# Finite Element Methods

Finite element methods discretize weak forms directly by choosing discrete subspaces for test and trial functions.  So an equation: find $u \in V$ such that
$$ \int_{\Omega} \nabla v \cdot \kappa \cdot \nabla u = 0, \quad \text{for all } v \in V $$
becomes: find $u_h \in V_h$ such that
$$ \int_{\Omega} \nabla v_h \cdot \kappa \cdot \nabla u_h = 0, \quad \text{for all } v_h \in V_h . $$
The integral is usually computed using quadrature which may or may not be exact.

To implement finite element methods, we need to define the discrete spaces $V_h$.  Usually these will be piecewise polynomial with some continuity between elements.  We'll start by working on a single element $(-1, 1)$ and will use a monomial basis for the moment, then provide a formal definition of finite elements and a more general construction.

## Quadrature

Finite element methods require numerical integration of weak forms.  This is usually done by **quadrature**

$$ \int_{-1}^1 f(x) \approx \sum_{i=1}^n w_i f(q_i) . $$
There are many ways to choose the points $q_i$ and weights $w_i$.  For example, the trapezoid rule is a quadrature.

```python
%matplotlib inline
import numpy
from matplotlib import pyplot
pyplot.style.use('ggplot')

def quad_trapezoid(n):
    q = numpy.linspace(-1, 1, n)
    w = 0*q + 2/(n-1)
    w[[0,-1]] /= 2
    return q, w

def plot_quad_accuracy(fs, quad):
    ns = numpy.logspace(.5, 2, 10, dtype=int)
    for f, F in fs:
        exact = F(1) - F(-1)
        def err(n):
            q, w = quad(n)
            return w.dot(f(q)) - exact
        errors = [numpy.abs(err(n)) for n in ns]
        pyplot.loglog(ns, errors, 'o', label=f.__name__)
    pyplot.loglog(ns, ns**(-2.), label='$n^{{-2}}$')
    pyplot.legend()

def poly(p):
    def f(x):
        return numpy.polyval(p, x)
    pint = numpy.polyint(p)
    def F(x):
        return numpy.polyval(pint, x)
    return f, F

plot_quad_accuracy([(numpy.exp, numpy.exp),
                    poly([1,2,3])],
                   quad_trapezoid)
```

### Gauss quadrature

Methods like the trapezoid rule are inefficient for integrating polynomials and related functions, in the sense that they need many points to reach desired accuracy tolerances.  Gauss quadrature is a spectrally accurate method that can exactly integrate polynomials of degree $2n-1$ using $n$ points.  To derive Gauss quadrature, we will need the Legendre polynomials $P_n(x)$, first discovered in the early 19th century as eigenfunctions (resonant modes) of the differential operator
$$ \frac{d}{d x} (1 - x^2) \frac{d P_n(x)}{dx} $$
on the interval $(-1, 1)$.
Legendre polynomials can also be derived by applying Gram-Schmidt orthogonalization to the monomials using the inner product
$$ \langle P_m, P_n \rangle = \int_{-1}^1 P_m(x) P_n(x) $$
and a normalization convention that $P_n(1) = 1$.
In practice, we will use the recursive definition
$$\begin{split}
P_0(x) &= 1 \\
P_1(x) &= x \\
(n+1) P_{n+1}(x) &= (2n+1) x P_n(x) - n P_{n-1}(x)
\end{split}$$

```python
def vander_legendre(x, n=None):
    if n is None:
        n = len(x)
    P = numpy.ones((len(x), n))
    if n > 1:
        P[:,1] = x
    for k in range(1,n-1):
        P[:,k+1] = ((2*k+1) * x * P[:,k] - k * P[:,k-1]) / (k + 1)
    return P

x = numpy.linspace(-1, 1)
P = vander_legendre(x, 6)
pyplot.figure()
pyplot.plot(x, P)
pyplot.title('Legendre Polynomials');
```

Since the polynomials $P_n$ are orthogonal to $P_0(x) = 1$ in particular,
$$ \int_{-1}^1 P_n(x) = 0, \quad n=1,2,\dotsc $$
which means that we can integrate an arbitrary function by writing the Lagrange interpolating polynomial in the Legendre basis
$$ p_n(x) = \sum_{i=0}^n c_i P_i(x) $$
and integrating
$$ \int_{-1}^1 p_n(x) = 2 c_0 . $$

```python
def quad_lin_legendre(n):
    q = numpy.linspace(-1, 1, n)
    P = vander_legendre(q)
    w = 2*numpy.linalg.solve(P.T, numpy.eye(n, 1)).flatten()
    return q, w

plot_quad_accuracy([(numpy.exp, numpy.exp),
                    poly([1,2,3,4,5])],
                   quad_lin_legendre)
```

This is quite accurate when using just a few points, but becomes unstable due to the [Runge phenomenon](https://en.wikipedia.org/wiki/Runge%27s_phenomenon).  We could control this by choosing `cosspace` points like we did for Chebyshev methods, but it is possible to do much better.  Suppose we write a polynomial on the interval $(-1,1)$ as

$$ p_{2n-1}(x) = P_n(x) q(x) + r(x) $$

where $P_n(x)$ is the $n$th Legendre polnomials and both $q(x)$ and $r(x)$ are polynomials of maximum degree $n-1$.

* Can every polynomials of degree $2n-1$ be written in the above form?
* How many roots does $P_n(x)$ have on the interval?
* Can we choose points $\{x_i\}$ such that the first term is 0?

If $P_n(x_i) = 0$ for each $x_i$, then we need only integrate $r(x)$, which is done exactly by integrating its interpolating polynomial.  How do we find these roots $x_i$?

#### Derivatives of Legendre Polynomials

The derivatives of Legendre polynomials satisfy a recurrence,
$$ P_{n+1}'(x) = (2n+1) P_n(x) + P_{n-1}'(x) . $$

With the ability to compute $P_n(x)$ and $P_n'(x)$, we can use Newton's method to compute the roots.
It turns out that `cos(linspace(.5/n, 1-.5/n, n) * pi)` is a good initial guess.

```python
def vander_legendre_deriv(x, n=None):
    if n is None:
        n = len(x)
    P = numpy.ones((len(x), n))
    dP = numpy.zeros_like(P)
    if n > 1:
        P[:,1] = x
        dP[:,1] = 1
    for k in range(1,n-1):
        P[:,k+1] = ((2*k+1) * x * P[:,k] - k * P[:,k-1]) / (k + 1)
        dP[:,k+1] = (2*k+1) * P[:,k] + dP[:,k-1]
    return P, dP

def quad_gauss_legendre(n, verbose=False):
    q = numpy.cos(numpy.linspace(-1+.5/n, -.5/n, n) * numpy.pi)
    for i in range(20):
        P, dP = vander_legendre_deriv(q, n+1)
        Pn = P[:,-1]
        dPn = dP[:,-1]
        if verbose:
            print('[{:d}] {:10e} {}'.format(i, numpy.linalg.norm(Pn), q))
        if numpy.linalg.norm(Pn) < 1e-12:
            break
        q -= Pn / dPn
    w1 = 2*numpy.linalg.solve(P[:,:-1].T, numpy.eye(n, 1)).flatten()
    w2 = 2 / ((1-q**2)*dPn**2) # There is a closed form solution
    if verbose:
        print('|w1 - w2| = {:10e}'.format(numpy.linalg.norm(w1 - w2)))
    return q, w2

quad_gauss_legendre(3, verbose=True)
plot_quad_accuracy([(numpy.exp, numpy.exp),
                    poly([1,2,3,4,5])],
                   quad_gauss_legendre)
```

## Interpolation nodes

We will define the finite element solution $u_h \in V_h$ by the Lagrange interpolating polynomials on some set of nodes $x_i$.

```python
%run fdtools.py

def cosspace(a, b, n=50):
    return (a + b)/2 + (b - a)/2 * (numpy.cos(numpy.linspace(-numpy.pi, 0, n)))

x = numpy.linspace(-1, 1, 10)
x = cosspace(-1, 1, 4)

def febasis(x, q):
    B = numpy.zeros((len(q), len(x)))
    D = numpy.empty_like(B)
    for i, qi in enumerate(q):
        c = fdstencil(qi, x, nderiv=1)
        B[i] = c[0]
        D[i] = c[1]
    return B, D

xx = numpy.linspace(-1, 1)
B, D = febasis(x, xx)
pyplot.plot(xx, B)
pyplot.plot(x, 0*x, 'ok')
pyplot.plot(x, 0*x+1, 'ok')
pyplot.title('Basis');
pyplot.figure()
pyplot.plot(xx, D)
pyplot.plot(x, 0*x, 'ok')
pyplot.title('Derivatives')
print('cond {:10e}'.format(numpy.linalg.cond(B)))
```

## Galerkin method

### $L^2$ projection

$L^2$ projection refers to the problem: find $u \in V_h$ such that
$$ \int_{-1}^1 v(x) \big[ u(x) - f(x) \big] = 0, \quad \text{for all } v \in V_h$$
and is an excellent first test problem because it does not require derivatives.  This projection should be exact if $f(x) \in V_h$.

```python
def L2_galerkin(n, f):
    x = cosspace(-1, 1, n)
    q, w = quad_gauss_legendre(n)
    B, D = febasis(x, q)
    rhs = B.T @ (w * f(q))
    A = B.T * w @ B
    u = numpy.linalg.solve(A, rhs)
    return x, u

def plot_galerkin(x, u, f):
    q = numpy.linspace(-1, 1)
    B, _ = febasis(x, q)
    pyplot.plot(x, u, 'ok')
    pyplot.plot(q, B@u, '-k')
    pyplot.plot(q, f(q))
    pyplot.title('Galerkin error {:8e}'.format(numpy.linalg.norm(B@u - f(q), numpy.inf)))

def tanh5(x):
    return numpy.tanh(5*x)
x, u = L2_galerkin(10, tanh5)
plot_galerkin(x, u, tanh5)
```

```python
def L2_galerkin_convergence(f):
    ns = numpy.logspace(.5, 2, 10, dtype=int)
    def error(n):
        x, u = L2_galerkin(n, f)
        return numpy.linalg.norm(u - f(x), numpy.inf)
    errors = [error(n) for n in ns]
    pyplot.semilogy(ns, errors, 'o')
    pyplot.xlabel('n')
    pyplot.ylabel('error')

L2_galerkin_convergence(tanh5)
```

```python
def laplace_galerkin(n, f):
    x = cosspace(-1, 1, n)
    q, w = quad_gauss_legendre(n)
    B, D = febasis(x, q)
    rhs = B.T @ (w * f(q))
    A = D.T * w @ D
    A[0] = 0
    A[0,0] = 1
    rhs[0] = 0
    A[-1] = 0
    A[-1,-1] = 1
    rhs[-1] = 0
    u = numpy.linalg.solve(A, rhs)
    return x, u

x, u = laplace_galerkin(5, lambda x: numpy.cos(x*numpy.pi/2))
plot_galerkin(x, u, lambda x: numpy.cos(x*numpy.pi/2)*(2/numpy.pi)**2)
```

```python
x, u = laplace_galerkin(3, lambda x:0*x+1)
plot_galerkin(x, u, lambda x:.5*(1 - x**2))
```

## Error estimates

We introduce the notation
$$ a(v, u) = \int_\Omega \nabla v(x) \cdot \nabla u(x) $$
and note that $a$ is
* bilinear (linear in each of its arguments)
* symmetric: $a(u, v) = a(v,u)$
* positive definite: $a(u, u) > 0$ when $u \ne 0$
thus defines an inner product on the function space $V$.
We also introduce the $L^2$ inner product
$$ \langle u, v \rangle = \int_\Omega u(x) v(x) $$
so that our continuous weak form is to find $u \in V$ such that
$$ a(v, u) = \langle v, f \rangle, \quad \forall v\in V. $$
Our Galerkin discretization is to find $u_h \in V_h \subset V$ such that
$$ a(v_h, u_h) = \langle v_h, f \rangle, \quad \forall v_h \in V_h . $$
Since $V_h \subset V$, we can subtract these two, yielding
$$ a(v_h, u_h - u) = 0, \quad \forall v_h \in V_h .$$
This says that the error in the discrete solution $u_h - u$ is $a$-orthogonal to all test functions $v_h$.

We can also define the "energy norm" or $a$-norm,
$$ \lVert u \rVert_a = \sqrt{a(u,u)} . $$
This norm satisfies the Cauchy-Schwarz inequality,
$$ \lvert a(u,v) \rvert \le \lVert u \rVert_a \lVert v \rVert_a . $$
Now,
\begin{align}
\lVert u_h - u \rVert_a^2 &= a(u_h - u, u_h - u) \\
&= a(u_h - v_h, u_h - u) + a(v_h - u, u_h - u) \\
&= a(v_h - u, u_h - u) \\
&\le \lVert v_h - u \rVert_a \lVert u_h - u \rVert_a .
\end{align}
In other words,
$$\lVert u_h - u \rVert_a \le \lVert v_h - u \rVert_a, \quad \forall v_h \in V_h .$$
So the solution $u_h$ computed by the Galerkin discretization is optimal over the subspace $V_h$ as measured in the $a$-norm.

#### Observations
* The Galerkin method computes the exact solution any time it resides in the subspace $V_h$.
* The Galerkin method is automatically symmetric any time the weak form is symmetric.
* The Galerkin method can be spectrally accurate, similar to the Chebyshev finite difference methods.
* For a nonlinear problem, discretization and differentiation will commute.

## Finite elements

### Ciarlet definition

Ciarlet (1978) defines a **finite element** as a triple $(K, P, N)$ where
* $K$ is a bounded subset of $R^n$.
* $P = \mathrm{span} \{p_i(x)\}$ is a finite dimensional space of functions on $\bar K$.  This is called the "prime basis" and is often a collection of polynomials.  It should be easy to evaluate and differentiate at arbitrary points.
* $N = \{ n_i \}$ is a set of nodes such that the square matrix $V_{ij} = n_i(p_j)$ is nonsingular.

Formally, nodes are elements of the "dual space" of $P$, i.e., linear functionals on $P$.  In practice, nodes are very often just point evaluation $n_i(p_j) = p_j(x_i)$.  Nodes are used to specify continuity between elements.  Other examples of nodes:

* $n_i(p_j) = p_j'(x_i)$: used in Hermite elements to ensure continuity of higher derivatives
* $n_i(p_j) = |\Gamma_i|^{-1} \int_{\Gamma_i} p_j$: used to ensure continuity of average values
* $n_i(p_j) = |\Gamma_i|^{-1} \int_{\Gamma_i} p_j \cdot \hat n$: used to ensure continuity of average values

A **nodal basis** $\{ \phi_j(x) \}$ is one that satisfies
$$ n_i(\phi_j) = \delta_{ij} . $$
We write $\phi_j$ in the prime basis by solving with the generalized Vandermonde matrix $V_{ij} = n_i(p_j)$,
$$ \phi_j(x) = \sum_k (V^{-1})_{j,k} p_k(x) . $$

#### Example: 1D Lagrange
* $K = (-1, 1)$
* $P = \{ 1, x, x^2, \dotsc \}$
* $N$ is the set of $|P|$ Gauss-Lobatto or Chebyshev-Lobatto points.

This produces the basis we constructed above.

#### Example: 2D Lagrange basis on triangles

* $K = \{ (x,y) : x > -1, y > -1, x+y < 0 \}$
* $P = \{ 1, x, y, x^2, xy, y^2, \dotsc \}$
* $N$ is pointwise evaluation at vertices plus points along the edges.

## Isoparametric mapping

We will represent the geometry of an element using a finite element basis.  The basis used for geometry need not be the same as the basis used for solution variables, but it must be possible to evaluate to the same quadrature points.  Given the reference coordinates $X \in K \subset R^n$ and physical coordinates $x(X)$ on the physical element $x(K)$, an integral on the physical element can be written
$$ \int_{x(K)} f(x) dx = \int_K \underbrace{\left\lvert \frac{\partial x}{\partial X} \right\rvert}_{\text{determinant}} f(x(X)) dX .$$

## General formulation

Given a weak form: find $u$ such that
$$ \int_\Omega v\cdot f_0(u, \nabla u) + \nabla v\cdot f_1(u, \nabla u) = 0, \quad \forall v$$
we discretize as
$$ \sum_e \mathcal E_e^T \Big( B^T W \left\lvert \frac{\partial x}{\partial X} \right\rvert f_0(\tilde u, \nabla \tilde u) + D^T \left(\frac{\partial X}{\partial x}\right)^{T} W \left\lvert \frac{\partial x}{\partial X} \right\rvert f_1(\tilde u, \nabla\tilde u) \Big) = 0 $$
where $\tilde u = B \mathcal E_e u$ and $\nabla \tilde u = \frac{\partial X}{\partial x} D \mathcal E_e u$ are the values and gradients evaluated at quadrature points.

| Notation | Meaning |
|---------|:-------------|
| $x$ | physical coordinates |
| $X$ | reference coordinates |
| $\mathcal E_e$ | restriction from global vector to element $e$ |
| $B$ | values of nodal basis functions at quadrature ponits on reference element |
| $D$ | gradients of nodal basis functions at quadrature points on reference element|
| $W$ | diagonal matrix of quadrature weights on reference element |
| $\frac{\partial x}{\partial X} = D \mathcal E_e x $ | gradient of physical coordinates with respect to reference coordinates |
| $\left\lvert \frac{\partial x}{\partial X}\right\rvert$ | determinant of coordinate transformation at each quadrature point |
| $\frac{\partial X}{\partial x} = \left(\frac{\partial x}{\partial X}\right)^{-1}$ | derivative of reference coordinates with respect to physical coordinates |

```python
class fe1:
    def __init__(self, p):
        self.p = p
        self.xref = cosspace(-1, 1, p+1)
        self.q, self.w = quad_gauss_legendre(p+1)
        self.B, self.D = febasis(self.xref, self.q)

def fe1_mesh(fe, nelem):
    "Create a mesh with nelem elements of type fe"
    Erestrict = (numpy.arange(0, nelem*fe.p, fe.p)[:,None]
                 + numpy.arange(fe.p+1))
    x1 = numpy.linspace(-1, 1, nelem+1)
    x = numpy.empty(fe.p * nelem + 1)
    for e, E in enumerate(Erestrict):
        x[E] = (x1[e+1] + x1[e])/2 + fe.xref * (x1[e+1] - x1[e])/2
    return x, Erestrict

x, Erestrict = fe1_mesh(fe1(2), 4)
print(x)
print(Erestrict)
print(x[Erestrict])
```

    [-1.   -0.75 -0.5  -0.25  0.    0.25  0.5   0.75  1.  ]
    [[0 1 2]
     [2 3 4]
     [4 5 6]
     [6 7 8]]
    [[-1.   -0.75 -0.5 ]
     [-0.5  -0.25  0.  ]
     [ 0.    0.25  0.5 ]
     [ 0.5   0.75  1.  ]]

```python
def fsolve_newton(F, J, u0, rtol=1e-10, maxit=50, verbose=False):
    u = u0.copy()
    Fu = F(u)
    norm0 = numpy.linalg.norm(Fu)
    for i in range(maxit):
        du = sp.linalg.spsolve(J(u), -Fu)
        u += du
        Fu = F(u)
        norm = numpy.linalg.norm(Fu)
        if verbose:
            print('Newton {:d} anorm {:6.2e} rnorm {:6.2e}'.
                  format(i+1, norm, norm/norm0))
        if norm < rtol * norm0:
            break
    return u, i
```

```python
import scipy.sparse as sp
import scipy.sparse.linalg
from scipy.optimize import fsolve

class projection:
    def __init__(self, k):
        self.k = k
    def form(self, x, u, Du):
        return (u - self.exact(x), 0), ((1, 0), (0, 0))
    def exact(self, x):
        return numpy.tanh(self.k * x)

class laplacian:
    def __init__(self, k):
        self.k = k
    def form(self, x, u, Du):
        k = self.k
        manufactured = 2*k**2 * (numpy.tanh(k*x)**2 - 1) * numpy.tanh(k*x)
        return (manufactured, Du), ((0, 0), (0, 1))
    def exact(self, x):
        return numpy.tanh(self.k * x)

def fe1_solve(fe, nelem, form, dirichlet):
    x, Erestrict = fe1_mesh(fe, nelem)
    q, w = quad_gauss_legendre(fe.p+1)
    B, D = febasis(fe.xref, q)
    W = numpy.empty((nelem, len(w)))
    dXdx = numpy.empty((nelem, len(w)))
    xq = numpy.empty((nelem, len(w)))
    for e, E in enumerate(Erestrict):
        xq[e] = B @ x[E]
        dxdX = D @ x[E]
        W[e] = w * dxdX # Quadrature weight on physical element
        dXdx[e] = 1/dxdX
    bcmask = numpy.zeros(nelem * fe.p + 1, dtype=bool)
    for indices, _ in dirichlet:
        bcmask[indices] = True

    def residual(u):
        ubc = u.copy()
        for indices, func in dirichlet:
            ubc[indices] = func(x[indices])
        v = u - ubc
        for e, E in enumerate(Erestrict):
            ue = ubc[E]
            f, _ = form(xq[e], B @ ue, dXdx[e] * (D @ ue))
            vE = B.T @ (W[e] * f[0]) + D.T @ (dXdx[e] * W[e] * f[1])
            vE[bcmask[E]] = 0
            v[E] += vE
        return v

    def jacobian(u):
        ai = []
        aj = []
        aa = []
        for e, E in enumerate(Erestrict):
            ue = u[E]
            _, df = form(xq[e], B @ ue, dXdx[e] * (D @ ue))
            Ae = (B.T * W[e] * df[0][0] @ B
                  + B.T * W[e] * df[0][1] * dXdx[e] @ D
                  + D.T * dXdx[e] * W[e] * df[1][0] @ B
                  + D.T * dXdx[e] * W[e] * df[1][1] * dXdx[e] @ D)
            Ae[bcmask[E],:] = 0
            Ae[:,bcmask[E]] = 0
            ai += numpy.outer(E, numpy.ones_like(E)).flatten().tolist()
            aj += numpy.outer(numpy.ones_like(E), E).flatten().tolist()
            aa += Ae.flatten().tolist()
        N = len(u)
        for indices, _ in dirichlet:
            ipos = [i % N for i in indices]
            ai += ipos
            aj += ipos
            aa += numpy.ones_like(indices).tolist()
        A = sp.csr_matrix((aa, (ai, aj)), shape=(N,N))
        return A

    u0 = numpy.zeros(nelem * fe.p + 1) # initial guess
    u, nit = fsolve_newton(residual, jacobian, u0, verbose=True)
    # Return solution evaluated at quadrature points
    return x, u, xq.flatten(), (u[Erestrict] @ B.T).flatten()

def fe1_plot(x, u, xq, uq, exact=None):
    pyplot.plot(xq, uq, '.')
    pyplot.plot(x, u, 'o')
    if exact is not None:
        uexact = exact(xq)
        pyplot.plot(xq, uexact)
        error = numpy.linalg.norm(uq - uexact, numpy.inf)
        pyplot.title('Error {:8e}'.format(error));

fe = fe1(5)
prob = projection(3)
x, u, xq, uq = fe1_solve(fe, 4, prob.form, ())
fe1_plot(x, u, xq, uq, prob.exact)

pyplot.figure()
prob = laplacian(3)
x, u, xq, uq = fe1_solve(fe, 4, prob.form, [([0,-1], prob.exact)])
fe1_plot(x, u, xq, uq, prob.exact)
```

```python
class vlaplacian:
    def __init__(self, k, eps = 1e-2):
        self.k = k
        self.eps = eps
    def form(self, x, u, Du):
        k = self.k
        forcing = 1
        diffusivity = numpy.where(numpy.abs(x) < .5, self.eps, 1)
        return (-forcing, diffusivity*Du), ((0, 0), (0, diffusivity))

prob = vlaplacian(3, 1e-3)
x, u, xq, uq = fe1_solve(fe, 4, prob.form, [([0,-1], numpy.zeros_like)])
fe1_plot(x, u, xq, uq)
```

```python
class plaplacian:
    def __init__(self, p, eps = 1e-2):
        self.p = p
        self.eps = eps
    def form(self, x, u, Du):
        p = self.p
        gamma = (.5*self.eps**2 + .5*Du*Du)
        dgamma = Du
        diffusivity = gamma**((p-2)/2)
        ddiffusivity = (p-2)/2 * gamma**((p-2)/2-1) * dgamma
        forcing = 1
        return (-forcing, diffusivity*Du), ((0, 0), (0, diffusivity + ddiffusivity*Du))

prob = plaplacian(3)
x, u, xq, uq = fe1_solve(fe, 4, prob.form, [([0,-1], numpy.zeros_like)])
fe1_plot(x, u, xq, uq)
```

## 2D

The main change will be in construction of a finite element basis.  We need to define a reference element and a quadrature (often called "cubature" in multiple dimensions).  This can be done by Kronecker product for quadrilaterals and hexahedra where we are interested in products of polynomials $p_n(x) q_n(y)$.  For multivariate polynomials of maximal degree (typically used for finite element methods on triangles and tetrahedra), the techniques are far more specialized.  A good modern source for computing such quadratures is [Witherden and Vincent (2015)](https://doi.org/10.1016/j.camwa.2015.03.017) and the associated [Polyquad](https://github.com/vincentlab/polyquad) software.

```python
def tri_quad4():
    q = numpy.array([[ -0.10810301816807,   -0.78379396366386  ],
                     [ -0.78379396366386,   -0.10810301816807  ],
                     [ -0.10810301816807,   -0.10810301816807  ],
                     [-0.816847572980458,   0.633695145960917  ],
                     [ 0.633695145960917,  -0.816847572980458  ],
                     [-0.816847572980458,  -0.816847572980458  ]])
    w = numpy.array([ 0.446763179356023,
                      0.446763179356023,
                      0.446763179356023,
                      0.219903487310644,
                      0.219903487310644,
                      0.219903487310644])
    return q, w

q, w = tri_quad4()
pyplot.plot(q[:3,0], q[:3,1], 'o', label='weight={:6f}'.format(w[0]))
pyplot.plot(q[3:,0], q[3:,1], 's', label='weight={:6f}'.format(w[3]))
pyplot.triplot([-1, -1, 1], [1, -1, -1])
pyplot.legend();
```

```python
class fe2tri:
    def __init__(self, p):
        x1 = numpy.array([[-1, 1], [-1, -1], [1, -1]])
        x2 = numpy.array([[-1, 0], [0, -1], [0, 0]])
        if p == 1:
            x = x1
        elif p == 2:
            x = numpy.vstack([x1, x2])
        self.p = p
        self.xref = x
        self.q, self.w = tri_quad4() # Could use fewer points for p==1
        V, _ = self.prime(x)
        Vinv = numpy.linalg.inv(V)
        Bprime, Dprime = self.prime(q)
        self.B = Bprime @ Vinv
        self.D = Dprime @ Vinv

    def prime(self, x):
        V = numpy.ones((len(x), len(self.xref)))
        dV = numpy.zeros((len(x), 2, len(self.xref)))
        V[:,1] = x[:,0]
        V[:,2] = x[:,1]
        # dV[:,2*i] is derivative in x direction, dV[:,2*i+1] is in y-direction
        dV[:,0,1] = 1
        dV[:,1,2] = 1
        if self.p > 1:
            V[:,3] = x[:,0]**2
            V[:,4] = x[:,0]*x[:,1]
            V[:,5] = x[:,1]**2
            dV[:,0,3] = 2*x[:,0]
            dV[:,0,4] = x[:,1]
            dV[:,1,4] = x[:,0]
            dV[:,1,5] = 2*x[:,1]
        return V, dV

    def meshref(self):
        # Mesh for plotting on reference element
        x1 = numpy.linspace(-1, 1)
        xx, yy = numpy.meshgrid(x1, x1)
        for i,y in enumerate(yy):
            xx[i] = numpy.linspace(-1, -y[0])
        return numpy.vstack([xx.flatten(), yy.flatten()]).T

    def plot(self):
        pyplot.plot(self.xref[:,0], self.xref[:,1], 'o')
        pyplot.plot(self.q[:,0], self.q[:,1], 's')
        pyplot.triplot([-1, -1, 1], [1, -1, -1])

        X = self.meshref()
        Vinv = numpy.linalg.inv(self.prime(self.xref)[0])
        Bprime = self.prime(X)[0]
        B = Bprime @ Vinv
        pyplot.figure()
        for i in range(6):
            from matplotlib import cm
            pyplot.subplot(2, 3, i+1)
            pyplot.tricontourf(X[:,0], X[:,1], B[:,i], 30, cmap=cm.seismic, vmin=-1, vmax=1)

fe2tri(2).plot()
```

```python
class Mesh:
    def __init__(self, lcar=.5, shape='circle', reshape_boundary=False):
        import pygmsh
        geom = pygmsh.built_in.Geometry()
        if shape == 'circle':
            geom.add_circle((0,0,0), 1, lcar)
        elif shape == 'rectangle':
            geom.add_rectangle(-1, 1, -.5, .5, 0, lcar)
        elif shape == 'eyes':
            holes = [geom.add_circle((c,0,0), .25, .25*lcar, make_surface=False)
                 for c in (-.5, .5)]
            geom.add_circle((0,0,0), 1, lcar, holes=holes)
        else:
            raise RuntimeError('Shape not recognized:', shape)
        points, elements, _, _, _ = pygmsh.generate_mesh(geom, verbose=False, dim=2)
        vtx = points[:,:2]
        tri = elements['triangle']
        # Gmsh doesn't guarantee consistent orientation so fix up any inverted elements
        orient = numpy.cross(vtx[tri[:,1]] - vtx[tri[:,0]],
                             vtx[tri[:,2]] - vtx[tri[:,1]]) < 0
        tri[orient] = tri[orient][:,[0,2,1]]
        # Create edges
        edges = tri[:,[0,1,1,2,2,0]].reshape((-1,2))
        edges.sort(axis=1)
        ind = numpy.lexsort((edges[:,1], edges[:,0]))
        edge2vertex, starts, perm, counts = numpy.unique(edges[ind], axis=0,
                        return_index=True, return_inverse=True, return_counts=True)
        cell2edge = numpy.empty(len(edges), dtype=int)
        cell2edge[ind] = perm
        cell2edge = cell2edge.reshape((-1, 3))
        edgenumbers, edgecount = numpy.unique(cell2edge.flatten(), return_counts=True)
        edgecenter = .5*(vtx[edge2vertex[:,0]] + vtx[edge2vertex[:,1]])

        centroids = (vtx[tri[:,0]] + vtx[tri[:,1]] + vtx[tri[:,2]]) / 3
        h = numpy.min(numpy.linalg.norm(numpy.kron([1,1,1], centroids).reshape((-1,2))
                                        - edgecenter[cell2edge.flatten()], axis=1))

        # Classify boundaries
        bedges = edgenumbers[edgecount == 1]
        if shape == 'eyes':
            def distance(c, r):
                return numpy.abs(numpy.linalg.norm(edgecenter[bedges] - c, axis=1) - r)
            mouter = distance((0,0), 1)
            mleft = distance((-.5,0), .25)
            mright = distance((.5,0), .25)
            boundary = dict(outer=bedges[mouter <= numpy.minimum(mleft, mright)],
                           left=bedges[mleft <= numpy.minimum(mouter, mright)],
                           right=bedges[mright <= numpy.minimum(mleft, mouter)])
        else:
            boundary = dict(outer=bedges)

        self.vtx = vtx
        self.tri = tri
        self.edge2vertex = edge2vertex
        self.cell2edge = cell2edge
        self.edgecenter = edgecenter
        self.boundary = boundary
        self.shape = shape
        self.nvtx = len(vtx)
        self.nface = len(edge2vertex)
        self.h = h
        if reshape_boundary:
            self.reshape_boundary()

    def reshape_boundary(self):
        def project_to_circle(label, c, r):
            edges = self.boundary[label]
            x = self.edgecenter[edges]
            self.edgecenter[edges] = c + r*(x-c) / numpy.linalg.norm(x-c, axis=1)[:,None]
        if self.shape == 'circle':
            project_to_circle('outer', (0,0), 1)
        elif self.shape == 'eyes':
            project_to_circle('outer', (0,0), 1)
            project_to_circle('left', (-.5,0), .25)
            project_to_circle('right', (.5,0), .25)

    def tri2(self):
        _, Erestrict = self.Erestrict(2)
        return Erestrict[:,[0,3,5, 1,4,3, 2,5,4, 3,4,5]].reshape(-1,3)

    def Erestrict(self, p):
        if p == 1:
            return self.vtx, self.tri
        elif p == 2:
            x = numpy.vstack([self.vtx, self.edgecenter])
            Erestrict = numpy.hstack([self.tri, self.nvtx+self.cell2edge])
            return x, Erestrict
        raise RuntimeError('Not implemented for order', p)

    def Frestrict(self, p):
        if p == 1:
            return self.edge2vertex
        elif p == 2:
            return numpy.hstack([self.edge2vertex,
                                 self.nvtx + numpy.arange(self.nface)[:,None]])
        raise RuntimeError('Not implemented for order', p)

    def plotmesh(self):
        pyplot.triplot(self.vtx[:,0], self.vtx[:,1], triangles=self.tri)
        x, _ = self.Erestrict(2)
        Frestrict = self.Frestrict(2)
        for label, faces in self.boundary.items():
            xF = x[Frestrict[faces,2]]
            pyplot.plot(xF[:,0], xF[:,1], 's', label=label)
            xFv = x[Frestrict[faces,:2].flatten()]
            pyplot.plot(xFv[:,0], xFv[:,1], '.k')
        pyplot.legend()

mesh = Mesh(shape='circle', reshape_boundary=True)
mesh.plotmesh()
```

```python

```

```python

```

```python
class projection2:
    def __init__(self, k):
        self.k = k
    def form(self, x, u, Du):
        return (u - self.exact(x), 0*Du), ((1, 0), (0, 0))
    def exact(self, x):
        return self.k * x[:,0] * x[:,1]
        return numpy.tanh(self.k * x[:,0])

class laplacian2:
    def __init__(self, k=1, exact='1'):
        self.k = k
        self.exact = getattr(self, 'exact_' + exact)
        self.forcing = getattr(self, 'forcing_' + exact)
    def form(self, x, u, Du):
        return (self.forcing(x), self.k*Du), ((0, 0), (0, self.k*numpy.eye(2)[None,:,:]))
    def exact_1(self, x):
        return x[:,0] + x[:,1]
    def forcing_1(self, x):
        return 0
    def exact_2(self, x):
        return x[:,0]*x[:,1] + 2*x[:,0]**2 - x[:,1]**2 + x[:,1]
    def forcing_2(self, x):
        return 2
    def exact_warp(self, xx):
        from numpy import tanh, exp
        k, x, y = self.k, xx[:,0], xx[:,1]
        return tanh(k*x) * exp(-4*y**2)
    def forcing_warp(self, xx):
        from numpy import tanh, exp
        k, x, y = self.k, xx[:,0], xx[:,1]
        return 2*k**2*(tanh(k*x)**2 - 1)*exp(-4*y**2)*tanh(k*x) + 8*(8*y**2 - 1)*exp(-4*y**2)*tanh(k*x)

def fe2_geom(fe, mesh):
    x, Erestrict = mesh.Erestrict(fe.p)
    nelem = len(Erestrict)
    Q = len(fe.w)
    B, D = fe.B, fe.D
    W = numpy.empty((nelem, Q))
    dXdx = numpy.empty((nelem, Q, 2, 2))
    xq = numpy.empty((nelem, Q, 2))
    for e, E in enumerate(Erestrict):
        xE = x[E,:]
        xq[e] = B @ xE
        dxdX = D @ xE # 2x2 matrices at each quadrature point
        det = numpy.linalg.det(dxdX)
        W[e] = w * det # Quadrature weight on physical element
        dXdx[e] = numpy.linalg.inv(dxdX)
    return xq, W, dXdx

def fe2_solve(fe, mesh, form, dirichlet={}, spy=False):
    x, Erestrict = mesh.Erestrict(fe.p)
    Frestrict = mesh.Frestrict(fe.p)
    Ndof = len(x)
    B, D = fe.B, fe.D
    xq, W, dXdx = fe2_geom(fe, mesh)
    dirichletidx = []
    bcmask = numpy.zeros(Ndof, dtype=bool)
    for label, func in dirichlet.items():
        indices = Frestrict[mesh.boundary[label]].flatten()
        dirichletidx.append((label, indices, func))
        bcmask[indices] = True

    def project_dirichlet(u): # Affine projector into space satisfying Dirichlet BC
        ubc = u.copy()
        for label, indices, func in dirichletidx:
            ubc[indices] = func(x[indices])
        return ubc

    def residual(u):
        ubc = project_dirichlet(u)
        v = u - ubc
        for e, E in enumerate(Erestrict):
            uE = ubc[E]
            uq = B @ uE
            Dxuq = numpy.einsum('ixX,iX->ix', dXdx[e], D @ uE)
            f, _ = form(xq[e], uq, Dxuq)
            vE = B.T @ (W[e] * f[0]) + numpy.einsum('iXp,ixX,ix->p',
                                                    D, dXdx[e], W[e,:,None] * f[1])
            vE[bcmask[E]] = 0
            v[E] += vE
        return v

    def jacobian(u):
        ubc = project_dirichlet(u)
        ai = []
        aj = []
        aa = []
        for e, E in enumerate(Erestrict):
            uE = ubc[E]
            Dx = numpy.einsum('ixX,iXp->ixp', dXdx[e], D)
            _, df = form(xq[e], B @ uE, Dx @ uE)
            Ae = (numpy.einsum('qi,q,qj->ij', B, W[e] * df[0][0], B)
                  + numpy.einsum('qi,qy,qyj->ij', B, W[e,:,None] * df[0][1], Dx)
                  + numpy.einsum('qxi,qx,qj->ij', Dx, W[e,:,None] * df[1][0], B)
                  + numpy.einsum('qxi,qxy,qyj->ij', Dx, W[e,:,None,None] * df[1][1], Dx))
            Ae[bcmask[E],:] = 0
            Ae[:,bcmask[E]] = 0
            ai += numpy.outer(E, numpy.ones_like(E)).flatten().tolist()
            aj += numpy.outer(numpy.ones_like(E), E).flatten().tolist()
            aa += Ae.flatten().tolist()
        bcidx = numpy.where(bcmask)[0].tolist()
        ai += bcidx
        aj += bcidx
        aa += numpy.ones_like(bcidx).tolist()
        A = sp.csr_matrix((aa, (ai, aj)), shape=(Ndof,Ndof))
        if spy:
            pyplot.spy(A)
        return A

    u0 = numpy.zeros(Ndof) # initial guess
    u, nit = fsolve_newton(residual, jacobian, u0, verbose=True)
    return x, u

mesh = Mesh(.5, reshape_boundary=True)
fe = fe2tri(2)
for prob in [projection2(3), laplacian2(exact='warp')]:
    x, u = fe2_solve(fe, mesh, prob.form, dict(outer=prob.exact), spy=False)
    pyplot.figure()
    pyplot.tricontourf(x[:,0], x[:,1], u)
    error = numpy.linalg.norm(u - prob.exact(x))
    pyplot.title('{} error {:8.2e} h={:.2e}'.format(type(prob).__name__, error, mesh.h))
    pyplot.colorbar();
```

```python
def fe2_Lpnorm_error(fe, mesh, u, exact, p=2):
    x, Erestrict = mesh.Erestrict(fe.p)
    B = fe.B
    xq, W, dXdx = fe2_geom(fe, mesh)
    isum = 0
    for e,E in enumerate(Erestrict):
        uq = B @ u[E]
        isum += numpy.sum(W[e] * numpy.abs((uq - exact(xq[e])))**p)
    return isum**(1/p)

fe2_Lpnorm_error(fe, mesh, u, prob.exact)
```

    0.0052247940091271683

```python
fe = fe2tri(2)
prob = laplacian2(exact='warp')
meshes = [Mesh(lcar, shape='circle', reshape_boundary=True)
          for lcar in numpy.geomspace(.05, 1, 6)[::-1]]
def err(mesh):
    x, u = fe2_solve(fe, mesh, prob.form, dict(outer=prob.exact))
    enorm = fe2_Lpnorm_error(fe, mesh, u, prob.exact)
    enorm = numpy.linalg.norm(u - prob.exact(x), numpy.inf)
    return [mesh.h, enorm]
r = numpy.array([err(mesh) for mesh in meshes])
hs, es = r[:,0], r[:,1]
pyplot.loglog(hs, es, 'o')
for p in (1,2,3):
    pyplot.loglog(hs, hs**p, label='$h^{}$'.format(p))
pyplot.xlabel('$h$')
pyplot.ylabel('max error')
pyplot.legend();
```

## Dirichlet boundary conditions

We have implemented boundary conditions using the following methodology.  Note that the true solution space $V$ for the weak form does not include any degrees of freedom on the Dirichlet boundary.  The ansatz space
$$\bar V = V \times V_{\Gamma} $$
is the product of this interior space with the space of functions on the Dirichlet boundary.
Specifically, $\bar V$ includes all discrete functions on $\bar \Omega$ regardless of whether they satisfy the boundary condition.  Now define

* $R_0: \bar V \to V_0$ restricts to the subspace with 0 at Dirichlet boundary,
* $R_D: \bar V \to V_D$ projects into the affine subspace with inhomogeneous Dirichlet boundary conditions.

Now given an interior residual $\tilde F(u)$, we extend it to a residual including boundary conditions as
$$ F(u) = R_0 \tilde F(R_D u) + (u - R_D u) . $$
The left term has no dependence on $u|_\Gamma$ (because of $R_D u$) so those columns of the Jacobian are zero.
It produces no contribution on the boundary (because of $R_0$) so those rows of the Jacobian are zero.
The Jacobian of the second term is the identity on the Dirichlet boundary degrees of freedom.

# Homework 5: due 2017-12-15

We consider steady-state flow in a fully saturated porous medium.  We will assume constant porosity and viscosity for simplicity, leading to [Darcy's law](https://en.wikipedia.org/wiki/Darcy%27s_law) producing a flux
$$ q = -\kappa \nabla p $$
in terms of pressure $p$ and permeability $\kappa$.
Conservation of mass yields a Laplacian
$$ \nabla\cdot q = -\nabla\cdot \big( \kappa \nabla p \big) = 0 . $$
The velocity of the fluid is
$$ u = \frac q \phi $$
where $\phi \in (0,1]$ is porosity.

Our flow will take place on the 2D unit disc with two holes removed
$$ \left\{ x \in \mathbb R^2 : |x|<1, \big|x-(1/2,0)| > 1/4\big|, \big|x-(-1/2,0)| > 1/4\big|  \right\} . $$
The outer boundary is Neumann, corresponding to an impermeable boundary.
The left "hole" is our inejection with a relative pressure of 1 and the right is extraction with a relative pressure of 0.
We can solve for pressure as follows.

```python
mesh = Mesh(shape='eyes', reshape_boundary=True)
fe = fe2tri(2)
prob = laplacian2()
x, pressure = fe2_solve(fe, mesh, prob.form, dict(left=lambda x:0*x[:,0]+1, right=lambda x:0*x[:,0]))
pyplot.tricontourf(x[:,0], x[:,1], pressure, triangles=mesh.tri2())
pyplot.title('Pressure')
pyplot.colorbar();
```

The velocity field can be computed by taking the gradient of pressure and dividing by permeability.
Now suppose that the fluid being injected at the left hole is hot while the outflow is a colder heat sink.
Temperature will satisfy the advection-diffusion equation (with appropriate choice of units)
$$ \nabla\cdot \big( - \nabla T + u T \big) = 0 $$
with boundary conditions $T=1$ at the left (inflow) hole, $T=0$ at the right (outflow) hole, and Neumann conditions at the (insulated) outer boundary.
The homework is to solve for temperature using the velocity field defined by the pressure solution above.

* You will need to modify `fe2_solve` so that you can evaluate velocity at the quadrature points.
* You will need to write a `form` function that implements the weak form of the advection-diffusion equation for temperature.

With respect to your numerically computed solutions, think about the following questions.

1. What is the order of accuracy of $p$, $u$, and $T$ for linear `fe2tri(1)` and quadratic `fe2tri(2)` elements? Rather than interpolating to compare in a norm, you can compare by computing moments of the error, such as
$$ \int_\Omega x (u_{2h} - u_h)(x,y) = \int_\Omega x u_{2h}(x,y) - \int_\Omega x u_h(x,y), $$
where the two integrals on the right can be carried out on the natural quadrature of each mesh independently.

2. When porosity $\phi$ decreases (at fixed permeability), the velocity increases.  Can you obtain an accurate solution for $T$ for any value of $\phi$ or is there a relationship between $\phi$ and the grid size $h$?

```python
pyplot.triplot(x[:,0], x[:,1], triangles=mesh.tri2());
```

```python

```


# Function rectangle

## Synopse
Create a binary rectangle image.

- **g = rectangle(s, r, c) **

    - **g**: Image.
    - **s**: Image. [rows cols], output image dimensions.
    - **r**: Double. [rrows ccols], rectangle image dimensions.
    - **c**: Image. [row0 col0], center of the rectangle.

## Description
Creates a binary image with dimensions given by s, rectangle dimensions given by r and center given by c. The pixels inside the rectangle are one and outside zero.

```python
import numpy as np

def rectangle(s, r, c):

    rows,  cols  = s[0], s[1]
    rrows, rcols = r[0], r[1]
    rr0,   cc0   = c[0], c[1]
    rr, cc = np.meshgrid(np.arange(rows), np.arange(cols), indexing='ij')

    min_row, max_row = rr0-rrows//2, rr0+rrows//2
    min_col, max_col = cc0-rcols//2, cc0+rcols//2

    g1 = (min_row <= rr) & (max_row > rr)
    g2 = (min_col <= cc) & (max_col > cc)

    g = g1 & g2
    return g
```

## Examples

```python
testing = (__name__ == "__main__")
if testing:
    ! jupyter nbconvert --to python rectangle.ipynb
    import numpy as np
    import sys,os
    ia898path = os.path.abspath('../../')
    if ia898path not in sys.path:
        sys.path.append(ia898path)
    import ia898.src as ia

```

    [NbConvertApp] Converting notebook rectangle.ipynb to python
    [NbConvertApp] Writing 1808 bytes to rectangle.py

### Example 1

```python
if testing:
    F = ia.rectangle([7,9], [3,2], [3,4])
    print(F)
```

    [[False False False False False False False False False]
     [False False False False False False False False False]
     [False False False  True  True False False False False]
     [False False False  True  True False False False False]
     [False False False False False False False False False]
     [False False False False False False False False False]
     [False False False False False False False False False]]

- **Example 2**

```python
if testing:
    F = ia.rectangle([200,300], [90,120], [70,120])
    ia.adshow(ia.normalize(F))
```

## Equation

\begin{equation}
  g(x,y)=\begin{cases}
    1, & \text{if } x_\text{min} \leq x < x_\text{max} \text{ and } y_\text{min} \leq y < y_\text{max}.\\
    0, & \text{otherwise}.
  \end{cases}
\end{equation}

## Contributions

Lucas de Vasconcellos Teixeira, 1st semester 2017


# Partition Problem

Eric Ren

Notes from Skiena, 2008.

## Recurrence Relation
Let's start with the recurrence relation.  Here is the math:

$$S = [s_1, s_1,..., s_n], s \in \mathbb{R} $$

\begin{align}
P(n, k) & = \min_{i=1}^{n} \max( P(i, k-1), \sum_{j=i+1}^n s_j ) \tag{0} \\
P(1, k) & = s_1  \tag{1} \\
P(n, 1) & = \sum_{i=1}^{n} s_i  \tag{2}
\end{align}

Unfortunately Python does not support pattern matching. Else we can write very idiomatic code that will look just like the recurrence above.

Now on to the different approaches:

## Recursive

```python

```

## Recursive w/ Memoization

```python

```

## Imperative

```python
# To better match the math equations above, collection starts at index 1 instead of 0

def partition(collection, n, k):
    if n == 0:
        return "No elements in collection to partition"

    # initialize matrix
    m = [[float('inf')] * k for _ in range(n+1)]
    d = [[-1] * k for _ in range(n+1)]

    # create prefix sums
    prefix_sum = [0] * (n+1)
    for i in range(1, n+1):
        prefix_sum[i] = prefix_sum[i-1] + collection[i]

    # Base case from eq (1) above
    for i in range(1, n+1):
        m[1][i] = collection[1]

    # Base case from eq (2) above
    for i in range(1, n+1):
        m[i][1] = prefix_sum[i]

    # eq 0
    for i in range(2, n+1):
        for j in range(2, k+1):
            for x in range(1, i):
                cost = max(m[x][j-1].cost, prefix_sum[i]-prefix_sum[x])
                if m[i][j] > cost:
                    m[i][j] = cost
                    d[i][j] = x

    reconstruct_partition(collection, d, n, k)

def reconstruct_partition(collection, dividers, n, k):

def print_items(collection, start, end):

```

## Less imperative

```python
def partition()
```

## More "functional"
This probably is not efficient in python. But in languages that do optimize for tail recursion, this can result in the same performance as using loops.

## Not enough memory... Generators to the rescue!

Now lets say that we can't initialize all that memory in the


# MATH 497: Final Project

Remark:

Please upload your solutions for this project to Canvas with a file named "Final_Project_yourname.ipynb".

=================================================================================================================

## Problem 1 [20%]:

Consider the following linear system

\begin{equation}\label{matrix}
A\ast u =f,
\end{equation}
or equivalently $u=\arg\min \frac{1}{2} (A* v,v)_F-(f,v)_F$, where $(f,v)_F =\sum\limits_{i,j=1}^{n}f_{i,j}v_{i,j}$ is the Frobenius inner product.
Here $\ast$ represents a convolution with one channel, stride one and zero padding one. The convolution kernel $A$ is given by
$$
A=\begin{bmatrix} 0 & -1 & 0 \\ -1 & 4 & -1 \\ 0 & -1 & 0 \end{bmatrix},~~
$$
the solution $ u \in \mathbb{R}^{n\times n} $, and the RHS $ f\in \mathbb{R}^{n\times n}$ is given by $f_{i,j}=\dfrac{1}{(n+1)^2}.$

### Tasks:
Set $J=4$, $n=2^J-1$ and the number of iterations $M=100$. Use the gradient descent method and the multigrid method to solve the above problem with a random initial guess $u^0$. Let $u_{GD}$ and $u_{MG}$ denote the solutions obtained by gradient descent and multigrid respectively.

* [5%] Plot the surface of solution $u_{GD}$ and $u_{MG}$.

* [10%] Define error $e_{GD}^m = \|A * u^{m}_{GD}- f\|_F=\sqrt{\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} |(A * u^{m}_{GD}- f)_{i,j}}|^2 $ for $m=0,1,2,3,...,M$. Similarly, we define the multigrid error $e_{MG}^m$. Plot the errors $e_{GD}^m$ and $e_{MG}^m$ as a function of the iteration $m$ (your x-axis is $m$ and your y-axis is the error). Put both plots together in the same figure.

* [5%] Find the minimal $m_1$ for which $e^{m_1}_{GD} <10^{-5}$ and the minimal $m_2$ for which $e^{m_2}_{MG} <10^{-5}$, and report the computational time for each method. Note that $m_1$ or $m_2$ may be greater than $M=100$, in this case you will have to run more iterations.

### Remark:

Below are examples of using gradient descent and multigrid iterations for M-times
* #### For gradient descent method with $\eta=\frac{1}{8}$, you need to write a code:

    Given initial guess $u^0$
$$
\begin{align}
&\text{for    }  m =  1,2,...,M\\
&~~~~\text{for    }  i,j = 1: n\\
&~~~~~~~~u_{i,j}^{m} = u_{i,j}^{m-1}-\eta(f_{i,j}-(A\ast u^{m-1})_{i,j})\\
&~~~~\text{endfor}\\
&\text{endfor}
\end{align}
$$

* #### For multigrid method, we have provided the framework code in F02_MultigridandMgNet.ipynb:

    Given initial guess $u^0$
$$
\begin{align}
&\text{for    }  m =  1,2,...,M\\
&~~~~u^{m} = MG1(u^{m-1},f, J, \nu)\\
&\text{endfor}
\end{align}
$$

=================================================================================================================

## Problem 2 [50%]:

Use SGD with momentum and weight decay to train MgNet on the Cifar10 dataset. Use 120 epochs, set the initial learning rate to 0.1, momentum to 0.9, weight decay to 0.0005, and divide the learning rate by 10 every 30 epochs. (The code to do this has been provided.) Let $b_i$ denote the test accuracy of the model after $i$ epochs, and let $b^*$ = $\max_i(b_i)$ be the best test accuracy attained during training.

### Tasks:
   * [30%] Train MgNet with the following three sets of hyper-parameters (As a reminder, the hyper-parameters of MgNet are $\nu$, the number of iterations of each layer, $c_u$, the number of channels for $u$, and $c_f$, the number of channels for $f$.):

    (1) $\nu=$[1,1,1,1], $c_u=c_f=64$.

    (2) $\nu=$[2,2,2,2], $c_u=c_f=64$.

    (3) $\nu=$[2,2,2,2], $c_u=c_f=64$, try to improve the test accuracy by implementing MgNet with $S^{l,i}$, which means different iterations in the same layer do not share the same $S^{l}$.

   * For each numerical experiment above, print the results with the following format:

       "Epoch: i, Learning rate: lr$_i$, Training accuracy: $a_i$, Test accuracy: $b_i$"

        where $i=1,2,3,...$ means the $i$-th epoch,  $a_i$ and $b_i$ are the training accuracy and test accuracy computed at the end of $i$-th epoch, and lr$_i$ is the learning rate of $i$-th epoch.

   * [10%] For each numerical experiment above, plot the test accuracy against the epoch count, i.e. the x-axis is the number of epochs $i$ and y-axis is the test accuracy $b_i$. An example plot is shown in the next cell.

   * [10%] Calculate the number of parameters that each of the above models has. Discuss why the number of parameters is different (or the same) for each of the models.

```python
from IPython.display import Image
Image(filename='plot_sample_code.png')

```

```python
# You can calculate the number of parameters of my_model by:
model_size = sum(param.numel() for param in my_model.parameters())

```

=================================================================================================================

## Problem 3 [25 %]:

Try to improve the MgNet Accuracy by increasing the number of channels. (We use the same notation as in the previous problem.) Double the number of channels to $c_u=c_f=128$ and try different $\nu$ to maximize the test accuracy.

### Tasks:
   * [20%] Report $b^{*}$, $\nu$ and the number of parameters of your model for each of the experiments you run.
   * [5%] For the best experiment, plot the test accuracy against the epoch count, i.e. the x-axis is the number of epochs $i$ and y-axis is the test accuracy $b_i$. (Same as for the previous problem.)

```python
# You can calculate the number of parameters of my_model by:
model_size = sum(param.numel() for param in my_model.parameters())

```

=================================================================================================================

## Problem 4 [5%]:

Continue testing larger MgNet models (i.e. increase the number of channels) to maximize the test accuracy. (Again, we use the same notation as in problem 2.)

### Tasks:

+  [5%] Try different training strategies and MgNet architectures with the goal of achieving $b^*>$ 95%. Hint: you can tune the number of epochs, the learning rate schedule, $c_u$, $c_f$, $\nu$, try different $S^{l,i}$ in the same layer $l$, etc...

=================================================================================================================


```julia
using RowEchelon
using Latexify

rref([1 2 3; 4 5 6])

display(latexify(rref([1 2 3; 4 5 6])))

display(latexify(rref([1 2 3; 4 5 6])))
```

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & -1.0 \\
0.0 & 1.0 & 2.0 \\
\end{array}
\right]
\end{equation}

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & -1.0 \\
0.0 & 1.0 & 2.0 \\
\end{array}
\right]
\end{equation}

```julia
# The Four Subspaces for A
using RowEchelon
using Latexify

A = [1 3 5 0 7;
     0 0 0 1 2;
     1 3 5 1 9]

At = transpose(A)
display(A)
display(rref(A))

display("+++++++++++A Transpose+++++++++++")
display(At)
display(latexify(rref(At)))
```

    3×5 Matrix{Int64}:
     1  3  5  0  7
     0  0  0  1  2
     1  3  5  1  9

    3×5 Matrix{Float64}:
     1.0  3.0  5.0  0.0  7.0
     0.0  0.0  0.0  1.0  2.0
     0.0  0.0  0.0  0.0  0.0

    "+++++++++++A Transpose+++++++++++"

    5×3 transpose(::Matrix{Int64}) with eltype Int64:
     1  0  1
     3  0  3
     5  0  5
     0  1  1
     7  2  9

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & 1.0 \\
0.0 & 1.0 & 1.0000000000000002 \\
0.0 & 0.0 & 0.0 \\
0.0 & 0.0 & 0.0 \\
0.0 & 0.0 & 0.0 \\
\end{array}
\right]
\end{equation}

```julia
A = [1 2 3;
     1 2 3;
     2 5 8]
display(A)
display(latexify(rref(A)))
```

    3×3 Matrix{Int64}:
     1  2  3
     1  2  3
     2  5  8

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & -1.0 \\
0.0 & 1.0 & 2.0 \\
0.0 & 0.0 & 0.0 \\
\end{array}
\right]
\end{equation}

```julia
using RowEchelon
using Latexify

A = [1 2 3 1;
     1 1 2 1;
     1 2 3 1]
display(transpose(A))
transpose(A) * [-1,0,1]

```

    4×3 transpose(::Matrix{Int64}) with eltype Int64:
     1  1  1
     2  1  2
     3  2  3
     1  1  1

    4-element Vector{Int64}:
     0
     0
     0
     0

```julia
using
```


## 3.2 Decentralized Gradient Descent (Adaptation-with-combination)

### 3.2.1 Problem and ATC-DGD

Consider $n$ computing nodes collaborate to solve the problem:

$$\min_{x\in \mathbb{R}^d} \quad \frac{1}{n}\sum_{i=1}^n f_i(x)$$

where $f_i(x)$ is a local and private function held by node $i$. Each node $i$ can access its own variable $x$ or gradient $\nabla f_i(x)$, but it has to communicate to access information from other nodes.

If each $f_i(x)$ is assumed to be smooth, the ATC-DGD has the following recursions

$$\begin{align}
x_i^{(k+1)} = \sum_{j=1}^n w_{ij} \Big(x_j^{(k)} - \alpha \nabla f_j(x_j^{(k)}) \Big) \quad \mbox{(ATC-DGD)}
\end{align}$$

### 3.2.2 Decentralized gradient descent (Adapt-with-combination)

This section discusses another form of decentralized gradient descent:

$$\begin{align}
x_i^{(k+1)} = \sum_{j=1}^n w_{ij} x_j^{(k)} - \alpha \nabla f_i(x_i^{(k)}) \quad \mbox{(AWC-DGD)}
\end{align}$$

Comparing with ATC-DGD, AWC-DGD mixes the combination and local adaptation within a single step. It is also worth noting that the combinination is not on the gradient, which is different from ATC-DGD. This will affects AWC-DGD's convergence slightly.

### 3.2.3 Convergence properties

The convergence property of AWC-DGD is very similar to that of ATC-DGD. For $L$-smooth and $\mu$-strongly convex problems, if the step-size $\alpha$ is sufficiently small, the ATC-DGD algorithm will converge as follows.

\begin{align}
\limsup_{k\to \infty} \frac{1}{n}\sum_{i=1}^n \|x_i^{(k)} - x^\star \|^2 = O\Big( (1-\alpha \mu)^{k} + \frac{\alpha^2  b^2}{(1-\rho)^2}\Big) \hspace{1cm} \mbox{(AWC-DGD-Convergence)}
\end{align}

where $x^\star$ is the glboal solution to the optimization problem, $\rho = \max\{|\lambda_2(W), \lambda_n(W)|\}$ and $b^2 = \frac{1}{n}\sum_{i=1}^n \|\nabla f_i(x^\star)\|^2$ denotes the data heterogeneity between nodes. Quantity $1-\rho$ measures the connectivity of the network topology. It is observed that AWC-DGD cannot converge exactly to the solution $x^\star$, but to a neighborhood around it. The limiting error is on the order of $O(\frac{\alpha^2 b^2}{(1-\rho)^2})$. When step-size $\alpha$ is small, or the data heterogeneity $b^2$ is small, or the network is well-connected, i.e., $\rho \to 0$, the limiting error can be negligible.

<!-- It is also observed that the limiting bias of AWC-DGD $O(\frac{\alpha^2 b^2}{(1-\rho)^2})$ is slightly larger than the that of ATC-DGD $O(\frac{\alpha^2 \rho^2 b^2}{(1-\rho)^2})$ when the same step-size $\alpha$ is used. In addition, it is also proved in literature \[Refs\] that the step-size stability range  -->

### 3.2.4 An example: least-square problem

In this section, we will show a demo on how to solve a least-square problem with AWC-DGD using BlueFog. Suppose $n$ computing nodes collaborate to solve the following problem:

$$\min_x \quad \frac{1}{n}\sum_{i=1}^n \|A_i x - b_i\|^2$$

where $\{A_i, b_i\}$ are local data held in node $i$.

#### 3.2.4.1 Set up BlueFog

In the following code, you should be able to see the id of your CPUs. We use 8 CPUs to conduct the following experiment.

```python
import ipyparallel as ipp

rc = ipp.Client(profile="bluefog")
dview = rc[:]  # A DirectView of all engines
dview.block = True
rc.ids
```

    [0, 1, 2, 3, 4, 5, 6, 7]

```python
%%px
import numpy as np
import bluefog.torch as bf
import torch
from bluefog.common import topology_util
import networkx as nx
import matplotlib.pyplot as plt

%matplotlib inline

bf.init()
print(f"Hello, I am {bf.rank()} among {bf.size()} processes")
```

    [stdout:0] Hello, I am 4 among 8 processes
    [stdout:1] Hello, I am 2 among 8 processes
    [stdout:2] Hello, I am 5 among 8 processes
    [stdout:3] Hello, I am 1 among 8 processes
    [stdout:4] Hello, I am 3 among 8 processes
    [stdout:5] Hello, I am 7 among 8 processes
    [stdout:6] Hello, I am 6 among 8 processes
    [stdout:7] Hello, I am 0 among 8 processes

#### 3.2.4.2 Generate local data $A_i$ and $b_i$

```python
%%px

def generate_data(m, n):

    A = torch.randn(m, n).to(torch.double)
    x_o = torch.randn(n, 1).to(torch.double)
    ns = 0.1 * torch.randn(m, 1).to(torch.double)
    b = A.mm(x_o) + ns

    return A, b
```

#### 3.2.4.3 Distributed gradient descent method

```python
%%px

def distributed_grad_descent(A, b, maxite=5000, alpha=1e-1):

    x_opt = torch.zeros(n, 1, dtype=torch.double)

    for _ in range(maxite):
        # calculate local gradient
        grad_local = A.t().mm(A.mm(x_opt) - b)

        # global gradient
        grad = bf.allreduce(grad_local, name="gradient")

        # distributed gradient descent
        x_opt = x_opt - alpha * grad

    grad_local = A.t().mm(A.mm(x_opt) - b)
    grad = bf.allreduce(grad_local, name="gradient")  # global gradient

    # evaluate the convergence of distributed gradient descent
    # the norm of global gradient is expected to 0 (optimality condition)
    global_grad_norm = torch.norm(grad, p=2)
    if bf.rank() == 0:
        print(
            "[Distributed Grad Descent] Rank {}: global gradient norm: {}".format(
                bf.rank(), global_grad_norm
            )
        )

    return x_opt
```

In the following code we run distributed gradient descent to achieve the global solution $x^\star$ to the optimization problem. To validate whether $x^\star$ is optimal, it is enough to examine $\frac{1}{n}\sum_{i=1}^n \nabla f_i(x^\star) = 0$.

```python
%%px

m, n = 20, 5
A, b = generate_data(m, n)
x_opt = distributed_grad_descent(A, b, maxite=200, alpha=1e-2)
```

    [stdout:7] [Distributed Grad Descent] Rank 0: global gradient norm: 3.2361430059369585e-14

#### 3.2.4.3 Decentralized gradient descent method

In this section, we depict the convergence curve of the decentralied gradient descent (the AWC version). We will utilize the $x^\star$ achieved by distributed gradient descent as the optimal solution. First, we define one step of the AWC-DGD method.

```python
%%px

def AWC_DGD_one_step(x, x_opt, A, b, alpha=1e-2):

    # one-step ATC-DGD.
    # The combination weights have been determined by the associated combination matrix.

    grad_local = A.t().mm(A.mm(x) - b)  # compute local grad
    x_new = bf.neighbor_allreduce(x) - alpha * grad_local  # AWC update

    # the relative error: |x^k-x_gloval_average|/|x_gloval_average|
    rel_error = torch.norm(x_new - x_opt, p=2) / torch.norm(x_opt, p=2)

    return x_new, rel_error
```

Next we run AWC-DGD algorithm.

```python
%%px

# Set topology as exponential-two topology.
G = topology_util.ExponentialTwoGraph(bf.size())
bf.set_topology(G)

maxite = 500
x = torch.zeros(n, 1, dtype=torch.double)  # Initialize x
rel_error = torch.zeros((maxite, 1))
for ite in range(maxite):

    if bf.rank() == 0:
        if ite % 10 == 0:
            print("Progress {}/{}".format(ite, maxite))

    x, rel_error[ite] = AWC_DGD_one_step(
        x, x_opt, A, b, alpha=5e-4
    )  # you can adjust alpha to different values
```

    [stdout:7]
    Progress 0/500
    Progress 10/500
    Progress 20/500
    Progress 30/500
    Progress 40/500
    Progress 50/500
    Progress 60/500
    Progress 70/500
    Progress 80/500
    Progress 90/500
    Progress 100/500
    Progress 110/500
    Progress 120/500
    Progress 130/500
    Progress 140/500
    Progress 150/500
    Progress 160/500
    Progress 170/500
    Progress 180/500
    Progress 190/500
    Progress 200/500
    Progress 210/500
    Progress 220/500
    Progress 230/500
    Progress 240/500
    Progress 250/500
    Progress 260/500
    Progress 270/500
    Progress 280/500
    Progress 290/500
    Progress 300/500
    Progress 310/500
    Progress 320/500
    Progress 330/500
    Progress 340/500
    Progress 350/500
    Progress 360/500
    Progress 370/500
    Progress 380/500
    Progress 390/500
    Progress 400/500
    Progress 410/500
    Progress 420/500
    Progress 430/500
    Progress 440/500
    Progress 450/500
    Progress 460/500
    Progress 470/500
    Progress 480/500
    Progress 490/500

```python
# collect relative error from node 0 for step-size 2e-3
rel_error_exp2_alpha2em3 = dview.pull("rel_error", block=True, targets=0)
```

```python
# collect relative error from node 0 for step-size 1e-3
rel_error_exp2_alpha1em3 = dview.pull("rel_error", block=True, targets=0)
```

```python
# collect relative error from node 0 for step-size 5e-4
rel_error_exp2_alpha5em4 = dview.pull("rel_error", block=True, targets=0)
```

```python
import matplotlib.pyplot as plt

%matplotlib inline

plt.semilogy(rel_error_exp2_alpha2em3)
plt.semilogy(rel_error_exp2_alpha1em3)
plt.semilogy(rel_error_exp2_alpha5em4)

plt.legend(["2e-3", "1e-3", "5e-4"], fontsize=16)

plt.xlabel("Iteration", fontsize=16)
plt.ylabel("Relative error", fontsize=16)
```

It is observed from the above figures that smaller alpha can lead to more accuate solution, but the convergence rate will get slower. This observation is consistent with Eq. (DGD-Convergence).


### Example 4: Burgers' equation

Now that we have seen how to construct the non-linear convection and diffusion examples, we can combine them to form Burgers' equations. We again create a set of coupled equations which are actually starting to form quite complicated stencil expressions, even if we are only using a low-order discretisations.

Let's start with the definition fo the governing equations:
$$ \frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} + v \frac{\partial u}{\partial y} = \nu \; \left(\frac{\partial ^2 u}{\partial x^2} + \frac{\partial ^2 u}{\partial y^2}\right)$$

$$ \frac{\partial v}{\partial t} + u \frac{\partial v}{\partial x} + v \frac{\partial v}{\partial y} = \nu \; \left(\frac{\partial ^2 v}{\partial x^2} + \frac{\partial ^2 v}{\partial y^2}\right)$$

The discretized and rearranged form then looks like this:

\begin{aligned}
u_{i,j}^{n+1} &= u_{i,j}^n - \frac{\Delta t}{\Delta x} u_{i,j}^n (u_{i,j}^n - u_{i-1,j}^n) - \frac{\Delta t}{\Delta y} v_{i,j}^n (u_{i,j}^n - u_{i,j-1}^n) \\
&+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n-2u_{i,j}^n+u_{i-1,j}^n) + \frac{\nu \Delta t}{\Delta y^2} (u_{i,j+1}^n - 2u_{i,j}^n + u_{i,j+1}^n)
\end{aligned}

\begin{aligned}
v_{i,j}^{n+1} &=  v_{i,j}^n - \frac{\Delta t}{\Delta x} u_{i,j}^n (v_{i,j}^n - v_{i-1,j}^n) - \frac{\Delta t}{\Delta y} v_{i,j}^n (v_{i,j}^n - v_{i,j-1}^n) \\
&+ \frac{\nu \Delta t}{\Delta x^2}(v_{i+1,j}^n-2v_{i,j}^n+v_{i-1,j}^n) + \frac{\nu \Delta t}{\Delta y^2} (v_{i,j+1}^n - 2v_{i,j}^n + v_{i,j+1}^n)
\end{aligned}

Great. Now before we look at the Devito implementation, let's re-create the NumPy-based implementation form the original.

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 41
ny = 41
nt = 120
c = 1
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .0009
nu = 0.01
dt = sigma * dx * dy / nu
```

```python
#NBVAL_IGNORE_OUTPUT

# Assign initial conditions
u = np.empty((nx, ny))
v = np.empty((nx, ny))

init_hat(field=u, dx=dx, dy=dy, value=2.)
init_hat(field=v, dx=dx, dy=dy, value=2.)

plot_field(u)
```

```python
#NBVAL_IGNORE_OUTPUT

for n in range(nt + 1): ##loop across number of time steps
    un = u.copy()
    vn = v.copy()

    u[1:-1, 1:-1] = (un[1:-1, 1:-1] -
                     dt / dx * un[1:-1, 1:-1] *
                     (un[1:-1, 1:-1] - un[1:-1, 0:-2]) -
                     dt / dy * vn[1:-1, 1:-1] *
                     (un[1:-1, 1:-1] - un[0:-2, 1:-1]) +
                     nu * dt / dx**2 *
                     (un[1:-1,2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                     nu * dt / dy**2 *
                     (un[2:, 1:-1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))

    v[1:-1, 1:-1] = (vn[1:-1, 1:-1] -
                     dt / dx * un[1:-1, 1:-1] *
                     (vn[1:-1, 1:-1] - vn[1:-1, 0:-2]) -
                     dt / dy * vn[1:-1, 1:-1] *
                    (vn[1:-1, 1:-1] - vn[0:-2, 1:-1]) +
                     nu * dt / dx**2 *
                     (vn[1:-1, 2:] - 2 * vn[1:-1, 1:-1] + vn[1:-1, 0:-2]) +
                     nu * dt / dy**2 *
                     (vn[2:, 1:-1] - 2 * vn[1:-1, 1:-1] + vn[0:-2, 1:-1]))

    u[0, :] = 1
    u[-1, :] = 1
    u[:, 0] = 1
    u[:, -1] = 1

    v[0, :] = 1
    v[-1, :] = 1
    v[:, 0] = 1
    v[:, -1] = 1

plot_field(u)
```

Nice, our wave looks just like the original. Now we shall attempt to write our entire Burgers' equation operator in a single cell - but before we can demonstrate this, there is one slight problem.

The diffusion term in our equation requires a second-order space discretisation on our velocity fields, which we set through the `TimeFunction` constructor for $u$ and $v$. The `TimeFunction` objects will store this dicretisation information and use it as default whenever we use the shorthand notations for derivative, like `u.dxl` or `u.dyl`. For the advection term, however, we want to use a first-order discretisation, which we now have to create by hand when combining terms with different stencil discretisations. To illustrate let's consider the following example:

```python
from devito import Grid, TimeFunction, first_derivative, left

grid = Grid(shape=(nx, ny), extent=(2., 2.))
x, y = grid.dimensions
t = grid.stepping_dim

u1 = TimeFunction(name='u1', grid=grid, space_order=1)
print("Space order 1:\n%s\n" % u1.dxl)

u2 = TimeFunction(name='u2', grid=grid, space_order=2)
print("Space order 2:\n%s\n" % u2.dxl)

# We use u2 to create the explicit first-order derivative
u1_dx = first_derivative(u2, dim=x, side=left, order=1)
print("Explicit space order 1:\n%s\n" % u1_dx)
```

    Space order 1:
    u1(t, x, y)/h_x - u1(t, x - h_x, y)/h_x

    Space order 2:
    3*u2(t, x, y)/(2*h_x) + u2(t, x - 2*h_x, y)/(2*h_x) - 2*u2(t, x - h_x, y)/h_x

    Explicit space order 1:
    u2(t, x, y)/h_x - u2(t, x - h_x, y)/h_x

Ok, so by constructing derivative terms explicitly we again have full control of the spatial discretisation - the power of symbolic computation. Armed with that trick, we can now build and execute our advection-diffusion operator from scratch in one cell.

```python
#NBVAL_IGNORE_OUTPUT
from sympy import Eq, solve
from devito import Operator, Constant

# Define our velocity fields and initialise with hat function
u = TimeFunction(name='u', grid=grid, space_order=2)
v = TimeFunction(name='v', grid=grid, space_order=2)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

# Write down the equations with explicit backward differences
a = Constant(name='a')
u_dx = first_derivative(u, dim=x, side=left, order=1)
u_dy = first_derivative(u, dim=y, side=left, order=1)
v_dx = first_derivative(v, dim=x, side=left, order=1)
v_dy = first_derivative(v, dim=y, side=left, order=1)
eq_u = Eq(u.dt + u*u_dx + v*u_dy, a*u.laplace)
eq_v = Eq(v.dt + u*v_dx + v*v_dy, a*v.laplace)

# Let SymPy rearrange our stencils to form the update expressions
stencil_u = solve(eq_u, u.forward)[0]
stencil_v = solve(eq_v, v.forward)[0]
update_u = Eq(u.forward, stencil_u)
update_v = Eq(v.forward, stencil_v)

# Create Dirichlet BC expressions using the low-level API
bc_u = [Eq(u.indexed[t+1, 0, y], 1.)]  # left
bc_u += [Eq(u.indexed[t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(u.indexed[t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(u.indexed[t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(v.indexed[t+1, 0, y], 1.)]  # left
bc_v += [Eq(v.indexed[t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(v.indexed[t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(v.indexed[t+1, x, 0], 1.)]  # bottom

# Create the operator
op = Operator([update_u, update_v] + bc_u + bc_v)

# Execute the operator for a number of timesteps
op(time=nt + 1, dt=dt, a=nu)

plot_field(u.data[0])
```


# Chapter 2

## 2E1

Which of the expressions below correspond to the statement: the probability of rain on Monday?

(1) $Pr(\text{rain})$
(2) $Pr(\text{rain|Monday})$
(3) $Pr(\text{Monday|rain})$
(4) $Pr(\text{rain, Monday})/ Pr(\text{Monday})$

### Ans

$$
\frac{Pr(\text{rain, Monday})}{Pr(\text{Monday})} = Pr(\text{rain} \mid \text{Monday})
$$

## 2E2

Which of the following statements corresponds to the expression: $Pr(\text{Monday} \mid \text{rain})$?

(1) The probability of rain on Monday.
(2) The probability of rain, given that it is Monday.
(3) The probability that it is Monday, given that it is raining.
(4) The probability that it is Monday and that it is raining.

### Ans

(3) The probability that it is Monday, given that it is raining.

## 2E3

2E3. Which of the expressions below correspond to the statement: _the probability that it is Monday, given that it is raining_?

(1) $Pr(\text{Monday} \mid \text{rain})$

(2) $Pr(\text{rain} \mid \text{Monday})$

(3) $Pr(\text{rain} \mid \text{Monday}) \cdot Pr(\text{Monday})$

(4) $Pr(\text{rain} \mid \text{Monday}) \cdot Pr(\text{Monday}) / Pr(\text{rain})$

(5) $Pr(\text{Monday} \mid \text{rain}) \cdot Pr(\text{rain}) / Pr(\text{Monday})$

### Ans

(1) $Pr(\text{Monday} \mid \text{rain})$

(4)
$$
\begin{equation}
\begin{aligned}
Pr(\text{rain} \mid \text{Monday}) \cdot Pr(\text{Monday}) / Pr(\text{rain})
&= \frac{Pr(\text{rain}, \text{Monday})}{Pr(\text{rain})} \\
&= Pr(\text{Monday} \mid \text{rain}) \\
\end{aligned}
\end{equation}
$$

## 2E4.

The Bayesian statistician Bruno de Finetti (1906–1985) began his 1973 book on probability theory with the declaration: “PROBABILITY DOES NOT EXIST.” The capitals appeared in the original,  so I imagine de Finetti wanted us to shout this statement. What he meant is that probability is a device for describing uncertainty from the perspective of an observer with limited knowledge; it has no  objective reality. Discuss the globe tossing example from the chapter, in light of this statement. What  does it mean to say “the probability of water is 0.7”?

### Ans

In the case of the globe tossing example, it means that there is some "true" proportion of water to land. However, "truth"s like thees are usually fully known to us for many reasons (e.g. due to finite sample size, measurement error, unobserved variables, missing data, etc.). We can model our uncertainty using probability theory. Hopefully our model correctly represents truths, such as the "true" proportion of water to land. However, that's not guaranteed. When we say "the probability of water is 0.7," colloquially, it means that we *think* that the probability of water is somewhere around 70%, given this point in time. We did not give a lot of significant digits. It might not actually be exactly 0.7, but we _think_ it is somewhere around there. It still denotes uncertainty of the observer, given what we currently know.

## 2M1.

2M1. Recall the globe tossing model from the chapter. Compute and plot the grid approximate  posterior distribution for each of the following sets of observations. In each case, assume a uniform  prior for p.

(1) W, W, W
(2) W, W, W, L
(3) L, W, W, L, W, W, W

### Ans

We'll instead use rejection sampling and ABC-SMC:

```python
import numpy as np
import pandas as pd
from unlikely.models import Models, Model
from unlikely.priors import Beta
from unlikely.engine import abc_smc
from unlikely.misc import create_images_from_data

num_particles = 2000
obs = np.array([
    1,1,1,
    1,1,1,0,
    0,1,1,0,1,1,1
])

epsilons_list = [[0], [3,2,1,0]]

def distance(x,y):
    """
    Compare the number of ones in one vs. the other.
    """
    return abs(x.sum() - y.sum())

def simulate(priors, obs):
    """
    Data is binomially distributed.
    """
    return np.random.binomial(n=1, p=priors['beta'], size=len(obs))

data_to_display = [
    [
        {
            'title': f"obs: {obs[:3]}",
            'data': [
                pd.DataFrame({
                    'reference': np.random.beta(
                        obs[:3].sum() + 1,
                        len(obs[:3]) - obs[:3].sum() + 1,
                        num_particles
                    )
                })

            ]
        },
        {
            'title': f"after {obs[3:7]}",
            'data': [
                pd.DataFrame({
                    'reference': np.random.beta(
                        obs[:7].sum() + 1,
                        len(obs[:7]) - obs[:7].sum() + 1,
                        num_particles
                    )
                })
            ]
        },
        {
            'title': f"after {obs[7:]}",
            'data': [
                pd.DataFrame({
                    'reference': np.random.beta(
                        obs.sum() + 1,
                        len(obs) - obs.sum() + 1,
                        num_particles
                    )
                })
            ]
        },
        {
            'title': "Full batch",
            'data': [
                pd.DataFrame({
                    'reference': np.random.beta(
                        obs.sum() + 1,
                        len(obs) - obs.sum() + 1,
                        num_particles
                    )
                })
            ]
        }
    ]
]

for row, epsilons in enumerate(epsilons_list):
    models = Models(
        [
            Model(
                name='flat prior',
                priors=[
                    Beta(alpha=1, beta=1, name="beta"),
                ],
                simulate=simulate,
                prior_model_proba=1,
            ),
        ]
    )

    # Update with 1st batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models,
        obs=obs[:3],
        distance=distance,
    )

    data_to_display[0][0]['data'].append(
        pd.DataFrame(models[0].prev_accepted_proposals).rename(
            columns={'beta': f'eps: {epsilons}'}
        )
    )

    # The posterior distribution becomes the prior
    models.use_distribution_from_samples()

    # Update with 2nd batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models,
        obs=obs[3:7],
        distance=distance,
    )

    # The posterior distribution becomes the prior
    models.use_distribution_from_samples()

    data_to_display[0][1]['data'].append(
        pd.DataFrame(
            models[0].prev_accepted_proposals).rename(
            columns={'beta': f'eps: {epsilons}'}
        )
    )

    # Update with 3rd batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models,
        obs=obs[7:],
        distance=distance,
    )

    data_to_display[0][2]['data'].append(
        pd.DataFrame(
            models[0].prev_accepted_proposals).rename(
            columns={'beta': f'eps: {epsilons}'}
        )
    )

    models_full_batch = Models(
        [
            Model(
                name='flat prior',
                priors=[
                    Beta(alpha=1, beta=1, name="beta"),
                ],
                simulate=simulate,
                prior_model_proba=1,
            ),
        ]
    )

    # Update full batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models_full_batch,
        obs=obs,
        distance=distance,
    )

    data_to_display[0][3]['data'].append(
        pd.DataFrame(
            models_full_batch[0].prev_accepted_proposals
        ).rename(columns={'beta': f'eps: {epsilons}'})
    )

create_images_from_data(
    data={
        'title': '3 batch updates',
        'data': data_to_display
    },
    xlim=(0,1),
    figsize_mult=(2,8)
)
```

Using the [unlikely](https://github.com/Edderic/unlikely) library, Bayesian updating with the full batch, along with Bayesian updating through mini batches with only epsilon 0, seems to be more accurate than updating through mini batches with many epsilons. Mini batch updating leads to overinflated posterior distributions.

## 2M2

Now assume a prior for p that is equal to zero when p < 0.5 and is a positive constant when  p ≥ 0.5. Again compute and plot the grid approximate posterior distribution for each of the sets of observations in the problem just above.

```python
import numpy as np
import pandas as pd
from unlikely.models import Models, Model
from unlikely.priors import Uniform
from unlikely.engine import abc_smc
from unlikely.misc import create_images_from_data
import pdb

num_particles = 2000
obs = np.array([
    1,1,1,
    1,1,1,0,
    0,1,1,0,1,1,1
])

epsilons_list = [[0], [3,2,1,0]]

def distance(x,y):
    """
    Compare the number of ones in one vs. the other.
    """
    return abs(x.sum() - y.sum())

def simulate(priors, obs):
    """
    Data is binomially distributed.
    """

    return np.random.binomial(n=1, p=priors['uniform'], size=len(obs))

data_to_display = [
    [
        {
            'title': f"obs: {obs[:3]}",
            'data': []
        },
        {
            'title': f"after {obs[3:7]}",
            'data': []
        },
        {
            'title': f"after {obs[7:]}",
            'data': []
        },
        {
            'title': "Full batch",
            'data': []
        }
    ]
]

for row, epsilons in enumerate(epsilons_list):
    models = Models(
        [
            Model(
                name='Uniform over (0.5, 1)',
                priors=[
                    Uniform(alpha=0.5, beta=1, name="uniform"),
                ],
                simulate=simulate,
                prior_model_proba=1,
            ),
        ]
    )

    # Update with 1st batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models,
        obs=obs[:3],
        distance=distance,
    )

    data_to_display[0][0]['data'].append(
        pd.DataFrame(models[0].prev_accepted_proposals).rename(
            columns={'uniform': f'eps: {epsilons}'}
        )
    )

    # The posterior distribution becomes the prior
    models.use_distribution_from_samples()

    # Update with 2nd batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models,
        obs=obs[3:7],
        distance=distance,
    )

    # The posterior distribution becomes the prior
    models.use_distribution_from_samples()

    data_to_display[0][1]['data'].append(
        pd.DataFrame(
            models[0].prev_accepted_proposals).rename(
            columns={'uniform': f'eps: {epsilons}'}
        )
    )

    # Update with 3rd batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models,
        obs=obs[7:],
        distance=distance,
    )

    data_to_display[0][2]['data'].append(
        pd.DataFrame(
            models[0].prev_accepted_proposals).rename(
            columns={'uniform': f'eps: {epsilons}'}
        )
    )

    models_full_batch = Models(
        [
            Model(
                name='flat prior',
                priors=[
                    Uniform(alpha=0.5, beta=1, name="uniform"),
                ],
                simulate=simulate,
                prior_model_proba=1,
            ),
        ]
    )

    # Update full batch
    abc_smc(
        num_particles=num_particles,
        epsilons=epsilons,
        models=models_full_batch,
        obs=obs,
        distance=distance,
    )

    data_to_display[0][3]['data'].append(
        pd.DataFrame(
            models_full_batch[0].prev_accepted_proposals
        ).rename(columns={'uniform': f'eps: {epsilons}'})
    )

create_images_from_data(
    data={
        'title': '3 batch updates',
        'data': data_to_display
    },
    xlim=(0,1),
    figsize_mult=(2,8)
)
```

## 2M3

Suppose there are two globes, one for Earth and one for Mars. The Earth globe is 70% covered  in water. The Mars globe is 100% land. Further suppose that one of these globes—you don’t know  which—was tossed in the air and produced a “land” observation. Assume that each globe was equally  likely to be tossed. Show that the posterior probability that the globe was the Earth, conditional on  seeing “land” (Pr(Earth|land)), is 0.23.

```python
Beta(alpha=700, beta=300, name="proba_water").get_name()
```

    'proba_water'

We can construct two models of the world: one corresponding to the Earth and one corresponding to Mars. We then run the ABC-SMC algorithm. Afterwards, we compute to see how many times the samples produced from Earth are accepted and compare that to how many times the samples from Mars were accepted. We then normalize those two numbers by the number of total samples.

```python
import numpy as np
import pandas as pd
from unlikely.models import Models, Model
from unlikely.priors import Beta
from unlikely.engine import abc_smc
from unlikely.misc import create_images_from_data

def simulate(priors, obs):
    """
    Data is binomially distributed.
    """

    return np.random.binomial(n=1, p=priors['proba_water'], size=len(obs))

def distance(x,y):
    """
    Compare the number of ones in one vs. the other.
    """
    return abs(x.sum() - y.sum())

obs = np.array([0]) # land

num_particles = 2000
epsilons = [0]

models = Models(
    [
        Model(
            name='Earth produced it',
            priors=[
                Beta(alpha=700 + 1, beta=300 + 1, name="proba_water")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        ),

        Model(
            name='Mars produced it',
            priors=[
                Beta(alpha=1, beta=1001, name="proba_water")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        )
    ]
)

abc_smc(
    num_particles=num_particles,
    epsilons=epsilons,
    models=models,
    obs=obs,
    distance=distance,
)

models.get_posterior_probabilities()

```

    100%|████████████████████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:01<00:00, 1950.46it/s]

    {'Earth produced it': 0.2325, 'Mars produced it': 0.7675}

$P(\text{Earth} \mid \text{Data}) \approx 0.23$

## 2M4

Suppose you have a deck with only three cards. Each card has two sides, and each side is either  black or white. One card has two black sides. The second card has one black and one white side. The  third card has two white sides. Now suppose all three cards are placed in a bag and shuffled. Someone  reaches into the bag and pulls out a card and places it flat on a table. A black side is shown facing up,  but you don’t know the color of the side facing down. Show that the probability that the other side is  also black is 2/3. Use the counting method (Section 2 of the chapter) to approach this problem. This  means counting up the ways that each card could produce the observed data (a black side facing up  on the table).

```python
np.random.randint(0,1)
```

    0

```python
import numpy as np

card_1 = [0, 0]
card_2 = [0, 1]
card_3 = [1, 1]

probas = [1/3, 1/3, 1/3]
cards = [card_1, card_2, card_3]

def simulate_card_selection(cards, probas):
    """
    Let Black be represented by 0, and White be represented by 1.
    """

    count_black_other_side = 0
    num_sims = 10000
    num_div = 0

    for i in range(num_sims):

        # choose a card randomly
        chosen_card_index = np.random.choice(list(range(len(cards))), p=probas)

        chosen_card = cards[chosen_card_index]

        # choose a side to display
        chosen_side_to_show_index = np.random.randint(0,2)
        other_side_index = 1 - chosen_side_to_show_index
        chosen_side_to_show = chosen_card[chosen_side_to_show_index]

        if chosen_side_to_show == 0:
            num_div += 1

            if chosen_card[other_side_index] == 0:
                count_black_other_side += 1

    return count_black_other_side / num_div

print(f"Probability that the other side is black: {round(simulate_card_selection(cards, probas), 2)}")
```

    Probability that the other side is black: 0.67

## 2M5

2M5. Now suppose there are four cards: B/B, B/W, W/W, and another B/B. Again suppose a card is  drawn from the bag and a black side appears face up. Again calculate the probability that the other  side is black.

```python
cards = [(0,0), (0,1), (1,1), (0,0)]
print(f"Probability that the other side is black: {round(simulate_card_selection(cards), 2)}")
```

    Probability that the other side is black: 0.81

## 2M6

Imagine that black ink is heavy, and so cards with black sides are heavier than cards with white  sides. As a result, it’s less likely that a card with black sides is pulled from the bag. So again assume  there are three cards: B/B, B/W, and W/W. After experimenting a number of times, you conclude that  for every way to pull the B/B card from the bag, there are 2 ways to pull the B/W card and 3 ways to  pull the W/W card. Again suppose that a card is pulled and a black side appears face up. Show that  the probability the other side is black is now 0.5. Use the counting method, as before.

```python
cards = [(0,0), (0,1), (1,1)]
probas = np.array([1, 2, 3])
probas = probas / probas.sum() # Normalize
print(f"Probability that the other side is black: {round(simulate_card_selection(cards, probas), 2)}")
```

    Probability that the other side is black: 0.49

## 2M7

Assume again the original card problem, with a single card showing a black side face up. Before  looking at the other side, we draw another card from the bag and lay it face up on the table. The face  that is shown on the new card is white. Show that the probability that the first card, the one showing  a black side, has black on its other side is now 0.75. Use the counting method, if you can. Hint: Treat  this like the sequence of globe tosses, counting all the ways to see each observation, for each possible  first card.

```python
def simulate_card_selection_2M7(cards, probas):
    """
    Let Black be represented by 0, and White be represented by 1.
    """

    count_black_other_side = 0
    num_sims = 10000
    num_div = 0

    for i in range(num_sims):

        # choose a card randomly
        chosen_card_index = np.random.choice(
            list(range(len(cards))),
            p=probas
        )

        chosen_card = cards[chosen_card_index]

        # choose a side to display
        chosen_side_to_show_index = np.random.randint(0,2)
        other_side_index = 1 - chosen_side_to_show_index
        chosen_side_to_show = chosen_card[chosen_side_to_show_index]

        possible_other_chosen_card_indices = list(
            set(list(range(len(cards)))) - set({chosen_card_index})
        )

        other_card_index = np.random.choice(possible_other_chosen_card_indices)
        other_card = cards[other_card_index]
        other_card_side_to_show_index = np.random.randint(0,2)
        other_card_side_to_hide_index = 1 - other_card_side_to_show_index

        other_card_side_to_show = other_card[other_card_side_to_show_index]
        other_card_side_to_hide = other_card[other_card_side_to_hide_index]

        if chosen_side_to_show == 0 and other_card_side_to_show == 1:
            num_div += 1

            if chosen_card[other_side_index] == 0:
                count_black_other_side += 1

    return count_black_other_side / num_div

cards = [(0,0), (0,1), (1,1)]
probas = np.array([1, 1, 1])
probas = probas / probas.sum()
print(f"Probability that the other side is black: {round(simulate_card_selection_2M7(cards, probas), 2)}")
```

    Probability that the other side is black: 0.76

## 2H1

Suppose there are two species of panda bear. Both are equally common in the wild and live in the same places. They look exactly alike and eat the same food, and there is yet no genetic assay  capable of telling them apart. They differ however in their family sizes. Species A gives birth to twins 10% of the time, otherwise birthing a single infant. Species B births twins 20% of the time, otherwise  birthing singleton infants. Assume these numbers are known with certainty, from many years of field research.

Now suppose you are managing a captive panda breeding program. You have a new female panda  of unknown species, and she has just given birth to twins. What is the probability that her next birth  will also be twins?

```python
import numpy as np
import pandas as pd
from unlikely.models import Models, Model
from unlikely.priors import Beta
from unlikely.engine import abc_smc
from unlikely.misc import create_images_from_data

def simulate(priors, obs):
    """
    Data is binomially distributed.
    """

    return np.random.binomial(n=1, p=priors['proba_twin'], size=len(obs))

def distance(x,y):
    """
    Compare the number of ones in one vs. the other.
    """
    return abs(x.sum() - y.sum())

obs = np.array([1]) # observe a twin birth

num_particles = 5000
epsilons = [0]

models = Models(
    [
        Model(
            name='Species A',
            priors=[
                # Species A gives birth to twins 10% of the time
                Beta(alpha=1000 + 1, beta=9000 + 1, name="proba_twin")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        ),

        Model(
            name='Species B',
            priors=[
                # Species B gives birth to twins 20% of the time
                Beta(alpha=2000 + 1, beta=8000 + 1, name="proba_twin")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        )
    ]
)

abc_smc(
    num_particles=num_particles,
    epsilons=epsilons,
    models=models,
    obs=obs,
    distance=distance,
)

models.get_posterior_probabilities()

```

    100%|█████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:14<00:00, 350.27it/s]

    {'Species A': 0.3294, 'Species B': 0.6706}

There's a 1/3 chance that the species of mother is A, while there's a 2/3 chance that it is B. The data generating process in terms of the DAG is:

```python
import graphviz

from graphviz import Digraph

dag = Digraph('species-birthing-dag')
dag.edge('Species', 'Twin(t=0) = 1')
dag.edge('Species', 'Twin(t=1)')
dag
```

$$
\begin{equation}
\begin{aligned}
  P(T_{t=1} \mid T_{t=0} = 1) &= \sum_s P(T_{t=1}, S=s \mid T_{t=0} = 1) \\
&= \sum_s P(T_{t=1} \mid S=s, T_{t=0} = 1) \cdot P(S=s \mid T_{t=1}) \\
&= \sum_s P(T_{t=1} \mid S=s) \cdot P(S=s \mid T_{t=0} = 1) & \text{Once you know the Species, a previous birth doesn't tell you anything about the next birth}\\
\end{aligned}
\end{equation}
$$

From running the above code, we have $P(S=a \mid T_{t=0}=1) \approx 0.334$ and $P(S=b \mid T_{t=0}=1) \approx 0.666$. We are also given $P(T_{t=1} \mid S=s)$. When $S=a$, we have  $P(T_{t=1} \mid S=a) = 0.10$. For $S=b$, it is double that: $P(T_{t=1} \mid S=a) = 0.20$.

Thus, the above equation reduces to:

$$
\begin{equation}
\begin{aligned}
\sum_s P(T_{t=1} \mid S=s) \cdot P(S=s \mid T_{t=0} = 1) &=
P(T_{t=1} \mid S=a) \cdot P(S=a \mid T_{t=0} = 1) \\
&\quad + P(T_{t=1} \mid S=b) \cdot P(S=b \mid T_{t=0} = 1) \\
&= 0.1 \cdot 0.334 + 0.2 \cdot 0.666 \\
&= 0.1666 \\
&\approx 0.17
\end{aligned}
\end{equation}
$$

We could find the same answer by using our updated model of the world ($P(S=s \mid T_{t=0} = 1)$) to produce imaginary data for the next birth:

```python
simulate(
    models[0].prev_accepted_proposals,
    models[0].prev_accepted_proposals
).mean() * models.get_posterior_probabilities()['Species A'] + simulate(
    models[1].prev_accepted_proposals,
    models[1].prev_accepted_proposals
).mean() * models.get_posterior_probabilities()['Species B']
```

    0.17020000000000002

## 2H2

Recall all the facts from the problem above. Now compute the probability that the panda we  have is from species A, assuming we have observed only the first birth and that it was twins.

```python
import numpy as np
import pandas as pd
from unlikely.models import Models, Model
from unlikely.priors import Beta
from unlikely.engine import abc_smc
from unlikely.misc import create_images_from_data

def simulate(priors, obs):
    """
    Data is binomially distributed.
    """

    return np.random.binomial(n=1, p=priors['proba_twin'], size=len(obs))

def distance(x,y):
    """
    Compare the number of ones in one vs. the other.
    """
    return abs(x.sum() - y.sum())

obs = np.array([1]) # observe a twin birth

num_particles = 5000
epsilons = [0]

models = Models(
    [
        Model(
            name='Species A',
            priors=[
                # Species A gives birth to twins 10% of the time
                Beta(alpha=1000 + 1, beta=9000 + 1, name="proba_twin")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        ),

        Model(
            name='Species B',
            priors=[
                # Species B gives birth to twins 20% of the time
                Beta(alpha=2000 + 1, beta=8000 + 1, name="proba_twin")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        )
    ]
)

abc_smc(
    num_particles=num_particles,
    epsilons=epsilons,
    models=models,
    obs=obs,
    distance=distance,
)

print(
    "The probabilities of the species of the mother, given we observed twins: "\
    + f"{models.get_posterior_probabilities()}"
)

```

    100%|█████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:13<00:00, 358.93it/s]

    The probabilities of the species of the mother, given we observed twins: {'Species A': 0.3496, 'Species B': 0.6504}

## 2H3

Continuing on from the previous problem, suppose the same panda mother has a second birth  and that it is not twins, but a singleton infant. Compute the posterior probability that this panda is species A.

```python
models.use_distribution_from_samples() # set posterior distribution samples as new prior.

abc_smc(
    num_particles=num_particles,
    epsilons=epsilons,
    models=models,
    obs=np.array([0]),
    distance=distance,
)

print(
    "The probabilities of the species of the mother, given we observed 1 twin and 1 singleton: "\
    + f"{models.get_posterior_probabilities()}"
)

```

    100%|████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:00<00:00, 5901.81it/s]

    The probabilities of the species of the mother, given we observed 1 twin and 1 singleton: {'Species A': 0.3918, 'Species B': 0.6082}

Using the posterior from 2H2 as the prior, and combining that with the observation of a singleton infant, Species A became more likely, but Species B is still the more probable.

## 2H4

A common boast of Bayesian statisticians is that Bayesian inference makes it easy to use all of  the data, even if the data are of different types.  So suppose now that a veterinarian comes along who has a new genetic test that she claims can  identify the species of our mother panda. But the test, like all tests, is imperfect. This is the information you have about the test:

• The probability it correctly identifies a species A panda is 0.8.
• The probability it correctly identifies a species B panda is 0.65.

The vet administers the test to your panda and tells you that the test is positive for species A. First  ignore your previous information from the births and compute the posterior probability that your  panda is species A. Then redo your calculation, now using the birth data as well.

### Using vet genetic test only:

```python
import numpy as np
import pandas as pd
from unlikely.models import Models, Model
from unlikely.priors import Beta
from unlikely.engine import abc_smc
from unlikely.misc import create_images_from_data

def simulate(priors, obs):
    return np.random.binomial(n=1, p=priors['doc_proba'], size=len(obs))

def distance(x,y):
    """
    Compare the number of ones in one vs. the other.
    """
    return abs(x.sum() - y.sum())

obs = np.array([1]) # Doctor claims a species A

num_particles = 5000
epsilons = [0]

models = Models(
    [
        Model(
            name='Species A',
            priors=[
                Beta(alpha=8000 + 1, beta=2000 + 1, name="doc_proba")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        ),

        Model(
            name='Species B',
            priors=[
                Beta(alpha=3500 + 1, beta=6500 + 1, name="doc_proba")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        )
    ]
)

abc_smc(
    num_particles=num_particles,
    epsilons=epsilons,
    models=models,
    obs=obs,
    distance=distance,
)

print(
    "The probabilities of the species of the mother, given the doctor's claim: "\
    + f"{models.get_posterior_probabilities()}"
)

```

    100%|████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:02<00:00, 1856.05it/s]

    The probabilities of the species of the mother, given the doctor's claim: {'Species A': 0.7004, 'Species B': 0.2996}

```python
list(np.array([1])) + list(np.array([1,2]))
```

    [1, 1, 2]

```python
import numpy as np
import pandas as pd
from unlikely.models import Models, Model
from unlikely.priors import Beta
from unlikely.engine import abc_smc
from unlikely.misc import create_images_from_data

def simulate(priors, obs):
    genetic_test = np.random.binomial(n=1, p=priors['genetic_test'], size=1)
    twin_birth = np.random.binomial(n=1, p=priors['twin_birth'], size=len(obs['twin_birth']))

    return {
        'genetic_test': genetic_test,
        'twin_birth': twin_birth
    }

def distance(x,y):
    """
    Compare the number of ones in one vs. the other.
    """

    x_list = list(x['genetic_test']) + list(x['twin_birth'])
    y_list = list(y['genetic_test']) + list(y['twin_birth'])

    return abs(x['genetic_test'] - y['genetic_test']) \
        + sum(abs(x['twin_birth'] - y['twin_birth']))

obs = {
    'genetic_test': [1], # Species A
    'twin_birth': [1, 0]
}

np.array([1]) # Doctor claims a species A

num_particles = 5000
epsilons = [0]

models = Models(
    [
        Model(
            name='Species A',
            priors=[
                Beta(alpha=8000 + 1, beta=2000 + 1, name="genetic_test"),
                Beta(alpha=1000 + 1, beta=9000 + 1, name="twin_birth")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        ),

        Model(
            name='Species B',
            priors=[
                Beta(alpha=3500 + 1, beta=6500 + 1, name="genetic_test"),
                Beta(alpha=2000 + 1, beta=8000 + 1, name="twin_birth")
            ],
            simulate=simulate,
            prior_model_proba=0.5,
        )
    ]
)

abc_smc(
    num_particles=num_particles,
    epsilons=epsilons,
    models=models,
    obs=obs,
    distance=distance,
)

print(
    "The probabilities of the species of the mother, given the doctor's claim and births: "\
    + f"{models.get_posterior_probabilities()}"
)

```

    100%|█████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:41<00:00, 121.86it/s]

    The probabilities of the species of the mother, given the doctor's claim and births: {'Species A': 0.5634, 'Species B': 0.4366}

```python

```