## setup and notebook configuration

```python
# scientific python stack
import numpy as np
import sympy as sym
import orthopy
import quadpy
from scipy.interpolate import lagrange
```

```python
# matplotlib, plotting setup
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns                       # nice plotting defaults
import cmocean as cmo                       # ocean colormaps
sym.init_printing(use_latex='mathjax')
sns.set()
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
```

    /Users/corbin/virtual_envs/FEMexperiment_env/lib/python3.6/site-packages/cmocean/tools.py:76: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
      if not mpl.cbook.is_string_like(rgbin[0]):

```python
%load_ext autoreload
%autoreload 2
```

```python
from src.fem_base.master.master_1D import Master1D
import src.fem_base.master.polynomials_1D as p1d
import src.fem_base.master.mk_master as mkm
```

# 1D nodal basis

We define the master 1D element as $I\in[-1, 1]$, so in this section we will develop the ideas behind building a 1D nodal basis on this interval.

## construction of vandermonde matrices

We will construct our nodal basis as a linear combination of orthonormal polynomials, namely, the Legendre polynomials, since they can be efficiently and stably computed. We use the open source library `orthopy` to generate the Jacobi polynomials as described in Hesthaven, Warburton.

```python
x = np.linspace(-1, 1, 100)
n_polys = 4
vals = orthopy.line_segment.tree_jacobi(x, n=n_polys, alpha=0, beta=0, standardization='normal')
for i in range(n_polys):
    plt.plot(x, vals[i], label='P_{}'.format(i))
plt.legend()

plt.title('normalized Legendre polynomials')
plt.show()
```

1. These polynomials form the 'modal' basis over [-1, 1], which we will connect to the nodal basis via the Vandermonde matrices described below.
2. The choice of an orthonormal (not simply orthogonal) basis provides some elegant ways of computing the mass and stiffness matrices.

We can create the Vandermonde matrix for the orthonormal modal basis in 1D. These functions can be found in `fem_base/master/polynomials_1D.py`.

```python
# %load -s Jacobi_Poly,P_tilde,Vandermonde1D ../src/fem_base/master/polynomials_1D.py
def Jacobi_Poly(r, alpha, beta, N):
    """ wraps orthopy to return Jacobi polynomial
    @param r pts at which to evaluate the jacobi polynomial
    """
    return orthopy.line_segment.tree_jacobi(r, n=N-1,
        alpha=alpha, beta=beta, standardization='normal')

def P_tilde(r, N):
    P = np.zeros((len(r), N))
    polyvals = Jacobi_Poly(r, alpha=0, beta=0, N=N)
    for j in range(N):
        P[:, j] = polyvals[j]
    return P.T

def Vandermonde1D(N, x):
    """ initialize 1D vandermonde Matrix Vij = phi_j(x_i)"""
    V1D = np.zeros((len(x), N))
    JacobiP = Jacobi_Poly(x, alpha=0, beta=0, N=N)
    for j, polyvals in enumerate(JacobiP):
        V1D[:, j] = polyvals
    return V1D
```

An important conceptual point is that the Vandermonde matrix here is NOT the shape function matrix, it's the Vandermonde matrix of the Orthonormal polynomial basis. We will see this later as we have to create the shape function matrices.

Additionally, we wrap the `quadpy` library to create two interface functions, which generate Legendre Gauss Lobatto points and weights, as well as Gauss Legendre points and weights.

```python
# %load -s LegendreGaussLobatto,GaussLegendre ../src/fem_base/master/nodal_basis_1D.py
def LegendreGaussLobatto(N):
    """ generates N Legendre Gauss Lobatto points on [-1, 1]"""
    GL = quadpy.line_segment.GaussLobatto(N, a=0., b=0.)
    return GL.points, GL.weights

def GaussLegendre(N):
    """ generates N Gauss Legendre points on [-1, 1] """
    GL = quadpy.line_segment.GaussLegendre(N)
    return GL.points, GL.weights

```

## properties / conditioning of vandermonde matrices

The Vandermonde matrices associated with the modal Legendre basis are sensitive to choice of points at which the Vandermonde matrix is evaluated. This is discussed in Hesthaven.

- equidistant nodal points are very easy to generate, but lead to poorly conditioned Vandermonde matrices due to the well-known Runge phenomenon.
- Legendre Gauss Lobatto points are more stable at high polynomial order.

We can see how the determinant of the Vandermonde matrix grows as we move to a large number of nodal points (corresponding to high order polynomial order schemes).

```python
equi_det, LGL_det = [], []
for N in range(2, 35):
    nb = N + 1
    equi_pts = np.linspace(-1, 1, nb)
    V = Vandermonde1D(nb, equi_pts)
    equi_det.append(np.linalg.det(V))

    LGL_pts, _ = LegendreGaussLobatto(nb)
    V = Vandermonde1D(nb, LGL_pts)
    LGL_det.append(np.linalg.det(V))
```

```python
plt.semilogy(list(range(2, 35)), equi_det, label='equidistant')
plt.semilogy(list(range(2, 35)), LGL_det, label='LGL nodes')
plt.legend()
plt.show()
```

This plotted result agrees with that in Hesthaven. We wrap the scipy lagrange polynomial routines in order to construct a lagrange basis over the [-1, 1] interval for comparison.

### Runge phenomenon for lagrange polynomials

In the following subsection, we examine the Runge phenomenon of the lagrange polynomial basis when defined on equidistant points vs. LGL points. We will later develop the relations that allow us to build these nodal basis functions ("shape functions") over [-1, 1] from the Vandermonde matrices directly; but for now, we can explicitly create a lagrange basis of our choosing using `scipy` for the purposes of comparison.

```python
# %load -s lagrange_polys,lagrange_basis_at_pts ../src/fem_base/master/polynomials_1D.py
def lagrange_polys(pts):
    lagrange_polys = []
    for i, pt in enumerate(pts):
        data = np.zeros_like(pts)
        data[i] = 1
        lagrange_polys.append(lagrange(pts, data))
    return lagrange_polys

def lagrange_basis_at_pts(lagrange_polys, eval_pts):
    """ evaluates lagrange polynomials at eval_pts"""
    result = np.zeros((len(lagrange_polys) ,len(eval_pts)))
    for i, poly in enumerate(lagrange_polys):
        result[i, :] = lagrange_polys[i](eval_pts)
    return result

```

```python
N = 5
lp = np.linspace(-1, 1, N)
lpolys = lagrange_polys(lp)

vN = 100
view_pts = np.linspace(-1, 1, vN)
li = lagrange_basis_at_pts(lpolys, view_pts)
plt.plot(view_pts, li.T)
plt.title('lagrange polynomials over equally spaced points')
plt.show()
```

```python
N = 5
lp, _ = LegendreGaussLobatto(N)
lpolys = lagrange_polys(lp)

vN = 100
view_pts = np.linspace(-1, 1, vN)
li = lagrange_basis_at_pts(lpolys, view_pts)
plt.plot(view_pts, li.T)
plt.title('lagrange polynomials over LGL points')
plt.show()
```

By choosing LGL points instead of equidistant points, our basis functions don't exceed 1, unlike in the above plot, where we are already seeing a slight Runge phenomenon, even at low order. We will later see that these effects are exaggerated when computing the derivatives of these nodal shape functions.

## vandermonde relations

### relationship between vandermonde $V$, orthonormal modal basis polynomials $\tilde{\mathbf{P}}$, and lagrange basis functions (shape functions) $\ell$

Hesthaven makes the claim that $V^T \mathbf{\ell}(r) = \tilde{\mathbf{P}}(r)$ in (3.3).

In Hesthaven's notation, $N$ denotes the polynomial order, $N_p$ denotes the number of nodal points (we would call $nb$), and let's call the number of "view points" `xx`, which are arbitrary.

Then the shapes of the Hesthaven structures are:
- $\mathbf{\ell}$, $\tilde{\mathbf{P}}$, $V$ are all (`nb`, `xx`)
- $V^T \ell$ is (`xx`, `nb`) x (`nb`, `xx`) $\rightarrow$ (`xx`, `xx`) where rows contain the values of polynomials $\tilde{\mathbf{P}}$

This works for either equidistant points or the LGL points. So in this case we start from the lagrange basis, and compute the orthonormal modal coefficients. This has important applications when we have a nodal solution, but would like to decompose the solution into its modal components for spectral analysis or filtering.

```python
N = 5
lp, _ = LegendreGaussLobatto(N)  # LGL points
#lp = np.linspace(-1, 1, N)      # Equidistant points
view_pts = np.linspace(-1, 1, 50)

l_polys = lagrange_polys(pts=lp)
ℓ = lagrange_basis_at_pts(l_polys, eval_pts=view_pts)
V = Vandermonde1D(N=len(view_pts), x=lp)
P = np.dot(V.T, ℓ)

# plot the result
plt.plot(view_pts, ℓ.T, '--')
plt.plot(view_pts, P[0:3,:].T)
plt.show()
```

We see that indeed we recover the orthonormal Legendre polynomials.

Going the other way, we can invert the relation to find that

$$\ell = (V^T)^{-1} \tilde{\mathbf{P}}$$

which allows us to create our nodal shape functions from the orthonormal modal polynomial coefficients.

```python
nb = 4
nodal_pts, _ = LegendreGaussLobatto(nb)
view_pts = np.linspace(-1, 1, 50)

# create the Vandermonde, P matrices
V = Vandermonde1D(N=nb, x=nodal_pts)
Vti = np.linalg.inv(V.T)
P = P_tilde(r=view_pts, N=nb)
print('shape of Vandermonde: {}'.format(V.shape))
print('shape of P: {}'.format(P.shape))

yy = np.dot(Vti, P)
plt.plot(view_pts, yy.T)
plt.title('nodal shape functions generated from orthogonal basis polynomials')
plt.show()
```

    shape of Vandermonde: (4, 4)
    shape of P: (4, 50)

This is how we can efficiently evaluate the nodal shape functions at arbitrary points on the master element. This will be important for generating the data structures used for mapping and quadrature on the master element.

### relationship between vandermonde $V$ and mass matrix

We can build on the relationship developed in the section above to form the mass matrix for a nodal basis. We note that

$M_{ij} = \int_{-1}^{1}\ell_i(r)\, \ell_j(r) \,dr = (\ell_i, \ell_j)_I$, and if we expand out $\ell = (V^T)^{-1}\tilde{\mathbf{P}}$, it turns out (page 51)

$$M = (V V^T)^{-1}$$

because of the orthogonal nature of our choice of basis function; the implication is that we can compute the integrals over the master element without the explicit need for quadrature points or weights. Note first that $\phi_i(\xi) = \sum_{n=1}^{nb} (V^T)_{in}^{-1} \tilde{P}_{n-1}(\xi)$. Then

\begin{align}
M_{ij} &= \int^{1}_{-1} \phi_i(\xi)\,\phi_j(\xi)\,d\xi
      = \int^{1}_{-1}\left[\sum_{k=1}^{nb} (V^T)_{ik}^{-1} \tilde{P}_{k-1}(\xi)
      \sum_{m=1}^{nb} (V^T)_{jm}^{-1} \tilde{P}_{m-1}(\xi) \right]\, d\xi \\
      &= \sum_{k=1}^{nb} \sum_{m=1}^{nb} (V^T)_{ik}^{-1}
       (V^T)_{jm}^{-1} \int^{1}_{-1}\tilde{P}_{k-1}(\xi) \tilde{P}_{m-1}(\xi)
       =\sum_{k=1}^{nb} \sum_{m=1}^{nb} (V^T)_{ik}^{-1}
       (V^T)_{jm}^{-1} \delta_{km} \\
       &=\sum_{k=1}^{nb} (V^T)_{im}^{-1}
       (V^T)_{jm}^{-1} = \sum_{k=1}^{nb} (V^T)_{mi}^{-1} (V)_{mj}^{-1} \\
       &= (V^{T})^{-1} V^{-1} = (VV^T)^{-1}
\end{align}

Where note we've used the cute trick that $\int_{-1}^1 \tilde{P}_m \tilde{P}_n = \delta_{mn}$, since we chose an __orthonormal__ modal basis. Orthogonal wouldn't have done it, but an orthonormal modal basis has this property.

We can check this relation against the more traditional way of constructing the mass matrix with quadrature. `master.shap` has dimensions of (`n_quad`, `nb`)

```python
order = 3
m1d = Master1D(p=order)
M_quadrature = m1d.M

Np = order + 1
nodal_points, _ = LegendreGaussLobatto(Np)
V = Vandermonde1D(N=Np, x=nodal_points)
M_vand = np.linalg.inv(np.dot(V, V.T))

# throw error if mass matrices created by quadrature / Hesthaven relation unequal
assert(np.allclose(M_quadrature, M_vand))
```

## efficient computation of derivatives of the basis functions

### derivatives of Legendre polynomials

In order to compute the derivatives of the shape functions (which are expressed via the vandermonde matrix $V$), we must take the derivatives with respect to the orthogonal basis polynomials. There is an identity (Hesthaven, p. 52)

$$ \frac{d \tilde{P}_n}{d r} = \sqrt{n(n+1)}\,\tilde{P}^{(1,1)}_{n-1}$$

This is in contrast to directly differentiating either the coefficients of $\tilde{P}$ or more directly the nodal shape functions $\ell$ if the explicit polynomial form is known (like in `scipy`, but this becomes trickier in multiple dimensions). As it turns out, the first approach is a very efficient way to compute these operators.

```python
def Jacobi_Poly_Derivative(r, alpha, beta, N):
    """ take a derivative of Jacobi Poly, more general than above
    copy the format of orthopy (list of arrays)
    """
    dp = [np.zeros_like(r)]
    Jacobi_P = Jacobi_Poly(r, alpha + 1, beta + 1, N)
    for n in range(1, N+1):
        gamma = np.sqrt(n * (n + alpha + beta + 1))
        dp.append(gamma * Jacobi_P[n-1])
    return dp

#def dP_tilde(r, N):
#    P = np.zeros((len(r), N))
#    polyvals = Jacobi_Poly_Derivative(r, alpha=0, beta=0, N=N)
#    for j in range(N):
#        P[:, j] = polyvals[j]
#    return P
```

We can examine some of the derivatives of the Legendre polynomials.

```python
xx = np.linspace(-1, 1, 50)
jpd = Jacobi_Poly_Derivative(xx, alpha=0, beta=0, N=3)
for i, polyder in enumerate(jpd):
    plt.plot(xx, polyder, label=r'$P_{}^\prime(x)$'.format(i))
plt.legend()
plt.show()
```

These look good. The derivative of the first Legendre polynomial is analytically $P_0^\prime=0$, $P_1^\prime = \sqrt{3/2}$, $P_2^\prime$ should be linear, $P_3^\prime$ should be quadratic, etc.

### discrete derivative operators

We can declare the derivative Vandermonde matrix, and invert it in the same manner to obtain the derivatives of the nodal shape functions.

This works because
$$V^T \ell = P \Rightarrow V^T \frac{d}{dx} \ell = \frac{d}{dx}P$$

Hence $$V_r\equiv V^T D_r^T, \qquad {V_r}_{(ij)} = \frac{d \tilde{P}_j(r_i)}{d x} $$

and finally $D_r = V_r V^{-1}$ (see Hesthaven, p. 53), as well as $S = M D_r $, where $S_{ij} = \left(\phi_i, \frac{d\phi_j}{dx}\right)_I$, and where $M$ is the mass matrix.

```python
def GradVandermonde1D(N, x):
    Vr = np.zeros((len(x), N))
    dJacobi_P = Jacobi_Poly_Derivative(x, alpha=0, beta=0, N=N-1)
    for j, polyder in enumerate(dJacobi_P):
        Vr[:,j] = polyder
    return Vr
```

```python
p = 3
nb = p+1
nodal_pts, _ = LegendreGaussLobatto(nb)
#nodal_pts = np.linspace(-1, 1, nb)
view_pts = np.linspace(-1, 1, 50)

# grad vandermonde
V  = Vandermonde1D(N=nb, x=nodal_pts)
Vr = GradVandermonde1D(N=nb, x=view_pts)
Vi = np.linalg.inv(V)
Dr = np.dot(Vr, Vi)

print('shape Vr:    {}'.format(Vr.shape))
print('shape V inv: {}'.format(Vi.shape))
print('shape Dr:    {}'.format(Dr.shape))

# shape functions
V = Vandermonde1D(N=nb, x=nodal_pts)
Vti = np.linalg.inv(V.T)
P = P_tilde(r=view_pts, N=nb)
shap = np.dot(Vti, P)

# shape functions at view points
fig, ax = plt.subplots(1, 2, figsize=(8, 4))
ax[0].plot(view_pts, shap.T, '--')
ax[0].set_title(r'nodal shape functions $\phi$')
ax[1].plot(view_pts, Dr)
ax[1].set_title(r'derivatives of nodal shape functions $\frac{d \phi}{dx}$')
plt.show()
```

    shape Vr:    (50, 4)
    shape V inv: (4, 4)
    shape Dr:    (50, 4)

As a remark, we can once again show the effect of using Legendre Gauss Lobatto points vs equally spaced nodal points.

```python
N = 8
nb = N+1
nodal_pts_LGL, _ = LegendreGaussLobatto(nb)
nodal_pts_equi = np.linspace(-1, 1, nb)
view_pts = np.linspace(-1, 1, 100)

# shape functions at view points
fig, ax = plt.subplots(figsize=(8, 8))
ax.set_yticks([])
ax.set_xticks([])
labels = ['LGL nodes', 'uniform nodes']
for idx, nodal_pts in enumerate([nodal_pts_LGL, nodal_pts_equi]):
    # grad vandermonde
    V  = Vandermonde1D(N=nb, x=nodal_pts)
    Vr = GradVandermonde1D(N=nb, x=view_pts)
    Vi = np.linalg.inv(V)
    Dr = np.dot(Vr, Vi)

    # shape functions
    V = Vandermonde1D(N=nb, x=nodal_pts)
    Vti = np.linalg.inv(V.T)
    P = P_tilde(r=view_pts, N=nb)
    shap = np.dot(Vti, P)

    # plot
    ax = fig.add_subplot(2, 2, idx*2+1)
    ax.plot(view_pts, shap.T)
    ax.set_yticks([0, 1])
    ax.set_title(r' $\phi$, {}'.format(labels[idx]))
    ax = fig.add_subplot(2, 2, idx*2+2)
    ax.plot(view_pts, Dr)
    ax.set_title(r'$\f{}$, {}'.format('rac{d \phi}{dx}',labels[idx]))
plt.subplots_adjust(wspace=0.2, hspace=0.2)
fig.suptitle(r'$\phi$ and $\frac{d\phi}{d x}$, LGL vs uniformly-spaced nodes')
plt.show()
```

### remarks on discrete derivative operators

Suppose we compute the derivative matrix $D_r$ at the nodal points for some order $p$. There are some interesting properties to understand about these derivative matrices.

#### annihilation of constant vectors

Note that if we represent a function nodally, i.e., $u = c_i \phi_i(x)$, then

$$ \frac{du}{dx} = \frac{d}{dx}(c_i \phi_i(x)) = c_i \frac{d\phi_i}{dx} $$

Therefore, if we want to discretely take a derivative of a function (we'll represent the function on the master element for now, but trivially, we could map it to some other region), it suffices to multiply the derivative operator with the nodal vector $D_r u$. It should be clear, then, that $D_r$ will annihilate any constant vector.

```python
p = 2
nb = p+1
nodal_pts, _ = LegendreGaussLobatto(nb)
u = np.ones_like(nodal_pts)

# grad vandermonde
V  = Vandermonde1D(N=nb, x=nodal_pts)
Vr = GradVandermonde1D(N=nb, x=nodal_pts)
Vi = np.linalg.inv(V)
Dr = np.dot(Vr, Vi)

duh = np.dot(Dr, u)
print(np.max(np.abs(duh)))
```

    1.1102230246251565e-16

#### exponential convergence in $p$ for smooth functions

If the function of which we are attempting to take the discrete derivative is smooth (in the sense of infinitely differentiable), then we will see exponential convergence to the analytical solution w/r/t polynomial order of the nodal basis.

```python
ps = [1, 2, 3, 4, 8, 12, 16, 18, 24, 32]
errs = []
fig, ax = plt.subplots(1, 3, figsize=(8, 3))
for p in ps:
    nb = p+1
    nodal_pts, _ = LegendreGaussLobatto(nb)
    view_pts = np.linspace(-1, 1, 100)

    # grad vandermonde
    V  = Vandermonde1D(N=nb, x=nodal_pts)
    Vr = GradVandermonde1D(N=nb, x=nodal_pts)
    Vi = np.linalg.inv(V)
    Dr = np.dot(Vr, Vi)

    # nodal shap
    V = Vandermonde1D(N=nb, x=nodal_pts)
    Vti = np.linalg.inv(V.T)
    P = P_tilde(r=view_pts, N=nb)
    view_shap = np.dot(Vti, P)

    u = np.sin(nodal_pts-np.pi/4.)
    du = np.cos(view_pts-np.pi/4)
    duh = np.dot(Dr, u)

    view_duh = np.dot(duh, view_shap)
    err = np.max(np.abs(view_duh - du))
    errs.append(err)

    # plot first few
    if p < 4:
        ax[p-1].plot(view_pts, np.sin(view_pts), label=r'$u$')
        ax[p-1].plot(view_pts, du, label=r'$u^\prime$')
        ax[p-1].plot(view_pts, view_duh, '--', label=r'$du_h$')
        ax[p-1].set_title(r'$p={}$'.format(p))
        ax[p-1].legend()
fig.suptitle('discrete derivative operator applied to {}'.format(r'$\sin(x - \pi/4)$'), y=1.03)
plt.show()
```

Note that this is over the master element; to apply this to a field, we would also need to apply the $J^{-1}$ factors.

```python
fig, ax = plt.subplots()
ax.semilogy(ps, errs)
ax.set_xticks(ps)
ax.set_ylabel(r'$||du - du_h||_{L_\infty}$')
ax.set_xlabel('polynomial order p')
ax.set_title('exponential convergence of discrete derivative')
plt.show()
```

# 1D master element

In `fem_base/master/nodal_basis_1D.py`, we define a simple nodal basis object for 1D problems -- no need to be fancy, pretty much the only thing we need this for is to get nodal shape functions and their derivatives efficiently. The underlying orthonormal Legendre polynomial basis is hidden to the user. This is the work horse for the master element.

```python
# %load -s NodalBasis1D ../src/fem_base/master/nodal_basis_1D.py
class NodalBasis1D(object):
    """ minimalist nodal basis object:
    efficiently computes shape functions and their derivatives
    detailed explanation in tutorial/1D_basis_and_master_element.ipynb
    """
    def __init__(self, p, node_spacing='GAUSS_LOBATTO'):
        self.nb = p + 1
        if node_spacing == 'GAUSS_LOBATTO':
            self.nodal_pts, _ = LegendreGaussLobatto(self.nb)
        elif node_spacing == 'EQUIDISTANT':
            self.nodal_pts = np.linspace(-1, 1, self.nb)
        else: raise ValueError('node_spacing {} not recognized'.format(node_spacing))

    def shape_functions_at_pts(self, pts):
        """ computes shape functions evaluated at pts on [-1, 1]
        @retval shap (len(pts), nb) phi_j(pts[i])
        """
        V = p1d.Vandermonde1D(N=self.nb, x=self.nodal_pts)
        VTinv = np.linalg.inv(V.T)
        P = p1d.P_tilde(r=pts, N=self.nb)
        shap = np.dot(VTinv, P)
        return shap.T

    def shape_function_derivatives_at_pts(self, pts):
        """ computes shape function derivatives w/r/t x on [-1, 1]
        @retval shap_der, (Dr in Hesthaven), (len(pts), nb) d/dx phi_j(pts[i])
        """
        V  = p1d.Vandermonde1D(N=self.nb, x=self.nodal_pts)
        Vx = p1d.GradVandermonde1D(N=self.nb, x=pts)
        Vinv = np.linalg.inv(V)
        shap_der = np.dot(Vx, Vinv)
        return shap_der

```

Define a 1D master element, which is built on top of the 1D basis.
- Precompute shape functions at the nodal points and Gauss Legendre quadrature points, both are useful for different types of schemes. We use Gauss Legendre points instead of Gauss Lobatto points because they can integrate degree $2n-1$ polynomials exactly instead of $2n - 3$, where $n$ is the number of integration points. We would like to integrate $(\phi_i, \phi_j)_{\hat{K}}$, which is order 2$p$, so to integrate the mass matrix exactly, we need $2p + 1$ points, and common practice is $2p+2$. Since quadrature in 1D is cheap, we opt for the latter.
- Precompute mass matrix $M_{ij} = (\phi_i, \phi_j)$ and stiffness matrices $S_{ij} = \left(\phi_i, \frac{d\phi_j}{dx}\right)$, $K_{ij} = \left(\frac{d\phi_i}{dx}, \frac{d\phi_j}{dx}\right)$. Additionally, store $M^{-1}$, as it is commonly used. Although Hesthaven's method for mass and stiffness matrices are elegant, they rely on the underlying choice of an orthanormal modal basis. Since this class could be overloaded to work with other choices of basis, better to simply compute these matrices with quadrature.

```python
# %load -s Master1D ../src/fem_base/master/master_1D.py
class Master1D(object):
    """ minimalist 1D master object, for use in purely 1D problems
    detailed explanation in tutorial/1D_basis_and_master_element.ipynb
    """
    def __init__(self, p, nquad_pts=None, *args, **kwargs):
        self.p, self.nb = p, p+1
        self.basis = NodalBasis1D(p=p, **kwargs)
        self.nodal_pts = self.basis.nodal_pts
        self.nq = 2*self.p + 2 if nquad_pts is None else nquad_pts
        self.quad_pts, self.wghts = GaussLegendre(self.nq)

        # shape functions at nodal and quadrature points
        self.shap_quad,  self.dshap_quad = self.mk_shap_and_dshap_at_pts(self.quad_pts)

        # mass, stiffness matrices
        self.M, self.S, self.K = self.mk_M(), self.mk_S(), self.mk_K()
        self.Minv = np.linalg.inv(self.M)

        # lifting permuatation matrix L (0s, 1s)
        self.L = self.mk_L()

    def mk_shap_and_dshap_at_pts(self, pts):
        shap = self.basis.shape_functions_at_pts(pts)
        dshap = self.basis.shape_function_derivatives_at_pts(pts)
        return shap, dshap

    def mk_M(self):
        """ the mass matrix, M_ij = (phi_i, phi_j) """
        shapw = np.dot(np.diag(self.wghts), self.shap_quad)
        M = np.dot(self.shap_quad.T, shapw)
        return M

    def mk_S(self):
        """ the stiffness matrix, S_ij = (phi_i, \frac{d\phi_j}{dx}) """
        dshapw = np.dot(np.diag(self.wghts), self.dshap_quad)
        S = np.dot(self.shap_quad.T, dshapw)
        return S

    def mk_K(self):
        """ the stiffness matrix, K_ij = (\frac{d\phi_i}{dx}, \frac{d\phi_j}{dx}) """
        dshapw = np.dot(np.diag(self.wghts), self.dshap_quad)
        K = np.dot(self.dshap_quad.T, dshapw)
        return K

    def mk_L(self):
        L = np.zeros((self.nb, 2))
        L[0, 0]  = 1
        L[-1, 1] = 1
        return L

    @property
    def shap_der(self):
        """ return the shape derivatives for apps expecting 2, 3D"""
        return [self.dshap_quad]

```

```python
p=2
m1d = Master1D(p=p)
print(m1d.S)

m2 = mkm.Master_nodal(order=p, dim=1, element=0)
print(m2.K[0])
```

    [[-5.00000000e-01  6.66666667e-01 -1.66666667e-01]
     [-6.66666667e-01 -8.40647907e-17  6.66666667e-01]
     [ 1.66666667e-01 -6.66666667e-01  5.00000000e-01]]
    [[-5.00000000e-01 -6.66666667e-01  1.66666667e-01]
     [ 6.66666667e-01 -1.82500513e-17 -6.66666667e-01]
     [-1.66666667e-01  6.66666667e-01  5.00000000e-01]]

```python
def JacobiP(pts, α, β, N):
    """ same functionality as Hesthaven """
    return p1d.Jacobi_Poly(pts, α, β, N+1)[-1]

def GradJacobiP(pts, α, β, N):
    """ copy functionality from hesthaven """
    dp = np.zeros(len(pts))
    if N==0:
        return dp
    else:
        dp = np.sqrt(N*(N+α+β+1)) * JacobiP(pts,α+1,β+1,N-1)
        return dp
```

```python
xx = np.linspace(-1, 1, 100)
for i in range(4):
    yy = JacobiP(xx, 0, 0, i)
    plt.plot(xx,yy, label=r'$P_{}$'.format(i))
plt.legend()
plt.show()
```

```python
for i in range(4):
    yy = GradJacobiP(xx, 0, 0, i)
    plt.plot(xx,yy, label=r'$P^\prime_{}$'.format(i))
plt.legend()
plt.show()
```

# TODO
- demonstration of lifting operator on master
- performance timings for creation of these base operators


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

```python
test = symbols('newbe')
```

```python
test2 = test+10
```

```python

```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

```python
test_sub = (test)/4
```

```python
type(test_sub)
```

    sympy.core.mul.Mul

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

```python
test_sub.subs(test,1)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

```python
test_a = symbols('a')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

```python
dfdtdt = diff(dfdt,t)
```

```python
epl2 = Eq(dfdtdt,alpha*f(t))
```

```python
solution_eql2 = dsolve(epl2)
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
symbols?
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

```python
solution_eq.subs(t,0)
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
p_0_eql = Eq(at_0,p_0)
```

```python
solve(p_0_eql,C1)
```

```python

```

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

```python
particular.subs(t,0)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
t = symbols('t')
```

```python
f = Function('f')
```

    f

```python
dfdt = diff(f(t),t)
```

```python
ft = f(t)
```

```python
a,b = symbols('alpha,beta')
```

```python
b
```

```python
ft**2
```

```python
a*ft+b*(ft**2)
```

```python
eql = Eq(dfdt,a*ft+b*(ft**2))
```

```python
solution = dsolve(eql)
```

```python
solution = simplify(solution)
```

```python
solution.subs(t,0)
```

```python
C1,p0 = symbols('C1,p_0')
```

```python
right_hands = solution.rhs
```

```python
p0_eql = Eq(right_hands.subs(t,0),p0)
```

```python
C1_solutions = solve(p0_eql,C1)
```

```python
C1_solutions[0]
```

```python
particular = solution.subs(C1,C1_solutions[0])
```

```python
particular = simplify(particular)
```

```python
# Solution goes here
```

```python
# Solution goes here
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```


# Section 8.4 $\quad$ Differential Equations

In this section, we consider the first order differential equations.

A simple example of a differential equation is
\begin{equation*}
  \frac{d}{dt}x(t) = 2x(t).
\end{equation*}

The first order homogeneous system of differential equations is
\begin{eqnarray*}
% \nonumber to remove numbering (before each equation)
  x_1'(t) &=& a_{11}x_1(t) + a_{12}x_2(t) + \cdots a_{1n}x_n(t) \\
  x_2'(t) &=& a_{21}x_1(t) + a_{22}x_2(t) + \cdots a_{2n}x_n(t) \\
          & & \vdots \\
  x_2'(t) &=& a_{n1}x_1(t) + a_{n2}x_2(t) + \cdots a_{nn}x_n(t)
\end{eqnarray*}

### Example 1

Find the solution of the following system of differential equations
\begin{equation*}
  \left[
    \begin{array}{c}
      x_1' \\
      x_2' \\
      x_3' \\
    \end{array}
  \right] =
  \left[
        \begin{array}{rrr}
          3 & 0 & 0\\
          0 & -2 & 0\\
          0 & 0 & 4 \\
        \end{array}
      \right]
  \left[
    \begin{array}{c}
      x_1 \\
      x_2 \\
      x_3 \\
    \end{array}
  \right]
\end{equation*}

```python
from sympy import *

t = symbols('t');
x1 = Function('x1');
x2 = Function('x2');
x3 = Function('x3');
eq1 = x1(t).diff(t) - 3*x1(t);
eq2 = x2(t).diff(t) + 2*x2(t);
eq3 = x3(t).diff(t) - 4*x3(t);

dsolve([eq1, eq2, eq3])
```

    [x1(t) == C1*exp(3*t), x2(t) == C2*exp(-2*t), x3(t) == C3*exp(4*t)]

In general $A$ is not a diagonal matrix. Suppose that $A$ is diagonalizable.

### Example 2

Find the solution of the following system of differential equations
\begin{equation*}
  \left[
    \begin{array}{c}
      x_1' \\
      x_2' \\
      x_3' \\
    \end{array}
  \right] =
  \left[
        \begin{array}{rrr}
          0 & 1 & 0\\
          0 & 0 & 1\\
          8 & -14 & 7 \\
        \end{array}
      \right]
  \left[
    \begin{array}{c}
      x_1 \\
      x_2 \\
      x_3 \\
    \end{array}
  \right]
\end{equation*}

```python
from sympy import *

t = symbols('t');
x1 = Function('x1');
x2 = Function('x2');
x3 = Function('x3');
eq1 = x1(t).diff(t) - x2(t);
eq2 = x2(t).diff(t) - x3(t);
eq3 = x3(t).diff(t) - 8*x1(t) + 14*x2(t) - 7*x3(t);

dsolve([eq1, eq2, eq3])
```

    [x1(t) == C1*exp(t) + C2*exp(2*t) + C3*exp(4*t),
     x2(t) == C1*exp(t) + 2*C2*exp(2*t) + 4*C3*exp(4*t),
     x3(t) == C1*exp(t) + 4*C2*exp(2*t) + 16*C3*exp(4*t)]

### Example 3

Find the solution of the following system of differential equations
\begin{equation*}
  \left[
    \begin{array}{c}
      x_1' \\
      x_2' \\
      x_3' \\
    \end{array}
  \right] =
  \left[
        \begin{array}{rrr}
          1 & 0 & 0\\
          0 & 3 & -2\\
          0 & -2 & 3 \\
        \end{array}
      \right]
  \left[
    \begin{array}{c}
      x_1 \\
      x_2 \\
      x_3 \\
    \end{array}
  \right]
\end{equation*}

```python
from sympy import *

t = symbols('t');
x1 = Function('x1');
x2 = Function('x2');
x3 = Function('x3');
eq1 = x1(t).diff(t) - x1(t);
eq2 = x2(t).diff(t) - 3*x2(t) + 2*x3(t);
eq3 = x3(t).diff(t) + 2*x2(t) - 3*x3(t);

dsolve([eq1, eq2, eq3])
```

    [x1(t) == C1*exp(t) + C2*t*exp(t),
     x2(t) == C2*exp(t) - C3*exp(5*t),
     x3(t) == C2*exp(t) + C3*exp(5*t)]


# About
This notebook was created to follow the sympy code generation tutorial at [this link](http://www.sympy.org/scipy-2017-codegen-tutorial/)

```python
from sympy import *
init_printing()
```

```python
pi.evalf(10)
```

```python
alpha, beta, gamma, x, y = symbols('alpha beta gamma x y')
alpha, beta
```

```python
f= Function('f')
```

```python
diff(sin(x+1)*cos(y), x, y)
```

```python
test = diff(f(x)+1,x)
test
```

```python
Md = Function('M_d')(x)
Md
```

```python
q1, q2, q3 = symbols('q_1 q_2 q_3')
q = Matrix([q1, q2, q3])
q
```

```python
acol = Matrix([q1**2+sin(q2), exp(q3), q2-q1])
J = acol.jacobian(q)
J
```

```python
n, m = symbols('n m', integer=True)
M = MatrixSymbol('M', n, m)
b = MatrixSymbol('b', m, 1)
M*b
```

```python
(M*b).T
```

```python
(M*b).T
```

```python
Transpose(M*b).doit()
```

```python
A = IndexedBase('A')
i = Idx('i')
A[2], A[i]
```

# Code printing

```python
x = symbols('x')
expr = abs(sin(x**2))
expr
```

```python
ccode(expr)
```

    'fabs(sin(pow(x, 2)))'

```python
tanh(x).rewrite(exp)
```

# make python functions from expressions the easy way

```python

```


# 07 Uncertainty Analysis

## Elemental Uncertainties
Uncertainty due to elemental contributors, $u_i$:
\begin{align}
u_{e} = \sqrt{\sum_i u_i^2}
\end{align}

begin{align}
u_{overall} & = \sqrt{\sum_{comp} u_{comp}^2}
\end{align}

### Pressure Regulator Calibration

It is desired to use an Equilibar pressure regulator QPV in gas flow experiment.  The specifications for this regulator are:
(https://www.equilibar.com/electronic-pressure-regulators/)

> accuracy = .25%FS

> repeatability = 0.02%FS

However, the flow meter accuracy is too low for our application and one decides to calibrate it with a flow calibrator that has an accuracy of 0.04 % of FS.

> What will be the accuracy of the calibrated instrument?

### Pressure Measurement

A pressure transducer is connected to a digital data-acquisition board on a computer.  The characteristics of the pressure transducer and the data-acquisition system (DAS) are given below.  All uncertainties are given at 95 % confidence level.
The environment temperature is uncertain by $\pm 10^\circ$C.

> Pressure Transducer Characteristics:

Range: 0-1,000 kPa

 Output: 0-5 V

 Linearity: $\pm 0.25$ %FS

 Repeatability: $\pm 0.06$ %FS

 Thermal-stability error: $\pm 0.01$ %FS/$^\circ$C

> DAS Characteristics:

 Number of bits: 10 bits

 Input range: $0-10,\, 0-1,\, 0-0.1,\, 0-0.01$ V

 Gain error: $\pm 2$ LSD

 Linearity: $\pm 2$ LSD

Overall Uncertainty for measurement: Combination of Pressure Transducer and DAS

\begin{align}
u_{overall} & = \sqrt{\sum_{comp} u_{comp}^2} \\
& = \sqrt{u_{PT}^2 + u_{DAS}^2}
\end{align}

Uncertainty of presssure transducer
\begin{align}
u_{PT} = \sqrt{\sum_i u_i^2}
\end{align}
with $u_i$ elemental error:
> linearity: $\pm 0.25$ %FS

> repeatability: $\pm 0.06$ %FS

> Thermal stability: $\pm 0.01$ %FS/$^\circ$C

LSD: Least Significant Digit

01101: binary word, LSD: bit associated with $2^0$ and MSD (most significant digit): bit associated with $2^{N-1}$ for $N$ bit word.

ie value of the first bin in your digitized word/measurement.

number of bins: $2^N$, $N$: number of bits

size of a bin: resolution, $\Delta V = \frac{V_{max} - V_{min}}{2^N}$

## Uncertainty of a result

\begin{align}
R = f(x_1, x_2, \dots) \quad u_R = \sqrt{\sum_i \left( u_{x_i} \frac{\partial f}{\partial x_i} \right)^2}
\end{align}

### Design of experiment

One wishes to determine the heat transfer coefficient, $h$, of a heated rod of length, $L$, and diameter, $D$, in a flow. By applying the first law of thermodynamics one can relate the input power of the experiment, $W$, to
$h$ by:

\begin{align*}
h = \frac{W}{\pi D L \Delta T}, \textrm{where $\Delta$T is the temperature difference between the rod and the surrounding fluid}
\end{align*}

The relative uncertainties are: $\frac{u_L}{L}=0.1\%$, $\frac{u_D}{D}=0.2\%$, $\frac{u_{\Delta T}}{\Delta T} = 1.0\%$.  One wishes to have a maximum relative inaccuracy of 2% of $h$.

What should be the relative accuracy of the power measurements, $u_W/W$?

```python

```


```python
from pyodesys.tests._robertson import get_ode_exprs
import sympy as sp
from sympy import symbols
sp.init_printing()
```

```python
fj = [get_ode_exprs(reduced=reduced) for reduced in range(4)]
```

```python
t = symbols('t')
y = A, B, C = symbols('A B C')
inits = symbols('A0 B0 C0')
p = symbols('k1 k2 k3')
p1 = p2 = p3 = p + inits
y1, y2, y3 = [B, C], [A, C], [A, B]
```

```python
f0, j0 = fj[0][0](t, y, p, backend=sp), fj[0][1](t, y, p, backend=sp)
f0, sp.Matrix(j0)
```

```python
f1, j1 = fj[1][0](t, y1, p1, backend=sp), fj[1][1](t, y1, p1, backend=sp)
f1, sp.Matrix(j1)
```

```python
f2, j2 = fj[2][0](t, y2, p2, backend=sp), fj[2][1](t, y2, p2, backend=sp)
f2, sp.Matrix(j2)
```

```python
f3, j3 = fj[3][0](t, y3, p3, backend=sp), fj[3][1](t, y3, p3, backend=sp)
f3, sp.Matrix(j3)
```

```python
diff1 = sp.Matrix(f1).jacobian(y1) - sp.Matrix(j1)
diff1.simplify()
diff1
```

```python
diff2 = sp.Matrix(f2).jacobian(y2) - sp.Matrix(j2)
diff2.simplify()
diff2
```

```python
diff3 = sp.Matrix(f3).jacobian(y3) - sp.Matrix(j3)
diff3.simplify()
diff3
```


# Bayesian Linear Regression

```python
import os, sys, re, random

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from time import time
import scipy as sp
import scipy.io as sio

plt.rcParams['figure.figsize'] = (16.0, 8.0)
```

## 1. Linear Regression

```python
olympics = sio.loadmat('olympics.mat')
```

```python
x = np.array(olympics['male100'][:,0])
y = np.array(olympics['male100'][:,1])
plt.scatter(x, y)
plt.xlabel('Years')
plt.ylabel('Time')
plt.show()
```

```python
def lsfit(x, t):
    xmu = np.sum(x) / len(x)
    tmu = np.sum(t) / len(t)
    xt = np.array([x_i * t_i for x_i, t_i in zip(x, t)])
    xtmu = np.sum(xt) / len(xt)
    x2mu = np.sum(x**2) / len(x)
    w1 = (xtmu - xmu * tmu) / (x2mu - xmu**2)
    w0 = tmu - w1 * xmu
    return w0, w1

def avg_loss(x, t, w0, w1):
    return np.sum([(t[i] - w0 - w1*x[i])**2 for i in range(len(x))]) / len(x)
```

### 1.1. Men's 100 meters

```python
x_male100 = np.array(olympics['male100'][:,0])
y_male100 = np.array(olympics['male100'][:,1])

w0_male100, w1_male100 = lsfit(x_male100, y_male100)
fig, ax = plt.subplots()
points = ax.scatter(x_male100, y_male100)
line = ax.plot(x_male100, [w1_male100 * x_i + w0_male100 for x_i in x_male100], color='red')
ax.set_xlabel('Years')
ax.set_ylabel('Time')

print('w0:', w0_male100, 'w1:', w1_male100)
print('Average loss:', avg_loss(x_male100, y_male100, w0_male100, w1_male100))
plt.show()
```

### 1.2. Women's 100 meters

```python
x_female100 = np.array(olympics['female100'][:,0])
y_female100 = np.array(olympics['female100'][:,1])

w0_female100, w1_female100 = lsfit(x_female100, y_female100)
fig, ax = plt.subplots()
points = ax.scatter(x_female100, y_female100)
line = ax.plot(x_female100, [w1_female100 * x_i + w0_female100 for x_i in x_female100], color='red')
ax.set_xlabel('Years')
ax.set_ylabel('Time')

print('w0:', w0_female100, 'w1:', w1_female100)
print('Average loss:', avg_loss(x_female100, y_female100, w0_female100, w1_female100))
plt.show()
```

### 1.3. Men vs Women's 100 meters

```python
fig, ax = plt.subplots()

ax.scatter(x_male100, y_male100, color='blue', label='male100')
ax.scatter(x_female100, y_female100, color='red', label='female100')

ax.plot(x_male100, [w1_male100 * x_i + w0_male100 for x_i in x_male100], color='lightblue')
ax.plot(x_female100, [w1_female100 * x_i + w0_female100 for x_i in x_female100], color='pink')

ax.set_xlabel('Years')
ax.set_ylabel('Time')
ax.legend()

plt.show()
```

#### 1.3.1. Intersection

$$
w_{1_{male}}x - y = -w_{0_{male}}\\
w_{1_{female}}x - y = -w_{0_{female}}
$$

We use `numpy` to solve these linear equations by passing the coefficients to `solve` function:

```python
x, y = np.linalg.solve([[w1_male100, -1], [w1_female100, -1]], [-w0_male100, -w0_female100])
print('Intersection at year %.2f with time %.2f' % (x, y))
```

    Intersection at year 2589.25 with time 1.90

### 1.4. Loss minimization in vector form

In the lectures, we derived the following expression for the least squares solution in vector form:

$$
\hat w = (X^TX)^{-1}X^Tt \\
X = \begin{bmatrix}
       x_1^0 & x_1^1 \\
       x_2^0 & x_2^1 \\
       \vdots & \vdots \\
       x_N^0 & x_N^1
     \end{bmatrix}
$$

Our first step is to create X. This can be done in many ways. Perhaps the easiest is with the following command:

```python
def get_design_matrix(x, k):
    return np.array([[x_i**n for n in range(k + 1)] for x_i in np.array(x).flat])

k = 2
X = get_design_matrix(x_male100, k) # second order
print(X)
```

    [[  1.00000000e+00   1.89600000e+03   3.59481600e+06]
     [  1.00000000e+00   1.90000000e+03   3.61000000e+06]
     [  1.00000000e+00   1.90400000e+03   3.62521600e+06]
     [  1.00000000e+00   1.90600000e+03   3.63283600e+06]
     [  1.00000000e+00   1.90800000e+03   3.64046400e+06]
     [  1.00000000e+00   1.91200000e+03   3.65574400e+06]
     [  1.00000000e+00   1.92000000e+03   3.68640000e+06]
     [  1.00000000e+00   1.92400000e+03   3.70177600e+06]
     [  1.00000000e+00   1.92800000e+03   3.71718400e+06]
     [  1.00000000e+00   1.93200000e+03   3.73262400e+06]
     [  1.00000000e+00   1.93600000e+03   3.74809600e+06]
     [  1.00000000e+00   1.94800000e+03   3.79470400e+06]
     [  1.00000000e+00   1.95200000e+03   3.81030400e+06]
     [  1.00000000e+00   1.95600000e+03   3.82593600e+06]
     [  1.00000000e+00   1.96000000e+03   3.84160000e+06]
     [  1.00000000e+00   1.96400000e+03   3.85729600e+06]
     [  1.00000000e+00   1.96800000e+03   3.87302400e+06]
     [  1.00000000e+00   1.97200000e+03   3.88878400e+06]
     [  1.00000000e+00   1.97600000e+03   3.90457600e+06]
     [  1.00000000e+00   1.98000000e+03   3.92040000e+06]
     [  1.00000000e+00   1.98400000e+03   3.93625600e+06]
     [  1.00000000e+00   1.98800000e+03   3.95214400e+06]
     [  1.00000000e+00   1.99200000e+03   3.96806400e+06]
     [  1.00000000e+00   1.99600000e+03   3.98401600e+06]
     [  1.00000000e+00   2.00000000e+03   4.00000000e+06]
     [  1.00000000e+00   2.00400000e+03   4.01601600e+06]
     [  1.00000000e+00   2.00800000e+03   4.03206400e+06]]

Given X, we can easily compute Equation 1 with the following command:

```python
def get_w_hat(X, t):
    return np.dot(np.linalg.inv(X.T.dot(X)), X.T.dot(t))

t = np.array(y_male100)
w_hat = get_w_hat(X, t)
print(w_hat)
```

    [  4.55597856e+02  -4.43160486e-01   1.10151552e-04]

```python
fig, ax = plt.subplots()
men_points = ax.scatter(x_male100, y_male100)
men_fit = ax.plot(x_male100, X.dot(w_hat), color='red')

ax.set_xlabel('Years')
ax.set_ylabel('Time')

plt.show()
```

To make predictions, at a column vector of test points, `x_test`, we need to create `X_test` and then multply it by `w_hat`. For example:

```python
x_test = np.arange(2012, 2016)
X_test = get_design_matrix(x_test, k)
predictions = X_test.dot(w_hat) # returns a column vector
print('Predictions:', predictions.T)
```

    Predictions: [ 9.86830317  9.86850269  9.8689225   9.86956262]

```python
def get_predictions(x, t, x_test, k = 0):
    X = get_design_matrix(x, k)
    X_test = get_design_matrix(x_test, k)
    w_hat = get_w_hat(X, t)
    return X_test.dot(w_hat)
```

We may find it interesting to test your code with different datasets. The following code will generate data from a third order polynomial:

```python
x = np.random.rand(1, 50)
x.sort()
x_test = np.random.rand(1, 200)
x_test.sort()
noise = 0.5
t = np.array(5 * x**3 - x**2 + x + np.random.rand(1, 50) * noise).T
tt = np.array(5 * x_test**3 - x_test**2 + x_test + np.random.rand(1, 200) * noise).T

k = 2
```

```python
fig, ax = plt.subplots()
points = ax.scatter(x, t)
points_test = ax.scatter(x_test, tt, color='pink')
X = get_design_matrix(x, k)
w_hat = get_w_hat(X, t)
fit = ax.plot(x.T, X.dot(w_hat), color='green')

ax.set_xlabel('T')
ax.set_ylabel('X')

plt.show()
```

You can compute the loss on the test data using:

```python
di = (get_predictions(x, t, x_test, k) - tt)**2;
mean_err = di.mean()
print('Mean error', mean_err)
```

    Mean error 0.031804680491

### 1.5. Cross-validation

```python
def cross_validation(x, t, k = 0):
    err = np.zeros(x.size)
    for i in range(x.size):
        train_x = np.delete(x, i)
        train_t = np.delete(t, i)
        test_x = np.take(x, i)
        test_t = np.take(t, i)
        pred = get_predictions(train_x, train_t, test_x, k)
        err[i] = ((pred - test_t) ** 2).mean()

    mean_err = err.mean()
    return mean_err

k = 2
noise = 0.5
x = np.random.rand(20)
x.sort()

t = np.array(5 * x**3 - x**2 + x + np.random.rand(20) * noise).T
di = (get_predictions(x, t, x_test, k) - tt)**2;
mean_err = di.mean()
print('Mean error: %.2f' % mean_err)
mean_err = cross_validation(x, t, k)
print('Mean error from cross-validation: %.2f' % mean_err)

X = get_design_matrix(x, k)
w_hat = get_w_hat(X, t)

fig, ax = plt.subplots()
points = ax.scatter(x, t)

fit = ax.plot(x, X.dot(w_hat), color='red')

plt.show()
```

Plot the average error for different polynomial orders. Does your code predict the correct one?

```python
orders = np.arange(10)
avg_err = np.zeros(10)
for k in orders:
    avg_err[k] = cross_validation(x, t, k)

print(avg_err)

plt.figure()
plt.plot(orders, avg_err)
plt.xlabel('k-order polynomial')
plt.ylabel('Error')
plt.yscale('log')
plt.show()
```

## 2. Bayesian Linear Regression

We want to look at the effect of increasing the polynomial order above 4, we'll need to rescale the $x$ (Olympic year) values (do this before you calculate all the $x^k$ stuff).

```python
# Load men's olympic data
x = np.array(olympics['male100'][:,0])
t = np.array(olympics['male100'][:,1])

min_val = x.min() # First Olympic year
x = x - min_val   # Subtract the first year
x = x / 4         # Olympics are every four years

test_x = (np.array(2012) - min_val) / 4
```

### 2.1. Maximum likelihood

Using the maximum likelihood estimators, we compute $\hat w$ and $\hat\sigma^2$ on the Olympic data for polynomials of first, second and third order.

\begin{align}
t &= w_0 + w_1 x \\
t &= w_0 + w_1 x + w_2 x^2 \\
t &= w_0 + w_1 x + w_2 x^2 + w_3 x^3
\end{align}

We first declare a function to compute $\hat\sigma^2$, the variance, which is defined as:

\begin{equation}
\hat\sigma^2 = \frac{1}{N}(\mathbf{t} - \mathbf{X\hat{w}})^T(\mathbf{t} - \mathbf{X\hat{w}})
\end{equation}

```python
def get_variance(X, t, w_hat):
    A = t - X.dot(w_hat)
    return A.T.dot(A / t.size)

K = 6
s_q = np.zeros(K)
for k in np.arange(K):
    X = get_design_matrix(x, k)
    w_hat = get_w_hat(X, t)
    s_q[k] = get_variance(X, t.T, w_hat).mean()

plt.figure()
plt.plot(np.arange(K), s_q)
plt.xlabel('k-order polynomial')
plt.ylabel('Variance [log]')
plt.yscale('log')
plt.show()
```

We perform a LOO CV analysis for maximum likelihood for polynomials of order `0` to `8`:

```python
K = 9
orders = np.arange(K)
avg_err = np.zeros(K)
for k in orders:
    avg_err[k] = cross_validation(x, t, k)

print(avg_err)

plt.figure()
plt.plot(orders, avg_err)
plt.xlabel('k-order polynomial')
plt.ylabel('Error')
plt.show()
```

## 2.2. Uncertainty in estimates

For the first order polynomial, we compute $\mathrm{cov}\{\hat w\}$ using $\hat\sigma^2$ in place of $\sigma^2$.

\begin{equation}
    \mathrm{cov}\{\hat w\} = \sigma^2 \left(\mathbf{X}^T \mathbf{X}\right)^{-1}
\end{equation}

```python
# Load men's olympic data
x = np.array(olympics['male100'][:,0])
t = np.array(olympics['male100'][:,1])

min_val = x.min() # First Olympic year
x = x - min_val   # Subtract the first year
x = x / 4         # Olympics are every four years
```

```python
def get_covariance(X, variance):
    return variance * np.linalg.inv(X.T.dot(X))

X = get_design_matrix(x, 1)
w_hat = get_w_hat(X, t)
ss = get_variance(X, t.T, w_hat).mean()
cov = get_covariance(X, ss)
print('Covariance:\n', cov)
```

    Covariance:
     [[ 806.56277261   -2.36552444]
     [  -2.36552444   99.3232368 ]]

```python
import numpy.matlib as npml

def gausssamp(mu, sigma, N, sigmachol=None):
    if sigmachol is None:
        sigmachol = np.linalg.cholesky(sigma).T
    q = np.random.rand(mu.size, N)
    g = npml.repmat(np.atleast_2d(mu).T, 1, N) + sigmachol.dot(q)
    return g.T

co_w = ss * np.linalg.inv(X.T.dot(X))
samps = gausssamp(w_hat, co_w, 10).T

plt.figure()
for i in np.arange(10):
    plt.plot(x, X.dot(samps[:,i]))
plt.scatter(x, t)
plt.show()
```

## 2.3. Uncertainty in predictions

We load `synthentic_data.mat` which contains the variables `x`, `t`, `testx`.

```python
synthentic_data = sio.loadmat('synthetic_data.mat')
```

```python
x = np.array(synthentic_data['x'])
t = np.array(synthentic_data['t'])
test_x = np.array(synthentic_data['testx'])
```

```python
order = 7
X = get_design_matrix(x, order)
w_hat = get_w_hat(X, t)
ss = get_variance(X, t.T, w_hat).mean()

fig, ax = plt.subplots()
points = ax.scatter(x, t)
fit = ax.plot(x, X.dot(w_hat), color='green')

ax.set_xlabel('x')
ax.set_ylabel('t')

plt.show()
```

```python
def get_variance_test(X_test, X, variance):
    cov = get_covariance(X, variance)
    for i, x_new in enumerate(X_test):
        ss_new[i] = x_new.dot(cov).dot(x_new).mean()
    return ss_new
```

```python
t_new = get_predictions(x, t, test_x, order)
X_new = get_design_matrix(test_x, order)

ss_new = get_variance_test(X_new, X, ss)
```

Predictive means are called `t_new`, variances `v_new`. Both are vectors of the same size (and orientation!)

```python
fig, ax = plt.subplots()
points = ax.scatter(x, t)
err = ax.errorbar(test_x, t_new, yerr=ss_new, color='#ff8888')

ax.set_xlabel('x')
ax.set_ylabel('t')

plt.show()
```

TODO: What do you notice – is it what you expect?

We now repeat this experiment for `synthetic_data_2.mat`.

```python
synthentic_data_2 = sio.loadmat('synthetic_data_2.mat')
```

```python
x = np.array(synthentic_data_2['x'])
t = np.array(synthentic_data_2['t'])
test_x = np.array(synthentic_data_2['testx'])
```

```python
order = 7
X = get_design_matrix(x, order)
w_hat = get_w_hat(X, t)
ss = get_variance(X, t.T, w_hat).mean()

t_new = get_predictions(x, t, test_x, order)
X_new = get_design_matrix(test_x, order)

ss_new = get_variance_test(X_new, X, ss)

fig, ax = plt.subplots()
points = ax.scatter(x, t)
err = ax.errorbar(test_x, t_new, yerr=ss_new, color='#ff8888')

ax.set_xlabel('x')
ax.set_ylabel('t')

plt.show()
```

TODO: What do you notice?

## 3. Bayesian inference for a coin tossing experiment

### 3.1. Computing posteriors

The beta probability density function is defined as:

\begin{equation}
p(r|\alpha,\beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} r^{\alpha - 1}(1 - r)^{\beta - 1}
\end{equation}

where $0 \leq r \leq 1$ and $\alpha, \beta > 0$.

Posterior, which we aim to find is:

\begin{equation}
p(r | y_N, \alpha, \beta) \propto r^{\gamma - 1}(1 - r)^{\delta - 1}
\end{equation}

```python
import scipy.stats as spstats
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = (16.0, 8.0)
```

```python
class Coin:
    def __init__(self):
        self.__r = np.random.uniform(0, 1)
    def __call__(self, N):
        return np.sum([i <= self.r for i in np.random.rand(N, 1)])
    @property
    def r(self):
        return self.__r
```

```python
def plot_beta(alpha, beta, ax = None):
    x = np.arange(0, 1.01, 0.01)
    if ax is None:
        fig, ax = plt.subplots()
    ax.plot(x, spstats.beta.pdf(x, alpha, beta))
    return ax

def fairground():
    return sum([i < 0.8 for i in np.random.rand(10, 1)]) <= 6

plot_beta(2, 2)
plt.show()
```

```python
N = 10000
coin = Coin()
yN = coin(N)
alpha1 = 2
beta1 = 2
alpha2 = 1
beta2 = 2
gamma1 = yN + alpha1
delta1 = beta1 + N + yN
gamma2 = yN + alpha2
delta2 = beta2 + N + yN

fig, ax = plt.subplots()
plot_beta(alpha1, beta1, ax)
plot_beta(gamma1, delta1, ax)
plt.show()
```

```python
print('The coin bias was', coin.r)
```

```python
from scipy.misc import comb

n_look = 50
coin1 = Coin()
y_n = coin1(n_look)
alpha1 = 2
beta1 = 2
gamma1 = y_n + alpha1
delta1 = beta1 + n_look - y_n

N = 10
p_ynew_se_six = 0

for ynew in np.arange(6):
    b = comb(N, ynew)
    num = np.random.gamma(gamma1 + delta1) * np.random.gamma(gamma1 + ynew) * np.random.gamma(delta1 + N - ynew)
    den = np.random.gamma(gamma1) * np.random.gamma(delta1) * np.random.gamma(gamma1 + delta1 + N)
    p_ynew_se_six += (b * num) / den

print('The coin bias was', coin1.r)
print(p_ynew_se_six)
```


# Rotational transformation to ship fixed.

# Purpose
Wind and waves are given in world coordinates, how to transform them to ship fixed?

# Methodology
Using scipy to do the rotational transformation

# Setup

```python
# %load imports.py
from typing import no_type_check
%matplotlib inline
%load_ext autoreload
%autoreload 2

import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sympy as sp

```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
from scipy.spatial.transform import Rotation as R
```

## Example usage

```python
psi = np.linspace(0,360,100)
r = R.from_euler('z', psi, degrees=True)
coords = r.apply([1,0,0])

fig,ax = plt.subplots()
ax.plot(coords[:,0], coords[:,1],'-');
ax.axis('equal');

r2 = r.inv()
coords = r2.apply(coords)

fig,ax = plt.subplots()
ax.plot(coords[:,0], coords[:,1],'o');
ax.set_xlim(-1,1)
ax.set_ylim(-1,1)
```

```python
from sympy.vector import CoordSys3D
W = CoordSys3D(name='W')
psi = sp.symbols('psi')
B = W.orient_new_axis('B', psi, W.k)
W.rotation_matrix(B)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\psi \right)} & - \sin{\left(\psi \right)} & 0\\\sin{\left(\psi \right)} & \cos{\left(\psi \right)} & 0\\0 & 0 & 1\end{matrix}\right]$

```python

```