```python
from numpy import array
from sympy import symbols, Eq, Matrix, diff, Derivative, simplify, factor, expand, latex, init_printing, collect
init_printing()
from IPython.display import display, Math
```

## Define the interpolation functions

```python
x1, y1, x2, y2, x3, y3, x4, y4 = symbols('x_1, y_1, x_2, y_2, x_3, y_3, x_4, y_4')
r, s = symbols('r, s')

# Define the interpolation functions
h1 = factor(1/4*(1+r)*(1+s))
h2 = factor(1/4*(1-r)*(1+s))
h3 = factor(1/4*(1-r)*(1-s))
h4 = factor(1/4*(1+r)*(1-s))

display(Math('h_1 = ' + latex(h1)))
display(Math('h_2 = ' + latex(h2)))
display(Math('h_2 = ' + latex(h3)))
display(Math('h_2 = ' + latex(h4)))
```

# Relate the (x, y) Local Coordinate System to the (r, s) Natural Coordinate System

```python
# Relate the (x, y) coordinate system to the (r, s) coordinate system
x = h1*x1 + h2*x2 + h3*x3 + h4*x4
y = h1*y1 + h2*y2 + h3*y3 + h4*y4

display(Math('x = ' + latex(x)))
display(Math('y = ' + latex(y)))
```

# Find the Jacobian Matrix
The Jacobian matrix converts derivatives with respect to x and y into derivatives with respect to r and s

$J = \begin{bmatrix} \frac{dx}{dr} & \frac{dy}{dr} \\ \frac{dx}{ds} & \frac{dy}{ds} \end{bmatrix}$

```python
# Calculate the Jacobian matrix
J = Matrix([[diff(x, r), diff(y, r)],
            [diff(x, s), diff(y, s)]])

display(Math('J = ' + latex(factor(J))))
```

```python
B_kappa = Matrix([[0,      0,      -diff(h1, r), 0,      0,      -diff(h2, r), 0,      0,      -diff(h2, r), 0,      0,      -diff(h4, r)],
                  [0, diff(h1, s),       0,      0, diff(h2, s),       0,      0, diff(h3, s),       0,      0, diff(h4, s),       0     ],
                  [0, diff(h1, r), -diff(h1, s), 0, diff(h2, r), -diff(h2, s), 0, diff(h3, r), -diff(h3, s), 0, diff(h4, r), -diff(h4, s)]])

display(Math('B_\kappa = J^{-1}(1/4)' + latex(B_kappa*4)))
```

```python
dH = Matrix([[diff(h1, r), diff(h2, r), diff(h3, r), diff(h4, r)],
             [diff(h1, s), diff(h2, s), diff(h3, s), diff(h4, s)]])

B_m = Matrix([[diff(h1, r),      0,      diff(h2, r),      0,      diff(h3, r),      0,      diff(h4, r),      0     ],
              [     0,      diff(h1, s),      0,      diff(h2, s),      0,      diff(h3, s),      0,      diff(h4, s)],
              [diff(h1, s), diff(h1, r), diff(h2, s), diff(h2, r), diff(h3, s), diff(h3, r), diff(h4, s), diff(h4, r)]])

display(Math('dH = (1/4)' + latex(dH*4)))
display(Math('B_m = J^{-1}(1/4)' + latex(B_m*4)))
print(B_m*4)
```

# Calculate the Tensor Shear Strain Components
The tensor shear strain components can be calculated using Reference 1, equations 5.102:

$\gamma_{xz}=\gamma_{rz}sin{\beta}-\gamma_{sz}sin{\alpha}$

$\gamma_{yz}=-\gamma_{rz}cos{\beta}+\gamma_{sz}cos{\alpha}$

Where the valus for $\gamma_{rz}$ and $\gamma_{sz}$ can be found from equations 5.103:

$\gamma_{rz}=\frac{\sqrt{(C_x+rB_x)^2+(C_y+rB_y)^2}}{8|J|}([1+s][\frac{w_1-w_2}{2}+\frac{x_1-x_2}{4}(\theta_y^1+\theta_y^2)-\frac{y_1-y_2}{4}(\theta_x^1+\theta_x^2)]+[1-s][\frac{w_4-w_3}{2}+\frac{x_4-x_3}{4}(\theta_y^4+\theta_y^3)-\frac{y_4-y_3}{4}(\theta_x^4+\theta_x^3)])$

$\gamma_{sz}=\frac{\sqrt{(A_x+sB_x)^2+(A_y+sB_y)^2}}{8|J|}([1+r][\frac{w_1-w_4}{2}+\frac{x_1-x_4}{4}(\theta_y^1+\theta_y^4)-\frac{y_1-y_4}{4}(\theta_x^1+\theta_x^4)]+[1-r][\frac{w_2-w_3}{2}+\frac{x_2-x_3}{4}(\theta_y^2+\theta_y^3)-\frac{y_2-y_3}{4}(\theta_x^2+\theta_x^3)])$

In order to express the tensor shear strains in matrix form we'll need to collect the terms associated with the displacement matrix [d]. The coefficients associated with these terms form the [B] matrix for shear.

$\gamma = \begin{bmatrix} \gamma_{xz} \\ \gamma_{yz} \end{bmatrix} = \begin{bmatrix} B_{\gamma_{xz}} \\ B_{\gamma_{yz}} \end{bmatrix} \begin{bmatrix} w_1 \\ \theta_x^1 \\ \theta_y^1 \\ w_2 \\ \theta_x^2 \\ \theta_y^2 \\ w_3 \\ \theta_x^3 \\ \theta_y^3 \\ w_4 \\ \theta_x^4 \\ \theta_y^4 \end{bmatrix}$

It's easier to convert $\gamma_{rz}$ and $\gamma_{sz}$ (as opposed to $\gamma_{xz}$ and $\gamma_{yz}$) into matrix format by collecting [d] terms, and then apply equations 5.102 afterward:

```python
r, s = symbols('r, s')
h = symbols('h')
x1, x2, x3, x4 = symbols('x1, x2, x3, x4')
y1, y2, y3, y4 = symbols('y1, y2, y3, y4')
w1, w2, w3, w4 = symbols('w1, w2, w3, w4')
theta_x1, theta_x2, theta_x3, theta_x4 = symbols('theta_x1, theta_x2, theta_x3, theta_x4')
theta_y1, theta_y2, theta_y3, theta_y4 = symbols('theta_y1, theta_y2, theta_y3, theta_y4')

gamma_rz_terms = (1 + s)*((w1 - w2)/2 + (x1 - x2)/4*(theta_y1 + theta_y2) - (y1 - y2)/4*(theta_x1 + theta_x2)) \
               + (1 - s)*((w4 - w3)/2 + (x4 - x3)/4*(theta_y4 + theta_y3) - (y4 - y3)/4*(theta_x4 + theta_x3))

gamma_sz_terms = (1 + r)*((w1 - w4)/2 + (x1 - x4)/4*(theta_y1 + theta_y4) - (y1 - y4)/4*(theta_x1 + theta_x4)) \
                 + (1 - r)*((w2 - w3)/2 + (x2 - x3)/4*(theta_y2 + theta_y3) - (y2 - y3)/4*(theta_x2 + theta_x3))

display(1/4*collect(expand(gamma_rz_terms*4), [w1, theta_x1, theta_y1, w2, theta_x2, theta_y2, w3, theta_x3, theta_y3, w4, theta_x4, theta_y4]))
print('')
display(1/4*collect(expand(gamma_sz_terms*4), [w1, theta_x1, theta_y1, w2, theta_x2, theta_y2, w3, theta_x3, theta_y3, w4, theta_x4, theta_y4]))
```

# References

The following references were used in the formulation of this element. The primary reference was the first one because it had a section directly relating to isoparametric general plate bending elements, but all three were used. The first reference is a free download from MIT's website.

1. "Finite Element Procedures, 2nd Edition", Klaus-Jurgen Bathe
2. "Finite Element Analysis Fundamentals", Richard H. Gallagher
3. "A First Course in the Finite Element Method, 4th Edition", Daryl L. Logan


# **A brief intro to the most useful kind of algebra (linear)**
---

If you've ever wondered how R or python gives you regression coefficients, the answer is linear algebra! Linear algebra operations are essential to almost all modern methods for analyzing or modeling data. In this post I will first go through through some linear algebra basics, then build up to finding the ordinary least squares estimate for multiple linear regression. To try and make this self-contained, I will start from the basics. For the linear regression part, I will just assume knowledge of basic derivative rules, e.g., finding the derivative of $(a - bx)^2$ with respect to $x$.

# Building up to matrices
We begin with the building block of *scalars*, which here will refer to any real number. A variable referring to a scalar will be a lower-case letter such as $x$. For example, $x$ can refer to the recording of a neuroimaging sensor at a particular time, or in general the value of a sample's measurement.

We can arrange a list of scalars into a *vector*. We can enumerate the entries of a vector through a subscript, e.g., $x_1$ refers to the first entry of a vector. Vectors will be denoted with a bold-faced lower-case letter, $\mathbf{x}$, and can be visualized in the following way:

\begin{equation*}
    \mathbf{x} = \begin{bmatrix} x_{1} \\x_{2} \\ \vdots \\ x_{n} \end{bmatrix}
\end{equation*}

Where we have a list of $n$ scalars. The number of scalars in a vector $n$ is referred to as the *dimensionality* of the vector. A more mathematically formal way of stating this is that $\mathbf{x} \in \mathbb{R}^n$, which states that the vector $\mathbf{x}$ belongs to a set of all (real) vectors with dimension $n$.

Importantly, vectors can be thought of as an arrow in $n$-dimensional space. The arrow goes from the origin of the space to the coordinates listed in the vector. A vector's $L_2$ norm or *length* is denoted by:

\begin{equation*}
\Vert \mathbf{x} \Vert = \sqrt{\sum_{i=1}^{n}x_i^2}
\end{equation*}

Which is the Euclidean distance from the origin to the vector tip.

(An aside: As human beings, we're mostly limited to reasoning about 2-dimensional and 3-dimensional spaces. However, the data we collect is often higher-dimensional. The wonder of linear algebra is that it allows us us to reason and have intuitions about much higher-dimensional vectors and spaces.)

## Basic vector operations
Some basic things we can do with a vector are:
- multiply the vector with a scalar
- add the vector to another vector
- find a scalar value called the *dot product* or *inner product* of the vector with another vector
 - (there's also an *outer product*, but we'll come back to that in a bit)

Multiplying a vector with a scalar produces another vector, and can be thought of as scaling the vector in space (it points in the same direction but is larger or smaller):

\begin{equation*}
  a \mathbf{x} = \begin{bmatrix} ax_{1} \\ ax_{2} \\ \vdots \\ ax_{n} \end{bmatrix}
\end{equation*}

We can add vectors, provided they have the same dimension:

\begin{equation*}
  \mathbf{x + y} =
  \begin{bmatrix} x_{1} \\ x_{2} \\ \vdots \\ x_{n} \end{bmatrix} +
  \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} =
  \begin{bmatrix} x_1 + y_{1} \\ x_2 + y_{2} \\ \vdots \\ x_n + y_{n} \end{bmatrix}
\end{equation*}

We can define a dot product between two same-sized vectors, which is a scalar that indicates something about the angle between the vectors ($\theta$) in space.

\begin{align*}
  \mathbf{x \cdot y} &= x_1 y_{1} + x_2 y_{2} + \cdots + x_n y_{n} = \sum_{i=1}^{n} x_iy_i  \\
  &= \Vert \mathbf{x} \Vert \Vert \mathbf{y} \Vert \cos(\theta)
\end{align*}

Thus, if two vectors are oriented at $90^\circ$ relative to each other - that is, they are *orthogonal* - their dot product is $0$. This last formulation allows us to define the concept of angles in higher than 3-dimensional space.

The dot product of a vector with itself is its length squared:

\begin{equation}
  \mathbf{x \cdot x} = \Vert \mathbf{x} \Vert^2 = \sum_{i=1}^{n} x_i^2
\end{equation}

## Matrices
A list of vectors of the same dimension can form another mathematical object called a *matrix*, commonly denoted as upper-case bold letters, such as $\mathbf{X}$:

\begin{equation*}
  \mathbf{X} =
  \begin{bmatrix}
    x_{1,1} & x_{1,2} & \cdots & x_{1,p} \\
    x_{2,1} & x_{2,2} & \cdots & x_{2,p} \\
    \vdots & \vdots & \ddots & \vdots \\
    x_{n,1} & x_{n,2} & \cdots & x_{n,p}
  \end{bmatrix}
\end{equation*}

In most data matrices, $x_{n,p}$ is a scalar variable indicating the measurement of the $p^{th}$ dimension of the $n^{th}$ sample. As with vectors, we can formally state $\textbf{X} \in \mathbb{R}^{n \times p}$, indicating that the matrix $\textbf{X}$ has $n$ rows and $p$ columns. A more common way to state this is that the shape of $\mathbf{X}$ is $(n, p)$.

We can think of matrices as a list of column vectors. For example, we could think of our data matrix as:

\begin{equation*}
  \mathbf{X} =
  \left[ \begin{array}{c | c | c | c }
      x_{1,1} & x_{1,2} & \cdots & x_{1,p} \\
      x_{2,1} & x_{2,2} & \cdots & x_{2,p} \\
      \vdots & \vdots & \ddots & \vdots \\
      x_{n,1} & x_{n,2} & \cdots & x_{n,p} \end{array}
  \right] =
  \begin{bmatrix}
    | & | & & | \\
    \mathbf{x}_1 & \mathbf{x}_2 & \cdots & \mathbf{x}_p \\
    | & | & & |
  \end{bmatrix}
\end{equation*}

Here we have a list of $p$ vectors $\mathbf{x}_1, \dots, \mathbf{x}_p$, all of which are $n$-dimensional. In other words, each vector $\mathbf{x}_i$ represents all of the samples' data for whatever the $i^{\textrm{th}}$ dimension was measuring.

We could also think of a matrix as a list of row vectors. A row of $\mathbf{X}$ indicates all of the dimensions of a particular sample:

\begin{equation*}
  \mathbf{X} =
  \left[ \begin{array}{c c c c }
      x_{1,1} & x_{1,2} & \cdots & x_{1,p} \\
      \hline
      x_{2,1} & x_{2,2} & \cdots & x_{2,p} \\
      \hline
      \vdots & \vdots & \ddots & \vdots \\
      \hline
      x_{n,1} & x_{n,2} & \cdots & x_{n,p} \end{array}
  \right] =
  \begin{bmatrix}
    - \; \mathbf{x}_1^\top \; - \\
    - \; \mathbf{x}_2^\top \; - \\
    \vdots \\
    - \; \mathbf{x}_n^\top \; -\\
  \end{bmatrix}
\end{equation*}

Here we have $n$ row vectors (they are different than column vectors), which are denoted as $\mathbf{x}_1^\top, \dots, \mathbf{x_n}^\top$. These $n$ vectors represent one sample each, and are of dimension $p$ which indicates that there were $p$ dimensions measured.

If you're wondering where the $^\top$ symbol comes from, it's actually the *transpose* of a matrix. It is denoted by a superscript $\mathbf{X}^\top \in \mathbb{R}^{p \times n}$ and is an operation which turns each of the rows of $\mathbf{X}$ into columns of $\mathbf{X}^\top$. In other words, if $\mathbf{X}$ is of shape $(n,p)$, then $\mathbf{X}^\top$ will be of shape $(p,n)$:

\begin{equation*}
  \mathbf{X}^\top =
  \begin{bmatrix}
    x_{1,1} & x_{2,1} & \cdots & x_{n,1} \\
    x_{1,2} & x_{2,2} & \cdots & x_{n,2} \\
    \vdots & \vdots & \ddots & \vdots \\
    x_{1,p} & x_{2,p} & \cdots & x_{n,p}
  \end{bmatrix}
\end{equation*}

Here we now have dimensions measured as columns and samples as rows. In essence, row vectors are transposes of column vectors.

```python
# working with vectors and matrices in numpy
import numpy as np

# defining vectors (1d array)
x = np.array([0, 1, 2, 3])
y = np.array([1, 2, 3, 4])

# somewhat confusingly, vectors (1d arrays) in numpy are printed horizontally
print('x is:\t', x)
print('y is:\t', y)
print('x shape:', x.shape)
print('---------')

# operations
print('5x = \t', 5*x)
print('x + y =\t', x+y)
print('x * y =\t', x.dot(y))
print('---------')

# defining a matrix (2d array)
n, p = 5, 3 # 5 samples, 3 dims
X = np.random.randint(10, size=(n,p))   # random integer matrix
print('X looks like:\n', X)
print('X is of shape: ', X.shape)
print('---------')

# transpose
print('X transpose:\n', X.T)            # numpy syntax for transpose
print('X transpose shape: ', X.T.shape)
```

    x is:	 [0 1 2 3]
    y is:	 [1 2 3 4]
    x shape: (4,)
    ---------
    5x = 	 [ 0  5 10 15]
    x + y =	 [1 3 5 7]
    x * y =	 20
    ---------
    X looks like:
     [[4 9 6]
     [2 1 7]
     [0 1 3]
     [0 9 9]
     [0 5 6]]
    X is of shape:  (5, 3)
    ---------
    X transpose:
     [[4 2 0 0 0]
     [9 1 1 9 5]
     [6 7 3 9 6]]
    X transpose shape:  (3, 5)

# Matrix-vector multiplication
Multiplying a matrix with a vector results in a new vector. A way to intuit matrix-vector multiplication is to consider the matrix as a a "machine" which acts on the input vector and produces an output vector of a particular dimension. In this case, the matrix must have the dimensionality of the row vectors be equal to the dimensionality of the input vector.

There are two ways to frame matrix-vector multiplications, which result from thinking about the matrix as a either a collection of column vectors or row vectors.
- We can get the output vector by weighting the columns of the matrix with the entries of the input vector and summing these weighted vectors.
- Alternatively, we can get each element of the output vector by doing the dot product of the rows of the matrix with the input vector.

Both perspectives will be useful for the rest of this post. Either way, if we have a size $(n, p)$ matrix, that means we can do matrix-vector multiplication by giving it a $p$-dimensional vector, and our output will be an $n$-dimensional vector. To illustrate, let's define a $(2 \times 3)$ matrix $\mathbf{A}$ and a $3$-dimensional vector $\mathbf{b}$:

\begin{align*}
  \mathbf{A} =
  \begin{bmatrix}
    \color{SteelBlue}{a_{1,1}} & \color{DarkGoldenrod}{a_{1,2}} & \color{green}{a_{1,3}} \\
    \color{DeepSkyBlue}{a_{2,1}} & \color{orange}{a_{2,2}} & \color{SpringGreen}{a_{2,3}} \\
  \end{bmatrix},
  \quad \quad
  \mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix}
\end{align*}

The different colors (blue, orange, green) are different columns, and darker/lighter are different rows. Now, we are looking to find the output vector $\mathbf{c = Ab}$:

\begin{equation}
  \mathbf{c} =
  \begin{bmatrix}
    \color{SteelBlue}{a_{1,1}} & \color{DarkGoldenrod}{a_{1,2}} & \color{green}{a_{1,3}} \\
    \color{DeepSkyBlue}{a_{2,1}} & \color{orange}{a_{2,2}} & \color{SpringGreen}{a_{2,3}} \\
  \end{bmatrix} \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix}
\end{equation}

How do we actually multiply these things together? Let's see the first way of framing matrix-vector multiplication, which is to consider weighting the columns of $\mathbf{A}$ with the elements of $\mathbf{b}$:

\begin{equation*}
  \mathbf{c} =
  \begin{bmatrix} c_{1} \\ c_{2} \end{bmatrix} =
  b_1 \color{SteelBlue}{{\begin{bmatrix} a_{1,1} \\ \color{DeepSkyBlue}{a_{2,1}} \end{bmatrix}}} +
  b_2 \color{DarkGoldenrod}{{\begin{bmatrix} a_{1,2} \\ \color{orange}{a_{2,2}} \end{bmatrix}}} +
  b_3 \color{green}{{\begin{bmatrix} a_{1,3} \\ \color{SpringGreen}{a_{2,3}} \end{bmatrix}}} =
  \mathbf{Ab}
\end{equation*}

For this case, imagine taking the elements of $\mathbf{b}$ and putting them in front of the columns of $\mathbf{A}$ as scalar multiples, then summing up those scaled columns. In some sense, the columns of matrix $\mathbf{A}$ provide a set of basis patterns which are weighted and summed to produce $\mathbf{c}$.

If we do a little rearranging of the above definition, we can get the other way of framing matrix-vector multiplication:

\begin{equation}
  \begin{bmatrix} c_1 \\ c_2 \end{bmatrix} =
  \begin{bmatrix}
    b_1 \color{SteelBlue}{a_{1,1}} + b_2 \color{DarkGoldenrod}{a_{1,2}} + b_3 \color{green}{a_{1,3}} \\
    b_1 \color{DeepSkyBlue}{a_{2,1}} + b_2 \color{orange}{a_{2,2}} + b_3 \color{SpringGreen}{a_{2,3}}
  \end{bmatrix}
\end{equation}

Where

\begin{align}
  c_1 &=
  b_1 \color{SteelBlue}{a_{1,1}} + b_2 \color{DarkGoldenrod}{a_{1,2}} + b_3 \color{green}{a_{1,3}} =
   \begin{bmatrix} \color{SteelBlue}{a_{1,1}} \\ \color{DarkGoldenrod}{a_{1,2}} \\ \color{green}{a_{1,3}} \end{bmatrix} \cdot \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix} \\ \\
  c_2 &= b_1 \color{DeepSkyBlue}{a_{2,1}} + b_2 \color{orange}{a_{2,2}} + b_3 \color{SpringGreen}{a_{2,3}} =
  \begin{bmatrix} \color{DeepSkyBlue}{a_{2,1}} \\ \color{orange}{a_{2,2}} \\ \color{SpringGreen}{a_{2,3}} \end{bmatrix} \cdot \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix}
\end{align}

That is, the elements of $\mathbf{c}$ are the dot products of the row vectors of $\mathbf{A}$ with $\mathbf{b}$. For this case, imagine taking the vector $\mathbf{b}$ and putting it sideways on top of the matrix $\mathbf{A}$. Then, the first element of $\mathbf{c}$ is the dot product of $\mathbf{b}$ with the first row of $\mathbf{A}$, the second element of $\mathbf{c}$ is the dot product of $\mathbf{b}$ with the second row of $\mathbf{A}$, and so on.

To recap, and in general for multiplying an $(n,p)$ matrix and a $p$ dimensional vector:

\begin{align*}
  \mathbf{c} &= \mathbf{Ab} \\
  \begin{bmatrix} c_1 \\ c_2 \\ \vdots \\ \vdots \\ c_n \end{bmatrix}
  &= \begin{bmatrix}
    | & | & | \\
    | & | & | \\
    \mathbf{a}_1 & \cdots & \mathbf{a}_p \\
    | & | & | \\
    | & | & | \\
  \end{bmatrix}
  \begin{bmatrix} b_{1} \\ \vdots \\ b_{p} \end{bmatrix} =
  b_1 \mathbf{a}_1 + \cdots + b_p \mathbf{a}_p \\ \\
  \begin{bmatrix} c_1 \\ c_2 \\ \vdots \\ \vdots \\ c_n \end{bmatrix} &= \begin{bmatrix}
    - \; \mathbf{a}_1^\top \; - \\
    - \; \mathbf{a}_2^\top \; - \\
    \vdots \\ \vdots \\
    - \; \mathbf{a}_n^\top \; - \\
  \end{bmatrix} \mathbf{b} =
  \begin{bmatrix} \mathbf{a}_1^\top\mathbf{b} \\ \mathbf{a}_2^\top\mathbf{b} \\ \vdots \\ \vdots \\ \mathbf{a}_n^\top\mathbf{b} \end{bmatrix}
\end{align*}

Either way, the output is an $n$-dimensional vector.

### Vectors and vector operations as 1-dimensional matrices
Imagine we have a single row vector $\mathbf{a}^\top$ and a column vector $\mathbf{b}$, both of dimension $n$. If we consider the vector $\mathbf{a}^\top$ to be a size $(1, n)$ matrix and $\mathbf{b}$ to be an $n$ dimensional vector, we can see that the matrix-vector product $\mathbf{a}^\top\mathbf{b}$ should be a size $(1 \times 1)$ vector, otherwise known as a scalar. As it turns out, we've already seen this before as the dot product (or inner product):

\begin{align*}
  \mathbf{a} \cdot \mathbf{b} &= \mathbf{a}^\top \mathbf{b} \\
  \sum_{i=1}^{n} a_n b_n &= \begin{bmatrix} a_1 & a_2 & \cdots & a_n \end{bmatrix}
  \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix}
\end{align*}

Let's now consider the outer product, sometimes denoted $\mathbf{a} \otimes \mathbf{b}$:

\begin{align*}
  \mathbf{a} \otimes \mathbf{b} =
  \mathbf{a} \mathbf{b}^\top =
  \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix}
  \begin{bmatrix} b_1 & b_2 & \cdots & b_n \end{bmatrix}
\end{align*}

What might it mean to multiply an $(n,1)$ matrix with a $(1,n)$ matrix?

## Matrix-matrix multiplication
We can now define how to multiply two matrices $\mathbf{A}$ and $\mathbf{B}$ together to get a matrix $\mathbf{C = AB}$. In fact, matrix-matrix multiplication is equivalent to doing a series of matrix-vector multiplications. That is, imagine $\mathbf{B}$ as a series of column vectors $\mathbf{b}_1, \cdots, \mathbf{b}_k$. Then, each column vector $\mathbf{c}_i$ will be equal to $\mathbf{Ab}_i$:

\begin{align*}
  \mathbf{C} &= \mathbf{AB} \\
  \begin{bmatrix}
    c_{1,1} & c_{1,2} & \cdots & c_{1,k} \\
    c_{2,1} & c_{2,2} & \cdots & c_{2,k} \\
    \vdots & \vdots & \ddots & \vdots \\
    c_{n,1} & c_{n,2} & \cdots & c_{n,k}
  \end{bmatrix}
  &= \begin{bmatrix}
    a_{1,1} & \cdots & a_{1,p} \\
    a_{2,1} & \cdots & a_{2,p} \\
    \vdots & \ddots & \vdots \\
    a_{n,1} & \cdots & a_{n,p}
  \end{bmatrix}
  \begin{bmatrix}
    b_{1,1} & b_{1,2} & \cdots & b_{1,k} \\
    \vdots & \vdots & \ddots & \vdots \\
    b_{p,1} & b_{p,2} & \cdots & b_{p,k}
  \end{bmatrix} \\ \\
  \begin{bmatrix} \mathbf{c}_1 & \mathbf{c}_2 & \cdots & \mathbf{c}_k \end{bmatrix}
  &= \begin{bmatrix} \mathbf{Ab}_1 & \mathbf{Ab}_2 & \cdots & \mathbf{Ab}_k \end{bmatrix}
\end{align*}

Thus, the first column of $\mathbf{C}$ is simply $\mathbf{Ab}_1$. This reveals one rule of matrix-matrix multiplication: the so-called *inner dimensions* of the matrices must agree in order for the multiplication to be defined. That is, $\mathbf{C = AB}$ is only defined when the matrix $\mathbf{A}$ has shape $(n, k)$ and the matrix $\mathbf{B}$ has shape $(k, p)$ for any $n$ and $p$. In other words, the rows of $\mathbf{A}$ must be of the same dimension as the columns of $\mathbf{B}$. Then, the output $\mathbf{C}$ will be of shape $(n,p)$.

Importantly, this tells us that matrix multiplication is *not* commutative. That is, $\mathbf{AB} = \mathbf{BA}$ is *not* generally true. In fact, if $\mathbf{B}$ is size $(k,p)$ and $\mathbf{A}$ is size $(n,k)$, then $\mathbf{BA}$ isn't even defined, as the inner dimensions do not agree. Thus, *left* multiplying a matrix is fundamentally different than *right* multiplying a matrix.

## The identity matrix and matrix inverses
An identity matrix is a matrix with ones along the diagonal and zeros everywhere else. For example, size $2$ identity matrix is defined as:

\begin{equation}
  \mathbf{I}_2 =
  \begin{bmatrix}
    1 & 0 \\
    0 & 1
  \end{bmatrix}
\end{equation}

In general:

\begin{equation}
  \mathbf{I}_n =
  \begin{bmatrix}
    1 & 0 & \cdots & 0 \\
    0 & 1 & \cdots & 0 \\
    \vdots & \vdots & \ddots & \vdots \\
    0 & 0 & \cdots & 1
  \end{bmatrix}
\end{equation}

We can think of the identity matrix as the matrix version of a 1 for matrix multiplication. In other words, $1*a = a$. Similarly, $\mathbf{Ib} = \mathbf{b}$, and $\mathbf{IA} = \mathbf{AI} = \mathbf{A}$.

The inverse of a matrix only exists for square matrices (those with size $(n,n)$), and only certain square matrices at that. An inverse $\mathbf{A}^{-1}$ of a matrix $\mathbf{A}$ is defined such that $\mathbf{A}^{-1} \mathbf{A} = \mathbf{A} \mathbf{A}^{-1} = \mathbf{I}$. If we have an equation like $\mathbf{Ax} = \mathbf{b}$, and we wish to get $\mathbf{x}$ alone, we can *left* multiply by $\mathbf{A}^{-1}$:

\begin{align}
  \mathbf{Ax} &= \mathbf{b} \\
  \mathbf{A}^{-1} \mathbf{Ax} &= \mathbf{A}^{-1} \mathbf{b} \\
  \mathbf{x} &= \mathbf{A}^{-1} \mathbf{b}
\end{align}

```python
# matrix-vector
n, p = 2, 3                       # 2 rows, 3 cols
A = np.arange(n*p).reshape((n,p)) # fast way to get 2d array with ordered ints
b = np.arange(p)                  # vectors don't have a second dimension in numpy
c = A @ b                         # syntax for matrix multiplication in numpy

print('A:\n', A)
print('------')
print('b:\n', b)   # remember b is a column vector, despite how it's printed
print('------')
print('c:\n', c)   # same with c
```

    A:
     [[0 1 2]
     [3 4 5]]
    ------
    b:
     [0 1 2]
    ------
    c:
     [ 5 14]

```python
# let's try multiplying A with a 4-dim vector for fun
x = np.arange(4)
c = A.dot(x)  # another syntax for matrix multiplication in numpy
```

```python
# matrix-matrix
n, p, k = 5, 3, 2
A = np.arange(n*p).reshape((n,p))
B = np.arange(p*k).reshape((p,k))
C = A @ B

print('A shape:', A.shape, '\n', A)
print('------')
print('B shape:', B.shape, '\n', B)
print('------')
print('C shape:', C.shape, '\n', C)
print('------')

# let's get first col of C:
b_1 = B[:, 0]
c_1 = A.dot(b_1)
print('c_1:\n', c_1)
```

    A shape: (5, 3)
     [[ 0  1  2]
     [ 3  4  5]
     [ 6  7  8]
     [ 9 10 11]
     [12 13 14]]
    ------
    B shape: (3, 2)
     [[0 1]
     [2 3]
     [4 5]]
    ------
    C shape: (5, 2)
     [[ 10  13]
     [ 28  40]
     [ 46  67]
     [ 64  94]
     [ 82 121]]
    ------
    c_1:
     [10 28 46 64 82]

# Estimating linear regression coefficients with matrix/vector operations
Let's put all the above to work to help us do linear regression.

We are all familiar with the most simplified univariate model: $y = mx + b$. To generalize this, we can consider more than one independent (input) variable by considering a model where we have a vector of $n$ observations $\mathbf{x}$ instead of a single observation $x$ and a vector of $n$ weights $\mathbf{b}$ instead of the single weight $m$. For instance, consider the model prediction for a single observation of $n$ regressors $x_1, x_2, \dots, x_p$:

\begin{equation*}
\hat{y}_i = b_0 + b_1 x_{1,i} + b_2 x_{2,i} + \cdots + b_p x_{p, i}
\end{equation*}

There's a lot of indices in that equation! We can simplify the notation using vectors, as this model is defined as the dot product of the regressors and the weights. We can also include the intercept $b_0$ in the vector of weights $\mathbf{b}$, so that it is of shape $(n, p+1)$:

\begin{align}
\hat{y}_i &= \mathbf{b} \cdot  \mathbf{x}_i = \mathbf{b}^\top \mathbf{x}_i
\end{align}

Where

\begin{equation*}
  \mathbf{b}^\top = \begin{bmatrix} b_0 & b_1 & b_2 & \cdots & b_p \end{bmatrix}
  \quad \textrm{and} \quad
  \mathbf{x}_i = \begin{bmatrix} 1 \\ x_{1,i} \\ x_{2,i} \\ \vdots \\ x_{p,i} \end{bmatrix}
\end{equation*}

If we want to estimate the coefficients $\mathbf{b}$ we can do a bit more linear algebra and some calculus to find the solution as an optimization problem. More specifically, we first have to define the objective, i.e., the thing we want to minimize. It stands to reason that we want to minimize the error of our model predictions over all our data. Then, we take the derivative of the error with respect to the parameters we want to find (the coefficients), and set that expression equal to 0. Finally, we find the values of $\mathbf{b}$ which solve those derivative equations.

To begin, let's define the error for sample $i$ as:

\begin{align*}
  \epsilon_i &= y_i - \hat{y}_i \\
  \epsilon_i &= y_i - \mathbf{b}^\top \mathbf{x}_i\\
  y_i &= \mathbf{b}^\top \mathbf{x}_i + \epsilon_i
\end{align*}

Now we need to find values for our coefficients $\mathbf{b}$ that minimize this loss over all samples. For a few reasons, we will minimize the sum of squared errors:

\begin{align}
  \underset{\mathbf{b} \in \mathbb{R}^n}{\textrm{minimize}} \sum_{i=1}^{p} \epsilon_i^2 =
  \sum_{i=1}^{p} (y_i - \mathbf{b}^\top \mathbf{x}_i)^2
\end{align}

We could continue solving this optimization problem by taking the derivative with respect to $b_0, \dots, b_p$, setting it to 0, and solving for the values of $\mathbf{b}$.  It would get a bit ugly with all the algebra, so the vector and matrix notation can help us out a bit here by stacking all of our $n$ observations. Define the $(n, 1)$ vector $\mathbf{y}$ as the vector of all the observed responses, the $(n, p+1)$ matrix $\mathbf{X}$ as the *design matrix* of observed regressors, and the $(n, 1)$ vector $\boldsymbol{\epsilon}$ as the error of all the samples:

\begin{align*}
  \mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ \vdots \\ y_n \end{bmatrix},
  \quad
  \mathbf{X} =
  \begin{bmatrix}
    1 & x_{1,1} & \cdots & x_{1,p} \\
    1 & x_{2,1} & \cdots & x_{2,p} \\
    \vdots & \vdots & \ddots & \vdots \\
    \vdots & \vdots & \ddots & \vdots \\
    1 & x_{n,1} & \cdots & x_{n,p}
  \end{bmatrix},
  \quad
  \mathbf{b} = \begin{bmatrix} b_0 \\ b_1 \\ \vdots \\ b_p \end{bmatrix},
  \quad
  \boldsymbol{\epsilon} = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \vdots \\ \epsilon_n \end{bmatrix}
\end{align*}

Recall that the column of $1$'s is there to account for the intercept $b_0$. Then,

\begin{align*}
  \mathbf{y} &= \mathbf{X} \mathbf{b} + \boldsymbol{\epsilon}
\end{align*}

That's pretty much the univariate multiple linear regression model. We can now rewrite the optimization problem. Recall that we want to minimize the sum of squared errors. As it turns out, this is equivalent to the dot product of $ \boldsymbol{\epsilon}$ with itself:

\begin{equation*}
\text{SSE} = \sum_{i=1}^{n} \epsilon_i^2 = \boldsymbol{\epsilon}^\top \boldsymbol{\epsilon}
\end{equation*}

Now our minimization now looks like this:

\begin{equation}
\underset{\mathbf{b} \in \mathbb{R}^n}{\textrm{minimize}} \quad \boldsymbol{\epsilon}^\top \boldsymbol{\epsilon}
\end{equation}

The setup is finally done. Now we substitute $\boldsymbol{\epsilon} = \mathbf{y} - \mathbf{X} \mathbf{b}$, take the derivative, set to 0, and solve for our coefficients. If you're worried about taking the derivative of an expression with vectors and matrices, it's not actually all that bad. Vector derivative rules share many similarities to the univariate derivative rules we learned in basic calculus. First, let's consider the derivative of $(y-xb)^2$ with respect to $b$:

\begin{align*}
  (y-xb)^2 &= (y - xb)(y - xb) &\quad \\
  &= y^2 - yxb - xby + x^2b^2 &\quad \text{multiply out}\\
  &= y^2 - 2bxy + x^2b^2 &\quad \text{middle terms are the same} \\
  \frac{d}{db}(y - xb)^2 &= \frac{d}{db}y^2 - \frac{d}{db}2bxy + \frac{d}{db}x^2b^2 &\quad \text{distribute derivative} \\
  &= 0 - 2xy + 2x^2b &\quad \text{derivative rules} \\
\end{align*}

Now back to our objective function for multiple linear regression:

\begin{align}
{\epsilon}^\top \boldsymbol{\epsilon} &= (\mathbf{y} - \mathbf{X} \mathbf{b})^\top (\mathbf{y} - \mathbf{X} \mathbf{b}) \\
  &= (\mathbf{y}^\top - (\mathbf{X}\mathbf{b})^\top)(\mathbf{y} - \mathbf{X} \mathbf{b}) &\quad \textrm{ distribute transpose} \\
  &= (\mathbf{y}^\top - \mathbf{b}^\top \mathbf{X}^\top)(\mathbf{y} - \mathbf{X} \mathbf{b}) &\quad \textrm{transpose again}  \\
  &= \mathbf{y}^\top\mathbf{y} - \mathbf{y}^\top \mathbf{X} \mathbf{b} - \mathbf{b}^\top \mathbf{X}^\top \mathbf{y} - \mathbf{b}^\top \mathbf{X}^\top \mathbf{X} \mathbf{b} &\quad \text{multiply out} \\
  &= \mathbf{y}^\top\mathbf{y} - 2 \mathbf{b}^\top \mathbf{X}^\top \mathbf{y} - \mathbf{b}^\top \mathbf{X}^\top \mathbf{X} \mathbf{b} &\quad \text{middle terms are sneakily the same} \\
  \frac{\partial}{\partial\mathbf{b}} \boldsymbol{\epsilon}^\top\boldsymbol{\epsilon}
  &= \frac{\partial}{\partial\mathbf{b}}
      \left( \mathbf{y}^\top\mathbf{y} \right) - \frac{\partial}{\partial\mathbf{b}} \left( 2\mathbf{b}^\top\mathbf{X}^\top\mathbf{y} \right) - \frac{\partial}{\partial\mathbf{b}} \left( \mathbf{b}^\top\mathbf{X}^\top\mathbf{X}\mathbf{b} \right) &\quad \textrm{distribute derivative} \\
  &= 0 -2\mathbf{X}^\top\mathbf{y} - 2 \mathbf{X}^\top\mathbf{X}\mathbf{b} &\quad \textrm{derivative rules}
\end{align}

We're almost there! Now we just set the derivative equal to 0 and solve for the values of $\mathbf{b}$:

\begin{align}
  0 &= -2\mathbf{X}^\top\mathbf{y} - 2 \mathbf{X}^\top\mathbf{X}\mathbf{b} \\
  \mathbf{X}^\top\mathbf{X}\mathbf{b} &= \mathbf{X}^\top\mathbf{y} &\quad \textrm{getting b by itself} \\
  \hat{\mathbf{b}} &= \left(\mathbf{X}^\top\mathbf{X}\right)^{-1} \mathbf{X}^\top\mathbf{y} &\quad \textrm{left multiply by inverse to get b} \\\\
\end{align}

And that's how we get the OLS estimator of the coefficients for multiple linear regression.

## Multivariate multiple linear regression
We can further define multivariate multiple linear regression, where there are multiple measured dependent variables and independent variables. In matrix form, we get something like this:

\begin{equation}
\mathbf{Y} = \mathbf{XB + E}
\end{equation}

The response variables are now in a matrix $\mathbf{Y}$, shape $(n, k)$, to account for the $k$-dimensional response observed per sample. The coefficient weights are also a matrix $\mathbf{B}$, shape $(p+1, k)$, to accomodate weights for each of the dimenions of the response variables. $\mathbf{E}$ is now the error matrix, which gives us the error for all dimensions of all samples. Finally, the design matrix $\mathbf{X}$ doesn't change at all.

If we decide to fit the coefficient matrix $\mathbf{B}$ by minimizing the squared loss, as we did for the univariate multiple regression, we get something that looks familiar:

\begin{equation}
\hat{\mathbf{B}} = \left(\mathbf{X}^\top\mathbf{X}\right)^{-1}\mathbf{X}^\top\mathbf{Y}
\end{equation}

Interpreting these coefficients might be hard, but estimating them can actually be pretty straightforward!

Finally, let's see a toy example of this in action. I'll do a simple linear regression model: $\mathbf{y} = b_0 + b_1\mathbf{x}$.

```python
# simulating simple linear regression
import matplotlib.pyplot as plt   # for plotting

# pick random b_0 and b_1 - let's see how we recover these
b = np.array([3.4, 12.35])    # slope, intercept
b = b[:, np.newaxis]

# simulate n datapoints. the x is normally distributed
n = 30
data_mean = 0
data_std = 1
data = np.random.normal(data_mean, data_std, size=(n,1)) # make it 2d array for ease

# remember that since we have b_0 in our vector weights, we add a column of 1s to our data matrix
ones = np.ones((n,1))
x = np.hstack((ones, data)) # x is our design matrix

# add gaussian noise
noise_loc = 0
noise_scale = 5
e = np.random.normal(loc=noise_loc, scale=noise_scale, size=(n,1))

# simulate y values
y = x.dot(b) + e

# get estimates for b, predictions
b_hat = np.linalg.inv(x.T @ x) @ x.T @ y
y_hat = x.dot(b_hat)

# R^2
SS_res = e.T @ e
std = y - y.mean()
SS_tot = std.T @ std
r2 = 1 - (SS_res / SS_tot)

# plot data
fig, ax = plt.subplots()
ax.scatter(data, y) # plotting data
ax.set(xlabel='x', ylabel='y', title='simple linear regression')

# plot true line
b_0, b_1 = b
ax.plot(data, b_0 + (b_1 * data), color='k', label='ground truth')

# plot our fit
ax.plot(data, y_hat, color='r', label='our fit')
ax.legend()

print('true b:\n', b)
print('our estimate:\n', b_hat)
print('r^2:', r2)
```


# Analytic approx. for filters

The aim here is to derive analytic formulae for products of the filtering, given $W(kR)$ models and (very) simple $P(k)$. These will be useful for basic testing (against known analytic solution), but also, if $P(k)$ can be set close enough to reasonable models, for checking appropriate resolution/limits for integration.

Our main targets will be the mass variance:

$$ \sigma^2_n(r) = \frac{1}{2\pi^2} \int_0^\infty dk\ k^{2(1+n)} P(k) W^2(kR), $$

and the log derivative:

$$ \frac{d\ln \sigma^2}{d\ln R} = \frac{1}{\pi^2\sigma^2} \int_0^\infty W(kR) \frac{dW(kR)}{d\ln(kR)} P(k)k^2 dk. $$

Typically we'll use a power-law for the power spectrum,

$$ P(k) = k^p. $$

```python
from sympy import *
init_session()
p = symbols("p")
k, x, R, P = symbols('k x R P',positive=True)
```

    IPython console for SymPy 1.0 (Python 2.7.12-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

```python
def sigma(W, n,p,kmin=0,kmax=1):
    P = k**p
    #return k**(2*(1+n)) * P * W**2/(2*pi**2)
    integ = k**(2*(1+n)) * P * W**2/(2*pi**2)
    integ = integ.subs(x,k*R)
    res = integrate(integ,(k,kmin,kmax))
    print res
    return res

def dw_dlnkr(W):
    return x*diff(W,x)

def dlnss_dlnr(W,p,kmin=0,kmax=1):
    P = k**p
    dwdlnx = dw_dlnkr(W)
    integ = (W * dwdlnx * P * k**2).subs(x,k*R)
    s = sigma(W,0,p,kmin,kmax)
    res = integrate(integ,(k,kmin,kmax))/(pi**2*s)
    print res
    return res
```

## TopHat

In this case, we have

$$ W(kR) = 3\frac{\sin x - x\cos x}{x^3}. $$

```python
W = 3*(sin(x) - x*cos(x))/x**3
```

```python
sigma(W,0,2,0,1)
```

```python
sigma(W,1,2)
```

## SharpK

In this case, we have

$$ W(kR) = \begin{cases} 1 & kR \geq 1 \\ 0 & kR < 1 \end{cases}. $$

This renders the solution very simple:

$$ \sigma^2(R) = \frac{1}{2\pi^2} \int_0^{1/R} k^{2(1+n)} k^p = \frac{1}{2\pi^2}\frac{1}{tR^t}, $$

where $t = 2(1+n) + p + 1$.

And

$$ \frac{d\ln \sigma^2}{d\ln r} = \frac{-1}{2\pi^2 \sigma^2 R^{3+p}}. $$

## Gaussian

In this case we have

$$ W(x=kR) = \exp(-x^2/2). $$

```python
W = exp(-x**2/2)
```

```python
sigma(W,0,-y,0,oo)
```

```python
sigma(W,1,2,0,oo)
```

```python
dlnss_dlnr(W,2,0,oo)
```


# Data Fitting Exercises

# 1. Linear Regression

A common part of data analysis is **fitting** a mathematical model to some experimental data. This might be to determine how well a particular model describes your experimental data, or to extract a numerical parameter; for example, fitting rate laws to reactant concentrations over time to obtain rate constants. Another example is the semester 1 exercise, where you fitted CO$_2$ vapour pressures to the Clausius-Clapeyron equation to obtain phase transition enthalpies.

A third example where fitting to simple exerimental data can give chemical information about a particular reaction involves the **Van't Hoff equation**. This relates the change in the equilibrium constant, $K$, for a reaction, to a change in temperature, providing the enthalpy change for the reaction, $\Delta H$, is constant:

\begin{equation}
\frac{\mathrm{d}\,\ln{K}}{\mathrm{d}\,T} = \frac{\Delta H}{RT^2}.\tag{1}
\end{equation}

The derivation of this equation comes from combining two standard equations for the Gibbs free energy change, $\Delta G$, for a reaction:

\begin{equation}
\Delta G = \Delta H - T \Delta S\tag{2}
\end{equation}

\begin{equation}
\Delta G = -RT \ln{K}\tag{3}
\end{equation}

Combining these two equations, we get

\begin{equation}
\ln{K} = -\frac{\Delta H}{RT} + \frac{\Delta S}{R}.\tag{4}
\end{equation}

This is the **linear** form of the Van't Hoff equation. Differentiating with respect to $\frac{1}{T}$ (keeping $\Delta H$ and $\Delta S$ constant) gives equation 1.

Because the linear form of the Van't Hoff equation gives the equation for a straight line (hence “linear”) a plot of $\ln{K}$ against $\frac{1}{T}$ should give a straight line, as long as the enthalpy and entropy of reaction are approximately constant over the temperature range of interest. Furthermore, a straight line fitted to these data will have a slope of $-\frac{\Delta H}{R}$ and an intercept of $\frac{\Delta S}{R}$. By performing a linear fit, it is therefore possible to extract values for $\Delta H$ and $\Delta S$.

In the semeseter 1 lab, you learned how to use the `linregress()` function (from `scipy.stats`) to perform a linear regression (fit a straight line to a set of data). This exercises will work through model fitting in a bit more detail, divided into three parts.
1. In the first part, you will have the opportunity to revise and practice working with data, and using `scipy.stats.linregress` to fit a straight line, to obtain the experimental reaction enthalpy and entropy for an equilibrium.
2. The second part will go into more detail about what we mean when we talk about “fitting&rqduo; a straight line, and will show you a different, more general, approach to solving the same problem, that involves writing your own functions.
3. In the third part, you will apply what you have learned to fit a **non-linear** model to some experimental data, to analyse a flash-photolysis experiment.

Review the basics of using matplotlib to plot graphs, and using scipy.stats.linregress for linear regression, using your notebooks from last semester.

## Exercise

The file `data/equilibrium_constant.dat` contains a set of equilibrium constants measured at different temperatures for the reaction

\begin{equation}
2\mathrm{NO}_2 \rightleftharpoons \mathrm{N}_2\mathrm{O}_4.
\end{equation}

By performing a linear regression with these data, calculate the enthalpy and entropy changes for this reaction.

### Instructions

The first thing you will need to do is set up your notebook for working with data, by importing `numpy` and the plotting functions from `matplotlib`.
>```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python

```

### Process

The data from this experiment are stored in a text file in `data/equilbirium_constant.dat`, which looks like

```
# equilibrium constant data for 2 NO2 => N2O4
# columns are: temperature (degrees Celsius), K

9   34.3
20  12
25  8.79
33  4.4
40  2.8
52  1.4
60  0.751
70  0.4
```

To be able to work with the temperature and equilibrium constant data, you want to load them into `numpy` arrays. You can read the data from the file using the `np.loadtxt()` function, e.g.

>```python
    data = np.loadtxt('data/equilibrium_constant.dat')
```

Load the dataset from the file into a numpy array called data.
Check that this is a 2D array containing both the temperatures and equilibrium constants.

```python

```

To make it easier to work with the temperature and equilibrium constant data separately, copy the columns from your data array into a pair of 1D arrays, named temperature and k_eq.
Remember that you can refer to a particular column or row in a 2D array using **array slicing**.
Convert your temperature array from Celsius into Kelvin.

```python

```

```python

```

Plot $\ln{K}$ against $1/T$, and check that this gives a straight line relationship.
Label your axes.

```python

```

Using scipy.stats.linregress fit a straight line to this dataset, and print the slope and intercept from your fit.

```python

```

Generate another $\ln{K}$ versus $1/T$ plot, showing the experimental data as points, and your “line of best fit” as a line.

A simple way to generate points that lie along a straight line is to define a `line` function:
>```python
def line( m, c, x ):
    y = m * x + c
    return y
```

This function takes three arguments; $m$, $c$, and $x$; and returns the corresponding $y$ values. By passing in `slope` as $m$, `intercept` as $c$, and `1/temperature` as $x$, you can calculate the $y$ coordinates to plot on your figure.

```python
# Define your function here
```

```python
# Plot your figure.
# You will need to use `plt.plot()` twice to plot the original data, and your fitting line.
```

Use your fitted values for the slope and intercept to calculate the enthalpy and entropy of this reaction.
Remember that you can import the gas constant, $R$, using

 from scipy.constants import R

Add more code cells as necessary to organise your calculation.

```python

```

Create a markdown cell below, and use it to comment on the signs of $\Delta H$ and $\Delta S$ for this reaction.


# Analytical Modelling

Often, a well testing problem can be efficiently investigated using analytical solutions but these may require a degree of sophistication that is cumbersome for hand or Excel calculation.

The purpose of this notebook is to demonstrate a few Python techniques for well test modelling.

## 1. Implementing a Theis solution

Theis is the workhorse of pump test analysis in well-confined aquifers. It's reasonably easy to implement once you understand how to code up the well function, $W(u)$

\begin{equation}
W(u)=\int\limits_{u}^{\infty}\frac{1}{y}e^{-y}dy
\end{equation}

Fortunately, the exponential integral above is already implemented in `scipy`. The cell below implements the well function as the Python function `W(u)`.

```python
from scipy.special import expi

def W(u):
    return -expi(-u)
```

Run the cell below to see what the well function looks like. You can see where the logarithmic approximation is valid, for $u<0.05$.

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np
f,ax=plt.subplots(1,1)
u = np.logspace(-2,1,101)
ax.semilogx(u, W(u),'k.')
ax.axvline(0.05, color = 'r', linestyle=':')
ax.set_xlabel('$u$'); ax.set_ylabel('$W(u)$')
```

The `Theis` function is now straightforward to implement (run the cell below)

```python
def Theis(r,t,Q,S,T):
    return Q/(4*np.pi*T)*W(r**2*S/(4*T*t))
```

The function is already "vectorized", which means any of the inputs can be passed as a vector. The plot below gives drawdown 100 m from a well pumped at 20 L/s, in a formation with $T$=2200 m$^2$/d and $S$=10$^{-4}$.

```python
f,ax=plt.subplots(1,1)
t = np.linspace(0.001,3,101)
r,Q,S,T = [100., 20, 1.e-4, 2200]
ax.plot(t, Theis(r,t,Q*1.e-3*24*3600,S,T),'k.')
ax.set_xlabel('time [days]'); ax.set_ylabel('drawdown [m]')
```

***Why did we apply the scaling `Q*1.e-3*24*3600`?***

***Create a plot of drawdown over time inside a well of radius 0.3 m pumped at 2 m$^3$/min, $S$=0.2, $T$=200 m$^2$/d***

***Create a plot of drawdown with distance for the same parameters, at `t=0.1` and `t=1.5` days.***

```python
# your code here
```

## 2. Superposition of elementary solutions

We can use superposition to:
1. Model the drawdown at a location due to multiple pumping wells.
2. Model the drawdown due a single well being pumped at different rates.

### 2.1 Multiple wells

Suppose we are monitoring an observation well that is:
1. 100 m from a well that has been pumping at 20 L/s for the last 3 days.
2. 50 m from a well that has been pumping at 30 L/s for only the last day.

What should the drawdown profile look like over time? (Assume the same aquifer parameters as the example in 1.)

```python
# APPROACH: create a time vector for the solution and use a loop to compute drawdown contributions from the individual wells
t = np.linspace(0.001,3,101)
h = 0.*t
S,T = [1.e-4, 2200]
r1,t1,Q1 = [100,0., 20]  # well 1 starts at t=0 (pumping for the last three days)
r2,t2,Q2 = [50, 2., 30]  # well 2 starts at t=2 (pumping for only the last day)

for i in range(len(t)):
    # contribution from well 1
    if t[i]>t1:
        h[i] = h[i] + Theis(r1, t[i]-t1, Q1*1.e-3*24*3600, S, T)
    # contribution from well 2
    if t[i]>t2:
        h[i] = h[i] + Theis(r2, t[i]-t2, Q2*1.e-3*24*3600, S, T)

f,ax=plt.subplots(1,1)
ax.plot(t, h,'k-')
ax.set_xlabel('time [days]'); ax.set_ylabel('drawdown [m]')
```

***Add a third well that has been pumping for 10 L/s for the last 2 days at a distance of 40 m from the observation well.***

```python
# your code here
```

### 2.2 Step-rate pumping

Suppose we pump a well at:
1. 10 L/s for 30 mins, then
2. 15 L/s for another 30 mins.

What should the drawdown profile look like in the well, if the diameter is 0.7 m and there are no well losses?

```python
# APPROACH: same as 2.1, except R1 = R2 = well radius and Q2 = size of pumping step.
# your code here
```

***Add a third pumping step, at 25 L/s for another 60 mins.***

***Pumping is halted and the well is allowed to recover. Model this as a negative pumping step of 25 L/s. Plot the recovery for the next 2 hours.***

## 3. Solving equations containing pumping solutions

Pumping solutions introduce all sorts of non-linearity and the potential for analytically non-invertible equations.

For example, suppose we are monitoring an observation bore. We have pumped a well 100 m away at 10 L/s for the last 30 mins and a second well 75 m away at 5 L/s for the last 60 mins. If the drawdown is 30 cm and the transmissivity is known to be 1650 m$^2$/d, what is the storativity, $S$?

Obviously, this is a complex (and contrived!) problem. Let's write out the superposition of Theis solutions

\begin{equation}
h = \underbrace{\frac{Q_1}{4\pi T}W\left(\frac{r_1^2S}{4T(t-t_1)}\right)}_{\text{first well}} + \underbrace{\frac{Q_2}{4\pi T}W\left(\frac{r_2^2S}{4T(t-t_2)}\right)}_{\text{second well}}
\end{equation}

Which with known quantities converted to metres and days, and then substituted is

\begin{equation}
0.3 = \frac{864}{4\pi\cdot 1650}W\left(\frac{100^2\cdot S}{4\cdot1650\cdot 0.021}\right) + \frac{432}{4\pi\cdot 1650}W\left(\frac{75^2\cdot S}{4\cdot1650\cdot0.042}\right)
\end{equation}

One way to solve for $S$ is by guess-and-check but that can take a while.

Another way is to use Python's root finding functions to solve the above equation in the form $LHS-RHS=0$.

```python
from scipy.optimize import fsolve

# define the known parameters
h,r1,r2,Q1,Q2,t1,t2,T = [0.3,100,75,864,432,1./48,1./24,1650]

# define the root function for minimising, LHS - RHS, with the function input the unknown quantity
def f(S):
    return h-Q1/(4*np.pi*T)*W(r1**2*S/(4*T*t1))-Q2/(4*np.pi*T)*W(r2**2*S/(4*T*t2))

# pass the FUNCTION HANDLE (name) and an initial guess [0.001] to the root function
S = fsolve(f,0.001)[0]
print("Storativity is", S)
```

    Storativity is 9.752853715820548e-05

***Given the values $\beta$=10$^{-3}$ d/m$^2$, $r_w$=0.25 m, $S$=10$^{-4}$ and $t_{test}$=30 mins, use `fsolve` to show that the transmissivity in the equation below is $T$=1280 m$^2$/d.***

\begin{equation}
\beta = \frac{1}{4\pi T}W\left(\frac{r_w^2 S}{4Tt_{test}}\right)
\end{equation}

***For $h_{max}$=0.5 m, $\alpha$=1.e-13, $n$=3.5 and $t_{pump}$=1 year, use `fsolve` to show that the max. pumping rate in the equation below is $Q_{max}$=311 m$^3$/d.***

\begin{equation}
h_{max} = \frac{Q_{max}}{4\pi T}W\left(\frac{r_w^2 S}{4Tt_{pump}}\right)+\alpha Q_{max}^n
\end{equation}

```python
# your code here
```

## (Extra) Fitting pumping solutions to data

We use pumping solutions to make sense of the data. One way to do this is the graphical method you learned in class.

A more general approach is to plot the data and a numerical model over top of each other. Then, make changes to the model parameters until a good match is achieved.

This can be done automatically with a Python function called `curve_fit`.

First, let's create some fake pumping data with *known* transmissivity $T$=1500 m$^2$/d and $S$=0.0034

```python
T,S = [1500,0.0034]
r,Q,tpump = [200., 25.*1.e-3*24*3600, 3.]    # 3 day test at 25 L/s, observed at 200 m
td = np.logspace(-2,np.log10(tpump), 31)      # 31 log-spaced measurements
hd = Theis(r,td,Q,S,T)                         # drawdown observations

hd = hd*(1.+0.1*np.random.randn(len(hd)))                 # add 10% normally distributed random noise - for a challenge

f,(ax1,ax2) = plt.subplots(1,2, figsize=(10,4))
ax1.plot(td,hd,'ko',mfc='w')
ax2.semilogx(td,hd,'ko',mfc='w')
for ax in [ax1,ax2]:ax.set_xlabel('time [days]');ax.set_ylabel('drawdown [m]')

# note, this is not the most accurate way to model pump test noise, it is for demonstration purposes only
```

Python `curve_fit` works by finding the parameters that minimize the sum-of-squares misfit between data and a model.

The 'model' must be expressed as a Python function, $f$, with a very particular input structure:
1. The first argument is the independent variable (time).
2. Subsequent arguments are parameters that `curve_fit` can play with ($T$ and $S$).
3. Any other parameters should be hard-coded.
4. The function must return the same thing measured by the data (drawdown).

See below for a function meeting this requirement.

```python
def pump_test_model(t, T, S):
    # note how r and Q have been hard-coded - we know what these are
    return Theis(200., t, 25.*1.e-3*24*3600, S, T)
```

Now we call `curve_fit`, passing it the model (function handle/name), the data, and our starting guess at the model parameters.

```python
from scipy.optimize import curve_fit
p,pcov = curve_fit(pump_test_model, td, hd, [1000, 1.e-2])
print('best-fit T =', p[0],'and S =', p[1])
```

    best-fit T = 1495.7358957318952 and S = 0.0034192470900673037

The first output is a vector of the estimated parameters. We'll use it to plot the 'best' model over the data.

```python
f,(ax1,ax2) = plt.subplots(1,2, figsize=(10,4))
ax1.plot(td,hd,'ko',mfc='w')
ax1.plot(td,pump_test_model(td, p[0], p[1]),'r-')
ax2.semilogx(td,hd,'ko',mfc='w')
ax2.semilogx(td,pump_test_model(td, p[0], p[1]),'r-')
for ax in [ax1,ax2]:ax.set_xlabel('time [days]');ax.set_ylabel('drawdown [m]')
```

## (Extra) Uncertainty of best-fit pumping solutions

If we are not precisely certain of the data, then it can be possible for more than one model to provide a credible fit to the data.

There are numerous ways to handle model uncertainty (and the best way depends on where the error is coming from - are the data noisy or is the conceptual model wrong?)

A simple way to get a first cut at model uncertainty is called Linear Sensitivity Analysis. `curve_fit` returns a second output called the covariance matrix, which gives some indication about how confident it is estimating the values of $T$ and $S$.

We can use the covariance matrix to generate "possible pairs" of $[T,S]$ and plot these models.

We can also use ranges of the sampled parameters to construct uncertainty estimates. In this case, we estimate transmissivity to be between 1375 and 1635 m$^2$/d with 90% confidence (in fact, it is 1500 m$^2$/d), and storativity between 2.8 and 4.0e-3 (in fact, it is 3.4e-3).

```python
from scipy.optimize import curve_fit
p,pcov = curve_fit(pump_test_model, td, hd, [1000, 1.e-2])
print('covariance matrix is', pcov)

f,(ax1,ax2) = plt.subplots(1,2, figsize=(10,4))
ax1.plot(td,hd,'ko',mfc='w')
ax2.semilogx(td,hd,'ko',mfc='w')

N = 100   # number of possible pairs to generate
Ts = []; Ss = []
for p in np.random.multivariate_normal(p, pcov, N):
    ax1.plot(t,pump_test_model(t, *p),'r-', lw=0.2, alpha=0.2)
    ax2.semilogx(t,pump_test_model(t, *p),'r-', lw=0.2, alpha=0.2)
    Ts.append(p[0]); Ss.append(p[1])

for ax in [ax1,ax2]:ax.set_xlabel('time [days]');ax.set_ylabel('drawdown [m]')

print('5 to 95-percentile range of T is [{:3.2f},{:3.2f}]'.format(*np.percentile(Ts,[5,95])))
print('5 to 95-percentile range of S is [{:3.2e},{:3.2e}]'.format(*np.percentile(Ss,[5,95])))

```


# Kurveintegral med Sympy

Vi har en kurve i rommet

$$
\vec{r}(t) = t\mathbf{i} + t\mathbf{j} + 2t^{3/2} \mathbf{k}, \quad t \in [0, 1]
$$

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
from sympy.vector import CoordSys3D
```

Start med å plotte kurven i 3D rom. Dette kan man gjøre med matplotlib på følgende måte.

```python
%matplotlib notebook
ax = plt.axes(projection='3d')
ti = np.linspace(0, 1, 100)
ax.plot3D(ti, ti, 2*ti**1.5)
```

    <IPython.core.display.Javascript object>

    [<mpl_toolkits.mplot3d.art3d.Line3D at 0x7f839062b820>]

Lag et koordinatsystem og posisjonsvektoren $\vec{r}$

```python
t = sp.Symbol('t', real=True)
N = CoordSys3D('N')
r = t*N.i + t*N.j + 2*t**sp.Rational(3,2)*N.k
```

Finn så den deriverte av $\vec{r}$ mhp t

```python
drdt = r.diff(t, 1)
drdt
```

$\displaystyle \mathbf{\hat{i}_{N}} + \mathbf{\hat{j}_{N}} + (3 \sqrt{t})\mathbf{\hat{k}_{N}}$

Integrer for å finne buelengden

```python
sp.integrate(sp.sqrt(drdt.dot(drdt)), (t, 0, 1))
```

$\displaystyle - \frac{4 \sqrt{2}}{27} + \frac{22 \sqrt{11}}{27}$


```python
import numpy as np
from scipy.linalg import lu
from sympy import Matrix
```

Solves Ax=B

We are trying to solve the linear system

 2x1 + 5x2 = 2</br>
-4x1 + 3x2 = -30

```python
A = Matrix([[2,5],[-4,3]])
```

```python
A
```

$\displaystyle \left[\begin{matrix}2 & 5\\-4 & 3\end{matrix}\right]$

```python
B= Matrix([[2,-30]])
```

```python
B = B.transpose()
```

```python
B
```

$\displaystyle \left[\begin{matrix}2\\-30\end{matrix}\right]$

```python
A.gauss_jordan_solve(B)
```

    (Matrix([
     [ 6],
     [-2]]),
     Matrix(0, 1, []))

x1 = 6 and x2 = -2

```python

```


# TDOA Localization

The time difference of arrival (TDOA) measurement is given as:

$d = |\vec{a} - \vec{n}| - |\vec{b} - \vec{n}|$

where $\vec{a}$ and $\vec{b}$ are anchor nodes, $\vec{n}$ is the vehicle, and d is the difference in time of arrival of the anchor messages at the vehicle, converted to $m$ using the speed of light.

We are going to linearize this measurement equation and then plot the localization accuracy in a given volume with known anchor locations.

See [Kalman Filtering](https://en.wikipedia.org/wiki/Kalman_filter)

See [Multilateration/TDOA](https://en.wikipedia.org/wiki/Multilateration)

```python
import sympy
from itertools import combinations
import numpy as np
import matplotlib.pyplot as plt
sympy.init_printing()

def derive_measurement_jacobian():
    """
    Compute the measurement jacobian.

    @return: Returns the symbolic Jacobian H, and a function that
        computes H given the anchors (a, b), and the vehicle/node (n).
    """
    a = sympy.Matrix(sympy.symbols('a_0:3', real=True))
    b = sympy.Matrix(sympy.symbols('b_0:3', real=True))
    n = sympy.Matrix(sympy.symbols('n_0:3', real=True))

    d = sympy.Matrix.norm(a - n) - sympy.Matrix.norm(b - n)

    da = sympy.symbols('da')
    db = sympy.symbols('db')

    H = sympy.Matrix([d]).jacobian(n)
    H.simplify()
    H
    f_H = sympy.lambdify((a, b, n), H)
    return {'H': H, 'f_H': f_H}

meas = derive_measurement_jacobian()
meas['H']
```

$\displaystyle \left[\begin{matrix}- \frac{a_{0} - n_{0}}{\sqrt{\left(a_{0} - n_{0}\right)^{2} + \left(a_{1} - n_{1}\right)^{2} + \left(a_{2} - n_{2}\right)^{2}}} + \frac{b_{0} - n_{0}}{\sqrt{\left(b_{0} - n_{0}\right)^{2} + \left(b_{1} - n_{1}\right)^{2} + \left(b_{2} - n_{2}\right)^{2}}} & - \frac{a_{1} - n_{1}}{\sqrt{\left(a_{0} - n_{0}\right)^{2} + \left(a_{1} - n_{1}\right)^{2} + \left(a_{2} - n_{2}\right)^{2}}} + \frac{b_{1} - n_{1}}{\sqrt{\left(b_{0} - n_{0}\right)^{2} + \left(b_{1} - n_{1}\right)^{2} + \left(b_{2} - n_{2}\right)^{2}}} & - \frac{a_{2} - n_{2}}{\sqrt{\left(a_{0} - n_{0}\right)^{2} + \left(a_{1} - n_{1}\right)^{2} + \left(a_{2} - n_{2}\right)^{2}}} + \frac{b_{2} - n_{2}}{\sqrt{\left(b_{0} - n_{0}\right)^{2} + \left(b_{1} - n_{1}\right)^{2} + \left(b_{2} - n_{2}\right)^{2}}}\end{matrix}\right]$

```python
def find_max_error(anchors, n):
    """
    Find the lhe maximum error of the UWB localization method.

    @param anchors: A list of 3d points representing anchor locations
    @param n: A 3d point representing the location of the node/ vehicle.
    """
    P = 100000*np.eye(3) # this represents that we don't know where we are initially, it is the covariance matrix
    R = (0.2)**2  # the measurement variance
    for a, b in combinations(anchors, 2):
        # if evaluating at an anchor, shift anchor slightly
        if np.linalg.norm(a-n) == 0:
            a = np.array(a) + 1e-6*np.array([1, 1, 1])
        if np.linalg.norm(b-n) == 0:
            b = np.array(b) + 1e-6*np.array([1, 1, 1])
        # Kalman correction
        H = meas['f_H'](a, b, n)
        S = H.dot(P).dot(H.T) + R
        K = P.dot(H.T).dot(np.linalg.inv(S))
        P = P - K.dot(H).dot(P)
    # Find max error standard deviation
    evals, evecs = np.linalg.eig(P)
    return np.sqrt(np.max(evals))

def plot_accuracy(anchors, nz, border=2, vmin=0, vmax=1):
    """
    Plot a contour of localization accuracy as a fixed height.

    @param anchors: A list of 3d points representing anchor locations
    @param nz: The z height of the node for the heat map.
    @param vmin: The min value of the heat map.
    @param vmax: The max value of the heat map.
    """
    nx_list = np.linspace(np.min(anchors[:, 0]) - border, np.max(anchors[:, 0]) + border, 20)
    ny_list = np.linspace(np.min(anchors[:, 1]) - border, np.max(anchors[:, 1]) + border, 20)
    levels=np.arange(vmin, vmax + 0.1, 0.1)
    data = []
    for ny in ny_list:
        x_data = []
        for nx in nx_list:
            x_data.append(find_max_error(anchors, [nx, ny, nz]))
        data.append(x_data)
    data = np.array(data)
    X, Y = np.meshgrid(nx_list, ny_list)
    plt.contourf(X, Y, data, cmap='jet', levels=levels, vmin=vmin, vmax=vmax)
    for anchor_id, anchor in enumerate(anchors):
        plt.text(anchor[0]+ 0.1*anchor[2], anchor[1], str(anchor_id))
        plt.plot(anchor[0], anchor[1], 'kx')
    plt.xlabel('x, m')
    plt.ylabel('y, m')
    cbar = plt.colorbar()
    return plt.gca()
```

Now we can plot a heat map of the localization accuracy of the method within the flight volume.

```python
sx = 3.8  # The x separation of the anchors.
sy = 6  # The y separation of the anchors.
z0 = 0.57 # lower height on tripod
z1 = 2.36 # upper anchor height on tripod
anchors = np.array([
    [0, 0, z0],
    [0, 0, z1],
    [0, sy, z0],
    [0, sy, z1],
    [sx, sy, z0],
    [sx, sy, z1],
    [sx, 0, z0],
    [sx, 0, z1]
])

z_list = [0, 1, 2, 3, 4]
for zi, z in enumerate(z_list):
    border = 2
    plt.figure()
    ax = plot_accuracy(anchors, nz=z, vmin=0, vmax=0.5, border=border)
    plt.title('TDOA Localization Accuracy at z={:g} m'.format(z));
    plt.axis('equal')
```

We will now try to move some of the anchors back and up and see how this changes the localization accuracy.

```python
sx = 2
sy = 5
z0 = 0.1
z1 = 2.3
anchors = np.array([
    [0, 0, z0],
    [0, 0, z1],
    [0, sy, z0],
    [0, sy, z1],
    #[sx, sy, z0],
    [sx, sy, z1],
    #[sx, 0, z0],
    [sx, 0, z1],
    [5, 0, 5],
    [5, sy, 5],
    #[10, 0, 10],
    #[10, sy, 10]
])

z_list = [0, 1, 2, 3, 4]
for zi, z in enumerate(z_list):
    plt.figure()
    ax = plot_accuracy(anchors, nz=z, vmin=0, vmax=0.5, border=2)
    plt.title('TDOA Localization Accuracy at z={:g} m'.format(z));
    plt.axis('equal')
```


# 15.3. Analyzing real-valued functions
https://ipython-books.github.io/153-analyzing-real-valued-functions/

## Ref

* SymPy integral transforms - http://docs.sympy.org/latest/modules/integrals/integrals.html
* Real analysis on Wikipedia, at https://en.wikipedia.org/wiki/Real_analysis#Bibliography
* Calculus on Wikibooks, at http://en.wikibooks.org/wiki/Calculus
* Real analysis on Awesome Math, at https://github.com/rossant/awesome-math/#real-analysis

```python
from sympy import *
init_printing()
```

```python
var('x z')
```

```python
f = 1 / (1 + x**2)
```

```python
type(f)
```

    sympy.core.power.Pow

```python
v = f.subs(x, 1)
v, type(v)
```

    (1/2, sympy.core.numbers.Half)

```python
diff(f, x)
```

```python
limit(f, x, oo)   # "oo" for infinity
```

```python
ts = series(f, x0=0, n=9)
ts
```

```python
type(ts)
```

    sympy.core.add.Add

How to remove higher order terms:
* https://stackoverflow.com/questions/25001896/sympy-drop-higher-order-terms-in-polynomial

```python
ts = series(f, x0=0, n=9).removeO()   # remove Big-O
ts
```

```python
type(ts)
```

    sympy.core.add.Add

```python
integrate(f, (x, -oo, oo))
```

```python
integrate(f, x)
```

```python
fourier_transform(f, x, z)
```

```python

```


# 1. Creating formulas
Write the following mathematical formula in Python:

\begin{align}
 result = 6a^3 - \frac{8b^2 }{4c} + 11
\end{align}

```python
a = 2
b = 3
c = 2
```

```python
# Your formula here:
result = 6*(a**3) - 8*(b**2)/(4*c) + 11
```

```python

assert result == 50
```

# 2. Floating point pitfalls
Show that `0.1 + 0.2 == 0.3`

```python
# Your solution here
a = 0.1
b = 0.2
c = a + 3
c

```

    3.1

```python
assert c == 0.3 #Fail, c = 3.1 because of binary floating point
```

```python
from decimal import *
Decimal(0.1) + Decimal(0.2) #Decimal('0.30') => combined total # decimal place in arithmetic calc
Decimal('0.1') + Decimal('0.2') #Decimal('0.3')

```

    Decimal('0.3')

```python
from decimal import *
getcontext().prec = 2
getcontext()
Decimal(0.1+0.2) #Decimal('0.3000000000000000444089209850062616169452667236328125')

```

    Decimal('0.3000000000000000444089209850062616169452667236328125')


# The Multivariate Gaussian distribution

The density of a multivariate Gaussian with mean vector $\mu$ and covariance matrix $\Sigma$ is given as

\begin{align}
\mathcal{N}(x; \mu, \Sigma) &= |2\pi \Sigma|^{-1/2} \exp\left( -\frac{1}{2} (x-\mu)^\top \Sigma^{-1} (x-\mu) \right) \\
& = \exp\left(-\frac{1}{2} x^\top \Sigma^{-1} x + \mu^\top \Sigma^{-1} x   - \frac{1}{2} \mu^\top \Sigma^{-1} \mu   -\frac{1}{2}\log \det(2\pi \Sigma) \right) \\
\end{align}

Here, $|X|$ denotes the determinant of a square matrix.

$\newcommand{\trace}{\mathop{Tr}}$

\begin{align}
{\cal N}(s; \mu, P) & =  |2\pi P|^{-1/2} \exp\left(-\frac{1}2 (s-\mu)^\top P^{-1} (s-\mu) \right)
\\
& =   \exp\left(
-\frac{1}{2}s^\top{P^{-1}}s  + \mu^\top P^{-1}s  { -\frac{1}{2}\mu^\top{P^{-1}\mu -\frac12|2\pi P|}}
\right) \\
\log {\cal N}(s; \mu, P) & =  -\frac{1}{2}s^\top{P^{-1}}s  + \mu^\top P^{-1}s + \text{ const} \\
& =  -\frac{1}{2}\trace {P^{-1}} s s^\top  + \mu^\top P^{-1}s + \text{ const} \\
\end{align}

## Special Cases

To gain the intuition, we take a look to a few special cases

### Bivariate Gaussian

#### Example 1: Identity covariance matrix

$
x = \left(\begin{array}{c} x_1 \\ x_2 \end{array} \right)
$

$
\mu = \left(\begin{array}{c} 0 \\ 0 \end{array} \right)
$

$
\Sigma = \left(\begin{array}{cc} 1& 0 \\ 0 & 1 \end{array} \right) = I_2
$

\begin{align}
\mathcal{N}(x; \mu, \Sigma) &= |2\pi I_{2}|^{-1/2} \exp\left( -\frac{1}{2} x^\top x \right)
= (2\pi)^{-1} \exp\left( -\frac{1}{2} \left( x_1^2 + x_2^2\right)  \right) =  (2\pi)^{-1/2} \exp\left( -\frac{1}{2} x_1^2  \right)(2\pi)^{-1/2} \exp\left( -\frac{1}{2} x_2^2 \right)\\
& = \mathcal{N}(x; 0, 1) \mathcal{N}(x; 0, 1)
\end{align}

#### Example 2: Diagonal covariance
$\newcommand{\diag}{\text{diag}}$

$
x = \left(\begin{array}{c} x_1 \\ x_2 \end{array} \right)
$

$
\mu = \left(\begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right)
$

$
\Sigma = \left(\begin{array}{cc} s_1 & 0 \\ 0 & s_2 \end{array} \right) = \diag(s_1, s_2)
$

\begin{eqnarray}
\mathcal{N}(x; \mu, \Sigma) &=& \left|2\pi \left(\begin{array}{cc} s_1 & 0 \\ 0 & s_2 \end{array} \right)\right|^{-1/2} \exp\left( -\frac{1}{2} \left(\begin{array}{c} x_1 - \mu_1 \\ x_2-\mu_2 \end{array} \right)^\top \left(\begin{array}{cc} 1/s_1 & 0 \\ 0 & 1/s_2 \end{array} \right) \left(\begin{array}{c} x_1 - \mu_1 \\ x_2-\mu_2 \end{array} \right) \right) \\
&=& ((2\pi)^2 s_1 s_2 )^{-1/2} \exp\left( -\frac{1}{2} \left( \frac{(x_1-\mu_1)^2}{s_1} + \frac{(x_2-\mu_2)^2}{s_2}\right)  \right) \\
& = &\mathcal{N}(x; \mu_1, s_1) \mathcal{N}(x; \mu_2, s_2)
\end{eqnarray}

#### Example 3:
$
x = \left(\begin{array}{c} x_1 \\ x_2 \end{array} \right)
$

$
\mu = \left(\begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right)
$

$
\Sigma = \left(\begin{array}{cc} 1 & \rho \\ \rho & 1 \end{array} \right)
$
for $1<\rho<-1$.

Need $K = \Sigma^{-1}$. When $|\Sigma| \neq 0$ we have $K\Sigma^{-1} = I$.

$
\left(\begin{array}{cc} 1 & \rho \\ \rho & 1 \end{array} \right) \left(\begin{array}{cc} k_{11} & k_{12} \\ k_{21} & k_{22} \end{array} \right) = \left(\begin{array}{cc} 1& 0 \\ 0 & 1 \end{array} \right)
$

\begin{align}
k_{11} &+ \rho k_{21} & & &=1 \\
\rho k_{11} &+ k_{21} & & &=0 \\
&& k_{12} &+ \rho k_{22} &=0 \\
&& \rho k_{12} &+ k_{22} &=1 \\
\end{align}

Solving these equations leads to the solution

$$
\left(\begin{array}{cc} k_{11} & k_{12} \\ k_{21} & k_{22} \end{array} \right) = \frac{1}{1-\rho^2}\left(\begin{array}{cc} 1 & -\rho \\ -\rho & 1 \end{array} \right)
$$

Plotting the Equal probability contours

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

from notes_utilities import pnorm_ball_points

RHO = np.arange(-0.9,1,0.3)
plt.figure(figsize=(20,20/len(RHO)))

plt.rc('text', usetex=True)
plt.rc('font', family='serif')

for i,rho in enumerate(RHO):
    plt.subplot(1,len(RHO),i+1)
    plt.axis('equal')
    ax = plt.gca()
    ax.set_xlim(-4,4)
    ax.set_ylim(-4,4)

    S = np.mat([[1, rho],[rho,1]])
    A = np.linalg.cholesky(S)
    dx,dy = pnorm_ball_points(3*A)
    plt.title(r'$\rho =$ '+str(rho if np.abs(rho)>1E-9 else 0), fontsize=16)
    ln = plt.Line2D(dx,dy,markeredgecolor='k', linewidth=1, color='b')
    ax.add_line(ln)
    ax.set_axis_off()
    #ax.set_visible(False)

plt.show()
```

```python
from ipywidgets import interact, interactive, fixed
import ipywidgets as widgets
from IPython.display import clear_output, display, HTML
from matplotlib import rc

from notes_utilities import bmatrix, pnorm_ball_line

rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
## for Palatino and other serif fonts use:
#rc('font',**{'family':'serif','serif':['Palatino']})
rc('text', usetex=True)

fig = plt.figure(figsize=(5,5))

S = np.array([[1,0],[0,1]])

dx,dy = pnorm_ball_points(S)
ln = plt.Line2D(dx,dy,markeredgecolor='k', linewidth=1, color='b')

dx,dy = pnorm_ball_points(np.eye(2))
ln2 = plt.Line2D(dx,dy,markeredgecolor='k', linewidth=1, color='k',linestyle=':')

plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
ax = fig.gca()
ax.set_xlim((-4,4))
ax.set_ylim((-4,4))
txt = ax.text(-1,-3,'$\left(\right)$',fontsize=15)
ax.add_line(ln)
ax.add_line(ln2)

plt.close(fig)

def set_line(s_1, s_2, rho, p, a, q):
    S = np.array([[s_1**2, rho*s_1*s_2],[rho*s_1*s_2, s_2**2]])
    A = np.linalg.cholesky(S)
    #S = A.dot(A.T)
    dx,dy = pnorm_ball_points(A,p=p)
    ln.set_xdata(dx)
    ln.set_ydata(dy)
    dx,dy = pnorm_ball_points(a*np.eye(2),p=q)
    ln2.set_xdata(dx)
    ln2.set_ydata(dy)

    txt.set_text(bmatrix(S))
    display(fig)
    ax.set_axis_off()

interact(set_line, s_1=(0.1,2,0.01), s_2=(0.1, 2, 0.01), rho=(-0.99, 0.99, 0.01), p=(0.1,4,0.1), a=(0.2,10,0.1), q=(0.1,4,0.1))

```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    <function __main__.set_line>

```python
%run plot_normballs.py
```

```python
%run matrix_norm_sliders.py

```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Exercise:

$
x = \left(\begin{array}{c} x_1 \\ x_2 \end{array} \right)
$

$
\mu = \left(\begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right)
$

$
\Sigma = \left(\begin{array}{cc} s_{11} & s_{12} \\ s_{12} & s_{22} \end{array} \right)
$

Need $K = \Sigma^{-1}$. When $|\Sigma| \neq 0$ we have $K\Sigma^{-1} = I$.

$
\left(\begin{array}{cc} s_{11} & s_{12} \\ s_{12} & s_{22} \end{array} \right) \left(\begin{array}{cc} k_{11} & k_{12} \\ k_{21} & k_{22} \end{array} \right) = \left(\begin{array}{cc} 1& 0 \\ 0 & 1 \end{array} \right)
$

Derive the result
$$
K = \left(\begin{array}{cc} k_{11} & k_{12} \\ k_{21} & k_{22} \end{array} \right)
$$

Step 1: Verify

$$
\left(\begin{array}{cc} s_{11} & s_{12} \\ s_{21} & s_{22} \end{array} \right) = \left(\begin{array}{cc} 1 & -s_{12}/s_{22} \\ 0 & 1 \end{array} \right) \left(\begin{array}{cc} s_{11}-s_{12}^2/s_{22} & 0 \\ 0 & s_{22} \end{array} \right) \left(\begin{array}{cc} 1 & 0 \\ -s_{12}/s_{22} & 1 \end{array} \right)
$$

Step 2: Show that
$$
\left(\begin{array}{cc} 1 & a\\ 0 & 1 \end{array} \right)^{-1} = \left(\begin{array}{cc} 1 & -a\\ 0 & 1 \end{array} \right)
$$
and
$$
\left(\begin{array}{cc} 1 & 0\\ b & 1 \end{array} \right)^{-1} = \left(\begin{array}{cc} 1 & 0\\ -b & 1 \end{array} \right)
$$

Step 3: Using the fact $(A B)^{-1} = B^{-1} A^{-1}$ and $s_{12}=s_{21}$, show that and simplify
$$
\left(\begin{array}{cc} s_{11} & s_{12} \\ s_{21} & s_{22} \end{array} \right)^{-1} =
\left(\begin{array}{cc} 1 & 0 \\ s_{12}/s_{22} & 1 \end{array} \right)
\left(\begin{array}{cc} 1/(s_{11}-s_{12}^2/s_{22}) & 0 \\ 0 & 1/s_{22} \end{array} \right) \left(\begin{array}{cc} 1 & s_{12}/s_{22} \\ 0 & 1 \end{array} \right)
$$

## Gaussian Processes Regression

In Bayesian machine learning, a frequent problem encountered is the regression problem where we are given a pairs of inputs $x_i \in \mathbb{R}^N$ and associated noisy observations $y_i \in \mathbb{R}$. We assume the following model

\begin{eqnarray*}
y_i &\sim&  {\cal N}(y_i; f(x_i), R)
\end{eqnarray*}

The interesting thing about a Gaussian process is that the function $f$ is not specified in close form, but we assume that the function values
\begin{eqnarray*}
f_i & = & f(x_i)
\end{eqnarray*}
are jointly Gaussian distributed as
\begin{eqnarray*}
\left(
  \begin{array}{c}
    f_1 \\
    \vdots \\
    f_L \\
  \end{array}
\right) & = & f_{1:L} \sim  {\cal N}(f_{1:L}; 0, \Sigma(x_{1:L}))
\end{eqnarray*}
Here, we define the entries of the covariance matrix $\Sigma(x_{1:L})$ as
\begin{eqnarray*}
\Sigma_{i,j} & = & K(x_i, x_j)
\end{eqnarray*}
for $i,j \in \{1, \dots, L\}$. Here, $K$ is a given covariance function. Now, if we wish to predict the value of $f$ for a new $x$, we simply form the following joint distribution:
\begin{eqnarray*}
\left(
  \begin{array}{c}
    f_1 \\
    f_2 \\
    \vdots \\
    f_L \\
    f \\
  \end{array}
\right) & \sim &  {\cal N}\left( \left(\begin{array}{c}
    0 \\
    0 \\
    \vdots \\
    0 \\
    0 \\
  \end{array}\right)
  , \left(\begin{array}{cccccc}
    K(x_1,x_1) & K(x_1,x_2) & \dots & K(x_1, x_L) &  K(x_1, x) \\
    K(x_2,x_1) & K(x_2,x_2) & \dots & K(x_2, x_L) & K(x_2, x) \\
    \vdots &\\
    K(x_L,x_1) & K(x_L,x_2) & \dots & K(x_L, x_L) &  K(x_L, x)   \\
    K(x,x_1) & K(x,x_2) & \dots & K(x, x_L) &  K(x, x)   \\
  \end{array}\right) \right) \\
\left(
\begin{array}{c}
    f_{1:L} \\
    f
  \end{array}
\right) & \sim &  {\cal N}\left( \left(\begin{array}{c}
    \mathbf{0} \\
    0 \\
  \end{array}\right)
  , \left(\begin{array}{cc}
    \Sigma(x_{1:L}) &  k(x_{1:L}, x) \\
    k(x_{1:L}, x)^\top &  K(x, x)   \\
  \end{array}\right) \right) \\
\end{eqnarray*}

Here,  $k(x_{1:L}, x)$ is a $L \times 1$ vector with entries $k_i$ where

\begin{eqnarray*}
k_i = K(x_i, x)
\end{eqnarray*}

Popular choices of covariance functions to generate smooth regression functions include a Bell shaped one
\begin{eqnarray*}
K_1(x_i, x_j) & = & \exp\left(-\frac{1}2 \| x_i - x_j \|^2 \right)
\end{eqnarray*}
and a Laplacian
\begin{eqnarray*}
K_2(x_i, x_j) & = & \exp\left(-\frac{1}2 \| x_i - x_j \| \right)
\end{eqnarray*}

where $\| x \| = \sqrt{x^\top x}$ is the Euclidian norm.

## Part 1
Derive the expressions to compute the predictive density
\begin{eqnarray*}
p(\hat{y}| y_{1:L}, x_{1:L}, \hat{x})
\end{eqnarray*}

\begin{eqnarray*}
p(y | y_{1:L}, x_{1:L}, x) &=& {\cal N}(y; m, S) \\
m & = & \\
S & = &
\end{eqnarray*}

## Part 2
Write a program to compute the mean and covariance of $p(\hat{y}| y_{1:L}, x_{1:L}, \hat{x})$ to generate a for the following data:

     x = [-2 -1 0 3.5 4]
     y = [4.1 0.9 2 12.3 15.8]

Try different covariance functions $K_1$ and $K_2$ and observation noise covariances $R$ and comment on the nature of the approximation.

## Part 3
Suppose we are using a covariance function parameterised by
\begin{eqnarray*}
K_\beta(x_i, x_j) & = & \exp\left(-\frac{1}\beta  \| x_i - x_j \|^2 \right)
\end{eqnarray*}
Find the optimum regularisation parameter $\beta^*(R)$ as a function of observation noise variance via maximisation of the marginal likelihood, i.e.
\begin{eqnarray*}
\beta^* & = & \arg\max_{\beta} p(y_{1:N}| x_{1:N}, \beta, R)
\end{eqnarray*}
Generate a plot of $b^*(R)$ for $R = 0.01, 0.02, \dots, 1$ for the dataset given in 2.

```python
def cov_fun_bell(x1,x2,delta=1):
    return np.exp(-0.5*np.abs(x1-x2)**2/delta)

def cov_fun_exp(x1,x2):
    return np.exp(-0.5*np.abs(x1-x2))

def cov_fun(x1,x2):
    return cov_fun_bell(x1,x2,delta=0.1)

R = 0.05

x = np.array([-2, -1, 0, 3.5, 4]);
y = np.array([4.1, 0.9, 2, 12.3, 15.8]);

Sig = cov_fun(x.reshape((len(x),1)),x.reshape((1,len(x)))) + R*np.eye(len(x))
SigI = np.linalg.inv(Sig)

xx = np.linspace(-10,10,100)
yy = np.zeros_like(xx)
ss = np.zeros_like(xx)
for i in range(len(xx)):
    z = np.r_[x,xx[i]]
    CrossSig = cov_fun(x,xx[i])
    PriorSig = cov_fun(xx[i],xx[i]) + R

    yy[i] = np.dot(np.dot(CrossSig, SigI),y)
    ss[i] = PriorSig - np.dot(np.dot(CrossSig, SigI),CrossSig)

plt.plot(x,y,'or')
plt.plot(xx,yy,'b.')
plt.plot(xx,yy+3*np.sqrt(ss),'b:')
plt.plot(xx,yy-3*np.sqrt(ss),'b:')
plt.show()

```


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Convolution

The convolution of two signals $s(t)$ and $g(t)$ is defined as

\begin{equation}
\int_{-\infty}^{\infty} s(\tau) \cdot g(t - \tau) \; d\tau = s(t) * g(t)
\end{equation}

where $*$ is a common short-hand notation of the convolution integral. As shown in the previous Section, the convolution is an important operation in the theory of signals and systems. Its properties are therefore of general interest.

### Properties

For the signals $s(t)$, $g(t)$, $h(t) \in \mathbb{C}$ the convolution shows the following properties

1. The Dirac impulse is the [identity element](https://en.wikipedia.org/wiki/Identity_element) of the convolution
    \begin{equation}
    s(t) * \delta(t) = s(t)
    \end{equation}

2. The convolution is [commutative](https://en.wikipedia.org/wiki/Commutative_property)
    \begin{equation}
    s(t) * g(t) = g(t) * s(t)
    \end{equation}

3. The convolution is [associative](https://en.wikipedia.org/wiki/Associative_property)
    \begin{equation}
    \left[ s(t) * g(t) \right] * h(t) = s(t) * \left[ g(t) * h(t) \right]
    \end{equation}

5. The convolution is [distributive](https://en.wikipedia.org/wiki/Distributive_property)
    \begin{equation}
    s(t) * \left[ g(t) + h(t) \right] = s(t) * g(t) + s(t) * h(t)
    \end{equation}

5. Multiplication with a scalar $a \in \mathbb{C}$
    \begin{equation}
    a \cdot \left[ s(t) * g(t) \right] = \left[ a \cdot s(t) \right] * g(t) = s(t) * \left[ a \cdot g(t) \right]
    \end{equation}

6. Derivative of the convolution
    \begin{equation}
    \frac{d}{dt} \left[ s(t) * g(t) \right] =  \frac{d s(t)}{dt} * g(t) = s(t) * \frac{d g(t)}{dt}
    \end{equation}

The first property is a consequence of the sifting property of the Dirac pulse, the second to fifth property can be proven by considering the definition of the convolution integral and the sixth property follows from the properties of the derivative of the Dirac delta function.

### Geometrical Interpretation

The convolution can be [interpreted in a graphical manner](https://en.wikipedia.org/wiki/Convolution#Visual_explanation). This provides valuable insights into its calculation and allows to estimate its result. The calculation of the convolution integral

\begin{equation}
y(t) = \int_{-\infty}^{\infty} x(\tau) \cdot h(t-\tau) \; d\tau
\end{equation}

can be decomposed into four steps:

1. substitute $t$ by $\tau$ in both $x(t)$ and $h(t)$,

2. time-reverse $h(\tau)$ (reflection at vertical axis),

3. shift $h(-\tau)$ by $t$ to the right to yield $h(t - \tau)$,

4. shift $h(t - \tau)$ from $t = -\infty$ to $t = \infty$, check if it overlaps with $x(\tau)$ and calculate the convolution integral for the overlapping sections.

**Example**

Above procedure is illustrated in the following example with

\begin{align}
h(t) &= e^{-t} \\
x(t) &= \text{rect} \left(t - \frac{1}{2}\right)
\end{align}

Before proceeding, helper functions for the rectangular signal and plotting of the signals are defined

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, tau = sym.symbols('t tau')
```

```python
class rect(sym.Function):

    @classmethod
    def eval(cls, arg):
        return sym.Heaviside(arg + sym.S.Half) - sym.Heaviside(arg - sym.S.Half)
```

```python
def plot_signals(x_axis, x, h, ylabel, xlabel):
    p1 = sym.plot(x, (x_axis, -5, 5), show=False, line_color='b', ylabel=ylabel, xlabel=xlabel)
    p2 = sym.plot(h, (x_axis, -5, 5), show=False, line_color='r')
    p1.extend(p2)
    p1.show()
```

Now lets define and plot the signals. In the following, the impulse response $h(t)$ is illustrated by the red graph and the input signal $x(t)$ by the blue graph.

```python
h = sym.exp(-t) * sym.Heaviside(t)
x = rect(t - 1/2)

plot_signals(t, x, h, r'$h(t)$, $x(t)$', r'$t$')
```

The **first step** is to substitute $t$ by $\tau$. The horizontal axis of the plot represents $\tau$

```python
h1 = h.subs(t, tau)
x1 = x.subs(t, tau)

plot_signals(tau, x1, h1, r'$h(\tau)$, $x(\tau)$', r'$\tau$')
```

The **second step** is to time-reverse $h(\tau)$

```python
h2 = h1.subs(tau, -tau)

plot_signals(tau, x1, h2, r'$h(-\tau)$, $x(\tau)$', r'$\tau$')
```

The impulse response $h(-\tau)$ is shifted by $t$ to the right in the **third step** to yield $h(t - \tau)$. This is illustrated for $t = -2$

```python
h3 = h2.subs(tau, tau-t)

plot_signals(tau, x1, h3.subs(t, -2), r'$h(t-\tau)$, $x(\tau)$', r'$\tau$')
```

It now becomes obvious that we have to consider three cases with respect to the overlap of $h(t-\tau)$ and $x(\tau)$

1. $t<0$: no overlap
2. $0 \leq t < 1$: partial overlap
3. $t > 0$: full overlap

The **fourth step**, the evaluation of the convolution integral for the three cases is left open as an exercise.

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# Chapter 3: Conditional Probability and Independence

```python
%%html
<style>
.tableHeader{
    font-size:16px;
}

.eqn{
    font-size:16px;
    color:blue;
}
</style>
```

---

## 3.2 Conditional probabilities
$P(A \mid B)$ refers to the conditional probability that **A** occurs given than **B** has occurred

If $P(B)>0$ then
$$
P(A \mid B) = \frac{P(AB)}{P(B)} \implies P(AB) = P(B)P(A \mid B)
$$

### The multiplication rule
$$
P(A_1A_2A_3 \cdots A_n) = P(A_1)P(A_2 \mid A_1)P(A_3 \mid A_1A_2) \cdots P(A_n \mid A_1 \cdots A_{n-1})
$$

---

## 3.3 Bayes's formula
Let **A** and **B** be events, We may express **A** as:
$$A = AB \cup AB^\complement$$
for, in order for an outcome to be in **A**, it must either be in both **A** and **B** or be in **A** but not in **B**.

### 3.3.1 Bayes
\begin{align}
P(A) &= P(AB) + P(AB^\complement) \\
&= P(A \mid B)P(B) + P(A \mid B^\complement)P(B^\complement) \\
&= P(A \mid B)P(B) + P(A \mid B^\complement)[1-P(B)]
\end{align}

### 3.3.2
\begin{align}
P(B \mid A) &= \frac{P(BA)}{P(A)} \\
&= \frac{P(A \mid B)P(B)}{P(A \mid B)P(B) + P(A \mid B^\complement)[1-P(B)]}
\end{align}

### Definition
$$
\frac{P(A)}{P(A^\complement)} = \frac{P(A)}{1-P(A)}
$$
The odds of an event **A** tell how much more likely it is that the event **A** occurs than it is that it does not occur

### 3.3.3
$$
\frac{P(B \mid A)}{P(B^\complement \mid A)} = \frac{P(B)}{P(B^\complement)} \frac{P(A \mid B)}{P(A \mid B^\complement)}
$$

### 3.3.4 1st Bayes
\begin{align}
P(A) &= \sum_{i=1}^{n} P(AB_i) \\
&= \sum_{i=1}^{n} P(A \mid B_i)P(B_i)
\end{align}

### 3.3.5 2nd Bayes
\begin{align}
P(B_j \mid A) &= \frac{P(AB_j)}{P(A)} \\
&= \frac{P(A \mid B_i)P(B_i)}{\sum_{i=1}^{n} P(A \mid B_i)P(B_i)}
\end{align}

---


# Performance metrics in multi-objective optimization
> Implementing and comparing various performance metrics in multi-objective optimization.

- toc: true
- categories: [multi-objective optimization]
- image: images/hypervolume.png

In multi-objective optimization, the goal is to simultaneously optimize $M$ different and generally conflicting objectives.
Since there is no single solution that is optimal in all objectives, we are instead interested in the Pareto front, composed of the set of points in objective space that are not dominated by any other point ($a$ dominates $b$ if $a$ is equal or better than $b$ in all objectives, and strictly better in at least one objective).
The goal is thus to find a good approximation of the true Pareto front with finite ressources.

To define what a good approximation is, there are three properties that we care for:
1. **Convergence**: The individual points should be optimal, meaning that the distance to the true Pareto front is minimal.
2. **Spread**: The entire front should be found. The distance between extreme solutions should be maximized.
3. **Distribution**: The Pareto front should be uniformly covered.

In the following we look at different performance metrics for approximate fronts.

## Illustrative example
As example, we consider two objectives to be minimized, where the true Pareto-front forms a quarter-circle of radius $r$ around a center point.
Some of the metrics require a reference front, for which we take the true Pareto-front sampled in 100 equidistant steps.

Now, some optimization algorithms we ran resulted in two approximate fronts $A_1$ and  $A_2$, that we wish to compare. In 2D we can still do this visually and see that set 1 is better in objective 1, whereas set 2 is better in objective 2. The solutions of set 1 are better on average, whereas set 2 has more diverse solutions.

```python
#collapse-hide
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('png')
import pygmo

# two approximate fronts
A1 = np.array([[0.22, 0.85], [0.3, 0.7], [0.4, 0.5], [0.5, 0.45], [0.6, 0.4], [0.8, 0.35]])
A2 = np.array([[0.3, 1.15], [0.35, 0.8], [0.5, 0.6], [0.7, 0.3], [1.0, 0.18], [1.1, 0.15]])

# true Pareto front uniformly sampled by 100 points
φ = np.linspace(0, np.pi / 2, 100)
R = np.array([1.2, 1.2]) - 1.1 * np.array([np.sin(φ), np.cos(φ)]).T

def plot_pareto_front(ax=None):
    if ax is None:
        fig, ax = plt.subplots(1, figsize=(6, 6))
    ax.fill_between(R.T[0], R.T[1], 1.2, color='gray', alpha=0.2)
    ax.plot(*R.T, color='k')
    ax.set(xlim=(0, 1.2), ylim=(0, 1.2), xlabel='$y_1$', ylabel='$y_2$')
    ax.grid()
    ax.set_axisbelow(True)
    return ax

def plot_attainment(A, ax=None):
    """Plot the attainment surface for m=2 objectives"""
    if ax is None:
        fig, ax = plt.subplots(1, figsize=(6, 6))
    N, m = A.shape
    assert m == 2, ValueError("A has to be of shape (n, 2)")
    S = np.zeros((N + 2, 2))
    S[1:-1] = A[np.argsort(A[:, 0])]  # sort along x-axis
    S[0] = np.min(A[:, 0]), 1.2
    S[-1]= 1.2, np.min(A[:, 1])
    ax.step(*S.T, where='post')

ax = plot_pareto_front()
ax.scatter(*A1.T, label='$A_1$')
plot_attainment(A1, ax=ax)
ax.scatter(*A2.T, label='$A_2$')
plot_attainment(A2, ax=ax)
ax.legend();
```

## Hypervolume indicator

The hypervolume indicator {% cite Zitzler1999 %} measures the volume in the objective space that is dominated by a set, bounded by a reference point.
Larger values indicate better convergence, spread and distribution.
The hypervolume is sensitive to the choice of the reference point. For more distant reference points, the difference in dominated hypervolume decreases.
In practice we can take a reference point that is slightly worse than the combination of the worst objective values from the evaluated Pareto front (the nadir).
A more principled choice of the reference point is given in {% cite Ishibuchi2018 %}.

Choosing (1.2, 1.2) as reference point we get that set 1 covers a significantly larger hypervolume than set 2.

```python
#collapse-hide
ref_point = [1.2, 1.2]
print(f'Set 1: volume={pygmo.hypervolume(A1).compute(ref_point):.3f}')
print(f'Set 2: volume={pygmo.hypervolume(A2).compute(ref_point):.3f}')
```

    Set 1: volume=0.723
    Set 2: volume=0.659

The difference in hypervolume between set 1 and 2 shrinks if we choose a more distant reference point.

```python
#collapse-hide
ref_point = [1.5, 1.5]
print(f'Set 1: volume={pygmo.hypervolume(A1).compute(ref_point):.3f}')
print(f'Set 2: volume={pygmo.hypervolume(A2).compute(ref_point):.3f}')
```

    Set 1: volume=1.362
    Set 2: volume=1.335

## Exclusive hypervolume contribution

The exclusive hypervolme contribution measures the volume that is exclusively dominated by a given point in a set.
For the extremal points a reference point is again necessary to define a volume. If the nadir is chosen as reference point, the extremal points have 0 exclusive contribution. For all non-extremal points the contribution is independent of the reference. Here, small values indicate that a point carries little additional information and can be pruned away.

```python
#collapse-hide
def plot_hypervolume(points, ref_point, ax=None):
    contributions = pygmo.hypervolume(points).contributions(ref_point)

    if ax is None:
        fig, ax = plt.subplots(1, figsize=(6, 6))

    # plot dominated volume for each point
    for p in points:
        w, h = ref_point - p
        ax.add_patch(plt.Rectangle(p, w, h, alpha=0.3))

    # annotate each point with its exclusive contribution
    for p, c in zip(points, contributions):
        ax.annotate(f"{c:.3f}", p)

    ax.plot(points[:, 0], points[:, 1], 'ko', label='front')
    ax.plot(ref_point[0], ref_point[1], 'ro', label='reference')

ref_point = [1.2, 1.2]
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12.5, 6))
plot_pareto_front(ax=ax1)
plot_pareto_front(ax=ax2)
plot_hypervolume(A1, ref_point, ax=ax1)
plot_hypervolume(A2, ref_point, ax=ax2)
ax1.set_title(f'$A_1$: volume={pygmo.hypervolume(A1).compute(ref_point):.3f}')
ax2.set_title(f'$A_2$: volume={pygmo.hypervolume(A2).compute(ref_point):.3f}');
```

## Crowding distance

The crowding distance {% cite Deb2000 %} calculates for each point in a non-dominated set the average side length of the cuboid formed by the neighboring points. This provides a measure of the local density at each point. Points with small crowding distance can be pruned out without losing too much information. In a uniformly populated front, each point will have a similar crowding distance.
The crowding distance is thus similar to the exclusive hypervolume contribution, only that here the side length and not the volume is considered.

```python
#collapse-show
def crowding_distance(A):
    N, m = A.shape

    # no crowding distance for lesss than 2 points
    if N <= 2:
        return np.full(N, np.inf)

    # sort points along each objective
    sort = np.argsort(A, axis=0)
    A = A[sort, np.arange(m)]

    # normalize all objectives
    norm = np.max(A, axis=0) - np.min(A, axis=0)
    A = A / norm
    A[:, norm == 0] = 0  # handle min = max

    # distance to previous and to next point along each objective
    d = np.diff(A, axis=0)
    inf = np.full((1, m), np.inf)
    d0 = np.concatenate([inf, d])
    d1 = np.concatenate([d, inf])

    # cuboid side length = sum of distance to previous and next point
    unsort = np.argsort(sort, axis=0)
    cuboid = d0[unsort, np.arange(m)] + d1[unsort, np.arange(m)]
    return np.mean(cuboid, axis=1)
```

Calculating the crowding distances in our example, we can quantify our observation that the points are more evenly distributed in set 2. In set 1 there is a point with a significantly smaller crowding distance we could prune away.

One might wonder why both sets have similar crowding distance values even though the cuboids (i.e. distances to neighboring points) are larger in set 2. This is because the sides of the cuboids are normalized to the objective value range in the approximate front and set 2 has a larger spread.

```python
#collapse-hide
def plot_crowding(A, ax=None):
    if ax is None:
        fig, ax = plt.subplots(1, figsize(6, 6))
    N, m = A.shape
    for i in range(1, N - 1):
        a0 = A[i - 1]
        a1 = A[i + 1]
        ax.add_patch(plt.Rectangle(a0, *(a1 - a0), alpha=0.3, linewidth=2))
    ax.scatter(*A.T, color='k')
    for a, d in zip(A, crowding_distance(A)):
        ax.annotate(f"{d:.2f}", a)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12.5, 6))
for ax, A in zip((ax1, ax2), (A1, A2)):
    plot_pareto_front(ax=ax)
    plot_crowding(A, ax=ax)
ax1.set_title(f'$A_1$')
ax2.set_title(f'$A_2$');
```

## Epsilon-indicator

An objective vector $a$ is said to $\epsilon$-dominate $b$ if $a_i < \epsilon \cdot b_i ~ \forall i=1 \ldots m$ where $\epsilon < 1$ is a scalar.
The binary $\epsilon$-indicator $I_\epsilon(A, B)$ is defined as the smallest $\epsilon$ value for which any vector in $B$ is $\epsilon$-dominated by a vector in $A$ {% cite Zitzler2003 %}.
\begin{align}
I_\epsilon(A, B) = \max_\limits{b \in B} \min_\limits{a \in A} \max_\limits{1 \leq i \leq m} \frac{a_i}{b_i}
\end{align}
The unary $\epsilon$-indicator is defined as $I_{\epsilon 1}(A) = I_\epsilon(A, R)$ where $R$ is the reference front (Pareto-front or a good approximation to it).

If we use the additive $\epsilon$-dominance ($a_i < b_i + \epsilon ~ \forall i = 1 \ldots m$) instead, we get the additive $\epsilon$-indicator.
\begin{align}
I_{\epsilon+}(A, B) = \max_\limits{b \in B} \min_\limits{a \in A} \max_\limits{1 \leq i \leq m} (a_i - b_i)
\end{align}

```python
#collapse-show
def epsilon_indicator(A, B, additive=False):
    if additive:
        r = A[:, np.newaxis, :] - B[np.newaxis, :, :]
    else:  # multiplicative
        r = A[:, np.newaxis, :] / B[np.newaxis, :, :]
    return r.max(axis=2).min(axis=0).max()
```

For the multiplicative $\epsilon$-indicator we get

```python
#collapse-hide
print(f'Set 1: I_ε1={epsilon_indicator(A1, R):.3f}')
print(f'Set 2: I_ε1={epsilon_indicator(A2, R):.3f}')
```

    Set 1: I_ε1=3.500
    Set 2: I_ε1=3.000

For the additive $\epsilon$-indicator we get

```python
#collapse-hide
print(f'Set 1: I_ε1={epsilon_indicator(A1, R, True):.3f}')
print(f'Set 2: I_ε1={epsilon_indicator(A2, R, True):.3f}')
```

    Set 1: I_ε1=0.250
    Set 2: I_ε1=0.220

## Generational distance

The generational distance (GD) {% cite Veldhuizen2000 %} is defined as the average distance of points in the approximate front $A$ to a reference front $R$, which is either the actual Pareto front or a very good approximation to it.

\begin{align}
\mathrm{GD}(A, R) = \frac{1}{N_A} \left(\sum\limits_{a \in A} d(a, R)^p \right)^{1/p}
\end{align}
where $d(a, R)$ is the euclidean distance of $a$ to the reference front.
With the default choice of $p=1$, GD is simply the average distance.

GD is a measure of the convergence.

```python
#collapse-show
def distance(A, B):
    """Calculate for each point in A the distance to B"""
    d = A[:, np.newaxis] - B[np.newaxis]  # shape (len(A), len(B), m)
    return np.linalg.norm(d, axis=2).min(axis=1)

def generational_distance(A, R, p=1):
    return np.linalg.norm(distance(A, R), p) / len(A)
```

In the example, set 1 achieves a better GD-value as its points are closer to the Pareto front (better convergence).

```python
#collapse-hide
print(f'GD(A,R1) = {generational_distance(A1, R):.3f}')
print(f'GD(A,R2) = {generational_distance(A2, R):.3f}')
```

    GD(A,R1) = 0.084
    GD(A,R2) = 0.119

## Inverted generational distance

The inverted generationl distance (IGD) {% cite Coello2004 %} simply reverses the role of approximate front and reference front, i.e. $\mathrm{IGD}(A, R) = \mathrm{GD}(R, A)$. Thus it calculates the average distance between each point in the reference front $R$ to the closest point in the approximated front $A$.

IGD simultaneously measures convergence, spread and distribution.
In the example, set 2 achieves a better IGD-value due to its better spread and distribution.

```python
#collapse-hide
print(f'IGD(A1,R) = {generational_distance(R, A1):.3f}')
print(f'IGD(A2,R) = {generational_distance(R, A2):.3f}')
```

    IGD(A1,R) = 0.174
    IGD(A2,R) = 0.157

## Modified (inverted) generational distance

The GD / IGD metrics cannot differentiate the quality of points in the approximate front when they are non-dominated by the reference front.
To fix this, a modified distance measure can be used {% cite Ishibuchi2015 %}
\begin{align}
d^+(a, R) = \min_{r \in R} \sqrt{ \sum_{i=1}^m \max(a_i - r_i, 0)^2 }
\end{align}
i.e. setting negative components of the difference $a - r$ to zero before calculating the euclidean distance.
With this modification we get the modified generational distance GD$^+$ and the modified inverted generational distance IGD$^+$.

```python
#collapse-show
def modified_distance(A, B):
    """Calculate for each point in A the modified distance to B"""
    d = (A[:, np.newaxis] - B[np.newaxis]).clip(0, None)
    return np.linalg.norm(d, axis=2).min(axis=1)

def modified_generational_distance(A, R, p=1):
    return np.linalg.norm(modified_distance(A, R), p) / len(A)
```

In our example the approximate fronts are dominated by the reference front, so the GD / GD$^+$ and IGD / IGD$^+$ are the same.
To see the effect we add the infeasible point (0.3, 0.3) to the approximate front.

```python
#collapse-hide
A3 = np.r_[A1, [[0.3, 0.3]]]
ax = plot_pareto_front()
ax.scatter(*A3.T, label='$A_3$')
ax.legend();
```

Although the approximate front is better with this point than without, the GD increases (IGD stays the same since for each point in $R$ there is a closer point than the newly added one). In contrast the modified versions correctly indicate an improvement.

```python
#collapse-hide
print(f'GD(A1,R) = {generational_distance(A1, R):.3f}')
print(f'GD(A3,R) = {generational_distance(A3, R):.3f}')
print(f'GD+(A3,R) = {modified_generational_distance(A3, R):.3f}')
```

    GD(A1,R) = 0.084
    GD(A3,R) = 0.097
    GD+(A3,R) = 0.072

```python
#collapse-hide
print(f'IGD(A1,R) = {generational_distance(R, A1):.3f}')
print(f'IGD(A3,R) = {generational_distance(R, A3):.3f}')
print(f'IGD+(A3,R) = {modified_generational_distance(R, A3):.3f}')
```

    IGD(A1,R) = 0.174
    IGD(A3,R) = 0.174
    IGD+(A3,R) = 0.086

## Attainment surface quantiles

The approximate front is also called attainment surface. For several runs of an multi-optimization algorithm, the *quantiles* of the approximate front can be calculated, indicating the boundaries of the dominated volume that are likely to be attained with some probability.

In 2 and 3 dimensions attainment surfaces can be used for visual inspection. For quantification and higher dimensions they can be used for binary metrics in the form of statistical comparisons.

## References

{% bibliography --cited %}