# Function Representation and Manipulation

```
%matplotlib inline
```

```
import numpy as np
import matplotlib
matplotlib.rcParams.update({'font.size': 14})
import matplotlib.pyplot as plt
```

From a mathematical point of view, a central point in numerical methods is how we represent a general function $f(x)$. As $x$ is a real number the function encodes an (uncountably) infinite amount of information: its value $f(x)$ for every point $x$. A computer can only usefully store and manipulate a finite amount of information, so it would appear impossible to represent even the simplest function of one variable.

Of course, most functions can be represented straightforwardly using little information. We often see functions that can be represented symbolically - $f(x) = x^2 + x$ is a simple example - but numerical methods are precisely for cases where symbolic methods are not applicable. Instead we must consider how a function is represented, or approximated, in terms of a finite amount of information.

Two standard representations can be used to make the central point. The first representation is in terms of polynomials, using Taylor series:

\begin{equation}
  f(x ; x_0) = \sum_{n=0}^{\infty} \frac{c_n}{n!} (x - x_0)^n.
\end{equation}

Here $x_0$ is a parameter about which the Taylor series is centred.

The second representation is in terms of trigonometric functions, using Fourier series. For simplicity we consider only the Fourier cosine series

\begin{equation}
  f(x ; L) = \frac{1}{2} a_0 + \sum_{n=1}^{\infty} a_n \cos \left( \frac{n \pi x}{L} \right).
\end{equation}

The Fourier series representation is $2L$-periodic.

Given a set of coefficients ($\{c_n\}$ for the Taylor series or $\{a_n\}$ for the Fourier series) we define a function.

```
an = np.zeros((10, 1))
cn = np.zeros_like(an)
for n in range(len(an)):
    an[n] = 1.0 / (n + 1)**2
    cn[n] = an[n]
x = np.linspace(-1.0, 1.0, 100)
taylor = an[0]*np.ones_like(x)
fourier = cn[0]*np.ones_like(x)
for n in range(1, len(an)):
    taylor += an[n] * x**n
    fourier += cn[n] * np.cos(n * np.pi * x)
fig = plt.figure(figsize = (8,6))
ax = fig.add_subplot(111)
ax.plot(x,taylor,label = 'Taylor')
ax.plot(x,fourier, label = 'Fourier')
plt.legend()
ax.set_xlabel('$x$')
plt.show()
```

Given a function, the standard analytical methods will give you the coefficients. This can rarely be directly applied in numerical methods - as noted above, it is usual that you're dealing with a function that isn't known in symbolic form.

Above, when we defined the function $f(x)$ from the coefficients, we were noting that we could evaluate the representation for any point $x$ given

1. the form of the representation (e.g., as a Taylor series about $x_0=0$), and
2. the value of the coefficients.

We can (nearly always) go in the opposite direction: given the value of the function at enough points $x_n$ we can compute the coefficients of the representation.

For example, let us assume that

1. the function will be represented as a polynomial of order 2, i.e. a Taylor series about $x_0=0$ such that all coefficients $c_n$ vanish for $n > 2$, and
2. the value of the function is known at three points $x_1, x_2, x_3$.

This is precisely the information needed to compute the coefficients $c_0, c_1, c_2$ and hence write the function as

\begin{equation}
  f(x) \approx c_0 + c_1 x + \frac{c_2}{2} x^2.
\end{equation}

Explicitly, we note that the function representation matches the known value of the function at the three points only if

\begin{equation}
  \begin{pmatrix} 1 & x_1 & \tfrac{1}{2} x_1^2 \\ 1 & x_2 & \tfrac{1}{2} x_2^2 \\ 1 & x_3 & \tfrac{1}{2} x_3^2 \end{pmatrix} \begin{pmatrix} c_0 \\ c_1 \\ c_2 \end{pmatrix} = \begin{pmatrix} f(x_1) \\ f(x_2) \\ f(x_3) \end{pmatrix}
\end{equation}

which, given the information we know, is a linear system for the coefficients.

Note in particular that the number of points at which the function is known must match the number of coefficients in the function representation.

The previous example hints at one of the standard methods of numerically representing functions. First, we take a representation that we know, given an infinite amount of information, converges to the function we want (in some sense). Second, assume that taking a *finite* number of terms is sufficient: e.g., truncate the sum in the Taylor series expansion after a finite number of terms. Then, using a finite amount of information about the function (such at its value at an appropriate number of points) we compute the coefficients of the representation. We then manipulate the representation *as if* it were the true function.

```

```

Lectures/Function Representation and Manipulation.ipynb


# Kinematics

```python
import numpy as np
from sympy import *
from IPython.display import display
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

%matplotlib notebook
init_printing()
```

### Helpers

Create rotational matrix generator.

```python
def rotate(p, theta, o, direction):
    """
    Rotate a point about an direction through a point.
    Represents a counterclockwise rotation by theta about the line passing through point pointing in direction.
    :param p: The point to be rotated.
    :param theta: How much to rotate p.
    :param o: The origin of the rotational axis.
    :param direction: The direction of the rotational axis.
    """

    # Ensure direction is matrix (for norm).
    direction = Matrix(direction)

    # Unpack.
    u, v, w = direction / direction.norm()
    x, y, z = p
    a, b, c = o

    x1 = (a*(v**2+w**2)-u*(b*v+c*w-u*x-v*y-w*z))*(1-cos(theta))+x*cos(theta)+(-c*v+b*w-w*y+v*z)*sin(theta)
    y1 = (b*(u**2+w**2)-v*(a*u+c*w-u*x-v*y-w*z))*(1-cos(theta))+y*cos(theta)+(c*u-a*w+w*x-u*z)*sin(theta)
    z1 = (c*(u**2+v**2)-w*(a*u+b*v-u*x-v*y-w*z))*(1-cos(theta))+z*cos(theta)+(-b*u+a*v-v*x+u*y)*sin(theta)

    return Matrix((x1, y1, z1))
```

Create pretty simplification function.

```python
def pretty(eqn, expressions, symbols):
    """
    Does pretty simplification by replacing expressions in a expression or matrix with symbols.
    :param eqn: The desired input expression or matrix.
    :param expressions: The expressions to replace.
    :param symbols: The symbols to use.
    """

    eqn = eqn.copy()

    for expr, sub in zip(expressions, symbols):
        eqn = eqn.subs(expr, sub)

    return simplify(eqn)
```

Create plotting functions.

```python
def equal_ax(ax):
    extents = np.array([getattr(ax, 'get_{}lim'.format(dim))() for dim in 'xyz'])
    sz = extents[:,1] - extents[:,0]
    centers = np.mean(extents, axis=1)
    maxsize = max(abs(sz))
    r = maxsize/2
    for ctr, dim in zip(centers, 'xyz'):
        getattr(ax, 'set_{}lim'.format(dim))(ctr - r, ctr + r)

def plot_arm(points):
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')

    ax.set_xlabel('X axis')
    ax.set_ylabel('Y axis')
    ax.set_zlabel('Z axis')

    xs, ys, zs = zip(*points)
    ax.plot(xs, ys, zs, 'go-')

    ax.view_init(30, 150)
    equal_ax(ax)

    plt.show()
```

### Preliminary Tests

Run test cases for rotations about origin. Ensure that the rotate function works.

```python
theta = Symbol('theta')
nx = (1, 0, 0)
ny = (0, 1, 0)
nz = (0, 0, 1)
o = (0, 0, 0)
```

```python
# Rotate p about x.
p = Matrix((1, 0, 0))
rotate(p, theta, o, nx)
```

$$\left[\begin{matrix}1\\0\\0\end{matrix}\right]$$

```python
# Rotate p about y.
p = Matrix((1, 0, 0))
rotate(p, theta, o, ny)
```

$$\left[\begin{matrix}\cos{\left (\theta \right )}\\0\\- \sin{\left (\theta \right )}\end{matrix}\right]$$

```python
# Rotate p about z.
p = Matrix((1, 0, 0))
rotate(p, theta, o, nz)
```

$$\left[\begin{matrix}\cos{\left (\theta \right )}\\\sin{\left (\theta \right )}\\0\end{matrix}\right]$$

### Forward Kinematics

Define lengths for arm segments. Graph points for base condition.

```python
l1 = 10
l2 = 10
l3 = 2
l4 = 5

p0 = (0, 0, 0)
p1 = (0, 0, l1)
p2 = (0, 0, l1 + l2)
p3 = (0, l3, l1 + l2)
p4 = (0, l3, l1 + l2 + l4)

points = [p0, p1, p2, p3, p4]

plot_arm(points)
```

    <IPython.core.display.Javascript object>

Define all variables and angles. $l_3$ can be negative depending on design and assembly.

```python
theta1, theta2, theta3, theta4, l3 = symbols('theta_1 theta_2 theta_3 theta_4 l_3', real=True)
l1, l2, l4 = symbols('l_1 l_2 l_4', positive=True)

o0 = (0, 0, 0)
o1 = (0, 0, l1)
o3 = (0, l3, l1 + l2)
```

Solving for $p_0$ is trivial. It is always at the defined origin.

```python
p0 = Matrix((0, 0, 0))
p0
```

$$\left[\begin{matrix}0\\0\\0\end{matrix}\right]$$

To solve for $p_1$, account for $\theta_1$ and $\theta_2$.

```python
p1 = Matrix((0, 0, l1))
p1_sym = symbols('p_1x p_1y p_1z')

p1 = rotate(p1, theta2, o0, ny)
p1 = rotate(p1, theta1, o0, nz)

p1 = p1.expand()

simplify(p1)
```

$$\left[\begin{matrix}l_{1} \sin{\left (\theta_{2} \right )} \cos{\left (\theta_{1} \right )}\\l_{1} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} \right )}\\l_{1} \cos{\left (\theta_{2} \right )}\end{matrix}\right]$$

To solve for $p_2$, account for $\theta_1$ through $\theta_3$ (inclusive).

```python
p2 = Matrix((0, 0, l1 + l2))
p2_sym = symbols('p_2x p_2y p_2z')

p2 = rotate(p2, theta3, o1, ny)
p2 = rotate(p2, theta2, o0, ny)
p2 = rotate(p2, theta1, o0, nz)

p2 = p2.expand()

pretty(p2, p1, p1_sym)
```

$$\left[\begin{matrix}l_{2} \sin{\left (\theta_{2} + \theta_{3} \right )} \cos{\left (\theta_{1} \right )} + p_{1x}\\l_{2} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} + \theta_{3} \right )} + p_{1y}\\l_{2} \cos{\left (\theta_{2} + \theta_{3} \right )} + p_{1z}\end{matrix}\right]$$

To solve for $p_3$, account for the same rotations as $p_2$.

```python
p3 = Matrix((0, l3, l1 + l2))
p3_sym = symbols('p_3x p_3y p_3z')

p3 = rotate(p3, theta3, o1, ny)
p3 = rotate(p3, theta2, o0, ny)
p3 = rotate(p3, theta1, o0, nz)

p3 = p3.expand()

pretty(p3, p2, p2_sym)
```

$$\left[\begin{matrix}- l_{3} \sin{\left (\theta_{1} \right )} + p_{2x}\\l_{3} \cos{\left (\theta_{1} \right )} + p_{2y}\\p_{2z}\end{matrix}\right]$$

To solve for $p_4$, account for the all rotations.

```python
p4 = Matrix((0, l3, l1 + l2 + l4))

p4 = rotate(p4, theta4, o3, ny)
p4 = rotate(p4, theta3, o1, ny)
p4 = rotate(p4, theta2, o0, ny)
p4 = rotate(p4, theta1, o0, nz)

p4 = p4.expand()

pretty(p4, p3, p3_sym)
```

$$\left[\begin{matrix}l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} \cos{\left (\theta_{1} \right )} + p_{3x}\\l_{4} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} + p_{3y}\\l_{4} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} + p_{3z}\end{matrix}\right]$$

Graph some test conditions.

```python
def graph_fk(lengths, angles):
    sub = [(theta1, angles[0]), (theta2, angles[1]), (theta3, angles[2]), (theta4, angles[3]),
           (l1, lengths[0]), (l2, lengths[1]), (l3, lengths[2]), (l4, lengths[3])]
    points = [p0, p1.subs(sub), p2.subs(sub), p3.subs(sub), p4.subs(sub)]

    plot_arm(points)

    return points
```

```python
lengths = (10, 10, 2, 5)
angles = (0, 0, 0, 0)
graph_fk(lengths, angles)
```

    <IPython.core.display.Javascript object>

$$\left [ \left[\begin{matrix}0\\0\\0\end{matrix}\right], \quad \left[\begin{matrix}0\\0\\10\end{matrix}\right], \quad \left[\begin{matrix}0\\0\\20\end{matrix}\right], \quad \left[\begin{matrix}0\\2\\20\end{matrix}\right], \quad \left[\begin{matrix}0\\2\\25\end{matrix}\right]\right ]$$

### Inverse Kinematics

Given a 4 DOF system, there needs to be additional constraints aside from a point in space to avoid obtaining infinite solutions. For a general hand-like robot, it is helpful to define the direction of the vector created by the last two points. Redefine points $p_0$ to $p_4$ as $A$ to $E$.

The directional constraint, $\phi$, defines the angle formed by the vector $\overrightarrow{DE}$ and the normal of the XY-plane (Z axis). The point $(x, y, z)$ defines the desired target point in space.

```python
x, y, z, phi = symbols('x y z phi', real=True)
```

Given $\phi$, it is possible to solve for $C$. Drawing out the diagram basically reveals two similar right triangles. Connecting all of the vertices reveals a trapazoid. The two diagonals are known. Two sides are known. Let the sides be $(a,b,c,d)$. $a = l_4\sin{\left(\phi\right)}$ and $d = \sqrt{a^2 + l_3^2}$. The diagonals are $l_3$ and the distance of the target from the origin. $c$ can be solved using the equations detailed [here](http://mathworld.wolfram.com/Trapezoid.html).

```python
a, b, c, d = symbols('a, b, c, d', real=True)
c1 = solve(l3**2 - (a*b**2-a**2*b-a*c**2+b*d**2)/(b-a), c)[1]
c1
```

```python
c2 = solve(x**2+y**2 - (a*b**2-a**2*b-a*d**2+b*c**2)/(b-a), c)[1]
c2
```

Setting both of the above equations equal to each other allows the solution for $b$ to be found. We need to find $\theta_0$, the angle difference caused by $l_3$. We can solve for the opposite leg of the right triangle and use it along with $b$ to find the desired angle.

```python
u = Symbol('u', positive=True)
u_soln = solve(a / (l3 - u) - b / u, u)[0]
u_soln
```

```python
b_soln = solve(c1 - c2, b)[1]
simplify(b_soln.expand())
```

```python
theta_0 = atan(u_soln.subs(b, b_soln) / b_soln).subs([(a, l4*sin(phi)),(d, sqrt(l3**2+(l4*sin(phi))**2))])
simplify(theta_0.expand())
```

We need to rewrite this equation because $\phi$ can cause this equation to be undefined and it is known analytically that $\phi$ does not play a role here.

```python
theta_0 = atan2(l3, sqrt(x**2+y**2-l3**2))
theta_0
```

The next step is to actually solve for C. We can define the vector $\overrightarrow{V} = \overrightarrow{CE}$. $C$ is really just $\overrightarrow{OC}$. Therefore, $\overrightarrow{C} = \overrightarrow{T} - \overrightarrow{CE}$. The vector $\overrightarrow{CE}$ can be described as two rotations of $\overrightarrow{{CE}_{0}}$ -- one rotation about the Z-axis and another rotation about a rotated Y-axis vector. $\overrightarrow{D}$ can be solved in a similar manner. Let $\overrightarrow{n}$ be the the Y-axis normal vector rotated by $R = \operatorname{atan2}{\left(y, x\right)} - \theta_0$.

The interesting thing to realize here is that there are two possible cases. In one case, the arm is not flipped. In the other case it is. We need to account for both cases.

```python
theta0 = Symbol('theta_0', real=True)

# Define CD vector.
CD = Matrix((0, 0, l4))

# Define D.
D = Matrix((x, y, z)) - CD
D_sym = symbols('D_x D_y D_z', real=True)

# Define oE.
oE = (x, y, z)

# Rotate ny about Z-axis.
R1 = atan2(y, x) - theta0
R2 = atan2(y, x) + theta0

n1 = rotate(ny, R1, o, nz)
n2 = rotate(ny, R2, o, nz)
n = [n1, n2]

# Rotate D about rotated Y-normal.
D1 = rotate(D, phi, oE, n1)
D2 = rotate(D, phi, oE, n2)

D1 = simplify(D1.expand())
D2 = simplify(D2.expand())

D = [D1, D2]

# Output.
D
```

$$\left [ \left[\begin{matrix}- l_{4} \sin{\left (\phi \right )} \cos{\left (\theta_{0} - \operatorname{atan_{2}}{\left (y,x \right )} \right )} + x\\l_{4} \sin{\left (\phi \right )} \sin{\left (\theta_{0} - \operatorname{atan_{2}}{\left (y,x \right )} \right )} + y\\- l_{4} \cos{\left (\phi \right )} + z\end{matrix}\right], \quad \left[\begin{matrix}- l_{4} \sin{\left (\phi \right )} \cos{\left (\theta_{0} + \operatorname{atan_{2}}{\left (y,x \right )} \right )} + x\\- l_{4} \sin{\left (\phi \right )} \sin{\left (\theta_{0} + \operatorname{atan_{2}}{\left (y,x \right )} \right )} + y\\- l_{4} \cos{\left (\phi \right )} + z\end{matrix}\right]\right ]$$

```python
n
```

$$\left [ \left[\begin{matrix}\sin{\left (\theta_{0} - \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\\cos{\left (\theta_{0} - \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\0\end{matrix}\right], \quad \left[\begin{matrix}- \sin{\left (\theta_{0} + \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\\cos{\left (\theta_{0} + \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\0\end{matrix}\right]\right ]$$

```python
# Define CE vector.
CE1 = Matrix((0, l3, l4))
CE2 = Matrix((0, -l3, l4))

# Define C.
C1 = Matrix((x, y, z)) - CE1
C2 = Matrix((x, y, z)) - CE2

C_sym = symbols('C_x C_y C_z', real=True)

# Rotate C about Z-axis.
C1 = rotate(C1, R1, oE, nz)
C2 = rotate(C2, R2, oE, nz)

# Rotate C about rotated Y-normal.
C1 = rotate(C1, phi, oE, n1)
C2 = rotate(C2, phi, oE, n2)

C1 = simplify(C1.expand())
C2 = simplify(C2.expand())

C = [C1, C2]

# Output.
[pretty(C[0], D[0], D_sym), pretty(C[1], D[1], D_sym)]
```

$$\left [ \left[\begin{matrix}D_{x} - l_{3} \sin{\left (\theta_{0} - \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\D_{y} - l_{3} \cos{\left (\theta_{0} - \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\D_{z}\end{matrix}\right], \quad \left[\begin{matrix}D_{x} - l_{3} \sin{\left (\theta_{0} + \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\D_{y} + l_{3} \cos{\left (\theta_{0} + \operatorname{atan_{2}}{\left (y,x \right )} \right )}\\D_{z}\end{matrix}\right]\right ]$$

Given $C$, it is possible to use the law of cosines to solve for $\theta_3$. $\theta_3$ can either be positive or negative.

```python
theta_3 = pi - acos((l1**2 + l2**2 - C[1].norm()**2) / (2 * l1 * l2))

sol1 = pretty(theta_3, C[1], C_sym)
sol2 = -sol1
theta_3 = [sol1, sol2]
theta_3
```

We can solve for $\theta_2$ by assuming that $B$ still rests on the Z-axis. Define $C^{\prime\prime}$ as the point $C$ with only $\theta_3$. Define $C^{\prime}$ as the point $C$ with only $\theta_2$ and $\theta_3$. $C^{\prime}_z$ must be equal to $C_z$ to form a radius for $\theta_1$ to complete. It is better to solve for the equation involving z rather than x and y, because $\theta_1$ does not impact z.

```python
C_pp = (l2 * sin(theta3), 0, l1 + l2 * cos(theta3))
C_p = rotate(C_pp, theta2, o, ny)
theta_2 = solve(C_p[2] - C_sym[2], theta2)
theta_2
```

It is not immediately evident which solution to pick for the non-flipped vs. flipped. In fact, it basically is not evident at all. Test shows that the first solution works for non-flipped and the second for flipped. However, they flip when $x^2 + y^2 < l_3^2 + (l_4 \sin{(\phi)})^2$ and $\sin{(\phi)} > 0$. This is an observed result from extensive testing. I am sure there is good mathematical justification, but it is probably to difficult to derive from equations.

$\theta_1$ can actually be by solved using the forward kinematic equations for $C$. Assuming $\theta_1$ is 0, compute the current angle with the x-axis. Then, simply swing around. It might be tempting to use $C$, but there can be issues because it can sometimes be on the Z-axis. Using $D$ is better as it cannot be on the Z-axis unless $l_3$ is zero (in that case it does't matter anyway since $\theta_0$ will be zero).

```python
theta_1 = atan2(D_sym[1], D_sym[0]) - atan2(p3[1].subs(theta1, 0), simplify(p3[0].subs(theta1, 0)))
theta_1
```

We can now solve for $\theta_4$. Define two vectors, $\overrightarrow{DE}$ and $\overrightarrow{BC}$. $B$ can be solved using the forward kinematics equation. The dot product can be used to determine the angle. The sign can be determined using the cross product. Define $s$ as the sign. Account for multiple solutions due to inverse cosine. Testing reveals that the second solution applies to the flipped case.

$$s = \operatorname{sgn}{\left[ \overrightarrow{n} \cdot \left( \overrightarrow{BC} \times \overrightarrow{DE} \right) \right] }$$

$$
\left[
s \cdot \operatorname{acos}{\left( \frac{\overrightarrow{BC} \cdot \overrightarrow{DE} }{l_2 l_4} \right)}, \quad
-s \cdot \operatorname{acos}{\left( \frac{\overrightarrow{BC} \cdot \overrightarrow{DE} }{l_2 l_4} \right)}
\right]
$$

For most constraint conditions of $(x, y, z, \phi)$, there are at most four solutions (and usually four solutions). The kinematic solution is stable around singularities. In cases where less than four solutions exist (i.e. one or two), all four equations will agree on these solutions. There is no known case of three unique solutions.

### Jacobian Analysis

We can define the forward Jacobian matrix to assist in computing velocities and torques. First, we need to compute the Jacobian for the linear velocity, $J_v(\theta)$.

```python
wrt = Matrix((theta1, theta2, theta3, theta4))
J_v = simplify(p4).jacobian(wrt)
J_v
```

$$\left[\begin{matrix}- l_{1} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} \right )} - l_{2} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} + \theta_{3} \right )} - l_{3} \cos{\left (\theta_{1} \right )} - l_{4} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{1} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} \right )} + l_{2} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{2} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{4} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )}\\l_{1} \sin{\left (\theta_{2} \right )} \cos{\left (\theta_{1} \right )} + l_{2} \sin{\left (\theta_{2} + \theta_{3} \right )} \cos{\left (\theta_{1} \right )} - l_{3} \sin{\left (\theta_{1} \right )} + l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} \cos{\left (\theta_{1} \right )} & l_{1} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} \right )} + l_{2} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{2} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{4} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )}\\0 & - l_{1} \sin{\left (\theta_{2} \right )} - l_{2} \sin{\left (\theta_{2} + \theta_{3} \right )} - l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & - l_{2} \sin{\left (\theta_{2} + \theta_{3} \right )} - l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & - l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )}\end{matrix}\right]$$

We also need to define the Jacobian of angular velocities, $J_\omega(\theta)$.

```python
w = Matrix((0, theta2 + theta3 + theta4, theta1))
J_w = w.jacobian(wrt)
J_w
```

$$\left[\begin{matrix}0 & 0 & 0 & 0\\0 & 1 & 1 & 1\\1 & 0 & 0 & 0\end{matrix}\right]$$

We can combine the two results to get the full Jacobian matrix, $J_{ee}(\theta)$.

```python
J_ee = Matrix((J_v, J_w))
J_ee
```

$$\left[\begin{matrix}- l_{1} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} \right )} - l_{2} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} + \theta_{3} \right )} - l_{3} \cos{\left (\theta_{1} \right )} - l_{4} \sin{\left (\theta_{1} \right )} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{1} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} \right )} + l_{2} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{2} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{4} \cos{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )}\\l_{1} \sin{\left (\theta_{2} \right )} \cos{\left (\theta_{1} \right )} + l_{2} \sin{\left (\theta_{2} + \theta_{3} \right )} \cos{\left (\theta_{1} \right )} - l_{3} \sin{\left (\theta_{1} \right )} + l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} \cos{\left (\theta_{1} \right )} & l_{1} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} \right )} + l_{2} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{2} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} \right )} + l_{4} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & l_{4} \sin{\left (\theta_{1} \right )} \cos{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )}\\0 & - l_{1} \sin{\left (\theta_{2} \right )} - l_{2} \sin{\left (\theta_{2} + \theta_{3} \right )} - l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & - l_{2} \sin{\left (\theta_{2} + \theta_{3} \right )} - l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )} & - l_{4} \sin{\left (\theta_{2} + \theta_{3} + \theta_{4} \right )}\\0 & 0 & 0 & 0\\0 & 1 & 1 & 1\\1 & 0 & 0 & 0\end{matrix}\right]$$

From the Jacobian, the velocities and torque can be related. This ensures if a particular operation is possible.

$$\dot{x} = J \dot{\theta}$$

$$\tau = J^T F$$


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.

# Spreading out

Welcome to the second lesson of the course module: "_Spreading out: parabolic PDEs."_ We're studying the heat equation in one spatial dimension:

\begin{equation}
\frac{\partial T}{\partial t} = \alpha \frac{\partial^2 T}{\partial x^2}
\end{equation}

where $\alpha$ is the thermal diffusivity and $T$ is the temperature.

In the previous lesson, we reviewed the numerical solution of the 1D diffusion equation with a forward-time, centered-space scheme: an _explicit_ scheme. What does that mean?

The solution for $T$ at timestep $t^{n+1}$ was calculated using different combinations of $T$ values from the *previous* timestep $t^n$.  We have complete knowledge of the parts that feed into the solution update at each spatial point.

*Implicit* methods work differently: we will use more data from the "future" in the update, including several values of $T$ at $t^{n+1}$. This will make the scheme more difficult to apply, but there are several reasons why it may be worth the effort.

In lesson 1, we discussed two disadvantages of explicit methods: (1) boundary effects drag behind by one time step; (2) stability requirements constrain the time-step to very small values. Both of these issues are resolved by implicit schemes.

## Implicit schemes

Let's move things around a bit and try combining the Euler time step with an evaluation of the spatial derivative on the *updated* solution at $t^{n+1}$. The discretized form of the equation is now as follows:

\begin{equation}\frac{T_{i}^{n+1}-T_{i}^{n}}{\Delta t}=\alpha\frac{T_{i+1}^{n+1}-2T_{i}^{n+1}+T_{i-1}^{n+1}}{\Delta x^2}\end{equation}

The stencil for this discretization doesn't look anything like the other stencils we've used until now. Check it out.

.
#### Figure 1. Stencil of the implicit central scheme.

From the previous time-step, we only know $T_i^{n}$, but what about $T_i^{n+1}$, $T_{i-1}^{n+1}$ and $T_{i+1}^{n+1}$?  What can we do?

No need to panic! Let's start by putting what we *do know* on the right-hand side of the equation and what we *don't know* on the left. We get:

\begin{equation} -T_{i-1}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{i}^{n+1} - T_{i+1}^{n+1} = T_{i}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

It looks like there are a lot of unknowns and just one equation!

What does it look like with $i=1$?

\begin{equation}-T_{0}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{1}^{n+1} - T_{2}^{n+1} = T_{1}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

and $i=2$?

\begin{equation}-T_{1}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{2}^{n+1} - T_{3}^{n+1} = T_{2}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

What about $i=3$?

\begin{equation}-T_{2}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{3}^{n+1} - T_{4}^{n+1} = T_{3}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

Can you see the common element across equations? Here's a little help:

$T_{i}^{n+1}$ also appears in the equation for $T_{i-1}^{n+1}$ and $T_{i+1}^{n+1}$. We might have enough equations if we apply this for all $i$-values at the same time, don't you think? In fact, this is a linear system of equations for the unknown values $T_{i}^{n+1}$ on the spatial grid.

### What about boundary conditions?

Let's look at the boundary points of the example from the previous lesson with a Dirichlet BC at $x=0$ and a Neumann BC at $x=1$, discretizing with $N$ mesh points.

The value $T_0^{n+1}$ is known at every time-step from the BC, so putting all unknown terms on the left-hand side of the equation and the known values on the right side yields the following for the $i=1$ equation:

\begin{equation}-T_{2}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{1}^{n+1} = T_{1}^{n}\frac{\Delta x^2}{\alpha\Delta t} + T_{0}^{n+1} \end{equation}

That was easy!

On the other hand, for $i=N-2$, the equation reads

\begin{equation} -T_{N-3}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{N-2}^{n+1} - T_{N-1}^{n+1} = T_{N-2}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

The discretized Neumann boundary condition on the right side of the rod is

\begin{equation}\frac{T^{n}_{N-1} - T^{n}_{N-2}}{\Delta x} = q.\end{equation}

But we can just as easily write that at time step $n+1$ (the boundary conditions apply at every time-step):

\begin{equation}\frac{T^{n+1}_{N-1} - T^{n+1}_{N-2}}{\Delta x} = q.\end{equation}

Inserting the Neumann boundary condition in the equation for $i=N-2$ yields

\begin{equation} - T_{N-3}^{n+1} + \left( 1 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{N-2}^{n+1} = T_{N-2}^{n}\frac{\Delta x^2}{\alpha\Delta t} + \Delta x q \end{equation}

Make sure you work this out with pen and paper: it's important to recognize where these terms come from!

Now we can write the linear system of equations in matrix form as follows:

$$[A][x] = [b]+[b]_{b.c.}$$

where the matrix of coefficients $[A]$ is a sparse matrix—most of the matrix elements are zero—with three non-zero diagonals. We write below the system expanded out, so you can see the structure of the matrix, with $\sigma=\frac{\alpha\Delta t}{\Delta x^2}$:

\begin{align}\left[ \begin{array}{cccccc}
 \left(2 + \frac{1}{\sigma}\right) & -1 & 0 & \cdots & & 0 \\
 -1 & \left(2 + \frac{1}{\sigma}\right) & -1 & 0 & \cdots & 0 \\
 0 & & \ddots& & & \vdots \\
 \vdots & & & & \left(2 + \frac{1}{\sigma}\right)& \\
 0 & \cdots & & & -1 & \left(1 + \frac{1}{\sigma}\right) \end{array} \right]
 \cdot
 \left[ \begin{array}{c}
 T_1^{n+1} \\ T_2^{n+1} \\ \vdots \\ \\ T_{N-2}^{n+1} \end{array} \right]
 =
 \left[ \begin{array}{c}
T_1^n \frac{1}{\sigma} \\ T_2^{n}\frac{1}{\sigma} \\ \vdots \\ \\ T_{N-2}^{n}\frac{1}{\sigma} \end{array} \right]
 +
 \begin{bmatrix}
  T_0^{n+1}\\
  0\\\\
  \vdots\\\\
  0\\
  q\Delta x
 \end{bmatrix}
 \end{align}

Notice that the Dirichlet boundary condition adds only a term to the right-hand side of the system.  The Neumann boundary condition both adds a term to the right-hand side and modifies the matrix $[A]$.

### Problem set up

We'll re-use the problem from lesson 1: we have a graphite rod, with [thermal diffusivity](http://en.wikipedia.org/wiki/Thermal_diffusivity) $\alpha=1.22\times10^{-3} {\rm m}^2/{\rm s}$, length $L=1{\rm m}$, and temperature held at $T=100{\rm C}$ on the left end, $x=0$, and $0{\rm C}$ everywhere else intially. We'll compute the evolution of temperature on the length of the rod.

Let's start like we did in the previous lesson: import your libraries and set up the discretization.

```
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

```
L = 1.
nt = 100
nx = 51.
alpha = 1.22e-3

dx = L/(nx-1)

Ti = numpy.zeros(nx)
Ti[0] = 100
```

### Solving a linear system

We need to solve the linear system of equations written above to advance the solution in time. Luckily, we can rely on our friends from SciPy who have developed some nice linear solvers, so we don't need to write our own.

From `scipy.linalg`, let's import `solve`: a function to solve linear systems. Make sure to explore the documentation of [`scipy.linalg`](http://docs.scipy.org/doc/scipy-0.14.0/reference/linalg.html). We'll need to define our own custom functions to generate the coefficient matrix and the right-hand side of the linear system. You should carefully study the code below.

```
from scipy.linalg import solve
```

```
def generateMatrix(N, sigma):
    """ Computes the matrix for the diffusion equation with backward Euler
        Dirichlet condition at i=0, Neumann at i=-1

    Parameters:
    ----------
    T: array of float
        Temperature at current time step
    sigma: float
        alpha*dt/dx^2

    Returns:
    -------
    A: 2D numpy array of float
        Matrix for diffusion equation
    """

    # Setup the diagonal
    d = numpy.diag(numpy.ones(N-2)*(2+1./sigma))

    # Consider Neumann BC
    d[-1,-1] = 1+1./sigma

    # Setup upper diagonal
    ud = numpy.diag(numpy.ones(N-3)*-1, 1)

    # Setup lower diagonal
    ld = numpy.diag(numpy.ones(N-3)*-1, -1)

    A = d + ud + ld

    return A
```

```
def generateRHS(T, sigma):
    """ Computes right-hand side of linear system for diffusion equation
        with backward Euler

    Parameters:
    ----------
    T: array of float
        Temperature at current time step
    sigma: float
        alpha*dt/dx^2

    Returns:
    -------
    b: array of float
        Right-hand side of diffusion equation with backward Euler
    """
    b = numpy.zeros_like(T)

    b = T[1:-1]*1./sigma
    # Consider Dirichlet BC
    b[0] += T[0]

    return b
```

Next, we'll define a function that steps in time using the implicit central-space scheme.  Remember that for an implicit method, a step in time is performed by solving the entire linear system. This is a fundamental difference between implicit and explicit methods, and implies a considerable computational cost.

```
def implicit_ftcs(T, A, nt, sigma):
    """ Advances diffusion equation in time with implicit central scheme

    Parameters:
    ----------
    T: array of float
        initial temperature profile
    A: 2D array of float
        Matrix with discretized diffusion equation
    nt: int
        number of time steps
    sigma: float
        alpha*td/dx^2

    Returns:
    -------
    T: array of floats
        temperature profile after nt time steps
    """

    for t in range(nt):
        Tn = T.copy()
        b = generateRHS(Tn, sigma)
        # Use numpy.linalg.solve
        T_interior = solve(A,b)
        T[1:-1] = T_interior
        # Enforce Neumann BC (Dirichlet is enforced automatically)
        T[-1] = T[-2]

    return T
```

We solve the linear system for every time step, but the $A$ matrix does not change. Thus, you can generate it only once and then use it as an input parameter to the time-stepping function. Let's try this out!

```
sigma = 0.5
dt = sigma * dx*dx/alpha
nt = 1000

A = generateMatrix(nx, sigma)
```

It's always a good idea to check your matrix...

```
print A
```

    [[ 4. -1.  0. ...,  0.  0.  0.]
     [-1.  4. -1. ...,  0.  0.  0.]
     [ 0. -1.  4. ...,  0.  0.  0.]
     ...,
     [ 0.  0.  0. ...,  4. -1.  0.]
     [ 0.  0.  0. ..., -1.  4. -1.]
     [ 0.  0.  0. ...,  0. -1.  3.]]

Looks good!

Now, we apply the time iteration with our very own function `implicit_ftcs()`.

```
T = Ti.copy()
T = implicit_ftcs(T, A, nt, sigma)
```

Now plot the solution!

```
pyplot.plot(numpy.linspace(0,1,nx), T, color='#003366', ls='-', lw=3);
```

Not too impressive, this looks just like the result from *explicit* forward in time, centered in space for $\alpha\frac{\Delta t}{\Delta x^2} = \frac{1}{2}$.

But try $\alpha\frac{\Delta t}{\Delta x^2} = 5$, which violates the stability condition of the *explicit* scheme:

```
sigma = 5.0

T = Ti.copy()
A = generateMatrix(nx, sigma)

T = implicit_ftcs(T, A, nt, sigma)

pyplot.plot(numpy.linspace(0,1,nx), T, color='#003366', ls='-', lw=3);
```

**It didn't blow up!**

We were not able to use such a large time step with the explicit scheme. You can try out other values of `sigma` and you'll get a stable solution. In fact, this is an *unconditionally stable* scheme—the most valuable feature of implicit methods is that they give stable solutions without a constraint on the choice of time step.

Using the implicit scheme, we can always advance in time using larger time steps. But each time step requires the solution of a linear system, which is computationally expensive. This is the trade-off between explicit and implicit methods.

##### A word of warning

Implicit methods allow you to use significantly larger time steps, because they are not subject to stability constraints. But that doesn't mean you can use just _any_ large time step! Remember that Euler's method is a first-order method, so the _accuracy_ gets worse as you increase the time step, in direct proportion. In fact, you can lose the ability to capture the correct physics if your time step is too large. Numerical stability does not imply accuracy!

##### Dig deeper

You see how matrix `A` is mostly full of zeros?  We call such a matrix *sparse*, and there are many ways to make more efficient calculations taking advantage of their particular structure. First of all, you can optimize the memory usage. Check out SciPy's [sparse-matrix storage formats](http://docs.scipy.org/doc/scipy-0.14.0/reference/sparse.html): you don't need too store $(N-2)^2$ elements! For example, a `coo_matrix` format stores only $3*N_\text{nonzero}$, where $N_\text{nonzero}$ is the number of non-zero elements in `A`. Make sure to explore this topic a little more. It's an important topic in numerical PDEs.

---
###### The cell below loads the style of the notebook

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>


```python
import matplotlib.pyplot as plt
import numpy as np
import scipy.special
from scipy.optimize import minimize
from scipy.stats import geom
from typing import Tuple
```

```python
cycles = np.array([i for i in range(1, 14)])
smokers = np.array([29, 16, 17, 4, 3, 9, 4, 5, 1, 1, 1, 3, 7])
non_smokers = np.array([198, 107, 55, 38, 18, 22, 7, 9, 5, 3, 6, 6, 12])
```

(a) Fitting a geometric model to the group of smokers and the group of non-smokers we can compare the estimated probability of pregnancy per cycle.

```python
def plot_pregnancy_per_cycle(
    cyles: np.ndarray,
    pregnancy_counts: np.ndarray,
    title_label: str) -> None:
    p = pregnancy_counts[0] / np.sum(pregnancy_counts)
    prob = geom.pmf(cycles, p)
    plt.bar(cycles, prob)
    plt.xlabel('Number of cycles')
    plt.ylabel('Probability p')
    plt.title(f'Probability of pregnancy per cycle for {title_label}');
```

```python
plot_pregnancy_per_cycle(cycles, smokers, 'smokers')
```

```python
plot_pregnancy_per_cycle(cycles, non_smokers, 'non-smokers')
```

From the plots it seems that the probability of getting pregnant in less cycles is higher for non-smokers.

(b) To check the adequacy of the geometric model we follow the Example (4.6) by plotting a Geometric plot. The idea is to plot the number of cycles k against a function of the counts $n_k$ that is expected to be a straight line for Geometric data. Since $p_k = P(X = k) = (1 - p)^{k - 1}p$, then

$$
\log p_k = k \log(1 - p) + \log \Big( \frac{p}{1 - p} \Big)
$$

indicating that we should plot $\log p_k$ versus $k$, where $p_k$ is estimated by $n_k \big/ \sum n_k$.

```python
def geometric_plot(cycles: np.ndarray, counts: np.ndarray) -> None:
    p_k = counts / np.sum(counts)
    plt.plot(cycles, np.log(p_k), 'o')
    plt.xlabel('Number of cycles k')
    plt.ylabel(r'Log($p_k$)');
    plt.title('(a) Geometric plot')
```

```python
geometric_plot(cycles, smokers)
```

```python
geometric_plot(cycles, non_smokers)
```

It is clear that for both smokers and non-smokers the data does not follow a Geometric prescription.

(c) We are told that conditional on $p$ to let the pregancy counts $X_p \sim geom(p)$, where the probability of pregnancy per cycle $p$ itself is random with beta distribution with density:

$$
f(p) = \frac{1}{B(\alpha, \beta)}p^{\alpha - 1} (1 - p)^{\beta - 1}
$$

The beta-geometric probabilities are given by

\begin{align}
P(X = k) & = E \big[ P (X_p = k | p) \big]  \\
& = E \big[ (1 - p)^{k-1}p \big] \\
& = \int_{0}^{1} (1 - p)^{k-1} p f(p) dp \\
& = \frac{1}{B(\alpha, \beta)} \int_{0}^{1} p^{\alpha - 1} (1 - p)^{\beta - 1} (1 - p)^{k - 1} p dp \\
& = \frac{1}{B(\alpha, \beta)} \int_{0}^{1} p^{(\alpha + 1) - 1} (1 - p)^{(\beta + k - 1) - 1} dp \\
& = \frac{B(\alpha + 1, \beta + k - 1)}{B(\alpha, \beta)}
\end{align}

for $k = 0, 1, ...$ where the definition of the [beta function](https://en.wikipedia.org/wiki/Beta_function) is used to evaluate the integral.

(d) To fit the beta-geometric model to the data, the log-likelihood of $(\alpha, \beta)$ is:

$$
\log L(\alpha, \beta) = \sum_{k=1}^{12}n_k \log P(X=k) + n_{[>12]} \log\{1 - P(X > 12)\}
$$

where $P(X = k)$ is given in (b) above. Numerical optimization of the log-likelihood yields.

```python
def neg_log_likelihood(x: np.ndarray, cycles: np.ndarray, counts: np.ndarray) -> float:
    alpha = x[0]
    beta = x[1]
    n_k = counts
    p_k = beta_geometric_probability(alpha, beta, cycles)
    neg_log_like = -np.sum(n_k[:-1] * p_k[:-1]) - n_k[-1] * np.log(1 - p_k[-1])
    return neg_log_like

def beta_geometric_probability(alpha: float, beta: float, k: np.ndarray) -> np.ndarray:
    prob = scipy.special.beta(alpha + 1, beta + k - 1) / scipy.special.beta(alpha, beta)
    return prob
```

```python
def fit_beta_geometric_model(
    cycles: np.ndarray, counts: np.ndarray) -> Tuple[float, float]:
    # Yikes the results are sensitive to these initial conditions! The max value is
    # consistent but occurs at different alpha and beta values.
    x0 = np.array([1.0, 1.0])
    bounds = ((1e-10, None), (1e-10, None))
    result = minimize(neg_log_likelihood, x0, args=(cycles, counts), bounds=bounds)
    #print(result)
    return result.x[0], result.x[1]

def print_mle(alpha_mle: float, beta_mle: float, title: str) -> None:
    (print(title + " alpha-hat = ", np.round(alpha_mle, 2), ", ",
           "beta-hat = ", np.round(beta_mle, 7)))
```

```python
alpha_hat, beta_hat = fit_beta_geometric_model(cycles, smokers)
print_mle(alpha_hat, beta_hat, 'Smokers')
```

```python
def expected_frequencies(mle_probs: np.ndarray, total_count: float) -> np.ndarray:
    freqs = mle_probs * total_count
    return freqs

def chi_squared(obs_freqs: np.ndarray, exp_freqs: np.ndarray) -> float:
    chi_sq = np.sum((obs_freqs - exp_freqs)**2 / exp_freqs)
    return chi_sq

def print_chi_squared(chi_sq: float) -> None:
    print("chi-squared = ", np.round(chi_sq, 1))
```

```python
N = np.sum(smokers)
mle_probs = beta_geometric_probability(alpha_hat, beta_hat, cycles)
print(mle_probs, np.sum(mle_probs))
exp_freqs = expected_frequencies(mle_probs, N)
chi_sq = chi_squared(smokers, exp_freqs)
print_chi_squared(chi_sq)
```

The MLE probabilities are clearly wrong as all the probability mass is put on the first cyle. The same happens for the non-smokers below. The problem seems to stem from the MLE values but I am not sure where the bug is in the code (my code or scipy) or possibly in my math. Anyway I think the general workflow is correct in this notebook.

```python
alpha_hat, beta_hat = fit_beta_geometric_model(cycles, non_smokers)
print_mle(alpha_hat, beta_hat, 'Non-smokers')
```

```python
N = np.sum(non_smokers)
mle_probs = beta_geometric_probability(alpha_hat, beta_hat, cycles)
print(mle_probs, np.sum(mle_probs))
exp_freqs = expected_frequencies(mle_probs, N)
chi_sq = chi_squared(non_smokers, exp_freqs)
print_chi_squared(chi_sq)
```


# One and two electron Hamiltonian

This model is valid for a double-dot system tuned to the transition from (1,0) to (0,1) or with two electrons for (1,1) to (2,0).

Author: Pieter Eendebak (pieter.eendebak@gmail.com), Bruno Buijtendorp (brunobuijtendorp@gmail.com)

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp
%matplotlib inline
sp.init_printing(use_latex='latex')
```

## One electron Hamiltonian

Define 1-electron double dot Hamiltonian
e is detuning, $t$ is tunnel coupling. The basis we work in is (1,0) and (0,1).

```python
e, t = sp.symbols('e t')
H = sp.Matrix([[e/2, t],[t, -e/2]])
sp.pprint(H)

#%% Get normalized eigenvectors and eigenvalues
eigvec_min = H.eigenvects()[0][2][0].normalized()
eigval_min = H.eigenvects()[0][0]

eigvec_plus = H.eigenvects()[1][2][0].normalized()
eigval_plus = H.eigenvects()[1][0]

#%% Lambdify eigenvalues to make them numerical functions of e and t (nicer plotting)
eigval_min_func = sp.lambdify((e,t), eigval_min , 'numpy')
eigval_plus_func = sp.lambdify((e,t), eigval_plus, 'numpy')

#%% Plot energy levels
t_value = 1
plot_x_limit = 5
Npoints_x = 1000
```

    ⎡e     ⎤
    ⎢─   t ⎥
    ⎢2     ⎥
    ⎢      ⎥
    ⎢   -e ⎥
    ⎢t  ───⎥
    ⎣    2 ⎦

```python
erange = np.linspace(-plot_x_limit, plot_x_limit, Npoints_x)
levelfig, levelax = plt.subplots()
levelax.plot(erange, eigval_min_func(erange , t_value), label='$S-$')
levelax.plot(erange, eigval_plus_func(erange, t_value), label ='$S+$')
levelax.set_title('Energy levels for double-dot in one-electron regime, t = %.1f' % t_value)
plt.plot(erange, erange/2, ':c', label='avoided crossing')
plt.plot(erange, -erange/2, ':c')
plt.legend()
levelax.set_xlabel('detuning $(uev)$')
levelax.set_ylabel('energy $(ueV)$')
_=plt.axis('tight')
```

```python
#%% Plot energy level differences
SminS = eigval_plus_func(erange , t_value) - eigval_min_func(erange, t_value)

plt.figure()
plt.plot(erange, SminS, label='$E_{S_+} - E_{S_-}$')
plt.title('Energy transitions for double-dot in one-electron regime, t = %.1f $\mu eV$' % (t_value))
plt.legend()
plt.ylabel('$\Delta E$ $ (\mu eV)$')
plt.xlabel('$\epsilon$ $ (\mu eV)$')

#%% Get S(1,0) component of eigenvectors
eigcomp_min = eigvec_min[0]
eigcomp_plus = eigvec_plus[0]

#%% Plot S(1,0) components squared (probabilities) of eigenvectors as function of detuning
t_value = 1
erange = np.linspace(-20,20,500)
plot_x_limit = 20

# Lambdify eigenvector components to make them functions of e and t
eigcompmin_func = sp.lambdify((e,t), eigcomp_min , 'numpy')
eigcompplus_func = sp.lambdify((e,t), eigcomp_plus, 'numpy')

fig2, ax2 = plt.subplots()
ax2.plot(erange,eigcompmin_func(erange, t_value)**2, label='$S_-$')
ax2.plot(erange,eigcompplus_func(erange, t_value)**2, label='$S_+$')
ax2.set_xlabel('detuning,  ($\mu$eV)')
ax2.set_ylabel('(1,0) coefficient squared')
_=plt.legend()
```

## Two-electron Hamiltonian

Define 2-electron double dot Hamiltonian
e is detuning, t is tunnel coupling. The basis we work in is: {S(2,0), S(1,1), T(1,1)}

```python
e, t = sp.symbols('e t')
# Basis: {S(2,0), S(1,1), T(1,1)}
H = sp.Matrix([[e, sp.sqrt(2)*t, 0],[sp.sqrt(2)*t, 0, 0],[0, 0, 0]])

#%% Get normalized eigenvectors and eigenvalues
eigvec_min = H.eigenvects()[1][2][0].normalized()
eigval_min = H.eigenvects()[1][0]

eigvec_plus = H.eigenvects()[2][2][0].normalized()
eigval_plus = H.eigenvects()[2][0]

eigvec_T = H.eigenvects()[0][2][0].normalized()
eigval_T = H.eigenvects()[0][0]

#%% Lambdify eigenvalues to make them numerical functions of e and t (nicer plotting)
eigval_min_func = sp.lambdify((e,t), eigval_min , 'numpy')
eigval_plus_func = sp.lambdify((e,t), eigval_plus, 'numpy')

#%% Plot energy levels
t_value = 1
plot_x_limit = 5
Npoints_x = 1000

erange = np.linspace(-plot_x_limit, plot_x_limit, Npoints_x)
levelfig, levelax = plt.subplots()
levelax.plot(erange, [eigval_T]*len(erange), label='T(1,1)')
levelax.plot(erange, eigval_min_func(erange , t_value), label='$S_-$')
levelax.plot(erange, eigval_plus_func(erange, t_value), label ='$S_+$')
levelax.set_title('Energy levels for double-dot in two-electron regime, t = %.1f' % t_value)
plt.legend()
levelax.set_xlabel('detuning $(uev)$')
levelax.set_ylabel('energy $(ueV)$')
plt.axis('tight')

#%% Plot energy level differences
SminS = eigval_plus_func(erange , t_value) - eigval_min_func(erange, t_value)
S20minT = eigval_plus_func(erange, t_value)
TminS11 = -eigval_min_func(erange, t_value)

plt.figure()
plt.plot(erange, SminS, label='$E_{S_+} - E_{S_-}$')
plt.plot(erange, S20minT, label = '$E_{S_+} - E_T$')
plt.plot(erange, TminS11, label = '$E_T - E_{S_-}$')
plt.title('Energy transitions for double-dot in two-electron regime, t = %.1f $\mu eV$' % (t_value))
plt.legend()
plt.ylabel('$\Delta E$ $ (\mu eV)$')
plt.xlabel('$\epsilon$ $ (\mu eV)$')

#%% Get S(2,0) component of eigenvectors
eigcomp_min = eigvec_min[0]
eigcomp_plus = eigvec_plus[0]
eigcomp_T = eigvec_T[0]

#%% Plot S(2,0) components squared (probabilities) of eigenvectors as function of detuning
t_value = 1
erange = np.linspace(-20,20,500)
plot_x_limit = 20

# Lambdify eigenvector components to make them functions of e and t
eigcompmin_func = sp.lambdify((e,t), eigcomp_min , 'numpy')
eigcompplus_func = sp.lambdify((e,t), eigcomp_plus, 'numpy')

fig2, ax2 = plt.subplots()
ax2.plot(erange,eigcompmin_func(erange, t_value)**2, label='$S_-$')
ax2.plot(erange,eigcompplus_func(erange, t_value)**2, label='$S_+$')
ax2.plot(erange,[eigcomp_T]*len(erange), label='$T$')
ax2.set_xlabel('Detuning ($\mu$eV)')
ax2.set_ylabel('S(2,0) coefficient squared')
_=plt.legend()
```

```python

```


```python
%matplotlib notebook

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy.stats
```

```python
data = pd.read_csv('data.csv')

print("data has {} measurements for {} variables".format(*data.shape))
print("\n{}\n...".format(data.head(8)))
```

    data has 5000 measurements for 2 variables

       country  language
    0        2         5
    1        4         2
    2        2         3
    3        1         1
    4        5         5
    5        4         5
    6        5         4
    7        4         5
    ...

```python
countries = ['Canada', 'USA', 'England', 'Italy', 'Switzerland']
languages = ['English', 'French', 'Spanish', 'German', 'Italian']

F = pd.crosstab(data.country, data.language, margins=True)
F.index = [*countries, 'col totals']
F.columns = [*languages, 'row totals']

print("{}".format(F))
```

                 English  French  Spanish  German  Italian  row totals
    Canada           688     280       10      11       11        1000
    USA              730      31      190       8       41        1000
    England          798      74       38      31       59        1000
    Italy             17      13       11      15      944        1000
    Switzerland       15     222       20     648       95        1000
    col totals      2248     620      269     713     1150        5000

```python
F = pd.crosstab(data.country, data.language, margins=False)
F.index = countries
F.columns = languages

chisq_stat, p_value, dof, E = scipy.stats.chi2_contingency(F)

print('Results of Chi-squared test of independence\n')
print('  Chi-squared test statistic: {:02.2F}'.format(chisq_stat))
print('  degrees of freedom: {}'.format(dof))
print('  p-value: {:02.6F}'.format(p_value))
```

    Results of Chi-squared test of independence

      Chi-squared test statistic: 7338.94
      degrees of freedom: 16
      p-value: 0.000000

```python
print('matrix of observations F:\n\n{}'.format(F))
```

    matrix of observations F:

                 English  French  Spanish  German  Italian
    Canada           688     280       10      11       11
    USA              730      31      190       8       41
    England          798      74       38      31       59
    Italy             17      13       11      15      944
    Switzerland       15     222       20     648       95

```python
P = F / F.sum().sum()

print('correspondence matrix P:\n\n{}'.format(P))
```

    correspondence matrix P:

                 English  French  Spanish  German  Italian
    Canada        0.1376  0.0560   0.0020  0.0022   0.0022
    USA           0.1460  0.0062   0.0380  0.0016   0.0082
    England       0.1596  0.0148   0.0076  0.0062   0.0118
    Italy         0.0034  0.0026   0.0022  0.0030   0.1888
    Switzerland   0.0030  0.0444   0.0040  0.1296   0.0190

```python
row_centroid = P.sum(axis=1)

print('row centroid (marginal frequency distribution over countries):\n\n{}'.format(row_centroid))
```

    row centroid (marginal frequency distribution over countries):

    Canada         0.2
    USA            0.2
    England        0.2
    Italy          0.2
    Switzerland    0.2
    dtype: float64

```python
col_centroid = P.sum(axis=0)

print('column centroid (marginal frequency distribution over languages):\n\n{}'.format(col_centroid))
```

    column centroid (marginal frequency distribution over languages):

    English    0.4496
    French     0.1240
    Spanish    0.0538
    German     0.1426
    Italian    0.2300
    dtype: float64

```python
row_totals = F.sum(axis=1)

print("row totals (marginal frequency distribution over the countries):\n\n{}".format(row_totals))
```

    row totals (marginal frequency distribution over the countries):

    Canada         1000
    USA            1000
    England        1000
    Italy          1000
    Switzerland    1000
    dtype: int64

```python
col_totals = F.sum(axis=0)

print("column totals (marginal frequency distribution over the languages):\n\n{}".format(col_totals))
```

    column totals (marginal frequency distribution over the languages):

    English    2248
    French      620
    Spanish     269
    German      713
    Italian    1150
    dtype: int64

```python
data = []
for _,row in P.iterrows():
    acc = []
    cntry_i = row.name
    p_iplus = row_centroid.ix[cntry_i]
    for  cntry_k in P.index:
        p_kplus = row_centroid.ix[cntry_k]
        chisqd = np.sqrt(np.sum(np.square(row/p_iplus - P.ix[cntry_k]/p_kplus) / col_centroid))
        acc.append(chisqd)
    data.append(acc)

row2row_chisqd = pd.DataFrame(data, index=P.index, columns=P.index)

print("row-to-row Chi-squared distance table:\n\n{}".format(row2row_chisqd))
```

    row-to-row Chi-squared distance table:

                   Canada       USA   England     Italy  Switzerland
    Canada       0.000000  1.053631  0.629709  2.315427     1.978023
    USA          1.053631  0.000000  0.678054  2.296625     2.203064
    England      0.629709  0.678054  0.000000  2.192568     2.054644
    Italy        2.315427  2.296625  2.192568  0.000000     2.509498
    Switzerland  1.978023  2.203064  2.054644  2.509498     0.000000

```python
PT = P.T

data = []
for _,row in PT.iterrows():
    acc = []
    lang_j = row.name
    p_plusj = col_centroid.ix[lang_j]
    for lang_k in PT.index:
        p_plusk = col_centroid.ix[lang_k]
        chisqd = np.sqrt(np.sum(np.square(row/p_plusj - PT.ix[lang_k]/p_plusk) / row_centroid))
        acc.append(chisqd)
    data.append(acc)

col2col_chisqd = pd.DataFrame(data, index=PT.index, columns=PT.index)

print("column-to-column Chi-squared distance table:\n\n{}".format(col2col_chisqd))
```

    column-to-column Chi-squared distance table:

              English    French   Spanish    German   Italian
    English  0.000000  1.174400  1.160320  2.338660  2.157459
    French   1.174400  0.000000  1.849170  1.582510  2.140049
    Spanish  1.160320  1.849170  0.000000  2.439221  2.309852
    German   2.338660  1.582510  2.439221  0.000000  2.572037
    Italian  2.157459  2.140049  2.309852  2.572037  0.000000

```python
Mu_ij = row_centroid.values.reshape((P.index.size,1)) * col_centroid.values.reshape((1,P.columns.size))

Lambda = (P - Mu_ij) / np.sqrt(Mu_ij)

print('inertia Lambda:\n\n{}'.format(Lambda))
```

    inertia Lambda:

                  English   French   Spanish    German   Italian
    Canada       0.159004  0.19812 -0.084450 -0.155852 -0.204219
    USA          0.187016 -0.11811  0.262604 -0.159404 -0.176243
    England      0.232370 -0.06350 -0.030464 -0.132166 -0.159458
    Italy       -0.288528 -0.14097 -0.082522 -0.151114  0.665808
    Switzerland -0.289862  0.12446 -0.065169  0.598536 -0.125888

```python
U,S,V = np.linalg.svd(Lambda)

num_sv = np.arange(1,len(S)+1)

cum_var_explained = [np.sum(np.square(S[0:n])) / np.sum(np.square(S)) for n in num_sv]

print('Using first singular value, {:0.3F}% variance explained'.format(cum_var_explained[0]))
print('Using first 2 singular values, {:0.3F}% variance explained'.format(cum_var_explained[1]))
print('Using first 3 singular values, {:0.3F}% variance explained'.format(cum_var_explained[2]))
```

    Using first singular value, 0.506% variance explained
    Using first 2 singular values, 0.916% variance explained
    Using first 3 singular values, 0.987% variance explained

```python
fig = plt.figure()
ax = fig.add_subplot(111)

plt.plot(num_sv, cum_var_explained, color='#2171b5', label='variance explained')
plt.scatter(num_sv, S, marker='s', color='#fc4e2a', label='singular values')

plt.legend(loc='lower left', scatterpoints=1)

ax.set_xticks(num_sv)
ax.set_xlim([0.8, 5.1])
ax.set_ylim([0.0, 1.1])
ax.set_xlabel('Number of singular values used')
ax.set_title('Singular values & cumulative variance explained',
             fontsize=16,
             y=1.03)

plt.grid()
```

    <IPython.core.display.Javascript object>

```python
cntry_x = U[:,0]
cntry_y = U[:,1]
cntry_z = U[:,2]

lang_x = V.T[:,0]
lang_y = V.T[:,1]
lang_z = V.T[:,2]
```

```python
import pylab
from mpl_toolkits.mplot3d import Axes3D, proj3d

fig = pylab.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(cntry_x, cntry_y, cntry_z, marker='s', s=50, c='#2171b5')
cntry_labels = []
for i,(x,y,z) in enumerate(zip(cntry_x,cntry_y,cntry_z)):
    x2, y2, _ = proj3d.proj_transform(x,y,z, ax.get_proj())

    label = pylab.annotate(Lambda.index[i],
                          xy=(x2,y2),
                          xytext=(-2,2),
                          textcoords='offset points',
                          ha='right',
                          va='bottom',
                          color='#2171b5')
    cntry_labels.append(label)

ax.scatter(lang_x, lang_y, lang_z, marker='o', s=50, c='#fc4e2a')
lang_labels = []
for i,(x,y,z) in enumerate(zip(lang_x,lang_y,lang_z)):
    x2, y2, _ = proj3d.proj_transform(x,y,z, ax.get_proj())

    label = pylab.annotate(Lambda.columns[i],
                          xy=(x2,y2),
                          xytext=(-2,2),
                          textcoords='offset points',
                          ha='right',
                          va='bottom',
                          color='#fc4e2a')
    lang_labels.append(label)

def update_position(e):
    for i,(x,y,z) in enumerate(zip(cntry_x,cntry_y,cntry_z)):
        x2, y2, _ = proj3d.proj_transform(x,y,z, ax.get_proj())
        cntry_labels[i].xy = x2, y2
    for i,(x,y,z) in enumerate(zip(lang_x,lang_y,lang_z)):
        x2, y2, _ = proj3d.proj_transform(x,y,z, ax.get_proj())
        lang_labels[i].xy = x2, y2
    fig.canvas.draw()
fig.canvas.mpl_connect('button_release_event', update_position)

ax.set_xlabel(r'$1^{st}$ singular value')
ax.set_xticks([-0.5, 0.0, 0.5])
ax.set_ylabel(r'$2^{nd}$ singular value')
ax.set_yticks([-0.5, 0.0, 0.4])
ax.set_zlabel(r'$3^{rd}$ singular value')
ax.set_zticks([-0.5, 0.0, 0.5])
ax.set_title('Correspondence Analysis with 3 Singular Values (3D)',
            fontsize=16,
            y=1.1)

pylab.show()
```

    <IPython.core.display.Javascript object>

----

http://www.mathematica-journal.com/2010/09/an-introduction-to-correspondence-analysis/

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3718710/

Next we derive the _correspondence matrix_ $P$ from $F$ with

\begin{align}
  P &= \left[ p_{ij} \right] \\
    &= \left[ \frac{f_{ij}}{n} \right] & \text{where } n = \sum_{i=1}^{I} \sum_{j=1}^{J} f_{ij}
\end{align}

----

The $\chi^2$ distances between rows gives us a clue as to how the countries relate to one another in terms of the primary spoken languages.

The $\chi^2$ distance between rows $i$ and $k$ is given by

\begin{align}
 d_{ik} &= \sqrt{\sum_{j=1}^{J} \frac{(p_{ij}/p_{i+} - p_{kj}/p_{k+})^2}{p_{+j}} }
\end{align}

We can see in this row-to-row $\chi^2$ distance table that for the Anglophonic countries, Canada, USA and England should be clustered near one another, while Italy and Switzerland are both separated from the other countries.

Conversely, the $\chi^2$ distances between columns gives us a clue as to how the languages relate to one another in terms of the countries.

The $\chi^2$ distance between columns $j$ and $k$ is given by

\begin{align}
 d_{jk} &= \sqrt{\sum_{i=1}^{I} \frac{(p_{ij}/p_{+j} - p_{kj}/p_{+k})^2}{p_{i+}} }
\end{align}

For the languages, we can see from the column-to-column $\chi^2$ distances that English and Spanish should be closely related, with French somewhere between English and German. Italian, however, should be sitting alone all by itself away from the others.

----

We start with a matrix of _standardized residuals_:

\begin{align}
  \Omega &= \left[ \frac{p_{ij} - \mu_{ij}}{\sqrt{\mu_{ij}}}  \right]
\end{align}

```python

```


# Outline:
1. [Vector Spaces](#Vector-Spaces)
2. [Linear Dependance & Independence](#Linear-dependence-&-independence)
3. [Change of Basis](#Change-of-Basis)
4. [Linear Transformation between Vector Spaces](#Linear-Transformation-between-Vector-Spaces)
5. [Nullspace and Range of a Linear Transformation](#Nullspace-and-Range-of-a-Linear-Transformation)
6. [Linear Transformation and Matrices: Relationships](#Linear-Transformation-and-Matrices:-Relationships)
7. [Application to Linear Systems: Geometric Interpretation](#Application-to-Linear-Systems:-Geometric-Interpretation)
8. [Representation of an Operator in different basis](#Representation-of-an-Operator-in-different-basis)
9. [Eigenvalue and Eigenvectors](#Eigenvalue-and-Eigenvectors)
10. [Diagonal form of an operator](#Diagonal-form-of-an-operator)
11. [Jordan form of a linear operator](#Jordan-form-of-a-linear-operator)

```python
from IPython.display import display
from sympy import init_printing, pprint

init_printing(use_latex='mathjax')

```

# Vector Spaces

**Definition:** A non-empty space $E$ is called a $\underline{\text{Vector Space (vs)}}$ on a field $ \mathbb{K}$ if between its elements ($\underline{\text{Vectors}}$), the following $\underline{\text{Compositional Laws}}$ are true:

## 1. (Inner) (+) → sum (+): $ E \times E \rightarrow E$

$ (\underline{u},\underline{v}) \rightarrow \underline{w} = \underline{u} + \underline{v}$

with the following properties:

1. Commutative:

    $ \underline{u} + \underline{v} = \underline{v} + \underline{u}, \quad \forall \underline{u}, \underline{v} \in E$

2. Associative:

    $ \underline{u} + \underline{v} + \underline{w} = \underline{u} + (\underline{v} + \underline{w}), \quad \forall \underline{u},\underline{v},\underline{w} \in E$

3. Neutral Element $\underline{0}$ s.t.

    $\underline{u} + \underline{0} = \underline{0} + \underline{u} = \underline{u}, \quad \forall \underline{u} \in E$

4. Inverse Element $(-\underline{u})$ s.t.

    $\underline{u} + -\underline{u} = -\underline{u} + \underline{u} = \underline{0}, \quad \forall \underline{u} \in E$

## 2. (External on the field $\mathbb{K}$): $\mathbb{K} \times E \rightarrow E$

$(\lambda, \underline{u}) \rightarrow \underline{w} = \lambda\underline{u}$

with the following properties:

5. Distributive w.r.t. the sum:

    $\lambda(\underline{u} + \underline{v}) = \lambda\underline{u} + \lambda\underline{v}, \quad \forall \lambda \in \mathbb{K}, \quad \forall \underline{u},\underline{v} \in E$

6. Distributive w.r.t. the sum in $\mathbb{K}$:

    $(\lambda + \mu)\underline{v} = \lambda\underline{v} + \mu\underline{v}$

7. Associative

    $\lambda (\mu \underline{v}) = (\lambda \mu) \underline{v}, \quad \forall \lambda,\mu \in \mathbb{K}, \quad \forall \underline{v} \in E$

8. Neutral element w.r.t. field $\mathbb{K}$

    $1 \cdot \underline{u} = \underline{u} \cdot 1 = \underline{u}, \quad \forall \underline{u} \in E$

**Note:** $E$ has an algebraic structure of "Albelian Group"

**Note:** $\mathbb{K}$ can be real $\mathbb{R}$ or complex $\mathbb{C}$

## Example A

---
$\mathbb{R}^n =
\begin{equation}
\underbrace{
\mathbb{R} \times \mathbb{R} \times \dots \times \mathbb{R}
}_{n-times}
\end{equation}
$
is the set of all $n$-ple of real numbers.  $(x_1, \dots, x_n) \in \mathbb{R}^n \Rightarrow $ it is a Vector Space $\rightarrow$ indeed:

$(x_1, \dots, x_n) + (y_1, \dots, y_n) = (x_1 + y_1, \dots, x_n + y_n)$ with $(0, \dots, 0)$ as netral element and $(-x_1, \dots, -x_m)$ as inverse element.

---

## Example B

---

Consider the set $C(a, b)$ of all functions that are continuous & real in $(a, b) \in \mathbb{R}$.  Such set is a _vector space_

$C(a, b) \times C(a, b) \rightarrow C(a, b)$

$\big(f(x), g(x)\big) \rightarrow h(x) = f(x) + g(x)$

with $0(x) = 0$ in $(a, b)$ is the neutral function and $-f(x)$ s.t. $f(x) + (-f(x)) = 0(x)$ is the inverse fun.

---

# Linear dependence & independence

**Definition:** Let $\lbrace \underline{v}_1, \dots, \underline{v}_r \rbrace$ be a set of r-vectors of $\mathbb{E}$.  The vector $\lambda_1\underline{v}_1 + \dots + \lambda_r\underline{v}_r$ is called the $\underline{\text{Linear Combination}}$ of the vectors on $\mathbb{K}$.

**Definition:** $r$ vectors of $\mathbb{E} \lbrace \underline{v}_1, \dots, \underline{v}_r \rbrace$ are said to be $\underline{\text{Linearly Dependent (LD)}}$ if $\exists \, \lambda_1, \dots, \lambda_r$ in $\mathbb{K}$ not **all** zero s.t.: $\lambda_1 \underline{v}_1 + \dots + \lambda_r \underline{v}_r = \underline{0}$

**Definition:** If $\lambda_1 = \dotsb \lambda_r = 0 \Rightarrow \lbrace \underline{v}_1, \dots, \underline{v}_2 \rbrace$ are $\underline{\text{Linearly Independent (LI)}}$.

**Definition:** We call the $\underline{\text{Basis}}$ of a vector space $\mathbb{E}$ to be any LI systems of vectors capable of generating the all space by linear combination:  $\mathbb{B} = \lbrace \underline{e}_1, \dots, \underline{e}_n \rbrace \Rightarrow \underline{v} = v_1\underline{e}_1 + \dots + v_n\underline{e}_n, \quad \forall \underline{v} \in \mathbb{E}$

$v_1, \dots v_n$ are the components of $\underline{v}$ w.r.t. the basis $\mathbb{B}$.  $n$ is the $\underline{\text{Dimension}}$ of the vector space $\mathbb{E}$.

***Theorem:*** Let $\mathbb{E}_n$ be a $vs^1$ of dimension $r$.  The vectors $\lbrace \underline{v}_1, ..., \underline{v}_r \rbrace$ are LI iff the rank of the following matrix is $r$:

$
\begin{vmatrix}
v_1^1 & \dots & v_r^1 \\
\vdots & \ddots & \dots \\
v_1^n & \dots & v_r^n \\
\end{vmatrix}\rightarrow$ Matrix formed with the components of the vector $\underline{v}_i\,, i = 1, \dots, r$ with respect to any basis $\mathbb{B} = \lbrace \underline{e}_1, \dots, \underline{e}_n \rbrace$

$
\underline{v}_i = \displaystyle\sum_{j=1}^n v_i^j \thinspace \underline{e}_j, \quad i = 1, \dots, r
$

# Change of Basis

Consider $\mathbb{E}_n$ to be a vector space and let $\lbrace \underline{e}_1, \dotsb, \underline{e_n}\rbrace = B \notin \lbrace \underline{e}_{1'}, \dotsb, \underline{e}_{n'} \rbrace = {B'}$ be two basis.  By definition, every vector of ${B'}$ can be expressed as a linear combination of $B$, i.e.

$\displaystyle\underline{e}_{i'} = \sum_{r=1}^n A_i^r, \underline{e}_r, \quad i' = 1, \dots, n$

Inversely we have:

$\displaystyle\underline{e}_s = \sum_{s=1}^n A_s^{i'} \underline{e}_{i'}, \quad s = 1, \dots, n$ (From ${B'} \rightarrow B$)

Substituting we have:

$\displaystyle\underline{e}_s = \sum_{s=1}^n \sum_{r=1}^n A_{i'}^r A_s^{i'} \underline{e}_r \Rightarrow$ but since $B$ & ${B'}$ are LI we have:

$\displaystyle \sum_{s=1}^n \sum_{r=1}^r A_{i'}^r A_s^{i'} = \delta_s^r =
\begin{cases}
    1 & \quad \text{if } r = s\\
    0 & \quad \text{if } r \neq s
\end{cases}
$

The matrices $\lbrace A_s^{i'} \rbrace$ and $\lbrace A_{i'}^r \rbrace$ are inverse to each other $\rightarrow$ we derive the following laws for change of basis:

$\lbrace B' = B\underline{\underline{A}} \Leftrightarrow B = B'\underline{\underline{A}}^{-1} \rbrace \text{& } \underline{\underline{A}}\underline{\underline{A}}^{-1} = \underline{\underline{I}}$

It is easy to show that the components fo a vector $v=E_n$ change in a <u>Controvariant</u> fashion with changing the basis, i.e.:

$\underline{u} = v' \underline{e}_1 + \dots + v^n \underline{e}_n = \displaystyle\sum_{i=1}^n v^i \underline{e}_i = \sum_{i'=1}^n v^{i'} \underline{e}_{i'}, \quad \text{and } \quad \lbrace V' = A^{-1}V \Leftrightarrow V = AV' \rbrace \Rightarrow$ Controvariant!

# Linear Transformation between Vector Spaces

**Definition:** Let $E$ and $F$ be two Vector Spaces on $\mathbb{K}$.  A $\underline{\text{Linear Transformation}}$ of $E$ in $F: L: E \rightarrow F$ is a map such that $\forall \underline{x}, \underline{y} \in E, \text{& } \forall \lambda \in \mathbb{K}$

1. $L(\underline{x} + \underline{y}) = L(\underline{x}) + L(\underline{y})$
2. $L(\lambda\underline{x}) = \lambda L(\underline{x})$

or equivalently,

$L(\lambda_1 \underline{x} + \lambda_2 \underline{y} = \lambda_1 L(\underline{x} + \lambda_2 L(\underline{y})$

## Example A

---
$L:
\begin{matrix}
    C^1(0, 1) \rightarrow C^0(0, 1) \\
    f(x) \rightarrow \frac{\mathrm d}{\mathrm d x}f(x)
\end{matrix}
$ is linear

$\frac{\mathrm d}{\mathrm d x} \big(\lambda_1 f(x) + \lambda_2 g(x) \big) = \lambda_1 f'(x) + \lambda_2 g'(x)$

---

## Example B

---

$L: \begin{matrix}
    C^0(0,1) \rightarrow C^0(0,1) \\
    f(x) \rightarrow \displaystyle\int_0^x f(x')dx'
\end{matrix}
$ is linear.

$\displaystyle \int_0^x \big(\lambda_1 f(x') + \lambda_2 g(x')\big) dx' = \lambda_1 \int_0^x f(x')dx' + \lambda_2 \int_0^x g(x') dx'$

# Nullspace and Range of a Linear Transformation

**Definition:**  Let $L: E_n \rightarrow F_m$ be a linear map.  The $\underline{\text{nullspace}} \text{ } N(L)$ (also called kernel) is the subspace of $E_n$ s.t. $N(L) = \big\lbrace \underline{x} \in E_n \, | \, L \underline{x} = \underline{0} \big\rbrace$

**Definition:** Let $L: E_n \rightarrow F_m$ be a linear map.  The $\underline{\text{Raupe}}$ space of $L$, ($R(L)$) is the subspace of $F_m$ s.t.

$R(L) = \big\lbrace \underline{y} \in F_m \, | \, L \underline{x} = \underline{y} \big\rbrace$

The dimension of $R(L)$ is called $\underline{\text{rank}}$

***Theorem:***  If Let $L: E_n \rightarrow F_m$ and $dim \, E_n = n, \, dim \, F_m = m$ then, $dim \, E_n = dim \, N(L) + dim \, R(L)$

***Theorem:*** Two spaces are isomorphic if they have the same dimension.

Isomorphism: Linear application
$L:
\begin{matrix}
    E \rightarrow F \\
    \underline{x} \rightarrow y = L \underline{x}
\end{matrix}
$ s.t.

1. one-to-one:

    $L(\underline{x}_1) \neq L(\underline{x}_2)$ if $\underline{x}_1 \neq \underline{x}_2 \quad \forall \underline{x}_1, \underline{x}_2 \in E_n$

2. Onto:

    For any $\underline{y} \in F, \exists\, \underline{x} \in E$ s.t. $L(\underline{x}) = \underline{y}$

    Note that the "onto" implies that $L(\underline{x})$ (Roupe) covers the all space $F$.  But then $dim \, R(L) = dim\, F = m = n \Rightarrow dim\, E_n = n = dim\, N(L) + dim\, R(L) \Rightarrow dim\, N(L) = 0$

# TODO: Add figures or python graphics

# Linear Transformation and Matrices: Relationships

Let $E_n$ and $F_m$ be two vector spaces on $\mathbb{K}$

***Theorem:*** if $\lbrace \underline{e}_1, \dots, \underline{e}_n \rbrace$ is a basis of $E_n$, the linear map $L: E_n \rightarrow F_m$ is uniquiely determined by the $n$-transformed vectors of the bases $\lbrace \underline{f}, \dots , \underline{f}_m \rbrace$.  The linear map $L$ is uniquely represented by a matrix $m \times n$.  Conversely, any matrix $(m \times n)$ represents a linear map $L: E_n \rightarrow F_m$

Set the following basis in $E_n \lbrace \underline{e}_1, \dots, \underline{e}_n \rbrace$.  Any vector $\underline{x}$ in $E_n$ can be written as: $\displaystyle \sum_{i=1}^n x_i \underline{e}_i$.  The transformed under $L$ are the following: $L \underline{x} = \displaystyle \sum_{i=1}^n x^i \begin{equation}\underbrace{L \underline{e}_i}_{\underline{\mathcal{E}}_i}\end{equation} = \sum_{i=1}^n x^i \underline{\mathcal{E}}_i$

$\mathcal{E}_i = L \underline{e}_i$ can be represented in the basis $\big\lbrace \underline{f}_1, \dots, \underline{f}_m \big\rbrace$

$\mathcal{E}_i = \displaystyle \sum_j^m \alpha_i^j\, \underline{f}_j, \quad i = 1, \dots, n \quad \quad \lbrace x_i^j \rbrace_{
\begin{matrix}
    i\, = 1,\, \dots\,,\, n \\
    j\, = 1,\, \dots\,,\, m
\end{matrix}
}$
is a $m \times n$ matrix!

The $\lbrace x_i^j \rbrace$ matrix uniqutely represents the transformation of L.

# Application to Linear Systems: Geometric Interpretation

$
\begin{cases}
   \begin{matrix}
       a_1^1 x^1 + & \dots & a_n^1 x^n = y^1
   \end{matrix} \\
   \begin{matrix}
       a_1^m x_1 + & \dots & a_m^n x^n = y^m
   \end{matrix}
\end{cases}
\Rightarrow \quad
\overbrace{A}^{(m \times n)} \underbrace{\underline{x}}_{(n \times 1)} = \underbrace{\underline{y}}_{(m \times 1)}$ is a linear system.

## Problem: Does the System Have a Solution?

If $A$ represents a map $\underline{x} \rightarrow \underline{y}$, solutions can exist if and only if $\underline{y}$ is in the range of $L$ or $\underline{y} \in R(A)$.  Since the range of $A$ is represented by the columns of $A$, that implies that $\underline{y}$ must be expressed as linear combinations of the column of $A$.

$
\begin{bmatrix}
    A\underline{e}_1 & A \underline{e}_2 & \dots & A \underline{e}_n \\
    \vdots           & \vdots            & \vdots& \vdots
\end{bmatrix}
\quad
\underline{x} = \underline{y} \Rightarrow x' (A \underline{e}_1) + \dots + x^n (A \underline{e}_m) = \underline{y}
$

Homegenous Systems: $A \underline{x} = \underline{0} \rightarrow$ Solutions of $m \times n$ homegenous systems form a vector space $\rightarrow$ Null space of the application.

# TODO: Insert Figure here

$A \underline{x} = \underline{0}$ if $m = n \notin \mathbb{R}(A) = rnk(A) = n \Rightarrow$

$\Rightarrow dim E_n = \dim N(A) + \dim \mathbb{R}(A) \Rightarrow n = \dim N(A) + n$

$\Rightarrow \dim N(A) = 0 \Rightarrow $ only possible solution for $A \underline{x} = \underline{0}$ is $\underline{x} = \underline{0}$

## General Solution for $A \underline{x} = \underline{y}$:

The general solution is the sum of **any** solution for $A \underline{x} = \underline{y}$ plus the general solution for the homogeneous system $A \underline{x} = \underline{0}$

Indeed: if $\underline{x} = \underline{x}_P + \underline{x}_H$, we have:

$A \underline{x} = A \underline{x}_P + A \underline{x}_H = \underline{y} + \underline{0}$

thus the solution can be expressed as:

$\underline{x} = \underline{x}_P + \underbrace{c_1 \underline{x}_1 + \dots + c_k \underline{x}_k}_{\text{Fundamental (LI) solutions of } N(A)} \quad \text{where } k = n - \dim \mathbb{R}(A)$

# Representation of an Operator in different basis

A linear transformation $L: E_n \rightarrow E_n$ is also called an $\underline{Operator}$.

Let ${\lbrace \underline{e}_i \rbrace}_{i = 1, \dots, n}$ and ${\lbrace \underline{e}_i' \rbrace}_{i' = 1, \dots, n}$ two basises of $E_n$.  Let $C$ be the matrix $(n \times n)$ representing the change of basis $B \rightarrow B'$.  That is:

$\lbrace \underline{e}_1', \dots, \underline{e}_n \rbrace = \lbrace \underline{e}_1, \dots, \underline{e}_n \rbrace C$

Let $\underline{\underline{A}}$ be the matrix represetnting a linear operator $L$ in $B$.  Conversely, let $\underline{\underline{A}}'$ be the matrix representing the smae operator in $B'$.  The following relationship holds:

$
\begin{cases}
    X' = C^{-1}X \\
    X = CX'
\end{cases} \text{,}\quad \text{if } Y = \underline{\underline{A}}X \text{ it is also true that }
\begin{cases}
    Y' = C^{-1}Y \\
    Y = CY'
\end{cases}
$

Thus we have:

$ Y = \underline{\underline{A}} X \Rightarrow CY' = \underline{\underline{A}} CX' \Rightarrow Y' = C^{-1} \underline{\underline{A}} C X' = \underline{\underline{A}}' X'$

thus the law of transformation is the following:

$\big\lbrace A' = C^{-1} A C \Longleftrightarrow A = C A'C^{-1}\big\rbrace$

**Definition**:  Two $(n \times n)$ matrices $A, B$ s.t. $A = CBC^{-1}$ where $C$ is an $(n \times n)$ invertible matrix, are said to be $\underline{similar}$

# Eigenvalue and Eigenvectors

**Definition:** For a given linear map $A: E_n \rightarrow E_n$ (operator), an element $\lambda \in \mathbb{K}$ is called an $\underline{eigenvalue}$ if $\exists \, {v} \in E_n, \,\, \underline{v} \neq \underline{0} \,\,$ s.t. $A \underline{v} = \lambda \underline{v}$ where $\underline{v}$ is called the $\underline{eigenvector}$

***Theroem:***  Let $E_\lambda$ be the set of all vectors $\underline{u}$, s.t. $A \underline{u}_i = \lambda \underline{u}_i$.  The set $E_\lambda$ plus $\underline{0}$ is a vector subspace of $E$.  $E_\lambda$ is called the $\underline{eigenspace}$ of $\lambda$

## How to find eigen-values (e-values):

Set the basis $\lbrace \underline{e}_i \rbrace_{i=1}^n$ for $E_n$.  $A$ is then represented by a matrix $A (n \times n)$.  To find the evalues, solve for the following:

$A \underline{v} = \lambda \underline{v}$ or $(A - \lambda I) \underline{v} = \underline{0} \rightarrow \text{ homogenous system of linear equations}.$

The solution is non-trivial iff $\det(A - \lambda I) = 0$

The equation $\det (A - \lambda I) = 0$ is an equation of depre $n$ called the $\underline{\text{characteristic equation}}$ and the $\det(A-\lambda I)$ is the $\underline{\text{characteristic polynomial}}$ of $A$, sometimes indicated with $P_A(\lambda)$

***Theroem:*** Similar matrices $A' = C^{-1} A C$ have the same characteristic Polynomial, i.e.:

$P_{A'}(\lambda) = P_A(\lambda)$

## Finding Eigenvalue and Eigenvectors using python:

Using the numpy linear algebra library, the eigen values and eigen vectors can be determined:

```python
import numpy as np

A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
v, k = np.linalg.eig(A)

print("The eigen values are: {v[0]}, {v[1]}, {v[2]}".format(v=v))
```

    The eigen values are: 16.116843969807043, -1.1168439698070427, -1.3036777264747022e-15

```python
from sympy import Matrix
K = Matrix(k)
display(K)  # use print or pprint in iPython console
```

$$\left[\begin{matrix}-0.231970687246286 & -0.785830238742067 & 0.408248290463864\\-0.525322093301234 & -0.0867513392566283 & -0.816496580927726\\-0.818673499356181 & 0.61232756022881 & 0.408248290463863\end{matrix}\right]$$

# Diagonal form of an operator

**Question:**  Is it possible to choose a basis in $E_n$ s.t. $A: E_n \rightarrow E_n$ is represented by the simplest matrix?

**Definition:**  A operator $A$ is aid to be $\underline{diagonizable}$ if $\exists$ a basis of $E_n$ s.t. the matrix $A \, (n \times n)$ is diagonal.

***Thereom:***  $A$ is diagonizable iff it admits $n$ linearly indpendent e-vectors, i.e. there exists a basis $E_n$ formed by e-vectors.

**Definition:**  We call $\underline{\text{Geometric Multiplicity}} \text{ (GM)}$ of e-value $\lambda$ for $A$, the dimension of $E(\lambda) = \lbrace \underline{x} \in E | A \underline{x} = \lambda\underline{x} \rbrace$  Since $A \underline{x} = \lambda \underline{x} \Rightarrow GM = \dim N(A - \lambda I)$

**Definition:**  We call $\underline{\text{Algebraic Multiplicity}} \text{ (AM)}$ of e-value $\lambda$ of $A$ the multiplicity of $\lambda$ a sroot of the characteristic equation $\det (A - \lambda I) = 0$

***Theroem:*** Let $A$ be an operator $A: E_n \rightarrow E_n$, and let $\lambda_1$ be an e-value.  Then:

$\begin{equation}AM(\lambda_1) \leq AM(\lambda_1)\end{equation}$

***Theroem:*** Let $A$ be an operator in $E_n$.  $A$ is $\underline{diagonizable}$ iff, for any evalue of $A \, \lambda_i$, then:

$GM(\lambda_i) = AM(\lambda_i) \quad \forall \lambda_i$ s.t. $A \underline{x} = \lambda_i \underline{x}$

If $A$ is diagonal then $\exists$ a basis of e-vectors such that $A$ **is similar** to a diagonal matrix:

$\begin{cases}
    D = C^{-1} A C && \underline{x} = \mathbb{R}\underline{z} \\
    A = C D C
\end{cases}$

# Jordan form of a linear operator

**Definition:**  We vall a $\underline{\text{Jordan matrix}}$ of order $n$, a matrix with the structure:

$J_n =
\begin{bmatrix}
    \lambda  && \dots  && 0 \\
    0        && \ddots && 0 \\
    0       && \dots   && \lambda
\end{bmatrix}
\quad$  Where $J_n$ is a $n \times n$ matrix.

***Theroem:***  Let $A$ be a linear operator in $E_n$ an $\displaystyle P_A(\lambda) = \prod_{i=1}^z(\lambda - \lambda_i)^{z_i}$.  Then $A$ is similar to a matrix comprising Jordan blocks.  I.e. there exists a basis of vectors of $E_n$ (called generalize e-vectors) represented by a matrix $C$, such that the operator $A$ is similar to the following:

$\displaystyle
J =
\begin{bmatrix}
    \begin{matrix} \lambda_1 && \dots && 0 \\ 0 && \ddots && 0 \\ 0 && \dots && \lambda_1 \end{matrix} && \dots && 0\\
    0 && \ddots && 0 \\
    0 && 0 && \begin{matrix} \lambda_z && \dots && 0 \\ 0 && \ddots && 0 \\ 0 && \dots && \lambda_z \end{matrix} \\
\end{bmatrix}
$

The similarity condition is expressed as:

$J = C^{-1} A C \Longleftrightarrow A = CJC^{-1}$

# Theorem: (Cayley - Hamilton)

**Definition:**  For a given polynomial $P(t) = a_0 t^n + a_1 t^{n-1} + \dots + a_{n-1}t + a_n$ and an operator $A$ in $E_n$, we define the $\underline{\text{polynomial of operator}} \, p(A)$, the following operator:

$p(A) = a_0 A^{n} + a_1 A^{n-1} + \dots + a_{n-1} A + a_n$

Consider the operator $A$ represented by matrix $\underline{\underline{A}}$ in a specified basis $\displaystyle \lbrace \underline{e}_i \rbrace_{i=1}^n$.  Then the following is true:

$P(\underline{\underline{A}}) = 0 \text{ where } P(\underline{\underline{A}}) = \det (\underline{\underline{A}} - \lambda \underline{\underline{I}})$

# Euclidean Vector Space

Let $E$ be a vector space on $\mathbb{R}$.

**Definition:**  We define the scalar product or inner product of $E$ as a bilinear transformation:  $g: E \times E \rightarrow \mathbb{R}$ s.t. the following properties are true:

1. $(\underline{u}, \underline{v} = (\underline{v}, \underline{u}), \quad \forall \underline{u}, \underline{v} \in E$ (Commulative)
2. $(\alpha \underline{u}, \underline{v}) = (\underline{u}, \alpha \underline{v}), \quad \forall \underline{u}, \underline{v} \in E, \quad \forall \alpha \in \mathbb{R}$
3. $(\underline{u}, \underline{v} + \underline{w}) = (\underline{u}, \underline{v}) + (\underline{u}, \underline{w}), \quad \forall \underline{u}, \underline{v}, \underline{w} \in E$
4. $(\underline{u}, \underline{u}) \geq 0, \quad\forall \underline{u} \in E$ and if $(\underline{u}, \underline{u}) = 0 \Rightarrow \underline{u} = \underline{0}$

Basically $ g(\underline{u}, \underline{v}) = \underline{u} \cdot \underline{v}$ is a bilinear form on $E \times E$ that is symmectical and positive definite.

**Definition:**  A vector space $E$ is said to be $\underline{Euclidean}$ if a scalar/inner product is defined on it.

**Definition:**  Two vectors in $E$, $\underline{u}, \underline{v}$ are said to be orthogonal iff $(\underline{u}, \underline{v}) = \underline{u} \cdot \underline{v} = 0$

**Definition:**  We define $\underline{norm}$ of a vector: $\underline{u} \in E$ the following $||\underline{u}|| = \sqrt{(\underline{u}, \underline{v})} = \sqrt{\underline{u} \cdot \underline{v}}$

**Definition:**  A set of vectors $\lbrace \underline{e}_i \rbrace_{i=1}^n$ is said to be orthonormal iff:

$(\underline{e}_i, \underline{e}_k) = \delta_{ik} \begin{cases} 1, && \text{if } i=k\\0, && \text{if } i \neq k\end{cases}$

# Adjoint Operators

**Note:** A vector space on $C$ is said to be a Hilbert space if there is a inner product $g: E \times E \rightarrow C$ that satisfies the following properties:

1. $(\underline{u}, \underline{v}) = (\overline{\underline{u}, \underline{v}}) \quad \forall \underline{u}, \underline{v} \in E$
2. $(\lambda \underline{u} + \mu \underline{v}, \underline{w}) = \lambda (\underline{u}, \underline{w}) + \mu (\underline{v}, \underline{w}) \quad \forall \underline{u}, \underline{v} \in E, \quad \forall \lambda, \mu \in C$
3. $(\underline{u}, \underline{u}) \geq 0 \text{ and } (\underline{u}, \underline{u} = 0 \Longleftrightarrow \underline{u} = \underline{0}$

Consider now a Hilbert (Euclidean) space equipped with inner product.

**Definition:**  We call $A^*$ adjoint operator of $A$, an operator that satisfies the following property:

$(A \underline{x}, \underline{y}) = (\underline{x}, A^*\underline{y}), \quad \forall \underline{x}, \underline{y} \in E$

An important class of operators are the $\underline{self-adjoint}$ operators, i.e. operators like $A = A^*$:

$(A\underline{x}, \underline{y}) = (\underline{x}, A \underline{y})$

Note that if the space $E$ is Euclidean, then $A^* = A^T$ and $(A\underline{x},\underline{y}) = (\underline{x}, A^T\underline{y})$

Self-adjoint operators are therefore $\underline{Symmetric}$ $A=A^T$

## Spectral Theorem

A symmetric operator $A=A^T$ is always diagonizable.  Its e-values are all **real** and the e-vectors are orthogonal, i.e. $\exists$ matrix $C$ made of e-vectors $C = \big[\underline{v}_1, \dots, \underline{v}_n\big]$ s.t. the operator $A$ is similar to a diagonal operator (can be represented by a diagonal matrix in the basis of e-vectors).

For $A\underline{v} = \lambda \underline{v}$ we can consider a transformation $x = M\underline{v}$ s.t.

$AM^{-1}\underline{z} = \lambda M^{-1}\underline{z} \Rightarrow MAM^{-1}\underline{z} = \lambda MM^{-1}\underline{z} \Rightarrow D\underline{z} = \lambda \underline{z}$

OR

$\big\lbrace MAM^{-1} = D \Longleftrightarrow A = M^{-1}DM \big\rbrace \rightarrow \text{ similarity condition}$


# One-dimensional convolutions

In this short post we build on our previous discussion of [single-dimensional moving averages](https://jermwatt.github.io/control-notes/posts/moving_averages/Moving_averages.html) by describing their generalization as single-dimensional *convolutions*.  As in that prior post here we too will be working with a generic ordered sequence of $P$ points denoted as

\begin{equation}
x_1,\, x_2,\, \ldots,\, x_P.
\end{equation}

You can skip around this document to particular subsections via the hyperlinks below.

-  [Generalizing the moving average](#generalizing-moving-average)
-  [Common convolution filters](#common-filters)

```python
## This code cell will not be shown in the HTML version of this notebook
# imports from custom library for animations
import sys
sys.path.append('../../')
from demo_libraries import timeseries_library as tlib
import matplotlib.pyplot as plt

# import standard libs
import numpy as np
import pandas as pd
from IPython.display import clear_output

# path to data
datapath = '../../datasets/plain_timeseries/'

# This is needed to compensate for matplotlib notebook's tendancy to blow up images when plotted inline
%matplotlib notebook
from matplotlib import rcParams
rcParams['figure.autolayout'] = True

%load_ext autoreload
%autoreload 2
```

## Generalizing the moving average

In our previous [post on moving averages](https://jermwatt.github.io/control-notes/posts/moving_averages/Moving_averages.html)) we saw the moving average of *order* or *window-length* $D$ is itself a sequence of the same length, with its first $D$ values equal to the sequence itself as

\begin{equation}
h_p = x_p \,\,\,\,\,\, p=1,...,D.
\end{equation}

and in general for $p \geq D+1$ we have

\begin{equation}
h_{p} = \frac{x_{p-1} + x_{p-2} + \cdots + x_{p-D}}{D}.
\end{equation}

The functionality used here (the simple *average*) can in principle be replaced with any general function one desires.  However if we swap out the average with something just slightly more complicated (relatively speaking) than a straight-up average - a *linear combination* - we would instead compute updates of the form

\begin{equation}
h_{p} = w_1x_{p-1} + w_2x_{p-2} + \cdots + w_{D}x_{p-D}.
\end{equation}

Here the weights $w_1,\,w_2,\,...,w_{D}$ can be set however we please, and if set to $w_d = \frac{1}{D}$ for all $d$ we recover the moving average update above.

The set of such linear combinations are often referred to as *convolutions*, and the weights $w_1,\,...,w_{D}$ as *filters* or *kernels* (see [endnotes](#endnotes) for further discussion of this term).

Like the moving average, convolutions *dynamic systems* that are often used to smooth and - more generally - clean up time series for further processing.  As with the moving average, general convolutions

## Common convolution filters

```python
## This code cell will not be shown in the HTML version of this notebook
def plot_weights(weights,title):
    # create figure
    fig = plt.figure(figsize = (9,4))
    O = len(weights)
    x = np.arange(1,O+1,1)

    # scatter weights
    plt.scatter(x,weights,s=80,c='k',edgecolor='w',linewidth=1,zorder=1)
    for i in range(len(weights)):
        xv = x[i]
        yv = weights[i]
        plt.plot((xv,xv),(0,yv),c='k',zorder=0,linestyle='--')

    # cleanup plot
    plt.plot((x[0]-1,x[-1]+1),(0,0),c='k',linewidth=1)
    plt.title(title,fontsize=18)
    plt.xlabel(r'$d$',fontsize=16)
    plt.ylabel(r'$w_d$',fontsize=16,rotation=0,labelpad=20)
    plt.show()
```

#### Example 1:   Average filter

Viewed through the lens of convolution, a moving average is often referred to as a "mean filter".  Below we produce two Python functions that illustrate this perspective, and allow for immediate generalization to other filters.  The  `mean_weights` function creates a set of "mean weights" used to produce a moving average - that is, the elements of a series in each window of length $D$ are multiplied by these weights.  This function takes in only one input: $D$ the order of the moving average.

```python
# make average weights
def mean_weights(D):
    weights = [1/D for v in range(D)]
    return weights
```

We can visualize these weights - as shown below for the case where $\mathcal{D} = 10$.  Here each black dot represents a weight value, and the dashed lines are drawn for visualization purposes only.

```python
## This code cell will not be shown in the HTML version of this notebook
D = 10
w = mean_weights(D)
plot_weights(w,title='mean filter')
```

    <IPython.core.display.Javascript object>

We can then produce a moving average via the `linear_filter` function below - which takes in a time series `x`,  a window length `O`, and a set of filter weights `w`.

```python
# general linear filter function
def linear_filter(x,w):
    # filter input signal
    D = len(w)
    y = [v for v in x[:D]]
    for p in range(len(x) - D):
        # make next element
        b = sum([a*b for a,b in zip(x[p:p+D],w)])
        y.append(b)
    return np.array(y)
```

This process, animated below for a particular time series, smooths the input time series.  The window in which each average is computed is straddled on both sides by vertical blue bars, and as the animation progresses from left to right across the series the resulting moving average is shown as an orange series.

```python
## This code cell will not be shown in the HTML version of this notebook
# load in data
csvname = datapath + 'ford_data.csv'
data = pd.read_csv(csvname)
x = np.array(data['Close'])    # date: 1975 to 2019

# produce moving average time series
D = 10
w = mean_weights(D)
y = linear_filter(x,w)

# run animator
demo = tlib.time_series_animators.Visualizer()
demo.animate_moving_ave(x,y,D,savepath='videos/animation_1.mp4')
clear_output()
```

```python
## This code cell will not be shown in the HTML version of this notebook
from IPython.display import HTML
HTML("""

""")
```

#### Example 2:   Sinc filter

Other popular choices for weights outside of the average include average cosine and sinc weights.  These arise naturally in the study of [signal processing](http://sam-koblenski.blogspot.com/2015/09/everyday-dsp-for-programmers-averaging.html) where they go by names like ["low pass filtering"](https://en.wikipedia.org/wiki/Low-pass_filter).  To create such weights we sample an elementary function, like for example the sinc function shown below for $D=30$, and multiply the resulting weights by windowed portions of our input series precisely as we did with the original moving average weights above.

```python
## This code cell will not be shown in the HTML version of this notebook
# compute centered sinc weights
def sinc_weights(D,**kwargs):
    v = np.arange(0.5-D/2,D/2,1)
    freq = 1
    if 'freq' in kwargs:
        freq = kwargs['freq']
    weights = []
    for o in v:
        spot = o/D
        if np.abs(spot) < 10**(-8):
            spot = 1
        h = np.sin(2*np.pi*freq*spot)/(2*np.pi*freq*spot)
        weights.append(h)

    # normalize
    h_sum = sum(weights)
    weights = [h/h_sum for h in weights]
    return weights
```

```python
## This code cell will not be shown in the HTML version of this notebook
D = 30
w = sinc_weights(D,freq=3)
plot_weights(w,title='sinc filter')
```

    <IPython.core.display.Javascript object>

To produce this set of sinc weights we simply adjust the basic sinc function, and divide off the order of the window as shown below.

```python
# compute centered sinc weights
def sinc_weights(D,**kwargs):
    v = np.arange(0.5-D/2,D/2,1)
    freq = 1
    if 'freq' in kwargs:
        freq = kwargs['freq']
    weights = []
    for o in v:
        spot = o/D
        if np.abs(spot) < 10**(-8):
            spot = 1
        h = np.sin(2*np.pi*freq*spot)/(2*np.pi*freq*spot)
        weights.append(h)

    # normalize
    h_sum = sum(weights)
    weights = [h/h_sum for h in weights]
    return weights
```

This set of weights will smooth an input series, leaving a more trigonometric fit than the moving average.

```python
## This code cell will not be shown in the HTML version of this notebook
# load in data
csvname = datapath + 'ford_data.csv'
data = pd.read_csv(csvname)
x = np.array(data['Close'])    # date: 1975 to 2019

# produce moving average time series
D = 10
w = sinc_weights(D,freq=1)
y = linear_filter(x,w)

# run animator
demo = tlib.time_series_animators.Visualizer()
demo.animate_moving_ave(x,y,D,savepath='videos/animation_2.mp4')
clear_output()
```

```python
## This code cell will not be shown in the HTML version of this notebook
from IPython.display import HTML
HTML("""

""")
```

#### Example 3:   Cosine filter

By similarly sampling a cosine we can create cosine filter - whose weights are shown below.

```python
## This code cell will not be shown in the HTML version of this notebook
# produce cosine weights
def cos_weights(D,**kwargs):
    v = np.linspace(-np.pi/2,np.pi/2,D)
    freq = 1
    if 'freq' in kwargs:
        freq = kwargs['freq']
    weights = []
    for o in v:
        h = np.cos(o)
        weights.append(h)

    # normalize
    h_sum = sum(weights)
    weights = [h/h_sum for h in weights]
    return weights

# plot weights
D = 30
w = cos_weights(D)
plot_weights(w,title='cos filter')
```

    <IPython.core.display.Javascript object>

## Endnotes

The general linear combination in Equation (4) is more accurately known as *cross-correlation*, which is closely related to the convolution operation which is more commonly defined as

\begin{equation}
h_{p} = w_{D}x_{p-1} + w_{D-1}x_{p-2} + \cdots + w_{1}x_{p-D}.
\end{equation}

Here the weight sequence $w$ has been reversed before being multiplied by the elements of $x$.  Flipping the order of the weights makes for a more awkward formula, and gaurntees certain mathematical niceties like   e.g., the [commutative property](https://en.wikipedia.org/wiki/Convolution).  But because these mathematical ideas do not concern us - and for the sake of clarity in notation - we mish-mash the terms corss-correlation and convolution.