## Belief Networks

```python
import graphviz as gz
import numpy as np
```

Belief networks are a way to depict the independence assumptions made in a distribution.

### Wet Grass Example (3.1.1)
$
\begin{align}
    R &\in {0,1} \quad &&\text{R = 1 means that it has been raining, and 0 otherwise} \\
    S &\in {0,1} \quad &&\text{S = 1 means that sprinkler was left on, and 0 otherwise} \\
    J &\in {0,1} \quad &&\text{J = 1 means that Jack's grass is wet, and 0 otherwise} \\
    T &\in {0,1} \quad &&\text{T = 1 means that Tracey's grass is wet, and 0 otherwise} \\
\end{align}
$

A model of the world corresponds to $p(T,J,R,S)$. One way to decompose this joint distribution is

$
\begin{align}
    p(T,J,R,S) &= p(T \mid J,R,S)p(J,R,S) \\
               &= p(T \mid J,R,S)p(J \mid R,S)p(R,S) \\
               &= p(T \mid J,R,S)p(J \mid R,S)p(R \mid S)p(S) \\
\end{align}
$

There may be other decompositions. For example, instead of conditioning on $J,R,S$ combined, we could have conditioned on $T,J,R$. However, in this example, conditioning in this way allows us to incorporate our independence assumptions into the model.

#### Conditional Independence Assumption 1
We assume that whether Tracey's grass is wet or not does not **directly depend** on whether Jack's grass is wet or not. That is $p(T \mid J,R,S) = p(T \mid R,S)$.

#### Conditional Independence Assumption 2
We assume that whether Jack's grass is wet or not does not **directly depend** on whether Tracey's sprinkler was left on or not. That is $p(J \mid R,S) = p(J \mid R)$.

#### Conditional Independence Assumption 3
We assume that rain does not **directly depend** on whether sprinkler was left on or not. That is $p(R \mid S) = p(R)$.

#### Final Model
In the end, our model of the world becomes

$$
p(T,J,R,S) = p(T \mid R,S)p(J \mid R)p(R)p(S)
$$

To completely specify the model, we need $2^2 + 2^1 + 2^0 + 2^0 = 8$ probability values in total.

#### Corresponding Bayesian Belief Network

```python
G = gz.Digraph()
G.edge('R', 'T')
G.edge('S', 'T')
G.edge('R', 'J')
G
```

Parents of node $T$ are the variables that $T$ is dependent ($R$ and $S$).

#### Complete Model
We specify all of the 8 values to complete our model.

$
\begin{align}
    &p(R = 1) &&= 0.2 \\
    &p(S = 1) &&= 0.1 \\
    &p(J = 1 \mid R = 1) &&= 1 \\
    &p(J = 1 \mid R = 0) &&= 0.2 \\
    &p(T = 1 \mid R = 1, S = 0) &&= 1 \\
    &p(T = 1 \mid R = 1, S = 1) &&= 1 \\
    &p(T = 1 \mid R = 0, S = 1) &&= 0.9 \\
    &p(T = 1 \mid R = 0, S = 0) &&= 0 \\
\end{align}
$

which can be written as follows:

```python
"""
   T | 0   1
 R,S |
 -------------
 0,0 | 1   0
 0,1 | 0.1 0.9
 1,0 | 0   1
 1,1 | 0   1
"""

p_R = np.array([0.8, 0.2])
p_S = np.array([0.9, 0.1])
p_J_given_R = np.array([[0.8, 0.2],
                        [0, 1]])
p_T_given_R_S = np.array([[1, 0],
                          [0.1, 0.9],
                          [0, 1],
                          [0, 1]])
```

#### Inference
After specifying the complete model, we can make inference by manipulating the expressions and using the values we have specified. For example,

$
\begin{align}
    p(S = 1 \mid T = 1, J = 1) &= \sum_R p(S = 1, R \mid T = 1, J = 1) \\
                               &= \frac{\sum_R p(S = 1, R, T = 1, J = 1)}{p(T = 1, J = 1)} \\
                               &= \frac{\sum_R p(S = 1, R, T = 1, J = 1)}{\sum_{R,S} p(T = 1, J = 1, R, S)} \\
                               &= \frac{\sum_R p(T = 1 \mid R, S = 1)p(J = 1 \mid R)p(R)p(S = 1)}{\sum_{R,S} p(T = 1 \mid R, S)p(J = 1 \mid R)p(R)p(S)} \\
\end{align}
$

```python
j1r = p_J_given_R[:, 1]
t1rs1 = p_T_given_R_S[[1, 3], 1]
r = p_R
s1 = p_S[1]
nom = np.sum(j1r*t1rs1*r*s1)
```

$p(J = 1 \mid R)$ = {{j1r}}

$p(T = 1 \mid R, S = 1)$ = {{t1rs1}}

$p(R)$ = {{r}}

$p(S = 1)$ = {{s1}}

$\sum_R p(J = 1 \mid R)p(T = 1 \mid R, S = 1)p(R)p(S = 1)$ = {{nom}}

```python
j1r1 = p_J_given_R[1, 1]
t1r1s = p_T_given_R_S[2:, 1]
r1 = p_R[1]
s = p_S
j1r0 = p_J_given_R[0, 1]
t1r0s = p_T_given_R_S[:2, 1]
r0 = p_R[0]
denom = np.sum(j1r1*t1r1s*r1*s + j1r0*t1r0s*r0*s)
```

1. ** R = 1 case **
  * $p(J = 1 \mid R = 1)$ = {{j1r1}}
  * $p(T = 1 \mid R = 1, S)$ = {{t1r1s}}
  * $p(R = 1)$ = {{r1}}
  * $p(S)$ = {{s}}

2. ** R = 0 case **
  * $p(J = 1 \mid R = 0)$ = {{j1r0}}
  * $p(T = 1 \mid R = 0, S)$ = {{t1r0s}}
  * $p(R = 0)$ = {{r0}}
  * $p(S)$ = {{s}}

$\sum_{R,S} p(J = 1 \mid R)p(T = 1 \mid R, S)p(R)p(S)$ = {{denom}}

```python
nom/denom
```

    0.16044776119402987

$p(S = 1 \mid T = 1, J = 1)$ = {{nom/denom}}

### Uncertain Evidence and Jeffrey's Rule
We don't know the exact state of the evidence variable, i.e. we have a probability distribution over the evidence variable. For example, if $dom(x) = \{red, blue, green\}$,
* **certain evidence** would give us a distribution of (0, 1, 0),
* **uncertain evidence** gives us a distribution of (0.2, 0.3, 0.5).

Performing inference with soft evidence can be achieved using Bayes' rule. For example, let our model be $p(x, y)$ and that we have some soft (uncertain) evidence $\tilde{y}$ about the variable $y$. We want to know how our beliefs are going to change with this new evidence, i.e. we want to calculate $p(x \mid \tilde{y})$.

#### Jeffrey's Rule
$$
p(x \mid \tilde{y}) = \sum_y p(x, y \mid \tilde{y}) = \sum_y p(x \mid y, \tilde{y})p(y \mid \tilde{y}) = \sum_y p(x \mid y)p(y \mid \tilde{y}).
$$

Here, $p(y = i \mid \tilde{y})$ represents the probability that $y$ is in state $i$ given the soft-evidence. For example, assume $dom(x) = \{red, blue, green\}$ and the soft evidence $\tilde{x}$ gives us the distribution $(0.2, 0.3, 0.5)$. Then, $p(x = blue \mid \tilde{x}) = 0.3$.

#### Example 3.2
$
\begin{align}
p(B = tr \mid W = tr, \tilde{G}) &= p(B = tr \mid W = tr, G)p(G \mid \tilde{G}) \\
                                 &= p(B = tr \mid W = tr, G = tr)p(G = tr \mid \tilde{G}) + p(B = tr \mid W = tr, G = fa)p(G = fa \mid \tilde{G}).
\end{align}
$

### Unreliable Evidence
We have evidence about $G$ but we are unsure about its accuracy. To solve this problem, we replace rv $G$ with a new variable $H$ such that

$
p(G \mid A) \rightarrow p(H \mid A), \quad \text{ where } p(H \mid A) = \begin{cases}
                                                                            0.8,& A = tr \\
                                                                            0.2,& A = fa \\
                                                                        \end{cases}
$

Hence, our new joint probability becomes $p(B, A, H, W) = p(A \mid B)p(B)p(W \mid A)p(H \mid A)$. $H$ is arbitrary and fixed, and its value doesn't change $p(H \mid A)$. Here, we use $p(H \mid A)$ to simply replace the evidence about $G$ with our own beliefs.

### Formal Definition
A belief network is a distribution of the form
$$
p(x_1, \dots, x_D) = \prod_{i=1}^D p(x_i \mid pa(x_i))
$$
where $pa(x_i)$ represents the parental variables of $x_i$. Represented as a digraph with arrows pointing from parents to children, a belief network corresponds to a **Directed Acyclic Graph (DAG)**, with the $i^{th}$ node in the graph corresponding to the factor $p(x_i \mid pa(x_i))$.

A BBN specifies which variables are the 'direct causes' of which variable, namely $pa(x_i)$ are the 'direct causes' of $x_i$. However, this does not necassarily mean that $x_i$ and $pa(pa(x_i))$ are independent.

#### BBN Example

```python
G = gz.Digraph()
G.edges(['AB', 'AC', 'AE', 'CD', 'BD', 'DE'])
G
```

corresponds to the following model
$$
p(A, B, C, D, E) = p(E \mid A, D)p(D \mid B, C)p(B \mid A)p(C \mid A)p(A)
$$

#### Same Distribution Multiple Graphs
Multiple BBNs may correspond to the same probability distribution. This can easily be verified by the following example
$$
p(x_1, x_2, x_3) = p(x_1 \mid x_2, x_3)p(x_2 \mid x_3)p(x_3) = p(x_3 \mid x_1, x_2)p(x_2 \mid x_1)p(x_1)
$$

#### Algorithm to Construct BBN on variables $x_1, \dots, x_n$
Suppose we have the joint distribution $p(x_1, \dots, x_n)$. Then an algorithm to construct a BBN is

1. write down the $n$-node cascade BBN,
2. label the nodes with the variables in any order,
3. if variables $x_i$ and $x_j$ are independent, delete the edge between $x_i$ and $x_j$.

#### Dependencies and the Markov Blanket
Consider a distribution on a set of variables $X$, the corresponding BBN DAG $G$, and $x_i \in X$. Let $MB(x_i)$ be the set of variables in the Markov Blanket of $x_i$. Then, for any variable $y \in X \setminus \{x_i \cup MB(x_i)\}, x_i \ci y \mid MB(x_i)$. That is, Markov Blanket of $x_i$ carries all information about $x_i$.

Graphical dependence doesn't always mean actual dependence. Specific distribution instances may result in graphically dependent variables being independent.

#### Collisions and Conditional Independence
Given a path $P$, a **collider** is a node $c$ on $P$ with neighbors $a$ and $b$ such that $a \rightarrow c \leftarrow b$. A collider is a path specific, i.e. a node can be a collider on path $P_1$ but may not be a collider on path $P_2$.

##### non-collider $z$ which is conditioned along the path between $x$ and $y$
This path cannot induce dependence between $x$ and $y$ since $z$ gives all the information about $y$ that $x$ could give.

```python
G = gz.Digraph()
G.node('z', color='green')
G.edges(['xz', 'zy'])
G
```

##### collider $z$ along the path which is not in conditioning set (and none of its descendants)
In this case, both $x$ and $y$ directly or indirectly affect the state of $z$. If we don't know anything about the state of $z$, we can't form a dependence between the root causes $x$ and $y$.

```python
G = gz.Digraph()
G.node('z', color='green')
G.edges(['xz', 'yz'])
G
```

##### path between $x$ and $y$ which contains no colliders and no conditioning variables
This path **d-connects** the variables $x$ and $y$ (makes them graphically dependent).

```python
G = gz.Digraph()
G.node('z', color='red')
with G.subgraph(name='cluster') as A:
    A.edges(['xh', 'hy'])
G.edge('z', 'y')
G
```

#### Effects of Marginalizing and Conditioning

##### Collider

```python
G = gz.Digraph()
G.node('C', color='red')
G.edges(['AC', 'BC'])
G
```

* **Marginalizing** over a variable can be viewed as not having any information about that variable.
* **Conditioning** on a variable can be viewed as having information about that variable.

* Initially, $A$ and $B$ are unconditionally independent.
* If we marginalize over $C$, we do not know anything about $C$, then we can't induce any dependence between $A$ and $B$.
* If we condition on $C$, we have some information about $C$. If we know something about $C$, we learn something about how the affects of $A$ and $B$ combine to produce that result. Hence, $A$ and $B$ becomes graphically dependent.
* Similarly, if we condition on a descendant of $C$, we learn something about $C$, and thus indirectly form a dependence between $A$ and $B$.

##### Non-collider
In non-collider cases, $C$ can be
1. result of $A$ and cause of $B$.
2. cause of $A$ and result of $B$,
3. root cause of both $A$ and $B$,

In all the cases,

* if we marginalize $C$, we don't have any information about $C$, yet still we know that $A$ and $B$ are related. These relations are
  1. $A$ indirectly causes $B$,
  2. $B$ indirectly causes $A$,
  3. $A$ and $B$ have the same root cause.
In all of these cases, removing $C$ out of the picture doesn't get rid of the fact that $A$ and $B$ are related. Hence, marginalizing $C$ induces a graphical dependence between $A$ and $B$.

* if we condition on $C$, we have information about $C$. In this case,
  1. $C$ explains $B$. Hence, $A$ and $B$ become independent,
  2. $C$ explains $A$. Hence, $A$ and $B$ become independent,
  3. $C$ explains both $A$ and $B$. Hence, $A$ and $B$ become independent.

#### d-Separation Definition
Let $X$, $Y$, $Z$ be disjoint vertex sets on DAG $G$.

For every variable $x \in X$ and $y \in Y$, check every path $U$ between $x$ and $y$. A path $U$ is **blocked** if there is a node $w$ on $U$ such that either

1. $w$ is a collider and neither $w$ nor any of its descendants is in $Z$, or
2. $w$ is not a collider on $U$ and $w$ is in $Z$.

A blocked path cannot induce dependence. If all paths between $X$ and $Y$ are blocked, then $X$ and $Y$ are **d-separated** by $Z$. If $X$ and $Y$ are not d-separated, they are **d-connected**.

\begin{remark}
If the variable sets $X$ and $Y$ are d-separated by $Z$, they are independent conditional on $Z$ in all probability distributions such a graph can represent.
\end{remark}

\begin{remark}
While d-separation implies independence between variables, d-connection does not always imply dependence between variables. Hence, while belief networks are a good tool to ensure independence assumptions in a model, they are not an appropriate tool to ensure dependence assumptions.
\end{remark}

#### Markov Equivalence in BBNs
Two (directed or undirected) graphs are Markov equivalent if they both represent the same set of conditional independence statements.

* **Immorality in a DAG:** A configuration of three nodes $A$, $B$, $C$ such that $C$ is a child of both $A$ and $B$, with $A$ and $B$ not directly connected.
* **Skeleton of a DAG:** Graph obtained by removing directions on the edges.

\begin{remark}
Two DAGs are Markov equivalent iff they have the same skeleton and the same set of immoralities.
\end{remark}

### Causality
* Formally, BNs only make independence statements, not causal ones.
* There is a difference between 'given that we see' (observational evidence) and 'given that we do' (interventional evidence).

#### Simpson's Paradox (3.4.1)
A model of the gender, drug and recovery is $p(G, D, R) = p(R \mid G, D)p(D \mid G)p(G)$. However, if we interpret the experiment from a causal perspective, we intervene and give the drug. In this case, $p(D \mid G)$ should not be used since we give the drug regardless of the gender. Hence, our new model of the experiment should be

$$
\tilde{p}(G, R \mid D) = p(R \mid G, D)p(G).
$$
Then, we can make inference according to the following formula
$$
\tilde{p}(R \mid D) = \sum_G \tilde{p}(R \mid D, G)p(G)
$$

#### do-calculus
In setting any variable into a particular state we need to remove all parental links to that variable. This is called the **do-operator**. With this operator, we differentiate an observational inference $p(x \mid y)$ from a causal inference $p(x \mid do(y))$.

Let all the variables $X = X_I \cup X_O$ be written in terms of intervention variables $X_I$ and observation variables $X_O$. For a belief network $p(X) = \prod_i p(X_i \mid pa(X_i))$, **inferring the effect of setting intervention variables to their corresponding states is equivalent to standard evidential inference in the post intervention distribution**:

$$
p(X_O \mid do(X_{I_1} = x_1), \dots, do(X_{I_k} = x_k)) = \prod_{X_j \in X_O} p(X_j \mid pa(X_j)).
$$

That is, intervention variables are set in particular states, and the corresponding terms $p(X_{I_i} \mid pa(X_{I_i})$ are removed from the original BBN. Graphically, this corresponds to

1. consider each intervention variable
2. cut the connections to its parents
3. set the variable to its intervention state

This operation intuitively corresponds to first setting the causal variables, and then performing the rest of the experiment with the non-causal variables.

#### Influence Diagrams
Another way to represent intervention is by appending a parential decision variable $F_X$ to each variable $X$ on which an intervention can be made. For example, we can modify $p(G, D, R) = p(R \mid G, D)p(D \mid G)p(G)$ to obtain

$$
\tilde{p}(G, D, R, F_D) = p(R \mid G, D)p(D \mid G, F_D)p(F_D)p(G).
$$
where
$$
p(D \mid G, F_D = \varnothing) = p(D \mid pa(D)), \quad p(D \mid G, F_D = d) = 1 \text{ for } D = d
$$

Here, by setting $F_D$ to a state of $D$, we can intervene and set $D$ to a specific value.


# AutoDiff by Symboic Representation in Julia

```julia
using Symbolics
```

```julia
i(x) = x
f(x) = 3x^2
g(x) = 2x^2
h(x) = x^2
w_vec = [i, h, g, f]

@variables x
```

\begin{equation}
\left[
\begin{array}{c}
x \\
\end{array}
\right]
\end{equation}

```julia
function forward_fn(w_vec, x, i::Int)
    y = w_vec[i](x)
    i == size(w_vec)[1] ? y : [y; forward_fn(w_vec,y,i+1)]
end
```

    forward_fn (generic function with 1 method)

```julia
x_vec = forward_fn(w_vec, x, 1)
display(x_vec)
```

\begin{equation}
\left[
\begin{array}{c}
x \\
x^{2} \\
2 x^{4} \\
12 x^{8} \\
\end{array}
\right]
\end{equation}

```julia
function gradient(w_i, x_i_1, x)
    dy = expand_derivatives(Differential(x)(w_i(x)))
    substitute(dy, (Dict(x=>x_i_1,)))
end

function reverse_autodiff(w_vec, x_vec, x, i::Int)
    if i == 1
        1
    else
        gradient(w_vec[i],x_vec[i-1],x) *
            reverse_autodiff(w_vec, x_vec, x, i-1)
    end
end
```

    reverse_autodiff (generic function with 1 method)

```julia
y_sad = x_vec[end]
display(y_sad)
dy_sad = reverse_autodiff(w_vec, x_vec, x, size(w_vec)[1])
display(dy_sad)
```

\begin{equation}
12 x^{8}
\end{equation}

\begin{equation}
96 x^{7}
\end{equation}

## Check by theory

```julia
y_th = f(g(h(x)))
display(y_th)
dy_th = expand_derivatives(Differential(x)(y_th))
display(dy_th)
```

\begin{equation}
12 x^{8}
\end{equation}

\begin{equation}
96 x^{7}
\end{equation}

```julia

```


# Shell geometry

## Init symbols for *sympy*

```python
from sympy import *
from geom_util import *
from sympy.vector import CoordSys3D
import matplotlib.pyplot as plt
import sys
sys.path.append("../")

%matplotlib inline

%reload_ext autoreload
%autoreload 2
%aimport geom_util
```

```python
# Any tweaks that normally go in .matplotlibrc, etc., should explicitly go here
%config InlineBackend.figure_format='retina'
plt.rcParams['figure.figsize'] = (12, 12)

plt.rc('text', usetex=True)

plt.rc('font', family='serif')
# SMALL_SIZE = 42
# MEDIUM_SIZE = 42
# BIGGER_SIZE = 42

# plt.rc('font', size=SMALL_SIZE)          # controls default text sizes
# plt.rc('axes', titlesize=SMALL_SIZE)     # fontsize of the axes title
# plt.rc('axes', labelsize=MEDIUM_SIZE)    # fontsize of the x and y labels
# plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
# plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
# plt.rc('legend', fontsize=SMALL_SIZE)    # legend fontsize
# plt.rc('figure', titlesize=BIGGER_SIZE)  # fontsize of the figure title

init_printing()
```

```python
N = CoordSys3D('N')
alpha1, alpha2, alpha3 = symbols("alpha_1 alpha_2 alpha_3", real = True, positive=True)
```

## Cylindrical coordinates

```python
R, L = symbols("R L", real = True, positive=True)
```

```python
a1 = pi / 2 + (L / 2 - alpha1)/R

x = (R + alpha3)* cos(a1)
y = alpha2
z = (R + alpha3) * sin(a1)

r = x*N.i + y*N.j + z*N.k
```

```python
R1=r.diff(alpha1)
R2=r.diff(alpha2)
R3=r.diff(alpha3)
```

```python
R1
```

```python
R2
```

```python
R3
```

### Draw

```python
import plot

%aimport plot

alpha1_x = lambdify([R, L, alpha1, alpha3], x, "numpy")
alpha3_z = lambdify([R, L, alpha1, alpha3], z, "numpy")

R_num = 1/0.8
L_num = 2
h_num = 0.1

x1_start = 0
x1_end = L_num
x3_start = -h_num/2
x3_end = h_num/2

def alpha_to_x(a1, a2, a3):
    x=alpha1_x(R_num, L_num, a1, a3)
    z=alpha3_z(R_num, L_num, a1, a3)
    return x, 0, z

plot.plot_init_geometry_2(x1_start, x1_end, x3_start, x3_end, alpha_to_x)
```

```python
%aimport plot

R3_1=R3.dot(N.i)
R3_3=R3.dot(N.k)

R3_1_x = lambdify([R, L, alpha1, alpha3], R3_1, "numpy")
R3_3_z = lambdify([R, L, alpha1, alpha3], R3_3, "numpy")

def R3_to_x(a1, a2, a3):
    x=R3_1_x(R_num, L_num, a1, a3)
    z=R3_3_z(R_num, L_num, a1, a3)
    return x, 0, z

plot.plot_vectors(x1_start, x1_end, 0, alpha_to_x, R3_to_x)
```

```python
%aimport plot

R1_1=R1.dot(N.i)
R1_3=R1.dot(N.k)

R1_1_x = lambdify([R, L, alpha1, alpha3], R1_1, "numpy")
R1_3_z = lambdify([R, L, alpha1, alpha3], R1_3, "numpy")

def R1_to_x(a1, a2, a3):
    x=R1_1_x(R_num, L_num, a1, a3)
    z=R1_3_z(R_num, L_num, a1, a3)
    return x, 0, z

plot.plot_vectors(x1_start, x1_end, 0, alpha_to_x, R1_to_x)
```

### Lame params

```python
H1 = 1+alpha3/R
H2=S(1)
H3=S(1)

H=[H1, H2, H3]
DIM=3
dH = zeros(DIM,DIM)
for i in range(DIM):
    dH[i,0]=H[i].diff(alpha1)
    dH[i,1]=H[i].diff(alpha2)
    dH[i,2]=H[i].diff(alpha3)

dH

```


# Deadbeat controller for the double integrator

Consider a mass of 1kg moving in one direction on a friction-free horizontal surface. We can apply a force to the mass (input signal $u$), and the mass is also subject to disturbance forces $v$. We are interested in controlling the position $z$ of the mass. In continuous time the dynamics are described by
$$ \ddot{z} = u + v. $$
Introducing the state variables $x_1=z$ and $x_2=\dot{z}$, the system can also be represented on state-space form with state vector $x = \begin{bmatrix} z & \dot{z}\end{bmatrix}^T$ as
\begin{align}
\dot{x} &= \underbrace{\begin{bmatrix} 0 & 1\\0 & 0\end{bmatrix}}_{A}x + \underbrace{\begin{bmatrix}0\\1\end{bmatrix}}_{B}u + \underbrace{\begin{bmatrix}0\\1\end{bmatrix}}_{B}v\\
y &= \underbrace{\begin{bmatrix}1 & 0 \end{bmatrix}}_C x
\end{align}

## Discrete-time state-space model
The discrete-time state-space model using a sampling period $h$ is
\begin{align}
x(k+1) &= \Phi(h)x(k) + \Gamma(h)u + \Gamma(h)v\\
y(k) &= Cx(k)
\end{align}
where
$$ \Phi(h) = \mathrm{e}^{Ah} = \begin{bmatrix} 1 & h\\0 & 1 \end{bmatrix}$$
and
$$ \Gamma(h) = \int_0^h \mathrm{e}^{As}B ds = \begin{bmatrix} \frac{h^2}{2}\\h \end{bmatrix}.$$
### Verification by symbolic computation

```python
import numpy as np
import sympy as sy
sy.init_printing(use_latex='mathjax', order='lex')

h = sy.symbols('h', real=True, positive=True)
A = sy.Matrix([[0,1], [0,0]])
B = sy.Matrix([[0],[1]])
Phi = sy.simplify(sy.exp(A*h))
Phi
```

$$\left[\begin{matrix}1 & h\\0 & 1\end{matrix}\right]$$

```python
s = sy.symbols('s')
Gamma = sy.integrate(sy.exp(A*s)*B, (s, 0, h))
Gamma
```

$$\left[\begin{matrix}\frac{h^{2}}{2}\\h\end{matrix}\right]$$

## Reachability
The controllability matrix for this second order system becomes
$$ W_c = \begin{bmatrix} \Gamma & \Phi\Gamma \end{bmatrix} = \begin{bmatrix} \frac{h^2}{2} & \frac{3h^2}{2}\\h & h\end{bmatrix}, $$
with determinant
$$\det W_c = \frac{h^3}{2}(1 - 3) = -h^3,$$
which is different from zero since $h>0$.
It is hence possible to reach any point in the state-space from any other point in just two steps (two sampling periods).
### Verification by symbolic computation

```python
Wc = sy.BlockMatrix([[Gamma, Phi*Gamma]]).as_explicit()
Wc
```

$$\left[\begin{matrix}\frac{h^{2}}{2} & \frac{3 h^{2}}{2}\\h & h\end{matrix}\right]$$

```python
sy.det(Wc)
```

$$- h^{3}$$

## Designing an input sequence
We now know that the system is reachable. This means that we can take the system from the origin in the state-space (position zero and velocity zero) to any other point in state-space. And it can be done in only two steps with the input sequence
$$ u(0), \, u(1).$$
Let's say we want to reach the point
$$ x_d = \begin{bmatrix} a\\b \end{bmatrix},$$
which in words is that we want the mass to be at $z=a$ and having the velocity $\dot{z}=b$. The general solution for an n-th order discrete-time state space system is
\begin{align}
x(n) &= \Phi^n x(0) + \Phi^{n-1}\Gamma u(0) + \Phi^{n-2}\Gamma u(1) + \cdots + \Gamma u(n-1)\\
&= \Phi^n x(0) + W_cU,
\end{align}
where
$$ U = \begin{bmatrix} u(n-1)\\u(n-2)\\\vdots\\u(0)\end{bmatrix}. $$
In the case here we have $x(0)=0$ and this leads to the equation
$$ W_cU = x_d, \qquad \text{with solution}$$
\begin{align}
U &= \begin{bmatrix}u(1)\\u(0)\end{bmatrix} = W_c^{-1}x_d = \begin{bmatrix} \frac{h^2}{2} & \frac{3h^2}{2}\\h & h\end{bmatrix}^{-1} \begin{bmatrix} a\\b \end{bmatrix}\\
&= \frac{1}{-h^3} \begin{bmatrix} h & -\frac{3h^2}{2}\\-h & \frac{h^2}{2} \end{bmatrix} \begin{bmatrix} a\\b \end{bmatrix}\\
&= \begin{bmatrix} -\frac{1}{h^2} & \frac{3}{2h}\\\frac{1}{h^2} & -\frac{1}{2h} \end{bmatrix} \begin{bmatrix} a\\b \end{bmatrix}\\
&= \begin{bmatrix} -\frac{a}{h^2} + \frac{3b}{2h}\\ \frac{a}{h^2} - \frac{b}{2h} \end{bmatrix}.
\end{align}
Thus the input sequence becomes $u(0) = \frac{a}{h^2} - \frac{b}{2h}$, $u(1) = \frac{-a}{h^2} + \frac{3b}{2h}$.
### Verification with symbolic computation

```python
# Verify
a,b = sy.symbols('a,b')
U = Wc.inv()*sy.Matrix([[a],[b]])
U
```

$$\left[\begin{matrix}- \frac{a}{h^{2}} + \frac{3 b}{2 h}\\\frac{a}{h^{2}} - \frac{b}{2 h}\end{matrix}\right]$$

```python
# Simulate
u0 = U[1,0]
u1 = U[0,0]
x0 = sy.Matrix([[0],[0]])
x1 = Phi*x0 + Gamma*u0
x2 = Phi*x1 + Gamma*u1
sy.simplify(x2)
```

$$\left[\begin{matrix}a\\b\end{matrix}\right]$$

## State feedback
Introducing the state-feedback control law
$$ u = -l_1x_1 - l_2 x_2 + l_0y_{ref} = -Lx + l_0y_{ref}$$
gives the closed-loop state-space system
\begin{align}
x(k+1) &= \Phi x(k) +\Gamma\big(-Lx(k) + l_0y_{ref}(k)\big) + \Gamma v(k) = \left( \Phi - \Gamma L \right) x(k) + l_0\Gamma y_{ref}(k) + \Gamma v(k)\\
y(k) &= C x(k)
\end{align}
with characteristic polynomial given by
\begin{align}
\det \left( zI - (\Phi-\Gamma L) \right) &= \det \left( \begin{bmatrix} z & 0\\0 & z \end{bmatrix} - \begin{bmatrix} 1 & h\\0 & 1 \end{bmatrix} + \begin{bmatrix} l_1\frac{h^2}{2} & l_2\frac{h^2}{2}\\ l_1h & l_2h \end{bmatrix} \right)\\
&= \det \begin{bmatrix} z-1+l_1\frac{h^2}{2} & -h+l_2\frac{h^2}{2}\\l_1h & z-1+l_2h
\end{bmatrix}\\
&= (z-1+l_1\frac{h^2}{2})(z-1+l_2h) - l_1h(-h + l_2\frac{h^2}{2})\\
&= z^2 + (-1+l_2h-1+l_1\frac{h^2}{2}) z + (1-l_2h - l_1\frac{h^2}{2} + l_1l_2\frac{h^3}{2} +l_1h^2 -l_1l_2\frac{h^3}{2})\\
&= z^2 + (l_1\frac{h^2}{2}+l_2h-2) z + (1 +l_1\frac{h^2}{2} -l_2h)
\end{align}
### Verification by symbolic computation

```python
l1, l2 = sy.symbols('l1, l2', real=True)
z = sy.symbols('z')
L = sy.Matrix([[l1, l2]])
ch_poly = sy.Poly((z*sy.eye(2) - (Phi - Gamma*L)).det(), z)
ch_poly.as_expr()
```

$$\frac{h^{2} l_{1}}{2} - h l_{2} + z^{2} + z \left(\frac{h^{2} l_{1}}{2} + h l_{2} - 2\right) + 1$$

### Desired closed-loop characteristic polynomial
Here we are interested in designing a deadbeat controller, so the desired closed-loop poles are
$$ p_1 = 0, \qquad p_2=0,$$
and the desired characteristic polynomial is
$$ A_c(z) = (z-p_1)(z-p_2) = z^2. $$
In the same spirit as when designing an RST controller using the polynomial approach, we set the calculated characteristic polynomial - obtained when introducing the linear state feedback- equal to the desired characteristic polynomial.
\begin{align}
z^1: \qquad l_1\frac{h^2}{2} + l_2h -2 &= 0\\
z^0: \qquad l_1\frac{h^2}{2} - l_2h+1 &= 0
\end{align}
which can be written as the system of equations
$$ \underbrace{\begin{bmatrix} \frac{h^2}{2} & h\\\frac{h^2}{2} & -h \end{bmatrix}}_{M} \underbrace{\begin{bmatrix} l_1\\l_2\end{bmatrix}}_{L^T} = \underbrace{\begin{bmatrix}2\\-1\end{bmatrix}}_{b} $$
with solution given by

$$L^T = M^{-1}b = \frac{1}{-h^3} \begin{bmatrix} -h & -h\\-\frac{h^2}{2} & \frac{h^2}{2} \end{bmatrix} \begin{bmatrix} 2\\-1 \end{bmatrix}$$

$$ = -\frac{1}{h^3} \begin{bmatrix} -2h+h\\-h^2-\frac{h^2}{2}\end{bmatrix} = \begin{bmatrix} \frac{1}{h^2}\\\frac{3}{2h} \end{bmatrix} $$
### Verification by symbolic calculation

```python
des_ch_poly = sy.Poly(z*z, z)
dioph_eqn = ch_poly - des_ch_poly
sol = sy.solve(dioph_eqn.coeffs(), (l1,l2))
sol
```

$$\left \{ l_{1} : \frac{1}{h^{2}}, \quad l_{2} : \frac{3}{2 h}\right \}$$

In the system of equations $ML^T=b$ above, note that the matrix $M$ can be written
$$ M = \begin{bmatrix} \frac{h^2}{2} & h\\\frac{h^2}{2} & -h \end{bmatrix} = \begin{bmatrix}1 & 0\\-2 & 1\end{bmatrix}\underbrace{\begin{bmatrix} \frac{h^2}{2} & h \\ \frac{3h^2}{2} & h\end{bmatrix}}_{W_c^T}, $$
so $M$ will be invertible if and only if $\det W_c^T = \det W_c \neq 0$.

## The resulting closed-loop system
So, we have found the control law
$$ u(k) = -Lx(k) + l_0y_{ref}(k) = -\begin{bmatrix} \frac{1}{h^2} & \frac{3}{2h} \end{bmatrix}x(k) + l_0 y_{ref}(k)$$
which gives a closed-loop system with poles in the origin, i.e. deadbeat control. The closed-loop system becomes
\begin{align*}
 x(k+1) &= \big( \Phi - \Gamma L \big) x(k) + \Gamma l_0 y_{ref}(k) + \Gamma v(k)\\
 &= \left( \begin{bmatrix} 1 & h\\0 & 1\end{bmatrix} - \begin{bmatrix} \frac{h^2}{2}\\h\end{bmatrix}\begin{bmatrix} \frac{1}{h^2} & \frac{3}{2h} \end{bmatrix}  \right) x(k) + \Gamma l_0 y_{ref}(k) + \Gamma v(k)\\
 &= \left( \begin{bmatrix} 1 & h\\0 & 1\end{bmatrix} - \begin{bmatrix} \frac{1}{2} & \frac{3h}{4}\\ \frac{1}{h} & \frac{3}{2}\end{bmatrix}\right) x(k) + \Gamma l_0 y_{ref}(k) + \Gamma v(k)\\
 &= \underbrace{\begin{bmatrix} \frac{1}{2} & \frac{h}{4} \\-\frac{1}{h} & -\frac{1}{2}\end{bmatrix}}_{\Phi_c}x(k) + \begin{bmatrix}\frac{h^2}{2}\\h\end{bmatrix} l_0 y_{ref}(k) + \begin{bmatrix}\frac{h^2}{2}\\h\end{bmatrix}  v(k)\\
 y(k) &= \begin{bmatrix} 1 & 0 \end{bmatrix} x(k)
 \end{align*}
 ### Verification using symbolic computations

```python
L = sy.Matrix([[sol[l1], sol[l2]]])
Phic = Phi - Gamma*L
Phic
```

$$\left[\begin{matrix}\frac{1}{2} & \frac{h}{4}\\- \frac{1}{h} & - \frac{1}{2}\end{matrix}\right]$$

## Determining the reference signal gain $l_0$
Consider the steady-state solution for a unit step in the reference signal. We set $y_{ref}=1$ and $v = 0$. This gives
$$ x(k+1) = \Phi_c x(k) + \Gamma l_0. $$
In steady-state there is no change in the state, so $x(k+1)=x(k)=x_{ss}$, which leads to
$$ x_{ss} = \Phi_c x_{ss} + \Gamma l_0$$
$$ (I - \Phi_c)x_{ss} = \Gamma l_0$$
\begin{align}
x_{ss} &= (I - \Phi_c)^{-1}\Gamma l_0\\
&= \begin{bmatrix} \frac{1}{2} &-\frac{h}{4}\\ \frac{1}{h} & \frac{3}{2} \end{bmatrix}^{-1} \begin{bmatrix} \frac{h^2}{2}\\h \end{bmatrix} l_0\\
&= \begin{bmatrix}\frac{3}{2} & \frac{h}{4}\\-\frac{1}{h} & \frac{1}{2} \end{bmatrix} \begin{bmatrix} \frac{h^2}{2}\\h\end{bmatrix} l_0\\
&= \begin{bmatrix}\frac{3h^2}{4} + \frac{h^2}{4}\\-\frac{h}{2} + \frac{h}{2} \end{bmatrix}l_0= \begin{bmatrix}h^2\\ 0 \end{bmatrix}l_0\\
\end{align}
which means that the steady-state velocity $\dot{z}(\infty) = x_2(\infty) = 0$. This makes sense.

We can now determine $l_0$. Since  $y(k)=x_1(k)$ then $y_{ss} = h^2 l_0$ for a unit step in the reference signal. We would like the steady-state value $y_{ss}$ to be the same as the reference signal (which is equal to one, of course) so this gives
$$ h^2l_0 = 1 \quad \Rightarrow \quad l_0 = \frac{1}{h^2}. $$

##  Simulate step responses (symbolically)
### Step response from the reference

```python
l0 = 1/(h*h)
C = sy.Matrix([[1,0]])
x = sy.Matrix([[0],[0]]) # Initial state
yref = sy.Matrix([[1]])
xs = [x] # List to hold state trajectory
us = [[0]] # and control signal
ys = [[0]] # and system output
for k in range(6): # No need to simulate too long. It is deadbeat control after all
    us.append(-L*x + l0*yref)
    x = Phic*x + Gamma*l0*yref
    xs.append(x)
    ys.append(C*x)
xs
```

$$\left [ \left[\begin{matrix}0\\0\end{matrix}\right], \quad \left[\begin{matrix}\frac{1}{2}\\\frac{1}{h}\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right]\right ]$$

```python
us
```

$$\left [ \left [ 0\right ], \quad \left[\begin{matrix}\frac{1}{h^{2}}\end{matrix}\right], \quad \left[\begin{matrix}- \frac{1}{h^{2}}\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right]\right ]$$

### Step response from the disturbance

```python
x = sy.Matrix([[0],[0]]) # Initial state
yref = sy.Matrix([[0]])
v = sy.Matrix([[1]])
xs = [x] # List to hold state trajectory
us = [[0]] # and control signal
ys = [[0]] # and system output
for k in range(6): # No need to simulate too long. It is deadbeat control after all
    us.append(-L*x + l0*yref)
    x = Phic*x + Gamma*l0*yref + Gamma*v
    xs.append(x)
    ys.append(C*x)
xs
```

$$\left [ \left[\begin{matrix}0\\0\end{matrix}\right], \quad \left[\begin{matrix}\frac{h^{2}}{2}\\h\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right]\right ]$$

## Simulate step-responses (numerically)

```python
import control as ctrl
import matplotlib.pyplot as plt
# Convert to from sympy matrices to numpy
hval = .1
Phi_np = np.array(Phi.subs({h:hval})).astype(np.float64)
Gamma_np = np.array(Gamma.subs({h:hval})).astype(np.float64)
L_np = np.array(L.subs({h:hval})).astype(np.float64)
l0_np = np.array(l0.subs({h:hval})).astype(np.float64)
Phic_np = Phi_np - Gamma_np*L_np
C_np = np.array(C).astype(np.float64)
D_np = np.array([[0]])
sys_c = ctrl.ss(Phic_np, Gamma_np*l0_np, C_np, D_np, hval) # From ref signal
sys_cv = ctrl.ss(Phic_np, Gamma_np, C_np, D_np, hval) # From disturbance signal

```

```python
tvec = np.asarray(np.arange(8))*hval
T, yout = ctrl.step_response(sys_c, tvec)
T, yout_v = ctrl.step_response(sys_cv, tvec)
```

```python
plt.figure(figsize=(14,6))
plt.step(tvec, yout.flatten())
plt.step(tvec, yout_v.flatten())
```

```python

```

state-space/.ipynb_checkpoints/Deadbeat controller for the double integrator-checkpoint.ipynb


# Matrixrekening

## Week 1

#### Matrix definitie

Een matrix wordt aangeduidt met:

$A = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \qquad B = \begin{pmatrix} -3 & \frac{2}{5} & \frac{1}{2} \\ 0 & -10 & 2 \end{pmatrix} \qquad C = \begin{pmatrix} 2 \\ -1 \\ 8 \end{pmatrix} \qquad D = \begin{pmatrix}1 & 3 & 4 \end{pmatrix}$

De afmeting van de matrix zijn als volgt:

* $A = 2 \times 2$
* $B = 2 \times 3$
* $C = 1 \times 3$

Hier tellen we eerst de _rijen_ en daarna de _kolommen_. Als het aantal rijen en kolommen gelijk is dan spreken we van een _vierkante_ matrix. Als een matrix $1$ rij (of kolom) heeft zoals matrix $C$ dan spreken we van een _vector_. $C$ is een _kolomvector_. Indien er maar $1$ rij zoals in $D$ is dan spreken we van een _rijvector_.

De verschillende _elementen_ in de matrix worden op de volgende manier benoemd:

$A = \begin{pmatrix} a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & \ddots && \vdots \\ \vdots && \ddots & \vdots \\ a_{m1} & a_{m2} & \ldots & a_{mn} \end{pmatrix}$

Met deze methode kunnen we de elementen in de onderstaande $3 \times 2$ matrix op de volgende manier noteren:

$L=\begin{pmatrix} 7 & 1 \\ 8 & 0 \\ 3 & -4 \end{pmatrix}$

* $l_{11} = 7$
* $l_{12} = 1$
* $l_{21} = 8$
* $l_{22} = 0$
* $l_{31} = 3$
* $l_{32} = -4$

#### Vierkante matrices

Zoals al eerder is genoemd, is een $n \times n$ matrix een _vierkante_ matrix. De hoofddiagonaal in een vierkante matrix is hieronder aangegeven:

$\begin{pmatrix} \color{red}{1} & 2 & 3 \\ 4 & \color{red}{5} & 6 \\ 7 & 8 & \color{red}{9} \end{pmatrix}$

De hoofddiagonaal bestaat uit de elementen $a_{11}, a_{22}, a_{33}, \ldots, a_{nn}$.

#### Nul-matrix

Een matrix waarvan alle elementen de waarde $0$ bevatten wordt de _nulmatrix_ genoemd. De afmetingen maken hierbij niet uit.

$\begin{pmatrix}0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0\end{pmatrix}$

#### Symmetrische matrix

Als een matrix symmetrisch is over de hoofddiagonaal dan wordt er gesproken van een _symmetrische matrix_.

$\begin{pmatrix}1 & 3 & \frac{2}{3} \\ 3 & 5 & 9 \\ \frac{2}{3} & 9 & 9 \end{pmatrix}$

Hiervoor geldt dat $a_{ij} = a_{ji}$. En vinden we de eigenschap $a_{ij} = a_{ji} \forall i,j$.

#### Transponeren

Als we de matrix $A$ _transponeren_ krijgen we een getransponeerde matrix, aangeduidt met $A^T$:

$A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix} \qquad A^T = \begin{pmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{pmatrix}$.

Hierbij geldt dat $m \times n \rightarrow n \times m$ en $a_{ij} \rightarrow a_{ji}$.

#### ($+$, $-$)

Om matrices bij elkaar op te tellen **moeten** deze dezelfde afmetingen hebben, indien dit niet zo is dan is deze _niet gedefinieerd_.

$A=\begin{pmatrix} 1 & 3 & 7 \\ 2 & 8 & 6 \end{pmatrix} \qquad B=\begin{pmatrix} 4 & 3 & 1 \\ 9 & 2 & 5 \end{pmatrix} \qquad A+B=\begin{pmatrix} 5 & 6 & 8 \\ 11 & 10 & 11 \end{pmatrix}$

#### ($\times$, $\div$)

Een matrix kunnen we vermenigvuldigen met een getal. Dit wordt een _scalaire_ matrix vermenigvuldigen genoemd. Het getal $2$ is een _scalar_.

$A = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \qquad 2 \cdot A = \begin{pmatrix} 2 & 4 \\ 6 & 8 \end{pmatrix} $

Om matrices met elkaar te vermenigvuldigen kunnen we dit doen a.d.h.v. de volgende definitie:

Als $A$ een $m \times n$ matrix en $B$ een $n \times q$ matrix is, dan is het product $C=A\cdot B$ de $m\times q$ matrix waarvan de elementen zijn

$\begin{align}c_{ij} = a_{i1}b_{1j} + a_{i2}b_{2j} + \ldots + a_{in}b_{nj} = \sum\limits_{k=1}^n a_{ik}b_{kj}\end{align}$.

Een voorbeeld hiervan is:

$\begin{pmatrix} 2 & 4 & 6 \\ 3 & 2 & 1 \end{pmatrix} \cdot \begin{pmatrix} 2 \\ 4 \\ 6 \end{pmatrix} = \begin{pmatrix} 2\cdot2+4\cdot4+6\cdot6 \\ 3\cdot2+2\cdot4+1\cdot6 \end{pmatrix} = \begin{pmatrix} 56 \\ 20\end{pmatrix}$

Van een $2\times3$ matrix kan alleen een product worden genomen met een $3\times n$ matrix en het resultaat heeft de afmeting $2\times n$.

Dit kan uiteraard ook met andere operaties worden gecombineerd zoals bijvoorbeeld:

$A^T \cdot A = \begin{pmatrix} 2 & 7 & 8 \end{pmatrix} \cdot \begin{pmatrix} 2 \\ 7 \\ 8 \end{pmatrix} = 127$

In een meer gegeneraliseerde vorm kunnen we beter zien dat het benaderen van elementen volgt dat $a_{\text{rij},\text{kolom}}$, ofterwel $a_{y,x}$:

$A = \begin{pmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{pmatrix} \qquad B = \begin{pmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \\ b_{31} & b_{32} & b_{33} \end{pmatrix}$

En bij een matrixvermenigvuldiging wordt het volgende algoritme gevolgd:

1. Voor elke kolom $j$ in $B$.
2. Voor elke rij $i$ in $A$.
3. Voor elk element $k$ in rij $i$.
4. $C_{ij} = k_1 \cdot B_{ij} + k_2 \cdot B_{ij} + \ldots + k_n \cdot B_{ij}$.

Hieruit volgt dat:

$A \cdot B = \begin{pmatrix} a_{11}b_{11} + a_{12}b_{21} + a_{13}b_{31} & a_{11}b_{12} + a_{12}b_{22} + a_{13}b_{32} & a_{11}b_{13} + a_{12}b_{23} + a_{13}b_{33} \\ a_{21}b_{11} + a_{22}b_{21} + a_{23}b_{31} & a_{11}b_{12} + a_{12}b_{22} + a_{13}b_{32} & a_{11}b_{13} + a_{12}b_{23} + a_{13}b_{33}\\ a_{31}b_{11} + a_{32}b_{21} + a_{33}b_{31} & a_{11}b_{12} + a_{12}b_{22} + a_{13}b_{32} & a_{11}b_{13} + a_{12}b_{23} + a_{13}b_{33}  \end{pmatrix}$

#### Eenheidsmatrix (identity)

Een eenheidsmatrix is altijd een vierkante matrix waarvan alle elementen $0$ zijn behalve de elementen van de hoofddiagonaal, deze zijn $1$.

$E = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}$

De eeheidsmatrix wordt aangeduidt met $A^E$ (eenheid) of $A^I$ (identity).

Wanneer er een matrixvermenigvuldiging wordt gedaan met de eenheidsmatrix dan komt er altijd de oorspronkelijk matrix uit. De volgorde waarin dit gebeurd maakt niet uit $A\cdot E = E \cdot A = A$.

#### Machtsverheffen

Machtsverheffen kan alleen met een vierkante matrix en hierbij gelden de regels $n \geq 0$, $n \in \mathbb{N}$. Ga na waarom je niet kunt machtsverheffen op een matrix die niet vierkant is.

* $A^0 = E$ (identiteitsmatrix)
* $A^1 = A$
* $A^2 = A\cdot A$
* $A^n = A \cdot A \cdot \ldots$

Een macht van een matrix kan geen gebroken exponenten hebben.

#### Commutativiteit

Bij matrixvermenigvuldigen geldt de commutatieve eigenschap $a \cdot b = b \cdot a$ niet!

#### Associativeit en distributiviteit

Bij matrixvermenigvuldigen gelden de associatieve $a \cdot b \cdot c = (a \cdot b) \cdot c = a \cdot (b \cdot c)$ en distributieve $a \cdot (b + c) = a\cdot b + a \cdot c$ eigenschappen wel.

#### Nulvector

Een vector in bijvoorbeeld $\mathbb{R}^3$ waarvan alle elementen $0$ zijn:

$\begin{pmatrix} 0 \\ 0 \\ 0\end{pmatrix}$

Deze vector geeft de oorsprong van het coordinatensysteem aan.

#### Eenheidsvectoren

Dit zijn de vectoren die de basis vormen van het coordinatensysteem waarin je werkt, voor $\mathbb{R}^2$ zijn dit de vectoren:

$\begin{pmatrix} 1 \\ 0\end{pmatrix} \qquad \begin{pmatrix} 0 \\ 1\end{pmatrix}$

#### Lineaire combinatie van eenheidsvectoren

Elke vector in $\mathbb{R}^2$ kan worden geschreven als een _lineaire combinatie_ van eenheidsvectoren, bijvoorbeeld:

$\begin{pmatrix} 3 \\ 4\end{pmatrix} = 3\cdot \begin{pmatrix} 1 \\ 0 \end{pmatrix} + 4 \cdot \begin{pmatrix} 0 \\ 1\end{pmatrix}$

En voor $\mathbb{R}^3$:

$\begin{pmatrix} 2 \\ -1 \\ 6 \end{pmatrix} = 2 \cdot \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} - \begin{pmatrix} 0 \\ 1 \\ 0\end{pmatrix} + 6 \cdot \begin{pmatrix} 0 \\ 0 \\ 1\end{pmatrix}$

## Week 2

#### 1e-graads vergelijking

Dit wordt ook wel een _lineaire vergelijking_ genoemd en dit is in de vorm van $5x+7y=11$. Een voorbeeld een vergelijking die niet lineair is zijn $5x^2 + y = 8$.

#### Equivalent $(\iff)$

Als stelsels equivalent (gelijkwaardig) wil dat zeggen dat ze dezelfde oplossing(en) hebben.

Twee stelsels zijn equivalent als:

* beide precies $1$, en dezelfde, oplossing hebben
* beide geen oplossing hebben
* beide oneindig veel, en dezelfde, oplossingen hebben

#### Standaard vorm

In alle vergelijkingen staan de variabelen aan de linker kant (op volgorde $x,y,z$) en alle constanten staan aan de rechterkant van het $=$ teken.

#### Elementaire bewerkingen (elementairy row operations)

Op lineaire stelsels kunnen de volgende drie elementaire bewerkingen worden uitgevoerd:

1. Een vergelijking met een getal vermenigvuldigen ($x\not=0$). (Scale)
2. Een vergelijking bij een andere vergelijking optellen of aftrekken. (Pivot)
3. De volgorde van de vergelijking verwisselen. (Swap)

Omdat dit al snel wat onoverzichtelijk wordt kan het handig zijn om met de volgende notatie aan te geven welke stappen er zijn uitgevoerd:

* $R_1 - 3R_2 \rightarrow R_1$, hiermee zeggen we dat we $3$ maar rij $2$ van rij $1$ aftrekken en deze plaatsen op rij $1$.
* $R_1 \iff R_2$, hiermee zeggen we dat wij de rijen $1$ en $2$ met elkaar verwisselen.

Met deze elementaire bewerkingen kun je stelsels van lineaire vergelijkingen oplossen.

#### Vegen

Vegen betekend achtereen volgend elimineren.

#### Pivot

De pivot is de variabele waarmee geveegd wordt. Als notaties digitaal zijn dan kan het helpen om de pivot dikgedrukt te maken.

#### Stelsels vergelijkingen als een matrixvergelijking

In plaats van $x$ schrijven we $x_1$, $y$ wordt $x_2$, en $z$ wordt $x_3$.

Het stelsel:

$\begin{cases}\begin{align}x_1 & -x_2 & +2x_3 &= 16 \\ -2x_1 & +4x_2 & +x_3 &= 5 \\ 3x_1 & -5x_2 & +3x_3 &= 25\end{align}\end{cases}$

kunnen we schrijven als $A \cdot X = B$, waarin:

$A=\begin{pmatrix}1 & -2 & 2 \\ -2 & 4 & 1 \\ 3 & -5 & 3\end{pmatrix}, X = \begin{pmatrix}x_1\\x_2\\x_3\end{pmatrix}, B=\begin{pmatrix}16\\5\\25\end{pmatrix}$

De matrix $A$ bevat de coefficienten van $x_1, x_2, x_3$ en heet daarom de _coefficient matrix_.

We plakken daarvoor $B$ als een kolom aan $A$, gescheiden door een vertical lijn.

Dit is de _aangevulde coefficient matrix_ (augmented matrix). Hiermee kunnen we de uitwerking van het stelsel verkorten. De pivot elementen zijn dikgedrukt.

#### Eenduidig oplosbaar

Stelsel heeft precies $1$ oplossing.

#### Strijdig stelsel

Als ergens bijvoorbeeld de vergelijking $0=1$ of $4=2$ (ongelijkheid) optreedt.

#### Afhankelijk stelsel

Heeft oneindig veel oplossingen.

#### Algemene oplossing

Dit is de verzameling van alle oplossingen. Elke afzonderlijke oplossing is een _particuliere oplossing_.

#### Oplossingsmogelijkheden

* Eenduidig oplosbaar stelsel heeft $1$ oplossing.
* Strijdig stelsel heeft geen oplossing.
* Afhandkelijk stelsen heeft oneindig veel oplossingen.

Voor het aantal oplossingen geldt dat ($\text{#vgl}=$ aantal vergelijkingen, $\text{#var}=$ aantal variabelen):

1. $\text{#vgl}=\text{#var} \rightarrow 0$ oplossingen in speciale gevallen oneindig veel.
2. $\text{#vgl}>\text{#var} \rightarrow$ in het algemeen geen oplossing (in speciale gevallen $1$ of oneindig).
3. $\text{#vgl}<\text{#var} \rightarrow$ oneindig veel oplossingen (in speciale gevallen onoplosbaar).

#### Overige

In het dictaar wordt er gebruik gemaakt van de letters $t,u,v$. Echter in andere boeken wordt er gebruik gemaakt van $\alpha, \beta, \gamma, \lambda, \mu$ (alpha, beta, gamma, lambda, mu).

## Week 3

#### (aangevulde) coefficient matrix

Een coefficient matrix is in de vorm van $A \times \vec{x} = \vec{b}$.

$A=\begin{pmatrix}1 & -1 & 2 \\ -2 & 4 & 1 \\ 3 & 5 & 3\end{pmatrix}$

Hierbij is de aangevulde coefficient matrix in de vorm van $\begin{pmatrix}A|\vec{b}\end{pmatrix}$.

#### Bovendriehoeksmatrix

Een bovendriehoeksmatrix (upper-triangular form) heeft alle elementen linksonder de hoofddiagonaal op $0$.

#### Strijdig/afhankelijk

Een _strijdig_ stelsel heeft geen oplossing. Een _afhankelijk_ stelsel heeft oneindig veel oplossingen.

#### Eenduidig

Een _eenduidig_ stelsel heeft $1$ oplossing.

#### Equivalent

Een stelsel is _equivalent_ als ze beide dezelfde algemene oplossing hebben.

* Geen oplossing
* 1 en dezelfde oplossing
* oneindig veel, en dezelfde, oplossingen

#### Algemene oplossing(en)

Dit zijn alle oplossing(en) van een stelsel. Hierbij wordt een enkele oplossing een _particuliere_ oplossing genoemd.

#### Algemene oplossing (bij afhankelijke stelsels)

Om aan te geven wat alle oplossingen zijn van een afhankelijk stelsel kunnen we de volgende notatie gebruiken. (Niet vergeten als je $t$ aangeeft in het stelsel dat je erbij vermeldt dat deze een willekeurige waarde kan bevatten.)

$\begin{pmatrix}x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} \frac{1}{4}-\frac{9}{4}t \\ \frac{5}{8}+\frac{7}{8}t \\ t \end{pmatrix} = \begin{pmatrix}\frac{1}{4}\\\frac{5}{8}\\0\end{pmatrix}+t\begin{pmatrix}\frac{1}{4}\\\frac{7}{8}\\1\end{pmatrix}$

Nu kunnen we op een gemakkelijke manier $t$ vervangen met een getal en uitrekenen welke oplossing er is.

## Week 4

### Determinanten

**Definitie**

De _determinant_ is een hulpmiddel om vast te stellen of een stelsel van lineaire vergelijking precies 1, geen, of oneindig veel oplossingen heeft.

Voor elke _vierkante matrix_ ($N \times N$) kunnen we $\det(A)$ berekenen.

**$2 \times 2$ matrix**

In het geval van een $2 \times 2$ matrix, $A=\begin{pmatrix}a&b\\c&d\end{pmatrix}$, dan is $\det(A) = ad-bc$.

**Voorbeeld:**

Om de determinant van $A=\begin{pmatrix}1&5\\2&7\end{pmatrix}$ te berekenen krijgen we $1 \cdot 7 - 5 \cdot 2 = -3$.

**$3 \times 3$ matrix**

Om de determinant van een $3 \times 3$ matrix te berekenen kunnen we de onderdeterminanten berekenen door af te wikkelen via een rij of kolom.

Voor matrix $A = \begin{pmatrix}a_1&b_1&c_1\\a_2&b_2&c_2\\a_3&b_3&c_3 \end{pmatrix}$ wordt dit op de volgende manier berekend:

$\det(A) = a_1(b_2c_3-c_2b_3) - a_2(b_1c_3 - b_3c_1) + a_3(b_1c_2-b_2c_1)$

**Determinant $N\times N$ matrix**

Bij het bereken van de determinant wordt het onderstaande patroon gevolgd:

$\begin{pmatrix}+&-&+&-&\cdots\\-&+&-&+&\cdots\\+&-&+&-&\cdots\\-&+&-&+&\ldots\\\cdots& \vdots &\vdots&\vdots&\ddots\end{pmatrix}$

**Eigenschappen van determinanten en veegprocedure**

1. Bij het verwisselen van twee rijen of kolommen in een determinant veranderd deze van teken.
2. Als je een determinant spiegeld over de hoofddiagonaal blijft deze gelijk.
3. Als je alle getallen in een rij (of kolom) met een willekeurig getal $k$ vermenigvuldigd, dan wordt de waarde van de determinant met $k$ vermenigvuldigd.
4. Als je in een determinant $k$ maal een rij (of kolom) optelt of aftrekt van een andere rij (of kolom), dan blijft de determinant gelijk.

Het is mogelijk om de veegprocedure op determinant toe te passen om in een rij/kolom meer nullen te krijgen. Hiervoor worden de bovenstaande regels gebruikt. Dit is handig omdat het afwikkelen van de rijen/kolommen hierdoor aanzienlijk wordt vereenvoudigd.

## Week 5

### Inverses

Voor een matrix kan een inverse worden bepaald als:

1. De matrix is vierkant ($N \times N$)
2. $\det(A) \not=0$

Een belangrijke eigenschap van de inverse is dat zodra deze met de oorspronkelijke matrix wordt vermenigvuldigd we de identiteitsmatrix krijgen. Ofterwel $A \times A^{-1} = E = A^{-1} \times A = E$. Deze operatie **wel** commutatief, in tegenstelling tot matrixvermenigvuldigen wat niet commutatief is.

Als een matrix een inverse heeft dan is de matrix _inverteerbaar (regulier)_. Als een matrix geen inverse heeft dan is de matrix _singulier_.

**Voorbeeld**

Stel we hebben $A=\begin{pmatrix}1&2\\1&3\end{pmatrix}$. Om de inverse te vinden zoeken we naar $A \times A^{-1} = \begin{pmatrix}1&0\\0&1\end{pmatrix}$.

Ofterwel, $\begin{pmatrix}1&2\\1&3\end{pmatrix} \cdot \begin{pmatrix}a&b\\c&d\end{pmatrix} = \begin{pmatrix}1&0\\0&1\end{pmatrix}$

Dit is gemakkelijk op te lossen met vegen. Hiervoor gebruiken we een aangevulde coefficientmatrix met links $A$ en rechts de identiteitsmatrix $E$. Vervolgens vegen we zodat de coefficientmatrix alleen maar $1$ heeft op de hoofddiagonaal. De oorspronkelijke identiteitsmatrix (rechterkant) is nu $A^{-1}$. Dit is ook weer te controleren door $A \times A^{-1} = E$.

**Eigenschappen**

* Een matrix heeft een inverse als $\det(A)\not=0$.
* Elke inverse is uniek.
* Als $A$ inverteerbaar is dan is $A^{-1}$ ook inverteerbaar.

**Oplossen stelsel**

Stel we hebben een vergelijking $Ax=b$, dan geldt $x = A^{-1}b$.

Bewijs:

$\begin{align}
Ax &= b \\
A^{-1}(Ax)&=A^{-1}b \quad &(\text{Add} \quad A^{-1}) \\
(A^{-1}A)x&=A^{-1}b \quad &(\text{By associativity}) \\
Ex &= A^{-1}b \quad &( A^{-1}A=E )\\
x &= A^{-1}b \quad &(Ex=x)
\end{align}$

```python

```


# Principal Component Analysis

## 1. The goal - dimensionality reduction

Principal Component Analysis (sometimes called Proper Orthogonal Decomposition) is a technique for reducing the dimensions of data.

If we think of our data as points in a coordinate system (easy with 2-D data, but harder to visualise in n-D), what PCA does is project the data onto a new set of axes, where the basis vectors for the new axes are a **linear combination** of the original coordinate system basis vectors.

The key to PCA is that the new basis vectors are chosen in a defined way: the basis vector of the first principle component is aligned so as to capture the maximum possible variation in the data; the second principle component captures the second largest variation in the data; and so on.

Above is a cartoon illustration. The data points have a large variation in both the x and y basis vectors; note, also, that 2 data points that differ in x will also differ in y - these changes are correlated. The principal components (right picture) are chosen so that the maximum variation in the data is obtained along the PC1 basis vector; also, there is no correlation between a change in PC2 and in PC1.

## 2. Link to the Singular Value Decomposition
Principal Component Analyis and the Singular Value Decomposition are closely related. The easiest way to illustrate this is by assuming the answer: for mean-centred data (each dimension has zero mean), the basis vectors required for PCA are the columns of $\mathbf{U}$ in the SVD.

Our data is stored in matrix $\mathbf{X}$,

\begin{equation}
\mathbf{X} = \begin{bmatrix}
\mid & \mid &  & \mid  \\
\mathbf{x_1} & \mathbf{x_2} & \cdots  & \mathbf{x_m} \\
\mid & \mid &  & \mid \end{bmatrix}\quad\text{.}
\end{equation}

Each column $\mathbf{x_k}$ is a new data point, and each row is a different dimension. For the 2-D example above, $\mathbf{X}$ would have 2 rows and as many columns as we have points.

We perform the SVD on $\mathbf{X}$, so that,

\begin{equation}
\mathbf{X} = \mathbf{U}\,\mathbf{\Sigma}\,\mathbf{V^T}\quad\text{.}
\end{equation}

Now, we recall that $\mathbf{U}$ (and also $\mathbf{V}$) is orthonormal: each column is a vector of unit magnitude, and all the column vectors are orthogonal. If we want to project our original data $\mathbf{X}$ on to the basis vectors formed by the columns of $\mathbf{U}$ (to obtain a new view of our data which we will call $\mathbf{Y}$) then all we have to do is,

\begin{equation}
\mathbf{Y}=\mathbf{U^T}\mathbf{X}\quad\text{.}
\end{equation}

To make further progress, we introduce *variance* and *covariance*. If we have two samples of data, each with zero mean, stored in vector $\mathbf{a}$ and a vector $\mathbf{b}$, then the variances are:

\begin{equation}
\sigma^2_\mathbf{a} = \frac{1}{n-1}\mathbf{a}\mathbf{a^T} \\
\sigma^2_\mathbf{b} = \frac{1}{n-1}\mathbf{b}\mathbf{b^T}
\end{equation}

(the n-1 is because we treat the vectors as a sample of data, and we seek an 'unbiased estimator').

The covariance of $\mathbf{a}$ and $\mathbf{b}$ is

\begin{equation}
\sigma^2_\mathbf{ab} = \frac{1}{n-1}\mathbf{a}\mathbf{b^T}\quad\text{.}
\end{equation}

If the data in $\mathbf{a}$ and $\mathbf{b}$ are correlated, the covariance will be high. If $\mathbf{a}$ and $\mathbf{b}$ are not correlated, the covariance will be low.

We can obtain the covariance of our transformed data matrix $\mathbf{Y}$ as follows,

\begin{equation}
\mathbf{C_Y}=\frac{1}{n-1}\mathbf{Y}\mathbf{Y^T}\quad\text{.}
\end{equation}

What do we want the covariance matrix, $\mathbf{C_Y}$, to look like? The entries on the diagonal will be the variances in each of our (transformed) measurement dimensions. If we have chosen these dimensions correctly, i.e. we are using the principal components of the data, then all of the off-diagonal entries will be zero (uncorrelated).

Since $\mathbf{Y} = \mathbf{U}^T\mathbf{X}$,

\begin{align}
\mathbf{C_Y} &= \frac{1}{n-1}\mathbf{Y}\mathbf{Y^T} \\
 &= \frac{1}{n-1} (\mathbf{U}^T\mathbf{X})(\mathbf{U}^T\mathbf{X})^T \\
 &= \frac{1}{n-1} \mathbf{U}^T\:\mathbf{X}\mathbf{X}^T\:\mathbf{U} \\
 &= \frac{1}{n-1} \mathbf{U}^T\:\mathbf{U}\mathbf{\Sigma}\mathbf{V^T}(\mathbf{U}\mathbf{\Sigma}\mathbf{V^T})^T\:\mathbf{U} \\
 &= \frac{1}{n-1} \mathbf{U}^T\:\mathbf{U}\mathbf{\Sigma}\mathbf{V^T}\mathbf{V}\mathbf{\Sigma}\mathbf{U^T}\:\mathbf{U} \\
 & = \frac{1}{n-1} \mathbf{\Sigma}^2
\end{align}

This is exactly what we want. $\mathbf{C_Y}$ is a matrix which only has entries on the diagonal. This means that the basis vectors on to which we have projected $\mathbf{X}$ must be the principal components. We also know that $\mathbf{\Sigma}$ has the singular values in order largest to smallest, so the columns of $\mathbf{U}$ are our principal component basis vectors arranged such that the first column gives the basis with the largest variance in $\mathbf{Y}$ and so on.

## 3. Simple demonstration of PCA

We can use Python to demonstrate PCA.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
%matplotlib inline
```

First, we construct an artifical set of data like the cartoon above.

```python
npts = 500                                        # number of points
angle = 20.                                       # this is the angle of our first principal component direction
angle_rad = angle/180.*np.pi
s = np.random.normal(size=npts)                   # the s and n coordinates are along, and perpendicular, to PC1
n = 0.1* np.random.normal(size=npts)
x = s*np.cos(angle_rad) - n*np.sin(angle_rad)     # obtain the x and y of each point
y = s*np.sin(angle_rad) + n*np.cos(angle_rad)
x = x - np.mean(x)                                # make sure x and y have zero mean
y = y - np.mean(y)
plt.scatter(x,y,alpha=0.4)
plt.axis("equal")
plt.xlabel("x")
plt.ylabel("y");
```

Construct our data matrix $\mathbf{X}$. The matrix will have two rows (corresponding to x and y) and the number of columns will be the number of data points.

```python
X = np.stack((x,y))                                # numpy has a handy function for combinine two row vectors into a matrix
```

```python
plt.scatter(X[0,:],X[1,:],alpha=0.4)
plt.axis("equal")
plt.xlabel("x")
plt.ylabel("y");
```

Obtain the SVD of $\mathbf{X}$

```python
U, S, VT = np.linalg.svd(X)
```

We can immediately check to see if the first principle component is at the expected angle.

```python
np.arctan(U[0,1]/U[0,0])/np.pi*180.             # U[0,1] is the y component of the first PC, U[0,0] is the x component
```

We now obtain $\mathbf{Y}$ by projecting our data matrix on to the column vectors of $\mathbf{U}$.

```python
Y = U.T @ X
```

And we can plot the data in this transformed basis

```python
plt.scatter(Y[0,:],Y[1,:],alpha=0.4)
plt.axis("equal")
plt.xlabel("PC 1")
plt.ylabel("PC 2");
```

## 4. 'Eigenwakes'

Demonstrations of PCA in the literature sometimes use an example with many photographs of faces. The principal component vectors (each of which is an image) are called [Eigenfaces](https://en.wikipedia.org/wiki/Eigenface). In the spirit of this example, we look at Eigenwakes!

First load a set of wakes 1188 wakes from CFD simulations of compressor stators.

```python
wakes = np.load("data/cfd_images.npy")
wakes = wakes.T                               # take the transpose so that each column of wakes is an individual wake
imsize = [56,56]                              # each wake is a 56 x 56 "image"
n_wakes = wakes.shape[1]
wakes.shape
```

The first column of wakes is a clean wake:

```python
nwake = 0
wake = np.reshape(wakes[:,nwake],imsize)
plt.imshow(wake, vmin=0, vmax=1.)
plt.colorbar()
plt.title("wake "+str(nwake));
```

Wake number 700, for example, has a large corner separation:

```python
nwake = 700
wake = np.reshape(wakes[:,nwake],imsize)
plt.imshow(wake, vmin=0, vmax=1.)
plt.colorbar()
plt.title("wake "+str(nwake));
```

To perform PCA, we need the mean of each row of our data matrix to be zero.
Obtain the mean wake:

```python
wake_mean = np.mean(wakes, axis = 1)
plt.imshow(np.reshape(wake_mean,imsize), vmin=0, vmax=1.)
plt.colorbar()
plt.title("mean wake");
```

Subtract the mean wake from each individual wake to obtain our data matrix, $\mathbf{X}$. Then perform the SVD.

```python
X = wakes - np.outer(wake_mean, np.ones([1,n_wakes]))
U, S, VT = np.linalg.svd(X)
```

We can plot the columns of $\mathbf{U}$ - our principal component vectors - as images.

```python
plt.figure(figsize=(15,8))
for i in range(4):
    plt.subplot(1,4,i+1)
    plt.imshow(np.reshape(U[:,i],imsize))
    plt.title('PC mode '+str(i+1))
```

Project $\mathbf{X}$ on to the columns of $\mathbf{U}$ to obtain $\mathbf{Y}$:

```python
Y = U.T @ X
```

What do the wakes look like if we reduce the number of dimensions to `nPC = 10` ? Remember that the number of dimensions of our original data was 56 x 56 = 3136.

```python
nPC = 10
```

Reconstruct wake 0 using only the first nPC Principal components:

```python
nwake = 0
wake_recon = wake_mean + U[:,:nPC] @ Y[:nPC, nwake]
plt.imshow(np.reshape(wake_recon,imsize), vmin=0, vmax=1.)
plt.title("Wake "+str(nwake)+", reconstructed with "+str(nPC)+" PC modes");
```

and wake 700:

```python
nwake = 700
wake_recon = wake_mean + U[:,:nPC] @ Y[:nPC, nwake]
plt.imshow(np.reshape(wake_recon,imsize), vmin=0, vmax=1.)
plt.title("Wwake "+str(nwake)+", reconstructed with "+str(nPC)+" PC modes");
```

We can plot the principal components (the values in $\mathbf{Y}$) that are needed to reconstruct the wakes.

```python
xx = np.arange(1, nPC+1)
plt.scatter(xx, Y[:nPC, 0],c='b')
plt.scatter(xx, Y[:nPC, 700],c='r')
plt.xlabel('PC number')
plt.ylabel('Value of PC');
```

In the above figure, the blue points are from wake 0 - a clean wake. The red points are from wake 700 - a corner separation.

Using a different technique (convolutional neural network image processing) all of the wakes in our data set have been labeled, and the results are in the cornsep.txt file. A value of 0 means the wake is clean, a value of 1 means a corner separation has been detected.

```python
cornsep = np.loadtxt("data/cornsep.txt", dtype=int)
```

Using these labels, we can plot the values of the principal components for all the wakes in $\mathbf{Y}$.

```python
for i in range(n_wakes):
    plt.scatter(xx,Y[:nPC,i],c=np.array(["b", "r"])[cornsep[i]],alpha=0.01)
plt.xlabel('PC number')
plt.ylabel('Value of PC');
```

You can see that, for some of the principal components, the blue and red dots do not overlap much - this is the case for PC2 for example. If a principal compoment, or a small set of principal components, can be used to differentiate between clean wakes and corner separated wakes, this could be used in an automated classification process.

To illustrate this, we plot each wake as a point in a 2D plot where the axes are PC1 and PC2:

```python
plt.scatter(Y[0,:],Y[1,:],c=np.array(["b", "r"])[cornsep],alpha=0.2)
plt.xlabel('PC 1')
plt.ylabel('PC 2');
```

or in a 3D plot where we use the first 3 principal components as coodinates:

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(Y[0,:],Y[1,:],Y[2,:],c=np.array(["b", "r"])[cornsep],alpha=0.2)
ax.set_xlabel('PC 1')
ax.set_ylabel('PC 2')
ax.set_zlabel('PC 3');

```

The PCA has helped us to see the low dimensional - reduced order - structure that is important in our original, high dimensional data.

```python

```


# Problem

```latex
%%latex
In Figure \ref{fig:probfig}, the radius of the small circle (black) is the same as the edge length of a square.
What is the value of $\tan(\angle \mathrm{ACB})$?

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{probfig.jpeg}
  \caption{Problem's figure}
  \label{fig:probfig}
\end{figure}

```

In Figure \ref{fig:probfig}, the radius of the small circle (black) is the same as the edge length of a square.
What is the value of $\tan(\angle \mathrm{ACB})$?

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{probfig.jpeg}
  \caption{Problem's figure}
  \label{fig:probfig}
\end{figure}

# Solution

```latex
%%latex
% https://tex.stackexchange.com/questions/86044/represent-an-arc-over-letters
\settowidth{\mywidth}{AB}
From Figure \ref{fig:probfig}, the inscribed angle $\angle \mathrm{ACB}$ has its intercepted arc $\bigfrown{\mathrm{AB}}$
(with angle $\angle \mathrm{AOB}$).
Because inscribed angle is half of its intercepted arc, we have:
    \begin{equation}
    \angle \mathrm{ACB} = \frac{1}{2} \times \angle \mathrm{AOB}.
    \end{equation}

Let point D (as shown in Figure \ref{fig:figwithsizes}: \Nameref{fig:figwithsizes}) be the bottom center (crossing point) of the 2 squares.
Due to symmetry $\angle \mathrm{AOD} = \frac{1}{2} \times \angle \mathrm{AOB}$, and hence:
    \begin{equation}
    \tan( \angle \mathrm{ACB} ) = \tan({\angle \mathrm{AOD}}).
    \end{equation}

Figure \ref{fig:figwithsizes} shows Figure \ref{fig:probfig} which is annotated with sizes relevant to the solution.
Let point O be the center of the big red circle. Also, let the edge length of the square (which is also the radius of the small black circle) and
the radius of the big red circle be $a$ and $R$, respectively.
```

% https://tex.stackexchange.com/questions/86044/represent-an-arc-over-letters
\settowidth{\mywidth}{AB}
From Figure \ref{fig:probfig}, the inscribed angle $\angle \mathrm{ACB}$ has its intercepted arc $\bigfrown{\mathrm{AB}}$
(with angle $\angle \mathrm{AOB}$).
Because inscribed angle is half of its intercepted arc, we have:
    \begin{equation}
    \angle \mathrm{ACB} = \frac{1}{2} \times \angle \mathrm{AOB}.
    \end{equation}

Let point D (as shown in Figure \ref{fig:figwithsizes}: \Nameref{fig:figwithsizes}) be the bottom center (crossing point) of the 2 squares.
Due to symmetry $\angle \mathrm{AOD} = \frac{1}{2} \times \angle \mathrm{AOB}$, and hence:
    \begin{equation}
    \tan( \angle \mathrm{ACB} ) = \tan({\angle \mathrm{AOD}}).
    \end{equation}

Figure \ref{fig:figwithsizes} shows Figure \ref{fig:probfig} which is annotated with sizes relevant to the solution.
Let point O be the center of the big red circle. Also, let the edge length of the square (which is also the radius of the small black circle) and
the radius of the big red circle be $a$ and $R$, respectively.

```python
%matplotlib inline
# patches: https://note.nkmk.me/python-matplotlib-patches-circle-rectangle/
# patches: https://www.codespeedy.com/how-to-draw-shapes-in-matplotlib-with-python/
# patches: https://nickcharlton.net/posts/drawing-animating-shapes-matplotlib.html
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as patches
import matplotlib

# matplotlib.rcParams['figure.figsize'] = 6, 6 # Resize the figure, in inches.
matplotlib.rcParams['figure.dpi']= 150
matplotlib.rcParams['mathtext.fontset'] = 'cm'  # or 'stix'
matplotlib.rcParams['font.family'] = 'STIXGeneral'

def annotate_dim(ax,xyfrom,xyto,text=None,fontsize=15, color='black', xoffset=0.0, yoffset=0.0, lnwid=1.0, boxfc='white'):

    if text is None:
        text = str(np.sqrt( (xyfrom[0]-xyto[0])**2 + (xyfrom[1]-xyto[1])**2 ))

    ax.annotate("",xyfrom,xyto,arrowprops=dict(color=color, arrowstyle='<->', shrinkA=0, shrinkB=0, lw=lnwid))
    ax.text((xyto[0]+xyfrom[0])/2+xoffset,(xyto[1]+xyfrom[1])/2+yoffset,text,fontsize=fontsize,color=color,
          bbox=dict(facecolor=boxfc, edgecolor='none', boxstyle='round,pad=0.02'))

def annotate_dim_anchor(ax, xyfrom, xyto, ancfrom, ancto,
                        text=None,fontsize=15, color='black', xoffset=0.0, yoffset=0.0, lnwid=1.0, boxfc='white',
                        anclnwid=0.2, extend=2.0):

    if text is None:
        text = str(np.sqrt( (xyfrom[0]-xyto[0])**2 + (xyfrom[1]-xyto[1])**2 ))

    # Line: ancfrom - xyfrom & ancto - xyto
    eps = 0.0001
    if abs(xyfrom[0] - ancfrom[0])>eps or abs(xyfrom[1] - ancfrom[1])>eps:
        extx = xyfrom[0]
        exty = xyfrom[1]
        if abs(xyfrom[0] - ancfrom[0])<=eps:
            exty = xyfrom[1] + extend * (xyfrom[1] - ancfrom[1])/abs(xyfrom[1] - ancfrom[1])
        elif abs(xyfrom[1] - ancfrom[1])<=eps:
            extx = xyfrom[0] + extend * (xyfrom[0] - ancfrom[0])/abs(xyfrom[0] - ancfrom[0])
        else:
            seglen = np.sqrt( (xyfrom[0]-ancfrom[0])**2 + (xyfrom[1]-ancfrom[1])**2 )
            extx = xyfrom[0] + extend * (xyfrom[0]-ancfrom[0]) / seglen
            exty = xyfrom[1] + extend * (xyfrom[1]-ancfrom[1]) / seglen
        lancfr = plt.Line2D( (ancfrom[0],  extx), (ancfrom[1],  exty), color=color, lw=anclnwid )
        ax.add_line(lancfr)

    if abs(xyto[0] - ancto[0])>eps or abs(xyto[1] - ancto[1])>eps:
        extx = xyto[0]
        exty = xyto[1]
        if abs(xyto[0] - ancto[0])<=eps:
            exty = xyto[1] + extend * (xyto[1] - ancto[1])/abs(xyto[1] - ancto[1])
        elif abs(xyto[1] - ancto[1])<=eps:
            extx = xyto[0] + extend * (xyto[0] - ancto[0])/abs(xyto[0] - ancto[0])
        else:
            seglen = np.sqrt( (xyto[0]-ancto[0])**2 + (xyto[1]-ancto[1])**2 )
            extx = xyto[0] + extend * (xyto[0]-ancto[0]) / seglen
            exty = xyto[1] + extend * (xyto[1]-ancto[1]) / seglen
        lancto = plt.Line2D( (ancto[0],  extx), (ancto[1],  exty), color=color, lw=anclnwid )
        ax.add_line(lancto)

    ax.annotate("",xyfrom,xyto,arrowprops=dict(color=color, arrowstyle='<->', shrinkA=0, shrinkB=0, lw=lnwid))
    ax.text((xyto[0]+xyfrom[0])/2+xoffset,(xyto[1]+xyfrom[1])/2+yoffset,text,fontsize=fontsize,color=color,
          bbox=dict(facecolor=boxfc, edgecolor='none', boxstyle='round,pad=0.02'))

# fig = plt.figure()
ax = plt.axes()

a = 3.0
R = 5.0*a/3.0
ptO = (a, 3*a-R)
vecOA = (0.0-ptO[0], a-ptO[1])
absvecOA = np.sqrt( (vecOA[0])**2 + (vecOA[1])**2 )
uvecOA = ( vecOA[0]/absvecOA, vecOA[1]/absvecOA )
ptA = ( ptO[0] + R*uvecOA[0],  ptO[1] + R*uvecOA[1] )
ptD = (a, 0.0)
ptTop = (a, 3*a)
ptE = (0.0, a)
ptF = (a, a)
ptG = (2*a, 0.0)
shplw=1.5

# fc = face color, ec = edge color
c = patches.Circle(xy=(a, 2*a), radius=a, fc='none', ec='k', lw=shplw, ls='-')  # default lw=1, ls='-'  or '-.' or '--'
cbig = patches.Circle(xy=ptO, radius=R, fc='none', ec='r', lw=shplw, ls='-')
r1 = patches.Rectangle(xy=(0, 0), width=a, height=a, ec='#000000', lw=shplw, fill=False)
r2 = patches.Rectangle(xy=(a, 0), width=a, height=a, ec='#000000', lw=shplw, fill=False)
lineOA = plt.Line2D((ptO[0], ptA[0]), (ptO[1], ptA[1]), ls='--', lw=shplw)
lineOF = plt.Line2D((ptO[0], ptF[0]), (ptO[1], ptF[1]), ls='--', lw=shplw, color='k')
ax.add_patch(cbig)
ax.add_patch(c)
ax.add_patch(r1)
ax.add_patch(r2)
ax.add_line(lineOA)
ax.add_line(lineOF)

plt.text(ptO[0]+0.2, ptO[1]+0.15, 'O', fontsize=12)
plt.text(ptA[0]-0.6, ptA[1]-0.4, 'A', fontsize=12)
plt.text(ptE[0]-0.35, ptE[1]+0.18, 'E', fontsize=12)
plt.text(ptF[0]+0.18, ptF[1]-0.6, 'F', fontsize=12)
plt.text(ptTop[0]-0.2, ptTop[1]+0.25, 'T', fontsize=12)
annotate_dim(ax, ptO, ptG, '$R$', fontsize=13, color='green', xoffset=0.0, yoffset=0.1, lnwid=1.0, boxfc='none')
annotate_dim(ax, ptO, ptTop, '$R$', fontsize=13, color='green', xoffset=0.05, yoffset=-0.1, lnwid=1.0, boxfc='none')
annotate_dim(ax, (0.0, 0.65*a), (a, 0.65*a), '$a$', fontsize=13, color='green', xoffset=-0.12, yoffset=0.17, lnwid=1.0, boxfc='none')

dimanclw = 0.25
# anchor and dimension point
dppos = 4.2
ancp1 = (2*a, a)
dp1 = (dppos*a, a)
dp2 = (dppos*a, 3*a)
hige = 0.5
annotate_dim_anchor(ax,  dp1, dp2, ancp1, ptTop,
             '$2a$', fontsize=13, color='green', xoffset=0.11, yoffset=-0.1, lnwid=1.0, boxfc='none',
            anclnwid=dimanclw, extend=hige)
dp3 = (dppos*a, 0.0)
annotate_dim_anchor(ax,  dp1, dp3, dp1, ptG,
             '$a$', fontsize=13, color='green', xoffset=0.11, yoffset=-0.1, lnwid=1.0, boxfc='none',
            anclnwid=dimanclw, extend=hige)

dppos2 = 3.0
dp4 = (dppos2*a, a)
dp5 = (dppos2*a, ptO[1])
annotate_dim_anchor(ax,  dp4, dp5, dp4, ptO,
             '$2a-R$', fontsize=13, color='green', xoffset=0.2, yoffset=-0.2, lnwid=1.0, boxfc='none',
            anclnwid=dimanclw, extend=hige)

dppos3 = -0.55
dpD = (ptD[0], dppos3*a)
dpG = (ptG[0], dppos3*a)
annotate_dim_anchor(ax,  dpD, dpG, ptD, ptG,
             '$a$', fontsize=13, color='green', xoffset=-0.12, yoffset=0.17, lnwid=1.0, boxfc='none',
            anclnwid=dimanclw, extend=hige)

# annotate_dim_anchor(ax,  (dpD[0]-1, dpD[1]-1), (dpG[0]+2, dpG[1]-3), ptD, ptG,
#              '$a$', fontsize=13, color='green', xoffset=-0.12, yoffset=0.17, lnwid=1.0, boxfc='none',
#             anclnwid=dimanclw, extend=hige)

plt.text(ptD[0]-0.55, ptD[1]-0.6, 'D', fontsize=12)
plt.text(ptG[0]+0.1, ptG[1]-0.6, 'G', fontsize=12)

plt.axis('off')
plt.axis('scaled')
ax.set_aspect('equal')
plt.show()
```

```latex
%%latex
Let $\overline{\mathrm{XY}}$ and $\lvert\overline{\mathrm{XY}}\rvert$ be line segment $\mathrm{XY}$ and its length, respectively.
Looking at right triangle $\triangle\mathrm{ODG}$ and applying Pythagoras' theorem:
\begin{align*}
\lvert\overline{\mathrm{OD}}\rvert^2 + \lvert\overline{\mathrm{DG}}\rvert^2 = \lvert\overline{\mathrm{OG}}\rvert^2
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert^2 + a^2 = R^2 \\
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert = \sqrt{R^2 - a^2}.
\addtocounter{equation}{1}
\label{eq:lenOD}  \tag{\theequation}
\end{align*}
Also from Figure \ref{fig:figwithsizes}, we have:
\begin{align*}
\lvert\overline{\mathrm{OD}}\rvert + \lvert\overline{\mathrm{OT}}\rvert = \lvert\overline{\mathrm{DT}}\rvert
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert + R = \lvert\overline{\mathrm{DF}}\rvert + \lvert\overline{\mathrm{FT}}\rvert \\
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert + R = a + 2a \\
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert = 3a - R.
\addtocounter{equation}{1}
\label{eq:lenOD2}  \tag{\theequation}
\end{align*}
From equation \eqref{eq:lenOD} and \eqref{eq:lenOD2}, we have:
\begin{align*}
% \require{cancel}
\sqrt{R^2 - a^2} = 3a - R
& \Rightarrow R^2 - a^2 = (3a - R)^2 \\
& \Rightarrow \cancel{R^2} - a^2 = 9a^2 - 6aR + \cancel{R^2} \\
& \Rightarrow 6aR = 9a^2 + a^2 = 10a^2  \\
& \Rightarrow 6\cancel{a}R = 10a^{\cancel{2}}  && (a\neq 0) \\
& \Rightarrow R = \frac{5}{3}a.
\addtocounter{equation}{1}
\label{eq:relation_Ra}  \tag{\theequation}
\end{align*}

```

Let $\overline{\mathrm{XY}}$ and $\lvert\overline{\mathrm{XY}}\rvert$ be line segment $\mathrm{XY}$ and its length, respectively.
Looking at right triangle $\triangle\mathrm{ODG}$ and applying Pythagoras' theorem:
\begin{align*}
\lvert\overline{\mathrm{OD}}\rvert^2 + \lvert\overline{\mathrm{DG}}\rvert^2 = \lvert\overline{\mathrm{OG}}\rvert^2
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert^2 + a^2 = R^2 \\
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert = \sqrt{R^2 - a^2}.
\addtocounter{equation}{1}
\label{eq:lenOD}  \tag{\theequation}
\end{align*}
Also from Figure \ref{fig:figwithsizes}, we have:
\begin{align*}
\lvert\overline{\mathrm{OD}}\rvert + \lvert\overline{\mathrm{OT}}\rvert = \lvert\overline{\mathrm{DT}}\rvert
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert + R = \lvert\overline{\mathrm{DF}}\rvert + \lvert\overline{\mathrm{FT}}\rvert \\
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert + R = a + 2a \\
& \Rightarrow \lvert\overline{\mathrm{OD}}\rvert = 3a - R.
\addtocounter{equation}{1}
\label{eq:lenOD2}  \tag{\theequation}
\end{align*}
From equation \eqref{eq:lenOD} and \eqref{eq:lenOD2}, we have:
\begin{align*}
% \require{cancel}
\sqrt{R^2 - a^2} = 3a - R
& \Rightarrow R^2 - a^2 = (3a - R)^2 \\
& \Rightarrow \cancel{R^2} - a^2 = 9a^2 - 6aR + \cancel{R^2} \\
& \Rightarrow 6aR = 9a^2 + a^2 = 10a^2  \\
& \Rightarrow 6\cancel{a}R = 10a^{\cancel{2}}  && (a\neq 0) \\
& \Rightarrow R = \frac{5}{3}a.
\addtocounter{equation}{1}
\label{eq:relation_Ra}  \tag{\theequation}
\end{align*}

```latex
%%latex
Now look at right triangle $\triangle\mathrm{EFO}$.
\begin{align*}
\tan(\angle \mathrm{AOD}) = \tan(\angle \mathrm{EOF})
    &=  \frac{\lvert\overline{\mathrm{EF}}\rvert}{\lvert\overline{\mathrm{FO}}\rvert} \\
    &= \frac{a}{2a - R} && (\text{from Figure }\ref{fig:figwithsizes}) \\
    &= \frac{a}{2a - \frac{5}{3}a} && (\text{substituting $R$ from equation \eqref{eq:relation_Ra}}) \\
    &=  \frac{\cancel{a}}{\frac{1}{3}\cancel{a}} \\
    &=  3.
\addtocounter{equation}{1}
\label{eq:answer}  \tag{\theequation}
\end{align*}
Hence, $\tan(\angle\mathrm{ACB}) = \tan(\angle \mathrm{AOD}) = 3$. $\blacksquare$
```

Now look at right triangle $\triangle\mathrm{EFO}$.
\begin{align*}
\tan(\angle \mathrm{AOD}) = \tan(\angle \mathrm{EOF})
    &=  \frac{\lvert\overline{\mathrm{EF}}\rvert}{\lvert\overline{\mathrm{FO}}\rvert} \\
    &= \frac{a}{2a - R} && (\text{from Figure }\ref{fig:figwithsizes}) \\
    &= \frac{a}{2a - \frac{5}{3}a} && (\text{substituting $R$ from equation \eqref{eq:relation_Ra}}) \\
    &=  \frac{\cancel{a}}{\frac{1}{3}\cancel{a}} \\
    &=  3.
\addtocounter{equation}{1}
\label{eq:answer}  \tag{\theequation}
\end{align*}
Hence, $\tan(\angle\mathrm{ACB}) = \tan(\angle \mathrm{AOD}) = 3$. $\blacksquare$