# Chapter 5: Why are deep neural networks hard to train?

## The vanishing gradient problem

## What's causing the vanishing gradient problem? Unstable gradients in deep neural nets

### Exercise 1 ([link](http://neuralnetworksanddeeplearning.com/chap5.html#exercise_255808)): avoiding the unstable gradient problem with a steeper activation function?

Using our random weight initialization by a Gaussian with mean $0$ and standard deviation $1$, our weights are "typically" the same order of magnitude as $1$ (actually, the mean size is $\sqrt{\frac 2 \pi} \approx 0.8$).

Ideally then, we'd like an activation function whose derivative is as often as possible close to $1$, so the learning speeds of each layer would be similar despite the product of many terms (all close to 1).

Such an activation function wouldn't solve the unstable gradient problem (as explained in the paragraph just above this exercise in the book, the product of terms make the situation intrinsically unstable; if for some reason the weights all grow, we'll still have an exploding gradient, and a vanishing gradient if they all decrease). But it would probably help.

### Problem 1 ([link](http://neuralnetworksanddeeplearning.com/chap5.html#problems_778071)): ranges of weights and activations avoiding the vanishing gradient problem

**(1)** Since $\forall z \in \mathbb{R}, \sigma'(z) \leq \frac 1 4$, we can only ever have $|w \sigma'(wa+b)| \geq 1$ when $|w| \geq 4$.

**(2)** Since $\sigma'$ is largest around $0$, we'll get greater intervals with $b = 0$.

So we want to solve:

\begin{equation*}
    \begin{aligned}
        | w \sigma'(wa)| \geq 1 &\iff \sigma'(wa) \geq \frac{1}{|w|} \\
        &\iff \frac{e^{-wa}}{(1 + e^{-wa})^2} \geq \frac{1}{|w|} \\
        &\iff 0 \geq 1 + ( 2 - |w|) e^{-wa} + (e^{-wa})^2
    \end{aligned}
\end{equation*}

Let's denote $x = e^{-wa}$.

We've got a quadratic inequation: $x^2 + (2 - |w|)x + 1 \leq 0$.

The discriminant of this equation is:

$$\Delta = (2 - |w|)^2 - 4 = |w|(|w| - 4)$$

If $|w| > 4$, then $\Delta > 0$ and we have two distinct real roots that we'll note $r_1$ and $r_2$, $r_1 < r_2$.

As the coefficient before $x^2$ is positive, this inequation will be satisfied for $r_1 \leq x \leq r_2$.

Translating into $a$:

\begin{equation*}
    \begin{aligned}
        r_1 \leq e^{-wa} \leq r_2 &\iff \ln r_1 \leq -wa \leq \ln r_2 \\
        &\iff - \ln r_2 \leq wa \leq - \ln r_1
    \end{aligned}
\end{equation*}

If $w > 0$, this is equivalent to:

$$- \frac 1 w \ln r_2 \leq a \leq - \frac 1 w \ln r_1$$

And so the width of the interval will be:

$$\frac 1 w (-\ln r_1 + \ln r_2) = \frac 1 w \ln \frac{r_2}{r_1} = \frac{1}{|w|} \ln \frac{r_2}{r_1}$$

If $w < 0$, this is equivalent to:

$$- \frac 1 w \ln r_1 \leq a \leq - \frac 1 w \ln r_2$$

And so the width of the interval will be:

$$- \frac 1 w (\ln r_2 - \ln r_1) = - \frac 1 w \ln \frac{r_2}{r_1} = \frac{1}{|w|} \ln \frac{r_2}{r_1}$$

So the width of the interval will always be (again, provided $|w| \geq 4$ and $b = 0$):

$$\frac{1}{|w|} \ln \frac{r_2}{r_1}$$

Now we know ([Vieta's formulas](https://en.wikipedia.org/wiki/Vieta%27s_formulas)) that the roots of the quadratic equation $a x^2 + bx + c = 0$ satisfy $r_1 r_2 = \frac c a$. In our case, $r_1 r_2 = 1$.

Therefore, $\frac{1}{r_1} = r_2$ and $\frac{r_2}{r_1} = r_2^2$. So the width of our interval can be written:

$$\frac{1}{|w|} \ln(r_2^2) = \frac{2}{|w|} \ln r_2$$

All we have to do now is compute $r_2$:

\begin{equation*}
    \begin{aligned}
        r_2 &= \frac{-b + \sqrt{\Delta}}{2a} \\
        &= \frac{|w| - 2 + \sqrt{|w|(|w| - 4)}}{2} \\
        &= \frac{|w| \left( 1 + \sqrt{1 - 4/|w|} \right)}{2} - 1
    \end{aligned}
\end{equation*}

Conclusion: the set of $a$ satisfying $|w\sigma'(wa+b)| \geq 1$ can range over an interval no greater in width than

$$\frac{2}{|w|} \ln\left( \frac{|w|(1+\sqrt{1-4/|w|})}{2}-1\right)$$

**(3)** Let's plot the function (which is a very imprecise, but visual, way of numerically determining a maximum):

```python
import numpy as np
import matplotlib.pyplot as plt
import math

def f(x):
    return 2 / x * math.log(x / 2 * (1 + math.sqrt(1 - 4 / x)) - 1)

X = np.arange(4.0, 14.0, 0.05)
Y = np.array([f(x) for x in X])

plt.plot(X, Y)
plt.show()
```

We see that the maximum is around $|w| = 7$. Being more precise:

```python
import numpy as np
import matplotlib.pyplot as plt
import math

def f(x):
    return 2 / x * math.log(x / 2 * (1 + math.sqrt(1 - 4 / x)) - 1)

X = np.arange(6.5, 7.5, 0.01)
Y = np.array([f(x) for x in X])

plt.plot(X, Y)
plt.show()
```

We see that the width is greatest at $|w| \approx 6.9$, where it takes a value $\approx 0.45$.

### Problem 2: constructing an identity neuron

**Note:** For this problem, I did not follow Nielsen's hint. The approximation I will construct seems satisfying to me, however I feel there might be a better solution. In particular, another solution might perhaps get arbitrarily close to the identity, which my solution can't do.

We start from a regular sigmoid (setting $w_1 = w_2 = 1$, $b = 0$):

```python
import numpy as np
import matplotlib.pyplot as plt
import math

def sigmoid(w1, w2, b, x):
    return w2 / (1 + math.exp(- w1 * x - b))

def plot(w1, w2, b):
    # Plot between -3 and 3:
    X1 = np.arange(-3.0, 3.0, 0.05)
    Y1 = np.array([sigmoid(w1, w2, b, x) for x in X1])

    # Plot between 0 and 1:
    X2 = np.arange(0.0, 1.0, 0.01)
    Y2 = np.array([sigmoid(w1, w2, b, x) for x in X2])
    Ylin = X2  # target function

    plt.subplot(1, 2, 1)  # 1 line, 2 columns, position 1
    plt.ylim(bottom=0, top=1)
    plt.plot(X1, Y1)

    plt.subplot(1, 2, 2)
    plt.ylim(bottom=0, top=1)
    plt.plot(X2, Y2, label="sigmoid")
    plt.plot(X2, Ylin, label="target")
    plt.legend()

    plt.show()
```

```python
w1 = 1
w2 = 1
b = 0
plot(w1, w2, b)
```

What happens if we impose the following 2 conditions on our final function?

* $w_2 \sigma(0.5 w_1 + b) = 0.5$
* $w_2 \sigma(w_1 + b) = 1$

Those conditions mean that the sigmoid approximation of the identity must be exact on abscisses $0.5$ and $1$.

In other terms,

\begin{equation}
\frac{w_2}{1 + e^{-0.5 w_1 - b}} = 0.5
\tag{1}\end{equation}

\begin{equation}
\frac{w_2}{1 + e^{w_1 - b}} = 1
\tag{2}\end{equation}

Dividing (2) by (1) we get:

\begin{equation*}
    \begin{aligned}
        2 &= \frac{1 + e^{-0.5 w_1} e^{-b}}{1 + e^{-w_1}e^{-b}} \\
        1 + 2 e^{-w_1} e^{-b} &= e^{-0.5 w_1} e^{-b} \\
        1 &= e^{-b} \left( e^{-0.5 w_1} - 2 e^{-w_1} \right)
    \end{aligned}
\end{equation*}

Supposing $w_1$ fixed, we obtain:

$$b = \ln \left( e^{-0.5 w_1} - 2 e^{-w_1} \right)$$

Note that this imposes $\left( e^{-0.5 w_1} - 2 e^{-w_1} \right) > 0$, and so $w_1 > 2 \ln 2 \approx 1.39$.

Now we can deduce $w_2$ using (2):

$$w_2 = 1 + e^{- w_1 - b}$$

Playing with the parameter $w_1$ gives rather satisfying approximations around $w_1 = 6$.

```python
w1 = 6
b = math.log(math.exp(- 0.5 * w1) - 2 * math.exp(- w1))
w2 = 1 + math.exp(-(w1 + b))
plot(w1, w2, b)
```

## Unstable gradients in more complex networks

## Other obstacles to deep learning


```python
import matplotlib.pyplot as plt
import numpy as np
```

# Item XIV

Show that the following Laurent expansion is valid in $1 < |z| < 2$:
$$
\frac{-1}{(z-1)(z-2)} = \sum_{n=0}^\infty \frac{z^n}{2^{n+1}} + \sum_{n=1}^\infty \frac{1}{z^n} \,,
$$
and draw an sketch of the region. Does it exist an expansion when $|z|>2$? If so, please compute it.

\* **Note**: I had to change the $1$ with $-1$, for the exercise to be correct.

---

We have that:
\begin{align*}
\sum_{n=0}^{\infty} \frac{z^n}{2^{n+1}} &= \frac{1}{2} \sum_{n=0}^{\infty} \frac{z^n}{2^n}
\\ &= \frac{1}{2} \sum_{n=0}^{\infty} \left( z/2 \right)^n
\\ &= \frac{1}{2} \frac{1}{1-z/2} \quad \text{as long as $|z/2|<1$.}
\\ &= \frac{1}{2-z}
\end{align*}
And for the other sum:
\begin{align*}
\sum_{n=1}^{\infty} \frac{1}{z^n}
&= -1 + \sum_{n=0}^{\infty} \frac{1}{z^n}
\\ &= -1 + \sum_{n=0}^{\infty} (1/z)^n
\\ &= -1 + \frac{1}{1-(1/z)} \quad \text{as long as $(|1/z|<1) \Leftrightarrow (z<-1) \vee (z>1)$.}
\\ &= \frac{1}{z-1}
\end{align*}
Adding both results:
\begin{align}
\frac{1}{2-z} + \frac{1}{z-1} &= \frac{-1}{(z-1)(z-2)}
\end{align}

```python
# Function to plot a function
def plot_fun(fn,xi,xf,vectorized=False,n=101,**args):
    xs = np.linspace(xi,xf,num=n)
    if vectorized:
        ys = fn(xs)
    else:
        ys = [fn(x) for x in xs]
    plt.plot(xs,ys,'-',**args)
```

```python
# The sums, limiting n:
def sums(z,n_max):
    ns = np.arange(n_max+1)
    sum1 = np.sum(z**ns/2**(ns+1))
    sum2 = np.sum(1/z**ns[1:])
    return sum1+sum2
```

```python
# We plot the region
plt.figure(figsize=(8,6))
plot_fun(lambda x: -1/((x-1)*(x-2)),0,3,vectorized=True,label="$\\frac{1}{(1-z)(2-z)}$")
plot_fun(lambda x: sums(x,2),0,3,label="sums up to $n=2$",c=(.6,.6,.6))
plot_fun(lambda x: sums(x,4),0,3,label="sums up to $n=4$",c=(.4,.4,.4))
plot_fun(lambda x: sums(x,8),0,3,label="sums up to $n=8$",c=(.2,.2,.2))
plot_fun(lambda x: sums(x,16),0,3,label="sums up to $n=16$",c=(.0,.0,.0))
plt.ylim((-20,20))
plt.legend()
plt.grid()
plt.show()
```

We can see that it converges in $1<|z|<2$ as $n \rightarrow \infty$.

---

We can get the Laurent expansion of the series when $|z|>2$, if we make use of:
$$
\frac{1}{1-z} = \sum_{n=1}^\infty \frac{1}{z^n} \quad \text{if $|z|>1$} \,.
$$
We proceed:
\begin{align*}
\frac{1}{(z-1)(z-2)} &= \frac{1}{2-z} + \frac{1}{z-1}
\\ &= \frac{1}{2} \cdot \frac{1}{1-\frac{z}{2}} - 1 \cdot \frac{1}{1-z}
\\ &= \frac{-1}{2} \sum_{n=1}^\infty \frac{1}{(z/2)^n} + 1 \sum_{n=1}^\infty \frac{1}{z^n}
\\ &= \frac{-1}{2} \sum_{n=1}^\infty \frac{2^n}{z^n} + 1 \sum_{n=1}^\infty \frac{1}{z^n}
\end{align*}

```python
# The new sums, limiting n:
def sums2(z,n_max):
    ns = np.arange(n_max+1)
    sum1 = -0.5*np.sum(2**ns[1:]/z**ns[1:])
    sum2 = np.sum(1/z**ns[1:])
    return sum1+sum2
```

```python
# We plot the region, again
plt.figure(figsize=(8,6))
plot_fun(lambda x: -1/((x-1)*(x-2)),0,3,vectorized=True,label="$\\frac{1}{(1-z)(2-z)}$")
plot_fun(lambda x: sums2(x,2),0,3,label="sums up to $n=2$",c=(.6,.6,.6))
plot_fun(lambda x: sums2(x,4),0,3,label="sums up to $n=4$",c=(.4,.4,.4))
plot_fun(lambda x: sums2(x,8),0,3,label="sums up to $n=8$",c=(.2,.2,.2))
plot_fun(lambda x: sums2(x,16),0,3,label="sums up to $n=16$",c=(.0,.0,.0))
plt.ylim((-20,20))
plt.legend()
plt.grid()
plt.show()
```

We can see that it converges to the function when $|z|>2$ as $n \rightarrow \infty$.

## References
* http://sym.lboro.ac.uk/resources/Handout-Laurent.pdf

```python

```


```python
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "last_expr"
import numpy as np
from matplotlib import pyplot as plt
```

# 变化率

对于函数 ${y = f(x)}$ 定义变化率为：

$$ roc = \frac{\Delta{y}}{\Delta{x}}  $$

代入函数曲线上的两点，可以计算 ${m}$ ：

\begin{equation}roc = \frac{f(x_{2}) - f(x_{1})}{x_{2} - x_{1}} \end{equation}

例如 ${f(x) = x^{2} + x}$

```python
def f(x):
    return (x)**2 + x

x = np.array(range(0, 11))
y = np.array([0,10])

plt.xlabel('x')
plt.ylabel('y')
plt.grid()

plt.plot(x, f(x), color='g')
plt.plot(y, f(y), color='m')

plt.show()
```

# 极限

例：
\begin{equation}\lim_{x \to 5} f(x)\end{equation}

```python
%matplotlib widget
x = [*range(0,5), *np.arange(4.25, 6, 0.25), *range(6, 11)]
y = [f(i) for i in x]

plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

plt.plot(x,y, color='lightgrey', marker='o', markeredgecolor='green', markerfacecolor='green')
plt.plot(5, f(5), color='red', marker='o', markersize=10)
plt.plot(5.25, f(5.25), color='blue', marker='<', markersize=10)
plt.plot(4.75, f(4.75), color='orange', marker='>', markersize=10)
plt.show()
```

##  连续性

对函数 ${g(x) = -(\frac{12}{2x})^{2},\;\; x \ne 0}$ 画出图形

```python
%matplotlib widget
def g(x):
    if x != 0:
        return -(12/(2*x))**2
x = range(-20, 21)
y = [g(a) for a in x]
plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()
plt.plot(x,y, color='g')
xy = (0,g(1))
plt.annotate('O',xy, xytext=(-0.7, -37),fontsize=14,color='b')
plt.show()
```

函数 ${h(x) = 2\sqrt{x},\;\; x \ge 0}$ 是不连续的。

```python
%matplotlib inline
def h(x):
    if x >= 0:
        import numpy as np
        return 2 * np.sqrt(x)

x = range(-20, 21)
y = [h(a) for a in x]

plt.xlabel('x')
plt.ylabel('h(x)')
plt.grid()

plt.plot(x,y, color='g')
plt.plot(0, h(0), color='g', marker='o', markerfacecolor='g', markersize=10)

plt.show()
```

函数 ${
k(x) = \begin{cases}
  x + 20, & \text{if } x \le 0, \\
  x - 100, & \text{otherwise }\end{cases}}$ 是不连续的。

```python
%matplotlib inline

def k(x):
    import numpy as np
    if x <= 0:
        return x + 20
    else:
        return x - 100

x1 = range(-20, 1)
x2 = range(1, 20)
y1 = [k(i) for i in x1]
y2 = [k(i) for i in x2]

plt.xlabel('x')
plt.ylabel('k(x)')
plt.grid()
plt.plot(x1,y1, color='g')
plt.plot(x2,y2, color='g')
plt.plot(0, k(0), color='g', marker='o', markerfacecolor='g', markersize=10)
plt.plot(0, k(0.0001), color='g', marker='o', markerfacecolor='w', markersize=10)

plt.show()
```

### 无穷

对
$$d(x) = \frac{4}{x - 25},\;\; x \ne 25$$

当 ${x \to 25}$ 时:

```python
%matplotlib inline

def d(x):
    if x != 25:
        return 4 / (x - 25)

x = [*range(-100, 24), *np.arange(24.9, 25.2, 0.1), *range(26, 101)]
y = [d(i) for i in x]

plt.xlabel('x')
plt.ylabel('d(x)')
plt.grid()

plt.plot(x,y, color='purple')
plt.show()
```

\begin{equation}\lim_{x \to 25^{+}} d(x) = \infty \end{equation}

\begin{equation}\lim_{x \to 25^{-}} d(x) = -\infty \end{equation}

对函数 ${e(x) = \begin{cases}
  5, & \text{if } x = 0, \\
  1 + x^{2}, & \text{otherwise }
\end{cases}}$ 求 ${{x\to0}}$ 时的极限：

$$\lim_{x \to 0} a(x) \ne a(0) $$

$$\lim_{x \to 0} a(x) = 1 $$

```python
%matplotlib inline

def e(x):
    if x == 0:
        return 5
    else:
        return 1 + x**2

x= [-1, -0.5, -0.2, -0.1, -0.01, 0.01, 0.1, 0.2, 0.5, 1]
y =[e(i) for i in x]

plt.xlabel('x')
plt.ylabel('e(x)')
plt.grid()

plt.plot(x, y, color='m')
plt.scatter(0, e(0), color='m')
plt.plot(0, 1, color='m', marker='o', markerfacecolor='w', markersize=10)
plt.show()
```

对函数

$$g(x) = \frac{x^{2} - 1}{x - 1}, x \ne 1$$

求 ${x \to 1}$ 时的极限：

\begin{equation}\lim_{x \to a} g(x) = \frac{(x-1)(x+1)}{x - 1}\end{equation}

\begin{equation}\lim_{x \to a} g(x)= x+1\end{equation}

\begin{equation}\lim_{x \to 1} g(x) = 2\end{equation}

类似的，对函数

\begin{equation}h(x) = \frac{\sqrt{x} - 2}{x - 4}, x \ne 4 \text{ and } x \ge 0\end{equation}

求 ${x \to 4}$ 时的极限:

\begin{equation}\lim_{x \to a}h(x) = \frac{1}{{\sqrt{x} + 2}}\end{equation}

\begin{equation}\lim_{x \to 4}h(x) = \frac{1}{{\sqrt{4} + 2}} = \frac{1}{4}\end{equation}

## 极限的算术运算

\begin{equation}\lim_{x \to a} (j(x) + l(x)) = \lim_{x \to a} j(x) + \lim_{x \to a} l(x)\end{equation}

\begin{equation}\lim_{x \to a} (j(x) - l(x)) = \lim_{x \to a} j(x) - \lim_{x \to a} l(x)\end{equation}

\begin{equation}\lim_{x \to a} (j(x) \cdot l(x)) = \lim_{x \to a} j(x) \cdot \lim_{x \to a} l(x)\end{equation}

\begin{equation}\lim_{x \to a} \frac{j(x)}{l(x)} = \frac{\lim_{x \to a} j(x)}{\lim_{x \to a} l(x)}\end{equation}

\begin{equation}\lim_{x \to a} (j(x))^{n} = \Big(\lim_{x \to a} j(x)\Big)^{n}\end{equation}

## 微分和求导

- 导函数的定义：

\begin{equation}\lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

\begin{equation}\frac{d}{dx}f(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

```python
%matplotlib inline

def f(x):
    return x**2 + x

x = list(range(0, 11))
y = [f(i) for i in x]
x1 = 3
y1 = f(x1)
h = 3
x2 = x1+h
y2 = f(x2)

plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()
plt.plot(x,y, color='green')
plt.scatter(x1,y1, c='red')
plt.annotate('(x,f(x))',(x1,y1), xytext=(x1-0.5, y1+3))

plt.scatter(x2,y2, c='red')
plt.annotate('(x+h, f(x+h))',(x2,y2), xytext=(x2+0.5, y2))
plt.show()
```

#### 导函数也可记为

\begin{equation}f'(\textbf{a}) = \lim_{h \to 0} \frac{f(\textbf{a} + h) - f(\textbf{a})}{h} \Leftrightarrow f'(a) = \lim_{x \to a} \frac{f(x) - f(a)}{x - a} \end{equation}

### 求导数

\begin{equation}f(x) = x^{2} + x\end{equation}

\begin{equation}f'(a) = \lim_{h \to 0} \frac{f(a + h) - f(a)}{h} \end{equation}

\begin{equation}f'(\textbf{2}) = \lim_{h \to 0} \frac{f(\textbf{2} + h) - f(\textbf{2})}{h} \end{equation}

\begin{equation}f'(2) = \lim_{h \to 0} \frac{((2+h)^{2} + 2 + h) - (2^{2} + 2)}{h} \end{equation}

\begin{equation}f'(2) = \lim_{h \to 0} \frac{(h^{2} + 5h + 6) - 6}{h} = \lim_{h \to 0} \frac{h^{2} + 5h}{h}  = \lim_{h \to 0} h + 5 = 5\end{equation}

```python
%matplotlib widget

def f(x):
    return x**2 + x

x = list(range(0, 11))
y = [f(i) for i in x]

x1 = 2
y1 = f(x1)

m = 5

plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

plt.plot(x,y, color='green')
plt.scatter(x1,y1, c='red')
plt.annotate('(x,f(x))',(x1,y1), xytext=(x1-0.5, y1+3))

xMin = x1 - 5
yMin = y1 - (5*m)
xMax = x1 + 5
yMax = y1 + (5*m)
plt.plot([xMin,xMax],[yMin,yMax], color='magenta')

plt.show()
```

### 求导函数

\begin{equation}f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} \frac{((x+h)^{2} + x + h) - (x^{2} + x)}{h} \end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} \frac{x^{2} + h^{2} + 2xh + x + h - x^{2} - x}{h} \end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} \frac{h^{2} + 2xh + h}{h} \end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} 2x + h + 1 \end{equation}

\begin{equation}f'(x) = 2x + 1 \end{equation}

## 可微

并不是所有的函数都是可以求导函数的。

可微的函数有下面的几何特征
- 连续
- 切线不竖直
- 光滑

\begin{equation}
q(x) = \begin{cases}
  \frac{40,000}{x^{2}}, & \text{if } x < -4, \\
  (x^{2} -2) \cdot (x - 1), & \text{if } x \ne 0 \text{ and } x \ge -4 \text{ and } x < 8, \\
  (x^{2} -2), & \text{if } x \ne 0 \text{ and } x \ge 8
\end{cases}
\end{equation}

```python
%matplotlib inline

def q(x):
    if x != 0:
        if x < -4:
            return 40000 / (x**2)
        elif x < 8:
            return (x**2 - 2) * x - 1
        else:
            return (x**2 - 2)

x = [*range(-10, -5), -4.01]
x2 = [*range(-4, 8), 7.9999, *range(8, 11)]

y = [q(i) for i in x]
y2 = [q(i) for i in x2]

plt.xlabel('x')
plt.ylabel('q(x)')
plt.grid()

plt.plot(x,y, color='purple')
plt.plot(x2,y2, color='purple')
plt.scatter(-4,q(-4), c='red')
plt.annotate('A (x= -4)',(-5,q(-3.9)), xytext=(-7, q(-3.9)))
plt.scatter(0,0, c='red')
plt.annotate('B (x= 0)',(0,0), xytext=(-1, 40))
plt.scatter(8,q(8), c='red')
plt.annotate('C (x= 8)',(8,q(8)), xytext=(8, 100))

plt.show()
```

- A不连续
- B不连续
- C不光滑

## 求导规则

### 基本规则
\begin{equation}f(x) = \pi \;\; \therefore \;\; f'(x) = 0 \end{equation}

\begin{equation}f(x) = 2g(x) \;\; \therefore \;\; f'(x) = 2g'(x) \end{equation}

\begin{equation}f(x) = g(x) + h(x) \;\; \therefore \;\; f'(x) = g'(x) + h'(x) \end{equation}

\begin{equation}f(x) = k(x) - l(x) \;\; \therefore \;\; f'(x) = k'(x) - l'(x) \end{equation}

\begin{equation}\frac{d}{dx}(2x + 6) = \frac{d}{dx} 2x +  \frac{d}{dx} 6 = 2\end{equation}

### 幂规则
> 极其常用的规则
\begin{equation}f(x) = x^{n} \;\; \therefore \;\; f'(x) = nx^{n-1}\end{equation}

例如：

\begin{equation}f(x) = x^{3} \;\; \therefore \;\; f'(x) = 3x^{2}\end{equation}

\begin{equation}f(x) = x^{-2} \;\; \therefore \;\; f'(x) = -2x^{-3}\end{equation}

\begin{equation}f(x) = x^{2} \;\; \therefore \;\; f'(x) = 2x\end{equation}

### 幂规则的推导

\begin{equation}f(x) = x^{2}\end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} \end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} \frac{(x + h)^{2} - x^{2}}{h} \end{equation}

\begin{equation}f'(x) = \lim_{h \to 0} \frac{x^{2} + h^{2} + 2xh - x^{2}}{h} = \lim_{h \to 0} \frac{h^{2} + 2xh}{h} = \lim_{h \to 0} h + 2x = 2x\end{equation}

### 乘法规则

\begin{equation}\frac{d}{dx}[f(x)g(x)] = f'(x)g(x) + f(x)g'(x) \end{equation}

例如

\begin{equation}f(x) = 2x^{2} \end{equation}

\begin{equation}g(x) = x + 1 \end{equation}

\begin{equation}f'(x) = 4x \end{equation}

\begin{equation}g'(x) = 1 \end{equation}

\begin{equation}\frac{d}{dx}[f(x)g(x)] = (4x \cdot (x + 1)) + (2x^{2} \cdot 1) \end{equation}

\begin{equation}\frac{d}{dx}[f(x)g(x)] = 6x^{2} + 4x \end{equation}

### 商数规则
\begin{equation}r(x) = \frac{s(x)}{t(x)} \end{equation}

\begin{equation}r'(x) = \frac{s'(x)t(x) - s(x)t'(x)}{（t(x)）^{2}} \end{equation}

例如

\begin{equation}s(x) = 3x^{2} \end{equation}

\begin{equation}t(x) = 2x\end{equation}

\begin{equation}r'(x) = \frac{(6x \cdot 2x) - (3x^{2} \cdot 2)}{（2x）^{2}} = \frac{6x^{2}}{4x^{2}} = \frac{3}{2}\end{equation}

### 链式规则

\begin{equation}\frac{d}{dx}[o(i(x))] = o'(i(x)) \cdot i'(x)\end{equation}

例如

\begin{equation}i(x) = x^{2} \end{equation}

\begin{equation}o(x) = 2x \end{equation}

\begin{equation}o'(x) = 2,  i'(x) = 2x\end{equation}

\begin{equation}\frac{d}{dx}[o(i(x))] = 4x\end{equation}

# 极值和优化

对函数${k(x) = -10x^{2} + 100x + 3}$ 有导函数：

\begin{equation}k'(x) = -20x + 100 \end{equation}

```python
%matplotlib inline

def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

x = list(range(0, 11))
y = [k(i) for i in x]

yd = [kd(i) for i in x]

plt.axhline()
plt.axvline()
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

plt.plot(x,y, color='green')

plt.plot(x,yd, color='purple')

x1 = 2
x2 = 5
x3 = 8
plt.plot([x1-1,x1+1],[k(x1)-(kd(x1)),k(x1)+(kd(x1))], color='r')
plt.plot([x2-1,x2+1],[k(x2)-(kd(x2)),k(x2)+(kd(x2))], color='r')
plt.plot([x3-1,x3+1],[k(x3)-(kd(x3)),k(x3)+(kd(x3))], color='r')

plt.show()
```

## 找最大值和最小值

对 ${k(x) = -10x^{2} + 100x + 3}$ 有 ${k'(x) = -20x + 100 }$

\begin{equation}-20x + 100 = 0 \end{equation}

\begin{equation}x = 5 \end{equation}

如果求二阶导数

\begin{equation}k'(x) = -20x + 100 \Rightarrow k''(x) = -20\end{equation}

根据二阶导数为常量，而且是负常量，得知一阶导函数是线性下降的。导函数为0的点就是极大值。

函数
\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

```python
%matplotlib inline

def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

x = list(range(-10, 11))
y = [w(i) for i in x]

yd = [wd(i) for i in x]

plt.axhline()
plt.axvline()
plt.xlabel('x')
plt.ylabel('w(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

plt.plot(x,y, color='g')
plt.plot(x,yd, color='m')
plt.show()
```

## 极值点

对函数
\begin{equation}v(x) = x^{3} - 2x + 100 \end{equation}

```python
%matplotlib widget

def v(x):
    return (x**3) - (2*x) + 100

def vd(x):
    return 3*(x**2) - 2

x = list(range(-10, 11))
y = [v(i) for i in x]

yd = [vd(i) for i in x]

plt.axhline()
plt.axvline()
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-1000, 2000, 100))
plt.grid()
plt.plot(x,y, color='g')
plt.plot(x,yd, color='m')
plt.show()
```

```python
%matplotlib inline

def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

def k2d(x):
    return -20

plt.axhline()
plt.axvline()
x = list(range(0, 11))
y = [k(i) for i in x]
yd = [kd(i) for i in x]
y2d = [k2d(i) for i in x]

plt.xlabel('x')
plt.ylabel('k(x)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()
plt.plot(x,y, color='g')
plt.plot(x,yd, color='r')
plt.plot(x,y2d, color='b')

plt.show()
```

练习：

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

```python
%matplotlib inline

def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

def w2d(x):
    return 2

x = list(range(-10, 11))
y = [w(i) for i in x]
yd = [wd(i) for i in x]
y2d = [w2d(i) for i in x]

plt.axhline()
plt.axvline()
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()
plt.plot(x,y, color='green')
plt.plot(x,yd, color='purple')
plt.plot(x,y2d, color='magenta')

plt.show()
```

## 极值点不一定是最大值或者最小值

\begin{equation}v(x) = x^{3} - 6x^{2} + 12x + 2 \end{equation}

\begin{equation}v'(x) = 3x^{2} - 12x + 12 = 0\end{equation}

${x = 2}$ 是极值点

```python
%matplotlib inline

def v(x):
    return (x**3) - (6*(x**2)) + (12*x) + 2

def vd(x):
    return (3*(x**2)) - (12*x) + 12

def v2d(x):
    return (3*(2*x)) - 12

from matplotlib import pyplot as plt

x = list(range(-5, 11))
y = [v(i) for i in x]
yd = [vd(i) for i in x]
y2d = [v2d(i) for i in x]

plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-2000, 2000, 50))
plt.grid()

plt.plot(x,y, color='green')
plt.plot(x,yd, color='purple')
plt.plot(x,y2d, color='magenta')
plt.show()

print ("v(2) = " + str(v(2)))

print ("v'(2) = " + str(vd(2)))

print ("v''(2) = " + str(v2d(2)))

```

## 优化算法

- 求一阶导数
- 求极值
- 求二阶导数检查是否为最大值

# 偏微分

对多变量函数

$$f(x,y) = x^2 + y^2$$

对于变量x, 定义

$$\frac{\partial f(x,y)}{\partial x} = \frac{\partial (x^2 + y^2)}{\partial x}$$

$$\frac{\partial x^2}{\partial x} = 2x，\;\;\; \frac{\partial y^2}{\partial x} = 0$$

$$\frac{\partial f(x,y)}{\partial x} = 2x + 0 = 2x$$

对于变量y

$$\frac{\partial f(x,y)}{\partial y} = 0 + 2y = 2y$$

## 计算梯度

> 梯度是多唯曲面的切面

$$\frac{\partial f(x,y)}{\partial x} = 2x \\
\frac{\partial f(x,y)}{\partial y} = 2y$$

梯度向量：
$$grad(f(x,y)) =  \vec{g(x,y)} = \begin{bmatrix}\frac{\partial f(x,y)}{\partial x} \\ \frac{\partial f(x,y)}{\partial y} \end{bmatrix} = \begin{bmatrix}2x \\ 2y \end{bmatrix} $$

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import math

el = np.arange(-5,6)
nx, ny = np.meshgrid(el, el, sparse=False, indexing='ij')

x_coord = []
y_coord = []
z = []
for i in range(11):
    for j in range(11):
        x_coord.append(float(-nx[i,j]))
        y_coord.append(float(-ny[i,j]))
        z.append(nx[i,j]**2 + ny[i,j]**2)

x_grad = [-2 * x for x in x_coord]
y_grad = [-2 * y for y in y_coord]

plt.xlim(-5.5,5.5)
plt.ylim(-5.5,5.5)
for x, y, xg, yg in zip(list(x_coord), list(y_coord), list(x_grad), list(y_grad)):
    if x != 0.0 or y != 0.0: ## Avoid the zero divide when scaling the arrow
        l = math.sqrt(xg**2 + yg**2)/2.0
        plt.quiver(x, y, xg, yg, width = l, units = 'dots')

z = np.array(z).reshape(11,11)
plt.contour(el, el, z)
```

图例
- 箭头是梯度方向
- 箭头宽度代表大小
- 梯度方向是垂直于曲面的

## 梯度下降算法

1. 从一个起点开始。
2. 计算这点的梯度。
3. 沿着梯度方向探索一步。
4. 判断梯度是否接近零，如果是，停止。
5. 否则，从第2步继续。

# 积分

对函数 ${f(x)=x}$ 有：

$$\int f(x)\;dx = \frac{1}{2} x^2$$

$$\int_0^2 f(x)\;dx = \frac{1}{2} x^2\ \big|_0^2 = \frac{4}{2} - \frac{0}{2} = 2$$

```python
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

def f(x):
    return x

x = range(0, 11)
y = [f(a) for a in x]

plt.plot(x,y, color='purple')

section = np.arange(0, 2, 1/20)
plt.fill_between(section,f(section), color='orange')

plt.show()
```

```python
import scipy.integrate as integrate
i, e = integrate.quad(lambda x: f(x), 0, 2)
print (i)
```

    2.0

$$\int_0^3 3x^2 + 2x + 1\;dx = \frac{3}{3} x^3 + \frac{2}{2} x^2 + x\ \big|_0^3
= 27 + 9 + 3 + 0 + 0 + 0
= 39$$

```python
from matplotlib.patches import Polygon

def g(x):
    return 3 * x**2 + 2 * x + 1

x = range(0, 11)
y = [g(a) for a in x]

fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

plt.plot(x,y, color='purple')
ix = np.linspace(0, 3)
iy = g(ix)
verts = [(0, 0)] + list(zip(ix, iy)) + [(3, 0)]
poly = Polygon(verts, facecolor='orange')
ax.add_patch(poly)

plt.show()
```

```python
i, e = integrate.quad(lambda x: 3 * x**2 + 2 * x + 1, 0, 3)
print(i)
print(e)
```

    38.99999999999999
    4.3298697960381095e-13

$$\int^{\infty}_0 e^{-5x} dx$$

```python
import numpy as np
i, e = integrate.quad(lambda x: np.exp(-x*5), 0, np.inf)

print('Integral: ' + str(i))
print('Absolute Error: ' + str(e))
```

    Integral: 0.20000000000000007
    Absolute Error: 1.560666811361375e-11

$$\int_{-\infty}^{\infty} \frac{1}{2 \pi} e^{\frac{-x^2}{\sqrt(2 \pi)}} dx$$

```python
import numpy as np
norms = lambda x: np.exp(-x**2/2.0)/np.sqrt(2.0 * 3.14159)
i, e = integrate.quad(norms, -np.inf, np.inf)

print('Integral: ' + str(i))
print('Absolute Error: ' + str(e))
```

    Integral: 1.0000004223321999
    Absolute Error: 1.0178195684846592e-08


# Matrix generation

## Init symbols for *sympy*

```python
from sympy import *
from geom_util import *
from sympy.vector import CoordSys3D
N = CoordSys3D('N')
alpha1, alpha2, alpha3 = symbols("alpha_1 alpha_2 alpha_3", real = True, positive=True)
init_printing()

%matplotlib inline

%reload_ext autoreload
%autoreload 2
%aimport geom_util
```

### Lame params

```python
# h1 = Function("H1")
# h2 = Function("H2")
# h3 = Function("H3")

# H1 = h1(alpha1, alpha2, alpha3)
# H2 = h2(alpha1, alpha2, alpha3)
# H3 = h3(alpha1, alpha2, alpha3)

H1,H2,H3=symbols('H1,H2,H3')

H=[H1, H2, H3]
DIM=3
dH = zeros(DIM,DIM)
for i in range(DIM):
    for j in range(DIM):
        dH[i,j]=Symbol('H_{{{},{}}}'.format(i+1,j+1))

dH

```

### Metric tensor

${\displaystyle \hat{G}=\sum_{i,j} g^{ij}\vec{R}_i\vec{R}_j}$

```python
G_up = getMetricTensorUpLame(H1, H2, H3)
```

${\displaystyle \hat{G}=\sum_{i,j} g_{ij}\vec{R}^i\vec{R}^j}$

```python
G_down = getMetricTensorDownLame(H1, H2, H3)
```

### Christoffel symbols

```python
DIM=3

G_down_diff = MutableDenseNDimArray.zeros(DIM, DIM, DIM)
for i in range(DIM):
    for j in range(DIM):
        for k in range(DIM):

            G_down_diff[i,i,k]=2*H[i]*dH[i,k]

GK = getChristoffelSymbols2(G_up, G_down_diff, (alpha1, alpha2, alpha3))
GK
```

### Gradient of vector

$
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)
=
B \cdot
\left(
\begin{array}{c}
u_1 \\
\frac { \partial u_1 } { \partial \alpha_1} \\
\frac { \partial u_1 } { \partial \alpha_2} \\
\frac { \partial u_1 } { \partial \alpha_3} \\
u_2 \\
\frac { \partial u_2 } { \partial \alpha_1} \\
\frac { \partial u_2 } { \partial \alpha_2} \\
\frac { \partial u_2 } { \partial \alpha_3} \\
u_3 \\
\frac { \partial u_3 } { \partial \alpha_1} \\
\frac { \partial u_3 } { \partial \alpha_2} \\
\frac { \partial u_3 } { \partial \alpha_3} \\
\end{array}
\right)
= B \cdot D \cdot
\left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right)
$

```python
def row_index_to_i_j_grad(i_row):
    return i_row // 3, i_row % 3

B = zeros(9, 12)
B[0,1] = S(1)
B[1,2] = S(1)

B[2,3] = S(1)

B[3,5] = S(1)
B[4,6] = S(1)
B[5,7] = S(1)

B[6,9] = S(1)
B[7,10] = S(1)
B[8,11] = S(1)

for row_index in range(9):
    i,j=row_index_to_i_j_grad(row_index)
    B[row_index, 0] = -GK[i,j,0]
    B[row_index, 4] = -GK[i,j,1]
    B[row_index, 8] = -GK[i,j,2]

B
```

### Strain tensor

$
\left(
\begin{array}{c}
\varepsilon_{11} \\
\varepsilon_{22} \\
\varepsilon_{33} \\
2\varepsilon_{12} \\
2\varepsilon_{13} \\
2\varepsilon_{23} \\
\end{array}
\right)
=
\left(E + E_{NL} \left( \nabla \vec{u} \right) \right) \cdot
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)$

```python
E=zeros(6,9)
E[0,0]=1
E[1,4]=1
E[2,8]=1
E[3,1]=1
E[3,3]=1
E[4,2]=1
E[4,6]=1
E[5,5]=1
E[5,7]=1
E
```

```python
def E_NonLinear(grad_u):
    N = 3

    du = zeros(N, N)

    #    print("===Deformations===")

    for i in range(N):
        for j in range(N):
            index = i*N+j
            du[j,i] = grad_u[index]

    #    print("========")

    I = eye(3)

    a_values = S(1)/S(2) * du * G_up

    E_NL = zeros(6,9)
    E_NL[0,0] = a_values[0,0]
    E_NL[0,3] = a_values[0,1]
    E_NL[0,6] = a_values[0,2]

    E_NL[1,1] = a_values[1,0]
    E_NL[1,4] = a_values[1,1]
    E_NL[1,7] = a_values[1,2]

    E_NL[2,2] = a_values[2,0]
    E_NL[2,5] = a_values[2,1]
    E_NL[2,8] = a_values[2,2]

    E_NL[3,1] = 2*a_values[0,0]
    E_NL[3,4] = 2*a_values[0,1]
    E_NL[3,7] = 2*a_values[0,2]

    E_NL[4,0] = 2*a_values[2,0]
    E_NL[4,3] = 2*a_values[2,1]
    E_NL[4,6] = 2*a_values[2,2]

    E_NL[5,2] = 2*a_values[1,0]
    E_NL[5,5] = 2*a_values[1,1]
    E_NL[5,8] = 2*a_values[1,2]

    return E_NL

%aimport geom_util
u=getUHat3D(alpha1, alpha2, alpha3)

# u=getUHatU3Main(alpha1, alpha2, alpha3)

gradu=B*u

E_NL = E_NonLinear(gradu)*B

E_NL

```

### Physical coordinates

$u_i=u_{[i]} H_i$

```python
P=zeros(12,12)
P[0,0]=H[0]
P[1,0]=dH[0,0]
P[1,1]=H[0]
P[2,0]=dH[0,1]
P[2,2]=H[0]
P[3,0]=dH[0,2]
P[3,3]=H[0]

P[4,4]=H[1]
P[5,4]=dH[1,0]
P[5,5]=H[1]
P[6,4]=dH[1,1]
P[6,6]=H[1]
P[7,4]=dH[1,2]
P[7,7]=H[1]

P[8,8]=H[2]
P[9,8]=dH[2,0]
P[9,9]=H[2]
P[10,8]=dH[2,1]
P[10,10]=H[2]
P[11,8]=dH[2,2]
P[11,11]=H[2]
P=simplify(P)
P
```

```python
B_P = zeros(9,9)

for i in range(3):
    for j in range(3):
        ratio=1
        if (i==0):
            ratio = ratio*H1
        elif (i==1):
            ratio = ratio*H2
        elif (i==2):
            ratio = ratio*H3

        if (j==0):
            ratio = ratio*H1
        elif (j==1):
            ratio = ratio*H2
        elif (j==2):
            ratio = ratio*H3

        row_index = i*3+j

        B_P[row_index, row_index] = ratio

Grad_U_P = simplify(B_P*B*P)
B_P
```

```python
StrainL=simplify(E*Grad_U_P)
StrainL
```

```python
%aimport geom_util

u=getUHatU3Main(alpha1, alpha2, alpha3)

gradup=B_P*B*P*u

E_NLp = E_NonLinear(gradup)*B*P*u

simplify(E_NLp)
```

### Tymoshenko theory

$u^1 \left( \alpha_1, \alpha_2, \alpha_3 \right)=u\left( \alpha_1 \right)+\alpha_3\gamma \left( \alpha_1 \right) $

$u^2 \left( \alpha_1, \alpha_2, \alpha_3 \right)=0 $

$u^3 \left( \alpha_1, \alpha_2, \alpha_3 \right)=w\left( \alpha_1 \right) $

$ \left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right) = T \cdot
\left(
\begin{array}{c}
u \\
\frac { \partial u } { \partial \alpha_1} \\
\gamma \\
\frac { \partial \gamma } { \partial \alpha_1} \\
w \\
\frac { \partial w } { \partial \alpha_1} \\
\end{array}
\right) $

```python
T=zeros(12,6)
T[0,0]=1
T[0,2]=alpha3
T[1,1]=1
T[1,3]=alpha3
T[3,2]=1

T[8,4]=1
T[9,5]=1
T
```

```python
D_p_T = StrainL*T
simplify(D_p_T)
```

```python
u = Function("u")
t = Function("theta")
w = Function("w")

u1=u(alpha1)+alpha3*t(alpha1)
u3=w(alpha1)

gu = zeros(12,1)
gu[0] = u1
gu[1] = u1.diff(alpha1)
gu[3] = u1.diff(alpha3)

gu[8] = u3
gu[9] = u3.diff(alpha1)

gradup=Grad_U_P*gu

# o20=(K*u(alpha1)-w(alpha1).diff(alpha1)+t(alpha1))/2
# o21=K*t(alpha1)
# O=1/2*o20*o20+alpha3*o20*o21-alpha3*K/2*o20*o20
# O=expand(O)
# O=collect(O,alpha3)
# simplify(O)

StrainNL = E_NonLinear(gradup)*gradup
simplify(StrainNL)
```

### Square theory

$u^1 \left( \alpha_1, \alpha_2, \alpha_3 \right)=u_{10}\left( \alpha_1 \right)p_0\left( \alpha_3 \right)+u_{11}\left( \alpha_1 \right)p_1\left( \alpha_3 \right)+u_{12}\left( \alpha_1 \right)p_2\left( \alpha_3 \right) $

$u^2 \left( \alpha_1, \alpha_2, \alpha_3 \right)=0 $

$u^3 \left( \alpha_1, \alpha_2, \alpha_3 \right)=u_{30}\left( \alpha_1 \right)p_0\left( \alpha_3 \right)+u_{31}\left( \alpha_1 \right)p_1\left( \alpha_3 \right)+u_{32}\left( \alpha_1 \right)p_2\left( \alpha_3 \right) $

$ \left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right) = L \cdot
\left(
\begin{array}{c}
u_{10} \\
\frac { \partial u_{10} } { \partial \alpha_1} \\
u_{11} \\
\frac { \partial u_{11} } { \partial \alpha_1} \\
u_{12} \\
\frac { \partial u_{12} } { \partial \alpha_1} \\
u_{30} \\
\frac { \partial u_{30} } { \partial \alpha_1} \\
u_{31} \\
\frac { \partial u_{31} } { \partial \alpha_1} \\
u_{32} \\
\frac { \partial u_{32} } { \partial \alpha_1} \\
\end{array}
\right) $

```python
L=zeros(12,12)
h=Symbol('h')
p0=1/2-alpha3/h
p1=1/2+alpha3/h
p2=1-(2*alpha3/h)**2

L[0,0]=p0
L[0,2]=p1
L[0,4]=p2

L[1,1]=p0
L[1,3]=p1
L[1,5]=p2

L[3,0]=p0.diff(alpha3)
L[3,2]=p1.diff(alpha3)
L[3,4]=p2.diff(alpha3)

L[8,6]=p0
L[8,8]=p1
L[8,10]=p2

L[9,7]=p0
L[9,9]=p1
L[9,11]=p2

L[11,6]=p0.diff(alpha3)
L[11,8]=p1.diff(alpha3)
L[11,10]=p2.diff(alpha3)

L
```

```python
B_General = zeros(9, 12)
B_General[0,1] = S(1)
B_General[1,2] = S(1)
B_General[2,3] = S(1)

B_General[3,5] = S(1)
B_General[4,6] = S(1)
B_General[5,7] = S(1)

B_General[6,9] = S(1)
B_General[7,10] = S(1)
B_General[8,11] = S(1)

for row_index in range(9):
    i,j=row_index_to_i_j_grad(row_index)

    B_General[row_index, 0] = -Symbol("G_{{{}{}}}^1".format(i+1,j+1))
    B_General[row_index, 4] = -Symbol("G_{{{}{}}}^2".format(i+1,j+1))
    B_General[row_index, 8] = -Symbol("G_{{{}{}}}^3".format(i+1,j+1))

B_General
```

```python
simplify(B_General*L)
```

```python
D_p_L = StrainL*L
simplify(D_p_L)
```

```python
h = 0.5
exp=(0.5-alpha3/h)*(1-(2*alpha3/h)**2)#/(1+alpha3*0.8)
p02=integrate(exp, (alpha3, -h/2, h/2))
integral = expand(simplify(p02))
integral
```

## Mass matrix

```python
rho=Symbol('rho')
B_h=zeros(3,12)
B_h[0,0]=1
B_h[1,4]=1
B_h[2,8]=1
M=simplify(rho*P.T*B_h.T*G_up*B_h*P)
M
```

```python
e1 = L.T*M*L/(1+alpha3*0.8)
e2 = L.T*M*L
e1r=integrate(e1, (alpha3, -thick/2, thick/2))
e2r=integrate(e2, (alpha3, -thick/2, thick/2))
```

```python
thick=0.1
e1r.subs(h, thick)-e2r.subs(h, thick)
```


```python
%pylab inline
import sympy
sympy.init_printing()
```

    Populating the interactive namespace from numpy and matplotlib

```python
t0, t1, tau, Q, P, dt, T = sympy.symbols(
    't0, t1, tau, Q, P, \Delta_t, T',
    real=True, positive=True)
```

```python
def Phi(t, tau):
    return sympy.exp(-(t - tau)/T)
```

```python
Q_d = sympy.integrate(Phi(t1, tau)*Q*Phi(t1, tau),
                    (tau, t0, t1)).simplify()
simp_subs = {
    2*(t1 - t0): 2*dt,
    (2*t0 - 2*t1)/T: -2*dt/T
}
Q_d = Q_d.simplify().subs(simp_subs)
Q_d
```

```python
dt = 1.0/250
T = 1000
gyro_sigma_rrw = 1e-5
gyro_sigma_rrw*gyro_sigma_rrw*T/2*(1 - exp(-2*dt/1000))
```


# Scientific Computing with Python (Second Edition)

# Chapter 02

*We start by importing all from Numpy. As explained in Chapter 01 the examples are written assuming this import is initially done.*

```python
from numpy import *
```

## 2.1 Variables

```python
a = 1

diameter = 3.

height = 5.

cylinder = [diameter, height] # reference to a list
```

```python
print(a, diameter, height,cylinder)
```

    1 3.0 5.0 [3.0, 5.0]

```python
x = 3 # integer (int)
y = 'sunny' # string (str)
```

```python
print(type(x), type(y))
```

    <class 'int'> <class 'str'>

```python
a = b = c = 1   # a, b and c get the same value 1
```

```python
print(a,b,c)
```

    1 1 1

```python
a = 1
a = a + 1   # a gets the value 2
a = 3 * a   # a gets the value 6
```

```python
print(a)
```

    6

```python
a = 1
a += 1  # same as a = a + 1
a *= 3  # same as a = 3 * a
```

```python
print(a)
```

    6

### 2.2 Numeric types
#### 2.2.1 Integers

```python
6 // 2  # 3  an integer value
```

    3

```python
7 // 2  # 3
```

    3

```python
7 / 2   # 3.5  a float value
```

    3.5

#### 2.2.2 Floating point numbers

```python
a = 3.0
```

```python
a = 30.0e-1
```

```python
a = 30.e-1
```

```python
0.4 - 0.3 # returns 0.10000000000000003
```

    0.10000000000000003

```python
0.4 - 0.3 == 0.1 # returns False
```

    False

```python
exp(1000.) # inf
```

    /home/claus/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: RuntimeWarning: overflow encountered in exp
      """Entry point for launching an IPython kernel.

    inf

```python
a = inf
```

```python
3 - a   # -inf
```

    -inf

```python
3 + a   # inf
```

    inf

```python
a + a # inf
```

    inf

```python
a - a # nan
```

    nan

```python
 a / a # nan
```

    nan

```python
x = nan
x < 0 # False
```

    False

```python
x > 0 # False
```

    False

```python
x == x # False
```

    False

```python
0 < inf     # True
```

    True

```python
inf <= inf  # True
```

    True

```python
 inf == inf  # True
```

    True

```python
-inf < inf  # True
```

    True

```python
inf - inf   # nan
```

    nan

```python
exp(-inf)   # 0
```

    0.0

```python
exp(1 / inf)  # 1
```

    1.0

*Remove in the next cell the comment sign to provoke the error*

```python
seterr(all = 'raise')
#exp(1000)
```

    {'divide': 'warn', 'over': 'warn', 'under': 'ignore', 'invalid': 'warn'}

```python
import sys
sys.float_info.epsilon # 2.220446049250313e-16
```

    2.220446049250313e-16

```python
a = float('1.356')
print(a)
```

    1.356

```python
a = pi            # returns 3.141592653589793
a
```

    3.141592653589793

```python
a1 = float64(a)   # returns 3.1415926535897931
a1
```

    3.141592653589793

```python
a2 = float32(a)   # returns 3.1415927
a2
```

    3.1415927

```python
a - a1            # returns 0.0
```

    0.0

```python
a - a2            # returns -8.7422780126189537e-08
```

    -8.742278012618954e-08

```python
f32 = finfo(float32)
f32.precision   # 6 (decimal digits)
```

    6

```python
f64 = finfo(float64)
f64.precision   # 15 (decimal digits)
```

    15

```python
f = finfo(float)
f.precision     # 15 (decimal digits)
```

    15

```python
f64.max         # 1.7976931348623157e+308 (largest number)
```

    1.7976931348623157e+308

```python
f32.max         # 3.4028235e+38 (largest number)
```

    3.4028235e+38

*Uncomment the next line to get the help text*

```python
# help(finfo)     # Check for more options
```

### 2.2.3 Complex numbers

```python
z = 5.2j
z
```

    5.2j

```python
z = 3.5 + 5.2j
z
```

    (3.5+5.2j)

```python
b = 5.2
```

*Uncomment the next line to provoke the NameError*

```python
#z = bj   # returns a NameError
```

*Uncomment the next line to provoke the NameError*

```python
#z = b*j  # returns a NameError
```

```python
z = b*1j # is correct
z
```

    5.2j

```python
z = 3.2 + 5.2j
z.conjugate() # returns (3.2-5.2j)
```

    (3.2-5.2j)

```python
z = 1j
z.real       # 0.0
```

    0.0

```python
z.imag       # 1.0
```

    1.0

*Remove in the next cell the comment sign to provoke the error*

```python
# z.imag = 2   # AttributeError: readonly attribute
```

```python
z = 1 + 0j
z == 1     # True
```

    True

*Remove in the next cell the comment sign to provoke the error*

```python
# float(z)   # TypeError
```

```python
from matplotlib.pyplot import *
N = 10
# the following vector contains the Nth roots of unity:
unity_roots = array([exp(1j*2*pi*k/N) for k in range(N)])
# access all the real or imaginary parts with real or imag:
axes(aspect='equal')
plot(unity_roots.real, unity_roots.imag, 'o')
allclose(unity_roots**N, 1) # True
```

```python
z = 3.2+5.2j
(z + z.conjugate()) / 2.   # returns (3.2+0j)
```

    (3.2+0j)

```python
((z + z.conjugate()) / 2.).real   # returns 3.2
```

    3.2

```python
(z - z.conjugate()) / 2.   # returns 5.2j
```

    5.2j

```python
((z - z.conjugate()) / 2.).imag   # returns 5.2
```

    5.2

```python
sqrt(z * z.conjugate())   # returns (6.1057350089894991+0j)
```

    (6.105735008989499+0j)

## 2.3 Booleans

```python
a = True
print(a)
b = 30 > 45   # b gets the value False
print(b)
```

    True
    False

```python
x= 5
if x > 0:
 print("positive")
else:
 print("nonpositive")
```

    positive

### 2.3.1 Boolean operators

```python
True and False # False
```

    False

```python
False or True # True
```

    True

```python
(30 > 45) or (27 < 30) # True
```

    True

```python
not True # False
```

    False

```python
not (3 > 4) # True
```

    True

```python
a=1; b=2; c=2
a < b < c     # same as: a < b and b < c
```

    False

```python
a < b <= c    # same as: a < b and b <= c (less or equal)
```

    True

```python
a == b == c   # same as: a == b and b == c
```

    False

### 2.3.2 Boolean casting

```python
bool([])   # False
```

    False

```python
bool(0)   # False
```

    False

```python
bool(' ')   # True
```

    True

```python
bool('')   # False
```

    False

```python
bool('hello')   # True
```

    True

```python
bool(1.2)   # True
```

    True

```python
bool(array([1]))   # True
```

    True

*Remove in the next cell the comment sign to provoke the error*

```python
# bool(array([1,2]))   # Exception raised!
```

### 2.3.3 Automatic Boolean casting

```python
a=10
if a:
   ...
if bool(a): # exactly the same as above
   ...
```

```python
L=[1,2,3]
if L:
    print("list not empty")
else:
    print("list is empty")
```

    list not empty

```python
L=[]
if L:
    print("list not empty")
else:
    print("list is empty")
```

    list is empty

```python
n = 20
if n % 2:      # the modulus operator
    print("n is odd")
else:
    print("n is even")
```

    n is even

```python
n = 21
if n % 2:
    print("n is odd")
else:
    print("n is even")
```

    n is odd

### 2.3.4 Return values of and and or

```python
x = 1; y = 0
x and y
```

    0

```python
def and_as_function(x,y):
    if not x:
        return x
    else:
        return y
```

```python
and_as_function(x,y)
```

    0

```python
x or y
```

    1

```python
y or x
```

    1

```python
def or_as_function(x,y):
    if x:
        return x
    else:
        return y
```

```python
or_as_function(x,y)
```

    1

```python
del x
True or xformatting
```

    True

```python
False and x
```

    False

```python
1 or 'a' # produces 1
```

    1

```python
'a' or 1 # produces 'a'
```

    'a'

### 2.3.5 Boolean and integer

```python
def print_ispositive(x):
    possibilities = ['nonpositive or zero', 'positive']
    return f"x is {possibilities[x>0]}"

print_ispositive(-10)
```

    'x is nonpositive or zero'

```python
isinstance(True, bool)
```

    True

```python
isinstance(False, bool)
```

    True

```python
isinstance(True, int)
```

    True

```python
isinstance(False, int)
```

    True

```python
True + 13
```

    14

## 2.4 Strings

```python
name = 'Johan Carlsson'
```

```python
child = "Åsa is Johan Carlsson's daughter"
```

```python
book = """Aunt Julia
       and the Scriptwriter"""
```

```python
book[-1] # returns 'r'
```

    'r'

```python
book[-12:] # returns 'Scriptwriter'
```

    'Scriptwriter'

*Remove in the next cell the comment sign to provoke the error*

```python
# book[1] = 'a'
```

### 2.4.1 Escape sequences and raw strings

```python
print('Temperature\t20\tC\nPressure\t5\tPa')
```

    Temperature	20	C
    Pressure	5	Pa

```python
a="""
A multi-line
example"""
a #  returns '\nA multi-line \nexample'
```

    ' \nA multi-line \nexample'

```python
print(a)
```

    A multi-line
    example

```python
latexfontsize="\\tiny"
print(latexfontsize)  # prints \tiny
```

    \tiny

```python
latexfs=r"\tiny"   # returns "\tiny"
latexfontsize == latexfs  # returns True
```

    True

```python
print(r"\"")   # returns  \"
```

    \"

```python
print(r"\\")   # returns  \
```

    \\

*Remove in the next cell the comment sign to provoke the error*

```python
# print(r"\")    # returns an error (why?)
```

```python
r"\"" == '\\"'
```

    True

```python
r"She: \"I am my dad's girl\"" == 'She: \\"I am my dad\'s girl\\"'
```

    True

### 2.4.2 Operations on strings and string methods

```python
last_name = 'Carlsson'
first_name = 'Johanna'
full_name = first_name + ' ' + last_name
                              # returns 'Johanna Carlsson'
full_name
```

    'Johanna Carlsson'

```python
game = 2 * 'Yo' # returns 'YoYo'
game
```

    'YoYo'

*Remove in the next cell the comment sign to provoke the error*

```python
# game = 2.7 * 'Yo' # returns 'YoYo'
```

```python
'Anna' > 'Arvi' # returns false
```

    False

```python
'ANNA' < 'anna'  # returns true
```

    True

```python
'10B' < '11A'    # returns true
```

    True

```python
text = 'quod erat    demonstrandum'
```

```python
text.split() # returns ['quod', 'erat', 'demonstrandum']
```

    ['quod', 'erat', 'demonstrandum']

```python
table = 'Johan;Carlsson;19890327'
table.split(';') # returns ['Johan','Carlsson','19890327']
```

    ['Johan', 'Carlsson', '19890327']

```python
king = 'CarlXVIGustaf'
king.split('XVI')  # returns ['Carl','Gustaf']
```

    ['Carl', 'Gustaf']

### 2.4.3 String formatting

```python
course_code = "NUMA01"
print("Course code: {}".format(course_code))    # Course code: NUMA01
```

    Course code: NUMA01

```python
course_code = "NUMA01"
print(f"Course code: {course_code}")            # Course code: NUMA01
```

    Course code: NUMA01

```python
quantity = 33.45
print("{:f}".format(quantity)) # 33.450000
print("{:1.1f}".format(quantity)) # 33.5
print("{:.2e}".format(quantity)) # 3.35e+01
```

    33.450000
    33.5
    3.35e+01

```python
quantity = 33.45
print(f"{quantity:1.1f}")         # 33.5
```

    33.5

```python
r"we {} in LaTeX \begin{{equation}}".format('like')
```

    'we like in LaTeX \\begin{equation}'

### 2.5 Summary
*No code*

## 2.6 Exercises

*Code variant 1*

```python
u = 1.0 # you have to use a float here!
uold = 10.
for iteration in range(2000):
   if not abs(u-uold) > 1.e-8:
      print('Convergence')
      break # sequence has converged
   uold = u
   u = 2*u
else:
   print('No convergence')
```

    Convergence

*Code variant 2*

```python
u = 1.0 # you have to use a float here!
uold = 10.
for iteration in range(2000):
   if abs(u-uold) < 1.e-8:
      print('Convergence')
      break # sequence has converged
   uold = u
   u = 2*u
else:
   print('No convergence')
```

    No convergence


In this example, an equation for a contour of an axially symmetric
beam propagating in free space is checked.

Suppose there is an axially symmetric charged particles beam propagating
in free space.
Under certain set of approximations, trajectory of a particle on the edge
of the beam can be described by the following expressions:

<!--  -->
\begin{align}
  & r = r(0) e^{u^2}
  \\
  & z =
  \left( \dfrac{m v^3}{q I} \right)^{1/2}
  r(0)
  \int_{0}^{u} e^{u^2} du
\end{align}

where $u$ is a parameter, $v$ is speed of particles along the $z$ axis,
$r(0)$ is initial radius and $I$ is full current of the beam.
Derivation of these relations can be found in supplementary notebook [link].

These equations define a contour of the beam $r(z)$. It is possible to perform
a numerical simulation of a beam propagating in free space and
plot particles coordinates on the X-Z or Y-Z plane.
This will allow to compare the numerical profile
with the analytical expression.

The beam will become $e = 2.71$ times wider than the initial at $u = 1$.
For this value of the parameter, the integral in the $z$ equation
[approximately equals to](https://www.wolframalpha.com/input/?i=integrate%28+e^u^2,+u,+0,+1+%29) $1.46$.
Since the speed in the $z$-direction is assumed constant, it is possible to estimate the time needed for
particles to pass this distance as

<!--  -->
\begin{align}
  t \approx
  1.5 \left( \dfrac{m v}{q I} \right)^{1/2}
  r(0)
\end{align}

Let's substitute some numerical values (see the cell below). Suppose there is an electron beam ($q = 4.8 \cdot 10^{-10} ~ [\mbox{cgs}], ~ m = 9.1 \cdot 10^{-28} ~ [\mbox{g}]$) with full current $I = 0.1 ~ [\mbox{A}] = 2.998 \cdot 10^{8} ~ [\mbox{cgs}]$ and initial radius $r(0) = 0.5 ~ [\mbox{cm}]$.
Energy of it's particles equals to 1 keV, so that speed is $v = 1.808 \cdot 10^{9} ~ [\mbox{cm/s}]$.
For this beam to get $2.71$ times wider (from $0.5 ~ [\mbox{cm}]$ to $1.36 ~ [\mbox{cm}]$), it takes $t = 2.63 \cdot 10^{-9} ~ [\mbox{s}]$ seconds.
This happens over a distance $z = 4.632 ~ [\mbox{cm}]$.

```python
from math import *

m = 9.8e-28
q = 4.8e-10
print( "q = {:.3e} [cgs]".format( q ) )
print( "m = {:.3e} [g]".format( m ) )

r_0 = 0.5
print( "beam_radius = {:.3e} [cm]".format( r_0 ) )

ampere_to_cgs = 2997924536.8431
I = 0.1 * ampere_to_cgs
print( "I = {:.3e} [A] = {:.3e} [cgs]".format( I / ampere_to_cgs, I ) )

ev_to_cgs = 1.60218e-12
E = 1000 * ev_to_cgs
v = sqrt( 2 * E / m )
print( "E = {:.3e} [eV] = {:.3e} [erg]".format( E / ev_to_cgs, E ) )
print( "v = {:.3e} [cm/s]".format( v ) )

r0_e_times_wider = e * r_0
z_e_times_wider = 1.46 * sqrt( m * v**3 / q / I ) * r_0
t_e_times_wider = 1.5 * sqrt( m * v / q / I ) * r_0
print( "r0_e_times_wider = {:.3e} [cm]".format( r0_e_times_wider ) )
print( "t_e_times_wider = {:.3e} [s]".format( t_e_times_wider ) )
print( "z_e_times_wider = {:.3e} [cm]".format( z_e_times_wider ) )
```

    q = 4.800e-10 [cgs]
    m = 9.800e-28 [g]
    beam_radius = 5.000e-01 [cm]
    I = 1.000e-01 [A] = 2.998e+08 [cgs]
    E = 1.000e+03 [eV] = 1.602e-09 [erg]
    v = 1.808e+09 [cm/s]
    r0_e_times_wider = 1.359e+00 [cm]
    t_e_times_wider = 2.632e-09 [s]
    z_e_times_wider = 4.632e+00 [cm]

These estimates can be used as a basis for config file parameters.

Full simulation time is set to 3.0e-9 [s]. There are 100 time steps, with each 10th step written to file.

```python
from ef.config.components import *

sim_time = 3.0e-9
n_of_steps = 100
dt = sim_time / n_of_steps
save_each_step = 10
dt_save = dt * save_each_step
time_grid = TimeGridConf(total=sim_time, step=dt, save_step=dt_save)
print(time_grid)
```

    ### TimeGridConf:
    total = 3e-09
    save_step = 3e-10
    step = 3e-11

The simulation domain is defined as

```python
mesh = SpatialMeshConf(size=(5, 5, 10), step=(.1, .1, .1))
print(mesh)
```

    ### SpatialMeshConf:
    size = array([ 5.,  5., 10.])
    step = array([0.1, 0.1, 0.1])

with z size approximately two times the estimated value, with 100 nodes in that direction. X and y sizes are 10 times the radius with 50 nodes in each direction.

The source is centered along the x and y axes and is close to origin along the z. For a time step dt = 3.00e-11 [s], to provide a current I = 0.1 [A] the source has to generate $n = I ~ dt ~/~ q = 1.87 \cdot 10^7$ particles each time step. This is not computationally feasible, so instead let's fix an amount of generated particles to 5000 at each step. The charge of macroparticles should be $Q = I ~ dt ~/~ n = 1.799 \cdot 10^{-6} ~ [\mbox{cgs}]$ To preserve charge-to-mass ratio, mass of the macroparticles should be set to $M = Q ~/~ q_e ~ m_e = 3.672 \cdot 10^{-24} ~ [\mbox{g}]$. To have the same initial velocity as electrons, mean momentum should be set to $p = M ~ v = 6.641 \cdot 10^{-15} ~ [\mbox{g * cm / s}]$.

```python
num_of_real_particles = I * dt / q
print( "num_of_real_particles = {:.3e}".format( num_of_real_particles ) )

num_of_macro_particles = 5000
macro_q = I * dt / num_of_macro_particles
macro_m = macro_q / q * m
macro_mean_momentum = macro_m * v
print( "num_of_macro_particles = {:d}".format( num_of_macro_particles ) )
print( "macro_q = {:.3e} [cgs]".format( macro_q ) )
print( "macro_m = {:.3e} [g]".format( macro_m ) )
print( "macro_mean_momentum = {:.3e} [g * cm / s]".format( macro_mean_momentum ) )

source = ParticleSourceConf(
    name = "cathode_emitter",
    shape = Cylinder(start=(mesh.size[0]/2., mesh.size[1]/2., 0.52),
                     end=(mesh.size[0]/2., mesh.size[1]/2., 0.51),
                     radius=0.5),
    initial_particles = num_of_macro_particles,
    particles_to_generate_each_step = num_of_macro_particles,
    momentum = (0, 0, macro_mean_momentum),
    temperature = 0.0,
    charge = -macro_q,
    mass = macro_m
)
```

```python
from ef.config.config import Config
from ef.config.visualizer import Visualizer3d
axially_symmetric_beam_conf = Config(time_grid, mesh, sources=[source])
vis = Visualizer3d()
axially_symmetric_beam_conf.visualize_all(vis)
```

    <Figure size 640x480 with 1 Axes>

Potential is zero on each boundary of the domain (by default).

Other config parameters:

```python
output_file = OutputFileConf(prefix = "contour_jup_", suffix = ".h5")
particle_interaction_model = ParticleInteractionModelConf(model = "PIC")
```

```python
axially_symmetric_beam_conf = Config(time_grid, mesh, sources=[source],
                                     particle_interaction_model=particle_interaction_model,
                                     output_file=output_file)
print(axially_symmetric_beam_conf.export_to_string())
```

    [TimeGrid]
    total_time = 3e-09
    time_save_step = 3e-10
    time_step_size = 3e-11

    [SpatialMesh]
    grid_x_size = 5.0
    grid_x_step = 0.1
    grid_y_size = 5.0
    grid_y_step = 0.1
    grid_z_size = 10.0
    grid_z_step = 0.1

    [ParticleSourceCylinder.cathode_emitter]
    cylinder_axis_start_x = 2.5
    cylinder_axis_start_y = 2.5
    cylinder_axis_start_z = 0.51
    cylinder_axis_end_x = 2.5
    cylinder_axis_end_y = 2.5
    cylinder_axis_end_z = 0.52
    cylinder_radius = 0.5
    initial_number_of_particles = 5000
    particles_to_generate_each_step = 5000
    mean_momentum_x = 0.0
    mean_momentum_y = 0.0
    mean_momentum_z = 6.640708217582056e-15
    temperature = 0.0
    charge = -1.7987547221058602e-06
    mass = 3.672457557632798e-24

    [OutputFilename]
    output_filename_prefix = contour_jup_
    output_filename_suffix = .h5

    [BoundaryConditions]
    boundary_phi_right = 0.0
    boundary_phi_left = 0.0
    boundary_phi_bottom = 0.0
    boundary_phi_top = 0.0
    boundary_phi_near = 0.0
    boundary_phi_far = 0.0

    [ParticleInteractionModel]
    particle_interaction_model = PIC

```python
axially_symmetric_beam_conf.make().start_pic_simulation()
```

```python
import h5py
import numpy as np
import matplotlib.pyplot as plt
import scipy.integrate

filename = 'contour_jup_0000100.h5'
h5file = h5py.File( filename, mode = "r" )

def get_source_current( h5file ):
    time_step = h5file["/TimeGrid"].attrs["time_step_size"]
    charge = h5file["/ParticleSources/cathode_emitter"].attrs["charge"]
    particles_per_step = h5file[
        "/ParticleSources/cathode_emitter"].attrs["particles_to_generate_each_step"]
    current = particles_per_step * charge / time_step
    return current

def get_source_geometry( h5file ):
    axis_start_x = \
        h5file["/ParticleSources/cathode_emitter"].attrs["cylinder_axis_start_x"]
    axis_start_z = \
        h5file["/ParticleSources/cathode_emitter"].attrs["cylinder_axis_start_z"]
    radius = h5file["/ParticleSources/cathode_emitter"].attrs["cylinder_radius"]
    return ( axis_start_x, axis_start_z, radius )

def get_source_particle_parameters( h5file ):
    mass = h5file["/ParticleSources/cathode_emitter"].attrs["mass"]
    charge = h5file["/ParticleSources/cathode_emitter"].attrs["charge"]
    momentum_z = h5file["/ParticleSources/cathode_emitter"].attrs["mean_momentum_z"]
    return ( mass, charge, momentum_z )

def beam_radius( u, r_0 ):
    return r_0 * np.exp( u ** 2 )

def beam_z( u, m, v, q, I, r_0 ):
    coeff = np.sqrt( m * v**3 / q / I ) * r_0
    subint = lambda t: np.exp( t * t )
    low_lim = 0
    up_lim = u
    integral_value = scipy.integrate.quad( subint, low_lim, up_lim )[0]
    return coeff * integral_value

beam_axis_x_pos, emitter_z_pos, r_0 = get_source_geometry( h5file )
I = get_source_current( h5file )
m, q, p = get_source_particle_parameters( h5file )
v = p / m

u_min = 0; u_max = 2; num_u_points = 100  # for u = 1, r = r(0) * 2.71812
u = np.linspace( u_min, u_max, num_u_points )
r_an = [ beam_radius( x, r_0 ) for x in u ]
r_an_upper = r_an + beam_axis_x_pos
r_an_lower = beam_axis_x_pos - r_an
z_an = [ beam_z( x, m = m, v = v, q = q, I = I, r_0 = r_0 ) for x in u ]
z_an = z_an + emitter_z_pos

r_num = h5file["/ParticleSources/cathode_emitter/position_x"]
z_num = h5file["/ParticleSources/cathode_emitter/position_z"]

z_volume_size = h5file["/SpatialMesh"].attrs["z_volume_size"]
x_volume_size = h5file["/SpatialMesh"].attrs["x_volume_size"]
plt.xlabel( "Z [cm]" )
plt.ylabel( "X [cm]" )
plt.ylim( 0, x_volume_size )
plt.xlim( 0, z_volume_size )
plt.plot( z_num, r_num, '.', label = "num" )
plt.plot( z_an, r_an_upper, label = "theory", color = "g" )
plt.plot( z_an, r_an_lower, color = "g" )
plt.legend()
h5file.close()

plt.savefig( "beam_contour_jup.png" )
plt.show()
```

```python

```


# Stereo Geometry

This notebook visualizes the geometry between two views called epipolar geometry.

**Subjects are covered:**
1. **Definitions of epipolar geometry, the Fundamental Matrix, and the Essential Matrix.**
2. **Visualizing epipolar geometry.**
3. **8 point algorithm for computing the Fundamental matrix.**
4. **Deriving relative camera poses from the essential matrix.**
5. **Conclusion**
6. **Sources**

Section 1 is meant as a theoretical underpinning of the methods used. It can be skipped by readers only interested in the algorithm's implementation. Many of the descriptions in this section I have taken, sometimes verbatim, from [Multiple View Geometry in Computer Vision](https://www.robots.ox.ac.uk/~vgg/hzbook/) by Richard Hartley and Andrew Zisserman, [epipolar geometry Wikipedia articles](https://en.wikipedia.org/wiki/Essential_matrix), and [Computer Vision Algorithms and Applications](https://szeliski.org/Book/) by Richard Szeliski.

**Why are we interested in these subjects?**
Structure and depth are inherently ambiguous from single views.
Understanding stereo geometry will allow us to overcome this ambiguity and estimate many useful values, such as:
* **The 3D structure of a scene.** Given calibrated cameras, the 3D reconstruction can be known up to a similarity transform.
    Additionally, if a landmark with known dimensions and pose is observed, a Euclidean reconstruction can be computed.
    This allows for applications such as:
    * measuring objects in the scene (photogrammetry)
    * collision avoidance for autonomous robots (depth perception)
    * inserting objects into a scene (virtual reality)
    * superimposing textures onto the scene (virtual reality)
    * creating 3D models of real-world objects (metric reconstruction).
* **The relative pose of a camera.** Knowing the relative pose will allow for a more targeted search of point correspondences. Stereo geometry provides closed-form solutions for calculating relative camera poses from point correspondences. Also, when performing camera pose estimation with iterative non-linear methods (e.g. Levenberg Marquardt in notebook 2), it is crucial to have an initial pose estimate.

**Further reading**
For a comprehensive treatment of epipolar geometry, see chapters nine to eleven of the textbook [Multiple view geometry in computer vision](https://www.robots.ox.ac.uk/~vgg/hzbook/) by Richard Hartley and Andrew Zisserman. For this subject, I would not recommend video lectures. The notations in many lectures was inconsistent, and I eventually found it was best understood through reading and drawing.

# Epipolar Geometry
Epipolar geometry is the intrinsic geometry between two views. It is independent of scene structure and only depends on the cameras' internal parameters and relative pose.
This geometry is essentially the geometry of the intersection of the two camera image planes with a pencil of planes having the baseline as axis. Here, the baseline is the line joining the camera centers. (images taken from [Multiple View Geometry in Computer Vision](https://www.robots.ox.ac.uk/~vgg/hzbook/)).

<center>

</center>

$ \mathbf {C,C}'$ - The camera projection centers
$ \mathbf {X}$ - A 3D point
$ \mathbf {x,x'}$ - The 2D images of $X$
$ \mathbf {\pi}$ - The epipolar plane
$ \mathbf {e, e'}$ - The epipoles
$ \mathbf {l, l'}$ - The epipolar lines
$ \mathbf {P, P'}$ - The camera matrices

* The epipole is the point of intersection of the line joining the camera centers (the baseline) with the image plane.
* Equivalently, the epipole is the image in one view of the other view's camera center.
* $ \mathbf {X,x,x',C, C'}$ are coplanar. This plane is denoted as $ \mathbf{\pi}$. The rays back-projected from $ \mathbf {x}$ and $ \mathbf {x'}$ intersect at $ \mathbf {X}$, and also lie in $ \mathbf {\pi}$.
* The epipolar plane must contain the baseline. Therefore, there is a one-parameter family (a pencil) of epipolar planes.
* The epipolar line is the intersection of an epipolar plane with the image plane.
* All epipolar lines intersect at the epipole.
* An epipolar plane defines the correspondence between the two epipolar lines.
* A segment of the epipolar line can be found by projecting the other camera's back-projected ray of the image of $ \mathbf {X}$.
* For each point $ \mathbf {x}$ in one image, there exists a corresponding epipolar line $ \mathbf {l'}$ in the other image. Any point $ \mathbf {x'}$ in the second image matching point $ \mathbf {x}$ must lie on the epipolar line $ \mathbf {l'}$.
* **Recap.** In homogenous coordinates, the cross product between two points is the line passing through those points. The cross product of two lines is the point where they intersect. The dot product of a line and a point on that line is always $0$.

# The Fundamental Matrix
The fundamental matrix is the algebraic representation of epipolar geometry. That is the above variables and their interrelations are captured in this matrix.

Mostly simply:

An epipolar line is a projection in the second image of the ray from the point $ \mathbf {x}$ through the camera center $ \mathbf {C}$ of the first camera. Thus, there is a map

$$ \mathbf {x \rightarrow l'}$$
This mapping is a projective mapping from points to lines, which can be represented by the Fundamental matrix $\mathbf{F}$.

$$ \mathbf{l'} = \mathbf{Fx} $$

Because each corresponding point $\mathbf{x'}$ lies on $\mathbf{l'}$ and has a dot product of 0, $\mathbf{F}$ defines the constraint

$$ \mathbf{x'} ^{\mathbf{T}} \mathbf{F} \mathbf{x} = 0$$

### Algebraic Derivation of the Fundamental Matrix
This derivation is taken from the book Multiple View Geometry in Computer Vision.

The ray back-projected from $\mathbf{x}$ is obtained by solving $\mathbf{PX = x}$. The one-parameter family of solutions is of the form

$$ \mathbf{X}(\lambda) = \mathbf{P^{+} x} + \lambda \mathbf{C}$$

Keep in mind these are homogenous coordinates, meaning the last coordinate is rescaled to 1. Essentially, we are choosing a point on a line segment between camera center $\mathbf{C}$ and 3D point $\mathbf{P^{+} x}$. In particular, two 3D points on the ray are $\mathbf{P^{+} x}$ at $\lambda = 0$, and $\mathbf{C}$ at $\lambda = \infty$. Here $\mathbf{P^{+}}$ is the psuedo-inverse of $\mathbf{P}$, and $\mathbf{C}$ is its null-vector, namely the camera center, defined by $\mathbf{PC}=\mathbf{0}$. These two points are imaged by the second camera $\mathbf{P'}$ at $\mathbf{P'}\mathbf{P^{+} x}$ and $\mathbf{P'C}$ respectively. The epipolar line is the line joining these two projeted points, namely $\mathbf{l'} = (\mathbf{P'C})\times  (\mathbf{P'}\mathbf{P^{+} x})$. The point $\mathbf{P'C}$ is the epipole in the second image, namely the projection of the first camera center, and may be denoted by $\mathbf{e'}$. Thus, $\mathbf{l'} = [\mathbf{e'}]_{\times} (\mathbf{P'}\mathbf{P^{+}) x} = \mathbf{Fx} $, where $\mathbf{F}$ is the matrix

$$ \mathbf{F} =  [\mathbf{e'}]_{\times} (\mathbf{P'}\mathbf{P^{+})} $$

Thus formalizing $ \mathbf {x \rightarrow l'}$ to the equation

$$ \mathbf{l'} = \mathbf{Fx}$$

Given that any point $w$ on $\mathbf{l'}$ will have the dot product $w \cdot \mathbf{l'} = 0$, we can infer the following:
The fundamental matrix satisfies the condition that for any pair of corresponding points $\mathbf{x} \leftrightarrow \mathbf{x'}$ in the two images

$$ \mathbf{x'} ^{\mathbf{T}} \mathbf{F} \mathbf{x} = 0$$

The importance of the relation is that it gives a way of characterizing the fundamental matrix without reference to the camera matrices, i.e. only in terms of corresponding image points. This enables $\mathbf{F}$ to be computed from image correspondences alone. To compute matrix $\mathbf{F}$ at least $7$ correspondences are
required.

### Properties

* $\mathbf{F}$ is a $3 \times 3$ matrix
* $\mathbf{F}$ is rank 2
* If $\mathbf{F}$ is the fundamental matrix of the pair of cameras $(\mathbf{P}, \mathbf{P'})$, then
$\mathbf{F^{T}}$ is the fundamental matrix of the pair in the opposite order $(\mathbf{P'}, \mathbf{P})$.
* $ \mathbf{l'} = \mathbf{Fx}$
* $ \mathbf{l} = \mathbf{F^Tx'}$
* For any point $ \mathbf{x}$ (other than $ \mathbf{e}$) the epipolar line $ \mathbf{l'} = \mathbf{F} \mathbf{x}$
contains epipole $\mathbf{e'}$
* $\mathbf{e'^T} (\mathbf{Fx}) = (\mathbf{e'^TF})\mathbf{x} = 0$ for all $\mathbf{x}$.
* $\mathbf{e'}$ is the left null-vector of $\mathbf{F}$.
* $\mathbf{e}$ is the right null-vector of $\mathbf{F}$.
* $\mathbf{e'} = \mathbf{P'C}$
* $\mathbf{e} = \mathbf{PC'}$
* $\mathbf{Fe} = 0$
* $\mathbf{F^{T}e'} = 0$
* $\mathbf{l'} = \mathbf{F}[\mathbf{e}]_{\times}\mathbf{l}$
* $\mathbf{l} = \mathbf{F^T}[\mathbf{e'}]_{\times}\mathbf{l'}$

# The Essential Matrix
The essential matrix is the specialization of the fundamental matrix to the case of
normalized image coordinates. Historically, the essential matrix was introduced (by Longuet-Higgins) before the fundamental matrix, and the fundamental
matrix may be thought of as the generalization of the essential matrix in which the
(inessential) assumption of calibrated cameras is removed. The essential matrix is defined as

$$ \mathbf{E} = \mathbf{K^{'T}FK}$$

The relation between the essential matrix and corresponding points is essentially ( ;] ) the same as for the fundamental matrix.

$$ \mathbf{x'} ^{\mathbf{T}} \mathbf{F} \mathbf{x} = 0$$

$$ \mathbf{x'} \mathbf{K^{'-T}K^{'T}FKK^{-1}} \mathbf{x} = 0$$

$$ \mathbf{y'} \mathbf{K^{'T}FK} \mathbf{y} = 0 $$

$$ \mathbf{y'} \mathbf{E} \mathbf{y} = 0 $$

Where $\mathbf{y}$ and $\mathbf{y'}$ are the **normalized coordinates**. These can be thought of as the image plane coordinates for a camera with the identity matrix as camera intrinsics.

# Visualizing epipolar geometry

We will now visualize epipolar geometry by plotting the epipolar lines for stereo point correspondences.
For this demonstration, we use a stereo setup from notebook 1.

<center>

</center>

```python
from notebook_functions import (init_3d_plot,
                                plot_chessboard,
                                plot_camera_wireframe,
                                plot_picture,
                                project_points_to_picture,
                                object_points,
                                images,
                                get_stereo_setup_with_correspondences,
                                triangulate)
import ipyvolume as ipv
import numpy as np
import cv2
import matplotlib.pyplot as plt
```

```python
images, extrinsics, cam_centers, intrinsics, match_coords, object_points = get_stereo_setup_with_correspondences()

init_3d_plot()
plot_chessboard(object_points)
vis_scale = 5

for idx, (cam_center, extrinsic, image) in enumerate(zip(cam_centers, extrinsics, images)):
    inv_extrinsic = np.linalg.inv(extrinsic)
    inv_intrinsics = np.linalg.inv(intrinsics)
    plot_camera_wireframe(cam_center, vis_scale, inv_extrinsic)
    plot_picture(image, inv_extrinsic, vis_scale)

ipv.show()
```

## Calculate the fundamental matrix from camera matrices
We will calculate the fundamental matrix using  $ \mathbf{F} =  [\mathbf{e'}]_{\times} (\mathbf{P'}\mathbf{P^{+})} $ and $\mathbf{e'} = \mathbf{P'C}$.
Where $ [T_{\times}] = \begin{pmatrix} 0 & -t_z & t_y\\ t_z & 0 & -t_x\\ -t_y & t_x & 0 \end{pmatrix} $

```python
extrinsic_1, extrinsic_2 = extrinsics
extrinsic_1_orig = extrinsic_1
extrinsic_2_orig = extrinsic_2
cam_center_1, cam_center_2 = cam_centers
cam_center_1_orig = cam_center_1
cam_center_2_orig = cam_center_2
proj_1 = intrinsics[:3, :3] @ extrinsic_1[:3, :4]
proj_2 = intrinsics[:3, :3] @ extrinsic_2[:3, :4]
pinv_proj_1 = np.linalg.pinv(proj_1)
pinv_proj_2 = np.linalg.pinv(proj_2)
e_1 = proj_1 @ cam_center_2[:4]
e_2 = proj_2 @ cam_center_1[:4]
e_1 = e_1 / e_1[2]
e_2 = e_2 / e_2[2]
x, y, z = e_2
cross_e_2 = np.array([[ 0, -z,  y],
                      [ z,  0, -x],
                      [-y,  x,  0]])
F = cross_e_2 @ (proj_2 @ pinv_proj_1)

print(f'Rank of F is very near 2 (ignoring numerical rounding errors): {np.isclose(np.linalg.svd(F)[1][2], 0)}')
print(f'Fe≈0 and F trans e\'≈ 0: {np.all(F @ e_1 < 0.0001), np.all(F.T @ e_2 < 0.0001)}')
```

## Draw the epipolar lines

The epipolar lines will now be drawn using the equation $ \mathbf{l'} = \mathbf{Fx}$ and $ \mathbf{l} = \mathbf{F^Tx'}$.

```python
def draw_homogenous_line(image, line):
    """ Draws a line represented by homogenous coordinates onto an image.

    Args:
        image (np.ndarray): An image array.
        line (np.ndarray): A 2D line represented by homgenous coordinates.
    Returns:
        image (np.ndarray): The image with the line drawn on it.
    """
    height, width, _ = image.shape
    a, b, c = line
    y_at_x0 = int(-c/b)
    y_at_xwidth =  int((-a * width -  c)/b)
    image = cv2.line(image, (0, y_at_x0), (width, y_at_xwidth), [255, 255, 255], 5)
    return image

def plot_epipolar_lines(image_1, image_2, points_1, points_2, F):
    """ Plots point correspondences along with  epipolar lines.

    Args:
        image_{1|2} (np.ndarray): The stereo images.
        points_{1|2} (np.ndarray): The stereo point correspondences.
        F (no.ndarray): The fundamental matrix.
    """
    for point_1, point_2 in zip(points_1, points_2):
        line_1 = F.T @ point_2[:3]
        line_2 = F @ point_1[:3]
        image_1 = cv2.circle(image_1, point_1[:2].astype(int), 15, [255,255,255], -1)
        image_2 = cv2.circle(image_2, point_2[:2].astype(int), 15, [255,255,255], -1)
        image_1 = draw_homogenous_line(image_1, line_1)
        image_2 = draw_homogenous_line(image_2, line_2)

    image_concat = cv2.hconcat([image_1, image_2])
    image_concat = cv2.cvtColor(image_concat, cv2.COLOR_BGR2RGB)
    plt.figure(figsize=(20,20))
    plt.imshow(image_concat)
    plt.show()
```

```python
im_1_points = match_coords[0]
im_2_points = match_coords[1]
image_1 = images[0]
image_2 = images[1]
plot_epipolar_lines(image_1.copy(), image_2.copy(), im_1_points, im_2_points, F)
```

# 8 point algorithm - Fundamental matrix from point correspondences

The 8 point algorithm is a straightforward approach for calculating the Fundamental matrix $\mathbf{F}$.

1. **Points**. Find $8$ or more stereo point correspondences. Here we will use SIFT features.
2. **Linear solution**. Use an SVD to find the least-squares solution to $\mathbf{F}$ given the $8$ or more correspondence constraints $\mathbf{x'} ^{\mathbf{T}} \mathbf{F} \mathbf{x} = 0$. That is, for SVD$= \mathbf {U\Sigma V^{T}}$, take the last column of $\mathbf{V}$ as $\mathbf{F}$.*
3. **Enforce Rank 2**. For noisy data, the linear solution $\mathbf{F}$ will not be of rank $2$ in general. To enforce this, use the SVD to perform a PCA and keep only the first $2$ principal axes when reconstructing $\mathbf{F}$.

*-See the appendix for an intuitive explanation and proof of why:
* The Singular Value Decomposition can be used to find a least-squares solution.
* The rank of the Fundamental matrix must be $2$.

### Constraint matrix

$
\begin{align}
\mathbf{x'} ^{\mathbf{T}} \mathbf{F} \mathbf{x} &= 0 \\\\
\begin{bmatrix}
x' & y' & w'
\end{bmatrix}
\begin{bmatrix}
f_1 & f_2 & f_3 \\
f_4 & f_5 & f_6 \\
f_7 & f_8 & f_9
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
w
\end{bmatrix} &= 0  \\\\
x' x f_1 + x' y f_2 + x' w f_3 \\
+ y' x f_4 + y' y f_5 + y' w f_6 \\
+ w' x f_7 + w' y f_8 + w' w f_9 &= 0 \\\\
\begin{bmatrix}
x'x & x'y & x'w & y'x & y'y & y'w & w'x & w'y & w'w
\end{bmatrix}
\begin{bmatrix}
f_1 \\ f_2 \\ f_3 \\ f_4 \\ f_5 \\ f_6 \\ f_7 \\ f_8 \\ f_9
\end{bmatrix} &= \begin{bmatrix} 0 \end{bmatrix}
\end{align}
$

The last equation is a linear homogeneous equation of form $\mathbf{A}b = 0$. Adding $7$ more constraints to $\mathbf{A}$ allows us to compute $b$. In this case, $b$ is the rolled-out fundamental matrix $\mathbf{F}$.

```python
# Linear solution
constraint_matrix = list()
for point_1, point_2 in zip(im_1_points, im_2_points):
    x1, y1, w1, _ = point_1
    x2, y2, w2, _ = point_2
    constraint = [x2*x1, x2*y1, x2*w1, y2*x1, y2*y1, y2*w1, w2*x1, w2*y1, w2*w1]
    constraint_matrix.append(constraint)
constraint_matrix = np.array(constraint_matrix)
u, s, v_t = np.linalg.svd(constraint_matrix)
least_squares = v_t[-1] # last column of V
F = least_squares.reshape((3, 3))

# Enforce rank 2
u, s, v_t = np.linalg.svd(F)
s = np.diag(s)
s[2, 2] = 0
F = u @ s @ v_t
plot_epipolar_lines(image_1.copy(), image_2.copy(), im_1_points, im_2_points, F)
```

# Deriving relative camera pose from the essential matrix.

For a proof of this result refer to 9.6.2 in Multiple View Geometry in Computer Vision and [this Wikipedia article](https://en.wikipedia.org/wiki/Essential_matrix#Extracting_rotation_and_translation). The proof is out-of-scope and would require too much space.

For a given essential matrix $\mathbf{E}$ with SVD decomposition $\mathbf{E} = \mathbf{U} \text{diag}(1,1,0)\mathbf{V^T}$ and first camera matrix $\mathbf{P} = [\mathbf{I} | \mathbf{0}]$, there are our possible choices for the second camera $\mathbf{P'}$, namely

$$ \mathbf{P'} = [\mathbf{UWV^T} | +\mathbf{u_3}] \hspace{0.5em} \text{ or } \hspace{0.5em} [\mathbf{UWV^T} | -\mathbf{u_3}] \hspace{0.5em} \text{ or } \hspace{0.5em} [\mathbf{UW^TV^T} | +\mathbf{u_3}] \hspace{0.5em} \text{ or } \hspace{0.5em} [\mathbf{UW^TV^T} | -\mathbf{u_3}]$$

Where $\mathbf{W} = \begin{bmatrix} 0 & -1 & 0 \\
                                    1 &  0 & 0 \\
                                    0 &  0 & 1  \end{bmatrix} $

This 4 solution ambiguity can be reduced to a single solution. This is the solution that places all points in front of the cameras after triangulation.
We compute all 4 solutions and check which solution has points with only positive z coordinates relative to the camera.

```python
def decompose_essential(essential):
    """ Decomposes an essential matrix into a relative camera pose.

    Args:
        essential (np.ndarray): The essential matrix.
    Returns:
        pose (np.ndarray): The extrinsic matrix of the second camera.
    """
    w = np.array([[0, -1, 0],
                  [1,  0, 0],
                  [0,  0, 1]])
    u, s, v_t = np.linalg.svd(essential)

    # Translation hypothesis
    u_3 = u[:, 2, np.newaxis]

    # Rotation hypothesis
    rot_a = u @ w @ v_t
    rot_b = u @ w.T @ v_t
    rot_a = -rot_a if np.linalg.det(rot_a) < 0 else rot_a
    rot_b = -rot_b if np.linalg.det(rot_b) < 0 else rot_b

    # 4 motion hypothoses as extrinsic matrices
    extr_2_a = np.hstack([rot_a, u_3])
    extr_2_b = np.hstack([rot_a, -u_3])
    extr_2_c = np.hstack([rot_b, u_3])
    extr_2_d = np.hstack([rot_b, -u_3])

    # Assume cam 1  = [I | 0]
    extr_1 = np.array([[1, 0, 0, 0],
                       [0, 1, 0, 0],
                       [0, 0, 1, 0]])
    cam_center_1 = np.zeros(3)
    inv_intrinsics = np.linalg.inv(intrinsics[:3, :3])
    inv_extrinsic_1 = np.linalg.pinv(extr_1)
    vec_in_cam_ref_1 = inv_intrinsics @ im_1_points.T[:3]
    vec_in_world_1 = inv_extrinsic_1 @ vec_in_cam_ref_1
    pose = None

    # Triangulate points for each of the 4 solutions
    for extr_2 in [extr_2_a, extr_2_b, extr_2_c, extr_2_d]:
        cam_center_2 = -extr_2[:3, :3] @ extr_2[:, 3].T
        inv_extrinsic_2 = np.linalg.pinv(extr_2)
        vec_in_cam_ref_2 = inv_intrinsics @ im_2_points.T[:3]
        vec_in_world_2 = inv_extrinsic_2 @ vec_in_cam_ref_2
        rel_zs = np.array([])
        for vec_1, vec_2 in zip(vec_in_world_1.T, vec_in_world_2.T):
            triangulated, _, _ = triangulate(cam_center_1, vec_1, cam_center_2, vec_2)
            triangulated = np.append(triangulated, 1)
            x1, y1, z1 = extr_1 @ triangulated # Triangulated point in camera 1 ref
            x2, y2, z2 = extr_2 @ triangulated # Triangulated point in camera 2 ref
            rel_zs = np.append(rel_zs, [z1, z2])
        if (rel_zs > 0).all():
            pose = extr_2

    return pose
```

# Visualize relative pose inferred from the essential matrix
The ground truth pose of the second camera is shown in red.
The inferred pose of the second camera is shown in blue.
As can be seen, the pose can be very decently inferred from point correspondences.
The careful reader will notice that the relative translation is artificially scaled by a "magic" number.
This is because the essential matrix has no information on the scale of the relative camera translation.
For details on why this is, see [this post](https://stackoverflow.com/questions/69742520/extracting-the-scale-of-translation-vector-that-i-got-from-the-essential-matrix/69980810#69980810).
We do however know the scale of the scene due to the chessboard calibration object, which is how I manually found the translation scale 8.6.

```python
init_3d_plot()
plot_chessboard(object_points)
plot_camera_wireframe(cam_center_2_orig, vis_scale, np.linalg.inv(extrinsic_2_orig), color='red')

essential = intrinsics[:3, :3].T @ F @ intrinsics[:3, :3]
rel_extrinsic_2 = decompose_essential(essential)

# Convert relative pose from essential matrix decomposition to absolute poses.
rel_extrinsic_2[:, 3] *= 8.6
extrinsic_2 = rel_extrinsic_2 @ extrinsic_1
cam_center_2 = -extrinsic_2[:3, :3].T @ extrinsic_2[:3, 3]
extrinsic_2 = np.vstack([extrinsic_2,  [0, 0, 0, 1]])
extrinsics[1] = extrinsic_2
cam_centers[1] = cam_center_2

for idx, (cam_center, extrinsic, image) in enumerate(zip(cam_centers, extrinsics, images)):
    inv_extrinsic = np.linalg.pinv(extrinsic)
    inv_intrinsics = np.linalg.inv(intrinsics)
    plot_camera_wireframe(cam_center, vis_scale, inv_extrinsic)
    plot_picture(image, inv_extrinsic, vis_scale)

ipv.show()
```

## Conclusion
In this notebook we
* visualized the epipolar lines for stereo point correspondences
* computed the fundamental matrix using the 8 point algorithm
* decomposed the essential matrix into a relative pose of a camera

Having an intuition of how epipolar geometry works will serve us well.
It is both a tool for problem-solving and a theoretical basis for many computer vision algorithms.

## Sources

1. [Szeliski, Richard. "Computer vision: algorithms and applications." Springer Science & Business Media, 2010](https://szeliski.org/Book/)
2. [Andrew, Alex M. "Multiple view geometry in computer vision." Kybernetes, 2001](https://www.robots.ox.ac.uk/~vgg/hzbook/)

## Appendix

#### Why does the SVD contain the least-squares solution?
This section gives an intuitive proof of why for a homogenous equation (or data) matrix $\mathbf{A}$ in $\mathbf{A}b = 0$, the last column of the SVD's $\mathbf{V}$ matrix is the least-squares solution.
For a homogenous least-squares problem, we are interested in finding a parameter vector $b$ such that

$$ \mathbf{A}b = 0 $$

subject to the constraint $ \lVert b \rVert ^{2}=1$ to avoid the arbitrary solution. Because there may be no exact solution, this is phrased as a minimization problem.

$$ \arg \underset{b}{\min}  \lVert \mathbf{A}b \rVert^2_2 $$

The SVD decomposes a matrix into two orthonormal matrices $ \mathbf {U, V^{T}} $ and one diagonal matrix with non-negative real numbers $\mathbf{\Sigma}$. Interpreting the SVD as a PCA, $\mathbf{V}$ encodes the principal axes of the data and $\mathbf{\Sigma}$ the amount of variance of the data when projected onto these principal axes in descending order, i.e. $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_n$(for details on the SVD $\leftrightarrow$PCA connection, see [this Princeton tutorial](https://www.cs.princeton.edu/courses/archive/spring12/cos598C/svdchapter.pdf)).

$$  \mathbf{A} \rightarrow \mathbf {U\Sigma V^{T}} $$

Matrix multiplication with an orthonormal matrix does not affect the norm of a vector, thus we can remove $\mathbf{U}$ from the equation.

$$ \arg \underset{b}{\min} \lVert \mathbf {U \Sigma V^{T}}b \rVert =  \arg \underset{b}{\min} \lVert \mathbf {\Sigma V^{T}}b \rVert$$

We further substitute with $y = \mathbf{V^{T}} b$, with constraint $ \lVert y \rVert ^{2}=1$ due to orthonormality of $\mathbf{V^{T}}$.

$$ \arg \underset{b}{\min} \lVert \mathbf {\Sigma V^{T}}b \rVert =arg \underset{y}{\min} \lVert \mathbf {\Sigma} y \rVert $$

Because the singular values $\sigma$ in $\mathbf{\Sigma}$ encode the variance in descending order, the optimal $y$ is trivially $(0,\dots,0,1)^T$.
$$ y =  (0,\dots,0,1)^T$$
$$ \mathbf{V^{T}} b = (0,\dots,0,1)^T$$
$$ \mathbf{V V^{T}}b = \mathbf{V} (0,\dots,0,1)^T $$
$$ b = \mathbf{V} (0,\dots,0,1)^T $$
$$ b = \text{last column of } \mathbf{V} $$
making the  optimal  $b$ the last column of  $\mathbf{V}$.

## Why is the fundamental matrix rank 2?
This proof was taken from [this Quora post by Samarth Brahmbhatt](https://www.quora.com/Why-is-the-fundamental-matrix-in-computer-vision-rank-2). Additional references to other proofs have been added where needed.

#### Intuition
A fundamental matrix is given by the equation $ \mathbf{x'}^T \mathbf{F} \mathbf{x} = 0 $. Now consider an epipolar line $ l' = \mathbf{F} \mathbf{x} $. The right epipole $  \mathbf{e'}  $ lies on this line, so $  \mathbf{e'}^T l' = 0 $ or $  \mathbf{e'} ^T \mathbf{F} \mathbf{x} = 0 $ for all $ \mathbf{x} $. This implies that $  \mathbf{e'} ^T \mathbf{F} = 0 $. Similarly, one can prove that $ \mathbf{F}  \mathbf{e}  = 0 $. Hence $ \mathbf{F} $ has a null space which is not just the zero vector. So $ \mathbf{F} $ is rank deficient.

#### Proof
The proof that $ \mathbf{F} $ has of rank exactly 2 comes from the fact that $ \mathbf{F} $ is constructed from the essential matrix $ \mathbf{E}$:

$$ \mathbf{F} = (\mathbf{K'^{-1}})^T \mathbf{E} \mathbf{K^{-1}} $$  where the $ \mathbf{K} $'s are intrinsic matrices of the two cameras. Now, $ \mathbf{E} = [T_{\times}]R $ where $ R $ is the rotation matrix relating the two camera co-ordinate systems and $ [T_{\times}] = \begin{pmatrix} 0 & -t_z & t_y\\ t_z & 0 & -t_x\\ -t_y & t_x & 0 \end{pmatrix} $. For a proof of this identity, see the [wikipedia page](https://en.wikipedia.org/wiki/Essential_matrix#Derivation_and_definition). A little bit of manipulation will show that one column of $ [T_{\times}] $ is a linear combination of the other two columns. So $ [T_{\times}] $ has rank 2. For a proof of this see [this row reduced solution](https://www.wolframalpha.com/input/?i2d=true&i=row+echelon+form+of+%7B%7B0%2C-z%2Cy%7D%2C%7Bz%2C0%2C-x%7D%2C%7B-y%2Cx%2C0%7D%7D).

Hence any matrix that you construct by multiplying other matrices with $ [T_{\times}] $ (such as $ \mathbf{E} $ and $ \mathbf{F} $) will also have at most rank 2.


```python
from sympy import symbols, expand, factor, solve, latex, Rational, init_printing
init_printing()  # Activates pretty printing
```

```python
x, y = symbols('x y')
expr = 3 * x **2 + 6 * x + 20
expr
```

```python
import matplotlib.pyplot as plt
import numpy as np

x = np.array(range(-50,50,1))
y = 3 * x **2 + 6 * x + 20

fig, ax = plt.subplots()

ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')

ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0)) # set position of x spine to x=0

ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))   # set position of y spine to y=0

ax.set_xlim([-10,10])
ax.set_ylim([-1,100])

ax.set_xticks(range(-10,10,1))
ax.set_title(r"$\bf "+latex(expr)+"$", size=20)

ax.grid(True)
ax.plot(x,y)
plt.show()
```