>>> Work in Progress (Following are the lecture notes of Prof Andrew Ng/Kian Katanforoosh - CS229 - Stanford. This is my interpretation of his excellent teaching and I take full responsibility of any misinterpretation/misinformation provided herein.)

## Lecture 12

#### Outline
- Logistic Regression as NN (last lecture)
- Neural Network (last lecture)
  - Backpropagation (this lecture)
- Improving your NN (this lecture)

### Backpropagation
- in order to define optimization problem and find right parameters, define a cost function
- forward propagating m examples at a time
  - cost function is the average of loss function
  > $J(\hat{y}, y) = \frac{1}{m}\sum\limits_{i=1}^{m}L^{(i)}$
  - where
  > $L^{(i)} = -[y^{(i)}\log\hat{y}^{(i)} + (1-y^{(i)})\log(1-\hat{y}^{(i)}) ]$
  - the parameters $W^{[1]}, b^{[1]}, W^{[2]}, b^{[2]}, W^{[3]}, b^{[3]}$
    - the square bracket denotes the layer
    - we need to train all these parameters
- backward propagation
  - here we start from back and move forward
  - we compute derivative of $W^{[3]}$ and $b^{[3]}$, then $W^{[2]}$ and $b^{[2]}$ and finally $W^{[1]}$ and $b^{[1]}$
  > $w^{[l]} = w^{[l]} - \alpha\frac{\partial L}{\partial w^{[l]}}$

#### Derivative of loss function wrt $W^{[3]}$
- Lets compute derivation of loss function wrt to $W^{[3]}$, then we can compute cost function as it will be summation over all training example
> $\frac{\partial L}{\partial W^{[3]}} = - [y^{(i)} \frac{\partial}{\partial W^{[3]}} (\log\sigma(W^{[3]} a^{[2]} + b^{[3]}))  +  (1-y^{(i)}) \frac{\partial}{\partial W^{[3]}} (\log(1-\sigma(W^{[3]} a^{[2]} + b^{[3]})))]$

- The reason we have $a^{[2]}$ in this equation is because  $a^{[2]}$ is one of the parameter and input for $Z^{[3]}$  activation node

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

#### Breaking down derivative
- There is a composition of three functions here:
  - logarithm
  - sigmoid
  - linear function
- Derivative of log function
> $(\log x)' = \frac{1}{x}$
- Derivative of sigmoid function $\sigma(x) = \frac{1}{1+e^{-x}}$
> $\sigma'(x) = \sigma(x)(1-\sigma(x))$
- Derivative of log of sigmoid function $\log\sigma(..)$ wrt $W^{[3]}$
> $\frac{\partial \log\sigma(..)}{\partial w} = \frac{1}{\log\sigma(..)} \frac{\partial\sigma(..)}{W^{[3]}}$
- Derivative of log of sigmoid function $\log\sigma(Z^{[3]})$ wrt $W^{[3]}$
> $\begin{equation}\\
\begin{aligned}\\
\frac{\partial \log\sigma(Z^{[3]})}{\partial W^{[3]}} &= \frac{1}{\sigma(Z^{[3]})} \frac{\partial\sigma(Z^{[3]})}{\partial W^{[3]}}\\
&= \frac{1}{a^{[3]}} \frac{\partial\sigma(W^{[3]}a^{[2]} + b^{[3]} )}{\partial W^{[3]}}\\
\end{aligned}\\
\end{equation}$
  - Applying derivative of sigmoid function from above
> $\begin{equation}\\
\begin{aligned}\\
&= \frac{1}{a^{[3]}}.{a^{[3]}(1-a^{[3]})}. \frac{\partial(W^{[3]}a^{[2]} + b^{[3]} )}{\partial W^{[3]}}\\
\end{aligned}\\
\end{equation}$
- Derivative of linear function $Z^{[3]}$ wrt $W^{[3]}$
> $\frac{\partial}{\partial W^{[3]}}(W^{[3]}a^{[2]} + b^{[3]}) = a^{[2]T}$
  - Matrix shape of linear function $(W^{[3]}a^{[2]} + b^{[3]})$ is (1,1)
  - Matrix shape of $W^{[3]}$ is (1,2) because it is connected to 2 neurons
    - So the matrix shape of whole derivative will be the shape of $W^{[3]}$ i.e., (1,2)
    - because we are taking the derivative of scalar (numerator) wrt higher dimension matrix, i.e., row vector(denominator) here.
  - if not in matrix form, derivative of linear function would be $a^{[2]}$. But the shape of $a^{[2]}$ is (2,1) which will not match with the overall derivative shape which is (1,2). So the result should be in $a^{[2]T}$ which will have the same shape as $W^{[3]}$
    - (1,2)
- Derivative of log of sigmoid function $\log\sigma(Z^{[3]})$ wrt $W^{[3]}$ (contd from above)
> $\begin{equation}\\
\begin{aligned}\\
\frac{\partial \log\sigma(Z^{[3]})}{\partial W^{[3]}} &= \frac{1}{a^{[3]}}.{a^{[3]}(1-a^{[3]})}.  \frac{\partial(W^{[3]}a^{[2]} + b^{[3]} )}{\partial W^{[3]}}\\
&= \frac{1}{a^{[3]}}.{a^{[3]}(1-a^{[3]})}.  a^{[2]T}\\
\end{aligned}\\
\end{equation}$
- Similarly derivative of 2nd log piece of sigmoid function $(1-\log\sigma(Z^{[3]}))$ wrt $W^{[3]}$
> $\begin{equation}\\
\begin{aligned}\\
\frac{\partial (1-\log\sigma(Z^{[3]}))}{\partial W^{[3]}} &= \frac{1}{1-a^{[3]}}.(-1){a^{[3]}(1-a^{[3]})}.  a^{[2]T}\\
\end{aligned}\\
\end{equation}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

- (Continuing from above) derivative of loss function wrt to $W^{[3]}$
> $\begin{equation}\\
\begin{aligned}\\
\frac{\partial L}{\partial W^{[3]}} &= - [y^{(i)} \frac{\partial}{\partial W^{[3]}} (\log\sigma(W^{[3]} a^{[2]} + b^{[3]})) + (1-y^{(i)}) \frac{\partial}{\partial W^{[3]}} (\log(1-\sigma(W^{[3]} a^{[2]} + b^{[3]})))]\\
&= - [y^{(i)} \frac{1}{a^{[3]}}.{a^{[3]}(1-a^{[3]})}. a^{[2]T} + (1-y^{(i)}) \frac{1}{1-a^{[3]}}.(-1){a^{[3]}(1-a^{[3]})}. a^{[2]T}]\\
&= -[y^{(i)}(1-a^{[3]})a^{[2]T} -  (1-y^{(i)})a^{[3]}a^{[2]T}]\\
&= -[y^{(i)}a^{[2]T} - a^{[3]}a^{[2]T}]\\
&= -(y^{(i)} - a^{[3]})a^{[2]T}\\
\end{aligned}\\
\end{equation}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

#### Derivative of cost function wrt $W^{[3]}$
> $\frac{\partial J}{\partial W^{[3]}} = -\frac{1}{m}\sum\limits_{i=1}^{m}(y^{(i)} - a^{[3]})a^{[2]T}$

#### Derivative of loss function wrt $W^{[2]}$

> $\frac{\partial L}{\partial W^{[2]}} = \frac{\partial L}{\partial a^{[3]}} \frac{\partial a^{[3]}}{\partial Z^{[3]}} \frac{\partial Z^{[3]}}{\partial a^{[2]}} \frac{\partial a^{[2]}}{\partial Z^{[2]}} \frac{\partial Z^{[2]}}{\partial W^{[2]}} $

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

#### Backpropagation using chain rule
- What is the thought process of writing chain rule?
  - We need to take derivative wrt variables thorough which errors will backpropagate. We dont want errors to get stuck. We need to go through variables that are connected to each other.
  - $\hat{y}$ also called $a^{[3]}$ is the first thing that is connected to loss function
  - $\frac{\partial L}{\partial a^{[3]}}$ - the output neuron is directly connected to loss function
  - $\frac{\partial a^{[3]}}{\partial Z^{[3]}}$ - $a^{[3]}$ which is the output activation of the last neuron is connected to the linear part of the last neuron which is $Z^{[3]}$
    - which will be derivative of sigmoid ($\sigma'(x)$) because $a^{[3]} = \sigma({Z^{[3]}})$ and $\sigma'(x) = \sigma(x)(1-\sigma(x))$
  - $\frac{\partial Z^{[3]}}{\partial a^{[2]}}$ - We know $Z^{[3]} = W^{[3]}a^{[2]} + b^{[3]}$
    - which path should we take in order to back propagate?
      - Not $W^{[3]}$, as we will be stuck
      - Not $b^{[3]}$, as we will be stuck
      - we should derivate wrt $a^{[2]}$, as it is connected to $Z^{[2]}$ and $Z^{[2]}$ is connected to $a^{[1]}$, and then we can backpropagate using this path
  - $\frac{\partial a^{[2]}}{\partial Z^{[2]}}$ - from above
  - $\frac{\partial Z^{[2]}}{\partial W^{[2]}}$ - from above

#### How to use the derivative of loss function wrt $W^{[3]}$ in derivative of loss function wrt $W^{[2]}$
- Term 1 and 2
  - We calculated earlier
  > $\frac{\partial L}{\partial W^{3}} = -(y^{(i)} - a^{[3]}) = (a^{[3]} - y^{(i)})$
  - But
  > $\frac{\partial L}{\partial W^{3}} = \frac{\partial L}{\partial a^{3}} \frac{\partial a^{3}}{\partial W^{3}} $
  - So
  > $\frac{\partial L}{\partial a^{3}} \frac{\partial a^{3}}{\partial W^{3}} = (a^{[3]} - y^{(i)})$
- Term 3
  - We calculated earlier
  > $Z^{[3]} = W^{[3]}a^{[2]}+b^{[3]}$
  - So
  > $\frac{\partial Z^{[3]}}{\partial a^{[2]}} = W^{[3]T}$
  - Why transpose?
    - Size of $Z^{[3]}$ - (1,1)
    - Size of $a^{[2]}$ - (2,1)
    - Size of $W^{[3]}$ - (1,2) - so we need to transpose this - $W^{[3]T}$
- Term 4
  - This is the derivative of sigmoid ($\sigma'(x)$) because $a^{[2]} = \sigma({Z^{[2]}})$ and $\sigma'(x) = \sigma(x)(1-\sigma(x))$
  > $\frac{\partial a^{[2]}}{\partial Z^{[2]}} = a^{[2]}(1 - a^{[2]})$
    - sigmoid times 1 minus sigmoid
- Term 5
  - As calculated earlier, derivative of linear function $Z^{[3]}$ wrt $W^{[3]}$
  > $\frac{\partial Z^{[3]}}{\partial W^{[3]}} = \frac{\partial}{\partial W^{[3]}}(W^{[3]}a^{[2]} + b^{[3]}) = a^{[2]T}$
  - So derivative of linear function $Z^{[2]}$ wrt $W^{[2]}$
  > $\frac{\partial Z^{[2]}}{\partial W^{[2]}} = \frac{\partial}{\partial W^{[2]}}(W^{[2]}a^{[1]} + b^{[2]}) = a^{[1]T}$

#### Derivative of loss function wrt $W^{[2]}$ (contd)
> $\begin{equation}\\
\begin{aligned}\\
\frac{\partial L}{\partial W^{[2]}} &= \frac{\partial L}{\partial a^{[3]}} \frac{\partial a^{[3]}}{\partial Z^{[3]}} \frac{\partial Z^{[3]}}{\partial a^{[2]}} \frac{\partial a^{[2]}}{\partial Z^{[2]}} \frac{\partial Z^{[2]}}{\partial W^{[2]}}\\
&= (a^{[3]} - y^{(i)}).W^{[3]T}.a^{[2]}(1 - a^{[2]}).a^{[1]T}\\
\end{aligned}\\
\end{equation}$
- Size of matrix:
> $(2,3) = (1,1).(2,1).(2,1).(1,3)$
  - This is not correct
- For derivative of sigmoid we use element wise product (Details in the note)
  > $W^{[3]T}*a^{[2]}(1 - a^{[2]})$
- Rearranging elements we get
  > $\{\{\{W^{[3]T}*a^{[2]}(1 - a^{[2]})\}.(a^{[3]} - y^{(i)})\}.a^{[1]T}\}$
- Matrix size
  > $\{\{\{(2,1)*(2,1)\}.(1,1)\}.(1,3)\}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

- The results must be cached during forward propagation so that it gets reused during backward propagation step
- It results in computational efficiency

### Improving Neural Networks
- Use different activation functions
  - One of the activation function we used was sigmoid function
    - Sigmoid $\sigma(z) = \frac{1}{1+e^{-z}}$
      - this ranges from 0 to 1
      - adv: it is used for classification as probability
      - disadv: if you have high or low z's (activation), the gradient is very close to 0
        - __it works very well in the linear regime but has trouble working in saturating regimes because the network doesn't update parameters properly__
        - __What is the problem with low gradients/saturating regimes?__
          - when we backpropagate, if z we cached was big, the gradient will be very small and it will be super hard to update parameters that are early in the network because the gradient will be negligible
    - $\sigma'(x) = \sigma(x)(1-\sigma(x))$
  - Other activation function we used was ReLU
    - ReLU(z) = $\begin{equation}
    \begin{cases}
      0 & \text{if } z \le 0\\
      z & \text{if } z \gt 0\\
    \end{cases} \end{equation}$
      - adv: if z is very big in positives, there is no saturation, the gradient will be 1
        - In house prediction problem, it makes sense to use ReLU compared to sigmoid or tanh, because the value is within 0 to $+\infty$
    - ReLU'(z) = $\mathbb 1\{z \gt 0\}$
  - one other commonly used activation function is tanh
    - tanh(z) = $\frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}$
      - this ranges from -1 to 1
      - adv: this is used in problems where there is positive reward or negative reward like in reinforcement learning
      - disadv: same as sigmoid
    - tanh'(z) = 1 - tanh(z)$^{2}$

#### Why do we need activation functions
- Lets say we use identity function as activation function $z \rightarrowtail z$
- Lets derive forward propagation

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

- from this we see that we need activation functions
- otherwise no matter how deep your network is, it will be equivalent to linear regression
- generally we use the same activation function in the whole network
- there are experiments using different activation function in different layers of network, but the consensus is using a single activation function

#### Initialization Methods
- If z is too small or too big, it will lead to saturation of network
- Inorder to avoid that we can use normalization of input
- One method is to take average of x's and take difference of x's with the average. This method brings examples on the axes, but they are still dissipated
- to fully solve the problem, divide the x's by standard dev. This method centers examples on the
- what is good about this method?
  - Earlier the loss function contours were ellipsoidal in nature, so gradient descent algorithm took multiple iteration to reach over the global minima
  - with examples normalized, the gradient descent points towards the middle
  - that is the reason why the second loss function is easier to train
- __Important point__: __the $\sigma$ and $\mu$ computed over the training set must be used over the test set as well.__ We should not compute the mean and standard deviation of the test set and normalize your test inputs through the network because your network is not used to seeing this type of transformation as an input. We want the distribution of inputs at the first neuron to be always the same, no matter if it is train or test set.

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

#### Vanishing/Exploding gradients
- Lets consider very deep network, with identity activation function and zero bias, the $\hat{y}$ and gradient will explode if the number is higher than 1 or the gradient will vanish if it is lower than 1.
- The solution is initialize the weights properly, with right range of values
- The weight range should be as close as 1, which will avoid the vanishing/exploding gradients problem

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

#### Initialization problem
- Example with 1 neuron
  - if the number of features are large, we should have small weight, otherwise z will explode

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

- Few other intializations

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

#### Optimization
- Gradient Descent
- Stochastic Gradient Descent
- Batch Gradient Descent
- Mini-Batch Gradient Descent
  - trade-off between stochasticity and vectorization

##### Notation
- m training example
> $X = \left(x^{(1)} x^{(2)} ... x^{(m)}\right)$
> $Y = \left(y^{(1)} y^{(2)} ... y^{(m)}\right)$
- T batches each containing 1000 training example
> $X = \left(X^{\{1\}} X^{\{2\}} ... X^{\{T\}}\right)$
> $Y = \left(Y^{\{1\}} Y^{\{2\}} ... Y^{\{T\}}\right)$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$

##### Momentum Gradient Descent
- loss is extended a lot in one direction compared to the other
- the gd will be orthogonal to the contour
- we would be interested in moving faster/larger update in horizontal line and slower/smaller update in vertical side
- momentum tells us look at the past update and take an average of the past update
> $v := \beta v + (1-\beta)\frac{\partial L}{\partial w}$
> $w := w - \alpha v$
- velocity variable tracks the direction that we should take for the current iteration with a factor/weight beta
- in terms of implementation/memory it is just one more line of code, but gives lot of efficiency

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng/Kian Katanforoosh}}$


```python
import numpy as np
import sympy as sp
import math

import Lagrange_poly3 as p1
import Lagrange_poly4 as p2
import Polynomial_vec as p3
import Polynomial_dict as p4
import Polynomial as p5

import matplotlib.pyplot as plt

# Needed only in Jupyter to render properly in-notebook
%matplotlib inline
```

# Chinmai Raman

## Homework 7

## 7.8 Wrapping functions in a class
Visualizes $p_L(x)$ for x between the first and last interpolation point on a given function

```python
xp = np.linspace(0, np.pi, 5)
yp = np.sin(xp)

#Lagrange's interpolation polynomial
p_L = p1.LagrangeInterpolation(xp, yp)
x = 1.2
print 'p_L(%g) = %g' % (x, p_L(x))
print 'sin(%g) = %g' % (x, np.sin(x))
p_L.plot()
```

## 7.9 Flexibly handling function arguments
The LagrangeInterpolation class is now flexible in handling function arguments

```python
def myfunction(x):
    return np.exp(-x/2.0)*np.sin(x)

p_L2 = p2.LagrangeInterpolation(myfunction, [0, np.pi], 11)
p_L2.plot()
```

## 7.27 Vectorizing a class for polynomials

$f(x)=10+5x+4x^2$ for $x = 3$

```python
poly = p3.Polynomial(np.array([10.0, 5.0, 4.0]))
poly(3)
```

    61.0

```python
poly = p3.Polynomial(np.array([10.0, 5.0, 4.0]))
str(poly)
```

    float64

    '10 + 5*x + 4*x^2'

```python
poly = p3.Polynomial(np.array([10.0, 5.0, 4.0]))
poly.differentiate()
str(poly)
```

    float64

    '5 + 8*x^1'

The calculations in a case where $n = 3$ using the vectorized statements produces the same results as the original differentiate method

### Unvectorized

```python
arr = []
for i in range(0, 1000):
    arr.append(i)
poly1 = p5.Polynomial(arr)

%timeit poly1 + poly1
```

    10000 loops, best of 3: 107 µs per loop

### Vectorized

```python
arr2 = np.zeros(1000)
for i in range(0, 1000):
    arr2[i] = i
poly2 = p3.Polynomial(arr2)

%timeit poly2 + poly2
```

    1000 loops, best of 3: 247 µs per loop

### Unvectorized

```python
%timeit poly1 * poly1
```

    1 loops, best of 3: 425 ms per loop

### Vectorized

```python
%timeit poly2 * poly2
```

    1 loops, best of 3: 512 ms per loop

The add and mul functions in the unvectorized Polynomial Class are running faster than their counterparts in the vectorized Polynomial class. Despite changing the length of the coefficients, the unvectorized class remains faster. Why this is is unclear.

## 7.28 Using a dictionary to hold polynomial coefficients

```python
poly = p4.Polynomial(np.array([1, 2, 0, 3]))
poly2 = p4.Polynomial(np.array([3, 4, 1, 0, 2]))
poly3 = poly + poly2
```

    {0: 1, 1: 2, 3: 3}
    {0: 3, 1: 4, 2: 1, 4: 2}
    {0.0: 4.0, 1.0: 6.0, 2.0: 1.0, 3.0: 3.0, 4.0: 2.0}

```python
poly = p4.Polynomial(np.array([0, 1, 0, 2]))
poly2 = p4.Polynomial(np.array([3, 0, 4]))
poly3 = poly * poly2
```

    {1: 1, 3: 2}
    {0: 3, 2: 4}
    {1.0: 3.0, 3.0: 10.0, 5.0: 8.0}

```python
arr3 = {}
for i in range(0, 1000):
    arr3[i] = i
poly3 = p4.Polynomial(arr3)

%timeit poly3 + poly3
```

    10 loops, best of 3: 22.7 ms per loop

```python
%timeit poly3 * poly3
```

    1 loops, best of 3: 331 ms per loop

The add function is significantly slower using the dictionary implementation than both the vectorized and unvectorized versions. However the multiply function is faster. The positive of this implementation is that one can input functions of high order with zero coefficients much faster. Memory is also saved within the program from not having to store coefficients when they are zero. The negative is that testing is more difficult, as for example, the 'numpy.allclose()' function only works with arrays, and not dictionaries.


```python
import sympy as sm
import sympy.physics.mechanics as me
me.init_vprinting()
```

```python
N, A, B = sm.symbols("N, A, B", cls=me.ReferenceFrame)
```

```python
alpha, beta = me.dynamicsymbols('alpha, beta')
d, w, c, l = sm.symbols('d, w, c, l')
```

```python
A.orient_axis(N, alpha, N.z)
B.orient_axis(A, beta, A.x)
```

```python
N_v_S = -d*alpha.diff()*A.y
N_v_S
```

```python
A.ang_vel_in(N) + B.ang_vel_in(A)
```

```python
B.ang_vel_in(N)
```

```python
r_S_Q = -w*B.x - (c + l/2)*B.z
r_S_Q
```

```python
N_v_Q = N_v_S + B.ang_vel_in(N).cross(r_S_Q)
N_v_Q
```

```python
N_v_Q.express(B)
```

```python
S, Q = sm.symbols('S, Q', cls=me.Point)
```

```python
Q.set_pos(S, r_S_Q)
```

```python
Q.pos_from(S)
```

```python
S.set_vel(N, N_v_S)
```

```python
S.vel(N)
```

```python
Q.v2pt_theory(S, N, B)
```

```python
s = me.dynamicsymbols('s')
```

```python
r_S_R = (s-w)*B.x + (l/2-c)*B.z
r_S_R
```

```python
B_v_R = s.diff()*B.x
B_v_R
```

```python
B.ang_vel_in(A)
```

```python
A_v_R = B_v_R + me.cross(B.ang_vel_in(A), r_S_R)
A_v_R
```

```python
me.cross(B.ang_acc_in(A), r_S_R)
```

```python
me.cross(B.ang_vel_in(A), me.cross(B.ang_vel_in(A), r_S_R))
```

```python
2*me.cross(B.ang_vel_in(A), B_v_R)
```

```python
B_v_R
```

```python
t = me.dynamicsymbols._t
```

```python
B_v_R.diff(t, B)
```

```python
R = me.Point('R')
```

```python
R.set_pos(S, r_S_R)
```

```python
R.set_vel(B, B_v_R)
```

```python
S.set_vel(A, 0)
```

```python
R.a1pt_theory(S, A, B)
```

```python
R.a1pt_theory(S, N, B)
```


# Warmup - Introduction to SymPy
This lab exercise uses the [SymPy](http://www.sympy.org/en/index.html) symbolic math library to model constraints in the problem.  To do that, we will use symbols (`sympy.Symbol`), functions (`sympy.Function`), and expressions (`sympy.Expr`) from sympy, and then we'll combine the function and expression classes to make constraints -- evaluatable symbolic functions.

In this warmup, you will be introduced to the syntax and functionality of SymPy:
- [Example 1](#Example-1:-Symbols): Creating [symbols](http://docs.sympy.org/dev/modules/core.html#module-sympy.core.symbol)
- [Example 2](#Example-2:-Expressions): Creating & manipulating [expressions](http://docs.sympy.org/dev/modules/core.html#id16) with [arithmetic & logical operators](http://docs.sympy.org/dev/modules/core.html#sympy-core)
- [Example 3](#Example-3:-Symbolic-substitution-and-expression-evaluation): Symbolic [substitution & evaluation](http://docs.sympy.org/dev/modules/core.html#subs)
- [Exercises](#SymPy-Exercises): Creating & manipulating constraints & [functions](http://docs.sympy.org/dev/modules/functions/index.html)

(See a list of common "gotchas" for sympy in their documentation: http://docs.sympy.org/dev/gotchas.html)

Start by reading and running the example cells, then complete the steps in the warmup cell.

```python
import matplotlib as mpl
import matplotlib.pyplot as plt

from util import constraint
from IPython.display import display
from sympy import *
init_printing()
```

## Example 1: Symbols
**Sympy provides the `Symbol` class to create symbolic variables.  Create individual symbols by calling the constructor with a symbol name.**  (Tip: Use the `display()` function to pretty-print symbolic terms.)

```python
x = Symbol('x')
display(x)
```

**You can also create symbols from an iterable sequence using the `symbols()` function.**

```python
i, j, k = symbols(['i', 'j', 'k'])  # use implicit unpacking to associate multiple symbols with variables
display((i, j, k))
```

**`symbols()` can also create subscripted sequences of symbolic variables.**

```python
X = symbols("X:3")
display(X)
```

## Example 2: Expressions

**A symbol is the most basic expression.**  (Tip: Jupyter notebooks show information about objects
using the `?` magic function)

```python
x = Symbol('x')
x?
display(x)
```

**You can also define expressions with relations between symbols.** (However, notice that expressions have no _names_...)

```python
x, y = symbols('x y')
or_relation = x | y
or_relation?
display(or_relation)
```

**Also, not all operators can be used in expressions.  The equal sign (=) performs assignment in python, so it cannot be used to make expressions.  Using `=` assigns a new python variable to an existing reference.**

```python
x, y = symbols("x y")
y = x   # now y references the same symbolic object as x
display(y)  # display(y) == x  ??!
```

**Use `sympy.Eq` for symbolic equality expressions:** (Tip: there are lots of expressions in the [sympy docs](http://docs.sympy.org/dev/modules/core.html#sympy-core))

```python
x, z = symbols("x z")
display(Eq(z, x))
```

**Sympy overloads standard python operators so that arithmetic and logical expressions can be constructed directly between symbolic objects.**

```python
x, y, z = symbols("x y z")
display([x**2, x - y, Ne(x, y), (~x & y & z)])
```

## Example 3: Symbolic substitution and expression evaluation

**Given an original expression...**

```python
x, y, z = symbols("x y z")
relation = Eq(x, y)
display(relation)
```

**Symbolic variables can be replaced by other variables, or by concrete values.** (Tip: use positional arguments in the `subs()` method to replace one symbol)

```python
display(relation.subs(x, z))  # Use positional arguments to substitute a single symbol
```

**But keep in mind that substitution returns a _copy_ of the expression -- it doesn't operate in-place.**  (Tip: as a result, you can use substitution on one expression bound to generic variables to generate new instances bound to specific variables.)

Look at what happens when we bind new variables to our equality relation:

```python
a = symbols("a:5")
b = symbols("b:5")
display([relation.subs({x: _a, y: _b}) for _a, _b in zip(a, b)])
```

**Symbol substitution returns an expression.** (Recall that Symbols _are_ expressions).

```python
print(type(relation), type(relation.subs(x, z)))
print(type(relation) == type(relation.subs(x, z)))
```

    <class 'sympy.core.relational.Equality'> <class 'sympy.core.relational.Equality'>
    True

**But substituting values for all symbols returns a value type.** (Tip: You can substitute multiple symbols in the `subs()` command by providing a mapping (dict) from current symbols to new symbols or values.)

```python
print(type(relation), type(relation.subs({x: 0, y: 1})))
print(type(relation) != type(relation.subs({x: 0, y: 1})))
```

    <class 'sympy.core.relational.Equality'> <class 'sympy.logic.boolalg.BooleanFalse'>
    True

## Example 4: Constraints

**Constraints are a construct of this lab exercise (not part of sympy) that combine symbolic Functions with Expressions for evaluation. The `constraint()` function (defined in the `util` module) takes a name and an expression and returns a "named expression" -- a constraint.**

```python
x, y = symbols(['x', 'y'])
sameAs = constraint("SameAs", Eq(x, y))
display(sameAs)
```

**Constraints are evaluated using the .subs method, just like an expression. If the resulting
expression has unbound (free) symbols, then the result is a new constraint.**

```python
display(sameAs.subs(x, 0), type(sameAs.subs(x, 0)))
```

**If the resulting expression has no free symbols, then the result is only the evaluated expression.**

```python
display(sameAs.subs({x: 0, y: 0}), type(sameAs.subs({x: 0, y: 0})))
```

## SymPy Exercises
Complete the following exercises to check your understanding of sympy symbols, expressions, and constraints:

**Question 1:** Create an array of subscripted symbols A0, A1, A2 stored in a variable named `A`

```python
A = symbols("A:3")

# test for completion
assert(len(A) == 3)
assert(all([type(v) == Symbol for v in A]))
print("All tests passed!")
```

    All tests passed!

**Question 2:** Create an expression E with two generic symbols (e.g., "a" and "b", etc.) that represents logical XOR

```python
a, b = symbols("a b")
E = Xor(x,y)

# test for completion
_vars = E.free_symbols
assert(len(_vars) == 2)
xor_table = {(0, 0): False, (0, 1): True, (1, 0): True, (1, 1): False}
assert(all(E.subs(zip(_vars, vals)) == truth for vals, truth in xor_table.items()))
print("All tests passed!")
```

    All tests passed!

**Question 3:** Create a constraint MaxAbsDiff with three generic arguments to test abs(a - b) < c, and create a copy of the constraint such that it tests abs(A[0] - A[1]) < A[2] from Q1

```python
a, b, c = symbols("a b c")
maxAbsDiff = constraint("less_than", Lt(Abs(a-b), c))
maxAbsDiff_copy = maxAbsDiff.subs({a: A[0], b:A[1], c:A[2]})

# test for completion
assert(maxAbsDiff.free_symbols != maxAbsDiff_copy.free_symbols)
assert(len(maxAbsDiff_copy.free_symbols) == len(maxAbsDiff_copy.args))
inputs = {(0, 6, 7): True, (6, 0, 7): True, (7, 6, 0): False}
assert(all(maxAbsDiff_copy.subs(zip(A[:3], vals)) == truth for vals, truth in inputs.items()))
print("All tests passed!")
```

    All tests passed!

**(Optional) Question 4:** Create a constraint AllDiff accepting the symbols in A as arguments, returning True if they are all different, and False if any pair is the same

```python
allDiff = constraint("all_differ", Ne(A[0],A[1]) & Ne(A[1], A[2]))

inputs = (([0, 1, 2], True), ([1, 1, 1], False), ([0, 1, 1], False))
assert(all(allDiff.subs(zip(A, vals)) == truth for vals, truth in inputs))
print("All tests passed!")
```

    All tests passed!


## Chemical kinetics
As we saw in the previous notebook, the rate of chemical reactions can be described by ordinary differential equations.

In this notebook we will look at a classic example in chemical kinetics:
[Robertson's example](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.53.8603).
It is a 3-species reaction system:

$$
A \overset{k_1}{\rightarrow} B \\
B + C \overset{k_2}{\rightarrow} A + C \\
2 B \overset{k_3}{\rightarrow} B + C
$$

where A, B and C represent three different chemical species (e.g. reactive molecules dissolved in water). What is special about this system is that the numerical stepping needs to be performed using an implicit method (requiring the Jacobian) when solving for large time scales.

The rate of each process follows the [law of mass action](https://en.wikipedia.org/wiki/Law_of_mass_action), i.e. the rate is proportional to the concentration of each reacting species (to the power of their multiplicity). The proportionality constant is known as the rate constant of the reaction ($k_1,\ k_2\ \&\ k_3$ in our case). If we denote the rate of each reaction:

$$
r_1 = k_1[A] \\
r_2 = k_2[B][C] \\
r_3 = k_3[B]^2
$$

$[A],\ [B],\ [C]$ denotes the concentration of respective species. We can now formulate a system of ordinary differential equations describing how the concentrations evolve over time:

$$
\frac{d[A]}{dt} = r_2 - r_1 \\
\frac{d[B]}{dt} = r_1 - r_2 - r_3 \\
\frac{d[C]}{dt} = r_3
$$

We will now express these differential equations (and their Jacobian) symbolically using SymPy:

```python
import sympy as sym
sym.init_printing()
```

```python
t = sym.symbols('t')  # not used, only a placeholder
c = cA, cB, cC = sym.symbols('[A] [B] [C]')  # concentrations
k = k1, k2, k3 = sym.symbols('k_1 k_2 k_3')
r1, r2, r3 = k1*cA, k2*cB*cC, k3*cB**2
ydot = r2 - r1, r1 - r2 - r3, r3
ydot
```

$\dot{\mathbf{y}}$ now represent our ODE system, where $\mathbf{y}$ is our state vector (concentrations). We will need a callback to evaluate $\dot{\mathbf{y}}$ when we integrate this ODE system numerically (using ``scipy.integrate.odeint``). As we have seen SymPy can provide us with this callback:

```python
import numpy as np
from scipy.integrate import odeint
```

```python
f = sym.lambdify((c, t) + k, ydot)

def integrate(tend=3600, t0=1e-6, **kwargs):
    tout = np.logspace(np.log10(t0), np.log10(tend))
    k_vals = (0.04, 1e4, 3e7)  # from the literature, k1 has another unit than k2 & k3
    c0 = [1, 0, 0]  # we start with unit concentration of A (number density)
    yout, info = odeint(f, c0, tout, args=k_vals, full_output=True, **kwargs)
    return tout, yout, info
```

The lambda function is needed to give f the signature asked for by ``odeint``. ``full_output=True`` makes odeint return a dictionary with information about the numerical integration.

```python
tout, yout, info = integrate(1e6, atol=1e-9, rtol=1e-9)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def plot_result(tout, yout, info=None):
    fig, axes = plt.subplots(1, 2, figsize=(14, 4))
    for i, label in enumerate('ABC'):
        for ax in axes:
            ax.plot(tout, yout[:, i], label=label)
    axes[1].set_xscale('log')
    axes[1].set_yscale('log')
    for ax in axes:
        ax.set_ylabel('$\mathrm{concentration / mol \cdot dm^{-3}}$')
        ax.set_xlabel('$\mathrm{time / s}$')
        ax.legend(loc='best')
    if info:
        print("The jacobian was evaluated %d times" % info['nje'][-1])
```

```python
plot_result(tout, yout, info)
```

If we look closer at the info-dictionary we will see that odeint (or rather LSODA which is the underlying package) switched method from an explicit Adams method to an implicit Backward Differentiation Formula (BDF). It is common for chemical kinetics problems that the problem becomes stiff. The larger number of species, the bigger is the Jacobian matrix.

By default the solver will approximate the elements in the Jacobian matrix by taking finite differences of $\mathbf{f}$. This is often works quite satisfactorily but for larger systems it sometimes fails. A more robust (and faster) approach is to provide a callback which evaluates an analytic Jacobian. Using SymPy we can do this quite effortlessly:

```python
J = sym.Matrix(ydot).jacobian(c)
J
```

```python
J_cb = sym.lambdify((c, t) + k, J)
```

```python
tout, yout, info = integrate(1e6, Dfun=J_cb)
```

```python
plot_result(tout, yout, info)
```

We see that the solver needed to evaluate the Jacobian fewer times (due to it being essentially exact this time around). For larger systems the impact of an analytic Jacobian is often even greater (being the difference between a failed and successful integration).

Benchmarking with and without the analytic Jacobian callback:

```python
%timeit integrate(1e6)
```

```python
%timeit integrate(1e6, Dfun=J_cb)
```


# Introduction to Geometric Algebra
Below I will try to motivate the case for Geometric Algebra (GA) using rotation of vectors in various spaces and compare to the other most common approaches.

(Source: https://commons.wikimedia.org/wiki/File:Counterclockwise_rotation.png)

Prerequisites
- Rotation matrices: good to know and to understand the motivation but not absolutely necessary
- Complex numbers: used to introduce Geometric Algebra rotors

TODO List
- Add graphics
- Add practical examples for PGA

## Rotating 2D vectors
Given a 2D vector $v = \begin{bmatrix} x \\ y \end{bmatrix} = x e_x + y e_y$ how do we rotate the vector given a rotation angle $\phi$? Let's look at some different solutions to this problem.

### Using 2x2 Rotation Matrices
A common solution is to use rotation matrices. In the 2D case it will be a 2x2 matrix of the following form

\begin{equation}
R(\phi) =
\begin{bmatrix}
cos(\phi) & -sin(\phi)\\
sin(\phi) & cos(\phi)
\end{bmatrix}
\end{equation}

Multiplying the vector and the matrix together we arrive at

\begin{equation}
v' = R(\phi) v = (x cos(\phi) - y sin(\phi)) e_x + (x sin(\phi) + y cos(\phi)) e_y
\end{equation}

Rotations can also be composed by multiplying two or more matrices together to obtain a single matrix

\begin{equation}
v' = R_a R_b v = R v$
\end{equation}

The downside of this method is that we need to calculate the `2x2` matrix which contains four numbers which seems more than actually necessary. Also while rotation matrices
do exist for other dimensions it is usually not easy to come up with them. They are often either memorized or require a lookup.

### Using Complex Numbers
Another common tool for 2D rotation are the complex numbers where we introduce an imaginary unit $i$ that squares to $-1$. First note that if we multiply a complex number with the imaginary unit $i$ the complex number will get rotated by 90°. To verify this we multiply $1 + i$ by $i$ the result will be $(1 + i) i = -1 + i$.

TODO: Add image of 90° rotation

By multiplying by integer powers of $i$ we can rotate by integer multiples of 90°, for example multiplying by $i^3$ will rotate by 270°.

TODO: Add image of 90° 180° 270° 360° rotations

However in order to create arbitrary rotations phi, we need to form the exponential $e^{i \phi}$ and multiply by it.
The resulting complex number of the exponential in terms of real and imaginary parts is given by [Euler's formula](https://en.wikipedia.org/wiki/Euler%27s_formula)

\begin{equation}
e^{i \phi} = cos(\phi) + i sin(\phi)
\end{equation}

Now in order to achieve what we wanted, we need to represent our vector $v$ as a complex number $c = x + i y$ (ie. the x coordinate becomes the real part and the y coordinate becomes the imaginary part).
Then we can apply the formula from above to rotate by an arbitrary angle $\phi$:

\begin{equation}
c' = c e^{i \phi} = (x + i y) e^{i \phi} = (x + i y) (cos(\phi) + i sin(\phi)) = (x cos(\phi) - y sin(\phi)) + i (x sin(\phi) + y cos(\phi))
\end{equation}

TODO: Show image of complex plane general rotation

If we compare this to what we arrived at using rotation matrices the result is identical. Similarly we can also compose rotations by multiplying two or more complex numbers together
to arrive at a single complex number that does both rotations. We now also only need to store two numbers for a rotation. Nevertheless the approach still leaves some questions. The conversion between vectors and complex numbers seems arbitrary. Furthermore this approach only works in 2 dimensions and it's not clear how to generalize it yet.

### Using 2D Geometric Algebra
Usually introductions to GA begin by defining various rules and going over derivations before doing anything useful with them. I will also define some rules but try to get to the interesting stuff more quickly.

Like for the standard 2D vector algebra in geometric algebra we have two basis vectors $e_x, e_y$ using which arbitrary vectors $v = x e_x + y e_y$ can be formed. Multiplying two same basis vectors together will result in $+1$ if they are the same (ie. $e_x e_x = 1, e_y e_y = 1$) similarly to how the dot product in standard vector algebra works.

What is new is that we can also multiply two different basis vectors and the result will not be zero, but can't be simplified further $e_x e_y = e_{xy}$. $e_xy$ here is not something new, it just stands for the two basis vectors multiplied together as a shorthand way of writing. These elements made up of two basis vectors are called bivectors.

Importantly the order of the product matters. A rule is that when you swap the factors of a product of basis vectors you pick up a minus sign, for example. $e_{xy} = e_x e_y = -e_y e_x = -e_{yx}$.

This kind of product is called the Geometric Product and it is the central idea of Geometric Algebra. Let's now use these three rules and see what some results are when we use them

\begin{aligned}
e_x e_y e_x & = & \text{(rewrite as shorthand)} \\
e_{xyx} & = & \text{(Swap neighbours on the right, pick up minus sign)} \\
-e_{xxy} & = & \text{(Multiplying same basis vectors results in 1, e_xx = e_x e_x = 1)} \\
-e_y &
\end{aligned}

Hopefully this gives some idea of how the rules work and how to use them. Now for something more interesting, let's see what happens if we multiply of these together, that is, squaring it:

\begin{equation}
e_{xy}^2 = e_{xy} e_{xy} = e_{xyxy} = -e_{xyyx} = -e_{xx} = -1
\end{equation}

We can see the square of the bivector $e_{xy}$ is $-1$, the same as the square of the imaginary unit of the complex numbers.

Back to the rotation problem, to resolve the disconnect of vectors and complex numbers we make use of the things we just learnt. Vectors are represented as before as $v = x e_x + y e_y$. As mentioned before $e_{xy}$ can be identified as the imaginary unit $i$ hence we present complex numbers as $a + b e_{xy}$ and a rotation by an arbitrary angle $\phi$ can be performed in the same way as before using Euler's formula

\begin{equation}
R(\phi) = e^{\phi e_{xy}} = cos(\phi) + e_{xy} sin(\phi)
\end{equation}

The entity $R$ you get after exponentiating is called a rotor (because it rotates when you multiply by it, d'uh). With this we don't need to perform the arbitrary conversion between vectors and complex numbers, we can simply multiply the vector and the rotor together as is

\begin{equation}
v R(\phi) = (x e_x + y e_y) (cos(\phi) + e_{xy} sin(\phi)) = e_x (x cos(\phi) - y sin(\phi)) + e_y (x sin(\phi) + y cos(\phi))
\end{equation}

This result is the same as the rotation formula we got for the rotation matrix and the procedure is pretty much identical to what we did with complex numbers. We can also see that the imaginary unit is in fact a rotation in the XY plane as the bivector $e_{xy}$ is formed by multiplying the two basis vectors together, so in some sense it represents the XY plane.

So far it seems that at best we gained some clarity, which is good, but might not justify using it over the well known complex numbers and vectors yet. The big advantage of this approach as we will see in the next
sections is that this generalizes to any dimension and does not require any memorization besides the basic geometric algebra rules we learnt above.

## Rotating 3D vectors
Now we go up one dimension to three dimensional space. We want to rotate a 3D vector $v = x e_x + y e_y + e_z$ by the angles $\alpha, \beta, \gamma$
around the three axes (or planes) of rotation. As we will see things will get a bit more complicated in the usual approaches.

### Using 3x3 Rotation Matrices
As in the 2D case we can look up the 3D rotation matrix formula on Wikipedia or some other place

\begin{equation}
R(\alpha, \beta, \gamma) =
\begin{bmatrix}
cos(\alpha) cos(\beta) & cos(\alpha) sin(\beta) sin(\gamma) - sin(\alpha) cos(\gamma) & cos(\alpha) sin(\beta) cos(\gamma) + sin(\alpha) sin(\gamma) \\
sin(\alpha) cos(\beta) & sin(\alpha) sin(\beta) sin(\gamma) + cos(\alpha) cos(\gamma) & sin(\alpha) sin(\beta) cos(\gamma) - cos(\alpha) sin(\gamma) \\
-sin(\beta) & cos(\beta) sin(\gamma) & cos(\beta) cos(\gamma)
\end{bmatrix}
\end{equation}

Multiplying a 3D vector with it will give us the desired result $v' = R(\alpha, \beta, \gamma) v$. Some people might be able to memorize this formula but I think it is obvious to see that the rotation matrices for higher dimensions
will be even harder to come up with. Additionally the `3x3` matrix now contains nine numbers which is way more than the original three we started with.

### Using 3D Geometric Algebra
We now have three basis vector $e_x, e_y, e_z$ and as a result three independent bivectors $e_{xy}, e_{xz}, e_{yz}$.
Let's see what the bivectors square to using the rules from before:

\begin{aligned}
e_{xy}^2 = e_{xyxy} = -e_{xyyx} = -1 \\
e_{xz}^2 = e_{xzxz} = -e_{xzzx} = -1 \\
e_{yz}^2 = e_{yzyz} = -e_{yzzy} = -1
\end{aligned}

All three of our bivectors square to $-1$. Some readers might be familiar with quaternions which have the same property for their three imaginary units $i, j, k$ and are like the 3D versions of the 2D complex numbers. Previously we demonstrated that $e_{xy}$ can be used to perform a rotation in the XY plane, so one might already suspect what $e_{xz}$ and $e_{yz}$ can be used for.

One might expect that the rotor $R_{xz} = e^{\pi e_{xz}}$ would produce a rotation by 90° ($\pi$ in degrees) in the XZ plane when multiplying a 3D vector by it.
This is almost correct. It turns out that the 2D case was special and the general formula for applying rotors requires a two sided product, also called the sandwich product

\begin{equation}
v' = R v \widetilde{R}
\end{equation}

where $\widetilde{R}$ means reversing the order of all basis vectors appearing in it (eg. $\widetilde{\phi e_{xz}} = \phi e_{zx} = -\phi e_{xz})$.

Another result of this is that the angle in the exponent needs to be half of the desired angle as we are now multiplying by it twice. As stated before the 2D case was special and one could then verify that this sandwich product yields the same results as the one sided product in 2D.

So, to put this all together, a 3D rotation in the XZ plane can be generated using

\begin{equation}
R_{xz} = e^{\frac{\phi}{2} e_{xz}} = cos(\frac{\phi}{2}) + e_{xz} sin(\frac{\phi}{2})
\end{equation}

by sandwiching a 3D vector with it

\begin{equation}
v' = R_{xz} v \widetilde{R}_{xz} = (cos(\frac{\phi}{2}) + e_{xz} sin(\frac{\phi}{2})) v \widetilde{(cos(\frac{\phi}{2}) + e_{xz} sin(\frac{\phi}{2}))} = (cos(\frac{\phi}{2}) + e_{xz} sin(\frac{\phi}{2})) x (cos(\frac{\phi}{2}) - e_{xz} sin(\frac{\phi}{2}))
\end{equation}

Now to perform arbitrary 3D rotations in multiple planes we can just apply multiple of these rotors, for example a rotation in XY plane then the XZ plane then YZ plane
by angles $\alpha, \beta, \gamma$ respectively can be performed using three sandwich products in the following way:

\begin{equation}
v' = R_{yz}(\gamma) R_{xz}(\beta) R_{xy}(\alpha) v \widetilde{R}_{xy}(\alpha) \widetilde{R}_{xz}(\beta) \widetilde{R}_{yz}(\gamma)
\end{equation}

The three rotors $R_{xy}(\alpha), R_{xz}(\beta), R_{yz}(\gamma)$ can also be combined into a single rotor $R(\alpha, \beta, \gamma)$ by multiplying them together. The same rotation around three axes can then be achieved
using a single sandwich product

\begin{equation}
v' = R(\alpha, \beta, \gamma) v \widetilde{R}(\alpha, \beta, \gamma)
\end{equation}

As we have three rotors and each rotor contains a unique bivector and a scalar part it turns out the combined rotor will contain four numbers, much less than the nine of the `3x3` rotation matrix.

Hopefully it is now possible to see that the Geometric Algebra approach will generalize to any dimension. For example in 4D we would have six planes of rotation and thus six bivectors, and one can generate rotations in them using exactly the same approach with the exponential and sandwich product.

## Translating and rotating 2D vectors

Besides rotation another common geometric operation is translation. In the following sections we will see where Geometric Algebra becomes really useful and provides more than just being insightful, minimal and generalizing well.
We're going back to two dimensions to keep things less verbose.We're going back to two dimensions to keep things less verbose. The goal now is to translate a vector $v = x e_x + y e_y$ by an offset $T = t_x e_x + t_y e_y$ and also rotate is as before by an angle $\phi$ in the three axes.

### Using Standard Vectors and Addition

The most straightforward way of achieving translation is simply adding the translation vector to the original vector

\begin{equation}
v' = v + T = (x + t_x) e_x + (y + t_y) e_y
\end{equation}

Now we also want to rotate the vector to arrive at the result. We can use the 2x2 rotation matrix from before to do this

\begin{equation}
v'' = R(\phi) v'
\end{equation}

This works, but has several issues. For one, we are using two different operations to represent translations and rotations. Furthermore it also makes combining these transformations (combined rotation and translation) harder. We can't just compose multiple transforms into a single one as we could do for rotation (eg. by multiplying two rotation matrices together we could get a single rotation matrix).

### Using Standard Vectors with homogenous coordinates and 3x3 matrices

To fix the issue of translation and rotation being represented in different ways using different operations (vector addition for translation, matrix multiplication for rotation) we want a way to unify them. For this the traditional solution is to use homogenous coordinates. Here we simply pad the 2D vector using a constant $1$ to get a vector with three elements

\begin{equation}
v_h = \begin{bmatrix} v \\ 1 \end{bmatrix} = \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}
\end{equation}

Translation can then be done using `3x3` matrices as the factors in the third column will add a term proportional to the constant $1$

\begin{aligned}
x' = R_{11} x + R_{12} y + R_{13} 1 \\
y' = R_{21} x + R_{22} y + R_{23} 1
\end{aligned}

The last row of the `3x3` matrix is usually chosen so that the resulting vector will also have a $1$ in the fourth position (ie. $R_{31} = R_{32} = 0, R_{33} = 1$).

Unlike the vector addition solution, we can now compose the transformations by multiplying `3x3` matrices together to get a single `3x3` matrix that does the combined rotation and translation. However now we have even more numbers.
A `3x3` matrix will contain nine numbers, far more than the minimal amount of three (two for translation, one for rotation).

### Using 2D Projective Geometric Algebra
In addition to the previous two basis vectors $e_x, e_y$ we now add a third basis vector $e_0$. The interesting thing about this basis vector is that instead of squaring to $+1$ it will square to $0$. This might seem strange at first but this allows performing translations in the same way we performed rotations using the exponential function and rotors. Remember, when we started with rotations we multiplied by $i$ (or the bivector $e_{xy}$ in GA) which generated a rotation by 90° and later we introduced the exponential with Euler's formula to produce a rotation of an arbitrary angle. Here we will do the same for translation.

Another change now is that previously we used $e_x, e_y$ as the two basis vectors to represent our points. Instead, we now use the bivectors $e_{y0}, e_{0x}$ instead and we will see that this seemingly unnatural choice will help us achieve the translation we wanted. We also add a third bivector $e_xy$ to the point representation with coefficient $1$. A point with coordinates $x, y$ is now represented as

\begin{equation}
P = x e_{y0} + y e_{0x} + e_{xy}
\end{equation}

Now to do the equivalent of the 90° rotation for translation, let's notice that right-multiplying the point with $(1 + e_{0x})$ generates a translation of $1$ in the x direction (ie. increments the coefficient of $e_{0y}$ by $1$)

\begin{aligned}
& (x e_{y0} + y e_{0x} + e_{xy}) (1 + e_{0x}) & = \\
& (x e_{y0} + y e_{0x} + e_{xy}) + e_{xy0x} & = \\
& (x e_{y0} + y e_{0x} + e_{xy}) + e_{y0} & = \\
& (x + 1) e_{y0} + y e_{0x} + e_{xy}
\end{aligned}

So now we can translate by $1$ in both the x and the y direction (a translation in the y direction would be done by right-multiplying with $(1 + e_{y0})$). As with rotation, to generate a translation by an arbitrary amount we now
form the exponential to get a rotor $R = e^{\frac{t_x}{2} e_{0x} + \frac{t_y}{2} e_{y0}}$. It turns out here we can't use Euler's formula to get the result of the exponential as it only applies for values that square to $-1$ such as our bivector (or the imaginary unit $i$). Instead the solution is a much simpler formula

\begin{equation}
e^{\frac{t_x}{2} e_{0x} + \frac{t_y}{2} e_{y0}} = 1 + (\frac{t_x}{2} e_{0x} + \frac{t_y}{2} e_{y0})
\end{equation}

so we just add a scalar $1$. This is also why there was a one plus in our x-translation by $1$ earlier as $e^{e_{0x}} = 1 + e_{0x}$. Now by multiplying a rotor that does rotation and a rotor that does translation, we can have
more general rotors that do both rotation and translation. These are often also called motors. In general they are applied as before using the sandwich product. A 2D motor contains four numbers (a scalar, two translation bivectors $e_{y0}, e_{0x}$ and a rotation bivector $e_{xy}$), very close to the minimum of three required for 2D translation and rotation.

## Applications of (Projective) Geometric Algebra

### Interpolating motors
A cool thing we can now do that we couldn't easily do before is interpolating between two motors.

Given two $m_1$ and $m_2$ we can interpolate them using the logarithm. Taking the logarithm of a motor allows us to get the term that was in the exponent before we took the exponential. Then we can linearly interpolate the two exponents and afterwards take the exponential again which should give us a nice and smooth interpolation that blends both translation and rotation. Putting those thoughts into a formula we arrive at

\begin{equation}
interpolate(m_1, m_2, \alpha) = e^{\alpha log(m_1) + (1 - \alpha) log(m_2)}
\end{equation}

with interpolation factor $\alpha \in [0, 1]$. This requires us to compute the logarithm of a motor. Fortunately for the 2D case there is a simple formula

\begin{equation}
log(m) = \frac{m}{||m||} = \frac{m}{\sqrt{m \widetilde{m}}}
\end{equation}

While in other dimensions we have [a way of computing the exponential of (most interesting) GA multivectors](https://www.euclideanspace.com/maths/algebra/clifford/algebra/functions/exponent/index.htm) the logarithm unfortunately doesn't generalize as easily. Formulas for the most important cases are known though including the 3D case.

## Other important things that were not explicitly mentioned
- Basis vectors can square to $[-1, 0, +1]$
- Geometric product can be decomposed into an inner an exterior product $a b = a \cdot b + a \wedge b$

## Further study
- [bivector.net](https://bivector.net/): Contains GA tools and links to various resources
- [CoffeeShop](https://enkimute.github.io/ganja.js/examples/coffeeshop.html): Interactive PGA examples
- [Geometric Algebra talk at SIGGRAPH2019](https://youtu.be/tX4H_ctggYo): Great introduction to PGA many motivating examples


## Performance Indicator

It is fundamental for any algorithm to measure the performance. In a multi-objective scenario, we can not simply calculate the distance to the true global optimum, but must consider a set of solutions. Moreover, sometimes the optimum is not even known and other techniques must be used.

First, let us consider a scenario, where the Pareto-front is known:

```python
import numpy as np
from pymoo.factory import get_problem
from pymoo.visualization.scatter import Scatter

# The pareto front of a scaled zdt1 problem
pf = get_problem("zdt1").pareto_front()

# The result found by an algorithm
A = pf[::10] * 1.1

# plot the result
Scatter(legend=True).add(pf, label="Pareto-front").add(A, label="Result").show()
```

### Generational Distance (GD)

The GD performance indicator  measure the distance from solution to the Pareto-front. Let us assume the points found by our algorithm are the objective vector set $A=\{a_1, a_2, \ldots, a_{|A|}\}$ and the reference points set (Pareto-front) is $Z=\{z_1, z_2, \ldots, z_{|Z|}\}$. Then,

\begin{align}
\begin{split}
\text{GD}(A) & = & \; \frac{1}{|A|} \; \bigg( \sum_{i=1}^{|A|} d_i^p \bigg)^{1/p}\\[2mm]
\end{split}
\end{align}

where $d_i$ represents the euclidean distance (p=2) from $a_i$ to its nearest reference point in $Z$. Basically, this results in the average distance from any point $A$ to the closest point in the Pareto-front.

```python
from pymoo.factory import get_performance_indicator

gd = get_performance_indicator("gd", pf)
print("GD", gd.calc(A))
```

    GD 0.05497689467314528

### Generational Distance Plus (GD+)

Ishibushi et. al. proposed in  GD+:

\begin{align}
\begin{split}
\text{GD}^+(A) & = & \; \frac{1}{|A|} \; \bigg( \sum_{i=1}^{|A|} {d_i^{+}}^2 \bigg)^{1/2}\\[2mm]
\end{split}
\end{align}

where for minimization $d_i^{+} = max \{ a_i - z_i, 0\}$ represents the modified distance from $a_i$ to its nearest reference point in $Z$ with the corresponding value $z_i$.

```python
from pymoo.factory import get_performance_indicator

gd_plus = get_performance_indicator("gd+", pf)
print("GD+", gd_plus.calc(A))
```

    GD+ 0.05497689467314528

### Inverted Generational Distance (IGD)

The IGD performance indicator  inverts the generational distance and measures the distance from any point in $Z$ to the closest point in $A$.

\begin{align}
\begin{split}
\text{IGD}(A) & = & \; \frac{1}{|Z|} \; \bigg( \sum_{i=1}^{|Z|} \hat{d_i}^p \bigg)^{1/p}\\[2mm]
\end{split}
\end{align}

where $\hat{d_i}$ represents the euclidean distance (p=2) from $z_i$ to its nearest reference point in $A$.

```python
from pymoo.factory import get_performance_indicator

igd = get_performance_indicator("igd", pf)
print("IGD", igd.calc(A))
```

    IGD 0.06690908300327662

### Inverted Generational Distance Plus (IGD+)

In  Ishibushi et. al. proposed IGD+ which is weakly Pareto compliant wheres the original IGD is not.

\begin{align}
\begin{split}
\text{IGD}^{+}(A) & = & \; \frac{1}{|Z|} \; \bigg( \sum_{i=1}^{|Z|} {d_i^{+}}^2 \bigg)^{1/2}\\[2mm]
\end{split}
\end{align}

where for minimization $d_i^{+} = max \{ a_i - z_i, 0\}$ represents the modified distance from $z_i$ to the closest solution in $A$ with the corresponding value $a_i$.

```python
from pymoo.factory import get_performance_indicator

igd_plus = get_performance_indicator("igd+", pf)
print("IGD+", igd_plus.calc(A))
```

    IGD+ 0.06466828842775944

### Hypervolume

For all performance indicators shown so far a target set needs to be known. For Hypervolume only a reference point needs to be provided. First, I would like to mention that we are using the Hypervolume implementation from [DEAP](https://deap.readthedocs.io/en/master/). It calculates the area/volume which is dominated by the provided set of solutions with respect to a reference point.

This image is taken from  and illustrates a two objective example where the area which is dominated by a set of points is shown in grey.
Whereas, for the other metrics the goal was to minimize the distance to the Pareto-front, here, we desire to maximize the performance metric.

```python
from pymoo.factory import get_performance_indicator

hv = get_performance_indicator("hv", ref_point=np.array([1.2, 1.2]))
print("hv", hv.calc(A))
```

    hv 0.9631646448182305


# Gradient Descent Derivation

We will derive the optimization algorithm from scratch

### Computing $\frac{\partial}{\partial w_0} L(w)$

\begin{align}
\frac{\partial}{\partial w_0} L(w)
&= \frac{\partial}{\partial w_0} \frac{1}{n} \sum_{i=1}^n\left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)^2 \\
&= \frac{1}{n} \sum_{i=1}^n \frac{\partial}{\partial w_0} \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)^2 \quad  \\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \frac{\partial}{\partial w_0} \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right) \quad\textit{# Chain Rule}\\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \left(0 - \frac{\partial}{\partial w_0}\sin\left(w_0 x_i + w_1 \right)\right) \\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \left(0 - \cos\left(w_0 x_i + w_1 \right) \frac{\partial}{\partial w_0}\left(w_0 x_i + w_1 \right) \right)  \quad\textit{# Chain Rule}\\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \left(0 - \cos\left(w_0 x_i + w_1 \right) \left(x_i + 0 \right) \right) \\
&= -\frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \cos\left(w_0 x_i + w_1 \right) x_i \quad\textit{# Simplified}
\end{align}

### Computing $\frac{\partial}{\partial w_1} L(w)$

\begin{align}
\frac{\partial}{\partial w_1} L(w)
&= \frac{\partial}{\partial w_1} \frac{1}{n} \sum_{i=1}^n\left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)^2 \\
&= \frac{1}{n} \sum_{i=1}^n \frac{\partial}{\partial w_1} \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)^2 \\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \frac{\partial}{\partial w_1} \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right) \quad\textit{# Chain Rule}\\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \left(0 - \frac{\partial}{\partial w_1}\sin\left(w_0 x_i + w_1 \right)\right) \\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \left(0 - \cos\left(w_0 x_i + w_1 \right) \frac{\partial}{\partial w_1}\left(w_0 x_i + w_1 \right) \right)  \quad\textit{# Chain Rule}\\
&= \frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \left(0 - \cos\left(w_0 x_i + w_1 \right) \left(0 + 1\right) \right) \\
&= -\frac{1}{n} \sum_{i=1}^n  2 \left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)
    \cos\left(w_0 x_i + w_1 \right) \quad\textit{# Simplified}
\end{align}

Finally, we get
$$
L(w) = \frac{1}{n} \sum_{i=1}^n\left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)^2
$$

$$
\nabla_w L(w) = \left[ \frac{\partial}{\partial w_1} L(w), \frac{\partial}{\partial w_2} L(w) \right]
$$

```
import plotly.offline as py
import plotly.graph_objects as go
import plotly.express as px
import plotly.figure_factory as ff
import numpy as np
import pandas as pd
```

```
def load_data():
  np.random.seed(42)
  n = 50
  x = np.sort(np.random.rand(n)*2.5 * np.pi)
  x = x - x.mean()/x.std()
  y = np.sin(2.2 * x + 1.8) + 0.2 * np.random.randn(n)
  return (x, y)
```

```
(x, y) = load_data()
print("Number of data points: ", x.shape[0])
```

    Number of data points:  50

```
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y, mode='markers', name='markers'))
fig.update_layout(title='Scatter')
html = plotly.offline.plot(fig, filename='3d-scatter-plotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

The data is scattered randomly and has high variance, resembling a sine function

```
def model(w, x):
  return np.sin(w[0] * x + w[1])
```

```
xtest = np.linspace(x.min()-0.1, x.max()+0.1, 100)

yhat1 = model([1, 1], xtest)
yhat2 = model([2, 1], xtest)
yhat3 = model([1, 2], xtest)
```

```
fig = px.scatter(x=x, y=y)
fig.add_trace(go.Scatter(x=xtest, y=yhat1, name="$f_{[1,1]}(x) = \sin(x+1)$"))
fig.add_trace(go.Scatter(x=xtest, y=yhat2, name="$f+{[2,1]}(x) = \sin(2x + 1)$"))
fig.add_trace(go.Scatter(x=xtest, y=yhat3, name="$f_{[1,2]}(x) = \sin(x+2)$"))
fig.update_layout(title='Scatter Sine Plot')
html = plotly.offline.plot(fig, filename='3d-scattersine-plotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

The sin function does not fit at all on the data. We need to find a parameter that is closer to the data, we need a loss function. So, we make use of average squared loss.

$$
L(w) = L\left(f_w, \mathcal{D} = \left\{(x_i, y_i \right\}_{i=1}^n\right) = \frac{1}{n} \sum_{i=1}^n\left(y_i - f_w\left(x_i\right)\right)^2
= \frac{1}{n} \sum_{i=1}^n\left(y_i - \sin\left(w_0 x_i + w_1 \right)\right)^2
$$

```
def avg_sq_loss(w):
  return np.mean((y - model(w, x))**2)
print(avg_sqloss([1,1]))
```

    0.8519315245287105

```
fig = px.bar(x=["$w=[1,1]$", "$w=[2,1]$", "$w=[1,2]$"],
       y=[avg_sq_loss([1,1]), avg_sq_loss([2,1]), avg_sq_loss([1,2])])
fig.update_yaxes(title="Average Squared Loss")
html = plotly.offline.plot(fig, filename='3d-scatter-avgsqloss-plotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

```
loss_trials = []
w0values = np.linspace(0, 3, 30)
w1values = np.linspace(0, 3, 30)
for w0 in w0values:
    for w1 in w1values:
        loss_trials.append([w0, w1, avg_sq_loss([w0, w1])])
```

```
loss_df = pd.DataFrame(loss_trials, columns=["w0", "w1", "loss"])
loss_df
```

      w0
      w1
      loss

      0
      0.0
      0.000000
      0.593748

      1
      0.0
      0.103448
      0.568623

      2
      0.0
      0.206897
      0.564753

      3
      0.0
      0.310345
      0.581171

      4
      0.0
      0.413793
      0.616070

      ...
      ...
      ...
      ...

      895
      3.0
      2.586207
      0.943073

      896
      3.0
      2.689655
      0.949557

      897
      3.0
      2.793103
      0.955011

      898
      3.0
      2.896552
      0.960128

      899
      3.0
      3.000000
      0.965686

900 rows × 3 columns

```
loss_df.sort_values("loss").head()
```

      w0
      w1
      loss

      648
      2.172414
      1.862069
      0.036495

      649
      2.172414
      1.965517
      0.041426

      647
      2.172414
      1.758621
      0.042194

      676
      2.275862
      1.655172
      0.042869

      675
      2.275862
      1.551724
      0.046697

```
# We wish to visualise the best brute force model
w_best_brute = loss_df.loc[loss_df["loss"].idxmin(), ["w0", "w1"]].to_numpy()
print(w_best_brute)
```

    [2.17241379 1.86206897]

```
yhat_brute = model(w_best_brute, xtest)

fig = px.scatter(x=x, y=y)
fig.add_trace(go.Scatter(x=xtest, y=yhat_brute, name="Brute Force Approach"))
html = plotly.offline.plot(fig, filename='3d-scatter-avgxysqloss-plotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

We will try to get a better fit using gradient descent

```
fig = go.Figure()
fig.add_trace(go.Surface(x=w0values, y=w1values,
                         z=loss_df["loss"].to_numpy().reshape((len(w0values), len(w1values)))))
fig.update_layout(margin=dict(l=0, r=0, t=0, b=0), height=600)
html = plotly.offline.plot(fig, filename='3d-scatter-avgxysqloss-plotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

```
fig = go.Figure()
fig.add_trace(go.Contour(x=loss_df["w0"], y=loss_df["w1"], z=loss_df["loss"]))
fig.update_layout(margin=dict(l=0, r=0, t=0, b=0),
                  height=500, width=1000)
html = plotly.offline.plot(fig, filename='3dcontours-plotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

```
def gradient(w):
  g0 = -np.mean(2 * (y - np.sin(w[0] * x + w[1]))*np.cos(w[0]*x + w[1]*x))
  g1 = -np.mean(2 * (y - np.sin(w[0] * x + w[1]))*np.cos(w[0]*x + w[1]))
  return np.array([g0, g1])
```

```
gradient([1., 1.])
```

    array([-0.68715648,  0.15427075])

```
loss_grad_df = loss_df.join(loss_df[['w0', 'w1']]
 .apply(lambda w: gradient(w), axis=1, result_type="expand")
 .rename(columns={0:"g0", 1:"g1"}))
loss_grad_df
```

      w0
      w1
      loss
      g0
      g1

      0
      0.0
      0.000000
      0.593748
      -0.346570
      -0.346570

      1
      0.0
      0.103448
      0.568623
      -0.121564
      -0.139293

      2
      0.0
      0.206897
      0.564753
      0.095250
      0.062907

      3
      0.0
      0.310345
      0.581171
      0.246164
      0.251583

      4
      0.0
      0.413793
      0.616070
      0.301754
      0.418980

      ...
      ...
      ...
      ...
      ...
      ...

      895
      3.0
      2.586207
      0.943073
      -0.182591
      0.069638

      896
      3.0
      2.689655
      0.949557
      -0.161851
      0.056666

      897
      3.0
      2.793103
      0.955011
      -0.104241
      0.049892

      898
      3.0
      2.896552
      0.960128
      -0.025551
      0.050302

      899
      3.0
      3.000000
      0.965686
      0.051813
      0.058474

900 rows × 5 columns

```
fig = go.Figure()
fig = ff.create_quiver(x=loss_grad_df['w0'], y=loss_grad_df['w1'],
                       u=loss_grad_df['g0'], v=loss_grad_df['g1'],
                       line_width=2, line_color="white",
                       scale = 0.1, arrow_scale=.2)
fig.add_trace(go.Contour(x=loss_grad_df['w0'], y=loss_grad_df['w1'], z=loss_grad_df['loss']))
fig.update_layout(margin=dict(l=0, r=0, t=0, b=0),
                  height=600, width=800)
fig.update_layout(xaxis_range=[w0values.min(), w0values.max()])
fig.update_layout(yaxis_range=[w1values.min(), w1values.max()])
html = plotly.offline.plot(fig, filename='3dplotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

```
def gradient_descent(w_0, lr = lambda t: 1./(t+1.), nepochs=10):
    w = w_0.copy()
    values = [w]
    for t in range(nepochs):
        w = w - lr(t) * gradient(w)
        values.append(w)
    return np.array(values)
```

```
values = gradient_descent(np.array([3.0, 0.0]),
                          nepochs=100,
                          lr =lambda t: 1./np.sqrt(t+1.))
```

```
fig = go.Figure()
fig.add_trace(go.Contour(x=loss_grad_df['w0'], y=loss_grad_df['w1'], z=loss_grad_df['loss']))
fig.add_trace(go.Scatter(x=values[:,0], y=values[:,1], name="Path", mode="markers+lines",
                     line=go.scatter.Line(color='white')))
fig.update_layout(margin=dict(l=0, r=0, t=0, b=0),
                  height=600, width=800)
fig.update_layout(xaxis_range=[w0values.min(), w0values.max()])
fig.update_layout(yaxis_range=[w1values.min(), w1values.max()])
html = plotly.offline.plot(fig, filename='3dplotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

```
fig = go.Figure()
fig.add_trace(
    go.Surface(x=w0values, y=w1values,
               z=loss_df["loss"].to_numpy().reshape((len(w0values), len(w1values)))))
fig.add_trace(
    go.Scatter3d(x=values[:,1], y=values[:,0], z=[avg_sq_loss(w) for w in values],
                line=dict(color='white')))
fig.update_layout(margin=dict(l=0, r=0, t=0, b=0),
                  height=600)
html = plotly.offline.plot(fig, filename='3dplotly.html',include_plotlyjs='cdn')
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

```
import noise
import numpy as np
import matplotlib
from mpl_toolkits.mplot3d import axes3d
```

```
shape = (50, 50)
scale = 100.0
octaves = 6
persistence = 0.5
lacunarity = 2.0
```

```
world = np.zeros(shape)
for i in range(shape[0]):
  for j in range(shape[1]):
    world[i][j] = noise.pnoise2(i/scale,
                                j/scale,
                                octaves=octaves,
                                persistence=persistence,
                                lacunarity=lacunarity,
                                repeatx=1024,
                                repeaty=1024,
                                base=42)
```

```
import matplotlib.pyplot as plt

plt.imshow(world, cmap='terrain')
```

```
lin_x = np.linspace(0,1,shape[0],endpoint=False)
lin_y = np.linspace(0,1,shape[1],endpoint=False)
x, y = np.meshgrid(lin_x, lin_y)
```

```
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.plot_surface(x, y, world, cmap="terrain")
```

```
terrain_cmap = matplotlib.cm.get_cmap('terrain')
def plt_to_plotly(cmap, pl_entries):
  h = 1.0/(pl_entries - 1)
  pl_colorscale = []

  for k in range(pl_entries):
    C = list(map(np.uint8, np.array(cmap(k*h)[:3])*255))
    pl_colorscale.append([k*h, 'rgb'+str((C[0], C[1], C[2]))])
  return pl_colorscale
terrain = plt_to_plotly(terrain_cmap, 255)
```

```
import plotly
import plotly.graph_objects as go
plotly.offline.init_notebook_mode(connected=True)

fig = go.Figure(data=[go.Surface(colorscale=terrain,z=world)])

fig.update_layout(title='Random 3D Terrain')
html = plotly.offline.plot(fig, filename='3d-terrain-plotly.html',include_plotlyjs='cdn')
```

```
from IPython.core.display import HTML
HTML(html)
```

<meta charset="utf-8" />

Let us visualise the 3D Terrains, a simple optimization of gradient descent

```
from IPython.core.display import HTML
import plotly
import plotly.graph_objects as go
import noise
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
%matplotlib inline
```

```
z = world
plt.imshow(z, origin='lower', cmap='terrain')

# Find maximum value index in numpy array
indices = np.where(z == z.max())
max_z_x_location, max_z_y_location = (indices[1][0], indices[0][0])
plt.plot(max_z_x_location, max_z_y_location, 'ro', markersize=15)

# Find minimum value index in numpy array
indices = np.where(z == z.min())
min_z_x_location, min_z_y_location = (indices[1][0], indices[0][0])
plt.plot(min_z_x_location, min_z_y_location, 'yo', markersize=15)
```

```
import numpy as np
from numpy.lib.stride_tricks import as_strided
```

```
def sliding_window(arr, window_size):
  """
  Slide Window view of array
  """
  arr = np.asarray(arr)
  window_size = int(window_size)
  if arr.ndim!=2:
    raise ValueError("Need 2D input")
  if not(window_size>0):
    raise ValueError("Need positive window size")
  shape = (arr.shape[0] - window_size + 1,
           arr.shape[1] - window_size + 1,
           window_size, window_size)
  if shape[0] <= 0:
    shape = (1, shape[1], arr.shape[0], shape[3])
  if shape[1] <= 0:
    shape = (shape[0], 1, shape[2], arr.shape[1])
  strides = (arr.shape[1]*arr.itemsize, arr.itemsize,
             arr.shape[1]*arr.itemsize, arr.itemsize)
  return as_strided(arr, shape=shape, strides=strides)

def cell_neighbours(arr, i, j, d):
    """Return d-th neighbors of cell (i, j)"""
    w = sliding_window(arr, 2*d+1)

    ix = np.clip(i - d, 0, w.shape[0]-1)
    jx = np.clip(j - d, 0, w.shape[1]-1)

    i0 = max(0, i - d - ix)
    j0 = max(0, j - d - jx)
    i1 = w.shape[2] - max(0, d - i + ix)
    j1 = w.shape[3] - max(0, d - j + jx)

    return w[ix, jx][i0:i1,j0:j1].ravel()
```

```
from dataclasses import dataclass

@dataclass
class descent_step:
  value: float
  x_index: float
  y_index: float

def gradient_descent_3D(array, x_start, y_start, steps=50, step_size=1, plot=False):
  step = descent_step(array[y_start][x_start], x_start, y_start)
  step_history = []
  step_history.append(step)

  # 2D Plotting of array with starting point as red marker
  if plot:
    plt.imshow(array, origin='lower', cmap='terrain')
    plt.plot(x_start, y_start, 'ro')
  current_x = x_start
  current_y = y_start

  for i in range(steps):
    prev_x = current_x
    prev_y = current_y

    neighbours = cell_neighbours(array, current_x, current_y, step_size)

    next_step = neighbours.min()
    indices = np.where(array == next_step)

    current_x, current_y = (indices[1][0], indices[0][0])
    step = descent_step(array[current_y][current_x], current_x, current_y)

    step_history.append(step)

    if plot:
      plt.plot([prev_x, current_x], [prev_y, current_y], 'k-')
      plt.plot(current_x, current_y, 'ro')

    if prev_y == current_y and prev_x == current_x:
      print(f"Converged in {i} steps")
      break
  return next_step, step_history
```

```
np.random.seed(42)
global_minimum = z.min()
indices = np.where(z == global_minimum)
print(f"Target: {global_minimum} @ {indices}")

step_size = 0
found_minimum = 99999

# Random starting point
start_x = np.random.randint(0,50)
start_y = np.random.randint(0,50)

# Increase step size until convergence on global minimum
while found_minimum != global_minimum:
    step_size += 1
    found_minimum,steps = gradient_descent_3D(z,start_x,start_y,step_size=step_size,plot=False)

print(f"Optimal step size {step_size}")
found_minimum,steps = gradient_descent_3D(z,start_x,start_y,step_size=step_size,plot=True)
print(f"Steps: {steps}")
```

```
def multiDimenDist(point1,point2):
   #find the difference between the two points, its really the same as below
   deltaVals = [point2[dimension]-point1[dimension] for dimension in range(len(point1))]
   runningSquared = 0
   #because the pythagarom theorm works for any dimension we can just use that
   for coOrd in deltaVals:
       runningSquared += coOrd**2
   return runningSquared**(1/2)
def findVec(point1,point2,unitSphere = False):
  #setting unitSphere to True will make the vector scaled down to a sphere with a radius one, instead of it's orginal length
  finalVector = [0 for coOrd in point1]
  for dimension, coOrd in enumerate(point1):
      #finding total difference for that co-ordinate(x,y,z...)
      deltaCoOrd = point2[dimension]-coOrd
      #adding total difference
      finalVector[dimension] = deltaCoOrd
  if unitSphere:
      totalDist = multiDimenDist(point1,point2)
      unitVector =[]
      for dimen in finalVector:
          unitVector.append( dimen/totalDist)
      return unitVector
  else:
      return finalVector
```

```
def generate_3d_plot(step_history):
    # Initialise empty lists for markers
    step_markers_x = []
    step_markers_y = []
    step_markers_z = []
    step_markers_u = []
    step_markers_v = []
    step_markers_w = []

    for index, step in enumerate(step_history):
        step_markers_x.append(step.x_index)
        step_markers_y.append(step.y_index)
        step_markers_z.append(step.value)

        if index < len(steps)-1:
            vec1 = [step.x_index,step.y_index,step.value]
            vec2 = [steps[index+1].x_index,steps[index+1].y_index,steps[index+1].value]

            result_vector = findVec(vec1,vec2)
            step_markers_u.append(result_vector[0])
            step_markers_v.append(result_vector[1])
            step_markers_w.append(result_vector[2])
        else:
            step_markers_u.append(0.1)
            step_markers_v.append(0.1)
            step_markers_w.append(0.1)

    fig = go.Figure(data=[
        go.Cone(
        x=step_markers_x,
        y=step_markers_y,
        z=step_markers_z,
        u=step_markers_u,
        v=step_markers_v,
        w=step_markers_w,
        sizemode="absolute",
        sizeref=2,
        anchor='tail'),

        go.Scatter3d(
        x=step_markers_x,
        y=step_markers_y,
        z=step_markers_z,
        mode='lines',
        line=dict(
            color='red',
            width=2
        )),

        go.Surface(colorscale=terrain,z=world,opacity=0.5)])

    fig.update_layout(
        title='Gradient Descent Steps',
        scene = dict(zaxis = dict(range=[world.min(),world.max()],),),)
    return fig

# Generate 3D plot from previous random starting location
fig = generate_3d_plot(steps)
HTML(plotly.offline.plot(fig, filename='random_starting_point_3d_gradient_descent.html',include_plotlyjs='cdn'))
```

<meta charset="utf-8" />

```
found_minimum,steps = gradient_descent_3D(z,max_z_x_location,max_z_y_location,step_size=5,plot=True)
fig = generate_3d_plot(steps)
HTML(plotly.offline.plot(fig, filename='maximum_starting_point_step_size_5_3d_gradient_descent.html',include_plotlyjs='cdn'))
```


```python
import numpy as np
import math
import pandas as pd
import scipy.optimize as opt
import matplotlib.pyplot as plt
from sympy import *
%matplotlib inline
```

# Questão 1

```python
def bissecao(f, a, b, tol=1e-10, iteracao=9999):
    i  = 0;
    erro, x_ant = 1, a

    while(erro > tol and i < iteracao):
        inf = np.sign(f(a))
        sup = np.sign(f(b))

        if (inf * sup != -1):
            print("Não há raiz nesse intervalo:")
            return
        else:
            x = (a + b)/2.
            if (f(x) == 0):
                print("A raíz é:", x)
                return
            elif (inf * np.sign(f(x)) == -1):
                b = x
            else: a = x
            i += 1
        erro = np.abs((x - x_ant)/np.abs(x))
        x_ant = x
    return x
```

```python
f = lambda x: x**5+2*x**3-5*x-2
```

```python
x = np.linspace(-3, 3, 100)
plt.plot(x, f(x))
plt.grid()
```

```python
bissecao(f, -2, 2, 1e-4)
```

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:21: RuntimeWarning: divide by zero encountered in double_scalars

    1.3197021484375

```python
f = lambda x: ( x**2 - 2 )
bissecao(f, -1, 8, 1e-4)
```

    1.4141082763671875

# Questão 2

```python
def function(x):
    return (-0.5 * (x**2)) + 2.5*x + 4.5
```

# (A)

```python
x = np.linspace(-5, 10)

r1 = opt.root(function, -2)
r2 = opt.root(function, 7)
rt = np.array([r1.x, r2.x])
fig = plt.figure()
a = fig.add_subplot()
plt.plot(x, function(x))
a.plot(rt, function(rt), 'o')
plt.grid(ls=':')
```

# (B)

```python
def quadratica(a, b, c):
    delta = (b**2) - (4 * a * c) # Δ=b²- 4.a.c
    x1 = (-b-math.sqrt(delta)) / (2 * a)
    x2 = (-b+math.sqrt(delta)) / (2 * a)
    return x1, x2

print(quadratica(-0.5, 2.5, 4.5))
x1, x2 = np.roots([-0.5, 2.5, 4.5])
print(x1, x2)
```

    (6.405124837953327, -1.405124837953327)

    6.405124837953327 -1.4051248379533272

# (C)

### Erro relativo entre cada iteração

```python
error = []
raizes = []

for i in range(1,5):
    if (i <= 3):
        raiz = opt.root_scalar(f,method='bisect',bracket=[5,10], maxiter=i).root
        x = opt.root_scalar(f,method='bisect',bracket=[5,10], maxiter=(i+1)).root
        raizes.append(raiz)
        error.append(np.absolute(x - raiz)/x)

for i in range(len(error)):
    print(f'Iteração: {i} \t Raiz: {raizes[i]} \t Erro: {error[i]}')
```

    Iteração: 0 	 Raiz: 5.0 	 Erro: 0.2
    Iteração: 1 	 Raiz: 6.25 	 Erro: 0.0
    Iteração: 2 	 Raiz: 6.25 	 Erro: 0.0

### Erro relativo como valor real

```python
error2 = []
raizes2 =[]

for i in range(1,5):
    if (i <= 3):
        raiz = opt.root_scalar(f,method='bisect',bracket=[5,10], maxiter=i).root
        raizes2.append(raiz)
        error2.append(np.absolute(6.405124837953327 - raiz)/6.405124837953327)

for i in range(len(erros)):
    print(f'Iteração: {i} \t Raiz: {raizes2[i]} \t Erro: {error2[i]}')
```

    Iteração: 0 	 Raiz: 5.0 	 Erro: 0.2193750900259293
    Iteração: 1 	 Raiz: 6.25 	 Erro: 0.024218862532411636
    Iteração: 2 	 Raiz: 6.25 	 Erro: 0.024218862532411636

# Questão 3

```python
f = lambda x: np.sin(x) - x**3
x = np.linspace(-np.pi, np.pi, 200)
x2 = f(x)
r1= opt.root(f, -2)
r2 = opt.root(f, 7)
mon = np.array([r1.x, r2.x])
fig = plt.figure()
a = fig.add_subplot()
plt.plot(x, x2)
a.plot(mon, f(mon), 'o')
plt.grid(ls=':')
```

# Questão 4

```python
#f (x) = −2x6 − 1.5x4 + 10x + 20
f = lambda x: (-2*x**6) - (1.5*x**4) + (10*x) + 20
x = Symbol('x')
y = (-2*x**6) - (1.5*x**4) + (10*x) + 20
der = y.diff(x)
y1 = lambda x: (-12*x**5) - (6*x**3) + 10
print(der)
print(bissecao(y1, 0, 1, 0.05))
```

    -12*x**5 - 6.0*x**3 + 10
    0.84375

# Questão 5

```python
def interp(f, xi, xs, tol=1e-10):
    i = 0;
    inf = (f(xi))
    sup = (f(xs))
    if (inf * sup >= 0):
        return -1
    erro, x_ant = 1, xi
    while (erro > tol):
        x = xs - ( ( f(xs)*(xi-xs) ) / (f(xi)-f(xs)))
        if ((f(x) * f(xi)) < 0):
            xs = x
        else: xi = x
        i += 1
        erro = np.abs((x - x_ant)/np.abs(x))
        x_ant = x
    return x

f = lambda x: ( x**5+2*x**3-5*x-2 )

print(bissecao(f, -2, 2))
print(interp(f, -2, 2))
```

    1.319641167181544
    1.3196411670137347

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:21: RuntimeWarning: divide by zero encountered in double_scalars


Inverse Problems

Probabilistic Inversion: Simulated annealing

##### Authors:
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))

* Kilian Gessele ([@KGessele](https://github.com/KGessele))

* Berta Vilacís ([@bvilacis](https://github.com/bvilacis))

---

This notebook presents a technique for finding the maximum likely point of a multi-modal function while sampling it. Our function is a probability density function (pdf) and the method used is called simulated annealing, a Markov Chain Monte Carlo method (MCMC).

This method is inspired by the annealing process in solid state physics. Annealing is a physical process of heating a solid until thermal stresses are released, then cooling it very slowly until the crystals are perfectly arranged, acheiving a minimum energy state. Depending on how fast or slow the temperature is cooled, the results will have worse or better the quality.

Photo: Gavin Smith - https://bit.ly/2IzAkcE}

The simulation of this process is known as Simualted Annealing. It is a mathematical analogy which can be used to sample highly nonlinear functions, multidimensional functions. The analogy of the slow temperature cooling is the decrese on the probablitiy of accepting worse solutions as the space is explored; instead, the probability for going to a better solution is kept to 1. At high temperatures the system is allowed to explore the whole space, and at low temperatures exist a restriction in the exploration. The decrease of the temperature has to be done carefully, due to the aim is to find the global minimum state of energy. If it is done too fast you can obtain a secondary minima, and if it is too slow you will waste a lot of forward calculations.

This analogy between the physical process of annealing and the mathematical problem of obtaining a global minimum of a function allow us to finding the maximum likelihood point.

First we will define an energy function:
\begin{equation}
S(m) = - T_0 log\frac{\sigma_M(m)}{\rho_M(m)}
\end{equation}
where $m$ is refered to the coordinates in the space, $T_0$ is a fixed positive number termed the ambient temperature (e.g. $T=1$), $\sigma_M(m)$ is the designed function and $\rho_M(m)$ is a parameter.
We obtain the probability density function as a function of the temperature of the system
\begin{equation}
\sigma_M(m,T)=\rho_M(m) exp \bigg( -\frac{S(m)}{T} \bigg) = \rho_M(m) exp \bigg( -\frac{-T_0 log\frac{\sigma_M(m)} {\rho_M(m)}}{T} \bigg)
\end{equation}

```python
# Import Libraries (PLEASE RUN THIS CODE FIRST!)
# ----------------------------------------------
import numpy as np
import matplotlib
## Show Plot in The Notebook
matplotlib.use('nbagg')
import matplotlib.pyplot as plt
from IPython import display

from mpl_toolkits.mplot3d import Axes3D
```

**PARAMETER CONFIGURATION**

The above expression helps us to understand the relationship between our pdf and the temperature. For this notebook is considered that $\rho_M(m) = 1$ and $T_0 = 1$, with the expresion beeing
\begin{equation}
\sigma_M(m,T)= exp \bigg( -\frac{-log(\sigma_M(m))}{T} \bigg)
\end{equation}

```python
# Parameter Configuration
# ----------------------

# Some global parameters

npp = 1000000  #number of tries to take
ns = 100   # number of samples to keep
T0 = 1.
T = T0
alpha = 0.99999 #cooling schedule

rho = 1 # manteining it constant
rho0 = 1 # mantening it constant
Tplots = 10 # inicial temperature for the plots
stepT = 4 #how many steps should the Temperature be *0.1 evolved
```

**GENERATION OF THE PDF**

Monte Carlo sampling of a probability density function, in this case for the purpose of the notebook it is used the absolute value of the  function "peaks" from Matlab in each point. The corresponding section can simply be replaced with the probability from a proper inverse problem (e.g. hypocenter location). This function has the following expression

\begin{equation}
\sigma_M(m) = \sigma(x,y) = | 3(1-x)^2 e^{-x^2-(y+1)^2}-10 \big( \frac{x}{5} -x^3-y^5\big) e^{-x^2-y^2} - \frac{1}{3} e^{-(x+1)^2-y^2} |
\end{equation}

```python
# Generate a pdf

# the following steps generate a pdf; this is equivalent to the function "peaks(n)" in matlab
n = 100 # number of dimension
pdf = np.zeros([n,n])
sigma = np.zeros([n,n])
s = np.zeros([n,n])
x = -3.
for i in range(0,n):
    y = -3.
    for j in range(0,n):
        pdf[j,i]=3.*(1-x)**2*np.exp(-(x**2)-(y+1)**2)-10.*(x/5-x**3-y**5)*np.exp(-x**2-y**2)-1./3*np.exp(-(x+1)**2-y**2)
        if pdf[j,i] < 0:
            pdf[j,i] = pdf[j,i] * (-1) # in contrast to the peaks function: all negative values are multiplied by (-1)
        y = y + 6./(n-1)
    x = x + 6./(n-1)

pdf = pdf / pdf.max()
s  = -T0*np.log(pdf/rho0)
```

```python
# Plot the 3D plot of pdf
# --------------------------

X = np.arange(0,100 + 100./(n-1), 100./(n-1))
Y = np.arange(0,100 + 100./(n-1), 100./(n-1))
fig0 = plt.figure()
ax = fig0.gca(projection='3d')
X, Y = np.meshgrid(X, Y)
surf = ax.plot_surface(Y, X, pdf,rstride=2, cstride=2, cmap=plt.cm.coolwarm, linewidth=0.1)
plt.gca().invert_xaxis()
plt.show()
```

    <IPython.core.display.Javascript object>

**The Energy function**

It gives the distribution of energy in our pdf.

```python
# Plot the 3D plot of Energy function
# --------------------------

X = np.arange(0,100 + 100./(n-1), 100./(n-1))
Y = np.arange(0,100 + 100./(n-1), 100./(n-1))
fig0 = plt.figure()
ax = fig0.gca(projection='3d')
X, Y = np.meshgrid(X, Y)
surf = ax.plot_surface(Y, X, s/s.max(),rstride=2, cstride=2, cmap=plt.cm.coolwarm, linewidth=0.1)
plt.gca().invert_xaxis()
plt.show()
```

    <IPython.core.display.Javascript object>

**THE HEAT BATH**

For understanding how the temperature interacts with the probability distribution function, the notebook plots where $\sigma_M(m,T)$ is allowed to go in a given temperature. Each line represents an increase of factor of 10 of the temperature, starting with a hihg temperature of $T=10$, and finishing with $T=0.01$. As simulated annealing we  Finally the ones in the middle are the "Heat bath graphs", in which varies their values in the space vary with each temperature.

```python
#The heat bath
a = 1
fig0 = plt.figure(figsize=(10,8))
for i in range(0,stepT):
    sigma = rho * np.exp(-(s)/Tplots)
    sigma= sigma / sigma.max()

    if Tplots < 1:
        temp='$T = $'+str(Tplots)+'\n'
    else:
        temp= '$T = $'+str(int(Tplots))+'\n'

    Tplots = Tplots * 0.1
    b = 0
    X = np.arange(0,100 + 100./(n-1), 100./(n-1))
    Y = np.arange(0,100 + 100./(n-1), 100./(n-1))
    ax = fig0.add_subplot(2, 2, a, projection='3d')
    X, Y = np.meshgrid(X, Y)
    ax.set_title(temp)
    ax.plot_surface(Y, X, sigma,rstride=2, cstride=2, cmap=plt.cm.coolwarm, linewidth=0, antialiased=False)
    plt.gca().invert_xaxis()
    a += 1
    plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.90, hspace=0.5, wspace=0.15)

plt.show()
```

    <IPython.core.display.Javascript object>

** SIMULATED ANNEALING: The algorithm **

Now that we saw how the temperature affects to the pdf we can develop the algorithm. The therminology will be the folowing:

   – $X$ is the random vector, or the configuration.

   – $S_m$ is the system energy, the function value.

   – $T$ is the system temperature.

   – $\Delta S$ is the difference in system energy between two steps ($\Delta S= S_m(X_{i+1}) - S_m (X_i)$).

At each iteration on the temperature the program creates a $X_{i}$ and evaluates the point using the simulation model (pdf), finding an associated system energy. This move has to be accepted or not with respect to the previous one.
  * if $S= S_m(X_{i+1}) < S_m (X_i)$ $\rightarrow$ accept $X_{i+1}$ as the new model (current_model =  new_model)
  * if $S= S_m(X_{i+1}) \geqslant S_m (X_i)$ $\rightarrow$ accept $X_{i+1}$ as the new model with probability $S =  exp (-\frac{\Delta S}{T})$ with $T = \alpha T$

beeing $\alpha$ the cooling schedule.

At the end, the system will have reduced the temperature, according to the cooling choosen, towards 0.

```python
# Simulated annealing and ploting the steps

# find an initial vector x
xcur = np.array([np.floor(np.random.uniform(0, 100)), np.floor(np.random.uniform(0, 100))])

# Let s get moving
iis = 0
npp = 0
xa = np.array([])
xa = np.hstack((xa, np.array([xcur[0],xcur[1]])))

## Ploting the steps (initializing the graph)
fig = plt.figure()
f1 = fig.add_subplot(111)
f1.imshow(pdf.transpose(),aspect='auto',interpolation='none', animated=True)
f1.set_xlim(0,99)
f1.set_ylim(99,0)
f1.set_title('Simulated annealing steps')
plt.ion() #set interactive mode
plt.show()

Pa = np.zeros(ns+1)
xnew = np.array([0.,0.])

Pa = [0]
while iis < ns:
    npp = npp+1
    #make a random choice for the next move
    xnew = np.array([np.floor(np.random.uniform(0, 100)), np.floor(np.random.uniform(0, 100))])
    #compare probabilities
    Pcur = pdf[int(xcur[0]), int(xcur[1])]
    Pnew = pdf[int(xnew[0]), int(xnew[1])]

    T = alpha * T
    if Pnew >= Pcur:
        xcur = xnew
       # print ' Made the %i-th move to [%i,%i] ' %(iis+1, xcur[0], xcur[1])
        # display move graphically
        f1.plot(xnew[0],xnew[1],'w+') #plot the steps with black +
        plt.gcf().canvas.draw() #show the update
        iis = iis + 1

    if Pnew < Pcur:
        deltaP = Pcur - Pnew
        P = np.exp(-1. * deltaP / T)
        test = np.random.uniform(0,1)
        if test <= P:
            xcur = xnew
            #print ' Made the %i-th move to [%i,%i] P= %g ' %(iis+1,xcur[0],xcur[1],P)
            # display move graphically
            f1.plot(xnew[0],xnew[1],'w+') # plot the steps with black +
            plt.gcf().canvas.draw()  #show the update
            iis = iis + 1

    xa = np.vstack((xa, np.array([xcur[0],xcur[1]])))
    Pa.append(Pnew)

plt.gcf().canvas.draw()
npp = npp + 1
Pa = np.array(Pa)
```

    <IPython.core.display.Javascript object>

**FINAL SAMPLING **

To show how the program has proceded it is shown all the places where the program has gone in the sampling space, and the the maximum probability point found. Also, it is ploted as function of the iterations the best model and the current model.

```python
# Final sampling

#get max of Pa and the index for the best model
y = Pa.max()
for i in range(0,npp):
    if Pa[i] == y:
        maxi = i

#Ploting the results
fig2 = plt.figure(figsize=(10,5))

#First graph
f1 = fig2.add_subplot(121)
f1.imshow(pdf.transpose(),aspect='auto', extent=[0,100,100,0],interpolation='none')
f1.plot(xa[:,0],xa[:,1],'w+') #Plot the steps with white +
f1.plot(xa[maxi,0],xa[maxi,1],'wo') #Maximum value acchieved ploted wiht the white cirlce
f1.set_ylabel('y')
f1.set_xlabel('x')

#Second graph
f2 = fig2.add_subplot(122)
# Show as a function of the iterations
# 1. the best model
# 2. the current model
best = np.zeros(npp)
for i in range(0,npp):
    a = Pa[0]
    for j in range(0,i+1):
        if Pa[j] > a:
            a = Pa[j]
    best[i] = a

f2.plot(range(0,npp),best,'r-', label='Best model')
f2.plot(range(0,npp),Pa,'b-', label='Current model')
f2.set_xlabel('# of points visited')
f2.yaxis.set_label_position("right")
f2.legend(loc='upper center', bbox_to_anchor=(0.6, 1.1), ncol=2, handletextpad=0.1)
f2.set_ylabel('Probability')

plt.show()
```

    <IPython.core.display.Javascript object>

notebooks/Seismic Inverse Problems/Probabilistic Inversion/pi_simann.ipynb


This notebook demonstrates using SciPy to solve ODEs and Matplotlib for plotting

# Geodesics in Schwarzschild spacetime

The Schwarzschild spacetime describes an isolated spherically symmetric black hole. The (timelike) geodesic equations give the motion of a test mass around the black hole. In this notebook we give the equations, place them into first-order form, numerically integrate them and then plot the resulting orbit.

In this notebook we use geometrized units such that $G = c = 1$. The Schwarzschild black hole has mass $M$, and the Schwarzschild coordinates are $(t,r,\theta, \phi)$.

## Equations of motion

Let the coordinates of the test body be denoted by $x^\mu_p(\tau) = \{t_p(\tau), r_p(\tau), \theta_p(\tau),\phi_p(\tau)\}$ where $\tau$ is the propertime (propertime is the time as mearsured by a clock movine along with the test body$. Using the Euler-Lagrange equations one can derive the equations of motion a test body in orbit about the black hole. These are given by the ODEs:

$$\begin{align}
\frac{dt}{d\tau} &= E^2(1-2M/r)^{-1}\\
\left(\frac{dr}{d\tau}\right)^2 &= E^2 - \left(1- \frac{2M}{r}\right)\left(1 + \frac{L^2}{r^2}\right)\\
\frac{d\phi}{d\tau} & = \frac{L}{r^2}
\end{align}$$

where $E$ and $L$ are the energy and angular momentum as measured at infinity.

Note due to spherical symmetry, without loss of generality we can set $\theta=\pi/2$ and $\theta$ does not evolve over the orbit.

## Manipulating the radial equation

The radial equation above is written in terms of first-order derivaties but it is not especially useful for numerical integration. That is because to use it you have to take a square root, which brings with it a $\pm$. The plus represents the outgoing motion from periastron to apastron, and the minus the ingoing motion. Switching between these two is difficult numerically.

A nice solution is to take a derviative of the whole radial equation w.r.t. $\tau$. This results in a second-order differenential equation which removes the troublesome $r'(\tau)$ term. The result of this operation is:

$$\frac{d^2r}{d\tau^2} = \frac{L^2(r-3M)}{r^4} - \frac{M}{r^2}$$

In order to numerically integrate this equation we now need to write this in first-order form.
To do this we introduce $r_1 = dr/d\tau$, then the above equation can be written as two coupled first-order equations:

$$\begin{align}
   \frac{dr}{d\tau}   &= r_1 \\
   \frac{dr_1}{d\tau} &= \frac{L^2(r-3M)}{r^4} - \frac{M}{r^2}
\end{align}$$

## The systems of equations and initial conditions

In order to plot the orbital trajectory we only need the $r(\tau)$ an $\phi(\tau)$ solutions. This makes our set of ODEs:

$$\begin{align}
   \frac{dr}{d\tau}   &= r_1 \\
   \frac{dr_1}{d\tau} &= \frac{L^2(r-3M)}{r^4} - \frac{M}{r^2} \\
   \frac{d\phi}{d\tau} & = \frac{L}{r^2}
\end{align}$$

In order to solve these equations we need to specify the initial conditions at $\tau = \tau_0 $. Without loss of generality we will take $\tau_0 = 0$. We can define $r_0 \equiv r_0$ and $\phi(0) \equiv \phi_0$. Again, w.l.o.g we can take $\phi_0 = 0$.

What about $r_1(0) = dr/d\tau(0) \equiv u^r_0$. For this we can use the original version of the radial equation. Thus we have

$$ u^r_0 = \pm\sqrt{E^2 - \left(1- \frac{2M}{r}\right)\left(1 + \frac{L^2}{r^2}\right)}$$

# The energy and angular-momentum

We will now restrict out attention to bound orbits which have a minimum and maximum radius which we denote by $r_\min/r_\max$, respectively. We can further define two geometric quantities, the semi-latus rectum, $p$, and the orbital eccentricity, $e$. This are given by

$$ p \equiv \frac{2r_\max r_\min}{M(r_\max + r_\min)},\qquad e \equiv \frac{r_\max - r_\min}{r_\max + r_\min}$$

Inverting these equations we get

$$ r_\max = \frac{p}{1-e},\qquad r_\min = \frac{p}{1+e}$$

By looking at the effective potential, the energy and angular momentum can be written in terms of $(p,e)$. This gives

$$ E = \sqrt{\frac{(p-2-2e)(p-2+2e)}{p(p-3-e^2)}}, \qquad L = \sqrt{\frac{p^2 M^2}{p-3-e^2}}$$

# Writing the code

We now want to write a code to compute the orbital trajectory. To do this we need to write a function that given $p$ and $e$:

1) computes $(E,L)$
2) computes $u^r_0$
3) integrates the ODEs
4) plots the result

In order to plot the results we define the cartesian coordinates via:

$$ x(\tau) = r(\tau) \cos[\phi(\tau)], \qquad y(\tau) = r(\tau) \sin[\phi(\tau)]$$

Next we load the libraries we will need and write the function that computes the geodesic and plots its spatial trajectory. Hereafter we will set $M=1$.

```python
from scipy.integrate import solve_ivp
import numpy as np
import matplotlib.pyplot as plt
```

```python
# Define the geodesic equations
def geodesicEqs(tau, y, L):
    r   = y[0]
    r1  = y[1]

    dr  = r1
    dr1 = L**2*(r - 3)/r**4 - 1/r**2

    dphi = L/r**2

    return [dr, dr1, dphi]
```

```python
def plotGeodesic(p, e, tau_max):

    # Compute the energy and angular momentum
    E = np.sqrt((p-2-2*e)*(p-2+2*e)/(p*(p-3-e**2)))
    L = np.sqrt(p**2/(p-3-e**2))

    # The initial conditions
    # Pick an r0 value between rmin and rmax
    r0 = p
    phi0 = 0

    # Calculate the initial value of the radial four-velocity
    ur0 = - np.sqrt(E**2 - (1-2/r0)*(L**2/r0**2 + 1) )

    # solve the geodesics with solve_ivp from SciPy
    sol = solve_ivp(geodesicEqs, [0, tau_max], [r0, ur0, 0], args = [L], t_eval = np.arange(0, tau_max, 1))

    rVals = sol.y[0]
    phiVals = sol.y[2]

    xVals = rVals*np.cos(phiVals)
    yVals = rVals*np.sin(phiVals)

    # Set the figure size
    plt.figure(figsize = (15,15))

    # Ensure the entire trajectory will be visible
    rmax = p/(1-e)

    plt.xlim(-rmax-1, rmax+1)
    plt.ylim(-rmax-1, rmax+1)

    # Draw a black disk for the black hole. The Schwarzschild radius is r_s = 2M.
    circle1 = plt.Circle((0, 0), 2, color='k')
    plt.gcf().gca().add_artist(circle1)

    # Plot the trajectory
    plt.plot(xVals, yVals)

```

```python
plotGeodesic(8,0.5,1000)
```

```python

```