#Spazi Vettoriali

##Introduzione ai vettori
I vettori sono utilizzato per esempio nella fisica.
$$
\vec{f} = m * \vec{a}
$$

##Vettori del piano
Come rappresentazione un vettore è una frecca orientata che nasce da un punto $O$ verso un punto $P$

$$
OP = \vec{v}
$$
Ogni vettore ha 3 elementi:
 * La direzione (Retta passante per O e P)
 * Il verso (Da O verso P)
 * Un intensità (La distanza da O a P)

##Somma di vettori
Dati due vettori con direzioni diverse la loro somma segue la regola del parallelogramma.

Il vettore somma è la diagonale del parallelogramma costruito usando i die vettori come lati.
Se hanno la stessa direzione muove la punta del primo vettore lungo il verso del secondo dell'intensità del secondo vettore.

### Proprietà della somma
#### Vettore Opposto
Secondo questa regola, avendo due vettori $\vec v_1, \vec v_2$ con la stessa direzione e intensità ma verso opposto, la loro somma è un vettore di intensità 0, il vettore nullo. Il vettore sarà quindi chiamato **vettore oppsoto**
$$
\vec v_1 + \vec v_2 = \vec0 \\
\vec v_2 = -\vec v_1
$$
#### Proprietà commutativa
Un'altra proprietà importante della somma dei vettori è la **proprietà commutativa**.
$$
\vec v + \vec w = \vec w + \vec v
$$
Non importa l'ordine con cui sommo i vettori, il risultato non cambia.
#### Proprietò associativa
Se devo sommare più di 2 vettori l'ordine con cui eseguo queste somme non cambia il risultato
$$
\vec v + \vec w + \vec z
= (\vec v + \vec w) + \vec z
= \vec v + (\vec w + \vec z)
$$

##Prodotto di un numero reale per un vettore
Dato un vettore $\vec v$ e un numero reale $a$ il loro prodotto è un vettore $a\vec v$ con le seguenti caratteristiche:
 * La stessa direzione di $\vec v$
 * La stessa direzione di $\vec v$ se $a$ è positivo, la direzione opposta se $a$ è negativo
 * L'intensità data dal prodotto fra $a$ è l'intensità di $\vec v = \lVert v \rVert$

###Proprietà del prodotto per un numero
####Azzeramento del prodotto
Il vettore $a\vec v$ sarò nullo se $a = 0$ o $\vec v = $ vettore nullo

####Proprietà distributiva
Il prodotto di un numero per la somma di due vettori è pari alla somma dei prodotti
$$
a(\vec v + \vec w) = a\vec v + a\vec w
$$
Il prodotto fra un vettore per la somma di due numeri reali è paro alla somma dei prodotti
$$
(a + v)\vec v = a\vec v + b\vec v
$$

##N-Uple (dieta)
Prendiamo l'esempio di una dieta.
Per ogni cibi mi servono 3 informazioni principali:
 * Quantità di grassi
 * Quantità di carboidrati
 * Quantità di proteine

Ogni cibo lo possiamo quindi rappresentare con una terna di valori che rappresentano queste quantità.
$$
cibo_A = (30g, 20g, 40g)
$$
Generalizzando il concetto posso rappresentare un cibo con una n-upla, cioè n numeri dove ogni numero ha un suo significato e il cui insieme mi rappresenta le caratteristiche di questo cibo che a me interessano.
###Coppie (in $R^2$)
Prendendo due coppie $(a, b), (c, d) \in R^2$
Posso definire delle operazioni
$$
\begin{align}
\text{somma} & = (a, b) + (c, d) = (a+c, b+d) \\
\text{prodotto con un numero} & = (a, b) * c = (ac, bc)
\end{align}
$$
Noto subito che anche qui si trovano delle proprietà
####Proprietà della somma
 |Proprietà              |Descrizione                                  |
 |-----------------------|---------------------------------------------|
 |Commutativa            |$(a, b) + (c, d) = (c, d) + (a, b)$          |
 |Associativa            |$((a,b)+(c,d))+(e,f) = (a,b)+((c,d)+(e,f))$  |
 |Esistenza del Neutro   |$(a,b) + (0,0) = (a,b)$                      |
 |Esistenza dell'Opposto |$(a,b) +(-a, -b) = (0, 0)$                   |

####Proprietà del prodotto per un numero
 |Proprietà                 |Descrizione                        |
 |--------------------------|-----------------------------------|
 |Distributiva alle coppie  | $m((a,b) + (c,d))= m(a,b)+m(c,d)$ |
 |Distributiva numeri       | $(m+n)(a,b) = m(a,b)+n(a,b)$      |
 |Esistenza del Neutro      | $1(a,b) = (a,b)$                  |
 |Composizione dei prodotti | $(mn)(a,b)=m(n(a,b))$             |

E' immediato vedere che:
 * Queste proprietà valgono indipendente del numero di elementi (n-upla)
 * Sono le stesse proprietà dei vettori del piano

Algebra 17 Introduzione concetto di spazio vettoriale.ipynb


# Bayesian filtering for nonlinear Gaussian dynamical systems

Wouter Kouw | Last update: 09-12-2021

Consider a discrete-time dynamic system with observations $y_k \in \mathbb{R}^{d_y}$ and latent states $x_k \in \mathbb{R}^{d_x}$. These are subject to the following dynamics:

$$\begin{align}
x_k =&\ f(x_{k-1}) + q_{k-1} \\
y_k =&\ h(x_k) + r_{k} \, ,
\end{align}$$

where $f$ and $h$ are static nonlinear functions, $q_{k-1}$ represents process noise and $r_{k}$ is measurement noise. Both sources of noise are Gaussian distributed with zero mean:

$$q_{k-1} \sim \mathcal{N}(0, Q) \, , \qquad r_k \sim \mathcal{N}(0, R) \, ,$$

with covariance matrices $Q$ and $R$.

## Problem: tracking a noisy pendulum

We will be tracking a pendulum whose observations are noisy. You can find this example in [Bayesian Filtering & Smoothing](https://www.cambridge.org/core/books/bayesian-filtering-and-smoothing/C372FB31C5D9A100F8476C1B23721A67) by Simon Särkkä (Ex 3.7). The state transition has the following form:

$$\begin{align*}
\underbrace{\begin{bmatrix} x_{1,k} \\ x_{2,k} \end{bmatrix}}_{x_k} = \underbrace{\begin{bmatrix} x_{1,k-1} + x_{2,k-1} \Delta t \\ x_{2,k-1} -G \sin(x_{1,k-1}) \Delta t \end{bmatrix}}_{f(x_{k-1})} + q_{k-1}
\end{align*}$$

where $x_1$ represents the angle of the pendulum, $x_2$ represents the change in angle and $G$ is gravitational acceleration.

The measurement function is described as:

$$\begin{align*}
y_k = \underbrace{\sin(x_{1,k})}_{h(x_k)} + r_k
\end{align*}$$

We cannot observe the change in angle directly, only the height of the angle itself (i.e. $\sin(x_{1})$).

#### Data-generating process

```julia
using LinearAlgebra
using ProgressMeter
using Plots
pyplot();
```

```julia
# Length of time-series
T = 400
Δt = 0.01

# Gravitational acceleration
G = 9.81

# Process noise covariance
qc = 1e-3
Q = [qc*Δt^3/3 qc*Δt^2/2; qc*Δt^2/2 qc*Δt]

# Measurement noise variance
R = 1e-2

# Initial states
x0 = [1.0, 0.0]

# Initialize data array
states = zeros(2,T)
observations = zeros(T,)

# Initialize previous state variable
prev_state = x0

for k = 1:T

    # State transition
    states[1,k] = prev_state[1] + prev_state[2]*Δt
    states[2,k] = prev_state[2] - G*sin(prev_state[1])*Δt

    # Add process noise
    states[:,k] = states[:,k] + cholesky(Q).L*randn(2)

    # Observation with added measurement noise
    observations[k] = sin(states[1,k]) + sqrt(R)*randn(1)[1]

    # Update "previous state"
    prev_state = states[:,k]

end
```

```julia
# Inspect data
plot((1:T).*Δt, states[1,:], linewidth=3, color="blue", xlabel="time [Δt = "*string(Δt)*"]", ylabel="height pendulum", label="states", size=(800,400))
scatter!((1:T).*Δt, observations, color="black", markersize=2,label="observed")
```

### Model specification

We can convert the state transition and observation likelihood from the system dynamics to probabilistic form by integrating out the noise variables:

$$\begin{align}
x_k \sim&\ \mathcal{N}(f(x_{k-1}), Q) \\
y_k \sim&\ \mathcal{N}(g(x_k), R) \, .
\end{align}$$

The noise covariance matrices are now the covariance matrices of the stochastic state transition and likelihood. In both cases, we will use an unscented transform to approximate the result of applying the nonlinear function with a Gaussian distribution.

We will need priors for the unknown variables. We will be doing recursive estimation (filtering) in this notebook, we need only define a prior for the "previous state" variable $x_{k-1}$:

$$\begin{align}
x_{k-1} \sim \mathcal{N}(m_k, W_k^{-1}) \, ,
\end{align}$$

where $W_k$ is a precision matrix. In Bayesian inference, we often work with precision matrices instead of covariance matrices as that tends to require less matrix inversions. We will model the unknown noise precision matrices with a gamma distribution (univariate measurement noise) and a Wishart distribution (multivariate process noise). These are the conditionally [conjugate priors](https://en.wikipedia.org/wiki/Conjugate_prior) for Gaussian likelihoods:

$$\begin{align}
Q^{-1} \sim&\ \mathcal{W}(V_Q, n_Q) \\
R^{-1} \sim&\ \Gamma(a_r, b_r) \, .
\end{align}$$

With a likelihood and a set of priors, we can specify a model in a probabilistic programming language (here [ForneyLab.jl](https://github.com/biaslab/ForneyLab.jl)) and automatically infer (approximate) posteriors.

```julia
using ForneyLab
```

```julia
fg = FactorGraph()

# Measurement noise precision
@RV Ri ~ Gamma(placeholder(:a_R), placeholder(:b_R))

# Process noise precision
@RV Qi ~ Wishart(placeholder(:V_Q, dims=(2,2)), placeholder(:n_Q))

# Previous state vector
@RV x_kmin1 ~ GaussianMeanPrecision(placeholder(:m_x, dims=(2,)), placeholder(:W_x, dims=(2,2)))

# Nonlinear state transition function
f(x) = [x[1] + x[2]*Δt, x[2] - G*sin(x[1])*Δt]
@RV fx_k ~ Nonlinear{Unscented}(x_kmin1; g=f)

# Add process noise
@RV x_k ~ GaussianMeanPrecision(fx_k, Qi)

# Mask the state vector
@RV x_1 = dot([1., 0.], x_k)

# Nonlinear observation function
h(x) = sin(x)
@RV hx_k ~ Nonlinear{Unscented}(x_1; g=h)

# Add measurement noise
@RV y_k ~ GaussianMeanPrecision(hx_k, Ri)

# Indicate y_k will be observed
placeholder(y_k, :y_k);

ForneyLab.draw(fg)
```

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.47.1 (20210417.1919)
 -->
<!-- Title: G Pages: 1 -->
<svg width="585pt" height="863pt"
 viewBox="0.00 0.00 584.50 863.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 859)">
<title>G</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-859 580.5,-859 580.5,4 -4,4"/>
<!-- 11769799603831642054 -->
<g id="node1" class="node">
<title>11769799603831642054</title>
<polygon fill="none" stroke="black" points="187.5,-288 115.5,-288 115.5,-216 187.5,-216 187.5,-288"/>
<text text-anchor="middle" x="151.5" y="-254.8" font-family="Times New Roman,serif" font-size="9.00">g{Unscented}</text>
<text text-anchor="middle" x="151.5" y="-244.8" font-family="Times New Roman,serif" font-size="9.00">nonlinear_1</text>
</g>
<!-- 7200964442915660904 -->
<g id="node13" class="node">
<title>7200964442915660904</title>
<polygon fill="none" stroke="black" points="188,-171 81,-171 81,-99 188,-99 188,-171"/>
<text text-anchor="middle" x="134.5" y="-137.8" font-family="Times New Roman,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="134.5" y="-127.8" font-family="Times New Roman,serif" font-size="9.00">gaussianmeanprecision_1</text>
</g>
<!-- 11769799603831642054--7200964442915660904 -->
<g id="edge9" class="edge">
<title>11769799603831642054--7200964442915660904</title>
<path fill="none" stroke="black" d="M146.31,-215.87C144.22,-201.74 141.81,-185.46 139.72,-171.31"/>
<text text-anchor="start" x="144.5" y="-191.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">x_kmin1</text>
<text text-anchor="start" x="118.72" y="-173.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="125.31" y="-209.47" font-family="Times New Roman,serif" font-size="8.00">2 in1 </text>
</g>
<!-- 15707164406108971955 -->
<g id="node2" class="node">
<title>15707164406108971955</title>
<polygon fill="none" stroke="black" points="167.5,-522 95.5,-522 95.5,-450 167.5,-450 167.5,-522"/>
<text text-anchor="middle" x="131.5" y="-488.8" font-family="Times New Roman,serif" font-size="9.00">dot</text>
<text text-anchor="middle" x="131.5" y="-478.8" font-family="Times New Roman,serif" font-size="9.00">dotproduct_1</text>
</g>
<!-- 13099004617794688251 -->
<g id="node8" class="node">
<title>13099004617794688251</title>
<polygon fill="lightgrey" stroke="black" points="83.5,-396 29.5,-396 29.5,-342 83.5,-342 83.5,-396"/>
<text text-anchor="middle" x="56.5" y="-366.8" font-family="Times New Roman,serif" font-size="9.00">clamp_1</text>
</g>
<!-- 15707164406108971955--13099004617794688251 -->
<g id="edge5" class="edge">
<title>15707164406108971955--13099004617794688251</title>
<path fill="none" stroke="black" d="M95.12,-469.18C79.98,-460.55 63.98,-448.26 55.5,-432 49.82,-421.12 49.4,-407.63 50.68,-396.02"/>
<text text-anchor="start" x="55.5" y="-425.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">clamp_1</text>
<text text-anchor="start" x="29.68" y="-398.62" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="74.12" y="-471.78" font-family="Times New Roman,serif" font-size="8.00">3 in2 </text>
</g>
<!-- 12615399910196930301 -->
<g id="node14" class="node">
<title>12615399910196930301</title>
<polygon fill="none" stroke="black" points="306,-405 199,-405 199,-333 306,-333 306,-405"/>
<text text-anchor="middle" x="252.5" y="-371.8" font-family="Times New Roman,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="252.5" y="-361.8" font-family="Times New Roman,serif" font-size="9.00">gaussianmeanprecision_2</text>
</g>
<!-- 15707164406108971955--12615399910196930301 -->
<g id="edge10" class="edge">
<title>15707164406108971955--12615399910196930301</title>
<path fill="none" stroke="black" d="M167.8,-450.5C182.84,-436.21 200.26,-419.65 215.38,-405.28"/>
<text text-anchor="start" x="196.5" y="-425.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">x_k</text>
<text text-anchor="start" x="194.38" y="-407.88" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="146.8" y="-444.1" font-family="Times New Roman,serif" font-size="8.00">2 in1 </text>
</g>
<!-- 7572076714573308162 -->
<g id="node3" class="node">
<title>7572076714573308162</title>
<polygon fill="lightgrey" stroke="black" points="383.5,-162 303.5,-162 303.5,-108 383.5,-108 383.5,-162"/>
<text text-anchor="middle" x="343.5" y="-132.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_V_Q</text>
</g>
<!-- 18314240599931604027 -->
<g id="node4" class="node">
<title>18314240599931604027</title>
<polygon fill="lightgrey" stroke="black" points="77,-54 0,-54 0,0 77,0 77,-54"/>
<text text-anchor="middle" x="38.5" y="-24.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_m_x</text>
</g>
<!-- 16861794541815506586 -->
<g id="node5" class="node">
<title>16861794541815506586</title>
<polygon fill="none" stroke="black" points="379.5,-288 307.5,-288 307.5,-216 379.5,-216 379.5,-288"/>
<text text-anchor="middle" x="343.5" y="-254.8" font-family="Times New Roman,serif" font-size="9.00">W</text>
<text text-anchor="middle" x="343.5" y="-244.8" font-family="Times New Roman,serif" font-size="9.00">wishart_1</text>
</g>
<!-- 16861794541815506586--7572076714573308162 -->
<g id="edge8" class="edge">
<title>16861794541815506586--7572076714573308162</title>
<path fill="none" stroke="black" d="M343.5,-215.87C343.5,-198.57 343.5,-178.06 343.5,-162.15"/>
<text text-anchor="start" x="343.5" y="-191.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">V_Q</text>
<text text-anchor="start" x="322.5" y="-164.75" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="329.5" y="-209.47" font-family="Times New Roman,serif" font-size="8.00">2 v </text>
</g>
<!-- 13333063496959055474 -->
<g id="node15" class="node">
<title>13333063496959055474</title>
<polygon fill="lightgrey" stroke="black" points="576.5,-162 498.5,-162 498.5,-108 576.5,-108 576.5,-162"/>
<text text-anchor="middle" x="537.5" y="-132.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_n_Q</text>
</g>
<!-- 16861794541815506586--13333063496959055474 -->
<g id="edge1" class="edge">
<title>16861794541815506586--13333063496959055474</title>
<path fill="none" stroke="black" d="M379.54,-229.64C413.28,-209.63 463.59,-179.81 498.33,-159.22"/>
<text text-anchor="start" x="446.5" y="-191.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">n_Q</text>
<text text-anchor="start" x="477.33" y="-161.82" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="379.54" y="-232.24" font-family="Times New Roman,serif" font-size="8.00">3 nu </text>
</g>
<!-- 14420421744326312879 -->
<g id="node6" class="node">
<title>14420421744326312879</title>
<polygon fill="lightgrey" stroke="black" points="271,-54 192,-54 192,0 271,0 271,-54"/>
<text text-anchor="middle" x="231.5" y="-24.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_W_x</text>
</g>
<!-- 2576489981286358758 -->
<g id="node7" class="node">
<title>2576489981286358758</title>
<polygon fill="lightgrey" stroke="black" points="360,-513 283,-513 283,-459 360,-459 360,-513"/>
<text text-anchor="middle" x="321.5" y="-483.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_b_R</text>
</g>
<!-- 18030336638603525491 -->
<g id="node9" class="node">
<title>18030336638603525491</title>
<polygon fill="none" stroke="black" points="169.5,-639 97.5,-639 97.5,-567 169.5,-567 169.5,-639"/>
<text text-anchor="middle" x="133.5" y="-605.8" font-family="Times New Roman,serif" font-size="9.00">g{Unscented}</text>
<text text-anchor="middle" x="133.5" y="-595.8" font-family="Times New Roman,serif" font-size="9.00">nonlinear_2</text>
</g>
<!-- 18030336638603525491--15707164406108971955 -->
<g id="edge2" class="edge">
<title>18030336638603525491--15707164406108971955</title>
<path fill="none" stroke="black" d="M132.89,-566.87C132.64,-552.74 132.36,-536.46 132.11,-522.31"/>
<text text-anchor="start" x="133.5" y="-542.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">x_1</text>
<text text-anchor="start" x="111.11" y="-524.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="111.89" y="-560.47" font-family="Times New Roman,serif" font-size="8.00">2 in1 </text>
</g>
<!-- 4789214632737044419 -->
<g id="node10" class="node">
<title>4789214632737044419</title>
<polygon fill="none" stroke="black" points="357.5,-639 285.5,-639 285.5,-567 357.5,-567 357.5,-639"/>
<text text-anchor="middle" x="321.5" y="-605.8" font-family="Times New Roman,serif" font-size="9.00">Gam</text>
<text text-anchor="middle" x="321.5" y="-595.8" font-family="Times New Roman,serif" font-size="9.00">gamma_1</text>
</g>
<!-- 4789214632737044419--2576489981286358758 -->
<g id="edge11" class="edge">
<title>4789214632737044419--2576489981286358758</title>
<path fill="none" stroke="black" d="M321.5,-566.87C321.5,-549.57 321.5,-529.06 321.5,-513.15"/>
<text text-anchor="start" x="321.5" y="-542.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">b_R</text>
<text text-anchor="start" x="300.5" y="-515.75" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="308.5" y="-560.47" font-family="Times New Roman,serif" font-size="8.00">3 b </text>
</g>
<!-- 11609603851023295769 -->
<g id="node12" class="node">
<title>11609603851023295769</title>
<polygon fill="lightgrey" stroke="black" points="552,-513 475,-513 475,-459 552,-459 552,-513"/>
<text text-anchor="middle" x="513.5" y="-483.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_a_R</text>
</g>
<!-- 4789214632737044419--11609603851023295769 -->
<g id="edge14" class="edge">
<title>4789214632737044419--11609603851023295769</title>
<path fill="none" stroke="black" d="M357.61,-580.37C391.07,-560.33 440.69,-530.61 474.91,-510.11"/>
<text text-anchor="start" x="423.5" y="-542.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">a_R</text>
<text text-anchor="start" x="453.91" y="-512.71" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="357.61" y="-582.97" font-family="Times New Roman,serif" font-size="8.00">2 a </text>
</g>
<!-- 10856907810642589554 -->
<g id="node11" class="node">
<title>10856907810642589554</title>
<polygon fill="lightgrey" stroke="black" points="265,-855 188,-855 188,-801 265,-801 265,-855"/>
<text text-anchor="middle" x="226.5" y="-825.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_y_k</text>
</g>
<!-- 2184084146676702280 -->
<g id="node16" class="node">
<title>2184084146676702280</title>
<polygon fill="none" stroke="black" points="280,-756 173,-756 173,-684 280,-684 280,-756"/>
<text text-anchor="middle" x="226.5" y="-722.8" font-family="Times New Roman,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="226.5" y="-712.8" font-family="Times New Roman,serif" font-size="9.00">gaussianmeanprecision_3</text>
</g>
<!-- 10856907810642589554--2184084146676702280 -->
<g id="edge7" class="edge">
<title>10856907810642589554--2184084146676702280</title>
<path fill="none" stroke="black" d="M226.5,-800.8C226.5,-787.31 226.5,-770.68 226.5,-756.1"/>
<text text-anchor="start" x="226.5" y="-776.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">y_k</text>
<text text-anchor="start" x="205.5" y="-758.7" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="205.5" y="-794.4" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
</g>
<!-- 7200964442915660904--18314240599931604027 -->
<g id="edge4" class="edge">
<title>7200964442915660904--18314240599931604027</title>
<path fill="none" stroke="black" d="M89.97,-98.74C83.82,-93.09 77.8,-87.08 72.5,-81 65.39,-72.84 58.69,-63.07 53.15,-54.15"/>
<text text-anchor="start" x="72.5" y="-74.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">m_x</text>
<text text-anchor="start" x="32.15" y="-56.75" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="73.97" y="-92.34" font-family="Times New Roman,serif" font-size="8.00">2 m </text>
</g>
<!-- 7200964442915660904--14420421744326312879 -->
<g id="edge12" class="edge">
<title>7200964442915660904--14420421744326312879</title>
<path fill="none" stroke="black" d="M166.53,-99C179.88,-84.4 195.13,-67.75 207.49,-54.23"/>
<text text-anchor="start" x="189.5" y="-74.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">W_x</text>
<text text-anchor="start" x="186.49" y="-56.83" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="151.53" y="-92.6" font-family="Times New Roman,serif" font-size="8.00">3 w </text>
</g>
<!-- 12615399910196930301--11769799603831642054 -->
<g id="edge15" class="edge">
<title>12615399910196930301--11769799603831642054</title>
<path fill="none" stroke="black" d="M221.65,-332.87C209.23,-318.74 194.94,-302.46 182.51,-288.31"/>
<text text-anchor="start" x="205.5" y="-308.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">fx_k</text>
<text text-anchor="start" x="161.51" y="-290.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="205.65" y="-326.47" font-family="Times New Roman,serif" font-size="8.00">2 m </text>
</g>
<!-- 12615399910196930301--16861794541815506586 -->
<g id="edge13" class="edge">
<title>12615399910196930301--16861794541815506586</title>
<path fill="none" stroke="black" d="M280.3,-332.87C291.48,-318.74 304.36,-302.46 315.56,-288.31"/>
<text text-anchor="start" x="301.5" y="-308.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">Qi</text>
<text text-anchor="start" x="294.56" y="-290.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="265.3" y="-326.47" font-family="Times New Roman,serif" font-size="8.00">3 w </text>
</g>
<!-- 2184084146676702280--18030336638603525491 -->
<g id="edge3" class="edge">
<title>2184084146676702280--18030336638603525491</title>
<path fill="none" stroke="black" d="M172.74,-690.34C163.56,-683.42 154.93,-675.29 148.5,-666 143.01,-658.08 139.53,-648.38 137.31,-639.03"/>
<text text-anchor="start" x="148.5" y="-659.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">hx_k</text>
<text text-anchor="start" x="116.31" y="-641.63" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="156.74" y="-692.94" font-family="Times New Roman,serif" font-size="8.00">2 m </text>
</g>
<!-- 2184084146676702280--4789214632737044419 -->
<g id="edge6" class="edge">
<title>2184084146676702280--4789214632737044419</title>
<path fill="none" stroke="black" d="M255.52,-683.87C267.2,-669.74 280.64,-653.46 292.33,-639.31"/>
<text text-anchor="start" x="277.5" y="-659.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">Ri</text>
<text text-anchor="start" x="271.33" y="-641.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="240.52" y="-677.47" font-family="Times New Roman,serif" font-size="8.00">3 w </text>
</g>
</g>
</svg>

### Compile inference algorithm

We cannot use exact Bayesian inference here and will therefore switch to an approximate inference scheme: variational message passing.

```julia
# Define a factorization for the posterior
q = PosteriorFactorization(x_k, fx_k, hx_k, Qi, Ri, ids=[:xk, :fk, :hk, :Qi, :Ri])

# Define a variational message passing procedure
algorithm = messagePassingAlgorithm([x_k, fx_k, hx_k, Qi, Ri], q)

# Compile message passing procedure to an inference algorithm
code = algorithmSourceCode(algorithm);

# Import compiled functions to workspace
eval(Meta.parse(code));

# Print compiled functions (uncomment when desired)
# println(code)
```

### Execute inference algorithm

We execute the algorithm in an online fashion: after each timestep the posteriors for the state and noise precisions are used as priors for the next timestep.

```julia
# Number of iterations
num_iterations = 5

# Parameters for prior distributions
m_x_0 = [1., 0.]
W_x_0 = [1. 0.;0. 1.]
a_R_0 = 10.0
b_R_0 = 0.1
V_Q_0 = [10. 0.1;0.1 1.]
n_Q_0 = 10.0

# Initialize array to track Free Energy objective
F = zeros(T,num_iterations)

# Initialize arrays for storing parameter estimates
m_x_k = zeros(2,T)
W_x_k = zeros(2,2,T)
a_R_k = zeros(T,)
b_R_k = zeros(T,)
V_Q_k = zeros(2,2,T)
n_Q_k = zeros(T,)

# Initialize previous parameter estimates
m_x_kmin1 = m_x_0
W_x_kmin1 = W_x_0
a_R_kmin1 = a_R_0
b_R_kmin1 = b_R_0
V_Q_kmin1 = V_Q_0
n_Q_kmin1 = n_Q_0

# Initialize data dictionary
data = Dict()

# Initialize marginal distributions
marginals = Dict()
marginals[:x_kmin1] = ProbabilityDistribution(Multivariate, GaussianMeanVariance, m=m_x_0, v=W_x_0)
marginals[:hx_k] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=[0., 0.], w=[1. 0.;0. 1.])
marginals[:fx_k] = ProbabilityDistribution(Multivariate, GaussianMeanVariance, m=[0., 0.], v=[1. 0.;0. 1.])
marginals[:x_k] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=[0., 0.], w=[1. 0.;0. 1.])
marginals[:x_1] = ProbabilityDistribution(Univariate, GaussianMeanVariance, m=0.0, v=1.0)
marginals[:Ri] = ProbabilityDistribution(Univariate, Gamma, a=a_R_0, b=b_R_0)
marginals[:Qi] = ProbabilityDistribution(Wishart, v=V_Q_0, nu=n_Q_0)

# Recursive estimation procedure (posteriors at k => priors at k+1)
@showprogress for k = 1:T

    # Store data for current time-step
    data[:y_k] = observations[k]
    data[:m_x] = m_x_kmin1
    data[:W_x] = W_x_kmin1
    data[:a_R] = a_R_kmin1
    data[:b_R] = b_R_kmin1
    data[:V_Q] = V_Q_kmin1
    data[:n_Q] = n_Q_kmin1

    # Iteratively update beliefs
    for tt = 1:num_iterations

        # Update recognition distributions
        stepxk!(data, marginals)
        stepfk!(data, marginals)
        stephk!(data, marginals)
        stepRi!(data, marginals)
        stepQi!(data, marginals)
    end

    # Update book-keeping
    m_x_kmin1 = m_x_k[:,k] = ForneyLab.unsafeMean(marginals[:x_k])
    W_x_kmin1 = W_x_k[:,:,k] = ForneyLab.unsafePrecision(marginals[:x_k])
    a_R_kmin1 = a_R_k[k] = marginals[:Ri].params[:a]
    b_R_kmin1 = b_R_k[k] = marginals[:Ri].params[:b]
    V_Q_kmin1 = V_Q_k[:,:,k] = marginals[:Qi].params[:v]
    n_Q_kmin1 = n_Q_k[k] = marginals[:Qi].params[:nu]

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:09

### Inspect results
We can plot the estimates of the angle and the change in the angle, over time, including the uncertainty of the estimates.

```julia
# Plot true states and overlay estimates
plot((1:T).*Δt, states[1,:], linewidth=2, xlabel="time (Δt = "*string(Δt)*")", ylabel="angle (α)", color="blue", label="states", size=(800,400))
plot!((1:T).*Δt, m_x_k[1,:], linewidth=3, color="red", ribbon=[sqrt.(inv.(W_x_k)[1,1,:]) sqrt.(inv.(W_x_k)[1,1,:])], alpha=0.6, label="estimated")
title!("Angle of the pendulum, over time")
```

```julia
# Plot true states and overlay estimates
plot((1:T).*Δt, states[2,:], linewidth=2, xlabel="time (Δt = "*string(Δt)*")", ylabel="change in angle (dα/dt)", color="blue", label="states", size=(800,400))
plot!((1:T).*Δt, m_x_k[2,:], linewidth=3, color="red", ribbon=[sqrt.(inv.(W_x_k)[2,2,:]) sqrt.(inv.(W_x_k)[2,2,:])], alpha=0.6, label="estimated", legend=:bottomright)
title!("Change in angle of the pendulum, over time")
```

Both the angle and the change in angle are tracked nicely.


Our goal is to simulate a isothermal sphere undergoing gravitational collapse. We can write down the force balance equation for the sphere, which describes the competition between gravitational infall versus thermal pressure support.
$$ -\frac{1}{\rho}\nabla P -\nabla \Phi_g =0$$
Using Poisson's equation ($\nabla \Phi_g^2 =4\pi G \rho$) and the isothermal equation of state ($P = \rho a_T^2$), we can rearrange the equation to the form of the isothermal Lane-Emden equation:

$$\frac{1}{\psi^2} \frac{d}{d\psi}\Bigg(\xi^2 \frac{d\psi}{d\xi}\Bigg)=e^{-\psi}$$
Using product rule, we can rewrite the Lane-Emden equation as:
\begin{align}
\frac{d^2\psi}{d\xi^2}+\frac{2}{\xi}\frac{d\psi}{d\xi}=e^{-\psi}
\end{align}
with boundary conditions as: $\psi(0)=0$ and $\frac{d\psi}{d\xi}(0)=0$.

The Lane-Emden equation describes a family of solution based on the single parameter $\xi_{max}$, which specifies the cloud radius and determine the density contrast $\rho_c/\rho_0$ (ratio between central density and edge of the sphere). Analytically, the sphere is marginally stable (i.e. just at the verge of collpasing) when $\rho_c/\rho_0$=14.1 [1]. This marginally-stable state is called the Bonner-Ebert sphere. Numerically, we can see that a cloud radius of  $\xi$=6.451 corresponds to a density contrast of $\rho_c/\rho_0$=14.038. The singular isothermal sphere case corresponds to $\xi_{max}=\infty$. [1]

We want to solve the Lane-Emden equation to figure out what is the density distribution that yields a marginally-stable sphere, but the Lane-Emden solution can not be solved in closed form, so instead we need to do numerical intergration to get the solution.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

We start integrating at a small poisitive number (1e-6) to avoid div-by-0 error and integrate up to xi_max, with interval of $\Delta \xi$ = 0.01.

```python
xi_max = 6.451
from scipy import integrate
def solvr(Y, t):
    return [Y[1], exp(-Y[0])-2/t*Y[1]]
xi = np.arange(1e-6, xi_max, 0.01)
IC = [0, 0] # initial conditions for psi and psi prime
psi = integrate.odeint(solvr, IC, xi)
```

For a spherically symmetric cloud, $\rho(r)=\rho_c e^{-\psi}$, so we can use the numerically integrated results for $\psi$ to compute the density profile.

```python
rho = np.exp(-psi[:,0])
plt.plot(xi,psi[:,0],label ='$\psi$')
plt.plot(xi,rho,label =r'$\rho/\rho_c$')
plt.xlabel(r"$\xi$",fontsize=15)
plt.xlim(0,xi_max)
from matplotlib.legend_handler import HandlerLine2D
plt.legend(loc='upper left',prop={'size':13},numpoints=1)
```

Note that since ``rmax`` is the only parameter that needs to be adjusted for different models of the Lane-Emden solution. To make sure that the boundary of the sphere is properly filled with the Bonnert-Ebert values, we extend the density profile in ``density.txt`` to $\xi$= 20.

```python
xi = np.arange(1e-6, 20, 0.01)
psi = integrate.odeint(solvr, IC, xi)
rho = np.exp(-psi[:,0])
np.savetxt("FLASH_4.3/density.txt",rho)
```

Inside ``Simulation_initBlock.F90``, we can read in the values of the array
~~~fortran
  open(12,file="../density.txt")
  read(12,*,IOSTAT=io) dens_arr
  close(12)
~~~

Now we have a 1D array of non-dimensional density values from numerical integration, we need to assign each cell in the simulation box with the density values that corresponds to the right $\rho(r)$. Since we are working with a radial distribution on a cartesian grid, we don't have the exact radius value at each cell, so need to do linear interpolation to map the cells to somewhere between their closest density values. As explained by [Wikipedia](https://en.wikipedia.org/wiki/Linear_interpolation), linear interpolation basically weighs what the y values are  based on how close the x values lie to the previous and subseqeunt datapoints. In our case,

$$ \rho(r) = \rho_0 + (\rho_1-\rho_0)\frac{r-r_0}{r_1-r_0}$$

Translating this into our Simulation_init.F90 and picking dr ($\Delta \xi$) as 0.01 to correspond with np.arange used in the numerical integration:

    if (rc <= rmax) then
       rc0 = int(rc/dr)+1 !to prevent from hitting index 0 which is yields zero density
       rho0 =  rho_c*dens_arr(rc0,1)
       rho1 = rho_c*dens_arr(rc0+1,1)
       rc0 = rc0*dr
       rc1 = rc0+dr
       rhoZone = fattening_factor*(rho0+(rho1-rho0)*(rc-rc0)/(rc1-rc0) )!linear interpolation

Simulation_init.F90 reads in values up to ``rmax`` $\xi$= 16.90 for initialize the sphere, note that this has a much higher density contrast than the Bonner-Ebert sphere. This ensures that the sphere in our toy problem will actually collapse in a short time. We also multiply the sphere's density by a "fattening_factor" of 100 for the same reason.

```python
idx = np.where(np.isclose(xi,16.9))[0][0]
print "At xi = ", xi[idx]
print "Density is: ", rho[idx]*rho_c, "g/cm^3"
print "density contrast is ", 1./rho[idx]
```

    At xi =  16.900001
    Density is:  7.2918475616e-20 g/cm^3
    density contrast is  150.853400418

We can now run our simulation to check that the sphere is initialized with the right density distribution.

```python
cd ~/dlee/FLASH4.3/object/
```

    /global/u2/d/dorislee/dlee/FLASH4.3/object

```python
import yt
from matplotlib.legend_handler import HandlerLine2D
from mpl_toolkits.axes_grid1 import AxesGrid
from yt.mods import *
yt.mylog.setLevel(50)
```

```python
def compareplot1Dprofile(timestep):
    fattening_factor=100
    rho_c = 1.1e-19*fattening_factor
    pf= yt.load("sphere_hdf5_chk_{}".format(str(timestep).zfill(4)))
    sp = pf.sphere(pf.domain_center, (0.32,"pc"))
    rp = yt.create_profile(sp,'radius','density')
    plt.plot(rp.x.value*1.05e-17,rp["density"].in_units("g/cm**3").value/rho_c,label="Simulation")
    xi_max = 6.451
    def solvr(Y, t):
        return [Y[1], exp(-Y[0])-2/t*Y[1]]
    xi = np.arange(1e-6, xi_max, 0.01) #start at small poisitive number to avoid div-by-0
    psi = integrate.odeint(solvr, [0, 0], xi)
    rho = np.exp(-psi[:,0])
    plt.plot(xi,rho,"--",label ='Integration',color="red",linewidth=5)
    plt.xlabel(r"$\xi$",fontsize=15)
    plt.ylabel(r"$\rho/\rho_c$",fontsize=15)
    plt.legend(loc='upper right',prop={'size':12},numpoints=1)
```

```python
compareplot1Dprofile(0)
```

## Reference:

[1] Stahler, Steven William., and F. Palla. *The Formation of Stars*. pg. 243-248. Weinheim: Wiley-VCH, 2004.
[2] Typanski,Nathan, *Solving a second-order ODE with NumPy and SciPy*. Online. [Link](http://www.nathantypanski.com/blog/2014-08-23-ode-solver-py.html)

tutorial/3-Lane Emden Numerical Solution.ipynb


<figure>
<center>

</center>
</figure>

# Regression with Automatic Differentiation in TensorFlow

# Task 1: TensorFlow

```python
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

print('Using TensorFlow version:', tf.__version__)
print('Devices available:', tf.config.list_physical_devices())
```

# Task 2: Constants

```python

```

```python

```

```python

```

```python

```

# Task 3: Variables

```python

```

```python

```

```python

```

# Task 4: Automatic Differentiation

Let's take a simple equation as an example:
\begin{equation}
y = x^3 ; \frac{dy}{dx} = 3x^2
\end{equation}

```python

```

What about higher order gradients?

\begin{equation}
y = x^3 ; \frac{dy}{dx} = 3x^2 ; \frac{d^2 y}{dx^2} = 6x
\end{equation}

```python

```

# Task 5: Watching Tensors

```python

```

```python

```

# Task 6: Persistent Tape

\begin{equation}
y = x^3 ; z = 2y ; \frac{dz}{dx} = \frac{dz}{dy} . \frac{dy}{dx}
\end{equation}

```python

```

# Task 7: Generating Data for Linear Regression

Solve a simple linear equation:

\begin{equation}
y = wx + b
\end{equation}

```python

```

```python

```

# Task 8: Linear Regression

```python

```

```python

```


```python
import numpy as np
A = np.random.rand(3,5)
A = np.arange(15).reshape(3,5)
B = np.random.rand(5,2)
B = np.arange(10).reshape(5,2)
M = np.empty((3,2))

# M_{ij} - \sum_{k} A_{ik}*B_{kj} = A_{i,}*B_{kj}
for i in range(3):
    for j in range(2):
        total = 0
        for k in range(5):
            total += A[i,k]*B[k,j]
        M[i,j] = total
M
```

    array([[ 60.,  70.],
           [160., 195.],
           [260., 320.]])

```python
# ik is A index
# kj is B index
# ij is outptu index
# i j is free index
# k is dummy index
#### repeating letter in a different term
####   will be multiplied and will be the output.
####   mulipied summation
# M_{ik} = A_{ik}B_{kj}
np.einsum('ik,kj->ij',A,B)
```

    array([[ 60,  70],
           [160, 195],
           [260, 320]])

```python
a = np.arange(5)
b = np.arange(3)
# free index: i,j(Used in output)
# Summation(dummy) index: None (All used in output)
# outer[i,j] = a[i] * b[j]
np.einsum('i,j->ij',a,b)
```

    array([[0, 0, 0],
           [0, 1, 2],
           [0, 2, 4],
           [0, 3, 6],
           [0, 4, 8]])

```python
outer_product = np.empty((5,3))
for i in range(5):
    for j in range(3):
        total = 0
        total += a[i]*b[j]
        outer_product[i,j] = total
outer_product
```

    array([[0., 0., 0.],
           [0., 1., 2.],
           [0., 2., 4.],
           [0., 3., 6.],
           [0., 4., 8.]])

```python
# Ommiting a letter measn that axis will be summed.
# \sum_{i}x[i]
x = np.ones(3)
np.einsum('i->',x)
```

    3.0

```python
x = np.arange(6).reshape(2,3)
np.einsum('ij->ji',x)

```

    array([[0, 2, 4],
           [1, 3, 5]])

```python
x
```

    array([[0, 1],
           [2, 3],
           [4, 5]])

```python
# Permutation of Tensors
np.einsum("ij->ji",x)
```

    array([[0, 2, 4],
           [1, 3, 5]])

```python
# Summation
np.einsum('ij->',x)
```

    15

```python
# column sum
np.einsum('ij->j',x)
```

    array([6, 9])

```python
# row sum
np.einsum('ij->i',x)
```

    array([1, 5, 9])

```python
# Matrix-Vector multiplication
v = np.arange(3)
# 3,2 -> 1,3
np.einsum('ij,i->ij',x,v)
```

    array([[ 0,  0],
           [ 2,  3],
           [ 8, 10]])

```python
x1 = np.arange(6).reshape(2,3)
# 2,3 -> 1,3
np.einsum('ij,k->ki',x1,v)
```

    array([[ 0,  0],
           [ 3, 12],
           [ 6, 24]])

```python
import sympy as sm
sm.Matrix(np.einsum('ij,jk->ik',x1,x))
```

$\displaystyle \left[\begin{matrix}10 & 13\\28 & 40\end{matrix}\right]$

```python

```

```python

```


# Toy SEIR model for COVID19 in Australia

Table of Contents

1  Context/Outline

2  Python set up

3  Assumptions - model/set-up

4  The model

4.1  The ODE model

4.2  Starting points

4.3  Use an ODE solver

4.4  Plot and describe

4.5  plot comparative

4.6  Run the model

## Context/Outline

This implements a simple SEIR model, where the population is compartmentalised into four mutually exclusive groups: S=susceptible, E=exposed, I=infected, and R=resistant. It is described by the following set of ordinary differential equations for a population of 1 (ie. we are working in population fractions):

$$\begin{align}
\frac{dS}{dt} & = -\beta SI \\
\frac{dE}{dt} & = \beta SI - \alpha E \\
\frac{dI}{dt} & = \alpha E - \gamma I \\
\frac{dR}{dt} & = \gamma I  \\
\end{align} $$
Where:
$$\begin{align}
\frac{d(S+E+I+R)}{dt} & = 0 \\
S+E+I+R & = 1 \\
\end{align}$$

## Python set up

```python
# imports
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from pathlib import Path
from io import StringIO
import scipy.integrate

#pandas
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999

# plotting
plt.style.use('ggplot')

# where to save the plot
CHART_DIRECTORY = '../charts'
Path(CHART_DIRECTORY).mkdir(parents=True, exist_ok=True)
CHART_DIRECTORY += '/!SIR-!ODE-'
```

## Assumptions - model/set-up

```python
N_DAYS = 500                # days - model period
STEP_SIZE = .01             # fraction of days
H = int(1 / STEP_SIZE)      # steps per day
POPULATION = 25_500_000     # Australian people
r0 = 2.5
e_period = 3                # exposure period - days
                            # to the point of being infectious,
                            # so less than to the point of having
                            # symptoms
i_period = 8                # infectious period - days

# Other
case_fatality_rate = 0.007  # proportion of those infected;
                            # I have seen estimates between
                            # 0.66% and 1.44%

# not used
hospitalisation_period = int(9 * H) # days
hospitalisation_rate = 0.06 # proportion of those infected
                            # hugely age dependent.
```

## The model

### The ODE model

```python
def SEIR_model(y, t, alpha, beta, gamma):
    s, e, i, r = y
    ds_dt = -beta * s * i
    de_dt = (beta * s * i) - (alpha * e)
    di_dt = (alpha * e) - (gamma * i)
    dr_dt = (gamma * i)
    return ds_dt, de_dt, di_dt, dr_dt
```

### Starting points

```python
def get_parameters(r0):
    # y - starting points - ensure sum to one
    y = np.zeros(4)
    y[1] = 1000 / POPULATION # e
    y[2] = 1000 / POPULATION # i
    y[3] = 0.0 # r
    y[0] = 1.0 - y[1] - y[2] - y[3] # s

    # t - times slices
    t = np.linspace(0, N_DAYS, N_DAYS*H)

    # parameters
    beta = r0 / i_period
    alpha = 1 / e_period
    gamma = 1 / i_period
    print(f'alpha: {alpha}, beta: {beta}, gamma: {gamma}')

    return y, t, alpha, beta, gamma
```

### Use an ODE solver

```python
def solve(r0):

    y0, t, alpha, beta, gamma = get_parameters(r0)

    solution = scipy.integrate.odeint(SEIR_model, y0,
        t, args=(alpha, beta, gamma))

    # put the solution into a DataFrame
    solution = pd.DataFrame(solution, columns = ['susceptible',
            'exposed', 'infectious', 'removed'])
    solution.index = solution.index / H # return index to days

    # calculate deaths
    solution['dead'] = solution['removed'] * case_fatality_rate
    solution['recovered'] = solution['removed'] - solution['dead']
    del solution['removed']

    return solution
```

### Plot and describe

```python
def plot_solution(solution, r0):
    ax = solution.plot(lw=2)
    t = f'COVID19: toy SEIR ODE solution, $R_0={r0}$'
    ax.set_title(t)
    ax.set_xlabel('Day')
    ax.set_ylabel('Population Proportion')
    fig = ax.figure
    fig.set_size_inches(8, 4)
    fig.tight_layout(pad=1)
    fig.savefig(f'{CHART_DIRECTORY}-{t}.png', dpi=125)
    plt.show()
    plt.close()

    # key results
    print('Susceptible at end: '+
          f'{int(solution.susceptible.iloc[-1]*POPULATION):,} '+
          f'({np.round(solution.susceptible.iloc[-1]*100, 1)}%)')
    print('Resistant at end: '+
          f'{int(solution.recovered.iloc[-1]*POPULATION):,} '+
          f'({np.round(solution.recovered.iloc[-1]*100, 1)}%)')
    print('Dead at end: '+
          f'{int(solution.dead.iloc[-1]*POPULATION):,} '+
          f'({np.round(solution.dead.iloc[-1]*100, 1)}%)')
```

### plot comparative

```python
def plot_comparative(data, title):

    data = data * POPULATION
    ax = data.plot(lw=2)
    ax.set_title(title)
    ax.set_xlabel('Day')
    ax.set_ylabel('Australian People')
    fig = ax.figure
    fig.set_size_inches(8, 4)
    fig.tight_layout(pad=1)
    fig.savefig(f'{CHART_DIRECTORY}-{title}.png', dpi=125)
    plt.show()
    plt.close()
```

### Run the model

```python
deaths = pd.DataFrame()
for r0 in [2.5, 1.5, 1.2]:
    print ('---------------')
    solution = solve(r0)
    plot_solution(solution, r0)
    deaths[f'$R_0={r0}$'] = solution['dead']

print ('---------------')
plot_comparative(deaths, 'Total Deaths')
plot_comparative(deaths.diff(H), 'Daily New Deaths')
```

The End


# Fourier smoothing

Notebook for AMath 570, Autumn Quarter 2015, R.J. LeVeque

See: http://faculty.washington.edu/rjl/classes/am570a2015/codes.html

### Aliasing in the DFT

For a periodic function $u(x)$ we can write
$$
u(x) = \frac{1}{2\pi}\sum_{k=-\infty}^\infty e^{ikx} \hat u_k, \quad\quad
\text{where}\quad \hat u_k = \int_0^{2\pi} e^{-ikx}u(x)\,dx.
$$
The Discrete Fourier Transform can be viewed as an approximation to this using $N$ points $x_j=jh$ with $h=2\pi/N$,
$$
v_j = u(x_j) = \frac{1}{2\pi} \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\, \hat v_k, \quad\quad
\text{where}\quad \hat v_k = \frac 1 N \sum_{j=1}^N e^{-ikx_j}\,v_j.
$$

Similar to Theorem 2 in SMM Chapter 4, there is an aliasing theorem that relates these:

**Theorem:**
$\quad\displaystyle \hat v_k = \sum_{\ell=-\infty}^\infty \hat u_{k + \ell N}.$

**Proof:**

$$
\begin{align}
v_j = u(x_j) &= \frac{1}{2\pi}\sum_{k=-\infty}^\infty e^{ikx} \hat u_k \\
&= \frac{1}{2\pi} \left( \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\, \hat u_k + \sum_{k=N/2+1}^{3N/2} e^{ikx_j}\, \hat u_k + \cdots \right) \\
\end{align}
$$
Now note that
$$
\sum_{k=N/2+1}^{3N/2} e^{ikx_j}\, \hat u_k = \sum_{k=-N/2+1}^{N/2} e^{i(k+N)x_j}\, \hat u_{k+N}
= \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\, \hat u_{k+N}
$$
since $e^{ikNx_j} = 1$ for all $j$.  Similarly for all other terms in the sum above, and so
$$
v_j = \frac{1}{2\pi} \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\,\left( \sum_{\ell=-\infty}^\infty \hat u_{k + \ell N} \right)
$$

We thus see how the Fourier series that *interpolates* at $N$ points is related to the full Fourier series.

### Trucation / projection

Another approach to getting a finite Fourier series would be to take the full series and *truncate* it to $M$ terms, defining
$$
u_M(x) = \frac{1}{2\pi} \sum_{k=-M/2+1}^{M/2} e^{ikx}\, \hat u_k, \quad\quad
\text{where}\quad \hat u_k = \int_0^{2\pi} e^{-ikx}u(x)\,dx.
$$
How does this relate to $v(x)$?  If we take $M=N$, then the function $u_N(v)$ has the same number of terms as $v(x)$, but different coefficients.  It no longer *interpolates* at equally spaced points.  Instead it gives the *best approximation* in the 2-norm (i.e. the *least squares* approximation.

If we let $F_M$ be the space of all Fourier sums with $M$ terms, then $u_M$ solves the problem
$$
\min_{p\in F_M} \|p - u\|_2, \quad\quad \text{where} \quad
\|w\|_2 = \left(\int_0^{2\pi} |w(x)|^2\, dx \right)^{1/2}.
$$
Recall that for complex valued functions, $|w(x)|^2 = w(x)\bar w(x)$ and also that the inner product of two functions can be defined by
$$
\langle w_1, w_2\rangle = \int_0^{2\pi} w_1(x) \bar w_2(x)\, dx
$$
We say two functions are *orthogonal* if their inner product is 0.  Note that $e^{ik_1x}$ and $e^{ik_2 x}$ are orthogonal for any two integers $k_1 \neq k_2$, since their inner product is
$$
\int_0^{2\pi} e^{ik_1x}e^{-ik_2x}\, dx = \int_0^{2\pi} e^{i(k_1-k_2)x}\,dx
$$
If $k_1 = k_2$ then the integral is $2\pi$, otherwise it is 0.

In general the solution to a least squares problem has the property that the residual $p(x) - u(x)$ must be orthogonal to the space $F_M$.  This can be seen by noting that the square of the error is $\langle p-u, p-u\rangle$, writing $p$ as a Fourier sum in terms of unknown coefficients $\hat p_k$, differentiating with respect to each $\hat p_k$ and setting these derivatives to zero.  You should find the requirement that $\langle p-u, e^{ikx}\rangle = 0$ for each wave number occuring in the sum.  Now writing out $p$ and $u$ as Fourier sums in this expression and using the orthogonality of the Fourier modes, this condition reduces to $\hat p_k = \hat u_k$ for the wave numbers in the finite sum $p$.

### Approximating $u_M$ by a finite sum

Truncating the full Fourier series requires being able to compute the integrals that define the $\hat u_k$.  If this can't be done, we could use a finite sum.  If we use $M$ function values then we are back to the discrete Fourier transform $v(x)$ defined above, which we might call $v_M(x)$.  But another approach is to compute $v_N(x)$ for some $N > M$ and then truncate this to $M$ terms.  If the original function is smooth enough that the coefficents decay quickly then $v_N$ will be very close to $u$.  This is often done as a way of smoothing noisy data.

## Exploration:

Below we start with a Gaussian function and add some high frequencies as a way of generating a function to experiment with.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

The next line makes it possible to zoom in on plots, only works with matplotlib 1.4 or later.

```python
# %matplotlib notebook   # uncomment to try this
```

```python
xfine = linspace(0, 2*pi, 10000)
```

```python
omega = [16]   # list of high frequencies to add (integers so noise is periodic)
m = len(omega)
a = m*[0.1]   # amplitudes

def u(x):
    u = exp(-(x-pi)**2)     # Gaussian
    for k in range(m):
        u = u + a[k] * cos(omega[k]*x)
    return u

figure(figsize=(10,5))
plot(xfine,u(xfine))
print 'wave numbers:', omega
print 'amplitudes:', a
```

```python
from scipy import fft,ifft
N = 34;
h = 2*pi/N
xj = linspace(0, 2*pi-h, N)  # note: fft assumes points are 0, h, 2h, ..., (N-1)h   (not h, 2h, ..., Nh)

M = N  # M = N ==> interpolation,  M < N gives smoothing  N and M shold be even

ik = 1j*hstack((range(0,N/2+1), range(-N/2+1,0)));   # i * wave number vector (matlab ordering)

vj = u(xj)
vj_hat = fft(vj) * h

def v(x):
    v = vj_hat[0] # constant term
    v += vj_hat[N/2] * cos(x*N/2) # N/2 term needs special handling: see SMM (3.4)

    for k in range(1, M/2):
        v += exp(1j*k*x)*vj_hat[k] + exp(-1j*k*x)*vj_hat[N-k]

        # alternatively, for real vj can instead write this as:
        # v += 2*real(vj_hat[k])*cos(k*x) - 2*imag(vj_hat[k])*sin(k*x)

    v = v / (2.*pi)
    v = real(v)  # should be real already up to rounding errors!
    return v

figure(figsize=(12,5))
subplot(1,2,1)
plot(xfine,u(xfine),'b')
plot(xj, u(xj), 'bo')
plot(xj, v(xj), 'r.')
plot(xfine,v(xfine),'r')
xlim(0, 2*pi)
title('Functions u(x) and v_M(x)')

subplot(1,2,2)
plot(abs(vj_hat[:N/2+1]))
title('Fourier transform')

```

## Things to try:

Try changing the cells above and re-executing.  Try to explain what you see in each case.

 - `omega = [16]` and $N = 34$ or $N = 30$ with $M=N$.  Note that in the latter case wave number 16 is aliased to 14.
 - `omega = [16]` and $N = 10$.  Now what aliasing do you see?
 - `omega = [16]` and $N = 34$ and $M = 10$.  Now there is no aliasing of wave number 16 and the smoothed function $v_M(x)$ is very close to the Gaussian.
 - `omega = [16, 25]` and $N = 64$ and $M = 12$.
 - `omega = [16, 25]` and $N = 30$ and $M = 12$.
 - `omega = [16, 25]` and $N = 30$ and $M = 30$.
 - `omega = [16, 25]` and $N = 12$ and $M = 12$.

```python

```


# Routh criterion calculator with symbolic variables

*Author: Rafael Ribeiro*

```python
from sympy import *
```

```python
def routh_criterion(coeffs):
    """
    Create a Routh-Hurwitz stability criterion matrix of a nth-degree polynomial

    Argument:
    coeffs -- list of coefficients from the denominator of a transfer function from the highest to the
              lowest power

    Return:

    routh_matrix -- Sympy matrix
    """

    first_line = []
    second_line = []

    for index in range(len(coeffs)):
        if (index%2 == 0) or (index==0):
            first_line.append(coeffs[index])
        else:
            second_line.append(coeffs[index])

    while(len(first_line)!=len(second_line)):
        second_line.append(0)

    routh_matrix = Matrix([first_line, second_line])

    line_n = []

    aux_matrix = [first_line, second_line]

    for line in range(len(coeffs)-2):

        for index in range(len(first_line)-1):

            line_n.append(
                simplify((routh_matrix[line+1,0]*routh_matrix[line,index+1] - routh_matrix[line+1,index+1]*routh_matrix[line,0] )\
                         /routh_matrix[line+1,0]))

        while(len(first_line)!=len(line_n)):

            line_n.append(0)

        aux_matrix.append(line_n)

        routh_matrix = Matrix(aux_matrix)

        line_n = []

    return routh_matrix
```

Let's take

$$s^{4} + 59s^{3} + 967s^{2} + (1000k + 3681)s + (50000k + 2772) $$

as an example. Note that the coefficients go from highest to lowest order.

```python
k = symbols('k')

coeffs = [1, 59, 967, 1000*k + 3681, 50000*k + 2772]
```

```python
routh_matrix = routh_criterion(coeffs)

routh_matrix
```

$\displaystyle \left[\begin{matrix}1 & 967 & 50000 k + 2772\\59 & 1000 k + 3681 & 0\\\frac{53372}{59} - \frac{1000 k}{59} & 50000 k + 2772 & 0\\\frac{250 \left(1000 k^{2} + 124359 k - 186813\right)}{250 k - 13343} & 0 & 0\\50000 k + 2772 & 0 & 0\end{matrix}\right]$

```python
routh_matrix[4,0]
```

$\displaystyle 50000 k + 2772$

```python
reduce_inequalities(routh_matrix[4,0]>0)
```

$\displaystyle - \frac{693}{12500} < k \wedge k < \infty$

```python
routh_matrix[3,0]
```

$\displaystyle \frac{250 \left(1000 k^{2} + 124359 k - 186813\right)}{250 k - 13343}$

```python
reduce_inequalities(routh_matrix[3,0]>0)
```

$\displaystyle \left(\frac{13343}{250} < k \wedge k < \infty\right) \vee \left(k < - \frac{124359}{2000} + \frac{1239 \sqrt{10561}}{2000} \wedge - \frac{1239 \sqrt{10561}}{2000} - \frac{124359}{2000} < k\right)$

```python

```


二分法

```python
from sympy import *
import math
```

```python
def bisection(f, a, b, eps, eta=1e-16, verbose=False):
    """二分法求根

    对方程 f(x) = 0 在区间 [a, b]，使用二分法求根。
    做 ceil((log(((b - a) / eps), 2) - 1) 次迭代，使结果满足精度 eps。

    实现参考: 数值分析[谷根代，杨晓忠 等著]2011年版.P18.算法1

    Args:
        f: function, 一元函数，表示要求根的方程: f(x) = 0.
        a, b: float, 有根区间 [a, b] 的端点.
        eps: float, 给定精度.

        eta: float, 当 abs(f(x)) <= eta 时停止计算, default 1e-16.
        verbose: bool, 打印出二分法计算的表格, default False.

    Returns:
        (x_final, N)

        x_final: float, 二分法求得的近似根
        N: int, 迭代次数

    Raises:
        ValueError: 给定的 f(a) * f(b) < 0 时无法求解，抛出异常
    """
    if f(a) * f(b) > 0:
        raise ValueError("rootless interval: f(a) * f(b) < 0")

    N = math.ceil((log(((b - a) / eps), 2) - 1).evalf(5))

    if verbose:
        print(f'N = {N}\n')

    x = (a + b) / 2

    if verbose:
        print(f'n \t (a, b) \t f(x_n)')
        print('-'*35)

    for n in range(N+1):
        if abs(f(x)) <= eta:  # f(x) == 0
            break

        if verbose:
            print(f'{n} \t ({a}, {b}) \t f({x})={f(x)}')

        if f(x) * f(a) < 0:
            b = x
        else:
            a = x
        x = (a + b) / 2
        n += 1

    if verbose:
        print(f'{n} \t ({a}, {b}) \t -')

    x_final = (a + b) / 2
    if verbose:
        print(f'\nresult: x = ({a}+{b})/2 = {x_final}')

    return x_final, N+1

```

```python
def f(x):
    return 2 * exp(-x) - sin(x)

bisection(f, 0, 1, 0.0005, verbose=True)
```

    N = 10

    n 	 (a, b) 	 f(x_n)
    -----------------------------------
    0 	 (0, 1) 	 f(0.5)=0.733635780821064
    1 	 (0.5, 1) 	 f(0.75)=0.263094345458695
    2 	 (0.75, 1) 	 f(0.875)=0.0661805371209897
    3 	 (0.875, 1) 	 f(0.9375)=-0.0228698549070950
    4 	 (0.875, 0.9375) 	 f(0.90625)=0.0208763999947352
    5 	 (0.90625, 0.9375) 	 f(0.921875)=-0.00119109771321235
    6 	 (0.90625, 0.921875) 	 f(0.9140625)=0.00979401298210625
    7 	 (0.9140625, 0.921875) 	 f(0.91796875)=0.00428930379438952
    8 	 (0.91796875, 0.921875) 	 f(0.919921875)=0.00154606529293533
    9 	 (0.919921875, 0.921875) 	 f(0.9208984375)=0.000176724441991016
    10 	 (0.9208984375, 0.921875) 	 f(0.92138671875)=-0.000507376461447939
    11 	 (0.9208984375, 0.92138671875) 	 -

    result: x = (0.9208984375+0.92138671875)/2 = 0.921142578125

    (0.921142578125, 11)

```python
def dichotomy(f, a, b, eps, eta=1e-16, verbose=False):
    """二分法求根

    对方程 f(x) = 0 在区间 [a, b]，使用二分法求根。
    一直迭代到 abs(b - a) <= eps 为止。

    实现参考：《实验二  非线性方程求根》

    Args:
        f: function, 一元函数，表示要求根的方程: f(x) = 0.
        a, b: float, 有根区间 [a, b] 的端点.
        eps: float, 根的容许误差.

        eta: float, abs(f(x)) 的容许误差, default 1e-16.
        verbose: bool, 打印出二分法计算的表格, default False.

    Returns:
        (x_final, N)

        x_final: float, 二分法求得的近似根
        N: int, 迭代次数

    Raises:
        ValueError: 给定的 f(a) * f(b) < 0 时无法求解，抛出异常
    """
    if f(a) * f(b) > 0:
        raise ValueError("rootless interval: f(a) * f(b) < 0")

    if verbose:
        print(f'n \t (a, b) \t f(x_n)')
        print('-'*35)

    n = 0
    while abs(b - a) > eps:
        x = (a + b) / 2
        fx = f(x)
        if verbose:
            print(f"{n}\t {a, b}\t f({x})={fx}")

        if abs(fx) <= eta:
            break

        if f(a) * fx < 0:
            b = x
        else:
            a = x

        n += 1

    x_final = (a + b) / 2
    if verbose:
        print(f"{n}\t {a, b}\t -")
        print(f'\nresult: x = ({a}+{b})/2 = {x_final}')

    return x_final, n

```

```python
dichotomy(f, 0, 1, 0.0005, 0)
```

    (0.921142578125, 11)

```python
def fixed_point_iter(phi, x_0, max_steps=25, verbose=False):
    """不动点迭代

    Args:
        phi: function, 迭代函数
        x_0: float, 初值

        max_steps: int, 最大迭代次数
        verbose: bool, 打印出每一步的值，default False.

    Returns:
        x_final: float, 最终的近似根 x
    """
    x = x_0
    for i in range(max_steps):
        if verbose:
            print(f'x_{i} \t {x}')
        x = phi(x)
    return x
```

```python
fixed_point_iter(lambda x: 1 + 1 / x**2, 1.5, max_steps=10, verbose=True)
```

    x_0 	 1.5
    x_1 	 1.4444444444444444
    x_2 	 1.4792899408284024
    x_3 	 1.456976
    x_4 	 1.4710805833200253
    x_5 	 1.4620905354712408
    x_6 	 1.4677905760195855
    x_7 	 1.464164380462178
    x_8 	 1.4664663557170745
    x_9 	 1.465003040566855

    1.4659324390818347

```python
def newton_iter(f, x_0, eps=0, eta=0, df=None, max_steps=20, frac=False, verbose=False):
    """牛顿迭代法

    Args:
        f: function, 迭代函数
        x_0: float, 初值
        eps: float, 根的容许误差, default 0.
        eta: float, abs(f(x)) 的容许误差, default 0.

        df: function, f 的导函数, 默认 None 表示自动调用 sympy.diff 求导(会导致后续迭代中使用分数运算)。
        max_steps: int, 最大迭代次数, default 20.
        frac: bool, True 则输出分数(仅对 df=None 时生效)，否则使用 float, default False.
        verbose: bool, 打印出每一步的值, default False.

    Returns:
        x_final: float, 最终的近似根 x
    """

    if df == None:
        __x = Symbol('x')
        __df = diff(f(__x), __x)
        df = lambda x: __df.subs(__x, x)

    x = x_0
    for i in range(max_steps):
        if verbose:
            print(i, x)

        x_next = x - f(x) / df(x)

        if abs(x_next - x) <= eps or abs(f(x)) <= eta:
           break

        x = x_next

    if not frac:
        x = float(x)
    return x
```

```python
def f(x):
    return x ** 3 - 2 * x - 5

def df(x):
    return 3 * x ** 2 - 2

# newton_iter(f, 2, eps=0.5e-3, frac=True, verbose=True)
newton_iter(f, 2, eps=0.5e-3, df=df, frac=True, verbose=True)
```

    0 2
    1 2.1
    2 2.094568121104185

    2.094568121104185

```python
def single_point_truncation(f, x_0, x_1, eps=0, eta=0, max_steps=20, verbose=False):
    """单点弦截法

    Args:
        f: function, 迭代函数
        x_0, x_1: float, 初值
        eps: float, 根的容许误差, default 0.
        eta: float, abs(f(x)) 的容许误差, default 0.

        max_steps: int, 最大迭代次数, default 20.
        verbose: bool, 打印出每一步的值, default False.

    Returns:
        x_final: float, 最终的近似根 x
    """

    f_x0 = f(x_0)

    x = x_1
    for i in range(max_steps):
        if verbose:
            print(i, x)

        x_next = x - f(x) / (f(x) - f_x0) * (x - x_0)

        if abs(x_next - x) <= eps or abs(f(x)) <= eta:
           break

        x = x_next

    return x
```

```python
def f(x):
    return x ** 3 - 2 * x - 5

single_point_truncation(f, 2, 1, eps=0.5e-3, verbose=True)
```

    0 1
    1 2.2
    2 2.088967971530249
    3 2.094861151990966

    2.094861151990966

```python
def secant_method(f, x0, x1, eps=0, eta=0, max_steps=20, verbose=False):
    """两点弦截法（割线法）

    Args:
        f: function, 迭代函数
        x0, x1: float, 初值
        eps: float, 根的容许误差, default 0.
        eta: float, abs(f(x)) 的容许误差, default 0.

        max_steps: int, 最大迭代次数, default 20.
        verbose: bool, 打印出每一步的值, default False.

    Returns:
        x_final: float, 最终的近似根 x
    """
    for i in range(max_steps):
        if verbose:
            print(i, x1)

        x2 = x1 - f(x1) * (x1 - x0) / (f(x1) - f(x0))
        x0, x1 = x1, x2

        if abs(x1 - x0) <= eps or abs(f(x1)) <= eta:
           break

    return x1
```

```python
secant_method(lambda x: x ** 2 - 612, 10, 30, max_steps=5, verbose=True) # Root: 24.738633748750722
```

    0 30
    1 22.8
    2 24.545454545454547
    3 24.746543778801843
    4 24.73860275369709

    24.738633748750722

题目
求方程 $f(x)=x^3+x^2-3x-3=0$ 在 1.5 附近的根.（误差限为 $\epsilon=1e-6$, $\eta=1e-9$）

```python
def f(x):
    return x ** 3 + x ** 2 - 3 * x - 3

x0 = 1.5
x1 = 2
eps = 1e-6
eta = 1e-9

result = {}
print("牛顿迭代法:")
result["牛顿迭代法"] = newton_iter(f, x0, eps=eps, eta=eta, verbose=True)
print("单点弦截法:")
result["单点弦截法"] = single_point_truncation(f, x0, x1, eps=eps, eta=eta, verbose=True)
print("两点弦截法:")
result["两点弦截法"] = secant_method(f, x0, x1, eps=eps, eta=eta, verbose=True)

print("\nresult:")
for k in result:
    print(f'{k}: {result[k]}')
```

    牛顿迭代法:
    0 1.5
    1 1.77777777777778
    2 1.73336066694000
    3 1.73205192940947
    4 1.73205080756970
    单点弦截法:
    0 2
    1 1.6923076923076923
    2 1.7390156515180086
    3 1.7308625826467308
    4 1.7322544663053168
    5 1.7320159286895187
    6 1.7320567817872679
    7 1.7320497843005194
    8 1.732050982835706
    两点弦截法:
    0 2
    1 1.6923076923076923
    2 1.7257977285018928
    3 1.7322172842612025
    4 1.732050123979108

    result:
    牛顿迭代法: 1.7320508075697012
    单点弦截法: 1.732050982835706
    两点弦截法: 1.7320508074943775

```python

```


# Convolutional Neural Networks

## Introduction
A neural network's hidden layers are used to learn feature detectors from input data. For simple input data types we can use fully connected hidden layers to allow us to learn features across any combination of input feature values. Thus one feature might be learned which is activated only when the first and last input feature values are both active. Another feature might be learned which is active when the average intensity across all input feature values is above a certain threshold.

The figure below illustrates this situations. Here neurons $\delta^{2}_{1}$ and $\delta^{2}_{2}$ for example might be defined to depend on the first and last input neurons, and mean values of input neurons respectively.

<!-- conv-1.png -->

While fully connected layers like this are very useful at learning features that might depend on any particular combination of input data values, they are computationally costly. The number of weights to be learned in a fully connected layer between input units $L1$ and hidden units $L2$ is:

\begin{equation}
(size(L1)+1) \times size(L2)
\end{equation}

where the addition of 1 to the left hand side is due to the fact we need to learn the weight to a bias unit on the input layer.

Lets consider for a moment the case of working with image data. Here if we assume a low resolution square image of edge 256 lines, and assume we want a hidden layer that can say pick out 64 features, then we require $((256 \times 256) + 1) \times 64$ weights, i.e., 4,194,368 weights. This is a very significant number of weights in comparison to the examples we have considered to date.

For complex input data types such as images we deal with this exponential growth in the number of weights by taking advantage of specific properties of the input data. In the case of images we can take advantage of two facts:
1. That in images the input is often translation invariant; and
2. In early layers we can focus on learning local features rather than global features.

To illustrate consider the case of analyising an image to find edges. An edge can be found by looking for a steep gradient in image pixel intensities in a local block of pixels. For example simple features for horizontal and vertical edge detection in a 3x3 block of pixels can be illustrated by the following weight matrices for linear feature detectors.

<!-- edge-detector-unattributed.png -->

Such a local feature is translation invariant -- a horizontal edge at the bottom left of the picture generally looks like a horizontal edge in any other part of the image when viewed as a steep gradient in pixel intensities. Such a property has been taken advantage of in the image processing domain for many years. For example consider the image below where vertical and horizontal edges are extracted from an image.

<!-- edges-unattributed.jpg -->

We can illustrae the use of a simple edge detector below by scanning over an image from the MNIST dataset based on the Gx filter shown above.

```python
# %matplotlib inline
import matplotlib.pyplot as plt
import matplotlib as mp
import numpy as np
import tensorflow as tf

# Get a copy of the mnist dataset container
mnist = tf.keras.datasets.mnist

# Pull out the training and test data
(x_train, y_train),(x_test, y_test) = mnist.load_data()
image = x_train[5]

# Reshape the image into 2D and normalize pixel intensities to between 0 and 1
flattened_image = np.reshape(image, (-1, 28))
flattened_image = flattened_image / 255.

# For illustration purposes display the flattened image of a digit
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.matshow(flattened_image, cmap = mp.cm.binary)
plt.xticks(np.array([]))
plt.yticks(np.array([]))
plt.show()

# design a filter which picks out verticle edges
# in this case it is a 3x3 filter
#weights = [-1,0,1,-1,0,1,-1,0,1]
weights = [-1,-1,-1,0,0,0,1,1,1]

# Construct a conainer for the result of filter application.
# Note that our output image will be slightly smaller than our input image
num_rows, num_cols = np.ma.shape(flattened_image)
edge_output = np.zeros((num_rows-2,num_cols-2))

# iterate over the rows in the image and apply the filter
for i,row in enumerate(flattened_image):
    if i < (num_rows - 2):
        # iterate over each cell in the given row
        for j,val in enumerate(row):
            if j < (num_cols - 2):
                # manually isolate the pixels that will have the filter applied to
                sample = [flattened_image[i][j],
                          flattened_image[i][j+1],
                          flattened_image[i][j+2],
                          flattened_image[i+1][j],
                          flattened_image[i+1][j+1],
                          flattened_image[i+1][j+2],
                          flattened_image[i+2][j],
                          flattened_image[i+2][j+1],
                          flattened_image[i+2][j+2],
                         ]
                # calculate and store a logistic function based on the sample and weights
                logit = np.matmul(sample,weights)
                edge_output[i,j] = 1 / (1 + np.exp(-logit))

# for illustration purposes display the results of the feature detector
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.matshow(edge_output, cmap = mp.cm.binary)
plt.xticks(np.array([]))
plt.yticks(np.array([]))
plt.show()
```

In the example above we say that our filter is being *convolved* over the input image. In other words the filter is being applied independently to each input.

Note that in the example code above we design our filter to only look at complete 3x3 blocks of our input image. This is a simplification to allow a quick and transparent implementation of the process. This results in our output image being slightly smaller than the input image. In this case our output image is 26x26 pixels rather than the 28x28 of our input image.

It is also worth noting that the filters above have no bias value. This is acceptable in this simple case since we are not training the filter. However later when we switch to a Neural Network implementation we need to introduce a bias unit.

## Convolutional Layer Design

In convolutional neural networks we apply the principle above to the design of hidden layers in the network. Specifically, instead of relying on neurons in a hidden layer that are fully connected to each neuron in the preceding layer, we instead design our hidden layer neurons in such a way that they are only exposed to a small sample of the input units. However we introduce clones of these neurons so that the entire set of inputs is covered by a given hidden layer neuron type.

We can achieve this goal in a Deep Neural Network architecture by simultaneously learning copies of simple feature detectors that are applied across a complete input vector (2D in the case of images). Each of these detectors share the same weights and can thus be thought of as being copies or clones of each other. The features detected are local and assumed to be translational invariant.

Rather than having a hidden layer that is fully connected to the input layer we end up with a hidden layer that consists of a set of neurons that each look at a slightly different subsample of the input image. The sub-sample might for example be a 3x3 grid of the input image. Each neuron in the convolutional neuron set is looking at a different part of the input, but crucially since all these neurons are forced to share the same weights, they all are looking for the exact same thing.

This principle is illustrated in the figure below where the same feature detector is applied to different samples of the input image to generate a feature map. Note that the weight marked is fixed to the same value across each application.

<!-- cnn-animated.gif -->

In a convolutional layer we typically train a number of these small local features. Each of these features in turn results in individual feature maps. Considering our example earlier we might for example construct a very simple convolutional layer with two features: one feature for vertical line detection and another feature for horizontal line detection. Such a simple convolutional layer is illustrated below.

<!-- cnn_example.png -->

We see in this example that there are in fact two neuron types in our convolutional layer. Each of these is characterissed by having its own set of weights, but is applied iteratively over the input image to produce a given feature map. The feature itself is a simple 3x3 feature, i.e. it contains only 9 weights that are applied to a local block of the input. The feature results in a Feature Map over our input. The Feature Map is our resultant image above.

### Convolutional Layer Parameters

Looking again at the example we can see that a convolutional layer has 3 key parameters that describe the dimensionality of the layer's output. The first two of these are **width** and **height** and are used to describe the dimensionality of a given feature map in the convolutional layer. The third dimensionality feature is **depth**. Depth is the number of filter or neuron types in the convolution layer. Each index in the depth of the convolution layer allows another filter type / another feature type to be learned. All neurons at the same depth index share the same weights.

The figure below shows a traditional illustration of one layer of a convolutional neural network. From this illustration we can see where the term depth has come to mean the number of features / feature maps in a convolutional layer.

<!-- cnn_depth.png.png -->

It should be noted however that CNNs are sometimes visualised as being orthogonal to our viewing plane. In this case the different feature types move from left to right across the screen as illustrated in this wikimedia image below. In that image the depth of the convolutional layer is 5 units and we see that all 5 units with the same height and width values are projections from the same scanned section of the input image.

<!-- cnn_alt.png -->

In our examples above our filter was applied with a sliding window of 1 over our input image. Thus the overlap between our application of the filters was maximized. It is possible to reduce this overlap and thus result in smaller convolutional feature maps. The amount of overlap is controlled by a parameter called **stride**. In our example above we used a stride of 1. A stride of 3 in the example above would result in no actual overlap of our filter over the input images. In our examples below we will stick with a stride of 1 for notational simplicity.

### From One to Multiple Channels

In our mnist example, our input is a black and white image. We talk about this image type as just having one channel -- just black and white intensity. Colour images on the other hand are usually built around three individual maps of red, green, and blue intensities. Our feature based approach is easily expanded to deal with multiple channels. In such case the information from all 3 channels is accounted for in a single kernel computation. For example, in the example above we use a 3x3 filter applied to our 1 channel information. This results in 9 parameters to be trained (plus a single bias value). We can apply the same kernel type instead to 3 channel information. In this case the number of parameters that our kernel has to learn is (3 x 3 x 3) + 1, i.e., 28. A single feature detector will still produce a single image map. The advantage of this approach is that our filters not only get the opportunity to learn spatial features, but they can learn features dependent on particular combinations of colour.

### Multiple Convolutional Layers

Rather than having a single convolutional layer in a network, we usually have a number of layers configured in a feedforward arrangement. Here a feature map at layer k with depth $d_{1}$ feeds can be used as input to any number of feature detectors in the next convolution layer.

In a typical design, we treat all the image maps is a layer as individual channels being passed into the next layer. This gives our network the opportunity to combine information from multiple trained features.

### Linking to Fully Connected Layers

Layers of convolutional neurons create more and more sophisticated feature detectors that operate locally. At a certain point we want to be able to investigate global connections in an image. For example we might want one feature that activates if a straight vertical line is detected in one area of an image and a horizontal line is detected to its upper right. For these global connections to be detected we need to once again introduce standard non-convolutional layers.

<!-- cnn_full.png -->

In the context of CNNs we refer to standard layers as fully connected layers. The reason for this is simply that all neurons in a standard layer are fully connected back to all units in the prior layer. Where the prior layer is a convolutional layer this means that all units in the convolution are connected. This can lead to an explosion of connections / weights. Consider the case where the convolutional layer has depth 20, height 20, and width 20. If our fully connected layer has say 30 neurons then we are talking about 240000 connections in one layer alone.

## Pooling Layers

With a stride of 1 our convolutional feature maps are almost as big as our input images. With a potentially large number of features in our convolutional layer this can result in a very large number of output values which would then have to be fed into layers further down the network. To limit this growth in connection number we can subsample our convolutional feature maps to reduce their size. In neural network terminology this subsampling is referred to as pooling.

We can illustrate this pooling method by applying a pooling layer to our image detection feature map as follows.

```python
# Construct a conainer for the result of pooling application.
num_rows, num_cols = np.ma.shape(edge_output)
mean_pool = np.zeros((num_rows//2,num_cols//2))
max_pool = np.zeros((num_rows//2,num_cols//2))

# iterate over the rows in the image and apply the filter
for i,row in enumerate(edge_output):
    if i % 2 == 0:
        # iterate over each cell in the given row
        for j,val in enumerate(row):
            if j % 2 == 0:
                # manually isolate the pixels that we will pool
                sample = [edge_output[i][j],
                          edge_output[i][j+1],
                          edge_output[i+1][j],
                          edge_output[i+1][j+1],
                         ]
                # store the pooled values
                mean_pool[i//2,j//2] = np.mean(sample)
                max_pool[i//2,j//2] = np.amax(sample)

# for illustration purposes display the results of the feature detector
fig = plt.figure()
ax = fig.add_subplot(1, 2, 1)
ax.matshow(max_pool, cmap = mp.cm.binary)
plt.xticks(np.array([]))
plt.yticks(np.array([]))
ax = fig.add_subplot(1, 2, 2)
ax.matshow(mean_pool, cmap = mp.cm.binary)
plt.xticks(np.array([]))
plt.yticks(np.array([]))
plt.show()
```

We pool by taking a clump of units and producing one aggregate value for those units. There are a number of ways in which we can aggregate or pool these values. One straightforward way is to simply calculate the mean activation over the clump of units. This is referred to as mean-pooling. Another popular method is to take the maximum value across the clump of units as the activation for the new aggregate unit. This is referred to as max-pooling. Max pooling rather than mean pooling has become the dominant method for pooling. One 'hand-waving' interpretation of why max-pooling might be most useful is that it preserves the identification of strong features rather than smoothing out the feature map.

It is important to note that whereas a convolutional layer has an actual activation function which can be anything from a logistic function to RELU, a pooling layer does not apply any function to the input data apart from the pooling function itself (usually max or mean).

The primary motivation for pooling is thus simply to reduce the size of feature maps and therefore reduce the number of connections in the network. Not only is this important in terms of reducing computational cost, but it also helps to reduce the potential for overfitting.

The figure below illustrates the application of pooling in a complete workflow.

<!-- cnn_pooling.png -->

## Convolution and Pooling in Tensorflow

TensorFlow provides a very neat implementation of Convolutional Neural Network and Pooling functionality where once again we just need to add in some extra layer definitions.

### MNIST in TensorFlow without Convolution

To illustrate the TensorFlow approach we will swap over to the use of the MNIST digits corpus. In the code below we first provide an implementation without the use of convoutional or pooling layers. This implementation uses 2 hidden layers of 256 RELU units each. The Adam Optimzier is also used. This is basically the same examples we have seen a couple of times over the last few weeks.

```python
# import tensorflow library
import tensorflow as tf

# Get a copy of the mnist dataset container
mnist = tf.keras.datasets.mnist

# Pull out the training and test data
(x_train, y_train),(x_test, y_test) = mnist.load_data()

# Normalize the training and test datasets
x_train = tf.keras.utils.normalize(x_train, axis=1)
x_test = tf.keras.utils.normalize(x_test, axis=1)

# Create a simple sequential network object
model = tf.keras.models.Sequential()

# Add layers to the network for processing the input data
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))

# Compile the model
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])

# Start the training process
model.fit(x=x_train, y=y_train, epochs=5)

model.summary()

# Evaluate the model performance with test data
test_loss, test_acc = model.evaluate(x=x_test, y=y_test,verbose=0)

# Print out the model accuracy
print('\nTest accuracy: ' + str(test_acc*100) + "%" )
```

    Epoch 1/5
    1875/1875 [==============================] - 5s 2ms/step - loss: 0.4701 - accuracy: 0.8648
    Epoch 2/5
    1875/1875 [==============================] - 4s 2ms/step - loss: 0.1156 - accuracy: 0.9641
    Epoch 3/5
    1875/1875 [==============================] - 4s 2ms/step - loss: 0.0730 - accuracy: 0.9778
    Epoch 4/5
    1875/1875 [==============================] - 4s 2ms/step - loss: 0.0524 - accuracy: 0.9836
    Epoch 5/5
    1875/1875 [==============================] - 4s 2ms/step - loss: 0.0389 - accuracy: 0.9873
    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    flatten (Flatten)            (32, 784)                 0
    _________________________________________________________________
    dense (Dense)                (32, 128)                 100480
    _________________________________________________________________
    dense_1 (Dense)              (32, 128)                 16512
    _________________________________________________________________
    dense_2 (Dense)              (32, 10)                  1290
    =================================================================
    Total params: 118,282
    Trainable params: 118,282
    Non-trainable params: 0
    _________________________________________________________________

    Test accuracy: 97.22999930381775%

The implementation above achieves very high accuracy in just 15 epochs. In fact we can see that the error even after completion of the 0th epoch is already down to below 20%. This isn't an error in the code, but is rather due to the use of mini-batch learning. In the code above we did not wait until collecting all of our error before making adjustments to our weights. Instead we split our training set into a number of mini-batches each of size 100. After collecting the errors for 100 examples we made adjustments to our weights. Given our training data size is very large, this means that we in fact make over 500 applications of the back-propagation of weights over every full epoch of training. Before proceeding, try adjusting the batch size and observer the error graph.

With a batch size of 10000, how many epochs does it take for the error to reduce below 10%?

### MNIST in TensorFlow with Convolution

Implementing a convolutional layer in TensorFlow is very straightforward. We can take our MNIST code from above and very quickly update it to incorporate a CNN. We implement the major changes in our ff_network function.

Note that we have to take our input images which had been flattened to 1D arrays and rework them into 2D for use with the convolutional layer. Similarly following the convolutional layer we also have to reshape the data back to 1D to feed it into a fully-connected layer.

```python
x_train = x_train.reshape(x_train.shape[0], 28, 28, 1)
x_test = x_test.reshape(x_test.shape[0], 28, 28, 1)

input_shape = (28, 28, 1)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(12, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(16, activation='relu'))
model.add(tf.keras.layers.Dense(10, activation='softmax'))

# Compile the model
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])

model.summary()

# Start the training process
model.fit(x=x_train, y=y_train, epochs=5)

# Evaluate the model performance with test data
test_loss, test_acc = model.evaluate(x=x_test, y=y_test,verbose=0)

# Print out the model accuracy
print('\nTest accuracy: ' + str(test_acc*100) + "%" )
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv2d (Conv2D)              (None, 26, 26, 12)        120
    _________________________________________________________________
    max_pooling2d (MaxPooling2D) (None, 13, 13, 12)        0
    _________________________________________________________________
    flatten_1 (Flatten)          (None, 2028)              0
    _________________________________________________________________
    dense_3 (Dense)              (None, 16)                32464
    _________________________________________________________________
    dense_4 (Dense)              (None, 10)                170
    =================================================================
    Total params: 32,754
    Trainable params: 32,754
    Non-trainable params: 0
    _________________________________________________________________
    Epoch 1/5
    1875/1875 [==============================] - 18s 9ms/step - loss: 0.6760 - accuracy: 0.7947
    Epoch 2/5
    1875/1875 [==============================] - 17s 9ms/step - loss: 0.1739 - accuracy: 0.9513
    Epoch 3/5
    1875/1875 [==============================] - 17s 9ms/step - loss: 0.1069 - accuracy: 0.9687
    Epoch 4/5
    1875/1875 [==============================] - 17s 9ms/step - loss: 0.0900 - accuracy: 0.9730
    Epoch 5/5
    1875/1875 [==============================] - 17s 9ms/step - loss: 0.0713 - accuracy: 0.9789

    Test accuracy: 97.49000072479248%

## Skip Connections & Residual Networks

CNNs allow us to build networks that can build general purpose feature detectors that can be applied over an entire image but yet only require a relatively small number of weights. This versatility lead to many different network architectures being developed around the basic CNN architecture over the last 10 years. However it is important to note that deeper and deeper networks with 50+ layers were often found to be very useful as they could detect more and more complex features from fed in image data. However it was equally found that well known challenges in building deeper and deeper networks were seen. These problems such as the 'vanishing gradient' problem limited the depth of Deep Learning networks in general and deep CNN networks in particular.

The so-called Skip Connection provided an essential next step in the development of very deep CNN based networks. The basic idea of the skip connection is that instead of having simply a single collection of neurons in a given hidden layer, that we instead build a special layer type that includes a typical layer architecture but also a specialist direct link between inputs and outputs that allowed signal to fully flow from inputs to outputs without the alteration caused by the activation function. These so-called skip connections have become an integral part of so-called Residual Networks and underpin the standard Deep CNN architectures at this point.

The figure below illustrates the general layout of a skip connection as used in a residual network.

<!-- cnn_pooling.png -->

The top of the image shows an extract from two standard layers of a network. Here we have an input x. For the moment we can assume this is either directly from an actual input, or perhaps is an output from another hidden layer that is not shown. In a normal way, that value x passes through two hidden layers in sequence. In each layer we first of all compute the value z from x and the weights, before passing this value, the logit, on to the unit's activiation function. Here we assume the activation function is the Rectified Linear Unit. The output of this unit is then passed forward as input to the next unit which in turn calculates the logit with its own weights before again calculating the activation function. This second activtion function can be thought of as the output of this extract, or block.

In the bottom of the image we show the altered form as used in a residual network. Again we have an input x, but in this case as well as being passed into the first unit, it is also copied around the first hidden unit and fed into the second unit directly. When it is fed into the second unit, it is important to note that the input is not fed into the unit's standard inputs before calculation of the logit. Instead it is added to the output of the logit calculation just before the calculation of the activation function.

While this explains the mechanism of what a skip connection is, it says little to us about the intuition of why this would work. In short, the intuition is that in backpropogation the skip connection is allowing more of the error signal to continue to backpropogate through the network than would be possible otherwise. The two RELU functions can still learn complex functions, but importantly we can now build longer deeper networks.

It should be noted that this is just one example of what a residual block can look like. There are many more variations of this general theme.

## The Inception Networks

While residual blocks solve many of the challenges in deep CNNs, they are not the end of the story. Making image classifiers more generic required a numbe of other engineering tricks that increased performance. The Incpetion Networks lead the way in many of these changes.

One of the most important changes introduced by the Inception Network concerned the issue of scale.

The core of the CNN is the feature kernel. In the examples above we suggested that feature kernels of size 3, 4, or 5 might be useful for image processing. In practice however different kernel sizes are better for different object analysis types. For example small kernels are good at identifying small patterns in data, where large kernels are better at identifying more global information. The most important contribution of the inception networks was the idea of having kernels of different sizes within the same layer of the network. This allowed the network to be far more scale invariant in training than was possible previously.

<!-- cnn_pooling.png -->

The figure above from Inception Network v1 illustrates how multiple different convolution filter sizes were used in a typical layer alongside max pooling layers to provide a more complex or rather wider range of analysis than was possible previously.

Inception introduced many other useful improvements that are beyond of what we can cover here. For anyone interested, a good blog review of the inception network versions and their individual improvements can be found here:

https://towardsdatascience.com/a-simple-guide-to-the-versions-of-the-inception-network-7fc52b863202#:~:text=The%20Inception%20network%20was%20an,Designing%20CNNs%20in%20a%20nutshell.

## CIFAR 10 Tutorial

The TensorFlow Website has a number of excellent tutorials which demonstrate the power of TensorFlow. CIFAR 10 is a basic dataset for image classification which is often used as a next step after MNIST classification.

Read, study and put to work the TensorFlow CIFAR 10 Tutorial as described here:
https://www.tensorflow.org/tutorials/images/cnn

In order to get your code to run on non GPU hardware it may be necessary for you to reduce the complexity of the network, or only training for a shorter period of time than would be possible.

## Suggested Tasks

1. Add a second CNN layer to the MNIST example above. How does performance and training compare to the 1 CNN variant?
2. Add dropout to the MNIST example. Once again, how does training speed and accuracy compare to the original variant?
3. Integrate the CIFAR-10 dataset into the MNIST example above.

Complete Modules/Deep Learning/week 7/CNNs.ipynb


sergazy.nurbavliyev@gmail.com © 2021

## Expected Number of flips until N heads

Question: Assume we have a fair coin. What is the expected number of coin flips needed to get $N$ consecutive heads.

### Intuition

Let us solve this for $N=2$ first. Then we will generalize the idea for any $N$. Let $X$ denote the number of flips that we need to get two consecutive heads. Then we want to find $\mathbb{E}[X]$. We will denote the flip that results in head with $H$ and in tails as $T$.
Then we can write $\mathbb{E}[X]$ as:
\begin{equation}
\mathbb{E}[X]=\mathbb{P}(H)(1+\mathbb{E}[X|H])+\mathbb{P}(T)(1+\mathbb{E}[X|T])=\frac{1}{2}(1+\mathbb{E}[X|H])+\frac{1}{2}(1+\mathbb{E}[X|T])
\end{equation}
where we conditioned on our first flip. Note that $\mathbb{E}[X|T])=\mathbb{E}[X])$ because after we see tails we need to start over again to see two consecutive heads. Then equation (1) becomes
\begin{equation}
\mathbb{E}[X]=\frac{1}{2}(1+\mathbb{E}[X|H])+\frac{1}{2}(1+\mathbb{E}[X])
\end{equation}
Now we need to solve $\mathbb{E}[X|H]$. We will conditioned further if we can get rid of these terms. Then we have
\begin{equation}
\mathbb{E}[X|H]=\mathbb{P}(H)(1+\mathbb{E}[X|HH])+\mathbb{P}(T)(1+\mathbb{E}[X|HT])=\frac{1}{2}(1+\mathbb{E}[X|HH])+\frac{1}{2}(1+\mathbb{E}[X|HT])
\end{equation}
From here we can see that $\mathbb{E}[X|HH]=0$ because we already see the pattern, and $\mathbb{E}[X|HT])=\mathbb{E}[X])$ since we need to start over again. The equation (3) will be
\begin{equation}
\mathbb{E}[X|H]=\frac{1}{2}(1+0)+\frac{1}{2}(1+\mathbb{E}[X])=1+\frac{1}{2}\mathbb{E}[X]
\end{equation}
So equation (2) will be
\begin{equation}
\mathbb{E}[X]=\frac{1}{2}(1+\mathbb{E}[X|H])+\frac{1}{2}(1+\mathbb{E}[X])=\frac{1}{2}(1+(1+\frac{1}{2}\mathbb{E}[X]))+\frac{1}{2}(1+\mathbb{E}[X])=\frac{1}{2}(2+\frac{1}{2}\mathbb{E}[X]))+\frac{1}{2}(1+\mathbb{E}[X])
\end{equation}
After multiplying by 8 both sides we get
\begin{equation}
8\mathbb{E}[X]=4(2+\frac{1}{2}\mathbb{E}[X]))+4(1+\mathbb{E}[X])=8+2\mathbb{E}[X]))+4+4\mathbb{E}[X])\Rightarrow 2\mathbb{E}[X] =12\Rightarrow \mathbb{E}[X] =6
\end{equation}

## Theoritical result

Lets denote $\mathbb{E}_N$ for $N$ consecutive heads. With the same logic above, if we get one more head after $\mathbb{E}_{N-1}$, then we are done because we would have $N$ consecutive heads.  But if it is a tail then  we need to start over again. Thus there are two cases:

If we have a head, $\mathbb{E}_{N-1}+1$.

If we have a tail, $\mathbb{E}_{N}+1$.

\begin{equation}
\mathbb{E}_{N}= \mathbb{P}(H)(\mathbb{E}_{N-1}+1)+\mathbb{P}(T)(\mathbb{E}_{N-1}+\mathbb{E}_{N}+1)=\frac{1}{2}(\mathbb{E}_{N-1}+1)+\frac{1}{2}(\mathbb{E}_{N-1}+\mathbb{E}_{N}+1)
\end{equation}
Multiply by 2 both sides to get
\begin{equation}
\mathbb{E}_{N}=2\mathbb{E}_{N-1}+2
\end{equation}

## Python code for simulation

```python
def expect(n):
    if n == 0:
        return 0

    return 2*expect(n-1)+2
```

```python
expect(2)
```

    6

```python
expect(3)
```

    14

```python
expect(4)
```

    30

```python
expect(5)
```

    62

```python
expect(11)
```

    4094

```python
expect(100)
```

    2535301200456458802993406410750

```python

```

Expected number of flips until N heads March 6 2021.ipynb


# Some bandwidth calculations

source: Hongler's class

The goal of this chapter is to find out how much packets we can push around, to try and estimate how much ranging sequences we can do per second.

The biggest issue when doing communication is that the radio communication system we use is half duplex: as every node wants to transmit or receive on the same medium (same frequency), two nodes cannot communicate at the same time in any direction. This also means that collisions can occur and will invalidate any communications happening at the same time. Therefore, we need to evaluate the likelihood of such interferences.

We start by assuming that the probability of a packet transmission by a node in an interval $t$ is given by a memoryless law:

\begin{equation}
f(t) = \lambda e^{-\lambda t}
\end{equation}

We note $\lambda$ the average time between two ranging measurements and $\mu$ the average time it takes for a full measurement sequence to complete.

One good approximation can be given by the M/M/1 queuing theory framework. Of course this is a bit simplistic, as the shared medium cannot hold ininitely many customers. We first start by computing the load factor $\rho = \frac{\lambda}{\mu}$.

From M/M/1, we know that $p_n = (1 - \rho) \rho^n$ is the probability of having $n$ ranging transactions at the same time. We are interested in the probability of having $n > 1$, as this creates a collision.

\begin{eqnarray}
p(n>1) &=& 1 - p_0 - p_1 \\
       &=& 1 - (1 - \rho) (1 + \rho)
\end{eqnarray}

The rate of successful measurements ($\lambda'$) is given by the rate of measurement attemps multiplied by the success rate:

\begin{equation}
\lambda' = \lambda \cdot (1 - p(n>1))
\end{equation}

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
sp.init_printing()
%matplotlib inline
```

```python
initiate_rate, completion_rate = sp.symbols('lambda mu')
rho = initiate_rate / completion_rate
```

```python
p_collision = 1 - (1 - rho) - (1 - rho) * rho
real_rate = (1 - p_collision) * initiate_rate
optimal_rate = sp.solve(sp.diff(real_rate, initiate_rate), initiate_rate)
optimal_rate
```

```python
real_rate_num = sp.lambdify([initiate_rate, completion_rate], real_rate)
durations = [5e-3, 10e-3, 30e-3]
legends = ['{} ms'.format(int(i * 1000)) for i in durations]
for measurement_sequence_duration in durations:
    f = lambda x: real_rate_num(x, 1 / measurement_sequence_duration)
    rates = np.logspace(0, 3, 1000)
    plt.loglog(rates, f(rates))
    optimal = 1 / (np.sqrt(3) * measurement_sequence_duration)
    print(measurement_sequence_duration, optimal,  f(optimal))

plt.xlabel('Distance measurement attempts rate $\lambda$ (Hz)')
plt.ylabel('Successful measurements $\lambda\'$ (Hz)')
#plt.title('Measurement rates for various measurements sequence duration')
plt.legend(legends)
plt.grid()
plt.savefig('measurement_rate.pdf')
```

```python

```

uwb-beacon-firmware/doc/report/models/Bandwidth in half duplex nodes.ipynb