```python
import numpy as np
import pandas as pd
import sympy as sym
from sympy import init_printing
from lgbayes.models import LinearGaussianBN
init_printing(use_latex=True)
%matplotlib inline

%load_ext autoreload
%autoreload 2
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
def sample():
    u = np.random.normal(0,s_u,size=n)
    z = w_zu*u + np.random.normal(0,s_z,size=n)
    x = w_xz*z+np.random.normal(0,s_x,size=n)
    y = w_yx*x + w_yz*z + w_yu*u+ np.random.normal(0,s_y,size=n)
    return {"U":u,"Z":z,"X":x,"Y":y}
```

```python
s_u = 1
s_z = 0.3
s_x = 1
s_y = 0.5

w_zu = 2
w_xz = 0.5
w_yx = 0.5
w_yz = -1.0
w_yu = 3.0

beta_x = 0.5
beta_z = -1
sigma = 0.5
n = 1000
```

```python
data_dict = sample()
df1 = pd.DataFrame(data_dict,columns=["U","Z","X","Y"])
df1.cov()
```

      U
      Z
      X
      Y

      U
      0.976613
      1.945132
      1.001346
      1.480123

      Z
      1.945132
      3.964203
      2.033707
      2.878700

      X
      1.001346
      2.033707
      2.067833
      2.034622

      Y
      1.480123
      2.878700
      2.034622
      2.836955

```python
model = LinearGaussianBN()
model.add_var("X",[])
model.add_var("Y",["X"])
print(model)
model.cov
a = model.observe(["X"],["x"])
a.cov
a.mu[0].simplify()
```

```python
model.mu
```

```python
model.cov
```

```python
model.information_matrix
```

```python
model = LinearGaussianBN()
model.add_var("U",None,[0],s_u**2)
model.add_var("Z",["U"],[0,w_zu],s_z**2)
model.add_var("X",["Z"],[0,w_xz],s_x**2)
model.add_var("Y",["U","Z","X"],[0,w_yu,w_yz,w_yx],s_y**2)

print (model)
data_dict = dict(zip(model.variables,model.sample(n).T))
df = pd.DataFrame(data_dict,columns=model.variables)
df.cov()
```

    U ~ N(0 ; 1)
    Z ~ N(2*U ; 0.09)
    X ~ N(0.5*Z ; 1)
    Y ~ N(3*U + -1*Z + 0.5*X ; 0.25)

      U
      Z
      X
      Y

      U
      1.004677
      2.000284
      0.997798
      1.499634

      Z
      2.000284
      4.073505
      2.018449
      2.893577

      X
      0.997798
      2.018449
      2.016385
      1.946916

      Y
      1.499634
      2.893577
      1.946916
      2.814356

```python

```


## Define profiles for runs based on regions in $\tau_v$ vs Z space.

$C(z) = C_0 + C'(z-Hs) + C''(z-Hs)^2$.

Hs: Shelf-break depth (147.5 m)

Hd: Depth of interest, >Hs, about 400 m.

Z: depth of upwelling

C_0 : Background concentration

**Two regimes**: Belly and No-belly (within $0\le z \le Hd$)

**Belly** $\rightarrow$ $C''\le \frac{C'}{2Hs}$, $C''\ge \frac{C'}{2(Hs-Hd)}$

**No-Belly** $\rightarrow$ $C''>\frac{C'}{2Hs}$, $C''< \frac{C'}{2(Hs-Hd)}$

** Non-dim number $\tau_v$**

$$\tau_v=Z\frac{\delta^2_vC}{\delta_vC}=Z\frac{\delta^2_vC}{\delta_vC}\rvert_{z=Hs}=Z\frac{2C''}{C'}$$

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sympy as sym
import seaborn as sns
sym.init_printing() # enable fancy printing
```

```python
# Set appearance options seaborn
sns.set_style('white')
sns.set_context('talk')
```

```python
tau,Z = sym.symbols('tau, Z')

# Tau_v vs Z space

p2 = sym.plot_implicit(sym.Or(Z/147.5 < tau, Z/(147.5-400)> tau), (Z, 0, 160),(tau,-0.7,1.5),
                      title='Tau as a function of Z',
                      ylabel=r'$\tau_v$',
                      xlabel='Z (m)')

```

```python
Hs = 147.5
Hd = 400.0

ZZ = np.linspace(0,160,50)
ZZ_exp = np.array([40,60,80,100,120,140])
Zbase = 120 # base case depth of upwelling
Zlow = 80 # base case depth of upwelling
Zlower = 100 # base case depth of upwelling

C1 = np.linspace(0,0.2,50)

#For now, choose only C'>0
C1exp = 0.047 # same as linear profile
C2exp = np.array([(C1exp/(2*Hs))+0.0002,(C1exp/(2*Hs))-0.0001,
                  (C1exp/(2*(Hs-Hd)))+0.00004,(C1exp/(2*(Hs-Hd)))-0.0002,0.0025,-2.5E-3])
colours = ['purple','blue','red','orange','teal','magenta']

fig,ax = plt.subplots(1,2,figsize=(14,6))

ax[0].plot(ZZ,ZZ/Hs,'g')
ax[0].plot(ZZ,ZZ/(Hs-Hd),'g')

for ii,cc in zip(range(len(C2exp)),colours):
    ax[0].plot(Zbase,(Zbase*2*C2exp[ii])/C1exp,'o',color=cc)
    ax[0].plot(Zlow,(Zlow*2*C2exp[ii])/C1exp,'o',color=cc)
    ax[0].plot(Zlower,(Zlower*2*C2exp[ii])/C1exp,'o',color=cc)
    print(C2exp[ii])

ax[0].axhline(y=0,color='k')
ax[0].set_ylabel(r'$\tau_v$')
ax[0].set_xlabel('Z (m)')
ax[0].text(0.9, 0.9,'III', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)
ax[0].text(0.9, 0.6,'I', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)
ax[0].text(0.9, 0.3,'II', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)
ax[0].text(0.9, 0.1,'IV', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)

ax[1].plot(C1,C1/(2*Hs),'g')
ax[1].plot(C1,C1/(2*(Hs-Hd)),'g')

for ii,cc in zip(range(len(C2exp)),colours):
    ax[1].plot(C1exp,C2exp[ii],'o',color=cc)

ax[1].axhline(y=0,color='k')
ax[1].set_xlim(0,0.2)
ax[1].set_xlabel('$C\'$')
ax[1].set_ylabel('$C\'\'$')

plt.tight_layout()
```

```python
Hs = 147.5
Hd = 400.0

ZZ = np.linspace(0,160,50)
ZZ_exp = np.array([40,60,80,100,120,140])
Zbase = 120 # base case depth of upwelling
Zlow = 80 # base case depth of upwelling
Zlower = 100 # base case depth of upwelling

C1 = np.linspace(0,0.2,50)

#For now, choose only C'>0
C1exp = 0.047 # same as linear profile
C2exp = np.array([3.6E-4,5.9E-5,-5.3E-5,-2.9E-4,0.0025,-2.5E-3])
colours = ['purple','blue','red','orange']

fig,ax = plt.subplots(1,2,figsize=(14,6))

ax[0].plot(ZZ,ZZ/Hs,'g')
ax[0].plot(ZZ,ZZ/(Hs-Hd),'g')

for ii,cc in zip(range(len(C2exp)),colours):
    ax[0].plot(Zbase,(Zbase*2*C2exp[ii])/C1exp,'o',color=cc)
    ax[0].plot(Zlow,(Zlow*2*C2exp[ii])/C1exp,'o',color=cc)
    ax[0].plot(Zlower,(Zlower*2*C2exp[ii])/C1exp,'o',color=cc)
    print(C2exp[ii])

ax[0].axhline(y=0,color='k')
ax[0].set_ylabel(r'$\tau_v$')
ax[0].set_xlabel('Z (m)')
ax[0].text(0.9, 0.9,'III', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)
ax[0].text(0.9, 0.6,'I', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)
ax[0].text(0.9, 0.3,'II', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)
ax[0].text(0.9, 0.1,'IV', horizontalalignment='center',verticalalignment='center',transform=ax[0].transAxes,fontsize=15)

ax[1].plot(C1,C1/(2*Hs),'g')
ax[1].plot(C1,C1/(2*(Hs-Hd)),'g')

for ii,cc in zip(range(len(C2exp)),colours):
    ax[1].plot(C1exp,C2exp[ii],'o',color=cc)

ax[1].axhline(y=0,color='k')
ax[1].set_xlim(0,0.2)
ax[1].set_xlabel('$C\'$')
ax[1].set_ylabel('$C\'\'$')

plt.tight_layout()
```

What do these profiles look like?

```python
Hs = 147.5
No = 0

z = np.linspace(0,400,50)

fig,ax = plt.subplots(1,1,figsize=(6,8))
labels = ['III','I','II','IV','Max curv', 'Min Curv']
colours = ['purple','blue','red','orange','teal','magenta']

for ii,cc,lab in zip(range(len(C2exp)),colours,labels):

    CC = C1exp*(z-Hs)+C2exp[ii]*((z-Hs)**2)
    ax.plot(CC,z, '-', color=cc,label=lab)

ax.plot(C1exp*(z-Hs),z,'-',color='green',label='Linear')

ax.set_xlabel('Concentration ($\mu$M)')
ax.set_ylabel('Depth (m)')
ax.invert_yaxis()
ax.axvline(0,linestyle='--' ,color='k')
ax.legend(loc=0)
```

```python
Hs = 147.5
No = 0

z = np.linspace(0,400,50)

fig,ax = plt.subplots(1,1,figsize=(6,8))
labels = ['III','I','II','IV']
colours = ['purple','blue','red','orange']

for ii,cc,lab in zip(range(len(C2exp)),colours,labels):

    CC = C1exp*(z-Hs)+C2exp[ii]*((z-Hs)**2)
    ax.plot(CC,z, '-', color=cc,label=lab)

ax.plot(C1exp*(z-Hs),z,'-',color='green',label='Linear')

ax.set_xlabel('Concentration ($\mu$M)')
ax.set_ylabel('Depth (m)')
ax.invert_yaxis()
ax.axvline(0,linestyle='--' ,color='k')
ax.legend(loc=0)
```

Add Co to get positive profiles everywhere:

```python
Hs = 147.5
Co = 13.5 # Add 13.5 [C] to get positive profiles

z = np.linspace(0,400,50)

fig,ax = plt.subplots(1,1,figsize=(6,8))
labels = ['III','I','II','IV','Max curv','Min Curv']
colours = ['purple','blue','red','orange','teal','magenta']

for ii,cc,lab in zip(range(len(C2exp)-1),colours,labels):

    CC = Co + C1exp*(z-Hs)+C2exp[ii]*((z-Hs)**2)
    ax.plot(CC,z, '-', color=cc,label=lab)

ax.plot(150 + C1exp*(z-Hs)+C2exp[ii+1]*((z-Hs)**2),z,'-',color='magenta',label='Min Curv')

ax.plot(Co + C1exp*(z-Hs),z,'-',color='green',label='Linear')

ax.set_xlabel('Concentration ($\mu$M)')
ax.set_ylabel('Depth (m)')
ax.invert_yaxis()
ax.axvline(0,linestyle='--' ,color='k')
ax.legend(loc=0)
```

```python

```

Or, same value of $\tau_v$ and see what it implies about C1 and C2? For instance, equivalent profiles. Given a value of $\tau_v$, if the upwelling regime changes (change Z) what will the new values of C1 and C2 be to keep the same $\tau_v$? (Why would this be important? Does it make sense to ask what does the profile have to look like to keep the same regime? I mean, we can't change that in real life...)

```python

```

```python

```

```python

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import proj3d, Axes3D
```

```python
from sympy.parsing.sympy_parser import parse_expr
from sympy import Matrix, symbols, expand
```

```python
from matplotlib.patches import FancyArrowPatch
class Arrow3D(FancyArrowPatch):
    def __init__(self, xs, ys, zs, *args, **kwargs):
        FancyArrowPatch.__init__(self, (0,0), (0,0), *args, **kwargs)
        self._verts3d = xs, ys, zs
    def draw(self, renderer):
        xs3d, ys3d, zs3d = self._verts3d
        xs, ys, zs = proj3d.proj_transform(xs3d, ys3d, zs3d, renderer.M)
        self.set_positions((xs[0],ys[0]),(xs[1],ys[1]))
        FancyArrowPatch.draw(self, renderer)
```

```python
P = parse_expr('Matrix([0,0])')
u = parse_expr('Matrix([-1,-1])')
#f = parse_expr('((x+2)**2 + x*y + y**2 + 10) / ((x+2)**2 + y**2 + 1)')
f = parse_expr('x**2 + y**3')

if not ('Matrix' in str(type(P)) and P.shape == (2, 1)):
    raise ValueError('Point P must be a 2x1 matrix.')

if len(P.free_symbols) != 0:
    raise ValueError('Point P cannot contain any variables.')

if not ('Matrix' in str(type(u)) and u.shape == (2, 1)):
    raise ValueError('Vector u must be a 2x1 matrix.')

if len(u.free_symbols) != 0:
    raise ValueError('Vector u cannot contain any variables.')

if len(f.free_symbols) != 2:
    raise ValueError('Function f requires two variables.')

variables = [str(s) for s in f.free_symbols]
variables.sort()
x, y = variables[0], variables[1]
t = symbols('t')
grad = Matrix([f.diff(x), f.diff(y)])

P_grad = grad.subs(x, P[0] + t).subs(y, P[1] + t)
Df = Matrix(np.dot(P_grad.T, u))

# create surface plot
fig = plt.figure(figsize=(12,12))
plt.subplot(211)

xs = np.linspace(-4, 4, 32)
ys = np.linspace(-4, 4, 32)
X, Y = np.meshgrid(xs, ys)
zs = np.array([expand(f).subs(x, xp).subs(y, yp).evalf() for xp, yp in zip(np.ravel(X), np.ravel(Y))]).astype('float')
Z = zs.reshape(X.shape)

minz = np.floor(np.min(Z))

ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X, Y, Z, cmap=cm.magma, lw=3, alpha=0.3)
ax.contour(X, Y, Z, 10, lw=3, cmap="magma", linestyles="solid", offset=minz)

# plot directional derivative
px, py, pz = P[0], P[1], expand(f).subs(x, P[0]).subs(y, P[1]).evalf()
dx, dy, dz = u[0] + px, u[1] + py, expand(f).subs(x, u[0] + px).subs(y, u[1] + py).evalf()

plot([px], [py], [pz], 'o', zorder=3, c='purple', markersize=8)
ax.add_artist(Arrow3D([float(px), float(u[0])], [float(py), float(u[1])], [minz]*2, mutation_scale=15, lw=2, arrowstyle="-|>", ls='dashed', color="gray", zorder=5))
ax.add_artist(Arrow3D([float(px), float(px)], [float(py), float(py)], [minz, float(pz)], mutation_scale=15, lw=2, ls='dashed', arrowstyle="-", color="gray", zorder=5))
```

```python
P[0]
```

    0

```python

```


Construir o gráfico e encontrar o foco e uma equação da diretriz.

3. $y^2 = -8x$

$2p = -8$,logo

$p = -4$

Calculando o foco

$F = \frac{p}{2}$

$F = \frac{-4}{2}$

$F = -2$

$F(-2,0)$

Calculando a diretriz

$d = -\frac{p}{2}$

$d = -(-2)$

$d : x = 2$

$V(0,0)$

$F(-2,0)$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((y-0)**2, -8*(x+0)), (x,-5,10), (y,-10,10),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/03.ipynb


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""
# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
half = S(1)/2

c, EI, l, F = var("c, EI, l, F")
a1, a2 = var("a_1, a_2")
x  = var("x")

# Ansatz:
(x2, x3) = (x*x, x*x*x)
w = (a1*x2 + a2*x3)
wl = w.subs(x,l)

# w', w'':
wp   = diff(w,x)
wpp  = diff(wp,x)
wpp2 = wpp*wpp

# Energy:
I = integrate(wpp2, x)
I = I.subs(x,l)
# Definite integral:
U1 = half*EI*I
U2 = half*c*wl*wl
UF = F*wl
U  = U1 + U2 - UF

pprint("\nU1:")
pprint(U1)
pprint("\nU2:")
pprint(U2)
pprint("\nUF:")
pprint(UF)

# Find Min:
eq1 = diff(U,a1)
eq2 = diff(U,a2)
pprint("\n")
pprint(eq1)
pprint("\n")
pprint(eq2)

Eqs = [eq1, eq2]

# Solve for a1 and a2:
sol=solve(Eqs,[a1,a2])
pprint("\nSolution:")
pprint(sol)

# U1:
#    ⎛    2               2        2  3⎞
# EI⋅⎝4⋅a₁ ⋅l + 12⋅a₁⋅a₂⋅l  + 12⋅a₂ ⋅l ⎠
# ──────────────────────────────────────
#                   2
#
# U2:
#                  2
#   ⎛    2       3⎞
# c⋅⎝a₁⋅l  + a₂⋅l ⎠
# ──────────────────
#         2
#
# UF:
#    ⎛    2       3⎞
# -F⋅⎝a₁⋅l  + a₂⋅l ⎠
#
# Solution:
# ⎧         3⋅F⋅l                -F      ⎫
# ⎪a₁: ───────────────, a₂: ─────────────⎪
# ⎨      ⎛          3⎞                  3⎬
# ⎪    2⋅⎝3⋅EI + c⋅l ⎠      6⋅EI + 2⋅c⋅l ⎪
# ⎩                                      ⎭

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
from sympy.parsing.sympy_parser import parse_expr
from sympy import Matrix, symbols, expand, latex
import numpy as np
import matplotlib.pyplot as plt

```

```python
f = parse_expr('2*sin(6 * phi)')
var = list(f.free_symbols)[0]

r = np.arange(0, 2*pi, 0.01)
theta = [f.subs(var, r).evalf() for r in r]

plt.figure(figsize=(10,10))
ax = plt.subplot(111, projection='polar')
ax.plot(r, theta, c='purple', lw=4)
ax.grid(True)

ax.set_title("Polat plot of ${}$".format(latex(f)), va='bottom')
plt.show()
```

```python

```


# Propriedade da multiplicação e janelas

Imagine que você tenha um fenômeno (ou sinal) com duração infinita, $x(t)$, que deseja observar (medir).

Quando medimos $x(t)$ por um tempo finito estamos observando o fenômeno por uma janela temporal $w(t)$ finita. Na prática, o sinal observado é:

\begin{equation}
x_o(t) = x(t)w(t)
\end{equation}

e pela propriedade da dualidade e do teorema da convolução, o espectro observado será

\begin{equation}
X_o(\mathrm{j} \omega) = X(\mathrm{j} \omega) * W(\mathrm{j} \omega)
\end{equation}

ou seja, o espectro observado terá em si as características do fenômeno em si e da janela de observação. Vamos investigar os espectros dos sinais $x(t)$, $w(t)$ e $x_o(t)$.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
from scipy import signal
```

# O sinal infinito (ou ao menos de longa duração)

```python
fs = 1000
time = np.arange(0, 1000, 1/fs)
xt = np.cos(2*np.pi*10*time)
# Espectro
Xjw = np.fft.fft(xt)
freq = np.linspace(0, (len(Xjw)-1)*fs/len(Xjw), len(Xjw))

# tempo
plt.figure(figsize=(12,4))
plt.plot(time, xt, '-b', linewidth = 1, label = r'$x(t)$')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 1000))

# frequencia
plt.figure(figsize=(12,4))
plt.semilogx(freq, 20*np.log10(2*np.abs(Xjw)/len(Xjw)), '-b', linewidth = 1, label = r'$|X(\mathrm{j} \omega)|$')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|X(\mathrm{j} \omega)|$ [-]')
plt.xlim((1, fs/2))
plt.ylim((-80, 10));
```

# Definamos algumas janelas

1. Retangular
2. Hanning
3. Hammming

```python
# Tempo de observação
Tp = 10
Np = len(time[time<Tp]) # numero de amostras contidas na janela de observação (início do sinal)
tw = np.linspace(0, Tp, Np) # vetor temporal de observação

# Janela retangular
w_ret = np.concatenate((np.ones(Np), np.zeros(len(xt)-Np)))
W_ret = np.fft.fft(w_ret)

# Janela Hanning
w_hann = np.concatenate((signal.hann(Np), np.zeros(len(xt)-Np)))
W_hann = np.fft.fft(w_hann)

# Janela Hamming
w_hamm = np.concatenate((signal.hamming(Np), np.zeros(len(xt)-Np)))
W_hamm = np.fft.fft(w_hamm)

# tempo
plt.figure(figsize=(12,4))
plt.plot(time, w_ret, '-r', linewidth = 3, label = 'Rect.')
plt.plot(time, w_hann, '-g', linewidth = 3, label = 'Hanning')
plt.plot(time, w_hamm, '-k', linewidth = 3, label = 'Hamming')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 2*Tp))
plt.ylim((0, 1.1))

# frequencia
plt.figure(figsize=(12,4))
plt.semilogx(freq, 20*np.log10(2*np.abs(W_ret)/Np), '-r', linewidth = 3, label = 'Rect')
plt.semilogx(freq, 20*np.log10(2*np.abs(W_hann)/Np), '-g', linewidth = 3, label = 'Hanning')
plt.semilogx(freq, 20*np.log10(2*np.abs(W_hamm)/Np), '-k', linewidth = 3, label = 'Hamming')
plt.legend(loc = 'lower left')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|W(\mathrm{j} \omega)|$ [-]')
plt.xlim((0.001, 10))
plt.ylim((-80, 10));

```

# Observemos o espectro resultante

```python
# Retangular
xo_ret = xt*w_ret
Xo_ret = np.fft.fft(xo_ret)

# Hanning
xo_hann = xt*w_hann
Xo_hann = np.fft.fft(xo_hann)
f_hann = np.sqrt(np.sum(w_ret**2)/np.sum(w_hann**2))

# Hamming
xo_hamm = xt*w_hamm
Xo_hamm = np.fft.fft(xo_hamm)
f_hamm = np.sqrt(np.sum(w_ret**2)/np.sum(w_hamm**2))

# tempo
plt.figure(figsize=(12,4))
plt.plot(time, xo_ret, '-r', linewidth = 2, label = 'Rect.')
plt.plot(time, xo_hann, '-g', linewidth = 2, label = 'Hanning')
plt.plot(time, xo_hamm, '-k', linewidth = 2, label = 'Hamming')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 2*Tp))
plt.ylim((-1.1, 1.1))

# frequencia
plt.figure(figsize=(12,4))
plt.semilogx(freq, 20*np.log10(2*np.abs(Xo_ret)/Np), '-r', linewidth = 2, label = 'Rect')
plt.semilogx(freq, 20*np.log10(2*f_hann*np.abs(Xo_hann)/Np), '-g', linewidth = 2, label = 'Hanning')
plt.semilogx(freq, 20*np.log10(2*f_hamm*np.abs(Xo_hamm)/Np), '-k', linewidth = 2, label = 'Hamming')
plt.legend(loc = 'lower left')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|W(\mathrm{j} \omega)|$ [-]')
plt.xlim((1, fs/2))
plt.ylim((-80, 10));

```

Aula 27 - propriedade da multiplicacao e janelas/Jamelas.ipynb


$$
\newcommand{\mat}[1]{\boldsymbol {#1}}
\newcommand{\mattr}[1]{\boldsymbol {#1}^\top}
\newcommand{\matinv}[1]{\boldsymbol {#1}^{-1}}
\newcommand{\vec}[1]{\boldsymbol {#1}}
\newcommand{\vectr}[1]{\boldsymbol {#1}^\top}
\newcommand{\rvar}[1]{\mathrm {#1}}
\newcommand{\rvec}[1]{\boldsymbol{\mathrm{#1}}}
\newcommand{\diag}{\mathop{\mathrm {diag}}}
\newcommand{\set}[1]{\mathbb {#1}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\pderiv}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\bb}[1]{\boldsymbol{#1}}
$$

# Tutorial 2: Multilayer Perceptron

## Todays Agenda
---
* recap on linear models
* Linear (fully connected) layers
* Activation functions
* 2-Layer MLP implementation from scratch (last time we use numpy for networks)
* N-layer MLP with PyTorch's `nn`, `autograd` and `optim` modules

## imports
---

```python
# Setup
%matplotlib inline
import os
import numpy as np
import sklearn
import torch
import matplotlib.pyplot as plt
from typing import Sequence
```

```python
plt.rcParams['font.size'] = 20
```

## Reminders and recaps
---
### Perceptrons and linear models

The following hypothesis class
$$
\mathcal{H} =
\left\{ h: \mathcal{X}\rightarrow\mathcal{Y}
~\vert~
h(\vec{x}) = \varphi(\vectr{w}\vec{x}+b); \vec{w}\in\set{R}^D,~b\in\set{R}\right\}
$$
where $\varphi(\cdot)$ is some nonlinear function, is composed of functions representing the **perceptron** model.

Schematic of a single perceptron, and it's inspiration, a biological neuron.

### logistic regression
we train a model using the logistic function (aka sigmoid):
$$\varphi(\vec{z})=\sigma(\vec{z})=\frac{1}{1+\exp(-\vec{z})}\in[0,1].$$

**Limitation**: logistic regression is still a linear classifier. In what sense is it linear if we use a non-linear function?

$$\hat{y} = \sigma(\vectr{w}\vec{x}+b)$$

**Linear** in the sense that output depends only on a linear combination of weights and inputs.
we can also call it- first moment of the input.

Decision boundaries are therefore straight lines, or can be transformed to linear via link function.

If we define $y(\vec{x})=\vectr{w}\vec{x}+b$, then the **decision surface** of the classifier is $y(\vec{x})=\text{const}$ (usually zero).

What if our data is not linearly separable?
in most cases in the real world, we want to learn a non-linear combination of the input, such as the examples below

or even something more abstract, like a corrolation between 2 or more inputs

Can we still use e.g. logistic regression?

What if we first apply a **fixed** nonlinear transformation to the data?

For some $\psi: \set{R}^d \rightarrow \set{R}^D$, what does the following classifier do?

$$\hat{y} = \varphi(\vectr{w}\psi(\vec{x})+b).$$

Now we can get nonlinear decision boundaries with respect to $\vec{x}$.

But how can we choose a nonlinear transformation?
in **Machine Learning** you learn to look at the data and manually craft those kernel functions.

* what if the input is from high dimention and we can't look at it?
* waht if we look at some graphs of diffrent inputs, but we're not sure witch ones are impotant? (x1 vs x2? x3 corrolation with x5?)
* what do we do when we look but we don't know how to model the relation? we choose a set of functions and do hypothesis testing?
* what if we want to **learn** the non-linearity?

## Multilayer Perceptron (MLP)
---

### Model

Composed of $L$ **layers**, each layer $l$ with $n_l$ **perceptron** ("neuron") units.

Each layer $l$ operates on the output of the previous layer ($\vec{y}_{l-1}$) and calculates:

$$
\vec{y}_l = \varphi\left( \mat{W}_l \vec{y}_{l-1} + \vec{b}_l \right),~
\mat{W}_l\in\set{R}^{n_{l}\times n_{l-1}},~ \vec{b}_l\in\set{R}^{n_l},~ l \in \{1,2,\dots,L\}.
$$

- Note that both input and output are **vectors**. We can think of the above equation as describing a layer of **multiple perceptrons**.
- We'll henceforth refer to such layers as **fully-connected** or FC layers.
- The first layer accepts the input of the model, i.e. $\vec{y}_0=\vec{x}\in\set{R}^d$.
- The last layer, $L$, is the output layer, so $y_L$ is the output of the model.
- The layers $1, 2, \dots, L-1$ are called hidden layers.

How powerful is an MLP model? I.e. which functions can it approximate?

**Universal approximator theorem**

Given enough parameters, an MLP with $L>1$ and any non-linear activation function, can approximate any continuous function up to any specified precision (Cybenko, 1989).

The MLP is therefore a **potent hypothesis class** (recall approximation error).

See [here](http://neuralnetworksanddeeplearning.com/chap4.html) for an intuitive explanation of the UAT.

Personally, i prefer to look at it as aproximation we do using non-linearities.
sort of in the notion of Taylor series expantion:

in taylor expantion we use all moments, but the idea remain the same, aproximation of a function when reducing the error.

Given an input sample $\vec{x}^i$, the computed function of an $L$-layer MLP is:
$$
\vec{y}_L^i= \varphi \left(
\mat{W}_L \varphi \left( \cdots
\varphi \left( \mat{W}_1 \vec{x}^i + \vec{b}_1 \right)
\cdots \right)
+ \vec{b}_L \right)
$$

This expression is fully differentiable w.r.t. parameters using the **Chain Rule**.

And notice that $\vec{y}_L^i = \varphi(\mat{W}_L \vec{y}_{L-1}^i + \vec{b}_L)$ is just the linear model we started with.

So, intuitively, we can think of $\vec{y}_{L-1}$ as **learned** non-linear features of the input! In other words, $\vec{y}_{L-1} = \psi_{\vec{\Theta}}(\vec{x})$.

Since an MLP it has a non-linear dependency on it's inputs, through a learned transformation, non-linear decision boundaries are possible.

Cool visualization of the effect of MLP layers: http://playground.tensorflow.org/

## Activation functions
---

An **activation function** is the non-linear elementwise function $\varphi(\cdot)$ which operates on the affine part of the perceptron model.

Why do we even need non-linearities in the first place? Isn't the depth enough?

Without them, the MLP model would be equivalent to a single affine transform.

Common choices for the activation functions are:

- The logistic function (sigmoid)
    $$ \varphi(t) = \sigma(t) = \frac{1}{1+e^{-t}} \in  [0,1] $$
- The hyperbolic tangent (a shifted and scaled sigmoid)
    $$ \varphi(t) = \mathrm{tanh}(t) = \frac{e^t - e^{-t}}{e^t +e^{-t}} \in [-1,1]$$

- ReLU, rectified linear unit
    $$ \varphi(t) = \max\{t,0\} $$
Note that ReLU is not strictly differentiable. However, sub-gradients exist. We will define its gradient as:
    $$ \pderiv{\varphi}{t} = \begin{cases} 1, & t\geq0 \\ 0, & t<0 \end{cases} $$

```python
# activation functions
def sigmoid(z, deriv=False):
    output = 1 / (1 + np.exp(-1.0 * z))
    if deriv:
        return output * (1 - output)
    return output

def tanh(z, deriv=False):
    output = np.tanh(z)
    if deriv:
        return 1 - np.square(output)
    return output

def relu(z, deriv=False):
    output = z if z > 0 else 0
    if deriv:
        return 1 if z > 0 else 0
    return output
```

```python
def plot_activations():
    x = np.linspace(-5, 5, 1000)
    y_sig = sigmoid(x)
    y_tanh = tanh(x)
    y_relu = list(map(lambda z: relu(z), x))
    fig = plt.figure(figsize=(8, 5))
    ax1 = fig.add_subplot(2,1,1)
    ax1.plot(x, y_sig, label='sigmoid')
    ax1.plot(x, y_tanh, label='tanh')
    ax1.plot(x, y_relu, label='relu')
    ax1.grid()
    ax1.legend()
    ax1.set_xlabel('x')
    ax1.set_ylabel('y')
    ax1.set_ylim([-2, 2])
    ax1.set_title('Activation Functions')

    y_sig_derv = sigmoid(x, deriv=True)
    y_tanh_derv = tanh(x, deriv=True)
    y_relu_derv = list(map(lambda z: relu(z, deriv=True), x))
    ax2 = fig.add_subplot(2,1,2)
    ax2.plot(x, y_sig_derv, label='sigmoid')
    ax2.plot(x, y_tanh_derv, label='tanh')
    ax2.plot(x, y_relu_derv, label='relu')
    ax2.grid()
    ax2.legend()
    ax2.set_xlabel('x')
    ax2.set_ylabel('y')
    # ax2.set_ylim([-2, 2])
    ax2.set_title('Activation Functions Derivatives')
    plt.tight_layout()
```

```python
plot_activations()
```

Some reasons people cite for using ReLU are:

- Does not suffer from vanishing gradient when $x$ is far from zero (though gradient can still be zero).

- Much faster to compute than sigmoid and tanh.

- Promotes sparse weight vectors: "dead neurons" arguably cause sparsity in the next layer.

## Two-layer MLP implementation
---

Let's solve a simple **regression** problem with a 2-layer MLP (one hidden layer, one output layer).

We're trying to learn a continuous and perhaps non-deterministic function $y=f(\vec{x})$.

- Domain: $\vec{x}^i \in \set{R}^{D_{\text{in}}}$

- Target: $y^i \in \set{R}^{D_{\text{out}}}$

- Model: $\hat{y} =\mat{W}_2~ \varphi(\mat{W}_1 \vec{x}+ \vec{b}_1) + \vec{b}_2$

  i.e. a 2-layer MLP, where:
    - $\vec{x}\in\set{R}^{D_{\text{in}}}$ sample (feature vector)
    - $\mat{W}_1\in\set{R}^{H\times D_{\text{in}}},\ \vec{b}_1\in\set{R}^{H}$
    - $\mat{W}_2\in\set{R}^{D_{\text{out}}\times H},\ \vec{b}_2\in\set{R}^{D_{\text{out}}}$
    - $\varphi(\cdot) = \mathrm{ReLU}(\cdot) = \max\{\cdot,0\}$
    - $H$ is the hidden dimension
    - We'll set $D_{\text{out}}=1$ so output is a scalar

- MSE loss with L2 regularization:
    $$
    \begin{align}
    \ell(\vec{y},\vec{\hat y}) &= \frac{1}{2}\norm{\vec{\hat y} - \vec{y}}^2 \\
    L_{\mathcal{S}} &= \frac{1}{N}\sum_{i=1}^{N}\ell(\vec{y},\vec{\hat y}) + \frac{\lambda}{2}\left(\norm{\mat{W}_1}_F^2 + \norm{\mat{W}_2}_F^2 \right)
    \end{align}
    $$
- Optimization scheme: Vanilla SGD

### Computing the loss gradients with backpropagation

Let's write our model as
$$
\hat{y} =\mat{W}_2~ \underbrace{\mathrm{relu}(\overbrace{\mat{W}_1 \vec{x}+ \vec{b}_1}^{\vec{z}})}_{\vec{a}} + \vec{b}_2,
$$

and manually derive the gradient of the point-wise loss $\ell(\vec{y},\vec{\hat y})$ using the **chain rule**.
Remember that to use SGD, we need the gradient of the loss w.r.t. our parameter tensors.

$$
\begin{align}
&\pderiv{\ell}{\vec{\hat y}}=2\cdot\frac{1}{2}(\vec{\hat y}-\vec{y}) = (\vec{\hat y}-\vec{y})\\
(\ast)~~&\pderiv{\ell}{\mat{W}_2}= \pderiv{\ell}{\vec{\hat y}}\pderiv{\vec{\hat y}}{\mat{W}_2}
=(\vec{\hat y}-\vec{y})\vectr{a}
\end{align}
$$

How do we continue into the nonlinearity? Recall that we defined

$$ \pderiv{\mathrm{relu}(x)}{x} = \begin{cases} 1, & x\geq0 \\ 0, & x<0 \end{cases} $$

and that we apply the non-linearity elementwise on input tensors.

Also remember that the gradient of a vector w.r.t. another vector is the Jacobian, a matrix of mixed derivatives:
$\pderiv{a_i}{z_j}$.

We have $\vec{a}=\mathrm{relu}(\vec{z})$. Thus,

$$
\pderiv{\vec{a}}{\vec{z}}=\mathrm{diag}(\mathbb{1}[\vec{z}>0]).
$$

And so,
$$
\begin{align}
&\pderiv{\ell}{\vec{a}}=\pderiv{\vec{\hat y}}{\vec{a}}\pderiv{\ell}{\vec{\hat y}}
=\mattr{W}_2(\vec{\hat y}-\vec{y})\\
&\pderiv{\ell}{\vec{z}}=\pderiv{\vec{a}}{\vec{z}}\pderiv{\ell}{\vec{a}}
=\mathrm{diag}(\mathbb{1}[\vec{z}>0])\mattr{W}_2(\vec{\hat y}-\vec{y})\\
(\ast)~~&\pderiv{\ell}{\mat{W}_1}=\pderiv{\ell}{\vec{z}}\pderiv{\vec{z}}{\mat{W}_1}
=\mathrm{diag}(\mathbb{1}[\vec{z}>0])\mattr{W}_2(\vec{\hat y}-\vec{y})\vectr{x}\\
\end{align}
$$

For the biases, we can easily see that:
$$
\begin{align}
(\ast)~~&\pderiv{\ell}{\vec{b}_2}=\pderiv{\vec{\hat y}}{\vec{b}_2}\pderiv{\ell}{\vec{\hat y}}
=I_{D_{\text{out}}} \pderiv{\ell}{\vec{\hat y}} \\
(\ast)~~&\pderiv{\ell}{\vec{b}_1}=\pderiv{\vec{z}}{\vec{b}_1}\pderiv{\ell}{\vec{z}}
=I_{H}\pderiv{\ell}{\vec{z}}\\
\end{align}
$$

The final gradients for weight update, including regularization will be

$$
\nabla_{\mat{W}_j}L_{\mathcal{S}}=\frac{1}{N}\sum_{i=1}^{N} \pderiv{\ell_i}{\mat{W}_j} + \lambda\mat{W}_j, \ j=1,2.
$$

Let's implement it from scratch using just `numpy`.

```python
# A simple MLP with one hidden layer

# N: batch size
# D_in: number of features
N, D_in =  64, 10
# H: hidden-layer
# D_out: output dimension
H, D_out =  100, 1

# Random input data
X = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Model weights and biases
wstd = 0.01
W1 = np.random.randn(H, D_in)*wstd
b1 = np.random.randn(H,)*wstd + 0.1
W2 = np.random.randn(D_out, H)*wstd
b2 = np.random.randn(D_out,)*wstd + 0.1

reg_lambda = 0.5
learning_rate = 1e-3
```

```python
losses = []
for epoch in range(250):
    # Forward pass, hidden layer: A = relu(X W1 + b1), Shape: (N, H)
    Z = X.dot(W1.T) + b1
    A = np.maximum(Z, 0)

    # Forward pass, output layer: Y_hat = A W2 + b2, Shape: (N, D_out)
    Y_hat = A.dot(W2.T) + b2

    # Loss calculation (MSE)
    loss = np.mean((Y_hat - y) ** 2); losses.append(loss) # (N, D_out)

    # Backward pass:  Output layer
    d_Y_hat = (1./N) * (Y_hat - y)     # (N, D_out)
    d_W2 = d_Y_hat.T.dot(A)            # (D_out, H)
    d_A = d_Y_hat.dot(W2)              # (N, H)
    d_b2 = np.sum(d_Y_hat, axis=0)     # (D_out,)

    # Backward pass: Hidden layer
    d_Z = d_A * np.array(Z > 0, dtype=float)  # (N, H)
    d_W1 = d_Z.T.dot(X)                          # (H, D_in)
    d_b1 = np.sum(d_Z, axis=0)                   # (H,)

    # Backward pass: Regularization term
    d_W2 += reg_lambda * W2
    d_W1 += reg_lambda * W1

    # Gradient descent step
    W2 -= d_W2 * learning_rate; b2 -= d_b2 * learning_rate
    W1 -= d_W1 * learning_rate; b1 -= d_b1 * learning_rate
    print(epoch+1, end=' ')
```

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250

```python
# Plot losses
_, ax = plt.subplots(figsize=(10,5))
ax.plot(losses)
ax.set_ylabel('MSE loss'); ax.set_xlabel('Epoch');
```

Note that this implementation is not ideal, as it's:

- Non modular (hard to switch components)
- Hard to extend (e.g. to add layers)
- Error prone (hard-coded manual calculations)

And now, we'll see how to address these issues using PyTorch's API.

##  N-Layer MLP using PyTorch
---

Let's create all our usual components:
- Dataset
- Model
- Loss function
- Optimizer

But this time we'll create a modular implementation where each of these components is separate and can be changed independently of the others.

### Dataset

As in the first tutorial we're goint to use  MNIST database of handwritten digits.

```python
import torch
import torch.utils.data
import torchvision #for dataset
import torchvision.transforms as tv_transforms

root_dir = os.path.expanduser('~/.pytorch-datasets/')
```

```python
tf_tensor = tv_transforms.ToTensor()

batch_size = 512
train_size = batch_size * 10
test_size = batch_size * 2

# Datasets and loaders
ds_train = torchvision.datasets.MNIST(root=root_dir, download=True, train=True, transform=tf_tensor)
ds_test =  torchvision.datasets.MNIST(root=root_dir, download=True, train=False, transform=tf_tensor)

dl_train = torch.utils.data.DataLoader(ds_train, batch_size,
                                       sampler=torch.utils.data.SubsetRandomSampler(range(0,train_size)))
dl_test = torch.utils.data.DataLoader(ds_test, batch_size,
                                       sampler=torch.utils.data.SubsetRandomSampler(range(0,test_size)))

x0, y0 = ds_train[0]
n_features = torch.numel(x0)
n_classes = 10

print(f'x0: {x0.shape}, y0: {y0}')
```

    x0: torch.Size([1, 28, 28]), y0: 5

```python
import sys
sys.path.append('..')
import T1.plot_utils as plot_utils
# Show first few samples
print(f'x0: {x0.shape}, y0: {y0}')
plot_utils.dataset_first_n(ds_train, 10, cmap='gray', show_classes=True);
```

### Model Implementation

- The `torch.nn` module contains building blocks such as neural network layers,
  loss functions, activations and more.
- In this section, we'll see various parts of the `torch.nn` API.
- We'll use `nn.Linear` which implements a single MLP layer.
- We'll implement our model as a subclass of `nn.Module`, which means:
    - Any tensors we set as properties will be registered as model parameters.
    - We can nest `nn.Modules` and get all model parameters from the top-level `nn.Module`.
    - Can be used as a function if we implement the `forward()` method.

To understand `nn.Module`, lets look at a very basic one: the [**fully-connected**](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear) layer.

```python
import torch.nn as nn

fc = nn.Linear(in_features=3, out_features=5, bias=True)

# 10 samples of 3 features
t = torch.randn(10, 3)

print(f'w: {fc.weight.shape} \nb: {fc.bias.shape}')

# Forward pass, notice that grad_fn exists
print(f'output {fc(t).shape}')
fc(t)
```

    w: torch.Size([5, 3])
    b: torch.Size([5])
    output torch.Size([10, 5])

    tensor([[-0.0394,  0.9521,  0.3728,  0.1874, -0.9546],
            [ 0.2046,  0.3222,  0.9931, -0.0036,  0.3895],
            [ 0.0217,  0.8108,  1.3113,  0.6841,  0.4009],
            [ 0.4667,  0.4600,  0.5504,  0.8537, -0.1427],
            [ 1.0021, -0.8150,  0.3949, -0.3162,  0.7448],
            [-0.2779,  0.6849,  1.1078, -0.6873,  0.0565],
            [-0.7619,  1.9060,  0.7816,  0.2172, -1.4030],
            [ 0.2288,  0.6012,  0.4152,  0.3095, -0.5350],
            [-0.4877,  1.3889,  0.7999, -0.0210, -0.9062],
            [ 0.3430,  0.6981,  0.7921,  1.1339, -0.0414]],
           grad_fn=<AddmmBackward>)

`nn.Modules` have registered **parameters**, which are tensors which `require_grad`.

```python
# Note parameter dimenstions
list(fc.parameters())
```

    [Parameter containing:
     tensor([[-0.3894,  0.3235, -0.2057],
             [ 0.4167, -0.1986,  0.5722],
             [-0.0529, -0.5532, -0.0751],
             [-0.3746,  0.2465,  0.5421],
             [-0.5388, -0.4311, -0.5707]], requires_grad=True),
     Parameter containing:
     tensor([ 0.3040,  0.5668,  0.3823,  0.4548, -0.5169], requires_grad=True)]

We can create custom `nn.Module`s with arbitrary logic.

Let's recreate a fully-connected layer ourselves:

```python
class MyFullyConnectedLayer(nn.Module):

    def __init__(self, in_features, out_features,bias=True):
        super().__init__() # don't forget this to inherent from nn.Module
        self.bias = bias
        # nn.Parameter just marks W,b for inclusion in list of parameters
        self.W = nn.Parameter(torch.randn(out_features, in_features, requires_grad=True))
        if self.bias:
            self.b = nn.Parameter(torch.randn(out_features, requires_grad=True))

    def forward(self, x):
        # x assumed to be (N, in_features)
        z = torch.matmul(x, self.W.transpose(0, 1)) + self.b
        if self.bias:
            z+= self.b
        return z
```

```python
myfc = MyFullyConnectedLayer(in_features=3, out_features=5)
```

```python
myfc(t)
```

    tensor([[-0.1501, -2.7414, -0.1758,  1.5974,  0.0062],
            [-0.4833, -2.7031,  0.0597, -0.2723, -1.3212],
            [-2.7898, -3.3118, -1.1418, -0.4562, -1.0730],
            [-1.9656, -4.5765, -1.0430, -1.8895,  0.2665],
            [ 1.4678, -3.8225,  0.9936, -3.2285, -0.8759],
            [ 0.9569, -0.7969,  0.8913,  2.7259, -2.1027],
            [-1.0669, -1.3888, -0.6013,  4.7076, -0.4420],
            [-0.4387, -3.3846, -0.2691,  0.0515, -0.0145],
            [-0.3859, -1.5120, -0.1348,  3.5276, -0.8080],
            [-3.0860, -4.7044, -1.5791, -1.8620,  0.1921]], grad_fn=<AddBackward0>)

```python
list(myfc.parameters())
```

    [Parameter containing:
     tensor([[ 0.9171,  0.2804, -1.3612],
             [ 1.2308, -1.1032, -0.4678],
             [ 0.4355, -0.1168, -0.8913],
             [ 2.4323, -1.1664,  0.5993],
             [-0.1950,  1.1053,  0.7663]], requires_grad=True),
     Parameter containing:
     tensor([-0.3707, -1.8719, -0.2272, -0.2174,  0.0869], requires_grad=True)]

Quick visualization of our custom module's **computation graph**:

```python
import torchviz
torchviz.make_dot(myfc(t), params=dict(W=myfc.W, b=myfc.b))
```

Now that we know about `nn.Module`s, lets create an fairly-general MLP for multiclass classification.

```python
class MLP(torch.nn.Module):
    NLS = {'relu': torch.nn.ReLU, 'tanh': nn.Tanh, 'sigmoid': nn.Sigmoid, 'softmax': nn.Softmax, 'logsoftmax': nn.LogSoftmax}

    def __init__(self, D_in: int, hidden_dims: Sequence[int], D_out: int, nonlin='relu'):
        super().__init__()

        all_dims = [D_in, *hidden_dims, D_out]
        non_linearity = MLP.NLS[nonlin]
        layers = []

        for in_dim, out_dim in zip(all_dims[:-1], all_dims[1:]):
            layers += [
                nn.Linear(in_dim, out_dim, bias=True),
                non_linearity()
            ]

        self.fc_layers = nn.Sequential(*layers[:-1])
        self.log_softmax = nn.LogSoftmax(dim=1)

    def forward(self, x):
        x = torch.reshape(x, (x.shape[0], -1)) #flatten x
        z = self.fc_layers(x)
        y_pred = self.log_softmax(z)
        # Output is always log-probability
        return y_pred
```

```python
# Create an instance of the model: 5-layer MLP
mlp5 = MLP(D_in=n_features, hidden_dims=[32, 64, 128, 64], D_out=n_classes, nonlin='tanh')

print(mlp5)
```

    MLP(
      (fc_layers): Sequential(
        (0): Linear(in_features=784, out_features=32, bias=True)
        (1): Tanh()
        (2): Linear(in_features=32, out_features=64, bias=True)
        (3): Tanh()
        (4): Linear(in_features=64, out_features=128, bias=True)
        (5): Tanh()
        (6): Linear(in_features=128, out_features=64, bias=True)
        (7): Tanh()
        (8): Linear(in_features=64, out_features=10, bias=True)
      )
      (log_softmax): LogSoftmax(dim=1)
    )

```python
print(f'number of parameter tensors: {len(list(mlp5.parameters()))}')
```

    number of parameter tensors: 10

```python
print(f'number of parameters: {np.sum([torch.numel(p) for p in mlp5.parameters()])}')
```

    number of parameters: 44458

```python
# Test a forward pass
y_hat0 = mlp5(x0)

print(f'{x0.shape=}\n')
print(f'{y_hat0.shape=}\n')
print(f'{y_hat0=}')
```

    x0.shape=torch.Size([1, 28, 28])

    y_hat0.shape=torch.Size([1, 10])

    y_hat0=tensor([[-2.2084, -2.3476, -2.4827, -2.4447, -2.2673, -2.2707, -2.2224, -2.2763,
             -2.2885, -2.2538]], grad_fn=<LogSoftmaxBackward>)

Quick visualization of our full MLP's **computation graph**:

```python
torchviz.make_dot(mlp5(x0), params=dict(mlp5.named_parameters()))
```

### Loss and Optimizer

For the loss function, we'll use PyTorch's built in negative log-likelihood loss since our model outputs probabilities.

Log Softmax is advantageous over softmax for numerical stability, optimisation and heavy penalisation for highly incorrect class

softmax for first class:
$$
\frac{e^{w_1x_1}}{\sum_i{e^{w_ix_i}}}
$$
log softmax for first class:
$$
\frac{log(e^{w_1x_1})}{log(\sum_i{e^{w_ix_i}})} = \frac{log(e^{w_1x_1})}{\sum_i{log(e^{w_ix_i}})} =\frac{log(e^{w_1x_1})}{\sum_i{w_ix_i}}
$$

```python
Probability =  torch.tensor([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
Log_Probability = torch.log(Probability)
print(f'{Probability=}\n {Log_Probability=}')
```

    Probability=tensor([0.1000, 0.2000, 0.3000, 0.4000, 0.5000, 0.6000, 0.7000, 0.8000, 0.9000,
            1.0000])
     Log_Probability=tensor([-2.3026, -1.6094, -1.2040, -0.9163, -0.6931, -0.5108, -0.3567, -0.2231,
            -0.1054,  0.0000])

```python
import torch.optim

# Loss:
# Note: NLLLoss assumes *log*-probabilities (given by our LogSoftmax layer)
loss_fn = nn.NLLLoss()

# Fake ground-truth labels
# Notice that we don't need to 1-hot encode them!
yt = torch.randint(low=0, high=n_classes, size=(y_hat0.shape[0],))

# Try out the loss
loss_fn(y_hat0, yt)
```

    tensor(2.2224, grad_fn=<NllLossBackward>)

As for the optimization scheme, we'll use a built in SGD optimizer from the `torch.optim` module.

We won't need to calculate the loss gradient this time, as we'll use `autograd` for automatic differentiation.

```python
torch.manual_seed(42)

# Model for training
model = MLP(D_in=n_features, hidden_dims=[32, 32, 32], D_out=n_classes, nonlin='relu')

# Optimizer over our model's parameters
optimizer = torch.optim.SGD(params=model.parameters(), lr=5e-2, weight_decay=0.01, momentum=0.9)
```

### Training loop

This time we'll train over lazy-loaded batches from our data loader.

Notice that except from our model's `__init__()` and `__forward()__`, we're using PyTorch facilities for the entire training implementation.

```python
num_epochs = 10
losses_mlp = []
for epoch_idx in range(num_epochs):
    total_loss = 0

    for batch_idx, (X, y) in enumerate(dl_train):
        # Forward pass
        y_pred = model(X) #btz x channels x input_w x input_h --> btz x num_classes

        # Compute loss
        loss = loss_fn(y_pred, y)
        total_loss += loss.item()
        losses_mlp.append(loss.item())

        # Backward pass
        optimizer.zero_grad() # Zero gradients of all parameters
        loss.backward()       # Run backprop algorithms to calculate gradients

        # Optimization step
        optimizer.step()      # Use gradients to update model parameters

    print(f'Epoch #{epoch_idx+1}: Avg. loss={total_loss/len(dl_train)}')
```

    Epoch #1: Avg. loss=2.3051342964172363
    Epoch #2: Avg. loss=2.2864507913589476
    Epoch #3: Avg. loss=2.256749725341797
    Epoch #4: Avg. loss=2.176988220214844
    Epoch #5: Avg. loss=1.9311089754104613
    Epoch #6: Avg. loss=1.416105604171753
    Epoch #7: Avg. loss=0.919141310453415
    Epoch #8: Avg. loss=0.6763370871543884
    Epoch #9: Avg. loss=0.561392143368721
    Epoch #10: Avg. loss=0.5014606237411499

```python
# Plot losses
_, ax = plt.subplots(figsize=(10,5))
ax.plot(losses_mlp)
ax.set_ylabel('MSE loss'); ax.set_xlabel('Epoch');
```

Using the basic PyTorch building blocks we have arrived at a much more robust implementation:
- Easy to change architecture: layers and activation functions
- Easy to change loss
- Easy to change optimization method
- No need for manual gradient derivations

## Further reading
---

### initialization

We saw today a FC layer (or MLP or Linear layer), Neural networks has other type of layers that we're going to see in the next couple of tutorials.

when we implemented it ourself, we used torch.randn, that widraw from the standart normal distribution $\mathcal{N}(0,1)$.

But why? or more important, since we optimize it, **why does it matter**?

The answer to the second questioin is quite simple. just like K-Means, Neural network is not a stable algorithem, since diffrent initilizations will bring a diffrent outcome.

why is that? remember that the loss function is not a simple convex function!
 if we start with diffrent initilizations we're probably going to end up at diffrent local minima.

**note** that i'm lying to you a little bit, since we usually use Batch optimization algorithms and diffrent batches already contain a little randomness so even if the initial point is the same, we're not going to get the same trajectory from them on the loss function, and therfore this problem might not exist...

The honest answer is that this is an active research topic and the jastifications adressing some isues that we will state here, but it's not a rigorous definition.

Intuitivly, first thing we would think of a constant value, like 0 for all weights.

The idea of no constant is to:
* accelerate training by using small values
* keep neumerical stability
* brake symmetry

Other problem that we did not adress yet, is that with long networks, the chain rule is pruned make the gradients too small or too big (**vanishing gradients** and **exploding gradients** respectively).

So what can we do if we want to sqeeze the best results we can?

#### Xavier
Briefly, Normal distribution is always a good start, and to avoid edges effect normelize with the input size would help.
however, activision functions turn to non-linear shift in distibusion in neural networks.

that's where **Xavier initilization** came to life in [Understanding the difficulty of training deep feedforward neural networks](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)

basically they did a uniform distibution in [-1,1] and then scaling by $\frac{1}{\sqrt{n}}$.

:

here you can see the gradients without xavier (left) and with xavier (right)

note that we can use this normalization with $~\mathcal(N)$ or $~\mathcal(U)$

* **Xavier Uniform** - Initialize with values using uniform distribution. The resulting tensor will have values sampled from $\mathcal{U}(-a, a)$ where $$ a = \text{gain} \times \sqrt{\frac{6}{\text{fan}_{in} + \text{fan}_{out}}} $$

* **Xavier ** - Initialize with values using a normal distribution. The resulting tensor will have values sampled from $\mathcal{N}(0,\text{std}^2)$ where $$ \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan}_{in} + \text{fan}_{out}}} $$

* `fan_in` is the number of input units in the weight tensor and `fan_out` is the number of output units in the weight tensor

```python
w = torch.empty(1, 1000)
xavier_u = torch.nn.init.xavier_uniform_(w, gain=1.0)
plt.hist(xavier_u);
```

```python
w = torch.empty(1, 1000)
xavier_n = torch.nn.init.xavier_normal_(w, gain=1.0)
plt.hist(xavier_n);
```

One more improvment is **Kaiming Initialization** from [Delving Deep into Rectifiers:
Surpassing Human-Level Performance on ImageNet Classification](https://arxiv.org/pdf/1502.01852.pdf).

The only diffrence is that modern neural networks usually use Relu, or other type of non-symetric activision functions.

we won't go into the math of the paper, but the only noticeble diffrence is that we scale by  $\frac{\sqrt{2}}{\sqrt{n}}$.

* **Kaiming  Uniform** - The resulting tensor will have values sampled from $\mathcal{U}(-\text{bound}, \text{bound})$ where $$ \text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan_mode}}} $$

* **Kaiming (He) Normal** - Initialize with values sampled from normal distribution. The resulting tensor will have values sampled from $\mathcal{N}(0,\text{std}^2)$ where $$ \text{std} = \frac{\text{gain}}{\sqrt{\text{fan_mode}}} $$

    * `fan_mode` – either `fan_in` (default) or `fan_out`. Choosing `fan_in` preserves the magnitude of the variance of the weights in the forward pass. Choosing `fan_out`

and ever since, this is part of the Pytorch NN package.

```python
#`nonlinearity` – the non-linear function (`nn.functional` name), recommended to use only with `relu` or `leaky_relu` (default).
w = torch.empty(2, 1000)
k_u = torch.nn.init.kaiming_uniform_(w, a=0, mode='fan_in', nonlinearity='relu')
plt.hist(k_u);
```

```python
w = torch.empty(3, 1000)
k_n = torch.nn.init.kaiming_normal_(w, a=0, mode='fan_in', nonlinearity='relu')
plt.hist(k_n);
```

Note that initilization work for each channel seperatly

If we would like to change back to Xavier from kaiming (mostly, since there is no noticable diffrence for layers without ReLU)

```python
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Linear') != -1:
        torch.nn.init.xavier_normal_(m.weight, gain=1.0)
model.apply(weights_init)
```

    MLP(
      (fc_layers): Sequential(
        (0): Linear(in_features=784, out_features=32, bias=True)
        (1): ReLU()
        (2): Linear(in_features=32, out_features=32, bias=True)
        (3): ReLU()
        (4): Linear(in_features=32, out_features=32, bias=True)
        (5): ReLU()
        (6): Linear(in_features=32, out_features=10, bias=True)
      )
      (log_softmax): LogSoftmax(dim=1)
    )

when we use diffrent activisions, or when we do non-standart work, we might consider Xavier or something else...

besides, you might come up with the next big thing and everyone will use **Cohen Initialization** in the next years :)

## Links
---
[3b1b backprop video](https://www.youtube.com/watch?v=Ilg3gGewQ5U&ab_channel=3Blue1Brown)

[advanced loss functions](https://www.kdnuggets.com/2019/11/research-guide-advanced-loss-functions-machine-learning-models.html)

[how to choose loss functions](https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/)

[Weight Initialization in Neural Networks: A Journey From the Basics to Kaiming](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79)

## Credits
---

This tutorial heavly based on [Technion IIT cs-236781 tutorial](https://github.com/vistalab-technion/cs236781-tutorials/blob/master/t03/tutorial3-MLP.ipynb) that written by [Aviv Rosenberg](https://avivr.net) and re-weitten by [Moshe Kimhi](https://www.linkedin.com/in/moshekimhi/)

To re-use, please provide attribution and link to the original.

Some images in this tutorial were taken and/or adapted from the following sources:

- MartinThoma [CC0], via Wikimedia Commons https://commons.wikimedia.org/wiki/File:Perceptron-unit.svg
- Pattern Recognition and Machine Learning, C. M. Bishop, Springer, 2006
- Sebastian Raschka https://sebastianraschka.com/Articles/2015_singlelayer_neurons.html
- Favio Vázquez https://towardsdatascience.com/a-conversation-about-deep-learning-9a915983107
- Fundamentals of Deep Learning, Nikhil Buduma, Oreilly 2017

```python

```


## Two-dimensional advection equation

We want to solve the following PDE:

\begin{equation}
    \frac{\partial \phi}{\partial t} + u \frac{\partial \phi}{\partial x} + v \frac{\partial \phi}{\partial y} = 0
\end{equation}

The independen variables (i.e, $x$, $y$ and $t$) are used as input values for the NN, and the solution (i.e. $\phi(x,y,t)$) is the output. In order to find the solution, at each step the NN outputs are derived w.r.t the inputs. Then, a loss function that matches the PDE is built and the weights are updated accordingly. If the loss function goes to zero, we can assume that our NN is indeed the solution to our PDE. We will try to find a general solution for different values of $u$, so it will be set also as an input.

```python
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
import torch
```

First we define our PDE and set the values for training.

```python
from nangs.pde import PDE
from nangs.bocos import PeriodicBoco, DirichletBoco
from nangs.solutions import MLP

class MyPDE(PDE):
    def __init__(self, inputs=None, outputs=None, params=None):
        super().__init__(inputs, outputs, params)
    def computePDELoss(self, grads, inputs, outputs, params):
        dpdt, dpdx, dpdy = grads['p']['t'], grads['p']['x'], grads['p']['y']
        u, v = params['u'], params['v']
        return [dpdt + u*dpdx + v*dpdy]

# instanciate pde
pde = MyPDE(inputs=['x', 'y', 't'], outputs=['p'], params=['u', 'v'])

# define input values
x = np.linspace(0,1,20)
y = np.linspace(0,1,20)
t = np.linspace(0,1,10)
u = np.array([1.0])
v = np.array([1.0])

pde.setValues({'x': x, 'y': y, 't': t, 'u': u, 'v': v})
pde.setValues({'x': x, 'y': y, 't': t}, train=False)
```

Boundary conditions.

```python
# periodic b.c for the space dimension
x1, x2 = np.array([0]), np.array([1])
boco = PeriodicBoco('boco_x', {'x': x1, 'y': y, 't': t}, {'x': x2, 'y':y, 't': t})
pde.addBoco(boco)

y1, y2 = np.array([0]), np.array([1])
boco = PeriodicBoco('boco_y', {'x': x, 'y': y1, 't': t}, {'x': x, 'y':y2, 't': t})
pde.addBoco(boco)

# initial condition (dirichlet for temporal dimension)
p0 = np.zeros((len(y)*len(x)))
for i, _y in enumerate(y):
    for j, _x in enumerate(x):
        p0[i*len(x) + j] = math.sin(2.*math.pi*_x)*math.sin(2.*math.pi*_y)

boco = DirichletBoco('initial_condition', {'x': x, 'y': y, 't': np.array([0])}, {'p': p0})
pde.addBoco(boco)
```

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm

fig = plt.figure()
ax = fig.gca(projection='3d')
X, Y = np.meshgrid(x, y)
surf = ax.plot_surface(X, Y, p0.reshape((len(y),len(x))), cmap=cm.coolwarm, linewidth=0, antialiased=False)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

Now we define a topology for our solution and set the training parameters. Then we can find a solution for our PDE.

```python
# define solution topology
mlp = MLP(pde.n_inputs, pde.n_outputs, 5, 2048)
optimizer = torch.optim.Adam(mlp.parameters(), lr=3e-4)
pde.compile(mlp, optimizer)
```

```python
# find the solution
hist = pde.solve(epochs=10)
```

```python
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))
ax1.plot(hist['train_loss'], label="train_loss")
ax1.plot(hist['val_loss'], label="val_loss")
ax1.grid(True)
ax1.legend()
ax1.set_yscale("log")
for boco in pde.bocos:
    ax2.plot(hist['bocos'][boco.name], label=boco.name)
ax2.legend()
ax2.grid(True)
ax2.set_yscale("log")
plt.show()
```

Finally, we can evaluate our solution.

```python
# evaluate the solution
x = np.linspace(0,1,40)
y = np.linspace(0,1,40)
t = np.linspace(0,1,40)
u, v = np.array([1]), np.array([1])
p, p0, l2 = [], [], []
for _t in t:
    _p0 = np.zeros((len(y)*len(x)))
    for i, _y in enumerate(y):
        for j, _x in enumerate(x):
            _p0[i*len(x) + j] = math.sin(2.*math.pi*(_x-u*_t))*math.sin(2.*math.pi*(_y-v*_t))
    pde.evaluate({'x': x, 'y': y, 't': np.array([_t])})
    _p = pde.outputs['p']
    _l2 = np.sqrt(np.mean((_p - _p0)**2))
    p.append(_p)
    p0.append(_p0)
    l2.append(_l2)

from matplotlib import animation, rc
rc('animation', html='html5')

def plot(ax1, ax2, ax3, p, p0, t, l2):
    ax1.clear()
    ax2.clear()
    ax3.clear()
    tit1 = ax1.set_title(f't = {t:.2f} (NN)')
    tit2 = ax2.set_title(f't = {t:.2f} (Exact)')
    tit3 = ax3.set_title(f't = {t:.2f} (l2 {l2:5f})')
    X, Y = np.meshgrid(x, y)
    ax1.plot_surface(X, Y, p.reshape((len(y),len(x))), cmap=cm.coolwarm, linewidth=0, antialiased=False)
    ax1.set_xlim([0,1])
    ax1.set_ylim([0,1])
    ax1.set_zlim([-1,1])
    ax2.plot_surface(X, Y, p0.reshape((len(y),len(x))), cmap=cm.coolwarm,  linewidth=0, antialiased=False)
    ax2.set_xlim([0,1])
    ax2.set_ylim([0,1])
    ax2.set_zlim([-1,1])
    ax3.plot_surface(X, Y, p.reshape((len(y),len(x)))-p0.reshape((len(y),len(x))), cmap=cm.coolwarm,  linewidth=0, antialiased=False)
    ax3.set_xlim([0,1])
    ax3.set_ylim([0,1])
    ax3.set_zlim([-0.1,0.1])
    return [tit1, tit2, tit3]

def get_anim(fig, ax1, ax2, ax3, p, p0, t, l2):
    def anim(i):
        return plot(ax1, ax2, ax3, p[i], p0[i], t[i], l2[i])
    return anim

fig = plt.figure(figsize=(20,5))
ax1 = fig.add_subplot(1, 3, 1, projection='3d')
ax2 = fig.add_subplot(1, 3, 2, projection='3d')
ax3 = fig.add_subplot(1, 3, 3, projection='3d')
animate = get_anim(fig, ax1, ax2, ax3, p, p0, t, l2)
anim = animation.FuncAnimation(fig, animate, frames=len(t), interval=300, blit=True)
```

```python
anim
```

```python

```


```julia
using JuMP, Cbc, Plots     # The usual packages
using DelimitedFiles       # IO reading/writing files
using LinearAlgebra        # For convenience (includes function norm(x))
using Combinatorics        # To perform permutations
```

# MS-E2121 - Linear optimization
## Exercise session 8

### Demo exercise: Travelling salesman problem (TSP) - MTZ formulation
#### Examining the formulation

Show that the following formulation $P_{MTZ}$ is valid for the TSP defined on a directed graph $G = (N,A)$ with $N = \{1,\dots,n\}$ cities and arcs $A = \{(i,j) : i,j\in N, i\neq j\}$ between cities.
$$
P_{MTZ} = \left\{
   \begin{array}{ll}
   \displaystyle \sum_{j \in N \setminus \{i\}} x_{ij} = 1, & \forall i \in N \\
   \displaystyle \sum_{j \in N \setminus \{i\}} x_{ji} = 1, & \forall i \in N \\
   \displaystyle u_{i} - u_{j} + (n-1) x_{ij} \leq n - 2, & \forall i,j \in N \setminus \{ 1 \} : i \neq j ~~(*)\\
    x_{ij} \in \{0,1\}, & \forall i,j \in N : i\neq j\\
  \end{array}
\right.
$$

where $x_{ij} = 1$ if city $j\in N$ is visited immediately after city $i \in N$, and $x_{ij} = 0$ otherwise. Constraints $(*)$ with the variables $u_i \in \mathbb{R}$ for all $i\in N$ are called *Miller-Tucker-Zemlin* (MTZ) subtour elimination constraints.

We want to show that
1. Constraints $(*)$ prevent subtours in any solution $x \in P_{MTZ}$.
2. Every TSP solution $x$ satisfies the constraints $(*)$.

To prove the first one, we first assume that a solution $x \in P_{MTZ}$ *has* a subtour with $k$ nodes and $k$ arcs between them, not going through node 1. For example, assume nodes 2, 3 and 5 form a subtour when $n=5$. Let's write the constraints $(*)$ corresponding to this subtour:

\begin{align}
    u_2 - u_3 + 4 \le 3 \\
    u_3 - u_5 + 4 \le 3 \\
    u_5 - u_2 + 4 \le 3
\end{align}

We observe that if $x_{ij} = 1$ such that $i,j \in N \setminus 1$, the constraint $(*)$ can be written as $u_i \le u_j - 1$, which for integer variables is the same as $u_i < u_j$. For a general result, we denote the nodes in the subtour by $\{i_1, ..., i_k\}$ and get $u_{i_1} < u_{i_k} < u_{i_1}$, which is a contradiction. This tells us that there can be no subtour ($k<n$) that doesn't contain node 1. A subtour containing node 1 would imply another subtour not containing node 1, so that is also forbidden by $(*)$. This proves the first part.

For the second part, we notice that the $u$-variables seem to imply an ordering for the nodes. Assume that all tours start from node 1 and $u_i, \ i \in N \setminus 1$ is the position of the node on the tour (the first node visited after the starting node 1 has $u$-value 2, the second one 3 and so on). For each arc $i \rightarrow j$, we have either $x_{ij}=0$ or $x_{ij}=1$.

If $x_{ij} = 0$: there is no arc from $i$ to $j$, and the constraint is $u_{i} - u_{j} \leq n - 2$, which holds, since we defined that $i$ and $j$ can't be 1, and also not greater than $n$. The upper bound of the difference between two $u$-values is thus $n-2$, and the constraint always holds if $x_{ij}=0$.

If $x_{ij} = 1$: there is an arc from $i$ to $j$, and the constraint is $u_{i} - u_{j} + n-1 \leq n - 2$ or $u_{i} - u_{j} \leq -1$, which looks familiar from before and actually holds when we do not have subtours since by our definition, $u_i - u_j = -1$. The key is that this time, we have no subtours that do not contain node 1, which is treated as a special case.

To conclude, we proved that this formulation does not allow subtours, and all valid solutions $x$ satisfy the constraints. Therefore, the formulation is valid.

#### MTZ and naive TSP implementation
We first write some helper functions, starting with one that computes the distances between coordinates:

```julia
## Function for getting the distances array
function get_dist(xycoord::Matrix{},n::Int)
    # Compute distance matrix (d[i,j]) from city coordinates

    dist = zeros(n,n)
    for i = 1:n
        for j = i:n
            d = norm(xycoord[i,:] - xycoord[j,:])
            dist[i,j] = d
            dist[j,i] = d
        end
    end
    return dist
end
```

A function to convert the adjacency matrix $x$ describing a tour to a vector representing the tour, starting from city 1.

```julia
# Get the optimal tour
# Input
#     x: solution matrix
#     n: number of cities
# Returns
#     tour: ordering of cities in the optimal tour
function gettour(x::Matrix{Int}, n::Int)
    tour = zeros(Int,n+1)   # Initialize tour vector (n+1 as city 1 appears twice)
    tour[1] = 1             # Set city 1 as first one in the tour
    k = 2                   # Index of vector tour[k]
    i = 1                   # Index of current city
    while k <= n + 1        # Find all n+1 tour nodes (city 1 is counted twice)
        for j = 1:n
            if x[i,j] == 1  # Find next city j visited immediately after i
                tour[k] = j # Set city j as the k:th city in the tour
                k = k + 1   # Update index k of tour[] vector
                i = j       # Move to next city
                break
            end
        end
    end
    return tour             # Return the optimal tour
end
```

```julia
## Defining the colors to be used
c_blue = palette(:auto)[1]                         # color :1
c_orange = palette(:auto)[2]                       # color :2
c_green = palette(:auto)[3];                       # color :3
```

The data is stored in csv-files, a common external data format. We use the DelimitedFiles package to read a file into a matrix ```data```.

```julia
# "data16a.csv" has 3 columns which are stored in (Nullable) Arrays
# data[1], data[2], data[3] after the function call
# data = CSV.read(...) below. The columns contain:
#
#     data[:,1]: all cities i in V
#     data[:,2]: x-coordinate of each city i in V
#     data[:,3]: y-coordinate of each city i in V
#

data  = readdlm("data16a.csv", ',')
# data  = readdlm("data16b.csv", ',')
n = 16                  # number of cities
# println(data)         # Look at the data in compact form
V = data[2:n+1,1]       # All cities i in V
x = data[2:n+1,2]       # x-coordinates of cities i in V
y = data[2:n+1,3]       # y-coordinates of cities i in V
xycoord = [x y];        # n x 2 coordinate matrix
```

```julia
function tsp_naive(xycoord::Matrix{}, n::Int)

    # Create a model
    m = Model(Cbc.Optimizer)

    # Here the costs c are the distances between cities
    c = get_dist(xycoord,n)

    ## Variables

    # x[i,j] = 1 if we travel from city i to city j, 0 otherwise.
    @variable(m, x[1:n,1:n], Bin)

    ## Objective

    # Minimize length of tour
    @objective(m, Min, dot(c,x))

    ## Constraints

    # Ignore self arcs: set x[i,i] = 0
    @constraint(m, sar[i = 1:n], x[i,i] == 0)

    # We must enter and leave every city exactly once
    @constraint(m, ji[i = 1:n], sum(x[j,i] for j = 1:n if j != i) == 1)
    @constraint(m, ij[i = 1:n], sum(x[i,j] for j = 1:n if j != i) == 1)

    optimize!(m)

    cost = objective_value(m)         # Optimal cost (length)
    sol_x = round.(Int, value.(x))    # Optimal solution vector

    return m, sol_x, cost
end;
```

```julia
## Solve the problem and evaluate time and memory with @time macro
(m_naive, x_naive, cost_naive) = @time tsp_naive(xycoord, n);

# Get the optimal tour
tour_naive = gettour(x_naive,n);
```

```julia
plt = scatter(xycoord[:,1],xycoord[:,2],
      markercolor = c_blue,
      markerstrokewidth = 0,
      legend = false
)
for i in 1:length(tour_naive)-1
    annotate!(xycoord[tour_naive[i],1]+50, xycoord[tour_naive[i],2]+50, ("$(tour_naive[i])", 7, :left))
    plot!(([xycoord[tour_naive[i],1],xycoord[tour_naive[i+1],1]] , [xycoord[tour_naive[i],2],xycoord[tour_naive[i+1],2]]), c = c_blue, label = "")
end
plt
```

```julia
plt = scatter(xycoord[:,1],xycoord[:,2],
      markercolor = c_blue,
      markerstrokewidth = 0,
      legend = false
)
for i in 1:n
    annotate!(xycoord[i,1]+50, xycoord[i,2]+50, ("$(i)", 7, :left))
    for j in 1:n
        if x_naive[i,j] == 1
            plot!(([xycoord[i,1],xycoord[j,1]] , [xycoord[i,2],xycoord[j,2]]), c = c_blue, label = "")
            break
        end
    end
end
plt
```

```julia
# Solve a directed, TSP instance (MTZ formulation)
# Input
#    xycoord: coordinates of city locations
#    n: number of cities
# Returns
#    tour: ordering of cities in the optimal tour
#    cost: cost (length) of the optimal tour
function tsp_mtz(xycoord::Matrix{}, n::Int)

    # Create a model
    m = Model(Cbc.Optimizer)

    # Here the costs c are the distances between cities
    c = get_dist(xycoord,n)

    ## Variables

    # x[i,j] = 1 if we travel from city i to city j, 0 otherwise.
    @variable(m, x[1:n,1:n], Bin)
    # Variables u for subtour elimination constraints
    @variable(m, u[2:n])

    ## Objective

    # Minimize length of tour
    @objective(m, Min, dot(c,x))

    ## Constraints

    # Ignore self arcs: set x[i,i] = 0
    @constraint(m, sar[i = 1:n], x[i,i] == 0)

    # We must enter and leave every city exactly once
    @constraint(m, ji[i = 1:n], sum(x[j,i] for j = 1:n if j != i) == 1)
    @constraint(m, ij[i = 1:n], sum(x[i,j] for j = 1:n if j != i) == 1)

    # MTZ subtour elimination constraints
    @constraint(m, sub[i = 2:n, j = 2:n, i != j], u[i] - u[j] + (n-1)*x[i,j] <= (n-2))

    optimize!(m)

    cost = objective_value(m)         # Optimal cost (length)
    sol_x = round.(Int, value.(x))    # Optimal solution vector

    return m, sol_x, cost
end;
```

```julia
## Solve the problem and evaluate time and memory with @time macro
(m_mtz, x_mtz, cost_mtz) = @time tsp_mtz(xycoord, n);

# Get the optimal tour
tour_mtz = gettour(x_mtz,n);
```

```julia
## Print the optimal tour and its cost
println("\nOptimal tour: $(tour_mtz)\n")
println("Optimal length: ", cost_mtz)
```

```julia
plt = scatter(xycoord[:,1],xycoord[:,2],
      markercolor = c_blue,
      markerstrokewidth = 0,
      legend = false
)
for i in 1:length(tour_mtz)-1
    annotate!(xycoord[tour_mtz[i],1]+50, xycoord[tour_mtz[i],2]+50, ("$(tour_mtz[i])", 7, :left))
    plot!(([xycoord[tour_mtz[i],1],xycoord[tour_mtz[i+1],1]] , [xycoord[tour_mtz[i],2],xycoord[tour_mtz[i+1],2]]), c = c_blue, label = "")
end
plt
```

Now that we have solved the problem using the MTZ formulation, let's try solving the same problem starting with the naive implementation and using successive cutset or subtour elimination constraints:

```julia
## Initialisation
(m_naive, x_naive, cost_naive) = tsp_naive(xycoord, n);
subnodes = []
count = 0
stop = 0
lim = 100
methods = [:elimination,:cutset]
method = methods[2];
```

```julia
## Perform cuts to break subtours until we got an optimal
@time while stop == 0 && count < lim

    S = collect(permutations(subnodes,2))     # Possible connections present in the naive implementation
    NS = setdiff(V,subnodes)                  # Nodes that are still not included in the tour

    if method == :cutset
        ## Cutset constraints
        if length(S) > 0
            @constraint(m_naive,sum(m_naive[:x][subnodes[i],NS[j]] for i in 1:length(subnodes), j in 1:length(NS)) >= 1)
        end
    else
        ## Subtour elimination constraints
        if length(S) > 0
            @constraint(m_naive,sum(m_naive[:x][S[i][1],S[i][2]] for i in 1:length(S)) <= length(subnodes)-1)
        end
    end

    set_silent(m_naive)
    optimize!(m_naive)

    cost2 = objective_value(m_naive)          # Optimal cost (length)
    sol_x = round.(Int, value.(m_naive[:x]))     # Optimal solution vector

    tour2 = gettour(sol_x,n)            # Get the optimal tour

    if length(unique(tour2)) < n
        count = count + 1
        println("Method used: ",method)
        println("Subtours present in node 1, update subnodes vector to break the subtour: ", tour2')
        println("\nIteration $(count); not optimal.\n")
        subnodes = unique(tour2);
    else
        println("Method used: ",method)
        println("Optimal tour: ", tour2')
        println("\n Took $(count) iterations to find the optimal solution.")
        S = []
        stop = 1
    end;
end;
```

### Scheduling problem
TODO


$
\newcommand{\ket}[1]{\lvert {#1}\rangle}
\newcommand{\ds}{\displaystyle}
$

# Shor's Algorithm

## Introduction
Problem
> Integer factorization

Use Case
> $n = p \cdot q$, where $p,q$ are large prime numbers, often used in cryptographic functions

Classical Solution
> **General Number Field Sieve** has best asymptotic behaviour
> Superpolynomial scaling:
> $\displaystyle O(e^{(\ln n)^\frac{1}{3}(\ln\ln n)^{\frac{2}{3}}})$

Shor's Algorthim
> Factorization algorithm with poylnomial complexity

> Partially on Quantum processor, pre & post processing is classical
> Quantum complexity: $O\big( (\log n)^2 (\log \log n) (\log \log \log n) \big)$
> Polynomial time achieved with Quantum Forier Transform

----

## Algorithm

### 1. Preconditions of $n$

Exit if $n$ is any of the following
- even
- prime
- prime power

### 2. Attempt brute-force solution

Pick a random integer $x \lt n$, and calculate $\gcd(x, n)$.

If this is not 1, then we have a factor of $n$.

### 3. Quantum Algorithm

1. Pick q as the smallest power of 2 where $n^2 \leq q \leq 2n^2$

2. Find the period $r$ of $x^a \mod n$

3. Measure $c$ which has property $\displaystyle \frac{c}{q} \approx \frac{d}{r}$, where $d \in \mathbb{N}$

### 4. Determine ratio

$r, d$ can be determined if $gcd(d, r) = 1$ using fraction expansion algorithm

### 5. Try to obtain factors

Return to step 2
> if $r$ is odd

> if $x^{\frac{r}{2}} \equiv -1 \mod n$

Factors are
>$\displaystyle p, q = \gcd(x^{\frac{r}{2}} \pm 1, n)$

----

### Quantum Fourier Transform (QFT)

Define QFT with respect to an ONB $\left\{\ket{x}\right\} = {\ket{0}, \dots, \ket{q-1}}$
> $\ds QFT: \ket{x} \mapsto \frac{1}{\sqrt{q}} \sum\limits_{y=0}^{q-1} \exp \left\{ \frac{2\pi i}{q}x \cdot y \right\} \ket{y} = \frac{1}{\sqrt{q}} \sum\limits_{y=0}^{q-1} \omega^{x \cdot y} \ket{y}$

Apply QFT to a general state $\ds\ket{\psi} = \sum_x \alpha_x\ket{x}$ :
> $\ds QFT(\ket{\psi}) = \frac{1}{\sqrt{q}} \sum\limits_{y=0}^{q-1} \beta_y\ket{y}$

> where the $\beta_y$'s are the discrete Fourier tansform of the amplitudes $\alpha_x$

The QFT is unitary
> $QFT^\dagger QFT\ket{x} = \ket{x}$

----

## References

- [Lecture Notes - Shor's Algorithm][1]
- [Continued Fraction Expansion][2]

 [1]: https://qudev.phys.ethz.ch/content/QSIT15/Shors%20Algorithm.pdf
 [2]: https://www.math.u-bordeaux.fr/~pjaming/M1/exposes/MA2.pdf

----

## Addendum I - Euclidean Algorithm

$
\gcd(a,b) =
\Bigg\{\begin{split}
&b  & \text{ if } a \bmod b = 0 \\
&\gcd(b, a \bmod b) & \text{ else } \\
\end{split}
$

### References

- [Euclidean Algorithm][1]

 [1]: https://www.khanacademy.org/computing/computer-science/cryptography/modarithmetic/a/the-euclidean-algorithm

```python
# Euclidian Algorithm
def gcd(a, b):
    h,l = (max(a,b),min(a,b))
    q = h % l
    if q == 0:
        return l
    return gcd(l,q)

def is_coprime(a, b):
    return (gcd(a, b) == 1)

```

----
$\newcommand{\ds}{\displaystyle}$
## Addendum II - Continued Fractions

Tools to approximate a Real number to a Rational number.

### Example 1 - 1 and 2

'1' can be written as a fraction
> $\ds1 = \frac{2}{3-1}$

and as a continued/infinite fraction
> $\ds1 = \frac{2}{3-\frac{2}{3-\frac{2}{3-\dots}}}$

interestingly, using 2 instead would suggest the contradiction $1=2$
> $\ds2 = \frac{2}{3-\frac{2}{3-\frac{2}{3-\dots}}}$

### Example 2 - $\sqrt{2}$

> $
\begin{split}
\sqrt{2} &= 1.414213  \\
&= 1 + 0.41421  \\
&= 1 + \frac{1}{\frac{1}{0.41421 }} \\
&= 1 + \frac{1}{2+ 0.41421 } \\
&= 1 + \frac{1}{2+ \frac{1}{2 + \frac{1}{2 + \dots}}} \\
\end{split}
$

> Sequence `11111111...`

### Example 3 - $\phi$ (Golden Ratio)
> $
\begin{split}
\phi &= \frac{1 + \sqrt{5}}{2} \\
&= 1 + \frac{1}{1+ \frac{1}{1 + \frac{1}{1 + \dots}}}
\end{split}
$

> Sequence `11111111...`

### Example 4 - $e$ (2.718)
> $
\begin{split}
e &= \frac{1 + \sqrt{5}}{2} \\
&= 2 + \frac{1}{1+ \frac{1}{2 + \frac{1}{1 + \frac{1}{1 + \frac{1}{4 + \dots}}}}}
\end{split}
$

> Sequence - `2114116118...`

### Example 5 - Rational $\frac{1473}{50}$
*Rational numbers tend to be finite*
> $29 + \frac{1}{2 + \frac{1}{5 + \frac{1}{1 + \frac{1}{3}}}}$

### Example 5 -$\pi$
> $
\begin{split}
\pi &= 3.14159265358979 \\
&= 3 + \frac{1}{7+ \frac{1}{15 + \frac{1}{1 + \frac{1}{292 + \dots}}}} \\
3 &= 3.0 \\
3 + \frac{1}{7} = \frac{22}{7} &= 3.14 \\
3 + \frac{1}{7+\frac{1}{15}} = \frac{333}{106} &= 3.1415 \\
3 + \frac{1}{7+\frac{1}{15 + 1}} = \frac{355}{113} &= 3.141592 \\
3 + \frac{1}{7+\frac{1}{15 + \frac{1}{1 + \frac{1}{292}}}} = \frac{103993}{33102} &= 3.14159265 \\
\end{split}
$

----

## Addendum III - Example, Factoring n=21

### 1. Choose a random integer $x$, where $1 < x < n$

> if x *is **not** coprime* with n (i.e. x=6)

> $\gcd(x,n) = \gcd(6, 21) = 3$

> $\therefore \frac{21}{3} = 7$

> The factors are 7 and 3

> if x *is coprime* with n (i.e. x=11)

> $\gcd(11,21) = 1$

> then continue...

### 2. Determine q

> Where q is the smallest power of 2 between $n^2$ and $2n^2$

> $441 \leq q < 882 \land \log_2(q) = l : l \in \mathbb{N}$

> $l=9, q=512$

> Initial state for 2 registers with length of 'l'

> $\ket{\Phi_i} = \ket{0}_{r_1} \ket{0}_{r_2} = \ket{0}^{\otimes 2^l}$

### 3. Initialize first register ($r_1$)

> Initialize first register with superposition of all stats $a \pmod{q}$
> $\ds\ket{\Phi_0} = \frac{1}{\sqrt{512}} \sum\limits_{a=0}^{511} \ket{a}\ket{0} $

> therefore all bits are $\frac{1}{\sqrt{2}}\Big(\ket{0} + \ket{1}\Big)$

### 4. Initialize second register ($r_2$)

> Initialize Register with superposition of all states $x^a \pmod{n}$
> $\ds
\begin{split}
\ket{\Phi_1} &= \frac{1}{\sqrt{512}} \sum\limits_{a=0}^{511} \ket{a}\ket{11^a(\bmod{21})} \\
&= \frac{1}{\sqrt{512}} \Big( \ket{0}\ket{1} + \ket{1}\ket{11} + \ket{2}\ket{16} + \ket{3}\ket{8} \dots \Big)
\end{split}$

> The sequence looks like this
>$
\begin{matrix}
0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & \dots \\
1 & 11 & 16 & 8 & 4 & 2 & 1 & 11 & 16 & 8 & 4 & \dots \\
\end{matrix}
$

> *The period is 6, but not yet observable*

### 5. Quantum Fourier Transform
> $\ds \overset{\sim}{\ket{\Phi}} = \frac{1}{512} \sum\limits_{a=0}^{511} \sum\limits_{c=0}^{511} e^{\big(\frac{1}{512} 2\pi i a c\big)} \ket{c}\ket{11^a(\bmod{21})}$

### 6. Measurement
> Probability for state $\ket{c, x^k\pmod{n}}$ e.g. $k=2 \rightarrow \ket{c,16}$

> $\ds
\begin{split}
p(c) &= \Bigg| {\frac{1}{512}}_{a:11^a} \sum\limits_{\bmod{21}=16}^{511} e^{\big(\frac{1}{512} 2\pi iac\big)} \Bigg|^2 \\
&= \Bigg| {\frac{1}{512}}_{a:11^a} \sum\limits_{b} e^{\big(\frac{1}{512} 2\pi i(6b+2)c\big)} \Bigg|^2\\
\end{split}$

>
> Simulation of the probability amplitude of a period of 6

```python
PERIOD=6

import sympy as sm
import numpy as np
import matplotlib.pyplot as plt

fig = plt.gcf()
fig.set_size_inches(8,5)

sm.var('x')
# This function is a simulation, it does not
# reflect a multi-modal normal distribution
f = lambda x: ((sm.cos((x*PERIOD*2*sm.pi)/512)+1)/2)**80/PERIOD

x = np.linspace(0,512,1000)
y = np.array([f(v) for v in x],dtype='float')

plt.title(r"Probaility Amplitudes with a period of 6 peaks fro $c =\frac{512}{6} \cdot d, d \in \mathbb{Z}$")
plt.plot(x,y,color='gray')
plt.fill_between(x,y,0,color='#c0f0c0')

plt.show()
```

### 7. Determine the period
Continued Fraction Expansion (determine r)
> $\ds \Bigg| \frac{c}{q} - \frac{d}{r} \Bigg| = \overset{!}{\leq} \frac{1}{1024}$

> Use continued fraction expansion

> $\ds
\begin{split}
\frac{c}{q} &= a_0 + \frac{1}{a_1 + \frac{1}{a_2 + \dots}}  \\
\tfrac{d_0}{r_0} &= \tfrac{a_0}{1} \\
\tfrac{d_1}{r_1} &= \tfrac{1+a_0a_1}{a_1} \\
\tfrac{d_n}{r_n} &= \tfrac{a_nd_{n-1} + d_{n-2}}{a_nr_{n-1} + r_{n-2}} \\
&\dots \\
\end{split}
$

Assume the reading was **427**

> So

> e = $\ds \Bigg| \frac{427}{512} - \frac{d}{r} \Bigg| \overset{!}{\leq} \frac{1}{1024}$

> Use continued fraction expansion

> $\ds
\begin{split}
\frac{427}{512} &= 0.83398\dots \\
&= 0 + \frac{1}{1 + \frac{1}{5 + \frac{1}{42 + \frac{1}{2}}}} \\
\tfrac{d_0}{r_0} &= \tfrac{0}{1} = 0 \;\;\; &e = \tfrac{427}{512} > \tfrac{1}{1204} \text{  no match}\\
\tfrac{d_1}{r_1} &= \tfrac{1}{1} = 1 \;\;\; &e = \tfrac{85}{512} > \tfrac{1}{1204} \text{  no match}\\
\tfrac{d_2}{r_2} &= \tfrac{5}{6} = 0.8\dot{3} \;\;\; &e = \tfrac{2}{3072} < \tfrac{1}{1204} \text{  match!}\\
\tfrac{d_3}{r_3} &= \tfrac{427}{512} = 0.83398\dots \;\;\; &e = 0 \\
\end{split}
$

> $\frac{5}{6}$ is a match,   $d(427)$ and $r(6)$ are coprime,   therefore the period **r = 6**

Assume the reading was **171**

> we would get $\frac{d}{r} = \frac{1}{3}$,   $d(171)$ and $r(3)$ are *not* coprime, therefore this does not work, try again.

### 8. Check r (determine factors)

Check r is even
> $6 \pmod{2} = 0$

Check if $x^{\tfrac{r}{2}} \pmod{n} \ne -1$
> $11^{\tfrac{6}{2}} \pmod{21} = 11^3 \pmod{21} = 1331 \pmod{21} = 8 \ne 1330$

Calculate factors via $x^{\tfrac{r}{2}}\pmod{n} \pm 1$
> $1331 \pmod{21} - 1 = 7$

> $1331 \pmod{21} + 1 = 9$

```python

```


**Exercise set 11**
==============

>In connection with experimental design, we have seen two approaches
>for checking if determined effects are important or not. These two
>approaches are based on creating a probability plot and performing ANOVA.
>The goal of this exercise is to learn how we use these two approaches in
>practice.

**Exercise 11.1**

In this part of the exercise, we will deal with the technical aspects of
creating a normal probability plot. Our final aim is here
to check if the data for some measured quantities (given in
the data files: `Data/data1.txt`, `Data/data2.txt`,
`Data/data3.txt`, and `Data/data4.txt`) comes
from a normal distribution.

**(a)**  Before we begin creating normal probability plots, we should
inspect the raw data. Plot histograms for the raw data. Based
on this, would you say that any of the data files contain numbers
that might come from a normal distribution?
You can also, for each data set, try to compare with
a normal distribution (using the mean and standard deviation
you can obtain from the raw data).

```python
# Your code here
```

**Your answer to question 11.1(a):** *Double click here*

**(b)**  For creating the normal probability plot, we need to be able
to obtain certain parameters for the standard normal distribution.
Here, we will investigate some of the functions that can give us
such parameters.

The standard normal probability density function ($\operatorname{PDF}$) is given by,

$\operatorname{PDF}(x) = \frac{1}{\sqrt{2 \pi}} \operatorname{e}^{-\tfrac{x^2}{2}},$

and the cumulative distribution function (CDF) is,

$
\operatorname{CDF}(x) = \int_{-\infty}^{x} \operatorname{PDF} (t)\,\operatorname{d}t =
\frac{1}{2} \left[ 1 + \operatorname{erf} \left( \frac{x}{\sqrt{2}} \right) \right],
$

where $\operatorname{erf}(\ldots)$ is the error function.
The cumulative distribution function gives the probability of observing a
value less than or equal to $x$: $P(X \leq x) = \operatorname{CDF}(x)$.

We can also turn this equation around: Given a probability $P$ what is
the value of $x$ that gives this probability? To answer this question,
we need the quantile function (also known as the percent-point function),
which is the inverse of the cumulative distribution function.
For the standard normal distribution, the percent-point
function ($\operatorname{PPF}$),
is given by,

$
\operatorname{PPF}(P) = \sqrt{2} \operatorname{erf}^{-1}(2P - 1).
$

If we make use of the `scipy` package, all these functions
are available to us:
```python
from scipy.stats import norm
import numpy as np

x = np.linspace(-2, 2, 100)
pdf = norm.pdf(x)  # Probability density function.
cdf = norm.cdf(x)  # Cumulative probability density function.
ppf = norm.ppf(x)  # Quantile function/percent-point function.
```

Use these methods to answer the following questions for the standard normal distribution:

* (i)  What is the probability of observing a $x \leq 1$?

* (ii)  What is the probability of observing a $x \leq 0$?

* (iii)  What is the probability of observing a $x \leq -2$?

* (iv)  Given that the probability of observing $x \leq \alpha$ is
$10$\%, what is $\alpha$?

* (v)  Given that the probability of observing $x \leq \alpha$ is
$90$\%, what is $\alpha$?

* (vi)  Given that the probability of observing $x \leq \alpha$ is
$99$\%, what is $\alpha$?

```python
# Your code here
```

**Your answer to question 11.1(b):** *Double click here*

**(c)**  To construct the normal probability plot, we will make use of
the $\operatorname{PPF}$.
If the data we are to investigate contain $n$ points, then
we need to figure out how these $n$ points would be placed
in the distribution we are going to compare with (here: the standard normal distribution).
In the following, we will call the $n$ points we have measured
for $y_1$, $y_2$, $\ldots$, $y_n$, and we assume that we have
sorted them so that $y_1 \leq y_2 \leq \ldots \leq y_n$.

We now need to check
how $n$ points drawn from a normal distribution would
be distributed, and compare this with how our measured data is distributed.
The simplest approach would be
(as we did in lecture $11$) to find the locations in the
normal distribution where the probabilities are $\tfrac{1}{n}$, $\tfrac{2}{n}$,
$\ldots$, $\tfrac{n}{n}$. This we can obtain using the $\operatorname{PPF}$.
If we call these locations $x_1$, $x_2$, $\ldots$,
$x_n$, then we expect to have seen
one data point ($y_1$) for $x \leq x_1$,
two data points ($y_1$ and $y_2$) for $x \leq x_2$, $\ldots$,
and $n$ points for $x \leq x_n$.
So far so good, but we have two complicating factors:
(i) $x_i$ is *not* necessarily the most *probable* location
for the $i$'th smallest point, and (ii) how do we deal with $x_n = \infty$?

To solve these two potential problems, we have to rely on some theoretical
results from statistics. The correct $x_i$'s we are looking for are actually
something called the order statistic medians and these are exactly
related to order statistic medians from a *uniform distribution*, $m_i$,
\begin{equation}
x_i = \operatorname{PPF}(m_i) .
\label{eq:orderstat}
\tag{1}\end{equation}

Thus we can potentially find $x_i$ by first obtaining the corresponding
$m_i$. Unfortunately, no analytical expression for $m_i$ exist and
we have to rely on approximate estimates. One such approximation
was
suggested by [Filliben](https://doi.org/10.1080/00401706.1975.10489279),

\begin{equation}
m_i =
\begin{cases}
1 - 0.5^{1/n} & \text{if } i = 1, \\
\frac{i - 0.3175}{n + 0.365} & \text{if } i = 2, 3, \ldots, n-1, \\
0.5^{1/n} & \text{if } i=n,
\end{cases}
\label{eq:uniformorderstat}
\tag{2}
\end{equation}

Thus, in summary, to create the normal probability plot we do the following:

* (i)  We sort our original data.

* (ii)  For each sorted data point, we calculate its
uniform statistic median, $m_i$,
using Eq. \eqref{eq:orderstat}.

* (iii)  For each sorted data point, we calculate its most
probable location, $x_i$, in a normal distribution using
Eq. \eqref{eq:uniformorderstat} and
the $m_i$ value we found in the previous step.

* (iv)  We plot the sorted data against the most probable locations
found in the previous step. That is, we plot the pairs ($x_i$, $y_i$).

Create the normal probability plots for the four data sets given
in `Data/data1.txt`, `Data/data2.txt`,
`Data/data3.txt`, and `Data/data4.txt`. Which
of these would you say are numbers that could originate from a
normal distribution?

```python
# Your code here
```

**Your answer to question 11.1(c):** *Double click here*

**(d)**  The method we have described above works for any distribution,
not just the normal distribution. We can create similar plots
for other distributions by changing the $\operatorname{PPF}$ function
to the corresponding function for the distribution we wish to check for.
Repeat the previous step, but use the `Gumbel distribution`
(in `scipy` this is available by
`from scipy.stats import gumbel_r`) in place of the
normal distribution. Based on the plots you create, would you say
that any of the data sets may contain numbers from a Gumbel distribution?

```python
# Your code here
```

**Your answer to question 11.1(d):** *Double click here*

**Exercise 11.2**

After running a set of experiments, you determine the effects
given in Table 1 for $4$ factors: A, B, C, and D.
Use a normal probability plot to identify the important effects among
the ones listed in this table.

|**Factor** | **Effect** |
|:---:|---:|
|A    | $-8.00$ |
|B    | $24.00$ |
|C    | $-2.25$ |
|D    | $-5.50$ |
|AB   | $ 1.00$ |
|AC   | $ 0.75$ |
|AD   | $ 0.00$ |
|BC   | $-1.25$ |
|BD   | $ 4.50$ |
|CD   | $-0.25$ |
|ABC  | $-0.75$ |
|ABD  | $ 0.50$ |
|ACD  | $-0.25$ |
|BCD  | $-0.75$ |
|ABCD | $-0.25$ |

| |
|---|
|**Table 1:** *Effects determined in a set of experiments.*|

```python
# Your code here
```

**Your answer to question 11.2:** *Double click here*

**Exercise 11.3**

The fertilizer magnesium ammonium phosphate MgNH$_4$PO$_4$ is an effective
supplier for nutrients necessary for plant growth. In an experiment, you have
tested the effect of this fertilizer on the growth of Chrysanthemums by
measuring the height of the plants after growing them for four weeks.
You have considered $4$ different concentrations of the fertilizer (measured in g/bu)
and you have measured the mean height by growing $10$ plants per concentration.
The measured data is given in Table 2, and is
also given in the file `Data/fertilizer.txt`.

|**50 g/bu** | **100 g/bu** | **200 g/bu** | **400 g/bu** |
|:---:|:---:|:---:|:---:|
|$13.2$ | $16.0$ | $ 7.8$ | $21.0$ |
|$12.4$ | $12.6$ | $14.4$ | $14.8$ |
|$12.8$ | $14.8$ | $20.0$ | $19.1$ |
|$17.2$ | $13.0$ | $15.8$ | $15.8$ |
|$13.0$ | $14.0$ | $17.0$ | $18.0$ |
|$14.0$ | $23.6$ | $27.0$ | $26.0$ |
|$14.2$ | $14.0$ | $19.6$ | $21.1$ |
|$21.6$ | $17.0$ | $18.0$ | $22.0$ |
|$15.0$ | $22.2$ | $20.2$ | $25.0$ |
|$20.0$ | $24.4$ | $23.2$ | $18.2$ |

| |
|---|
|**Table 2:** *Measured plant heights (in cm) as a function of the fertilizer concentration (in g/bu).*|

**(a)**  Here, we will test the hypothesis that the mean height
of the plants is not affected by the amount of fertilizer. We
are going to test this with a specified significance level, $\alpha$.
What is the meaning of $\alpha$ in connection with a hypothesis test?

```python
# Your code here
```

**Your answer to question 11.3(a):** *Double click here*

**(b)**  Based on this data, can you conclude on the $0.05$ level of significance that
different concentrations of the fertilizer affect the mean attained height
of the plants? What concentration, if any, appears to give the tallest plants?

```python
# Your code here
```

**Your answer to question 11.3(b):** *Double click here*

**(c)**  Would your conclusion change with a significance level of $0.10$?

```python
# Your code here
```

**Your answer to question 11.3(c):** *Double click here*

The relevant critical values for the $f$ distribution with $3$ and $36$ degrees of
freedom  are: $f_{\alpha = 0.05}(3, 36) = 2.866$ and $f_{\alpha=0.10}(3, 36) = 2.243$.

**Exercise 11.4**

From a $2^2$ factorial experiment replicated three times you have obtained
the data given in Table 3. We use here a short-hand notation
for the $4$ possible combinations of the variables: $(1)$, $a$, $b$, and $ab$.
In this notation $(1)$ is the experiment where all factors are at their low level. For the
other cases, the absence of a letter means that the corresponding factor was at a low level, and
the presence of a letter means that the corresponding factor was at a high level.

|**Experiment** | **Replicate 1** | **Replicate 2** | **Replicate 2** |
|:---:|:---:|:---:|:---:|
|$(1)$ | $9$  | $10$ | $11$ |
|$a$   | $30$ | $31$ | $29$ |
|$b$   | $19$ | $20$ | $21$ |
|$ab$  | $5$  | $6$  | $4$  |

| |
|---|
|**Table 3:** *Results from a $2^2$ factorial experiment, repeated $3$ times.*|

**(a)**  Calculate the effects. Here you should average the effects over the
replicates.

```python
# Your code here
```

**Your answer to question 11.4(a):** *Double click here*

**(b)**  Use ANOVA to investigate which effects are important in this case. For
a significance level of $\alpha = 0.01$, the relevant critical
$f$-value is $f_{\alpha=0.01}(1, 8) = 11.259$ with $1$ and $8$ degrees
of freedom.

```python
# Your code here
```

**Your answer to question 11.4(b):** *Double click here*