```python
# El amplificador puente es como se observa en la imagen.
import numpy as np
import sympy as sym
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from IPython.display import Image
sym.init_printing()
#%matplotlib widget
%matplotlib inline
Image(filename='amp_puente.png',width=300)
```

```python
# VERIFICAR QUE Vo RESPONDE A LA SIGUIENTE ECUACION
sym.var('Va, Vb, Vc, Vd, Vo, Vp')
sym.var('R1, R2, R3, R4, R5, R6')
sym.var('Vo_')
display(sym.Eq(Vo_,sym.fu(((1+R6/R5)*(R2/(R1+R2)-R4/(R3+R4))*Vp))))
Vo_=sym.fu(((1+R6/R5)*(R2/(R1+R2)-R4/(R3+R4))*Vp))
```

```python
find=sym.Matrix(([Va],[Vb],[Vc],[Vd],[Vo])) #Incognitas
#Se escriben tantas ecuacionenes como nodos haya
ec_nodo_0=sym.Eq(Vd,0)
ec_nodo_1=sym.Eq(Vb-Vc,Vp)
ec_nodo_2=sym.Eq((Vb-Vd)/R3+(Vc-Vd)/R4,0)
ec_nodo_3=sym.Eq(Va/R5+(Va-Vo)/R6,0)
ec_nodo_4=sym.Eq((Vb-Va)/R1+(Vb-Vd)/R3,(Va-Vc)/R2+(Vd-Vc)/R4)#Caso especial de superNodo
display(sym.Eq(Vo,sym.factor(sym.solve([ec_nodo_0,ec_nodo_1,ec_nodo_2,ec_nodo_3,ec_nodo_4],find)[Vo])))
Vo=sym.simplify(sym.factor(sym.solve([ec_nodo_0,ec_nodo_1,ec_nodo_2,ec_nodo_3,ec_nodo_4],find)[Vo]))
```

```python
print('Se valida la ecuación?',np.invert(np.bool_(sym.simplify(Vo_-Vo))))
sym.simplify(Vo_-Vo)
```

```python
sym.var('Av,R, D, Vo_calc') # Si Av es la ganancia Av=(1+R6/R5)   R1=R-D (Contrae) R2=R+D R1/R2=R4/R3

display(sym.Eq(Vo_calc,sym.simplify((Vo.subs({(R1,R-D),(R2,R+D),(R3,R+D),(R4,R-D),(R6,(Av-1)*R5)})))))
Vo_calc=sym.simplify((Vo.subs({(R1,R-D),(R2,R+D),(R3,R+D),(R4,R-D),(R6,(Av-1)*R5)})))
```


```python
import sympy as sm
import matplotlib.pyplot as plt
```

1. Write down a 2-D function, like z= f(x,y), it may be polynomials, sinusoidal, exponential, logarithm ….

```python
x, y = sm.symbols('x, y')
z = x**2 + y**2
```

2. Plot the 3-D figure , or 2-D figure (like an image)

```python
%matplotlib inline
fig = sm.plotting.plot3d(z, x, y)
```

3. Find the maximal, minimal point and circle them

```python
img = plt.imread('hw2_2.jpg')
fig = plt.imshow(img)
```

4. Find the maximal and minimal point by solving

```python
dx = z.diff(x)
dx
```

$\displaystyle 2 x$

```python
dy = z.diff(y)
dy
```

$\displaystyle 2 y$

```python
sm.solve([dx, dy], x, y, dict=True)
```

    [{x: 0, y: 0}]

### Read the following blog post and practice through it and then write down a notebook and codes for presentation

#### Case study 1: finding the general form of a function and its inverse

What’s the inverse of this little quadratic function? We could find it by hand, but let’s just fire up SymPy:

```python
import sympy

# create a symbolic variable for each symbol in our equation
y, x, k = sympy.symbols('y, x, k', real=True)

# define the equation y = kx - (1-k)x^2
fwd_equation = sympy.Eq(y, k*x - (k - 1)*x**2)

# solve the equation for x and print solutions
inverse = sympy.solve(fwd_equation, x)
print('found {} solutions for x:'.format(len(inverse)))
print('\n'.join([str(s) for s in inverse]))
```

    found 2 solutions for x:
    (k - sqrt(k**2 - 4*k*y + 4*y))/(2*(k - 1))
    (k + sqrt(k**2 - 4*k*y + 4*y))/(2*(k - 1))

Let’s see how that first solution looks when we substitute in k=1.5 or k=1.375:

```python
print(inverse[0].subs(k, 1.5).simplify())
print(inverse[0].subs(k, 1.375).simplify())
```

    1.5 - 1.5*sqrt(1 - 0.888888888888889*y)
    1.83333333333333 - 1.83333333333333*sqrt(1 - 0.793388429752066*y)

Let’s conclude this example with a nice trick for any users of LaTeX or MathJax: we’ll ask SymPy to typeset the inverse formula for us.

```python
print('x =', sympy.latex(inverse[0]))
```

    x = \frac{k - \sqrt{k^{2} - 4 k y + 4 y}}{2 \left(k - 1\right)}

#### Case study 2: solving systems of equations

Note that taking the dot product of two planes’ normal vectors gives the negative cosine of their dihedral angle, so the middle constraint above can be expressed as ℓQ∙ℓR=−cosp. Here’s the corresponding SymPy code:

```python
p, q, r = sympy.symbols('p, q, r', real=True)

# create some symbols for unknown elements of lQ
x1, y1, z1 = sympy.symbols('x1, y1, z1')

# define vectors we know so far
P = sympy.Matrix([0, 0, 1])
lR = sympy.Matrix([1, 0, 0])
lQ = sympy.Matrix([x1, y1, z1])

lQ_equations = [
    sympy.Eq(lQ.dot(P), 0),              # lQ contains P
    sympy.Eq(lQ.dot(lR), -sympy.cos(p)), # angle at point P
    sympy.Eq(lQ.dot(lQ), 1)              # lQ is a unit vector
]

S = sympy.solve(lQ_equations, x1, y1, z1, dict=True, simplify=True)
print('found {} solutions for lQ:'.format(len(S)))
print('\n'.join([sympy.pretty(sln) for sln in S])) # ask for pretty output

lQ = lQ.subs(S[1])
print('now lQ is {}'.format(lQ))
```

    found 2 solutions for lQ:
    {x₁: -cos(p), y₁: -│sin(p)│, z₁: 0}
    {x₁: -cos(p), y₁: │sin(p)│, z₁: 0}
    now lQ is Matrix([[-cos(p)], [Abs(sin(p))], [0]])

One of SymPy’s quirks is that it sometimes adds extraneous details to solutions like the absolute value above (not needed here because ±sinp and ±|sinp| are equivalent sets of values). Removing these is quick once you notice them. Here’s the code to get rid of the stray Abs:

```python
lQ = lQ.subs(sympy.Abs(sympy.sin(p)), sympy.sin(p))
print('after subbing out abs, lQ is {}'.format(lQ))
```

    after subbing out abs, lQ is Matrix([[-cos(p)], [sin(p)], [0]])

Anyways, let’s manually verify that the solution worked. This is absolutely unnecessary because sympy.solve doesn’t return incorrect solutions, but it can be a nice sanity check, and the code is short, anyways:

```python
print('checking our work:')
print('  lQ . P  =', lQ.dot(P))
print('  lQ . lR =', lQ.dot(lR))
print('  lQ . lQ =', lQ.dot(lQ))
```

    checking our work:
      lQ . P  = 0
      lQ . lR = -cos(p)
      lQ . lQ = sin(p)**2 + cos(p)**2

Oops, looks like that last item could be simplified. Let’s try that again:

```python
print('  lQ . lQ =', lQ.dot(lQ).simplify())
```

      lQ . lQ = 1

All the constraints were satisfied, and I got to show you sympy.simplify in action.
Now we can go through a similar process to solve for ℓP, except this time we will explicitly encode the unit-length constraint into its z-coordinate.2

```python
x2, y2 = sympy.symbols('x2, y2')
z2 = sympy.sqrt(1 - x2**2 - y2**2)

lP = sympy.Matrix([x2, y2, z2])
print('||lP||^2 =', lP.dot(lP))

lP_equations = [
    sympy.Eq(lP.dot(lR), -sympy.cos(q)),
    sympy.Eq(lP.dot(lQ), -sympy.cos(r)),
]

S = sympy.solve(lP_equations, x2, y2, dict=True, simplify=True)
print('got {} solutions for lP'.format(len(S)))
print('\n'.join([sympy.pretty(sln) for sln in S]))

lP = lP.subs(S[0])
print('now lP is {}'.format(lP))
```

    ||lP||^2 = 1
    got 1 solutions for lP
    ⎧                 -(cos(p)⋅cos(q) + cos(r)) ⎫
    ⎨x₂: -cos(q), y₂: ──────────────────────────⎬
    ⎩                           sin(p)          ⎭
    now lP is Matrix([[-cos(q)], [-(cos(p)*cos(q) + cos(r))/sin(p)], [sqrt(-(cos(p)*cos(q) + cos(r))**2/sin(p)**2 - cos(q)**2 + 1)]])

Here’s the relvant GLSL snippet from the shader linked above, reflecting the full solution we obtained:
```
vec3 lr = vec3(1, 0, 0);
vec3 lq = vec3(-cp, sp, 0);
vec3 lp = vec3(-cq, -(cr + cp*cq)/sp, 0);
lp.z = sqrt(1.0 - dot(lp.xy, lp.xy));

vec3 P = normalize(cross(lr, lq));
vec3 Q = normalize(cross(lp, lr));
vec3 R = normalize(cross(lq, lp));
```

```python

```

#### Case study 3: Jacobians for nonlinear least squares

We start by just implementing the function f(θ,x,y) defined above:

```python
x, y, u, v, h, s, t, l, rho, phi = sympy.symbols(
    'x, y, u, v, h, s, t, l, rho, phi', real=True)

cr = sympy.cos(rho)
sr = sympy.sin(rho)

xp =  (x - u) * cr + (y - v) * sr
yp = -(x - u) * sr + (y - v) * cr

f = ( h * sympy.exp(-xp**2 / (2*s**2) - yp**2 / (2*t**2) ) *
      sympy.cos( 2 * sympy.pi * xp / l + phi ) )
```

Next, we can just print out the partial derivatives, one by one. Since we are targeting a C++ program, we can ask SymPy to directly emit C code:

```python
theta = (u, v, h, s, t, l, rho, phi)

for i, var in enumerate(theta):
    deriv = f.diff(var)
    print('grad[{}]'.format(i), '=', sympy.ccode(deriv) + ';')
```

    grad[0] = h*(-((u - x)*sin(rho) + (-v + y)*cos(rho))*sin(rho)/pow(t, 2) + ((-u + x)*cos(rho) + (-v + y)*sin(rho))*cos(rho)/pow(s, 2))*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*cos(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l) + 2*M_PI*h*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*sin(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l)*cos(rho)/l;
    grad[1] = h*(((u - x)*sin(rho) + (-v + y)*cos(rho))*cos(rho)/pow(t, 2) + ((-u + x)*cos(rho) + (-v + y)*sin(rho))*sin(rho)/pow(s, 2))*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*cos(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l) + 2*M_PI*h*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*sin(rho)*sin(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l)/l;
    grad[2] = exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*cos(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l);
    grad[3] = h*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*cos(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l)/pow(s, 3);
    grad[4] = h*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*cos(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l)/pow(t, 3);
    grad[5] = 2*M_PI*h*((-u + x)*cos(rho) + (-v + y)*sin(rho))*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*sin(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l)/pow(l, 2);
    grad[6] = h*(-1.0/2.0*((u - x)*sin(rho) + (-v + y)*cos(rho))*(2*(u - x)*cos(rho) - 2*(-v + y)*sin(rho))/pow(t, 2) - 1.0/2.0*(-2*(-u + x)*sin(rho) + 2*(-v + y)*cos(rho))*((-u + x)*cos(rho) + (-v + y)*sin(rho))/pow(s, 2))*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*cos(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l) - 2*M_PI*h*(-(-u + x)*sin(rho) + (-v + y)*cos(rho))*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*sin(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l)/l;
    grad[7] = -h*exp(-1.0/2.0*pow((u - x)*sin(rho) + (-v + y)*cos(rho), 2)/pow(t, 2) - 1.0/2.0*pow((-u + x)*cos(rho) + (-v + y)*sin(rho), 2)/pow(s, 2))*sin(phi + 2*M_PI*((-u + x)*cos(rho) + (-v + y)*sin(rho))/l);

Instead, we will get SymPy to do the common subexpression elimination for us using sympy.cse:

```python
derivs = [ f.diff(var) for var in theta ]

variable_namer = sympy.numbered_symbols('sigma_')
replacements, reduced = sympy.cse(derivs, symbols=variable_namer)

for key, val in replacements:
    print('double', key, '=', sympy.ccode(val) + ';')

print()

for i, r in enumerate(reduced):
    print('grad[{}]'.format(i), '=', sympy.ccode(r) + ';')
```

    double sigma_0 = cos(rho);
    double sigma_1 = 2*sigma_0;
    double sigma_2 = -u + x;
    double sigma_3 = sin(rho);
    double sigma_4 = -v + y;
    double sigma_5 = sigma_3*sigma_4;
    double sigma_6 = sigma_0*sigma_2 + sigma_5;
    double sigma_7 = M_PI/l;
    double sigma_8 = 2*sigma_7;
    double sigma_9 = phi + sigma_6*sigma_8;
    double sigma_10 = pow(s, -2);
    double sigma_11 = pow(sigma_6, 2);
    double sigma_12 = pow(t, -2);
    double sigma_13 = u - x;
    double sigma_14 = sigma_0*sigma_4;
    double sigma_15 = sigma_13*sigma_3 + sigma_14;
    double sigma_16 = pow(sigma_15, 2);
    double sigma_17 = exp(-1.0/2.0*sigma_10*sigma_11 - 1.0/2.0*sigma_12*sigma_16);
    double sigma_18 = h*sigma_17*sin(sigma_9);
    double sigma_19 = sigma_10*sigma_6;
    double sigma_20 = sigma_12*sigma_15;
    double sigma_21 = sigma_17*cos(sigma_9);
    double sigma_22 = h*sigma_21;
    double sigma_23 = sigma_18*sigma_8;
    double sigma_24 = sigma_2*sigma_3;

    grad[0] = sigma_1*sigma_18*sigma_7 + sigma_22*(sigma_0*sigma_19 - sigma_20*sigma_3);
    grad[1] = sigma_22*(sigma_0*sigma_20 + sigma_19*sigma_3) + sigma_23*sigma_3;
    grad[2] = sigma_21;
    grad[3] = sigma_11*sigma_22/pow(s, 3);
    grad[4] = sigma_16*sigma_22/pow(t, 3);
    grad[5] = 2*M_PI*sigma_18*sigma_6/pow(l, 2);
    grad[6] = sigma_22*(-1.0/2.0*sigma_19*(2*sigma_14 - 2*sigma_24) - 1.0/2.0*sigma_20*(sigma_1*sigma_13 - 2*sigma_5)) - sigma_23*(sigma_14 - sigma_24);
    grad[7] = -sigma_18;

```python

```


# Fourier Series
Any general waveform of finity energy $x(t)$ (square integrable) can be represented as weighted sum of trigonometric functions.
\begin{equation}
f(x) \simeq \frac{a_0}{2} + \sum_{n=1}^{\infty}\left\{a_n\cos(\frac{n\pi{x}}{p}) + b_n\sin(\frac{n\pi{x}}{p})\right\}
\end{equation}
where the Fourier coefficients $a_n, b_n$ are defined by
\begin{equation}
a_n = \frac{1}{p}\int_{-p}^{p}f(x)\cos(\frac{n\pi{x}}{p})dx, n\geq0\\
b_n = \frac{1}{p}\int_{-p}^{p}f(x)\sin(\frac{n\pi{x}}{p})dx, n\geq1
\end{equation}

## Square function (even)
\begin{equation}
f(x) = \left\{ \begin{array}{rcl}
A, & -\frac{1}{2}T_p + nT < x < \frac{1}{2}T_p + nT  \\
0, & \frac{1}{2}T_p + nT < x < \frac{3}{2}T_p + nT
\end{array}\right.
\end{equation}
where $n \in R$, $T_p = \pi$ and $T = 2\pi$.

\begin{eqnarray}
a_0 &=& \frac{1}{T}\int_0^T f(t)dt \\
&=& \frac{1}{T}\int_{-\frac{T_p}{2}}^{\frac{T_p}{2}} Adt \\
&=& \frac{T_p}{T}A \\
&=& \frac{A}{2}
\end{eqnarray}

\begin{eqnarray}
a_n &=& \frac{2}{T}\int_0^T f(t)\cos(\omega_nt)dt \\
&=& \frac{2}{T}\int_{-\frac{T_p}{2}}^{\frac{T_p}{2}} A\cos(\omega_nt)dt \\
&=& \frac{2A}{\omega_n T}\int_{-\frac{T_p}{2}}^{\frac{T_p}{2}} \cos(\omega_nt)d\omega_nt \\
&=& \frac{4A}{\omega_n T}\sin(\frac{\omega_n T_p}{2}) \\
&=& \frac{2A}{n\pi}\sin(\frac{n\pi}{2})
\end{eqnarray}

\begin{equation}
a_n = \left\{ \begin{array}{lcl}
0, & n & even, n\neq 0  \\
(-1)^{\frac{n-1}{2}}\frac{2A}{n\pi}, & n & odd
\end{array}\right.
\end{equation}

\begin{eqnarray}
b_n &=& \frac{2}{T}\int_0^T f(t)\sin(\omega_nt)dt \\
&=& \frac{2}{T}\int_{-\frac{T_p}{2}}^{\frac{T_p}{2}} A\sin(\omega_nt)dt \\
&=& 0
\end{eqnarray}

```python
import numpy as np
import matplotlib.pyplot as plt

# Fourier coefficients
def a0(A, X):
    return A/2 * np.ones(len(X))

def an(A, n):
    if n%2==0:
        return 0
    return (-1)**((n-1)/2)*(2*A)/(n*np.pi)

def Plot(x, y, s, T):
    plt.figure(figsize=(10,5))
    ax = plt.gca()
    ax.grid(color='#b7b7b7', linestyle='-', linewidth=0.5, alpha=0.5)
    plt.plot(x,y,'-', color='black', lw=2,label="Signal")
    plt.plot(x,s,'-', color='#d3d3d3', lw=2,label="Fourier series approximation")
    plt.xticks([-5/2*np.pi, -3/2*np.pi, -1/2*np.pi, 0, 1/2*np.pi, 3/2*np.pi, 5/2*np.pi],
               [r'$-5/2\pi$', r'$-3/2\pi$', r'$-1/2\pi$', r'$0$',r'$1/2\pi$', r'$3/2\pi$', r'$5/2\pi$'])

    # the arrow
    ax.annotate('', xy=(-T/2,0.5), xytext=(-T/2,0),
                arrowprops={'arrowstyle': '-', 'ls': 'dashed', 'ec': '#333333'}, va='center')
    ax.annotate('', xy=(T/2,0.5), xytext=(T/2,0),
                arrowprops={'arrowstyle': '-', 'ls': 'dashed', 'ec': '#333333'}, va='center')
    ax.annotate('', xy=(T/2,0.5), xytext=(-T/2,0.5),
                arrowprops={'arrowstyle': '<->'}, va='center')
    ax.annotate('', xy=(T/4,1.5), xytext=(-T/4,1.5),
                arrowprops={'arrowstyle': '<->'}, va='center')
    plt.text(-0.2,  1.55, r'$T_p$', {'color': 'k', 'fontsize': 12})
    plt.text(-0.2,  0.55, r'$T$', {'color': 'k', 'fontsize': 12})
    plt.show()

def Series(n):
    # Initial parameters
    A = 2
    N = 1000
    T = 2*np.pi
    f = 1/T

    x = np.linspace(-5/2*np.pi,5/2*np.pi,N)
    y = A/2*(1 + np.sign(np.cos(2*np.pi*f*x)))

    s =  a0(A, x) + sum(an(A, i) * np.cos(i*x) for i in range(1,n+1))
    Plot(x,y,s,T)

Series(19)
```

## Sawtooth function (odd)
\begin{equation}
f(x) = \frac{2A}{T}x \quad (n - \frac{1}{2})T < x < (n - \frac{1}{2})T
\end{equation}
where $n \in R$, and $T = 2\pi$.

\begin{eqnarray}
a_0 &=& \frac{1}{T}\int_0^T f(t)dt \\
&=& \frac{1}{T}\int_{-\frac{T}{2}}^{\frac{T}{2}} \frac{2A}{T}tdt \\
&=& 0
\end{eqnarray}

\begin{eqnarray}
a_n &=& \frac{2}{T}\int_0^T f(t)\cos(\omega_nt)dt \\
&=& \frac{2}{T}\int_{-\frac{T}{2}}^{\frac{T}{2}} \frac{2A}{T}t\cos(\omega_nt)dt \\
&=& 0
\end{eqnarray}

\begin{eqnarray}
b_n &=& \frac{2}{T}\int_0^T f(t)\sin(\omega_nt)dt \\
&=& \frac{2}{T}\int_{-\frac{T}{2}}^{\frac{T}{2}} \frac{2A}{T}t\sin(\omega_nt)dt \\
&=& -\frac{4A}{\omega_nT^2}\int_{-\frac{T}{2}}^{\frac{T}{2}}td\cos(\omega_nt) \\
&=& -\frac{4A}{\omega_nT^2} \left[ t\cdot\cos(\omega_nt)\big\vert_{-\frac{T}{2}}^{\frac{T}{2}} - \int_{-\frac{T}{2}}^{\frac{T}{2}} \cos(\omega_nt)dt \right] \\
&=& -\frac{4A}{\omega_nT^2} \left[ T\cos(\frac{\omega_n T}{2}) -\frac{2}{\omega_n}\sin(\frac{\omega_n T}{2}) \right]
\end{eqnarray}

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import signal

# Fourier coefficients
def bn(T, A, n):
    w_n = 2*np.pi*n/T
    return -(4*A)/(w_n*T**2)*(T*np.cos(w_n*T/2) - 2/(w_n)*np.sin(w_n*T/2))

def Plot(x,y,s,T):
    plt.figure(figsize=(10,5))
    ax = plt.gca()
    ax.grid(color='#b7b7b7', linestyle='-', linewidth=0.5, alpha=0.5)
    plt.plot(x,y,'-', color='black', lw=2,label="Signal")
    plt.plot(x,s,'-', color='#d3d3d3', lw=2,label="Fourier series approximation")
    plt.xticks([-5/2*np.pi, -3/2*np.pi, -1/2*np.pi, 0, 1/2*np.pi, 3/2*np.pi, 5/2*np.pi],
               [r'$-5/2\pi$', r'$-3/2\pi$', r'$-1/2\pi$', r'$0$',r'$1/2\pi$', r'$3/2\pi$', r'$5/2\pi$'])

    # the arrow
    ax.annotate('', xy=(T/2,-1), xytext=(-T/2,-1),
                arrowprops={'arrowstyle': '<->'}, va='center')
    plt.text(-0.2,  -0.95, r'$T$', {'color': 'k', 'fontsize': 12})

def Series(n):
    A = 2
    T = 2*np.pi
    f = 1/T

    x = np.linspace(-5/2*np.pi,5/2*np.pi,1000)
    y = A * signal.sawtooth(x-T/2)

    s =  sum(bn(T, A, i) * np.sin(i*x) for i in range(1,n+1))
    Plot(x,y,s,T)

Series(19)
```

## Delta function
Delta function can be represented as the generalized form of the square function.

```python
# general form
import numpy as np
import matplotlib.pyplot as plt

# Fourier coefficients
def a0(T, T_p, A, X):
    return T_p/T *A * np.ones(len(X))

def an(T, T_p, A, n):
    w_n = 2*m.pi*n/T
    return (4*A)/(w_n * T)*np.sin((w_n * T_p)/2)

def Plot(x,y,s,T,T_p):
    plt.figure(figsize=(10,5))
    ax = plt.gca()
    ax.grid(color='#b7b7b7', linestyle='-', linewidth=0.5, alpha=0.5)
    plt.plot(x,y,'-', color='black', lw=2,label="Signal")
    plt.plot(x,s,'-', color='#d3d3d3', lw=2,label="Fourier series approximation")
    plt.xticks([-5/2*np.pi, -3/2*np.pi, -1/2*np.pi, 0, 1/2*np.pi, 3/2*np.pi, 5/2*np.pi],
               [r'$-5/2\pi$', r'$-3/2\pi$', r'$-1/2\pi$', r'$0$',r'$1/2\pi$', r'$3/2\pi$', r'$5/2\pi$'])

    # the arrow
    ax.annotate('', xy=(-T/2,0.5), xytext=(-T/2,0),
                arrowprops={'arrowstyle': '-', 'ls': 'dashed', 'ec': '#333333'}, va='center')
    ax.annotate('', xy=(T/2,0.5), xytext=(T/2,0),
                arrowprops={'arrowstyle': '-', 'ls': 'dashed', 'ec': '#333333'}, va='center')
    ax.annotate('', xy=(T/2,0.5), xytext=(-T/2,0.5),
                arrowprops={'arrowstyle': '<->'}, va='center')
    ax.annotate('', xy=(T_p/2,1.5), xytext=(-T_p/2,1.5),
                arrowprops={'arrowstyle': '<->'}, va='center')
    plt.text(-0.2,  1.55, r'$T_p$', {'color': 'k', 'fontsize': 12})
    plt.text(-0.2,  0.55, r'$T$', {'color': 'k', 'fontsize': 12})
    plt.show()

def Series(n, delta):
    # Initial parameters
    A = 2
    T = 2*m.pi
    T_p = m.pi / delta
    f = 1/T

    x = np.linspace(-5/2*np.pi,5/2*np.pi,1000)
    y = []
    for i in X:
        if i > (-1/2*T_p) and i < (1/2*T_p) or i > (-1/2*T_p + T) and i < (1/2*T_p + T) or i > (-1/2*T_p - T) and i < (1/2*T_p - T):
            y.append(A)
        else:
            y.append(0)

    s =  a0(T, T_p, A, X) + sum(an(T, T_p, A, i) * np.cos(i*x) for i in range(1,n+1))
    Plot(x,y,s,T,T_p)

Series(60, 4)
```

## Rectified Linear Unit function (aperiod)
\begin{equation}
f(x) = \left\{\begin{array}{cl}
0,&-\pi < x \leq0\\
x,&0 < x < \pi\\
\end{array}\right.\
\end{equation}
\begin{equation}
f(x) \simeq \frac{a_0}{2} + \sum_{n=1}^{\infty}\left\{a_n\cos(nx) + b_n\sin(nx)\right\}
\end{equation}
\begin{eqnarray}
\frac{a_0}{2} &=& \frac{1}{2\pi}\int_{-\pi}^{\pi}f(x)dx \\
&=& \frac{1}{2\pi}\int_{0}^{\pi}xdx \\
&=& \frac{\pi}{4}
\end{eqnarray}
\begin{eqnarray}
a_n &=& \frac{1}{\pi}\int_{-\pi}^{\pi}f(x)\cos(nx)dx, n\geq1\\
&=& \frac{1}{\pi}\int_{0}^{\pi}x\cos(nx)dx\\
&=& \frac{1}{n\pi}\int_{0}^{\pi}xd\sin(nx)\\
&=& \frac{1}{n\pi}\left\{ \left[x\sin(nx)\right]_0^\pi - \int_{0}^{\pi}\sin(nx)dx \right\}\\
&=& -\frac{1}{n^2\pi}\int_{0}^{\pi}\sin(nx)dnx \\
&=& \frac{1}{n^2\pi}\left[\cos(nx)\right]_0^\pi \\
&=& \frac{-2}{n^2\pi},(n\geq1,odd)
\end{eqnarray}
\begin{eqnarray}
b_n &=& \frac{1}{\pi}\int_{-\pi}^{\pi}f(x)\sin(nx)dx, n\geq1\\
&=& \frac{1}{\pi}\int_{0}^{\pi}x\sin(nx)dx\\
&=& -\frac{1}{n\pi}\int_{0}^{\pi}xd\cos(nx)\\
&=& -\frac{1}{n\pi}\left\{ \left[x\cos(nx)\right]_0^\pi - \int_{0}^{\pi}\cos(nx)dx \right\}\\
&=& -\frac{1}{n\pi}\left[(-i)^n\pi-\frac{1}{n}\int_{0}^{\pi}\cos(nx)dnx\right] \\
&=& -\frac{1}{n\pi}\left\{(-i)^n\pi-\frac{1}{n}\left[\sin(nx)\right]_0^\pi\right\} \\
&=& \frac{(-1)^{n+1}}{n},(n\geq1)
\end{eqnarray}
\begin{equation}
f(x) \simeq \frac{\pi}{4} + \sum_{n\geq1,odd}\frac{-2}{n^2\pi}\cos(nx) + \sum_{n\geq1}\frac{(-1)^{n+1}}{n}\sin(nx)
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt

# Fourier coefficients
def a0(x):
    return np.pi/4 * np.ones(len(x))

def an(n,x):
    if n%2==0:
        return 0
    return (-2)/(n**2*np.pi) * np.ones(len(x))

def bn(n,x):
    return (-1)**(n+1)/n * np.ones(len(x))

def Plot(x,y,s):
    plt.figure(figsize=(10,5))
    ax = plt.gca()
    ax.grid(color='#b7b7b7', linestyle='-', linewidth=0.5, alpha=0.5)
    plt.plot(x,y,'-', color='black', lw=2,label="RELU")
    plt.plot(x,s,'-', color='#b7b7b7', lw=1,label="f2")

    plt.xticks([-np.pi, -1/2*np.pi, 0, 1/2*np.pi, np.pi], [r'$-\pi$', r'$-1/2\pi$', r'$0$',r'$1/2\pi$', r'$\pi$'])
    plt.show()

def Series(n):
    # Initial parameters
    N = 1000
    x = np.linspace(-np.pi, np.pi, N)
    y = np.where(x<0,0,x)

    s =  a0(x) + sum(an(i,x) * np.cos(i*x) + bn(i,x) * np.sin(i*x) for i in range(1,n+1))
    Plot(x,y,s)

Series(30)
```

## Reference:

- [Mathematics of the discrete Fourier transform](https://ccrma.stanford.edu/~jos/st/)
- [Fourier series - Wikipedia](https://en.wikipedia.org/wiki/Fourier_series)
- [An Interactive Guide To The Fourier Transform](https://betterexplained.com/articles/an-interactive-guide-to-the-fourier-transform/)
- [Fourier Series: Basic Results](http://www.sosmath.com/fourier/fourier1/fourier1.html)


# Solving AOC 19 with optimization

Day 19 of this year's Advent of Code is a nice geometrical problem. We are given sets of points detected by different scanners in $3d$ space. Each scanner has its own coordinate system and does **not** know its own position with respect to the others. Based on some common detected points that scanners share, the task is to retrieve their relative relative positions in a single coordinate system. A longer and more accurate description can be found at https://adventofcode.com/2021/day/19.

This task can be split into several smaller steps:

- identify the scanners which share common points
- choose a reference coordinate system
- determine the transformation needed to translate all the sets of points into the chosen coordinate system

In this post, I'm going to talk about how I solved this problem using some algorithms from convex optimization :)

## Reading the input data

First, we should read and analyze the data to know what we're dealing with. I saved the example data which is given in the problem description in a text file, which can be easily parsed as done below

```python
import numpy as np

def read_input(text_file):
    scanners = list()
    with open(text_file) as f:
        for line in f:
            if "scanner" in line:
                scanners.append(list())
            elif len(line.strip()) == 0:
                scanners[-1] = np.array(scanners[-1])
            else:
                scanners[-1].append(tuple(int(i) for i in line.strip().split(",")))
    scanners[-1] = np.array(scanners[-1])
    return scanners

points = read_input("test.txt")
```

We can also take a look at the some of the points detected by the scanners

```python
points[0]
```

    array([[ 404, -588, -901],
           [ 528, -643,  409],
           [-838,  591,  734],
           [ 390, -675, -793],
           [-537, -823, -458],
           [-485, -357,  347],
           [-345, -311,  381],
           [-661, -816, -575],
           [-876,  649,  763],
           [-618, -824, -621],
           [ 553,  345, -567],
           [ 474,  580,  667],
           [-447, -329,  318],
           [-584,  868, -557],
           [ 544, -627, -890],
           [ 564,  392, -477],
           [ 455,  729,  728],
           [-892,  524,  684],
           [-689,  845, -530],
           [ 423, -701,  434],
           [   7,  -33,  -71],
           [ 630,  319, -379],
           [ 443,  580,  662],
           [-789,  900, -551],
           [ 459, -707,  401]])

## Find the relations between different sets

Once we read the data, we should find the sets which share at least 12 points, as stated in the problem description. To do so, we can compute the pairwise distances among all points in the same set (according to some norm). Then, if two sets share at least 66 such distances, it means there are at least 12 points in common between them (since the number of pairwise distances between 12 points is 66, i.e., 12 choose 2). We call sets which share at least 12 points a "couple". Note that I'm using python sets, which could give wrong results. Indeed, if more than 2 points are at the same distance (for example (0,0,0), (1,1,1), (2,2,2)), only one distance would be counted, and we would potentially miss some related sets (if this causes the shared distances to be less than 66). Neverthless, this is easy to fix, altough I didn't do it and got the right answer anyway!

```python
from scipy.spatial.distance import pdist
from itertools import combinations

def find_connected_sets(scanners):
    connected = []
    for cloud_1, cloud_2 in combinations(range(len(scanners)), 2):
        dist_1 = pdist(scanners[cloud_1])
        dist_2 = pdist(scanners[cloud_2])
        shared_distances = set(dist_1) & set(dist_2)
        if len(shared_distances) >= 66:
            connected.append((cloud_1, cloud_2))
            connected.append((cloud_2, cloud_1))
    return connected

couples = find_connected_sets(points)
couples
```

    [(0, 1), (1, 0), (1, 3), (3, 1), (1, 4), (4, 1), (2, 4), (4, 2)]

Another problem we might encounter with the above approach is that two different sets of points which share the same distances might be considered overlapping even if they're not. For this advent of code puzzle, I'm not really sure this is a problem (how else could we detect that the sets share some points if not considering pairwise distances?), but in a real situation it could be.

Once we have all the connected "couples" of sets, we can build a graph. This graph will be used later, to determine the order to visit the sets of points. For example, the sets in the example data give rise to the following graph.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import networkx as nx

G=nx.Graph()
# Add nodes and edges
for couple in find_connected_sets(points):
    G.add_edge(couple[0], couple[1])

nx.draw(G, with_labels=True)
plt.show()
```

As already said, we need to determine the order to visit the sets of points. Indeed, in order to translate all the points in a single coordinate system, we can start from a set of points and iteratively transform the other sets into the system of the first set, following their relations in the graph we built. For example, in the example dataset, we can start from 0. From here, we can only visit 1 (since it's the only set which share at least 12 points with 0). From 1, we can either go to 4 or 3. Suppose we go to 3, then there are no other sets left. At this point we visit 4 and finally 2. This procedure corresponds to a Breadth First Search (BFS) visit of the graph of relations.

```python
def get_visit_path(relations, path, visited=None, couple=(-1, 0)):
    couples = [i for i in relations if i[0] == couple[1]]
    for new_couple in couples:
        if new_couple[1] not in visited and new_couple != couple[::-1]:
            path.append(new_couple)
            visited.add(new_couple[1])
            # recursively find next scanners
            get_visit_path(relations, path, visited, couple=new_couple)

path = []
vis = set()
get_visit_path(couples, path, vis)
path
```

    [(0, 1), (1, 3), (1, 4), (4, 2)]

## Find the correct transformation

Up to this point, we didn't describe how to translate a coordinate system to another one given that 2 sets share at least 12 points. We are going to do it next. However, first we need to identify common points between two sets which share at least 12 points. This is a bit boring: I do it by doing some manipulations on the indices in the (symmetric) matrix of pairwise distances. If you're interested in the process, the code is given below:

```python
from scipy.spatial.distance import squareform
from collections import defaultdict, Counter
from itertools import product

def find_shared_points(points_1, points_2):
    pairwise_dist_1 = pdist(points_1, 'sqeuclidean')
    pairwise_dist_2 = pdist(points_2, 'sqeuclidean')
    shared_distances = set(pairwise_dist_1) & set(pairwise_dist_2)
    if len(shared_distances) < 66:
        raise ValueError("At least 12 points are required!")
    dist_1 = squareform(pairwise_dist_1)
    dist_1[np.tril_indices(len(dist_1))] = 0
    dist_2 = squareform(pairwise_dist_2)
    dist_2[np.tril_indices(len(dist_2))] = 0
    graph_dict = defaultdict(Counter)
    for dist in shared_distances:
        first_group = np.where(dist == dist_1)
        second_group = np.where(dist == dist_2)

        for node1 in first_group:
            for node2 in second_group:
                try:
                    graph_dict[node1.item()][node2.item()] += 1
                except ValueError:
                    # if there's more than 1 index from np.where
                    for couple in product(node1, node2):
                        graph_dict[couple[0]][couple[1]] += 1
    graph_dict = {
        node: graph_dict[node].most_common(1)[0][0]
        for node in graph_dict.keys()
        if graph_dict[node].most_common(1)[0][1] >= 11
    }
    return graph_dict

find_shared_points(points[0], points[1])
for key, value in find_shared_points(points[0], points[1]).items():
    print(points[0][key], points[1][value])
```

    [ 528 -643  409] [-460  603 -452]
    [-485 -357  347] [ 553  889 -390]
    [-661 -816 -575] [729 430 532]
    [-345 -311  381] [ 413  935 -424]
    [-618 -824 -621] [686 422 578]
    [ 404 -588 -901] [-336  658  858]
    [ 459 -707  401] [-391  539 -444]
    [ 544 -627 -890] [-476  619  847]
    [-447 -329  318] [ 515  917 -361]
    [ 423 -701  434] [-355  545 -477]
    [ 390 -675 -793] [-322  571  750]
    [-537 -823 -458] [605 423 415]

Here comes the interesting part! Once we have the shared points, we can actually solve the problem. At this point, we have to find the transformation which transforms one point cloud to the other. This transformation is given by a rotation matrix and a translation vector. If we denote by $x$ and $x'$ the points in the two different coordinate systems, we will have:

$$ x = R x' + t $$

where $R$ is a $ 3 \times 3 $ rotation matrix and $ t $ is a translation vector. We can actually focus on the rotation matrix, as the translation vector is easy to recover once the points are "aligned".

The problem of finding the rotation matrix can be formalized as an optimization problem. First, we center the point clouds by subtracting them their centroids, i.e., their mean vectors. We define the centered points as $\{y_i\}_{i=1}^n$ and $\{y_i' \}_{i=1}^n$. Then we want to minimize (with respect to the rotation matrix) the sum of the distances between every couple of points. In mathematical terms, we have the following objective to minimize

$$ f(A) = \frac{1}{n} \sum_{i=1}^n \| y_i - A y_i' \| $$

where $A$ belong to the set of real-valued 3 by 3 matrices.

Is this a convex objective? It easy to verify. Consider the function $g_i(A) = \| y_i - A y_i' \|$. From the definition of convexity:

$$
\begin{align}
g_i(\lambda A_1 + (1-\lambda) A_2) &= \| y_i - \lambda A_1  - (1-\lambda) A_2 y_i' \| \\
    &= \| \lambda (y_i - A_1 y_i') + (1-\lambda) (y_i - A_2 y_i') \| \\
    &\leq \lambda \| y_i - A_1 y_i' \| + (1-\lambda) \| y_i - A_2 y_i' \| = \lambda g_i(A_1) + (1-\lambda) g_i(A_2)
\end{align}
$$

where we used the triangle inequality in the third step. Note that $f(A) = \frac{1}{n} \sum_{i=1}^n g_i(A)$ and the sum of convex functions is still a convex function. Hence, we have that $f$ is a convex function and we are guaranteed to find a global optimum.

We didn't specify the norm used in $f$, but it could be any norm. For simplicity, we will consider the Euclidean norm. The correct matrix $R$ will be the one minimizing the objective function $f$,

$$ R = \arg\min_{A \in \mathbb{R}^{3 \times 3}} f(A) $$

To be more precise, the set over which we are minimizing should be the one of the rotation matrices in $3d$ space, i.e., orthogonal matrices 3 by 3 with determinant equal to 1. However, if we use a gradient-based method, this more accurate formulation would involve an expensive projection step on the mentioned set in every step of the optimization process. Also, I found out that the minimization problem above worked well for the task at hand.

Now that we have our optimization problem formulated, we can try to solve it by using any optimization algorithm. To this aim, we can for example try any of the optimization algorithms contained in pytorch. One nice feature of pytorch is that it computes the gradients automatically, once we define the loss function. Below, we use most famous optimization algorithm, Stochastic Gradient Descent (SGD). The condition we used to determine when to stop the iterations of the optimization algorithm is to consider when the rounded matrix multiplied by the points to be rotated give back the points we want to recover. If this does not happen after 10 thousands steps, an error is raised.

```python
import torch
from functools import partial

def loss_f(x1, x2, model):
    return torch.linalg.norm(x1 - x2 @ model, axis=1).sum() / len(x1)

def optimize(cloud_1, cloud_2, opt_algo, save_history=True):
    centered_cloud_1 = cloud_1 - cloud_1.mean(axis=0)
    centered_cloud_2 = cloud_2 - cloud_2.mean(axis=0)
    centered_cloud_1 = torch.tensor(centered_cloud_1).float()
    centered_cloud_2 = torch.tensor(centered_cloud_2).float()
    history = [centered_cloud_1, centered_cloud_2]
    model = torch.eye(3, requires_grad=True)
    optimizer = opt_algo(params=[model])
    iterations = 10000
    cum_loss = 0
    for step in range(iterations):
        optimizer.zero_grad()
        loss = loss_f(centered_cloud_1, centered_cloud_2, model)
        loss.backward()
        optimizer.step()
        cum_loss += loss.item()
        converged = torch.all(torch.isclose(centered_cloud_1, centered_cloud_2 @ torch.round(model.detach())))
        if save_history:
            history.append(centered_cloud_2 @ model)
        if converged:
            break

    if not converged:
        raise ValueError(
            f"Method has not converged!\n"
            f"Rotation matrix:\n"
            f"{torch.round(model.detach())}\n"
            f"Loss at step {step}: {loss.item()}"
        )
    rot_mat = np.rint(model.detach().numpy()).astype(int)
    transl_vec = cloud_1[0] - cloud_2[0] @ rot_mat
    return rot_mat, transl_vec, step, history

def find_transf(points_1, points_2, algo, save_history=True):
    # retrieve shared points between point clouds
    graph_dict = find_shared_points(points_1, points_2)
    # retrieve the transformation
    first = points_1[list(graph_dict.keys())]
    second = points_2[list(graph_dict.values())]
    rot_mat, transl_vec, steps, history = optimize(first, second, algo, save_history=save_history)
    return rot_mat, transl_vec, steps, history

```

We can try to visualize the optimization process, by plotting the the transformed points as the objective function is minimized. We can see that as the optimal solution is approached, the transformed red points approach the original blue points. This only works in an interactive Jupyter notebook though. In the next cell we export the result to an HTML video (see below).

```python
%matplotlib notebook
from IPython.display import HTML
import matplotlib.animation as animation

def update(i, X, Y, Z, cloud1, cloud2):
    cloud2[0].set_data(X[i], Y[i])
    cloud2[0].set_3d_properties(Z[i])
    return cloud1, cloud2

def make_animation(data):
    data = np.array([i.detach().numpy() for i in data])
    xs = data[:, :, 0]
    ys = data[:, :, 1]
    zs = data[:, :, 2]
    fig = plt.figure()
    fig.set_size_inches(10, 10)
    ax = fig.add_subplot(111, projection='3d')
    cloud_1 = ax.plot(xs[0], ys[0], zs[0], "o", color='blue', markersize=5)
    cloud_2 = ax.plot([], [], [], "o", color='red', markersize=5)

    # ax limits
    x_max = max(xs.flatten())
    x_min = min(xs.flatten())
    y_max = max(ys.flatten())
    y_min = min(ys.flatten())
    z_max = max(zs.flatten())
    z_min = min(zs.flatten())
    ax.set_xlim(x_min - 10, x_max + 10)
    ax.set_ylim(y_min - 10, y_max + 10)
    ax.set_zlim(z_min - 10, z_max + 10)

    ani = animation.FuncAnimation(
        fig,
        update,
        frames=len(data) - 1,
        fargs=(xs, ys, zs, cloud_1, cloud_2),
        interval=100,
        blit=True,
        repeat=True,
    )
    # ani.save('rotation.gif', writer='imagemagick', fps=200)
    return ani

sgd = partial(torch.optim.SGD, lr=1e-4)
_, _, _, hist = find_transf(points[0], points[1], sgd)
ani = make_animation(hist)
```

    <IPython.core.display.Javascript object>

```python
HTML(ani.to_html5_video())
```

One disadvantage of SGD is that it has a learning rate to be set, which can influence the speed of convergence of the optimization process. Even more bad, if the learning rate is completely wrong, the solution will never converge to the optimal one (for example, you can [download the notebook](https://nicolaus93.github.io/assets/aoc19.ipynb) and try to use lr=1e-2)! For this reason, finding a good learning rate is a time consuming process, since different values have to be tried until a good one is found. On the other hand, there are also other optimization algorithms which do not need parameters such as the learning rate to be tuned and can still converge to the optimal solution. One such algorithm is Cocob, which has been used also to train neural networks. Let's see how it performs on this problem.

```python
%%capture
from optimal_pytorch.coin_betting.torch import Cocob

_, _, _, hist = find_transf(points[0], points[1], Cocob)
ani = make_animation(hist)
```

Note how we don't have to set any $lr$ parameter above!

```python
HTML(ani.to_html5_video())
```

There's also another subtle difference which can be noticed by looking at the videos above. If you look at the speed of the moving points when using SGD and Cocob, you can see that it is not the same. In SGD it is constant, and the red points slowly approach the blue ones. On the other hand, with Cocob the red points start moving slowly but then their speed greatly increases as they approach the blue ones. What is going on here?

The explanation is that the learning rate in SGD is fixed (at least in the pytorch version), while in Cocob it is not! But.. Didn't we say the Cocob does not have any learning rate? Well, Cocob doesn't have any learning rate to be **set**, but that doesn't mean it has no learning rate at all. The truth is that this algorithm is able to determine automatically the optimal learning rate, which is not fixed in advance, but changes depending on the data fed to it. In particular, it can be shown that finding the optimal learning rate is more or less equivalent to betting on the outcome of a coin in a game where the goal of the algorithm is to increase its total amount of money, hence the name Cocob (which stands for Continuous **Coin Betting**). Furthermore, if the objective function is fixed, the amount of money which is the algorithm is betting on (which is equivalent to learning rate) will grow exponentially and the algorithm will approach the optimal solution exponentially fast! If you're interested in a more detailed explanation you can watch [this video](https://youtu.be/f4KTZoClfs4?t=948). Coin betting is a fascinating and theoretically grounded area of research in optimization!

## Solving the original problem

Now that we have all the pieces needed, we can solve the original problem! In particular, there are 2 tasks:

- Translate all the points to a single coordinate system and count the number of unique points
- Find the couple of scanners which are at the largest Manhattan distance apart. What is such distance?

Below, we show how to use the functions described until now to solve the 2 tasks:

```python
def solve(data_file, algo, save_iterations=False):
    point_clouds = read_input(data_file)
    couples = find_connected_sets(point_clouds)
    visit_path = []
    vis = set()
    get_visit_path(couples, visit_path, vis)  # modify path inplace
    transl_vectors = []
    tot_iterations = 0
    for couple in visit_path:
        first, second = couple
        rot_mat, transl_vec, iterations, history = find_transf(
            point_clouds[first],
            point_clouds[second],
            algo,
            save_history=save_iterations
        )
        point_clouds[second] = point_clouds[second] @ rot_mat + transl_vec
        transl_vectors.append(transl_vec)
        tot_iterations += len(history) - 1

    common_points = set()
    for cloud in point_clouds:
        for point in cloud:
            common_points.add(tuple(point))

    max_dist = max(pdist(transl_vectors, 'cityblock'))
    if save_iterations:
        print(f"Total number of iterations: {tot_iterations}")
    return len(common_points), int(max_dist)

part_1, part_2 = solve("test.txt", Cocob)
print(f"part1: {part_1}")
print(f"part2: {part_2}")
```

    part1: 79
    part2: 3621

```python
part_1, part_2 = solve("test.txt", partial(torch.optim.SGD, lr=1e-4))
print(f"part1: {part_1}")
print(f"part2: {part_2}")
```

    part1: 79
    part2: 3621

And now the real puzzle, both with Cocob and SGD:

```python
part_1, part_2 = solve("input_puzzle.txt", Cocob)
print(f"part1: {part_1}")
print(f"part2: {part_2}")
```

    part1: 400
    part2: 12168

```python
part_1, part_2 = solve("input_puzzle.txt", partial(torch.optim.SGD, lr=1e-4))
print(f"part1: {part_1}")
print(f"part2: {part_2}")
```

    part1: 400
    part2: 12168

We can also measure how long it takes for the optimization algorithms to find the solution in terms of time:

```python
%%timeit
solve("input_puzzle.txt", Cocob)
```

    508 ms ± 4.33 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%%timeit
solve("input_puzzle.txt", partial(torch.optim.SGD, lr=1e-4))
```

    958 ms ± 37.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

It seems that Cocob is faster. However, the time taken to find a solution is not really meaningful since it is subject to many different variables. Instead, we can measure how many iterations the algorithms need:

```python
solve("input_puzzle.txt", Cocob, save_iterations=True)
```

    Total number of iterations: 2069

    (400, 12168)

```python
solve("input_puzzle.txt", partial(torch.optim.SGD, lr=1e-4), save_iterations=True)
```

    Total number of iterations: 4936

    (400, 12168)

## Summary

This was definitely a cool problem to solve in this year's Advent of Code! I hope you enjoyed the explanation on optimization algorithms and especially **coin betting**. I didn't go into much detail, but if you're interested this is a very nice topic in optimization and machine learning. You can find many resources online, and there even was a [tutorial](https://www.youtube.com/watch?v=UT_ziU3nIwU&t=619s) at last year's ICML. Despite their [practical success](https://github.com/Arturus/kaggle-web-traffic/blob/master/how_it_works.md#training-and-validation), I believe these algorithms are still not very well known to the practicioners. One of the reason is that there are not many implementations available. For this reason, last year we started implementing some of them in pytorch, and currently a [library](https://github.com/Nicolaus93/coin_betting) on Github is available on my profile. The plan is to add more parameter-free algorithms to the library (not only in torch but also JAX) and extensively test them against other more famous optimization algorithms such as SGD or Adam.


```python

```

```python
from sympy import *
from math import *
x, y = Symbol('x'), Symbol('y')

from sympy.plotting import plot

from sympy.vector import Vector
from sympy.vector import CoordSys3D

from sympy.geometry import Point

N = CoordSys3D('N')

```

```python
class taylor_polys:
    def __init__(self):
        pass

    def taylor_eval(self, f, a, n):
        """
        Evaluates the Taylor Polynomial of a given function n times centered around a

        A Taylor polynomial is defined as:
                n
        Pn(x) = Σ((deriv(f(a), i)(x-a)^n)/(n!))
               i=0

        f = function
        n = # of iterations
        a = center point

        Ex: P2(x) @ a = 3

        = ((deriv(f(a), 0)(x-3)^0)/(0!)) + ((deriv(f(a), 1)(x-3)^1)/(1!)) + ((deriv(f(a), 2)(x-3)^2)/(2!))
        = 0 + deriv(f(a), 1)(x-3) + ((deriv(f(a), 2)(x-3)^2)/6)
        """
        return sum([(lambdify(x, diff(f, x, i))(a)*(x-a)**i)/(factorial(i)) for i in range(n+1)])

    def langr_error(self, f, a, n, x_est):
        """
        Evaluates the Langrange Error of a function given an estimate of x

        The Langrange Error Bound for Pn(x) is defined as:

        abs(En(x)) = abs(f(x) - Pn(x)) <= (M*(abs(x-a))**n+1)/((n+1)!)

        M = max(deriv(f(abs(a)), n+1), deriv(f(abs(x_est)), n+1))

        M = is the bigger # between the n+1th derivative of the function of the absolute values of a and the estimate of x
        x_est = the estimate of x

        """

        M = max(lambdify(x, diff(f, x, n+1))(abs(a)),lambdify(x, diff(f, x, n+1))(abs(x_est)))
        return (M*(abs(x_est-a)**(n+1)))/(factorial(n+1))
```

```python
model = taylor_polys()
model.langr_error(e**x, 0, 3, -.1)
```

    4.6048788253152e-06

```python
ans_eq = 2*x**3 + 4*x**2 + 1*x
approx_eq = model.taylor_eval(ans_eq,2,2)
approx_eq
```

$\displaystyle 41 x + 16 \left(x - 2\right)^{2} - 48$

```python

plot(approx_eq ,ans_eq, xlim=[-10,10], ylim=[-10,10])
```

```python
testeq = lambdify([x,y],x**y)(1,1)
testeq
```

    1

```python
eq1 = Eq(y**2, 2*x+3)
solve(eq1,y)
```

    [-sqrt(2*x + 3), sqrt(2*x + 3)]

```python
n = Symbol('n')
def ratio_test(eq):

    ans = solveset(abs(limit(eq.subs(n,n+1)/eq, n, float('inf'))) < 1, x, S.Reals)
    return ans, (ans.end-ans.start)/2
```

```python
ratio_test(((1)**n*(2*x-3)**(n+1))/(2**n*(n+1)))
```

    (Interval.open(1/2, 5/2), 1)

```python
def projAtoB(a, b): return (a.dot(b)/(b.magnitude()**2)) * b
```

```python
a = 6*N.i - 3*N.j + 2*N.k
b = 2*N.i + 1*N.j - 2*N.k
projAtoB(a, b)
```

$\displaystyle (\frac{10}{9})\mathbf{\hat{i}_{N}} + (\frac{5}{9})\mathbf{\hat{j}_{N}} + (- \frac{10}{9})\mathbf{\hat{k}_{N}}$

```python
t = Symbol('t')
def collision_test(p1, p2):
    # Collision Test

    col = dict((str(x), solve(Eq(p1[x], p2[x]))) for x in range(3))

    for i in col['0']:
        if i in col['1'] and i in col['2']:
            return Point([lambdify(t,p1[x])(i) for x in range(3)]),True

    return (None, False)
```

```python
pointA = Point(t, t**2, t**3)
pointB = Point(1 - t, (1-t)/2, (1-t)/4)

collision_test(pointA, pointB)
```

    (Point3D(1/2, 1/4, 1/8), True)

```python
def normalize(x):
    return x/x.magnitude()
```

```python
a = 1*N.i + 1*N.j - 1*N.k
normalize(a)
```

$\displaystyle (\frac{\sqrt{3}}{3})\mathbf{\hat{i}_{N}} + (\frac{\sqrt{3}}{3})\mathbf{\hat{j}_{N}} + (- \frac{\sqrt{3}}{3})\mathbf{\hat{k}_{N}}$

```python

```


# Symbolické počty
V tomto tutoriálu je představen modul [Sympy](http://www.sympy.org/en/index.html), který slouží k počtům se symbolickou proměnnou v Pythonu. Kód pro import Sympy a nastavení Pylabu následuje.

```python
# inline plots
%matplotlib inline
# import sympy
import sympy as sp
```

Aby bylo možné zobrazit výstup ze Sympy pomocí Latex rovnicí v Jupyter Notebook, následující nastavení musí být provedeno.

```python
sp.init_printing(use_latex='mathjax')
```

***
## Práce se Sympy
Na začátku je potřeba si vytvořit symbolické proměnné. Příklad následuje.

```python
x, y, z = sp.symbols('x y z')
```

Od teď je možné proměnné `x, y, z` (Sympy objekty proměnná) používat při počtech.

```python
sp.Eq(x + y, z) # create equation
```

$$x + y = z$$

```python
sp.simplify(x**2 + y - 2*z / (x*z)) # simplify expression
```

$$x^{2} + y - \frac{2}{x}$$

Některé funkce Sympy dokáží také pracovat s výrazem/rovnicí zadanou jako text. V tomto případě není potřeba proměnné vytvářet předem. V následujícím příkladu užijeme proměnné *a* a *b*, které předtím nijak nedefinujeme!

```python
sp.simplify("a + b**2")
```

$$a + b^{2}$$

Některé funkce vrací svůj výsledek ve formě objektu, který je možný následně použít. Ukázáno na příkladu.

```python
f = sp.simplify("a + b**2 / (a*b)") # creation of Sympy object - expression
type(f)
```

    sympy.core.add.Add

```python
f.cancel()
```

$$\frac{1}{a} \left(a^{2} + b\right)$$

Poznámka: alternativní použití podobných funkcí je možno přímo s textem jako argument - příklad následuje:

```python
sp.cancel("a + b**2 / (a*b)")
```

$$\frac{1}{a} \left(a^{2} + b\right)$$

***
## Úpravy výrazů
Následují příklady úprav výrazů na výrazu: $\frac{z+\frac{x^3 + 1}{y^2}}{2(1-x)+x^2}$

```python
f = '(1+(x**3+1/x**2))/(2*(1-x)+x**2)'
```

### Zjednodušení výrazu

```python
sp.sympify(f)
```

$$\frac{x^{3} + 1 + \frac{1}{x^{2}}}{x^{2} - 2 x + 2}$$

### Převedení na kanonickou formu

```python
sp.cancel(f)
```

$$\frac{x^{5} + x^{2} + 1}{x^{4} - 2 x^{3} + 2 x^{2}}$$

### Rozložení na faktory

```python
sp.factor(f)
```

$$\frac{x^{5} + x^{2} + 1}{x^{2} \left(x^{2} - 2 x + 2\right)}$$

### Rozklad

```python
sp.expand(f)
```

$$\frac{x^{3}}{x^{2} - 2 x + 2} + \frac{1}{x^{4} - 2 x^{3} + 2 x^{2}} + \frac{1}{x^{2} - 2 x + 2}$$

### Rozklad na parciální zlomky

```python
fs = sp.simplify(f)
sp.apart(fs)
```

$$x + \frac{3 x - 5}{2 x^{2} - 4 x + 4} + 2 + \frac{1}{2 x} + \frac{1}{2 x^{2}}$$

### Dosazení hodnoty

```python
fs.subs("x", 5)
```

$$\frac{3151}{425}$$

***
## Řešení rovnic
### Kořeny rovnice
Následují dva příklady jak získat kořeny rovnice.

```python
f = "x**2 +3*x -4"
```

```python
sp.roots(f)
```

$$\left \{ -4 : 1, \quad 1 : 1\right \}$$

Nebo:

```python
sp.solve(f)

```

$$\left [ -4, \quad 1\right ]$$

### Řešení soustavy rovnic
Pomocí Sympy je možné zadat soustavu rovnic více způsoby. Následuje příklad jak vytvořit soustavu:

$ a + b = 1 $

$ a^4 = c $

$ b - 5/2 = 3 $

pomocí listu rovnic.

```python
a, b, c = sp.symbols("a, b, c")
equations = [
    sp.Eq(a + b, 1),
    sp.Eq(a**4, c),
    sp.Eq(b - 5/2, 3),
]
sp.solve(equations)
```

$$\left [ \left \{ a : -4.5, \quad b : 5.5, \quad c : 410.0625\right \}\right ]$$

Poznámka: všiměte si, že vrácen je slovnik uvnitř listu.

***
## Kalkulus

### Derivace

```python
fd = sp.Derivative('2*x*sqrt(1/x)',x)
fd
```

$$\frac{d}{d x}\left(2 x \sqrt{\frac{1}{x}}\right)$$

```python
fd.doit()
```

$$\sqrt{\frac{1}{x}}$$

### Integrál

```python
fi = sp.Integral('2*x*sqrt(1/x)',x)
fi
```

$$\int 2 x \sqrt{\frac{1}{x}}\, dx$$

```python
fi.doit()
```

$$\frac{4 x^{2}}{3} \sqrt{\frac{1}{x}}$$

### Limity

```python
fl = sp.Limit('sin(x)/x', x, 0)
fl
```

$$\lim_{x \to 0^+}\left(\frac{1}{x} \sin{\left (x \right )}\right)$$

```python
fl.doit()
```

$$1$$

***
## Kreslení grafů
Sympy používá Matplotlib ke kreslení grafů. Samotný Matplotlib a jeho pokročilé možnosti jsou předmětem jiného tutoriálů. Následuje několik jednoduchých příkladů, jak použít Matplotlib skrze Sympy.

```python
sp.plot(x**2)
```

```python
sp.plot(x**3)
```

***
## Vektory a Matice

```python
A = sp.Matrix([[x, -1], [3, x], [1, 2]])
A
```

$$\left[\begin{matrix}x & -1\\3 & x\\1 & 2\end{matrix}\right]$$

```python
B = sp.Matrix([[1, -1, 0], [3, 2, 4]])
B
```

$$\left[\begin{matrix}1 & -1 & 0\\3 & 2 & 4\end{matrix}\right]$$

```python
(A * B)**2
```

$$\left[\begin{matrix}\left(- x - 2\right) \left(3 x + 3\right) + \left(x - 3\right)^{2} - 28 & \left(- x - 2\right) \left(x - 3\right) + \left(- x - 2\right) \left(2 x - 3\right) - 12 & 4 x \left(- x - 2\right) - 4 x - 20\\28 x + \left(x - 3\right) \left(3 x + 3\right) + \left(2 x - 3\right) \left(3 x + 3\right) & 12 x + \left(- x - 2\right) \left(3 x + 3\right) + \left(2 x - 3\right)^{2} & 4 x \left(2 x - 3\right) + 20 x - 12\\16 x + 44 & - x + 1 & 12 x + 36\end{matrix}\right]$$

```python
2.5 * (B * A)
```

$$\left[\begin{matrix}2.5 x - 7.5 & - 2.5 x - 2.5\\7.5 x + 25.0 & 5.0 x + 12.5\end{matrix}\right]$$

### Užitečné konstruktory matic

```python
sp.eye(3)
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
sp.ones(3,2)
```

$$\left[\begin{matrix}1 & 1\\1 & 1\\1 & 1\end{matrix}\right]$$

```python
sp.zeros(2,3)
```

$$\left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$$

### Determinant a charakteristický polynom matice
Následuje ukázka jak získat determinand matice.

```python
C = sp.Matrix([[1, -x, 0], [x, 2, 4], [-3, 2, -4]])
C.det()
```

$$- 4 x^{2} + 12 x - 16$$

Následující kód získa charakteristický polynom matice *C*.

```python
p = C.charpoly()
sp.factor(p)
```

$$\lambda^{3} + \lambda^{2} + \lambda x^{2} - 18 \lambda + 4 x^{2} - 12 x + 16$$

### Vlastní čísla, vektory
Následují příklady jak spočítat vlastní čísla (*eigenvalues*) a vlastní vektory (*eigenvectors*) matice.

```python
D = sp.Matrix([[1, -1], [x, 2]])
D.eigenvals()
```

$$\left \{ - \frac{1}{2} \sqrt{- 4 x + 1} + \frac{3}{2} : 1, \quad \frac{1}{2} \sqrt{- 4 x + 1} + \frac{3}{2} : 1\right \}$$

```python
D.eigenvects()
```

$$\left [ \left ( - \frac{1}{2} \sqrt{- 4 x + 1} + \frac{3}{2}, \quad 1, \quad \left [ \left[\begin{matrix}\frac{1}{\frac{1}{2} \sqrt{- 4 x + 1} - \frac{1}{2}}\\1\end{matrix}\right]\right ]\right ), \quad \left ( \frac{1}{2} \sqrt{- 4 x + 1} + \frac{3}{2}, \quad 1, \quad \left [ \left[\begin{matrix}\frac{1}{- \frac{1}{2} \sqrt{- 4 x + 1} - \frac{1}{2}}\\1\end{matrix}\right]\right ]\right )\right ]$$

### Řešení soustavy rovnic
Hledáme $x$ a $y$ pro následující soustavu rovnic pro jakékoliv $z$:

$ 5x -3y = z $

$ -4x + 3y = 2 $

Pro řešení je soustava nejdříve přepsána maticové podoby:

$AX = B$

kde $X = [x, y]$. Řešení je potom:

$X = A^{-1}B$

Realizace pomocí Sympy následuje:

```python
A = sp.Matrix([[5, -3], [-4, 3]])
B = sp.Matrix([z, 2])
X = A**-1 * B
X
```

$$\left[\begin{matrix}z + 2\\\frac{4 z}{3} + \frac{10}{3}\end{matrix}\right]$$


```python
from sympy.abc import x,y,z
from sympy import sin,Matrix
from sympy import *
import numpy as np
```

### Jacobian
Use sympy to replicate the MATLAB documentation https://uk.mathworks.com/help/symbolic/jacobian.html

#### 1. Jacobian of Vector Function

```python
X = Matrix([x*y*z,y**2,x+z])
Y = Matrix([x,y,z])
X.jacobian(Y)
```

$\displaystyle \left[\begin{matrix}y z & x z & x y\\0 & 2 y & 0\\1 & 0 & 1\end{matrix}\right]$

#### 2. Jacobian of Scalar Function

```python
X = Matrix([2*x+3*y+4*z])
Y = Matrix([x,y,z])
X.jacobian(Y)
```

$\displaystyle \left[\begin{matrix}2 & 3 & 4\end{matrix}\right]$

#### 3. Jacobian with Respect to Scalar

```python
X = Matrix([x**2*y,x*sin(y)])
Y = Matrix([x])
a = X.jacobian(Y)
```

```python
print(a)
```

    Matrix([[2*x*y], [sin(y)]])

```python
b = a.subs([(x,0),(y,0)])
```

```python
print(b)
```

    Matrix([[0], [0]])

```python
np.array(b).astype(int).squeeze()+1
```

    array([1, 1])

#### 4. Others

```python
x, y = symbols('x y')
f = x + y
f.subs([(x,10),(y,20)])
f
```

$\displaystyle x + y$

```python
x, y, z, t = symbols('x y z t')
expr = cos(x)+1
expr.subs(x,y)
```

$\displaystyle \cos{\left(y \right)} + 1$

```python
a = expr.subs(x,0)
```

```python
print(a+1)
```

    3

```python

```


# Euler Problem 203

    The binomial coefficients nCk can be arranged in triangular form, Pascal's triangle, like this:
                                    1
                                1		1
                            1		2		1
                        1		3		3		1
                    1		4		6		4		1
                1		5		10		10		5		1
            1		6		15		20		15		6		1
        1		7		21		35		35		21		7		1
                                .........

    It can be seen that the first eight rows of Pascal's triangle contain twelve distinct numbers:
    1, 2, 3, 4, 5, 6, 7, 10, 15, 20, 21 and 35.

    A positive integer n is called squarefree if no square of a prime divides n. Of the twelve distinct
    numbers in the first eight rows of Pascal's triangle, all except 4 and 20 are squarefree. The sum of
    the distinct squarefree numbers in the first eight rows is 105.

    Find the sum of the distinct squarefree numbers in the first 51 rows of Pascal's triangle.

```python
N = 51

def binom(n, k):
    if k > n // 2:
        k = n - k
    b = 1
    for i in range(k):
        b = (b * (n - i)) // (i + 1)
    return b
```

```python
from sympy import primerange, nextprime

MAXIMUM_ROW_NUMBER = 100
smallprimes = list(primerange(2, MAXIMUM_ROW_NUMBER + 1))
```

```python
def squarefree_binom(n, k):
    assert n <= MAXIMUM_ROW_NUMBER
    for p in smallprimes:
        if p > n:
            return True
        if binom_order(n, k, p) > 1:
            return False
    return True

```

The exponent of a prime $p$ in the prime factorization of $n!$ is equal to $(n-s)/(p-1)$, where $s$ is the sum of the base-$p$ digits of $n$. This is a consequence of [Legendre's formula](https://en.wikipedia.org/wiki/Legendre%27s_formula). We can use this formula to calculate the exponent of $p$ in the prime factorization of a binomial coefficient.

```python
def sum_of_digits(n, p):
    return n if n < p else (n % p) + sum_of_digits(n // p, p)

def fact_order(n, p):
    return (n - sum_of_digits(n, p)) // (p - 1)

def binom_order(n, k, p):
    return fact_order(n, p) - fact_order(k, p) - fact_order(n-k, p)

```

```python
results = set()
for n in range(N):
    for k in range(n//2 + 1):
        if squarefree_binom(n, k):
            results.add(binom(n, k))
print(sum(results))
```

    34029210557338

Euler 203 - Squarefree Binomial Coefficients.ipynb


# POL280 Bayesian Modeling Homework 1

### Gento Kato (May 2, 2017)

---

## Q1
*Use the Gamma-Poisson conjugate speciﬁcation to analyze data on the number of presidential appointments from 1960 to 2000. The data are in the ﬁle called **appointments.dta** in the Dropbox folder.*

---

To start with, it is shown in the class that $gamma(\alpha, \beta)$ prior distribution and poisson likelihood produces $gamma(\alpha + \Sigma y, \beta + n)$ posterior distribution, as follows:

* **Gamma (Prior) - Poisson (Likelihood) $\Rightarrow$ Gamma Posterior**

\begin{align}
\mbox{Prior Distribution} = \mbox{Gamma}(\alpha, \beta) &= \frac{\beta^{\alpha}}{\Gamma (\alpha)} \theta^{\alpha-1} e^{- \beta \theta} \\
\mbox{Poisson PMF } &= p(y | \theta) = \frac{e^{-\theta} \theta^{y_i}}{y_i !} \\
\mbox{Poisson Likelihood } &= \mathit{L}(\theta | y) = \hat{\Pi}_{i=1}^n \frac{e^{-\theta} \theta^{y_i}}{y_i !} \\
&= \frac{e^{-\theta n} \theta^{\sum_{i=1}^{n} y_i} }{y_1 ! y_2 ! \dots y_n !} \\
\mbox{Posterior Distribution } \pi(\theta | y) &\propto \frac{\beta^{\alpha}}{\Gamma (\alpha)} \theta^{\alpha-1} e^{- \beta \theta} \times \frac{e^{-\theta n} \theta^{\sum_{i=1}^{n} y_i} }{y_1 ! y_2 ! \dots y_n !} \\
&\propto \theta^{\alpha - 1 + \Sigma y} e^{- \theta (\beta + n)} \\
&\propto \mbox{Gamma }(\alpha + \Sigma y, \beta + n)
\end{align}

Now, the observed posterior value from appointments.dta can be extracted as follows:

```R
library(foreign)
appdta <- read.dta("../data/POL280/appointments.dta") ## Open Data
y <- appdta$appoints ; y ## Store observed y
```

2

3

3

2

0

1

2

1

2

1

Then, using R function rgamma(alpha, beta), we can create function to produce posterior distribution:

```R
posterior <- function(sample, y, alpha, beta){
    n = length(y); sigmay = sum(y)
    # n is y's sample size, sigmay is the sum of all y values
    return(rgamma(sample, alpha+sigmay, beta+n))
    # generate posterior distribution from prior's parameters
}
```

I will use the above function in Q3 and Q4.

## Q2
*The posterior distribution for $\theta$ is $gamma(\delta_1, \delta_2)$ according to some parameters $\delta_1$ and $\delta_2$, which of course depend on your choice of the parameters for the gamma prior. You should model $\theta$ using two sets of priors: one which speciﬁes a great deal of certainty regarding your best guess as to the value of $\theta$ and one that represents ignorance regarding this value. *

---

Before turning to the analysis, for the gamma distribution specification in Q1, the mean and variance of the distribution are defined as follows:

\begin{align}
Mean\left[gamma(\alpha, \beta)\right] &= \frac{\alpha}{\beta}\\
Var\left[gamma(\alpha, \beta)\right] &= \frac{\alpha}{\beta^2}\\
\end{align}

Now, suppose that there are two prior belief gamma distribution with the same mean (let's say 5), but different variance (certainty), as follows:

\begin{align}
\mbox{Prior 1 (ignorant): Mean } &= \frac{\alpha}{\beta} = \frac{5}{1} = 5 \\
\mbox{Variance } &= \frac{\alpha}{\beta^2} = \frac{5}{1} = 5\\
\mbox{Prior 2 (certain): Mean } &= \frac{\alpha}{\beta} = \frac{50}{10} = 5 \\
\mbox{Variance } &= \frac{\alpha}{\beta^2} = \frac{50}{100} = \frac{1}{2}
\end{align}

We can generate those two prior distributions in R as follows

```R
## Set Alpha and Betas
a1 = 5; b1 = 1 ## Ignorant
a2 = 50; b2 = 10 ## Certain

## Generate Prior distribution
set.seed(27674) # Make this replicable
prior1 <- rgamma(10000, a1, b1) # Ignorant
prior2 <- rgamma(10000, a2, b2) # Certain

## Check Result
paste("For prior 1, mean is ", round(mean(prior1),2),
      ", variance is ", round(var(prior1),2))
paste("For prior 2, mean is ", round(mean(prior2),2),
      ", variance is ", round(var(prior2),2))
```

Now, the above two distributions can be plotted as follows:

```R
library(ggplot2);
source("https://raw.githubusercontent.com/gentok/Method_Notes/master/sources/gktheme.R")

bayesdata <- data.frame(prior1 = prior1, prior2 = prior2)

## Plot Result ##
bgraph <- ggplot(bayesdata) + gktheme +
    geom_density(aes(prior1, fill="1"), alpha = 0.5, size=0.5) +
    geom_density(aes(prior2, fill="2"), alpha = 0.5, size=0.5) +
    scale_y_continuous(limits=c(0,0.6),breaks=c(0,0.2,0.4,0.6))+
    scale_x_continuous(limits=c(0,16.03),breaks=c(0,2.5,5,7.5,10,12.5,15))+
    scale_linetype_manual(name="Gamma Parameters",values=c(1,2),
      labels = c(expression(paste("1. Ignorant: ", alpha == 5, "; " , beta == 1)),
                 expression(paste("2. Certain: ", alpha == 50, "; " , beta == 10))))+
    scale_fill_manual(name="Gamma Parameters",values=c(1,2),
      labels = c(expression(paste("1. Ignorant: ", alpha == 5, "; " , beta == 1)),
                 expression(paste("2. Certain: ", alpha == 50, "; " , beta == 10))))+
    xlab("Prior Belief")+
    ylab("Density")+
    ggtitle("Prior Distributions by Different Parameters")+
    theme(legend.position = "right")
```

```R
options(repr.plot.width=7, repr.plot.height=3.5)
bgraph
```

## Q3
*Generate a large number of values from this distribution in **R**, say 10, 000 or so, using the command:*

**posterior.sample <- rgamma(10000,d1,d2)**

---

Note that in Q1, I created following objects:

 * y variable, which is number of appointments by each president.
 * posterior function, to generate posterior distribution from gamma prior parameters and poisson likelihood

posterior function utilizes rgamma, so those two functions are essentially the same. Therefore, I use posterior function to generate posterior distribution, as follows:

```R
set.seed(8900786) # Make this replicable
#posterior(y, alpha, beta) ## Alpha and Beta from prior distribution
posterior1 <- posterior(10000, y, a1, b1)
posterior2 <- posterior(10000, y, a2, b2)
```

## Q4

Summarize the posteriors with quantities of interest such as means, medians, and
variances. Also supply plots of the density of the posterior distributions.

---

Before starting the analysis, note that mean, median and variance of observed y is shown as follows:

```R
paste("For observed y, mean is ", round(mean(y),2),
      ", median is ", round(median(y),2),
      ", variance is ", round(var(y),2))
```

'For observed y, mean is  1.7 , median is  2 , variance is  0.9'

Now the characteristics of posterior distribution can be extracted as follows:

```R
## Check Result
paste("For posterior 1, mean is ", round(mean(posterior1),2),
      ", median is ", round(median(posterior1),2),
      ", variance is ", round(var(posterior1),2))
paste("For posterior 2, mean is ", round(mean(posterior2),2),
      ", median is ", round(median(posterior2),2),
      ", variance is ", round(var(posterior2),2))
```

'For posterior 1, mean is  2 , median is  1.96 , variance is  0.18'

'For posterior 2, mean is  3.35 , median is  3.33 , variance is  0.17'

From the above result, the mean of posterior distrbution from more ignorant prior distribution (i.e., prior 1) is more strongly pulled by the observed y values than the mean of posterior distibution from certain prior distributon (i.e., prior 2). In other words, posterior distribution 1 has closer mean (i.e., mean is 2) to observed y (i.e., mean is 1.7) than posterior distribution 2 (i.e, mean is 3.35). The observed y has stronger influence on ignorant (high variance) prior belief than on certain (low variance) prior belief. Note that the variance for two posterior distributions are identical.

Now the Result can be plotted as follows:

```R
bayesdata$posterior1 <- posterior1
bayesdata$posterior2 <- posterior2

## Plot Result ##
bgraph2 <- ggplot(bayesdata) + gktheme +
    geom_density(aes(posterior1, fill="1"), alpha = 0.5, size=0.5) +
    geom_density(aes(posterior2, fill="2"), alpha = 0.5, size=0.5) +
    scale_y_continuous(limits=c(0,1),breaks=c(0,0.25,0.5,0.75,1))+
    scale_x_continuous(limits=c(0,6),breaks=c(0,1,2,3,4,5,6))+
    scale_linetype_manual(name="Gamma Prior Parameters",values=c(1,2),
      labels = c(expression(paste("1. Ignorant: ", alpha == 5, "; " , beta == 1)),
                 expression(paste("2. Certain: ", alpha == 50, "; " , beta == 10))))+
    scale_fill_manual(name="Gamma Prior Parameters",values=c(1,2),
      labels = c(expression(paste("1. Ignorant: ", alpha == 5, "; " , beta == 1)),
                 expression(paste("2. Certain: ", alpha == 50, "; " , beta == 10))))+
    xlab("Posterior Belief")+
    ylab("Density")+
    ggtitle("Posterior Distributions by Prior Parameters")+
    theme(legend.position = "right")
```

```R
options(repr.plot.width=7, repr.plot.height=3.5)
bgraph2
```

The above plot further confirmes the implication. While the shape of two posterior distributions are almost identical, the posterior distribution for ignorant prior is placed left of the posterior distribution for certain prior. Given that both prior distribution had the same mean of 5, ignorant prior holders are more strongly pulled by the observation of y (mean of 2) than certain prior holders.


# Multivariate Gaussian with full covariance

In this reading you will learn how you can use TensorFlow to specify any multivariate Gaussian distribution.

```python
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

print("TF version:", tf.__version__)
print("TFP version:", tfp.__version__)
```

    TF version: 2.7.0
    TFP version: 0.15.0

So far, you've seen how to define multivariate Gaussian distributions using `tfd.MultivariateNormalDiag`. This class allows you to specify a multivariate Gaussian with a diagonal covariance matrix $\Sigma$.

In cases where the variance is the same for each component, i.e. $\Sigma = \sigma^2 I$, this is known as a _spherical_ or _isotropic_ Gaussian. This name comes from the spherical (or circular) contours of its probability density function, as you can see from the plot below for the two-dimensional case.

```python
# Plot the approximate density contours of a 2d spherical Gaussian

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

spherical_2d_gaussian = tfd.MultivariateNormalDiag(loc=[0., 0.])

N = 100000
x = spherical_2d_gaussian.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
sns.jointplot(x1, x2, kind='kde', space=0, );
```

As you know, a diagonal covariance matrix results in the components of the random vector being independent.

## Full covariance with `MultivariateNormalFullTriL`

You can define a full covariance Gaussian distribution in TensorFlow using the Distribution `tfd.MultivariateNormalTriL`.

Mathematically, the parameters of a multivariate Gaussian are a mean $\mu$ and a covariance matrix $\Sigma$, and so the `tfd.MultivariateNormalTriL` constructor requires two arguments:

- `loc`, a Tensor of floats corresponding to $\mu$,
- `scale_tril`, a a lower-triangular matrix $L$ such that $LL^T = \Sigma$.

For a $d$-dimensional random variable, the lower-triangular matrix $L$ looks like this:

\begin{equation}
    L = \begin{bmatrix}
            l_{1, 1} & 0 & 0 & \cdots & 0 \\
            l_{2, 1} & l_{2, 2} & 0 & \cdots & 0  \\
            l_{3, 1} & l_{3, 2} & l_{3, 3} & \cdots & 0  \\
            \vdots  & \vdots & \vdots & \ddots & \vdots \\
            l_{d, 1} & l_{d, 2} & l_{d, 3} & \cdots & l_{d, d}
        \end{bmatrix},
\end{equation}

where the diagonal entries are positive: $l_{i, i} > 0$ for $i=1,\ldots,d$.

Here is an example of creating a two-dimensional Gaussian with non-diagonal covariance:

```python
# Set the mean and covariance parameters

mu = [0., 0.]  # mean
scale_tril = [[1.,  0.],
              [0.6, 0.8]]

sigma = tf.matmul(tf.constant(scale_tril), tf.transpose(tf.constant(scale_tril)))  # covariance matrix
print(sigma)
```

    tf.Tensor(
    [[1.  0.6]
     [0.6 1. ]], shape=(2, 2), dtype=float32)

```python
# Create the 2D Gaussian with full covariance

nonspherical_2d_gaussian = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)
nonspherical_2d_gaussian
```

    <tfp.distributions.MultivariateNormalTriL 'MultivariateNormalTriL' batch_shape=[] event_shape=[2] dtype=float32>

```python
# Check the Distribution mean

nonspherical_2d_gaussian.mean()
```

    <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0., 0.], dtype=float32)>

```python
# Check the Distribution covariance

nonspherical_2d_gaussian.covariance()
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[1. , 0.6],
           [0.6, 1. ]], dtype=float32)>

```python
# Plot its approximate density contours

x = nonspherical_2d_gaussian.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
sns.jointplot(x1, x2, kind='kde', space=0, color='r');
```

As you can see, the approximate density contours are now elliptical rather than circular. This is because the components of the Gaussian are correlated.

Also note that the marginal distributions (shown on the sides of the plot) are both univariate Gaussian distributions.

## The Cholesky decomposition

In the above example, we defined the lower triangular matrix $L$ and used that to build the multivariate Gaussian distribution. The covariance matrix is easily computed from $L$ as $\Sigma = LL^T$.

The reason that we define the multivariate Gaussian distribution in this way - as opposed to directly passing in the covariance matrix - is that not every matrix is a valid covariance matrix. The covariance matrix must have the following properties:

1. It is symmetric
2. It is positive (semi-)definite

_NB: A symmetric matrix $M \in \mathbb{R}^{d\times d}$ is positive semi-definite if it satisfies $b^TMb \ge 0$ for all nonzero $b\in\mathbb{R}^d$. If, in addition, we have $b^TMb = 0 \Rightarrow b=0$ then $M$ is positive definite._

The Cholesky decomposition is a useful way of writing a covariance matrix. The decomposition is described by this result:

> For every real-valued symmetric positive-definite matrix $M$, there is a unique lower-diagonal matrix $L$ that has  positive diagonal entries for which
>
> \begin{equation}
     LL^T = M
 \end{equation}
> This is called the _Cholesky decomposition_ of $M$.

This result shows us why Gaussian distributions with full covariance are completely represented by the `MultivariateNormalTriL` Distribution.

### `tf.linalg.cholesky`

In case you have a valid covariance matrix $\Sigma$ and would like to compute the lower triangular matrix $L$ above to instantiate a `MultivariateNormalTriL` object, this can be done with the `tf.linalg.cholesky` function.

```python
# Define a symmetric positive-definite matrix

sigma = [[10., 5.], [5., 10.]]
```

```python
# Compute the lower triangular matrix L from the Cholesky decomposition

scale_tril = tf.linalg.cholesky(sigma)
scale_tril
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[3.1622777, 0.       ],
           [1.5811388, 2.738613 ]], dtype=float32)>

```python
# Check that LL^T = Sigma

tf.linalg.matmul(scale_tril, tf.transpose(scale_tril))
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[10.,  5.],
           [ 5., 10.]], dtype=float32)>

If the argument to the `tf.linalg.cholesky` is not positive definite, then it will fail:

```python
# Try to compute the Cholesky decomposition for a matrix with negative eigenvalues

bad_sigma = [[10., 11.], [11., 10.]]

try:
    scale_tril = tf.linalg.cholesky(bad_sigma)
except Exception as e:
    print(e)
```

### What about positive semi-definite matrices?

In cases where the matrix is only positive semi-definite, the Cholesky decomposition exists (if the diagonal entries of $L$ can be zero) but it is not unique.

For covariance matrices, this corresponds to the degenerate case where the probability density function collapses to a subspace of the event space. This is demonstrated in the following example:

```python
# Create a multivariate Gaussian with a positive semi-definite covariance matrix

psd_mvn = tfd.MultivariateNormalTriL(loc=[0., 0.], scale_tril=[[1., 0.], [0.4, 0.]])
psd_mvn
```

    <tfp.distributions.MultivariateNormalTriL 'MultivariateNormalTriL' batch_shape=[] event_shape=[2] dtype=float32>

```python
# Plot samples from this distribution

x = psd_mvn.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
plt.xlim(-5, 5)
plt.ylim(-5, 5)
plt.title("Scatter plot of samples")
plt.scatter(x1, x2, alpha=0.5);
```

If the input to the function `tf.linalg.cholesky` is positive semi-definite but not positive definite, it will also fail:

```python
# Try to compute the Cholesky decomposition for a positive semi-definite matrix

another_bad_sigma = [[10., 0.], [0., 0.]]

try:
    scale_tril = tf.linalg.cholesky(another_bad_sigma)
except Exception as e:
    print(e)
```

In summary: if the covariance matrix $\Sigma$ for your multivariate Gaussian distribution is positive-definite, then an algorithm that computes the Cholesky decomposition of $\Sigma$ returns a lower-triangular matrix $L$ such that $LL^T = \Sigma$. This $L$ can then be passed as the `scale_tril` of `MultivariateNormalTriL`.

## Putting it all together

You are now ready to put everything that you have learned in this reading together.

To create a multivariate Gaussian distribution with full covariance you need to:

1. Specify parameters $\mu$ and either $\Sigma$ (a symmetric positive definite matrix) or $L$ (a lower triangular matrix with positive diagonal elements), such that $\Sigma = LL^T$.

2. If only $\Sigma$ is specified, compute `scale_tril = tf.linalg.cholesky(sigma)`.

3. Create the distribution: `multivariate_normal = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)`.

```python
# Create a multivariate Gaussian distribution

mu = [1., 2., 3.]
sigma = [[0.5, 0.1, 0.1],
         [0.1,  1., 0.6],
         [0.1, 0.6, 2.]]

scale_tril = tf.linalg.cholesky(sigma)

multivariate_normal = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)
```

```python
# Check the covariance matrix

multivariate_normal.covariance()
```

    <tf.Tensor: shape=(3, 3), dtype=float32, numpy=
    array([[0.49999997, 0.1       , 0.1       ],
           [0.1       , 1.0000001 , 0.6       ],
           [0.1       , 0.6       , 2.        ]], dtype=float32)>

```python
# Check the mean

multivariate_normal.mean()
```

    <tf.Tensor: shape=(3,), dtype=float32, numpy=array([1., 2., 3.], dtype=float32)>

## Deprecated: `MultivariateNormalFullCovariance`

There was previously a class called `tfd.MultivariateNormalFullCovariance` which takes the full covariance matrix in its constructor, but this is being deprecated. Two reasons for this are:

* covariance matrices are symmetric, so specifying one directly involves passing redundant information, which involves writing unnecessary code.
* it is easier to enforce positive-definiteness through constraints on the elements of a decomposition than through a covariance matrix itself. The decomposition's only constraint is that its diagonal elements are positive, a condition that is easy to parameterize for.

### Further reading and resources
* https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/MultivariateNormalTriL
* https://www.tensorflow.org/api_docs/python/tf/linalg/cholesky


# Numerical methods for 2nd-order ODEs

We've gone over how to solve 1st-order ODEs using numerical methods, but what about 2nd-order or any higher-order ODEs? We can use the same methods we've already discussed by transforming our higher-order ODEs into a **system of first-order ODEs**.

Meaning, if we are given a 2nd-order ODE
\begin{equation}
\frac{d^2 y}{dx^2} = y^{\prime\prime} = f(x, y, y^{\prime})
\end{equation}
we can transform this into a system of **two 1st-order ODEs** that are coupled:
\begin{align}
\frac{dy}{dx} &= y^{\prime} = u \\
\frac{du}{dx} &= u^{\prime} = y^{\prime\prime} = f(x, y, u)
\end{align}
where $f(x, y, u)$ is the same as that given above for $\frac{d^2 y}{dx^2}$.

Thus, instead of a 2nd-order ODE to solve, we have two 1st-order ODEs:
\begin{align}
y^{\prime} &= u \\
u^{\prime} &= f(x, y, u)
\end{align}

So, we can use all of the methods we have talked about so far to solve 2nd-order ODEs by transforming the one equation into a system of two 1st-order equations.

## Higher-order ODEs

This works for higher-order ODEs too! For example, if we have a 3rd-order ODE, we can transform it into a system of three 1st-order ODEs:
\begin{align}
\frac{d^3 y}{dx^3} &= f(x, y, y^{\prime}, y^{\prime\prime}) \\
\rightarrow y^{\prime} &= u \\
u^{\prime} &= y^{\prime\prime} = w \\
w^{\prime} &= y^{\prime\prime\prime} = f(x, y, u, w)
\end{align}

## Example: mass-spring problem

For example, let's solve a forced damped mass-spring problem given by a 2nd-order ODE:
\begin{equation}
y^{\prime\prime} + 5y^{\prime} + 6y = 10 \sin \omega t
\end{equation}
with the initial conditions $y(0) = 0$ and $y^{\prime}(0) = 5$.

We start by transforming the equation into two 1st-order ODEs. Let's use the variables $z_1 = y$ and $z_2 = y^{\prime}$:
\begin{align}
\frac{dz_1}{dt} &= z_1^{\prime} = z_2 \\
\frac{dz_2}{dt} &= z_2^{\prime} = y^{\prime\prime} = 10 \sin \omega t - 5z_2 - 6z_1
\end{align}

### Forward Euler

Then, let's solve numerically using the forward Euler method. Recall that the recursion formula for forward Euler is:
\begin{equation}
y_{i+1} = y_i + \Delta x f(x_i, y_i)
\end{equation}
where $f(x,y) = \frac{dy}{dx}$.

Let's solve using $\omega = 1$ and with a step size of $\Delta t = 0.1$, over $0 \leq t \leq 3$.

We can compare this against the exact solution, obtainable using the method of undetermined coefficients:
\begin{equation}
y(t) = -6 e^{-3t} + 7 e^{-2t} + \sin t - \cos t
\end{equation}

```matlab
% plot exact solution first
t = linspace(0, 3);
y_exact = -6*exp(-3*t) + 7*exp(-2*t) + sin(t) - cos(t);
plot(t, y_exact); hold on

omega = 1;

dt = 0.1;
t = [0 : dt : 3];

f = @(t,z1,z2) 10*sin(omega*t) - 5*z2 - 6*z1;

z1 = zeros(length(t), 1);
z2 = zeros(length(t), 1);
z1(1) = 0;
z2(1) = 5;
for i = 1 : length(t)-1
    z1(i+1) = z1(i) + dt * z2(i);
    z2(i+1) = z2(i) + dt * f(t(i), z1(i), z2(i));
end

plot(t, z1, 'o--')
xlabel('time'); ylabel('displacement')
legend('Exact', 'Forward Euler', 'Location','southeast')
```

### Heun's Method

For schemes that involve more than one stage, like Heun's method, we'll need to implement both stages for each 1st-order ODE. For example:

```matlab
clear
% plot exact solution first
t = linspace(0, 3);
y_exact = -6*exp(-3*t) + 7*exp(-2*t) + sin(t) - cos(t);
plot(t, y_exact); hold on

omega = 1;

dt = 0.1;
t = [0 : dt : 3];

f = @(t,z1,z2) 10*sin(omega*t) - 5*z2 - 6*z1;

z1 = zeros(length(t), 1);
z2 = zeros(length(t), 1);
z1(1) = 0;
z2(1) = 5;
for i = 1 : length(t)-1
    % predictor
    z1p = z1(i) + z2(i)*dt;
    z2p = z2(i) + f(t(i), z1(i), z2(i))*dt;

    % corrector
    z1(i+1) = z1(i) + 0.5*dt*(z2(i) + z2p);
    z2(i+1) = z2(i) + 0.5*dt*(f(t(i), z1(i), z2(i)) + f(t(i+1), z1p, z2p));
end
plot(t, z1, 'o')
xlabel('time'); ylabel('displacement')
legend('Exact', 'Heuns', 'Location','southeast')
```

### Runge-Kutta: `ode45`

We can also solve using `ode45`, by providing a separate function file that defines the system of 1st-order ODEs. In this case, we'll need to use a single **array** variable, `Z`, to store $z_1$ and $z_2$. The first column of `Z` will store $z_1$ (`Z(:,1)`) and the second column will store $z_2$ (`Z(:,2)`).

```matlab
%%file mass_spring.m
function dzdt = mass_spring(t, z)
    omega = 1;
    dzdt = zeros(2,1);

    dzdt(1) = z(2);
    dzdt(2) = 10*sin(omega*t) - 6*z(1) - 5*z(2);
end
```

    Created file '/Users/kyle/projects/ME373/docs/mass_spring.m'.

```matlab
% plot exact solution first
t = linspace(0, 3);
y_exact = -6*exp(-3*t) + 7*exp(-2*t) + sin(t) - cos(t);
plot(t, y_exact); hold on

% solution via ode45:
[T, Z] = ode45('mass_spring', [0 3], [0 5]);

plot(T, Z(:,1), 'o')
xlabel('time'); ylabel('displacement')
legend('Exact', 'ode45', 'Location','southeast')
```

```matlab

```


# Curvilinear coordinates

Shenfun automates curvilinear coordinates! To this end we make heavily use of [sympy](https://sympy.org) for symbolic mathematics in Python. First some background on curvilinear coordinates:

The position vector $\mathbf{r}$ in three-dimensional Cartesian coordinates is given as

$$
\mathbf{r} = x^1 \mathbf{i}_1 + x^2 \mathbf{i}_2 + x^3 \mathbf{i}_3
$$

where $\mathbf{i}_1, \mathbf{i}_2, \mathbf{i}_3$ are the unit basisvectors for the three spatial dimensions. This is very often written as

$$
\mathbf{r} = x \mathbf{i} + y \mathbf{j} + z \mathbf{k},
$$

but we will use the numbered version below, because it is numbered.

We write for a point $\mathbf{x}$

$$
\mathbf{x} = (x^1, x^2, x^3).
$$

A new curvilinear coordinate system with coordinate curves $q^1=q^1(\mathbf{x}), q^2=q^2(\mathbf{x})$ and $ q^3=q^3(\mathbf{x})$ can be defined, and we can get back to the Cartesian coordinate system through the inverse maps

$$
x^1=x^1(\mathbf{q}) \quad x^2=x^2(\mathbf{q}) \quad x^3=x^3(\mathbf{q}),
$$

using notation

$$
\mathbf{q} = (q^1, q^2, q^3).
$$

The position vector can now be written as a function of the new coordinates

$$
\mathbf{r} = x^i(\mathbf{q}) \,\mathbf{i}_i,
$$

with summation on repeated indices $i$.

For example, for cylindrical coordinates we have

$$
 q^1, q^2, q^3 = r, \theta, z,
$$

where $r, \theta, z$ are the radial position, the azimuthal angle, and the position along the length of the cylinder, respectively. The position vector is given in terms of these new coordinates as

$$
\mathbf{r} = r\cos \theta \,\mathbf{i}_1 + r\sin \theta \,\mathbf{i}_2 + z \,\mathbf{i}_2.
$$

For spherical coordinates we have

$$
q^1, q^2, q^3 = r, \theta, \phi,
$$

and the position vector is

$$
\mathbf{r} = r\sin \theta \cos \phi \,\mathbf{i}_1 + r\sin \theta \sin \phi \,\mathbf{i}_2 + r \cos \theta \,\mathbf{i}_3.
$$

By defining this position vector as a map between curvilinear and Cartesian coordinates, we can now find new basis vectors that point along the curvilinear coordinate curves as

$$
\mathbf{b}_i = \frac{\partial \mathbf{r}}{\partial q^{i}}, \quad \text{for}\, i \, \in (1, 2, 3).
$$

These basis vectors are the covariant basis vectors of the new coordinate system, and they are not normalized. The basis vectors are tangent to the coordinate curves. Along the coordinate curve $q^1$, both $q^2$ and $q^3$ are constant, and likewise for the other two coordinate curves $q^2$ and $q^3$.

Contravariant basis vectors are defined with a superscript index as

$$
\mathbf{b}^{i} = \nabla q^{i}.
$$

The covariant and contravariant basis vectors satisfy

$$
\mathbf{b}^{i} \cdot \mathbf{b}_{j} = \delta_{j}^{i}.
$$

A vector $\mathbf{v}$ can be given in either basis as

$$
\mathbf{v} = v^{i} \mathbf{b}_i = v_i \mathbf{b}^{i}.
$$

where $v_{i}$ and $v^{i}$ are co- and contravariant vector components, respectively.

In shenfun we normally use the (natural) covariant basis vectors. The magnitude of the covariant basis vectors are the scaling factors

$$
h_i = |\mathbf{b}_i|, \quad \text{for}\, i \, \in (1, 2, 3).
$$

The co- and contravariant metric tensors $g_{ij}$ and $g^{ij}$ are defined as

\begin{align*}
g_{ij} &= \mathbf{b}_i \cdot \mathbf{b}_j, \\
g^{ij} &= \mathbf{b}^{i} \cdot \mathbf{b}^{j}. \\
\end{align*}

The determinant of the matrix $G=\{g_{ij}\}_{i,j = 1,2,3}$ is termed $g$

$$
g = \det(G).
$$

The term $\sqrt{g}$ is also commonly called the Jacobian, $J$, which is defined as the determinant of the matrix $\partial x^{i}/ \partial q^j$.

This is really all we need to define the linear operators in the new coordinate system.

The gradient of a scalar $f$ is termed $\nabla f$ and equals

\begin{equation}
\nabla f = \frac{\partial f}{\partial q^{i}}\,\mathbf{b}^{i} = \frac{\partial f}{\partial q^{i}} g^{ij} \,\mathbf{b}_{j}.
\end{equation}

The divergence of a vector $\mathbf{v}$ is termed $\nabla \cdot \mathbf{v}$ and is given as

\begin{equation}
\nabla \cdot \mathbf{v} = \frac{1}{\sqrt{g}} \frac{\partial v^{i} \sqrt{g}}{\partial q^{i}}.
\end{equation}

This leads to the Laplace operator

\begin{equation}
\nabla^2 f = \frac{1}{\sqrt{g}}\frac{\partial}{\partial q^{i}}\left( g^{li} \sqrt{g} \frac{\partial f}{\partial q^{l}}\right).
\end{equation}

and the biharmonic operator

\begin{equation*}
\nabla^4 f= \frac{1}{\sqrt{g}}\frac{\partial}{\partial q^{i}}\left( g^{li}\sqrt{g} \frac{\partial}{\partial q^{l}} \left( \frac{1}{\sqrt{g}}\frac{\partial}{\partial q^{j}}\left( g^{kj}\sqrt{g} \frac{\partial f}{\partial q^{k}}\right) \right)\right).
\end{equation*}

Unless otherwise stated there is summation on repeated indices. So the biharmonic equation can have at most 81 different terms!

Note

Everything needed to do Curvilinear coordinates is achieved by taking derivatives of the position vector $\mathbf{r}$! These derivatives are automated by Shenfun through Sympy.

The Cartesian domain is $\Omega$ and the computational domain is $D$. The computational domain is a straight line, a rectangle or a box, whereas the Cartesian domain can be curved. An integral over the domain is computed with a change of variables as

\begin{equation}
\int_{\Omega} u(\mathbf{x}) d\sigma = \int_{D} u(\mathbf{x}(\mathbf{q})) \sqrt{g} d\mathbf{q}.
\end{equation}

where $d\mathbf{q}=\prod_{i} dq^i$.

Note

I should now probably have introduce transformed variables, like

$$
\tilde{u}(\mathbf{q}) = u(\mathbf{x}(\mathbf{q})).
$$

The spectral Galerkin approximation to the function $u$ in curvilinear coordinates will be

\begin{equation*}
u(\mathbf{x}(\mathbf{q})) = \sum_{i}\sum_{j}\sum_{k} \hat{u}_{ijk} \phi_i(q^1) \psi_j(q^2) \gamma_k(q^3)
\end{equation*}

for some basis functions $\phi, \psi$ and $\gamma$.

## Consider spherical coordinates

$$
\mathbf{r} = r\sin \theta \cos \phi \,\mathbf{i}_1 + r\sin \theta \sin \phi \,\mathbf{i}_2 + r \cos \theta \,\mathbf{i}_3.
$$

```python
from shenfun import *
import sympy as sp
config['basisvectors'] = 'covariant'

r, theta, phi = psi = sp.symbols('x,y,z', real=True, positive=True)
rv = (r*sp.sin(theta)*sp.cos(phi), r*sp.sin(theta)*sp.sin(phi), r*sp.cos(theta))
N = 6
F = FunctionSpace(N, 'F', dtype='d')
L0 = FunctionSpace(N, 'L', domain=(0, 1))
L1 = FunctionSpace(N, 'L', domain=(0, np.pi))
T = TensorProductSpace(comm, (L0, L1, F), coordinates=(psi, rv, sp.Q.positive(sp.sin(theta))))
V = VectorSpace(T)
u = TrialFunction(V)
```

```python
T.coors.get_covariant_basis()
```

Pretty-print

```python
from IPython.display import Math
Math(T.coors.latex_basis_vectors(symbol_names={r: 'r', theta: '\\theta', phi: '\\phi'}))
```

```python
T.coors.get_sqrt_det_g()
```

```python
Math((div(u)).tolatex(funcname='u', symbol_names={r: 'r', theta: '\\theta', phi: '\\phi'}))
```

We now want to make sure that the following vector identity holds

$$
\nabla^2 \mathbf{u} = \nabla( \nabla \cdot \mathbf{u}) - \nabla \times \nabla \times \mathbf{u}
$$

```python
du = div(grad(u))
dv = grad(div(u)) - curl(curl(u))
dv.simplify()
dw = du-dv
dw.simplify()
Math(dw.tolatex(funcname='u', symbol_names={r: 'r', theta: '\\theta', phi: '\\phi'}))
```

which is not bad considering the look of the vector Laplacian:

```python
Math(du.tolatex(funcname='u', symbol_names={r: 'r', theta: '\\theta', phi: '\\phi'}))
```

All of this is automated in Shenfun. You just use the generic operators `div, grad` and `curl` as usual. Non-orthogonal coordinates are possible as well, but not highly tested.

```python

```