# completing the square
> As always, we need our friends.

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy
from sympy import Integer as I, Rational as R, solve, S, N
import plotutils as pu
%matplotlib inline
```

This is just a quick reference on how to solve quadratic equations using [square completion](https://en.wikipedia.org/wiki/Completing_the_square). We'll use `sympy` and `matplotlib` to verify our results.

**Completing the square** is a basic method of solving a quadratic equation. In fact, the [quadratic formula](https://en.wikipedia.org/wiki/Quadratic_formula) can be proven using the method of completing the square.

>$(x + a)^2 = x^2 + 2ax + a^2$

We'll start with a basic *quadratic equation* by writing down a random one: $x^2 + 5x - 18 = 0$

Note that you can write down anything like $ax^2 + bx + c$ and just substitude $a$, $b$ and $c$ with some random numbers and it will work. You might run into negative roots but then you can just involve the [imaginary number](https://en.wikipedia.org/wiki/Imaginary_number) and solve it anyways (for example, see the second example below).

What we are aiming for is to write it in a square form such as $(x + a)^2 = y$ so we can actually solve it by saying $x + a = \sqrt{y}$.

Let's get back to our equation $x^2 + 5x - 18 = 0$. We'll start by adding $18$ to each side:

$$
\begin{align}
x^2 + 5x - 18 & = 0 \\
x^2 + 5x & = 18
\end{align}
$$

Just as a reminder:

> $(x + a)^2 = (x + a) \times (x + a) = x \times x + a \times x + a \times x + a \times a =  x^2 + 2ax + a^2$

In this case $2ax = 5x$ so in order to figure out what $a$ is we need to divide both sides by $2x$. This gives us $a = \frac{5}{2} = 2\frac{1}{2}$. We need to add $a^2 = {2\frac{1}{2}}^2 = \frac{25}{4} = 6\frac{1}{4}$ to *complete the square* and the rules of algebra tell us that we need to add it to each side:

$$
\begin{align}
x^2 + 5x & = 18 \\
x^2 + 5x + 6\frac{1}{4} & = 18 + 6\frac{1}{4} \\
x^2 + 5x + 6\frac{1}{4} & = 24\frac{1}{4}
\end{align}
$$

Now we have a constant number on the right-hand side. It's a rational but that's no problem. At least its positive so we don't need $i$ for this. More importantly though, the left-hand side is in a $x^2 + 2ax + a^2$ configuration. That means we can now rewrite it as $(x + a)^2$.

$$
\begin{align}
x^2 + 5x + 6\frac{1}{4} & = 24\frac{1}{4} \\
(x - 2\frac{1}{2})^2 & = 24\frac{1}{4}
\end{align}
$$

So now that we have a square expression on the left and a constant on the right we can start to formulate an answer:

$$
\begin{align}
(x + 2\frac{1}{2})
= \pm\sqrt{24\frac{1}{4}} \implies x & = -2\frac{1}{2} - \sqrt{24\frac{1}{4}} \cup -2\frac{1}{2} + \sqrt{24\frac{1}{4}} \\
& = -2\frac{1}{2} - \frac{1}{2}\sqrt{97} \cup -2\frac{1}{2} + \frac{1}{2}\sqrt{97}
\end{align}
$$

We can check this using `sympy`.

```python
expr = S('x^2 + 5*x - 18')
solve(expr)
```

    [-5/2 + sqrt(97)/2, -sqrt(97)/2 - 5/2]

Bcause we *can* and because it's kinda fun and easy to plot we'll try to visualize the solution as well.

```python
with plt.xkcd():
    ax = plt.gca()
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['left'].set_position(('data', 0))
    ax.spines['bottom'].set_position(('data', 0))
    ax.xaxis.set_ticks_position('bottom')
    ax.xaxis.set_ticks([-5/2-np.sqrt(97)/2, -5/2+np.sqrt(97)/2])
    ax.yaxis.set_ticks([])
    ax.xaxis.set_ticklabels(['-5/2-sqrt(97)/2', '-5/2+sqrt(97)/2'])
    ax.tick_params(direction='out', width=1.0, length=20.0)
    x = np.linspace(-10, 5, 100)
    y = x**2 + 5*x - 18
    plt.plot(x, y, label='y = x*x + 5*x - 18')
    plt.ylim(-30, 60)
    plt.legend(loc=2)
```

# imaginary example
So let's try another one. Again, we'll just jot down some random quadratic equation: $3x^2 - 3x + 5 = 0$. We want to get rid of that $3$ at the beginning so we'll start by scaling down the whole by a factor of $3$.

$\frac{3x^2 - 3x + 5}{3} = 0 \implies x^2 - x + \frac{5}{3} = 0 $

Now that we have something a bit easier to work with, the next thing we want is to write the left hand side as $x^2 + 2ax + a^2$.

$$
\begin{align}
x^2 - x + \frac{5}{3} & = 0 \\
x^2 - x & = -\frac{5}{3} & \text{subtract $\frac{5}{3}$} \\
x^2 - x + \frac{1}{4} & = -\frac{5}{3} + \frac{1}{4} & \text{add $a^2$ with $a = \frac{1}{2}$} \\
x^2 - x + \frac{1}{4} & = -\frac{17}{12} & \text{simplify} \\
(x - \frac{1}{2})^2 & = -\frac{17}{12} & \text{write as square} \\
x - \frac{1}{2} & = \pm \sqrt{-\frac{17}{12}} & \text{unsquare left-hand side} \\
x & = \frac{1}{2} \pm \sqrt{\frac{17}{12}}i & \text{remove negative root using $i$} \\
x & = \frac{1}{2} \pm \sqrt{51}\frac{i}{6} & \text{simplify}
\end{align}
$$

Let's check if `sympy` agrees with our answer.

```python
solve(S('x^2 - x + 5/3'))
```

    [1/2 - sqrt(51)*I/6, 1/2 + sqrt(51)*I/6]

Looks like we got it. To be honest, I kinda cheated on the last simplify step by using `sympy` to simplify the square root of the rational. Let's make sure our initial answer was correct.

```python
initial = sympy.sqrt(R(17, 12))*sympy.I
final = sympy.sqrt(51)*sympy.I/6
initial == final
```

    True


# Runge Kutta módszerek (Explicit)
Készítette: Sykora Henrik - BME-GPK Műszaki Mechanikai Tanszék (sykora@mm.bme.hu)

Forrás: https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods

### Kezdetiérték feladat (KDE):
$\dot{y}(t) = f(y,t)$

### Runge Kutta módszerek:
Közelítsük a megoldást diszkrét időpontokban ($t_n = nh$):
$y_n \approx y(t_n)$

$\begin{align}
y_{n+1} = y_{n} + h \sum_{i=1}^s b_i k_i,
\end{align}$
ahol
$\begin{align}
k_1 &= \left(t_n,y_n\right), \\
k_2 &= \left(t_n+c_2 h,y_n+\left(a_{21}k_1\right) h\right), \\
k_3 &= \left(t_n+c_3 h,y_n+\left(a_{31}k_1+a_{32}k_2\right)h\right), \\
&\;\vdots \\
k_s &= \left(t_n+c_s h,y_n+\left(a_{s1}k_1+\ldots+a_{s,s-1}k_{s-1}\right)h\right).
\end{align}$

Az $a_i$, $b_i$ és $c_i$ paramétereket az úgynevezett *Butcher tábla* (*Butcher tableau*) tartalmazza:

$\begin{equation}
\renewcommand\arraystretch{1.2}
\begin{array}
{c|cccc}
0\\
c_2 & a_{2,1}\\
c_3 & a_{3,1} & a_{3,2} \\
\vdots & \vdots& & \ddots \\
c_s & a_{s,1} & a_{s,2} & \dots & a_{s,1}\\
\hline
& b_1 & b_2 & \dots & b_{s-1} & b_s
\end{array}
\end{equation}$

#### Egyszerű esetek:
Euler módszer:
$\begin{equation}
\renewcommand\arraystretch{1.2}
\begin{array}
{c|cccc}
0\\
\hline
& 1
\end{array}
\end{equation}$
$\begin{align}
y_{n+1} &= y_{n} + h  \cdot 1 \cdot k_1,\\
k_1 &= f(t_n, y_n).
\end{align}$

Negyedrendű Runge-Kutta módszer:
$\begin{equation}
\renewcommand\arraystretch{1.2}
\begin{array}
{c|cccc}
0\\
\frac{1}{2} & \frac{1}{2}\\
\frac{1}{2} &0 &\frac{1}{2} \\
1& 0& 0& 1\\
\hline
& \frac{1}{6} &\frac{1}{3} &\frac{1}{3} &\frac{1}{6}
\end{array}
\end{equation}$
$\begin{align}
y_{n+1} &= y_{n} + \frac{h}{6} \left(k_1+2k_2 + 2k_3 + k_4\right),\\
k_1 &= \left(t_n,y_n \right), \\
k_2 &= \left(t_n+\frac{1}{2} h, y_n+\frac{1}{2}k_1 h\right), \\
k_3 &= \left(t_n+\frac{1}{2} h, y_n+\frac{1}{2}k_2 h\right), \\
k_4 &= \left(t_n+ h, y_n+k_3 h\right).
\end{align}$

#### Megjegyzés: Adaptív Runge-Kutta módszerek
$\begin{equation}
\renewcommand\arraystretch{1.2}
\begin{array}
{c|cccc}
0\\
c_2 & a_{2,1}\\
c_3 & a_{3,1} & a_{3,2} \\
\vdots & \vdots& & \ddots \\
c_s & a_{s,1} & a_{s,2} & \dots & a_{s,1}\\
\hline
& b_1 & b_2 & \dots & b_{s-1} & b_s\\
& b_1^\ast & b_2^\ast & \dots & b_{s-1}^\ast & b_s^\ast
\end{array}
\end{equation}$
Hibafüggvény ami alapján eldöntjük, hogy kell-e felezni/duplázni $h$ értékét:
$e_{n+1} = y_n+1-y^\ast_n+1 = h \sum_{i=s}^s\left(b_i-b_i^\ast\right)k_i$

# Implementáció

```python
# Számításokhoz és az eredmények megjelenítéséhez szükséges csomagok behívása
import numpy as np
import matplotlib.pyplot as mpl
# Segédfüggvény
def getzero(ll):
  return np.zeros(len(ll))

# Runge Kutta lépés osztály
class RKstep:
  def __init__(self,a,b,c,h):
    self.a = a
    self.b = b
    self.c = c
    self.h = h
    self.nk = len(c)

  def dy(self,f,t,y): # f(t_n,y_n) -> k
      ks=[getzero(y) for i in range(self.nk)]
      for (i,ci) in enumerate(self.c):
        ks[i]= f(t+ci*self.h,y + self.h*sum(self.a[i][j]*ks[j] for j in range(i)))
      return self.h*sum(k*self.b[i] for (i,k) in enumerate(ks))

# RK4 módszer
def RK4(h):
  a=[[],[1/2],[0,1/2],[0,0,1/2]]
  b=[1/6,1/3,1/3,1/6]
  c=[0,1/2,1/2,1]
  return RKstep(a,b,c,h)

# Euler módszer
def Euler(h):
  a=[[]]
  b=[1.]
  c=[0.]
  return RKstep(a,b,c,h)

# Közönséges differenciálegyenlet kezdetiérték problémájának megadása/megoldásfüggvénye osztályként
class ODEProblem:
  def __init__(self,f,y0,tspan):
    self.f = f
    self.y0 = y0 if type(y0)==np.ndarray else np.array(y0)
    self.tspan = tspan

  def solve(self,method):
    t0,T = self.tspan
    ts = np.arange(t0,T+100*np.finfo(float).eps,method.h)
    ys=[self.y0 for (i,t) in enumerate(ts)]
    for i in range(len(ys)-1):
      ys[i+1] = ys[i] + method.dy(self.f,ts[i],ys[i])
    return (ts,ys)
```

# Alkalmazás

Pl.:
$\dot{y}(t) = y(t), \quad y(0) = y_0, \quad t \in \left[0,T\right]$

```python
# Függvény és ODEProblem megadása
foo = lambda t,y: y#*np.sin(4*t)
y0=1.
T=3.
prob=ODEProblem(foo,[y0],(0.,T));

# Lépésköz + numerikus megoldás
h=0.1
# Runge Kutta megoldás
ts,ys=prob.solve(RK4(h))
#Euler módszer megoldás
tsE,ysE=prob.solve(Euler(h))

# Ábrázolás
mpl.plot(ts,np.exp(ts))
mpl.scatter(ts,[y[0] for y in ys],color='r',s=10)
mpl.scatter(tsE,[y[0] for y in ysE],color='g',s=10)
mpl.legend(['Pontos','RK4','Euler'])
mpl.show()
```


# Marginal likelihood for Bayesian linear regression

Author: [Zeel B Patel](https://patel-zeel.github.io/), [Nipun Batra](https://nipunbatra.github.io/)

Bayesian linear regression is defined as below,

\begin{align}
\mathbf{y} &= X\boldsymbol{\theta} + \epsilon\\
\epsilon &\sim \mathcal{N}(0, \sigma_n^2)\\
\theta &\sim \mathcal{N}(\mathbf{m}_0, S_0)
\end{align}

For a Gaussian random variable $\mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}, \Sigma)$, $A\mathbf{z} + \mathbf{b}$ is also a Gaussian random variable.

\begin{align}
\mathbf{y} = X\mathbf{\theta} + \boldsymbol{\epsilon} &\sim \mathcal{N}(\boldsymbol{\mu}', \Sigma')\\
\boldsymbol{\mu}' &= \mathbb{E}_{\theta, \epsilon}(X\mathbf{\theta}+\boldsymbol{\epsilon})\\
                  &= X\mathbb{E}(\mathbf{\theta}) + \mathbb{E}(\mathbf{\epsilon})\\
                  &= X\mathbf{m}_0\\
                  \\
\Sigma' &= V(X\mathbf{\theta}+\boldsymbol{\epsilon})\\
        &= XV(\mathbf{\theta})X^T+V(\boldsymbol{\epsilon})\\
        &= XS_0X^T + \sigma_n^2I
\end{align}

Marginal likelihood is $p(\mathbf{y})$ so,

\begin{align}
p(\mathbf{y}) &= \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma'|^{\frac{1}{2}}}\exp\left[-\frac{1}{2}(\mathbf{y}-\boldsymbol{\mu}')^T\Sigma'^{-1}(\mathbf{y}-\boldsymbol{\mu}')\right]\\
              &= \frac{1}{(2\pi)^{\frac{N}{2}}|XS_0X^T + \sigma_n^2I|^{\frac{1}{2}}}\exp\left[-\frac{1}{2}(\mathbf{y}-X\mathbf{m}_0)^T(XS_0X^T + \sigma_n^2I)^{-1}(\mathbf{y}-X\mathbf{m}_0)\right]
\end{align}

## Multiplication of two Gaussians (work in progress)

We need Gaussian pdf over same variables to evaluate their multiplication. Let us convert $y$ into $\theta$.

\begin{align}
\mathbf{y} &= X\theta + \boldsymbol{\epsilon}\\
\theta &= (X^TX)^{-1}X^T(\mathbf{y} - \boldsymbol{\epsilon})\\
\text{Deriving mean and covariance of }\theta\\
E(\theta) &= (X^TX)^{-1}X^T\mathbf{y}\\
V(\theta) &= \sigma_n^2\left[(X^TX)^{-1}X^T\right]\left[(X^TX)^{-1}X^T\right]^T\\
          &= \sigma_n^2(X^TX)^{-1}X^TX(X^TX)^{-1}\\
          &= \sigma_n^2(X^TX)^{-1}
\end{align}

Now, we have both $p(\mathbf{y}|\boldsymbol{\theta})$ and $p(\boldsymbol{\theta})$ in terms of $\boldsymbol{\theta}$. We can apply the rules from 6.5.2 of MML book. Writing our results in terminology of 6.5.2.

\begin{align}
\mathcal{N}(x|a, A) &== \mathcal{N}(\theta|(X^TX)^{-1}X^T\mathbf{y}, \sigma_n^2(X^TX)^{-1})\\
\mathcal{N}(x|b, B) &== \mathcal{N}(\theta|\mathbf{m}_0, S_0)
\end{align}

we know that,

$$
c\mathcal{N}(\theta|\mathbf{c}, C) = \mathcal{N}(x|a, A)\mathcal{N}(x|b, B)\\
\mathcal{N}(\theta|\mathbf{c}, C) = \frac{\mathcal{N}(x|a, A)\mathcal{N}(x|b, B)}{c}
$$

In the Bayesian setting,

\begin{align}
Prior &\sim \mathcal{N}(x|b, B) == \mathcal{N}(\theta|\mathbf{m}_0, S_0)\\
Likelihood &\sim \mathcal{N}(x|a, A) == \mathcal{N}(\theta|(X^TX)^{-1}X^T\mathbf{y}, \sigma_n^2(X^TX)^{-1})\\
Posterior &\sim \mathcal{N}(\theta|\mathbf{c}, C) == \mathcal{N}(\theta|\mathbf{m}_n, S_n)\\
\text{last but not the least}\\
Marginal\;likelihood &\sim c == \mathcal{N}(\mathbf{y}|\boldsymbol{\mu}, \Sigma)
\end{align}

Let us evaluate the posterior,

\begin{align}
Posterior &\sim \mathcal{N}(\theta|\mathbf{c}, C)\\
S_n = C &= (A^{-1} + B^{-1})^{-1}\\
  &= \left(\frac{X^TX}{\sigma_n^2} + S_0^{-1}\right)^{-1}\\
\mathbf{m_n} = \mathbf{c} &= C(A^{-1}a + B^{-1}b)\\
           &= S_n\left(\frac{X^TX}{\sigma_n^2}(X^TX)^{-1}X^T\mathbf{y} + S_0^{-1}\mathbf{m}_0\right)\\
           &= S_n\left(\frac{X^T\mathbf{y}}{\sigma_n^2} + S_0^{-1}\mathbf{m}_0\right)
\end{align}

Now, we evaluate the marginal likelihood,

\begin{align}
c &= \mathcal{N}(\mathbf{y}|\boldsymbol{\mu}, \Sigma)\\
  &= (2\pi)^{-\frac{D}{2}}|A+B|^{-\frac{1}{2}}\exp\left(-\frac{1}{2}(a-b)^T(A+B)^{-1}(a-b)\right)\\
  &= (2\pi)^{-\frac{D}{2}}|\sigma_n^2(X^TX)^{-1}+S_0|^{-\frac{1}{2}}\exp\left(-\frac{1}{2}((X^TX)^{-1}X^T\mathbf{y}-\mathbf{m}_0)^T(\sigma_n^2(X^TX)^{-1}+S_0)^{-1}((X^TX)^{-1}X^T\mathbf{y}-\mathbf{m}_0)\right)
\end{align}

Another well-known formulation of marginal likelihood is the following,

$$
p(\mathbf{y}) \sim \mathcal{N}(X\mathbf{m}_0, XS_0X^T + \sigma_n^2I)
$$

Let us verify if both are the same, empirically,

```python
import numpy as np
import scipy.stats
np.random.seed(0)

def ML1(X, y, m0, S0, sigma_n):
    N = len(y)
    return scipy.stats.multivariate_normal.pdf(y.ravel(), (X@m0).squeeze(), X@S0@X.T + np.eye(N)*sigma_n**2)

def ML2(X, y, m0, S0, sigma_n):
    D = len(m0)
    a = np.linalg.inv(X.T@X)@X.T@y
    b = m0
    A = np.linalg.inv(X.T@X)*sigma_n**2
    B = S0
    return scipy.stats.multivariate_normal.pdf(a.ravel(), b.ravel(), A+B)

def ML3(X, y, m0, S0, sigma_n):
    N = len(y)
    Sn = np.linalg.inv((X.T@X)/(sigma_n**2) + np.linalg.inv(S0))
    Mn = Sn@((X.T@y)/(sigma_n**2) + np.linalg.inv(S0)@m0)
    LML = -0.5*N*np.log(2*np.pi) - 0.5*N*np.log(sigma_n**2) - 0.5*np.log(np.linalg.det(S0)/np.linalg.det(Sn)) - 0.5*(y.T@y)/sigma_n**2 + 0.5*(Mn.T@np.linalg.inv(Sn)@Mn)
    return np.exp(LML)

X = np.random.rand(10,2)
m0 = np.random.rand(2,1)
s0 = np.random.rand(2,2)
S0 = s0@s0.T
sigma_n = 10
y = np.random.rand(10,1)

ML1(X, y, m0, S0, sigma_n), ML2(X, y, m0, S0, sigma_n),  ML3(X, y, m0, S0, sigma_n)
```

    (9.577110083272389e-15, 0.0034284478634232078, array([[2.08309892e-14]]))

### Products of Gaussian PDFs (Work under progress)

Product of two Gaussians $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}_0, \Sigma_0)$ and $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}_1, \Sigma_1)$ is an unnormalized Gaussian.

\begin{align}
f(\mathbf{x}) &= \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma_0|^{\frac{1}{2}}}\exp\left[-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_0)^T\Sigma_0^{-1}(\mathbf{x}-\boldsymbol{\mu}_0)\right]\\
g(\mathbf{x}) &= \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma_1|^{\frac{1}{2}}}\exp\left[-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_1)^T\Sigma_1^{-1}(\mathbf{x}-\boldsymbol{\mu}_1)\right]\\
\int h(x) = \frac{1}{c}\int f(\mathbf{x})g(\mathbf{x})d\mathbf{x} &= 1
\end{align}

We need to find figure out value of $c$ to solve the integration.

\begin{align}
h(x) &= \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma|^{\frac{1}{2}}}\exp\left[-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})\right] =  \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma|^{\frac{1}{2}}}\exp\left[-\frac{1}{2}\left(\mathbf{x}^T\Sigma^{-1}\mathbf{x} - 2\boldsymbol{\mu}^T\Sigma^{-1}\mathbf{x} + \boldsymbol{\mu}^T\Sigma^{-1}\boldsymbol{\mu}\right)\right]\\
f(x)g(x) &= \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma_0|^{\frac{1}{2}}(2\pi)^{\frac{N}{2}}|\Sigma_1|^{\frac{1}{2}}}\exp\left[
-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_0)^T\Sigma_0^{-1}(\mathbf{x}-\boldsymbol{\mu}_0)
-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_1)^T\Sigma_1^{-1}(\mathbf{x}-\boldsymbol{\mu}_1)\right]\\
         &= \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma_0|^{\frac{1}{2}}(2\pi)^{\frac{N}{2}}|\Sigma_1|^{\frac{1}{2}}}\exp\left[
-\frac{1}{2}\left(\mathbf{x}^T(\Sigma_0^{-1}+\Sigma_1^{-1})\mathbf{x}- 2\boldsymbol{\mu}^T(\Sigma_0^{-1}+\Sigma_1^{-1})\mathbf{x} + \boldsymbol{\mu}^T(\Sigma_0^{-1}+\Sigma_1^{-1})\boldsymbol{\mu}\right)
\right]\\
\end{align}

We can compare the exponent terms directly. We get the following results by doing that

\begin{align}
\Sigma^{-1} &= \Sigma_0^{-1} + \Sigma_1^{-1}\\
\Sigma &= \left(\Sigma_0^{-1} + \Sigma_1^{-1}\right)^{-1}\\
\\
\boldsymbol{\mu}^T\Sigma^{-1}\mathbf{x} &= \boldsymbol{\mu_0}^T\Sigma_0^{-1}\mathbf{x} + \boldsymbol{\mu_1}^T\Sigma_1^{-1}\mathbf{x}\\
\left(\boldsymbol{\mu}^T\Sigma^{-1}\right)\mathbf{x} &= \left(\boldsymbol{\mu_0}^T\Sigma_0^{-1} + \boldsymbol{\mu_1}^T\Sigma_1^{-1}\right)\mathbf{x}\\
\boldsymbol{\mu}^T\Sigma^{-1} &= \boldsymbol{\mu_0}^T\Sigma_0^{-1} + \boldsymbol{\mu_1}^T\Sigma_1^{-1}\\
\text{Applying transpose on both sides,}\\
\Sigma^{-1}\boldsymbol{\mu} &= \Sigma_0^{-1}\boldsymbol{\mu}_0 + \Sigma_1^{-1}\boldsymbol{\mu}_1\\
\boldsymbol{\mu} &= \Sigma\left(\Sigma_0^{-1}\boldsymbol{\mu}_0 + \Sigma_1^{-1}\boldsymbol{\mu}_1\right)
\end{align}

Now, solving for the normalizing constant $c$,

\begin{align}
\frac{c}{(2\pi)^{\frac{N}{2}}|\Sigma|^{\frac{1}{2}}} &= \frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma_0|^{\frac{1}{2}}(2\pi)^{\frac{N}{2}}|\Sigma_1|^{\frac{1}{2}}}\\
c &=  \frac{|\Sigma|^{\frac{1}{2}}}{(2\pi)^{\frac{N}{2}}|\Sigma_0|^{\frac{1}{2}}|\Sigma_1|^{\frac{1}{2}}}
\end{align}

If we have two Gaussians $\mathcal{N}(\mathbf{a}, A)$ and $\mathcal{N}(\mathbf{b}, B)$ for same random variable $\mathbf{x}$, Marginal likelihood can be given as,

$$
c = (2\pi)^{-N/2}|A+B|^{-1/2}\exp -\frac{1}{2}\left[(\mathbf{a} - \mathbf{b})^T(A+B)^{-1}(\mathbf{a} - \mathbf{b})\right]
$$

Here, we have two Gaussians $\mathcal{N}(0, \sigma^2I)$ and $\mathcal{N}((X^TX)^{-1}X^T\mathbf{y}, \frac{(X^TX)^{-1}}{\sigma_n^2} )$ for same random variable $\boldsymbol{\theta}$, Marginal likelihood can be given as,

$$
$$

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
```

```python
np.random.seed(0)
N = 10
D = 5
sigma_n = 0.1 # noise
sigma = 1 # variance in parameters
m0 = np.random.rand(D)
S0 = np.eye(D)*sigma**2

x = np.random.rand(N,D)
theta = np.random.rand(D,1)
y = x@theta + np.random.multivariate_normal(np.zeros(N), np.eye(N)*sigma_n**2, size=1).T
plt.scatter(x[:,0], x[:,1], c=y)
x.shape, theta.shape, y.shape
```

```python
a = np.linalg.inv(x.T@x)@x.T@y
b = m0.reshape(-1,1)
A = np.linalg.inv(x.T@x)/(sigma_n**2)
B = S0
A_inv = np.linalg.inv(A)
B_inv = np.linalg.inv(B)

c_cov = np.linalg.inv(A_inv + B_inv)
c_mean = c_cov@(A_inv@a + B_inv@b)
a.shape, A.shape, b.shape, B.shape, c_mean.shape, c_cov.shape
```

    ((5, 1), (5, 5), (5, 1), (5, 5), (5, 1), (5, 5))

```python
c_denom = 1/(((2*np.pi)**(D/2))*(np.linalg.det(c_cov)**0.5))
b_denom = 1/(((2*np.pi)**(D/2))*(np.linalg.det(B)**0.5))
a_denom = 1/(((2*np.pi)**(D/2))*(np.linalg.det(A)**0.5))
a_denom, b_denom, c_denom, 1/c_denom
```

    (1.5040129154541655e-07,
     0.010105326013811642,
     0.0110028525380197,
     90.88552232655665)

```python
normalizer_c = (1/(((2*np.pi)**(D/2))*(np.linalg.det(A+B)**0.5)))*np.exp(-0.5*((a-b).T@np.linalg.inv(A+B)@(a-b)))
norm_c_a_given_b = scipy.stats.multivariate_normal.pdf(a.squeeze(), b.squeeze(), A+B)
norm_c_b_given_a = scipy.stats.multivariate_normal.pdf(b.squeeze(), a.squeeze(), A+B)
normalizer_c, norm_c_a_given_b, norm_c_b_given_a, 1/normalizer_c
```

    (array([[1.35765194e-07]]),
     1.357651942204283e-07,
     1.357651942204283e-07,
     array([[7365658.08152844]]))

```python
a_pdf = scipy.stats.multivariate_normal.pdf(theta.squeeze(), a.squeeze(), A)
b_pdf = scipy.stats.multivariate_normal.pdf(theta.squeeze(), b.squeeze(), B)
c_pdf = scipy.stats.multivariate_normal.pdf(theta.squeeze(), c_mean.squeeze(), c_cov)

a_pdf, b_pdf, c_pdf, np.allclose(a_pdf*b_pdf, normalizer_c*c_pdf)
```

    (1.5039199356435742e-07, 0.008635160418150373, 0.00956547808509135, True)

```python
K = x@S0@x.T + np.eye(N)*sigma_n**2
marginal_Likelihood_closed_form = scipy.stats.multivariate_normal.pdf(y.squeeze(), (x@m0).squeeze(), K)
marginal_Likelihood_closed_form, 1/normalizer_c
```

    (1.8288404157840938, array([[7365658.08152844]]))

```python
from sklearn.model_selection import KFold
from sklearn.linear_model import LinearRegression

splitter = KFold(n_splits=100)
for train_ind, test_ind in splitter(x):
    train_x, train_y = x[train_ind], y[train_ind]
    test_x, test_y = x[test_ind], y[test_ind]
    model = LinearRegression()
    model.fit(train_x, train_y)

```

## What is the relationship between marginal_Likelihood_closed_form and any calculations done in multiplications of two gaussians?


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## Graded Homework 1

For this homework, we'll be using Jupyter notebooks for a few problems:
- Plotting functions (this notebook)
- Unit conversions
- Significant figures
---

Before you turn this problem in, make sure everything runs as expected. First, restart the kernel (in the menubar, select Kernel → Restart & Run All) and then check to make sure you don't see any error messages (which will stop the "Run All" command.

Make sure you fill in any place that says YOUR CODE HERE or "YOUR ANSWER HERE", as well as your name and collaborators (i.e., anyone you discussed this with) below:

```python
NAME = ""
COLLABORATORS = ""
```

```python
# First, let's import some modules for plotting functions
# import sympy and start a session
from sympy import init_session
init_session()

# insert any graphs into our notebooks directly
%matplotlib inline
# high resolution plots
%config InlineBackend.figure_format = 'retina'

from sympy.plotting import plot
```

### Part A (2 points)

The [Michaelis-Menten](https://en.wikipedia.org/wiki/Michaelis%E2%80%93Menten_kinetics) equation models the kinetics of enzymatic reactions, e.g.

$$
S \rightarrow P
$$

$$
v = \frac{d[P]}{dt} = \frac{V_{max} [S]}{K_m + [S]}
$$

where $v$ is the rate of the reaction, converting substrate S to product, $V_{max}$ is the maximum rate, and thhe constant $K_m$ is the substrate concentration at whhich the reaction rate is at half its maximum value.

Plot $v$ as a function of $[S]$ with $K_m = 0.04 M$ and $V_{max} = 0.1 M/s$.

```python
# use "x" as the variable instead of [S]

## BEGIN SOLUTION

## END SOLUTION

plot(v, xlim=(0.0, 3.0), ylim=(0,0.5))
```

## Part B (3 points)

The [**sinc** function](https://en.wikipedia.org/wiki/Sinc_filter) is used frequently in audio and signal processing, particularly to remove high-frequency noise.

$$
\mathrm{sinc} \; x = \frac{\sin \pi x}{\pi x}
$$

Plot **sinc x** from -4.0 to 4.0:

```python
## BEGIN SOLUTION

## END SOLUTION
```

**Concept** Is the $\mathrm{sinc}$ function an even or odd function? Explain.

YOUR ANSWER HERE:
## BEGIN SOLUTION
## END SOLUTION

## Part C (3 points)

Consider the function:

$$f(x) = \ln\left(\frac{1}{\cos^2 x} \right)$$

Since this will be periodic, we'll need to plot across a wide range, say -20 to +20.

Plot the function:

```python
## BEGIN SOLUTION

## END SOLUTION
```

**Concept:** Explain what happens to the function at $x = \pm \pi$, $\pm 2\pi$, etc.

YOUR ANSWER HERE:
## BEGIN SOLUTION
## END SOLUTION

### Part D: Significant Figures (3 points)

I know, it's everyone's favorite 😆 topic...

Make sure to print all three of the following calculations, with rounding to produce the right number of significant figures:

```python
from sigfig import round

# add and subtract some numbers (e.g., masses)
example_1 = 17.13+14.6751+3.123+7.654−8.123

# find the volume of a box:
# length is given as 7.78 m
# width is given as 3.486 m
# and whose height is 1.367 m
volume =

# find the pressure of a gas with PV = nRT
# V is 0.200 m**3,
# the temperature T is 298.15 K,
# and the amount of gas n is 1.000 mol
V =
T =
n =
R = 8.31446261815324 # m**3 • Pa (pascals) / (mol•K)
pressure = n * R * T / V # you can leave the pressure in Pa

## BEGIN SOLUTION
## END SOLUTION
```

### Polar 2D Coordinates (4 points)

As a reminder, we sometimes need to convert between x, y, Cartesian 2D coordinates and polar r, $\theta$ coordinates.

Please interconvert the following x, y coordinates and r, $\theta$ coordinates:

```python
x1 = 2
y1 = 10
r1 =
theta1 =

x2 = -2
y2 = -5
r2 =
theta2 =

r3 = 1.0
theta3 = 0.52
x3 =
y3 =

r4 = 5.0
theta = 5.87
x4 =
y4 =
```

**Concept**: What are the ranges of possible r and theta values?

R:
Theta:

## Spherical Coordinates (5 Points)

Much like in 2D XY coordinates, we sometimes need to interconvert 3D cartesian and spherical coordinates in chemistry.

Please interconvert XYZ to r, $\theta$, $\varphi$ or vice-versa

```python
x = 1.0
y = 2.0
z = 3.0
r =
theta =
phi =

r2 = 1.0
theta2 = 2.5 # radians
phi2 = -1.0 # radians
x2 =
y2 =
z2 =
```

**Concept**: What are the ranges of possible r, theta, and phi values?

R:

Theta:

Phi:


# Homework 4 - Hard Coding

## Group #12: Vishal Matta, Michele Meo, Giulio Rago

# Hashing Task

### Import data and libraries

```python
import os
import time
import math
import matplotlib.pyplot as plt
import numpy as np
from scipy import stats
```

We import the **hashing_lib** module that contains all the functions used to solve the hashing task:
you can find it in the **utils** folder of the repository.
It is suggested to read the script every time a function imported from it is used to better understand the code.

```python
from utils import hashing_lib as hs
```

We import the data in two differents lists, then we delete the last element as it contains an empty string.
We find out that the passwords to be stored in the filter are 100000000 and the passwords we want
to compare with the filter, once we stored the first set of passwords, are 39000000.

```python
with open ("/Users/Michele/PycharmProjects/ADM-HW4/DATA/passwords1.txt", "r") as file:
    passwords1 = list(file.read().split("\n"))

with open ("/Users/Michele/PycharmProjects/ADM-HW4/DATA/passwords2.txt", "r") as file:
    passwords2 = list(file.read().split("\n"))

passwords1.pop()
passwords2.pop();
```

```python
print(len(passwords1))
print(len(passwords2))
```

    100000000
    39000000

### Preliminary considerations

Given the false positive rate we are willing to have, $p$, and the number of items that will be stored in filter, $n$,
the website https://hackernoon.com/probabilistic-data-structures-bloom-filter-5374112a7832  provides us some
quantitative methods to estimate the needed size of the filter, $m$, and the number of hash functions to be applied, $k$, to have the required $p$.
In particular, we have chosen to accept a rate of false positives:

\begin{equation}
p = 0.01  -->  p = 1\%.
\end{equation}

Given $n=100000000$, we estimate the size of the filter and the number of needed hash functions with the following formulas:

\begin{equation}
m \approx -{{nlnp}\over{(ln2)^2}}
\end{equation}

\begin{equation}
k \approx {{m}\over{n}}ln2
\end{equation}

as we do in the following code cell.

```python
p = 0.01
n = len(passwords1)
m = int(-(n*math.log(p))/(math.log(2)**2))
k = round((m/n)*math.log(2))
print("We choose a rate of false positives of about", p, ": given the number of passwords "
      "to be registered in the filter,", n, ", we need at least a filter of size", m, "and", k,
      "different hash functions.")
```

    We choose a rate of false positives of about 0.01 : given the number of passwords to be registered in the filter, 100000000 , we need at least a filter of size 958505837 and 7 different hash functions.

Given the estimate obtained for the filter size, we choose a size $m$ equal to the first power of 2 higher than such estimate.
For this choice of size the number of necessary hash functions does not change with respect to the previous estimate and remains $k = 7$; moreover, by choosing a larger size we are more confident about possible collisions.

```python
m = 2**30
```

### Hash Function discussion

The code for our own hash function is written in the function *hash_function* in the **hashing_lib.py** script.
Here we describe the concept behind our hash function and, to be clearer, we apply it by hand on a string example '/DXhI;GNl*nbCzKso=?@'.
Given an index of the string, $i$, the first prime number smaller than $m$, ${prime}_0$, and another prime number near the size of the filter, $prime$, the proposed hash function does (in the example we choose $i=1$):
- takes the i-th-last character of the password and converts it to ascii code, in the example @-->64;
- takes the last character of the ascii number obtained before, in the example 64-->4;
- takes the (10+i)-th-last characters of the passwords, converts them in their ascii code and pick only the last numbers for each ascii number obtained, in the example:
@-->64-->4; ?-->63-->3; =-->61-->1; o-->111-->1 .......... G-->71-->1;
- joins all the numbers obtained by the previous step in a string and converts this string in an int number:
43115527802881
- converts the i-th character of the password in its ascii code, in the example /-->47, then apply the following formula to this number and the number obtained the previous step, in the example:
\begin{equation}
(43115527802881\cdot{prime}+47)\;\textrm{mod}({prime}_0)
\end{equation}
and we have our hash value given the password.

With this method it is possible to obtain different hash functions by changing the $i$ and $prime$ parameters described above: we do it in the *multi_hash_function* in **hashing_lib** module and to evaluate prime numbers near the size of the filter we use the funcion *LastPrimes* in the same module.
The fact of considering a number of different numbers based on the ascii code of the i-th-last character, allows us to change the order of magnitude of the hash value for each different passowrd.
Furthermore, always considering only the last number of the ascii conversion allows us to obtain a uniform distribution.

### Testing our Hash Function

Let's create a BloomFilter object of size $m$ (read the script **hashing_lib.py**).

```python
bloom_filter = hs.BloomFilter(m)
```

To minimize the probability of having collisions, the hash function must generate values uniformly distributed between 0 and the filter size.
Below, we verify that the values calculated on the first 1000000 passwords are uniformly distributed.

```python
hash_values = bloom_filter.testing_hash_function(passwords1[:1000000])
```

```python
fig = plt.figure(figsize=(8,5))
plt.xlabel("Hash Values")
plt.ylabel("Frequency")
plt.hist(hash_values, bins=100)
plt.show()
```

### Bloom Filter implementation

We apply 7 different hash functions to the set of passwords1 to store them on our filter, then we apply the same 7 hash functions to the set of passwords2 to verify which of these are already stored in the filter.
Furthermore, as requested, we calculate the execution time: this could be higher than the one strictly necessary to store passwords1 in the filter and check passwords2 because we also store all the passwords that are duplicated in a list (for a possible bonus task).

```python
start = time.time()

bloom_filter.multi_hash_functions(passwords1, 7)
positives = bloom_filter.hash_searching(passwords2, 7)

end = time.time()
```

Finally we can save the probable duplicates in a txt file and evaluate the execution time of our algorithm.

```python
with open('duplicates.txt', 'w') as f:
    for item in bloom_filter.duplicates:
        f.write("%s\n" % item)
```

```python
run_time = int(end-start)
h = run_time/(60*60)
m = (h-int(h))*60
s = int((m-int(m))*60)
```

```python
h, m = int(h), int(m)
```

Some characteristic quantities of our Bloom Filter are the following.

```python
print("The number of hash functions we have used is:", k)
print("The number of possible duplicates that we found is:", positives)
print("The probability of finding a false positive is:", p)
print("An estimate for the number of true duplicates is:", int(positives-positives*p))
print("The execution time of our algorithm is about:", run_time, "s (", h,"h",m,"m",s,"s)")
```

    The number of hash functions we have used is: 7
    The number of possible duplicates that we found is: 14145218
    The probability of finding a false positive is: 0.01
    An estimate for the number of true duplicates is: 14003765
    The execution time of our algorithm is about: 12651 s ( 3 h 30 m 51 s)

# Alphabetical Sort

## Build your own implementation of Counting Sort. If it's based (based, not copied!) on some reference on Internet, please cite it.

### I saw a implementation of this algorith in https://www.geeksforgeeks.org/counting-sort/, our algorith takes a list of numbers, we have to store an array of lenght equals of the maximun value of the list.

```python

def Counting_Sort(arr):

    #we store two arrays of lenght equals to the max value of the input list
    output = [0] * (max(arr)+1)

    count = [0] * (max(arr)+1)

    ans = [0] * len(arr)

    #for each number we count the occurrence and insert it into the i-th cell
    for i in arr:
        count[i] += 1
    #we fill cells with the sum of the previous cells
    for i in range(max(arr)+1):
        count[i] += count[i-1]

    #insert in the output vector the numbers in the respective cells
    #(obviously the occurrences are respected)
    for i in range(len(arr)):
        output[count[arr[i]]-1] = arr[i]
        count[arr[i]] -= 1

    #the ans vector is the final one
    #with a length equal to that of the initial vector(the output vector have in the final a lot of 0)
    for i in range(len(arr)):
        ans[i] = output[i]
    return ans[1:]

```

```python
import random
l = [random.randrange(0,1000) for i in range(100)]

```

## Build an algorithm, based on your implementation of Counting Sort, that receives in input a list with all the letters of the alphabet (not in alphabetical order), and returns the list ordered according to alphabetical order. Discuss time complexity (theoretically and empirically).

Very similar with the first one, we used the unicode indices of these characters to get a range on which to work the counting sort

```python
def List_of_letters(arr):
    arr = list(map(str.lower,arr))

    output = [0] * 256

    count = [0] * 256

    ans = [0] * len(arr)

    for i in arr:
        count[ord(i)] += 1

    for i in range(256):
        count[i] += count[i-1]

    for i in range(len(arr)):
        output[count[ord(arr[i])]-1] = arr[i]
        count[ord(arr[i])] -= 1

    new_idx = [0]*len(arr)
    for i in range(len(arr)):
        ans[i] = output[i]
    return ans

```

```python
l = ['g','s','y','r','t','y','h','s','f','a','e']
List_of_letters(l)
```

    ['a', 'e', 'f', 'g', 'h', 'r', 's', 's', 't', 'y', 'y']

## This is a variation of the second exercise, this function return the index of the character in the original list, we need this aux function fro the third exercise becouse we want to remember the initial index

```python
def List_of_letters_aux(arr):
    arr = list(map(str.lower,arr))

    output = [0] * 256

    count = [0] * 256

    ans = [0] * len(arr)
    index = ["0"]*(256) # This list will be useful for part 3, in order to retrieve the initial position of each letter in a list of words

    for i in arr:
        count[ord(i)] += 1

    for i in range(256):
        count[i] += count[i-1]

    for i in range(len(arr)):
        output[count[ord(arr[i])]-1] = arr[i]
        index[count[ord(arr[i])]-1] = i
        count[ord(arr[i])] -= 1

    new_idx = [0]*len(arr)
    for i in range(len(arr)):
        ans[i] = output[i]
        new_idx[i] = index[i]
    return new_idx

```

## Build an algorithm, based on your implementation of Counting Sort, that receives in input a list of length m, that contains words with maximum length equal to n, and returns the list ordered according to alphabetical order. Discuss time complexity (theoretically and empirically).

```python
def Counting_Sort_word(l):

    #if the list have only one element return it
    if len(l) < 2:
        return l

    elif all(len(set([l[j][0] for j in range(len(l))]))==len(l) for j in range(len(l))):

        first = [l[j][0] for j in range(len(l))]
        return([l[i] for i in List_of_letters_aux(first)])

    #trasform the list with only lower character
    array = list(map(str.lower,l))

    #We create two list,one with the input list, the other with the trasformed one
    head, *tail = array
    h_old, *t_old = l

    sort_1 = []
    sort_2 = []

    for k in range(len(tail)):
        i = 0
        if tail[k][0] == head[i] and len(tail[k]) > 1:
            # If the characters in the head is the same of in tail we dont need to sort them
            while i<len(tail[k])-1 and i<len(head)-1 and tail[k][i] == head[i]:
                i+=1

        if i == len(tail[k])-1 and i != len(head)-1:
            sort_1.append(t_old[k])
        elif i == len(head)-1 and i != len(tail[k])-1:
            sort_2.append(t_old[k])

        else:
            tosort = [tail[k][i],head[i]]

            # We sort the two words according to the first different character

            ind = List_of_letters_aux(tosort)

            sortedl = [tosort[u] for u in ind]

            # if tosort == sorted1 means that the two lists are sorted
            if tosort == sortedl :
                sort_1.append(t_old[k])
            else:
                sort_2.append(t_old[k])

    less = Counting_Sort_word(sort_1)
    more = Counting_Sort_word(sort_2)
    return less + [h_old] + more

```

```python
l1 = input("Enter a list of words: ").split(" ")
```

    Enter a list of words: cas grwe gweds g wejuy ngfdsa ddgtr

```python
Counting_Sort_word(l1)
```

    ['g', 'cas', 'ddgtr', 'grwe', 'gweds', 'ngfdsa', 'wejuy']

# Complexity

In this exercise we have developed an algorithm that can sort a list of words using what was seen in the counting_sort, the power of the Counting sort is in fact the possibility to order a vector of objects without having to compare them.
In the first two exercise we can sort the two list in linear time; now for sorting a list of words is much more complex.
Let's start by analyzing the algorithm, obviously the best case is when all the words have the first different letters, in this case we only use one time the __aux Function__.
the peggire case occurs when all the words have the same characters.

# Find similar wines

### Import data and libraries

```python
import pandas as pd
import math
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
```

We import the **clustering_lib** module that contains all the functions used to implement the k-means algorithm: you can find it in the **utils** folder of the repository. It is suggested to read the script every time a function imported from it is used to better understand the code.

```python
from utils import clustering_lib as clust
```

We import the dataset and we label the features as described in the website of the dataset.

```python
features = ["Class", "Alcohol", "Malic acid", "Ash", "Alcalinity of ash", "Magnesium", "Total phenols", "Flavanoids",
            "Nonflavanoid phenols", "Proanthocyanins", "Color intensity", "Hue", "OD280/OD315 of diluted wines",
            "Proline"]
```

```python
data = pd.read_csv("/Users/Michele/PycharmProjects/ADM-HW4/DATA/wine.data", names = features)
```

We delete the "class" column because it doesn't contain a wine feature, it's only a pre-existing classification.
Then we look at the data.

```python
data.drop(columns=["Class"], inplace=True)
data.head()
```

      Alcohol
      Malic acid
      Ash
      Alcalinity of ash
      Magnesium
      Total phenols
      Flavanoids
      Nonflavanoid phenols
      Proanthocyanins
      Color intensity
      Hue
      OD280/OD315 of diluted wines
      Proline

      0
      14.23
      1.71
      2.43
      15.6
      127
      2.80
      3.06
      0.28
      2.29
      5.64
      1.04
      3.92
      1065

      1
      13.20
      1.78
      2.14
      11.2
      100
      2.65
      2.76
      0.26
      1.28
      4.38
      1.05
      3.40
      1050

      2
      13.16
      2.36
      2.67
      18.6
      101
      2.80
      3.24
      0.30
      2.81
      5.68
      1.03
      3.17
      1185

      3
      14.37
      1.95
      2.50
      16.8
      113
      3.85
      3.49
      0.24
      2.18
      7.80
      0.86
      3.45
      1480

      4
      13.24
      2.59
      2.87
      21.0
      118
      2.80
      2.69
      0.39
      1.82
      4.32
      1.04
      2.93
      735

The data have 13 features, so they are points (or vectors) that live in a 13 dimensional space.

```python
features = ["Alcohol", "Malic acid", "Ash", "Alcalinity of ash", "Magnesium", "Total phenols", "Flavanoids",
            "Nonflavanoid phenols", "Proanthocyanins", "Color intensity", "Hue", "OD280/OD315 of diluted wines",
            "Proline"]
```

### Pre-Processing of the data

We preferred to work on a numpy matrix structure instead of a pandas dataframe.
So we convert the dataframe in a numpy array and we standardize the data.

```python
data = np.array(data)
data_matrix = StandardScaler().fit_transform(data)
```

### K-means implementation

We create a Kmeans object, choosing a number of 3 clusters because, due to experience gained on the code, this is the best choice to obtain a nice classification (read the script **clustering_lib.py**).

```python
C = clust.Kmeans(3, len(features))
C.Cluster
```

    {1: [], 2: [], 3: []}

We initialize the centroids randomly, choosing for each of their components a number between the maximum and the minimum value taken from each feature of the data (each column of the standardized data frame).
We do it with the random_centers function defined in the Kmeans class; then we can evaluate the clusters on this first random choice.

```python
C.random_centers(data_matrix)
C.clustering(data_matrix)
```

We define a *cost_function_values* numpy array to store the value of the cost function for each iteration.
Since the algorithm always converges in a few steps (always less than 10), we can create a 100 components array, that are more than enough.

```python
cost_function_values = np.zeros(100)
```

Now we want to improve the classification, so we implement the kmean algorithm step by step, updating the centroids with the geometric means of each cluster.
To not fix a number of steps chosen, we use a while loop in which the algorithm stops when the result becomes stable, that is when the difference between the cost function calculated at the current iteration and the previous one is less than a specific threshold, in our case 0.1.

```python
up = 0

while (up <= 1) or (abs(cost_function_values[up-1]-cost_function_values[up-2]) > 0.1):
    cost_function_values[up] = C.cost_function()
    C.update_centers()
    C.clustering(data_matrix)
    up += 1
```

The best classification that we can obtain with this algorithm is shown below through the data graphs on 3 different pairs of features.
From the graphs it is clear that the clusters are well separated from each other; if some points seem to overlap the clusters, it is probably because they are actually distant in other dimensions.
However this algorithm does not always lead to this good solution: this is better discussed in exercise 4.

```python
fig = plt.figure(figsize=(9,6))

for i in range(C.K):
    cluster = np.array(C.Cluster[i+1])
    plt.scatter(cluster[:,6], cluster[:,7], s=45, label="cluster "+str(i+1))
plt.scatter(C.C[:,6], C.C[:,7], c="black", s=200)
plt.xlabel("Flavanoids (standard.)")
plt.ylabel("Nonflavanoid phenols (standard.)")
plt.legend()
plt.show()
```

```python
fig = plt.figure(figsize=(9,6))

for i in range(C.K):
    cluster = np.array(C.Cluster[i+1])
    plt.scatter(cluster[:,6], cluster[:,8], s=45, label="cluster "+str(i+1))
plt.scatter(C.C[:,6], C.C[:,8], c="black", s=200)
plt.xlabel("Flavanoids (standard.)")
plt.ylabel("Proanthocyanins (standard.)")
plt.legend()
plt.show()
```

```python
fig = plt.figure(figsize=(9,6))

for i in range(C.K):
    cluster = np.array(C.Cluster[i+1])
    plt.scatter(cluster[:,0], cluster[:,2], s=45, label="cluster "+str(i+1))
plt.scatter(C.C[:,0], C.C[:,2], c="black", s=200)
plt.xlabel("Alcohol (standard.)")
plt.ylabel("Ash (standard.)")
plt.legend()
plt.show()
```

# K-means can go wrong

We run another time the k-means algorithm code (written in a short and not commented way), only to show one case where our k-means implementation can go wrong.
In the graph shown below, it's clear that this time k-means doesn't provide us a good classification and a valid separation among the data.
This happens because of the strong dependence of the k-means from the initial condition: in this case, the centroids have been randomly chosen too close together and this means that, step by step, one of these two clusters will prevail over the other, leaving it with few points or worse with no point.

```python
C = clust.Kmeans(3, len(features))

C.random_centers(data_matrix)
C.clustering(data_matrix)

cost_function_values = np.zeros(100)

up = 0

while (up <= 1) or (abs(cost_function_values[up-1]-cost_function_values[up-2]) > 0.1):
    cost_function_values[up] = C.cost_function()
    C.update_centers()
    C.clustering(data_matrix)
    up += 1
```

```python
fig = plt.figure(figsize=(9,6))

for i in range(C.K):
    cluster = np.array(C.Cluster[i+1])
    plt.scatter(cluster[:,6], cluster[:,7], s=45, label="cluster "+str(i+1))
plt.scatter(C.C[:,6], C.C[:,7], c="black", s=200)
plt.xlabel("Flavanoids (standard.)")
plt.ylabel("Nonflavanoid phenols (standard.)")
plt.legend()
plt.show()
```

The __K-means algorithm__ is one of the most popular clustering algorithms however its operation is heavily influenced by the initialization, therefore a correct initialization can reduce the computational cost of the algorithm.
So to see if a better initialization works, we use k-means++ to choose the initial values (or seeds) for the k-means clustering algorithm.

```python
import pandas as pd
import numpy as np
import time
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
```

```python
# Import the dataset
df = pd.read_csv("../Crime2001.csv")
```

    C:\Users\Giulio\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py:3058: DtypeWarning: Columns (21) have mixed types. Specify dtype option on import or set low_memory=False.
      interactivity=interactivity, compiler=compiler, result=result)

### We need a big DataFrame, we can use the dataframe used during the 28/11 lab with information about Chicago crimes

```python
#we need only the latitude and the longitude to verify our hypothesis
col_n = ['Longitude', 'Latitude']
df = pd.DataFrame(df,columns=col_n)
#We drop the null values
#and those smaller than 91 to avoid errors (same thing used during the lesson of the laboratory)
df = df.dropna(0)
df.drop( df[ df['Longitude'] < -91].index , inplace=True)
```

### In this case used the random init

```python
start = time.time()

# Create our k-mean model with random init
model = KMeans(n_clusters = 10, init = 'random')

# Fit our data in the model
model.fit(df[['Latitude', 'Longitude']])

centroids = model.cluster_centers_

fig = plt.figure(figsize = (15,10))
plt.scatter(df['Latitude'],df['Longitude'])
plt.scatter(centroids[:,0], centroids[:,1], marker='x')

plt.show()

time_rand = time.time() - start
print('Rand time: ', time_rand)

```

```python
start = time.time()

# Create our k-mean model with k-means++ init
model1 = KMeans(n_clusters = 10, init = 'k-means++')

# Fit our data in the model
model1.fit(df[['Latitude', 'Longitude']])

centroids1 = model.cluster_centers_

fig1 = plt.figure(figsize = (15,10))
plt.scatter(df['Latitude'],df['Longitude'])
plt.scatter(centroids1[:,0], centroids1[:,1], marker='x')
plt.show()

time_k = time.time() - start
print('K-means++ time: ', time_k)
```

## Conclusion

Using the k-means++ we have a speedup of:

```python
print(100*(1-time_k/time_rand), " %")
```

    14.158383999815705  %

```python

```


```python
import numpy as np
import scipy.signal as sl
import scipy.linalg as la
import pandas as pd
from autocorr import acf

from scipy.fftpack import fft, ifft, dct, idct

from matplotlib import pyplot as plt
from matplotlib.colors import to_rgba

%matplotlib inline

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
```

```python
def figure(figsize=(10, 5)):
    plt.figure(figsize=figsize)
```

```python
c1 = '#2b2a2b'
c2 = '#863234'
c3 = '#9f847b'
```

# Correlation Function

\begin{align}
     C_{AB}(\tau) \equiv \langle A B \rangle(\tau)
         = \lim_{t_0 \rightarrow \infty} ~ \frac{1}{t_0} \int_0^{t_0} \mathrm d t ~
             A (t + \tau) ~ B (t)
\end{align}

```python
def correlate(f1, f2, window=True):
    t0 = min(len(f1), len(f2))

    corr = np.zeros(t0)

    for t in range(t0):
        corr[t] = f1[t : t0] @ f2[: t0 - t]
        # normalize
        corr[t] /= (t0 - t)

    if window:
        # apply window function
        wd = np.hanning(2 * t0)
        corr *= wd[t0:]

    return corr
```

```python
# Reference data: fast + slow oscillation

t_max = 20
dt = 0.01
w1 = 1 / 1
w2 = 1/ 13

A1 = 1
A2 = 0.33

t = np.arange(0, t_max+dt, dt)
N = len(t)
```

```python
# noise
rand = np.random.RandomState(0)

noise = (1 * rand.randn(len(t)) * ( 1 - np.exp(- t / 10)))
```

```python
f = A1 * np.cos(2*np.pi * w1 * t) + A2 * np.cos(2*np.pi * w2 * t) + noise

figure()
plt.plot(t, f)
```

```python
# mean and variance
print(f'{f.mean():.5f} +- {f.var():.5f}')
```

    -0.01435 +- 0.90553

```python
C = correlate(f, f, window=False)
C2 = np.correlate(f, f, 'full')[len(f)-1:] / len(f)

C2s = sl.correlate(f, f)[len(f)-1:]
# Normalize
C2s /= np.arange(len(f), 0, -1)
# window
# C2s *= np.hanning(2 * len(f))[len(f):]
C2s *= sl.windows.hann(2 * len(f))[len(f):]

# Yair's correlation function (includes proper normalization and windowing and)
C3 = acf(f[:, None], lag=len(f))

fig, (ax1, ax2, ax3, ax4, ax5) = plt.subplots(nrows=5, sharex=True)

ax1.plot(t, C)
ax2.plot(t, C2)
ax3.plot(t, C2s)
ax4.plot(t, C3)
ax5.plot(t, (C - C2))
ax5.plot(t, (C - C2s))
ax5.set_ylim(0.1 * np.array([-1, 1]))
```

```python
np.arange(len(f), 0, -1)
```

    array([2002, 2001, 2000, ...,    3,    2,    1])

```python
ref_C = correlate(f, f)

for c in (C2, C2s, C3):
    print(la.norm(ref_C - c) < 1e-12)
```

    False
    True
    True

# Spectral Density / Power Spectrum

\begin{align}
     S(\omega)
         = \left\vert ~
         \int_{-\infty}^{\infty} \mathrm d t ~ f(t) ~ \mathrm e^{- \mathrm i \omega t}
         ~ \right\vert^2
\end{align}

```python
k = np.linspace(0, 1 / dt, N)

S = abs(fft(f))**2
```

```python
tm = 200
plt.plot(k[:tm], S[:tm], color=c1)
plt.axvline(w1, color=c2, alpha=0.75)
plt.axvline(w2, color=c2, alpha=0.75)
```

# Wiener Khintchine Theorem

\begin{align}
     S(\omega)
         = \int_{-\infty}^{\infty} \mathrm d \tau ~
             C(\tau) ~ \mathrm e^{- \mathrm i \omega \tau}
         = 2 \int_{0}^{\infty} \mathrm d \tau ~
             C(\tau) ~ \cos(\omega \tau)
\end{align}

```python
k = np.linspace(0, 1 / 2 / dt, N)
```

```python
SWK = 2 * dct(C2s)
```

```python
tm = 200
plt.plot(k[:tm], SWK[:tm], color=c1)
plt.axvline(w1, color=c2, alpha=0.75)
plt.axvline(w2, color=c2, alpha=0.75)
```

\begin{align}
     C(\tau)
         = \frac{1}{2 \pi} \int_{-\infty}^{\infty} \mathrm d \omega ~
             S(\omega) ~ \cos(\omega \tau)
\end{align}

```python
C3 = idct(S)
C4 = idct(SWK)
```

```python
fig, (ax1, ax2) = plt.subplots(nrows=2, sharex=True)
ax1.plot(C3)
ax2.plot(C4)
```

# Telegraph Equation
\begin{align}
     X(t) = (-1)^{N(t)}~,
\end{align}

where $N(t)$ is the number of events generated by a Poisson process in the time interval $[0, t)$.

A Poisson process can be modelled as and infinitely dense series of coin tosses with a probability

$$ P(\text{Head}) = \lambda \mathrm d t $$
to flip a head.

The probability of tossing $n$ times a head within $[0, t)$ is then given by

$$ P(N=n, t) = \frac{(\lambda t)^n ~ \mathrm{e}^{-\lambda t}}{n!}~.$$

See https://www.probabilitycourse.com/chapter11/11_1_2_basic_concepts_of_the_poisson_process.php

```python
lam = 1

nt = 10000
ts = np.linspace(0, 500, nt)
dt = ts[1] - ts[0]
dt, lam * dt
```

    (0.05000500050005, 0.05000500050005)

```python
sample = np.zeros(nt)

rand = np.random.RandomState(1)
rands = rand.rand(nt)
sample[rands < lam*dt] = 1
sample.sum()
```

    499.0

```python
signal = (-1) ** sample.cumsum()
```

```python
plt.plot(ts, signal)
plt.xlim([0, 20])
```

```python
%%time
C_telegraph = correlate(signal, signal)
```

    CPU times: user 201 ms, sys: 308 µs, total: 202 ms
    Wall time: 50.5 ms

```python
%%time
C_telegraph = np.correlate(signal, signal, mode='full')[nt-1:] / nt
```

    CPU times: user 99.6 ms, sys: 0 ns, total: 99.6 ms
    Wall time: 24.8 ms

```python
%%time
C_telegraph = sl.correlate(signal, signal)[nt-1:] / nt
```

    CPU times: user 8.34 ms, sys: 0 ns, total: 8.34 ms
    Wall time: 1.84 ms

```python
fig, ax = plt.subplots()

ax.plot(ts, np.exp(-  2 * ts * lam), linewidth=3)
ax.plot(ts, C_telegraph, '.')
ax.set_xlim([0, 5 * 1 / lam])
ax.axvline(1 / lam)

ax.legend([r'$\exp(- \lambda \tau)$', r'$C (\tau)$', '1 / $\lambda$'])
ax.set_xlabel(r'$\tau$')

plt.show()
```


# Introduction to Deep Learning with PyTorch

In this notebook, you'll get introduced to [PyTorch](http://pytorch.org/), a framework for building and training neural networks. PyTorch in a lot of ways behaves like the arrays you love from Numpy. These Numpy arrays, after all, are just tensors. PyTorch takes these tensors and makes it simple to move them to GPUs for the faster processing needed when training neural networks. It also provides a module that automatically calculates gradients (for backpropagation!) and another module specifically for building neural networks. All together, PyTorch ends up being more coherent with Python and the Numpy/Scipy stack compared to TensorFlow and other frameworks.

## Neural Networks

Deep Learning is based on artificial neural networks which have been around in some form since the late 1950s. The networks are built from individual parts approximating neurons, typically called units or simply "neurons." Each unit has some number of weighted inputs. These weighted inputs are summed together (a linear combination) then passed through an activation function to get the unit's output.

Mathematically this looks like:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$

With vectors this is the dot/inner product of two vectors:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

## Tensors

It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.

With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network.

```python
# First, import PyTorch
import torch
```

```python
def activation(x):
    """ Sigmoid activation function

        Arguments
        ---------
        x: torch.Tensor
    """
    return 1/(1+torch.exp(-x))
```

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 5 random normal variables
features = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(features)
# and a true bias term
bias = torch.randn((1, 1))
```

Above I generated data we can use to get the output of our simple network. This is all just random for now, going forward we'll start using normal data. Going through each relevant line:

`features = torch.randn((1, 5))` creates a tensor with shape `(1, 5)`, one row and five columns, that contains values randomly distributed according to the normal distribution with a mean of zero and standard deviation of one.

`weights = torch.randn_like(features)` creates another tensor with the same shape as `features`, again containing values from a normal distribution.

Finally, `bias = torch.randn((1, 1))` creates a single value from a normal distribution.

PyTorch tensors can be added, multiplied, subtracted, etc, just like Numpy arrays. In general, you'll use PyTorch tensors pretty much the same way you'd use Numpy arrays. They come with some nice benefits though such as GPU acceleration which we'll get to later. For now, use the generated data to calculate the output of this simple single layer network.
> **Exercise**: Calculate the output of the network with input features `features`, weights `weights`, and bias `bias`. Similar to Numpy, PyTorch has a [`torch.sum()`](https://pytorch.org/docs/stable/torch.html#torch.sum) function, as well as a `.sum()` method on tensors, for taking sums. Use the function `activation` defined above as the activation function.

```python
## Calculate the output of this network using the weights and bias tensors
y = activation(torch.sum(features*weights)+bias)
```

You can do the multiplication and sum in the same operation using a matrix multiplication. In general, you'll want to use matrix multiplications since they are more efficient and accelerated using modern libraries and high-performance computing on GPUs.

Here, we want to do a matrix multiplication of the features and the weights. For this we can use [`torch.mm()`](https://pytorch.org/docs/stable/torch.html#torch.mm) or [`torch.matmul()`](https://pytorch.org/docs/stable/torch.html#torch.matmul) which is somewhat more complicated and supports broadcasting. If we try to do it with `features` and `weights` as they are, we'll get an error

```python
>> torch.mm(features, weights)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-15d592eb5279> in <module>()
----> 1 torch.mm(features, weights)

RuntimeError: size mismatch, m1: [1 x 5], m2: [1 x 5] at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:2033
```

As you're building neural networks in any framework, you'll see this often. Really often. What's happening here is our tensors aren't the correct shapes to perform a matrix multiplication. Remember that for matrix multiplications, the number of columns in the first tensor must equal to the number of rows in the second column. Both `features` and `weights` have the same shape, `(1, 5)`. This means we need to change the shape of `weights` to get the matrix multiplication to work.

**Note:** To see the shape of a tensor called `tensor`, use `tensor.shape`. If you're building neural networks, you'll be using this method often.

There are a few options here: [`weights.reshape()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.reshape), [`weights.resize_()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.resize_), and [`weights.view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view).

* `weights.reshape(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)` sometimes, and sometimes a clone, as in it copies the data to another part of memory.
* `weights.resize_(a, b)` returns the same tensor with a different shape. However, if the new shape results in fewer elements than the original tensor, some elements will be removed from the tensor (but not from memory). If the new shape results in more elements than the original tensor, new elements will be uninitialized in memory. Here I should note that the underscore at the end of the method denotes that this method is performed **in-place**. Here is a great forum thread to [read more about in-place operations](https://discuss.pytorch.org/t/what-is-in-place-operation/16244) in PyTorch.
* `weights.view(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)`.

I usually use `.view()`, but any of the three methods will work for this. So, now we can reshape `weights` to have five rows and one column with something like `weights.view(5, 1)`.

> **Exercise**: Calculate the output of our little network using matrix multiplication.

```python
## Calculate the output of this network using matrix multiplication
y = activation(torch.mm(features,weights.view(5,1))+bias)
```

### Stack them up!

That's how you can calculate the output for a single neuron. The real power of this algorithm happens when you start stacking these individual units into layers and stacks of layers, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.

The first layer shown on the bottom here are the inputs, understandably called the **input layer**. The middle layer is called the **hidden layer**, and the final layer (on the right) is the **output layer**. We can express this network mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated

$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 3 random normal variables
features = torch.randn((1, 3))

# Define the size of each layer in our network
n_input = features.shape[1]     # Number of input units, must match number of input features
n_hidden = 2                    # Number of hidden units
n_output = 1                    # Number of output units

# Weights for inputs to hidden layer
W1 = torch.randn(n_input, n_hidden)
# Weights for hidden layer to output layer
W2 = torch.randn(n_hidden, n_output)

# and bias terms for hidden and output layers
B1 = torch.randn((1, n_hidden))
B2 = torch.randn((1, n_output))
```

> **Exercise:** Calculate the output for this multi-layer network using the weights `W1` & `W2`, and the biases, `B1` & `B2`.

```python
## Your solution here
h = activation(torch.mm(features, W1)+B1)
output = activation(torch.mm(h, W2)+B2)
print(output)
```

    tensor([[0.3171]])

If you did this correctly, you should see the output `tensor([[ 0.3171]])`.

The number of hidden units a parameter of the network, often called a **hyperparameter** to differentiate it from the weights and biases parameters. As you'll see later when we discuss training a neural network, the more hidden units a network has, and the more layers, the better able it is to learn from data and make accurate predictions.

## Numpy to Torch and back

Special bonus section! PyTorch has a great feature for converting between Numpy arrays and Torch tensors. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method.

```python
import numpy as np
a = np.random.rand(4,3)
a
```

    array([[0.07475861, 0.18944136, 0.18681591],
           [0.61698615, 0.56980633, 0.70194791],
           [0.19777638, 0.53466202, 0.11567961],
           [0.57604775, 0.88436445, 0.84050182]])

```python
b = torch.from_numpy(a)
b
```

    tensor([[0.0748, 0.1894, 0.1868],
            [0.6170, 0.5698, 0.7019],
            [0.1978, 0.5347, 0.1157],
            [0.5760, 0.8844, 0.8405]], dtype=torch.float64)

```python
b.numpy()
```

    array([[0.07475861, 0.18944136, 0.18681591],
           [0.61698615, 0.56980633, 0.70194791],
           [0.19777638, 0.53466202, 0.11567961],
           [0.57604775, 0.88436445, 0.84050182]])

```python
# Numpy array matches new values from Tensor
a
```

    array([[0.07475861, 0.18944136, 0.18681591],
           [0.61698615, 0.56980633, 0.70194791],
           [0.19777638, 0.53466202, 0.11567961],
           [0.57604775, 0.88436445, 0.84050182]])

```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(2)
```

    tensor([[0.1495, 0.3789, 0.3736],
            [1.2340, 1.1396, 1.4039],
            [0.3956, 1.0693, 0.2314],
            [1.1521, 1.7687, 1.6810]], dtype=torch.float64)

The memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well.

intro-to-pytorch/Part 1 - Tensors in PyTorch (Exercises).ipynb


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
# IMG_FILEPATH = "../../01_deep-learning-from-scratch/images/04_extensions/"
```

# Tanh and derivative

```python
a = np.arange(-5, 5, 0.01)
plt.plot(a, np.tanh(a))
plt.title("Tanh activation")
plt.xlabel("$x$")
plt.ylabel("$Tanh(x)$");
# plt.savefig(IMG_FILEPATH + "05_tanh_activation.png")
```

```python
(a[501] - a[500]) / 0.01
```

    0.9999999999999787

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

a = np.arange(-5, 5, 0.01)
plt.plot(a, sigmoid(a));
```

```python
def relu(x):
    return np.array([el if el > 0 else 0 for el in x])

a = np.arange(-5, 5, 0.01)
plt.plot(a, relu(a))
plt.title("ReLU activation function")
plt.xlabel("X")
plt.ylabel("$ReLU(x)$");
# plt.savefig(IMG_FILEPATH + "04_relu_activation.png")
```

```python
a = np.arange(-5, 5, 0.01)
plt.figure(figsize=(8,6))
plt.plot(a, sigmoid(a) * (1-(sigmoid(a))))
plt.plot(a, 1 - (np.tanh(a) **2))
plt.legend(['Derivative of $sigmoid(x)$',
            'Derivative of $Tanh(x)$'])
plt.title("Derivatives of $sigmoid(x)$ and $Tanh(x)$")
plt.xlabel("x");
# plt.savefig(IMG_FILEPATH + "06_sigmoid_tanh_derivative.png")
```

# Cross entropy/softmax

$$ \text{CE}(p_i, y_i) = - y_i * \text{log}(p_i) - (1 - y_i) * \text{log}(1-p_i) $$

$$ CE(x, 0) = \sum_{i}{(- y_i * log(p_i) - (1 - y_i) * log(1-p_i))} $$

$$ \begin{align} = &\sum_{i}{(- 0 * log(p_i) - (1 - 0) * log(1-p_i))} \\
=  &\sum_{i}{- log(1-p_i)} \end{align}$$

$$ CE(x, 1) = \sum_{i}{(- y_i * log(p_i) - (1 - y_i) * log(1-p_i))} $$

$$ \begin{align} = &\sum_{i}{(- 1 * log(p_i) - (1 - 1) * log(1-p_i))} \\
=  &\sum_{i}{- log(p_i)} \end{align}$$

$$ \text{Normalize}(\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix})  = \begin{bmatrix} \frac{x_1}{x_1 + x_2 + x_3} \\
\frac{x_2}{x_1 + x_2 + x_3} \\
\frac{x_3}{x_1 + x_2 + x_3}
\end{bmatrix} $$

$$ \text{Softmax}(\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix})  = \begin{bmatrix} \frac{e^{x_1}}{e^{x_1} + e^{x_2} + e^{x_3}} \\
\frac{e^{x_2}}{e^{x_1} + e^{x_2} + e^{x_3}} \\
\frac{e^{x_3}}{e^{x_1} + e^{x_2} + e^{x_3}}
\end{bmatrix} $$

$$ S(\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \vdots \\ x_n \end{bmatrix})  = \begin{bmatrix} \frac{e^{x_1}}{e^{x_1} + e^{x_2} + e^{x_3} + \ldots + e^{x_n}} \\
\frac{e^{x_2}}{e^{x_1} + e^{x_2} + e^{x_3} + \ldots + e^{x_n}} \\
\frac{e^{x_3}}{e^{x_1} + e^{x_2} + e^{x_3} + \ldots + e^{x_n}} \\
\ldots \\
\frac{e^{x_n}}{e^{x_1} + e^{x_2} + e^{x_3} + \ldots + e^{x_n}}
\end{bmatrix} $$

$$ {SCE}_1 = - y_1 * log(\frac{e^{x_1}}{e^{x_1} + e^{x_2} + e^{x_3}}) - (1 - y_1) * log(1-\frac{e^{x_1}}{e^{x_1} + e^{x_2} + e^{x_3}}) $$

$$ \text{softmax}(\begin{bmatrix} p_1 \\ p_2 \\ p_3 \end{bmatrix}) - \begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} $$

# Plot of log loss when $y_i = 0$

```python
x = np.linspace(0.01, 0.99, 99)
y1 = -1.0 * np.log(1 - x)
y2 = (x - 0) ** 2
plt.plot(x, y1);
plt.plot(x, y2);
plt.legend(['Cross entropy loss', 'Mean squared error'])

plt.title("Cross entropy loss vs. MSE when $y = 0$")
plt.xlabel("Prediction ($p$)")
plt.ylabel("Loss values");
# plt.savefig(IMG_FILEPATH + "04_Log_loss_vs_MSE_y_eq_0.png")
```

When $y=0$, loss can become theoretically infinite as $p$ approaches 1.

# Plot of loss when $y_i = 1$

```python
x = np.linspace(0.01, 0.99, 99)
y = -1.0 * np.log((x))

plt.plot(x, y);
plt.title("Log loss for $y = 1$")
plt.xlabel("Prediction ($p$)")
plt.ylabel("Log loss");
```

# Derivation of Softmax Cross Entropy derivative (simple case, two probabilities)

$$ C(p, y) = - y * \text{log}(p) - (1 - y) * \text{log}(1-p) $$

$$
C(p,y)=
\begin{cases}
-log(1-p) & \text{if }  y = 0\\
-log(p) & \text{if }  y = 1
\end{cases}
$$

$$ \text{SCE}(\begin{bmatrix} p_1 \\ p_2 \end{bmatrix}, \begin{bmatrix} y_1 \\ y_2 \end{bmatrix})_1 = - y_1 * log(S(\begin{bmatrix} p_1 \\ p_2 \end{bmatrix})_1) - (1 - y_1) * log(1-S(\begin{bmatrix} p_1 \\ p_2 \end{bmatrix})_1) = - y_1 * log(\frac{e^{p_1}}{e^{p_1} + e^{p_2}}) - (1 - y_1) * log(1-\frac{e^{p_1}}{e^{p_1} + e^{p_2}}) $$

Structure:

$ y_1 = a $, $e^{x_2} = b$

$$ SC(x_1) = - a * log(\frac{e^{x_1}}{e^{x_1} + b}) - (1 - a) * log(1-\frac{e^{x_1}}{e^{x_1} + b}) $$

**Quotient rule:**

$$ f(x) = \frac{g(x)}{h(x)} $$

$$ f'(x) = \frac{g'(x) * h(x) - g(x) * h'(x)}{(h(x))^2} $$

If

$$ f(x) = \frac{e^x}{e^x + b} $$

$$ \begin{align} f'(x) =& \frac{e^x * (e^x + b) - (e^x * e^x)}{(e^x + b)^2} \\
=& \frac{e^x * (e^x + b - e^ x)}{(e^x + b)^2} \\
=& \frac{e^x * b}{(e^x + b)^2}\end{align} $$

And if:

$$ g(x) = - a * log(f(x)) - (1 - a) * log(1-f(x)) $$

then:

$$ g'(x) = - a * \frac{f'(x)}{f(x)} - (1 - a) * \frac{-1 * f'(x)}{1-f(x)} $$

First, we'll compute $\frac{f'(x)}{f(x)}$:

$$ \begin{align}
\frac{f'(x)}{f(x)} =& \frac{\frac{e^x * b}{(e^x + b)^2}}{\frac{e^x}{e^x + b}} \\\\
=& \frac{-e^x * b}{(e^x + b)^2} * \frac{e^x + b}{e^x}
 \\
=& \frac{b}{e^x + b} \end{align} $$

Now, in this next part, we'll use the fact that:

$$ \frac{b}{e^x + b} = 1 - \frac{e^x}{e^x + b} $$

$ \begin{align}
\frac{-1 * f'(x)}{1 - f(x)} =& \frac{-1 * \frac{e^x * b}{(e^x + b)^2}}{1 - \frac{e^x}{e^x + b}} \\
=& \frac{\frac{-e^x * b}{(e^x + b)^2}}{\frac{b}{e^x + b}} \\
=& \frac{-e^x * b}{(e^x + b)^2} * \frac{e^x + b}{b}
 \\
=& \frac{-e^x}{e^x + b} \end{align}$

Finally, putting these pieces together:

$$ \begin{align} SC'(x) =& - a * \frac{f'(x)}{f(x)} - (1 - a) * \frac{-1 * f'(x)}{1-f(x)} \\
=& -a * \frac{b}{e^x + b} - (1 - a) * \frac{-e^x}{e^x + b} \\
=& -a * \frac{b}{e^x + b} + \frac{e^x}{e^x + b} - a * \frac{-e^x}{e^x + b} \\
=& -a * (1 - \frac{e^x}{e^x + b}) + \frac{e^x}{e^x + b} - a * \frac{-e^x}{e^x + b} \\
=& -a + a * \frac{e^x}{e^x + b} + \frac{e^x}{e^x + b} - a * \frac{-e^x}{e^x + b} \\
=& -a + \frac{e^x}{e^x + b} \\
\end{align} \\
$$

That's right, the derivative to be sent backward from the softmax layer is simply:

$$ S - Y = s(\begin{bmatrix} p_1 \\ p_2 \end{bmatrix}) - \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = \begin{bmatrix} \frac{e^{p_1}}{e^{p_1} + e^{p_2}} - y_1 \\ \frac{e^{p_2}}{e^{p_1} + e^{p_2}} - y_2 \end{bmatrix} $$

This makes sense:

* The softmax output will always be between 0 and 1.
* If $y_i$ is 0, then $ s(x_1) - y_1 $ will be a positive number: because indeed, if we increase the value of $x_1$, the loss will increase. Conversely if $y_i$ is one.
* Note that this loss function only makes sense because $ s(x_i) $ is always between 0 and 1.

This, by the way, is why TensorFlow has a function called `softmax_cross_entropy_with_logits`!

# Numerically stable softmax math

$$ log(softmax(x_j)) = log(\frac{e^{x_j}}{\sum_i^n e^{x_j}}) = x_j - logsumexp(X) $$

$$ softmax(x_j) = e^{log(softmax(x_j))} = e^{x - logsumexp(x)} $$

$$ logsumexp(X) = log(\sum_i^n e^{x_i}) = c + log(\sum_i^n e^{x_i - c}) $$

# One hot encoding

$$ [0, 2, 1] \Rightarrow \begin{bmatrix} 1 & 0 & 0 & \ldots & 0 \\ 0 & 0 & 1 & \ldots & 0 \\ 0 & 1 & 0 & \ldots & 0 \end{bmatrix} $$

# Learning rate decay

## Linear decay

$$ \alpha_{t} = \alpha_{start} - (\alpha_{start} - \alpha_{end}) * \frac{t}{N} $$

## Exponential decay

$$ \alpha_{t} = \alpha_{start} * ((\frac{\alpha_{end}}{\alpha_{start}})^\frac{1}{N})^t $$

# SGD Momentum gradient formula

$ \text{update} = \nabla_t + \mu * \nabla_{t-1} + \mu^2 * \nabla_{t-2} + \ldots $

# Weight init illustrations

## Feature level math

$ f_n = w_{1, n}* x_1 + \ldots + w_{784, n}* x_{784} + b_n $

$ \text{Var}(w_{i,j}) = 1 $

$ \text{Var}(X_1 + X_2) = \text{Var}(X_1) + \text{Var}(X_2) $

```python
n_feat = 784
n_hidden = 256

np.random.seed(190131)
```

```python
a = np.random.randn(1, n_feat)
```

```python
b = np.random.randn(n_feat, n_hidden)
```

```python
out = np.dot(a, b).reshape(n_hidden)
```

```python
fig, ax = plt.subplots(1, 2, figsize=(12, 5))
ax[0].hist(out)
ax[0].set_title("Distribution of inputs to activation function\nfor layer with 784 inputs")
ax[1].hist(np.tanh(out))
ax[1].set_title("Distribution of outputs of tanh function\nfor layer with 784 inputs");
# fig.savefig(IMG_FILEPATH + "01_weight_init_activation_illustration.png")
```


```python
# https://colab.research.google.com/github/kassbohm/wb-snippets/blob/master/ipynb/HTM_03/Selbst/4.4.ipynb

from sympy import *
sa_subs, ca_subs = var("s, c")

pprint("\nJ: Inertia Tensor wrt CG.")
pprint("\nJK: K-Components of J.")
pprint("\nJI: I-Components of J.")

pprint("\nJK / (ml²):")
JK = Matrix([
    [  96,   0, -72 ],
    [   0, 150,   0 ],
    [ -72,   0,  54 ],
    ])
pprint(JK)

alpha = var("alpha")
ca, sa = cos(alpha), sin(alpha)
R = Matrix([
    [ ca, sa, 0],
    [-sa, ca, 0],
    [  0,  0, 1],
    ])
Rt = R.transpose()
JI = Rt * JK * R

pprint("\nJI / (ml²):")

tmp = JI.subs([
    (sa, sa_subs),
    (ca, ca_subs),
    ])
pprint(tmp)

# pprint("\nEigenvalue of JK / ( ml² ), Multiplicity, Eigenvector:")
# tmp = JK.eigenvects()
# pprint(tmp)

A, JE = JK.diagonalize(normalize=True)

pprint("\nJE: JK diagonalized / ( ml² ):")
pprint(JE)

pprint("\nA, so that JK = A JE Aᵀ:")
pprint(A)

pprint("\nChecking: JK must be equal to A JE Aᵀ:")
tmp = A*JE*A.transpose()
pprint(tmp)

# JK: K-Components of JC / ( ml² ):
# ⎡96    0   -72⎤
# ⎢             ⎥
# ⎢ 0   150   0 ⎥
# ⎢             ⎥
# ⎣-72   0   54 ⎦
#
# JI: I-Components of JC / ( ml² ):
# ⎡    2        2                       ⎤
# ⎢96⋅c  + 150⋅s      -54⋅c⋅s      -72⋅c⎥
# ⎢                                     ⎥
# ⎢                     2       2       ⎥
# ⎢   -54⋅c⋅s      150⋅c  + 96⋅s   -72⋅s⎥
# ⎢                                     ⎥
# ⎣    -72⋅c           -72⋅s        54  ⎦
#
# Eigenvalue / ( ml² ), multiplicity, eigenvector:
# ⎡⎛      ⎡⎡3/4⎤⎤⎞  ⎛        ⎡⎡0⎤  ⎡-4/3⎤⎤⎞⎤
# ⎢⎜      ⎢⎢   ⎥⎥⎟  ⎜        ⎢⎢ ⎥  ⎢    ⎥⎥⎟⎥
# ⎢⎜0, 1, ⎢⎢ 0 ⎥⎥⎟, ⎜150, 2, ⎢⎢1⎥, ⎢ 0  ⎥⎥⎟⎥
# ⎢⎜      ⎢⎢   ⎥⎥⎟  ⎜        ⎢⎢ ⎥  ⎢    ⎥⎥⎟⎥
# ⎣⎝      ⎣⎣ 1 ⎦⎦⎠  ⎝        ⎣⎣0⎦  ⎣ 1  ⎦⎦⎠⎦
#
# JE; Eigenbasis-Components of JC / ( ml² ):
# ⎡0   0    0 ⎤
# ⎢           ⎥
# ⎢0  150   0 ⎥
# ⎢           ⎥
# ⎣0   0   150⎦
#
# A, so that JK = A JE Aᵀ:
# ⎡3/5  0  -4/5⎤
# ⎢            ⎥
# ⎢ 0   1   0  ⎥
# ⎢            ⎥
# ⎣4/5  0  3/5 ⎦
#
# Checking: JK must be equal to A JE Aᵀ:
# ⎡96    0   -72⎤
# ⎢             ⎥
# ⎢ 0   150   0 ⎥
# ⎢             ⎥
# ⎣-72   0   54 ⎦

```