# Métodos Numéricos Aplicados à Transferência de Calor

## Introdução

O normal do nosso fluxo de trabalho é iniciar importando as bibliotecas que vamos utilizar no decorrer do material.
Um maior detalhamento sobre elas já foi feito na aula anterior, de modo que agora podemos utilizar diretamente:

```python
import handcalcs.render
import matplotlib.pyplot as plt
import numpy as np
import plotly.express as px
import plotly.figure_factory as ff
import plotly.graph_objects as go
import plotly.io as pio
from tqdm.notebook import tqdm
```

O bloco a seguir é opcional, ele vai alterar o estilo padrão de nossas figuras, e aumentar um pouco o seu tamanho, melhorando a apresentação em nossa aula:

```python
# Definindo um novo estilo para as figuras [opcional]
# Esse bloco modifica alguns dos valores padrões para

plt.rcdefaults()

# https://matplotlib.org/3.1.0/gallery/style_sheets/style_sheets_reference.html
plt.style.use("ggplot")

# https://matplotlib.org/3.1.1/tutorials/introductory/customizing.html
plt.rcParams.update({"figure.dpi": 100, "figure.figsize": (6, 6)})

px.defaults.template = "ggplot2"
px.defaults.height = 600
pio.templates.default = "ggplot2"
```

## Exercícios Resolvidos

### Convecção e difusão transiente unidimensional

Resolver a EDP:

\begin{equation}
    \dfrac{\partial T}{\partial t} = \alpha \dfrac{\partial^2 T}{\partial x^2} - u\dfrac{\partial T}{\partial x}, \quad 0\leq x \leq 1 ; 0\leq t \leq 8
\end{equation}

Condições de contorno:

\begin{equation}
    T(0,t)=T(1,t)=0
\end{equation}

Condição inicial:

\begin{equation}
    T(x,0) =  1 - ( 10 x - 1 )^2 \quad \text{ se $0 \leq x \leq 0,2$}, \quad \text{ senão } T(x,0) = 0
\end{equation}

Discretizando com as derivadas espaciais numa representação por diferença central e a derivada temporal com diferença ascendente:

\begin{equation}
\dfrac{T_{i,n+1}-T_{i,n}}{\Delta t}=\alpha \dfrac{T_{i-1,n}-2T_{i,n}+T_{i+1,n}}{(\Delta x)^2} -u\dfrac{T_{i+1,n}-T_{i-1,n}}{2\Delta x}, \quad 1 \le i \le I - 2, \quad n > 0,
\end{equation}

\begin{equation}
T_{i=0,n} = 0,
\end{equation}
\begin{equation}
T_{i=I-1,n} = 0.
\end{equation}

Agora devemos isolar a incógnita do nosso problema: o termo $T_{i,n+1}$. Perceba que todos os termos à direita são conhecidos, e usamos essa informação para avançar progressivamente no tempo:

\begin{equation}
T_{i,n+1} = T_{i,n} + \Delta t \left( \alpha \dfrac{T_{i-1,n}-2T_{i,n}+T_{i+1,n}}{(\Delta x)^2} -u\dfrac{T_{i+1,n}-T_{i-1,n}}{2\Delta x} \right), \quad 1 \le i \le I - 2, \quad n > 0,
\end{equation}

Veja como podemos programar a solução para o problema, e note, principalmente, como ela se assemelha muito com a notação discretizada da equação acima. Aqui está o código:

```python
def equação_exemplo_1(coord_x, coord_t, alpha, u):

    # Condição inicial
    T = np.zeros(shape = (coord_x.size, coord_t.size))

    for i, x in enumerate(coord_x):
        if x <= 0.2:
            T[i] = 1. - (10. * x - 1) ** 2.

    # Condições de contorno
    T[0, :] = 0.0
    T[-1, :] = 0.0

    # Passo de tempo e resolução da malha
    dt = coord_t[1] - coord_t[0]
    dx = coord_x[1] - coord_x[0]

    # Aqui resolve-se a equação
    for n in tqdm(range(0, coord_t.size - 1)):
        for i in range(1, coord_x.size - 1):

                T[i, n + 1] = T[i, n] + dt * (
                    alpha * (T[i - 1, n] - 2.0 * T[i, n] + T[i + 1, n]) / dx**2.0
                    - u * (T[i + 1, n] - T[i - 1, n]) / (2.0 * dx)
                )

    return T
```

Hora de executar o cálculo utilizando a função que definimos no bloco anterior:

```python
coord_x = np.linspace(0., 1., num=101)
coord_t = np.linspace(0., 8., num=2001)

T = equação_exemplo_1(
    coord_x,
    coord_t,
    alpha = 0.001,
    u = 0.08
)
```

Por fim, exemplificamos a exibição dos resultados com `plotly`:

```python
fig = go.Figure()

for n in range(0, coord_t.size, 500):

    fig.add_trace(
        go.Scatter(
            x=coord_x, y=T[:,n],
            mode='lines',
            name=f't={coord_t[n]}'
        )
    )

fig.update_layout(
    title='Convecção e difusão transiente',
    xaxis_title='x',
    yaxis_title='Temperatura'
)

fig.show()
```

```python
fig = px.imshow(
    T.T,
    x = coord_x,
    y = coord_t,
    color_continuous_scale = 'RdBu_r',
    title = "Temperatura",
    labels = dict(x = "x", y = "tempo"),
    aspect = "auto",
    origin = "lower"
)
fig.show()
```

### Escoamento em Cavidade com Transferência de Calor

Aqui está o sistema de equações diferenciais: a equação da continuidade, duas equações para os componentes de velocidade $u,v$ e uma equação para a temperatura $\Theta$:

$$ \frac{\partial u}{\partial x} + \frac{\partial v}{\partial y} = 0 $$

$$\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y} = - \frac{\partial p}{\partial x}+ \frac{1}{Re} \left(\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2} \right) $$

$$\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial x}+v\frac{\partial v}{\partial y} = - \frac{\partial p}{\partial y}+ \frac{1}{Re} \left(\frac{\partial^2 v}{\partial x^2}+\frac{\partial^2 v}{\partial y^2}\right) + Ri ~ \Theta $$

$$ \frac{\partial \Theta}{\partial t} + u\frac{\partial \Theta}{\partial x} + v\frac{\partial \Theta}{\partial y} = \frac{1}{Re ~ Pr} \left(\frac{\partial^2 \Theta}{\partial x^2}+\frac{\partial^2 \Theta}{\partial y^2} \right) $$

* Equações discretas:

Primeiro, vamos discretizar a equação do momento para $u$, da seguinte maneira:

$$
\begin{split}
& \frac{u_{i,j}^{n+1}-u_{i,j}^{n}}{\Delta t}+u_{i,j}^{n}\frac{u_{i+1,j}^{n}-u_{i-1,j}^{n}}{2 \Delta x}+v_{i,j}^{n}\frac{u_{i,j+1}^{n}-u_{i,j-1}^{n}}{2\Delta y} = \\
& \qquad -\frac{p_{i+1,j}^{n}-p_{i-1,j}^{n}}{2\Delta x}+\frac{1}{Re}\left(\frac{u_{i+1,j}^{n}-2u_{i,j}^{n}+u_{i-1,j}^{n}}{\Delta x^2}+\frac{u_{i,j+1}^{n}-2u_{i,j}^{n}+u_{i,j-1}^{n}}{\Delta y^2}\right)
\end{split}
$$

Da mesma forma para a equação do momento para $v$:

$$
\begin{split}
&\frac{v_{i,j}^{n+1}-v_{i,j}^{n}}{\Delta t}+u_{i,j}^{n}\frac{v_{i+1,j}^{n}-v_{i-1,j}^{n}}{2\Delta x}+v_{i,j}^{n}\frac{v_{i,j+1}^{n}-v_{i,j-1}^{n}}{2\Delta y} = \\
& \qquad - \frac{p_{i,j+1}^{n}-p_{i,j-1}^{n}}{2\Delta y}
+\frac{1}{Re}\left(\frac{v_{i+1,j}^{n}-2v_{i,j}^{n}+v_{i-1,j}^{n}}{\Delta x^2}+\frac{v_{i,j+1}^{n}-2v_{i,j}^{n}+v_{i,j-1}^{n}}{\Delta y^2}\right) + Ri ~ \Theta_{i,j}
\end{split}
$$

A obtenção da equação de pressão-Poisson discretizada está fora do escopo dessa aula, por ser parte da disciplina de mecânica dos fluidos computacional, mais detalhes podem ser obtidos nos passos 10 e 11 do curso [CFD com Python](https://github.com/fschuch/CFDPython-BR). De qualquer maneira, a equação pode ser escrita assim:

$$
\begin{split}
& \frac{p_{i+1,j}^{n}-2p_{i,j}^{n}+p_{i-1,j}^{n}}{\Delta x^2}+\frac{p_{i,j+1}^{n}-2p_{i,j}^{n}+p_{i,j-1}^{n}}{\Delta y^2} = \\
& \qquad \left[ \frac{1}{\Delta t}\left(\frac{u_{i+1,j}-u_{i-1,j}}{2\Delta x}+\frac{v_{i,j+1}-v_{i,j-1}}{2\Delta y}\right) -\frac{u_{i+1,j}-u_{i-1,j}}{2\Delta x}\frac{u_{i+1,j}-u_{i-1,j}}{2\Delta x} - 2\frac{u_{i,j+1}-u_{i,j-1}}{2\Delta y}\frac{v_{i+1,j}-v_{i-1,j}}{2\Delta x} - \frac{v_{i,j+1}-v_{i,j-1}}{2\Delta y}\frac{v_{i,j+1}-v_{i,j-1}}{2\Delta y}\right]
\end{split}
$$

Por fim, a equação para temperatura:

$$
\begin{split}
&\frac{\Theta_{i,j}^{n+1}-\Theta_{i,j}^{n}}{\Delta t}+u_{i,j}^{n}\frac{\Theta_{i+1,j}^{n}-\Theta_{i-1,j}^{n}}{2\Delta x}+v_{i,j}^{n}\frac{\Theta_{i,j+1}^{n}-\Theta_{i,j-1}^{n}}{2\Delta y} = \\
& \qquad + \frac{1}{Re ~ Pr}\left(\frac{\Theta_{i+1,j}^{n}-2\Theta_{i,j}^{n}+\Theta_{i-1,j}^{n}}{\Delta x^2}+\frac{\Theta_{i,j+1}^{n}-2\Theta_{i,j}^{n}+\Theta_{i,j-1}^{n}}{\Delta y^2}\right)
\end{split}
$$

**Exercitando:** Você pode escrever essas equações em suas próprias anotações, manualmente, seguindo cada termo mentalmente à medida que as escreve.

Como antes, vamos reorganizar as equações da maneira que as iterações devem proceder no código. Primeiro, as equações de momento para a velocidade no passo de tempo subsequente.

A equação do momento na direção de $u$:

$$
\begin{split}
u_{i,j}^{n+1} = u_{i,j}^{n} & - u_{i,j}^{n} \frac{\Delta t}{2\Delta x} \left(u_{i+1,j}^{n}-u_{i-1,j}^{n}\right) - v_{i,j}^{n} \frac{\Delta t}{2\Delta y} \left(u_{i,j+1}^{n}-u_{i,j-1}^{n}\right) \\
& - \frac{\Delta t}{2\Delta x} \left(p_{i+1,j}^{n}-p_{i-1,j}^{n}\right) \\
& + \frac{1}{Re} \left(\frac{\Delta t}{\Delta x^2} \left(u_{i+1,j}^{n}-2u_{i,j}^{n}+u_{i-1,j}^{n}\right) + \frac{\Delta t}{\Delta y^2} \left(u_{i,j+1}^{n}-2u_{i,j}^{n}+u_{i,j-1}^{n}\right)\right)
\end{split}
$$

```python
def cavidade_u(u, v, dt, dx, dy, p, re):
    return (
        u[1:-1, 1:-1]
        - u[1:-1, 1:-1] * dt / (2*dx) * (u[2:, 1:-1] - u[0:-2, 1:-1])
        - v[1:-1, 1:-1] * dt / (2*dy) * (u[1:-1, 2:] - u[1:-1, 0:-2])
        - dt / (2 * dx) * (p[2:, 1:-1] - p[0:-2, 1:-1])
        + (1.0 / re)
        * (
            dt / dx ** 2 * (u[2:, 1:-1] - 2 * u[1:-1, 1:-1] + u[0:-2, 1:-1])
            + dt / dy ** 2 * (u[1:-1, 2:] - 2 * u[1:-1, 1:-1] + u[1:-1, 0:-2])
        )
    )
```

A equação do momento na direção de $v$:

$$
\begin{split}
v_{i,j}^{n+1} = v_{i,j}^{n} & - u_{i,j}^{n} \frac{\Delta t}{2\Delta x} \left(v_{i+1,j}^{n}-v_{i-1,j}^{n}\right) - v_{i,j}^{n} \frac{\Delta t}{2\Delta y} \left(v_{i,j+1}^{n}-v_{i,j-1}^{n})\right) \\
& - \frac{\Delta t}{2\Delta y} \left(p_{i,j+1}^{n}-p_{i,j-1}^{n}\right) \\
& + \frac{1}{Re} \left(\frac{\Delta t}{\Delta x^2} \left(v_{i+1,j}^{n}-2v_{i,j}^{n}+v_{i-1,j}^{n}\right) + \frac{\Delta t}{\Delta y^2} \left(v_{i,j+1}^{n}-2v_{i,j}^{n}+v_{i,j-1}^{n}\right)\right) + \Delta t ~ Ri ~ \Theta_{i,j}
\end{split}
$$

```python
def cavidade_v(u, v, theta, dt, dx, dy, p, re, ri):
    return (
        v[1:-1, 1:-1]
        - u[1:-1, 1:-1] * dt / (2*dx) * (v[2:, 1:-1] - v[0:-2, 1:-1])
        - v[1:-1, 1:-1] * dt / (2*dy) * (v[1:-1, 2:] - v[1:-1, 0:-2])
        - dt / (2 * dy) * (p[1:-1, 2:] - p[1:-1, 0:-2])
        + (1.0 / re)
        * (
            dt / dx ** 2 * (v[2:, 1:-1] - 2 * v[1:-1, 1:-1] + v[0:-2, 1:-1])
            + dt / dy ** 2 * (v[1:-1, 2:] - 2 * v[1:-1, 1:-1] + v[1:-1, 0:-2])
        )
    ) + ri * dt * theta[1:-1, 1:-1]
```

Agora, reorganizamos a equação de pressão-Poisson:

$$
\begin{split}
p_{i,j}^{n} = & \frac{\left(p_{i+1,j}^{n}+p_{i-1,j}^{n}\right) \Delta y^2 + \left(p_{i,j+1}^{n}+p_{i,j-1}^{n}\right) \Delta x^2}{2\left(\Delta x^2+\Delta y^2\right)} \\
& -\frac{\Delta x^2\Delta y^2}{2\left(\Delta x^2+\Delta y^2\right)} \\
& \times \left[\frac{1}{\Delta t}\left(\frac{u_{i+1,j}-u_{i-1,j}}{2\Delta x}+\frac{v_{i,j+1}-v_{i,j-1}}{2\Delta y}\right)-\frac{u_{i+1,j}-u_{i-1,j}}{2\Delta x}\frac{u_{i+1,j}-u_{i-1,j}}{2\Delta x} -2\frac{u_{i,j+1}-u_{i,j-1}}{2\Delta y}\frac{v_{i+1,j}-v_{i-1,j}}{2\Delta x}-\frac{v_{i,j+1}-v_{i,j-1}}{2\Delta y}\frac{v_{i,j+1}-v_{i,j-1}}{2\Delta y}\right]
\end{split}
$$

```python
def build_up_b(u, v, dt, dx, dy):
    """Constrói o termo entre chaves na equação acima"""

    b = np.zeros_like(u)

    b[1:-1, 1:-1] = (
        1
        / dt
        * (
            (u[2:, 1:-1] - u[0:-2, 1:-1]) / (2 * dx)
            + (v[1:-1, 2:] - v[1:-1, 0:-2]) / (2 * dy)
        )
        - ((u[2:, 1:-1] - u[0:-2, 1:-1]) / (2 * dx)) ** 2
        - 2
        * (
            (u[1:-1, 2:] - u[1:-1, 0:-2])
            / (2 * dy)
            * (v[2:, 1:-1] - v[0:-2, 1:-1])
            / (2 * dx)
        )
        - ((v[1:-1, 2:] - v[1:-1, 0:-2]) / (2 * dy)) ** 2
    )

    return b

def pressure_poisson(u, v, dt, dx, dy, p, nit=100):

    b = build_up_b(u, v, dt, dx, dy)

    for q in range(nit):
        pn = p.copy()
        p[1:-1, 1:-1] = (
            (
                (pn[2:, 1:-1] + pn[0:-2, 1:-1]) * dy ** 2
                + (pn[1:-1, 2:] + pn[1:-1, 0:-2]) * dx ** 2
            )
            / (2 * (dx ** 2 + dy ** 2))
            - dx ** 2 * dy ** 2 / (2 * (dx ** 2 + dy ** 2)) * b[1:-1, 1:-1]
        )

        p[-1, :] = p[-2, :]  # dp/dx = 0 at x = 2
        p[:, 0] = p[:, 1]  # dp/dy = 0 at y = 0
        p[0, :] = p[1, :]  # dp/dx = 0 at x = 0
        p[:, -1] = 0  # p = 0 at y = 2

    return p
```

Quase lá! Só falta a equação para temperatura:

$$
\begin{split}
\Theta_{i,j}^{n+1} = \Theta_{i,j}^{n} & - u_{i,j}^{n} \frac{\Delta t}{2\Delta x} \left(\Theta_{i+1,j}^{n}-\Theta_{i-1,j}^{n}\right) - v_{i,j}^{n} \frac{\Delta t}{2\Delta y} \left(\Theta_{i,j+1}^{n}-\Theta_{i,j-1}^{n})\right) \\
& + \frac{1}{Re ~ Pr} \left(\frac{\Delta t}{\Delta x^2} \left(\Theta_{i+1,j}^{n}-2\Theta_{i,j}^{n}+\Theta_{i-1,j}^{n}\right) + \frac{\Delta t}{\Delta y^2} \left(\Theta_{i,j+1}^{n}-2\Theta_{i,j}^{n}+\Theta_{i,j-1}^{n}\right)\right)
\end{split}
$$

```python
def cavidade_theta(u, v, theta, dt, dx, dy, p, re, pr):
    return (
        theta[1:-1, 1:-1]
        - u[1:-1, 1:-1] * dt / (2*dx) * (theta[2:, 1:-1] - theta[0:-2, 1:-1])
        - v[1:-1, 1:-1] * dt / (2*dy) * (theta[1:-1, 2:] - theta[1:-1, 0:-2])
        + (1.0 / re / pr)
        * (
            dt / dx ** 2 * (theta[2:, 1:-1] - 2 * theta[1:-1, 1:-1] + theta[0:-2, 1:-1])
            + dt
            / dy ** 2
            * (theta[1:-1, 2:] - 2 * theta[1:-1, 1:-1] + theta[1:-1, 0:-2])
        )
    )
```

A condição inicial é $u, v, p, \Theta = 0 $ em todos os lugares, e as condições de contorno são:

$u=1$ e $\Theta=0$ em $y=2$ (a "tampa");

$u, v = 0$ e $\Theta=1$ nas fronteiras restantes;

$\frac{\partial p}{\partial y}=0$ em $y=0$;

$p=0$ em $y=2$

$\frac{\partial p}{\partial x}=0$ em $x=0$ e $x=2$

Agora todos juntos em uma nova função para efetivamente resolver o escoamento em uma cavidade:

```python
def cavidade(x, y, t, re, ri, pr):

    # Condição inicial
    u = np.zeros((x.size, y.size))
    v = np.zeros((x.size, y.size))
    p = np.zeros((x.size, y.size))
    theta = np.zeros((x.size, y.size))

    # Passo de tempo e resolução da malha
    dt = t[1] - t[0]
    dx = x[1] - x[0]
    dy = y[1] - y[0]

    # Laço temporal
    for n in tqdm(range(t.size)):
        un = u.copy()
        vn = v.copy()
        thetan = theta.copy()

        p = pressure_poisson(un, vn, dt, dx, dy, p)
        u[1:-1, 1:-1] = cavidade_u(un, vn, dt, dx, dy, p, re)
        v[1:-1, 1:-1] = cavidade_v(un, vn, thetan, dt, dx, dy, p, re, ri)
        theta[1:-1, 1:-1] = cavidade_theta(un, vn, thetan, dt, dx, dy, p, re, pr)

        # Condições de contorno
        u[:, 0] = 0
        u[0, :] = 0
        u[-1, :] = 0
        u[:, -1] = 1  # Definir velocidade na tampa da cavidade igual a 1
        v[:, 0] = 0
        v[:, -1] = 0
        v[0, :] = 0
        v[-1, :] = 0
        theta[-1, :] = 1
        theta[:, 0] = 1
        theta[0, :] = 1
        theta[:, -1] = 0  # Definir theta na tampa da cavidade igual a 1

    return u, v, theta, p
```

Hora da ação:

```python
# Coordenadas
x = np.linspace(start = 0.0, stop = 2.0, num=21)
y = np.linspace(start = 0.0, stop = 2.0, num=21)
t = np.arange(start = 0.0, stop = 5.0, step = 0.001)

u, v, theta, p = cavidade(x, y, t, re = 40.0, ri = 0.0, pr = 1.0)
```

Verificando os resultados:

```python
fig = px.imshow(
    theta.T,
    x=x,
    y=y,
    color_continuous_scale="RdBu_r",
    title="Temperatura",
    labels=dict(x="x", y="y"),
    origin="lower",
)
fig.show()
```

```python
fig = ff.create_streamline(
    x[1:-1], y[1:-1], u.T[1:-1, 1:-1], v.T[1:-1, 1:-1], marker_color="black",
)

fig.add_trace(
    go.Contour(
        z=theta.T,
        x=x,
        y=y,
        colorscale="RdBu_r",
        colorbar=dict(title="Temperatura", titleside="right"),
    )
)

fig.update_layout(
    title="Escoamento em Cavidade com Transferência de Calor",
    xaxis_title="x",
    yaxis_title="y",
    autosize=False,
    width=800,
    height=800,
)

fig.show()
```

> Leitura recomendada:
> * Gostaria de se apronfundar mais no assunto? Veja a aula [Métodos Numéricos com Python](https://github.com/fschuch/metodos-numericos-com-python), que inclui a solução do problema da cavidade com transferência de calor utilizando conceitos Python mais avançados, como a criação e manipulação de classes em programação orientada ao objeto (OOP).

-----

> **Felipe N. Schuch**,

> Pesquisador em Fluidodinâmica Computacional na PUCRS, com interesse em: Escoamentos turbulentos, transferência de calor e massa, e interação fluido-estrutura; Processamento e visualização de dados em Python; Jupyter Notebook como uma ferramenta de colaboração, pesquisa e ensino.

> [felipeschuch@outlook.com](mailto:felipeschuch@outlook.com "Email") [@fschuch](https://twitter.com/fschuch "Twitter") [Aprenda.py](https://fschuch.github.io/aprenda.py "Blog") [@aprenda.py](https://www.instagram.com/aprenda.py/ "Instagram")

-----


# Implementation of Linear Kalman Filter using Constant Acceleration Motion Model and Linear Measurement Model

```python
import math
import matplotlib.pyplot as plt
import numpy as np
from scipy.sparse.linalg import expm
import scipy.integrate as integrate
from scipy.linalg import sqrtm
import sympy as sp
from sympy import init_printing
init_printing(use_latex=True)
from matplotlib import animation, rc
from IPython.display import HTML, Image
```

```python
# initalize global variables
dt = 0.01  # seconds
N = 100  # number of samples
qc = 0.1  # process noise magnitude
z_noise = 1  # measurement noise magnitude
```

## Available sensors:
1. GNSS, measuring x axis position, y axis position
2. Accelerometer, measuring x axis acceleration, y axis acceleration

## Linear Kalman Filter Prediction Step:
1. $x_{k|k-1} = a_{k-1} \cdot x_{k-1|k-1}$
2. $P_{k|k-1} = a_{k-1} \cdot P_{k-1|k-1} \cdot a_{k-1}^T + q_{k-1}$
\
\
In this project, we assume that an object is travelling in a linear path $x=y$, and we obtain noisy measurements of x and y positions from the GNSS sensor. The velocity is assumed to be constant and thus the accelerations would be zero. Hence, the accelerometer would always measure zero acceleration with added noise.

## Linear Kalman Filter Update Step:
1. $k_k = P_{k|k-1} \cdot h_k^T \cdot s_k^{-1}$
2. $v_k = z_k - h_k \cdot x_{k|k-1}$
3. $s_k = h_k \cdot P_{k|k-1} \cdot h_k^T + r_k$
4. $x_{k|k} = x_{k|k-1} + k_{k} \cdot v_{k}$
5. $P_{k|k} = P_{k|k-1} - k_k \cdot s_k \cdot k_k^T$
\
\
In this project, we assume that the state variables - x & y positions and x & y accelerations are directly measurable by the given sensors, with some added noise.

```python
px, py, vx, vy, ax, ay, T = sp.symbols('p_x p_y v_x v_y a_x a_y T')
x_k = sp.Matrix([px, py, vx, vy, ax, ay])
a = sp.Matrix([[0.0, 0.0, 1.0, 0.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 1.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]])
a = sp.exp(T * a)
gamma = sp.Matrix([[0.0], [0.0], [0.0], [0.0], [1.0], [1.0]])
q = qc * gamma @ np.transpose(gamma)
q_euler = T * q
q_exact = sp.integrate(sp.exp(T*a) * q * sp.exp(T*a.T), T)
h = sp.Matrix([[1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
              [0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 1.0]])
h_k = sp.Matrix([px, py, ax, ay])
```

## State Matrix ($x_k$):

```python
x_k
```

$\displaystyle \left[\begin{matrix}p_{x}\\p_{y}\\v_{x}\\v_{y}\\a_{x}\\a_{y}\end{matrix}\right]$

## State Transition Matrix ($a$):

```python
a
```

$\displaystyle \left[\begin{matrix}1 & 0 & T & 0 & 0.5 T^{2} & 0\\0 & 1 & 0 & T & 0 & 0.5 T^{2}\\0 & 0 & 1 & 0 & 1.0 T & 0\\0 & 0 & 0 & 1 & 0 & 1.0 T\\0 & 0 & 0 & 0 & 1.0 & 0\\0 & 0 & 0 & 0 & 0 & 1.0\end{matrix}\right]$

## Process Noise ($q$):

```python
q_euler
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0.1 T & 0.1 T\\0 & 0 & 0 & 0 & 0.1 T & 0.1 T\end{matrix}\right]$

```python
q_exact
```

$\displaystyle \left[\begin{matrix}1.0 \left(0.0125 T^{8} - 0.025 T^{7} + 0.1 T^{6} - 0.3 T^{5} + 0.75 T^{4} - 1.5 T^{3} + 2.25 T^{2} - 2.25 T + 1.125\right) e^{2 T} & 1.0 \left(0.0125 T^{8} - 0.025 T^{7} + 0.1 T^{6} - 0.3 T^{5} + 0.75 T^{4} - 1.5 T^{3} + 2.25 T^{2} - 2.25 T + 1.125\right) e^{2 T} & 1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T} & 1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T}\\1.0 \left(0.0125 T^{8} - 0.025 T^{7} + 0.1 T^{6} - 0.3 T^{5} + 0.75 T^{4} - 1.5 T^{3} + 2.25 T^{2} - 2.25 T + 1.125\right) e^{2 T} & 1.0 \left(0.0125 T^{8} - 0.025 T^{7} + 0.1 T^{6} - 0.3 T^{5} + 0.75 T^{4} - 1.5 T^{3} + 2.25 T^{2} - 2.25 T + 1.125\right) e^{2 T} & 1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T} & 1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T}\\1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.05 T^{4} - 0.1 T^{3} + 0.15 T^{2} - 0.15 T + 0.075\right) e^{2 T} & 1.0 \left(0.05 T^{4} - 0.1 T^{3} + 0.15 T^{2} - 0.15 T + 0.075\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T}\\1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.025 T^{6} - 0.05 T^{5} + 0.125 T^{4} - 0.25 T^{3} + 0.375 T^{2} - 0.375 T + 0.1875\right) e^{2 T} & 1.0 \left(0.05 T^{4} - 0.1 T^{3} + 0.15 T^{2} - 0.15 T + 0.075\right) e^{2 T} & 1.0 \left(0.05 T^{4} - 0.1 T^{3} + 0.15 T^{2} - 0.15 T + 0.075\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T}\\1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T} & 1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T} & 0.05 e^{2 T} & 0.05 e^{2 T}\\1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T} & 1.0 \left(0.025 T^{4} - 0.025 T^{3} + 0.0375 T^{2} - 0.0375 T + 0.01875\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T} & 1.0 \left(0.05 T^{2} - 0.05 T + 0.025\right) e^{2 T} & 0.05 e^{2 T} & 0.05 e^{2 T}\end{matrix}\right]$

## Measurements ($h_k$):

```python
h_k
```

$\displaystyle \left[\begin{matrix}p_{x}\\p_{y}\\a_{x}\\a_{y}\end{matrix}\right]$

## Measurement Matrix ($h$):

```python
h
```

$\displaystyle \left[\begin{matrix}1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0\end{matrix}\right]$

```python
# prior mean
x_0 = np.array([[0.0],  # x position
                [0.0],  # y position
                [0.0],  # x velocity
                [0.0],  # y velocity
                [0.0],  # x acceleration
                [0.0]  # y acceleration
                ])

# prior covariance
p_0 = np.array([[1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
                [0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
                [0.0, 0.0, 1.0, 0.0, 0.0, 0.0],
                [0.0, 0.0, 0.0, 1.0, 0.0, 0.0],
                [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
                [0.0, 0.0, 0.0, 0.0, 0.0, 1.0]])

# a matrix - discrete time motion model
a = np.array([[0.0, 0.0, 1.0, 0.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 1.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]])

a = expm(dt * a)
```

```python
# q matrix - continuous time process noise covariance
gamma = np.array([[0.0], [0.0], [0.0], [0.0], [1.0], [1.0]])
q = qc * gamma @ np.transpose(gamma)
q_euler = dt * q

# h matrix - measurement model
h = np.array([[1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
              [0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
              [0.0, 0.0, 0.0, 0.0, 0.0, 1.0]])

# r matrix - measurement noise covariance
r = np.array([[0.010, 0.0, 0.0, 0.0],
              [0.0, 0.015, 0.0, 0.0],
              [0.0, 0.0, 0.1, 0.0],
              [0.0, 0.0, 0.0, 0.1]])**2
```

```python
# linear kalman filter
def kalman_filter(x, p, z):
    x_pred, p_pred = linear_prediction(a, x, p, q_euler)
    x_upd, p_upd = linear_update(x_pred, p_pred, z, h, r)
    return x_upd, p_upd
```

```python
# linear kalman filter prediction step
def linear_prediction(a, x_hat, p_hat, q):
    x_pred = a @ x_hat
    p_pred = a @ p_hat @ np.transpose(a) + q
    return x_pred, p_pred
```

```python
# linear kalman filter update step
def linear_update(x_hat, p_hat, y, h, r):
    s = h @ p_hat @ np.transpose(h) + r
    k = p_hat @ np.transpose(h) @ np.linalg.pinv(s)
    v = y - h @ x_hat

    x_upd = x_hat + k @ v
    p_upd = p_hat - k @ s @ np.transpose(k)
    return x_upd, p_upd
```

```python
# generate ground truth position x_true and noisy position z
def gen_measurement(i):
    x_true = np.array([[i], [i], [0.0], [0.0]])
    z = x_true + z_noise * np.random.randn(4, 1)
    return z, x_true
```

```python
def plot_animation():
    plt.plot(x_true[0], x_true[1], '.r')
    plt.plot(x_est[0], x_est[1], '.b')
    plt.plot(z[0], z[1], '+g')
    plt.grid(True)
    plt.pause(0.001)
```

```python
# animation function. This is called sequentially
def animate(i):
    x = np.linspace(0, 2, 1000)
    y = np.sin(2 * np.pi * (x - 0.01 * i))
    line.set_data(x, y)
    return (line,)
```

```python
# run the kalman filter
x_est = x_0
p_est = p_0
x_true_cat = np.array([x_0[0, 0], x_0[1, 0]])
x_est_cat = np.array([x_0[0, 0], x_0[1, 0]])
z_cat = np.array([x_0[0, 0], x_0[1, 0]])
for i in range(N):
    z, x_true = gen_measurement(i)
    x_est, p_est = kalman_filter(x_est, p_est, z)
    x_true_cat = np.vstack((x_true_cat, np.transpose(x_true[0:2])))
    z_cat = np.vstack((z_cat, np.transpose(z[0:2])))
    x_est_cat = np.vstack((x_est_cat, np.transpose(x_est[0:2])))
print('KF Over')
```

    KF Over

## Ground Truth Trajectory, Noisy Measurements, Estimated Trajectory:

```python
fig = plt.figure()
f = fig.add_subplot(111)
f.plot(x_true_cat[0:, 0], x_true_cat[0:, 1], 'r', label='True Position')
f.set_xlabel('x [m]')
f.set_ylabel('y [m]')
f.legend(loc='upper left', shadow=True, fontsize='small')
plt.grid(True)

fig = plt.figure()
f = fig.add_subplot(111)
f.plot(z_cat[0:, 0], z_cat[0:, 1], '+g', label='Noisy Measurements')
f.set_xlabel('x [m]')
f.set_ylabel('y [m]')
f.legend(loc='upper left', shadow=True, fontsize='small')
plt.grid(True)

fig = plt.figure()
f = fig.add_subplot(111)
f.plot(x_est_cat[0:, 0], x_est_cat[0:, 1], 'b', label='Estimated Position')
f.set_xlabel('x [m]')
f.set_ylabel('y [m]')
f.legend(loc='upper left', shadow=True, fontsize='small')
plt.grid(True)

plt.show()
```

## Final Output:

```python
fig = plt.figure()
f = fig.add_subplot(111)
f.plot(x_true_cat[0:, 0], x_true_cat[0:, 1], 'r', label='True Position')
f.plot(x_est_cat[0:, 0], x_est_cat[0:, 1], 'b', label='Estimated Position')
f.plot(z_cat[0:, 0], z_cat[0:, 1], '+g', label='Noisy Measurements')
f.set_xlabel('x [m]')
f.set_ylabel('y [m]')
f.set_title('Linear Kalman Filter - Constant Acceleration Model')
f.legend(loc='upper left', shadow=True, fontsize='small')
plt.grid(True)
plt.show()
```


# Probni prvi kolokvij

## Nakić 1

Pomoću for petlji napišite ekvivalent donjeg koda
```py
q = [r**2 for r in [10**i for i in range(5)]]
```

```python
unutarnja = [10**i for i in range(5)]
q = [r**2 for r in unutarnja]
```

```python
unutarnja = []
for i in range(5):
    unutarnja.append(10**i)
q = []
for r in unutarnja:
    q.append(r**2)
```

```python
q
```

    [1, 100, 10000, 1000000, 100000000]

```python
q = []
for i in range(5):
    r = 10**i
    q.append(r**2)
```

```python
q = []
for i in range(0, 10, 2): q.append(10**i)
```

```python
q
```

    [1, 100, 10000, 1000000, 100000000]

```python
skup = {broj + 5 for broj in range(5, 99, 7) if broj % 3 == 1}
```

```python
skup
```

    {24, 45, 66, 87}

```python
skup = set()
for broj in range(5, 99, 7):
    if broj % 3 == 1:
        skup.add(broj + 5)
```

```python
skup
```

    {24, 45, 66, 87}

```python
rječnik = {ime: len(ime) for ime in 'Borna Ivka Mare Silvana'.split()}
```

```python
rječnik
```

    {'Borna': 5, 'Ivka': 4, 'Mare': 4, 'Silvana': 7}

```python
rječnik = {}
for ime in 'Borna Ivka Mare Silvana'.split():
    rječnik[ime] = len(ime)
```

```python
rječnik
```

    {'Borna': 5, 'Ivka': 4, 'Mare': 4, 'Silvana': 7}

```python
velika = [i + j for i in range(10) for j in range(i) if j % 3 != i % 4]
```

```python
print(velika)
```

    [1, 2, 3, 3, 4, 5, 5, 6, 5, 7, 8, 6, 7, 9, 10, 7, 8, 9, 10, 11, 12, 13, 9, 10, 12, 13, 15, 9, 11, 12, 14, 15, 17]

```python
velika = []
for i in range(10):
    for j in range(i):
        if j % 3 != i % 4:
            velika.append(i + j)
```

```python
print(velika)
```

    [1, 2, 3, 3, 4, 5, 5, 6, 5, 7, 8, 6, 7, 9, 10, 7, 8, 9, 10, 11, 12, 13, 9, 10, 12, 13, 15, 9, 11, 12, 14, 15, 17]

## Nakić 2

Napišite funkciju koja za ulaz prima broj $N$. Funkcija treba kreirati slučajan `numpy` niz duljine $N$ s uniformnom distribucijom, zatim treba naći broj $M$ brojeva u nizu koji leže između $0.5$ i $0.6$, te treba vratiti broj $\frac{M}{N}$. Učinite funkciju interaktivnom.

```python
import numpy as np
```

```python
import numpy as np
def proporcija(N):
    niz = np.random.uniform(size=N)
    M = ((niz > .5) & (niz < .6)).sum()
    return M / N
```

```python
import ipywidgets
slajder = ipywidgets.IntSlider(min=1, max=100, step=1, value=10)
ipywidgets.interact(proporcija, N=slajder)
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    <function __main__.proporcija>

## Nakić 3

Riješite sljedeću ODJ koristeći Runge-Kutta metodu četvrtog stupnja
na segmentu $[0,4]$:

```python
import sympy as sp
sp.init_printing()
y = sp.Function('y')
x = sp.Symbol('x')
epsilon = sp.Symbol('varepsilon')

(sp.Eq(sp.Derivative(y(x),x),1/sp.factor(2*(y(x)-1))),
 sp.Eq(y(0), 1 + sp.sqrt(epsilon)))
```

Ovdje je $ε=10^{−3}$.

```python
from scipy.integrate import ode
```

```python
def desna_strana(x, y):
    return 1 / (2*(y-1))
```

```python
epsilon = 10**-3
jednadžba = ode(desna_strana)
jednadžba.set_integrator('dopri5')
jednadžba.set_initial_value(np.sqrt(epsilon) + 1)
jednadžba.integrate(4)
```

    array([3.00024858])

### Pokušaj provjere

```python
import sympy as sp
sp.init_printing()
```

```python
y = sp.Function('y')
x = sp.Symbol('x')
epsilon = sp.Symbol('varepsilon')

jednadžba = sp.Eq(sp.Derivative(y(x),x),1/sp.factor(2*(y(x)-1)))
jednadžba
```

```python
sva_rješenja = sp.dsolve(jednadžba)
sva_rješenja
```

```python
opće_rj = sva_rješenja[1]
opće_rj
```

```python
početni = opće_rj.subs({x: 0, y(x): 1 + sp.sqrt(epsilon)})
početni
```

```python
partikularno = opće_rj.subs(sp.solve(početni)[0])
partikularno
```

```python
import decimal
decimal.Decimal(10**-3)
```

    Decimal('0.001000000000000000020816681711721685132943093776702880859375')

```python
(10**-3).hex()
```

    '0x1.0624dd2f1a9fcp-10'

```python
10**-3
```

```python
import sys
sys.float_repr_style
```

    'short'

```python
partikularno.subs({epsilon: sp.S(10)**-3, x: 4}).n(59)
```

## Nakić 4

Nacrtajte funkciju $v(x)$
za $μ=1$, $μ=0.01$ te $μ=0.001$
na segmentu $[0,1]$ koristeći $10000$ točaka.

```python
v = sp.Function('v')
mu = sp.Symbol('mu')

definicija = sp.Eq(v(x), (1-sp.E**(x/mu))/(1-sp.E**(1/mu)))
definicija
```

```python
domena = np.linspace(0, 1, 10_000)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# plt.plot([0, 1, 1], [0, 0, 1], label='$\mu=0.001$')
for mi in 0.01, 1, 0.1, 0.005:
    slika = (1 - np.e ** (domena / mi)) / (1 - np.e ** (1 / mi))
    plt.plot(domena, slika, label=f'$\mu={mi}$')
plt.legend()
```

```python
plt.plot([0, 1, 1], [0, 0, 1], label='$\mu=0.001$')
for mi in 0.01, 1:
    slika = (1 - np.e ** (domena / mi)) / (1 - np.e ** (1 / mi))
    plt.plot(domena, slika, label=f'$\mu={mi}$')
plt.legend()
```

```python
mi = sp.Symbol('mu')
t = sp.Symbol('t')
y = sp.Symbol('y', positive=True)
sp.Limit(definicija.rhs, mi, 0, '+')
sp.Limit(definicija.rhs.subs({1/mi: t, x: 1-y}), t, sp.oo)
```

## Nakić 5

Pomoću `sympy`ja, izraz `log(x**y*a**b)` pojednostavite do izraza `b*log(a) + y*log(x)`.

```python
import sympy as sp
sp.init_printing()
```

```python
x, a = sp.symbols('x a', positive=True)
y, b = sp.symbols('y b', real=True)
izraz = sp.log(x**y * a**b)
izraz
```

```python
sp.expand_log(izraz)
```

```python
x, y, a, b = sp.symbols('x y a b')
sp.expand_log(sp.log(x**y * a**b), force=True)
```

## Čačić 1

Što ispisuje sljedeći kod i zašto?

```python
a = b = 5
a += 3
print(b)

a = b = [5]
a += [3]
print(b)
```

    5
    [5, 3]

```python
mojalista = [2]
list.__add__(mojalista, [3])
mojalista
```

## Čačić 2

Simulirajte bacanje pristranog novčića (vjerojatnost glave je 55%) 1000 puta. Kolika je vjerojatnost da u tih 1000 bacanja broj glava bude veći od broja pisama? Rezultat odredite numerički iz simulacije tako da ponovite pokus (svih 1000 bacanja) 100 000 puta.

```python
import numpy as np
```

```python
broj_bacanja = 1000
broj_pokusa = 100_000
vjerojatnost_glave = .55

pokusi = np.random.uniform(size=(broj_bacanja, broj_pokusa))
glave = pokusi < vjerojatnost_glave
```

```python
broj_glava = glave.sum(axis=0)
broj_pisama = broj_bacanja - broj_glava
(broj_glava > broj_pisama).mean()
```

Domaća zadaća: dodajte interaktivnost. :-)
Alternativno: nacrtajte graf ovisnosti o pristranosti novčića.

## Čačić 3

Zadan je sljedeći DataFrame (u datoteci `pdzad.csv`):

```python
%%file pdzad.csv
2016-07-17	450	628.578622	girl	red
2014-02-17	418	226.950670	boy	red
2016-03-03	418	208.042709	girl	red
2014-04-28	26	915.238500	boy	blue
2014-09-07	182	744.946901	boy	green
2014-07-21	345	542.665316	girl	blue
2014-01-21	807	396.394572	boy	yellow
2016-01-23	36	792.745992	boy	blue
2015-07-07	44	616.078044	girl	bluedatum
2014-08-18	390	527.061301	boy	yellow
2014-04-02	456	4.716575	girl	green
2015-12-20	584	577.939367	boy	blue
2015-03-18	718	919.344036	girl	yellow
2015-04-26	792	623.799887	boy	red
2015-08-23	938	927.630485	boy	yellow
2015-09-27	14	40.380074	girl	blue
2014-07-15	26	451.519075	girl	blue
2016-05-15	116	969.555910	boy	red
2015-04-15	209	827.094101	boy	yellow
2014-03-03	202	811.039542	boy	red
2015-01-07	254	512.529009	girl	blue
2015-09-01	2	703.001673	girl	yellow
2016-04-26	667	187.608667	boy	red
2016-01-23	312	884.018403	girl	blue
2014-10-11	792	585.769718	girl	purple
2014-03-30	507	373.570850	girl	yellow
2015-03-27	78	635.718898	girl	purple
2014-12-27	100	41.486492	girl	purple
2016-04-17	683	138.001860	girl	red
2016-03-08	300	925.708079	boy	green
2016-01-16	925	971.141551	boy	red
2016-06-23	317	223.288738	boy	green
2016-01-26	783	819.096804	boy	blue
2015-01-28	71	309.760628	boy	blue
2016-03-27	325	444.532115	boy	purple
2015-07-10	552	517.586297	boy	red
2014-05-19	740	988.004758	boy	yellow
2014-06-07	463	61.233638	boy	green
2016-06-02	518	256.335606	boy	yellow
2014-03-14	528	114.734319	girl	red
2015-06-08	207	731.211508	boy	yellow
2015-10-09	211	715.706315	boy	red
2014-07-08	672	302.847557	boy	yellow
2014-05-06	375	957.917383	girl	yellow
2016-06-09	530	205.990431	girl	purple
2016-09-06	294	682.120222	girl	blue
2015-09-15	166	190.032588	boy	red
2015-01-12	591	978.913511	boy	blue
2015-10-21	571	259.954915	girl	green
2015-09-29	75	414.686734	boy	purple
```

    Writing pdzad.csv

Za najkasniji (datumi su u prvom stupcu) redak koji u predzadnjem stupcu ima `girl` a u zadnjem `yellow`, nađite umnožak drugog i trećeg stupca.

```python
import pandas as pd
```

```python
'datum prvi drugi spol boja'.split()
```

    ['datum', 'prvi', 'drugi', 'spol', 'boja']

```python
header = 'datum prvi drugi spol boja'.split()
df = pd.read_csv('pdzad.csv', delimiter='\t', names=header,
                 index_col=0, parse_dates=True)
```

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 50 entries, 2016-07-17 to 2015-09-29
    Data columns (total 4 columns):
    prvi     50 non-null int64
    drugi    50 non-null float64
    spol     50 non-null object
    boja     50 non-null object
    dtypes: float64(1), int64(1), object(2)
    memory usage: 2.0+ KB

```python
subdf = df[(df['spol'] == 'girl') & (df['boja'] == 'yellow')]
subdf
```

      prvi
      drugi
      spol
      boja

      datum

      2015-03-18
      718
      919.344036
      girl
      yellow

      2015-09-01
      2
      703.001673
      girl
      yellow

      2014-03-30
      507
      373.570850
      girl
      yellow

      2014-05-06
      375
      957.917383
      girl
      yellow

```python
subdf.loc[subdf.index.max()]
```

    prvi           2
    drugi    703.002
    spol        girl
    boja      yellow
    Name: 2015-09-01 00:00:00, dtype: object

```python
traženi = subdf.loc[subdf.index.max()]
traženi['prvi'] * traženi['drugi']
```

## Čačić 4

Nacrtajte u polarnom koordinatnom sustavu grafove $r=\frac{\left|\sin\vartheta\right|}{\vartheta}$ i $r=\ln(1+\vartheta)$.

```python
theta = np.linspace(0, 2*np.pi)[1:]
r1 = np.abs(np.sin(theta)) / theta
r2 = np.log(1 + theta)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.polar(theta, r1)
plt.polar(theta, r2)
```

## Čačić 5

Napišite funkciju koja prima simboličku funkciju, integrira je simbolički po varijabli `x` (ako može), dobiveni rezultat derivira simbolički i uspoređuje s originalnim ulazom. Na kraju treba ispisati jednadžbe koje proizlaze iz integriranja odnosno deriviranja.

```python
import sympy as sp
sp.init_printing()
x = sp.Symbol('x')
from IPython.display import display

def provjeri_integral(fja):
    integral = sp.Integral(fja, x)
    intfja = integral.doit()
    prva = sp.Eq(integral, intfja, evaluate=False)

    derivacija = sp.Derivative(intfja, x)
    derfja = derivacija.doit()
    druga = sp.Eq(derivacija, derfja, evaluate=False)

    if derfja == fja:
        print('Sintaksno su jednake')
    else:
        print('Nisu sintaksno jednake')

    display(prva)
    display(druga)
```

```python
provjeri_integral(x*(x+1))
```

```python
provjeri_integral(sp.sin(x))
```

```python
provjeri_integral(sp.sqrt(sp.sin(x)))
```

Domaća zadaća: pokušajte centrirati te dvije jednadžbe.
Ustanovite programski zna li sympy integrirati vašu funkciju.

```python

```


MLHEP 2019

Seminar: Unsupervised Learning

# About

The goal of this seminar is to consider main domains of unsupervised learning and demonstrate algorithms implemented in [scikit-learn](https://scikit-learn.org) library.

Topics:
- Clustering
- Data Scaling
- Principal Component Analysis (PCA) (Optionally)
- Anomalies Detection (Optionally)

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
```

# Part 1: Clustering

## Data Preparation

```python
from sklearn import datasets

n_samples = 1500
random_state = 170
X, y = datasets.make_blobs(centers=3, n_samples=n_samples, random_state=random_state, center_box=(-10, 10))

# To play with
# X, y = datasets.make_circles(n_samples=n_samples, factor=.5, noise=.05)
# X, y = datasets.make_moons(n_samples=n_samples, noise=.05)
# X = np.random.rand(n_samples, 2)
```

```python
X[:5]
```

```python
y[:5]
```

```python
def plot_clusters(X, y):

    # Create an figure with a custom size
    # plt.figure(figsize=(6, 4))

    if y is not None:
        for cluster_label in np.unique(y):
            # Plot all objects with y == i (class 0)
            plt.scatter(X[y == cluster_label, 0],     # selects all objects with y == i and the 1st column of X
                        X[y == cluster_label, 1],     # selects all objects with y == i and the 2nd column of X
                        label=str(cluster_label))     # label for the plot legend
    else:
        plt.scatter(X[:, 0], X[:, 1], label='samples')

    plt.xlabel('X1', size=12) # set up X-axis label
    plt.ylabel('X2', size=12) # set up Y-axis label
    plt.xticks(size=12)
    plt.yticks(size=12)

    plt.legend(loc='best', fontsize=12) # create the plot legend and set up it position
    plt.grid(b=1) # create grid on the plot

    plt.show() # display the plot
```

```python
plot_clusters(X, y=None)
```

## Clustering: K-Means

Suppose we have $N$ samples and $K$ clusters. Each cluster is described by its center (centroid) with coordinates $\mu_{j}$. The centroids are estimated by minimizing **within-cluster distance criterion**:

$$
L = \sum_{i=1}^{N} \min_{\mu_{k}} \rho(x_{i}, \mu_{k}) \to \min_{\mu_{1}, ..., \mu_{K}}
$$

$$
\rho(x_{i}, \mu_{k}) = || x_{i} - \mu_{k} ||^{2}
$$

where $x_{i}$ is a sample coordinates, $\rho(x_{i}, \mu_{k})$ is distance between the $i$-th sample and the $k$-th cluster's centroid.

**K-Means algorithm:**

```python
from sklearn.utils import resample

class MyKmeans(object):

    def __init__(self, n_clusters=2, max_iter=10, n_init=10):
        """
        K-Means clustering algorithms implementation.

        Parameters:
        -----------
        n_clusters: int
            Number of clusters.
        max_iters: int
            Number of iterations of the centroids search.
        n_init: int
            Number of different initializations of the centroids.
        """

        self.n_clusters = n_clusters
        self.max_iter = max_iter
        self.n_init = n_init

    def _predict_for_centers(self, cluster_centers, X):
        """
        Predict cluster labels based on their centroids.

        Parameters:
        -----------
        cluster_centers: numpy.array
            Array of the cluster centers.
        X: numpy.array
            Samples coordinates.

        Returns:
        --------
        labels: numpy.array
            Predicted cluster labels. Example: labels = [0, 0, 1, 1, 0, 2, ...].
        """

        object_distances2 = []
        for one_cluster_center in cluster_centers:
            dist2 = ((X - one_cluster_center)**2).sum(axis=1)
            object_distances2.append(dist2)
        object_distances2 = np.array(object_distances2)
        labels = np.argmin(object_distances2, axis=0)

        return labels

    def _calculate_cluster_centers(self, X, y):
        """
        Estimate cluster centers based on samples in these clusters.

        Parameters:
        -----------
        X: numpy.array
            Samples coordinates.
        y: numpy.array
            Cluster labels of the samples.

        Returns:
        --------
        cluster_centers: numpy.array
            Estimated cluster centers.
        """

        cluster_centers = []
        cluster_labels = np.unique(y)

        for one_cluster_label in cluster_labels:
            one_cluster_center = X[y == one_cluster_label].mean(axis=0)
            cluster_centers.append(one_cluster_center)

        return np.array(cluster_centers)

    def _calculate_cluster_metric(self, cluster_centers, X):
        """
        Calculate within-cluster distance criterion.

        Parameters:
        -----------
        cluster_centers: numpy.array
            Array of the cluster centers.
        X: numpy.array
            Samples coordinates.

        Returns:
        --------
        criterion: float
            The criterion value.
        """

        object_distances2 = []
        for one_cluster_center in cluster_centers:
            dist2 = ((X - one_cluster_center)**2).sum(axis=1)
            object_distances2.append(dist2)
        object_distances2 = np.array(object_distances2)
        min_dists2 = np.min(object_distances2, axis=0)
        criterion = min_dists2.mean()

        return criterion

    def _fit_one_init(self, X):
        """
        Run k-Means algorithm for randomly init cluster centers.

        Parameters:
        -----------
        X: numpy.array
            Samples coordinates.

        Returns:
        --------
        cluster_centers: numpy.array
            Estimated cluster centers.
        metric: float
            Within-cluster distance criterion criterion value.
        """

        # Init cluster centers
        cluster_centers = resample(X, n_samples=self.n_clusters, random_state=None, replace=False)

        # Search for cluster centers
        for i in range(self.max_iter):
            labels = self._predict_for_centers(cluster_centers, X)
            cluster_centers = self._calculate_cluster_centers(X, labels)

        # Calculate within-cluster distance criterion
        metric = self._calculate_cluster_metric(cluster_centers, X)

        return cluster_centers, metric

    def fit(self, X):
        """
        Run k-Means algorithm.

        Parameters:
        -----------
        X: numpy.array
            Samples coordinates.
        """

        self.best_cluster_centers = None
        self.best_metric = np.inf

        for i in range(self.n_init):

            # Run K-Means algorithms for randomly init cluster centers
            cluster_centers, metric = self._fit_one_init(X)

            # Save the best clusters
            if metric < self.best_metric:
                self.best_metric = metric
                self.best_cluster_centers = cluster_centers

    def predict(self, X):
        """
        Predict cluster labels.

        Parameters:
        -----------
        X: numpy.array
            Samples coordinates.

        Returns:
        --------
        y: numpy.array
            Predicted cluster labels. Example: labels = [0, 0, 1, 1, 0, 2, ...].
        """

        y = self._predict_for_centers(self.best_cluster_centers, X)

        return y
```

```python
clusterer = MyKmeans(n_clusters=3, max_iter=20, n_init=10)
clusterer.fit(X)
y_pred = clusterer.predict(X)
```

```python
y_pred[:10]
```

```python
plot_clusters(X, y_pred)
```

## Metrics

**Silhouette Score:**

$$
s = \frac{b - a}{max(a, b)}
$$

- **a**: The mean distance between a sample and all other points in the same class.
- **b**: The mean distance between a sample and all other points in the next nearest cluster.

**Adjusted Rand Index (ARI):**

$$
ARI = \frac{RI - Expected\_RI}{max(RI) - Expected\_RI}
$$

$$
RI = \frac{a + b}{a + b + c + d}
$$

- a, the number of pairs of elements in S that are in the same subset in X and in the same subset in Y
- b, the number of pairs of elements in S that are in different subsets in X and in different subsets in Y
- c, the number of pairs of elements in S that are in the same subset in X and in different subsets in Y
- d, the number of pairs of elements in S that are in different subsets in X and in the same subset in Y

```python
from sklearn import metrics

silhouette_score_values = []
adjusted_rand_score_values = []
within_cluster_dist_values = []
n_clusters = np.arange(2, 21)

for n in n_clusters:

    clusterer = MyKmeans(n_clusters=n, max_iter=10, n_init=10)
    clusterer.fit(X)
    y_pred = clusterer.predict(X)

    score1 = metrics.silhouette_score(X, y_pred)
    silhouette_score_values.append(score1)

    score2 = metrics.adjusted_rand_score(y, y_pred)
    adjusted_rand_score_values.append(score2)

    score3 = clusterer.best_metric
    within_cluster_dist_values.append(score3)
```

```python
plt.figure(figsize=(9, 6))
plt.plot(n_clusters, silhouette_score_values, linewidth=3, label='Silhouette score')
plt.plot(n_clusters, adjusted_rand_score_values, linewidth=3, label='Adjusted rand score')
plt.xlabel('Number of clusters', size=16)
plt.ylabel('Score', size=16)
plt.xticks(n_clusters, size=16)
plt.yticks(size=16)
plt.legend(loc='best', fontsize=16)
plt.grid(b=1)
plt.show()

plt.figure(figsize=(9, 6))
plt.plot(n_clusters, within_cluster_dist_values, linewidth=3, label='Within-cluster distance')
plt.xlabel('Number of clusters', size=16)
plt.ylabel('Score', size=16)
plt.xticks(n_clusters, size=16)
plt.yticks(size=16)
plt.legend(loc='best', fontsize=16)
plt.grid(b=1)
plt.show()
```

## Other Clustering Algorithms

Short overview of other clustering algorithms you can find in `scikit-learn` library [here](https://scikit-learn.org/stable/modules/clustering.html):

```python
from sklearn import datasets

n_samples = 1500
random_state = 170
X, y = datasets.make_blobs(centers=3, n_samples=n_samples, random_state=random_state, center_box=(-10, 10))

# To play with
# X, y = datasets.make_circles(n_samples=n_samples, factor=.5, noise=.05)
# X, y = datasets.make_moons(n_samples=n_samples, noise=.05)
# X = np.random.rand(n_samples, 2)

plot_clusters(X, None)
```

```python
# Import clustering algorithms
from sklearn import cluster
```

```python
# MiniBatchKMeans

# Run clusterer
clusterer = cluster.MiniBatchKMeans(n_clusters=3, batch_size=100)
clusterer.fit(X)
y_pred = clusterer.predict(X)

# Plot clustering results
plot_clusters(X, y_pred)
```

**DBSCAN:**

```python
# DBSCAN

# Run clusterer
clusterer = cluster.DBSCAN(eps=0.5, min_samples=5)
y_pred = clusterer.fit_predict(X)

# Plot clustering results
plot_clusters(X, y_pred)
```

**Agglomerative Clustering:**

[image link](https://quantdare.com/hierarchical-clustering/)

```python
# AgglomerativeClustering

# Run clusterer
clusterer = cluster.AgglomerativeClustering(n_clusters=3)
y_pred = clusterer.fit_predict(X)

# Plot clustering results
plot_clusters(X, y_pred)
```

## Tasks:

- Rerun cells above for other datasets. Explain the clustering results.
- Try different number of clusters and other options. How can you explain what you see?

---

---

---

# Part 2: Scaling

## Data Preparation

Multiply one of the sample features by a large number.

```python
X_scaled = X.copy()
X_scaled[:, 1] *= 100
```

```python
plot_clusters(X_scaled, None)
```

## Clustering without scaling

All clustering algorithms are based on distances between objects $\rho(x_{i}, x_{j})$. For an axample in 2D case:

$$
\rho(x_{i}, x_{j}) = \sqrt{ (x_{1i} - x_{1j})^{2} + (x_{2i} - x_{2j})^{2} }
$$

where $x_{1i}$ is the 1st input feature, $x_{2i}$ is the 2nd one.

Suppose, that the features have different scales:

$$
\frac{x_{2i}}{x_{1i}} = 1000
$$

Then

$$
\rho(x_{i}, x_{j}) = \sqrt{ (x_{1i} - x_{1j})^{2} + (x_{2i} - x_{2j})^{2} } \approx\sqrt{ (x_{2i} - x_{2j})^{2} }
$$

So, the 1st feature will not be taken into account by clustering algorithms.

```python
from sklearn import cluster

# Run clustering algorithm
clusterer = cluster.KMeans(n_clusters=3, n_init=10)
clusterer.fit(X_scaled)
y_pred = clusterer.predict(X_scaled)

# Show clustering results
plot_clusters(X_scaled, y_pred)
```

## Clustering with Standard Scaler

Satndard Scaler transforms feature $x$ to a new feature $x_{new}$ with zero mean and unit variance by the following way:

$$
x_{new} = \frac{ x - \mu }{ \sigma }
$$

where

$$
\mu = \frac{1}{N} \sum_{i=1}^{N}x_{i}
$$

$$
\sigma = \sqrt{ \frac{1}{N-1} \sum_{i=1}^{N} (x_{i} - \mu)^{2} }
$$

This transforms all input features to the same scale.

```python
from sklearn.preprocessing import StandardScaler

ss = StandardScaler()
ss.fit(X_scaled)
X_scaled_ss = ss.transform(X_scaled)
```

```python
from sklearn import cluster

# Run clustering algorithm
clusterer = cluster.KMeans(n_clusters=3, n_init=10)
clusterer.fit(X_scaled_ss)
y_pred = clusterer.predict(X_scaled_ss)

# Show clustering results
plot_clusters(X_scaled, y_pred)
```

---

---

---

# Part 3: Principal Component Analysis (PCA)

## Data Preparation

```python
from sklearn import datasets

# Generate 2D Gaussian distribution
n_samples = 2000
X, y = datasets.make_blobs(n_samples=n_samples, random_state=42, centers=[[0, 0]])

# Apply coordiantes transformation
transformation = [[0.6, 0.4],
                  [0.4, 0.6]]
X_aniso = np.dot(X, transformation)
```

```python
plot_clusters(X_aniso, None)
```

## Principal Component Analysis (PCA)

Find directions along which our datapoints have the greatest variance:

These directions are principal components. Principal components $a_{1},a_{2},...a_{D}\in\mathbb{R}^{D}$ are orthonormal:

$$
\langle a_{i},a_{j}\rangle=\begin{cases}
1, & i=j\\
0 & i\ne j
\end{cases}
$$

## PCA algorithm (detailed):

### Step 1:
Calculate variance across a principal component $a$ assuming that $X$ is centralized:
$$
\begin{align} \sigma^2_a & = \frac{1}{n}\sum\limits_{i=1}^n(a^\top x_i - \mu)^2 \\
& = \frac{1}{n}\sum\limits_{i=1}^n(a^\top x_i - 0)^2 \\
& = \frac{1}{n}\sum\limits_{i=1}^n a^\top( x_i x_i^\top) a \\
& = a^\top \left(\frac{1}{n}\sum\limits_{i=1}^n x_i x_i^\top \right) a \\
& = a^\top X^\top X a \\
\end{align}
$$

### Step 2:
Find $a_1$ that maximizes the variance:

$$
\begin{equation}
\begin{cases}
a_1^\top X^\top X a_1 \rightarrow \max_{a_1} \\
a_1^\top a_1 = 1
\end{cases}
\end{equation}
$$

Lagrangian of optimization problem:
$$ \mathcal{L}(a_1, \nu) = a_1^\top X^\top X a_1 - \nu (a_1^\top a_1 - 1) \rightarrow max_{a_1, \nu}$$

Derivative w.r.t. $a_1$:
$$ \frac{\partial\mathcal{L}}{\partial a_1} = 2X^\top X a_1 - 2\nu a_1 = 0 $$

$$X^\top X a_1 = \nu a_1$$

---
#### Note:
So $a_1$ is selected from a set of eigenvectors of  $X^\top X$. But which one?

$$ a_1^\top X^\top X a_1 = \nu a_1^\top a_1 = \nu \rightarrow \max$$

That means:
* $\nu$ should be the greatest eigenvalue of matrix $X^\top X$, which is $\lambda_1$
* $a_1$ is eigenvector, correspondent to $\lambda_1$
---

### Step 3:

Similarly for $a_{2}$:

$$
\begin{equation}
\begin{cases}
a_2^\top X^\top X a_2 \rightarrow \max_{a_2} \\
a_2^\top a_2 = 1 \\
a_2^\top a_1 = 0
\end{cases}
\end{equation}
$$

...

## PCA algorithm (short)

1. Center (and scale) dataset
2. Calculate covariance matrix $С=X^\top X$
3. Find first $k$ eigenvalues and eigenvectors
$$A =
\left[
  \begin{array}{cccc}
    \mid & \mid & & \mid\\
    a_{1} & a_{2} & \ldots & a_{k} \\
    \mid & \mid & & \mid
  \end{array}
\right]
$$
4. Perform projection:
$$ Z = XA $$

```python
# Import PCA
from sklearn.decomposition import PCA

# Fit PCA
pca = PCA(n_components=2)
pca.fit(X_aniso)

# Apply PCA
X_aniso_pca = pca.transform(X_aniso)
```

```python
X_aniso_pca
```

```python
pca_1 = PCA(n_components=1)
pca_1.fit(X_aniso)
X_aniso_pca_1 = pca_1.transform(X_aniso)

pca_2 = PCA(n_components=2)
pca_2.fit(X_aniso)
X_aniso_pca_2 = pca_2.transform(X_aniso)
```

```python

plt.figure(figsize=(15, 5))

# Plot original X with eigenvectors
plt.subplot(1, 3, 1)
plt.scatter(X_aniso[:, 0], X_aniso[:, 1], color='b')
for vector in pca_2.components_:
    plt.arrow(0, 0, vector[0], vector[1], head_width=0.2, head_length=0.2, fc='k', ec='k')
plt.title("Original X", size=14)
plt.xticks(size=14)
plt.yticks(size=14)
#plt.grid(b=1)
plt.legend(loc='best')

# Plot for PCA with n_components=2
plt.subplot(1, 3, 2)
plt.scatter(X_aniso_pca_2[:, 0], X_aniso_pca_2[:, 1], color='b')
for vector in pca_2.transform(pca_2.components_):
    plt.arrow(0, 0, vector[0], vector[1], head_width=0.2, head_length=0.2, fc='k', ec='k', linewidth=2)
plt.title("PCA with n_component = 2", size=14)
plt.xticks(size=14)
plt.yticks(size=14)
plt.ylim(-4.5, 4.5)
#plt.grid(b=1)

# Plot for PCA with n_components=1
plt.subplot(1, 3, 3)
plt.scatter(X_aniso_pca_1[:, 0], [0]*len(X_aniso_pca_1), color='b')
for vector in pca_1.transform(pca_1.components_):
    if vector[0] <= 0.5: continue
    plt.arrow(0, 0, vector[0], 0, head_width=0.2, head_length=0.2, fc='k', ec='k', linewidth=2)
plt.title("PCA with n_component = 1", size=14)
plt.xticks(size=14)
plt.yticks(size=14)
plt.xlim(-4.5, 4.5)
plt.ylim(-4.5, 4.5)
#plt.grid(b=1)

plt.tight_layout()
plt.show()
```

## Real Data Example

## Gender Recognition by Voice

This database was created to identify a voice as male or female, based upon acoustic properties of the voice and speech. The dataset consists of 3,168 recorded voice samples, collected from male and female speakers. The voice samples are pre-processed by acoustic analysis in R using the seewave and tuneR packages, with an analyzed frequency range of 0hz-280hz (human vocal range).

The following acoustic properties of each voice are measured and included within the CSV:

* meanfreq: mean frequency (in kHz)
* sd: standard deviation of frequency
* median: median frequency (in kHz)
* Q25: first quantile (in kHz)
* Q75: third quantile (in kHz)
* IQR: interquantile range (in kHz)
* skew: skewness (see note in specprop description)
* kurt: kurtosis (see note in specprop description)
* sp.ent: spectral entropy
* sfm: spectral flatness
* mode: mode frequency
* centroid: frequency centroid (see specprop)
* peakf: peak frequency (frequency with highest energy)
* meanfun: average of fundamental frequency measured across acoustic signal
* minfun: minimum fundamental frequency measured across acoustic signal
* maxfun: maximum fundamental frequency measured across acoustic signal
* meandom: average of dominant frequency measured across acoustic signal
* mindom: minimum of dominant frequency measured across acoustic signal
* maxdom: maximum of dominant frequency measured across acoustic signal
* dfrange: range of dominant frequency measured across acoustic signal
* modindx: modulation index. Calculated as the accumulated absolute difference between adjacent measurements of fundamental frequencies divided by the frequency range
* label: male or female

```python
! wget https://raw.githubusercontent.com/yandexdataschool/mlhep2019/master/notebooks/day-2/Clustering/data/voice.csv
```

```python
# Read data sample
data = pd.read_csv("voice.csv")
print("DataFrame shape: ", data.shape)
data.head()
```

```python
# Get feature names
feature_names = data.columns.drop(['label'])
print("Feature names: ", feature_names)
```

```python
# Prepare X and y
X = data[feature_names].values
y = 1. * (data['label'].values == 'male')
```

## Train / Test Split + Standardization

```python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Split data into train and test samples
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)

# Standardization
ss = StandardScaler()
ss.fit(X_train)

X_train = ss.transform(X_train)
X_test = ss.transform(X_test)
```

## Train Classifier

```python
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier

# You can play with other classifiers
clf = LogisticRegression(solver='lbfgs')
clf.fit(X_train, y_train)
```

```python
y_test_predict = clf.predict(X_test)
y_test_proba = clf.predict_proba(X_test)[:, 1]
```

```python
from sklearn.metrics import accuracy_score, roc_auc_score

accuracy = accuracy_score(y_test, y_test_predict)
auc = roc_auc_score(y_test, y_test_proba)
print("Accuracy: ", accuracy)
print("ROC AUC: ", auc)
```

## Apply PCA

```python
pca_accuracies = []
pca_aucs = []
pca_components = np.arange(1, 21)

for n_components in pca_components:

    # For each n_components run PCA
    pca = PCA(n_components=n_components)
    pca.fit(X_train)
    X_train_pca = pca.transform(X_train)
    X_test_pca = pca.transform(X_test)

    # Fit a classifier
    clf = LogisticRegression(solver='lbfgs')
    clf.fit(X_train_pca, y_train)

    # Make predictions
    y_test_predict = clf.predict(X_test_pca)
    y_test_proba = clf.predict_proba(X_test_pca)[:, 1]

    # Calculate quality metrics
    accuracy = accuracy_score(y_test, y_test_predict)
    pca_accuracies.append(accuracy)

    auc = roc_auc_score(y_test, y_test_proba)
    pca_aucs.append(auc)
```

```python
plt.figure(figsize=(9, 6))
plt.plot(pca_components, pca_accuracies, label='Accuracy', color='b', linewidth=3)
plt.plot(pca_components, pca_aucs, label='ROC AUC', color='r', linewidth=3)
plt.xticks(pca_components, size=14)
plt.xlabel("N components of PCA", size=14)
plt.yticks(size=14)
plt.ylabel("Metric values", size=14)
plt.legend(loc='best', fontsize=14)
plt.grid(b=1)
plt.show()
```

## Explained variance

Explained variance for $a_i$ can be calculated as the following ratio:
$$
\frac{\lambda_{i}}{\sum_{d=1}^{D}\lambda_{d}}
$$

where $\lambda_{i}$ is an eigenvalue.

```python
# Fit PCA
pca = PCA(n_components=20)
pca.fit(X_train)

# Take all eigenvalues (sorted)
eigenvalues = pca.explained_variance_
```

```python
eigenvalues
```

```python
pca_components = np.arange(1, 21)

# Calculate explained variance
explained_variance = eigenvalues / eigenvalues.sum()

# Calculate cumulative explained variance
cumsum_explained_variance = np.cumsum(explained_variance)
```

```python
plt.figure(figsize=(18, 6))

plt.subplot(1, 2, 1)
plt.plot(pca_components, explained_variance, color='b', linewidth=3)
plt.xticks(pca_components, size=14)
plt.xlabel("N components of PCA", size=14)
plt.yticks(size=14)
plt.ylabel("Explained Variance", size=14)
plt.title("PCA Explained Variance", size=14)
plt.grid(b=1)

plt.subplot(1, 2, 2)
plt.plot(pca_components, cumsum_explained_variance, color='b', linewidth=3)
plt.xticks(pca_components, size=14)
plt.xlabel("N components of PCA", size=14)
plt.yticks(size=14)
plt.ylabel("Explained Variance", size=14)
plt.title("PCA Cumulative Explained Variance", size=14)
plt.grid(b=1)

plt.show()
```

----

----

----

# Part 4: Anomalies Detection

`Scikit-learn` has several anomalies detection algorithms. Their description and examples are provided on [this page](https://scikit-learn.org/stable/modules/outlier_detection.html):

## Data Preparation

```python
from sklearn import datasets

# Define key constants
n_samples = 500
outliers_fraction = 0.15
n_outliers = int(outliers_fraction * n_samples)

# Generate sample
X = 4 * (datasets.make_moons(n_samples=n_samples, noise=.05, random_state=0)[0] - np.array([0.5, 0.25]))

# Add outliers
X = np.concatenate([X, np.random.RandomState(42).uniform(low=-6, high=6, size=(n_outliers, 2))], axis=0)
```

```python
def plot_anomalies(X, ano):

    if ano is not None:
        try:
            y = ano.predict(X)
        except:
            y = ano.fit_predict(X)
    else:
        y = None

    # Create an figure with a custom size
    # plt.figure(figsize=(9, 6))

    if y is not None:
        for cluster_label in np.unique(y):

            if cluster_label == -1:
                suff = " (Anomaly)"
            else:
                suff = " (Normal)"

            # Plot all objects with y == i (class 0)
            plt.scatter(X[y == cluster_label, 0],          # selects all objects with y == i and the 1st column of X
                        X[y == cluster_label, 1],          # selects all objects with y == i and the 2nd column of X
                        label=str(cluster_label)+suff)     # label for the plot legend
    else:
        plt.scatter(X[:, 0], X[:, 1], label='samples')

    # Plot decision doundary
    if ano is not None:
        xx, yy = np.meshgrid(np.linspace(-7, 7, 150), np.linspace(-7, 7, 150))
        try:
            Z = ano.predict(np.c_[xx.ravel(), yy.ravel()])
            Z = Z.reshape(xx.shape)
            plt.contour(xx, yy, Z, levels=[0], linewidths=2, colors='black')
        except:
            pass

    plt.xlabel('X1', size=12) # set up X-axis label
    plt.ylabel('X2', size=12) # set up Y-axis label
    plt.xticks(size=12)
    plt.yticks(size=12)

    plt.legend(loc='best', fontsize=12) # create the plot legend and set up it position
    plt.grid(b=1) # create grid on the plot

    plt.show() # display the plot
```

```python
plot_anomalies(X, ano=None)
```

## Run Anomalies Detection

```python
# IsolationForest

from sklearn.ensemble import IsolationForest

# Run anomalies detection algorithm
ano = IsolationForest(behaviour='new', contamination=outliers_fraction, random_state=42)
ano.fit(X)

# Detect anomalies
y_pred = ano.predict(X)

# Plot detected anomalies
plot_anomalies(X, ano)
```

```python
# EllipticEnvelope (Robust covariance)

from sklearn.covariance import EllipticEnvelope

ano = EllipticEnvelope(contamination=outliers_fraction)
ano.fit(X)

# Detect anomalies
y_pred = ano.predict(X)

# Plot detected anomalies
plot_anomalies(X, ano)
```

```python
# DBSCAN

# Run anomalies detection algorithm
ano = cluster.DBSCAN(eps=0.5, min_samples=5)
ano.fit(X)

# Detect anomalies
y_pred = ano.fit_predict(X)

# Plot detected anomalies
plot_anomalies(X, ano)
```

Task:
* Change `contamination` parameter. How can you explain the results?

```python

```


# Ensemble Learning

* The basic idea of ensemble learning is to have multiple learning algorithms for the same problem and combine their results to make a final prediction

* There are multiple types on ensemble learning.  Common approaches include:
    * Boosting
    * Bagging/Bootstrapping
    * Random Forests
    * Mixture of Experts

## Bagging

* When you have one data set, usually you may train an algorithm and learn a single set of parameters.  However, when we do this, we have no idea how stable/variable those parameters that we estimated are.
* Boostrapping can show us the variation in estimated parameter values given a particular data set. Sometimes, it can also help to improve our predictions.
* Essentially, to perform bootstrapping, you sample from your data set *with replacement* and train your algorithm to estimate the parameters with each sampled subset.  You can then look at how much the parameters vary with each sampled subset and you can also combine your estimates from each trained method by averaging over all of the results for regression:
\begin{equation}
y_{com}(\mathbf{x}) = \frac{1}{M} \sum_{m=1}^M y_m(\mathbf{x})
\end{equation}
* You can aggregate results over all your bootstrap samples using majority vote for classification.

```python
import numpy as np
import matplotlib.pyplot as plt
import math
import textwrap
%matplotlib inline

def generateRandData(N, l, u, gVar):
	'''generateRandData(N, l, u, gVar): Generate N uniformly random data points in the range [l,u) with zero-mean Gaussian random noise with variance gVar'''
	x = np.random.uniform(l,u,N)
	e = np.random.normal(0,gVar,N)
	t = np.sin(2*math.pi*x) + e
	return x,t

def fitdataReg(x,t,M,la):
	'''fitdata(x,t,M): Fit a polynomial of order M to the data (x,t)'''
	X = np.array([x**m for m in range(M+1)]).T
	w = np.linalg.inv(X.T@X+(la*np.identity(M+1)))@X.T@t
	return w

def plotPoly(x,t,xrange, y, esty, subplotloc,la=0):
	#plot everything
	plt.subplot(*subplotloc) #identify the subplot to use
	# plt.tight_layout()
	p1 = plt.plot(xrange, y, 'g') #plot true value
	p2 = plt.plot(x, t, 'bo') #plot training data
	p3 = plt.plot(xrange, esty, 'r') #plot estimated value

	#add title, legend and axes labels
	plt.ylabel('t') #label x and y axes
	plt.xlabel('x')

def bootstrapRegression(M, numData,percentSample,numSamples):

    #generate data
	x,t = generateRandData(numData,0,1,1)
	numDataSamples = round(percentSample*numData)
	subplotloc = [2, round(numSamples/2), 1]
	fig = plt.figure()
	xrange = np.arange(0.05,.95,0.001)  #get equally spaced points in the xrange
	esty = np.empty([numSamples, xrange.shape[0]])

	for iter in range(numSamples):
        #select a random subset of the data
		rp = np.random.permutation(numData)
		x_sub = x[rp[0:numDataSamples-1]]
		t_sub = t[rp[0:numDataSamples-1]]

        #fit the random subset
		w = fitdataReg(x_sub,t_sub,M,0)

        #plot results
		subplotloc[2] = iter+1
		y = np.sin(2*math.pi*xrange) #compute the true function value
		X = np.array([xrange**m for m in range(w.shape[0])]).T
		esty[iter,:] = X@w #compute the predicted value
		plotPoly(x_sub,t_sub,xrange,y,esty[iter,:],subplotloc)

    #combine the bootstrapped results
	comy = esty.mean(0)

    # compare to full data set
	fig = plt.figure()
	plotPoly(x,t,xrange,y,comy,[1, 1, 1])

	fig = plt.figure()
	w = fitdataReg(x,t,M,0)
	y = np.sin(2*math.pi*xrange) #compute the true function value
	X = np.array([xrange**m for m in range(w.shape[0])]).T
	yy = X@w #compute the predicted value
	plotPoly(x,t,xrange,y,yy, [1, 1, 1])

#Figure 1.7 from text
bootstrapRegression(4, 50,.15,10)

```

# Boosting: AdaBoost

* Goal:  Combine base (``weak'') classifiers to form a committee whose performance is better than any of the single base classifiers.
* The base classifiers are trained in sequence (not in parallel like in bootstrapping)
* Each base classifier is trained using a weighted data set (different weights for each base classifier)
* Points that are misclassified by a base classifier are weighted more heavily while training the next base classifier

* Consider a two class classification problem with $\mathbf{X} = \left\{ \mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n\right\}$ with corresponding labels $y_i \in \left\{ -1,1\right\}$.
* The goal is to construct a classifier of the form:
\begin{equation}
f(\mathbf{x}) = sign(F(\mathbf{x}))
\end{equation}
where
\begin{equation}
F(\mathbf{x}) = \sum_{k=1}^K \frac{1}{2}\alpha_k \phi(\mathbf{x}; \theta_k)
\end{equation}
where $\phi(\mathbf{x}; \theta_k)$ is the base classifier.

* We need to determine the parameter values for each base classifier:
\begin{eqnarray}
\arg \min_{\alpha_k, \theta_k} \sum_{i=1}^N \exp\left(-y_i F(\mathbf{x}_i) \right)
\end{eqnarray}
* This cost function penalizes the samples that are incorrectly classified ($y_iF(\mathbf{x}_i) < 0$) heavily
* Direct optimization of all $\alpha$s and $\theta$s is difficult.   So, we iteratively optimize (which is sub-optimal). At each stage, we train one base classifier holding fixed all those that have already been trained.

* Let:
\begin{eqnarray}
F_m(\mathbf{x}) &=& \sum_{k=1}^m \frac{1}{2}\alpha_k \phi(\mathbf{x}; \theta_k)\\
&=& F_{m-1}(\mathbf{x}) + \frac{1}{2}\alpha_m \phi(\mathbf{x}; \theta_m)
\end{eqnarray}
* At step $m$, we optimize for $\alpha_m$ and $\theta_m$ where $F_{m-1}(\mathbf{x})$ is fixed:
\begin{eqnarray}
(\alpha_m, \theta_m) &=& \arg \min_{\alpha, \theta} J(\alpha, \theta)\\
&=& \arg \min_{\alpha, \theta} \sum_{i=1}^N \exp\left( -\frac{1}{2}y_i\left( F_{m-1}(\mathbf{x}_i) + \alpha\phi(\mathbf{x}_i; \theta)\right)\right)
\end{eqnarray}
* So, let's optimize this in two steps: first $\theta_m$ and then $\alpha_m$
\begin{eqnarray}
\theta_m &=&  \arg \min_{\theta} \sum_{i=1}^N \exp\left( -y_i\left( F_{m-1}(\mathbf{x}_i) + \frac{1}{2}\alpha\phi(\mathbf{x}_i; \theta)\right)\right)\\
&=& \arg \min_{\theta} \sum_{i=1}^N w_i^{(m)} \exp\left( -\frac{1}{2}y_i\alpha\phi(\mathbf{x}_i; \theta)\right)
\end{eqnarray}
where
\begin{equation}
w_i^{(m)} = \exp\left(-y_iF_{m-1}(\mathbf{x}_i)\right)
\end{equation}

* This can be re-written as:

\begin{eqnarray}
\theta_m &=&  \arg \min_{\theta} \exp\left(-\alpha_m/2\right)\sum_{n \in T_m}w_n^{(m)} + \exp\left(\alpha_m/2\right)\sum_{n \in M_m}w_n^{(m)} \nonumber \\
&=& \left( \exp\left(\alpha_m/2\right) - \exp\left(-\alpha_m/2\right)\right)\sum_{i=1}^Nw_i^{(m)} I(\phi_m(\mathbf{x}_i;\theta) \ne y_i) + \exp\left(-\alpha_m/2\right)\sum_{i=1}^Nw_i^{(m)}
\end{eqnarray}

* This is equivalent to minimizing

\begin{equation}
\arg \min_{\theta} \sum_{i=1}^N w_i^{(m)} I(\phi_m(\mathbf{x}_i;\theta) \ne y_i)
\end{equation}

* Once we have the optimal classifier at step $m$ (i.e., $\theta_m$), then we determine the $\alpha_m$ values
\begin{eqnarray}
\sum_{y_i\phi(\mathbf{x}_i;\theta_m)<0}w_i^{(m)} = P_m\\
\sum_{y_i\phi(\mathbf{x}_i;\theta_m)>0}w_i^{(m)} = 1 - P_m
\end{eqnarray}
* Plugging this into J, we get:
\begin{eqnarray}
\alpha_m = \arg\min_{\alpha} \left\{ \exp(-\alpha)(1-P_m) + \exp(\alpha)P_m\right\}
\end{eqnarray}
* Take the derivative with respect to $\alpha$, set to zero, we get:
\begin{equation}
\alpha_m = \frac{1}{2}\ln\frac{1-P_m}{P_m}
\end{equation}
* Once you get $\theta_m$ and $\alpha_m$, you compute the weights for the next step:
\begin{equation}
w_i^{(m+1)} = \frac{\exp(-y_iF_m(\mathbf{x}_i))}{Z_m} = \frac{\exp(-y_i\alpha_m\phi(\mathbf{x}_i;\theta_m))}{Z_m}
\end{equation}
where
\begin{equation}
Z_m = \sum_{i=1}^N w_i^{(m)}\exp\left(-y_i\alpha_m\phi(\mathbf{x}_i;\phi_m)\right)
\end{equation}
* Notice that the weight corresponding to a sample is increased (or decreased) with respect to its value in the previous iteration
* Notice that the amount of increase or decrease depends on $\alpha_m$ which controls the relative importance of the $m^{th}$ term in building up the final classifier

```python

```

## Random Forests
* A forest is made up of many trees...
* For classification/regression, put an input vector down each of the trees in the forest.  For classification, classify the data point using majority vote.  For regression, average the values
* Each tree is grown using:
    * Sample $N$ data points (with replacement, i.e., a bootstrap sample) from the full training data set
    * Specify a number $d << D$.  $d$ variables are selected at random out of all $D$ features to determine the split on the node.  Select the best of the $d$ features to split at that node
    * Grow each tree as much as possible (i.e., no pruning or stopping early)

* Error relates to correlation between the trees.  Greater correlation leads to greater error.  *Does this make sense?*
* Error also relates to the strength of each individual tree.  Better individual trees lead to lower error
* https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm

# Dropout

* This is a method to help prevent overfitting and regularize a network.
* The approach attempts to minimize co-dependencies between neurons and enhance robustness of network
* Dropout has one parameter $p$.  In each iteration, you randomly exclude each neuron with probability $1-p$ during the training pass (in both forward and backward propagation).  Each iteration, you resample which neurons to keep and which to dropout.
* Dropout is related to the concept of ensemble learning with the unique case that the various models in the ensemble share parameters and these models are "combined" into a single model/network at test as opposed to training a fusion model or doing a simple average between outputs.
* During test, you use all neurons all the time.
* Please see and read: https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf

```python

```


# Tutorial: optimal piecewise binning with continuous target

## Basic

To get us started, let's load a well-known dataset from the UCI repository and transform the data into a ``pandas.DataFrame``.

```python
import pandas as pd
from sklearn.datasets import load_boston
```

```python
data = load_boston()
df = pd.DataFrame(data.data, columns=data.feature_names)
```

We choose a variable to discretize and the continuous target.

```python
variable = "LSTAT"
x = df[variable].values
y = data.target
```

Import and instantiate an ``ContinuousOptimalPWBinning`` object class and we pass the variable name. The ``ContinuousOptimalPWBinning`` can **ONLY** handle numerical variables. This differs from the ``ContinuousOptimalBinning`` object class.

```python
from optbinning import ContinuousOptimalPWBinning
```

```python
optb = ContinuousOptimalPWBinning(name=variable)
```

We fit the optimal binning object with arrays ``x`` and ``y``.

```python
optb.fit(x, y)
```

    ContinuousOptimalPWBinning(name='LSTAT')

You can check if an optimal solution has been found via the ``status`` attribute:

```python
optb.status
```

    'OPTIMAL'

You can also retrieve the optimal split points via the ``splits`` attribute:

```python
optb.splits
```

    array([ 4.6500001 ,  5.49499989,  6.86500001,  9.7249999 , 11.67499971,
           13.0999999 , 16.08500004, 19.89999962, 23.31500053])

#### The binning table

The optimal binning algorithms return a binning table; a binning table displays the binned data and several metrics for each bin. Class ``ContinuousOptimalPWBinning`` returns an object ``PWContinuousBinningTable`` via the ``binning_table`` attribute.

```python
binning_table = optb.binning_table
```

```python
type(binning_table)
```

    optbinning.binning.piecewise.binning_statistics.PWContinuousBinningTable

The `binning_table` is instantiated, but not built. Therefore, the first step is to call the method `build`, which returns a ``pandas.DataFrame``.

```python
binning_table.build()
```

      Bin
      Count
      Count (%)
      Sum
      Std
      Min
      Max
      Zeros count
      c0
      c1

      0
      (-inf, 4.65)
      50
      0.098814
      1985.9
      8.198651
      22.8
      50.0
      0
      56.750078
      -4.823690

      1
      [4.65, 5.49)
      28
      0.055336
      853.2
      6.123541
      21.9
      50.0
      0
      74.546279
      -8.650830

      2
      [5.49, 6.87)
      45
      0.088933
      1188.6
      5.136259
      20.6
      48.8
      0
      27.338265
      -0.059744

      3
      [6.87, 9.72)
      89
      0.175889
      2274.9
      6.845250
      11.9
      50.0
      0
      35.670910
      -1.273531

      4
      [9.72, 11.67)
      49
      0.096838
      1057.9
      2.994842
      15.0
      31.0
      0
      36.084157
      -1.316024

      5
      [11.67, 13.10)
      35
      0.069170
      697.5
      2.592139
      14.5
      27.9
      0
      20.719580
      -0.000000

      6
      [13.10, 16.09)
      66
      0.130435
      1289.9
      3.541705
      10.2
      30.7
      0
      36.325829
      -1.191317

      7
      [16.09, 19.90)
      69
      0.136364
      1129.3
      3.607949
      8.3
      27.5
      0
      25.801457
      -0.537020

      8
      [19.90, 23.32)
      28
      0.055336
      368.4
      3.912839
      5.0
      21.7
      0
      32.669942
      -0.882170

      9
      [23.32, inf)
      47
      0.092885
      556.0
      4.006586
      5.0
      23.7
      0
      13.271245
      -0.050143

      10
      Special
      0
      0.000000
      0.0
      NaN
      NaN
      NaN
      0
      0.000000
      0.000000

      11
      Missing
      0
      0.000000
      0.0
      NaN
      NaN
      NaN
      0
      0.000000
      0.000000

      Totals

      506
      1.000000
      11401.6

      5.0
      50.0
      0
      -
      -

Let's describe the columns of this binning table:

- Bin: the intervals delimited by the optimal split points.
- Count: the number of records for each bin.
- Count (%): the percentage of records for each bin.
- Sum: the target sum for each bin.
- Std: the target std for each bin.
- Min: the target min value for each bin.
- Max: the target max value for each bin.
- Zeros count: the number of zeros for each bin.
- $c_0$: the first coefficient of the prediction polynomial.
- $c_1$: the second coefficient of the prediction polynomial.

The  for bin $i$ is defined as $P_i = c_0 + c_1 x_i$, where $x_i \in \text{Bin}_{i}$. In general,
\begin{equation}
P_i = \sum_{j=0}^d c_j x_i^j,
\end{equation}
where $d$ is the degree of the event rate polynomial.

The last row shows the total number of records, sum and mean.

You can use the method ``plot`` to visualize the histogram and mean curve.

```python
binning_table.plot()
```

##### Mean transformation

Now that we have checked the binned data, we can transform our original data into mean values.

```python
x_transform_mean = optb.transform(x)
```

## Advanced

Many of the advanced options have been covered in the previous tutorials with a binary target. **Check it out!** In this section, we focus on the mean monotonicity trend and the mean difference between bins.

#### Binning table statistical analysis

The ``analysis`` method performs a statistical analysis of the binning table, computing the Information Value (IV) and Herfindahl-Hirschman Index (HHI). Additionally, several statistical significance tests between consecutive bins of the contingency table are performed using the Student's t-test. The piecewise binning also includes several performance metrics for regression problems.

```python
binning_table.analysis()
```

    -------------------------------------------------
    OptimalBinning: Continuous Binning Table Analysis
    -------------------------------------------------

      General metrics

        Mean absolute error              3.68295105
        Mean squared error              26.03331040
        Median absolute error            2.72954866
        Explained variance               0.69161991
        R^2                              0.69161991
        MPE                             -0.05213932
        MAPE                             0.17801859
        SMAPE                            0.08405929
        MdAPE                            0.13009259
        SMdAPE                           0.06660004
        HHI                              0.11313253
        HHI (normalized)                 0.03250821
        Quality score                    0.21464367

      Significance tests

        Bin A  Bin B  t-statistic      p-value
            0      1     5.644492 3.313748e-07
            1      2     2.924528 5.175586e-03
            2      3     0.808313 4.206096e-01
            3      4     4.714124 6.127108e-06
            4      5     2.712699 8.189719e-03
            5      6     0.622294 5.353397e-01
            6      7     5.162973 8.663417e-07
            7      8     3.742511 4.988278e-04
            8      9     1.408305 1.643801e-01

#### Mean monotonicity

The monotonic_trend option permits forcing a monotonic trend to the mean curve. The default setting “auto” should be the preferred option, however, some business constraints might require to impose different trends. The default setting “auto” chooses the monotonic trend most likely to minimize the L1-norm from the options “ascending”, “descending”, “peak” and “valley” using a machine-learning-based classifier.

```python
variable = "INDUS"
x = df[variable].values
y = data.target
```

```python
optb = ContinuousOptimalPWBinning(name=variable, monotonic_trend="auto")
optb.fit(x, y)
```

    ContinuousOptimalPWBinning(name='INDUS')

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Sum
      Std
      Min
      Max
      Zeros count
      c0
      c1

      0
      (-inf, 3.35)
      63
      0.124506
      1994.0
      8.569841
      16.5
      50.0
      0
      31.428209
      -0.000000

      1
      [3.35, 5.04)
      57
      0.112648
      1615.2
      8.072710
      17.2
      50.0
      0
      42.928218
      -3.432839

      2
      [5.04, 6.66)
      66
      0.130435
      1723.7
      7.879078
      16.0
      50.0
      0
      25.626711
      0.000000

      3
      [6.66, 8.01)
      31
      0.061265
      692.0
      4.604886
      14.4
      35.2
      0
      52.694010
      -4.064159

      4
      [8.01, 16.57)
      100
      0.197628
      2045.5
      3.547348
      11.9
      28.7
      0
      21.858675
      -0.212150

      5
      [16.57, 18.84)
      132
      0.260870
      2165.3
      8.507336
      5.0
      50.0
      0
      21.603058
      -0.196723

      6
      [18.84, inf)
      57
      0.112648
      1165.9
      9.519086
      7.0
      50.0
      0
      17.896791
      0.000000

      7
      Special
      0
      0.000000
      0.0
      NaN
      NaN
      NaN
      0
      0.000000
      0.000000

      8
      Missing
      0
      0.000000
      0.0
      NaN
      NaN
      NaN
      0
      0.000000
      0.000000

      Totals

      506
      1.000000
      11401.6

      5.0
      50.0
      0
      -
      -

```python
binning_table.plot()
```

```python
binning_table.analysis()
```

    -------------------------------------------------
    OptimalBinning: Continuous Binning Table Analysis
    -------------------------------------------------

      General metrics

        Mean absolute error              5.38628982
        Mean squared error              59.01564719
        Median absolute error            4.02296554
        Explained variance               0.30092446
        R^2                              0.30092446
        MPE                             -0.12086396
        MAPE                             0.27859705
        SMAPE                            0.12156690
        MdAPE                            0.18811572
        SMdAPE                           0.09658000
        HHI                              0.16875752
        HHI (normalized)                 0.06485221
        Quality score                    0.74831364

      Significance tests

        Bin A  Bin B  t-statistic  p-value
            0      1     2.180865 0.031181
            1      2     1.537968 0.126745
            2      3     2.976660 0.003739
            3      4     2.075258 0.044180
            4      5     4.934158 0.000002
            5      6    -2.770230 0.006721

A smoother curve, keeping the valley monotonicity, can be achieved by using ``monotonic_trend="convex"``.

```python
optb = ContinuousOptimalPWBinning(name=variable, monotonic_trend="convex")
optb.fit(x, y)
```

    ContinuousOptimalPWBinning(monotonic_trend='convex', name='INDUS')

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Sum
      Std
      Min
      Max
      Zeros count
      c0
      c1

      0
      (-inf, 3.35)
      63
      0.124506
      1994.0
      8.569841
      16.5
      50.0
      0
      35.478615
      -1.760813

      1
      [3.35, 5.04)
      57
      0.112648
      1615.2
      8.072710
      17.2
      50.0
      0
      35.478615
      -1.760813

      2
      [5.04, 6.66)
      66
      0.130435
      1723.7
      7.879078
      16.0
      50.0
      0
      35.478615
      -1.760813

      3
      [6.66, 8.01)
      31
      0.061265
      692.0
      4.604886
      14.4
      35.2
      0
      35.478615
      -1.760813

      4
      [8.01, 16.57)
      100
      0.197628
      2045.5
      3.547348
      11.9
      28.7
      0
      24.492313
      -0.388383

      5
      [16.57, 18.84)
      132
      0.260870
      2165.3
      8.507336
      5.0
      50.0
      0
      18.898426
      -0.050792

      6
      [18.84, inf)
      57
      0.112648
      1165.9
      9.519086
      7.0
      50.0
      0
      18.898426
      -0.050792

      7
      Special
      0
      0.000000
      0.0
      NaN
      NaN
      NaN
      0
      0.000000
      0.000000

      8
      Missing
      0
      0.000000
      0.0
      NaN
      NaN
      NaN
      0
      0.000000
      0.000000

      Totals

      506
      1.000000
      11401.6

      5.0
      50.0
      0
      -
      -

```python
binning_table.plot()
```

```python
binning_table.analysis()
```

    -------------------------------------------------
    OptimalBinning: Continuous Binning Table Analysis
    -------------------------------------------------

      General metrics

        Mean absolute error              5.46871836
        Mean squared error              60.47066184
        Median absolute error            4.02086478
        Explained variance               0.28368894
        R^2                              0.28368894
        MPE                             -0.12278289
        MAPE                             0.28138163
        SMAPE                            0.12314489
        MdAPE                            0.19095659
        SMdAPE                           0.09398799
        HHI                              0.16875752
        HHI (normalized)                 0.06485221
        Quality score                    0.74831364

      Significance tests

        Bin A  Bin B  t-statistic  p-value
            0      1     2.180865 0.031181
            1      2     1.537968 0.126745
            2      3     2.976660 0.003739
            3      4     2.075258 0.044180
            4      5     4.934158 0.000002
            5      6    -2.770230 0.006721


```python
import sympy as sm
import numpy as np
from scipy import optimize
from scipy import interpolate
from bokeh.io import output_notebook,push_notebook,show
from bokeh.plotting import figure, show, output_file
from bokeh.layouts import row
from bokeh.models import ColumnDataSource, HoverTool, NumeralTickFormatter

import ipywidgets as widgets
from ipywidgets import interact

output_notebook()

```

    Loading BokehJS ...

```python
def total_utility(c, weight, theta):
    '''
    Sums utility for c for multiple years
    c is an array
    '''
    if theta == 1:
        uts =np.log(c)
    else:
        uts = (c**(1-theta)-1)/(1-theta)
    # sum of utitity
    t_u = np.dot(uts,weight)

    return t_u

def prod(k,l,alpha,a):
    return (k**alpha)*((a*l)**(1-alpha))

def tot_ut_multiple_sks_quick(sks, k0, l, b, weight, theta, alpha, delta):
    '''
    Finds total utitilty for a set of years with a savingsrate for each year
    '''
    t = len(sks)
    k_short = np.empty(t)
    k_short[0] = k0

    for i in range(1,t):
        k_short[i]=sks[i-1]*prod(k_short[i-1],l[i-1],alpha,b)+(1-delta)*k_short[i-1]

    y_short = prod(k_short,l,alpha,b)

    return total_utility(y_short*(1-sks)/l, weight, theta)

```

```python
def optimal_sks(t, b, l, weight, delta, alpha, theta, k0, first=True):
    '''
    Optimizes the utility of the representative household
    by simulating the predicted future
    '''
    obj = lambda sks: -tot_ut_multiple_sks_quick(sks, k0, l, b, weight, theta, alpha, delta)
    sks0 = np.linspace(alpha,0,t)

    bounds = np.full((t,2),[1e-8,0.99999])
    res = optimize.minimize(obj, sks0, method='SLSQP',
        bounds=bounds,)

    if res.success:
        if first:
            return res.x[0]
        else:
            return res.x
    else:
        print('Optimization was sadly not succesfull')
```

```python
def find_ssk_sk(k,b,delta,n,alpha):
    return (k**(1-alpha)*(delta+n))/b
```

```python

def plotting(x,y_names,  x_array, y_arrays,y_name ='Savings rate', title='Figure',
                colors= ['red','blue','green','purple','yellow'],
                legendlocation="top_center",tools="pan,wheel_zoom,box_zoom,reset,save",
                width=400, height=500):

    '''
    Bokeh plotting
    '''

     # Bokeh needs a name for the data that neither has spaces nor numbers
    # because we want the option to do this we define abitrairy calls via the alphabeth.
    calls = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o',
     'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
    y_calls = []

    for i in range(len(y_names)):
        y_calls.append(calls[i])

    tooltips=[(f'{x}','@x{0,0.00}')]

    for yn,yc in zip(y_names,y_calls):
        text = '@'+f'{yc}'+'{0.00}'
        tooltips.append((f'{y_name} for {yn}',text))

    hover = HoverTool(tooltips=tooltips)
    data = {'x': x_array}
    for yc, y_array in zip(y_calls,y_arrays):
        data[f'{yc}']=y_array

    source = ColumnDataSource(data)

    p = figure(plot_width=width, plot_height=height, title=f'{title}',
        tools=[hover,tools], x_axis_label=f'{x}', y_axis_label=f'{y_name}')

    for i,(yc,yn) in enumerate(zip(y_calls,y_names)):
        p.line(x='x', y=f'{yc}', source=source,
           legend= f'{yn}', color = colors[i])

    p.legend.location = legendlocation

    return p

```

```python
def solving_and_plotting_interactive_micro(
        t, b, n, weight, delta, alpha, l, k0s_bounds=[0.001,30], theta_bounds=[0.3,1.5], precision=100,
        width=1000, height=600):

    # data:
    k0s = np.linspace(k0s_bounds[0],k0s_bounds[1],precision)
    sks_plot = np.zeros(precision)
    theta = 0.5
    for i in range(k_n):
            sks_plot[i] = optimal_sks(t, b, l, weight, delta, alpha, theta, k0s[i])

    steadystates = find_ssk_sk(k0s,b,delta,n,alpha)

    '''
    Bokeh plotting
    '''
     # Bokeh needs a name for the data that neither has spaces nor numbers
    # because we want the option to do this we define abitrairy calls via the alphabeth.

    tooltips=[('Capital pr. capita','@x{0,0.00}'),('Savings rate in steady state','@y{0.00}'),
             (f'Savingsrate for \u03B8 = {theta}','@z{0.00}')]

    hover = HoverTool(tooltips=tooltips)
    tools="pan,wheel_zoom,box_zoom,reset,save"

    data = {'x' : k0s,'y' : steadystates, 'z' : sks_plot}

    source = ColumnDataSource(data)

    p = figure(plot_width=width, plot_height=height, title=f'Optimal savings rate for varying \u03B8 and capital pr. capita',
        tools=[hover,tools], x_axis_label=f'Capital pr. capita', y_axis_label=f'Savings rate')

    p.line(x='x', y='y', source=source, legend= 'Steady state', color = 'yellow')
    p.line(x='x', y='z', source=source, legend= f'\u03B8 = {theta}', color = 'blue')

    p.legend.location = 'top_center'

    def update_theta(new_theta):
        theta = new_theta

        # push_notebook is af bokeh->jupyter-specific comand to tell python to update the bokeh plot in jupyter
        push_notebook()

    # show the plot, notebook_handle is again bokeh->jupyter-specific to show the plot in the notebook
    show(p,notebook_handle=True)

    # Make a dropdown-widget, so the user can choose different stocks, the layout option, makes the options box wider
    # the style options make the desciption box wide enough for the description string to be read fully
    drop_down = widgets.Dropdown(options=stock_list, layout = {'width':'50%'},\
        description='Choose a stock or index:',style = {'description_width': 'initial'})

    # Call the function
    interact(update_name, stock = drop_down)

    return p

```

**Interactive Micro**

```python
# Parameters
# Macro:

alpha = 1/3
delta = 0.05
n = 0.008
b = 1
k0 = 5
l0 = 1

#Micro:
theta = 0.5
beta = 0.99

t = 100

# time preference utility weights and population growth
# are precomputed to save time:
weight = np.array([beta**i for i in range(t)])
l = np.array([l0*(1+n)**i for i in range(t)])
```

```python
k0s_bounds=[0.001,30]
theta_bounds=[0.3,1.5]
precision=50
width=600
height=600
tools="pan,wheel_zoom,box_zoom,reset,save"
```

```python
# data:
k0s = np.linspace(k0s_bounds[0],k0s_bounds[1],precision)
sks_plot = np.zeros(precision)
theta = 0.5
for i in range(precision):
        sks_plot[i] = optimal_sks(t, b, l, weight, delta, alpha, theta, k0s[i])

steadystates = find_ssk_sk(k0s,b,delta,n,alpha)

tooltips=[('Capital pr. capita','@x{0,0.00}'),('Savings rate in steady state','@y{0.00}'),
            (f'Optimal savingsrate','@z{0.00}')]

hover = HoverTool(tooltips=tooltips)

data = {'x' : k0s,'y' : steadystates, 'z' : sks_plot,'theta_legend': [f'\u03B8 = {theta}' for i in range(precision)]}

source = ColumnDataSource(data)

p = figure(plot_width=width, plot_height=height, title=f'Optimal savings rate for varying \u03B8 and capital pr. capita',
        tools=[hover,tools], x_axis_label=f'Capital pr. capita', y_axis_label=f'Savings rate')

p.line(x='x', y='y', source=source, legend= 'Steady state', color = 'yellow')
p.line(x='x', y='z', source=source, legend= 'theta_legend', color = 'blue')

p.legend.location = 'top_center'

def update_theta(Theta=0.5):

    for i in range(precision):
        sks_plot[i] = optimal_sks(t, b, l, weight, delta, alpha, Theta, k0s[i])
    data['z']=sks_plot
    data['theta_legend']= [f'\u03B8 = {Theta}' for i in range(precision)]
    source.data = ColumnDataSource(data).data

    push_notebook()

show(p,notebook_handle=True)

interact(update_theta, Theta = (0,6,0.1))
```

    interactive(children=(FloatSlider(value=0.5, description='Theta', max=6.0), Output()), _dom_classes=('widget-i…

    <function __main__.update_theta(Theta=0.5)>

Too slow

```python
def ut(c,theta):
    if theta == 1:
        return np.log(c)
    else:
        return (c**(1-theta)-1)/(1-theta)
```

```python
thetas=[0.25,0.5,1,1.5,2]
c = np.linspace(0.01,6,1000)
us = []
for i in range(5):
    u = ut(c,thetas[i])
    us.append(u)
p = plotting('C',['0.5','0.7','1','1.5','5'],c,us,y_name='utility')
show(p,notebook_handle=True)
```

<Bokeh Notebook handle for In[61]>