## Disordered and complex systems

# Phase transition in the Ising model.

**Author:** João António - joaoantant@gmail.com \& github.com/JoaoAnt/.

**Based on:** the instructions for Project 4 from professor Alexander V. Goltsev.

**Note:**
For simplicity, in the graphs, I have used the color "**IndianRed**","**MediumOrchid**" and "**DarkCyan**" when it is a value obtained through this simulation and have used the color "**RoyalBlue**" for theoretical predictions. I have also used the notation M Teor. and $\chi$ Teor. for the theoretical predictions and used M and $\chi$ for the simulated ones.

```python
import numpy as np
import matplotlib.pyplot as plt
```

# Ising model on the ring
The energy of the Ising model on a ring is
\begin{equation}
E=-J \sum_{n-1}^{N}S_nS_{n+1}-H\sum_{n=1}^N S_n,
\end{equation}

where $S_n=\pm 1$, $S_{N+1}=S_1$ and $J$ is the exchange integral.

The energy of spin $S_n$ interacting with nearest-neighbouring spins in a magnetic field H is:

\begin{equation}
E_n(S_n)=-JS_n(S_{n-1}+S_{n+1})-HS_n.
\end{equation}

To calculate the magnetization I have used:

\begin{equation}
M^{(\alpha)} = \frac{1}{N} \sum_{n=1}^N S_n^{(\alpha)},
\end{equation}

where $\alpha$ is the counter of microstates, goes from 1 until m.

And lastly, to calculate the average over microstates:

\begin{equation}
<M>=\frac{1}{m}\sum_{\alpha=1}^m M^{(\alpha)}.
\end{equation}

All this equations where used as functions, except the first one.

**Find the temperature dependence of the magnetization.**

### Functions Used

```python
def calcEnergy(spin,RaSpin):
    '''Energy of a given configuration'''
    energy = -J*spin[RaSpin]*(spin[(RaSpin-1)%N]+spin[(RaSpin+1)%N])-H*spin[RaSpin]
    return energy

def calcMag(spin):
    '''Magnetization of a given configuration'''
    mag = 1/N*np.sum(spin)
    return mag

def calcAvMag(Magn,K):
    '''Calculate the average magnetization for a given K'''
    AvMag = 1/(K*100)*np.sum(Magn[100:K*100])
    return AvMag
```

### Input Table

| Parameters 	| Symbol 	| = 	| Value 	|
|:-------------------------------:	|:---------:	|:-:	|:---------:	|
| Exchange integral 	| J 	| = 	| 1 	|
| Number of spins 	| N 	| = 	| 1000 	|
| Number of generated microstates 	| m 	| = 	| 100 000 	|
| Temperature range 	| T 	| = 	| [0.1, 10] 	|
| Temperature step 	| $\Delta$T 	| = 	| 0.05 	|
| Magnetic field 	| H 	| = 	| 0.1 	|

```python
J=1
N=1000
m=100000
deltaT=0.05
T=0.1
Tfin=10
H=0.1
```

### Initializing variables that vary with T

```python
AverageMagn=np.zeros(int(m/100))
AverageMagnA=np.zeros(int(m/100))
AverageMagnB=np.zeros(int(m/100))
AverageMagn2=np.zeros(int((Tfin-T)/deltaT))
FinalMagn=np.zeros(int((Tfin-T)/deltaT))
```

### Metropolis algorithm and average magnetization calculus

```python
for count in range(int((Tfin-T)/deltaT)):

    #Initial Conditions
    Spins=np.ones(N)
    Magn=np.ones(m)

    for alpha in range(m):
        #Metropolis Algorithm
        RaSpin=np.random.randint(0,N-1)
        Old=calcEnergy(Spins,RaSpin)
        Spins[RaSpin]*=-1
        EnergyDiference=calcEnergy(Spins,RaSpin)-Old

    #if EnergyDiference<=0:        #Do Nothing
        if EnergyDiference>0:
            w=np.exp(-1/T*EnergyDiference)
            r=np.random.uniform(0, 1)
        #if r<=w:            #Do Nothing
            if r>w:
                Spins[RaSpin]*=-1
        Magn[alpha]=calcMag(Spins)

    #Data for the 1st Plot
    if T==0.1:
        for K in range(2,int(m/100)):
            AverageMagn[K]=calcAvMag(Magn,K)
    if round(T,3)==1:
        for K in range(2,int(m/100)):
            AverageMagnA[K]=calcAvMag(Magn,K)
    if round(T,3)==2:
        for K in range(2,int(m/100)):
            AverageMagnB[K]=calcAvMag(Magn,K)
    #Data for 2nd plot
    AverageMagn2[count]=calcAvMag(Magn,N)
    FinalMagn[count]=Magn[-1]
    T+=0.05;
```

### Plotting the data
### Average Magnetization vs Microstates

```python
f = plt.figure(figsize=(10, 4));
X = np.linspace(2*100, N*100, 1000);
plt.scatter(X[2:], AverageMagn[2:], s=50, marker='o', color='IndianRed', label='T=0.1')
plt.scatter(X[2:], AverageMagnA[2:], s=50, marker='o', color='MediumOrchid', label='T=1')
plt.scatter(X[2:], AverageMagnB[2:], s=50, marker='o', color='DarkCyan', label='T=2')
plt.xlabel("m", fontsize=20);
plt.ylabel("$<M>$", fontsize=20);
plt.axis('tight');
plt.legend(loc="upper right");
plt.title(r"$\bf{Fig.1.1:}$"+'How <M> tends to equilibrium for H=0.1 vs microstates')
plt.show()
```

In $\bf{Fig.1.1}$ it is possible to see how the average magnetization ($<M>$) varies with the microstates ($m$). As we can see for values below the critical temperature (T=0.1) it seems to stabilize near the value of 1, and as we increase the temperature until the critical temperature, the more microstates it takes to stabilize, and stabilizes at a lower value than previous. Lastly we have the value of (T=2) where we can see that after the critical temperature the average magnetization stabilizes for even smaller values.

### Magnetization vs Temperature

```python
f = plt.figure(figsize=(10, 4));
T = np.arange(0.1, 10, 0.05);
plt.scatter(T, AverageMagn2, s=50, marker='o', color='IndianRed')
plt.xlabel("T", fontsize=20);
plt.ylabel("$<M>$", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.1.2:}$"+'Average magnetization vs Temperature in H=0.1')

f = plt.figure(figsize=(10, 4));
T = np.arange(0.1, 10, 0.05);
plt.scatter(T, FinalMagn, s=50, marker='o', color='IndianRed')
plt.xlabel("T", fontsize=20);
plt.ylabel("$M$", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.1.3:}$"+'Magnetization vs Temperature in H=0.1')

f = plt.figure(figsize=(10, 4));
Magn2T=np.sinh(1/T*H)/(np.sqrt((np.sinh(1/T*H))**2+np.exp(-4*1/T*J)))
plt.scatter(T, Magn2T, s=50, marker='o', color='RoyalBlue')
plt.xlabel("T", fontsize=20);
plt.ylabel("$M$ Teor.", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.1.4:}$"+'Theoretical magnetization vs Temperature in H=0.1');
plt.show()
```

In the next $\bf{Fig.1.2}$ it is possible to see the average magnetization vs Temperature obtained through the simulation, this one is the one that should be compared to $\bf{Fig.1.4}$ where we have the theoretical prediction of the magnetization vs temperature for the same H.

In the $\bf{Fig.1.3}$ we can see the magnetization vs temperature obtained through simulation, despite having more or less the same behavior as the other two just mentioned, it has more noise to the curve (more fluctuations), since the minimum of energy is very unstable, all states have a certain probability of being visited even if they are not the minimum energy state.

As this is a simple one dimensional Ising Model, there is no phase transition.

### Magnetizations vs Temperature

```python
f = plt.figure(figsize=(10, 4));
plt.scatter(T, Magn2T, s=50, marker='o', color='RoyalBlue',label='M Teor.')
plt.scatter(T, FinalMagn, s=50, marker='o', color='MediumOrchid',label='M')
plt.scatter(T, AverageMagn2, s=50, marker='o', color='IndianRed',label='<M>')
plt.xlabel("T", fontsize=20);
plt.ylabel("$M$ Teor., $M$ and $<M>$", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.1.5:}$"+'Magnetizations vs Temperature in H=0.1');
plt.legend(loc="upper right");
plt.show()
```

In this $\bf{Fig.1.5:}$ we can see that the M Teor. and the $<M>$ have the same behaviour, like explained previously, and the M as nearly the same also, but with a lot more noise.

It is harder to see but the $<M>$ is slightly more to the right when compared to the M Teor., this is a effect of a non-infinite system and non-infinite time, if we had infinite time and infinite size system the curves would coincide perfectly.

# Ising model with all-to-all interaction (long-ranged interaction)

The energy of the Ising model with the long-ranged interaction is:

\begin{equation}
E=\frac{-J}{N}\sum_{n=1}^{N-1}\sum_{m=n+1}^N S_nS_m -H\sum_{n=1}^N S_n,
\end{equation}

where $S_n=\pm1$ and J is the exchange integral again.

The energy of spin $S_n$ interacting with the other spins in a magnetic
field H is:

\begin{equation}
E_n(S_n)=\frac{-J}{N}S_n\sum_{m=1, m\neq n}^N S_m-HS_n.
\end{equation}

To calculate the magnetization in the microstate $\alpha$ (goes from 1 untill m), I have used:

\begin{equation}
M^{(\alpha)} = \frac{1}{N} \sum_{n=1}^N S_n^{(\alpha)}.
\end{equation}

To calculate the average over microstates:

\begin{equation}
<M>=\frac{1}{m}\sum_{\alpha=1}^m M^{(\alpha)}.
\end{equation}

To calculate the susceptability:
\begin{equation}
\chi ^{(\alpha)}=\frac{N}{T} (\frac{1}{m}\sum_{\alpha=1}^{m}(M^{(a)})^2-<M>^2)
\end{equation}

All this equations where used as functions.

 Plot M and $\chi$ versus T. Analyse a temperature dependence of M and $\chi$ near the critical point T_c = J.

### Functions Used

```python
def calcEnergy(spin,RaSpin):
    '''Energy of a given configuration'''
    energy = -J/N*spin[RaSpin]*(spin.sum()-spin[RaSpin])-H*spin[RaSpin]
    return energy

def calcMag(spin):
    '''Magnetization of a given configuration'''
    mag = 1/N*np.sum(spin)
    return mag

def calcAvMag(Magn):
    '''Calculate the average magnetization'''
    AvMag = 1/m*np.sum(Magn)
    return AvMag

def calcSusc(Magn,AvMag):
    '''Calculate the average magnetization'''
    Temp=0
    for alpha in range(m):
        Temp+=Magn[alpha]**2
    susc=N/T*(1/m*Temp-AvMag**2)
    return susc
```

### Input Table

| Parameters 	| Symbol 	| = 	| Value 	|
|:-------------------------------:	|:---------:	|:-:	|:---------:	|
| Exchange integral 	| J 	| = 	| 1 	|
| Number of spins 	| N 	| = 	| 1000 	|
| Number of generated microstates 	| m 	| = 	| 50 000 	|
| Temperature range 	| T 	| = 	| [0.1, 10] 	|
| Temperature step 	| $\Delta$T 	| = 	| 0.05 	|
| Magnetic field 	| H 	| = 	| 0 and 0.001 	|

```python
J=1
N=1000
m=50000
deltaT=0.05
T=0.1
Tfin=10
H=0
```

## Using H=0
### Initializing variables that vary with T

```python
AvMag2=np.zeros(int((Tfin-T)/deltaT))
Suscp2=np.zeros(int((Tfin-T)/deltaT))
```

### Metropolis algorithm, average magnetization and susceptibility calculus

```python
for count in range(int((Tfin-T)/deltaT)):

    #Initial Conditions
    Spins=np.ones(N)
    Magn=np.ones(m)

    for alpha in range(m):
        #Metropolis Algorithm
        RaSpin=np.random.randint(0,N-1)
        Old=calcEnergy(Spins,RaSpin)
        Spins[RaSpin]*=-1
        EnergyDiference=calcEnergy(Spins,RaSpin)-Old

    #if EnergyDiference<=0:        #Do Nothing
        if EnergyDiference>0:
            w=np.exp(-1/T*EnergyDiference)
            r=np.random.uniform(0, 1)
        #if r<=w:            #Do Nothing
            if r>w:
                Spins[RaSpin]*=-1
        Magn[alpha]=calcMag(Spins)

    #Data for 2nd plot
    AvMag2[count]=calcAvMag(Magn)
    Suscp2[count]=calcSusc(Magn,AvMag2[count])
    T+=0.05;
```

### Plotting the data
### Magnetization vs Temperature

```python
f = plt.figure(figsize=(10, 4));
T = np.arange(0.1, 10, 0.05);
plt.scatter(T, AvMag2, s=50, marker='o', color='IndianRed')
plt.xlabel("T", fontsize=20);
plt.ylabel("$<M>$", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.1:}$"+'Average magnetization vs Temperature in H=0')

f = plt.figure(figsize=(10, 4));
Magn2T=1
for i in range(500):
    Magn2T=np.tanh(1/T*J*Magn2T+1/T*H)
plt.scatter(T, Magn2T, s=50, marker='o', color='RoyalBlue')
plt.xlabel("T", fontsize=20);
plt.ylabel("$M$ Teor.", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.2:}$"+'Theoretical magnetization vs Temperature in H=0');
plt.show()

f = plt.figure(figsize=(10, 4));
plt.scatter(T, AvMag2, s=50, marker='o', color='IndianRed',label='<M>')
plt.scatter(T, Magn2T, s=50, marker='o', color='RoyalBlue',label='M Teor.')
plt.xlabel("T", fontsize=20);
plt.ylabel("$<M>$ and $M$ Teor", fontsize=20);
plt.axis('tight');
plt.legend(loc="upper right");
plt.title(r"$\bf{Fig.2.3:}$"+'Magnetizations vs Temperature in H=0')
plt.show()
```

In the next $\bf{Fig.2.1}$ it is possible to see the average magnetization vs temperature obtained through the simulation, to be compared to $\bf{Fig.2.2}$ where we have the theoretical prediction of the magnetization vs temperature for the same H.

For the theoretical value of magnetization, since it is a recursive equation, I have iterated it over 500 times, to obtain the values that are plotted.

Lastly in $\bf{Fig.2.3}$ I have placed both curves in the same plot, as we can see the transition is not as sharp in the simulation as it is predicted, this is once more an effect of non-infinite size and non-infinite time.

### Susceptibility vs Temperature

```python
f = plt.figure(figsize=(10, 4));
plt.scatter(T, Suscp2, s=50, marker='o', color='IndianRed')
plt.xlabel("T", fontsize=20);
plt.ylabel("$\chi$", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.4:}$"+'Susceptibility vs Temperature in H=0')
plt.show()

f = plt.figure(figsize=(10, 4));
SuscpT=1/T/(np.cosh(1/T*J*Magn2T+H/T)**2-J/T)
plt.scatter(T, SuscpT, s=50, marker='o', color='RoyalBlue')
plt.xlabel("T", fontsize=20);
plt.ylabel("$\chi$ Teor.", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.5:}$"+'Theoretical susceptibility vs Temperature in H=0');
plt.show()

f = plt.figure(figsize=(10, 4));
plt.scatter(T, Suscp2, s=50, marker='o', color='IndianRed',label='$\chi$')
plt.scatter(T, SuscpT, s=50, marker='o', color='RoyalBlue',label='$\chi$ Teor.')
plt.xlabel("T", fontsize=20);
plt.ylabel("$\chi$, $\chi$ Teor.", fontsize=20);
plt.axis('tight');
plt.legend(loc="upper right");
plt.title(r"$\bf{Fig.2.6:}$"+'Susceptibilitys vs Temperature in H=0')
plt.show()
```

In the $\bf{Fig.2.4}$ I have plotted the susceptibility vs temperature for a H equal to zero. In the $\bf{Fig.2.5}$ we have the theoretical prediction, this two images are relative similar despite being a bit different near the critical temperature, this shows that near the critical temperature the system is even more unstable.

With the all to all interaction is now possible to see the phase transitions at the $T_c$ in this case in $T_c$ is in $T=1$, the magnetization shows a simple phase transition that is continuous, as for the susceptibility it also presents a phase transition, but this time is not a continuous one, the susceptibility in the $T_c$ goes to infinity.

Once more I have created the combination of the curves in $\bf{Fig.2.6}$, where we can see once more the $<M>$ going more to the left than the theoretical, but also this peak of magnetization has more width than the theoretical one, so it also goes more to the right, i.e., takes more temperature than the theoretical to reach zero (for the same amount of steps), and the phase transition starts earlier (with less temperature).

## Using H=0.01##
### Initializing variables that vary with T

```python
H=0.01
T=0.1
AvMag=np.zeros(int((Tfin-T)/deltaT))
Suscp=np.zeros(int((Tfin-T)/deltaT))
```

### Metropolis algorithm, average magnetization and susceptibility calculus

```python
for count in range(int((Tfin-T)/deltaT)):

    #Initial Conditions
    Spins=np.ones(N)
    Magn=np.ones(m)

    for alpha in range(m):
        #Metropolis Algorithm
        RaSpin=np.random.randint(0,N-1)
        Old=calcEnergy(Spins,RaSpin)
        Spins[RaSpin]*=-1
        EnergyDiference=calcEnergy(Spins,RaSpin)-Old

    #if EnergyDiference<=0:        #Do Nothing
        if EnergyDiference>0:
            w=np.exp(-1/T*EnergyDiference)
            r=np.random.uniform(0, 1)
        #if r<=w:            #Do Nothing
            if r>w:
                Spins[RaSpin]*=-1
        Magn[alpha]=calcMag(Spins)

    #Data for 2nd plot
    AvMag[count]=calcAvMag(Magn)
    Suscp[count]=calcSusc(Magn,AvMag[count])
    T+=0.05;
```

### Plotting the data
### Magnetization vs Temperature

```python
f = plt.figure(figsize=(10, 4));
T = np.arange(0.1, 10, 0.05);
plt.scatter(T, AvMag, s=50, marker='o', color='IndianRed')
plt.xlabel("T", fontsize=20);
plt.ylabel("$<M>$", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.7:}$"+'Average magnetization vs Temperature in H=0.01')
plt.show()

f = plt.figure(figsize=(10, 4));
Magn2T=1
for i in range(500):
    Magn2T=np.tanh(1/T*J*Magn2T+1/T*H)
plt.scatter(T, Magn2T, s=50, marker='o', color='RoyalBlue')
plt.xlabel("T", fontsize=20);
plt.ylabel("$M$ Teor.", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.8:}$"+'Theoretical magnetization vs Temperature in H=0.01');
plt.show()

f = plt.figure(figsize=(10, 4));
plt.scatter(T, AvMag, s=50, marker='o', color='IndianRed',label='<M>')
plt.scatter(T, Magn2T, s=50, marker='o', color='RoyalBlue',label='M Teor.')
plt.xlabel("T", fontsize=20);
plt.ylabel("$<M>$, $M$ Teor.", fontsize=20);
plt.axis('tight');
plt.legend(loc="upper right");
plt.title(r"$\bf{Fig.2.9:}$"+'Magnetizations vs Temperature in H=0.01')
plt.show()
```

Once again, in the next $\bf{Fig.2.7}$ it is possible to see the average magnetization vs temperature obtained through the simulation, to be compared to $\bf{Fig.2.8}$ where we have the theoretical prediction of the magnetization vs temperature for the same H (now H=0.01).

Lastly in $\bf{Fig.2.9}$ I have placed both curves in the same plot, as we can see the transition is not as sharp in the simulation as it is predicted, this is once more an effect of non-infinite size and non-infinite time. It's not as evident as in the $\bf{Fig.2.3}$.

### Susceptibility vs Temperature

```python
f = plt.figure(figsize=(10, 4));
plt.scatter(T, Suscp, s=50, marker='o', color='IndianRed')
plt.xlabel("T", fontsize=20);
plt.ylabel("$\chi$", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.10:}$"+'Susceptibility vs Temperature in H=0.01')
plt.show()

f = plt.figure(figsize=(10, 4));
SuscpT=1/T/(np.cosh(1/T*J*Magn2T+H/T)**2-J/T)
plt.scatter(T, SuscpT, s=50, marker='o', color='RoyalBlue')
plt.xlabel("T", fontsize=20);
plt.ylabel("$\chi$ Teor.", fontsize=20);
plt.axis('tight');
plt.title(r"$\bf{Fig.2.11:}$"+'Theoretical susceptibility vs Temperature in H=0.01');
plt.show()

f = plt.figure(figsize=(10, 4));
plt.scatter(T, Suscp, s=50, marker='o', color='IndianRed',label='$\chi$')
plt.scatter(T, SuscpT, s=50, marker='o', color='RoyalBlue',label='$\chi$ Teor.')
plt.xlabel("T", fontsize=20);
plt.ylabel("$\chi$, $\chi$ Teor.", fontsize=20);
plt.axis('tight');
plt.legend(loc="upper right");
plt.title(r"$\bf{Fig.2.12:}$"+'Susceptibilitys vs Temperature in H=0.01')
plt.show()
```

In the $\bf{Fig.2.10}$ I have plotted the susceptibility vs temperature for a H equal to zero. In the $\bf{Fig.2.11}$ we have the theoretical prediction, this two images are relative similar despite being a bit different near the critical temperature.

Finally in $\bf{Fig.2.11}$ I have both curves for comparison, once more we can see that the phase transition starts earlier (with less T, i.e. more to the left), that has more width (requires more T to restabilize after phase transition). All of this are effects of non-infinity size, and non-infinity time, like in the previous comparisons.

Project Notes/2019 DSC3 Phase transition in the Ising model.ipynb


```python
# importamos las librerías necesarias
%matplotlib inline
import random
import tsfresh
import os
import math
from scipy import stats
from scipy.spatial.distance import pdist
from math import sqrt, log, floor
from fastdtw import fastdtw
import ipywidgets as widgets
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import numpy as np
import pandas as pd
import seaborn as sns
from statistics import mean
from scipy.spatial.distance import euclidean
import scipy.cluster.hierarchy as hac
from scipy.cluster.hierarchy import fcluster
from sklearn.metrics.pairwise import pairwise_distances
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN
from sklearn.manifold import TSNE
from sklearn.metrics import normalized_mutual_info_score, adjusted_rand_score, silhouette_score, silhouette_samples
from sklearn.metrics import mean_squared_error
from scipy.spatial import distance

sns.set(style='white')
# "fix" the randomness for reproducibility
random.seed(42)
```

```python
!pip install ipywidgets
```

    Collecting ipywidgets
      Downloading ipywidgets-7.6.3-py2.py3-none-any.whl (121 kB)
    Requirement already satisfied: ipykernel>=4.5.1 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipywidgets) (5.3.4)
    Collecting jupyterlab-widgets>=1.0.0
      Downloading jupyterlab_widgets-1.0.0-py3-none-any.whl (243 kB)
    Requirement already satisfied: nbformat>=4.2.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipywidgets) (5.1.2)
    Collecting widgetsnbextension~=3.5.0
      Downloading widgetsnbextension-3.5.1-py2.py3-none-any.whl (2.2 MB)
    Requirement already satisfied: traitlets>=4.3.1 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipywidgets) (5.0.5)
    Requirement already satisfied: ipython>=4.0.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipywidgets) (7.21.0)
    Requirement already satisfied: jupyter-client in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipykernel>=4.5.1->ipywidgets) (6.1.7)
    Requirement already satisfied: tornado>=4.2 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipykernel>=4.5.1->ipywidgets) (6.1)
    Requirement already satisfied: backcall in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (0.2.0)
    Requirement already satisfied: jedi>=0.16 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (0.17.0)
    Requirement already satisfied: colorama in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (0.4.4)
    Requirement already satisfied: decorator in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (4.4.2)
    Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (3.0.8)
    Requirement already satisfied: setuptools>=18.5 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (52.0.0.post20210125)
    Requirement already satisfied: pygments in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (2.8.0)
    Requirement already satisfied: pickleshare in d:\userfiles\anaconda\envs\ia\lib\site-packages (from ipython>=4.0.0->ipywidgets) (0.7.5)
    Requirement already satisfied: parso>=0.7.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets) (0.8.1)
    Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbformat>=4.2.0->ipywidgets) (3.2.0)
    Requirement already satisfied: ipython-genutils in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbformat>=4.2.0->ipywidgets) (0.2.0)
    Requirement already satisfied: jupyter-core in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbformat>=4.2.0->ipywidgets) (4.7.1)
    Requirement already satisfied: attrs>=17.4.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (20.3.0)
    Requirement already satisfied: six>=1.11.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (1.15.0)
    Requirement already satisfied: pyrsistent>=0.14.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (0.17.3)
    Requirement already satisfied: wcwidth in d:\userfiles\anaconda\envs\ia\lib\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets) (0.2.5)
    Requirement already satisfied: notebook>=4.4.1 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from widgetsnbextension~=3.5.0->ipywidgets) (6.2.0)
    Requirement already satisfied: jinja2 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (2.11.3)
    Requirement already satisfied: terminado>=0.8.3 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.9.2)
    Requirement already satisfied: argon2-cffi in d:\userfiles\anaconda\envs\ia\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (20.1.0)
    Requirement already satisfied: prometheus-client in d:\userfiles\anaconda\envs\ia\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.9.0)
    Requirement already satisfied: pyzmq>=17 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (20.0.0)
    Requirement already satisfied: Send2Trash>=1.5.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.5.0)
    Requirement already satisfied: nbconvert in d:\userfiles\anaconda\envs\ia\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (6.0.7)
    Requirement already satisfied: python-dateutil>=2.1 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets) (2.8.1)
    Requirement already satisfied: pywin32>=1.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from jupyter-core->nbformat>=4.2.0->ipywidgets) (227)
    Requirement already satisfied: pywinpty>=0.5 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from terminado>=0.8.3->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.5.7)
    Requirement already satisfied: cffi>=1.0.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.14.5)
    Requirement already satisfied: pycparser in d:\userfiles\anaconda\envs\ia\lib\site-packages (from cffi>=1.0.0->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (2.20)
    Requirement already satisfied: MarkupSafe>=0.23 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.1.1)
    Requirement already satisfied: mistune<2,>=0.8.1 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.8.4)
    Requirement already satisfied: bleach in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (3.3.0)
    Requirement already satisfied: entrypoints>=0.2.2 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.3)
    Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.5.3)
    Requirement already satisfied: jupyterlab-pygments in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.1.2)
    Requirement already satisfied: pandocfilters>=1.4.1 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.4.3)
    Requirement already satisfied: defusedxml in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.6.0)
    Requirement already satisfied: testpath in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.4.4)
    Requirement already satisfied: async-generator in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.10)
    Requirement already satisfied: nest-asyncio in d:\userfiles\anaconda\envs\ia\lib\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.5.1)
    Requirement already satisfied: webencodings in d:\userfiles\anaconda\envs\ia\lib\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.5.1)
    Requirement already satisfied: packaging in d:\userfiles\anaconda\envs\ia\lib\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (20.9)
    Requirement already satisfied: pyparsing>=2.0.2 in d:\userfiles\anaconda\envs\ia\lib\site-packages (from packaging->bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (2.4.7)
    Installing collected packages: widgetsnbextension, jupyterlab-widgets, ipywidgets
    Successfully installed ipywidgets-7.6.3 jupyterlab-widgets-1.0.0 widgetsnbextension-3.5.1

### Dataset

Los datos son series temporales (casos semanales de Dengue) de distintos distritos de Paraguay

```python
path = "./data/Notificaciones/"
filename_read = os.path.join(path,"normalizado.csv")
notificaciones = pd.read_csv(filename_read,delimiter=",",engine='python')
notificaciones.shape
```

```python
listaMunicp = notificaciones['distrito_nombre'].tolist()
listaMunicp = list(dict.fromkeys(listaMunicp))
print('Son ', len(listaMunicp), ' distritos')
listaMunicp.sort()
print(listaMunicp)
```

A continuación tomamos las series temporales que leímos y vemos como quedan

```python
timeSeries = pd.DataFrame()
for muni in listaMunicp:
    municipio=notificaciones['distrito_nombre']==muni
    notif_x_municp=notificaciones[municipio]
    notif_x_municp = notif_x_municp.reset_index(drop=True)
    notif_x_municp = notif_x_municp['incidencia']
    notif_x_municp = notif_x_municp.replace('nan', np.nan).fillna(0.000001)
    notif_x_municp = notif_x_municp.replace([np.inf, -np.inf], np.nan).fillna(0.000001)
    timeSeries = timeSeries.append(notif_x_municp)
    ax = sns.tsplot(ax=None, data=notif_x_municp.values, err_style="unit_traces")
plt.show()
```

```python
#timeseries shape
n=217
```

```python
timeSeries.shape
```

```python
timeSeries.describe()
```

### Análisis de grupos (Clustering)

El Clustering o la clusterización es un proceso importante dentro del Machine learning. Este proceso desarrolla una acción fundamental que le permite a los algoritmos de aprendizaje automatizado entrenar y conocer de forma adecuada los datos con los que desarrollan sus actividades. Tiene como finalidad principal lograr el agrupamiento de conjuntos de objetos no etiquetados, para lograr construir subconjuntos de datos conocidos como Clusters. Cada cluster dentro de un grafo está formado por una colección de objetos o datos que a términos de análisis resultan similares entre si, pero que poseen elementos diferenciales con respecto a otros objetos pertenecientes al conjunto de datos y que pueden conformar un cluster independiente.

Aunque los datos no necesariamente son tan fáciles de agrupar

### Métricas de similitud

  Para medir lo similares ( o disimilares) que son los individuos existe una enorme cantidad de índices de similaridad y de disimilaridad o divergencia. Todos ellos tienen propiedades y utilidades distintas y habrá que ser consciente de ellas para su correcta aplicación al caso que nos ocupe.
La mayor parte de estos índices serán o bien, indicadores basados en la distancia (considerando a los individuos como vectores en el espacio de las variables) (en este sentido un elevado valor de la distancia entre dos individuos nos indicará un alto grado de disimilaridad entre ellos); o bien, indicadores basados en coeficientes de correlación ; o bien basados en tablas de datos de posesión o no de una serie de atributos.
A continuación mostramos las funciones de:
* Distancia Euclidiana
* Error cuadrático medio
* Fast Dynamic Time Warping
* Correlación de Pearson y
* Correlación de Spearman.

Existen muchas otras métricas y depende de la naturaleza de cada problema decidir cuál usar. Por ejemplo, *Fast Dymanic Time Warping* es una medida de similitud diseña especialmente para series temporales.

```python
#Euclidean
def euclidean(x, y):
    r=np.linalg.norm(x-y)
    if math.isnan(r):
        r=1
    #print(r)
    return r
```

```python
#RMSE
def rmse(x, y):
    r=sqrt(mean_squared_error(x,y))
    if math.isnan(r):
        r=1
    #print(r)
    return r
```

```python
#Fast Dynamic time warping
def fast_DTW(x, y):
    r, _ = fastdtw(x, y, dist=euclidean)
    if math.isnan(r):
        r=1
    #print(r)
    return r
```

```python
#Correlation
def corr(x, y):
    r=np.dot(x-mean(x),y-mean(y))/((np.linalg.norm(x-mean(x)))*(np.linalg.norm(y-mean(y))))
    if math.isnan(r):
        r=0
    #print(r)
    return 1 - r
```

```python
#Spearman
def scorr(x, y):
    r = stats.spearmanr(x, y)[0]
    if math.isnan(r):
        r=0
    #print(r)
    return 1 - r
```

```python
# compute distances using LCSS

# function for LCSS computation
# based on implementation from
# https://rosettacode.org/wiki/Longest_common_subsequence
def lcs(a, b):
    lengths = [[0 for j in range(len(b)+1)] for i in range(len(a)+1)]
    # row 0 and column 0 are initialized to 0 already
    for i, x in enumerate(a):
        for j, y in enumerate(b):
            if x == y:
                lengths[i+1][j+1] = lengths[i][j] + 1
            else:
                lengths[i+1][j+1] = max(lengths[i+1][j], lengths[i][j+1])
    x, y = len(a), len(b)
    result = lengths[x][y]
    return result

def discretise(x):
    return int(x * 10)

def multidim_lcs(a, b):
    a = a.applymap(discretise)
    b = b.applymap(discretise)
    rows, dims = a.shape
    lcss = [lcs(a[i+2], b[i+2]) for i in range(dims)]
    return 1 - sum(lcss) / (rows * dims)
```

```python
#Distancias para kmeans
#Euclidean
euclidean_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        euclidean_dist[i,j] = euclidean(timeSeries.iloc[i].values.flatten(), timeSeries.iloc[j].values.flatten())
#RMSE
rmse_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        rmse_dist[i,j] = rmse(timeSeries.iloc[i].values.flatten(), timeSeries.iloc[j].values.flatten())
#Corr
corr_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        corr_dist[i,j] = corr(timeSeries.iloc[i].values.flatten(), timeSeries.iloc[j].values.flatten())
#scorr
scorr_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        scorr_dist[i,j] = scorr(timeSeries.iloc[i].values.flatten(), timeSeries.iloc[j].values.flatten())
#DTW
dtw_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        dtw_dist[i,j] = fast_DTW(timeSeries.iloc[i].values.flatten(), timeSeries.iloc[j].values.flatten())
```

### Determinar el número de clusters a formar

La mayoría de las técnicas de clustering necesitan como *input* el número de clusters a formar, para eso lo que se hace es hacer una prueba con diferentes números de cluster y nos quedamos con el que dió menor error en general. Para medir ese error utilizamos **Silhouette score**.

El **Silhoutte score** se puede utilizar para estudiar la distancia de separación entre los clusters resultantes, especialmente si no hay conocimiento previo de cuáles son los verdaderos grupos para cada objeto, que es el caso más común en aplicaciones reales.

El Silhouette score $s(i)$ se calcula:
\begin{equation}
s(i)=\dfrac{b(i)-a(i)}{max(b(i),a(i))}
\end{equation}

Definamos $a (i)$ como la distancia media del punto $(i)$ a todos los demás puntos del grupo que se le asignó ($A$). Podemos interpretar $a (i)$ como qué tan bien se asigna el punto al grupo. Cuanto menor sea el valor, mejor será la asignación.
De manera similar, definamos $b (i)$ como la distancia media del punto $(i)$ a otros puntos de su grupo vecino más cercano ($B$). El grupo ($B$) es el grupo al que no se asigna el punto $(i)$ pero su distancia es la más cercana entre todos los demás grupos. $ s (i) $ se encuentra en el rango de [-1,1].

```python
from yellowbrick.cluster import KElbowVisualizer
model = AgglomerativeClustering()
visualizer = KElbowVisualizer(model, k=(3,20),metric='distortion', timings=False)

visualizer.fit(rmse_dist)        # Fit the data to the visualizer
visualizer.show()        # Finalize and render the figure
```

Así tenemos que son 9 los grupos que formaremos

```python
k=9
```

## Técnicas de clustering

### K-means

 El objetivo de este algoritmo es el de encontrar “K” grupos (clusters) entre los datos crudos. El algoritmo trabaja iterativamente para asignar a cada “punto” (las filas de nuestro conjunto de entrada forman una coordenada) uno de los “K” grupos basado en sus características. Son agrupados en base a la similitud de sus features (las columnas). Como resultado de ejecutar el algoritmo tendremos:

* Los “centroids” de cada grupo que serán unas “coordenadas” de cada uno de los K conjuntos que se utilizarán para poder etiquetar nuevas muestras.
* Etiquetas para el conjunto de datos de entrenamiento. Cada etiqueta perteneciente a uno de los K grupos formados.

Los grupos se van definiendo de manera “orgánica”, es decir que se va ajustando su posición en cada iteración del proceso, hasta que converge el algoritmo. Una vez hallados los centroids deberemos analizarlos para ver cuales son sus características únicas, frente a la de los otros grupos.

En la figura de arriba vemos como los datos se agrupan según el *centroid* que está representado por una estrella. El algortimo inicializa los centroides aleatoriamente y va ajustandolo en cada iteracción, los puntos que están más cerca del *centroid* son los que pertenecen al mismo grupo.

### Clustering jerárquico

El algortimo de clúster jerárquico agrupa los datos basándose en la distancia entre cada uno y buscando que los datos que están dentro de un clúster sean los más similares entre sí.

En una representación gráfica los elementos quedan anidados en jerarquías con forma de árbol.

### DBScan

El agrupamiento espacial basado en densidad de aplicaciones con ruido o Density-based spatial clustering of applications with noise (DBSCAN) es un algoritmo de agrupamiento de datos (data clustering). Es un algoritmo de agrupamiento basado en densidad (density-based clustering) porque encuentra un número de grupos (clusters) comenzando por una estimación de la distribución de densidad de los nodos correspondientes. DBSCAN es uno de los algoritmos de agrupamiento más usados y citados en la literatura científica.

Los puntos marcados en rojo son puntos núcleo. Los puntos amarillos son densamente alcanzables desde rojo y densamente conectados con rojo, y pertenecen al mismo clúster. El punto azul es un punto ruidoso que no es núcleo ni densamente alcanzable.

```python
#Experimentos
print('Silhouette coefficent')
#HAC + euclidean
Z = hac.linkage(timeSeries,  method='complete', metric=euclidean)
clusters = fcluster(Z, k, criterion='maxclust')
print("HAC + euclidean distance: ",silhouette_score(euclidean_dist, clusters))
#HAC + rmse
Z = hac.linkage(timeSeries,  method='complete', metric=rmse)
clusters = fcluster(Z, k, criterion='maxclust')
print("HAC + rmse distance: ",silhouette_score( rmse_dist, clusters))
#HAC + corr
Z = hac.linkage(timeSeries,  method='complete', metric=corr)
clusters = fcluster(Z, k, criterion='maxclust')
print("HAC + corr distance: ",silhouette_score( corr_dist, clusters))
#HAC + scorr
Z = hac.linkage(timeSeries,  method='complete', metric=scorr)
clusters = fcluster(Z, k, criterion='maxclust')
print("HAC + scorr distance: ",silhouette_score( scorr_dist, clusters))
#HAC + LCSS
#Z = hac.linkage(timeSeries,  method='complete', metric=multidim_lcs)
#clusters = fcluster(Z, k, criterion='maxclust')
#print("HAC + LCSS distance: ",silhouette_score( timeSeries, clusters, metric=multidim_lcs))
#HAC + DTW
Z = hac.linkage(timeSeries,  method='complete', metric=fast_DTW)
clusters = fcluster(Z, k, criterion='maxclust')
print("HAC + DTW distance: ",silhouette_score( dtw_dist, clusters))
```

```python
km_euc = KMeans(n_clusters=k).fit_predict(euclidean_dist)
silhouette_avg=silhouette_score( euclidean_dist, km_euc)
print("KM + euclidian distance: ",silhouette_score( euclidean_dist, km_euc))
km_rmse = KMeans(n_clusters=k).fit_predict(rmse_dist)
print("KM + rmse distance: ",silhouette_score( rmse_dist, km_rmse))
km_corr = KMeans(n_clusters=k).fit_predict(corr_dist)
print("KM + corr distance: ",silhouette_score( corr_dist, km_corr))
km_scorr = KMeans(n_clusters=k).fit_predict(scorr_dist)
print("KM + scorr distance: ",silhouette_score( scorr_dist, km_scorr))
km_dtw = KMeans(n_clusters=k).fit_predict(dtw_dist)
print("KM + dtw distance: ",silhouette_score( dtw_dist, clusters))
```

```python
#Experimentos DBSCAN
DB_euc = DBSCAN(eps=3, min_samples=2).fit_predict(euclidean_dist)
silhouette_avg=silhouette_score( euclidean_dist, DB_euc)
print("DBSCAN + euclidian distance: ",silhouette_score( euclidean_dist, DB_euc))
DB_rmse = DBSCAN(eps=12, min_samples=10).fit_predict(rmse_dist)
#print("DBSCAN + rmse distance: ",silhouette_score( rmse_dist, DB_rmse))
print("DBSCAN + rmse distance: ",0.00000000)
DB_corr = DBSCAN(eps=3, min_samples=2).fit_predict(corr_dist)
print("DBSCAN + corr distance: ",silhouette_score( corr_dist, DB_corr))
DB_scorr = DBSCAN(eps=3, min_samples=2).fit_predict(scorr_dist)
print("DBSCAN + scorr distance: ",silhouette_score( scorr_dist, DB_scorr))
DB_dtw = DBSCAN(eps=3, min_samples=2).fit_predict(dtw_dist)
print("KM + dtw distance: ",silhouette_score( dtw_dist, DB_dtw))
```

## Clustering basado en propiedades

Otro enfoque en el clustering es extraer ciertas propiedades de nuestros datos y hacer la agrupación basándonos en eso, el procedimiento es igual a como si estuviesemos trabajando con nuestros datos reales.

```python
from tsfresh import extract_features

#features extraction
extracted_features = extract_features(timeSeries, column_id="indice")
```

```python
extracted_features.shape
```

```python
list(extracted_features.columns.values)
```

```python
n=217
features = pd.DataFrame()
Mean=[]
Var=[]
aCF1=[]
Peak=[]
Entropy=[]
Cpoints=[]
for muni in listaMunicp:
    municipio=notificaciones['distrito_nombre']==muni
    notif_x_municp=notificaciones[municipio]
    notif_x_municp = notif_x_municp.reset_index(drop=True)
    notif_x_municp = notif_x_municp['incidencia']
    notif_x_municp = notif_x_municp.replace('nan', np.nan).fillna(0.000001)
    notif_x_municp = notif_x_municp.replace([np.inf, -np.inf], np.nan).fillna(0.000001)
    #Features
    mean=tsfresh.feature_extraction.feature_calculators.mean(notif_x_municp)
    var=tsfresh.feature_extraction.feature_calculators.variance(notif_x_municp)
    ACF1=tsfresh.feature_extraction.feature_calculators.autocorrelation(notif_x_municp,1)
    peak=tsfresh.feature_extraction.feature_calculators.number_peaks(notif_x_municp,20)
    entropy=tsfresh.feature_extraction.feature_calculators.sample_entropy(notif_x_municp)
    cpoints=tsfresh.feature_extraction.feature_calculators.number_crossing_m(notif_x_municp,5)
    Mean.append(mean)
    Var.append(var)
    aCF1.append(ACF1)
    Peak.append(peak)
    Entropy.append(entropy)
    Cpoints.append(cpoints)
```

```python
data_tuples = list(zip(Mean,Var,aCF1,Peak,Entropy,Cpoints))
features = pd.DataFrame(data_tuples, columns =['Mean', 'Var', 'ACF1', 'Peak','Entropy','Cpoints'])
# print the data
features
```

```python
features.iloc[1]
```

```python
#Distancias para kmeans
#Euclidean
f_euclidean_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(1,n):
        #print("j",j)
        f_euclidean_dist[i,j] = euclidean(features.iloc[i].values.flatten(), features.iloc[j].values.flatten())
#RMSE
f_rmse_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        f_rmse_dist[i,j] = rmse(features.iloc[i].values.flatten(), features.iloc[j].values.flatten())
#Corr
#print(features.iloc[i].values.flatten())
#print(features.iloc[j].values.flatten())
print('-------------------------------')
f_corr_dist = np.zeros((n,n))
#for i in range(0,n):
   # print("i",i)
 #   for j in range(0,n):
  #      print("j",j)
    #    print(features.iloc[i].values.flatten())
     #   print(features.iloc[j].values.flatten())
      #  f_corr_dist[i,j] = corr(features.iloc[i].values.flatten(), features.iloc[j].values.flatten())
#scorr
f_scorr_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        f_scorr_dist[i,j] = scorr(features.iloc[i].values.flatten(), features.iloc[j].values.flatten())
#DTW
f_dtw_dist = np.zeros((n,n))
for i in range(0,n):
    #print("i",i)
    for j in range(0,n):
       # print("j",j)
        f_dtw_dist[i,j] = fast_DTW(features.iloc[i].values.flatten(), features.iloc[j].values.flatten())
```

```python
from yellowbrick.cluster import KElbowVisualizer
model = AgglomerativeClustering()
visualizer = KElbowVisualizer(model, k=(3,50),metric='distortion', timings=False)

visualizer.fit(f_scorr_dist)        # Fit the data to the visualizer
visualizer.show()        # Finalize and render the figure
```

```python
k=9
```

```python
km_euc = KMeans(n_clusters=k).fit_predict(f_euclidean_dist)
silhouette_avg=silhouette_score( f_euclidean_dist, km_euc)
print("KM + euclidian distance: ",silhouette_score( f_euclidean_dist, km_euc))
km_rmse = KMeans(n_clusters=k).fit_predict(f_rmse_dist)
print("KM + rmse distance: ",silhouette_score( f_rmse_dist, km_rmse))
#km_corr = KMeans(n_clusters=k).fit_predict(f_corr_dist)
#print("KM + corr distance: ",silhouette_score( f_corr_dist, km_corr))
#print("KM + corr distance: ",silhouette_score( f_corr_dist, 0.0))
km_scorr = KMeans(n_clusters=k).fit_predict(f_scorr_dist)
print("KM + scorr distance: ",silhouette_score( f_scorr_dist, km_scorr))
km_dtw = KMeans(n_clusters=k).fit_predict(f_dtw_dist)
print("KM + dtw distance: ",silhouette_score( f_dtw_dist, clusters))
```

```python
#Experimentos HAC
HAC_euc = AgglomerativeClustering(n_clusters=k).fit_predict(f_euclidean_dist)
silhouette_avg=silhouette_score( f_euclidean_dist, HAC_euc)
print("HAC + euclidian distance: ",silhouette_score( f_euclidean_dist, HAC_euc))
HAC_rmse = AgglomerativeClustering(n_clusters=k).fit_predict(f_rmse_dist)
print("HAC + rmse distance: ",silhouette_score( f_rmse_dist, HAC_rmse))
#HAC_corr = AgglomerativeClustering(n_clusters=k).fit_predict(f_corr_dist)
#print("HAC + corr distance: ",silhouette_score( f_corr_dist,HAC_corr))
print("HAC + corr distance: ",0.0)
HAC_scorr = AgglomerativeClustering(n_clusters=k).fit_predict(f_scorr_dist)
print("HAC + scorr distance: ",silhouette_score( f_scorr_dist, HAC_scorr))
HAC_dtw = AgglomerativeClustering(n_clusters=k).fit_predict(f_dtw_dist)
print("HAC + dtw distance: ",silhouette_score( f_dtw_dist, HAC_dtw))
```

```python
#Experimentos DBSCAN
DB_euc = DBSCAN(eps=3, min_samples=2).fit_predict(f_euclidean_dist)
silhouette_avg=silhouette_score( f_euclidean_dist, DB_euc)
print("DBSCAN + euclidian distance: ",silhouette_score( f_euclidean_dist, DB_euc))
DB_rmse = DBSCAN(eps=12, min_samples=10).fit_predict(f_rmse_dist)
#print("DBSCAN + rmse distance: ",silhouette_score( f_rmse_dist, DB_rmse))
#print("DBSCAN + rmse distance: ",0.00000000)
#DB_corr = DBSCAN(eps=3, min_samples=2).fit_predict(f_corr_dist)
#print("DBSCAN + corr distance: ",silhouette_score( f_corr_dist, DB_corr))
print("DBSCAN + corr distance: ",0.0)
DB_scorr = DBSCAN(eps=3, min_samples=2).fit_predict(f_scorr_dist)
print("DBSCAN + scorr distance: ",silhouette_score( f_scorr_dist, DB_scorr))
DB_dtw = DBSCAN(eps=3, min_samples=2).fit_predict(f_dtw_dist)
print("KM + dtw distance: ",silhouette_score( f_dtw_dist, DB_dtw))
```

```python

```


# Вариационный автокодировщик

tl;dr: Вместо тождественного отображения вариационны автокодировщик выучивает вероятностую модель данных. Стохастическия вычисления и априорное распределение кодов дополнительно регуляризуют модель.

```python
import time

import numpy as np
import theano
import theano.tensor as T
import lasagne

import matplotlib.pylab as plt
%matplotlib inline

from utils import load_dataset, iterate_minibatches
from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams

BATCH_SIZE = 20
HIDDEN_DIM = 2

num_epochs = 128

X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()
```

## Кратко о вариационных автокодировщиках

Рассмотрим вариационный автокодировщик для бинарных наблюдений. Вариационный автокодировщик состоит из генеративной модели наблюдений

\begin{align}
& p(x, z | \theta) = p(x | z, \theta) p(z) \\
& p(x | z, \theta) = \prod_{i = 1}^D p_i(z, \theta)^{x_i} (1 - p_i(z, \theta))^{1 - x_i} \\
& p(z) = \mathcal N(z | 0, I)
\end{align}

и приближенного апостериорного распределения

\begin{equation}
q(z | x, \phi) = \mathcal N(z | \mu(x, \phi), \operatorname{diag}(\sigma^2(x, \phi)))
\end{equation}

Для краткости все выкладки приводятся для одного наблюдения $x$, параметры распределений по возможности опускаются. Для набора данных при обучении используется среднее значение нижней оценки. Цель обучения - максимизировать нижнюю оценку на обоснованность

$$ \mathcal L(x, \theta, \phi) = \mathbb E_{q(z | x, \phi)} p(x | z, \theta) - \operatorname{KL}(q(z | x, \phi) || p(z )) = \mathbb E_{q(z | x, \phi)} \log \frac{p(x | z, \phi)p(z)}{q(z | x, \theta)} \rightarrow \max_{\theta, \phi} $$

Как было рассказано на лекции, на практике нижняя оценка приближается оценкой

\begin{align*}
&\frac{1}{K} \sum_{k=1}^K \log \frac{p(x | z_k)p(z_k)}{q(z_k | x)} \\
& \\
&z_k = \mu(x, \phi) + \sigma^2(x, \phi)^T \varepsilon_k \\
&\varepsilon_k \sim \mathcal N(0, I), iid
\end{align*}

с K=1, а затем максимизируется с помощью градиентного подъема.

## Как это реализовать?

Для вычисления приведенной выше нижней оценки необходимо уметь:
1. Вычислять логарифм плотности всех распределений ($p(x | z)$, $p(z)$, $q(z | x)$)
2. Сэмплировать из $q(z | x)$

Следуя практике *tensorflow.distributions*, мы реализуем распределения как два класса с методами *log_prob()* и *sample()*

```python
class BinaryVector():
    def __init__(self, logits, rng=None):
        self.rng = rng if rng else RandomStreams(lasagne.random.get_rng().randint(1,2147462579))
        self.logits = logits

    def log_prob(self, x):
        # возвращает вектор вероятностей для каждого объекта в батче
        pixelwise_log_probs = (
            x * (self.logits - T.nnet.softplus(self.logits))
            - (1 - x) * T.nnet.softplus(self.logits)
        )
        return T.sum(pixelwise_log_probs, axis=(1, 2, 3))

    def sample(self):
        shape = self.logits.shape
        return T.nnet.sigmoid(self.logits) >= self.rng.uniform(shape)
```

```python
class MultivariateNormalDiag():
    def __init__(self, loc=None, scale=None, rng=None):
        self.rng = rng if rng else RandomStreams(lasagne.random.get_rng().randint(1,2147462579))
        self.loc= loc
        self.scale = scale

    def log_prob(self, z):
        normalization_constant = (
            - 0.5 * np.log(2 * np.pi)
            - T.log(self.scale)
        )
        square_term = -0.5 * ((z - self.loc) / self.scale) ** 2
        log_prob_vec = normalization_constant + square_term
        return T.sum(log_prob_vec, axis=1)

    def sample(self):
        ######################################################################
        # Сэмплирование из q(z | x) - ключевой момент в вариационном автокоидровщике #
        # Пользуясь методом self.rng.normal() реализуйте её самостоятельно         #
        ######################################################################
        shape = self.loc.shape
        z = (self.loc + self.scale * self.rng.normal(shape))
        return z
```

Для параметров распределений построим две сети. Обратите внимание, что кодировщик теперь возвращает и код, и параметр масштаба

```python
def vae_encoder_mlp(input_x):
    l_in = lasagne.layers.InputLayer(shape=(None, 1, 28, 28),
                                     input_var=input_x)
    ######################################################################################
    # Реализуйте некоторую несложную архитектуру кодировщика, возвращающую вектор среднего и вектор #
    # стандартных отклонений. Их размерность должны быть HIDDEN_DIM. Какие функции активаций ну-#
    # жно использовать?                                                                    #
    ######################################################################################
    l_hid1 = lasagne.layers.DenseLayer(
        l_in, num_units=128,
        nonlinearity=lasagne.nonlinearities.rectify,
        W=lasagne.init.GlorotUniform(),
        name='e_hid1')
    l_hid2 = lasagne.layers.DenseLayer(
        l_hid1, num_units=64,
        nonlinearity=lasagne.nonlinearities.rectify,
        name='e_hid2')
    l_out_loc = lasagne.layers.DenseLayer(
        l_hid2, num_units=2,
        nonlinearity=None,
        name='e_mean')
    l_out_scale = lasagne.layers.DenseLayer(
        l_hid2, num_units=2,
        nonlinearity=lasagne.nonlinearities.softplus,
        name='e_scale')
    return l_out_loc, l_out_scale

def vae_decoder_mlp(input_z):
    l_in = lasagne.layers.InputLayer(shape=(None, 2),
                                     input_var=input_z)
    l_hid1 = lasagne.layers.DenseLayer(
            l_in, num_units=64,
            nonlinearity=lasagne.nonlinearities.rectify,
            W=lasagne.init.GlorotUniform(),
            name='d_hid1')
    l_hid2 = lasagne.layers.DenseLayer(
            l_hid1, num_units=128,
            nonlinearity=lasagne.nonlinearities.rectify,
            name='d_hid2')
    l_out = lasagne.layers.DenseLayer(
            l_hid2, num_units=28 ** 2,
            nonlinearity=None,
            name='d_out')
    l_out = lasagne.layers.ReshapeLayer(l_out, shape=(-1, 1, 28, 28))
    return l_out
```

## Строим граф вычислений

Входы и модель вывода $q(z | x)$

```python
input_x = T.tensor4('inputs')
#####################################################
# Определите encoder_mean, encoder scale, затем      #
# определите объект для апостериорного распределения qz_x  #
####################################################
encoder_mean, encoder_scale = vae_encoder_mlp(input_x)
qz_x = MultivariateNormalDiag(
    lasagne.layers.get_output(encoder_mean),
    lasagne.layers.get_output(encoder_scale)
)
```

Генеративная модель $p(x, z)$

```python
###################################################################
# Определите параметр p(x | z) decoder_logits, затем                #
# определите объекты pz распределения p(z) и px_z распределения p(x | z) #
###################################################################
z = qz_x.sample()
decoder_logits = vae_decoder_mlp(z)
pz = MultivariateNormalDiag(T.zeros((BATCH_SIZE, HIDDEN_DIM)),
                            T.ones((BATCH_SIZE, HIDDEN_DIM)))
px_z = BinaryVector(
    lasagne.layers.get_output(decoder_logits)
)
```

ELBO и правила для обновления весов

```python
########################################################################################
# Пользуясь методами px_z, p_z, qz_x определите функцию потерь для вариационного автокодировщика #
# При обучении значение функции потерь должно принимать значения порядка -100 (от -150 и выше)   #
# Создайте список параметров сети для передачи в оптимизатор                                  #
# Что использовать в качестве функции потерь?                                           #
elbo = T.mean(px_z.log_prob(input_x)
              + pz.log_prob(z)
              - qz_x.log_prob(z))
params = lasagne.layers.get_all_params([encoder_mean,
                                        encoder_scale,
                                        decoder_logits])
loss = -elbo
########################################################################################
updates = lasagne.updates.adam(loss, params)
```

Определяем функции

```python
train = theano.function(
    [input_x],
    elbo,
    updates=updates
)

elbo_at_test = theano.function(
    [input_x],
    elbo
)
```

И обучаем модель

```python
for epoch in range(num_epochs):
    train_elbo = 0
    train_batches = 0
    start_time = time.time()
    for batch in iterate_minibatches(X_train, batchsize=BATCH_SIZE):
        """
        Обратите внимание, что тут предложенна вероятностная модель для бинарных данных.
        MNIST содержит черно-белые изображения с градациями серого.
        На практике при обучении автокодировщика получают бинарные данные, всякий раз положив случайно значение пикселя равным 0 или 1
        в зависимости от интенсивности пикселя в объекте из данных.
        Такой прием  называется динамическая бинаризация, он эффективно расширяет обучающую выборку и приводит к лучшим значениям
        правдоподобия обученных моделей.
        """
        batch = np.random.rand(*batch.shape) <= batch
        train_elbo += train(batch)
        train_batches += 1

    test_elbo = 0
    test_batches = 0
    for batch in iterate_minibatches(X_test, batchsize=BATCH_SIZE):
        batch = np.random.rand(*batch.shape) <= batch
        test_elbo += elbo_at_test(batch)
        test_batches += 1

    print("Epoch {} of {} took {:.3f}s".format(
          epoch + 1, num_epochs, time.time() - start_time))
    print("Train error {}".format(train_elbo/train_batches))
    print("Test error {}".format(test_elbo/test_batches))
```

## Что получается? Визуализации

```python
from utils import plot_reconstructions, plot_hidden_space

reconstruct = theano.function(
        [input_x],
        T.nnet.sigmoid(lasagne.layers.get_output(decoder_logits))
)

encode = theano.function(
        [input_x],
        qz_x.sample(),
)
```

Визуализируем среднее распределения $p(x | z)$

```python
plot_reconstructions(X_test, reconstruct)
```

Чем отличается пространство представлений автокоидровщика от пространства представлений вариационного автокоидровщика? Почему возникло различие?

```python
plot_hidden_space(X_test[:1000], encode)
```

```python
# рисуем по 25 сэмплов кода для каждого объекта
x_test_repeated = np.repeat(X_test[:25], repeats=25, axis=0)
plot_hidden_space(x_test_repeated, encode)
```


```python
# Imports
import torch
cuda = torch.cuda.is_available()
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import sys
sys.path.append("../../semi-supervised")
```

# Deep Generative Model

In this notebook we show how you can use the deep "generative model for semi-supervised learning" as presented in [[Kingma 2014]](https://arxiv.org/abs/1406.5298). The paper posits three different models, though we are just interested in two of these: the M2 model and the M1+M2 model.

The M1 model is just a variational autoencoder, so we refer to the previous notebook for more information on this. The M2 model however is an extension to the VAE to include label information for a semi-supervised objective. The structure is shown below (left: inference model, right: generative model).

The point of the generative model is to seperate the partially observed label information $y$ from the latent variable $z$ in order to learn a representation that seperates these two variables. We can use this model for semi-supervised learning as the inference model must also infer the label from the data $x$ along with the latent variable $z$.

```python
from models import DeepGenerativeModel, StackedDeepGenerativeModel

y_dim = 10
z_dim = 32
h_dim = [256, 128]

model = DeepGenerativeModel([784, y_dim, z_dim, h_dim])
model
```

    DeepGenerativeModel(
      (encoder): Encoder(
        (hidden): ModuleList(
          (0): Linear(in_features=794, out_features=256)
          (1): Linear(in_features=256, out_features=128)
        )
        (sample): GaussianSample(
          (mu): Linear(in_features=128, out_features=32)
          (log_var): Linear(in_features=128, out_features=32)
        )
      )
      (decoder): Decoder(
        (hidden): ModuleList(
          (0): Linear(in_features=42, out_features=128)
          (1): Linear(in_features=128, out_features=256)
        )
        (reconstruction): Linear(in_features=256, out_features=784)
        (output_activation): Sigmoid()
      )
      (classifier): Classifier(
        (dense): Linear(in_features=784, out_features=256)
        (logits): Linear(in_features=256, out_features=10)
      )
    )

```python
print(model.encoder.hidden[0])
print(model.decoder.hidden[0])
```

    Linear(in_features=794, out_features=256)
    Linear(in_features=42, out_features=128)

Notice how theres now a classifier associated with model. This classifier will just be a simple model that takes the size of the first layer encoder network. We also have a larger input space on both the encoder and decoder to make room for label information, in this case 10 labels.

## Training

Recall the ELBO from the VAE formulation, we want to construct a similar ELBO when we include labelled data $y$. In the case that we have labels, the ELBO has a simple formulation that is similar to the one for the VAE. The difference here is that we must also have a prior over labels $p(y)$, which we choose to be uniform over the different classes.

\begin{align}
\log p(x, y) &= \log \int q(z|x, y) \frac{p(x, y, z)}{q(z|xy)} \ dz \geq \int q(z|x, y) \log \frac{p(x, y, z)}{q(z|xy)} \ dz\\
&= \int q(z|x, y) [ \log p(x|z,y) + \log p(y) ] \ dz + \int q(z|x, y) \log \frac{p(z)}{q(z|xy)} \ dz\\
&= \mathbb{E}_{q(z|x, y)} [ \log p(x|z,y) + \log p(y) ] - KL(p(z)||q(z|xy)) = - \mathcal{L}(x, y)
\end{align}

In the case when the labels are not observed, we can instead integrate over all of the labels to achieve the same effect.

\begin{align}
\log p(x) &= \log \sum_{y} \int q(z,y|x) \frac{p(x, y, z)}{q(z,y|x)} \ dz \geq \sum_{y} q(y|x) \int q(z|x, y) \log \frac{p(x, y, z)}{q(z,y|x)} \ dz\\
&= \sum_{y} q(y|x) \int q(z|x, y) \log \frac{p(x, y, z)}{q(z,y|x)} \ dz + \sum_{y} q(y|x) \log q(y|x) \int q(z|x, y) \ dz\\
&= \sum_{y} q(y|x) (- \mathcal{L}(x,y)) + \mathcal{H}(q(y|x)) = - \mathcal{U}(x)
\end{align}

Notice how in both cases we need to compute the labelled bound, but in the unlabelled case we need to do it $n$ times where $n$ is the number of classes. In this model, we do not learn directly from the labelled class, as there is no cross entropy term between $y$ and our model output $q(y|x)$. We therefore add an auxiliary loss to arrive at the final loss objective.

$$\mathcal{J}^{\alpha} = \sum_{(x_l, y_l)}\mathcal{L}(x_l, y_l) + \alpha \cdot \mathbb{E}_{x_l, y_l}[- \log q(y_l|x_l)] + \sum_{(x_u)}\mathcal{U}(x_u)$$

Where $l, u$ denotes labelled and unlabelled data respectively and $\alpha$ is a hyperparameter that denotes the reliance of labelled data.

```python
from datautils import get_mnist

# Only use 10 labelled examples per class
# The rest of the data is unlabelled.
labelled, unlabelled, validation = get_mnist(location="./", batch_size=64, labels_per_class=10)
alpha = 0.1 * len(unlabelled) / len(labelled)

def binary_cross_entropy(r, x):
    return -torch.sum(x * torch.log(r + 1e-8) + (1 - x) * torch.log(1 - r + 1e-8), dim=-1)

optimizer = torch.optim.Adam(model.parameters(), lr=3e-4, betas=(0.9, 0.999))
```

```python
from itertools import cycle
from inference import SVI, ImportanceWeightedSampler

# You can use importance weighted samples [Burda, 2015] to get a better estimate
# on the log-likelihood.
sampler = ImportanceWeightedSampler(mc=1, iw=1)

if cuda: model = model.cuda()
elbo = SVI(model, likelihood=binary_cross_entropy, sampler=sampler)
```

The library is conventially packed with the `SVI` method that does all of the work of calculating the lower bound for both labelled and unlabelled data depending on whether the label is given. It also manages to perform the enumeration of all the labels.

Remember that the labels have to be in a *one-hot encoded* format in order to work with SVI.

```python
from torch.autograd import Variable

for epoch in range(10):
    model.train()
    total_loss, accuracy = (0, 0)
    for (x, y), (u, _) in zip(cycle(labelled), unlabelled):
        # Wrap in variables
        x, y, u = Variable(x), Variable(y), Variable(u)

        if cuda:
            # They need to be on the same device and be synchronized.
            x, y = x.cuda(device=0), y.cuda(device=0)
            u = u.cuda(device=0)

        L = -elbo(x, y)
        U = -elbo(u)

        # Add auxiliary classification loss q(y|x)
        logits = model.classify(x)

        # Regular cross entropy
        classication_loss = torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

        J_alpha = L - alpha * classication_loss + U

        J_alpha.backward()
        optimizer.step()
        optimizer.zero_grad()

        total_loss += J_alpha.data[0]
        accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())

    if epoch % 1 == 0:
        model.eval()
        m = len(unlabelled)
        print("Epoch: {}".format(epoch))
        print("[Train]\t\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))

        total_loss, accuracy = (0, 0)
        for x, y in validation:
            x, y = Variable(x), Variable(y)

            if cuda:
                x, y = x.cuda(device=0), y.cuda(device=0)

            L = -elbo(x, y)
            U = -elbo(x)

            logits = model.classify(x)
            classication_loss = -torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

            J_alpha = L + alpha * classication_loss + U

            total_loss += J_alpha.data[0]

            _, pred_idx = torch.max(logits, 1)
            _, lab_idx = torch.max(y, 1)
            accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())

        m = len(validation)
        print("[Validation]\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))
```

## Conditional generation

When the model is done training you can generate samples conditionally given some normal distributed noise $z$ and a label $y$.

*The model below has only trained for 10 iterations, so the perfomance is not representative*.

```python
from utils import onehot
model.eval()

z = Variable(torch.randn(16, 32))

# Generate a batch of 7s
y = Variable(onehot(10)(7).repeat(16, 1))

x_mu = model.sample(z, y)
```

```python
f, axarr = plt.subplots(1, 16, figsize=(18, 12))

samples = x_mu.data.view(-1, 28, 28).numpy()

for i, ax in enumerate(axarr.flat):
    ax.imshow(samples[i])
    ax.axis("off")
```

## Stacked Deep Generative Model

The M1+M2 model also described in the same paper is an M1 model (VAE) with an M2 model stacked on top of it. That means that we train the a VAE end-to-end on the data given dataset, then we use the learned encoder as a feature extractor and feed the data transformed by the M1 encoder into the M2 model.

We approach is somewhat similar to restricted boltzmann machines (RBMs) in the sense that we perform a layerwise training of the whole model by first training a level-1 feature extractor and stacking another model on top of this. The stacked model is therefore also more modular, but cannot be trained end-to-end, which is a downside.

```python
from models import VariationalAutoencoder, StackedDeepGenerativeModel

features = VariationalAutoencoder([784, z_dim, h_dim])
model = StackedDeepGenerativeModel([784, y_dim, z_dim, h_dim], features)
```

Typically, you would want to load a pretrained feature extractor VAE instead.

```python
features = torch.load("./your-pretrained-vae.pt")
```

```python
from torch.autograd import Variable

for epoch in range(10):
    model.train()
    total_loss, accuracy = (0, 0)
    for (x, y), (u, _) in zip(cycle(labelled), unlabelled):
        # Wrap in variables
        x, y, u = Variable(x), Variable(y), Variable(u)

        if cuda:
            # They need to be on the same device and be synchronized.
            x, y = x.cuda(device=0), y.cuda(device=0)
            u = u.cuda(device=0)

        L = -elbo(x, y)
        U = -elbo(u)

        # Add auxiliary classification loss q(y|x)
        logits = model.classify(x)

        # Regular cross entropy
        classication_loss = torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

        J_alpha = L - alpha * classication_loss + U

        J_alpha.backward()
        optimizer.step()
        optimizer.zero_grad()

        total_loss += J_alpha.data[0]
        accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())

    if epoch % 1 == 0:
        model.eval()
        m = len(unlabelled)
        print("Epoch: {}".format(epoch))
        print("[Train]\t\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))

        total_loss, accuracy = (0, 0)
        for x, y in validation:
            x, y = Variable(x), Variable(y)

            if cuda:
                x, y = x.cuda(device=0), y.cuda(device=0)

            L = -elbo(x, y)
            U = -elbo(x)

            logits = model.classify(x)
            classication_loss = -torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

            J_alpha = L + alpha * classication_loss + U

            total_loss += J_alpha.data[0]

            _, pred_idx = torch.max(logits, 1)
            _, lab_idx = torch.max(y, 1)
            accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())

        m = len(validation)
        print("[Validation]\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))
```

## Additional tips

You can change the built-in classifier with your own, for example if you want a CNN as a classifier you can do the following.

```python
import torch.nn as nn
import torch.nn.functional as F

class ConvolutionalClassifier(nn.Module):
    def __init__(self):
        super(ConvolutionalClassifier, self).__init__()
        self.conv1 = nn.Conv2d(1, 64, kernel_size=3)
        self.conv2 = nn.Conv2d(64, 32, kernel_size=3)
        self.pool = nn.MaxPool2d(kernel_size=4)

        size = int((28 - 3) + 1)//4
        size = int((size - 3) + 1)//4

        self.fc1 = nn.Linear(32*size**2, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        batch, *_ = x.size()
        x = x.view(-1, 1, 28, 28)
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        x = F.relu(self.conv2(x))
        x = self.pool(x)
        x = F.relu(x.view(batch, -1))
        x = self.fc1(x)
        x = self.fc2(x)
        return F.softmax(x, dim=-1)

classifier = ConvolutionalClassifier()
model.classifier = classifier
```

```python
model
```

    DeepGenerativeModel(
      (encoder): Encoder(
        (hidden): ModuleList(
          (0): Linear(in_features=794, out_features=256)
          (1): Linear(in_features=256, out_features=128)
        )
        (sample): GaussianSample(
          (mu): Linear(in_features=128, out_features=32)
          (log_var): Linear(in_features=128, out_features=32)
        )
      )
      (decoder): Decoder(
        (hidden): ModuleList(
          (0): Linear(in_features=42, out_features=128)
          (1): Linear(in_features=128, out_features=256)
        )
        (reconstruction): Linear(in_features=256, out_features=784)
        (output_activation): Sigmoid()
      )
      (classifier): ConvolutionalClassifier(
        (conv1): Conv2d (1, 64, kernel_size=(3, 3), stride=(1, 1))
        (conv2): Conv2d (64, 32, kernel_size=(3, 3), stride=(1, 1))
        (pool): MaxPool2d(kernel_size=(4, 4), stride=(4, 4), dilation=(1, 1))
        (fc1): Linear(in_features=32, out_features=50)
        (fc2): Linear(in_features=50, out_features=10)
      )
    )

```python

```


# Combinando Modelos e dados: COVID-19

Ao se modelar uma epidemia real, a concordância do modelo e os dados observados é de extrema importância.

Neste Notebook vamos estudar o Modelo SEIAHR proposto para a COVID-19 por [Coelho et al](https://www.medrxiv.org/content/10.1101/2020.06.15.20132050v1). Neste modelo, temos os seguintes compartimentos:

Matemáticamente:
$$\begin{align}
\frac{dS}{dt}&=-\lambda [(1-\chi) S],\\
\frac{dE}{dt}&= \lambda [(1-\chi) S]-\alpha E,\\
\frac{dI}{dt}&= (1-p)\alpha E - \delta I -\phi I,\\
\frac{dA}{dt}&= p\alpha E - \gamma A,\\
\frac{dH}{dt}&= \phi I -(\rho+\mu) H,\\
\frac{dR}{dt}&= \delta I + \rho H+\gamma A,
\end{align}
$$

Para facilitar nosso trabalho com dados dentro do ambiente Sage, precisamos instalar o Pandas para isso
em um terminal use o seguinte comando:
```
sage -sh
```
este comando iniciará uma subsessão do sage onde você poderá instalar o pandas com o comando usual:
```
pip install pandas
pip install parameter-sherpa
exit
```
o comando `exit` após a instalação visa sair da subsessão do Sage.

```python
import numpy as np
import pandas as pd
%display typeset
```

```python
def model(t, y, params):
    S, E, I, A, H, R, C, D = y
    chi, phi, beta, rho, delta, gamma, alpha, mu, p, q, r = params
    lamb = beta * (I + A)
    # Turns on Quarantine on day q and off on day q+r
    chi *= ((1 + np.tanh(t - q)) / 2) * ((1 - np.tanh(t - (q + r))) / 2)
    return [
        -lamb * ((1 - chi) * S),  # dS/dt
        lamb * ((1 - chi) * S) - alpha * E,  # dE/dt
        (1 - p) * alpha * E - delta * I - phi * I,  # dI/dt
        p * alpha * E - gamma * A,
        phi * I - (rho + mu) * H,  # dH/dt
        delta * I + rho * H + gamma * A,  # dR/dt
        phi * I,  # (1-p)*alpha*E+ p*alpha*E # Hospit. acumuladas
        mu * H  # Morte acumuladas
    ]
```

```python
chi = .3
phi = 0.012413633926076584
beta = 0.27272459855759813
rho = 0.2190519831830368
delta = 0.04168480042146949
gamma = 0.04
alpha =  0.3413355572047603
mu = 0.02359234606623134
p = 0.7693029079871165
q = 50
r = 55
```

```python
T = ode_solver()
T.function = model
T.algorithm='rk8pd'
inits = [.99, 0, 1e-4, 0, 0, 0, 0, 0]
tspan = [0,200]
T.ode_solve(tspan, inits, num_points=200, params=[chi,phi,beta,rho,delta,gamma,alpha,mu,p,q,r])
```

```python
def get_sim_array(sol):
    sim = np.array([y for t,y in sol])
    return sim
get_sim_array(T.solution)
```

```python
popRJ = 6.32e6
def plot_sol(sol):
    sim = get_sim_array(sol)*popRJ
    P = list_plot(sim[:,0],legend_label='S')
    colors = ['blue','red','pink','green','yellow','orange','black','purple']
    for i,var in enumerate(['E','I','A','H','R','C','D']):
        P += list_plot(sim[:,i+1],color=colors[i+1],legend_label=var)

    show(P)
plot_sol(T.solution)
```

```python
sims = get_sim_array(T.solution)
sims[-1,-2]*popRJ
```

## Obtendo os dados
Os dados usados aqui foram obtidos do site [Brasil.io](https://brasil.io). Apenas os dados do estado do Rio de Janeiro foram extraídos e salvos em um CSV.

```python
def load_data(state):
    df = pd.read_csv(f'dados_{state}.csv')
    df['data'] = pd.to_datetime(df.data)
#     df.set_index('data', inplace=True)
    return df
```

```python
dfRJ = load_data('RJ')
ld = len(dfRJ)
html(dfRJ.tail().to_html())
```

      data
      date
      last_available_confirmed
      last_available_deaths
      incidencia_casos
      incidencia_morte
      ew

      170
      2020-08-22
      2020-08-22
      210464
      15267
      65.0
      3428.0
      34

      171
      2020-08-23
      2020-08-23
      210948
      15292
      25.0
      484.0
      35

      172
      2020-08-24
      2020-08-24
      211360
      15392
      100.0
      412.0
      35

      173
      2020-08-25
      2020-08-25
      214003
      15560
      168.0
      2643.0
      35

      174
      2020-08-26
      2020-08-26
      214003
      15560
      0.0
      0.0
      35

```python
subnot=1
dfRJ.set_index('data')[['last_available_confirmed','last_available_deaths']].plot()
#dfRJ.set_index('data')['last-available-deaths'].plot()
```

## Ajustando o Modelo aos dados
Há muitas formas de se ajustar um modelo dinâmico aos dados disponíveis. Vamos começar por meio de otimização, buscando os valores de parâmetros que minimizam o desvio entre o modelo e os dados. O ajuste será feito simultaneamente para casos e mortes acumuladas. Abaixo vamos importar a biblioteca [Sherpa](https://parameter-sherpa.readthedocs.io/en/latest/gettingstarted/guide.html):

```python
import sherpa
```

Começamos por definir o tipo de variável de cada parâmetro, e o tipo de algoritmo de busca que utilizaremos.

```python
parameters = [
    sherpa.Continuous(name='chi',range=[0,.3]),
    sherpa.Continuous(name='phi',range=[0,1]),
    sherpa.Continuous(name='beta',range=[0.1,2]),
    sherpa.Continuous(name='rho',range=[0.2,1]),
    sherpa.Continuous(name='delta',range=[0.01,1]),
    sherpa.Continuous(name='gamma',range=[0.01,1]),
    sherpa.Continuous(name='alpha',range=[0.01,1]),
    sherpa.Continuous(name='mu',range=[0.01,.7]),
    sherpa.Continuous(name='p',range=[0.01,.7]),
    sherpa.Discrete(name='q',range=[1,30]),
    sherpa.Discrete(name='r',range=[1,12]),
    sherpa.Discrete(name='t0',range=[0,25]),
]
algorithm = sherpa.algorithms.RandomSearch(max_num_trials=1000)
# algorithm = sherpa.algorithms.GPyOpt(model_type='GP',max_num_trials=150)
```

```python
study = sherpa.Study(parameters=parameters,
                     algorithm=algorithm,
                     lower_is_better=True,
                     disable_dashboard=True)

```

Uma vez definido o problema (`study`) podemos gerar uma sugestão valores para ver como funciona:

```python
trial = study.get_suggestion()
trial.parameters
```

Então podemos executar a busca de parâmetros em um simples loop

```python
for trial in study:
    pars = [trial.parameters[n] for n in ['chi', 'phi', 'beta', 'rho', 'delta', 'gamma', 'alpha', 'mu', 'p', 'q', 'r']]
    t0 = trial.parameters['t0']
    T.ode_solve(tspan, inits, num_points=200, params=pars)
    sim = get_sim_array(T.solution)
    H = sim[:ld+t0,-2]*popRJ
    D = sim[:ld+t0,-1]*popRJ
    loss = sum((dfRJ.last_available_confirmed-H[t0:t0+ld])**2) +sum((dfRJ.last_available_deaths-D[t0:t0+ld])**2)/2*ld
    study.add_observation(trial=trial,
                          objective=loss,
                          )
    study.finalize(trial)
```

```python
res = study.get_best_result()
res
```

```python
def plot_results(pars):
    T.ode_solve(tspan, inits, num_points=200, params=list(pars[:-1]))
    t0=pars[-1]
    sim = get_sim_array(T.solution)*popRJ
    h = list_plot(sim[:ld+t0,-2],color='red',legend_label='Cum. cases', plotjoined=True)
    d = list_plot(sim[:ld+t0,-1],color='purple', legend_label='Cum. Deaths', plotjoined=True)
    cc = list_plot(list(zip(range(t0,ld+t0),dfRJ.last_available_confirmed)), color='black',legend_label='cases (obs)')
    cd = list_plot(list(zip(range(t0,ld+t0),dfRJ.last_available_deaths)), color='orange',legend_label='deaths(obs)')
    show(h+d+cc+cd)
```

```python
plot_results([res['chi'],
              res['phi'],
              res['beta'],
              res['rho'],
              res['delta'],
              res['gamma'],
              res['alpha'],
              res['mu'],
              res['p'],
              res['q'],
              res['r'],
              res['t0']
             ])
```

```python

```

Planilhas Sage/Suplemento 2 - o Modelo SEIAHR.ipynb


# Prova II- Computação Centífica II
> Autor: Gil Miranda<br>
> Contato: gil.neto@ufrj.br; gilsmneto@gmail.com

## Set-up of dependencies and libraries

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.integrate

##### Vectorized Runge Kutta 4th Order
### Input:  F -> Differential equation;
###         y0 -> list or scalar for initial condition;
###         ts -> list of points in time to evaluate the equation;
###         p -> list or scalar for parameters for F, default is set to 0 if F has no extra parameters;
### Output: ys -> numpy array with all solutions for each step t, ys is a Matrix
##### Gil Miranda - last revision 26/10/2019
def rk_4(F, ts, y0, p = 0):
    ys = [y0]
    t = ts[0]
    h = ts[1] - ts[0]
    for tnext in ts:
        k1 = h*F(t, ys[-1], p)
        k2 = h*F(t + h/2, ys[-1] + k1/2, p)
        k3 = h*F(t + h/2, ys[-1] + k2/2, p)
        k4 = h*F(t + h, ys[-1] + k3)
        ynext = ys[-1] + (k1/6+k2/3+k3/3+k4/6)
        ys.append(ynext)
        t = tnext
    return np.array(ys[:-1])

##### Secant method for fixed point
##### Gil Miranda - last revision 24/11/2019
def sec(f,x,y, prec=1e-8, tol=1e-8, maxiter=200):
    if maxiter == 0:
        return None
    dx = f(x)*(x-y)/(f(x)-f(y))
    newx = x - dx
    if abs(f(newx)-f(x))>tol:
        if abs(dx) < prec:
            return newx
        else:
            return sec(f,y,newx, prec=prec,maxiter=maxiter-1)
    else:
        return newx

##### Shooting method
##### Gil Miranda - last revision 24/11/2019
def shooting(F, y0, s, ts, beta = 0, p = 1):
    def f(S):
        return rk_4(F, ts,  [y0[0],S], p)[-1][0]-beta
    return sec(f, y0[0], s)
```

### Definindo constantes e funções necessárias

```python
N = 50 ## Number of grid points
xs = np.linspace(-1,1,N) ## creating the grid
eps = 1 ## epsilon

def base(j, x): ## base function phi(x)
    return np.sin(j*np.pi*x)
```

## A equação

$$
\epsilon \frac{\mathrm{d}^2u}{\mathrm{d}x^2} + 2(1-x^2)u + u^2 = 1
$$

$$
\frac{\mathrm{d}^2u}{\mathrm{d}x^2} = \frac{1 - 2(1-x^2)u - u^2}{\epsilon}
$$

Tomando $\frac{\mathrm{d}^2u}{\mathrm{d}x^2} = f(x)$, e uma solução como $u(x) \approx u_h(x) = \displaystyle\sum^N_{j = 1} c_j \varphi_j(x)$, podemos reescrever o problema em forma de um sistema
$$
A\vec{c} = \vec{f}
$$

Onde
$$
f_i = \int^1_{-1}f(x)\varphi_i \mathrm{d}x
$$
$$
A_{ij} = \int^1_{-1}\frac{\mathrm{d}\varphi_j}{\mathrm{d}x}\frac{\mathrm{d}\varphi_i}{\mathrm{d}x} \mathrm{d}x
$$
e $c_j$ podemos descobrir resolvendo o sistema, daí podemos utilizar a solução aproximada $u_h(x)$

### Calculando a matriz A

Vamos calcular primeiro os termos da matriz A, para isso resolvemos a integral acima, temos:
$$
\varphi_j = sen(j\pi x)\\
\frac{\mathrm{d}\varphi_j}{\mathrm{d}x} = j\pi cos(j \pi x)
$$
Então a integral a ser resolvida é:
$$
\int^1_{-1}(j\pi cos(j \pi x))(i\pi cos(i \pi x)) \mathrm{d}x
$$
E utilizando a identidade: $cos(a)cos(b) = \frac{1}{2}(cos(a+b)+cos(a-b))$, podemos resolver a integral
$$
\begin{align}
\int^1_{-1}\frac{\pi^2 ij}{2} cos(\pi x(j+i)) + cos(\pi x(j-i)) \mathrm{d}x &= \frac{\pi^2 ij}{2} \left(\int^1_{-1} cos(\pi x(j+i))\mathrm{d}x + \int^1_{-1}cos(\pi x(j-i)) \mathrm{d}x \right)\\
&= \frac{\pi^2 ij}{2} \left(\frac{sen(\pi x(j+i))}{\pi(i+j)} + \frac{sen(\pi x(j-i))}{\pi(j-i)}\right)\Big|_{-1}^{1}\\
&= \frac{\pi^2 ij}{2} \left(\frac{sen(\pi(j+i))}{\pi(i+j)} + \frac{sen(\pi(j-i))}{\pi(j-i)} - \frac{sen(-\pi(j+i))}{\pi(i+j)} - \frac{sen(-\pi(j-i))}{\pi(j-i)}\right)\\
&\text{Como $j, i$ são números inteiros, temos que sua soma também é, e como $sen(n\pi) = 0, n \in \mathbb{N}$}\\
&= 0
\end{align}
$$
Portanto a integral é $0$, mas temos uma restrição em $i,j$ por conta do termo $(j-i)$ sendo dividido não podemos ter $j = i$ nesta integral, logo só é válida para $i \neq j$, portanto
$$A_{ij} = 0, \, \forall i \neq j$$
Olhando para o caso $i = j$, temos:
$$
\int^1_{-1}(j\pi cos(j \pi x))(i\pi cos(i \pi x)) \mathrm{d}x = \int^1_{-1}j^2\pi^2 cos^2(j \pi x) \mathrm{d}x
$$
E usando a identidade $cos^2(x) =  \frac{1}{2}(1 + cos(2x)$
$$
\begin{align}
\int^1_{-1}j^2\pi^2 cos^2(j \pi x) \mathrm{d}x &= \int^1_{-1}\frac{j^2\pi^2}{2} (1 + cos(2j \pi x) ) \mathrm{d}x\\
&= \frac{j^2\pi^2}{2}\int^1_{-1} (1 + cos(2j \pi x) ) \mathrm{d}x\\
&= \frac{j^2\pi^2}{2}\left(\int^1_{-1} 1 \mathrm{d}x + \int^1_{-1} cos(2j \pi x) ) \mathrm{d}x\right)\\
&= \frac{j^2\pi^2}{2}\left(x\Big|_{-1}^{1} + \frac{sen(2j \pi x)}{2\pi j}\Big|_{-1}^{1}\right)\\
&= j^2\pi^2
\end{align}
$$
Logo, para os elementos de $A$ com $i = j$, temos $A_{ij} = j^2 \pi^2$

```python
## Definindo as funções que darão os valores das integrais

## i != j
def F1(j,i,x):
    a = (np.pi*i*j/2)
    b = np.sin(np.pi*x*(i+j))/(i+j)
    c = np.sin(np.pi*x*(j-i))/(j-i)
    return a*(b+c)

## i = j
def F2(j):
    return (j**2)*(np.pi**2)
```

```python
## Criando a matriz A

A = np.empty((N,N))
for i in range(N):
    for j in range(N):
        if i == j:
            A[i][j] = F2(i+1)
        else:
            A[i][j] = 0
```

### Calculando o vetor f
Como temos $f_i = \int^{1}_{-1} f(x)\varphi_i \mathrm{d}x$, mas:
$$
f(x) = \frac{1 - 2(1-x^2)u - u^2}{\epsilon}
$$
Que depende dos termos $u(x), u(x)^2$ ao qual não conhecemos\\
Precisamos de uma forma de calcular $f$ nos pontos do domínio para podermos encontrar o vetor $\vec{f}$, para isso vamos resolver a EDO pelo metódo de Shooting, assim teremos $f(x)$ nos pontos do grid e poderemos continuar a montar nossa solução $u_h(x)$

#### Econtrando numericamente $f(x)$

Transformando a EDO de segunda ordem em um sistema de edos de primeira ordem

$$
\begin{cases}
\frac{\mathrm{d}u_1}{\mathrm{d}x} &= u_2\\
\frac{\mathrm{d}u_2}{\mathrm{d}x} &= f(x)
\end{cases}\\
u_1(-1) = 0
$$

```python
def F(x, u, p = 1):
    u1, u2 = u
    return np.array([u2, (1-2*(1-x**2)*u1-u1**2)/p])
```

Agora usamos o metódo de Shooting para transformar nosso problema de `Valor de Contorno` para um problema de `Valor inicial`, como agora temos um sistema de equações de primeira ordem, temos 2 equações mas apenas condição inicial de uma delas, com o `shooting` iremos encontrar o valor inicial da segunda

```python
iv = shooting(F, [0,0], 1, xs, 0, eps)
sol = rk_4(F, xs, [0,iv], eps)
print(f"u_2(-1) = {iv}")
```

    u_2(-1) = 2.1413073185710805

Com isso temos os valores da solução da Equação no grid $xs$, agora podemos utilizar esses valores para produzir a função $f(x) = \frac{1 - 2(1-x^2)u - u^2}{\epsilon}$

```python
def func_f(x,u):
    return (1-2*(1-x**2)*u-u**2)/eps

fx = func_f(xs, sol[:,0])
```

```python
plt.figure(figsize=(15,8))
plt.plot(xs, fx, label = '$f(x)$')
plt.title(f"$f(x)$ com $\epsilon = {eps}$")
plt.grid(alpha = 0.5)
plt.legend()
plt.show()
```

Agora temos os valores $f(x) = \frac{\mathrm{d}^2u}{\mathrm{d}x^2}$ no grid $x$, podemos construir o vetor $\vec{f}: f_i = \int^1_{-1}f(x)\varphi_i \mathrm{d}x$, para isso vamos fazer a integração numericamente utilizando o metódo do `Trapézio`

```python
f_i = np.zeros(N)
for i in range(N):
    int_f = fx * base(i,xs)
    f_i[i] = scipy.integrate.trapz(int_f, xs)
```

### Encontrando o vetor $\vec{c}$
Do fato de que
$$
A \vec{c} = \vec{f}
$$
podemos tirar $\vec{c}$ resolvendo o sistema

```python
c_i = np.linalg.solve(A,f_i)
```

### Resolvendo a Equação do Problema com Elementos Finitos
Usando o fato de que:  $u(x) \approx u_h(x) = \displaystyle\sum^N_{j = 1} c_j \varphi_j(x)$, temos agora todos os elementos para criar nossa função que aproxima a solução da equação diferencial

```python
def u_h(c):
    def f(x):
        soma = 0
        for i in range(len(c)):
            partial = c[i]*base(i+1,x)
        soma += partial
        return soma
    return f
```

```python
plt.figure(figsize=(16,8))
plt.plot(xs, u_h(c_i)(xs))
plt.grid(alpha = 0.5)
plt.title('Solução da Equação Diferencial Utilizando F.E.M. | $\epsilon = 1$')
plt.show()
```