# Quantization of Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Quantization Error of a Linear Uniform Quantizer

As illustrated in the [preceding section](linear_uniform_characteristic.ipynb), quantization results in two different types of distortions. Overload distortions are a consequence of exceeding the minimum/maximum amplitude of the quantizer. Granular distortions are a consequence of the quantization process when no clipping occurs. Various measures are used to quantify the distortions of a quantizer. We limit ourselves to the signal-to-noise ratio as commonly used measure.

### Signal-to-Noise Ratio

A quantizer can be evaluated by its [signal-to-noise ratio](https://en.wikipedia.org/wiki/Signal-to-noise_ratio) (SNR), which is defined as the power of the continuous amplitude signal $x[k]$ divided by the power of the quantization error $e[k]$. Under the assumption that both signals are drawn from a zero-mean wide-sense stationary (WSS) process, the average SNR is given as

\begin{equation}
SNR = 10 \cdot \log_{10} \left( \frac{\sigma_x^2}{\sigma_e^2} \right) \quad \text{ in dB}
\end{equation}

where $\sigma_x^2$ and $\sigma_e^2$ denote the variances of the signals $x[k]$ and $e[k]$, respectively. The SNR quantifies the average impact of the distortions introduced by quantization. The statistical properties of the signal $x[k]$ and the quantization error $e[k]$ are required in order to evaluate the SNR of a quantizer. First, a statistical model for the quantization error is introduced.

### Model for the Quantization Error

In order to derive the statistical properties of the quantization error, the probability density functions (PDFs) of the quantized signal $x_\text{Q}[k]$ and the error $e[k]$, as well as its bivariate PDFs have to be derived. The underlying calculus is quite tedious due to the nonlinear nature of quantization. Please refer to [[Widrow](../index.ipynb#Literature)] for a detailed treatment. The resulting model is summarized in the following. We focus on the non-clipping case $x_\text{min} \leq x[k] < x_\text{max}$ first, hence on granular distortions. Here the quantization error is in general bounded $|e[k]| < \frac{Q}{2}$.

Under the assumption that the input signal has a wide dynamic range compared to the quantization step size $Q$, the quantization error $e[k]$ can be approximated by the following statistical model

1. The quantization error $e[k]$ is not correlated with the input signal $x[k]$

2. The quantization error is [white](../random_signals/white_noise.ipynb)

    $$ \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sigma_e^2 $$

3. The probability density function (PDF) of the quantization error is given by the zero-mean [uniform distribution](../random_signals/important_distributions.ipynb#Uniform-Distribution)

    $$ p_e(\theta) = \frac{1}{Q} \cdot \text{rect} \left( \frac{\theta}{Q} \right) $$

The variance of the quantization error is then [derived from its PDF](../random_signals/important_distributions.ipynb#Uniform-Distribution) as

\begin{equation}
\sigma_e^2 = \frac{Q^2}{12}
\end{equation}

Let's assume that the quantization index is represented as binary or [fixed-point number](https://en.wikipedia.org/wiki/Fixed-point_arithmetic) with $w$-bits. The common notation for the mid-tread quantizer is that $x_\text{min}$ can be represented exactly. Half of the $2^w$ quantization indexes is used for the negative signal values, the other half for the positive ones including zero. The quantization step is then given as

\begin{equation}
Q = \frac{ |x_\text{min}|}{2^{w-1}} = \frac{ x_\text{max}}{2^{w-1} - 1}
\end{equation}

where $x_\text{max} = |x_\text{min}| - Q$. Introducing the quantization step, the variance of the quantization error can be expressed by the word length $w$ as

\begin{equation}
\sigma_e^2 = \frac{x^2_\text{max}}{3 \cdot 2^{2w}}
\end{equation}

The average power of the quantization error quarters per additional bit spend. Introducing the variance into the definition of the SNR yields

\begin{equation}
\begin{split}
SNR &= 10 \cdot \log_{10} \left( \frac{3 \sigma_x^2}{x^2_\text{max}}  \right) + 10 \cdot \log_{10} \left( 2^{2w} \right) \\
& \approx 10 \cdot \log_{10} \left( \frac{3 \sigma_x^2}{x^2_\text{max}}  \right) + 6.02 w \quad \text{in dB}
\end{split}
\end{equation}

It now can be concluded that the SNR decays approximately by 6 dB per additional bit spend. This is often referred to as the 6 dB per bit rule of thumb for linear uniform quantization. Note, this holds only under the assumptions stated above.

### Uniformly Distributed Signal

A statistical model for the input signal $x[k]$ is required in order to calculate the average SNR of a linear uniform quantizer. For a signal that conforms to a zero-mean uniform distribution and under the assumption $x_\text{max} \gg Q$ its PDF is given as

\begin{equation}
p_x(\theta) = \frac{1}{2 x_\text{max}} \text{rect}\left( \frac{\theta}{2 x_\text{max}} \right)
\end{equation}

Hence, all amplitudes between $-x_\text{max}$ and $x_\text{max}$ occur with the same probability. The variance of the signal is then calculated to

\begin{equation}
\sigma_x^2 = \frac{4 x_\text{max}^2}{12}
\end{equation}

Introducing $\sigma_x^2$ and $\sigma_e^2$ into the definition of the SNR yields

\begin{equation}
SNR = 10 \cdot \log_{10} \left( 2^{2 w} \right) \approx 6.02 \, w \quad \text{in dB}
\end{equation}

The word length $w$ and resulting SNRs for some typical digital signal representations are

| | $w$ | SNR |
|----|:----:|:----:|
| Compact Disc (CD) | 16 bit | 96 dB |
| Digital Video Disc (DVD) | 24 bit | 144 dB |
| Video Signals | 8 bit | 48 dB |

Note that the SNR values hold only if the continuous amplitude signal conforms reasonably well to a uniform PDF and if it uses the full amplitude range of the quantizer. If the latter is not the case this can be considered by introducing the level $0 < A \leq 1$ into above considerations, such that $x_\text{min} \leq \frac{x[k]}{A} < x_\text{max}$. The resulting variance is given as

\begin{equation}
\sigma_x^2 = \frac{4 x_\text{max}^2 A^2}{12}
\end{equation}

introduced into the definition of the SNR yields

\begin{equation}
SNR = 10 \cdot \log_{10} \left( 2^{2 w} \right) + 20 \cdot \log_{10} ( A ) \approx 6.02 \, w + 20 \cdot \log_{10} ( A ) \quad \text{in dB}
\end{equation}

From this it can be concluded that a level of -6 dB is equivalent to a loss of one bit in terms of SNR of the quantized signal.

#### Example - Quantization of a uniformly distributed signal

In this example the linear uniform quantization of a random signal drawn from a uniform distribution is evaluated. The amplitude range of the quantizer is $x_\text{min} = -1$ and $x_\text{max} = 1 - Q$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

w = 8  # wordlength of the quantized signal
xmin = -1  # mimimum amplitude of input signal
N = 8192  # number of samples
K = 30  # maximum lag for cross-correlation

def uniform_midtread_quantizer(x, Q):
    # limiter
    x = np.copy(x)
    idx = np.where(x <= -1)
    x[idx] = -1
    idx = np.where(x > 1 - Q)
    x[idx] = 1 - Q
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

def analyze_quantizer(x, e):
    # estimated PDF of error signal
    pe, bins = np.histogram(e, bins=20, density=True, range=(-Q, Q))
    # estimate cross-correlation between input and error
    ccf = 1/len(x) * np.correlate(x, e, mode='full')
    # estimate PSD of error signal
    nf, Pee = sig.welch(e, nperseg=128)
    # estimate SNR
    SNR = 10*np.log10((np.var(x)/np.var(e)))
    print('SNR = %f in dB' %SNR)

    # plot statistical properties of error signal
    plt.figure(figsize=(9,4))

    plt.subplot(121)
    plt.bar(bins[:-1]/Q, pe*Q, width = 2/len(pe))
    plt.title('Estimated histogram of quantization error')
    plt.xlabel(r'$\theta / Q$')
    plt.ylabel(r'$\hat{p}_x(\theta) / Q$')
    plt.axis([-1, 1, 0, 1.2])

    plt.subplot(122)
    plt.plot(nf*2*np.pi, Pee*6/Q**2)
    plt.title('Estimated PSD of quantization error')
    plt.xlabel(r'$\Omega$')
    plt.ylabel(r'$\hat{\Phi}_{ee}(e^{j \Omega}) / \sigma_e^2$')
    plt.axis([0, np.pi, 0, 2])
    plt.tight_layout()

    plt.figure(figsize=(10,6))
    ccf = ccf[N-K-1:N+K-1]
    kappa = np.arange(-len(ccf)//2,len(ccf)//2)
    plt.stem(kappa, ccf)
    plt.title('Cross-correlation function between input signal and error')
    plt.xlabel(r'$\kappa$')
    plt.ylabel(r'$\varphi_{xe}[\kappa]$')

# quantization step
Q = 1/(2**(w-1))
# compute input signal
np.random.seed(1)
x = np.random.uniform(size=N, low=xmin, high=(-xmin-Q))
# quantize signal
xQ = uniform_midtread_quantizer(x, Q)
e = xQ - x
# analyze quantizer
analyze_quantizer(x, e)
```

**Exercise**

* Change the number of bits `w` and check if the derived SNR holds
* How does the SNR change if you lower the magnitude of the minimum amplitude `xmin` of the input signal?
* What happens if you chose the magnitude of the minimum amplitude `xmin` in the range of the quantization step? Why?

Solution: The numerically computed SNR conforms well to the theoretic result derived above. Lowering the magnitude of the minimum amplitude results in a lower SNR as predicted above. The input signal $x[k]$ is correlated to the quantization error $e[k]$ if the magnitude of the minimum amplitude is lowered such that it is close to the quantization step. Here the assumptions made for the statistical model of the quantization error do not hold.

### Harmonic Signal

For a harmonic input signal $x[k] = x_\text{max} \cdot \cos[\Omega_0 k]$ the variance $\sigma_x^2$ is given by its squared [root mean square](https://en.wikipedia.org/wiki/Root_mean_square) (RMS) value

\begin{equation}
\sigma_x^2 = \frac{x_\text{max}^2}{2}
\end{equation}

Introducing this into the definition of the SNR together with the variance $\sigma_e^2$ of the quantization error yields

\begin{equation}
SNR = 10 \cdot \log_{10} \left(2^{2 w} \cdot \frac{3}{2} \right) \approx 6.02 \, w + 1.76 \quad \text{in dB}
\end{equation}

The gain of 1.76 dB with respect to the case of a uniformly distributed input signal is due to the fact that the amplitude distribution of a harmonic signal is not uniform

\begin{equation}
p_x(\theta) = \frac{1}{\pi \sqrt{1 - (\frac{\theta}{x_\text{max}})^2}}
\end{equation}

for $|\theta| < x_\text{max}$. High amplitudes are more likely to occur. The relative power of the quantization error is lower for higher amplitudes which results in an increase of the average SNR.

### Normally Distributed Signal

So far, we did not consider clipping of the input signal $x[k]$, e.g. by ensuring that its minimum/maximum values do not exceed the limits of the quantizer. However, this cannot always be ensured for practical signals. Moreover, many practical signals cannot be modeled as a uniform distribution. For instance, a [normally distributed](../random_signals/important_distributions.ipynb#Normal-Distribution) random signal exceeds a given maximum value with non-zero probability. Hence, clipping will occur for such an input signal. Clipping results in overload distortions whose amplitude can be much higher that $\frac{Q}{2}$. For the overall average SNR both granular and overload distortions have to be included.

The root mean square (RMS) of the normal distributed input signal is given by its standard deviation $\sigma_x$. The RMS level $A$ of the input signal normalized to the maximum level of the quantizer as

\begin{equation}
A = \frac{\sigma_x}{x_\text{max}}
\end{equation}

The probability that clipping occurs can be derived from the [cumulative distribution function](../random_signals/important_distributions.ipynb#Normal-Distribution) (CDF) of the normal distribution as

\begin{equation}
\Pr \{ |x[k]| > x_\text{max} \} = 1 + \text{erf} \left( \frac{-1}{\sqrt{2} A} \right)
\end{equation}

where $x_\text{max} = - x_\text{min}$ was assumed. For a normally distributed signal with a given probability that clipping occurs $\Pr \{ |x[k]| > x_\text{max} \} = 10^{-5}$ the SNR can be approximately calculated to [[Zölzer](../index.ipynb#Literature)]

\begin{equation}
SNR \approx 6.02 \, w - 8.5 \quad \text{in dB}
\end{equation}

The reduction of the SNR by 8.5 dB results from the fact that small signal values are more likely to occur for a normally distributed signal. The relative quantization error for small signals is higher, which results in a lower average SNR. Overload distortions due to clipping result in a further reduction of the average SNR.

#### Example - Quantization of a normal distributed signal

The following example evaluates the SNR of a linear uniform quantizer with $w=8$ for a normally distributed signal $x[k]$. The SNR is computed and plotted for various RMS levels, the probabilities for clipping are shown additionally.

```python
from scipy.special import erf

w = 8  # wordlength of the quantizer
A = np.logspace(-2, 0, num=500)  # RMS levels
N = int(1e6)  # number of samples
np.random.seed(1)

def compute_SNR(a):
    # compute input signal
    x = np.random.normal(size=N, scale=a)
    # quantize signal
    xQ = uniform_midtread_quantizer(x, Q)
    e = xQ - x
    # compute SNR
    SNR = 10*np.log10((np.var(x)/np.var(e)))

    return SNR

def plot_SNR(A, SNR):
    # plot results
    plt.figure(figsize=(8,4))
    plt.plot(20*np.log10(A), SNR)
    plt.xlabel(r'RMS level $\sigma_x / x_\mathrm{min}$ in dB')
    plt.ylabel('SNR in dB')
    plt.grid()

# quantization step
Q = 1/(2**(w-1))
# compute SNR for given RMS levels
SNR = [compute_SNR(a) for a in A]
# plot results
plot_SNR(A, SNR)
# find maximum SNR
Amax = A[np.argmax(SNR)]
Pc = 1 + erf(-1/(np.sqrt(2)*Amax))
print(r'Maximum SNR = {0:2.3f} dB for A = {1:2.1f} dB with clipping probability {2:2.1e}'
      .format(np.array(SNR).max(), 20*np.log10(Amax), Pc))
```

**Exercise**

* Can you explain the overall shape of the SNR?
* For which RMS level and probability of clipping is the SNR optimal?
* Change the wordlength `w` of the quantizer. How does the SNR change?

Solution: The SNR is low for low RMS levels of the input signal since the relative level of the quantization error is high. The SNR increases with increasing level until the clipping errors become dominant which make the SNR decay after its maximum. The SNR is optimal for $A \approx -12$ dB which is equivalent to $\Pr \{ |x[k]| > x_\text{max} \} \approx 10^{-4}$. Increasing the wordlength by one bit increases the SNR approximately by 6 dB.

### Laplace Distributed Signal

The [Laplace distribution](../random_signals/important_distributions.ipynb#Laplace-Distribution) is a commonly applied model for speech and music signals. As for the normal distribution, clipping will occur with a non-zero probability. The probability that clipping occurs can be derived from the [cumulative distribution function](../random_signals/important_distributions.ipynb#Laplace-Distribution) (CDF) of the normal distribution as

\begin{equation}
\Pr \{ |x[k]| > x_\text{max} \} = e^{- \frac{\sqrt{2}}{A}}
\end{equation}

The SNR for a Laplace distributed signal is in general lower compared to a normal distributed signal. The reason for this is, that the Laplace distribution features low signal values with a higher and large values with a lower probability in comparison to the normal distribution. The relative quantization error for small signals is higher, which results in a lower average SNR. The probability of overload distortions is also higher compared to the normal distribution.

#### Example - Quantization of a Laplace distributed signal

The following example evaluates the SNR of a linear uniform quantizer with $w=8$ for a Laplace distributed signal $x[k]$. The SNR is computed and plotted for various RMS levels.

```python
w = 8  # wordlength of the quantizer
A = np.logspace(-2, 0, num=500)  # relative RMS levels
N = int(1e6)  # number of samples
np.random.seed(1)

def compute_SNR(a):
    # compute input signal
    x = np.random.laplace(size=N, scale=a/np.sqrt(2))
    # quantize signal
    xQ = uniform_midtread_quantizer(x, Q)
    e = xQ - x
    # compute SNR
    SNR = 10*np.log10((np.var(x)/np.var(e)))

    return SNR

# quantization step
Q = 1/(2**(w-1))
# compute SNR for given RMS levels
SNR = [compute_SNR(a) for a in A]
# plot results
plot_SNR(A, SNR)
# find maximum SNR
Amax = A[np.argmax(SNR)]
Pc = np.exp(-np.sqrt(2)/Amax)
print(r'Maximum SNR = {0:2.3f} dB for A = {1:2.1f} dB with clipping probability {2:2.1e}'
      .format(np.array(SNR).max(), 20*np.log10(Amax), Pc))
```

**Exercise**

* Compare the SNR for the Laplace distributed signal to the case of a normally distributed signal. What is different?

Solution: The overall SNR is lower compared to the case of a normally distributed signal. Its maximum is also at lower RMS levels. Both can be explained by the properties of the Laplace distribution discussed above.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


```python
from IPython.core.display import HTML

import sympy as sp
from sympy import oo
from utils import symplot, symdisp, round_expr
import numpy as np

HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

# *Circuitos Elétricos I - Semana 6*

### Problema 1

Para o circuito abaixo, tem-se que $v(t)=-1800te^{-20t}$ para $t\geq0$ e $i_1(0)=4\;A$ e $i_2(0)=-16\;A$

a) Determine $i_1(t)$ e $i_2(t)$ para $t\geq0$.\
b) Determine a energia fornecida à fonte de tensão no intervalo $0\leq t \leq\infty$.\
c) Determine a energia inicial armazenada nos indutores.\
d) Determine a energia final armazenada nos indutores.

```python
# define as variáveis
t = sp.symbols('t', real=True)

# expressão para a tensão v(t)
v = -1800*t*sp.exp(-20*t)

symdisp('v(t) = ', v, 'V')
```

```python
tmax = 0.5
intervalo = np.linspace(0, tmax, num=1000)
symplot(t, v, intervalo, funLabel= 'v(t)')
```

```python
# valores das indutâncias
L1 = 10
L2 = 30

# valores iniciais das correntes nos indutores
i1_0 = 4
i2_0 = -16

#i1_0 = 1
#i2_0 = 2
```

```python
# correntes nos indutores em função da tensão aplicada aos terminais
i1 = -(1/L1)*sp.integrate(v, (t, 0, t)) + i1_0
i2 = -(1/L2)*sp.integrate(v, (t, 0, t)) + i2_0

print('Correntes nos indutores:')
symdisp('i_1(t) = ', round_expr(i1, 2) , ' A')
symdisp('i_2(t) = ', round_expr(i2, 2) , ' A')
```

```python
tmax = 0.5
intervalo = np.linspace(0, tmax, num=1000)
symplot(t, i1, intervalo, funLabel= '$i_1(t)$')
```

```python
symplot(t, i2, intervalo, funLabel= '$i_2(t)$')
```

```python
# LKC
i = i1 + i2

# potência desenvolvida pela fonte
p = v*i

symdisp('p(t) = ', round_expr(p.simplify(),2), 'W')
```

```python
symplot(t, p, intervalo, funLabel= '$p(t)$')
```

```python
# energia entrege à fonte
E = sp.integrate(p, (t, 0, oo))

print('Energia entrege à fonte quando t tende a infinito:')
symdisp('E = ', E, 'J')
```

```python
# calculando os valores de energia em t=0

E1_0 = (1/2)*L1*(i1.evalf(subs={t:0}))**2
E2_0 = (1/2)*L2*(i2.evalf(subs={t:0}))**2

print('Energia inicial armazenada nos indutores:')
symdisp('E_1(0) = ', E1_0, 'J')
symdisp('E_2(0) = ', E2_0, 'J')
```

```python
# calculando os valores de energia em t =oo

E1_inf = (1/2)*L1*(i1.evalf(subs={t:100}))**2
E2_inf = (1/2)*L2*(i2.evalf(subs={t:100}))**2

print('Energia final armazenada nos indutores:')
symdisp('E_1(\infty) = ', round_expr(E1_inf, 2), 'J')
symdisp('E_2(\infty) = ', round_expr(E2_inf, 2), 'J')
```

```python
# calculando a variação de energia nos indutores

ΔE = (E1_inf-E1_0) + (E2_inf-E2_0)

print('Variação da energia armazenada nos indutores:')
symdisp('ΔE = ', round_expr(ΔE,2), 'J')
```

### Problema 2

Obtendo expressões para as indutâncias equivalentes em circuitos com acoplamento magnético

#### Associação em paralelo

$$
\begin{aligned}
&v_L = L_{1} \frac{d i_{1}}{d t}+M \frac{d i_{2}}{d t}\\
&v_L = L_{2} \frac{d i_{2}}{d t}+M \frac{d i_{1}}{d t}
\end{aligned}
$$

#### Definindo as equações do circuito na forma matricial

```python
L1, L2, M, vL, t = sp.symbols('L_1, L_2, M, v_L, t', real=True)
```

```python
# define a variável tempo
t = sp.symbols('t', real=True)

# define as indutâncias
L1, L2, M = sp.symbols('L_1, L_2, M', real=True, positive=True)

# define as correntes i1 e i2
i1 = sp.Function('i_1')(t)
i2 = sp.Function('i_2')(t)

# define a tensão vL
vL = sp.Function('v_L')(t)

# define as equações do circuito no formato matricial
A  = sp.Matrix([[L1, -M],[-M, L2]])
V  = sp.Matrix([[vL],[vL]])

I  = sp.Matrix([[i1],[i2]])
dI = sp.diff(I, t)

symdisp('A = ', A)
symdisp('V = ', V)
symdisp(r'\frac{dI}{dt} = ', dI)
```

$\displaystyle A = \left[\begin{matrix}L_{1} & - M\\- M & L_{2}\end{matrix}\right]\; $

$\displaystyle V = \left[\begin{matrix}\operatorname{v_{L}}{\left(t \right)}\\\operatorname{v_{L}}{\left(t \right)}\end{matrix}\right]\; $

$\displaystyle \frac{dI}{dt} = \left[\begin{matrix}\frac{d}{d t} \operatorname{i_{1}}{\left(t \right)}\\\frac{d}{d t} \operatorname{i_{2}}{\left(t \right)}\end{matrix}\right]\; $

#### Equação da tensão em função das correntes na forma matricial

```python
sp.Eq(V, A*dI)
```

$\displaystyle \left[\begin{matrix}\operatorname{v_{L}}{\left(t \right)}\\\operatorname{v_{L}}{\left(t \right)}\end{matrix}\right] = \left[\begin{matrix}L_{1} \frac{d}{d t} \operatorname{i_{1}}{\left(t \right)} - M \frac{d}{d t} \operatorname{i_{2}}{\left(t \right)}\\L_{2} \frac{d}{d t} \operatorname{i_{2}}{\left(t \right)} - M \frac{d}{d t} \operatorname{i_{1}}{\left(t \right)}\end{matrix}\right]$

#### Determinado a inversa da matriz de indutâncias $A$

```python
# matriz inversa de A
symdisp('A^{-1} = ' , A**-1)
```

$\displaystyle A^{-1} = \left[\begin{matrix}\frac{L_{2}}{L_{1} L_{2} - M^{2}} & \frac{M}{L_{1} L_{2} - M^{2}}\\\frac{M}{L_{1} L_{2} - M^{2}} & \frac{L_{1}}{L_{1} L_{2} - M^{2}}\end{matrix}\right]\; $

#### Determinando o vetor de derivadas das correntes

```python
# calcula o vetor de derivadas das correntes
dI = (A**-1)*V

dI.simplify()

symdisp(r'\frac{dI}{dt} = ', dI)
```

$\displaystyle \frac{dI}{dt} = \left[\begin{matrix}\frac{\left(L_{2} + M\right) \operatorname{v_{L}}{\left(t \right)}}{L_{1} L_{2} - M^{2}}\\\frac{\left(L_{1} + M\right) \operatorname{v_{L}}{\left(t \right)}}{L_{1} L_{2} - M^{2}}\end{matrix}\right]\; $

#### LKC

```python
# di0/dt = di1/dt + di2/dt
dI0 = dI[0] + dI[1]

symdisp(r'\frac{di_0}{dt} = \frac{di_1}{dt} + \frac{di_2}{dt} =  ', dI0)
```

$\displaystyle \frac{di_0}{dt} = \frac{di_1}{dt} + \frac{di_2}{dt} =  \frac{\left(L_{1} + M\right) \operatorname{v_{L}}{\left(t \right)}}{L_{1} L_{2} - M^{2}} + \frac{\left(L_{2} + M\right) \operatorname{v_{L}}{\left(t \right)}}{L_{1} L_{2} - M^{2}}\; $

#### Obtendo a expressão para a indutância equivalente

```python
# indutância equivalente: vL = Leq*di0/dt -> Leq = vL/di0/dt
Leq = vL/dI0

symdisp('L_{eq} = ', Leq.simplify())
```

$\displaystyle L_{eq} = \frac{L_{1} L_{2} - M^{2}}{L_{1} + L_{2} + 2 M}\; $

Exemplos de circuitos com indutância mútua:

Em paralelo: https://tinyurl.com/y9zo85wm \
Em série: https://tinyurl.com/y7jrvv2y

Jupyter notebooks/Circuitos Eletricos I - Semana 6.1.ipynb


```python
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
import random
import sympy
```

    Bad key "text.kerning_factor" on line 4 in
    /home/sc/anaconda3/envs/old_nx/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test_patch.mplstyle.
    You probably need to get an updated matplotlibrc file from
    http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
    or from the matplotlib source distribution

```python
max_B = 10000 # maximum budget
no_runs = 100 # no. of runs to average
K = 4 # controls number of restarts

np.random.seed(0)
random.seed(42)
```

```python
edges = np.loadtxt('facebook_combined.txt',dtype=int)

G = nx.Graph()
G.add_edges_from(edges)

G_no_edges=G.number_of_edges()
G_no_nodes=G.number_of_nodes()
print(G_no_nodes,G_no_edges)
```

    4039 88234

```python
from louvain import detect_communities, modularity
def draw_communities(G, node_size=90, alpha=1, k=None, randomized=False):
    partition = detect_communities(G, randomized=randomized)
    print("Modularity for best partition:", modularity(G, partition))
    community_map = {}
    for community, nodes in enumerate(partition):
        for node in nodes:
            community_map[node] = community

    cmap = plt.get_cmap("jet")
    plt.figure(figsize=(15, 15))
    pos = nx.spring_layout(G, k=k)
    indexed = [community_map.get(node) for node in G]
    plt.axis("off")
    nx.draw_networkx_nodes(G, pos=pos, cmap=cmap, node_color=indexed, node_size=node_size, alpha=alpha)
    nx.draw_networkx_edges(G, pos=pos, alpha=0.2)
    for x in range(len(pos)):
        pos[x] = pos[x]+np.array([0.02,0])
    labels = dict([(n, n) for n in G.nodes()]) #add labels
    #_ = nx.draw_networkx_labels(G, pos, labels=labels, font_color='#000000', font_size=15) #draw labels
draw_communities(G)
```

```python
partition = detect_communities(G, randomized=False)
community_map = {}
node_map = {}

for community, nodes in enumerate(partition):
    for node in nodes:
        node_map[node] = community
    community_map[community] = nodes
```

```python
def MH_sampling(G,B):
    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        neighbors = list(nx.neighbors(G,sample))
        sample_t = np.random.choice(neighbors)
        if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
            sample = sample_t
        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)

def RDS_sampling(G,B):
    est_RW = []
    est_RW_t1 = 0
    est_RW_t2 = 0
    sample = np.random.choice(G.nodes())
    deg_pr_sent = G.degree(sample)
    est_RW_t1 += node_fn(sample)/deg_pr_sent
    est_RW_t2 += 1/deg_pr_sent
    est_RW.append(est_RW_t1/est_RW_t2)
    for ii in range(2,B+1):
        neighbors = list(nx.neighbors(G,sample))
        sample = np.random.choice(neighbors)
        deg_pr_sent = G.degree(sample)
        est_RW_t1 += node_fn(sample)/deg_pr_sent
        est_RW_t2 += 1/deg_pr_sent
        est_RW.append(est_RW_t1/est_RW_t2)
    return np.array(est_RW)
```

```python
def RDSRR_sampling(G,B,U=None):
    restart_ind = [10]
    t=20
    while restart_ind[-1]<B:
        restart_ind.append(restart_ind[-1]+int(80*np.log(t)))
        t+=1

    est_RW = []
    est_RW_t1 = 0
    est_RW_t2 = 0
    sample = np.random.choice(G.nodes())
    deg_pr_sent = G.degree(sample)
    est_RW_t1 += node_fn(sample)/deg_pr_sent
    est_RW_t2 += 1/deg_pr_sent
    est_RW.append(est_RW_t1/est_RW_t2)
    for ii in range(2,B+1):
        if ii in restart_ind:
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                z = [node_map[x] for x in U].index(node_map[sample])
                sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample = np.random.choice(neighbors)

        deg_pr_sent = G.degree(sample)
        est_RW_t1 += node_fn(sample)/deg_pr_sent
        est_RW_t2 += 1/deg_pr_sent
        est_RW.append(est_RW_t1/est_RW_t2)
    return np.array(est_RW)

def MHRR_sampling(G,B,U=None):
    restart_ind = [10]
    t=20
    while restart_ind[-1]<B:
        restart_ind.append(restart_ind[-1]+int(K*np.log(t)))
        t+=1

    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        if ii in restart_ind:
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                z = [node_map[x] for x in U].index(node_map[sample])
                sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample_t = np.random.choice(neighbors)
            if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
                sample = sample_t

        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)
```

## F(v) = int(node_map[node] == 1)

```python
def node_fn(node):
    return int(node_map[node]==1)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.10670958157959891

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MH')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDS')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=2.5)
plt.grid()
```

## F(v) = int(G.degree(node)>100)

```python
def node_fn(node):
    return int(G.degree(node)>100)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.11908888338697697

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MH')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDS')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=2)
plt.grid()
```

## F(v) = isprime(v)

```python
def node_fn(node):
    return int(sympy.isprime(G.degree(node)))
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.2904184204010894

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MH')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDS')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR')
plt.ylim(0,0.5)

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.grid()
```

## F(v) = random

```python
fn_mapping = np.random.exponential(1,size=(G_no_nodes))
def node_fn(node):
    return fn_mapping[node]

F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    1.0110090445802433

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MH')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDS')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(0,0.3)
plt.grid()
```


# Assignment 4: Word Embeddings

Welcome to the fourth (and last) programming assignment of Course 2!

In this assignment, you will practice how to compute word embeddings and use them for sentiment analysis.
- To implement sentiment analysis, you can go beyond counting the number of positive words and negative words.
- You can find a way to represent each word numerically, by a vector.
- The vector could then represent syntactic (i.e. parts of speech) and semantic (i.e. meaning) structures.

In this assignment, you will explore a classic way of generating word embeddings or representations.
- You will implement a famous model called the continuous bag of words (CBOW) model.

By completing this assignment you will:

- Train word vectors from scratch.
- Learn how to create batches of data.
- Understand how backpropagation works.
- Plot and visualize your learned word vectors.

Knowing how to train these models will give you a better understanding of word vectors, which are building blocks to many applications in natural language processing.

## Outline

- [1 The Continuous bag of words model](#1)
- [2 Training the Model](#2)
    - [2.0 Initialize the model](#2)
        - [Exercise 01](#ex-01)
    - [2.1 Softmax Function](#2.1)
        - [Exercise 02](#ex-02)
    - [2.2 Forward Propagation](#2.2)
        - [Exercise 03](#ex-03)
    - [2.3 Cost Function](#2.3)
    - [2.4 Backproagation](#2.4)
        - [Exercise 04](#ex-04)
    - [2.5 Gradient Descent](#2.5)
        - [Exercise 05](#ex-05)
- [3 Visualizing the word vectors](#3)

# 1. The Continuous bag of words model

Let's take a look at the following sentence:
>**'I am happy because I am learning'**.

- In continuous bag of words (CBOW) modeling, we try to predict the center word given a few context words (the words around the center word).
- For example, if you were to choose a context half-size of say $C = 2$, then you would try to predict the word **happy** given the context that includes 2 words before and 2 words after the center word:

> $C$ words before: [I, am]

> $C$ words after: [because, I]

- In other words:

$$context = [I,am, because, I]$$
$$target = happy$$

The structure of your model will look like this:

Where $\bar x$ is the average of all the one hot vectors of the context words.

Once you have encoded all the context words, you can use $\bar x$ as the input to your model.

The architecture you will be implementing is as follows:

\begin{align}
 h &= W_1 \  X + b_1  \tag{1} \\
 a &= ReLU(h)  \tag{2} \\
 z &= W_2 \  a + b_2   \tag{3} \\
 \hat y &= softmax(z)   \tag{4} \\
\end{align}

```
%%capture
!wget https://raw.githubusercontent.com/martin-fabbri/colab-notebooks/master/deeplearning.ai/nlp/datasets/shakespeare.txt
```

```
from collections import Counter, defaultdict

import nltk
import re
import numpy as np
import scipy as linalg
from nltk.tokenize import word_tokenize
```

```
#@markdown Utility functions
#@markdown - sigmoid(z)
#@markdown - get_idx(words, word2Ind)
#@markdown - pack_idx_with_frequency(context_words, word2Ind)
#@markdown - get_vectors(data, word2Ind, V, C)
#@markdown - get_batches(data, word2Ind, V, C, batch_size)
#@markdown - compute_pca(data, n_components=2)
#@markdown - get_dict(data)

def sigmoid(z):
    # sigmoid function
    return 1.0/(1.0+np.exp(-z))

def get_idx(words, word2Ind):
    idx = []
    for word in words:
        idx = idx + [word2Ind[word]]
    return idx

def pack_idx_with_frequency(context_words, word2Ind):
    freq_dict = defaultdict(int)
    for word in context_words:
        freq_dict[word] += 1
    idxs = get_idx(context_words, word2Ind)
    packed = []
    for i in range(len(idxs)):
        idx = idxs[i]
        freq = freq_dict[context_words[i]]
        packed.append((idx, freq))
    return packed

def get_vectors(data, word2Ind, V, C):
    i = C
    while True:
        y = np.zeros(V)
        x = np.zeros(V)
        center_word = data[i]
        y[word2Ind[center_word]] = 1
        context_words = data[(i - C):i] + data[(i+1):(i+C+1)]
        num_ctx_words = len(context_words)
        for idx, freq in pack_idx_with_frequency(context_words, word2Ind):
            x[idx] = freq/num_ctx_words
        yield x, y
        i += 1
        if i >= len(data):
            print('i is being set to 0')
            i = 0

def get_batches(data, word2Ind, V, C, batch_size):
    batch_x = []
    batch_y = []
    for x, y in get_vectors(data, word2Ind, V, C):
        while len(batch_x) < batch_size:
            batch_x.append(x)
            batch_y.append(y)
        else:
            yield np.array(batch_x).T, np.array(batch_y).T
            batch = []

def compute_pca(data, n_components=2):
    """
    Input:
        data: of dimension (m,n) where each row corresponds to a word vector
        n_components: Number of components you want to keep.
    Output:
        X_reduced: data transformed in 2 dims/columns + regenerated original data
    pass in: data as 2D NumPy array
    """

    m, n = data.shape

    ### START CODE HERE ###
    # mean center the data
    data -= data.mean(axis=0)
    # calculate the covariance matrix
    R = np.cov(data, rowvar=False)
    # calculate eigenvectors & eigenvalues of the covariance matrix
    # use 'eigh' rather than 'eig' since R is symmetric,
    # the performance gain is substantial
    evals, evecs = linalg.eigh(R)
    # sort eigenvalue in decreasing order
    # this returns the corresponding indices of evals and evecs
    idx = np.argsort(evals)[::-1]

    evecs = evecs[:, idx]
    # sort eigenvectors according to same index
    evals = evals[idx]
    # select the first n eigenvectors (n is desired dimension
    # of rescaled data array, or dims_rescaled_data)
    evecs = evecs[:, :n_components]
    ### END CODE HERE ###
    return np.dot(evecs.T, data.T).T

def get_dict(data):
    """
    Input:
        K: the number of negative samples
        data: the data you want to pull from
        indices: a list of word indices
    Output:
        word_dict: a dictionary with the weighted probabilities of each word
        word2Ind: returns dictionary mapping the word to its index
        Ind2Word: returns dictionary mapping the index to its word
    """
    #
#     words = nltk.word_tokenize(data)
    words = sorted(list(set(data)))
    n = len(words)
    idx = 0
    # return these correctly
    word2Ind = {}
    Ind2word = {}
    for k in words:
        word2Ind[k] = idx
        Ind2word[idx] = k
        idx += 1
    return word2Ind, Ind2word
```

```
# Download sentence tokenizer
nltk.data.path.append('.')
nltk.download('punkt')
```

    [nltk_data] Downloading package punkt to /root/nltk_data...
    [nltk_data]   Package punkt is already up-to-date!

    True

```
with open("shakespeare.txt") as f:
    data = f.read()                     #  Read in the data
data = re.sub(r"[,!?;-]", ".", data)    #  Punktuations are replaced by .
data = nltk.word_tokenize(data)         #  Tokenize string to words
data = [
    ch.lower() for ch in data if ch.isalpha() or ch == "."
] #  Lower case and drop non-alphabetical tokens
print("Number of tokens:", len(data), "\n", data[:15])
```

    Number of tokens: 60933
     ['o', 'for', 'a', 'muse', 'of', 'fire', '.', 'that', 'would', 'ascend', 'the', 'brightest', 'heaven', 'of', 'invention']

```
# Compute the frequency distribution of the words in the dataset (vocabulary)
fdist = nltk.FreqDist(word for word in data)
print("Size of vocabulary: ", len(fdist))
print(
    "Most frequent tokens: ", fdist.most_common(20)
)  # print the 20 most frequent words and their freq.
```

    Size of vocabulary:  5772
    Most frequent tokens:  [('.', 9630), ('the', 1521), ('and', 1394), ('i', 1252), ('to', 1159), ('of', 1093), ('my', 857), ('that', 781), ('in', 770), ('you', 748), ('a', 742), ('is', 630), ('not', 559), ('for', 467), ('it', 460), ('with', 441), ('his', 434), ('but', 417), ('me', 417), ('your', 397)]

#### Mapping words to indices and indices to words
We provide a helper function to create a dictionary that maps words to indices and indices to words.

```
# get_dict creates two dictionaries, converting words to indices and viceversa.
word2Ind, Ind2word = get_dict(data)
V = len(word2Ind)
print("Size of vocabulary: ", V)
```

    Size of vocabulary:  5772

```
# example of word to index mapping
print("Index of the word 'king' :  ",word2Ind['king'] )
print("Word which has index 2743:  ",Ind2word[2743] )
```

    Index of the word 'king' :   2743
    Word which has index 2743:   king

# 2 Training the Model

###  Initializing the model

You will now initialize two matrices and two vectors.
- The first matrix ($W_1$) is of dimension $N \times V$, where $V$ is the number of words in your vocabulary and $N$ is the dimension of your word vector.
- The second matrix ($W_2$) is of dimension $V \times N$.
- Vector $b_1$ has dimensions $N\times 1$
- Vector $b_2$ has dimensions  $V\times 1$.
- $b_1$ and $b_2$ are the bias vectors of the linear layers from matrices $W_1$ and $W_2$.

The overall structure of the model will look as in Figure 1, but at this stage we are just initializing the parameters.

### Exercise 01
Please use [numpy.random.rand](https://numpy.org/doc/stable/reference/random/generated/numpy.random.rand.html) to generate matrices that are initialized with random values from a uniform distribution, ranging between 0 and 1.

**Note:** In the next cell you will encounter a random seed. Please **DO NOT** modify this seed so your solution can be tested correctly.

```
# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: initialize_model
def initialize_model(N,V, random_seed=1):
    '''
    Inputs:
        N:  dimension of hidden vector
        V:  dimension of vocabulary
        random_seed: random seed for consistent results in the unit tests
     Outputs:
        W1, W2, b1, b2: initialized weights and biases
    '''

    np.random.seed(random_seed)

    ### START CODE HERE (Replace instances of 'None' with your code) ###
    # W1 has shape (N,V)
    W1 = np.random.rand(N, V)
    # W2 has shape (V,N)
    W2 = np.random.rand(V, N)
    # b1 has shape (N,1)
    b1 = np.random.rand(N, 1)
    # b2 has shape (V,1)
    b2 = np.random.rand(V, 1)
    ### END CODE HERE ###

    return W1, W2, b1, b2
```

```
# Test your function example.
tmp_N = 4
tmp_V = 10
tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(tmp_N,tmp_V)
assert tmp_W1.shape == ((tmp_N,tmp_V))
assert tmp_W2.shape == ((tmp_V,tmp_N))
print(f"tmp_W1.shape: {tmp_W1.shape}")
print(f"tmp_W2.shape: {tmp_W2.shape}")
print(f"tmp_b1.shape: {tmp_b1.shape}")
print(f"tmp_b2.shape: {tmp_b2.shape}")
```

    tmp_W1.shape: (4, 10)
    tmp_W2.shape: (10, 4)
    tmp_b1.shape: (4, 1)
    tmp_b2.shape: (10, 1)

##### Expected Output

```CPP
tmp_W1.shape: (4, 10)
tmp_W2.shape: (10, 4)
tmp_b1.shape: (4, 1)
tmp_b2.shape: (10, 1)
```

### 2.1 Softmax
Before we can start training the model, we need to implement the softmax function as defined in equation 5:

$$ \text{softmax}(z_i) = \frac{e^{z_i} }{\sum_{i=0}^{V-1} e^{z_i} }  \tag{5} $$

- Array indexing in code starts at 0.
- $V$ is the number of words in the vocabulary (which is also the number of rows of $z$).
- $i$ goes from 0 to |V| - 1.

### Exercise 02
**Instructions**: Implement the softmax function below.

- Assume that the input $z$ to `softmax` is a 2D array
- Each training example is represented by a column of shape (V, 1) in this 2D array.
- There may be more than one column, in the 2D array, because you can put in a batch of examples to increase efficiency.  Let's call the batch size lowercase $m$, so the $z$ array has shape (V, m)
- When taking the sum from $i=1 \cdots V-1$, take the sum for each column (each example) separately.

Please use
- numpy.exp
- numpy.sum (set the axis so that you take the sum of each column in z)

```
# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: softmax
def softmax(z):
    '''
    Inputs:
        z: output scores from the hidden layer [shape (V, m)] where m = #batch
    Outputs:
        yhat: prediction (estimate of y)
    '''

    ### START CODE HERE (Replace instances of 'None' with your own code) ###

    # Calculate yhat (softmax)
    z_exp = np.exp(z)
    z_exp_sum = np.sum(z_exp, axis=0)
    yhat = z_exp / z_exp_sum

    ### END CODE HERE ###

    return yhat
```

```
# Test the function
tmp = np.array([[1,2,3],
                [1,1,1]
               ])
tmp_sm = softmax(tmp)
display(tmp_sm)
```

    array([[0.5       , 0.73105858, 0.88079708],
           [0.5       , 0.26894142, 0.11920292]])

##### Expected Ouput

```CPP
array([[0.5       , 0.73105858, 0.88079708],
       [0.5       , 0.26894142, 0.11920292]])
```

### 2.2 Forward propagation

### Exercise 03
Implement the forward propagation $z$ according to equations (1) to (3).

\begin{align}
 h &= W_1 \  X + b_1  \tag{1} \\
 a &= ReLU(h)  \tag{2} \\
 z &= W_2 \  a + b_2   \tag{3} \\
\end{align}

For that, you will use as activation the Rectified Linear Unit (ReLU) given by:

$$f(h)=\max (0,h) \tag{6}$$

    Hints

You can use numpy.maximum(x1,x2) to get the maximum of two values

Use numpy.dot(A,B) to matrix multiply A and B

```
# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: forward_prop
def forward_prop(x, W1, W2, b1, b2):
    '''
    Inputs:
        x:  average one hot vector for the context
        W1, W2, b1, b2:  matrices and biases to be learned
     Outputs:
        z:  output score vector
    '''

    ### START CODE HERE (Replace instances of 'None' with your own code) ###

    # Calculate h
    h = np.dot(W1, x) + b1

    # Apply the relu on h (store result in h)
    h = np.maximum(0, h)

    # Calculate z
    z = np.dot(W2, h) + b2

    ### END CODE HERE ###

    return z, h
```

```
# Test the function

# Create some inputs
tmp_N = 2
tmp_V = 3
tmp_x = np.array([[0,1,0]]).T
tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(N=tmp_N,V=tmp_V, random_seed=1)

print(f"x has shape {tmp_x.shape}")
print(f"N is {tmp_N} and vocabulary size V is {tmp_V}")

# call function
tmp_z, tmp_h = forward_prop(tmp_x, tmp_W1, tmp_W2, tmp_b1, tmp_b2)
print("call forward_prop")
print()
# Look at output
print(f"z has shape {tmp_z.shape}")
print("z has values:")
print(tmp_z)

print()

print(f"h has shape {tmp_h.shape}")
print("h has values:")
print(tmp_h)
```

    x has shape (3, 1)
    N is 2 and vocabulary size V is 3
    call forward_prop

    z has shape (3, 1)
    z has values:
    [[0.55379268]
     [1.58960774]
     [1.50722933]]

    h has shape (2, 1)
    h has values:
    [[0.92477674]
     [1.02487333]]

##### Expected output
```CPP
x has shape (3, 1)
N is 2 and vocabulary size V is 3
call forward_prop

z has shape (3, 1)
z has values:
[[0.55379268]
 [1.58960774]
 [1.50722933]]

h has shape (2, 1)
h has values:
[[0.92477674]
 [1.02487333]]
```

## 2.3 Cost function

- We have implemented the *cross-entropy* cost function for you.

```
# compute_cost: cross-entropy cost functioN
def compute_cost(y, yhat, batch_size):
    # cost function
    logprobs = np.multiply(np.log(yhat),y) + np.multiply(np.log(1 - yhat), 1 - y)
    cost = - 1/batch_size * np.sum(logprobs)
    cost = np.squeeze(cost)
    return cost
```

```
# Test the function
tmp_C = 2
tmp_N = 50
tmp_batch_size = 4
tmp_word2Ind, tmp_Ind2word = get_dict(data)
tmp_V = len(word2Ind)

tmp_x, tmp_y = next(get_batches(data, tmp_word2Ind, tmp_V,tmp_C, tmp_batch_size))

print(f"tmp_x.shape {tmp_x.shape}")
print(f"tmp_y.shape {tmp_y.shape}")

tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(tmp_N,tmp_V)

print(f"tmp_W1.shape {tmp_W1.shape}")
print(f"tmp_W2.shape {tmp_W2.shape}")
print(f"tmp_b1.shape {tmp_b1.shape}")
print(f"tmp_b2.shape {tmp_b2.shape}")

tmp_z, tmp_h = forward_prop(tmp_x, tmp_W1, tmp_W2, tmp_b1, tmp_b2)
print(f"tmp_z.shape: {tmp_z.shape}")
print(f"tmp_h.shape: {tmp_h.shape}")

tmp_yhat = softmax(tmp_z)
print(f"tmp_yhat.shape: {tmp_yhat.shape}")

tmp_cost = compute_cost(tmp_y, tmp_yhat, tmp_batch_size)
print("call compute_cost")
print(f"tmp_cost {tmp_cost:.4f}")
```

    tmp_x.shape (5772, 4)
    tmp_y.shape (5772, 4)
    tmp_W1.shape (50, 5772)
    tmp_W2.shape (5772, 50)
    tmp_b1.shape (50, 1)
    tmp_b2.shape (5772, 1)
    tmp_z.shape: (5772, 4)
    tmp_h.shape: (50, 4)
    tmp_yhat.shape: (5772, 4)
    call compute_cost
    tmp_cost 12.9825

##### Expected output

```CPP
tmp_x.shape (5778, 4)
tmp_y.shape (5778, 4)
tmp_W1.shape (50, 5778)
tmp_W2.shape (5778, 50)
tmp_b1.shape (50, 1)
tmp_b2.shape (5778, 1)
tmp_z.shape: (5778, 4)
tmp_h.shape: (50, 4)
tmp_yhat.shape: (5778, 4)
call compute_cost
tmp_cost 9.9560
```

## 2.4 Training the Model - Backpropagation

### Exercise 04
Now that you have understood how the CBOW model works, you will train it.

You created a function for the forward propagation. Now you will implement a function that computes the gradients to backpropagate the errors.

```
# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: back_prop
def back_prop(x, yhat, y, h, W1, W2, b1, b2, batch_size):
    '''
    Inputs:
        x:  average one hot vector for the context
        yhat: prediction (estimate of y)
        y:  target vector
        h:  hidden vector (see eq. 1)
        W1, W2, b1, b2:  matrices and biases
        batch_size: batch size
     Outputs:
        grad_W1, grad_W2, grad_b1, grad_b2:  gradients of matrices and biases
    '''
    ### START CODE HERE (Replace instanes of 'None' with your code) ###

    # Compute l1 as W2^T (Yhat - Y)
    # Re-use it whenever you see W2^T (Yhat - Y) used to compute a gradient
    l1 = np.dot(W2.T, yhat - y)
    # Apply relu to l1
    l1 = np.maximum(0, l1)
    # Compute the gradient of W1
    grad_W1 = 1/batch_size * np.dot(l1, x.T)
    # Compute the gradient of W2
    grad_W2 = 1/batch_size * np.dot(yhat - y, h.T)
    # Compute the gradient of b1
    grad_b1 = 1/batch_size * np.sum(l1, axis=1, keepdims=True)
    # Compute the gradient of b2
    grad_b2 = 1/batch_size * np.sum(yhat - y, axis=1, keepdims=True)
    ### END CODE HERE ###

    return grad_W1, grad_W2, grad_b1, grad_b2
```

```
# Test the function
tmp_C = 2
tmp_N = 50
tmp_batch_size = 4
tmp_word2Ind, tmp_Ind2word = get_dict(data)
tmp_V = len(word2Ind)

# get a batch of data
tmp_x, tmp_y = next(get_batches(data, tmp_word2Ind, tmp_V,tmp_C, tmp_batch_size))

print("get a batch of data")
print(f"tmp_x.shape {tmp_x.shape}")
print(f"tmp_y.shape {tmp_y.shape}")

print()
print("Initialize weights and biases")
tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(tmp_N,tmp_V)

print(f"tmp_W1.shape {tmp_W1.shape}")
print(f"tmp_W2.shape {tmp_W2.shape}")
print(f"tmp_b1.shape {tmp_b1.shape}")
print(f"tmp_b2.shape {tmp_b2.shape}")

print()
print("Forwad prop to get z and h")
tmp_z, tmp_h = forward_prop(tmp_x, tmp_W1, tmp_W2, tmp_b1, tmp_b2)
print(f"tmp_z.shape: {tmp_z.shape}")
print(f"tmp_h.shape: {tmp_h.shape}")

print()
print("Get yhat by calling softmax")
tmp_yhat = softmax(tmp_z)
print(f"tmp_yhat.shape: {tmp_yhat.shape}")

tmp_m = (2*tmp_C)
tmp_grad_W1, tmp_grad_W2, tmp_grad_b1, tmp_grad_b2 = back_prop(tmp_x, tmp_yhat, tmp_y, tmp_h, tmp_W1, tmp_W2, tmp_b1, tmp_b2, tmp_batch_size)

print()
print("call back_prop")
print(f"tmp_grad_W1.shape {tmp_grad_W1.shape}")
print(f"tmp_grad_W2.shape {tmp_grad_W2.shape}")
print(f"tmp_grad_b1.shape {tmp_grad_b1.shape}")
print(f"tmp_grad_b2.shape {tmp_grad_b2.shape}")
```

    get a batch of data
    tmp_x.shape (5772, 4)
    tmp_y.shape (5772, 4)

    Initialize weights and biases
    tmp_W1.shape (50, 5772)
    tmp_W2.shape (5772, 50)
    tmp_b1.shape (50, 1)
    tmp_b2.shape (5772, 1)

    Forwad prop to get z and h
    tmp_z.shape: (5772, 4)
    tmp_h.shape: (50, 4)

    Get yhat by calling softmax
    tmp_yhat.shape: (5772, 4)

    call back_prop
    tmp_grad_W1.shape (50, 5772)
    tmp_grad_W2.shape (5772, 50)
    tmp_grad_b1.shape (50, 1)
    tmp_grad_b2.shape (5772, 1)

## Gradient Descent

### Exercise 05
Now that you have implemented a function to compute the gradients, you will implement batch gradient descent over your training set.

**Hint:** For that, you will use `initialize_model` and the `back_prop` functions which you just created (and the `compute_cost` function). You can also use the provided `get_batches` helper function:

```for x, y in get_batches(data, word2Ind, V, C, batch_size):```

```...```

Also: print the cost after each batch is processed (use batch size = 128)

```
# UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: gradient_descent
def gradient_descent(data, word2Ind, N, V, num_iters, alpha=0.03):

    '''
    This is the gradient_descent function

      Inputs:
        data:      text
        word2Ind:  words to Indices
        N:         dimension of hidden vector
        V:         dimension of vocabulary
        num_iters: number of iterations
     Outputs:
        W1, W2, b1, b2:  updated matrices and biases

    '''
    W1, W2, b1, b2 = initialize_model(N,V, random_seed=282)
    batch_size = 128
    iters = 0
    C = 2
    for x, y in get_batches(data, word2Ind, V, C, batch_size):
        ### START CODE HERE (Replace instances of 'None' with your own code) ###
        # Get z and h
        z, h = forward_prop(x, W1, W2, b1, b2)
        # Get yhat
        yhat = softmax(z)
        # Get cost
        cost = compute_cost(y, yhat, batch_size)
        if ( (iters+1) % 10 == 0):
            print(f"iters: {iters + 1} cost: {cost:.6f}")
        # Get gradients
        grad_W1, grad_W2, grad_b1, grad_b2 = back_prop(x, yhat, y, h, W1, W2, b1, b2, batch_size)

        # Update weights and biases
        W1 = W1 - alpha * grad_W1
        W2 = W2 - alpha * grad_W2
        b1 = b1 - alpha * grad_b1
        b2 = b2 - alpha * grad_b2

        ### END CODE HERE ###

        iters += 1
        if iters == num_iters:
            break
        if iters % 100 == 0:
            alpha *= 0.66

    return W1, W2, b1, b2
```

```
# test your function
C = 2
N = 50
word2Ind, Ind2word = get_dict(data)
V = len(word2Ind)
num_iters = 150
print("Call gradient_descent")
W1, W2, b1, b2 = gradient_descent(data, word2Ind, N, V, num_iters)
```

    Call gradient_descent
    iters: 10 cost: 0.543259
    iters: 20 cost: 0.105205
    iters: 30 cost: 0.057815
    iters: 40 cost: 0.039841
    iters: 50 cost: 0.030391
    iters: 60 cost: 0.024565
    iters: 70 cost: 0.020614
    iters: 80 cost: 0.017758
    iters: 90 cost: 0.015598
    iters: 100 cost: 0.013906
    iters: 110 cost: 0.012934
    iters: 120 cost: 0.012129
    iters: 130 cost: 0.011417
    iters: 140 cost: 0.010785
    iters: 150 cost: 0.010219

##### Expected Output

```CPP
iters: 10 cost: 0.789141
iters: 20 cost: 0.105543
iters: 30 cost: 0.056008
iters: 40 cost: 0.038101
iters: 50 cost: 0.028868
iters: 60 cost: 0.023237
iters: 70 cost: 0.019444
iters: 80 cost: 0.016716
iters: 90 cost: 0.014660
iters: 100 cost: 0.013054
iters: 110 cost: 0.012133
iters: 120 cost: 0.011370
iters: 130 cost: 0.010698
iters: 140 cost: 0.010100
iters: 150 cost: 0.009566
```

Your numbers may differ a bit depending on which version of Python you're using.


# Ensemble models from scratch
> Implementation of Decision Trees, Random Forests and Adaboost model from scratch using Pytorch.
- author: "Axel Mendoza"
- categories: [decision-tree, random-forest, adaboost, heart-disease, pytorch, eda, from-scratch]
- toc: false
- comments: true
- badges: true
- image: images/trees.jpg

During this experiment, we will investigate the maths and implement Decision Trees, Random Forests and Adaboost. We will use Pytorch to create and train these algorithms to predict heart diseases.

## Dataset
The models will be trained using the [Heart Disease Dataset](https://archive.ics.uci.edu/ml/datasets/heart+Disease).
Fourteen features were selected among 76 attributes:
1. age: age in years
2. sex:
    - 1: male
    - 0: female
3. cp: chest pain type
    - 0: asymptomatic
    - 1: atypical angina
    - 2: non-anginal pain
    - 3: typical angina
4. trestbps: resting blood pressure
5. chol: serum cholestoral in mg/dl
6. fbs: fasting blood sugar > 120 mg/dl
7. restecg: resting electrocardiographic results
    - 0: showing probable or definite left ventricular hypertrophy by Estes' criteria
    - 1: normal
    - 2: having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV)
8. thalach: maximum heart rate achieved
9. exang: exercise induced angina
10. oldpeak: ST depression induced by exercise relative to rest
11. slope: the slope of the peak exercise ST segment
    - 1: upsloping
    - 2: flat
    - 3: downsloping
12. ca: number of major vessels (0-3) colored by flourosopy
13. thal:
    - 3 : fixed defect
    - 6 : normal
    - 7 : reversable defect
14. disease:
    - 0: artery diameter narrowing < 50%
    - 1-3: artery diameter narrowing > 50%, close to 3 is very severe

```python
import os
import torch
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from pprint import pprint
from sklearn.model_selection import train_test_split
from sklearn.base import BaseEstimator
from sklearn.model_selection import cross_val_score
```

### Read csv using Pandas

```python
hdisease = pd.read_csv(os.path.join('data', 'processed.cleveland.data'))
```

```python

# The data does not include the columns names, we did a little trick to add them
columns = {
    '63.0' : 'age',
    '1.0' : 'sex',
    '1.0.1' : 'cp',
    '145.0' : 'trestbps',
    '233.0' : 'chol',
    '1.0.2' : 'fbs',
    '2.0' : 'restecg',
    '150.0' : 'thalach',
    '0.0' : 'exang',
    '2.3' : 'oldpeak',
    '3.0' : 'slope',
    '0.0.1' : 'ca',
    '6.0' : 'thal',
    '0' : 'disease'
}
hdisease.rename(columns=columns, inplace=True)
```

### Replacing missing values

```python

# Remove the severity of the disease, 0: healthy, 1: sick
hdisease.loc[hdisease['disease'] > 1, 'disease'] = 1

# Setting gather unknown value and assign a new class for them
hdisease.loc[hdisease['ca'] == '?', 'ca'] = 4.0
hdisease.loc[hdisease['thal'] == '?', 'thal'] = 0.0

# Setting all values to float
hdisease = hdisease.astype(float)
```

```python
hdisease.head()
```

      age
      sex
      cp
      trestbps
      chol
      fbs
      restecg
      thalach
      exang
      oldpeak
      slope
      ca
      thal
      disease

      0
      67.0
      1.0
      4.0
      160.0
      286.0
      0.0
      2.0
      108.0
      1.0
      1.5
      2.0
      3.0
      3.0
      1.0

      1
      67.0
      1.0
      4.0
      120.0
      229.0
      0.0
      2.0
      129.0
      1.0
      2.6
      2.0
      2.0
      7.0
      1.0

      2
      37.0
      1.0
      3.0
      130.0
      250.0
      0.0
      0.0
      187.0
      0.0
      3.5
      3.0
      0.0
      3.0
      0.0

      3
      41.0
      0.0
      2.0
      130.0
      204.0
      0.0
      2.0
      172.0
      0.0
      1.4
      1.0
      0.0
      3.0
      0.0

      4
      56.0
      1.0
      2.0
      120.0
      236.0
      0.0
      0.0
      178.0
      0.0
      0.8
      1.0
      0.0
      3.0
      0.0

### Correlation Heatmap

```python
plt.figure(figsize=(20, 8))
sns.heatmap(hdisease.corr(), annot=True, square=True, fmt='0.2f')
```

```python
_, axes = plt.subplots(2, 2, figsize=(12, 10))
sns.barplot(x='disease', y='exang', data=hdisease, ax=axes[0, 0])
sns.barplot(x='disease', y='cp', data=hdisease, ax=axes[0, 1])
sns.barplot(x='disease', y='thalach', data=hdisease, ax=axes[1, 0])
sns.barplot(x='disease', y='chol', data=hdisease, ax=axes[1, 1])
```

Most features are highly correlated to the disease column.
By interpreting the barplots of healthy and sick patients, it is likely that a patient with heart disease will have the following caracteristics:
- cp: more chest pain
- thalach: a lower maximum heart rate, surprisingly, high beats per minute is a signal of heart health
- exang: increase pain when exercising
- cholesterol levels are similar through sick and healthy patient: no direct link with heart disease

### Decision Trees
Decision trees are defined by recursively partitioning the input space into regions. Each region is partitioned into sub-regions up to a leaf node.
Space is sub-divided into non overlapping regions based on some criteria at each node. Once created, a tree can be navigated with a new sample of data following each branch according to the conditions. Each leaf node maps the input to a class or continuous value depending on the task.
To understand how to grow a tree, we must introduce the gini index first.
Assuming that $D$ is the data in a leaf and $N$ the number of samples in $D$, we estimate the class-conditional probability as follow:
$$
    \hat{\pi}_c = \frac{1}{N} \sum_{i \in D}\mathbb{I}(y_i = c)
$$
where $c$ are the classes of the target. The Gini index of a given leaf is:
$$
    G_{l} = \sum_{c=1}^{C}\hat{\pi}_c(1 -\hat{\pi}_c) = 1 - \sum_{c=1}^{C}\hat{\pi}^2_c
$$

To have the Gini index of a split, we compute the Gini index on each of its leaves and multiply each of them by the proportion of samples in each leaves and sum both values:

$$
    G_{Node} = \sum_{i=1}^{K} \frac{k_{li}}{U}G_{li}
$$
where $K$ is the number of leaves of the node, $k_{li}$ is the number of samples in the leaf $i^{th}$ leaf and $U$ is the number of sample to split.

In the case of categorical inputs, the most common approach is to consider the split condition as $x_{ij} = c_k$ or $x_{ij} \neq c_k$, for each possible class label $c_k$.
The rule to perform a split on a node is as follows:
- if the node is in the maximum depth of the tree, keep it as a leaf.
- for each attribute, we try all the possible thresholds and compute the Gini index of each split, the threshold across all attributes that minimize the Gini index is selected.
- a node is said pure if the split generates a empty leaf, if a node is pure, it is set as a leaf.
- if the node has a lowest score than the Gini index of the best split, we keep it as a leaf.

```python

# Check which data are categorical or not
is_categorical = []
for col in hdisease.columns:
    # If less than 9 unique values, than the feature is categorical
    is_categorical.append(int(len(np.unique(hdisease[col])) < 9))
# Convert to Pytorch tensor
is_categorical = torch.tensor(is_categorical)[:-1]
```

```python

# Convert data to numpy
data = hdisease.values
# Get unique values of the target
utarget = torch.from_numpy(np.unique(data[:, -1]))
```

### Convert data to Pytorch

```python
X = torch.from_numpy(data[:, :-1])
y = torch.from_numpy(data[:, -1]).int()
```

### Node class

```python
class Node():
    """Base class for a tree node

    We will use a single class to have the options to build a Decision Tree,
    a Random Forest and Adaboost.

    Attributes:
        X: A torch tensor for the data.
        y: A torch tensor for the labels.
        is_categorical_list: A list of boolean, 1 if the features is categorical,
            0 otherwise.
        max_depth: A positive integer for the depth of the tree.
        columns: A list of the name of each column.
        depth: A positive integer for the depth of the node.
        nb_features: A positive integer for selecting a number of random
            features for the split. Only used when training a Random Forest.
        weights: A torch tensor for computing the weighted gini index.
            Only used when training Adaboost.
        col_perm: A torch tensor of the permuted column. Only for Random Forests.
        X_full: A torch tensor that is keeping the whole data in case we shuffle
            the input for training a Random Forest.
        col_full: A list that keeps the whole feature names. Only for Random Forests.
        is_catfull: A list that keeps the whole is_categorical_list if training a
            Random Forest.
        size: A positive integer for the number of samples.
        nb_features: A positive integer for the number of features
        cutoff: A float for the splitting value of the node.
        col: A positive integer as an index for which column to apply the condition
            for the split.
        left: A Node class for the left child node.
        right: A Node class for the right child node.

    """
    def __init__(self, X, y, is_categorical, max_depth, columns, depth=0, nb_features=0, weights=None):
        """Init function of the class

        This function builds the whole tree, recursively creating each child node down
        to the leaves.

        Args:
            Described in the attribute section of the class
        """
        # If training random forest
        if nb_features != 0:
            self.col_perm = torch.randperm(X.shape[1])[:nb_features]
            # We have to keep the permutated data as well as the full data
            # because we have to pass the full data to the child nodes
            self.X = X[:, self.col_perm]
            self.X_full = X
            self.columns = columns[self.col_perm]
            self.col_full = columns
            self.is_categorical_list = is_categorical[self.col_perm]
            self.is_catfull = is_categorical
        else:
            # Regular training of the decision tree
            self.X = X
            self.columns = columns
            self.is_categorical_list = is_categorical
        # Weights are used to compute a weighted gini index
        # Only when training AdaBoost
        self.weights = weights
        self.y = y
        self.size = float(X.shape[0])
        # Prediction if the node will turn into a leaf, or split value
        self.cutoff = None
        # Column to check for splitting the data
        self.col = None
        # Child nodes
        self.left = None
        self.right = None
        # Whether or not the split value is categorical
        self.depth = depth
        self.nb_features = nb_features
        # If the node contains only one label, it is set as a leaf
        if self.is_pure() or depth == max_depth or self.X.shape[0] == 1:
            # Select the predominant categorical value and set it as the prediction
            self.make_leaf()
            return

        # Computing the gini index on the population before the split
        gini_n = self.gini(y, self.weights)
        params = self.find_best_split()

        gini_s = params[0]
        # If no improvement, make the node as a leaf
        if gini_s >= gini_n:
            self.make_leaf()
        else:
            self.make_split(params, max_depth)

    def gini(self, y, weights=None):
        """Computes the gini index

        Args:
            y: A torch tensor for the labels of the data.
            weights: If training a Random Forest, the weights associated
                with each sample.
        """
        if weights is None:
            # Regular gini index
            pi = y.bincount() / float(y.shape[0])
        else:
            # Weighted gini index
            pi = self.get_weighted_population(y, weights)
        return 1 - pi.pow(2).sum()

    # For weighted gini index:
    def get_weighted_population(self, y, weights):
        """Computes the weighted gini index

        Instead of counting the samples for each class
        we count the weights of each class and divide by the sum of the weights

        Args:
            y: A torch tensor for the labels of the data.
            weights: If training a Random Forest, the weights associated
        """
        pi = torch.zeros((2,), dtype=torch.float32)
        idx_0 = torch.where(y == 0)[0]
        idx_1 = torch.where(y == 1)[0]
        pi[0] = weights[idx_0].sum()
        pi[1] = weights[idx_1].sum()
        return pi / weights.sum()

    def clean_attributes(self):
        """Cleans variables that are not usefull to predict"""
        del(self.X)
        del(self.y)
        del(self.weights)
        del(self.size)
        if self.nb_features != 0:
            del(self.X_full)
            del(self.col_full)
            del(self.is_catfull)
            del(self.colperm)
        del(self.nb_features)

    def is_pure(self):
        """Checks if the node is pure

        The node is pure if there is only one label in the population
        """
        return len(self.y.unique()) == 1

    def get_label(self):
        """Returns the most present label as a prediction if the node turns to a leaf"""
        if self.weights is None:
            return self.y.bincount().argmax().item()
        else:
            return self.get_weighted_population(self.y, self.weights).argmax().item()

    def make_leaf(self):
        """Makes the node a leaf"""
        self.cutoff = self.get_label()
        self.clean_attributes()

    def make_split(self, params, max_depth):
        """Performs the split

        Args:
            params: See find_best_split function
            max_depht: A positive integer for the maximum of the tree.
        """
        self.col = params[1]
        self.cutoff = params[2]
        # Save the categorical boolean of the selected column for the predict method
        self.var_categorical = self.is_categorical_list[self.col].item()
        # Recursively split by creating two instances of the Node class using the two groups
        if self.nb_features != 0:
            cols = self.col_full
            categorical_list = self.is_catfull
        else:
            cols = self.columns
            categorical_list = self.is_categorical_list
        # Creating child nodes based on the best params
        # If training Random Forest, we pass nb_features
        # If training AdaBoost, we pass the weights
        self.left = Node(X=params[3][0],
                         y=params[3][1],
                         is_categorical=categorical_list,
                         max_depth=max_depth,
                         columns=cols,
                         depth=self.depth + 1,
                         nb_features=self.nb_features,
                         weights=params[5])
        self.right = Node(X=params[4][0],
                          y=params[4][1],
                          is_categorical=categorical_list,
                          max_depth=max_depth,
                          columns=cols,
                          depth=self.depth + 1,
                          nb_features=self.nb_features,
                          weights=params[6])
        self.clean_attributes()

    def gini_split(self, idx_g_1, idx_g_2, cutoff, feature_idx, best_params):
        """Computes the gini index of the future split

        Args:
            idx_g_1: A torch tensor for the indices of the first group.
            idx_g_2: A torch tensor for the indices of the second group.
            cutoff: A float for the split value.
            feature_idx: A positive integer for the index of the feature to test.
            best_params: See function find_best_split
        """
        g_1 = self.y[idx_g_1].squeeze(1)
        g_2 = self.y[idx_g_2].squeeze(1)
        if self.weights is  None:
            #Gini index
            gini_g1 = (float(g_1.shape[0]) / self.size) * self.gini(g_1)
            gini_g2 = (float(g_2.shape[0]) / self.size) * self.gini(g_2)
        else:
            # Weighted gini index
            g_1_w = self.weights[idx_g_1]
            g_2_w = self.weights[idx_g_2]
            w_sum = self.weights.sum()
            gini_g1 = (g_1_w.sum() / w_sum) * self.gini(g_1, g_1_w)
            gini_g2 = (g_2_w.sum() / w_sum) * self.gini(g_2, g_2_w)

        gini_split = (gini_g1 + gini_g2)
        if gini_split < best_params[0]:
            best_params[0] = gini_split.item()
            best_params[1] = feature_idx
            best_params[2] = cutoff.item()
            # If training a base learner of a random forest
            # pass the full data to child nodes
            if self.nb_features != 0:
                best_params[3] = [self.X_full[idx_g_1].squeeze(1), g_1]
                best_params[4] = [self.X_full[idx_g_2].squeeze(1), g_2]
            else:
                best_params[3] = [self.X[idx_g_1].squeeze(1), g_1]
                best_params[4] = [self.X[idx_g_2].squeeze(1), g_2]
            if self.weights is not None:
                # Gather weights of each groups to pass them to the child nodes
                # for their own weighted gini index
                best_params[5] = g_1_w
                best_params[6] = g_2_w
        return best_params

    def find_best_split(self):
        """Finds the best split

            Creates a parameter list to store the parameters of the best split
            It contains:
            0: best gini index
            1: column index of the best split
            2: value of the best split
            3: left group [X, y], less than equal to #3 or belongs to the class #3 if categorical
            4: right group [X, y], greater than #3 or does not belong to the class #3
            5 : weights of the first group
            6 : weights of the second group
        """
        best_params = [2, -1, -1, None, None, None, None]
        for i in range(self.X.shape[1]):
            vals = self.X[:, i]
            if self.is_categorical_list[i]:
                for cutoff in vals.unique():
                    idx_uv = (vals == cutoff).nonzero()
                    idx_uv_not = (vals != cutoff).nonzero()
                    best_params = self.gini_split(idx_uv, idx_uv_not, cutoff, i, best_params)
            else:
                for cutoff in vals.unique():
                    idx_leq = (vals <= cutoff).nonzero()
                    idx_ge = (vals > cutoff).nonzero()
                    best_params = self.gini_split(idx_leq, idx_ge, cutoff, i, best_params)
        return best_params

    def get_dict(self):
        """Returns a dictionary containing nodes and their information"""
        node_dict = {}
        if self.left is None and self.right is None:
            node_dict['pred'] = self.cutoff
        else:
            node_dict['cutoff'] = self.cutoff
            node_dict['feature'] = self.columns[self.col]
            node_dict['categorical'] = self.var_categorical
            node_dict['left'] = self.left.get_dict()
            node_dict['right'] = self.right.get_dict()
        return node_dict

    def predict(self, sample):
        """Takes a single input and predicts its class

            Follows the tree based on the conditions
        """
        if self.nb_features != 0:
            sample_in = sample[self.col_perm]
        else:
            sample_in = sample
        if self.left is None and self.right is None:
            return self.cutoff
        if self.var_categorical:
            if sample_in[self.col] == self.cutoff:
                return self.left.predict(sample)
            else:
                return self.right.predict(sample)
        else:
            if sample_in[self.col] <= self.cutoff:
                return self.left.predict(sample)
            else:
                return self.right.predict(sample)
```

### Decision Tree Class

```python
class DecisionTreeClassifier(BaseEstimator):
    """Class for the Decision Tree Classifier

    Class that implements the sklearn methods
    Just a wrapper for our node class

    Attributes:
        max_depth: A positive integer for the maximum depth of the tree.
        columns: A list of the name of each column.
        nb_features: A positive integer for the number of features
        is_categorical: A list of boolean, 1 if the features is categorical,
    """
    def __init__(self, max_depth, is_categorical, columns, nb_features=0):
        """Inits the Decision Tree class

        Args:
            See attributes section.
        """
        if nb_features < 0:
            raise ValueError('negative integer passed to nb_features.')
        self.max_depth = max_depth
        # Wether or not each column is a categorical value
        self.is_categorical = is_categorical
        self.columns = columns
        self.root = None
        # Number of random features to select
        # Only used when building a random forest base learner
        # If 0 then train a decision tree using all the features availables
        self.nb_features = nb_features

    def fit(self, X, y, **kwargs):
        """Trains the model

        Needs to get the 'sample_weight' key in kwargs.
        Mandatory for using sklearn cross validation.

        Args:
            X: A torch tensor for the data.
            y: A torch tensor for the labels.
        """
        if self.nb_features > X.shape[1]:
            raise ValueError('parameter np_features should be less than equal to the number of features')
        if 'sample_weight' in kwargs.keys():
            weights = kwargs['sample_weight']
        else:
            weights = None
        self.root = Node(X, y, self.is_categorical, self.max_depth, self.columns, 0, self.nb_features, weights)

    def predict(self, X):
        """Predicts the labels of a batch of input samples"""
        if len(X.shape) == 0:
            return 'error: can not predict on empty input.'
        if len(X.shape) == 1:
            return self.root.predict(X)

        pred = torch.zeros((X.shape[0],), dtype=torch.int32)
        if self.root == None:
            return 'error: use the fit method before using predict.'
        for i in range(X.shape[0]):
            sample = X[i, :]
            pred[i] = self.root.predict(sample)
        return pred

    def __str__(self):
        """Method for printing the tree"""
        if self.root == None:
            return 'error: use the fit method to print the result of the training.'
        tree_dict = self.root.get_dict()
        pprint(tree_dict)
        return ''
```

```python
dt = DecisionTreeClassifier(max_depth=3,
                            is_categorical=is_categorical,
                            columns=hdisease.columns,
                            nb_features=13)
dt.fit(X, y)
print(dt)
```

    {'categorical': 1,
     'cutoff': 3.0,
     'feature': 'thal',
     'left': {'categorical': 1,
              'cutoff': 0.0,
              'feature': 'ca',
              'left': {'categorical': 0,
                       'cutoff': 156.0,
                       'feature': 'trestbps',
                       'left': {'pred': 0},
                       'right': {'pred': 1}},
              'right': {'categorical': 1,
                        'cutoff': 4.0,
                        'feature': 'cp',
                        'left': {'pred': 1},
                        'right': {'pred': 0}}},
     'right': {'categorical': 1,
               'cutoff': 4.0,
               'feature': 'cp',
               'left': {'categorical': 0,
                        'cutoff': 0.5,
                        'feature': 'oldpeak',
                        'left': {'pred': 1},
                        'right': {'pred': 1}},
               'right': {'categorical': 1,
                         'cutoff': 0.0,
                         'feature': 'ca',
                         'left': {'pred': 0},
                         'right': {'pred': 1}}}}

We just fit the whole data to the model to visualize the structure of the tree.
In the next section, we will validate the performance of the model using cross validation and have an interpretation of the tree structure.

### Cross Validation

```python
dt = DecisionTreeClassifier(max_depth=3,
                            is_categorical=is_categorical,
                            columns=hdisease.columns,
                            nb_features=13)
# Using 5 K-folds
test_acc = cross_val_score(dt, X, y, scoring='accuracy', cv=5).mean()
print('Accuracy on the test set:', test_acc)
```

    Accuracy on the test set: 0.8110382513661202

Using cross validation and accuracy as our metric, the model performed well for such a simple classifier.
Indeed, 81% accuracy on a 5 k-fold cross validation is acceptable given the complexity of predicting heart disease.
Looking at the tree structure, the first split is done on the 'thal' feature which has the highest correlation with the target.
The most discriminative features are then 'ca', 'cp' and 'oldpeak' and they all have more than 40% correlation with the target.
The model uses lower correlated variable such as 'chol' and 'testbps' inside the deeper nodes.

The 'thal' feature contributes the most in the decision. It refers to the Thalium stress test result, consisting in injecting radioactive element into the bloodstream of the patient. The 'thal' categorical values refers to the quality of blood circulation. It makes sense that a patient with poor blood circulation is very likely to have a heart disease.

The performance using a decision tree is acceptable, but let's try to do better using a Random Forest.

### Random Forest
Random forest is a technique known as bootstrap aggregating (bagging) which consists in taking multiple different models and compute the ensemble prediction:

$$
    f(\boldsymbol{x}) = \sum_{i = 0}^{M}f_m(\boldsymbol{x})
$$

where $f_m$ is the $m^{th}$ model.
The random forest technique is an ensemble model using decision trees as base learners. This ensemble model tries to decorrelate the base
learners by learning trees on a randomly chosen subset of features, as well as a randomly chosen subset of samples.

In other words, to train a random forest, we create $m$ subsets of the training set where we can select multiple times the same samples. These subsets are known as boostrapped datasets. Finally, each base learner is trained in the same fashion as a decision tree except that at each node, when we perform the split, we randomly choose a fixed number of features from the data.

```python
class RandomForestClassifier(BaseEstimator):
    """Class for the Random Forest model

    Implements the BaseEstimator sklearn class to use cross validation.

    Attributes:
        max_depth: A positive integer for the maximum depth of the tree.
        columns: A list of the name of each column.
        nb_features: A positive integer for the number of features
        is_categorical: A list of boolean, 1 if the features is categorical,
        n_estimators: A positive integer for the number of estimators in the ensemble.
        trees: A list of DecisionTreeClassifiers class of the ensemble.
    """
    def __init__(self, max_depth, is_categorical, columns, nb_features, n_estimators):
        """Inits the Random Forest class

        Args:
            See attribute section.
        """
        self.max_depth = max_depth
        self.is_categorical = is_categorical
        self.columns = columns
        self.nb_features = nb_features
        # Number of trees in the ensemble model
        self.n_estimators = n_estimators
        self.trees = []

    def fit(self, X, y, **kwargs):
        """Trains the random forest"""
        if self.nb_features > X.shape[1]:
            raise ValueError('parameter np_features should be less than equal to the number of features')
        for i in range(self.n_estimators):
            # Create boostrapped subset
            X_bstrp, y_bstrp = self.get_boostrap_data(X, y)
            dt = DecisionTreeClassifier(self.max_depth, self.is_categorical, self.columns, self.nb_features)
            # Train on the boostrapped subst
            dt.fit(X_bstrp, y_bstrp)
            self.trees.append(dt)

    def get_boostrap_data(self, X, y):
        """Returns the data and labels randomly shuffled"""
        idx = torch.randint(0, X.shape[0] - 1, size=(X.shape[0],))
        return X[idx, :], y[idx]

    def predict_sample(self, X):
        """Predicts on one sample"""
        preds = np.array([0, 0])
        # Predict with each base learner and return the most voted class
        for tree in self.trees:
            preds[tree.predict(X)] += 1
        return preds.argmax()

    def predict(self, X):
        """Predicts on a batch of samples"""
        if len(X.shape) == 0:
            return 'error: can not predict on empty input.'
        if len(X.shape) == 1:
            return self.predict_sample(X)

        pred = torch.zeros((X.shape[0],), dtype=torch.int32)
        if len(self.trees) == 0:
            return 'error: use the fit method before using predict.'
        for i in range(X.shape[0]):
            sample = X[i, :]
            pred[i] = self.predict_sample(sample)
        return pred
```

```python
max_depth_params = [3, 4]
nb_features_params = [i for i in range(2, 10)]
for _ in range(20):
    max_depth = np.random.choice(max_depth_params)
    nb_features = np.random.choice(nb_features_params)
    rf = RandomForestClassifier(max_depth=max_depth,
                                is_categorical=is_categorical,
                                columns=hdisease.columns,
                                nb_features=nb_features,
                                n_estimators=100)
    # Using 5 K-folds
    test_acc = cross_val_score(rf, X, y, scoring='accuracy', cv=5).mean()
    print('Test accuracy:\t' + str(test_acc) + ', nb features:\t' + str(nb_features) + ', max depth:\t', str(max_depth))
```

    Test accuracy:	0.8342622950819673, nb features:	7, max depth:	 3
    Test accuracy:	0.8243169398907104, nb features:	6, max depth:	 4
    Test accuracy:	0.8242076502732241, nb features:	6, max depth:	 3
    Test accuracy:	0.8243715846994537, nb features:	8, max depth:	 3
    Test accuracy:	0.8408743169398907, nb features:	6, max depth:	 3
    Test accuracy:	0.8275956284153005, nb features:	8, max depth:	 3
    Test accuracy:	0.834153005464481, nb features:	8, max depth:	 3
    Test accuracy:	0.8243169398907104, nb features:	4, max depth:	 3
    Test accuracy:	0.830928961748634, nb features:	6, max depth:	 3
    Test accuracy:	0.8342076502732242, nb features:	2, max depth:	 3
    Test accuracy:	0.8340983606557376, nb features:	7, max depth:	 4
    Test accuracy:	0.8175409836065575, nb features:	8, max depth:	 4
    Test accuracy:	0.8375956284153006, nb features:	4, max depth:	 3
    Test accuracy:	0.8341530054644808, nb features:	9, max depth:	 3
    Test accuracy:	0.8308743169398907, nb features:	5, max depth:	 3
    Test accuracy:	0.8275956284153004, nb features:	7, max depth:	 4
    Test accuracy:	0.8243169398907104, nb features:	2, max depth:	 4
    Test accuracy:	0.8441530054644808, nb features:	4, max depth:	 4
    Test accuracy:	0.8375409836065575, nb features:	3, max depth:	 4
    Test accuracy:	0.8342076502732242, nb features:	7, max depth:	 3

After a quick parameter search, the best model achieves 84.4% accuracy. A great improvement of 3% comparing to a single decision tree.
To go further we will try a technique known as boosting.

### Boosting
Boosting is an algorithm using multiple weak learners. A weak learner is defined to be slightly better than random guessing. The most common weak learner is known as a  stamp. A stamp is an univariate tree with only one split. The purpose of boosting is to sequentially apply the weak classification algorithm to repeatedly modified versions of the data.

The predictions from the weak learners are combined through a weighted majority vote:

$$
   f(\boldsymbol{x}) = \text{sign}\left(\sum_{m=1}^{M}\alpha_m f_m(\boldsymbol{x})\right)
$$

where $f(\boldsymbol{x})$ is the prediction of the final model, $f_m(\boldsymbol{x})$ is the prediction of the weak learner $m$ and $\{\alpha_1, \dots, \alpha_M\}$ are the weights computed by the boosting algorithm. Their effect is to give higher influence to the more accurate
learners. This formula represent the prediction in case of a binary classification $\{-1, 1\}$

The data modification consists of applying weights $\{\boldsymbol{w_1}, \dots, \boldsymbol{w_N}\}$ to each training samples. Initially the weights are set to $\frac{1}{N}$ where $N$ is the number of samples. At iteration $k$, we perform a data transformation using the weights on each training samples. We fit the data into the model and the samples that were misclassified by the learner $f_{k-1}$ have their weights increased while the samples that were well classified by $f_{k-1}$ have their weights decreased. The importance of the harder samples sequentially increase and the learners are forced to concentrate on the observation that are missed on the previous iterations.

AdaBoost, short for Adaptative Boosting, is the first boosting technique that got popular for being able to adapt to the weak learners.
In this algorithm, after fitting the weak learner $m$, we compute the weighted error:

$$
    \text{err}_m = \frac{1}{N}\sum_{i=1}^{N}w_i I\left(y_i \neq f_m(\boldsymbol{x_i})\right)
$$

Then we set the weights of the model in the final prediction based on its performance:

$$
    \alpha_m = \log\left(\frac{1 - \text{err}_m}{\text{err}_m}\right)
$$

Finally, we update the weights of each samples as follow:

$$
    \begin{align}
    w_i &= w_i * \exp(\alpha_m I\left(y_i \neq f_m(\boldsymbol{x}_i)\right)\\
    w_i &= w_i * \exp(-\alpha_m I\left(y_i = f_m(\boldsymbol{x}_i)\right)
    \end{align}
$$

The importance of the weak learner's weight is proportional to its performance during training.
In fact, if $err_m$ is low then $\alpha_m$ is high and the values of the sample weights $w_1, \dots, w_N$ get modified by a higher margin.
The misclassified samples are scaled up and the well classified samples are scaled down so that the learner will focus on the weakness of the previous one.

Inside the stamps, the splits are selected according to the weighted gini split:

$$
    \begin{align}
    \hat{\pi}_c = \frac{1}{\sum_{i=1}^{N}w_i} \sum_{i \in D}w_i\mathbb{I}(y_i = c)\\
    \text{G} = 1 - \sum_{c=1}^{C}\hat{\pi}^2_c
    \end{align}
$$

Here the samples with high weights contribute more to the gini index than the lower ones.

```python
class AdaBoostClassifier(BaseEstimator):
    """Class of the Adaboost model

    Attributes:
        columns: A list of the name of each column.
        is_categorical: A list of boolean, 1 if the features is categorical,
        n_estimators: A positive integer for the number of estimators in the ensemble.
        learners: A list of DecisionTreeClassifiers for the stamps of the ensemble.
        alphas: A torch tensor for the weights defining the importance of each learner
            in the final prediction.
    """
    def __init__(self, n_estimators=50, is_categorical=None, columns=None):
        self.n_estimators = n_estimators
        self.learners = []
        self.alphas = None

    def compute_weighted_err(self, y_true, y_pred, weights):
        """Computes a weighted error based on the weight associated with the samples"""
        idx_neq = torch.where(y_true != y_pred)[0]
        return weights[idx_neq].sum() / weights.sum()

    def set_alpha(self, err):
        """Updates the new weights of the models"""
        return torch.log((1 - err) / (err + 1e-10)) / 2

    def update_weights(self, weights, y_true, y_pred, alpha):
        """Updates the weights of the samples

        Args:
            weights: A torch tensor for the weights of each samples.
            y_true: A torch tensor for the labels of the data.
            y_pred: A torch tensor for the predictions of the model.
            alpha: A float for the weight of a learner.
        """
        if len(y_true.shape) == 1:
            y_true = y_true.unsqueeze(1)
        if len(weights.shape) == 1:
            weights = weights.unsqueeze(1)
        weights *= torch.exp(alpha * (y_pred != y_true).int())
        weights *= torch.exp(-alpha * (y_pred == y_true).int())
        weights /= weights.sum()
        return weights

    def fit(self, X, y, **kwargs):
        """Trains the model

        Args:
            X: A torch tensor for the data.
            y: A torch tensor for the labels.
        """
        if len(y.shape) == 1:
            y = y.unsqueeze(1)

        size = X.shape[0]
        # Initialize weights to 1/size
        weights = torch.empty((size,)).fill_(1 / size)
        alphas = torch.zeros((self.n_estimators, 1))

        for i in range(self.n_estimators):
            if len(weights.shape) > 1:
                weights = weights.squeeze(1)

            # Each learner is a stamp:
            # it has only one split and is using only one feature
            dtc = DecisionTreeClassifier(max_depth=1, is_categorical=is_categorical, columns=columns, nb_features=0)
            dtc.fit(X, y, sample_weight=weights)
            pred = dtc.predict(X).unsqueeze(1)
            self.learners.append(dtc)

            # Compute weighted error
            err = self.compute_weighted_err(y, pred, weights)
            # Set weight of the learner
            new_alpha = self.set_alpha(err)
            alphas[i] = new_alpha
            # Update weights
            weights = self.update_weights(weights, y, pred, new_alpha)
        self.alphas = alphas

    def predict(self, X):
        """Predicts the labels of a batch of data"""
        size = X.shape[0]
        preds = torch.zeros((size, 1), dtype=torch.float32)
        for i in range(self.n_estimators):
            pred = self.learners[i].predict(X).unsqueeze(1)
            # Change zeros to -ones to apply to sign formula discussed above
            pred[pred == 0] = -1
            preds += self.alphas[i] * pred
        preds = torch.sign(preds)
        preds[preds == -1] = 0
        return preds
```

```python
for i in range(10, 150, 10):
    adaboost = AdaBoostClassifier(n_estimators=i, is_categorical=is_categorical, columns=hdisease.columns)
    fit_params = {
        'n_estimators' : i,
        'is_categorical' : is_categorical,
        'columns' : hdisease.columns
    }
    test_acc = cross_val_score(adaboost, X, y, scoring='accuracy', cv=5).mean()
    print('Test accuracry: ' + str(test_acc) + ', n_estimators: ' + str(i))
```

    Test accuracry: 0.8374316939890709, n_estimators: 10
    Test accuracry: 0.8340437158469944, n_estimators: 20
    Test accuracry: 0.837377049180328, n_estimators: 30
    Test accuracry: 0.8274863387978142, n_estimators: 40
    Test accuracry: 0.8440437158469946, n_estimators: 50
    Test accuracry: 0.827431693989071, n_estimators: 60
    Test accuracry: 0.8341530054644808, n_estimators: 70
    Test accuracry: 0.8275409836065574, n_estimators: 80
    Test accuracry: 0.8308743169398907, n_estimators: 90
    Test accuracry: 0.8308196721311475, n_estimators: 100
    Test accuracry: 0.8274863387978142, n_estimators: 110
    Test accuracry: 0.8175956284153004, n_estimators: 120
    Test accuracry: 0.8142622950819671, n_estimators: 130
    Test accuracry: 0.8274863387978142, n_estimators: 140

AdaBoost achieved 84.4% accuracy on this data, the exact same performance as the random forest.

### Conclusion
Decision Trees are very prized by the machine learning community for their interpretability and their handy automatic feature selection.
Unfortunately, the decision trees tend to overfit. It is mandatory to limit the max depth parameter to avoid this behavior. Using the ensemble technique is a great way to deal with the tendency of the decision trees to overfit. However, as the model is getting more complex due to aggregation of multiple learners, it is more challenging to interpret a Random Forest or an AdaBoost model.