# Resolução da DFT e adição de zeros

Neste notebook avaliaremos aspectos de resolução da DFT e da adição de zeros (zero padding).

Notamos que um sinal com $N$ amostras, amostrado a uma taxa $F_s$ {Hz] possuirá um vetor de frequências

\begin{equation}
f_{Hz}^{(k)} = \left\{\frac{0F_{s}}{N}, \ \frac{1F_{s}}{N}, \  \frac{2F_{s}}{N}, \ ... \ \frac{(N-1)F_{s}}{N}\right\}
\end{equation}

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
import IPython.display as ipd # to play signals
import sounddevice as sd
import soundfile as sf
```

## Vamos criar uma resposta ao impulso ideal

Esta é a resposta ao impulso de um sistema massa-mola (1DOF)

\begin{equation}
h(t) = \frac{A}{\omega_d} \mathrm{e}^{-\zeta \omega_n t} \mathrm{sin}(\omega_d t)
\end{equation}
com $\zeta$ sendo o amortecimento, $\omega_n = 2 \pi f_1$ e $\omega_d = \omega_n\sqrt{1-\zeta^2}$, que possui TF, dada por

\begin{equation}
H(\mathrm{j}\omega) = \frac{A}{\omega_n^2 - \omega^2+ \mathrm{j}2\zeta\omega_n\omega}
\end{equation}

```python
# Define the sampling rate fs = 100 Hz and total record time T = 5 seconds
Fs=100
T=5
# time vector
time = np.arange(0, T, 1/Fs) # Ts = 0.01 [s]

# Sinal puro (Fenômeno) - original - h(t) de um S1GL (massa-mola)
A=200
zeta=0.3
wn=2*np.pi*10
wd=np.sqrt(1-zeta**2)*wn
# Impulse response
ht=(A/wd)*np.exp(-zeta*wn*time)*np.sin(wd*time)
# FRF teorica
freq = np.arange(0, Fs, 0.1)

Hw=A/(wn**2 - (2*np.pi*freq)**2+ 1j*2*zeta*wn*(2*np.pi*freq));

plt.figure(figsize=(12,4))
plt.subplot(1,2,1)
plt.title(r'$h(t)$ - (fenômeno original)')
plt.plot(time, ht, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, time[-1]))

plt.subplot(1,2,2)
plt.title(r'$|H(j\omega)|$ - (fenômeno original)')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs))
plt.ylim((-80, -20))
plt.tight_layout()
plt.show()
```

## Adicionamos ruído a resposta ao impulso (fenômeno ideal)

Tipicamente, é isso que acontece em uma medição. Ela sempre estará contaminada por ruído em algum grau. O ruído vai ter característica de ruído branco (distribuição normal com média zero e uma variância).

```python
noise = np.random.normal(loc = 0, scale = 0.01, size = len(time))
ht_med = ht+noise
```

```python
plt.figure()
plt.title(r'$h(t)$ - (fenômeno original vs. medido)')
plt.plot(time, ht, '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(time, ht_med, '-r', linewidth = 1, label = 'Fenômeno medido')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, time[-1]));
```

## Agora podemos investigar a FFT e sua resolução para alguns casos

### 1. Resolução original

"If n is not given, the length of the input along the axis specified by axis is used."

Se n não for dado, to comprimento da entrada (ao longo do eixo especificado) é usado - Em outras palavras, o espectro tem o mesmo número de pontos que o sinal no domínio do tempo.

```python
N = len(ht) # Número de pontos da FFT
Hw_med = np.fft.fft(ht_med)
freq_vec = np.linspace(0, (N-1)*Fs/N, N)
print("Vetor de frequência: {:.2f}, {:.2f}, {:.2f}, ... {:.2f}".format(freq_vec[0], freq_vec[1], freq_vec[2], freq_vec[-1]))
print("Resolução do espectro: {:.2f} [Hz]".format(freq_vec[1]-freq_vec[0]))

plt.figure(figsize=(10,6))
plt.title(r'$|H(j\omega)|$')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(freq_vec, 20*np.log10(np.abs(Hw_med)/Fs), '-r', linewidth = 1, label = 'Fenômeno medido')
plt.legend(loc = 'lower left')
plt.axvline(Fs/2, color='grey',linestyle = '--', linewidth = 4, alpha = 0.4)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs))
plt.ylim((-80, -20))
plt.tight_layout()
#plt.savefig('dft_mag_par.pdf')
plt.show()
```

## Cheque o teorema de Parseval

```python
# Domínio do tempo
Et = np.sum(ht_med**2)
# Domínio da frequência
Ef = np.sum(np.abs(Hw_med)**2)

print("A energia no domínio do tempo é {:.2f}".format(Et))
print("A energia no domínio da frequência é {:.2f}".format(Ef/N))
```

    A energia no domínio do tempo é 13.55
    A energia no domínio da frequência é 13.55

### 2. Sinal truncado

Agora vamos truncar $h(t)$. Note que o ruído tende a dominar a resposta ao impulso a partir de $0.4$ [s]

"If n is smaller than the length of the input, the input is cropped."

Se n for menor que que o comprimento do sinal, o sinal será truncado. É indiferente fazer:

a. Truncar o sinal no tempo:
- ht_med_trunc = ht_med[time<=0.4]
- Hw_med_trunc = np.fft.fft(ht_med_trunc)

b. Passar Nt como o segundo argumento da FFT
- Hw_med_trunc = np.fft.fft(ht_med, Nt)

```python
ht_med_trunc = ht_med[time<=0.4]

Nt = len(time[time<=0.4]) # Número de pontos da FFT
# Hw_med_trunc = np.fft.fft(ht_med_trunc) # Tente esta linha depois
Hw_med_trunc = np.fft.fft(ht_med, Nt)
freq_vec_t = np.linspace(0, (Nt-1)*Fs/Nt, Nt)
print("O número de pontos na DFT é {}".format(Nt))
print("Vetor de frequência: {:.2f}, {:.2f}, {:.2f}, ... {:.2f}".format(freq_vec_t[0], freq_vec_t[1],
    freq_vec_t[2], freq_vec_t[-1]))
print("Resolução do espectro: {:.2f} [Hz]".format(freq_vec_t[1]-freq_vec_t[0]))

plt.figure(figsize=(10,6))
plt.title(r'$|H(j\omega)|$ - (fenômeno original)')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(freq_vec_t, 20*np.log10(np.abs(Hw_med_trunc)/Fs), '-r', linewidth = 1, label = 'Fenômeno medido truncado')
plt.legend(loc = 'lower left')
plt.axvline(Fs/2, color='grey',linestyle = '--', linewidth = 4, alpha = 0.4)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs/2)) # Extenda até Fs se desejar
plt.ylim((-80, -20))
plt.tight_layout()
plt.show()
```

### 3. Sinal truncado e com zeros no final.

Agora vamos truncar $h(t)$ e adicionar zeros no final. Note que a resolução do espectro no caso anterior diminuiu (maior $\Delta f$).

"If n is larger, the input is padded with zeros."

Se n for maior que que o comprimento do sinal, o sinal será completado com zeros. É indiferente fazer:

a. Truncar o sinal no tempo:
- ht_med_trunc = ht_med[time<=0.4]
- Nt = len(time[time<=0.4])
- ht_med_zp = np.concatenate((ht_med_trunc, np.zeros(N-Nt)))
- Hw_med_zp = np.fft.fft(ht_med_zp) # Tente esta linha depois

b. Passar Nt como o segundo argumento da FFT
- Hw_med_zp = np.fft.fft(ht_med_trunc, N)

```python
ht_med_trunc = ht_med[time<=0.4]
Nt = len(time[time<=0.4])

ht_med_zp = np.concatenate((ht_med_trunc, np.zeros(N-Nt)))

#Hw_med_zp = np.fft.fft(ht_med_zp) # Tente esta linha depois
Hw_med_zp = np.fft.fft(ht_med_trunc, N)

freq_vec_zp = np.linspace(0, (N-1)*Fs/N, N)
print("O número de pontos na DFT é {}".format(len(ht_med_zp)))
print("Vetor de frequência: {:.2f}, {:.2f}, {:.2f}, ... {:.2f}".format(freq_vec_zp[0], freq_vec_zp[1],
    freq_vec_zp[2], freq_vec_zp[-1]))
print("Resolução do espectro: {:.2f} [Hz]".format(freq_vec_zp[1]-freq_vec_zp[0]))

plt.figure(figsize=(10,6))
plt.title(r'$|H(j\omega)|$ - (fenômeno original)')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(freq_vec_zp, 20*np.log10(np.abs(Hw_med_zp)/Fs), '-r', linewidth = 1, label = 'Fenômeno med. trunc. comp. zeros')
plt.legend(loc = 'lower left')
plt.axvline(Fs/2, color='grey',linestyle = '--', linewidth = 4, alpha = 0.4)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs/2)) # Extenda até Fs se desejar
plt.ylim((-80, -20))
plt.tight_layout()
plt.show()
```

## Trabalhando na IFFT

Vamos tratar rapidamente da transformada inversa e recuperar $h(t)$, a partir do espectro do sinal truncado e completado com zeros.

```python
ht_rec = np.fft.ifft(Hw_med_zp) # ifft
N_rec = len(ht_rec) # Número de amostras
t_mrec = (N_rec-1)/Fs # tempo máximo da ifft
time_rec = np.linspace(0, t_mrec, N_rec) # vetor temporal recuperado da ifft

plt.figure()
plt.title(r'$h(t)$ - (fenômeno original vs. IFFT do medido)')
plt.plot(time, ht, '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(time_rec, ht_rec, '-r', linewidth = 1, label = 'Fenômeno med. trunc. comp. zeros')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, time[-1]));
```

### Inspecione ht_rec

Você vai notar que:

1. O sinal recuperado da IFFT tem uma pequena parte imaginária
2. Você precisa do espectro de $0$ a $F_s$ pra operar a IFFT corretamente.

```python
ht_rec
```

    array([-1.31132980e-02-1.17683641e-17j,  1.56959697e+00+1.99285033e-17j,
            2.13357231e+00-5.46229728e-17j,  1.85113899e+00-4.88498131e-18j,
            1.06820611e+00-4.61120258e-17j,  1.98120119e-01-6.11798383e-18j,
           -4.78505416e-01-1.01804867e-17j, -7.91131898e-01-6.08427302e-18j,
           -7.24098739e-01-1.25559123e-17j, -4.93910808e-01+1.72125087e-17j,
           -1.32704465e-01-2.86079657e-17j,  1.35637166e-01+1.20231820e-17j,
            2.76690411e-01+1.97785491e-17j,  2.95206460e-01+2.44084806e-17j,
            2.02346788e-01-4.89000670e-18j,  7.19680486e-02+1.40362100e-17j,
           -2.26187658e-02+3.60721981e-17j, -9.15255500e-02+3.29928923e-18j,
           -1.17728629e-01-1.42474272e-17j, -8.90958560e-02+3.10673846e-17j,
           -6.63669849e-02-1.38476497e-17j,  1.08737487e-02-3.44088395e-17j,
            4.10848395e-02+6.28770210e-17j,  1.67814858e-02-8.17560688e-18j,
            2.27334309e-02-5.36159096e-17j,  1.33162996e-02+1.80890186e-17j,
            3.74385851e-04-2.27768346e-17j, -2.15409295e-02+3.53391134e-17j,
           -1.72967383e-02+5.21207735e-17j, -1.13996031e-02+5.29818087e-17j,
           -1.35199385e-02-1.39161500e-17j,  1.50149986e-03+5.21023465e-18j,
            9.12087023e-03+8.87078273e-18j,  7.95219739e-04+5.94773069e-18j,
            9.04785179e-03-2.94302101e-17j,  3.30061332e-03+5.12852942e-18j,
           -3.53646761e-03+4.37775093e-18j,  1.09633824e-02-1.38374022e-17j,
           -1.60729782e-04-2.27345293e-17j,  6.31880965e-03-6.46046418e-18j,
           -5.61484240e-03-6.46272429e-18j, -7.56516003e-18-1.83749122e-17j,
            4.08553566e-17-3.68559056e-17j, -1.82583428e-17+6.24486024e-18j,
            3.35206595e-17+8.40763483e-18j, -1.88587252e-17-2.89926379e-18j,
           -5.79546276e-17-2.98434143e-17j, -1.08190408e-17+4.96960086e-18j,
           -1.25896501e-17-6.11476647e-18j, -1.49106554e-18-9.92818376e-18j,
            2.88161769e-17+7.96589131e-18j,  4.07791942e-18+3.56987630e-17j,
            1.01910213e-16-5.53172811e-17j, -4.40581505e-17-4.39485653e-17j,
            3.55951258e-17+3.04973817e-17j,  1.87163541e-17+2.95058533e-18j,
           -9.65959177e-18+2.83841105e-17j, -9.31656225e-18-3.45002176e-17j,
           -1.83245071e-18-9.51735005e-18j, -4.27776378e-17-4.88094961e-18j,
           -2.22613471e-17-2.09692180e-17j, -3.38095603e-17+1.08559595e-17j,
           -7.46894396e-18-4.44822306e-17j, -3.52029539e-17-2.17423740e-17j,
            1.66221545e-17+5.93896478e-17j, -1.96727285e-17-1.76348465e-17j,
            7.33143811e-18+2.22431124e-17j, -5.33127515e-18-9.98554725e-18j,
           -3.01124828e-17+2.40831593e-18j, -9.32144189e-18+4.61400012e-17j,
           -1.09809125e-17+2.85650948e-17j, -1.13707199e-17+2.02535640e-17j,
            2.23989949e-17+4.52872920e-18j,  1.37197862e-17-7.99815414e-18j,
           -6.29709303e-18-4.90238756e-18j, -1.66796250e-17+7.07040514e-18j,
            5.96560982e-17+8.09962525e-18j,  4.99764017e-18-4.72525360e-17j,
           -1.45794778e-17+2.85225747e-17j,  3.06316147e-17-4.32257231e-17j,
            3.09545678e-18-4.65858864e-18j, -2.35056896e-17-4.10247489e-17j,
           -1.07140814e-17-9.84314478e-17j, -7.75068813e-17-7.43542699e-18j,
           -2.30513252e-17+1.09051638e-17j,  4.59827561e-18+3.22289065e-17j,
            1.65299928e-17-3.17367709e-17j,  1.11714071e-17+5.58277295e-18j,
            3.36504771e-17+1.41275231e-17j, -4.67123227e-18-2.69097723e-17j,
           -2.37003436e-17+5.21243902e-17j, -1.84093460e-17-1.46042687e-17j,
           -5.92017233e-18+1.56967264e-17j,  5.24886940e-19+2.22789245e-17j,
           -1.63699644e-17-2.21584655e-17j,  1.83087651e-17+9.58540129e-18j,
            4.91371792e-18-3.33508873e-17j,  8.31986121e-18-1.22609639e-17j,
            1.75778094e-17-2.40311292e-17j,  1.20591088e-17+1.54980838e-17j,
           -1.99972739e-17-1.20733909e-17j, -2.18369662e-16-8.57117196e-17j,
            1.08290384e-16-6.16137416e-17j,  9.52584472e-18+5.77512487e-17j,
           -8.19437290e-17+4.08077208e-17j, -1.00070411e-17+6.54863410e-18j,
            2.36766897e-18+8.70145305e-18j, -5.59903765e-17+8.44475160e-18j,
            8.75795994e-18-1.27988814e-17j,  3.06844216e-18-4.56147158e-18j,
           -9.72973431e-18-6.00300911e-18j,  1.90572750e-17-1.24709948e-17j,
            1.37168287e-17-1.66605566e-17j, -1.88070300e-18-5.03402739e-17j,
            3.80153376e-17-3.89407015e-17j, -3.17009072e-18-3.92600229e-17j,
            7.89081266e-18-1.02893513e-17j, -4.99378010e-17+3.50663243e-18j,
           -1.90060405e-17+4.04414637e-18j, -2.73745707e-17+6.25516616e-18j,
            7.52579198e-18+9.95826180e-18j, -2.52803035e-17-2.21023195e-17j,
           -4.22220589e-18-9.70366092e-18j,  3.01857127e-17+8.35608318e-18j,
            3.03744870e-18-7.95950177e-18j,  5.92779329e-18+2.01899303e-18j,
            4.11166147e-17+1.11050497e-17j, -5.78508559e-17-2.99379529e-17j,
           -3.52498976e-17-8.45401630e-18j, -1.47233890e-17+9.34243931e-18j,
           -1.79090412e-17+9.95975492e-18j, -1.09233966e-17+5.06592966e-18j,
            1.83524026e-17-4.28598853e-17j, -2.77875684e-17+2.81916401e-17j,
           -2.43203834e-17-1.24549078e-17j, -7.28540544e-19+1.87953999e-17j,
            2.98344799e-19-3.39592658e-18j,  1.10901245e-18+1.33090396e-17j,
            4.19794338e-17-3.25933304e-17j, -6.08319649e-18-6.60761441e-18j,
            1.26356141e-18+6.02444484e-19j, -6.76415288e-18+8.89853803e-18j,
           -2.99491088e-17+3.54559026e-17j, -1.98909367e-17-3.96176439e-17j,
            4.01292543e-17+1.84573975e-17j, -1.53610221e-17+4.54968592e-18j,
           -4.13836671e-18-1.03590628e-17j,  4.36088383e-17-7.75545022e-18j,
            9.09515317e-18+1.68049196e-17j, -1.44715993e-17+2.02351159e-17j,
            3.64686331e-18-7.28433695e-18j,  9.75664525e-18-5.32046241e-18j,
           -7.34947937e-17-4.45205233e-17j,  3.15576667e-17-4.51644084e-17j,
           -2.73647413e-17-4.26116131e-18j,  6.74035159e-18+1.14325244e-17j,
            5.98649665e-18-1.06425396e-17j, -2.22824412e-17-4.15310723e-17j,
            1.09182665e-17+4.47905826e-18j,  1.17566970e-17-3.99659543e-17j,
            7.04831945e-18-1.21643967e-17j,  2.77258131e-17-2.59172584e-17j,
           -7.01922235e-17-2.73256121e-18j,  4.67689708e-18-2.30327684e-17j,
            2.57802156e-17+7.61563030e-20j, -3.35458338e-18-7.58356790e-18j,
           -1.25535437e-17+2.00518790e-17j, -1.49217230e-17+1.78015531e-17j,
           -3.20693097e-18+1.56085053e-17j,  3.67069045e-17+2.25644929e-17j,
            3.86069267e-17+1.15646196e-17j, -1.30801627e-17+4.53695540e-17j,
            5.00692134e-17-3.24987795e-19j, -5.03024294e-19-9.24086367e-18j,
            6.20505503e-18-5.41326602e-19j, -2.76364249e-17-5.47995529e-18j,
            4.96336408e-17-7.76648797e-18j, -9.86172900e-17+5.85609492e-17j,
           -4.17193862e-17-3.31837657e-17j,  1.44452463e-17-5.48509588e-18j,
           -3.72549728e-18-1.82543981e-17j,  6.27928341e-17-2.55755885e-17j,
            3.20874193e-17+7.36500057e-17j,  2.52935121e-17-8.03977189e-18j,
            4.91019261e-18-4.43265858e-17j, -2.09126778e-17+3.10250812e-17j,
            5.63706277e-17+5.32110824e-18j,  3.17824052e-17+5.84347817e-17j,
            2.51873732e-17-5.05608236e-17j,  2.40247056e-17-2.81780169e-17j,
           -4.02079849e-18+6.45395020e-18j, -6.21365133e-18-1.87955679e-17j,
           -1.35704087e-17+1.28761546e-17j,  2.29054738e-17+2.46478862e-18j,
            1.49251861e-17+2.23085057e-17j,  5.14575343e-18-2.02522850e-17j,
           -2.81853907e-17-1.38255223e-17j, -2.01367755e-17+1.59685756e-17j,
           -3.51462302e-18-2.68710735e-17j, -2.62135925e-17+2.24695640e-17j,
            5.97089671e-18-1.32696656e-17j,  9.26539106e-17+8.74904443e-19j,
            1.29029367e-16-2.40829288e-17j,  2.53752724e-17+3.66051613e-17j,
            3.97580888e-17-5.70919498e-17j,  1.23917477e-17+1.22567083e-18j,
           -5.68825284e-17-1.22979168e-17j,  1.33921553e-17-2.71573399e-17j,
            4.24535935e-18-2.39702851e-17j,  3.45824135e-17+3.32057436e-18j,
           -1.18415180e-18+5.13004099e-18j, -1.13923550e-17+4.55617121e-18j,
            4.55466286e-18+1.29435046e-17j, -2.17660503e-17+6.84981965e-18j,
            4.53671222e-17-1.47989903e-18j,  1.09015174e-17+8.47775996e-19j,
            1.17574782e-17+3.65082980e-18j, -6.23647779e-18-5.98555367e-18j,
           -1.83179216e-17-7.79340769e-18j,  4.45349748e-17+8.08222022e-18j,
            1.51553356e-18-7.47704258e-18j, -4.60651340e-18-1.10989316e-17j,
           -4.14491811e-17+2.50221193e-17j, -1.26879701e-18-4.62634677e-17j,
            8.01024465e-18-4.15478267e-17j,  3.04395933e-18-2.51322021e-17j,
           -1.28801613e-17-3.86753957e-17j, -1.56790721e-17+1.91072614e-17j,
           -1.77447636e-17-4.66490752e-17j,  3.13408429e-17+5.79168916e-17j,
            2.59206521e-18+1.32327036e-17j,  7.51450469e-18+1.32970353e-17j,
            2.89923401e-17+6.91980966e-18j,  2.50145262e-17+3.12648746e-18j,
           -1.83360239e-18-2.05429361e-17j, -9.81625049e-18+5.93699472e-18j,
           -1.96432926e-17-1.14707407e-18j, -1.47976809e-17-5.12079245e-17j,
            9.51015754e-18-3.55180991e-19j,  4.47281081e-17+1.45013024e-18j,
           -1.16521709e-17+1.00951724e-18j, -3.77468911e-18+3.92584811e-17j,
            1.05349861e-17+2.98322952e-17j, -1.66639183e-17-5.36941548e-18j,
           -1.21763710e-17+2.69308108e-17j, -5.84339769e-18-8.03691332e-18j,
            1.43555245e-17+9.80634955e-18j,  1.02339942e-17-7.86472792e-18j,
           -4.94262175e-17-2.42096699e-17j,  9.32031704e-18-2.11875910e-17j,
            2.00306255e-18-1.57505072e-18j,  1.24548212e-17+4.39758835e-18j,
            8.74301590e-17+3.00308221e-18j,  1.31221087e-16+2.82535695e-17j,
           -3.53455816e-17+1.77473227e-18j,  2.89253748e-17+6.55685736e-18j,
           -2.01172641e-17-1.21408742e-17j,  8.97663963e-18-1.97818596e-17j,
            2.83460366e-17-2.86788472e-17j,  3.56104117e-17-7.50670223e-18j,
            3.16561966e-18-2.76152516e-17j, -1.83956972e-17-2.45992918e-17j,
            6.20449599e-18-3.86434472e-17j, -1.47826461e-17-8.72679158e-19j,
           -4.23394002e-17-2.96435947e-18j, -4.56899795e-19+1.39823307e-17j,
            1.80428589e-17-4.16168704e-18j, -8.12567344e-18+5.49111827e-17j,
           -1.44180421e-17-1.96132806e-17j, -1.21022877e-17+2.09510242e-17j,
           -1.16757158e-18-1.86661396e-17j, -8.88920602e-18+2.38425937e-17j,
           -5.51021469e-19-7.06977952e-18j, -2.90685340e-17-1.78213934e-18j,
           -1.89170645e-17+1.72624756e-17j,  1.62301976e-17-1.20549703e-17j,
           -1.38602358e-17+8.39941261e-18j, -3.94374256e-18+3.77981394e-17j,
            9.69857083e-18+7.61852948e-17j,  2.31585578e-17-1.54352064e-18j,
            5.25475525e-18+2.13549487e-18j,  1.54341297e-17+3.88840945e-17j,
            9.64247194e-21-4.08827175e-17j, -7.14026557e-18+1.21478150e-17j,
            2.22456227e-17+2.80014702e-17j, -3.37059077e-17+1.20631371e-17j,
           -1.38414852e-17+8.87208595e-18j,  9.20441644e-17-1.21531569e-17j,
           -1.57212542e-17+2.97165157e-17j, -3.10421465e-18-5.75767416e-17j,
            2.64052377e-17+1.44464697e-17j, -8.33221481e-18-1.96887711e-17j,
           -3.07405918e-17+1.15108313e-17j,  4.32174223e-17+3.44821646e-17j,
            4.97594337e-18+6.32020145e-18j,  2.76658655e-17+2.16680756e-17j,
           -1.07749466e-17+2.32442754e-17j, -2.41373068e-17+1.74510960e-17j,
           -3.69848475e-17+2.12112298e-17j, -9.43857962e-18-3.94862139e-17j,
           -5.52680750e-18+1.51679583e-17j,  7.78763459e-17-4.04242568e-17j,
            9.83443081e-17+4.27405178e-17j,  8.83115653e-17-3.02182853e-17j,
            1.70853301e-17+5.10306580e-17j, -2.66026024e-17+6.38806598e-18j,
           -1.41717452e-17+1.47327352e-17j,  7.18729730e-17+6.25438337e-18j,
           -4.24535935e-18-2.39702851e-17j, -3.45824135e-17+3.32057436e-18j,
            1.18415180e-18-1.25617504e-17j, -2.81849971e-18+2.74103578e-17j,
           -1.87655176e-17-1.72776831e-17j, -6.65565914e-18+8.06849584e-18j,
           -2.73455801e-18+6.87303180e-18j,  1.04147647e-17+1.45711409e-18j,
            1.57760528e-17-5.93451320e-18j, -2.57379453e-17+1.26921675e-17j,
            4.31869173e-17-1.29558028e-17j, -5.45512430e-18+9.06814996e-18j,
           -8.62096092e-18-1.51701351e-18j,  2.81432415e-17+1.11321808e-17j,
            5.74363927e-17-3.98569398e-17j,  5.70968911e-18+1.92471548e-17j,
           -4.45753097e-18+5.03481483e-17j,  1.87164120e-17+1.15239988e-17j,
           -2.66296108e-18-4.45039479e-18j, -2.51771352e-17-6.44717441e-17j,
            3.19556184e-17+2.03244444e-18j,  4.18629389e-18-1.40578375e-17j,
           -2.14797600e-18-1.55746440e-17j,  7.08492809e-18-4.14030280e-17j,
            5.20252908e-18+8.32678167e-18j,  2.43348872e-17-1.66300384e-17j,
            2.84789550e-17-5.30910107e-18j,  1.62555440e-17-2.29946553e-17j,
           -1.54397549e-17+1.10849264e-17j,  7.24816437e-18-7.71789334e-18j,
            1.29857365e-17+1.56074088e-18j, -3.07392980e-17+1.67310392e-17j,
            8.54354640e-18+9.85541293e-18j, -7.41893788e-19+2.03585585e-17j,
           -7.05990343e-17+3.15005437e-17j, -1.43276807e-17+1.49552952e-18j,
            6.42762326e-17+6.06415849e-18j, -1.16793124e-17+8.57360025e-18j,
            1.21250300e-17-1.76804713e-18j, -1.81127180e-17+1.04309961e-18j,
           -7.75572916e-18+3.53363116e-17j,  2.53028126e-17+2.30051386e-17j,
           -6.78064484e-18+1.21931074e-17j,  4.97533525e-18-1.46538738e-17j,
           -1.60734600e-17+4.95478962e-17j, -1.40681156e-17+3.23684158e-17j,
            1.92182118e-17+2.86034985e-18j,  2.77070934e-18+1.72319540e-17j,
           -1.05221424e-18+1.34635374e-17j,  3.31686217e-17-9.10400876e-18j,
            1.02412892e-17-5.15659630e-17j,  3.97623586e-17+8.93882920e-18j,
           -1.88369592e-17+1.92218328e-17j, -4.29352887e-17-7.71110859e-18j,
           -4.88998162e-19-3.34554876e-17j, -7.31052569e-18+6.00776329e-17j,
            1.13470453e-17-3.14184665e-18j,  5.23045235e-19-6.97556079e-18j,
            2.99374125e-17-1.79233425e-17j, -1.34079266e-17+5.15857099e-18j,
           -5.72139211e-18-1.90144861e-17j,  3.52605626e-18-1.16882159e-17j,
           -1.15508520e-18+1.94160230e-17j, -1.87952925e-17-1.10132098e-17j,
            2.41607871e-17+1.02676482e-17j, -8.06435921e-18+2.55258775e-17j,
           -4.00723210e-17-1.05247394e-17j, -1.08580332e-17-1.70303576e-17j,
            6.02579537e-17-4.61838597e-17j, -5.38736425e-17+6.76038650e-17j,
            4.66103203e-17-3.23480559e-17j,  2.38586751e-21-3.46767925e-17j,
           -2.55542130e-17+1.74263460e-17j, -6.92912687e-19+1.70250995e-17j,
           -2.24125018e-18-2.11237966e-18j, -5.09090965e-17+1.06285546e-17j,
           -7.73921559e-19+2.04694285e-17j, -1.58995509e-17+3.14930325e-17j,
           -2.45834425e-17+8.47235432e-18j, -7.56794469e-18-1.13126746e-17j,
            1.21570690e-17+1.43316918e-18j,  1.39345407e-17+1.29648187e-17j,
            2.63772776e-17+2.28799902e-17j, -3.17099733e-18+8.26904155e-18j,
           -9.52305721e-18+3.18701605e-18j, -6.18254301e-18+8.00208465e-18j,
           -1.49687998e-17-4.32882826e-17j,  2.68933330e-17-1.66837372e-17j,
           -4.46329949e-18-1.13692690e-17j, -1.29494661e-17-3.89169794e-18j,
           -1.18623867e-18+3.20797444e-17j, -3.42849445e-17+2.01149556e-17j,
            5.78641921e-18+2.63843543e-17j, -2.36377689e-16+8.92343348e-17j,
            1.19083291e-16+4.65088662e-17j, -9.52584472e-18-5.03713591e-17j,
           -3.17431088e-17-2.60157258e-17j,  1.00070411e-17-1.53196227e-17j,
            5.44757499e-17-8.00440860e-18j, -5.76964612e-17+6.25060555e-17j,
           -8.75795994e-18-1.27988814e-17j, -3.06844216e-18-4.56147158e-18j,
           -4.48112040e-18-4.40774128e-18j, -4.84642028e-18-3.59345194e-17j,
            1.47048808e-17-3.53382777e-17j, -2.65410064e-17+4.22951485e-17j,
           -9.59362819e-18-2.54253755e-17j, -1.10407640e-17+7.05768827e-18j,
           -1.67725969e-17+1.19777225e-17j,  1.99809505e-19-2.67145552e-17j,
            1.90060405e-17+2.59124032e-17j, -8.15256609e-18-2.23707536e-17j,
            1.37904901e-17-3.06409931e-18j, -2.80104016e-17+3.65342668e-17j,
           -2.88322147e-18+8.17575359e-17j, -2.84093559e-17+4.25791764e-17j,
           -1.90246603e-17+1.22912619e-17j,  2.95399091e-18+1.78870148e-17j,
           -1.52484182e-17-4.52944100e-19j,  9.88922120e-18+1.23778527e-17j,
           -2.05359598e-18+2.31666492e-18j,  4.75859905e-17-3.72286223e-17j,
            3.69818650e-18-1.77632704e-17j, -5.95199341e-18-1.98508706e-18j,
           -5.02972634e-18+1.23037215e-17j, -2.70574490e-17-2.27813839e-17j,
           -3.42993923e-17-6.69428718e-18j, -8.59733286e-18-3.13224356e-17j,
           -1.27328427e-17-2.49237626e-18j, -2.88536929e-18+3.07903099e-17j,
           -2.50207771e-17-4.69649278e-17j,  8.74773175e-18-1.02149477e-17j,
            6.73004436e-18-1.95189433e-19j, -3.72203669e-17-1.47930328e-17j,
            5.02680235e-17+3.08820955e-17j, -5.90898809e-17+2.81646679e-17j,
           -3.47114874e-17-3.77871586e-17j, -3.21358239e-18+1.08188328e-17j,
           -9.43832889e-18+6.70406497e-18j,  1.28365091e-17-2.61691515e-18j,
           -1.70391681e-17+3.27052676e-17j,  1.68666720e-17+1.38048078e-17j,
           -1.76934507e-17-1.74852929e-17j,  1.66830357e-17-3.39196635e-17j,
            3.73404254e-17+1.08191730e-16j, -5.02168644e-17-3.01636600e-17j,
            3.67627840e-17+3.89254519e-18j, -4.24423348e-17+1.33590068e-18j,
            3.60955811e-18-1.20858604e-17j, -1.05462579e-17-2.64177826e-17j,
           -7.94797393e-18+2.01846453e-17j,  5.33065080e-18+9.84268266e-18j,
           -1.85205575e-17+7.32759775e-18j, -1.86275512e-17+1.51427020e-17j,
           -2.18827000e-19+4.30700416e-17j, -3.23128328e-17-2.35118130e-17j,
            3.96602440e-17+1.04778993e-17j,  6.86293257e-18+1.18586045e-17j,
           -3.32345339e-17-4.56700714e-17j,  4.03901530e-18-4.55718323e-18j,
           -4.64612242e-17+2.04623659e-17j, -3.65727996e-17-1.33200512e-17j,
           -8.65001210e-18+1.23552046e-17j, -2.24577284e-17+2.55382124e-17j,
           -2.26412834e-18-1.00851662e-17j, -3.05194913e-17+2.08347638e-17j,
            4.79791932e-17+2.05638378e-17j, -1.79550444e-17+7.49231230e-18j,
           -6.77879367e-18+5.11683046e-17j,  3.59738403e-17-1.12362209e-17j,
            9.11461833e-18+1.37130677e-19j, -3.35433351e-17-4.57823676e-17j,
            3.16592986e-18-1.61149116e-19j,  1.92463581e-17+3.25121545e-17j,
           -2.80235143e-17-8.24339048e-17j, -7.84751830e-17+8.44216917e-18j,
           -1.66532449e-17+1.20433253e-17j, -4.39922187e-17-4.17371103e-19j,
           -1.83774589e-17-4.61792972e-17j, -4.74939743e-17-2.14428898e-17j,
            2.68828388e-17-6.25558081e-18j,  1.86660137e-17+4.64246585e-18j,
           -1.61795889e-17-4.93380064e-18j,  4.39611281e-20+4.55960447e-17j,
            9.12409672e-19+3.62846145e-17j, -2.04972111e-17-5.78506605e-18j,
           -1.88504163e-18+1.70572407e-17j,  9.41634605e-18-1.11781186e-17j,
            9.92167603e-18+3.81048097e-17j, -1.21585792e-17-4.05304921e-17j,
            3.25652359e-18+6.54655757e-17j,  7.84171339e-17+3.63384440e-17j])

Aula 44 - DFT resolucao e zero padding/.ipynb_checkpoints/DFT - resolucao e zero padding-checkpoint.ipynb


# 0. Imports

```python
import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split
from metrics import euclidean_similarity, pearson_similarity
import numpy.ma as ma

import scipy.sparse as scp

from metrics import mapk

from elementary import ElementaryRecommendation
```

# 1. Models description

Let $U$ be the set of users, $I$ - set of Items

## Baseline

### **Elementary Recommendation**

The most simlest way to predict ratings could be just a predicting average rating of all items, but this is not good at all. So for a baseline we can consider users and items bias. For example, some users rate the movies only in range 1-4 out of 10: rating 5 for them is something phenomenal; or users can also rate movies only highly.

Let $\mu$ be the average rating of all items, $r_{ui}$ - rating of item $i$ by user $u$, $I_u$ - items that were rated by user $u$. Then we can calculate user bias $b_u$ as:

$$
\begin{align*}
b_u=&\frac{1}{|I_u|+\alpha}\sum_{i\in{}I_u}(r_{ui} - \mu)
\end{align*},
$$

where $\alpha$ is smoothing coefficient.

We can similarly define item bias, but now let's take into account calculated users biases. Let $U_i$ be the users that rated item $i$. So then item bias $b_i$ is:

$$
\begin{align*}
b_i=&\frac{1}{|U_i|+\alpha}\sum_{u\in{}U_i}(r_{ui} - b_u - \mu)
\end{align*}
$$

**Final prediction** will be:

$$
\begin{align*}
r_{ui} = \mu+b_u+b_i
\end{align*}
$$

> ##### See the implementation of this approach in **elementary.py** file.

## Similarity based

### **Metrics**

How can the similarity of users/items be compared? Let's make an assumption that two users are similar if they rate items in a similar way and two items are similar if they are rated by users in a similar way. In this case we will use similarity metrics based on Euclidean distance and Pearson correlation coefficient which are suitable for this approach.

#### **Euclidean similarity**

Euclidean distance between two points $x$ and $y$ in Euclidean n-space can be calculated by the following equation:

$$
\begin{equation}
d(x, y) = \sqrt{\sum_{k=1}^n (x_k-y_k)^2}
\end{equation}
$$

Euclidean distance is inversely propotional to similarity so we should use inverted value of $d(x, y)$. Adding 1 in the denominator to avoid zero division we get the Euclidean similarity:

$$
\begin{equation}
sim(p, q) = \frac{1}{1 + d(p, q)}
\end{equation}
$$

#### **Pearson similarity**

Let $I_{uv}$ be the items that were rated both by users $u$ and $v$, $U_{ij}$ - users that rated both items $i$ and $j$:

$$
I_{uv} = \{i \in I \space | \space \exists r_{ui} \space \& \space \exists r_{vi}\}
$$
$$
U_{ij} = \{u \in U \space | \space \exists r_{ui} \space \& \space \exists r_{uj}\}
$$

Then the Pearson similarity between two users $u$ and $v$ can be calculated as:

$$
\begin{equation}
sim(u, v) = \frac {\sum_{i \in I_{uv}} \left( r_{ui} - \mu_u \right) \left( r_{vi} - \mu_v \right)}
                  {\sqrt{\sum_{i \in I_{uv}} \left( r_{ui} - \mu_u \right)^2}
                   \sqrt{\sum_{i \in I_{uv}} \left( r_{vi} - \mu_v \right)^2}}
\end{equation}
$$

And between two items $i$ and $j$:

$$
\begin{equation}
sim(i, j) = \frac {\sum_{u \in U_{ij}} \left( r_{ui} - \mu_i \right) \left( r_{uj} - \mu_j \right)}
                  {\sqrt{\sum_{u \in U_{ij}} \left( r_{ui} - \mu_i \right)^2}
                   \sqrt{\sum_{u \in U_{ij}} \left( r_{uj} - \mu_j \right)^2}}
\end{equation}
$$

> ##### See the implementation of these metrics in **metrics.py** file.

### **Trivial Recommendation**

Now let's consider an approach using similarity.

Let $i_0$ be the item that was highly rated by user $u_0$. We can find all users that rated this item (some kind of collaboration):

$$
U(i_0) = \{ u \in U \space | \space \exists r_{ui_0} \}
$$

Next we can comprise the set of items that were rated by users from $U(i_0)$:

$$
I(i_0) = \{ i \in I \space | \space \exists r_{ui}, \space u \in U(i_0) \}
$$

**Final prediction** will be top $N$ items from the set of items $I(i_0)$ sorted by similarity $sim(i, i_0)$ to the starting item $i_0$ .

> ##### See the implementation of this approach in **trivial.py** file.

### **User-based Recommendation**

Let $u_0$ be the user to whom we want to recommend. Let's comprize $U(u_0)$ as a set of top $K$ users that are most similar to $u_0$. We can find $U(u_0)$ as Nearest Neighbors to $u_0$ by one of the similarity metrics defined above.

The value of ratings user $u_0$ supposedly gives to item $i$ is calculated as an aggregation of some similar users rating of the item. In this case we will use:

$$
r_{u_{0}i}=\bar{r}_{u_0} + \frac{\sum_{u \in U(u_0)} sim(u, u_0)(r_{ui}-\bar{r}_u)}{\sum_{u \in U(u_0)}|sim(u, u_0)|},
$$

where $\bar{r}_{u^\prime}$ is the average rating of user $u^\prime$ for all the items rated by $u^\prime$.

**Final prediction** will be top $N$ items that have the highest predicted rating.

> ##### See the implementation of this approach in **user_based.py** file.

# 2. Data preparing

We will use subset of **The Movies Dataset** from **kaggle** to evaluate our models.

```python
data = pd.read_csv('ratings_small.csv', index_col=False)
data.shape
```

    (100004, 4)

```python
data.head()
```

      userId
      movieId
      rating
      timestamp

      0
      1
      31
      2.5
      1260759144

      1
      1
      1029
      3.0
      1260759179

      2
      1
      1061
      3.0
      1260759182

      3
      1
      1129
      2.0
      1260759185

      4
      1
      1172
      4.0
      1260759205

```python
print((data.userId.min(), data.userId.max(), len(data.userId.unique())))
print((data.movieId.min(), data.movieId.max(), len(data.movieId.unique())))
```

    (1, 671, 671)
    (1, 163949, 9066)

There are many missings id's so it would be more convenient if we changed the numbering.

```python
user_to_idx = {userId : idx for idx, userId in enumerate(data.userId.unique())}
movie_to_idx = {movieId : idx for idx, movieId in enumerate(data.movieId.unique())}

data.replace({'userId': user_to_idx, 'movieId': movie_to_idx}, inplace=True)
```

```python
print((data.userId.min(), data.userId.max(), len(data.userId.unique())))
print((data.movieId.min(), data.movieId.max(), len(data.movieId.unique())))
```

    (0, 670, 671)
    (0, 9065, 9066)

Let's split our data into train and test this way: we will remove the top 5 ratings from the most active users and put them in test data.

```python
train_data = data.copy()
test_data = pd.DataFrame([], columns=train_data.columns)

active_users = train_data.userId.value_counts()[:10].index

for user_id in active_users:
    test = train_data[train_data.userId == user_id].sort_values(by='rating', ascending=False)[:5]
    test_data = test_data.append(test, ignore_index=True)
    train_data = train_data[~((train_data.userId == user_id) & (train_data.movieId.isin(test.movieId.values)))]

train_data.shape, test_data.shape
```

    ((99954, 4), (50, 4))

It would be rather well to use [sparse matrix](https://en.wikipedia.org/wiki/Sparse_matrix) for storing all ratings, but in this case let's use dense one for simplifying some actions.

```python
X = scp.coo_matrix(
    (
        train_data['rating'],  # users ratings
        (train_data['userId'], train_data['movieId'])  # users and movies id's whose ratings are known
        ),
        shape=(len(user_to_idx), len(movie_to_idx))  # ratings matrix shape
    ).tocsr().A
```

```python
X
```

    array([[2.5, 3. , 3. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           ...,
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ]])

# 3. Quantifying the quality of recommendations.

```python
predicted = []

er = ElementaryRecommendation()
er.fit(X)

for id in test_data.userId.unique():
    predicted.append(er.make_recommendation(user_id=id, n_recommendations=100))
```

```python
actual = []

for id in test_data.userId.unique():
    actual.append(test_data[test_data.userId == id]['movieId'].astype(int).values)
```

```python
print(mapk(actual=actual, predicted=predicted, k=5))
print(mapk(actual=actual, predicted=predicted, k=10))
print(mapk(actual=actual, predicted=predicted, k=100))
```

    0.0
    0.0
    2.2222222222222223e-05

```python
from metrics import recsys_euclidean_similarity
```

```python

```


[Table of contents](../toc.ipynb)

# Ordinary differential equations

We will compare different discretization schemes to solve ordinary differential equations (ODE) in this mini project. The goal is that you have gathered practical experience in
 * explicit Euler,
 * and Runge-Kutta solver for ODEs.

We will solve and compare the solution of the basic ODE

$y'(t) = y(t)$

with exact solution, and numerical ODE solvers like it was done in section 1.2.6 [[Burg2013]](../references.bib) for Matlab.

```python
%matplotlib inline
from matplotlib import pyplot as plt

import sympy as sp
import numpy as np
```

## Exact solution

The exact solution is likely very simple to solve for you and you do not need Python for it, but let us compute with SymPy briefly.

First, we need to define a symbol and a function, which should result in the given function $y'(t) = y(t)$.

```python
y = sp.Function('y')
t = sp.symbols('t')

dgl = sp.Eq(y(t).diff(t), y(t))
dgl
```

$\displaystyle \frac{d}{d t} y{\left(t \right)} = y{\left(t \right)}$

We can compute now with SymPy the exact solution with and without initial condition.

```python
sp.dsolve(dgl, y(t))
```

$\displaystyle y{\left(t \right)} = C_{1} e^{t}$

The solution with initial condition becomes:

```python
sp.dsolve(dgl, y(t), ics={y(0): 3})
```

$\displaystyle y{\left(t \right)} = 3 e^{t}$

It is obvious that the integration constant $C_1$ from the solution without initial condition has always the value
$C_1 = y(0)$.

Therefore, we can define a function to compute the exact solution on different initial conditions.

```python
def exact_solution(time, init):
    return init * np.exp(time)
```

And a loop computes the solutions for each initial condition.

```python
time = np.arange(0, 3, 0.1)
init = np.arange(0, 4, 0.5)

dgl_exact = np.zeros((init.size, time.size))

for idx, value in enumerate(init):
    dgl_exact[idx,:] = exact_solution(time, value)
```

The next plot shows the solutions.

```python
plt.figure()
for row in np.ndindex(dgl_exact.shape[0]):
    plt.plot(time, dgl_exact[row], 'k-')
plt.ylim([0, 10])
plt.xlabel("time")
plt.ylabel("$y$")
plt.legend(["Exact solutions"])

plt.show()
```

## Numerical solution

### Runge-Kutta

Now we want to apply SciPy to compute the numerical solution with Runge-Kutta numerical solver. SciPy uses the `RK45`, which is an explicit Runge-Kutta of order four, where the step size is approximated from order five method. The classical Runge-Kutta of order four follows

$\begin{align}
r_1 &= f(t_i, y_i)\\
r_2 &= f(t_i + \frac{1}{2} \Delta t, y_i + \frac{1}{2} \Delta t r_1)\\
r_3 &= f(t_i + \frac{1}{2} \Delta t, y_i + \frac{1}{2} \Delta t r_2)\\
r_4 &= f(t_i + \Delta t, y_i + \Delta t r_3)\\
y_{i+1} &= y_i + \frac{1}{6} \Delta t (r_1 + 2 r_2 + 2 r_3 + r_4),
\end{align}$

see equation (1.105) in [[Burg2013]](../references.bib).

First, we need to define the function which should be solved.

```python
def equation(t, y):
    """Return right hand side of d/dt y(t) = y(t)"""
    return y
```

Now, we can use `scipy.intergrate` to solve the solution with Runge-Kutta solver.

```python
from scipy.integrate import solve_ivp

time_span = [0, 3]
init = [1]
dgl_RK45 = solve_ivp(equation, time_span, init, method='RK45', t_eval=time)
```

And now we plot this solution for initial value $y(0)=1$.

```python
plt.figure()
plt.plot(time, dgl_RK45.y[0, :])
plt.legend(["$y$ Runge-Kutta"])
plt.xlabel("Time")
plt.ylabel("$y$")
plt.ylim([0, 10])
plt.show()
```

### Explicit Euler

The explicit Euler method (forward Euler) is the simplest one step numerical solver we can implement by ourself.

It is based on the approximation

$y'(t_i) \approx \dfrac{y(t_{i+1}) - y(t_i)}{\Delta t}$,

$y(t_{i+1}) - y(t_i) \approx \Delta t f(t_i, y(t_i))$,

and the solution becomes,

$y_{i+1} = y_i + \Delta t f(t_i, y_i)$,

where $\Delta t$ is the step size and $f(t_i, y_i)$ is right hand side of $y'(t) = f(t, y(t))$, see section 1.2.6 [[Burg2013]](../references.bib).

```python
def solve_eeuler(func, time_vec, delta_t, inc):
    """Forwar Euler for DGL of type d/dt y = f(t, y(t))"""
    y = np.zeros(time_vec.size)
    y[0] = inc

    for idx, t in enumerate(time_vec[1:]):
        f = func(t, np.array([y[idx]]))
        y[idx + 1] = y[idx] + delta_t * f
    return y
```

```python
dgl_eeuler = solve_eeuler(func=equation, time_vec=time, delta_t=0.1, inc=1.0)
```

```python
time2 = np.arange(0, 3, 0.3)
dgl_eeuler2 = solve_eeuler(func=equation, time_vec=time2, delta_t=0.3, inc=1.0)
```

### Implicit Euler

The implicit Euler or backward Euler solver looks very similar to explicit Euler. However, here the computation $f(t_{i+1}, y_{i+1})$ is required which makes this method computational more expensive but ensure stability. The implicit Euler is

$y_{t+1} = y_i + \Delta t f(t_{i+1}, y_{i+1})$.

However, we will skip the implementation of implicit Euler herein.

### Results

The next plot shows all solutions together. The large step size in `dgl_eeuler2` causes large discretization error. Add to this, Euler method can also cause unstable solutions if the step size is to large for the given problem.

```python
plt.figure()
plt.plot(time, dgl_exact[2], 'k-')
plt.plot(time, dgl_RK45.y[0, :], '--')
plt.plot(time, dgl_eeuler, '--')
plt.plot(time2, dgl_eeuler2, '--')
plt.legend(["Exact solution", "$y$ Runge-Kutta", "$y$ Forward Euler", "$y$ Forward Euler2"])
plt.xlabel("Time")
plt.ylabel("$y$")
plt.ylim([0, 10])
plt.show()
```

Finally, let us compute the error of Runge-Kutta and forward Euler solver compared with exact solution.

```python
plt.figure()
plt.plot(time, dgl_exact[2] - dgl_RK45.y[0, :], '--')
plt.legend(["Runge-Kutta"])
plt.xlabel("Time")
plt.ylabel("Discretization error")
plt.show()
```

```python
plt.figure()
plt.plot(time, dgl_exact[2] - dgl_eeuler, '--')
plt.legend(["Forward Euler"])
plt.xlabel("Time")
plt.ylabel("Discretization error")
plt.show()
```

It is clearly visible that Runge-Kutta provides much better accuracy than explicit Euler. For more ODE solvers and their accuracy consult pages 75-80 in [[Burg2013]](../references.bib).

## Exercise:  ODE mini project

Please find the exact and Runge-Kutta solution for the mechanical oscillator system

$m\ddot{x}(t) + r \dot{x}(t) + k x(t) = K(t)$,

where $m$ denotes mass, $r$ damping constant, $k$ spring constant, and $K(t)$ is a time varying external force. We are looking for the displacement $x(t)$.

You can assume that $K(t) = K_0 \cos \omega t$.


# Solving Growth Models

Contents:

- [Solving Growth Models](#Solving-Growth-Models)
  - [A Basic Growth Model](#A-Basic-Growth-Model)
  - [Transition Equations](#Transition-Equations)
  - [Steady State](#Steady-State)
  - [Solving the Model](#Solving-the-Model)
  - [Generating Optimal Paths](#Generating-Optimal-Paths)

This lab includes:

(1) An overview of a basic non-stochastic growth model;

(2) Finding the transition equations of a growth model using dynamic programming;

(3) Solving for the policy functions of a growth model using value function iteration;

(4) Generating the optimal paths of variables using policy functions.

## A Basic Growth Model

Consider the following economy.

There is a representative agent who has preferences given by

\begin{align*}
    \sum_{t=0}^\infty \beta^t u(c_t) \, ,
\end{align*}

where $u(c_t) = \log(c_t)$ and $\beta \in (0,1)$ is the discount factor.

The technology in this economy is given by

\begin{align*}
    c_t + i_t &= y_t \, , \\
    y_t &= A k_{t}^\alpha \, , \\
    k_{t+1} &= i_t + (1-\delta) k_t \, ,
\end{align*}

where $c_t$ is consumption, $i_t$ is investment, $y_t$ is output, $k_t$ is capital, $\alpha \in (0,1)$, $A \in \mathbb{R}_+$ is the productivity factor, and $\delta \in (0,1)$ is the depreciation rate.

The social planner's problem may be expressed by the following Bellman equation:

\begin{align*}
    V(k) = \max_{c,i,k'} \left\{ \log(c) + \beta V(k') \right\}
\end{align*}

subject to

\begin{align}
    c + i &= A k^\alpha \, , \label{eq:flow_constraint} \\
    k' &= (1-\delta) k + i \, . \label{eq:capital_accumulation}
\end{align}

## Transition Equations

We can simplify this system by using the first constraint to solve for $i$:

\begin{align*}
    i = A k^\alpha - c \, ,
\end{align*}

which can be substituted into the second constraint to yield

\begin{align*}
    k' = (1-\delta) k + A k^\alpha - c  \, .
\end{align*}

The above equation for $k'$ can be substituted into the Bellman equation to yield

\begin{align*}
    V(k) = \max_{c} \left\{ \log(c) + \beta V\left[ (1-\delta)k + Ak^\alpha - c \right] \right\} \, .
\end{align*}

Using the Bellman equation we may derive the following first-order condition for $c$:

\begin{align*}
    0 = \frac{1}{c} + \beta V'(k')(-1) \, ,
\end{align*}

which is equivalent to

\begin{align}
\label{eq:foc}
    \beta V'(k') = \frac{1}{c} \, .
\end{align}

We may also use the Bellman eqation to obtain the following envelope condition:

\begin{align*}
V'(k) = \beta V'(k') (1 - \delta + \alpha A k^{\alpha - 1} ) \, .
\end{align*}

Eq. \eqref{eq:foc} can then be substituted into the RHS of the envelope condition to yield

\begin{align*}
    V'(k) = \frac{1}{c} \left( 1 - \delta + \alpha A k^{\alpha-1} \right) \, .
\end{align*}

This equation can now be "stepped-up" by a period ($k'$ and $c'$ instead of $k$ and $c$) to give us

\begin{align*}
    V'(k') = \frac{1}{c'} \left( 1 - \delta + \alpha A k'^{\alpha-1} \right) \, .
\end{align*}

We once again substitute Eq. \eqref{eq:foc} into the LHS of the above equation to obtain

\begin{align*}
    \frac{1}{\beta c} = \frac{1}{c'} \left( 1 - \delta + \alpha A k'^{\alpha-1} \right) \, ,
\end{align*}

which can be used to solve for $c'$:

\begin{align*}
    c' = c \beta \left( 1 - \delta + \alpha A k'^{\alpha - 1} \right) \, .
\end{align*}

We can now attach time indices to the above equation to obtain the Euler equation:

\begin{align}
\label{eq:euler}
    c_{t+1} = c_t \beta \left(1 - \delta + \alpha A k_{t+1}^{\alpha-1} \right) \, .
\end{align}

Therefore, the economy is governed by the system of transition equations made up by Eqs. \eqref{eq:flow_constraint}, \eqref{eq:capital_accumulation}, and \eqref{eq:euler}.

I repeat all equations below for easy reference.

**Capital accumulation:**

\begin{align*}
k_{t+1} = (1-\delta) k_t + Ak_t^{\alpha} - c_t
\end{align*}

**Euler equation:**

\begin{align*}
c_{t+1} = c_t \beta \left( 1 - \delta + \alpha A k_{t+1}^{\alpha-1} \right)
\end{align*}

**Output:**

\begin{align*}
y_t = A k_t^\alpha
\end{align*}

**Flow budget constraint / Investment:**

\begin{align*}
i_t = A k_t^{\alpha} - c_t
\end{align*}

## Steady State

In the steady state (SS), we have $c_{t+1} = c_t = \overline{c}$ and $k_{t+1} = k_t = \overline{k}$, which also implies SS output

\begin{align*}
\overline{y} = A \overline{k}^\alpha
\end{align*}

and SS investment

\begin{align*}
\overline{i} = \overline{y} - \overline{c} \, .
\end{align*}

We now need to solve for $\overline{c}$ and $\overline{k}$.

Applying a bit of algebra to the Euler and capital accumulation equations yields

\begin{align*}
        \overline{k} = \left( \frac{\alpha A}{\beta^{-1} - 1 + \delta} \right)^\frac{1}{1-\alpha} \, ,
\end{align*}

along with

\begin{align*}
    \overline{c} = A \overline{k}^\alpha - \delta \overline{k} \, .
\end{align*}

Let's write a function named `steadyState` that will output all model parameters and steady state values:

```julia
"""
   steadyState(α, β, δ, A)

Solves for the steady state of the growth model.

# Arguments
* `α` -- production function parameter
* `β` -- discount factor
* `δ` -- depreciation rate
* `A` -- productivity factor

"""
function steadyState(α = 0.3, β = 0.98, δ = 0.02, A = 1)

    # Solve for k̄
    k̄ = (  (α * A) / (β^(-1) - 1 + δ) )^(1/(1-α))

    # Solve for ȳ
    ȳ = A * k̄^α

    # Solve for c̄
    c̄ = ȳ - δ * k̄

    # Solve for ī
    ī = ȳ - c̄

    # Return model parameters and steady state variables
    return (α = α, β = β, δ = δ, A = A, c̄ = c̄, k̄ = k̄, ȳ = ȳ, ī = ī)
end
```

    steadyState

We run the function and store all outputs:

```julia
# Generate and save steady state objects
α, β, δ, A, c̄, k̄, ȳ, ī = steadyState()

# Print steady state values
@show c̄
@show k̄
@show ȳ
@show ī;
```

    c̄ = 2.0106154404398704
    k̄ = 17.53027697180669
    ȳ = 2.361220979876004
    ī = 0.3506055394361338

## Solving the Model

Let's load the packages we might need.

```julia
# Load all packages we might need
using LinearAlgebra, Plots, LaTeXStrings
```

Our goal now is to computationally solve for the policy function using value function iteration.

We start by constructing a function named `growthBellmanMap` that applies a single iteration of the Bellman map.

```julia
"""
   growthBellmanMap(V, kgrid, α, β, δ, A)

Applies a single iteration of the Bellman map to the model.
Returns a new iteration of the value and policy functions.

# Arguments
* `V`     -- An initial value function
* `kgrid` -- A grid of k values over some interval
* `α`     -- production function parameter
* `β`     -- discount factor
* `δ`     -- depreciation rate
* `A`     -- productivity factor

"""
function growthBellmanMap(V, kgrid, α, β, δ, A)

    N     = length(kgrid) #k grid length
    V_out = zeros(N) #new value function
    n_pol = zeros(Int,N) #policy rule for grid points
    k_pol = zeros(N) #policy rule for capital
    c_pol = zeros(N) #consumption rule
    obj   = zeros(N) #objective to be maximized wrt k′

    for n in 1:N #iterate for each initial capital

        for nprime in 1:N #iterate for choice of capital this period

            # Compute optimal consumption level
            c = (1-δ) * kgrid[n] + A * kgrid[n]^(α) - kgrid[nprime]

            if c <= 0
                obj[nprime] = -Inf #penalty if consumption <0
            else
                obj[nprime] = log(c) + β * V[nprime] #otherwise RHS of bellman equation
            end
        end

        V_out[n], n_pol[n] = findmax(obj) #find optimal value and the choice that gives it
        k_pol[n]           = kgrid[n_pol[n]] #record capital policy
        c_pol[n]           =(1-δ) * kgrid[n] + A * kgrid[n]^(α) - k_pol[n] #record consumption policy

    end

    return V_out, n_pol, k_pol, c_pol
end
```

    growthBellmanMap

Then we create a function named `solveGrowthBellman` that iteratively applies `growthBellmanMap` until reaching convergence.

The output of the function will contain a vector representing the value function, the policy functions for $k$ and $c$, as well as an index representing the policy map from $k_t$ to $k_{t+1}$ on a given grid for $k$.

```julia
"""
   solveGrowthBellman(V0, kgrid, α, β, δ, A, ϵ)

Applies the Bellman map until convergence.
Returns final iterations of the value and policy functions.

# Arguments
* `V`     -- initial value function
* `kgrid` -- grid of k values over some interval
* `α`     -- production function parameter
* `β`     -- discount factor
* `δ`     -- depreciation rate
* `A`     -- productivity factor
* `ϵ`     -- convergence condition

"""
function solveGrowthBellman(V0, kgrid, α, β, δ, A, ϵ=1e-6)

    # Initialize the convergence metric
    diff = 0.1

    # Apply the first iteration of the Bellman map
    V, n_pol, k_pol, c_pol = growthBellmanMap(V0, kgrid, α, β, δ, A)

    # Apply the Bellman map until convergence
    while diff > ϵ

        V_new, n_pol, k_pol, c_pol  = growthBellmanMap(V, kgrid, α, β, δ, A)

        diff = norm(V_new-V,Inf) #update convergence metric

        V = V_new #update value function
    end

    # Return final value function, policy function map indexes,
    # and policy functions for k and c
    return V, n_pol, k_pol, c_pol
end
```

    solveGrowthBellman

We now have everything we need to solve the model using value function iteration:

```julia
# Create a grid for k
N     = 501
kgrid = LinRange(k̄-10,k̄+10,N) #center grid on k̄

# Generate value function iteration solutions
V, n_pol, k_pol, c_pol = solveGrowthBellman(zeros(N), kgrid, α, β, δ, A);
```

We can plot the policy function for $k$:

```julia
plot(kgrid, kgrid, label = "45° line", color = "black")
plot!(kgrid, fill(k̄, length(kgrid)), label = "k̄", color = "gray", lw = 2, linestyle = :dash)
vline!([k̄], label = "", color = "gray",lw = 2, linestyle = :dash)
plot!(kgrid, k_pol, label = "Optimal k'", color = "orange", lw = 3)
title!("\$ \\textrm{Policy Function for } k_{t+1} \$")
xlabel!("\$k_{t}\$")
ylabel!("\$k_{t+1}\$")
```

We may also plot the policy function for $c$:

```julia
plot(title = "\$ \\textrm{Policy Function for } c_t \$")
plot!(kgrid, fill(c̄, length(kgrid)), label = "c̄", color = "green", lw = 2, linestyle = :dash)
vline!([k̄], label = "k̄", color = "gray", lw = 2,  linestyle = :dash)
plot!(kgrid, c_pol, label = "Optimal c", color = "green", lw = 3)
xlabel!("\$k_t\$")
ylabel!("\$c_t\$")
```

## Generating Optimal Paths

Let's create a function that generates the path of the economy over $T$ time periods given the relevant policy functions and $k_0$ .

```julia
"""
   growthBellmanPath(kgrid, n_pol, k_pol, c_pol, grid_initial, T)

Applies the Bellman map until convergence.
Returns final iterations of the value and policy functions.

# Arguments
* `kgrid`        -- grid of k values over some interval
* `n_pol`        -- capital policy grid index map
* `k_pol`        -- capital policy function
* `c_pol`        -- consumption policy function
* `grid_initial` -- initial index of k
* `T`            -- length of time interval

"""
function growthBellmanPath(kgrid, n_pol, k_pol, c_pol, grid_initial::Int64, T::Int64)

    index = fill(0,T+1) # capital grid index path
    c = zeros(T)        # empty vector to store consumption path
    k = zeros(T+1)      # empty vector to store capital path

    index[1] = grid_initial # store t=0 capital index

    k[1] = kgrid[index[1]] # store capital at t=0

    for t in 1:T
        c[t] = c_pol[index[t]] # store consumption at t
        k[t+1] = k_pol[index[t]] # store capital t+1
        index[t+1] = n_pol[index[t]] # store t+1 capital index
    end

    k = k[1:T]             # cut the last entry of k path
    grid_pts = index[1:T]  # cut the last entry of capital index path

    # Return capital, consumption, and capital index paths
    return k, c, grid_pts
end
```

    growthBellmanPath

First we plot the paths of $k$ and $c$ given the lowest possible $k_0$ from our grid.

```julia
T = 100
k_path, c_path, grid_path = growthBellmanPath(kgrid, n_pol, k_pol, c_pol, 1, T)
time = 1:T
plot(time, c_path./c̄, label = "Consumption", lw=3)
plot!(time, k_path./k̄, label = "Capital", lw =3, title = "\$ \\textrm{Optimal Path Given Low } k_0 \$")
plot!(time, (A.*k_path.^α)./ȳ, label = "Output", lw=3)
plot!(time, (A.*k_path.^α - c_path)/ī, label = "Investment", lw=3)
xlabel!("Time")
```

Notice that a low initial capital implies a low initial consumption level.

Over time, both variable seem to grow uniformly and ultimately convergence to their respective steady states.

Next, we plot the paths of $k$ and $c$ once again, but this time given the highest possible value for $k_0$ from our grid.

```julia
T = 100
k_path, c_path, grid_path = growthBellmanPath(kgrid, n_pol, k_pol, c_pol, length(kgrid), T)
time = 1:T
plot(time, c_path./c̄, label = "Consumption", lw = 3)
plot!(time, k_path./k̄, label = "Capital", lw = 3, title = "\$ \\textrm{Optimal Path Given High } k_0 \$")
plot!(time, (A.*k_path.^α)./ȳ, label = "Output", lw=3)
plot!(time, (A.*k_path.^α - c_path)/ī, label = "Investment", lw=3)
xlabel!("Time")
```

Notice that a high initial capital implies a high initial consumption level.

Over time, both variable seem to experience uniform negative growth until they convergence to their respective steady states.


# Linear SVM

```{note}
The intuition behind Support Vector Machine(SVM) is to separate positive&negative samples by large margins.
```

## Margin

Suppose we have a binary classification problem with $x \in \mathbb{R}^{d}, y \in \{-1, 1\}$. Try to solve this by linear classifier $h(x) = w^{T}x + b$, if $h(x) >= 0$ predict 1, else -1. Intuitively, in addition to correctly separate the data, we want to separate the data by large margins.

```{image} images/margin.png
:alt: margin
:width: 500px
:align: center
```

Geometric margin of the classifier:

$$\gamma^{(i)}=\frac{y^{(i)}(w^{T}x^{(i)} + b)}{\left \| w \right \| }$$

We want to maximize geometic margin:

$$
\begin{equation}
\begin{split}
\underset{\gamma, w, b}{\max}\ &\gamma \\
\mbox{s.t.}\ &\frac{y^{(i)}(w^{T}x^{(i)} + b)}{\left \| w \right \| } >= \gamma
\end{split}
\end{equation}
$$

Without loss of generality, set $\gamma\left \| w \right \|=1$, then the above is equivalent to:

$$
\begin{equation}
\begin{split}
\underset{w, b}{\min}\ &\frac{1}{2}{\left \| w \right \|}^2 \\
\mbox{s.t.}\ &{y^{(i)}(w^{T}x^{(i)} + b)} >= 1
\end{split}
\end{equation}
$$

## Dual Problem

We can write the constraints as:

$$g_{i}(w) = 1 - y^{(i)}(w^{T}x^{(i)} + b) \le 0$$

The generalized Lagrangian of the above optimal problem:

$$
L(w,b,\alpha )=\frac{1}{2}\left \| w \right \|^{2} + \sum_{i=1}^{n}\alpha_{i}\left [ 1 - y^{(i)}(w^{T}x^{(i)} + b) \right ]
$$

Primal problem:

$$\underset{w, b}{\min}\ \underset{\alpha_{i}\ge0}{\max}\ L(w, b, \alpha)$$

The KKT condition:

1. $\frac{1}{2}\left \| w \right \|^{2}$ is convex.
2. $g_{i}$'s are convex.
3. the constraints $g_{i}$'s are strictly feasible.

is satisfied in the separable case, so the primal problem is equivalent to the dual problem:

$$\underset{\alpha_{i}\ge0}{\max}\ \underset{w, b}{\min}\ L(w, b, \alpha)$$

Minimize $\nabla_{w}L(w, b, \alpha)$:

$$\nabla_{w}L(w, b, \alpha) = w - \sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)} = 0 \Rightarrow w = \sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)}$$

$$\nabla_{b}L(w, b, \alpha) = \sum_{i=1}^{n}\alpha_{i}y^{(i)}=0$$

Plug back to Lagrangian:

$$
\begin{equation}
\begin{split}
L(w, b, \alpha) =& \frac{1}{2}\left \| w \right \|^{2} + \sum_{i=1}^{n}\alpha_{i}\left [ 1 - y^{(i)}(w^{T}x^{(i)} + b)\right ]\\
=& \sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}(x^{(i)})^{T}x^{(i)}
\end{split}
\end{equation}
$$

Dual problem:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{\max}\ &\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle x^{(i)},x^{(j)} \right \rangle \\
\mbox{s.t.}\ &\alpha_{i}\ge{0}, i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

## Non-separable Case

For non-separable case, we add the soft term $\xi$, thus transform the optimization:

$$
\begin{equation}
\begin{split}
\underset{w, b}{\min}\ &\frac{1}{2}{\left \| w \right \|}^2 + C\sum_{i=1}^{n}\xi_{i}\\
\mbox{s.t.}\ &{y^{(i)}(w^{T}x^{(i)} + b)} >= 1 - \xi_{i},\ i=1,...,n\\
&\xi_{i} \ge 0,\ i=1,...,n.
\end{split}
\end{equation}
$$

Lagrangian:

$$
L(w,b,\xi,\alpha,r )=\frac{1}{2}\left \| w \right \|^{2} + C\sum_{i=1}^{n}\xi_{i} + \sum_{i=1}^{n}\alpha_{i}\left [1 - \xi -y^{(i)}(w^{T}x^{(i)} + b)\right ] -\sum_{i=1}^{n}r_{i}\xi_{i}
$$

Dual problem of the non-separable case:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{\max}\ &\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle x^{(i)},x^{(j)} \right \rangle \\
\mbox{s.t.}\ &0 \le \alpha_{i}\le{C},\ i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

## Examples

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_features=4, random_state=0)
```

```python
"""preprocessing + svm"""
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

clf = make_pipeline(StandardScaler(), LinearSVC(random_state=0, tol=1e-5))
clf.fit(X, y)
```

    Pipeline(steps=[('standardscaler', StandardScaler()),
                    ('linearsvc', LinearSVC(random_state=0, tol=1e-05))])

```python
clf.named_steps['linearsvc'].coef_
```

    array([[0.14144316, 0.52678399, 0.67978685, 0.49307524]])

```python
clf.predict([[0, 0, 0, 0]])
```

    array([1])

## SMO algorithm

We can not directly use coordinate ascent in SVM:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{\max}\ &\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle x^{(i)},x^{(j)} \right \rangle \\
\mbox{s.t.}\ &0 \le \alpha_{i}\le{C},\ i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

If we want to update some $\alpha_{i}$, we must update at least two of them simutaneously, coordinate ascent change to:

1. select some pair $\alpha_{i}, \alpha_{j}$.
2. optimize with respect to $\alpha_{i}, \alpha_{j}$ while holding other $\alpha_{k}$ fixed.

We can easily change step-2 into a quatratic optimization problem.

Remaining the choice of $\alpha_{i},\alpha_{j}$ , in fact this step is heuristic.

```python

```


```python
import numpy as np, cmath,scipy as sp
import scipy.io
from matplotlib import pyplot as plt

from numpy import pi, sin, cos, exp, sqrt, log, random  #import basic functions from numpy that we'll need

%matplotlib inline
```

```python
import seaborn as sns
sns.set_palette('muted')
sns.set_style('darkgrid')
```

```python
data=scipy.io.loadmat('sampleEEGdata')
```

```python
EEGdata=data["EEG"][0,0]["data"]
srate = float(data["EEG"][0,0]["srate"][0,0])
```

###Figure 12.1
The real component of a Morlet wavelet is a cosine multiplied by a Gaussian window:
\begin{equation}
Re(\text{Morlet wavelet}) = \cos(2\pi f t) e^{-t^2/(2s^2)}
\end{equation}

```python
time = np.arange(-1,1+1/srate,1/srate)

f=4 #sinewave frequency, Hz

#create a sinewave (cosine wave)
sine_wave = cos(2*pi*f*time)

#create a Gaussian

s=4/(2*pi*f) #standard deviation

gaussian_win = exp(-time**2/(2*s**2))

#plot our first wavelet!
_=plt.plot(time,sine_wave*gaussian_win)
```

###Figure 12.2

```python
fig=plt.figure()

fig.add_subplot(share_x=True)

plt.subplot(511)
plt.plot(np.squeeze(EEGdata[46,:,0]))

plt.subplot(512)
sine_wave = cos(2*pi*12*time) # 12Hz cosine wave
plt.plot(time,sine_wave)

plt.subplot(513)
#boxcar envelope
boxcar = np.zeros(len(sine_wave))
midpoint = (len(time)-1)/2.
boxcar[midpoint-np.round(srate/12./5.):midpoint+np.round(srate/12./1.25)] = 1
plt.plot(time,sine_wave*boxcar)

plt.subplot(514)
#boxcar of different length
boxcar = np.zeros(len(sine_wave))
midpoint = (len(time)-1)/2.
boxcar[midpoint-50:midpoint+50] = 1
plt.plot(time,sine_wave*boxcar)

plt.subplot(515)
s = 1.5/(2*pi*f)
gaussian_win = exp(-time**2/(2*s**2))
plt.plot(time,sine_wave*gaussian_win)

plt.tight_layout()
```

###Figure12.3

```python
srate = 500. #sample rate in Hz
f = 10
time = np.arange(-1,1,1/srate)

#complex sinusoid
sine_wave = exp(2*pi*1j*f*time)

#Gaussian window
s = 6/(2*pi*f)
gaussian_win = exp(-time**2/(2*s**2))

#together they make a complex morlet wavelet!
wavelet = sine_wave*gaussian_win

#create plots for each component
fig = plt.figure()
fig.add_subplot(share_x=True)

plt.subplot(311)
plt.plot(time,np.real(sine_wave))
plt.title("sine wave")

plt.subplot(312)
plt.plot(time,gaussian_win)
plt.title("gaussian window")

plt.subplot(313)
plt.plot(time,np.real(wavelet))
plt.title("my first wavelet")
_=plt.xlabel("time (ms)")

plt.tight_layout()
```

###Figure 12.4

```python
num_wavelets = 80 # number of frequency bands
lowest_frequency = 2 #in Hz
highest_frequency = 100 # in Hz

#(linear) equally spaced frequencies for our wavelet family
frequencies = np.linspace(lowest_frequency,highest_frequency,num_wavelets)

plt.figure()
plt.plot(frequencies)
plt.xlabel("Frequency order")
_=plt.ylabel("Frequency in Hz")
```

```python
#initialize our wavelet family
wavelet_family = np.zeros([num_wavelets,len(time)])*1j #1j is to create a complex array of zeros

#iterate through freqs and make a wavelet family

for fi in xrange(num_wavelets):

    #create a sine wave
    sinewave = exp(2*1j*pi*frequencies[fi]*time)

    #create gaussian window
    gaus_win = exp(-time**2/(2*(6/(2*pi*frequencies[fi]))**2))

    #create wavelet by multiplying our sine wave by the gaussian window
    wavelet_family[fi,:] = sinewave*gaus_win

    #this could be accomplished on one line
#     wavelet_family[fi,:] = exp(2*1j*pi*frequencies[fi]*time) * exp(-time**2/(2*(6/(2*pi*frequencies[fi]))**2))

#plot some of our wavelet family
fig=plt.figure()
fig.add_subplot(share_x=True)
plt.subplot(211)
_=plt.plot(time,np.real(wavelet_family[::np.round(random.rand()*30),:].T))

plt.subplot(212)
plt.plot(time,np.real(wavelet_family[30,:]))
plt.plot(time,np.imag(wavelet_family[30,:]),'r:')
plt.title("real and imaginary parts of one wavelet")
plt.legend(["real","imaginary"])
plt.tight_layout()
```

```python
fig=plt.figure(figsize=(6,6))
plt.imshow(np.real(wavelet_family),
           extent=[time[0], time[-1], frequencies[0], frequencies[-1]],
           aspect="auto",
           cmap=plt.get_cmap("hot"),
           origin="lower")

plt.xlabel("time (s)")
_=plt.ylabel("frequency (Hz)")
```

###Figure 12.5

```python
from numpy.fft import fft, ifft #import fft functions for ease of use
from scipy import signal as sig
#EEG data from one trial (electrode FCz)
eegdata = np.squeeze(EEGdata[46,:,9])
EEGpnts = data["EEG"][0,0]["pnts"][0,0] #number of points in EEG data
EEGtimes = data["EEG"][0,0]["times"][0]
EEGsrate = float(data["EEG"][0,0]["srate"][0])

#create wavelet
time = np.arange(-1,1 + 1/EEGsrate,1/EEGsrate)
f = 6 #frequency in Hz
sine_wave = exp(2*1j*pi*f*time)

#compute gaussian
s=4.5/(2*pi*f)
gaussian_win = exp(-time**2/(2*s**2))
#window the sinewave by a gaussian to create complex morlet wavelet
wavelet = sine_wave * gaussian_win

#half of wavelet size, useful for chopping off edges after convolution
halfwaveletsize = np.ceil(len(wavelet)/2)

#convolve with data
n_conv = len(wavelet) + EEGpnts - 1 #number of points in our convolution

fft_w = fft(wavelet,n_conv)
fft_e = fft(eegdata,n_conv)
#convolution theorem -- convolution = pointwise multiplication in frequency-space
ift = ifft(fft_e*fft_w,n_conv)*sqrt(s)/10 #sqrt(s)/20 is empirical scaling factor (sqrt(s)/10 in the book)
wavelet_conv_data = np.real(ift[halfwaveletsize: - halfwaveletsize]) #take middle portion of convolution
wavelet_conv_data = np.real(ift[halfwaveletsize:-halfwaveletsize]) #take middle portion of convolution

#create a filter to apply to data
nyquist = EEGsrate/2
transition_width = 0.2 #percent
filter_low = 4 #Hz
filter_high = 8 #Hz

ffrequencies = np.array([0 ,filter_low*(1-transition_width),
                         filter_low, filter_high, filter_high*(1+transition_width), nyquist])/nyquist
ideal_response = np.array([0, 0, 1, 1, 0, 0])

#there doesn't seem to be a python equivalent to MATLAB's firls function,
#so I am going to use butterworth filter as a close approximation.

b, a = sig.butter(5, np.array([filter_low*(1-transition_width),filter_high*(1+transition_width)])/nyquist,btype="bandpass")
eeg_4to8 = sig.filtfilt(b, a, eegdata, padlen=150)

plt.plot(EEGtimes,eegdata)
plt.plot(EEGtimes,wavelet_conv_data,'r')
plt.plot(EEGtimes,eeg_4to8,'g')
plt.axis([-200,1200,-40,40])
plt.xlabel("time (ms)")
plt.ylabel("voltage (mV)")
_=plt.legend(["raw","wavelet conv","band-passed"])
```

###Figure 12.6

```python
time = np.arange(-1,1+1/EEGsrate,1/EEGsrate)
n_conv = EEGpnts + len(time) -1
n2p1 = np.floor(n_conv/2)+1

f = 6 #hz
s = 6/(2*pi*f)
wavelet = exp(2*pi*1j*f*time) * exp(-time**2/(2*s**2))
halfwaveletsize = np.ceil(len(wavelet)/2)

eegdata = np.squeeze(EEGdata[46,:,9])

plt.figure()
plt.subplot(311)
plt.plot(EEGtimes,eegdata)
plt.xlim([-500,1200])
plt.title("raw")

plt.subplot(323)
fft_w = fft(wavelet,n_conv)
hz    = np.linspace(0,EEGsrate/2.,n2p1)
plt.plot(hz,np.absolute(fft_w[:n2p1])/np.max(np.absolute(fft_w[:n2p1])),'b')

fft_e = fft(eegdata,n_conv)
plt.plot(hz,np.absolute(fft_e[:n2p1])/np.max(np.absolute(fft_e[:n2p1])),'g')
plt.axis([0,40,0,1.05])
plt.title("individual power spectra")

plt.subplot(324)
plt.plot(hz,np.absolute(fft_e[:n2p1]*np.absolute(fft_w[:n2p1])))
plt.xlim([0, 40])
plt.title("convolved power spectrum")

plt.subplot(313)
plt.plot(EEGtimes,eegdata)
ift = ifft(fft_e*fft_w,n_conv)*sqrt(s)/10 #sqrt(s)/20 is empirical scaling factor (sqrt(s)/10 in the book)
plt.plot(EEGtimes,np.real(ift[halfwaveletsize:-halfwaveletsize]),'r')
plt.title("wavelet filtered")

plt.tight_layout()
```

###Figure 12.7

```python
#create 10Hz wavelet kernel

time = np.arange(-(EEGpnts/EEGsrate/2),EEGpnts/EEGsrate/2 + 1/EEGsrate,1/EEGsrate)
f = 10. #hz
s = 4/(2*pi*f) #sd of gaussian
wavelet = cos(2*pi*f*time) * exp(-time**2/(2*s**2))

#signal is one sine cycle
timeS = np.arange(0,1/f + 1/EEGsrate,1/EEGsrate)
signal = sin(2*pi*f*timeS)

#zeropad the signal
zz = np.zeros(EEGpnts/2 - len(timeS)/2)
signal = np.concatenate([zz,signal,zz])

plt.figure(figsize=(6,6))

#plot waves
plt.subplot(321)
plt.plot(wavelet,'r')
plt.xlim(200, len(time) - 200)
plt.title("wavelet")

plt.subplot(323)
plt.plot(signal)
plt.xlim([200, len(time)-200])
plt.title("1 cycle of signal")

plt.subplot(325)
plt.plot(np.convolve(wavelet,signal,mode="same"),'purple')
plt.axis([200,len(time)-200,-12,12])
plt.title("convolved wavelet and signal")

#plot the dot products at selected phase lags
plt.subplot(322)
plt.plot(wavelet[np.round(100/f)-2-1:],'r')
plt.plot(signal)
plt.xlim([200,len(time)-200])
plt.title("dot product: " + str( np.fix(np.sum(wavelet[np.round(100/f)-2-1:]*signal[:-np.round(100/f)+3]))))
plt.legend(["wavelet","signal"])

plt.subplot(324)
plt.plot(wavelet[np.round(2.3*100/f)-2-1:],'r')
plt.plot(signal)
plt.xlim([200,len(time)-200])
plt.title("dot product: " + str( np.fix(np.sum(
                wavelet[np.round(2.3*100/f)-2-1:]*signal[:-np.round(2.3*100/f)+3]))))

plt.subplot(326)
plt.plot(wavelet,'r')
plt.plot(signal)
plt.xlim([200,len(time)-200])
plt.title("dot product: " + str( np.fix(np.sum(
                wavelet*signal))))

plt.tight_layout()
```


# Tutorial rápido de Python para Matemáticos

© Ricardo Miranda Martins, 2022 - http://www.ime.unicamp.br/~rmiranda/

## Índice

1. [Introdução](1-intro.html)
2. [Python é uma boa calculadora!](2-calculadora.html) [(código fonte)](2-calculadora.ipynb)
3. [Resolvendo equações](3-resolvendo-eqs.html)  [(código fonte)](3-resolvendo-eqs.ipynb)
4. [Gráficos](4-graficos.html)  [(código fonte)](4-graficos.ipynb)
5. [Sistemas lineares e matrizes](5-lineares-e-matrizes.html)  [(código fonte)](5-lineares-e-matrizes.ipynb)
6. [Limites, derivadas e integrais](6-limites-derivadas-integrais.html)  [(código fonte)](6-limites-derivadas-integrais.ipynb)
7. **[Equações diferenciais](7-equacoes-diferenciais.html)**  [(código fonte)](7-equacoes-diferenciais.ipynb)

# Equações diferenciais

Chegamos no capítulo que motivou essas notas: vamos usar o Pythonn para resolver equações diferenciais!

Começaremos resolvendo algumas equações usando o SymPy. Ele não é a melhor ferramenta para isso, pois só procura soluções "algébricas" (no sentido de poderem ser explicitadas), mas para começar a brincar, já funciona bem.

## Equações lineares de primeira e segunda ordens

Primeiro vamos resolver uma equação simples: encontrar $f(x)$ tal que $$f'(x)+f(x)=0.$$ A notação para $f'(x)$ é um pouco estranha: para representar $f(t)$, devemos digitar ```f(t).diff(t)```. Se quiser a derivada de segunda ordem, $f''(t)$, entre ```f(t).diff(t,t)``` e por aí vai.

Depois disso, é só usar o comando ```dsolve```.

```python
import sympy as sp

# definindo a variavel independente
t = sp.symbols('t')

# definindo símbolos para as funcoes que estarão envolvidas
# nas equações diferenciais.
# note que precisamos já declarar esses símbolos como sendo
# da classe "função" com a adição cls=sp.Function
f = sp.symbols('f', cls=sp.Function)

# defina a equacao diferencial. para representar f'(t)
# deve-se usar f(t).diff(t).
# para indicar f''(t) usamos f(t).diff(t,t).
# a equacao sera na forma F=0. defina eq=F.
eq=f(t).diff(t) + f(t)

# resolvendo a equacao
sp.dsolve(eq, f(t))
```

Pronto! Está aí a solução da equação diferencial, já com a constante de integração. Vamos resolver agora uma equação de segunda ordem, $$g''(t)-2g'(t)+g(t)=0.$$

```python
import sympy as sp
t = sp.symbols('t')
g = sp.symbols('g', cls=sp.Function)
eq=g(t).diff(t,t) -2*g(t).diff(t) +g(t)
sp.dsolve(eq, g(t))
```

Vamos piorar um pouco a equação diferencial e tentar resolver $$q''(t)-6q'(t)+2q(t)-t\cos(t)=0.$$

```python
import sympy as sp
t = sp.symbols('t')
q = sp.symbols('q', cls=sp.Function)
eq=q(t).diff(t,t) -6*q(t).diff(t) +2*q(t)-t*sp.cos(t)
sp.dsolve(eq, q(t))
```

Para resolver adicionando condições iniciais, podemos complementar o comando. Vamos fazer isso, só para poder plotar um gráfico - adoramos gráficos. A condição inicial $q(0)=1$, $q'(0)=1$ é entrada como a opção ```ics={q(0): 1, q(t).diff(t).subs(t, 0): 1}``` dentro do ```dsolve```.

```python
import sympy as sp
t = sp.symbols('t')
q = sp.symbols('q', cls=sp.Function)
eq=q(t).diff(t,t) -6*q(t).diff(t) +2*q(t)-t*sp.cos(t)

# encontrando a solucao e armazenando no nome "solucao"
solucao=sp.dsolve(eq, q(t), ics={q(0): 1, q(t).diff(t).subs(t, 0): 1})

sp.plot(solucao.rhs,(t,-2,2),ylim=[-5,5])
```

## Sistemas de equações diferenciais

Para resolver/plotar sistemas de equações diferenciais, até podemos usar o SymPy, mas é muito mais eficiente usar o NumPy (ou uma mistura dele com o SciPy). Parte do código abaixo foi inspirado [desse site](https://danielmuellerkomorowska.com/2021/02/11/differential-equations-in-python-with-scipy/). Vamos usar como exemplo o que é, talvez, o sistema de equações diferenciais cujo retrato de fase ilustra mais livros em todo o mundo!

O sistema de Lorenz é definido como
$$\left\{
\begin{array}{lcl}
\dot x&=&\sigma (y-x),\\
\dot y&=& x(\rho-z)-y,\\
\dot z&=&xy-\beta z,
\end{array}
\right.
$$
onde $\sigma,\rho,\beta$ são parâmetros. O sistema de Lorenz é uma simplificação de um modelo atmosférico. Os valores mais utilizados desses parâmetros, como feito pelo próprio Lorenz, são $\sigma=10$, $\beta=8/3$ e $\rho=28$.

O comando base para resolver sistemas de equações é o ```odeint```, mas ele tem alguns detalhes.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint

def lorenz(state, t, sigma, beta, rho):
    x, y, z = state
    dx = sigma * (y - x)
    dy = x * (rho - z) - y
    dz = x * y - beta * z
    return [dx, dy, dz]

# valores dos parametros
sigma = 10.0
beta = 8.0 / 3.0
rho = 28.0

# parametros
p = (sigma, beta, rho)

# condicao inicial
ic = [1.0, 1.0, 1.0]

# tempo inicial, tempo final, tamanho da partição.
# se você colocar uma partição muito grande, então serão
# usados menos pontos para avaliar a função, e com isso
# terá a impressão de que são vários segmentos de reta.
# experimente um pouco com isso.
t = np.arange(0.0, 60.0, 0.001)

# integrando
result = odeint(lorenz, ic, t, p)

# plotando
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot(result[:, 0], result[:, 1], result[:, 2])

fig2 = plt.figure()
plt.plot(t,result[:, 0])
plt.plot(t,result[:, 1])
plt.plot(t,result[:, 2])
```

Soluções para sistemas bidimensionais também podem ser plotadas de forma muito similar. Como antes, vamos plotar a curva solução $(x(t),y(t))$ e depois, num mesmo gráfico, $x(t)$ e $y(t)$. Nosso eleito para o exemplo bidimensional é o oscilador de van der Pol.

A equação de segunda ordem $$x''-\mu(1-x^2)x'+x=0$$ é conhecida como oscilador de van der Pol, em homenagem ao matemático holandês Balthasar van der Pol (ok, ele era engenheiro e físico, mas acho que não ficaria triste em ser chamado de matemático) que a deduziu na década de 20, enquanto trabalhava na Philips com circuitos elétricos (veja detalhes [aqui](https://en.wikipedia.org/wiki/Van_der_Pol_oscillator) na Wikipedia). Essa equação de segunda ordem pode ser escrita como um sistema de equações de primeira ordem:
$$\left\{
\begin{array}{lcl}
\dot x&=&\mu(1-y^2)x-y,\\
\dot y&=& x.\\
\end{array}
\right.
$$

É fato conhecido que o oscilador de van der Pol admite um ciclo limite (uma órbita periódica atratora) para certos valores do parâmetro $\mu$. Vejamos como isso acontece ao fazer o gráfico: a solução começa em um certo ponto inicial e depois tende a uma curva fechada. Essa curva não pode ser expressada em termos algébricos, ela é uma curva transcendente. Vamos ao código:

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint

# define o sistema de van der pol
def sistema(variaveis, t):
    x, y = variaveis
    dx = 3*(1-y**2)*x-y
    dy = x
    return [dx, dy]

# define a condicao inicial, onde
# ic=[ x(0), y(0)]
ic = [0.1,0.2]

# define o tempo - será usado pelo integrador e também
# pelo plot
t = np.arange(0, 40, 0.0001)

# integrando
result = odeint(sistema, ic, t)

# primeiro plot
fig = plt.figure()
ax = plt.axes()
# plotando a curva-solucao
ax.plot(result[:, 0], result[:, 1])

# agora plotando x(t) e y(t) separadamente num segundo plot
fig2=plt.figure()
plt.plot(t,result[:, 0])
plt.plot(t,result[:, 1])
```

Note como no sistema de van der Pol, o plot das soluções $x(t),y(t)$ a partir de um certo $t$ parece o de uma função periódica, o que não acontece no sistema de Lorenz. Isso é justificado pela presença de um ciclo limite atrator no sistema de van der Pol, enquanto o sistema de Lorenz apresenta comportamento caótico.

Acima usamos o ```odeint``` para encontrar a solução de forma numérica antes de plotá-la, e o plot era sempre com $t\in[0,t_0]$, com $t_0>0$. O motivo para isso é que o ```odeint``` não funciona bem com tempos negativos. Uma possibilidade para resolver isso é usar o ```solve_ivp```. A página com manual dele [pode ser vista nesse link](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html), e é bem detalhada. Vamos resolver novamente o oscilador de van der Pol, agora com o ```solve_ivp```:

```python
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt

# note que agora que queremos usar o solve_ivp precisamos
# colocar o t como primeira variável, ao contrário de quando
# queríamos usar o odeint.
def sistema(t, variaveis):
    x, y = variaveis
    dx = 3*(1-y**2)*x-y
    dy = x
    return [dx, dy]

# condicao inicial
ic=[0.1,0.2]

# basicamente a sintaxe eh solve_ivp(sistema [t0,t1], ic). adicionamos
# a opcao dense_output para ficar mais facil recuperar as solucoes para
# que sejam plotadas.

solucao = solve_ivp(sistema, [-40, 40], ic,dense_output=True)

# agora discretizamos o tempo para plotar. é preciso ser compatível com
# o intervalo do tempo que foi passado para o comando solve_ivp
t = np.arange(-40, 40, 0.0001)

# a propriedade sol(t) da solucao carrega em suas colunas os valores
# de x(t) e y(t). para usar no plot, precisamos que isso esteja nas linhas,
# por isso usamos o .T no final, para calcular a transposta. isso vai plotar
# as solucoes x(t), y(t)
plt.plot(t,solucao.sol(t).T)

# agora criamos outra janela gráfica e plotamos a curva parametrizada.
# usamos [:,0] para acessar a primeira linha da matriz solucao.sol(t)
# e [:,1] para acessar a segunda linha. isso vai produzir um plot de
# curva parametrizada, como já sabemos fazer.
fig=plt.figure()
plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])
```

Compare os gráficos e as curvas: são bem parecidas! (Claro!!)

## Retratos de fase e campos de direções

Quando estamos estudando o comportamento de uma função $y=f(x)$, analisar o gráfico da função é um bom ponto de partida - e vimos como fazer gráficos no Python lá no começo desse tutorial.

No caso de equações diferenciais e PVIs, podemos encontrar e plotar a solução, seja na forma de curva parametrizada, seja em termos das coordenadas.

Já para sistemas de equações diferenciais, duas representações gráficas são muito eficientes:

1. o retrato de fase do sistema, que basicamente é o conjunto de todas as soluções, e pode ser representado graficamente/computacionalmente fazendo o esboço de "algumas" curvas-solução (como curvas parametrizadas), de modo a representar o comportamento gloal do sistema e
2. o campo de direções, que é uma representação gráfica do sistema de equações diferenciais (ordinárias e autônomas, claro) como o campo vetorial adjacente, ou seja, se o sistema tem a forma
$$\left\{
\begin{array}{lcl}
\dot x&=&P(x,y),\\
\dot y&=&Q(x,y)
\end{array}
\right.$$
então construímos o campo vetorial $X(x,y)=(P(x,y),Q(x,y))$ e em cada ponto $(a,b)$ do plano cartesiano colocamos o vetor $X(a,b)$. Como o campo vetorial $X$ é tangente às soluções da equação diferencial, isso nos dará uma boa ideia sobre o comportamento qualitativo das soluções da equação.

Para o retrato de fase, já sabemos o que fazer: basta colocar, num mesmo sistema de eixos, várias soluções (ou seja, soluções passando por várias condições iniciais). Isso pode ser feito com um ```for``` para pegar várias condições iniciais e ir plotando, como fazemos no comando abaixo. Confira [esse site aqui](http://www.doc.mmu.ac.uk/STAFF/S.Lynch/DSAP_Jupyter_Notebook.html) para mais alguns exemplos.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint

fig = plt.figure(num=1)
ax=fig.add_subplot(111)

def sistema(t, variaveis):
    x, y = variaveis
    dx = (1-y**2)*x-y
    dy = x
    return [dx, dy]

# trajetórias em tempo positivo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,10], [P,Q],dense_output=True)
    t = np.linspace(0, 10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# trajetórias em tempo negativo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,-10], [P,Q],dense_output=True)
    t = np.linspace(0, -10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# limita a janela de visualização e mostra o plot
plt.xlim(-5,5)
plt.ylim(-5,5)
plt.show()
```

Vamos a mais dois exemplo, agora considerando um sistema de equações lineares: primeiro o sistema
$$\left\{
\begin{array}{lcl}
\dot x&=&-y,\\
\dot y&=&x,
\end{array}
\right.
$$ que só tem soluções periódicas, e depois o sistema
e depois o sistema
$$\left\{
\begin{array}{lcl}
\dot x&=&y,\\
\dot y&=&x,
\end{array}
\right.
$$ que tem um ponto de sela.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint

fig = plt.figure(num=1)
ax=fig.add_subplot(111)

# um sistema que só tem órbitas periódicas
def sistema(t, variaveis):
    x, y = variaveis
    dx = -y
    dy = x
    return [dx, dy]

# trajetórias em tempo positivo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,10], [P,Q],dense_output=True)
    t = np.linspace(0, 10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# trajetórias em tempo negativo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,-10], [P,Q],dense_output=True)
    t = np.linspace(0, -10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# limita a janela de visualização e mostra o plot
plt.xlim(-5,5)
plt.ylim(-5,5)
plt.show()
```

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
from scipy.integrate import solve_ivp

fig = plt.figure(num=1)
ax=fig.add_subplot(111)

# um sistema que só tem órbitas periódicas
def sistema(t, variaveis):
    x, y = variaveis
    dx = y
    dy = x
    return [dx, dy]

# trajetórias em tempo positivo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,10], [P,Q],dense_output=True)
    t = np.linspace(0, 10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# trajetórias em tempo negativo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,-10], [P,Q],dense_output=True)
    t = np.linspace(0, -10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# limita a janela de visualização e mostra o plot
plt.xlim(-5,5)
plt.ylim(-5,5)
plt.show()
```

Por fim, um exemplo com MUITAS órbitas traçadas, e que possui tanto uma órbita homoclínica quanto algumas órbitas periódicas.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint

fig = plt.figure(num=1)
ax=fig.add_subplot(111)

# um sistema hamiltoniano com órbitas periódicas e uma órbita homoclínica
def sistema(t, variaveis):
    x, y = variaveis
    dx = -y
    dy = -x-x**2
    return [dx, dy]

# trajetórias em tempo positivo
for P in np.linspace(-2, 2, 20):
  for Q in np.linspace(-2, 2, 20):
    solucao = solve_ivp(sistema, [0,10], [P,Q],dense_output=True)
    t = np.linspace(0, 10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# trajetórias em tempo negativo
for P in np.linspace(-2, 2, 20):
  for Q in np.linspace(-2, 2, 20):
    solucao = solve_ivp(sistema, [0,-10], [P,Q],dense_output=True)
    t = np.linspace(0, -10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# limita a janela de visualização e mostra o plot
plt.xlim(-2,2)
plt.ylim(-2,2)
plt.show()
```

Agora que já sabemos plotar retratos de fase, vamos plotar campos de direções, e depois os dois juntos. Vamos usar como base a sela linear. O comando para plotar o campo vetorial é o ```quiver```, dentro do matplotlib. A sintaxe é bem simples.

```python
import matplotlib.pyplot as plt
import numpy as np

# criamos uma malha em R2, com x e y indo de -5 a 5, contendo
# um total de 10 pontos em cada coordenada
x,y = np.meshgrid( np.linspace(-5,5,10),np.linspace(-5,5,10) )

# calculamos o campo vetorial X(x,y)=(y,x) na malha
u = y
v = x

#N = np.sqrt(u**2+v**2)
#U2, V2 = u/N, v/N

# plotando
plt.quiver( x,y,u, v)
```

Em alguns casos, pode ser uma boa estratégia normalizar o campo vetorial para que a figura fique melhor. Isso obviamente deforma o campo vetorial, mas para visualização inicial, pode ser útil:

```python
import matplotlib.pyplot as plt
import numpy as np

# criamos uma malha em R2, com x e y indo de -5 a 5, contendo
# um total de 10 pontos em cada coordenada
x,y = np.meshgrid( np.linspace(-5,5,10),np.linspace(-5,5,10) )

# calculamos o campo vetorial X(x,y)=(y,x) na malha
u = y
v = x

# normalizando o campo
N = np.sqrt(u**2+v**2)
U, V = u/N, v/N

# plotando
plt.quiver( x,y,U, V)
```

Agora nosso *gran finale*: vamos plotar o campo de direções e o retrato de fase num só gráfico. Isso nos dá muita informação sobre o sistema dinâmico.

O código abaixo é o melhor que eu sei fazer, mas provavelmente deve existir uma forma mais eficiente: em particular, eu preciso definir separadamente tanto o sistema de equações diferenciais quanto o campo vetorial e isso não é a forma mais inteligente de fazer isso.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint

fig = plt.figure(num=1)
ax=fig.add_subplot(111)

# um sistema que só tem órbitas periódicas
def sistema(t, variaveis):
    x, y = variaveis
    dx = y
    dy = x
    return [dx, dy]

# calculando e plotando trajetórias em tempo positivo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,10], [P,Q],dense_output=True)
    t = np.linspace(0, 10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# calculando e plotando trajetórias em tempo negativo
for P in range(-5,5,1):
  for Q in range(-5,5,1):
    solucao = solve_ivp(sistema, [0,-10], [P,Q],dense_output=True)
    t = np.linspace(0, -10, 500)
    plt.plot(solucao.sol(t).T[:, 0],solucao.sol(t).T[:, 1])

# grid para os vetores
x,y = np.meshgrid( np.linspace(-5,5,20),np.linspace(-5,5,20) )

# calculando o campo vetorial X(x,y)=(y,x) na malha
u = y
v = x

# normalizando o campo (opcional)
N = np.sqrt(u**2+v**2)
u, v = u/N, v/N

# plotando o campo de direcoes
plt.quiver( x,y,u, v)

# limita a janela de visualização e mostra o plot
plt.xlim(-5,5)
plt.ylim(-5,5)
plt.show()
```

## EDPs

Se obter de forma explícita soluções para EDOs não é uma coisa simples, para EDPs se torna uma tarefa impossível. Então já partiremos para soluções numéricas desde o começo. Vamos estudar como obter a solução da equação do calor, e nosso objetivo será obter uma representação gráfica animada da evolução da equação (ou seja, como o "calor" se espalha numa região).

Os códigos abaixo foram adaptados dos códigos originais [daqui](https://levelup.gitconnected.com/solving-2d-heat-equation-numerically-using-python-3334004aa01a) e [daqui](http://firsttimeprogrammer.blogspot.com/2015/07/the-heat-equation-python-implementation.html).

Vamos começar com a equação do calor em dimensão 1. A ideia é simples: considere uma barra de ferro (ou outro material) que está fixada pelos dois extremos. Suponha que existe uma distribuição inicial de temperatura. Com o passar do tempo, como a temperatura evolui?

A solução do código abaixo é um pouco fake: nós já pegamos uma animação da solução com o passar do tempo e plotamos.

```python
%matplotlib inline
%matplotlib tk

import numpy as np
from numpy import pi
import matplotlib.pyplot as plt
import matplotlib.animation as animation

fig = plt.figure()
fig.set_dpi(100)
ax1 = fig.add_subplot(1,1,1)

#Diffusion constant
k = 2

#Scaling factor (for visualisation purposes)
scale = 5

#Length of the rod (0,L) on the x axis
L = pi

#Initial contitions u(0,t) = u(L,t) = 0. Temperature at x=0 and x=L is fixed
x0 = np.linspace(0,L+1,10000)
t0 = 0
temp0 = 5  #Temperature of the rod at rest (before heating)

#Increment
dt = 0.01

# Solução da equação do calor
def u(x,t):
    return temp0 + scale*np.exp(-k*t)*np.sin(x)

#Gradient of u
def grad_u(x,t):
                               #du/dx              #du/dt
    return scale*np.array([np.exp(-k*t)*np.cos(x),-k*np.exp(-k*t)*np.sin(x)])

a = []
t = []

for i in range(500):
    value = u(x0,t0) + grad_u(x0,t0)[1]*dt
    t.append(t0)
    t0 = t0 + dt
    a.append(value)

k = 0
def animate(i):         #The plot shows the temperature evolving with time
    global k            #at each point x in the rod
    x = a[k]            #The ends of the rod are kept at temperature temp0
    k += 1              #The rod is heated in one spot, then it cools down
    ax1.clear()
    plt.plot(x0,x,color='red',label='Temperatura em cada ponto x')
    plt.plot(0,0,color='red',label='Tempo decorrido '+str(round(t[k],2)))
    plt.grid(True)
    plt.ylim([temp0-2,2.5*scale])
    plt.xlim([0,L])
    plt.title('Evolução da equação do calor')
    plt.legend()

anim = animation.FuncAnimation(fig,animate,frames=360,interval=10)
plt.show()

```

É muito mais complicado resolver a equação do calor em umd domínio bidimensional. O código abaixo resolve numericamente a equação e plota o resultado numa animação.

```python
%matplotlib inline
%matplotlib tk

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from matplotlib.animation import FuncAnimation

print("2D heat equation solver")

plate_length = 50
max_iter_time = 750

alpha = 2
delta_x = 1

delta_t = (delta_x ** 2)/(4 * alpha)
gamma = (alpha * delta_t) / (delta_x ** 2)

# Initialize solution: the grid of u(k, i, j)
u = np.empty((max_iter_time, plate_length, plate_length))

# Initial condition everywhere inside the grid
u_initial = 0

# Boundary conditions
u_top = 100.0
u_left = 0.0
u_bottom = 0.0
u_right = 0.0

# Set the initial condition
u.fill(u_initial)

# Set the boundary conditions
u[:, (plate_length-1):, :] = u_top
u[:, :, :1] = u_left
u[:, :1, 1:] = u_bottom
u[:, :, (plate_length-1):] = u_right

def calculate(u):
    for k in range(0, max_iter_time-1, 1):
        for i in range(1, plate_length-1, delta_x):
            for j in range(1, plate_length-1, delta_x):
                u[k + 1, i, j] = gamma * (u[k][i+1][j] + u[k][i-1][j] + u[k][i][j+1] + u[k][i][j-1] - 4*u[k][i][j]) + u[k][i][j]

    return u

def plotheatmap(u_k, k):
    plt.clf()

    plt.title(f"Gráfico da temperatura no instante t = {k*delta_t:.3f} unit time")
    plt.xlabel("x")
    plt.ylabel("y")

    # This is to plot u_k (u at time-step k)
    plt.pcolormesh(u_k, cmap=plt.cm.jet, vmin=0, vmax=100)
    plt.colorbar()

    return plt

# Do the calculation here
u = calculate(u)

def animate(k):
    plotheatmap(u[k], k)

anim = animation.FuncAnimation(plt.figure(), animate, interval=1, frames=max_iter_time, repeat=False)

```


Solution to: [Day 4: Geometric Distribution II](https://www.hackerrank.com/challenges/s10-geometric-distribution-2/problem)

Table of Contents

- Table of Contents
- Math Solution
- Solution
    - Imports
    - Input
    - Geometric Distribution
    - Format
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Math Solution

The probability that a machine produces a defective product is 1/3. What is the probability that the 1st defect is found during the first 5 inspections?

There are two methods to solve this:
- use a summation of the geom_prob() function
- or, because the trials are independent, we can instead calculate the complement of the success rate to the power of n

\begin{equation}
= 1 - (1 - \frac{1}{3}) ^ {5}
\end{equation}

# Solution

## Imports

```python
from typing import Tuple
```

## Input

```python
def get_input() -> Tuple[float, int]:
	"""Return input for Day 4: Geom Dist II.

	Returns:
		Tuple[float, int]: p and n, respectively
	"""
	num, denom = [int(x) for x in input().split()]
	p = num/denom
	n = int(input())
	return (p, n)
```

## Geometric Distribution

```python
def calc_geom_dist(n: int, p: float) -> float:
	"""Returns probability mass of geom distribution for n, p

	Args:
		n (int): number of trials
		p (float): p(x)

	Returns:
		float: Probability of geom dist given n, p
	"""
	q = 1 - p
	return q ** (n-1) * p
```

```python
def get_cum_geom_prob(n: int, p: float) -> float:
	"""Returns cumulative probability of geometric distribution in n trials

	Args:
		n (int): total number of trials
		p (float): P(x)

	Returns:
		float: Cumulative probability of x in n trials
	"""
	cum_prob = 0

	for i in range(1, n + 1):
		geom_prob = calc_geom_dist(i, p)
		cum_prob += geom_prob
	return cum_prob
```

## Format

```python
def format_scale(num: float) -> float:
	"""Returns number with decimals formatted to scale

	Args:
		num (float): Number to format

	Returns:
		float: Number formatted to scale
	"""
	return f'{num:.3f}'
```

## Main

```python
def main():
	p, n = get_input()

	cum_prob = get_cum_geom_prob(n, p)
	print(format_scale(cum_prob))
```

```python
if __name__ == "__main__":
	main()
```

    1 3
    5
    0.868


# <center> ЛАБОРАТОРНАЯ  РАБОТА 4 </center>
# <center> РЕШЕНИЕ  СИСТЕМ НЕЛИНЕЙНЫХ  УРАВНЕНИЙ </center>

Теоретический материал к данной теме содержится в [1, глава 7].

```python
from sympy.solvers import solve
from sympy import exp, sin, cos, sqrt, log, ln, pi
from sympy import Rational as syR
from sympy import Symbol, diff
from scipy.misc import derivative

import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline
```

## Задача 1
Найти с точностью $\epsilon = 10^{-6}$ все корни системы нелинейных уравнений:
$$f_1(x_1, x_2)=0,$$
$$f_2(x_1, x_2)=0$$
используя метод Ньютона для системы нелинейных уравнений. Найти корни с помощью встроенного блока решения уравнений.
## Порядок решения задачи
1. Локализовать корни системы уравнений графически.
2. Составить программу-функцию, вычисляющую корень системы двух нелинейных уравнений по методу Ньютона с точностью $\epsilon$. Предусмотреть подсчет количества итераций.
3. Используя составленную программу, вычислить все корни заданной системы с точностью $\epsilon$.
4. Используя встроенный блок, найти все корни системы с точностью $\epsilon$. Сравнить с результатами, полученнными в п. 3.

### УКАЗАНИЕ.
В п. 1 привеcти уравнения системы к виду $x_2=g_i(x_1)$(либо $x_1=g_i(x_2)$), $i=1, 2$.

Ваша система уравнений:
$f1(x1,x2) = {Task1[TASK_VARIANT][f1_latex]}$
$f2(x1,x2) = {Task1[TASK_VARIANT][f2_latex]}$

```python
epsilon1 = 10**(-6)
# Функции
f1 = {Task1[TASK_VARIANT][f1_latex]}
f2 = {Task1[TASK_VARIANT][f2_latex]}
```

Локализуйте корни уравнения графически, используя пакет matplotlib

```python
##график
```

Реализуйте функцию, вычисляющую корень системы двух нелинейных уравнений по методу Ньютона с точностью $\varepsilon$. Не забудьте предусмотерть подсчет количества итераций.

```python
def meth_newt(x,y, epsilon):

```

Вычислите корни заданной системы с нужной точностью.

```python
##два ответа с корнями
```

Проверьте ответ, полученный Вашей программой, с ответом, полученной встроенной функцией (например, nsolve).

```python

```

#### ЛИТЕРАТУРА:
1. Амосов А.А., Дубинский Ю.А., Копченова Н.В. Вычислительные методы для инженеров. М.: Высшая школа, 1994.

```python

```