# Resolução da DFT e adição de zeros

Neste notebook avaliaremos aspectos de resolução da DFT e da adição de zeros (zero padding).

Notamos que um sinal com $N$ amostras, amostrado a uma taxa $F_s$ {Hz] possuirá um vetor de frequências

\begin{equation}
f_{Hz}^{(k)} = \left\{\frac{0F_{s}}{N}, \ \frac{1F_{s}}{N}, \  \frac{2F_{s}}{N}, \ ... \ \frac{(N-1)F_{s}}{N}\right\}
\end{equation}

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
import IPython.display as ipd # to play signals
import sounddevice as sd
import soundfile as sf
```

## Vamos criar uma resposta ao impulso ideal

Esta é a resposta ao impulso de um sistema massa-mola (1DOF)

\begin{equation}
h(t) = \frac{A}{\omega_d} \mathrm{e}^{-\zeta \omega_n t} \mathrm{sin}(\omega_d t)
\end{equation}
com $\zeta$ sendo o amortecimento, $\omega_n = 2 \pi f_1$ e $\omega_d = \omega_n\sqrt{1-\zeta^2}$, que possui TF, dada por

\begin{equation}
H(\mathrm{j}\omega) = \frac{A}{\omega_n^2 - \omega^2+ \mathrm{j}2\zeta\omega_n\omega}
\end{equation}

```python
# Define the sampling rate fs = 100 Hz and total record time T = 5 seconds
Fs=100
T=5
# time vector
time = np.arange(0, T, 1/Fs) # Ts = 0.01 [s]

# Sinal puro (Fenômeno) - original - h(t) de um S1GL (massa-mola)
A=200
zeta=0.3
wn=2*np.pi*10
wd=np.sqrt(1-zeta**2)*wn
# Impulse response
ht=(A/wd)*np.exp(-zeta*wn*time)*np.sin(wd*time)
# FRF teorica
freq = np.arange(0, Fs, 0.1)

Hw=A/(wn**2 - (2*np.pi*freq)**2+ 1j*2*zeta*wn*(2*np.pi*freq));

plt.figure(figsize=(12,4))
plt.subplot(1,2,1)
plt.title(r'$h(t)$ - (fenômeno original)')
plt.plot(time, ht, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, time[-1]))

plt.subplot(1,2,2)
plt.title(r'$|H(j\omega)|$ - (fenômeno original)')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs))
plt.ylim((-80, -20))
plt.tight_layout()
plt.show()
```

## Adicionamos ruído a resposta ao impulso (fenômeno ideal)

Tipicamente, é isso que acontece em uma medição. Ela sempre estará contaminada por ruído em algum grau. O ruído vai ter característica de ruído branco (distribuição normal com média zero e uma variância).

```python
noise = np.random.normal(loc = 0, scale = 0.01, size = len(time))
ht_med = ht+noise
```

```python
plt.figure()
plt.title(r'$h(t)$ - (fenômeno original vs. medido)')
plt.plot(time, ht, '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(time, ht_med, '-r', linewidth = 1, label = 'Fenômeno medido')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, time[-1]));
```

## Agora podemos investigar a FFT e sua resolução para alguns casos

### 1. Resolução original

"If n is not given, the length of the input along the axis specified by axis is used."

Se n não for dado, to comprimento da entrada (ao longo do eixo especificado) é usado - Em outras palavras, o espectro tem o mesmo número de pontos que o sinal no domínio do tempo.

```python
N = len(ht) # Número de pontos da FFT
Hw_med = np.fft.fft(ht_med)
freq_vec = np.linspace(0, (N-1)*Fs/N, N)
print("Vetor de frequência: {:.2f}, {:.2f}, {:.2f}, ... {:.2f}".format(freq_vec[0], freq_vec[1], freq_vec[2], freq_vec[-1]))
print("Resolução do espectro: {:.2f} [Hz]".format(freq_vec[1]-freq_vec[0]))

plt.figure(figsize=(10,6))
plt.title(r'$|H(j\omega)|$')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(freq_vec, 20*np.log10(np.abs(Hw_med)/Fs), '-r', linewidth = 1, label = 'Fenômeno medido')
plt.legend(loc = 'lower left')
plt.axvline(Fs/2, color='grey',linestyle = '--', linewidth = 4, alpha = 0.4)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs))
plt.ylim((-80, -20))
plt.tight_layout()
#plt.savefig('dft_mag_par.pdf')
plt.show()
```

## Cheque o teorema de Parseval

\begin{equation}
\sum\limits_{n=-\infty}^{\infty}|x[n]|^{2} = \frac{1}{N} \sum\limits_{k=-\infty}^{\infty}|X(k)|^{2}
\end{equation}

```python
# Domínio do tempo
Et = np.sum(ht_med**2)
# Domínio da frequência
Ef = np.sum(np.abs(Hw_med)**2)

print("A energia no domínio do tempo é {:.2f}".format(Et))
print("A energia no domínio da frequência é {:.2f}".format(Ef/N))
```

    A energia no domínio do tempo é 13.45
    A energia no domínio da frequência é 13.45

### 2. Sinal truncado

Agora vamos truncar $h(t)$. Note que o ruído tende a dominar a resposta ao impulso a partir de $0.4$ [s]

"If n is smaller than the length of the input, the input is cropped."

Se n for menor que que o comprimento do sinal, o sinal será truncado. É indiferente fazer:

a. Truncar o sinal no tempo:
- ht_med_trunc = ht_med[time<=0.4]
- Hw_med_trunc = np.fft.fft(ht_med_trunc)

b. Passar Nt como o segundo argumento da FFT
- Hw_med_trunc = np.fft.fft(ht_med, Nt)

```python
ht_med_trunc = ht_med[time<=0.4]

Nt = len(time[time<=0.4]) # Número de pontos da FFT
# Hw_med_trunc = np.fft.fft(ht_med_trunc) # Tente esta linha depois
Hw_med_trunc = np.fft.fft(ht_med, Nt)
freq_vec_t = np.linspace(0, (Nt-1)*Fs/Nt, Nt)
print("O número de pontos na DFT é {}".format(Nt))
print("Vetor de frequência: {:.2f}, {:.2f}, {:.2f}, ... {:.2f}".format(freq_vec_t[0], freq_vec_t[1],
    freq_vec_t[2], freq_vec_t[-1]))
print("Resolução do espectro: {:.2f} [Hz]".format(freq_vec_t[1]-freq_vec_t[0]))

plt.figure(figsize=(10,6))
plt.title(r'$|H(j\omega)|$ - (fenômeno original)')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(freq_vec_t, 20*np.log10(np.abs(Hw_med_trunc)/Fs), '-r', linewidth = 1, label = 'Fenômeno medido truncado')
plt.legend(loc = 'lower left')
plt.axvline(Fs/2, color='grey',linestyle = '--', linewidth = 4, alpha = 0.4)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs/2)) # Extenda até Fs se desejar
plt.ylim((-60, -20))
plt.tight_layout()
plt.show()
```

### 3. Sinal truncado e com zeros no final.

Agora vamos truncar $h(t)$ e adicionar zeros no final. Note que a resolução do espectro no caso anterior diminuiu (maior $\Delta f$).

"If n is larger, the input is padded with zeros."

Se n for maior que que o comprimento do sinal, o sinal será completado com zeros. É indiferente fazer:

a. Truncar o sinal no tempo:
- ht_med_trunc = ht_med[time<=0.4]
- Nt = len(time[time<=0.4])
- ht_med_zp = np.concatenate((ht_med_trunc, np.zeros(N-Nt)))
- Hw_med_zp = np.fft.fft(ht_med_zp) # Tente esta linha depois

b. Passar Nt como o segundo argumento da FFT
- Hw_med_zp = np.fft.fft(ht_med_trunc, N)

```python
ht_med_trunc = ht_med[time<=0.4]
Nt = len(time[time<=0.4])

ht_med_zp = np.concatenate((ht_med_trunc, np.zeros(N-Nt)))

#Hw_med_zp = np.fft.fft(ht_med_zp) # Tente esta linha depois
Hw_med_zp = np.fft.fft(ht_med_trunc, N)

freq_vec_zp = np.linspace(0, (N-1)*Fs/N, N)
print("O número de pontos na DFT é {}".format(len(ht_med_zp)))
print("Vetor de frequência: {:.2f}, {:.2f}, {:.2f}, ... {:.2f}".format(freq_vec_zp[0], freq_vec_zp[1],
    freq_vec_zp[2], freq_vec_zp[-1]))
print("Resolução do espectro: {:.2f} [Hz]".format(freq_vec_zp[1]-freq_vec_zp[0]))

plt.figure(figsize=(10,6))
plt.title(r'$|H(j\omega)|$ - (fenômeno original)')
plt.plot(freq, 20*np.log10(np.abs(Hw)), '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(freq_vec_zp, 20*np.log10(np.abs(Hw_med_zp)/Fs), '-r', linewidth = 1, label = 'Fenômeno med. trunc. comp. zeros')
plt.legend(loc = 'lower left')
plt.axvline(Fs/2, color='grey',linestyle = '--', linewidth = 4, alpha = 0.4)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel('Magnitude [dB]')
plt.xlim((0, Fs/2)) # Extenda até Fs se desejar
plt.ylim((-60, -20))
plt.tight_layout()
plt.show()
```

## Trabalhando na IFFT

Vamos tratar rapidamente da transformada inversa e recuperar $h(t)$, a partir do espectro do sinal truncado e completado com zeros.

```python
ht_rec = np.fft.ifft(Hw_med_zp) # ifft
N_rec = len(ht_rec) # Número de amostras
t_mrec = (N_rec-1)/Fs # tempo máximo da ifft
time_rec = np.linspace(0, t_mrec, N_rec) # vetor temporal recuperado da ifft

plt.figure()
plt.title(r'$h(t)$ - (fenômeno original vs. IFFT do medido)')
plt.plot(time, ht, '-b', linewidth = 2, label = 'Fenômeno original')
plt.plot(time_rec, ht_rec, '-r', linewidth = 1, label = 'Fenômeno med. trunc. comp. zeros')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, time[-1]));
```

### Inspecione ht_rec

Você vai notar que:

1. O sinal recuperado da IFFT tem uma pequena parte imaginária
2. Você precisa do espectro de $0$ a $F_s$ pra operar a IFFT corretamente.

```python
ht_rec
```

    array([ 3.03473634e-03-4.44089210e-19j,  1.57135236e+00+2.19269047e-17j,
            2.12465154e+00-6.51700915e-17j,  1.83876444e+00+3.07531778e-17j,
            1.07222624e+00-4.10326538e-17j,  1.93397750e-01-3.45487781e-17j,
           -4.74665289e-01+5.11553631e-18j, -7.83418685e-01+3.51304474e-17j,
           -7.33526554e-01+2.19443448e-17j, -4.70259620e-01-6.02252471e-18j,
           -1.43474479e-01-3.69773687e-17j,  1.15733472e-01+1.94508565e-17j,
            2.53649923e-01+1.38079301e-17j,  2.78635127e-01+1.86513978e-17j,
            1.85013369e-01-2.58488803e-17j,  7.18095183e-02+2.81220863e-18j,
           -1.48454576e-02-6.84186881e-18j, -9.47827683e-02-1.77983237e-17j,
           -9.96100019e-02-2.62776591e-17j, -9.44913691e-02+6.44431131e-18j,
           -3.01236414e-02+1.81815443e-17j,  1.41454977e-02-6.77124340e-18j,
            3.52049718e-02+2.22568781e-17j,  3.74063838e-02+1.41984806e-17j,
            2.27410337e-02-4.47826638e-17j,  2.45316427e-02+2.31801495e-18j,
            6.14179695e-03-3.64602687e-17j,  6.09361575e-03+9.55502480e-18j,
           -1.46307978e-02+2.10068252e-17j, -7.76829581e-03-1.72079678e-17j,
            2.92373209e-03+1.33171131e-17j, -3.84174250e-03-2.07897556e-17j,
           -3.75142945e-03-2.47233292e-17j,  6.07448275e-03-3.02423424e-17j,
            1.58512460e-02-6.42362947e-18j,  2.12363096e-02-6.47906880e-18j,
            2.36137125e-03+5.49647394e-18j, -4.99812340e-03+8.92287868e-18j,
           -1.41746782e-04+1.97094542e-17j,  4.34323607e-03+2.56991411e-17j,
            4.68851936e-03+1.33268321e-17j,  1.93949391e-17+1.35941541e-17j,
            1.44110928e-17-2.41953402e-17j, -2.16895365e-17-2.11359456e-17j,
            3.23660073e-17+7.26372982e-18j, -7.14582697e-18+3.74789039e-18j,
           -1.61356509e-17+4.90110362e-18j,  1.26553712e-17-3.67115732e-18j,
           -5.68369543e-19-6.46577611e-18j, -1.61814385e-17-1.95162973e-17j,
            2.40867200e-17+6.93345143e-18j, -3.93618182e-17-3.03449884e-18j,
            1.79660887e-17-9.23198636e-18j,  5.39504176e-18-1.80402300e-17j,
            4.40324057e-17-5.05823316e-18j,  2.53458298e-17-8.75945661e-18j,
           -7.88724984e-18+7.31161266e-18j, -2.99023373e-17-1.85176889e-17j,
           -3.89350084e-17-2.81215739e-17j, -3.55305865e-17-3.20192125e-17j,
           -1.08563958e-17+2.18044771e-19j, -1.32324453e-17+3.20621735e-17j,
            1.76236574e-17+3.79333123e-17j, -4.60752290e-17-3.07638100e-17j,
            5.53162842e-17+2.10895179e-17j,  3.94136988e-17-2.06016506e-17j,
            2.96593123e-17-6.27856759e-17j,  2.47728089e-17+3.20291929e-17j,
            1.89060109e-17+2.08007475e-17j, -2.51487625e-17-1.47162809e-17j,
           -1.00224844e-17+1.31565096e-17j, -2.07513184e-17-2.10825447e-17j,
            2.07335966e-17-6.31795905e-17j,  1.39443699e-18-2.44673938e-17j,
           -2.09374895e-17+1.93964997e-17j, -1.78210232e-17-2.25708577e-17j,
            3.08099087e-17+4.76010326e-17j, -1.42259548e-17-1.84998472e-17j,
           -2.03893537e-18+5.80106862e-17j,  3.91697248e-17+1.49731376e-17j,
           -1.20913641e-17-3.80086239e-18j, -2.96196372e-17+2.11152629e-17j,
           -3.15080291e-17-9.54348889e-17j, -2.73025787e-17+5.58313090e-17j,
            1.79580762e-17+2.89424083e-18j,  6.29795065e-19-1.42136132e-17j,
            1.49716614e-17-2.72970632e-18j,  3.92732353e-17+7.42739006e-18j,
            4.33436509e-17-7.54679250e-18j, -3.80140979e-17-1.17962472e-17j,
           -3.33046196e-17+1.59618139e-17j, -7.22579868e-18-1.61713662e-17j,
            1.61768056e-17-3.33774780e-17j,  1.13875477e-17+1.66321843e-17j,
            3.00673592e-17-4.02597917e-17j, -4.32070998e-18-8.72287755e-18j,
            5.97164713e-18-2.26354739e-18j,  1.98437645e-17+2.02425262e-17j,
            1.55063745e-17-1.99665739e-18j,  2.52020735e-17+4.03680170e-17j,
           -9.47302283e-19-3.29295747e-18j,  1.83436696e-17-5.43149608e-17j,
           -1.37605572e-16+9.00947044e-17j, -2.58814299e-18-3.14710085e-18j,
            7.20190673e-17-2.47922771e-17j, -2.91545180e-18+5.82078739e-18j,
            6.46101995e-17-5.31639151e-18j, -9.31532359e-18+4.61166827e-17j,
           -2.14607774e-17-1.76840775e-17j, -5.28634198e-17+1.39818742e-17j,
           -2.43644854e-17-7.21459470e-18j,  1.86452185e-17-1.93462963e-17j,
            4.79308169e-17-3.15106123e-17j,  2.90419188e-17-6.00480838e-19j,
           -5.88960675e-18+2.51738341e-17j,  1.66061434e-17-2.88888539e-18j,
            1.74096520e-17-1.58984085e-17j, -9.30716218e-18+4.58798806e-17j,
           -4.18635837e-17-2.05645626e-18j, -1.05396320e-17-1.55339882e-17j,
           -1.50824952e-17+2.08284482e-17j, -1.09982377e-17+2.97485005e-18j,
            4.56081460e-17-8.82012200e-17j,  2.10783399e-17+1.19253188e-17j,
           -1.03736415e-17-8.66146288e-18j, -3.18800172e-18+7.02599744e-18j,
            6.87860077e-17-7.89165097e-18j, -8.87762988e-17-1.01708767e-16j,
            5.96827069e-17-7.98954685e-17j, -2.14788472e-17+1.07201292e-17j,
           -2.20258062e-17-1.61257194e-17j,  1.93020569e-17+8.34111758e-18j,
            2.29922116e-17+4.48062895e-17j, -1.38814270e-17+1.68253786e-17j,
           -2.75595431e-18+2.16453811e-17j, -5.70997963e-18+3.47193272e-18j,
           -1.60374492e-18+2.58587155e-17j,  6.48340088e-18-4.35431300e-17j,
            3.90155760e-17+3.76315079e-19j,  1.24035804e-18+1.02316125e-17j,
           -1.05101155e-17-1.58916984e-17j,  3.02923963e-17-3.68641160e-17j,
            4.32273194e-17+6.33886994e-17j, -5.12971379e-17-3.54331897e-18j,
            6.08765062e-17+1.96153002e-17j, -1.15197228e-17-5.61823420e-18j,
           -1.09722281e-17+5.29353961e-17j, -5.62849246e-18-3.00268992e-17j,
            7.22272414e-18+1.95492052e-17j, -2.46026853e-17+4.64760734e-18j,
            3.78354987e-17-8.70330791e-18j, -1.31698721e-18-2.91415106e-17j,
            2.71009997e-17-2.30431872e-18j, -3.90242141e-17-5.31996557e-17j,
            3.30036355e-17+4.00646773e-17j, -4.48600240e-17-3.28542889e-18j,
            4.95022180e-18-3.02436429e-18j, -1.28402142e-17+4.13088538e-18j,
           -1.10239823e-17+1.53219852e-17j,  6.97197088e-17+2.79079771e-17j,
            2.73072646e-18+2.27022083e-17j,  5.28406143e-17-1.57695140e-17j,
           -5.32322367e-18-1.45450694e-17j, -1.02409197e-17+1.05033035e-17j,
            9.48317210e-18+1.04410957e-17j, -1.18364000e-17+2.03419417e-17j,
           -3.83412942e-17+3.11215589e-18j,  2.32550170e-18+1.99259417e-17j,
            1.22913507e-17-5.57794309e-17j,  2.55950182e-17+3.78743521e-17j,
            2.88885492e-18-2.25457235e-17j,  1.61504539e-17+1.05429727e-17j,
           -4.10792645e-17-2.30701721e-17j, -2.91978965e-17-1.35669838e-17j,
            1.86810597e-17+9.03753454e-18j, -1.88022339e-17-1.86941382e-17j,
            4.42513924e-18+1.88734787e-17j, -3.08822933e-17+2.17835379e-17j,
           -4.13020070e-17-1.09838321e-17j,  1.75308646e-17-2.18338128e-17j,
           -3.37106915e-19-3.53156920e-18j, -2.05409562e-17+5.36783814e-17j,
            2.86393893e-17-1.23650824e-17j,  3.98532764e-17-4.25144873e-18j,
           -4.77942829e-17-2.99079535e-17j, -3.06541729e-18+1.37260277e-17j,
            3.79094770e-17-3.78221738e-17j,  3.11386766e-17-1.42205689e-17j,
            1.56994699e-18+8.99832038e-18j,  1.01438728e-17-1.15288812e-17j,
           -2.24919325e-17-2.08719426e-18j, -1.01941316e-17+1.61148098e-17j,
            4.47805625e-18+4.69405142e-17j, -1.21649453e-17-2.18414123e-17j,
            3.07409937e-17+1.35381047e-17j, -2.75279006e-17+6.42714422e-18j,
           -5.73387068e-18+1.38247121e-17j,  8.06164023e-18+1.64437826e-17j,
            1.26570754e-17+1.68198606e-18j, -2.01010488e-17-2.75381873e-18j,
            1.06344439e-17-1.02266124e-17j,  3.01958858e-18-5.38359702e-17j,
           -7.22096881e-17-2.51031258e-17j, -1.45537819e-17-3.55441716e-17j,
            2.45547149e-17-5.53036388e-18j, -3.61109866e-17+1.53674071e-17j,
           -4.49719138e-17+2.72048059e-17j,  3.78423784e-18-4.28019525e-17j,
           -8.01038081e-18+3.38724187e-17j,  3.15664448e-17+2.05714092e-17j,
            1.30735883e-17-9.02167454e-18j, -6.05143050e-18-1.78587419e-18j,
            2.99607764e-17-1.46081223e-18j, -1.46017805e-17+1.21033455e-18j,
            2.74365324e-18+1.42761471e-17j, -4.29797558e-18+5.41117176e-18j,
           -3.38948618e-17-6.20777399e-18j,  4.79487645e-17+1.40812492e-17j,
            5.34821454e-17+1.19389094e-17j,  2.43741981e-17+1.57259679e-17j,
           -3.41519813e-18+1.76211629e-17j, -4.28039163e-17-1.74261714e-17j,
           -3.40259464e-17-1.11512809e-17j, -2.53271677e-17-2.13101551e-17j,
            2.69428856e-17-2.79778634e-18j, -2.57974711e-17+1.48179242e-18j,
            2.49024823e-17-1.23243633e-17j,  2.17584724e-17+7.17872890e-17j,
           -1.79985740e-17+2.06417929e-17j,  1.51112754e-17+1.44066878e-17j,
            3.09663475e-17-2.34610280e-17j, -8.57862893e-19+3.74664003e-17j,
           -1.34473790e-18+9.15678956e-18j, -1.77851220e-17+2.87381904e-17j,
           -7.10250394e-18+6.01273485e-18j,  1.90677212e-17+1.61035343e-17j,
            1.29964184e-17-2.19880083e-17j,  7.67066825e-18+3.06463447e-17j,
            1.59651618e-17+1.56092437e-17j,  3.53957086e-17-1.74933127e-17j,
           -1.00704895e-17+6.28771158e-18j,  1.40758623e-17+1.74708533e-17j,
           -4.73488575e-17-1.56307670e-17j, -3.05003593e-17+2.78463970e-19j,
           -1.91359717e-17-1.71325130e-17j,  1.19265623e-17+3.28157874e-17j,
            1.50544954e-18-2.56558614e-17j,  3.29152165e-17-1.06991826e-17j,
            7.12714591e-18+4.12440464e-18j, -2.53071294e-17+3.97262804e-18j,
            2.01791681e-17+1.97523219e-17j, -1.17660075e-16-2.30011308e-17j,
            2.96133231e-17-7.25200938e-18j, -5.97178779e-17-3.91179202e-18j,
            3.23125496e-17-1.31265136e-17j,  1.95720776e-17+3.26451454e-17j,
           -6.10072198e-19+2.59526735e-17j,  2.99711172e-18-4.18495964e-17j,
            5.82545509e-17-2.90873474e-17j,  3.13190425e-17-8.28616375e-18j,
           -3.30824138e-18-1.44382986e-17j,  3.72066085e-17-5.51905618e-17j,
           -4.33224363e-17-6.74944851e-18j, -5.46727069e-17+6.99444724e-18j,
           -3.39704330e-17-2.59905867e-17j,  2.06409908e-17-2.73452339e-20j,
            3.12596418e-17+2.38171388e-17j,  1.89849283e-17+9.67497674e-18j,
            9.54632689e-18-1.72545500e-18j,  2.85881996e-17-1.74323253e-17j,
           -1.43974585e-17+1.77495555e-18j,  2.14826820e-17-2.13252851e-17j,
           -9.00229369e-18-1.23531890e-17j, -7.64158956e-17-1.47102303e-17j,
           -7.02189115e-18+3.04783775e-17j, -1.67332185e-18+1.61137300e-17j,
           -1.46600973e-17-1.04742052e-17j,  1.50591054e-18-4.29551383e-17j,
            4.90072924e-17-1.03660403e-18j,  1.29644417e-17-2.18834329e-17j,
           -5.33491284e-18-2.07655437e-17j,  1.60567727e-18-5.15217509e-18j,
           -4.18296697e-18-2.47517172e-17j, -2.17635488e-17-3.61813894e-17j,
           -1.98799420e-18-2.67408213e-17j,  1.55063043e-17+2.41083210e-17j,
           -2.95673643e-17+4.94079875e-18j,  3.53772431e-17-1.61895889e-17j,
           -1.33154640e-17-1.30018039e-17j, -3.44220633e-17-2.07488660e-18j,
           -5.60178757e-18-1.17931313e-17j, -1.06349694e-17+2.66840978e-17j,
           -1.84598494e-17+2.05378711e-17j, -8.64860846e-18+3.49301125e-17j,
            1.73908548e-17-1.06724624e-17j,  9.61107701e-19-9.44777396e-18j,
           -4.70986463e-18+7.36647324e-18j, -6.63612530e-18+8.20172947e-19j,
           -1.86886002e-17-7.40541886e-18j, -1.42644405e-18+1.54819624e-17j,
            4.79765609e-18-2.69684353e-17j, -3.01958858e-18+1.29874764e-17j,
           -4.14771496e-17+1.61960355e-17j, -4.22896369e-17-3.55441716e-17j,
           -2.45547149e-17+2.78813594e-17j, -2.78378597e-17-1.33845456e-18j,
           -2.60823598e-17+2.72048059e-17j,  2.46374716e-17+1.12593515e-17j,
            8.01038081e-18-2.01888852e-17j,  3.94878287e-17-2.31651044e-17j,
            1.13726646e-18-1.16028721e-17j, -1.05399686e-18-2.04635953e-17j,
           -1.57499217e-17+1.52450494e-17j,  3.90925772e-19-1.23049914e-17j,
           -9.84908060e-18+5.92321630e-18j, -9.91287914e-18-6.81355543e-18j,
           -2.52045338e-18+1.10104048e-18j, -8.86891399e-18-4.21988031e-18j,
            6.91398717e-18+2.70495032e-17j, -6.61062968e-18+1.73212357e-17j,
            3.41519813e-18-1.64718211e-17j, -3.84644091e-17-2.31397180e-17j,
            4.82368011e-17+1.22043598e-17j,  4.30907361e-17+3.37952652e-17j,
           -2.87192425e-17+3.02510813e-17j, -3.51241672e-18+1.05741706e-17j,
           -6.47278008e-18-2.66955098e-17j,  3.24204112e-17+1.44579705e-17j,
            4.55321050e-17+2.22166686e-18j, -1.64435430e-17-2.60998948e-17j,
           -1.06492661e-17-1.28685911e-17j, -4.11085674e-17+1.31249846e-17j,
           -1.73070089e-17+3.31693433e-17j, -3.32851371e-17-3.25343461e-17j,
           -1.28815105e-17+3.14951929e-17j,  6.08683366e-17-2.21659151e-17j,
            1.14284882e-17-5.29068377e-18j,  3.69602973e-17+4.49099427e-19j,
           -1.94137921e-17+2.25045343e-17j,  7.97562000e-19-1.16918303e-17j,
           -8.35921275e-18-2.54930331e-17j,  2.07218318e-17+1.74306974e-17j,
           -2.53016197e-17-9.90729804e-18j, -2.09044217e-17-2.19998747e-17j,
           -7.84456255e-18+1.41605619e-17j, -5.27686196e-18+1.53138484e-17j,
           -3.52414823e-17-5.59857718e-18j,  6.42819498e-17-3.14822871e-17j,
            4.18342171e-18+2.52491442e-17j,  3.08314636e-17-3.97733824e-17j,
            4.42780287e-17-9.84448043e-18j, -4.21900753e-17+3.48314409e-17j,
            4.53187184e-17-5.55046002e-17j, -8.46634367e-17+2.13425863e-17j,
            2.04170299e-17-5.89820435e-18j,  3.10570235e-17-1.53498121e-17j,
            9.35507327e-18-5.69074181e-17j, -7.38526989e-17+1.69437679e-17j,
           -5.40413975e-18-2.42040982e-17j,  2.62230059e-18-5.99303539e-18j,
            9.60405712e-18+1.59846494e-18j,  6.74479164e-18+2.83460014e-17j,
            9.09327345e-17-3.11147341e-17j, -2.21401718e-17+1.74925203e-17j,
            5.28486532e-17-1.56006130e-17j, -1.69802384e-17+2.88744996e-17j,
            7.63221487e-18-1.05691506e-17j,  2.43172908e-17+1.36655409e-17j,
            6.34144224e-18-2.38066773e-17j,  1.92534254e-17-4.39196823e-17j,
           -3.21137466e-18-1.71275415e-17j,  1.88441503e-17-2.81613637e-17j,
           -4.58454378e-19-1.39732053e-17j,  5.82576503e-19+3.37156121e-17j,
            1.05575206e-17-1.24947730e-17j,  9.69947404e-18+1.22164185e-18j,
            6.24043580e-17+2.22974716e-17j, -1.65160021e-16+7.38702376e-17j,
            2.79991923e-17+1.03009040e-16j,  8.05493406e-18-8.65452784e-18j,
            7.96410951e-18+1.14257329e-18j, -1.69246991e-17-1.51291452e-17j,
            2.36619181e-17+1.06512421e-16j, -2.48285584e-17-6.37725188e-18j,
           -4.29626433e-17-4.62725739e-18j, -8.88762584e-18-2.73421601e-17j,
           -3.74950703e-18-1.01952765e-17j,  5.14113147e-17-6.80184687e-18j,
           -1.63255857e-17+2.61573497e-17j, -2.00499274e-17+1.15230027e-17j,
            3.91680994e-17-1.09133279e-17j, -5.24534152e-18+3.51071159e-17j,
            2.31042768e-17+1.98350899e-17j,  2.95714488e-18-3.44952852e-19j,
           -2.30882155e-17+4.05826504e-18j, -2.14858103e-18-3.14378391e-17j,
            3.80591359e-17+5.48524362e-18j, -1.71047146e-17+2.14588847e-18j,
           -2.97952813e-17-1.20925029e-17j,  1.10925747e-17+1.43521760e-17j,
            8.49681885e-18+1.87276339e-17j, -1.83436696e-17+5.38076470e-17j,
           -8.97681039e-17-8.48513500e-17j,  2.58814299e-18-3.14710085e-18j,
            4.16677704e-17+2.92690268e-17j,  2.91545180e-18-2.12098646e-17j,
           -7.76678060e-18-5.31639151e-18j,  9.31532359e-18+1.27049594e-17j,
            2.14607774e-17+1.57276458e-17j, -6.08234179e-17+3.65708388e-18j,
            1.01536307e-17+3.71953370e-18j, -4.43436377e-18+1.08748913e-17j,
            8.91260195e-18-4.47996031e-18j,  2.78015001e-17+7.75244999e-18j,
           -8.32124797e-18+1.16585081e-17j, -2.39528864e-18-3.01876478e-21j,
            3.53916371e-19-4.07249826e-18j,  3.06234442e-17+1.15743723e-18j,
           -1.49798352e-17+7.28240431e-18j,  3.89613415e-17+2.15780318e-18j,
            1.86352088e-17+1.85519196e-17j, -2.09761855e-17+8.84073120e-18j,
           -3.85027187e-17+8.18066964e-17j,  1.08960832e-17-1.97969887e-17j,
           -1.44953543e-17-8.07435009e-18j, -1.81282804e-17-9.28116428e-19j,
           -3.96269410e-19-3.86999514e-17j,  1.77220253e-17-2.54197843e-17j,
            1.13715667e-17-1.71474714e-17j,  4.45714862e-17-4.53767771e-17j,
            1.55865127e-17-1.36251842e-17j,  1.17841878e-17+4.49810548e-17j,
            6.13847383e-17+3.17167514e-17j,  2.80922817e-17-1.24291716e-17j,
           -3.09948256e-17-2.21202258e-17j,  1.28154070e-17-5.56225880e-18j,
           -3.25911242e-17-1.08722636e-17j, -2.38028801e-17+1.09349991e-18j,
           -3.41583502e-17+3.98359801e-17j,  1.96318348e-17+7.81345679e-18j,
           -2.81256084e-18+8.71115168e-18j, -7.87964074e-18-1.37113550e-17j,
            2.27836261e-17-1.70492897e-18j, -5.10237824e-17+4.53523229e-17j,
            8.12296336e-18-1.82517142e-17j,  1.20158494e-17+1.36864548e-17j,
           -1.29859195e-17+1.27932507e-17j,  9.29625928e-18+1.24378930e-18j,
            3.81013405e-17+1.09550132e-17j, -2.44146981e-17+4.20014770e-17j,
            1.57291316e-17+1.01613971e-18j,  5.68481502e-18-1.66230886e-17j,
           -2.53526171e-17+4.45838652e-17j,  1.09219218e-17+1.55972730e-17j,
            3.28820524e-17-7.05823358e-18j,  5.26449354e-19-1.05262155e-17j,
           -2.57919875e-17-1.59872619e-17j,  2.22822949e-17-5.39034157e-18j,
            3.58343494e-18-7.78664577e-19j, -2.16398762e-17-9.14930894e-18j,
            1.01565263e-17-2.06746312e-17j, -1.80564107e-17-1.73398411e-17j,
            3.37198049e-18+2.64263049e-17j, -4.67271026e-17-5.27481349e-18j,
            1.95730647e-17+1.57159508e-17j, -4.59269867e-18+1.04759742e-17j,
           -6.29614540e-17-1.55641012e-17j, -6.28515005e-18+6.10419764e-18j,
           -3.20971199e-17-8.32446508e-18j, -6.37696157e-18+1.66199812e-17j,
           -3.29891346e-17-6.34538438e-18j,  3.68694961e-18+2.30574328e-17j,
           -1.15493918e-17+8.61144293e-19j,  4.27181795e-18+4.34641586e-17j,
            4.82702697e-18-2.45881582e-17j,  2.58215473e-17+1.88892262e-17j,
            1.58305405e-17+4.39496321e-18j,  1.09119843e-16+2.82300658e-17j,
           -1.79359372e-17+2.16462581e-17j, -4.42744965e-17-1.37236287e-17j,
           -1.90665243e-17+1.41018565e-17j,  1.66298022e-17+1.93074085e-17j,
           -3.28763509e-18+4.29547009e-17j,  8.73321147e-17+9.79070176e-18j,
           -9.59010557e-18+2.43166096e-17j,  9.13962001e-18-9.59503138e-18j,
            2.26242079e-17-1.36184051e-17j,  4.51199872e-19-5.12846453e-17j,
            1.80341426e-17-3.89210624e-17j,  1.51737228e-17+4.98983174e-18j,
            7.79734088e-18+4.28607798e-18j, -4.74684838e-17+2.13891443e-17j,
            1.24482937e-17+2.08308655e-17j,  1.59057569e-17+4.23467221e-17j,
           -3.73474237e-17+9.27469584e-18j, -4.71149889e-18-1.82901986e-17j,
            3.74672259e-17+2.99460490e-17j, -1.22848199e-21+8.45332843e-18j,
           -6.45595881e-17-1.72707993e-17j,  9.26469222e-17+3.17269847e-17j])

Aula 44 - DFT resolucao e zero padding/DFT - resolucao e zero padding.ipynb


```julia
using Distributions
using LinearAlgebra
using Optim
using Plots
using ForwardDiff
using Statistics
```

## Method of Moments

A **moment condition**, $\bar{m}_n(\theta)=\bar{m}_n(Z_n,\theta)$ is a vector-valued function of the data $Z_n$ and the parameter $\theta$ that has mean zero, under the model, when evaluated at the true parameter value $\theta_0$, and expectation differenet from zero when evaluated at other parameter values:
$$E\big[\bar{m}_n(Z_n,\theta_0)=0\big]$$
$$E\big[\bar{m}_n(Z_n,\theta)\neq0,\quad \theta \neq \theta_0 \big]$$

The **method of moments principle** is to choose the estimator of the parameter to set the memonet condition equal to zero: $\bar{m}_{n}(\hat{\theta})\equiv0$.

### Example (OLS)

The classical linear model. Let $\bar{m}_{n}(\beta)=\frac{1}{n}\sum_{t}x_{t}(y_{t}-x_{t}^{\prime}\beta).$
So the moment contributions are $m_{t}(\beta)=x_{t}(y_{t}-x_{t}^{\prime}\beta)$.
When $\beta=\beta_{0},$ $y_{t}-x_{t}^{\prime}\beta_{0}=\epsilon_{t},$
and $m_{t}=x_{t}\epsilon_{t}$. We know that $E(x_{t}\epsilon_{t})=0,$
by the weak exogeneity assumption. Thus, the moment contributions,
and the moment condition, which is their average, have expectation
zero when evaluated at the true parameter value.

### Example ($\chi^2$)
Suppose we draw a random sample of $y_{t}$
from the $\chi^{2}(\theta_{0})$ distribution. Here, $\theta_{0}$
is the parameter of interest.

**(Version: mean)** If $Y\sim\chi^{2}(\theta_{0})$, then
the mean $E(Y)=\theta_{0}$. Let the moment contribution be

$$
m_{t}(\theta)=y_{t}-\theta
$$
Then

$$
\bar{m}_{n}(\theta) =\frac{1}{n}\sum_{t=1}^{n}m_{t}(\theta)=\bar{y}-\theta
$$
We know that the $E(\bar{y})=\theta_{0}$.

- Thus, $E\bar{m}_{n}(\theta_{0})=0.$
- However, $E\bar{m}_{n}(\theta)=\theta_{0}-\theta\ne$0 if $\theta\ne\theta_{0}.$

```julia
using Distributions
n = 10
theta = 3

# the distribution doesn't matter, what matters is
# that the moments are correctly specified
y = rand(Chisq(theta), n)
#y = rand(Normal(theta, sqrt(2*theta)), n)

# a MM estimator
thetahat = mean(y)
println("mm estimator based on mean: ", thetahat)
```

    mm estimator based on mean: 2.5892407845249705

**(Version: variance)** The variance of a $\chi^{2}(\theta_{0})$
r.v. is
$$
V\left(Y\right)=E\left(Y-\theta_{0}\right)^{2}=2\theta_{0}
$$

Let $m_{t}(\theta) =\frac{n}{n-1}\left(y_{t}-\bar{y}\right)^{2}-2\theta$
Then
$$
\bar{m}_{n}(\theta)=\frac{\sum_{t=1}^{n}\left(y_{t}-\bar{y}\right)^{2}}{n-1}-2\theta.
$$
The first term is the unbiased formula for the sample variance, and
thus has expectation equal to $2\theta_{0}.$ So if we evaluate $\bar{m}_{n}(\theta)$
at $\theta_{0},$ the expectation is zero.

The MM estimator using the variance would set
$$
\bar{m}_{n}(\hat{\theta})=\frac{\sum_{t=1}^{n}\left(y_{t}-\bar{y}\right)^{2}}{n-1}-2\hat{\theta}\equiv0.
$$
Solving for the estimator, it is half the sample variance:
$$
\hat{\theta}=\frac{1}{2}\frac{\sum_{t=1}^{n}\left(y_{t}-\bar{y}\right)^{2}}{n-\text{1}}.
$$
Again, by the LLN, the sample variance is consistent for the true
variance, that is,
$$
\frac{\sum_{t=1}^{n}\left(y_{t}-\bar{y}\right)^{2}}{n}\stackrel{p}{\rightarrow}2\theta_{0}.
$$
So, this MM is also consistent for $\theta_{0}$.

```julia
n = 10000
theta = 3

# the distribution doesn't matter, what matters is
# that the moments are correctly specified
y = rand(Chisq(theta), n)
#y = rand(Normal(theta, sqrt(2*theta)), n)

thetahat = 0.5*var(y)
println("mm estimator based on variance: ", thetahat)
```

    mm estimator based on variance: 2.888258871002374

- For the $\chi^{2}$ example, we have two alternative moment conditions
and only one parameter: we have **overidentification**, which means
that we have more information than is strictly necessary for consistent
estimation of the parameter.

**To summarize** a moment condition is a vector valued function
which has expectation zero at the true parameter value. We have seen
some examples of where we might get such functions, and more will
follow. For now, let's take moment conditions as given, and work out
the properties of the estimator

## Generalized Method of Moments

For the purposes of this course, the following definition of the GMM
estimator is sufficiently general:
\begin{defn}
\label{GMM estimator (defn)}The GMM estimator of the $k$ -dimensional
parameter vector $\theta_{0}$,

$$
\hat{\theta}\equiv\arg\min_{\Theta}\bar{m}_{n}(\theta)^{\prime}W_{n}\bar{m}_{n}(\theta)
$$

- where $\bar{m}_{n}(\theta)=\frac{1}{n}\sum_{t=1}^{n}m(Z_{t},\theta)$
is a $g$-vector valued function, $g\geq k,$ with $\mathcal{E}m(\theta_{0})=0,$
- and $W_{n}$ converges almost surely to a finite $g\times g$ symmetric
positive definite matrix $W_{\infty}$.

```julia
function fminunc(obj, x; tol = 1e-08)
    results = Optim.optimize(obj, x, LBFGS(),
                            Optim.Options(
                            g_tol = tol,
                            x_tol=tol,
                            f_tol=tol))
    return results.minimizer, results.minimum, Optim.converged(results)
    #xopt, objvalue, flag = fmincon(obj, x, tol=tol)
    #return xopt, objvalue, flag
end

```

    fminunc (generic function with 1 method)

### Consistency

```julia
n = 10000
theta = 3

# the distribution doesn't matter, what matters is
# that the moments are correctly specified
y = rand(Chisq(theta), n)
W = I(2)
obj = theta -> ([theta.-mean(y); theta.-0.5*var(y)]'W*[theta.-mean(y); theta.-0.5*var(y)])
thetahat, junk, junk = fminunc(obj, [2.0])
println("GMM estimator based on mean and variance: ", thetahat)

```

    GMM estimator based on mean and variance: [3.0859903675869997]

What's the reason for using GMM if MLE is asymptotically efficient?
- Robustness: GMM is based upon a limited set of moment conditions.
For consistency, only these moment conditions need to be correctly
specified, whereas MLE in effect requires correct specification of
**every conceivable** moment condition. GMM is \emph{robust with
respect to distributional misspecification.} The price for robustness
is usually a loss of efficiency with respect to the MLE estimator.
Keep in mind that the true distribution is \uline{not known} so
if we erroneously specify a distribution and estimate by MLE, the
estimator will be inconsistent in general (not always).
- Feasibility: in some cases the MLE estimator is not available, because
we are not able to deduce or compute the likelihood function. More
on this in the section on simulation-based estimation. The GMM estimator
may still be feasible even though MLE is not available.

### Asymptotic Normality

```julia
n = 30
theta = 3.0
reps = 1000
results = zeros(reps)
W = I(2)
for i = 1:reps
    y = rand(Chisq(theta), n)
    obj = theta -> ([theta.-mean(y); theta.-0.5*var(y)]'W*[theta.-mean(y); theta.-0.5*var(y)])
    thetahat, junk, junk = fminunc(obj, [5.0])
    results[i] = sqrt(n)*(thetahat[1]-theta)
end
histogram(results,nbins=50,legend=false)
plot!(size=(500,300))
```

```julia
std(results)
```

    4.652988476057202

```julia
n = 10000
theta = 3.0
reps = 1000
results = zeros(reps)
W = I(2)
for i = 1:reps
    y = rand(Chisq(theta), n)
    obj = theta -> ([theta.-mean(y); theta.-0.5*var(y)]'W*[theta.-mean(y); theta.-0.5*var(y)])
    thetahat, junk, junk = fminunc(obj, [3.0])
    results[i] = sqrt(n)*(thetahat[1]-theta)
end
histogram(results,nbins=50,legend=false)
plot!(size=(500,300))
```

### Choosing the Weighting Matrix

$W$ is a weighting matrix, which determines the relative
importance of violations of the individual moment conditions. For
example, if we are much more sure of the first moment condition, which
is based upon the variance, than of the second, which is based upon
the fourth moment, we could set
$$
W=\left[\begin{array}{cc}
a & 0\\
0 & b
\end{array}\right]
$$
 with $a$ much larger than $b.$ In this case, errors in the second
moment condition have less weight in the objective function.

How to estimate efficient GMM estimator?
### Two Step GMM estimator

The most common way to do efficient GMM estimation is the two step
GMM estimator:
1. Set the weight matrix to some positive definite matrix. Most commonly,
one uses an identity matrix. Obtain the GMM estimator
that minimizes $Q_{n}(\theta)=m_{n}(\theta)^{\prime}Wm_{n}(\theta)$
2. Based on this initial estimate, $\hat{\theta}$, say, compute the
moment contributions $m_{t}(\hat{\theta}),\,t=1,2,...,n$. Compute
an estimate of $\Omega_{\infty}$ based on the moment contributions,
say $\hat{\Omega}^{-1}$. The exact way to do this will depend upon
the assumptions of the model. Given the
estimate, compute the efficient GMM estimator which minimizes
$$
Q_{n}(\theta)=m_{n}(\theta)^{\prime}\hat{\Omega}^{-1}m_{n}(\theta).
$$
Note that $\hat{\Omega}^{-1}$ is fixed while numeric minimization
finds the second step estimator. The result is the two step estimator.

```julia
function gmm(moments, theta, weight)
    # average moments
    m = theta -> vec(mean(moments(theta),dims=1)) # 1Xg
    # moment contributions
    momentcontrib = theta -> moments(theta) # nXg
    # GMM criterion
    obj = theta -> ((m(theta))'weight*m(theta))
    # do minimization
    thetahat, objvalue, converged = fminunc(obj, theta)
    # derivative of average moments
    D = (ForwardDiff.jacobian(m, vec(thetahat)))'
    # moment contributions at estimate
    ms = momentcontrib(thetahat)
    return thetahat, objvalue, D, ms, converged
end
```

    gmm (generic function with 1 method)

```julia
n = 100
theta = 3.0
reps = 1000
results = zeros(reps,2)
for i = 1:1000
    y = rand(Chisq(theta), n)
    moments = theta -> [theta.-y  theta.-0.5.*(y .- mean(y)).^2.0]
    thetahat, junk, junk, ms, junk = gmm(moments, [3.0], I(2))
    results[i,1] = sqrt(n)*(thetahat[1]-theta)
    W = inv(cov(ms))
    thetahat, junk, junk, ms, junk = gmm(moments, thetahat, W)
    results[i,2] = sqrt(n)*(thetahat[1]-theta)
end
```

Let's look closely into it:

```julia
y = rand(Chisq(theta), n)
moments = theta -> [theta.-y  theta.-0.5.*(y .- mean(y)).^2.0]
```

    #85 (generic function with 1 method)

```julia
thetahat, junk, junk, ms, junk = gmm(moments, [3.0], I(2))
```

    ([3.330423129165346], 0.07597476105060097, [1.0 1.0], [-3.4375971075234237 -3.2671072605594325; 1.561283610540204 2.3969258477031445; … ; -2.7291879277801345 -0.9447322564482312; 2.220622514937011 1.2786543977131046], true)

```julia
W = inv(cov(ms))
thetahat, junk, junk, ms, junk = gmm(moments, thetahat, W)
```

    ([3.064864471476671], 0.003172090752355508, [1.0 1.0], [-3.7031557652120988 -3.5326659182481075; 1.2957249528515289 2.1313671900144695; … ; -2.9947465854688096 -1.2102909141369063; 1.955063857248336 1.0130957400244296], true)

Aggregating the result:

```julia
histogram(results[:,1],legend=false,title="Inefficiet theta")
plot!(size=(500,300))
```

```julia
histogram(results[:,2],legend=false,title="Efficiet theta")
plot!(size=(500,300))
```

### Continuously Updated GMM Estimator (CUE)
The continuously updated estimator solves a minimization problem where the efficient weight matrix is
estimated at each iteration of the numeric optimization process. The
CUE estimator solves the minimization problem
$$
Q_{n}(\theta)=m_{n}(\theta)^{\prime}\hat{\Omega}(\theta)^{-1}m_{n}(\theta).
$$

- Note that the covariance of the moment conditions will be updated
at each trial value of the objective function during the course of
minimization.
- This estimator is equivalent to an iterated version of the two step
estimator.
- The CUE estimator can be shown to have a smaller bias than does the
two step estimator, which may have a large small sample bias.

### Generalized instrumental variables estimator for linear models
The IV estimator may appear a bit unusual at first, but it will grow
on you over time.

Let's look at the previous section's results in more detail, for the
commonly encountered special case of a linear model with iid errors,
but with correlation between regressors and errors:
$$
\begin{eqnarray*}
y_{t} & = & x_{t}^{\prime}\theta+\varepsilon_{t}\\
\mathcal{E}(x_{t}\varepsilon_{t}) & \neq & 0
\end{eqnarray*}
$$

- Let's assume, just to keep things simple, that the errors are iid
- The model in matrix form is $y=X\theta+\epsilon$

We have seen some cases where this problem arises:
- measurement error of regressors
- lagged dependent variable and autocorrelated errors
- simultaneous equations

Let $K=dim(x_{t}).$ Consider some vector $z_{t}$ of dimension
$G\times1$, where $G\ge K.$ Assume that $E(z_{t}\epsilon_{t})=0.$
The variables $z_{t}$ are **instrumental variables**.

Consider the moment conditions
$$
\begin{align*}
m_{t}(\theta) & =z_{t}\epsilon_{t}\\
 & =z_{t}\left(y_{t}-x_{t}^{\prime}\theta\right)
\end{align*}
$$

We can arrange the instruments in the $n\times G$ matrix
$$
\begin{eqnarray*}
Z & = & \left[\begin{array}{c}
z_{1}^{\prime}\\
z_{2}^{\prime}\\
\vdots\\
z_{n}^{\prime}
\end{array}\right]
\end{eqnarray*}
$$

The average moment conditions are
$$
\begin{align*}
\bar{m}_{n}(\theta) & =\frac{1}{n}Z^{\prime}\epsilon\\
 & =\frac{1}{n}(Z^{\prime}y-Z^{\prime}X\theta)
\end{align*}
$$

Exercise: Derive the below representation of IV estimator from previous class, or Creel.
$$
\begin{equation}
\hat{\theta}_{IV}=\left(X^{\prime}Z\left(Z^{\prime}Z\right)^{-1}Z^{\prime}X\right)^{-1}X^{\prime}Z\left(Z^{\prime}Z\right)^{-1}Z^{\prime}y\label{eq:GIV estimator}
\end{equation}
$$

As an exercise, you can verify GIV estimator is
1. Consistent and
2. Asymptotically normally distributed

### Example (GIV)

Suppose the model
is
$$
\begin{eqnarray*}
y_{t}^{*} & = & \alpha+\rho y_{t-1}^{*}+\beta x_{t}+\epsilon_{t}\\
y_{t} & = & y_{t}^{*}+\upsilon_{t}
\end{eqnarray*}
$$
where $\epsilon_{t}$ and $\upsilon_{t}$ are independent Gaussian
white noise errors. Suppose that $y_{t}^{*}$ is not observed, and
instead we observe $y_{t}$. If we estimate the equation
$$
y_{t}=\alpha+\rho y_{t-1}+\beta x_{t}+\nu_{t}
$$
this the estimator is biased and inconsistent.

What about using the GIV
estimator?

Consider using as instruments $Z=\left[1\,x_{t}\,x_{t-1}\,x_{t-2}\right]$.
The lags of $x_{t}$ are correlated with $y_{t-1}$ as long as $\beta$
is different from zero, and by assumption $x_{t}$ and its lags are
uncorrelated with $\epsilon_{t}$ and $\upsilon_{t}$ (and thus they're
also uncorrelated with $\nu_{t})$. Thus, these are legitimate instruments.
As we have 4 instruments and 3 parameters, this is an overidentified
situation.

```julia
function lag(x::Array{Float64,2},p::Int64)
	n,k = size(x)
	lagged_x = [ones(p,k); x[1:n-p,:]]
end

function lag(x::Array{Float64,1},p::Int64)
	n = size(x,1)
	lagged_x = [ones(p); x[1:n-p]]
end

function  lags(x::Array{Float64,2},p)
	n, k = size(x)
	lagged_x = zeros(eltype(x),n,p*k)
	for i = 1:p
		lagged_x[:,i*k-k+1:i*k] = lag(x,i)
	end
    return lagged_x
end

function  lags(x::Array{Float64,1},p)
	n = size(x,1)
	lagged_x = zeros(eltype(x), n,p)
	for i = 1:p
		lagged_x[:,i] = lag(x,i)
	end
    return lagged_x
end
```

    lags (generic function with 2 methods)

```julia
# moment condition
function GIVmoments(theta, data)
	data = [data lags(data,2)]
    data = data[3:end,:] # get rid of missings
	n = size(data,1)
	y = data[:,1]
	ylag = data[:,2]
	x = data[:,3]
	xlag = data[:,6]
	xlag2 = data[:,9]
	X = [ones(n,1) ylag x]
	e = y - X*theta
	Z = [ones(n,1) x xlag xlag2]
	m = e.*Z
end

# do the Monte Carlo
n = 100
sig = 1
reps = 1000
results = zeros(reps,3)
for rep = 1:reps
	x = randn(n) # an exogenous regressor
	e = randn(n) # the error term
	ystar = zeros(n)
	# generate the dep var
	for t = 2:n
	  ystar[t] = 0.0 + 0.9*ystar[t-1] + 1.0*x[t] + e[t]
	end
    # add measurement error
	y = ystar + sig*randn(n)
	# now do GMM, using the data with meas. error in both dep. var. and regressor
	ylag = lag(y,1)
	data = [y ylag x];
    data = data[2:end,:] # drop first obs, missing due to lag
	theta = [0, 0.9, 1]
	weight = 1
    # note to self: this is very slow, because it uses the general GMM method, instead of analytic GIV
    # also, the weight matrix is not optimal
    moments = theta-> GIVmoments(theta, data)
	b, junk, junk, junk, junk = gmm(moments, theta, weight)
	b = b - [0.0, 0.9, 1.0] # subtract true values, so mean should be approx. zero if consistent
    results[rep,:] = b
end

```

```julia
mean(results,dims=1)
```

    1×3 Array{Float64,2}:
     0.00227442  -0.0229675  -0.0078547

```julia
std(results,dims=1)
```

    1×3 Array{Float64,2}:
     0.23251  0.139008  0.180733

```julia
histogram(results[:,1],nbins = 30,title="givconstant",legend=false)
plot!(size=(500,300))
```

```julia
histogram(results[:,2],nbins = 30,title="givrho",legend=false)
plot!(size=(500,300))
```


# 運動学的方程式の導出 （XYZオイラー角）

吉田勝俊（宇都宮大学）

## 参考情報
- [SymPyで代数演算してみる - Qiita](https://qiita.com/zawawahoge/items/1be137a8147902a5e6cb)
- [Matrices (linear algebra) — SymPy 1.6.2 documentation](https://docs.sympy.org/latest/modules/matrices/matrices.html)

```python
import sympy as sym #数式処理ライブラリ
sym.init_printing() #数式を綺麗に表示する設定
```

## Sympy 変数・関数

```python
om1, om2, om3 = sym.symbols('ω_1 ω_2 ω_3')
th1, th2, th3 = sym.symbols('θ_1 θ_2 θ_3', Real=True, positive=True)
dth1, dth2, dth3 = sym.symbols('\dot{θ}_1 \dot{θ}_2 \dot{θ}_3', Real=True, positive=True)
```

```python
th = sym.Matrix([th1,th2,th3]) #オイラー角
dth = sym.Matrix([dth1,dth2,dth3]) #その時間微分

display(th)
display(dth)
```

#### テスト

```python
th.dot(dth)
```

## ZYZオイラー角による回転行列

```python
def RotX(th): #X軸回転の行列
    return sym.Matrix([
        [1, 0, 0],
        [0, sym.cos(th), -sym.sin(th)],
        [0, sym.sin(th),  sym.cos(th)],
    ])

def RotY(th): #Y軸回転の行列
    return sym.Matrix([
        [sym.cos(th), 0, sym.sin(th)],
        [0, 1, 0],
        [-sym.sin(th), 0, sym.cos(th)],
    ])

def RotZ(th): #Z軸回転の行列
    return sym.Matrix([
        [sym.cos(th), -sym.sin(th), 0],
        [sym.sin(th),  sym.cos(th), 0],
        [0, 0, 1],
    ])

Rot = RotZ(th3)*RotY(th2)*RotX(th1)
Rot
```

### LaTeXソースの出力

```python
print(sym.latex(Rot))
```

### 回転行列の時間微分

```python
def sympy_dt_param(expr, param_vec, dparam_vec):
    """
    パラメータを時間の関数として時間微分する関数
        param_vec: パラメータベクトル sympy.Matrix([sympyのシンボル達])
        dparam_vec: パラメータベクトルの微分 sympy.Matrix([sympyのシンボル達])
    """
    dim = len(param_vec)
    result = expr

    # 仮の時間変数
    t = sym.symbols('t')

    # パラメータを時間関数で置換
    func_vec = []
    for i in range(dim):
        lab = 'param' + str(i) #仮の変数表示名
        fun = sym.symbols(lab, cls=sym.Function, Real=True)
        func_vec.append( fun(t) )

    func_vec = sym.Matrix(func_vec) #sympy型へ変換

    for i in range(dim):
        result = result.replace(param_vec[i],func_vec[i])

    # 時間微分を実行
    result = result.diff()
    dfunc_vec = func_vec.diff()

    # 時間関数を元のパラメータに戻す
    for i in range(dim):
        result = result.replace(dfunc_vec[i], dparam_vec[i])
        result = result.replace(func_vec[i], param_vec[i])

    return result
```

回転行列を実際に時間微分する

```python
dRot = sympy_dt_param(Rot, th, dth)
dRot
```

### 回転行列の逆行列

```python
invRot = Rot.transpose() #回転行列なので転置で求まる
invRot
```

### 角速度ベクトルの外積行列 $[\boldsymbol{\omega}\times]:=R^{-1}\dot{R}$

```python
OMcross = sym.simplify(invRot*dRot)
```

### 外積行列 $[\boldsymbol{\omega}\times]:=R^{-1}\dot{R}$ から角速度ベクトル $\boldsymbol{\omega}$ を抽出

```python
OMvec = sym.Matrix([OMcross[2,1],OMcross[0,2],OMcross[1,0]])
OMvec
```

### 行列表示 $\boldsymbol{\omega}=\Omega \dot{\boldsymbol{\theta}}$ の表現行列 $\Omega$

```python
tmp = OMvec
OMmat = tmp.jacobian(dth)
OMmat
```

### LaTeXソースの出力

```python
print(sym.latex(OMmat))
```

### $\Omega(\boldsymbol{\theta},\dot{\boldsymbol{\theta}})$ を $\dot{\boldsymbol{\theta}}$ について解き，運動学的方程式 $\dot{\boldsymbol{\theta}}=K(\boldsymbol{\theta},\boldsymbol{\omega})$ を導く

#### $\dot{\boldsymbol{\theta}}$ について解く

```python
var = dth
oms = sym.Matrix([om1, om2, om3])
equations = OMmat*var - oms
result = sym.solve(equations, var)
result
```

#### 解から$K(\boldsymbol{\theta},\boldsymbol{\omega})$を作る

```python
Kfunc = sym.Matrix([result[dth[i]] for i in range(3)])
Kfunc = sym.simplify(Kfunc)
Kfunc
```

## 運動学的方程式 $\dot{\boldsymbol{\theta}}=K(\boldsymbol{\theta},\boldsymbol{\omega})$ の行列表示

- 線形変換 $K(\boldsymbol{q},\boldsymbol{\omega})$ のヤコビ行列＝表現行列なので，ヤコビ行列を求める関数 [jacobian(X)](https://docs.sympy.org/latest/modules/matrices/matrices.html#sympy.matrices.matrices.MatrixCalculus.jacobian) を流用しました．

### $\dot{\boldsymbol{\theta}} = K_{\omega}\boldsymbol{\theta}$ 型の表現行列

オイラー角は，$\theta_1$, $\theta_2$, $\theta_3$ に関して非線形（三角関数の中身）なので，この型の表現行列は取れません．

### $\dot{\boldsymbol{\theta}} = K_{\boldsymbol{\theta}}\boldsymbol{\omega}$ 型の表現行列

```python
Kth = Kfunc.jacobian(oms)
Kth
```

### LaTeXソースの出力

```python
print(sym.latex(Kth))
```


Grover's Algorithm

https://en.wikipedia.org/wiki/Grover%27s_algorithm

## Imports

```python
%matplotlib inline
```

```python
from IPython.display import display
```

```python
from sympy import init_printing
init_printing(use_latex=True)
```

```python
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

## Grover's algorithm for 3 qubits

```python
nqubits = 3
```

Grover's algorithm is a quantum algorithm which searches an unordered database (inverts a function). It provides polynomial speedup over classical brute-force search ($O(\sqrt{N}) vs. O(N))$

Define a black box function that returns True if it is passed the state we are searching for.

```python
def black_box(qubits):
    return True if qubits == IntQubit(1, qubits.nqubits) else False
```

Build a uniform superposition state to start the search.

```python
psi = superposition_basis(nqubits)
psi
```

```python
v = OracleGate(nqubits, black_box)
```

Perform two iterations of Grover's algorithm.  Each iteration, the amplitude of the target state increases.

```python
iter1 = qapply(grover_iteration(psi, v))
iter1
```

```python
iter2 = qapply(grover_iteration(iter1, v))
iter2
```

A single shot measurement is performed to retrieve the target state.

```python
measure_all_oneshot(iter2)
```


# Computación neutrosófica mediante Sympy

Paper at:[Computación neutrosófica mediante Sympy](http://fs.unm.edu/NCML/ComputacionNeutrosoficaMediante.pdf)

## Introduccion a los números neutrosóficos

```
from sympy import var
```

```
i = var('i')
i+2
```

    i + 2

# Múltiplicación por un escalar

```
2*(2+i)

```

    2*i + 4

#  De-neutrosificación  con mpmath

Ejercicio. Realice la de-neutrosoficacion del número $ 3+2*i $

con $i{\in}[10,30]$ Utilice la libreria mpmath

```
from mpmath import *
mp.dps = 15 #Establece la precisión
i=mpi(10, 30)
3 + 2*i
```

    mpi('23.0', '63.0')

```
mp.dps = 15 #Establece la precisión
i=mpi(2, 3)
2*i + 4
```

    mpi('8.0', '10.0')

Matrices neutrosóficas

Resolver el siguiente sistema de ecuaciones lineales.
* $ x + 4 y =  2$

* $-2 x +   y = 14$

```
from sympy import Matrix, solve_linear_system
from sympy.abc import x, y
i = var('i')
system = Matrix(( (1, 4, 2), (-2, 1, 14)))
solve_linear_system(system, x, y)

```

    {x: -6, y: 2}

#Introduciendo indeterminación
Resolver el siguiente sistema de ecuaciones lineales.
* $ x + 4 y =  2+i$

* $-2 x +   y = 14+i$

```

```

```

from sympy import Matrix, solve_linear_system
from sympy.abc import x, y
i = var('i')
system = Matrix(( (1, 4, 2+i), (-2, 1, 14+i)))
solve_linear_system(system, x, y)

```

    {x: -i/3 - 6, y: i/3 + 2}

```
from sympy import Matrix, solve_linear_system
from sympy.abc import x, y, z

system = Matrix(( (1,0,0, 1100), (1, 1, 0, 1300), (0, 1, 2,3400)))

solve_linear_system(system, x, y, z)
```

    {x: 1100, y: 200, z: 1600}

```
from sympy import Matrix, solve_linear_system
from sympy.abc import x, y, z

system = Matrix(( (1,0,0, 1100-i), (1, 1, 0, 1300), (0, 1, 2,3400-i)))

solve_linear_system(system, x, y, z)
```

    {x: -i + 1100, y: i + 200, z: -i + 1600}

Cite: Leyva-Vázquez, M., & Smarandache, F. (2018). Computación neutrosófica mediante Sympy. Neutrosophic Computing and Machine Learning, 04, 16-21. doi: 10.5281/zenodo.2139584


# Homework 05: Conjugate Gradient Descent

In this homework, we will implement the conjugate graident descent algorithm. While you should nearly always use an optimization routine from a library for practical data analyiss, this exercise is useful because it will make concepts from multivariatble calculus and linear algebra covered in the lectrures concrete for you. Also, it brings you up the learning curve for the implementaiton of more complex algorithms than the ones you have been exposed to so far.

Note: The exercise assumes that we can calculate the gradient and Hessian of the fucntion we are trying to minimize. This can be computationally expensive or not even possible for soeme functions. Approximate methods can then be used; we do not go into such complexities here.

**Conjugate gradient descent**

We want to implement the line search method

$$
x_{k+1} = x_k + \alpha_k p_k
$$

where $\alpha_k$ is the step size and $p_k$ is the search direction.

In particular, we want the search directions $p_k$ to be *conjugate*, as this will allow us to find the minimum in $n$ steps for $x \in \mathbb{R}^n$ if $f(x)$ is a quadratic function.

The following exercises will unpack this:

- What quadratic functions are
- What conjugate vectors are
- How to find conjugate vectors by Gram-Schmidt process
- How to find the step size $\alpha_k$

and finally wrap them all into a conjugate gradient algorithm.

**Quadratic function surfaces**

Recall that our objective is to minimize a scalar valued function which maps $\mathbb{R}^n \mapsto \mathbb{R}$, for example, a log likelihoood function (for MLE) or unnormalized posterior distribution (for MAP). Geometrically, we are tring to find the value of the lowest point of some surface. The conjugate gradient algorihtm assumes that the surface can be approximated by the quadratic expression (say, by using a Taylor series expansion about $x$)

$$
f(x) = \frac{1}{2}x^TAx - b^Tx + c
$$

and that

$$
\nabla f = Ax - b = 0
$$

at the minimum (if A is positive definite). Note that $A$ is a matrix, $b$ is a vector, and $c$ is a scalar.  Also, note that the matrix $A$ is the Hessian of the quadratic function.For simplicity, we'll work in $\mathbb{R}^2$ so we can visualize the surface, so that $x$ is a 2-vector.

Note: A **form** is a polynomial function where every term has the same degree - for example, $x^2 + 2xy + y^2$ is a quadratic form, whcih can be rewritten as
$$
\begin{pmatrix}
x & y
\end{pmatrix}
\begin{pmatrix}
    1 & 1\\
    1 & 1
\end{pmatrix}
\begin{pmatrix}
x \\
y
\end{pmatrix}
$$

That is, $x^TAx$ is a quadratic form.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy.linalg as la
```

**Exercise 1 (20 points)**

We will work with function $f_1$

$$
f1(x) = \frac{1}{2} x^T \pmatrix{1 & 0 \\ 0 & 1}x
$$

and function $f_2$

$$
f2(x) = \frac{1}{2} x^T \pmatrix{1 & 0 \\ 0 & 3}x
$$

- Plot the labeled contours of the quadratic functions
- Use a `streamplot` to show the gradient vector field of the above quadratic functions.

```python
def f1(x, y):
    '''function f1'''

    return 0.5 * (x**2 + y**2)

def f2(x, y):
    '''fucntion f2'''

    return 0.5 * (x**2 + 3 * y**2)

x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)
plt.contour(X, Y, f1(X, Y), 10)
plt.contour(X, Y, f2(X, Y), 10)
plt.axis('square')
pass
```

```python
def u1(x, y):
    '''df1/px'''

    return x

def v1(x, y):
    '''df1/dy'''

    return y

def u2(x, y):
    '''df2/dx'''

    return x

def v2(x, y):
    '''df2/dy'''

    return 3 * y

plt.streamplot(X, Y, u1(X, Y), v1(X, Y), density = 1)
plt.streamplot(X, Y, u2(X, Y), v2(X, Y), density = 1)
pass
```

**Gram-Schmidt**

The way to numerically find conjugate vectors is to use the Gram-Schmidt process. Here, instead of the usual projection

$$
\text{proj}_u(v) = \frac{u \cdot v}{u \cdot u} \, u
$$

we use the generalized projection
$$
\text{proj}_u(v) = \frac{uA^Tv}{uA^Tu} \, u
$$

**Exercise 2 (30 points)**

The vectors $u$ and $v$ are orthogonal i.e. $u^Tv = 0$ and conjugate with respect to $A$ if $u^TAv = 0$. The geometric intuition for conjugate vecrors $u$ and $v$ is that $u$ and $v$ would be orthogonal if we stretched the contour plots so that it became isotropic (same in all directions, just like when A=𝟙). Write a Gram-Schmidt function to find orthogonal and conjuate vectors with the following signature

```python
def gram_schmidt(U, inner):
    """Return an orthogonal matrix.

    U is a matrix of (column) vectors.
    inner is a function that calculates the inner product.

    Returns an orthogonal matrix of the same shape as U.
    """
```

Use this function and the appropiate inner product to plot

- An orhtogonal set of basis vectors for $f_1$
- A conjugate set of basic vectors for $f_2$

where the first basis vector is to parallel to $\pmatrix{1 \\ 1}$.

```python
def inner_f1(x, y):

    A = np.array([[1,0],[0,1]])
    return (x.T @ A.T @ y).reshape(-1)

def inner_f2(x, y):

    A = np.array([[1,0],[0,3]])
    return (x.T @ A.T @ y).reshape(-1)

def gram_schmidt(U, inner):
    """Return an orthogonal matrix.

    U is a matrix of (column) vectors.
    inner is a function that calculates the inner product.

    Returns an orthogonal matrix of the same shape as U.
    """

    V = np.zeros((U.shape[0], U.shape[1]), dtype = 'float64')
    V[:,0] = U[:,0]

    for i in range(1, U.shape[1]):
        V[:,i] = U[:,i]

        for j in range(0, i):
            V[:,i] -= (inner(V[:,i], V[:,j]) / inner(V[:,j], V[:,j])) * V[:,j]

    return V
```

```python
U = np.array([[1,0],[1,1]])
basis_f1 = gram_schmidt(U, inner_f1)
basis_f2 = gram_schmidt(U, inner_f2)
print(basis_f1)
print(basis_f2)

plt.figure(figsize=(4,4))
plt.quiver(0, 0, basis_f1[0,0],basis_f1[1,0],color='red',angles='xy',scale_units='xy',scale=2);
plt.quiver(0, 0, basis_f1[0,1],basis_f1[1,1],color='red',angles='xy',scale_units='xy',scale=2);
plt.quiver(0, 0, basis_f2[0,0],basis_f2[1,0],color='blue',angles='xy',scale_units='xy',scale=2);
plt.quiver(0, 0, basis_f2[0,1],basis_f2[1,1],color='blue',angles='xy',scale_units='xy',scale=2);
```

**Exercise 3 (20 points)**

We now need to find the "step size" $\alpha$ to take in the direction of the search vector $p$. We can get a quadratic approximation to a general nonliner function $f$ by taking the Taylor series in the driection of $p$

$$
f(x + \alpha p) = f(x) + \alpha [f'(x)]^T p + \frac{\alpha^2}{2} p^T f''(x) p
$$

Find the derivative with respect to $\alpha$ and use this to find the optimal value for $\alpha$ with respect to the quadratic approcimaiton.

* Write the derivation of the optimal step size $\alpha$ using LaTeX in a Markdown cell.
* Write a funciton that returns $\alpha$ for a quadratic funciton with the following signature

```python
def step(x, p, A, b):
    """Returns the optimal step size to take in line search on a quadratic.

    A and b are the coefficients of the quadartic expression
    $$
    f(x) = \frac{1}{2}x^TAx - b^Tx + c
    $$
    p is the search direction
    x is the current location
    """
```

$$
\begin{align}
f(x+\alpha p)&=f(x)+\alpha[f'(x)]^Tp+\frac{\alpha^2}{2}p^Tf''(x)p\\
\frac{df(x+\alpha p)}{d\alpha}&=[f'(x)]^Tp+\alpha p^Tf''(x)p\\
\end{align}
$$
Solve for $\frac{df(x+\alpha p)}{d\alpha}=0$, we get
$$
\begin{align}
\alpha&=-\frac{[f'(x)]^Tp}{p^Tf''(x)p}\\
&=-\frac{\nabla f^T p}{p^T A p}\\
&=\frac{(b - Ax)^T p}{p^T A p}
\end{align}
$$

```python
def step(x, p, A, b):
    """Returns the optimal step size to take in line search on a quadratic.

    A and b are the coefficients of the quadartic expression
    $$
    f(x) = \frac{1}{2}x^TAx - b^Tx + c
    $$
    p is the search direction
    x is the current location
    """

    alpha = ((b - A @ x).T @ p) / (p.T @ A @ p)

    return alpha
```

**Line search**

We now know how to find a search direction $p_k$ - this is a vector that is conjugate to the previous search direction. The first search direction is usually set to be the gradient. Next we need to find out how far along $p_k$ we need to travel, i.e., we need to find $\alpha_k$. First we take a Taylor expansion in the direction of $p$

$$
f(x + \alpha p) = f(x) + \alpha [f'(x)]^T p + \frac{\alpha^2}{2} p^T f''(x) p
$$

followed by finding the derivative with respect to $\alpha$

$$
\frac{d}{d\alpha} f(x + \alpha p) = [f'(x)]^T p + \alpha p^T f''(x) p
$$

Solvign for $\frac{d}{d\alpha} f(x + \alpha p) = 0$, we get

$$
\alpha = - \frac{[f'(x)]^T p}{p^T f''(x) p} \\
= - \frac{\nabla f^T p}{p^T A p} \\
= \frac{(b - Ax)^T p}{p^T A p}
$$

**Exercise 4 (30 points)**

Implement the conjugate grdient descent algorithm with the following signature

```python
def cg(x, A, b, c, max_iter=100, tol=1e-3):
    """Conjugate gradient descent on a quadratic function surface.

    x is the starting position
    A, b and c are the coefficients of the quadartic expression
    $$
    f(x) = \frac{1}{2}x^TAx - b^Tx + c
    $$
    max_iter is the maximum number of iterations to take
    tol is the tolerance (stop if the length of the gradient is smaller than tol)

    Returns the number of steps taken and the list of all positions visited.
    """
```

- Use cg to find the minimum of the funciton $f_2$ from Exercise 1, starting from $\pmatrix{6 \\ 7}$.

- Plot the contour of the funciton f and the trajectory taken from the inital starting poitn $x$ to the final position, inlcuding all the intermediate steps.

We are not particularly concerned about efficiency here, so don't worry about JIT/AOT/C++ level optimization.

```python
def cg(x, A, b, c, max_iter=100, tol=1e-3):
    """Conjugate gradient descent on a quadratic function surface.

    x is the starting position
    A, b and c are the coefficients of the quadartic expression
    $$
    f(x) = \frac{1}{2}x^TAx - b^Tx + c
    $$
    max_iter is the maximum number of iterations to take
    tol is the tolerance (stop if the length of the gradient is smaller than tol)

    Returns the number of steps taken and the list of all positions visited.
    """

    def f(x):
        '''return f(x)'''

        return 0.5 * x.T @ A @ x - b.T @ x + c

    orbit = np.zeros((max_iter + 1, len(x)))
    p = np.zeros((max_iter + 1, len(x)))
    r = np.zeros((max_iter + 1, len(x)))

    orbit[0] = x.ravel()
    r[0] = b - A @ orbit[0]
    if la.norm(r[0]) < tol:
        return f(orbit[i])

    p[0] = r[0]
    for i in range(max_iter):

        alpha = step(orbit[i], p[i], A, b)
        orbit[i+1] = (orbit[i] + alpha * p[i]).ravel()
        r[i+1] = r[i] - alpha * (A @ p[i])

        if la.norm(r[i+1]) < tol:
            break

        beta = (r[i+1].T @ r[i+1]) / (r[i].T @ r[i])
        p[i+1] = r[i+1] + beta * p[i]

    return orbit[:i+2,], f(orbit[i+1])
```

```python
x0 = np.array([6, 7])
A = np.array([[1, 0], [0, 3]])
b = np.array([0,0])
c = 0.0

xs, minf = cg(x0, A, b, c)
print(xs)
print(minf)
```

    [[ 6.00000000e+00  7.00000000e+00]
     [ 3.89403974e+00 -3.70860927e-01]
     [ 4.44089210e-16  2.22044605e-16]]
    1.7256332301709633e-31

```python
x = np.linspace(-10, 10, 100)
y = np.linspace(-10, 10, 100)
X, Y = np.meshgrid(x, y)
plt.contour(X, Y, f2(X, Y), 10)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
pass
```

```python

```


```python

```

# sympy.ntheory に脇道

https://docs.sympy.org/latest/modules/ntheory.html#module-sympy.ntheory.modular

```python
# エラトステネスの篩
from sympy import sieve
sieve._reset()
display(25 in sieve) # 25 は素数表 sieve にあるか => False
display(sieve) # 単純な素数表ではないのだ!!!!
display(type(sieve))
display(type(sieve._list))
display(sieve._list)
display(sieve[3])
display([i for i in sieve._list])
```

    False

    <prime sieve (9): 2, 3, 5, ... 19, 23
    totient sieve (6): 0, 1, 1, ... 2, 4
    mobius sieve (6): 0, 1, -1, ... 0, -1>

    sympy.ntheory.generate.Sieve

    array.array

    array('l', [2, 3, 5, 7, 11, 13, 17, 19, 23])

    5

    [2, 3, 5, 7, 11, 13, 17, 19, 23]

```python
# same as 30 in sieve without return
from sympy import sieve
sieve._reset()
sieve.extend(30)
display(sieve[3]==5)
display(sieve._list)
```

    True

    array('l', [2, 3, 5, 7, 11, 13, 17, 19, 23, 29])

```python
# nth prime
from sympy import sieve, prime
sieve._reset()
n=20
sieve.extend_to_no(n)
display(sieve[n])
display(prime(n))
display(sieve._list)
```

    71

    71

    array('l', [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89])

```python
# isprime
from sympy import sieve,prime,isprime
sieve._reset()
display(isprime(1299709))
display(sieve._list)
display(sieve[10])
display(sieve._list)
```

    True

    array('l', [2, 3, 5, 7, 11, 13])

    29

    array('l', [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31])

```python
# mobiusrange(a,b) # b is outside of range
# 下の例では 7,8,9,10,11,12,13,14,15,16,17 についてメビウス関数の値を示す
# メビウス関数は素因数分解して中に平方を含んでいれば 0 => 8,9,12,16
# 因子が奇数個の場合は -1 => 7,11,13,17
# 因子が偶数個の場合は 1 => 10,14,15
# 従って素数とは関係ない
from sympy import sieve
print([i for i in sieve.mobiusrange(7, 18)])

```

    [-1, 0, 0, 1, -1, 0, -1, 1, 1, 0, -1]

```python
# primerange
from sympy import sieve
print([i for i in sieve.primerange(7, 18)])
```

    [7, 11, 13, 17]

```python
# primerange と同じものを書いてみる
from sympy import sieve
n=30
sieve.extend_to_no(n)
display([i for i in sieve._list if (i >= 7) & (i < 18)])
```

    [7, 11, 13, 17]

```python
# search 与えられた値の上下の素数を tuple で返す
from sympy import sieve
n = 8
display(sieve.search(n))
a, b = sieve.search(n)
display(sieve[a], sieve[b])
```

    (4, 5)

    7

    11

totient

オイラーのトーシェント関数とは、正の整数 $n$ に対して、 $n$ と互いに素である 1 以上 $n$ 以下の自然数の個数 $\varphi (n)$ を与える数論的関数 $\varphi$ である

```python
# totientrange range について totient 数を返す
# 素数の場合 n-1 個になる
from sympy import sieve
print([i for i in sieve.totientrange(7, 18)])
```

    [6, 4, 6, 4, 10, 4, 12, 6, 8, 8, 16]

```python
# nth prime n番目の素数
from sympy import prime
display(prime(10))
display(prime(1))
display(prime(100000))

```

    29

    31

    1299709

```python
#primepi n を含み n より小さい素数の数 primepi(10) => 2,3,5,7 => 4 個
from sympy import *
n = 10
primepi(n)
```

$\displaystyle 4$

```python
# 対数積分 logarithmic integral function オイラーの対数積分
from sympy import *
n = 10
print(li(n))
li(n).evalf()
```

    li(10)

$\displaystyle 6.1655995047873$

```python
# nextprime
from sympy import nextprime
[(i,nextprime(i)) for i in range (10,15)]
```

    [(10, 11), (11, 13), (12, 13), (13, 17), (14, 17)]

```python
# prevprime
from sympy import prevprime
[(i,prevprime(i)) for i in range (10,15)]
```

    [(10, 7), (11, 7), (12, 11), (13, 11), (14, 13)]

```python
# primerange, sieve
from sympy import primerange, sieve
print([i for i in primerange(1,30)])
print([i for i in sieve.primerange(1,30)])
print(list(sieve.primerange(1,30)))

```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]

```python
# randprime, isprime
from sympy import randprime, isprime
print(randprime(1,30))
print(isprime(randprime(1,30)))
```

    17
    True

```python
from sympy.ntheory.generate import primorial, primerange
from sympy import factorint, Mul, primefactors, sqrt
print(primorial(4)) # the first 4 primes are 2, 3, 5, 7
print(primorial(4, nth=False)) # primes <= 4 are 2 and 3
print(primorial(1))
print(primorial(1, nth=False))
print(sqrt(101).evalf())
print(primorial(sqrt(101), nth=False))
print(factorint(primorial(4) + 1))
print(factorint(210))
print(factorint(primorial(4) - 1))
p = list(primerange(10, 20)) # generate all primes in a given range
print(p)
print(sorted(set(primefactors(Mul(*p) + 1)).difference(set(p))))

```

    210
    6
    2
    1
    10.0498756211209
    210
    {211: 1}
    {2: 1, 3: 1, 5: 1, 7: 1}
    {11: 1, 19: 1}
    [11, 13, 17, 19]
    [2, 5, 31, 149]

```python
from sympy.ntheory.generate import cycle_length
func = lambda i: (i**2 + 1) % 51
print(next(cycle_length(func, 4)))
n = cycle_length(func, 4, values=True)
print(list(ni for ni in n))
print(next(cycle_length(func,4,nmax=4)))
print([ni for ni in cycle_length(func,4,nmax=4,values=True)])
```

    (6, 2)
    [17, 35, 2, 5, 26, 14, 44, 50, 2, 5, 26, 14]
    (4, None)
    [17, 35, 2, 5]

```python
from sympy import composite
print(composite(36))
print(composite(1))
print(composite(17737))

```

    52
    4
    20000

```python
from sympy import compositepi
print(compositepi(25))
print(compositepi(1))
print(compositepi(1000))
```

    15
    0
    831

```python
from sympy.ntheory.factor_  import smoothness
print(smoothness(2**7*3**2))
print(smoothness(2**4*13))
print(smoothness(2))
```

    (3, 128)
    (13, 16)
    (2, 2)

```python
from sympy.ntheory.factor_  import smoothness_p
print(smoothness_p(10431, m=1))
print(smoothness_p(10431))
print(smoothness_p(10431,power=1))
print(smoothness_p(21477639576571, visual=1))
```

    (1, [(3, (2, 2, 4)), (19, (1, 5, 5)), (61, (1, 31, 31))])
    (-1, [(3, (2, 2, 2)), (19, (1, 3, 9)), (61, (1, 5, 5))])
    (-1, [(3, (2, 2, 2)), (61, (1, 5, 5)), (19, (1, 3, 9))])
    p**i=4410317**1 has p-1 B=1787, B-pow=1787
    p**i=4869863**1 has p-1 B=2434931, B-pow=2434931

```python
print(factorint(17*9))
print(smoothness_p(factorint(17*9)))
print(smoothness_p(smoothness_p(factorint(17*9))))
```

    {3: 2, 17: 1}
    p**i=3**2 has p-1 B=2, B-pow=2
    p**i=17**1 has p-1 B=2, B-pow=16
    {3: 2, 17: 1}

```python
from sympy import trailing
print(trailing(128))
print(trailing(63))

```

    7
    0

```python
from sympy.ntheory import multiplicity
from sympy.core.numbers import Rational as R
print([multiplicity(5, n) for n in [8, 5, 25, 125, 250]])
print(multiplicity(3, R(1, 9)))

```

    [0, 1, 2, 3, 3]
    -2

https://docs.sympy.org/latest/modules/ntheory.html#module-sympy.ntheory.modular


#  Taylor Examples 5.4 and 5.5

last revised: 17-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

Here we are exploring the Fourier series for a waveform defined to be even about the origin, so $f(-t) = +f(t)$, with period $\tau$.  That means that the integrand for the $b_m$ coefficients is odd and so all of the corresponding integrals vanish.

The particular wave of interest here is a periodic rectangular pulse, such that in the interval $-\tau/2 \leq t \leq \tau/2$, the function takes the form:

$\newcommand{\fmax}{f_{\textrm{max}}}$
$\begin{align}
  f(t) = \left\{ \begin{array}{lc}
                   \fmax & -\Delta\tau/2 < t < \Delta\tau/2 \\
                   0 & |t| > \Delta\tau/2
                 \end{array}
                   \right.
\end{align}$

As usual, we do the $a_0$ integral separately and then $a_m$ for $m \geq 1$.  Here $\omega \equiv 2\pi/\tau$.  The results are (also see the text):

$\begin{align}
  a_0 = \frac{1}{\tau} \int_{-\tau/2}^{\tau/2} f(t)\, dt = \frac{\fmax\Delta\tau}{\tau}
\end{align}$

$\begin{align}
  a_m = \frac{2}{\tau} \int_{-\tau/2}^{\tau/2} \cos(m\omega t) f(t)\, dt =
      \frac{4\fmax}{\tau} \int_0^{\Delta\tau/2} \cos(2\pi m t/\tau) \, dt
    %  &= - \frac{2\fmax}{m\pi}\left[\cos(m\pi t)\right]^1_0 \\
      =
        \frac{2 \fmax}{m\pi} \sin\left(\frac{\pi m \Delta\tau}{\tau}\right)
\end{align}$

## Define the functions we'll need

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import quad
```

We start by defining a function for the rectangular pulse at any $t$.  The definition here is for a scalar function.  That is, it won't work to call it with $t$ and array of time points, unlike other functions we have defined.  It is possible to make it work, but then the function will be much less clear.  When we need to evaluate it for all elements of an array `t_pts`, we will use the construction: <br/> `np.array([rectangular(t) for t in t_pts])`.

```python
def rectangular(t, tau=1., Delta_tau=0.25, f_max=1.):
    """Returns the triangular wave of amplitude f_max and even about the
        origin at time t.  The period is tau.  It is defined as a scalar
        function (i.e., only one value of t can be passed at a time).
        **There must be a better way to define this!**
    """
    t_adjust = t - round(t)  # shift t so we can test for being within Delta_tau

    if np.fabs(t_adjust) < Delta_tau/2.:
        return f_max
    else:
        return 0.
```

Now a function that creates an array of Fourier coefficients for the triangular wave up to order N_max.

```python
def rectangular_coeffs_by_hand(N_max, tau=1., Delta_tau=0.25, f_max=1):
    """Fourier coefficients calculated by hand and loaded into an array
        as pairs in the form (a_n, b_n).
    """
    # initialize the array with the vaues of a_0 and b_0
    coeffs_array = [(f_max * Delta_tau / tau, 0)]
    for n in np.arange(1, N_max, 1):
        # these are pre-calcualated values of a_n and b_n
        a_n = 2.*f_max/(n * np.pi) * np.sin((np.pi * n * Delta_tau) / tau)
        b_n = 0.
        coeffs_array.append((a_n, b_n))  # add them to the end of the array
    return np.array(coeffs_array)  # convert to a numpy array to be safe

```

We would like a general way to construct the array of Fourier coefficients given any periodic function.  Our first pass at that uses a class definition and the scipy integration function quad.

```python
class FourierSeries():
    """
    Fourier series class finds the coefficients in a Fourier series with
    period tau up to a specified order.
    Assume these imports:
        from scipy.integrate import quad
        import numpy as np
    ***Need to add args to the function.***
    """
    def __init__(self,
                 function,
                 tau=1.,
                 N_max=10
                ):
        self.function = function
        self.tau = tau
        self.omega = 2. * np.pi / tau
        self.N_max = N_max
        # add something to quit if Nmax < 0 or not an integer (try and except)

    def a0_calc(self):
        """Calculate the constant Fourier coefficient by integration"""
        answer, error = quad(self.function, -tau/2., tau/2., args=(tau,))
        return (1./self.tau) * answer

    def an_integrand(self, t, n):
        """Integrand for the nth cosine coefficient"""
        return self.function(t, tau) * np.cos(n * self.omega * t)

    def an_calc(self, n):
        """Calculate the nth cosine coefficient (n > 0)"""
        # note comma after n in args
        answer, error = quad(self.an_integrand, -tau/2., tau/2., args=(n,))
        return (2./self.tau) * answer

    def bn_integrand(self, t, n):
        """Integrand for the nth cosine coefficient"""
        return self.function(t, tau) * np.sin(n * self.omega * t)

    def bn_calc(self, n):
        """Calculate the nth cosine coefficient (n > 0)"""
        answer, error = quad(self.bn_integrand, -tau/2., tau/2., args=(n,))
        return (2./self.tau) * answer

    def coeffs_upto_Nmax(self):
        """Calculate the Fourier series up to Nmax"""
        # first generate the coefficient
        coeffs_array = [(self.a0_calc(), 0)]   # a_0 and b_0

        for n in np.arange(1, N_max, 1):
            a_n = self.an_calc(n)
            b_n = self.bn_calc(n)
            coeffs_array.append((a_n, b_n))  # append a tuple of coefficients

        return np.array(coeffs_array)  # convert to a numpy array

```

Finally, we need a function that can take as input an array of t values and an array of Fourier coefficients and return the function at those t values with terms up to order N_max.

```python

def Fourier_reconstruct(t_pts, coeffs_array, tau, N_max):
    """Sum up the Fourier series up to n = N_max terms."""
    omega = 2. * np.pi / tau   # definition of omega
    result = 0.
    # iterate over coefficients but only up to N_max
    for n, (a,b) in enumerate(coeffs_array[:N_max+1]):
        result = result + a * np.cos(n * omega * t_pts) \
                        + b * np.sin(n * omega * t_pts)
    return result
```

## Example 5.4

Ok, now we can do Example 5.4.  Calculate the coefficients both ways.

```python
N_max = 20
tau = 1.
f_max = 1.
Delta_tau = 0.25
coeffs_by_hand = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

fs = FourierSeries(rectangular, tau, N_max)
coeffs_by_quad = fs.coeffs_upto_Nmax()
```

Let's check that the exact and numerical calculation of the coefficients agree.

(Note the space in the formats below, e.g., `{a1: .6f}`.  This means to leave an extra space for a positive number so that it aligns at the decimal point with negative numbers.)

```python
print(' n   a_exact     a_quad      b_exact     b_quad')
for n, ((a1,b1), (a2,b2)) in enumerate(zip(coeffs_by_hand, coeffs_by_quad)):
    print(f'{n:2d}  {a1: .6f}  {a2: .6f}    {b1: .6f}  {b2: .6f}')
```

     n   a_exact     a_quad      b_exact     b_quad
     0   0.250000   0.250000     0.000000   0.000000
     1   0.450158   0.450158     0.000000   0.000000
     2   0.318310   0.318310     0.000000   0.000000
     3   0.150053   0.150053     0.000000   0.000000
     4   0.000000   0.000000     0.000000   0.000000
     5  -0.090032  -0.090032     0.000000   0.000000
     6  -0.106103  -0.106103     0.000000   0.000000
     7  -0.064308  -0.064308     0.000000   0.000000
     8  -0.000000   0.000000     0.000000   0.000000
     9   0.050018   0.050018     0.000000   0.000000
    10   0.063662   0.063662     0.000000   0.000000
    11   0.040923   0.040923     0.000000   0.000000
    12   0.000000   0.000000     0.000000   0.000000
    13  -0.034628  -0.034628     0.000000   0.000000
    14  -0.045473  -0.045473     0.000000   0.000000
    15  -0.030011  -0.030011     0.000000   0.000000
    16  -0.000000   0.000000     0.000000   0.000000
    17   0.026480   0.026480     0.000000   0.000000
    18   0.035368   0.035368     0.000000   0.000000
    19   0.023693   0.023693     0.000000   0.000000

Make the comparison plot requested: N_max = 2 vs. N_max = 6.

```python
t_pts = np.arange(-2., 4., .01)
f_pts_2 = Fourier_reconstruct(t_pts, coeffs_by_hand, tau, 2)
f_pts_6 = Fourier_reconstruct(t_pts, coeffs_by_hand, tau, 6)

# Python way to evaluate the rectangular function at an array of points:
#   * np.array creates a numpy array;
#   * note the []s around the inner statement;
#   * rectangular(t) for t in t_pts
#       means step through each element of t_pts, call it t, and
#       evaluate rectangular at that t.
#   * This is called a list comprehension.  There are more compact ways,
#      but this is clear and easy to debug.
rectangular_t_pts = np.array([rectangular(t, tau, Delta_tau, f_max) \
                              for t in t_pts])
```

```python
# It looks like we should define a function for the axis!

fig_1 = plt.figure(figsize=(10,5))

ax_1 = fig_1.add_subplot(1,2,1)
ax_1.plot(t_pts, f_pts_2, label='N = 2', color='blue')
ax_1.plot(t_pts, rectangular_t_pts, label='exact', color='red')
ax_1.set_xlim(-1.1,4.1)
ax_1.set_xlabel('t')
ax_1.set_ylabel('f(t)')
ax_1.set_title('N = 2')
ax_1.legend()

ax_2 = fig_1.add_subplot(1,2,2)
ax_2.plot(t_pts, f_pts_6, label='N = 6', color='blue')
ax_2.plot(t_pts, rectangular_t_pts, label='exact', color='red')
ax_2.set_xlim(-1.1,4.1)
ax_2.set_xlabel('t')
ax_2.set_ylabel('f(t)')
ax_2.set_title('N = 6')
ax_2.legend();

fig_1.tight_layout()
fig_1.savefig('example_5.4.png')
```

## Example 5.5

We consider a driven oscillator with an even driving force so only cosine terms are non-zero in the Fourier transform of $f(t)$.  Linearity of the equation then means that the solution takes the form

$\begin{align}
  x(t) = \sum_{n=0}^{\infty} A_n \cos(n\omega t - \delta_n) \;,
\end{align}$

where

$\begin{align}
  A_n &= \frac{f_n}{\sqrt{(\omega_0^2 - n^2 \omega^2)^2 + 4\beta^2 n^2 \omega^2}}  \\
  \delta_n &= \arctan\left(\frac{2\beta n\omega}{\omega_0^2 - n^2 \omega^2}  \right)
\end{align}$

For Example 5.5 we apply this to the rectangular driving force, so we can use the coefficients from Example 5.4, which play the role of the $f_n$ coefficients here.

```python
def driven_coefficients(tau_0, tau, beta, coeffs):
    """Calculate the coefficients A_n and delta_n for a driven oscillator."""
    omega_0 = 2. * np.pi / tau_0
    omega = 2. * np.pi / tau

    A_n = []
    delta_n = []

    # the second entry in the tuple is called dummy because it is not used
    for n, (f_n, dummy) in enumerate(coeffs):
        A_n.append(f_n / np.sqrt((omega_0**2 - n**2 * omega**2)**2 +
                                  (2. * beta * n *omega)**2))
        delta_n.append(np.arctan2(2. * beta * n *omega,
                                  omega_0**2 - n**2 * omega**2 ))
    return np.array(A_n), np.array(delta_n)

def x_drive(t_pts, tau_0, tau, beta, N_max, coeffs):
    """Sum up the Fourier series for x(t) up to n = N_max terms."""
    omega = 2. * np.pi / tau
    A_n_array, delta_n_array = driven_coefficients(tau_0, tau, beta, coeffs)

    result = 0.
    # iterate over coefficients but only up to N_max
    for n, (A_n, delta_n) in enumerate(zip(A_n_array[:N_max+1],
                                           delta_n_array[:N_max+1])):
        result = result + A_n * np.cos( n * omega * t_pts - delta_n)
    return result
```

Ok, let's do Example 5.5!

```python
N_max = 20
tau = 1.
f_max = 1.
Delta_tau = 0.25
coeffs_by_hand = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

```

```python
t_pts = np.arange(0., 7., .01)

# first set of parameters
N_max = 20
f_max = 1.
Delta_tau = 0.25
tau = 1.
coeffs_by_hand_1 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_1 = 1.
beta = 0.2
A_n_array_1, delta_n_array_1 = driven_coefficients(tau_0_1, tau, beta,
                                                   coeffs_by_hand_1)
x_pts_1 = x_drive(t_pts, tau_0_1, tau, beta, N_max, coeffs_by_hand_1)

# now only tau is varied
tau = 1.5
coeffs_by_hand_2 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_2 = 1.
beta = 0.2
A_n_array_2, delta_n_array_2 = driven_coefficients(tau_0_2, tau, beta,
                                                   coeffs_by_hand_2)
x_pts_2 = x_drive(t_pts, tau_0_2, tau, beta, N_max, coeffs_by_hand_2)

# now only tau is varied
tau = 2.0
coeffs_by_hand_3 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_3 = 1.
beta = 0.2
A_n_array_3, delta_n_array_3 = driven_coefficients(tau_0_3, tau, beta,
                                                   coeffs_by_hand_3)
x_pts_3 = x_drive(t_pts, tau_0_3, tau, beta, N_max, coeffs_by_hand_3)

# now only tau is varied
tau = 2.5
coeffs_by_hand_4 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_4 = 1.
beta = 0.2
A_n_array_4, delta_n_array_4 = driven_coefficients(tau_0_4, tau, beta,
                                                   coeffs_by_hand_4)
x_pts_4 = x_drive(t_pts, tau_0_4, tau, beta, N_max, coeffs_by_hand_4)

```

How would you code the last cell differently so that you could just give an array of $\tau$ values?

Compare coefficients for the cases from parts 1 to 4: large $A_n$ coefficients when that component is in resonance.

```python
print(' n      case 1        case 2        case 3        case 4')
for n, (A_n_1, A_n_2, A_n_3, A_n_4) in enumerate(zip(A_n_array_1, A_n_array_2,
                                                     A_n_array_3, A_n_array_4)):
    print(f'{n:2d}   {A_n_1: .8f}   {A_n_2: .8f}   {A_n_3: .8f}   {A_n_4: .8f}')
```

     n      case 1        case 2        case 3        case 4
     0    0.00633257    0.00422172    0.00316629    0.00253303
     1    0.17911224    0.01447102    0.00822068    0.00592958
     2    0.00268521    0.00892472    0.08955612    0.01303477
     3    0.00047497    0.00179014    0.00396133    0.00973767
     4    0.00000000    0.00057109    0.00134261    0.00245255
     5   -0.00009501    0.00015945    0.00056729    0.00107408
     6   -0.00007678    0.00000000    0.00023749    0.00053672
     7   -0.00003393   -0.00005543    0.00007835    0.00027238
     8   -0.00000000   -0.00006360    0.00000000    0.00012820
     9    0.00001584   -0.00005119   -0.00003562    0.00004629
    10    0.00001629   -0.00003214   -0.00004751    0.00000000
    11    0.00000864   -0.00001389   -0.00004630   -0.00002467
    12    0.00000000   -0.00000000   -0.00003839   -0.00003583
    13   -0.00000522    0.00000837   -0.00002778   -0.00003854
    14   -0.00000591    0.00001158   -0.00001697   -0.00003608
    15   -0.00000339    0.00001086   -0.00000745   -0.00003071
    16   -0.00000000    0.00000774   -0.00000000   -0.00002399
    17    0.00000233    0.00000372    0.00000509   -0.00001696
    18    0.00000277    0.00000000    0.00000792   -0.00001036
    19    0.00000167   -0.00000266    0.00000879   -0.00000462

Now for the graphs in Figure 5.25:

```python
fig_2 = plt.figure(figsize=(10,5))

ax_1 = fig_2.add_subplot(2,2,1)
ax_1.plot(t_pts, x_pts_1, color='blue')
ax_1.set_xlim(0., 7.)
ax_1.set_ylim(-0.2, 0.2)
ax_1.set_xlabel('t')
ax_1.set_ylabel('x(t)')
ax_1.set_title(r'case (1): $\tau = 1$')

ax_2 = fig_2.add_subplot(2,2,2)
ax_2.plot(t_pts, x_pts_2, color='blue')
ax_2.set_xlim(0., 7.)
ax_2.set_ylim(-0.2, 0.2)
ax_2.set_xlabel('t')
ax_2.set_ylabel('x(t)')
ax_2.set_title(r'case (2): $\tau = 1.5$')

ax_3 = fig_2.add_subplot(2,2,3)
ax_3.plot(t_pts, x_pts_3, color='blue')
ax_3.set_xlim(0., 7.)
ax_3.set_ylim(-0.2, 0.2)
ax_3.set_xlabel('t')
ax_3.set_ylabel('x(t)')
ax_3.set_title(r'case (3): $\tau = 2.0$')

ax_4 = fig_2.add_subplot(2,2,4)
ax_4.plot(t_pts, x_pts_4, color='blue')
ax_4.set_xlim(0., 7.)
ax_4.set_ylim(-0.2, 0.2)
ax_4.set_xlabel('t')
ax_4.set_ylabel('x(t)')
ax_4.set_title(r'case (4): $\tau = 2.5$')

fig_2.tight_layout()
fig_2.savefig('Example_.png')
```

To add additional graphs with more $\tau$ values, we should really define a function that produces the plot given a $\tau$.

```python

```