This is a wish-list of models to still be implemented computationally (from Mellow & Beranek 2012):

> circular piston in a closed circular baffle
> circular piston in a rigid sphere
>

### Troubleshooting ideas:
> Theano numeric computation: https://docs.sympy.org/latest/modules/numeric-computation.html

### Circular piston in a rigid sphere

eqn. 12.111: $D(\theta)=-\frac{4}{k^{2}R^{2}sin^{2}\alpha} \sum_{n=0}^{N} A_{n}j^{n}P_{n}cos(\theta) $

* Here the unknown is $A_{n}$
    * There is an $A_{n}$ defined in eqn. 12.102 -- it is the result of a matrix inversion $a=M^{-1}.b$

From the explanations of 12.7-12.8 we know tht $P_{n}$ is the legendre function, $j_{n}$ is the Bessel function and $h_{n}^{(2)}$ is the spherical Hankel function (eqn.2.18)

```python
#from joblib import Parallel, delayed
#from gmpy2 import *
import matplotlib.pyplot as plt
import mpmath
dps = 150
mpmath.mp.dps = dps
import numpy as np
from scipy.special import jv as bessel_firstkind
#from symengine import *
import sympy
from sympy import besselj, symbols, hankel2, legendre, sin, cos, tan, summation, I, oo, diff, pi
from sympy import factor_terms, Matrix
from sympy import Abs, lambdify, integrate, expand,integrate, Integral
from sympy.printing.theanocode import theano_function
from sympy.utilities.autowrap import autowrap
#import tqdm

```

```python
mpmath.mp.dps
```

    50

```python
x, alpha, k, m,n, r1, R, theta = symbols('x alpha k m n r1 R theta')
r1 = (R*cos(alpha))/cos(theta)
```

```python
legendre_func = lambdify((m, x), legendre(m, x))
```

```python
r1
```

$\displaystyle \frac{R \cos{\left(\alpha \right)}}{\cos{\left(\theta \right)}}$

### Getting $A_{n}$
To get $A_{n}$, we need to solve a matrix equation $\textbf{a}=\textbf{M}^{-1}.\textbf{b}$.

Here, $M$ is given by a term that involves the $I_{mn}$ and $K_{mn}$ variables, while $b$ involves the $L_{m}$ equation.

```python
from sympy import N
```

```python
# eqn 12.108
Lm_expr = expand(legendre(m, cos(theta))*(r1**2/R**2)*tan(theta))#

Lm = Integral(Lm_expr, (theta,0,alpha))#.doit(meijerg=True)
Lm_func = lambdify((m,R,alpha), Lm)
# !!! Lm.subs({'alpha':np.radians(10),'m':10,'R':0.1}).evalf() --- this code runs the actual evaluation
def Lm_func_high(mv,Rv,alphav):
    return N(Lm.subs({'R':Rv,'m':mv,'alpha':alphav}),dps)
```

```python
%%time
Lm_func_high(1,0.1,0.02)

```

    Wall time: 1.48 s

$\displaystyle 0.00019995333608880826128958077496034040466962346673268$

```python
from sympy.utilities.autowrap import binary_function
```

```python

Lm_expr = expand(legendre(m, cos(theta))*(r1**2/R**2)*tan(theta))#

Lm = Integral(Lm_expr, (theta,0,alpha))#.doit(meijerg=True)ttt = theta*alpha - R
```

```python
# eqn 12.107
Kmn_expr = expand(legendre(n, cos(theta))*legendre(m, cos(theta))*sin(theta) )
Kmn = Integral(Kmn_expr, (theta, alpha, pi)).doit(meijerg=True)
# Kmn.subs({'alpha':np.radians(10),'m':1,'n':10}).evalf()
Kmn_func = lambdify((m,n,alpha), Kmn)
# Kmn_func(1050,500,np.radians(10))
def Kmn_func_high(mv,nv,alphav):
    return N(Kmn.subs({'m':mv, 'n':nv, 'alpha':alphav}),dps)
```

```python
%%time
Kmn_func_high(20,5,np.radians(10))
```

    Wall time: 16 s

$\displaystyle -0.0011604202936401205578419788497048185692103074007768$

```python

```

```python
%%time
Kmn_func(10,20,mpmath.pi/18)
```

    Wall time: 15.6 ms

    -0.0017275474626486497

```python
Kmn
```

$\displaystyle \int\limits_{\alpha}^{\pi} \sin{\left(\theta \right)} P_{m}\left(\cos{\left(\theta \right)}\right) P_{n}\left(\cos{\left(\theta \right)}\right)\, d\theta$

```python
Imn_part1 = (n*hankel2(n-1,k*r1)-(n+1)*hankel2(n+1,k*r1))*legendre(n, cos(theta))*cos(theta)
Imn_part2 = n*(n+1)*hankel2(n, k*r1)*(legendre(n-1, cos(theta)-legendre(n+1, cos(theta))))/k*r1
```

```python
Imn_parts = expand(Imn_part1+Imn_part2)
Imn_expr = expand(Imn_parts*legendre(m,cos(theta))*(r1**2/R**2)*tan(theta))
Imn = Integral(Imn_expr, (theta, 0, alpha)).doit(meijerg=True)
Imn_func = lambdify((alpha,R, k,m,n), Imn,'mpmath')

def Imn_func_high(alphav, Rv, kv, mv, nv):
    return N( Imn.subs({'alpha':alphav,'R':Rv, 'k':kv, 'm':mv, 'n':nv}),dps)

```

```python
%%time
Imn_func(mpmath.pi/18,mpmath.mpf(0.1),mpmath.mpf(20),20,0)
```

    Wall time: 23.1 s

    mpc(real='-0.000481745016183522122698045278138546527289926984966478791046650405130998691266830729500442525581983133148112640669110781418632370398942177784095492511574623', imag='-0.000124351573321090795887446460006512869271041273011308622103014789453656548348031722339368859322341381158873137319718527970215362686202390275236048001567065')

mpc(real='-0.000481745016183522122698045278138546527289926984966478791046650405130998691266830729500442525581983133148112640669110781418632370398942177784095492511574623', imag='-0.000124351573321090795887446460006512869271041273011308622103014789453656548348031722339368859322341381158873137319718527970215362686202390275236048001567065')

```python
Imn
```

$\displaystyle - \frac{\left(\int\limits_{0}^{\alpha} \frac{k \tan{\left(\theta \right)} H^{(2)}_{n + 1}\left(\frac{R k \cos{\left(\alpha \right)}}{\cos{\left(\theta \right)}}\right) P_{m}\left(\cos{\left(\theta \right)}\right) P_{n}\left(\cos{\left(\theta \right)}\right)}{\cos{\left(\theta \right)}}\, d\theta + \int\limits_{0}^{\alpha} \left(- \frac{k n \tan{\left(\theta \right)} H^{(2)}_{n - 1}\left(\frac{R k \cos{\left(\alpha \right)}}{\cos{\left(\theta \right)}}\right) P_{m}\left(\cos{\left(\theta \right)}\right) P_{n}\left(\cos{\left(\theta \right)}\right)}{\cos{\left(\theta \right)}}\right)\, d\theta + \int\limits_{0}^{\alpha} \frac{k n \tan{\left(\theta \right)} H^{(2)}_{n + 1}\left(\frac{R k \cos{\left(\alpha \right)}}{\cos{\left(\theta \right)}}\right) P_{m}\left(\cos{\left(\theta \right)}\right) P_{n}\left(\cos{\left(\theta \right)}\right)}{\cos{\left(\theta \right)}}\, d\theta + \int\limits_{0}^{\alpha} \left(- \frac{R n \cos{\left(\alpha \right)} \tan{\left(\theta \right)} H^{(2)}_{n}\left(\frac{R k \cos{\left(\alpha \right)}}{\cos{\left(\theta \right)}}\right) P_{m}\left(\cos{\left(\theta \right)}\right) P_{n - 1}\left(\cos{\left(\theta \right)} - P_{n + 1}\left(\cos{\left(\theta \right)}\right)\right)}{\cos^{3}{\left(\theta \right)}}\right)\, d\theta + \int\limits_{0}^{\alpha} \left(- \frac{R n^{2} \cos{\left(\alpha \right)} \tan{\left(\theta \right)} H^{(2)}_{n}\left(\frac{R k \cos{\left(\alpha \right)}}{\cos{\left(\theta \right)}}\right) P_{m}\left(\cos{\left(\theta \right)}\right) P_{n - 1}\left(\cos{\left(\theta \right)} - P_{n + 1}\left(\cos{\left(\theta \right)}\right)\right)}{\cos^{3}{\left(\theta \right)}}\right)\, d\theta\right) \cos^{2}{\left(\alpha \right)}}{k}$

```python
M_mn = (Imn + (n*hankel2(n-1,k*R) - (n+1)*hankel2(n+1,k*R) )*Kmn)/(2*n+1).doit(meijerg=True)
M_mn_func = lambdify((m,n,k,R,alpha), M_mn,'sympy')
def M_mn_func_high(mv,nv,kv,Rv,alphav):
    return N(M_mn.subs({'m':mv,'n':nv,'k':kv,'R':Rv,'alpha':alphav}),dps)

```

```python
b = -I*Lm
b_func = lambdify((m, alpha),b,'sympy')
```

```python
b_func(1, 0.1)
```

$\displaystyle - 0.990033288920621 i \int\limits_{0}^{0.1} \frac{\tan{\left(\theta \right)}}{\cos{\left(\theta \right)}}\, d\theta$

```python
Nv = 10
M_matrix = np.zeros((Nv,Nv),dtype='complex')
b_matrix = np.zeros(Nv,dtype='complex')

frequency = 50*10**3 # kHz
vsound = 330 # m/s
wavelength = vsound/frequency
alpha_value = np.radians(60)
k_value = 2*np.pi/(wavelength)
ka = 5
a_value = ka/k_value
R_value = a_value/np.sin(alpha_value) # m

```

```python
M_matrix = Matrix.zeros(Nv, Nv)
b_matrix = Matrix.zeros(Nv, 1)

rowcol = [ ]
for eachn in range(Nv):
    for eachm in range(Nv):
        rowcol.append((eachm,eachn))

#M_values = Parallel(n_jobs=4)( delayed(M_mn_func)(pos[0], pos[1], k_value, R_value, alpha_value)  for pos in tqdm.tqdm(rowcol))
M_values = [M_mn_func(pos[0], pos[1], k_value, R_value, alpha_value)  for pos in tqdm.tqdm(rowcol)]

```

    100%|██████████| 100/100 [00:40<00:00,  2.46it/s]

```python
for each_m in range(Nv):
    #M_matrix[each_m, each_n] = M_mn_func(each_m, each_n, k_value, R_value, alpha_value)
    b_matrix[each_m] = b_func(each_m, alpha_value)
for i,pos in enumerate(rowcol):
    M_matrix[pos[0],pos[1]] = M_values[i]
```

```python
M_mn_sol = M_mn.doit()
```

```python
N(M_mn_sol.subs({'m':0,'n':1,'k':k_value,'R':R_value,'alpha':alpha_value}))
```

$\displaystyle -0.105381314271948 - 0.089262973728071 i$

```python
M_mn_func_high(0,1,k_value,R_value,alpha_value)
```

$\displaystyle 0.11320907906257618111534459172734423801984539569991 - 0.053267986414244894183365876387248819434337199587735 i$

```python
M_matrix.shape
```

    (10, 10)

```python
a_matrix = M_matrix.LUsolve(b_matrix)
```

```python
b_matrix[0]
```

$\displaystyle - 0.25 i \int\limits_{0}^{1.0471975511966} \frac{\tan{\left(\theta \right)}}{\cos^{2}{\left(\theta \right)}}\, d\theta$

```python
M_matrix_inv = np.linalg.inv(M_matrix)
a_matrix = np.linalg.solve(M_matrix, b_matrix)
#a_matrix = np.matmul(M_matrix_inv,b_matrix)
```

```python
Ma = np.matmul(M_matrix,a_matrix)
```

```python
def d_theta(angle,k_v,R_v,alpha_v,An):
    num = 4
    N_v = An.size
    denom  = (k_v**2)*(R_v**2)*np.sin(alpha_v)**2
    part1 = num/denom
    jn_matrix = np.array([1j**f for f in range(N_v)])
    legendre_matrix = np.array([legendre_func(n_v, np.cos(angle)) for n_v in range(N_v)])
    part2_matrix = np.column_stack((An, jn_matrix, legendre_matrix))
    part2 = np.sum(np.apply_along_axis(lambda X: X[0]*X[1]*X[2], 1, part2_matrix))
    rel_level = - part1*part2
    return rel_level

def d_zero(k_v,R_v,alpha_v,An):
    num = 4
    N_v = An.size
    denom  = (k_v**2)*(R_v**2)*np.sin(alpha_v)**2
    part1 = num/denom
    jn_matrix = np.array([1j**f for f in range(N_v)])
    part2_matrix = np.column_stack((An, jn_matrix))
    part2 = np.sum(np.apply_along_axis(lambda X: X[0]*X[1], 1, part2_matrix))
    rel_level = - part1*part2
    return rel_level

def relative_directivity_db(angle,k_v,R_v,alpha_v,An):
    off_axis = d_theta(angle,k_v,R_v,alpha_v,An)
    on_axis = d_zero(k_v,R_v,alpha_v,An)
    rel_level = 20*np.log10(abs(off_axis/on_axis))
    return rel_level
```

```python
angles = np.linspace(0,2*np.pi,200)
dirnlty = [relative_directivity_db(angle_v, k_value, R_value,alpha_value,a_matrix) for angle_v in angles]

```

```python
# compare with Fig. 12.23
plt.figure()
a0 = plt.subplot(111, projection='polar')
plt.plot(angles, dirnlty)
plt.ylim(-40,0);plt.yticks(np.arange(-40,10,10))
plt.xticks(np.radians(np.arange(0,210,30)))
```

### Progress notes
* After having implemented the code in Chapter 12, I'm still getting weird directivity results. Even though I increase the $N$ from 10 to 50, and even 100, it doesn't seem to change the final results, which don't match Fig. 12.23 at all. Perhaps $N$ is not the issue at hand but rather the numerical precision at which the numbers need to be represented.  This guess is based on the Tim Mellow code, which specified very high numeric precisions with 300 place decimal representations. The standard ```lambdify``` default is to convert the expression into a numpy or scipy float-based function.
*

```python
from sympy import sinc
```

```python
t = symbols('t', real=True)
def nsinc(t):
    "normalized sinc function"
    return sinc(pi * t)

nsinc(t).integrate(meijerg=True)

```

$\displaystyle \frac{\operatorname{Si}{\left(\pi t \right)}}{\pi}$

```python
Integral(sinc(pi*t)).doit(meijerg=True)
```

$\displaystyle \frac{\operatorname{Si}{\left(\pi t \right)}}{\pi}$

```python

```


# Thomson Scattering: Spectral Density

[thomson]: ../diagnostics/thomson.rst
[spectral-density]: ../api/plasmapy.diagnostics.thomson.spectral_density.rst#spectral-density
[sheffield]: https://www.sciencedirect.com/book/9780123748775/plasma-scattering-of-electromagnetic-radiation

The [thomson.spectral_density][spectral-density] function calculates the [spectral density function S(k,w)][sheffield], which is one of several terms that determine the scattered power spectrum for the Thomson scattering of a probe laser beam by a plasma. In particular, this function calculates $S(k,w)$ for the case of a plasma consisting of one or more ion species and a neutralizing electron fluid under the assumption that all of the ion species and the electron fluid have Maxwellian velocity distribution functions. In this regime, the spectral density is given by the equation:

\begin{equation}
S(k,\omega) = \frac{2\pi}{k} \bigg |1 - \frac{\chi_e}{\epsilon} \bigg |^2 f_{e0}\bigg ( \frac{\omega}{k} \bigg ) + \sum_i \frac{2\pi Z_i}{k} \bigg | \frac{\chi_e}{\epsilon} \bigg |^2 f_{i0, i} \bigg ( \frac{\omega}{k} \bigg )
\end{equation}

where $\chi_e$ is the electron component susceptibility of the plasma and $\epsilon = 1 + \chi_e + \sum_i \chi_i$ is the total plasma dielectric function (with $\chi_i$ being the ion component of the susceptibility), $Z_i$ is the charge of each ion, $k$ is the scattering wavenumber, $\omega$ is the scattering frequency, and the functions $f_{e0}$ and $f_{i0,i}$ are the Maxwellian velocity distributions for the electrons and ion species respectively.

Thomson scattering can be either non-collective (the scattered spectrum is a linear sum of the light scattered by individual particles) or collective (the scattered spectrum is dominated by scattering off of collective plasma waves). The [thomson.spectral_density][spectral-density] function can be used in both cases. These regimes are delineated by the dimensionless constant $\alpha$:

\begin{equation}
\alpha = \frac{1}{k \lambda_{De}}
\end{equation}

where $\lambda_{De}$ is the Debye length. $\alpha > 1$ corresponds to collective scattering, while $\alpha < 1$ corresponds to non-collective scattering. Depending on which of these regimes applies, fitting the scattered spectrum can provide the electron (and sometimes ion) density and temperature. Doppler shifting of the spectrum can also provide a measurement of the drift velocity of each plasma species.

For a detailed explanation of the underlying physics (and derivations of these expressions), see ["Plasma Scattering of Electromagnetic Radiation" by Sheffield et al.][sheffield]

```python
%matplotlib inline

import astropy.units as u
import matplotlib.pyplot as plt
import numpy as np
import warnings

from plasmapy.diagnostics import thomson
from plasmapy.utils.exceptions import ImplicitUnitConversionWarning
```

Construct parameters that define the Thomson diagnostic setup, the probing beam and scattering collection.  These parameters will be used for all examples.

```python
# The probe wavelength can in theory be anything, but in practice integer frequency multiples of the Nd:YAG wavelength
# 1064 nm are used (532 corresponds to a frequency-doubled probe beam from such a laser).
probe_wavelength = 532*u.nm

# Array of wavelengths over which to calcualte the spectral distribution
wavelengths = np.arange(probe_wavelength.value-60, probe_wavelength.value+60, 0.01)*u.nm

# The scattering geometry is defined by unit vectors for the orientation of the probe laser beam (probe_n) and
# the path from the scattering volume (where the measurement is made) to the detector (scatter_n).
# These can be setup for any experimental geometry.
probe_vec = np.array([1, 0, 0])
scattering_angle = np.deg2rad(63)
scatter_vec = np.array([np.cos(scattering_angle), np.sin(scattering_angle), 0])
```

In order to calcluate the scattered spectrum, we must also include some information about the plasma. For this plot we'll allow the ``fract``, ``ion_species``, ``fluid_vel``, and ``ion_vel`` keywords to keep their default values, describing a single-species H+ plasma at rest in the laboratory frame.

```python
ne = 2e17*u.cm**-3
Te = 12*u.eV
Ti = 10*u.eV

# This warning filter catches an ImplicitUnitConversionWarning that results from specifying
# temperatures in eV instead of Kelvin.
with warnings.catch_warnings():
    warnings.simplefilter("ignore", ImplicitUnitConversionWarning)

    alpha, Skw = thomson.spectral_density(wavelengths, probe_wavelength,
                                          ne, Te, Ti, probe_vec=probe_vec,
                                          scatter_vec=scatter_vec)

fig, ax = plt.subplots()
ax.plot(wavelengths, Skw, lw=2)
ax.set_xlim(probe_wavelength.value-10, probe_wavelength.value+10)
ax.set_ylim(0, 1e-13)
ax.set_xlabel('$\lambda$ (nm)')
ax.set_ylabel('S(k,w)')
ax.set_title('Thomson Scattering Spectral Density')

```

### Example Cases in Different Scattering Regimes

We will now consider several example cases in different scattering regimes. In order to facilitate this, we'll set up each example as a dictionary of plasma parameters:

A single-species, stationary hydrogen plasma with a density and temperature that results in a scattering spectrum dominated by scattering off of single electrons.

```python
non_collective = {
    'name': 'Non-Collective Regime',
    'ne': 5e15*u.cm**-3,
    'Te': 40*u.eV,
    'Ti': np.array([10])*u.eV,
    'fract': np.array([1]),
    'ion_species': ['H+'],
    'fluid_vel': np.array([0, 0, 0])*u.km/u.s,
    'ion_vel': np.array([[0, 0, 0]])*u.km/u.s,
}
```

A single-species, stationary hydrogen plasma with a density and temperature that result in weakly collective scattering (scattering paramter $\alpha$ approaching 1)

```python
weakly_collective = {
    'name': 'Weakly Collective Regime',
    'ne': 2e17*u.cm**-3,
    'Te': 20*u.eV,
    'Ti': np.array([10])*u.eV,
    'fract': np.array([1]),
    'ion_species': ['H+'],
    'fluid_vel': np.array([0, 0, 0])*u.km/u.s,
    'ion_vel': np.array([[0, 0, 0]])*u.km/u.s,
}
```

A single-species, stationary hydrogen plasma with a density and temperature that result in a spectrum dominated by multi-particle scattering, including scattering off of ions.

```python
collective = {
    'name': 'Collective Regime',
    'ne': 5e17*u.cm**-3,
    'Te': 10*u.eV,
    'Ti': np.array([4])*u.eV,
    'fract': np.array([1]),
    'ion_species': ['H+'],
    'fluid_vel': np.array([0, 0, 0])*u.km/u.s,
    'ion_vel': np.array([[0, 0, 0]])*u.km/u.s,
}
```

A case identical to the collective example above, except that now the electron fluid has a substantial drift velocity parallel to the probe laser and the ions have a drift (relative to the electrons) at an angle.

```python
drifts = {
    'name': 'Drift Velocities',
    'ne': 5e17*u.cm**-3,
    'Te': 10*u.eV,
    'Ti': np.array([10])*u.eV,
    'fract': np.array([1]),
    'ion_species': ['H+'],
    'fluid_vel': np.array([700, 0, 0])*u.km/u.s,
    'ion_vel': np.array([[-600, -100, 0]])*u.km/u.s,
}
```

A case identical to the collective example, except that now the plasma consists 25% He+1 and 75% C+5

```python
two_species = {
    'name': 'Two Ion Species',
    'ne': 5e17*u.cm**-3,
    'Te': 10*u.eV,
    'Ti': np.array([10, 50])*u.eV,
    'fract': np.array([.25, .75]),
    'ion_species': ['He-4 1+', 'C-12 5+'],
    'fluid_vel': np.array([0, 0, 0])*u.km/u.s,
    'ion_vel': np.array([[0, 0, 0],[0, 0, 0]])*u.km/u.s,
}
```

```python
examples = [non_collective, weakly_collective, collective, drifts, two_species]
```

For each example, plot the the spectral distribution function over a large range to show the broad electron scattering feature (top row) and a narrow range around the probe wavelength to show the ion scattering feature (bottom row)

```python
fig, ax = plt.subplots(ncols=len(examples), nrows=2, figsize=[25,10])
fig.subplots_adjust( wspace=0.4, hspace=0.4)

lbls = 'abcdefg'

for i, x in enumerate(examples):

    with warnings.catch_warnings():
        warnings.simplefilter("ignore", ImplicitUnitConversionWarning)

        alpha, Skw = thomson.spectral_density(wavelengths, probe_wavelength,
                         x['ne'], x['Te'], x['Ti'], fract=x['fract'],
                         ion_species=x['ion_species'], fluid_vel=x['fluid_vel'],
                         probe_vec=probe_vec, scatter_vec=scatter_vec)

    ax[0][i].axvline(x=probe_wavelength.value, color='red')  # Mark the probe wavelength
    ax[0][i].plot(wavelengths, Skw)
    ax[0][i].set_xlim(probe_wavelength.value-15, probe_wavelength.value+15)
    ax[0][i].set_ylim(0, 1e-13)
    ax[0][i].set_xlabel('$\lambda$ (nm)')

    ax[0][i].set_title( lbls[i] + ') ' + x['name'] + '\n$\\alpha$={:.4f}'.format(alpha))

    ax[1][i].axvline(x=probe_wavelength.value, color='red')  # Mark the probe wavelength
    ax[1][i].plot(wavelengths, Skw)
    ax[1][i].set_xlim(probe_wavelength.value-1, probe_wavelength.value+1)
    ax[1][i].set_ylim(0, 1.1*np.max(Skw.value))
    ax[1][i].set_xlabel('$\lambda$ (nm)')

```

Plots of the spectral density function (Skw) which determines the amount of light scattered into different wavelengths.

a. In the non-collective regime only the electron feature is visible.

b. In the weakly collective regime (alpha approaches 1) an ion feature starts to appear and the electron feature is distorted

c. In the collective regime both features split into two peaks, corresponding to scattering off of forward and backwards propagating plasma oscillations.

d. The introduction of drift velocities introduces several Doppler shifts in the calculations, resulting in a shifted spectrum.

e. Changing the ion species present changes the resulting ion feature.


# Initialisation

```
!pip install nengolib
```

    Requirement already satisfied: nengolib in /usr/local/lib/python3.7/dist-packages (0.5.2)
    Requirement already satisfied: numpy>=1.13 in /usr/local/lib/python3.7/dist-packages (from nengolib) (1.19.5)
    Requirement already satisfied: nengo<3.0,>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from nengolib) (2.8.0)
    Requirement already satisfied: scipy>=0.19.0 in /usr/local/lib/python3.7/dist-packages (from nengolib) (1.4.1)

# LMU

```
import torch
import torch.nn as nn
from sympy.matrices import Matrix, eye, zeros, ones, diag, GramSchmidt
import numpy as np
from functools import partial
import torch.nn.functional as F
import math

from nengolib.signal import Identity, cont2discrete
from nengolib.synapses import LegendreDelay
from functools import partial

def lecun_uniform(tensor):
    fan_in = nn.init._calculate_correct_fan(tensor, 'fan_in')
    nn.init.uniform_(tensor, -math.sqrt(3 / fan_in), math.sqrt(3 / fan_in))

class LegendreMemoryUnitCell(nn.Module):
  def __init__(self, input_dim, units , order, theta,
                 input_encoders_initializer=lecun_uniform,
                 hidden_encoders_initializer=lecun_uniform,
                 memory_encoders_initializer=partial(torch.nn.init.constant_, val=0),
                 input_kernel_initializer=torch.nn.init.xavier_normal_,
                 hidden_kernel_initializer=torch.nn.init.xavier_normal_,
                 memory_kernel_initializer=torch.nn.init.xavier_normal_):
    super(LegendreMemoryUnitCell, self).__init__()

    self.order = order
    self.theta = theta
    self.units = units

    realizer = Identity()
    self._realizer_result = realizer(LegendreDelay(theta=theta, order=self.order))

    self._ss = cont2discrete(self._realizer_result.realization, dt=1., method='zoh')

    self._A = self._ss.A - np.eye(order)
    self._B = self._ss.B
    self._C = self._ss.C

    self.AT = nn.Parameter(torch.Tensor(self._A), requires_grad=False)
    self.BT = nn.Parameter(torch.Tensor(self._B), requires_grad=False)

    self.encoder_input = nn.Parameter(torch.Tensor(1,input_dim), requires_grad=True)
    self.encoder_hidden = nn.Parameter(torch.Tensor(1,self.units), requires_grad=True)
    self.encoder_memory = nn.Parameter(torch.Tensor(1,self.order ), requires_grad=True)
    self.kernel_input = nn.Parameter(torch.Tensor(self.units, input_dim), requires_grad=True)
    self.kernel_hidden = nn.Parameter(torch.Tensor(self.units, self.units), requires_grad=True)
    self.kernel_memory = nn.Parameter(torch.Tensor(self.units, self.order), requires_grad=True)

    input_encoders_initializer(self.encoder_input)
    hidden_encoders_initializer(self.encoder_hidden)
    memory_encoders_initializer(self.encoder_memory)
    input_kernel_initializer(self.kernel_input)
    hidden_kernel_initializer(self.kernel_hidden)
    memory_kernel_initializer(self.kernel_memory)

  def EulerOdeSolver(self):
    A_hat = (self.step_delta_t/self.theta)*self.AT + torch.eye(self.order,self.d_order_ode)
    B_hat = (self.step_delta_t/self.theta)*self.BT

    return A_hat, B_hat

  def forward(self, xt, states):
    ht, mt = states

    ut = F.linear(xt, self.encoder_input)\
     \
     +\
      F.linear(ht, self.encoder_hidden) \
      + \
     F.linear(mt, self.encoder_memory)

    mt = mt + F.linear(mt, self.AT) + F.linear(ut, self.BT)

    ht = nn.Tanh()(F.linear(xt, self.kernel_input) + F.linear(ht, self.kernel_hidden) + F.linear(mt, self.kernel_memory))

    return ht, (ht, mt)

class LegendreMemoryUnit(nn.Module):
  def __init__(self, input_dim, units , order, theta):
    super(LegendreMemoryUnit, self).__init__()

    self.units = units
    self.order = order

    self.lmucell = LegendreMemoryUnitCell(input_dim, units , order, theta)

  def forward(self, xt):
    outputs = []

    h0 = torch.zeros(xt.size(0),self.units).cuda()
    m0 = torch.zeros(xt.size(0),self.order).cuda()
    states = (h0,m0)
    for i in range(xt.size(1)):
      out, states = self.lmucell(xt[:,i,:], states)
      outputs += [out]
    return torch.stack(outputs).permute(1,0,2), states
    # return torch.stack(outputs).permute(1,0,2), states

```

# Test LMU output

```
x = torch.rand(64,5000,1).to(torch.device("cuda:0"))
h0 = torch.rand(64,49).to(torch.device("cuda:0"))
m0 = torch.rand(64,4).to(torch.device("cuda:0"))
model = LegendreMemoryUnit(1,49,4,4).to(torch.device("cuda:0"))
res, _ = model(x)
print(res.shape)
```

    torch.Size([64, 5000, 49])

# Models LSTM - LMU

```
class LMUModel(nn.Module):
  def __init__(self):
    super(LMUModel, self).__init__()
    self.LMU = LegendreMemoryUnit(1,49,4,4)
    self.dense = nn.Linear(49,1)

  def forward(self,x):
    x, _ = self.LMU(x)
    x = self.dense(x)

    return x

model_lstm = LMUModel()

print("Nombre de paramètres : ", sum(p.numel() for p in model_lstm.parameters() if p.requires_grad))
print()

```

    Nombre de paramètres :  2750

```
class LSTMModel(nn.Module):
  def __init__(self):
    super(LSTMModel, self).__init__()
    self.LSTM = nn.LSTM(1,25,1,batch_first=True)
    self.dense = nn.Linear(25,1)

  def forward(self,x):
    x, _ = self.LSTM(x)
    x = self.dense(x)

    return x

model_lstm = LSTMModel()

print("Nombre de paramètres : ", sum(p.numel() for p in model_lstm.parameters() if p.requires_grad))
print()

```

    Nombre de paramètres :  2826

# MackeyGlass Datas

Took from official LMU (tensorflow) repo

```
import collections
import matplotlib.pyplot as plt

import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns

from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM, RNN
from keras.initializers import RandomUniform
from keras.models import Sequential
from sklearn.metrics import mean_squared_error
from torchsummary import summary

def mackey_glass(sample_len=1000, tau=17, delta_t=10, seed=None, n_samples=1):
    # Adapted from https://github.com/mila-iqia/summerschool2015/blob/master/rnn_tutorial/synthetic.py
    '''
    mackey_glass(sample_len=1000, tau=17, seed = None, n_samples = 1) -> input
    Generate the Mackey Glass time-series. Parameters are:
        - sample_len: length of the time-series in timesteps. Default is 1000.
        - tau: delay of the MG - system. Commonly used values are tau=17 (mild
          chaos) and tau=30 (moderate chaos). Default is 17.
        - seed: to seed the random generator, can be used to generate the same
          timeseries at each invocation.
        - n_samples : number of samples to generate
    '''
    history_len = tau * delta_t
    # Initial conditions for the history of the system
    timeseries = 1.2

    if seed is not None:
        np.random.seed(seed)

    samples = []

    for _ in range(n_samples):
        history = collections.deque(1.2 * np.ones(history_len) + 0.2 * \
                                    (np.random.rand(history_len) - 0.5))
        # Preallocate the array for the time-series
        inp = np.zeros((sample_len,1))

        for timestep in range(sample_len):
            for _ in range(delta_t):
                xtau = history.popleft()
                history.append(timeseries)
                timeseries = history[-1] + (0.2 * xtau / (1.0 + xtau ** 10) - \
                             0.1 * history[-1]) / delta_t
            inp[timestep] = timeseries

        # Squash timeseries through tanh
        inp = np.tanh(inp - 1)
        samples.append(inp)
    return samples

def generate_data(n_batches, length, split=0.5, seed=0,
                  predict_length=15, tau=17, washout=100, delta_t=1,
                  center=True):
    X = np.asarray(mackey_glass(
        sample_len=length+predict_length+washout, tau=tau,
        seed=seed, n_samples=n_batches))
    X = X[:, washout:, :]
    cutoff = int(split*n_batches)
    if center:
        X -= np.mean(X)  # global mean over all batches, approx -0.066
    Y = X[:, :-predict_length, :]
    X = X[:, predict_length:, :]
    assert X.shape == Y.shape
    return ((X[:cutoff], Y[:cutoff]),
            (X[cutoff:], Y[cutoff:]))

(train_X, train_Y), (test_X, test_Y) = generate_data(128, 5000)
print(train_X.shape, test_X.shape)

def cool_plot(X, Y, title=""):
    plt.figure(figsize=(14, 8))
    plt.title(title)
    plt.scatter(X[:, 0], Y[:, 0] - X[:, 0], s=8, alpha=0.7,
                c=np.arange(X.shape[0]), cmap=sns.cubehelix_palette(as_cmap=True))
    plt.plot(X[:, 0], Y[:, 0] - X[:, 0], c='black', alpha=0.2)
    plt.xlabel("$x(t)$")
    plt.ylabel("$y(t) - x(t)$")
    sns.despine(offset=15)

    plt.show()

# cool_plot(train_X[0], train_Y[0])

plt.plot(train_X[0][0:100])
plt.plot(train_Y[0][0:100])
plt.show()

```

# Utils

```
def train(model, epochs, dataset, dataset_valid = None):
  for e in range(epochs):
      model.train()
      running_loss = 0
      with tqdm(total=len(dataset)) as bar:
        for i, (X, y) in enumerate(dataset):
            optimizer.zero_grad()

            output = model(X.cuda())

            loss_ll = criterion(output, y.cuda())

            loss_ll.backward()

            optimizer.step()

            running_loss += loss_ll.item()

            bar.update(1)
            bar.set_description("Epoch {} - Training loss: {}".format(e, running_loss/len(dataset)))

      model.eval()
      running_loss = 0
      outs = []
      with tqdm(total=len(dataset_valid)) as bar:
        for i, (X, y) in enumerate(dataset_valid):
            optimizer.zero_grad()

            output = model(X.cuda())

            loss_ll = criterion(output, y.cuda())

            running_loss += loss_ll.item()

            outs.append(output)
            bar.update(1)
            bar.set_description("Epoch {} - Training loss: {}".format(e, running_loss/len(dataset_valid)))

```

# Test LMU vs LSTM on MackeyGlass

```
from tqdm import tqdm
import torch.utils.data as data

model = LMUModel().cuda()
print("\n\nNombre de paramètres : ", sum(p.numel() for p in model.parameters() if p.requires_grad))
print("\n\n")
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
model.train()

dataset = torch.utils.data.TensorDataset(torch.Tensor(train_X).cuda(), torch.Tensor(train_Y).cuda())
dataset = data.DataLoader(dataset, batch_size=16, shuffle=True)

dataset_valid = torch.utils.data.TensorDataset(torch.Tensor(test_X).cuda(), torch.Tensor(test_Y).cuda())
dataset_valid = data.DataLoader(dataset_valid, batch_size=16, shuffle=False)

train(model, 100, dataset, dataset_valid)
```

      0%|          | 0/4 [00:00<?, ?it/s]

    Nombre de paramètres :  2750

    Epoch 0 - Training loss: 0.05387168284505606: 100%|██████████| 4/4 [00:12<00:00,  3.21s/it]
    Epoch 0 - Training loss: 0.03753741178661585: 100%|██████████| 4/4 [00:05<00:00,  1.42s/it]
    Epoch 1 - Training loss: 0.026890650391578674: 100%|██████████| 4/4 [00:12<00:00,  3.13s/it]
    Epoch 1 - Training loss: 0.01199188269674778: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 2 - Training loss: 0.012843496166169643: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 2 - Training loss: 0.017223669681698084: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 3 - Training loss: 0.014944496098905802: 100%|██████████| 4/4 [00:12<00:00,  3.18s/it]
    Epoch 3 - Training loss: 0.009903115686029196: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 4 - Training loss: 0.009571623057126999: 100%|██████████| 4/4 [00:12<00:00,  3.13s/it]
    Epoch 4 - Training loss: 0.010275870328769088: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 5 - Training loss: 0.010369897820055485: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 5 - Training loss: 0.009227692848071456: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 6 - Training loss: 0.008216657908633351: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 6 - Training loss: 0.006964302621781826: 100%|██████████| 4/4 [00:05<00:00,  1.48s/it]
    Epoch 7 - Training loss: 0.007081472547724843: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 7 - Training loss: 0.006996552110649645: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 8 - Training loss: 0.0065329637145623565: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 8 - Training loss: 0.005560776800848544: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 9 - Training loss: 0.0055028460919857025: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 9 - Training loss: 0.005370369413867593: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 10 - Training loss: 0.0050738140707835555: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 10 - Training loss: 0.004352257237769663: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 11 - Training loss: 0.00426543690264225: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 11 - Training loss: 0.003930750186555088: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 12 - Training loss: 0.003619474475272: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 12 - Training loss: 0.0031912961858324707: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 13 - Training loss: 0.0029909488512203097: 100%|██████████| 4/4 [00:12<00:00,  3.16s/it]
    Epoch 13 - Training loss: 0.002459310577251017: 100%|██████████| 4/4 [00:05<00:00,  1.43s/it]
    Epoch 14 - Training loss: 0.00231863628141582: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 14 - Training loss: 0.0018753806652966887: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 15 - Training loss: 0.0017779157788027078: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 15 - Training loss: 0.001495080505264923: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 16 - Training loss: 0.0014176419063005596: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 16 - Training loss: 0.0011975888046436012: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 17 - Training loss: 0.0010795327252708375: 100%|██████████| 4/4 [00:12<00:00,  3.17s/it]
    Epoch 17 - Training loss: 0.0008423931576544419: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 18 - Training loss: 0.0008409871079493314: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 18 - Training loss: 0.0007657954411115497: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 19 - Training loss: 0.0007972515450092033: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 19 - Training loss: 0.0007360217714449391: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 20 - Training loss: 0.0007329464133363217: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 20 - Training loss: 0.0006381762068485841: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 21 - Training loss: 0.0006486060592578724: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 21 - Training loss: 0.0005785990651929751: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 22 - Training loss: 0.0005930171610089019: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 22 - Training loss: 0.0005226586654316634: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 23 - Training loss: 0.0005493425705935806: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 23 - Training loss: 0.0004992043832316995: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 24 - Training loss: 0.0005248152956482954: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 24 - Training loss: 0.00047154480853350833: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 25 - Training loss: 0.0004959086872986518: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 25 - Training loss: 0.00044671470095636323: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 26 - Training loss: 0.00047382627963088453: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 26 - Training loss: 0.0004302551024011336: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 27 - Training loss: 0.00045760826469631866: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 27 - Training loss: 0.00041388445970369503: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 28 - Training loss: 0.00044146575237391517: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 28 - Training loss: 0.0004003809008281678: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 29 - Training loss: 0.00042694048897828907: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 29 - Training loss: 0.00038599530671490356: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 30 - Training loss: 0.0004141425743000582: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 30 - Training loss: 0.0003743963097804226: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 31 - Training loss: 0.0004031861390103586: 100%|██████████| 4/4 [00:12<00:00,  3.14s/it]
    Epoch 31 - Training loss: 0.0003638174821389839: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 32 - Training loss: 0.00039235021540662274: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 32 - Training loss: 0.00035547458537621424: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 33 - Training loss: 0.00038299430161714554: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 33 - Training loss: 0.0003450366493780166: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 34 - Training loss: 0.0003738810191862285: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 34 - Training loss: 0.0003371613420313224: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 35 - Training loss: 0.0003660653528640978: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 35 - Training loss: 0.0003299886957393028: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 36 - Training loss: 0.0003588964173104614: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 36 - Training loss: 0.0003234985488234088: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 37 - Training loss: 0.00035264520556665957: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 37 - Training loss: 0.0003179324703523889: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 38 - Training loss: 0.0003470203882898204: 100%|██████████| 4/4 [00:12<00:00,  3.13s/it]
    Epoch 38 - Training loss: 0.00031237094663083553: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 39 - Training loss: 0.0003418081032577902: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 39 - Training loss: 0.00030762660753680393: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 40 - Training loss: 0.00033712707954691723: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 40 - Training loss: 0.00030333370523294434: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 41 - Training loss: 0.00033267146500293165: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 41 - Training loss: 0.00029972322954563424: 100%|██████████| 4/4 [00:05<00:00,  1.36s/it]
    Epoch 42 - Training loss: 0.0003289942760602571: 100%|██████████| 4/4 [00:12<00:00,  3.05s/it]
    Epoch 42 - Training loss: 0.0002961409481940791: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 43 - Training loss: 0.0003253015674999915: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 43 - Training loss: 0.0002927118621300906: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 44 - Training loss: 0.00032213145459536463: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 44 - Training loss: 0.00029005866235820577: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 45 - Training loss: 0.0003192283838870935: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 45 - Training loss: 0.0002871526958188042: 100%|██████████| 4/4 [00:05<00:00,  1.36s/it]
    Epoch 46 - Training loss: 0.00031644020782550797: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 46 - Training loss: 0.00028461402689572424: 100%|██████████| 4/4 [00:05<00:00,  1.36s/it]
    Epoch 47 - Training loss: 0.0003139621185255237: 100%|██████████| 4/4 [00:12<00:00,  3.05s/it]
    Epoch 47 - Training loss: 0.0002824096445692703: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 48 - Training loss: 0.0003117293745162897: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 48 - Training loss: 0.0002802089002216235: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 49 - Training loss: 0.00030963614699430764: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 49 - Training loss: 0.00027882401627721265: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 50 - Training loss: 0.00030783392139710486: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 50 - Training loss: 0.0002762377553153783: 100%|██████████| 4/4 [00:05<00:00,  1.36s/it]
    Epoch 51 - Training loss: 0.00030665023950859904: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 51 - Training loss: 0.00027731294539989904: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 52 - Training loss: 0.000305190038488945: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 52 - Training loss: 0.000272716220933944: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 53 - Training loss: 0.00030259248160291463: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 53 - Training loss: 0.000274106947472319: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 54 - Training loss: 0.00030306357075460255: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 54 - Training loss: 0.00027225780650042: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 55 - Training loss: 0.0003011923035955988: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 55 - Training loss: 0.0002712893401621841: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 56 - Training loss: 0.00030196602892829105: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 56 - Training loss: 0.00027693801530404016: 100%|██████████| 4/4 [00:05<00:00,  1.36s/it]
    Epoch 57 - Training loss: 0.00031192531605483964: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 57 - Training loss: 0.0002865242422558367: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 58 - Training loss: 0.0003053729305975139: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 58 - Training loss: 0.00026451364101376384: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 59 - Training loss: 0.0003009338106494397: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 59 - Training loss: 0.0002758288901532069: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 60 - Training loss: 0.00029704142798436806: 100%|██████████| 4/4 [00:12<00:00,  3.05s/it]
    Epoch 60 - Training loss: 0.00026829808484762907: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 61 - Training loss: 0.00030043520382605493: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 61 - Training loss: 0.0002626121568027884: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 62 - Training loss: 0.00029239941795822233: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 62 - Training loss: 0.00026878572680288926: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 63 - Training loss: 0.0002926946908701211: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 63 - Training loss: 0.00025890492543112487: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 64 - Training loss: 0.0002897622325690463: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 64 - Training loss: 0.0002600965235615149: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 65 - Training loss: 0.00028692022169707343: 100%|██████████| 4/4 [00:12<00:00,  3.04s/it]
    Epoch 65 - Training loss: 0.0002560044886195101: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 66 - Training loss: 0.0002849468364729546: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 66 - Training loss: 0.00025571134756319225: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 67 - Training loss: 0.00028445664793252945: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 67 - Training loss: 0.0002548941774875857: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 68 - Training loss: 0.0002832992759067565: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 68 - Training loss: 0.00025346604525111616: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 69 - Training loss: 0.00028191067394800484: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 69 - Training loss: 0.0002527419856050983: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 70 - Training loss: 0.0002844357441063039: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 70 - Training loss: 0.0002656178330653347: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 71 - Training loss: 0.00030451132261077873: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 71 - Training loss: 0.0002796902772388421: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 72 - Training loss: 0.00029376726160990074: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 72 - Training loss: 0.0002495473418093752: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 73 - Training loss: 0.0002833925136656035: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 73 - Training loss: 0.0002585462207207456: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 74 - Training loss: 0.0002812048769555986: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 74 - Training loss: 0.0002492799576430116: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 75 - Training loss: 0.00028055982693331316: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 75 - Training loss: 0.00024996833235491067: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 76 - Training loss: 0.0002761335745162796: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 76 - Training loss: 0.00024774327903287485: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 77 - Training loss: 0.0002764774108072743: 100%|██████████| 4/4 [00:12<00:00,  3.05s/it]
    Epoch 77 - Training loss: 0.0002464139033691026: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 78 - Training loss: 0.00027369093731977046: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 78 - Training loss: 0.0002448815430398099: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 79 - Training loss: 0.00027374048659112304: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 79 - Training loss: 0.0002462286538502667: 100%|██████████| 4/4 [00:05<00:00,  1.42s/it]
    Epoch 80 - Training loss: 0.00027402315026847646: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 80 - Training loss: 0.00024391540500801057: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 81 - Training loss: 0.000271519718808122: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 81 - Training loss: 0.00024219185070251115: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 82 - Training loss: 0.000270053085841937: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 82 - Training loss: 0.0002418425829091575: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 83 - Training loss: 0.0002709827895159833: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 83 - Training loss: 0.00024584443963249214: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 84 - Training loss: 0.0002817051463352982: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 84 - Training loss: 0.00027960335864918306: 100%|██████████| 4/4 [00:05<00:00,  1.38s/it]
    Epoch 85 - Training loss: 0.00032525850110687315: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 85 - Training loss: 0.0002743011646089144: 100%|██████████| 4/4 [00:05<00:00,  1.41s/it]
    Epoch 86 - Training loss: 0.00028338372794678435: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 86 - Training loss: 0.0002717555107665248: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 87 - Training loss: 0.00028214404301252216: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 87 - Training loss: 0.0002537468244554475: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 88 - Training loss: 0.00027709093410521746: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 88 - Training loss: 0.00024339293304365128: 100%|██████████| 4/4 [00:05<00:00,  1.44s/it]
    Epoch 89 - Training loss: 0.0002728996623773128: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 89 - Training loss: 0.00023865257026045583: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 90 - Training loss: 0.00026949252787744626: 100%|██████████| 4/4 [00:12<00:00,  3.15s/it]
    Epoch 90 - Training loss: 0.00023664915352128446: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 91 - Training loss: 0.00026714117848314345: 100%|██████████| 4/4 [00:12<00:00,  3.10s/it]
    Epoch 91 - Training loss: 0.00023604380476172082: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 92 - Training loss: 0.00026580518897389993: 100%|██████████| 4/4 [00:12<00:00,  3.09s/it]
    Epoch 92 - Training loss: 0.00023694843548582867: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 93 - Training loss: 0.00026481267559574917: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 93 - Training loss: 0.0002384236577199772: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 94 - Training loss: 0.00026400198839837685: 100%|██████████| 4/4 [00:12<00:00,  3.08s/it]
    Epoch 94 - Training loss: 0.00023845406030886807: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 95 - Training loss: 0.00026328339299652725: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 95 - Training loss: 0.00023523561685578898: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 96 - Training loss: 0.00026321163386455737: 100%|██████████| 4/4 [00:12<00:00,  3.12s/it]
    Epoch 96 - Training loss: 0.00023322572451434098: 100%|██████████| 4/4 [00:05<00:00,  1.40s/it]
    Epoch 97 - Training loss: 0.00026061605603899807: 100%|██████████| 4/4 [00:12<00:00,  3.11s/it]
    Epoch 97 - Training loss: 0.00023372500800178386: 100%|██████████| 4/4 [00:05<00:00,  1.39s/it]
    Epoch 98 - Training loss: 0.0002603484426799696: 100%|██████████| 4/4 [00:12<00:00,  3.06s/it]
    Epoch 98 - Training loss: 0.00023183693701867014: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]
    Epoch 99 - Training loss: 0.0002601470478111878: 100%|██████████| 4/4 [00:12<00:00,  3.07s/it]
    Epoch 99 - Training loss: 0.00023260670059244148: 100%|██████████| 4/4 [00:05<00:00,  1.37s/it]

```
from tqdm import tqdm
import torch.utils.data as data

model = LSTMModel().cuda()
print("\n\nNombre de paramètres : ", sum(p.numel() for p in model.parameters() if p.requires_grad))
print("\n\n")
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
model.train()

dataset = torch.utils.data.TensorDataset(torch.Tensor(train_X).cuda(), torch.Tensor(train_Y).cuda())
dataset = data.DataLoader(dataset, batch_size=16, shuffle=True)

dataset_valid = torch.utils.data.TensorDataset(torch.Tensor(test_X).cuda(), torch.Tensor(test_Y).cuda())
dataset_valid = data.DataLoader(dataset_valid, batch_size=16, shuffle=False)

train(model, 100, dataset, dataset_valid)
```

    Epoch 0 - Training loss: 0.011813671328127384:  25%|██▌       | 1/4 [00:00<00:00,  7.53it/s]

    Nombre de paramètres :  2826

    Epoch 0 - Training loss: 0.046802151016891: 100%|██████████| 4/4 [00:00<00:00,  7.53it/s]
    Epoch 0 - Training loss: 0.046247866936028004: 100%|██████████| 4/4 [00:00<00:00, 16.11it/s]
    Epoch 1 - Training loss: 0.04624266363680363: 100%|██████████| 4/4 [00:00<00:00,  7.49it/s]
    Epoch 1 - Training loss: 0.04614768456667662: 100%|██████████| 4/4 [00:00<00:00, 16.33it/s]
    Epoch 2 - Training loss: 0.046114287339150906: 100%|██████████| 4/4 [00:00<00:00,  7.65it/s]
    Epoch 2 - Training loss: 0.04586911480873823: 100%|██████████| 4/4 [00:00<00:00, 15.75it/s]
    Epoch 3 - Training loss: 0.04578931536525488: 100%|██████████| 4/4 [00:00<00:00,  7.68it/s]
    Epoch 3 - Training loss: 0.04555390030145645: 100%|██████████| 4/4 [00:00<00:00, 16.11it/s]
    Epoch 4 - Training loss: 0.04551477171480656: 100%|██████████| 4/4 [00:00<00:00,  7.61it/s]
    Epoch 4 - Training loss: 0.045330765657126904: 100%|██████████| 4/4 [00:00<00:00, 16.33it/s]
    Epoch 5 - Training loss: 0.04527045786380768: 100%|██████████| 4/4 [00:00<00:00,  7.69it/s]
    Epoch 5 - Training loss: 0.04503007512539625: 100%|██████████| 4/4 [00:00<00:00, 16.14it/s]
    Epoch 6 - Training loss: 0.04493964184075594: 100%|██████████| 4/4 [00:00<00:00,  7.39it/s]
    Epoch 6 - Training loss: 0.04466285090893507: 100%|██████████| 4/4 [00:00<00:00, 16.40it/s]
    Epoch 7 - Training loss: 0.04456302337348461: 100%|██████████| 4/4 [00:00<00:00,  7.62it/s]
    Epoch 7 - Training loss: 0.04424550198018551: 100%|██████████| 4/4 [00:00<00:00, 16.13it/s]
    Epoch 8 - Training loss: 0.04409331176429987: 100%|██████████| 4/4 [00:00<00:00,  7.49it/s]
    Epoch 8 - Training loss: 0.04366330895572901: 100%|██████████| 4/4 [00:00<00:00, 16.13it/s]
    Epoch 9 - Training loss: 0.04342217277735472: 100%|██████████| 4/4 [00:00<00:00,  7.58it/s]
    Epoch 9 - Training loss: 0.042824034579098225: 100%|██████████| 4/4 [00:00<00:00, 16.50it/s]
    Epoch 10 - Training loss: 0.04242550954222679: 100%|██████████| 4/4 [00:00<00:00,  7.49it/s]
    Epoch 10 - Training loss: 0.04149808920919895: 100%|██████████| 4/4 [00:00<00:00, 16.58it/s]
    Epoch 11 - Training loss: 0.04079277068376541: 100%|██████████| 4/4 [00:00<00:00,  7.64it/s]
    Epoch 11 - Training loss: 0.03923314530402422: 100%|██████████| 4/4 [00:00<00:00, 16.21it/s]
    Epoch 12 - Training loss: 0.03793259151279926: 100%|██████████| 4/4 [00:00<00:00,  7.68it/s]
    Epoch 12 - Training loss: 0.03510310687124729: 100%|██████████| 4/4 [00:00<00:00, 16.31it/s]
    Epoch 13 - Training loss: 0.03257376095280051: 100%|██████████| 4/4 [00:00<00:00,  7.72it/s]
    Epoch 13 - Training loss: 0.027179218363016844: 100%|██████████| 4/4 [00:00<00:00, 15.47it/s]
    Epoch 14 - Training loss: 0.022414845880120993: 100%|██████████| 4/4 [00:00<00:00,  7.41it/s]
    Epoch 14 - Training loss: 0.013185449643060565: 100%|██████████| 4/4 [00:00<00:00, 15.53it/s]
    Epoch 15 - Training loss: 0.00822982401587069: 100%|██████████| 4/4 [00:00<00:00,  7.38it/s]
    Epoch 15 - Training loss: 0.004749904735945165: 100%|██████████| 4/4 [00:00<00:00, 16.04it/s]
    Epoch 16 - Training loss: 0.006897906074300408: 100%|██████████| 4/4 [00:00<00:00,  7.46it/s]
    Epoch 16 - Training loss: 0.007393145002424717: 100%|██████████| 4/4 [00:00<00:00, 16.10it/s]
    Epoch 17 - Training loss: 0.005986468051560223: 100%|██████████| 4/4 [00:00<00:00,  7.45it/s]
    Epoch 17 - Training loss: 0.004386574379168451: 100%|██████████| 4/4 [00:00<00:00, 16.80it/s]
    Epoch 18 - Training loss: 0.004102321516256779: 100%|██████████| 4/4 [00:00<00:00,  7.39it/s]
    Epoch 18 - Training loss: 0.003871064109262079: 100%|██████████| 4/4 [00:00<00:00, 16.54it/s]
    Epoch 19 - Training loss: 0.0038970985333435237: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 19 - Training loss: 0.0038985730498097837: 100%|██████████| 4/4 [00:00<00:00, 16.87it/s]
    Epoch 20 - Training loss: 0.0038104478735476732: 100%|██████████| 4/4 [00:00<00:00,  7.57it/s]
    Epoch 20 - Training loss: 0.003526116954162717: 100%|██████████| 4/4 [00:00<00:00, 16.05it/s]
    Epoch 21 - Training loss: 0.003210261231288314: 100%|██████████| 4/4 [00:00<00:00,  7.43it/s]
    Epoch 21 - Training loss: 0.0027287285774946213: 100%|██████████| 4/4 [00:00<00:00, 16.60it/s]
    Epoch 22 - Training loss: 0.0026558273239061236: 100%|██████████| 4/4 [00:00<00:00,  7.45it/s]
    Epoch 22 - Training loss: 0.0027365919086150825: 100%|██████████| 4/4 [00:00<00:00, 16.65it/s]
    Epoch 23 - Training loss: 0.0027639803010970354: 100%|██████████| 4/4 [00:00<00:00,  7.30it/s]
    Epoch 23 - Training loss: 0.0026207848568446934: 100%|██████████| 4/4 [00:00<00:00, 16.34it/s]
    Epoch 24 - Training loss: 0.0024585392675362527: 100%|██████████| 4/4 [00:00<00:00,  7.35it/s]
    Epoch 24 - Training loss: 0.0022943864460103214: 100%|██████████| 4/4 [00:00<00:00, 14.81it/s]
    Epoch 25 - Training loss: 0.002309341449290514: 100%|██████████| 4/4 [00:00<00:00,  7.40it/s]
    Epoch 25 - Training loss: 0.002335866854991764: 100%|██████████| 4/4 [00:00<00:00, 16.31it/s]
    Epoch 26 - Training loss: 0.002297056373208761: 100%|██████████| 4/4 [00:00<00:00,  7.24it/s]
    Epoch 26 - Training loss: 0.0022075606393627822: 100%|██████████| 4/4 [00:00<00:00, 16.83it/s]
    Epoch 27 - Training loss: 0.0021657930337823927: 100%|██████████| 4/4 [00:00<00:00,  7.30it/s]
    Epoch 27 - Training loss: 0.0021303174435161054: 100%|██████████| 4/4 [00:00<00:00, 15.16it/s]
    Epoch 28 - Training loss: 0.0021280248765833676: 100%|██████████| 4/4 [00:00<00:00,  7.17it/s]
    Epoch 28 - Training loss: 0.0021088813664391637: 100%|██████████| 4/4 [00:00<00:00, 15.83it/s]
    Epoch 29 - Training loss: 0.0020920083625242114: 100%|██████████| 4/4 [00:00<00:00,  7.42it/s]
    Epoch 29 - Training loss: 0.002059364749584347: 100%|██████████| 4/4 [00:00<00:00, 15.90it/s]
    Epoch 30 - Training loss: 0.002052450436167419: 100%|██████████| 4/4 [00:00<00:00,  7.45it/s]
    Epoch 30 - Training loss: 0.0020339690963737667: 100%|██████████| 4/4 [00:00<00:00, 16.54it/s]
    Epoch 31 - Training loss: 0.0020319863688200712: 100%|██████████| 4/4 [00:00<00:00,  7.46it/s]
    Epoch 31 - Training loss: 0.0020165679161436856: 100%|██████████| 4/4 [00:00<00:00, 16.37it/s]
    Epoch 32 - Training loss: 0.002013405959587544: 100%|██████████| 4/4 [00:00<00:00,  7.15it/s]
    Epoch 32 - Training loss: 0.001994386489968747: 100%|██████████| 4/4 [00:00<00:00, 16.45it/s]
    Epoch 33 - Training loss: 0.0019947029068134725: 100%|██████████| 4/4 [00:00<00:00,  7.29it/s]
    Epoch 33 - Training loss: 0.0019821292371489108: 100%|██████████| 4/4 [00:00<00:00, 16.16it/s]
    Epoch 34 - Training loss: 0.00198433204786852: 100%|██████████| 4/4 [00:00<00:00,  7.42it/s]
    Epoch 34 - Training loss: 0.0019695968367159367: 100%|██████████| 4/4 [00:00<00:00, 16.29it/s]
    Epoch 35 - Training loss: 0.0019707051978912205: 100%|██████████| 4/4 [00:00<00:00,  7.47it/s]
    Epoch 35 - Training loss: 0.0019579711952246726: 100%|██████████| 4/4 [00:00<00:00, 16.74it/s]
    Epoch 36 - Training loss: 0.001960664492798969: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 36 - Training loss: 0.0019489610276650637: 100%|██████████| 4/4 [00:00<00:00, 15.92it/s]
    Epoch 37 - Training loss: 0.001950057689100504: 100%|██████████| 4/4 [00:00<00:00,  7.41it/s]
    Epoch 37 - Training loss: 0.001938168628839776: 100%|██████████| 4/4 [00:00<00:00, 16.31it/s]
    Epoch 38 - Training loss: 0.0019397947180550545: 100%|██████████| 4/4 [00:00<00:00,  7.32it/s]
    Epoch 38 - Training loss: 0.0019292588403914124: 100%|██████████| 4/4 [00:00<00:00, 15.50it/s]
    Epoch 39 - Training loss: 0.0019301071588415653: 100%|██████████| 4/4 [00:00<00:00,  7.35it/s]
    Epoch 39 - Training loss: 0.0019198548689018935: 100%|██████████| 4/4 [00:00<00:00, 16.12it/s]
    Epoch 40 - Training loss: 0.0019200504757463932: 100%|██████████| 4/4 [00:00<00:00,  7.32it/s]
    Epoch 40 - Training loss: 0.0019107795087620616: 100%|██████████| 4/4 [00:00<00:00, 15.19it/s]
    Epoch 41 - Training loss: 0.0019104121602140367: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 41 - Training loss: 0.0019017902086488903: 100%|██████████| 4/4 [00:00<00:00, 16.20it/s]
    Epoch 42 - Training loss: 0.0019007523660548031: 100%|██████████| 4/4 [00:00<00:00,  7.35it/s]
    Epoch 42 - Training loss: 0.0018928625213447958: 100%|██████████| 4/4 [00:00<00:00, 16.51it/s]
    Epoch 43 - Training loss: 0.0018912689702119678: 100%|██████████| 4/4 [00:00<00:00,  7.30it/s]
    Epoch 43 - Training loss: 0.0018840281700249761: 100%|██████████| 4/4 [00:00<00:00, 15.99it/s]
    Epoch 44 - Training loss: 0.0018818468670360744: 100%|██████████| 4/4 [00:00<00:00,  7.46it/s]
    Epoch 44 - Training loss: 0.0018752702453639358: 100%|██████████| 4/4 [00:00<00:00, 16.20it/s]
    Epoch 45 - Training loss: 0.0018725153349805623: 100%|██████████| 4/4 [00:00<00:00,  7.35it/s]
    Epoch 45 - Training loss: 0.0018666451214812696: 100%|██████████| 4/4 [00:00<00:00, 15.33it/s]
    Epoch 46 - Training loss: 0.001863187353592366: 100%|██████████| 4/4 [00:00<00:00,  7.44it/s]
    Epoch 46 - Training loss: 0.0018580170581117272: 100%|██████████| 4/4 [00:00<00:00, 16.98it/s]
    Epoch 47 - Training loss: 0.0018539235170464963: 100%|██████████| 4/4 [00:00<00:00,  7.18it/s]
    Epoch 47 - Training loss: 0.0018495105614420027: 100%|██████████| 4/4 [00:00<00:00, 16.17it/s]
    Epoch 48 - Training loss: 0.0018449842755217105: 100%|██████████| 4/4 [00:00<00:00,  7.31it/s]
    Epoch 48 - Training loss: 0.0018410282500553876: 100%|██████████| 4/4 [00:00<00:00, 16.13it/s]
    Epoch 49 - Training loss: 0.0018358205852564424: 100%|██████████| 4/4 [00:00<00:00,  7.43it/s]
    Epoch 49 - Training loss: 0.0018329707090742886: 100%|██████████| 4/4 [00:00<00:00, 16.45it/s]
    Epoch 50 - Training loss: 0.0018269540159963071: 100%|██████████| 4/4 [00:00<00:00,  7.44it/s]
    Epoch 50 - Training loss: 0.0018243280937895179: 100%|██████████| 4/4 [00:00<00:00, 16.09it/s]
    Epoch 51 - Training loss: 0.0018179765320383012: 100%|██████████| 4/4 [00:00<00:00,  7.18it/s]
    Epoch 51 - Training loss: 0.0018161163898184896: 100%|██████████| 4/4 [00:00<00:00, 16.11it/s]
    Epoch 52 - Training loss: 0.0018091454112436622: 100%|██████████| 4/4 [00:00<00:00,  7.22it/s]
    Epoch 52 - Training loss: 0.001807879947591573: 100%|██████████| 4/4 [00:00<00:00, 16.36it/s]
    Epoch 53 - Training loss: 0.0018003683653660119: 100%|██████████| 4/4 [00:00<00:00,  7.44it/s]
    Epoch 53 - Training loss: 0.0017996571550611407: 100%|██████████| 4/4 [00:00<00:00, 14.65it/s]
    Epoch 54 - Training loss: 0.0017917337827384472: 100%|██████████| 4/4 [00:00<00:00,  7.43it/s]
    Epoch 54 - Training loss: 0.001791690126992762: 100%|██████████| 4/4 [00:00<00:00, 16.35it/s]
    Epoch 55 - Training loss: 0.0017828882555477321: 100%|██████████| 4/4 [00:00<00:00,  7.42it/s]
    Epoch 55 - Training loss: 0.001783599000191316: 100%|██████████| 4/4 [00:00<00:00, 16.01it/s]
    Epoch 56 - Training loss: 0.0017742907220963389: 100%|██████████| 4/4 [00:00<00:00,  7.30it/s]
    Epoch 56 - Training loss: 0.0017754001601133496: 100%|██████████| 4/4 [00:00<00:00, 16.70it/s]
    Epoch 57 - Training loss: 0.0017657425196375698: 100%|██████████| 4/4 [00:00<00:00,  7.19it/s]
    Epoch 57 - Training loss: 0.0017673789407126606: 100%|██████████| 4/4 [00:00<00:00, 15.92it/s]
    Epoch 58 - Training loss: 0.0017570871277712286: 100%|██████████| 4/4 [00:00<00:00,  7.25it/s]
    Epoch 58 - Training loss: 0.0017595317913219333: 100%|██████████| 4/4 [00:00<00:00, 15.72it/s]
    Epoch 59 - Training loss: 0.0017485136340837926: 100%|██████████| 4/4 [00:00<00:00,  7.44it/s]
    Epoch 59 - Training loss: 0.0017514571372885257: 100%|██████████| 4/4 [00:00<00:00, 16.79it/s]
    Epoch 60 - Training loss: 0.0017400124634150416: 100%|██████████| 4/4 [00:00<00:00,  7.38it/s]
    Epoch 60 - Training loss: 0.0017435102199669927: 100%|██████████| 4/4 [00:00<00:00, 16.67it/s]
    Epoch 61 - Training loss: 0.0017314227588940412: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 61 - Training loss: 0.0017358755867462605: 100%|██████████| 4/4 [00:00<00:00, 16.56it/s]
    Epoch 62 - Training loss: 0.00172309490153566: 100%|██████████| 4/4 [00:00<00:00,  7.36it/s]
    Epoch 62 - Training loss: 0.0017276867292821407: 100%|██████████| 4/4 [00:00<00:00, 16.01it/s]
    Epoch 63 - Training loss: 0.0017147923645097762: 100%|██████████| 4/4 [00:00<00:00,  7.45it/s]
    Epoch 63 - Training loss: 0.0017201715672854334: 100%|██████████| 4/4 [00:00<00:00, 16.38it/s]
    Epoch 64 - Training loss: 0.001706323935650289: 100%|██████████| 4/4 [00:00<00:00,  7.41it/s]
    Epoch 64 - Training loss: 0.0017122822464443743: 100%|██████████| 4/4 [00:00<00:00, 16.10it/s]
    Epoch 65 - Training loss: 0.001697781786788255: 100%|██████████| 4/4 [00:00<00:00,  7.38it/s]
    Epoch 65 - Training loss: 0.0017044447595253587: 100%|██████████| 4/4 [00:00<00:00, 16.93it/s]
    Epoch 66 - Training loss: 0.001689404743956402: 100%|██████████| 4/4 [00:00<00:00,  7.36it/s]
    Epoch 66 - Training loss: 0.001696308027021587: 100%|██████████| 4/4 [00:00<00:00, 16.24it/s]
    Epoch 67 - Training loss: 0.0016809095977805555: 100%|██████████| 4/4 [00:00<00:00,  7.30it/s]
    Epoch 67 - Training loss: 0.0016883467324078083: 100%|██████████| 4/4 [00:00<00:00, 16.32it/s]
    Epoch 68 - Training loss: 0.0016724035958759487: 100%|██████████| 4/4 [00:00<00:00,  7.49it/s]
    Epoch 68 - Training loss: 0.0016807140782475471: 100%|██████████| 4/4 [00:00<00:00, 16.46it/s]
    Epoch 69 - Training loss: 0.0016640470130369067: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 69 - Training loss: 0.0016728248447179794: 100%|██████████| 4/4 [00:00<00:00, 16.42it/s]
    Epoch 70 - Training loss: 0.0016556020709685981: 100%|██████████| 4/4 [00:00<00:00,  7.35it/s]
    Epoch 70 - Training loss: 0.0016647337761241943: 100%|██████████| 4/4 [00:00<00:00, 16.40it/s]
    Epoch 71 - Training loss: 0.001647163589950651: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 71 - Training loss: 0.0016568727151025087: 100%|██████████| 4/4 [00:00<00:00, 15.57it/s]
    Epoch 72 - Training loss: 0.0016387711511924863: 100%|██████████| 4/4 [00:00<00:00,  7.35it/s]
    Epoch 72 - Training loss: 0.0016489873523823917: 100%|██████████| 4/4 [00:00<00:00, 16.77it/s]
    Epoch 73 - Training loss: 0.001630245620617643: 100%|██████████| 4/4 [00:00<00:00,  7.32it/s]
    Epoch 73 - Training loss: 0.0016410812095273286: 100%|██████████| 4/4 [00:00<00:00, 16.93it/s]
    Epoch 74 - Training loss: 0.001621665433049202: 100%|██████████| 4/4 [00:00<00:00,  7.48it/s]
    Epoch 74 - Training loss: 0.0016331682563759387: 100%|██████████| 4/4 [00:00<00:00, 16.80it/s]
    Epoch 75 - Training loss: 0.0016131284937728196: 100%|██████████| 4/4 [00:00<00:00,  7.39it/s]
    Epoch 75 - Training loss: 0.0016254328074865043: 100%|██████████| 4/4 [00:00<00:00, 16.86it/s]
    Epoch 76 - Training loss: 0.0016047552635427564: 100%|██████████| 4/4 [00:00<00:00,  7.44it/s]
    Epoch 76 - Training loss: 0.001617209374671802: 100%|██████████| 4/4 [00:00<00:00, 15.74it/s]
    Epoch 77 - Training loss: 0.0015960769960656762: 100%|██████████| 4/4 [00:00<00:00,  7.41it/s]
    Epoch 77 - Training loss: 0.0016090570425149053: 100%|██████████| 4/4 [00:00<00:00, 16.77it/s]
    Epoch 78 - Training loss: 0.0015874719538260251: 100%|██████████| 4/4 [00:00<00:00,  7.37it/s]
    Epoch 78 - Training loss: 0.0016009735991246998: 100%|██████████| 4/4 [00:00<00:00, 16.38it/s]
    Epoch 79 - Training loss: 0.0015788046875968575: 100%|██████████| 4/4 [00:00<00:00,  7.43it/s]
    Epoch 79 - Training loss: 0.0015928774955682456: 100%|██████████| 4/4 [00:00<00:00, 16.08it/s]
    Epoch 80 - Training loss: 0.0015702011587563902: 100%|██████████| 4/4 [00:00<00:00,  7.33it/s]
    Epoch 80 - Training loss: 0.0015847833128646016: 100%|██████████| 4/4 [00:00<00:00, 15.88it/s]
    Epoch 81 - Training loss: 0.0015614784497302026: 100%|██████████| 4/4 [00:00<00:00,  7.40it/s]
    Epoch 81 - Training loss: 0.00157659497926943: 100%|██████████| 4/4 [00:00<00:00, 16.56it/s]
    Epoch 82 - Training loss: 0.001552743517095223: 100%|██████████| 4/4 [00:00<00:00,  7.31it/s]
    Epoch 82 - Training loss: 0.0015683765232097358: 100%|██████████| 4/4 [00:00<00:00, 16.63it/s]
    Epoch 83 - Training loss: 0.00154375410056673: 100%|██████████| 4/4 [00:00<00:00,  7.38it/s]
    Epoch 83 - Training loss: 0.001559967058710754: 100%|██████████| 4/4 [00:00<00:00, 16.79it/s]
    Epoch 84 - Training loss: 0.0015348998131230474: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 84 - Training loss: 0.0015516088751610368: 100%|██████████| 4/4 [00:00<00:00, 17.02it/s]
    Epoch 85 - Training loss: 0.001525905419839546: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 85 - Training loss: 0.0015431527572218329: 100%|██████████| 4/4 [00:00<00:00, 16.30it/s]
    Epoch 86 - Training loss: 0.0015168407408054918: 100%|██████████| 4/4 [00:00<00:00,  7.39it/s]
    Epoch 86 - Training loss: 0.0015347360749728978: 100%|██████████| 4/4 [00:00<00:00, 16.32it/s]
    Epoch 87 - Training loss: 0.0015076906129252166: 100%|██████████| 4/4 [00:00<00:00,  7.35it/s]
    Epoch 87 - Training loss: 0.0015260436339303851: 100%|██████████| 4/4 [00:00<00:00, 16.51it/s]
    Epoch 88 - Training loss: 0.0014984908048063517: 100%|██████████| 4/4 [00:00<00:00,  7.24it/s]
    Epoch 88 - Training loss: 0.0015173557330854237: 100%|██████████| 4/4 [00:00<00:00, 16.72it/s]
    Epoch 89 - Training loss: 0.001489295915234834: 100%|██████████| 4/4 [00:00<00:00,  7.41it/s]
    Epoch 89 - Training loss: 0.0015086512721609324: 100%|██████████| 4/4 [00:00<00:00, 16.80it/s]
    Epoch 90 - Training loss: 0.0014799039927311242: 100%|██████████| 4/4 [00:00<00:00,  7.43it/s]
    Epoch 90 - Training loss: 0.0014998327242210507: 100%|██████████| 4/4 [00:00<00:00, 15.20it/s]
    Epoch 91 - Training loss: 0.0014703980705235153: 100%|██████████| 4/4 [00:00<00:00,  7.42it/s]
    Epoch 91 - Training loss: 0.0014907537261024117: 100%|██████████| 4/4 [00:00<00:00, 16.51it/s]
    Epoch 92 - Training loss: 0.0014607641787733883: 100%|██████████| 4/4 [00:00<00:00,  7.34it/s]
    Epoch 92 - Training loss: 0.0014817501360084862: 100%|██████████| 4/4 [00:00<00:00, 16.19it/s]
    Epoch 93 - Training loss: 0.001451130141504109: 100%|██████████| 4/4 [00:00<00:00,  7.54it/s]
    Epoch 93 - Training loss: 0.0014726218942087144: 100%|██████████| 4/4 [00:00<00:00, 16.41it/s]
    Epoch 94 - Training loss: 0.0014411707234103233: 100%|██████████| 4/4 [00:00<00:00,  7.47it/s]
    Epoch 94 - Training loss: 0.0014632090460509062: 100%|██████████| 4/4 [00:00<00:00, 16.25it/s]
    Epoch 95 - Training loss: 0.0014311287377495319: 100%|██████████| 4/4 [00:00<00:00,  7.50it/s]
    Epoch 95 - Training loss: 0.001453739416319877: 100%|██████████| 4/4 [00:00<00:00, 15.89it/s]
    Epoch 96 - Training loss: 0.001421066845068708: 100%|██████████| 4/4 [00:00<00:00,  7.36it/s]
    Epoch 96 - Training loss: 0.0014441101520787925: 100%|██████████| 4/4 [00:00<00:00, 17.07it/s]
    Epoch 97 - Training loss: 0.0014107958995737135: 100%|██████████| 4/4 [00:00<00:00,  7.22it/s]
    Epoch 97 - Training loss: 0.0014343376969918609: 100%|██████████| 4/4 [00:00<00:00, 16.49it/s]
    Epoch 98 - Training loss: 0.001400229608407244: 100%|██████████| 4/4 [00:00<00:00,  7.46it/s]
    Epoch 98 - Training loss: 0.0014243807236198336: 100%|██████████| 4/4 [00:00<00:00, 16.72it/s]
    Epoch 99 - Training loss: 0.0013895573792979121: 100%|██████████| 4/4 [00:00<00:00,  7.30it/s]
    Epoch 99 - Training loss: 0.0014142740401439369: 100%|██████████| 4/4 [00:00<00:00, 14.54it/s]