Simulación matemática 2018

Lázaro Alonso

Email:   `alonsosilva@iteso.mx, lazarus.alon@gmail.com`

< [git GitHub tutorial 2](Clase2_GitTutorial2.ipynb) | [Guía](Clase0_GuiaSimulacionM.ipynb) | [Programación Lineal](Clase5_ProgramacionLineal.ipynb) >
___

### Optimización de funciones escalares diferenciables con `SymPy`

> - Mediante optimización se obtienen soluciones elegantes tanto en teoría como en ciertas aplicaciones.
> - La teoría de optimización usa elementos comenzando con cálculo elemental y álgebra lineal básica, y luego se extiende con análisis funcional y convexo.
> - Las aplicaciones en optimización involucran ciencia, ingeniería, economía, finanzas e industria.
> - El amplio y creciente uso de la optimización lo hace escencial para estudiantes y profesionales de cualquier rama de la ciencia y la tecnología.

**Referencia:**
- http://www.math.uwaterloo.ca/~hwolkowi//henry/reports/talks.d/t06talks.d/06msribirs.d/optimportance.shtml

Algunas aplicaciones son:

1. Ingeniería
 - Encontrar la composición de equilibrio de una mezcla de diferentes átomos.
 - Planeación de ruta para un robot (o vehículo aéreo no tripulado).
 - Planeación de la mano de obra óptima en una construcción o planta de producción.
2. Distribución óptima de recursos.
 - Distribución de rutas de vuelo.
 - Encontrar una dieta óptima.
 - Planeación de ruta óptima.
3. Optimización financiera
 - Administración de riesgos.
 - Portafolios de inversión.

En esta clase veremos aspectos básicos de optimización. En específico, veremos cómo obtener máximos y mínimos de una función escalar de una variable (como en cálculo diferencial).

___
## 0. Librerías que usaremos

Como ya dijimos en la primer clase `python` es el lenguaje de programación (el cual es de alto nivel). Sin embargo, `python` solo tiene unos pocos comandos primitivos y para hacer más fácil su uso en nuestras actividades de simulación en ingeniería, otras personas ya han escrito ciertas librerías por nosotros.

### 0.1 `NumPy`

`NumPy` (Numerical Python) es la librería fundamental para computación científica (numérica) con `Python`. Contiene, entre otras cosas:
- un objeto tipo arreglo N-dimensional muy poderoso
- funciones sofisticadas
- funciones de álgebra lineal, transformada de Fourier y números aleatorios.

Por lo anterior, `NumPy` es de amplio uso entre la comunidad científica e ingenieril (por su manejo de cantidades vectoriales). De la misma manera, se usa para guardar datos. Para nuestros propósitos, se puede usar libremente.

**Referencia:**
- http://www.numpy.org/

`NumPy` ya viene incluido en la instalación estándar de Anaconda por defecto. Para comenzar a usarlo, solo debemos de importarlo:

```python
# importar la librería numpy
import numpy as np
```

### 0.2 `SymPy`

`SymPy` (Symbolic Python) es una librería de `Python` para matemáticas simbólicas. Su objetivo es convertirse en un sistema de álgebra computacional con las mejores características, manteniendo el código lo más simple posible para que sea comprensible.

**Referencia:**
- http://www.sympy.org/en/index.html

`SymPy` ya viene incluido en la instalación estándar de Anaconda por defecto. Para comenzar a usarlo, solo debemos de importarlo:

```python
# importar la librería sympy
import sympy as sym
# imprimir en formato latex
sym.init_printing(use_latex='mathjax')
```

La funcionalidad de imprimir en formato LaTex que nos da `SymPy` mediante el proyecto `mathjax` hace de `SymPy` una herramienta muy atractiva...

Notar que en `SymPy` y en `NumPy` existen funciones con el mismo nombre, pero reciben tipos de datos diferentes...

Explicar el uso de la sintaxis `from numpy import *` y sus peligros (no recomendable).

```python
# diferencias de funciones de sympy y numpy
sym.var('x')
sym.sin(x)*np.sin(3)
```

$$0.141120008059867 \sin{\left (x \right )}$$

```python
# importar con * y ver que pasa
# from numpy import *
# from sympy import *
# No recomendado
```

### 0.3 `PyPlot` de `matplotlib`

El módulo `PyPlot` de la librería `matplotlib` contiene funciones que nos permite generar una gran cantidad de gráficas rápidamente. Las funciones de este módulo están escritas con el mismo nombre que las funciones para graficar en `Matlab`.

**Referencia:**
- https://matplotlib.org/api/pyplot_summary.html

```python
# importar matplotlib.pyplot
import matplotlib.pyplot as plt
# comando para que las gráficas salgan en la misma ventana
%matplotlib inline
```

```python
### Style, plots
```

```python

```

Ya que revisamos todas las librerías que usaremos, empecemos con la clase como tal...

___
Basamos todos los resultados en los siguientes teoremas:

## 1. Teorema de Fermat (análisis)

Si una función $f(x)$ alcanza un máximo o mínimo local en $x=c$, y si la derivada $f'(c)$ existe en el punto $c$, entonces $f'(c) = 0$.

### Ejemplo

Sabemos que la función $f(x)=x^2$ tiene un mínimo global en $x=0$, pues

$$f(x)=x^2\geq0,\qquad\text{y}\qquad f(x)=x^2=0 \qquad\text{si y solo si}\qquad x=0.$$

```python
# declarar la variable real x
#t = sym.Symbol('x')
#t
sym.var('x', real = True)
```

$$x$$

```python
# declarar ahora f=x^2 y mostrar
f = x**2
f
```

$$x^{2}$$

```python
# derivar f respecto a x y mostrar
df = sym.diff(f, x)
df
```

$$2 x$$

```python
# resolver f'(x)=0 y mostrar soluciones
xc = sym.solve(df, x)
xc
```

$$\left [ 0\right ]$$

Veamos la gráfica...

```python
# convertir f e una función que se pueda evaluar numéricamente
#(función lambdify de la librería sympy)
f_num = sym.lambdify([x], f, 'numpy')
x_vec = np.linspace(-5, 5, 100)

# graficar
plt.plot(x_vec, f_num(x_vec))
plt.xlabel('$x$')
plt.ylabel('$x^2$')
plt.show()
```

Ver diferencias entre f y f_num

```python
# intentar evaluar f y f_num

```

**Otra manera de hacer lo anterior**

Concepto de función...

```python
def f(x):
    return x**2
```

```python
f(x)
```

$$x^{2}$$

```python
f(np.array([5.124, 2.5436]))
```

    array([ 26.255376  ,   6.46990096])

```python
df = sym.diff(f(x), x)
df
```

$$2 x$$

```python
x_c = sym.solve(df, x)
x_c[0]
```

$$0$$

```python
# graficar
plt.plot(x_vec, f(x_vec))
plt.xlabel('$x$')
plt.ylabel('$x^2$')
plt.show()
```

El converso del teorema anterior no es cierto.

### Actividad
Considere $g(x)=x^3$.
- Usando `sympy`, muestre que $g'(0)=0$.
- Sin embargo, descartar que $x=0$ es un extremo de $g(x)$ viendo su **gráfica**.

```python
def g(x):
    return x**3
```

```python
dg = sym.diff(g(x), x)
dg
```

$$3 x^{2}$$

```python
puntos_criticos = sym.solve(dg, x)
puntos_criticos
```

$$\left [ 0\right ]$$

```python
dg_eval0 = dg.subs(x, puntos_criticos[0])
dg_eval0
```

$$0$$

```python
# graficar
plt.plot(x_vec, g(x_vec))
plt.xlabel('$x$')
plt.ylabel('$x^3$')
plt.show()
```

## 2. Criterio de la segunda derivada

Sea $f(x)$ una función tal que $f’(c)=0$ y cuya segunda derivada existe en un intervalo abierto que contiene a $c$.
- Si $f’’(c)>0$, entonces $f(c)$ es un mínimo relativo.
- Si $f’’(c)<0$, entonces $f(c)$ es un máximo relativo.
- Si $f’’(c)=0$, entonces el criterio no decide.

### Ejemplo

Mostrar, usando `sympy`, que la función $f(x)=x^2$ tiene un mínimo relativo en $x=0$.

Ya vimos que $f'(0)=0$. Notemos que:

```python
f = x**2
#d2f = sym.diff(f, x, x)
d2f = sym.diff(f, x, 2)
d2f
```

```python
d2f>0
```

Por tanto, por el criterio de la segunda derivada, $f(0)=0$ es un mínimo relativo (en efecto, el mínimo global).

### Ejemplo
¿Qué pasa con $g(x)=x^3$ al intentar utilizar el criterio de la segunda derivada? (usar `sympy`).

```python
g(x)
```

```python
d2g = sym.diff(g(x), x, x)
d2g
```

```python
d2g.subs(x, 0)
```

### Actividad

¿Qué pasa con $h(x)=x^4$ al intentar utilizar el criterio de la segunda derivada?.

```python
def h(x):
    return x**4
```

```python
dh = sym.diff(h(x), x, 1)
dh
```

```python
p_c_h = sym.solve(dh, x)
p_c_h
```

```python
d2h = sym.diff(h(x), x, 2)
d2h
```

```python
d2h.subs(x, p_c_h[0])
```

Como la segunda derivada en el punto crítico es cero, el criterio de la segunda derivada no decide.

## 3. Método para determinar extremos absolutos de una función continua y=f(x) en [a,b]

- Determinar todos los valores críticos $c_1, c_2, c_3, \dots, c_n$ en $(a,b)$.
- Evaluar $f$ en todos los valores críticos y en los extremos $x=a$ y $x=b$.
- El más grande y el más pequeño de los valores de la lista $f(a), f(b), f(c_1), f(c_2), \dots, f(c_n)$ son el máximo absoluto y el mínimo absoluto, respectivamente, de f en el intervalo [a,b].

### Ejemplo

Determinar los extremos absolutos de $f(x)=x^2-6x$ en $\left[0,5\right]$.

Obtenemos los puntos críticos de $f$ en $\left[0,5\right]$:

```python
f = x**2-6*x
f
```

```python
df = sym.diff(f, x)
df
```

```python
x_c = sym.solve(df, x)
x_c
```

Evaluamos $f$ en los extremos y en los puntos críticos:

```python
f.subs(x, 0), f.subs(x, 5), f.subs(x, x_c[0])
```

Concluimos que el máximo absoluto de $f$ en $\left[0,5\right]$ es $0$ y se alcanza en $x=0$, y que el mínimo absoluto es $-9$ y se alcanza en $x=3$.

```python
f_num = sym.lambdify([x], f, 'numpy')
x_vec = np.linspace(0, 5, 100)

plt.figure(figsize=(8,6))
plt.plot(x_vec, f_num(x_vec), 'k--', label = '$y=f(x)$')
plt.plot([0], [0], '*r', label = '$(0,0=\max_{0\leq x\leq 5} f(x))$')
plt.plot([3], [-9], '*b', label = '$(3,-9=\min_{0\leq x\leq 5} f(x))$')
plt.legend(loc='best')
plt.xlabel('x')
plt.grid()
plt.show()
```

### Actividad
Determinar los valores extremos absolutos de $h(x)=x^3-3x$ en $\left[-2.2,1.8\right]$, usando `sympy`. Mostrar en una gráfica.

```python
sym.var('x', real = True)
```

```python
def h(x):
    return x**3-3*x
```

```python
dh = sym.diff(h(x), x)
dh
```

```python
p_c = sym.solve(dh, x)
p_c
```

```python
h(-1), h(1), h(-2.2), h(1.8)
```

```python
x_vec = np.linspace(-2.2, 1.8, 50)

plt.figure(figsize=(7,5))
plt.plot(x_vec, h(x_vec), 'b:', label = '$y=x^3-3x$')
plt.plot([-2.2], [h(-2.2)], 'mo', label = '(-2.2,-4.048)')
plt.plot([-1], [h(-1)], 'yo', label = '(-1,2)')
plt.legend(loc='best')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.grid()
plt.show()
```

### En varias variables...

El procedimiento es análogo.

Si una función $f:\mathbb{R}^n\to\mathbb{R}$ alcanza un máximo o mínimo local en $\boldsymbol{x}=\boldsymbol{c}\in\mathbb{R}^n$, y $f$ es diferenciable en el punto $\boldsymbol{x}=\boldsymbol{c}$, entonces $\left.\frac{\partial f}{\partial \boldsymbol{x}}\right|_{\boldsymbol{x}=\boldsymbol{c}}=\boldsymbol{0}$ (todas las derivadas parciales en el punto $\boldsymbol{x}=\boldsymbol{c}$ son cero).

**Criterio de la segunda derivada:** para ver si es máximo o mínimo, se toma la segunda derivada (matriz jacobiana) y se verifica definición negativa o positiva, respectivamente.

Si se restringe a cierta región, hay ciertas técnicas. La más general, pero también la más compleja es la de **multiplicadores de Lagrange**.

```python
sym.var('x y')
x, y
```

```python
def f(x, y):
    return x**2 + y**2
```

```python
dfx = sym.diff(f(x,y), x)
dfy = sym.diff(f(x,y), y)
dfx, dfy
```

```python
xy_c = sym.solve([dfx, dfy], [x, y])
xy_c
```

```python
x_c, y_c = xy_c[x], xy_c[y]
x_c, y_c
```

```python
d2fx = sym.diff(f(x,y), x, 2)
d2fy = sym.diff(f(x,y), y, 2)
dfxy = sym.diff(f(x,y), x, y)

Jf = sym.Matrix([[d2fx, dfxy], [dfxy, d2fy]])
Jf.eigenvals()
```

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

x = np.linspace(-2, 2, 100)
y = x
X, Y = np.meshgrid(x, y)

ax.plot_surface(X, Y, f(X, Y))
ax.plot([x_c], [y_c], [f(x_c,y_c)], '*r');
```

### Tareas.

1. Por parejas harán un proyecto colaborativo.
2. Introducción a numpy. Obtener máximos y mínimos absolutos, usando `sympy`, de las funciones dadas en los intervalos dados, y graficar la función en dicho intervalo señalando los puntos máximo y mínimo absolutos.

**Definir fecha de las tareas**

___

< [git GitHub tutorial 2](Clase2_GitTutorial2.ipynb) | [Guía](Clase0_GuiaSimulacionM.ipynb) | [Programación Lineal](Clase5_ProgramacionLineal.ipynb) >

Created with Jupyter by Lázaro Alonso.
  Copyright:  Public Domain como en [CC](https://creativecommons.org/licenses/by/2.0/) (Exepto donde se indique lo contrario)


# Training a model on forces and energies

In addition to the energy, machine learning models can also be used to model molecular forces.
These are $N_\mathrm{atoms} \times 3$ arrays describing the Cartesian force acting on each atom due to the overall (potential) energy.
They are formally defined as the negative gradient of the energy $E_\mathrm{pot}$ with respect to the nuclear positions $\mathbf{R}$
\begin{equation}
\mathbf{F}^{(\alpha)} = -\frac{\partial E_\mathrm{pot}}{\partial \mathbf{R}^{(\alpha)}},
\end{equation}
where $\alpha$ is the index of the nucleus.

The above expression offers a straightforward way to include forces in machine learning models by simply defining a model for the energy and taking the appropriate derivatives.
The resulting model can directly be trained on energies and forces.
Moreover, in this manner energy conservation and the correct behaviour under rotations of the molecule is guaranteed.

Using forces in addition to energies to construct a machine learning model offers several advantages.
Accurate force predictions are important for molecular dynamics simulations, which will be covered in the subsequent tutorial.
Forces also encode a greater wealth of information than the energies.
For every molecule, only one energy is present, while there are $3N_\mathrm{atoms}$ force entries.
This property, combined with the fact that reference forces can be computed at the same cost as energies, makes models trained on forces and energies very data efficient.

In the following, we will show how to train such force models and how to use them in practical applications.

## Preparing the data

The process of preparing the data is smilar to the tutorial on [QM9](tutorial_02_qm9.ipynb). We begin by importing all relevant packages and generating a directory for the tutorial experiments.

```python
import schnetpack as spk
import os

forcetut = './forcetut'
if not os.path.exists(forcetut):
    os.makedirs(forcetut)
```

Next, the data needs to be loaded from a suitable dataset.
For convenience, we use the MD17 dataset class provided in SchNetPack, which automtically downloads and builds suitable databases containing energies and forces for a range of small organic molecules.
In this case, we use the ethanol molecule as an example.

```python
from schnetpack.datasets import MD17

ethanol_data = MD17(os.path.join(forcetut,'ethanol.db'), molecule='ethanol')
```

For custom datasets, the data would have to be loaded via the SchNetPack `AtomsData` class.
In this case, one needs to make sure, that the naming of properties is kept consistent. The `schnetpack.Properties` module provides standard names for a wide range of properties.
Here, we use the definitions provided with the `MD17` class.

In order to train force models, forces need to be included in the reference data.
Once the dataset was loaded, this can e.g. be checked by calling the `get_properties` function for sample entries.
Here we look at the configuration with the index 0.
The function returns an ASE `Atoms` object and a dictionary containing the loaded properties:

```python
atoms, properties = ethanol_data.get_properties(0)

print('Loaded properties:\n', *['{:s}\n'.format(i) for i in properties.keys()])
```

    Loaded properties:
     energy
     forces
     _atomic_numbers
     _positions
     _cell
     _neighbors
     _cell_offset

As can be seen,  `energy` and `forces` are included in the properties dictionary. To have a look at the `forces` array and check whether it has the expected dimensions, we can call:

```python
print('Forces:\n', properties[MD17.forces])
print('Shape:\n', properties[MD17.forces].shape)
```

    Forces:
     tensor([[ 1.4517e+00,  6.0192e+00,  5.2068e-07],
            [ 1.7953e+01, -5.1624e+00,  3.4900e-07],
            [-4.0884e+00,  2.2590e+01,  3.3088e-06],
            [-1.1416e+00, -9.7469e+00,  7.6473e+00],
            [-1.1416e+00, -9.7469e+00, -7.6473e+00],
            [-2.4821e+00,  4.9335e+00,  4.3700e+00],
            [-2.4821e+00,  4.9335e+00, -4.3700e+00],
            [-5.5148e+00, -3.0207e+00, -8.9093e-09],
            [-2.4393e+00, -1.0838e+01, -6.0721e-08]])
    Shape:
     torch.Size([9, 3])

The `atoms` object can e.g. be used to visualize the ethanol molecule:

```python
from ase.visualize import view
view(atoms, viewer='x3d')
```

<html>

 <head>

  <title>ASE atomic visualization</title>

  <link rel="stylesheet" type="text/css"

   href="https://www.x3dom.org/x3dom/release/x3dom.css">

  </link>

 </head>

 <body>

  <X3D style="margin:0; padding:0; width:100%; height:100%; border:none;">

   <Scene>

    <Transform translation="0.01 -0.57 0.00">

     <Shape>

      <Appearance>

       <Material diffuseColor="0.565 0.565 0.565" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.76">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="-1.29 0.25 0.00">

     <Shape>

      <Appearance>

       <Material diffuseColor="0.565 0.565 0.565" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.76">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="1.13 0.31 0.00">

     <Shape>

      <Appearance>

       <Material diffuseColor="1.000 0.051 0.051" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.66">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="0.04 -1.20 0.89">

     <Shape>

      <Appearance>

       <Material diffuseColor="1.000 1.000 1.000" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.31">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="0.04 -1.20 -0.89">

     <Shape>

      <Appearance>

       <Material diffuseColor="1.000 1.000 1.000" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.31">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="-1.32 0.88 0.89">

     <Shape>

      <Appearance>

       <Material diffuseColor="1.000 1.000 1.000" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.31">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="-1.32 0.88 -0.89">

     <Shape>

      <Appearance>

       <Material diffuseColor="1.000 1.000 1.000" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.31">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="-2.14 -0.42 0.00">

     <Shape>

      <Appearance>

       <Material diffuseColor="1.000 1.000 1.000" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.31">

      </Sphere>

     </Shape>

    </Transform>

    <Transform translation="1.99 -0.14 0.00">

     <Shape>

      <Appearance>

       <Material diffuseColor="1.000 1.000 1.000" specularColor="0.5 0.5 0.5">

       </Material>

      </Appearance>

      <Sphere radius="0.31">

      </Sphere>

     </Shape>

    </Transform>

   </Scene>

  </X3D>

 </body>

</html>

Next, the data is split into training (1000 points), test (500 points) and validation set (remainder) and data loaders are created. This is done in the same way as described in the [QM9 tutorial](tutorial_02_qm9.ipynb).

```python
train, val, test = spk.train_test_split(
        data=ethanol_data,
        num_train=1000,
        num_val=500,
        split_file=os.path.join(forcetut, "split.npz"),
    )

train_loader = spk.AtomsLoader(train, batch_size=100, shuffle=True)
val_loader = spk.AtomsLoader(val, batch_size=100)
```

Once again, we want to use the mean and standardeviation of the energies in the training data to precondition our model.
This only needs to be done for the energies, since the forces are obtained as derivatives and automatically capture the scale of the data.
Unlike in the case of QM9, the subtraction of atomic reference energies is not necessary, since only configurations of the same molecule are loaded.
All this can be done via the `get_statistics` function of the `AtomsLoader` class:

```python
means, stddevs = train_loader.get_statistics(
    spk.datasets.MD17.energy, divide_by_atoms=True
)

print('Mean atomization energy / atom:      {:12.4f} [kcal/mol]'.format(means[MD17.energy][0]))
print('Std. dev. atomization energy / atom: {:12.4f} [kcal/mol]'.format(stddevs[MD17.energy][0]))
```

    Mean atomization energy / atom:       -10799.5244 [kcal/mol]
    Std. dev. atomization energy / atom:       0.4679 [kcal/mol]

## Building the model

After having prepared the data in the above way, we can now build and train the force model.
This is done in the same two steps as described in [QM9 tutorial](tutorial_02_qm9.ipynb):

1. Building the representation
2. Defining an output module

For the representation we can use the same `SchNet` layer as in the previous tutorial:

```python
n_features = 128

schnet = spk.representation.SchNet(
    n_atom_basis=n_features,
    n_filters=n_features,
    n_gaussians=25,
    n_interactions=3,
    cutoff=5.,
    cutoff_network=spk.nn.cutoff.CosineCutoff
)
```

Since we want to model forces, the `Atomwise` output module needs to be adapted slightly.
We will still use one module to predict the energy, preconditioning with the mean and standard deviation per atom of the energy.

However, since the forces should be described as the derivative of the energy, we have to indicate that the corresponding derviative of the model should be computed.
This is done by specifying `derivative=MD17.forces`, which also assigns the computed derivative to the property `MD17.forces`.
Since the forces are the negative gradient, we also need to enable `negative_dr=True`, which simply multipiies the derviative with -1.

```python
energy_model = spk.atomistic.Atomwise(
    n_in=n_features,
    property=MD17.energy,
    mean=means[MD17.energy],
    stddev=stddevs[MD17.energy],
    derivative=MD17.forces,
    negative_dr=True
)
```

 Both modules are then combined to an `AtomisticModel`.

```python
model = spk.AtomisticModel(representation=schnet, output_modules=energy_model)
```

## Training the model

To train the model on energies and forces, we need to update the loss function to include the latter.
This combined loss function is:
\begin{equation}
\mathcal{L}(E_\mathrm{ref},\mathbf{F}_\mathrm{ref},E_\mathrm{pred}, \mathbf{F}_\mathrm{pred}) = \frac{1}{n_\text{train}} \sum_{n=1}^{n_\text{train}} \left[  \rho \left( E_\mathrm{ref} - E_\mathrm{pred} \right)^2  +  \frac{(1-\rho)}{3N_\mathrm{atoms}} \sum^{N_\mathrm{atoms}}_\alpha \left\| \mathbf{F}_\mathrm{ref}^{(\alpha)} - \mathbf{F}_\mathrm{pred}^{(\alpha)} \right\|^2 \right],
\end{equation}
where we take the predicted forces to be:
\begin{equation}
\mathbf{F}_\mathrm{pred}^{(\alpha)} = -\frac{\partial E_\mathrm{pred}}{\partial \mathbf{R}^{(\alpha)}}.
\end{equation}

We have introduced a parameter $\rho$ in order to control the tradeoff between energy and force loss.
By varying this parameter, the accuracy on energies and forces can be tuned.
Setting $\rho=0$ only forces are trained, while in the case of $\rho=1$ only energies are learned.
Using PyTorch, we can implement this loss function in the following way:

```python
import torch

# tradeoff
rho_tradeoff = 0.1

# loss function
def loss(batch, result):
    # compute the mean squared error on the energies
    diff_energy = batch[MD17.energy]-result[MD17.energy]
    err_sq_energy = torch.mean(diff_energy ** 2)

    # compute the mean squared error on the forces
    diff_forces = batch[MD17.forces]-result[MD17.forces]
    err_sq_forces = torch.mean(diff_forces ** 2)

    # build the combined loss function
    err_sq = rho_tradeoff*err_sq_energy + (1-rho_tradeoff)*err_sq_forces

    return err_sq
```

Next, we procede in the same manner as in the [QM9 tutorial](tutorial_02_qm9.ipynb). First, we specify that the Adam optimizer from PyTorch should be used to train the model:

```python
from torch.optim import Adam

# build optimizer
optimizer = Adam(model.parameters(), lr=5e-4)
```

Then, we construct the trainer hooks to monitor the training process and anneal the learning rate.
Since we also learn forces in addition to the energies, we include a corresponding metric into the logger.

```python
# before setting up the trainer, remove previous training checkpoints and logs
%rm -rf ./forcetut/checkpoints
%rm -rf ./forcetut/log.csv

import schnetpack.train as trn

# set up metrics
metrics = [
    spk.metrics.MeanAbsoluteError(MD17.energy),
    spk.metrics.MeanAbsoluteError(MD17.forces)
]

# construct hooks
hooks = [
    trn.CSVHook(log_path=forcetut, metrics=metrics),
    trn.ReduceLROnPlateauHook(
        optimizer,
        patience=5, factor=0.8, min_lr=1e-6,
        stop_after_min=True
    )
]
```

Finally, we build the SchNetPack `Trainer` and pass the optimizer, loss function, hooks and data loaders.

```python
trainer = trn.Trainer(
    model_path=forcetut,
    model=model,
    hooks=hooks,
    loss_fn=loss,
    optimizer=optimizer,
    train_loader=train_loader,
    validation_loader=val_loader,
)
```

We then train our model for 300 epochs, which should take approximately 10 minutes on a notebook GPU.

```python
# check if a GPU is available and use a CPU otherwise
if torch.cuda.is_available():
    device = "cuda"
else:
    device = "cpu"

# determine number of epochs and train
n_epochs = 300
trainer.train(device=device, n_epochs=n_epochs)
```

Training will produce several files in the `model_path` directory, which is `forcetut` in our case.
The split is stored in `split.npz`.
Checkpoints are written to `checkpoints` periodically, which can be used to restart training.
A copy of the best model is stored in `best_model`, which can directly be accessed using the `torch.load` function.
Since we specified the CSV logger, the training progress is saved to `log.csv`.

Using the CSV file, the training progress can be vizualized.
An example showing the evolution of the mean absolute errors (MAEs) during training is shown below.
Besides the `schnetpack.train.CSVHook`, it is also possible to use the `schnetpack.train.tensorboardHook`.
This makes it possible to monitor the training in real time with TensorBoard.

```python
import numpy as np
import matplotlib.pyplot as plt
from ase.units import kcal, mol

# Load logged results
results = np.loadtxt(os.path.join(forcetut, 'log.csv'), skiprows=1, delimiter=',')

# Determine time axis
time = results[:,0]-results[0,0]

# Load the validation MAEs
energy_mae = results[:,4]
forces_mae = results[:,5]

# Get final validation errors
print('Validation MAE:')
print('    energy: {:10.3f} kcal/mol'.format(energy_mae[-1]))
print('    forces: {:10.3f} kcal/mol/\u212B'.format(forces_mae[-1]))

# Construct figure
plt.figure(figsize=(14,5))

# Plot energies
plt.subplot(1,2,1)
plt.plot(time, energy_mae)
plt.title('Energy')
plt.ylabel('MAE [kcal/mol]')
plt.xlabel('Time [s]')

# Plot forces
plt.subplot(1,2,2)
plt.plot(time, forces_mae)
plt.title('Forces')
plt.ylabel('MAE [kcal/mol/\u212B]')
plt.xlabel('Time [s]')

plt.show()
```

It should be noted, that the model trained here is used exclusively for demonstrative purposes.
Accordingly, its size and the training time have been reduced significantly.
This puts strong constraints on the accuracy that can be obtained.
For practical applications, one would e.g. increase the number of features, the interaction layers, the learning rate schedule and train until convergence (removing the `n_epochs` keyword from the `trainer`).

SchNetPack also provides the script `schnetpack_md17.py`, which fully automates the whole training process for MD17 and comes with a series of reasonable default settings.

## Using the model

Since all models in SchNetPack are stored in the same way, we can use the trained force model in exactly the same manner as described in the [QM9 tutorial](tutorial_02_qm9.ipynb).

To load the model stored in the `best_model` file, we use the `torch.load` function.
It will automatically be moved to the device it was trained on.

```python
best_model = torch.load(os.path.join(forcetut, 'best_model'))
```

To evaluate its performance on the test data, we use an adapted version of the loop introduced in the previous tutorial.
Since the MD17 dataset of ethanol contains approximately 550 000 configurations, the full evaluation takes approximately 10 minutes on GPU.

```python
test_loader = spk.AtomsLoader(test, batch_size=100)

energy_error = 0.0
forces_error = 0.0

for count, batch in enumerate(test_loader):
    # move batch to GPU, if necessary
    batch = {k: v.to(device) for k, v in batch.items()}

    # apply model
    pred = best_model(batch)

    # calculate absolute error of energies
    tmp_energy = torch.sum(torch.abs(pred[MD17.energy] - batch[MD17.energy]))
    tmp_energy = tmp_energy.detach().cpu().numpy() # detach from graph & convert to numpy
    energy_error += tmp_energy

    # calculate absolute error of forces, where we compute the mean over the n_atoms x 3 dimensions
    tmp_forces = torch.sum(
        torch.mean(torch.abs(pred[MD17.forces] - batch[MD17.forces]), dim=(1,2))
    )
    tmp_forces = tmp_forces.detach().cpu().numpy() # detach from graph & convert to numpy
    forces_error += tmp_forces

    # log progress
    percent = '{:3.2f}'.format(count/len(test_loader)*100)
    print('Progress:', percent+'%'+' '*(5-len(percent)), end="\r")

energy_error /= len(test)
forces_error /= len(test)

print('\nTest MAE:')
print('    energy: {:10.3f} kcal/mol'.format(energy_error))
print('    forces: {:10.3f} kcal/mol/\u212B'.format(forces_error))
```

    Progress: 99.98%
    Test MAE:
        energy:      0.396 kcal/mol
        forces:      1.100 kcal/mol/Å

## Interface to ASE

As was shown in the [QM9 tutorial](tutorial_02_qm9.ipynb), one can also use the `AtomsConverter` to directly operate on ASE atoms objects.

Having access to molecular forces also makes it possible to perform a variety of different simulations.
The `SpkCalculator` offers a simple way to perform all computations available in the ASE package.
Below, we create an ASE calculator from the trained model and the previously generated `atoms` object (see [Preparing the data](#Preparing-the-data)).
One important point is, that the MD17 dataset uses kcal/mol and kcal/mol/Å as units for energies and forces.
For the ASE interface, these need to be converted to the standard internal ASE units eV and eV/Å.
This can be done by either passing the conversion factor or a string denoting the unit to the keywords `energy_units` and `force_units`.

```python
calculator = spk.interfaces.SpkCalculator(
    model=best_model,
    device=device,
    energy=MD17.energy,
    forces=MD17.forces,
    energy_units='kcal/mol',
    forces_units='kcal/mol/A'
)

atoms.set_calculator(calculator)

print('Prediction:')
print('energy:', atoms.get_total_energy())
print('forces:', atoms.get_forces())
```

    Prediction:
    energy: [-4215.326]
    forces: [[ 4.8149731e-02  2.2211398e-01 -8.2710464e-08]
     [ 7.8540379e-01 -2.2908500e-01 -3.1016423e-08]
     [-9.7170897e-02  1.0150807e+00 -4.1355232e-08]
     [-2.6240392e-02 -4.0961695e-01  3.1710267e-01]
     [-2.6240392e-02 -4.0961662e-01 -3.1710231e-01]
     [-1.2042553e-01  2.2401318e-01  1.8506417e-01]
     [-1.2042536e-01  2.2401318e-01 -1.8506417e-01]
     [-3.1431168e-01 -1.4146599e-01 -2.2616142e-08]
     [-1.2873909e-01 -4.9543568e-01 -4.2001407e-09]]

Among the simulations which can be done by using ASE and a force model are geometry optimisation, normal mode analysis and simple molecular dynamics simulations.

The `AseInterface` of SchNetPack offers a convenient way to perform basic versions of these computations.
Only a file specifying the geometry of the molecule and a pretrained model are needed.

We will first generate a XYZ file containing an ethanol configuration:

```python
from ase import io

# Generate a directory for the ASE computations
ase_dir = os.path.join(forcetut, 'ase_calcs')

if not os.path.exists(ase_dir):
    os.mkdir(ase_dir)

# Write a sample molecule
molecule_path = os.path.join( ase_dir, 'ethanol.xyz')
io.write(molecule_path, atoms, format='xyz')
```

The `AseInterface` is initialized by passing the path to the molecule, the model and a computation directory.
In addion, the computation device for the force model and how energies and forces are called in the output, as well as their units, need to be provided.

```python
ethanol_ase = spk.interfaces.AseInterface(
    molecule_path,
    best_model,
    ase_dir,
    device,
    energy=MD17.energy,
    forces=MD17.forces,
    energy_units='kcal/mol',
    forces_units='kcal/mol/A'
)
```

### Geometry optimization

For some applications it is neccessary to relax a molecule to an energy minimum.
In order to perform this optimization of the molecular geometry, we can simply call

```python
ethanol_ase.optimize(fmax=1e-4)
```

                    Step[ FC]     Time          Energy          fmax
    BFGSLineSearch:    0[  0] 14:04:31    -4215.326172        1.0197
    BFGSLineSearch:    1[  2] 14:04:31    -4215.358887        0.6782
    BFGSLineSearch:    2[  4] 14:04:31    -4215.370117        0.3471
    BFGSLineSearch:    3[  6] 14:04:31    -4215.373535        0.2088
    BFGSLineSearch:    4[  8] 14:04:31    -4215.376465        0.1770
    BFGSLineSearch:    5[ 10] 14:04:31    -4215.377441        0.0837
    BFGSLineSearch:    6[ 12] 14:04:31    -4215.377930        0.0413
    BFGSLineSearch:    7[ 15] 14:04:31    -4215.377930        0.0323
    BFGSLineSearch:    8[ 16] 14:04:31    -4215.378418        0.0184
    BFGSLineSearch:    9[ 17] 14:04:31    -4215.378418        0.0042
    BFGSLineSearch:   10[ 19] 14:04:31    -4215.378418        0.0022
    BFGSLineSearch:   11[ 20] 14:04:31    -4215.378418        0.0016
    BFGSLineSearch:   12[ 22] 14:04:31    -4215.378418        0.0028
    BFGSLineSearch:   13[ 23] 14:04:31    -4215.378418        0.0022
    BFGSLineSearch:   14[ 25] 14:04:31    -4215.378418        0.0018
    BFGSLineSearch:   15[ 28] 14:04:31    -4215.378418        0.0006
    BFGSLineSearch:   16[ 29] 14:04:31    -4215.378418        0.0001

Since we trained only a reduced model, the accuracy of energies and forces is not optimal and several steps are needed to optimize the geometry.

### Normal mode analysis

Once the geometry was optimized, normal mode frequencies can be obtained from the Hessian (matrix of second derivatives) of the molecule.
The Hessian is a measure of the curvature of the potential energy surface and normal mode frequencies are useful for determining, whether an optimization has reached a minimum.
Using the `AseInterface`, normal mode frequencies can be obtained via:

```python
ethanol_ase.compute_normal_modes()
```

    Writing ./forcetut/ase_calcs/normal_modes.eq.pckl
    Writing ./forcetut/ase_calcs/normal_modes.0x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.0x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.0y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.0y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.0z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.0z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.1x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.1x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.1y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.1y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.1z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.1z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.2x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.2x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.2y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.2y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.2z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.2z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.3x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.3x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.3y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.3y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.3z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.3z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.4x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.4x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.4y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.4y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.4z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.4z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.5x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.5x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.5y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.5y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.5z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.5z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.6x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.6x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.6y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.6y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.6z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.6z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.7x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.7x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.7y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.7y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.7z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.7z+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.8x-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.8x+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.8y-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.8y+.pckl
    Writing ./forcetut/ase_calcs/normal_modes.8z-.pckl
    Writing ./forcetut/ase_calcs/normal_modes.8z+.pckl
    ---------------------
      #    meV     cm^-1
    ---------------------
      0    2.3i     18.3i
      1    1.2i      9.7i
      2    0.1i      0.6i
      3    0.0i      0.2i
      4    0.1       0.7
      5    0.7       5.4
      6   23.0     185.5
      7   29.1     234.8
      8   51.1     412.1
      9  102.0     823.0
     10  108.7     876.9
     11  126.9    1023.1
     12  131.6    1061.6
     13  143.6    1158.3
     14  153.6    1238.6
     15  155.4    1253.3
     16  165.4    1334.4
     17  173.9    1402.6
     18  177.4    1430.8
     19  182.4    1471.0
     20  185.4    1495.2
     21  360.7    2909.2
     22  366.1    2953.0
     23  372.4    3004.0
     24  382.1    3081.9
     25  382.6    3085.6
     26  466.8    3764.8
    ---------------------
    Zero-point energy: 2.120 eV

    /home/mitx/anaconda3/envs/schnet_debug/lib/python3.6/site-packages/ase/vibrations/vibrations.py:355: ComplexWarning: Casting complex values to real discards the imaginary part
      fd.write('Mode #%d, f = %.1f%s cm^-1' % (n, f[n], c))

Imaginary frequencies indicate, that the geometry optimisation has not yet reached a minimum.
The `AseInterface` also creates an `normal_modes.xyz` file which can be used to visualize the vibrations with jmol.

### Molecular dynamics

Finally, it is also possible to basic run molecular dynamics simulations using this interface.
To do so, we first need to prepare the system, where we specify the simulation file.
This routine automatically initializes the velocities of the atoms to a random number corresponding to a certain average kinetic energy.

```python
ethanol_ase.init_md(
    'simulation'
)
```

The actual simulation is performed by calling the function `run_md` with a certain number of steps:

```python
ethanol_ase.run_md(1000)
```

During simulation, energies and geometries are logged to `simulation.log` and `simulation.traj`, respectively.

We can for example visualize the evolution of the systems total and potential energies as

```python
# Load logged results
results = np.loadtxt(os.path.join(ase_dir, 'simulation.log'), skiprows=1)

# Determine time axis
time = results[:,0]

# Load energies
energy_tot = results[:,1]
energy_pot = results[:,2]
energy_kin = results[:,3]

# Construct figure
plt.figure(figsize=(14,6))

# Plot energies
plt.subplot(2,1,1)
plt.plot(time, energy_tot, label='Total energy')
plt.plot(time, energy_pot, label='Potential energy')
plt.ylabel('E [eV]')
plt.legend()

plt.subplot(2,1,2)
plt.plot(time, energy_kin, label='Kinetic energy')
plt.ylabel('E [eV]')
plt.xlabel('Time [ps]')
plt.legend()

temperature = results[:,4]
print('Average temperature: {:10.2f} K'.format(np.mean(temperature)))

plt.show()
```

As can be seen, the potential and kinetic energies fluctuate, while the total energy (sum of potential and kinetic energy) remains approximately constant.
This is a good demonstration for the energy conservation obtained by modeling forces as energy derivatives.
Unfortunately, this also means that energy conservation is not a sufficient measure for the quality of the potential.

However, frequently one is interested in simulations where the system is coupled to an external heat bath.
This is the same as saying that we wish to keep the average kinetic energy of the system and hence temperature close a certain value.
Currently the average temperature only depends on the random velocities drawn during the initialization of the dynamics.
Keeping a constant temperature average be achived by using a so-called thermostat.
In the `AseInterface`, simulations with a thermostat (to be precise a Langevin thermostat) can be carried out by providing the `temp_bath` keyword.
A simulation with e.g. the target temperature of 300K is performed via:

```python
ethanol_ase.init_md(
    'simulation_300K',
    temp_bath=300,
    reset=True
)
ethanol_ase.run_md(20000)
```

We can now once again plot total and potential energies.
Instead of the kinetic energy, we now plot the temperature (both quantities are directly related).

```python
# Load logged results
results = np.loadtxt(os.path.join(ase_dir, 'simulation_300K.log'), skiprows=1)

# Determine time axis
time = results[:,0]
#0.02585
# Load energies
energy_tot = results[:,1]
energy_pot = results[:,2]

# Construct figure
plt.figure(figsize=(14,6))

# Plot energies
plt.subplot(2,1,1)
plt.plot(time, energy_tot, label='Total energy')
plt.plot(time, energy_pot, label='Potential energy')
plt.ylabel('Energies [eV]')
plt.legend()

# Plot Temperature
temperature = results[:,4]

# Compute average temperature
print('Average temperature: {:10.2f} K'.format(np.mean(temperature)))

plt.subplot(2,1,2)
plt.plot(time, temperature, label='Simulation')
plt.ylabel('Temperature [K]')
plt.xlabel('Time [ps]')
plt.plot(time, np.ones_like(temperature)*300, label='Target')
plt.legend()
plt.show()
```

Since our molecule is now subjected to external influences via the thermostat the total energy is no longer conserved.
However, the simulation temperature now fluctuates near to the requested 300K.
This can also be seen by computing the temperature average over time, which is now close to the desired value in contrast to the previous simulation.

## Summary

In this tutorial, we have trained a SchNet model on energies and forces using the MD17 ethanol dataset as an example.
We have then evaluated the performance of the model and performed geometry optimisation, normal mode analysis and basic molecular dynamic simulations using the SchNetPack ASE interface.

While these simulations can already be useful for practical applications, SchNetPack also comes with its own molecular dynamics package.
This package makes it possible to run efficient simulations on GPU and also offers access to advanced techniques, such as ring polymer dynamics.
In the next tutorial, we will cover how to perform molecular dynamics simulations directly with SchNetPack.


```
#!/usr/bin/python

import petsc4py
import sys

petsc4py.init(sys.argv)

from petsc4py import PETSc
from dolfin import *
from dolfin import plot as dplot

import mshr
Print = PETSc.Sys.Print
# from MatrixOperations import *
import numpy as np
import PETScIO as IO
import common
import scipy
import scipy.io
import time

import BiLinear as forms
import IterOperations as Iter
import MatrixOperations as MO
import CheckPetsc4py as CP
import ExactSol
import Solver as S
import NSprecondSetup
import memory_profiler
import gc
import sympy as sy
#@profile
m = 2

set_log_active(False)
errL2u =np.zeros((m-1,1))
errH1u =np.zeros((m-1,1))

l2uorder =  np.zeros((m-1,1))
H1uorder =np.zeros((m-1,1))

NN = np.zeros((m-1,1))
Vdim = np.zeros((m-1,1))
level = np.zeros((m-1,1))

nn = 2

dim = 2
ShowResultPlots = 'yes'
split = 'Linear'

for xx in xrange(1,m):
    print xx
    # parameters["form_compiler"]["quadrature_degree"] = -1

    level[xx-1] = xx + 2
    nn = 2**(level[xx-1])

    # Create mesh and define function space
    nn = int(nn)
    NN[xx-1] = nn/2

    # domain = mshr.Rectangle(Point(-1., -1.), Point(1., 1.)) - mshr.Rectangle(Point(0., -1.), Point(1., 0.) )
    # mesh = mshr.generate_mesh(domain, nn)

        # set_log_level(WARNING)
    mesh = RectangleMesh(-1,-1,1,1,nn,nn, 'left')
    cell_f = CellFunction('size_t', mesh, 0)
    for cell in cells(mesh):
        v = cell.get_vertex_coordinates()
        y = v[np.arange(0,6,2)]
        x = v[np.arange(1,6,2)]
        xone = np.ones(3)
        xone[x > 0] = 0
        yone = np.ones(3)
        yone[y < 0] = 0
        if np.sum(xone)+ np.sum(yone)>5.5:
            cell_f[cell] = 1
    mesh = SubMesh(mesh, cell_f, 0)

    cell_markers = CellFunction("bool", mesh)
    cell_markers.set_all(False)
    origin = Point(0., 0.)
    for cell in cells(mesh):
        p = cell.midpoint()
        if abs(p.distance(origin)) < 0.6:
            cell_markers[cell] = True

    mesh = refine(mesh, cell_markers)

    cell_markers = CellFunction("bool", mesh)
    cell_markers.set_all(False)
    origin = Point(0., 0.)
    for cell in cells(mesh):
        p = cell.midpoint()
        if abs(p.distance(origin)) < 0.4:
            cell_markers[cell] = True

    mesh = refine(mesh, cell_markers)

    cell_markers = CellFunction("bool", mesh)
    cell_markers.set_all(False)
    origin = Point(0., 0.)
    for cell in cells(mesh):
        p = cell.midpoint()
        if abs(p.distance(origin)) < 0.2:
            cell_markers[cell] = True

    mesh = refine(mesh, cell_markers)

    class Left(SubDomain):
        def inside(self, x, on_boundary):
            return near(x[0], -1.0)

    class Right(SubDomain):
        def inside(self, x, on_boundary):
            return near(x[0], 1.0)

    class Bottom(SubDomain):
        def inside(self, x, on_boundary):
            return near(x[1], -1.0)

    class Top(SubDomain):
        def inside(self, x, on_boundary):
            return near(x[1], 1.0)

    class CornerTop(SubDomain):
        def inside(self, x, on_boundary):
            return near(x[1], 0.0)

    class CornerLeft(SubDomain):
        def inside(self, x, on_boundary):
            return near(x[0], 0.0)

    left = Left()
    top = Top()
    right = Right()
    bottom = Bottom()
    cleft = CornerLeft()
    ctop = CornerTop()

    # Initialize mesh function for the domain
    domains = CellFunction("size_t", mesh)
    domains.set_all(0)

    # Initialize mesh function for boundary domains
    boundaries = FacetFunction("size_t", mesh)
    boundaries.set_all(0)

    left.mark(boundaries, 1)
    top.mark(boundaries, 1)
    bottom.mark(boundaries, 1)
    right.mark(boundaries, 1)

    cleft.mark(boundaries, 2)
    ctop.mark(boundaries, 2)

    order = 1
    parameters['reorder_dofs_serial'] = False
    V = VectorFunctionSpace(mesh, "CG", order)
    Vdim[xx-1] = V.dim()

    phi = sy.symbols('x[1]')
    rho = sy.symbols('x[0]')
    u =  pow(rho,2./3)*sy.sin(2*phi/3)
    v =  pow(rho,2./3)*sy.sin(2*phi/3)
    u = Expression((sy.ccode(u),sy.ccode(v)))
    # v = Expression(sy.ccode(v))

    class u0(Expression):
        def __init__(self, mesh, u, v):
            self.mesh = mesh
            self.u = u
            self.v = v
        def eval_cell(self, values, x, ufc_cell):
            if abs(x[0]) < 1e-8 and abs(x[1]) < 1e-8:
                values[0] = 0.0
                values[1] = 0.0

            else:
                r = sqrt(x[0]**2 + x[1]**2)
                theta = np.arctan2(x[1],x[0])
                # print theta
                if theta < 0:
                    theta += 2*np.pi
                values[0] = self.u(r, theta)[0]
                values[1] = self.u(r, theta)[1]
        def value_shape(self):
            return (2,)

    u0 = u0(mesh, u, v)

    b = TrialFunction(V)
    c = TestFunction(V)
    a = inner(grad(b), grad(c))*dx
    f = Expression(("0.0","0.0"))
    L = inner(f,c)*dx

    def boundary(x, on_boundary):
        return on_boundary
    bc1 = DirichletBC(V, u0, boundaries, 21)
    bc2 = DirichletBC(V, f, boundaries, 2)
    bc = [bc1, bc2]
    A, b = assemble_system(a, L, bc)
    A, b = CP.Assemble(A, b)
    u = b.duplicate()

    ksp = PETSc.KSP()
    ksp.create(comm=PETSc.COMM_WORLD)
    pc = ksp.getPC()
    ksp.setType('cg')
    pc.setType('gamg')
    ksp.setTolerances(1e-8)
    OptDB = PETSc.Options()
    # OptDB['pc_factor_mat_solver_package']  = "mumps"
    # OptDB['pc_factor_mat_ordering_type']  = "rcm"
    ksp.setFromOptions()
    scale = b.norm()
    b = b/scale
    ksp.setOperators(A,A)
    del A
    ksp.solve(b,u)
    # Mits +=dodim
    u = u*scale
    # parameters["form_compiler"]["quadrature_degree"] = 10

    Ve = VectorFunctionSpace(mesh,"CG",order+3)
    ue = interpolate(u0, Ve)

    ua = Function(V)
    ua.vector()[:] = u.array
    ErrorU = ua-ue
    errL2u[xx-1] = sqrt(abs(assemble(inner(ErrorU, ErrorU)*dx)))
    errH1u[xx-1] = sqrt(abs(assemble(inner(grad(ErrorU), grad(ErrorU))*dx)))

    # errL2u[xx-1] = errornorm(ue, ua, norm_type='L2', degree_rise=8)
    # errH1u[xx-1] = errornorm(ue, ua, norm_type='H10', degree_rise=8)

    if xx > 1:
       l2uorder[xx-1] =  np.abs(np.log2(errL2u[xx-2]/errL2u[xx-1])/np.log2(sqrt(float(Vdim[xx-1][0])/Vdim[xx-2][0])))
       H1uorder[xx-1] =  np.abs(np.log2(errH1u[xx-2]/errH1u[xx-1])/np.log2(sqrt(float(Vdim[xx-1][0])/Vdim[xx-2][0])))

import pandas as pd

# dplot(ua, interactive=True)

LatexTitles = ["l","DoFu","V-L2","L2-order","V-H1","H1-order"]
LatexValues = np.concatenate((level,Vdim,errL2u,l2uorder,errH1u,H1uorder), axis=1)
LatexTable = pd.DataFrame(LatexValues, columns = LatexTitles)
pd.set_option('precision',3)
LatexTable = MO.PandasFormat(LatexTable,"V-L2","%2.4e")
LatexTable = MO.PandasFormat(LatexTable,'V-H1',"%2.4e")
LatexTable = MO.PandasFormat(LatexTable,"H1-order","%1.2f")
LatexTable = MO.PandasFormat(LatexTable,'L2-order',"%1.2f")
print LatexTable

```

```

```

```
plot(mesh)
```

    <dolfin.cpp.io.VTKPlotter; proxy of <Swig Object of type 'std::shared_ptr< dolfin::VTKPlotter > *' at 0x105d8dde0> >

```

```


# Script 1

Based on https://www.geekstips.com/battery-life-calculator-sleep-mode/.

This version does a more liberal calculation.

[Github Backup](https://github.com/jmBSU/maskMisc/blob/main/batterylife.ipynb)

---
## Motivation
This was originally on Matlab but to change values and see the results would require you to download it and run it. It was not worth the extra hassle. Thus, it is here instead.

---
## Warning Popup
Just click "run anyway". For some odd reason when I was copying from my test environment, it thought that it was malicious.

---
## Current Values
If this works, this should plot the battery life assuming the following:

* Battery: 2 AAA in Series for 3.0 to 3.2V nominal
  * Rated for 1200 mAh total
* Sleep cycle consumes 18mA
* Wake cycle consumes 30mA
* 2 minute uptime followed by 8 minute sleep
* Calculation includes a derate of 15% on the battery

```python
%matplotlib inline

import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
import numpy as np
import sympy as sym

#Batter Capacity in mAh
batteryCap = 1200;

#Power draw in mA
wakePower = 30;
sleepPower = 18;

#wake and sleep time in Minutes
wakeTime = 2;
sleepTime = 8;

#For loop for 0 to 70 hours
x = [];

for i in range(71*60):
  x.append(i/60)

#Calculates y for each x manually
yarray = [];

timeCounter = 0;
timeMax = wakeTime + sleepTime;
y = batteryCap * 0.85 * 60;
yarray.append(y/60);

for i in range(71*60):
  #ignores 1st data entry
  if i != 0:
    #y must be positive - makes all <0 to =0
    if y > 0:
      #Clears internal count
      if timeCounter == timeMax:
        timeCounter = 0;
      #Assumes sleep first - starts decrementing per minute
      if timeCounter < sleepTime:
        y -= sleepPower;
        yarray.append(y/60);
      #This is for when it is awake - aka the remainder of timeMax
      elif timeCounter < timeMax:
        y -= wakePower;
        yarray.append(y/60);
      timeCounter+=1;
    else:
      yarray.append(0)

#Makes image big and good lookin - 4x3 at 300dpi
figure(figsize=(4,3), dpi=300)

#Plots a linear function of battery
plt.plot(x,yarray);
plt.xlabel("Time (hrs)");
plt.ylabel("mAh");
plt.title("Battery Life");
plt.grid();

#Finds the first 0, the index indicates the minute it is dead
hourDead = yarray.index(0)/60;
dayDead = hourDead/24;

print("Hours till battery is dead:","%0.2f" % hourDead,"or", "%0.2f" % dayDead, "days");

```

# Script 2 (Old)

This bottom script was the initial version where it plots the battery life via a linear equation. The main issue with this code was that it was scaled by the hour. This meant that uptime and sleeptime must add up and be repeatable to the hour.

i.e:
* 2 min up + 8 min sleep = 10 min total
  * 10 min * 6 runs in an hour = 60 minutes

Thus 12 min up in a hour, and 48 min sleep in an hour.

This converts to 0.2 uptime in an hour and 0.8 sleep in an hour.

These decimals are fed into the code and plotted.

```python
%matplotlib inline

import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
import numpy as np
import sympy as sym

#Batter Capacity in mAh
batteryCap = 1200;

#Power draw in mA
wakePower = 30;
sleepPower = 18;

#Fraction on a per hour basis
wakeTime = 0.2;
sleepTime = 0.8;

x = [];

#For loop for 0 to 70 hours
for i in range(71):
  x.append(i)

#Makes image big and good lookin - 4x3 at 300dpi
figure(figsize=(4,3), dpi=300)

#Plots a linear function of battery life
x = np.asarray(x);
plt.plot(x,batteryCap*0.85- sleepPower*sleepTime*x-wakePower*wakeTime*x);
plt.xlabel("Time (hr)");
plt.ylabel("mAh");
plt.title("Battery Life");
plt.grid();
plt.axis([0,70,0,1200]);

#Solves the hour and day battery is dead.
sym.var('hour');
eqn = sym.Eq(0,batteryCap*0.85- sleepPower*sleepTime*hour-wakePower*wakeTime*hour);
hourDead = sym.solve(eqn,hour);
dayDead = hourDead[0] / 24;
print("Hours till battery is dead:","%0.2f" % hourDead[0],"or", "%0.2f" % dayDead, "days");

```


```python
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
```

```python
import numpy as np
import numpy.linalg as la
```

# Dimension Reduction

```python
np.random.seed(123)
np.set_printoptions(3)
```

### PCA from scratch

Principal Components Analysis (PCA) basically means to find and rank all the eigenvalues and eigenvectors of a covariance matrix. This is useful because high-dimensional data (with $p$ features) may have nearly all their variation in a small number of dimensions $k$, i.e. in the subspace spanned by the eigenvectors of the covariance matrix that have the $k$ largest eigenvalues. If we project the original data into this subspace, we can have a dimension reduction (from $p$ to $k$) with hopefully little loss of information.

For zero-centered vectors,

\begin{align}
\text{Cov}(X, Y) &= \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1} \\
  &= \frac{\sum_{i=1}^nX_iY_i}{n-1} \\
  &= \frac{XY^T}{n-1}
\end{align}

and so the covariance matrix for a data set X that has zero mean in each feature vector is just $XX^T/(n-1)$.

In other words, we can also get the eigendecomposition of the covariance matrix from the positive semi-definite matrix $XX^T$.

We will take advantage of this when we cover the SVD later in the course.

Note: Here we are using a matrix of **row** vectors

```python
n = 100
x1, x2, x3 = np.random.normal(0, 10, (3, n))
x4 = x1 + np.random.normal(size=x1.shape)
x5 = (x1 + x2)/2 + np.random.normal(size=x1.shape)
x6 = (x1 + x2 + x3)/3 + np.random.normal(size=x1.shape)
```

#### For PCA calculations, each column is an observation

```python
xs = np.c_[x1, x2, x3, x4, x5, x6].T
```

```python
xs[:, :10]
```

    array([[-10.856,   9.973,   2.83 , -15.063,  -5.786,  16.514, -24.267,
             -4.289,  12.659,  -8.667],
           [  6.421, -19.779,   7.123,  25.983,  -0.246,   0.341,   1.795,
            -18.62 ,   4.261, -16.054],
           [  7.033,  -5.981,  22.007,   6.883,  -0.063,  -2.067,  -0.865,
             -9.153,  -0.952,   2.787],
           [-10.091,   9.144,   2.171, -14.452,  -5.93 ,  17.831, -24.971,
             -3.539,  13.002,  -8.794],
           [ -0.684,  -5.433,   4.485,   4.151,  -3.025,   9.405, -12.987,
            -12.12 ,   8.496, -11.511],
           [  1.618,  -5.193,  10.388,   6.864,  -0.771,   6.267,  -8.769,
            -11.222,   3.621,  -7.55 ]])

#### Center each observation

```python
xc = xs - np.mean(xs, 1)[:, np.newaxis]
```

```python
xc[:, :10]
```

    array([[-1.113e+01,  9.702e+00,  2.559e+00, -1.533e+01, -6.057e+00,
             1.624e+01, -2.454e+01, -4.560e+00,  1.239e+01, -8.938e+00],
           [ 6.616e+00, -1.958e+01,  7.318e+00,  2.618e+01, -5.090e-02,
             5.368e-01,  1.991e+00, -1.842e+01,  4.457e+00, -1.586e+01],
           [ 7.984e+00, -5.030e+00,  2.296e+01,  7.834e+00,  8.882e-01,
            -1.115e+00,  8.609e-02, -8.202e+00, -7.123e-04,  3.738e+00],
           [-1.023e+01,  9.007e+00,  2.033e+00, -1.459e+01, -6.068e+00,
             1.769e+01, -2.511e+01, -3.676e+00,  1.286e+01, -8.931e+00],
           [-7.496e-01, -5.498e+00,  4.419e+00,  4.085e+00, -3.091e+00,
             9.339e+00, -1.305e+01, -1.219e+01,  8.431e+00, -1.158e+01],
           [ 1.801e+00, -5.009e+00,  1.057e+01,  7.048e+00, -5.873e-01,
             6.451e+00, -8.585e+00, -1.104e+01,  3.805e+00, -7.366e+00]])

#### Covariance

Remember the formula for covariance

$$
\text{Cov}(X, Y) = \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1}
$$

where $\text{Cov}(X, X)$ is the sample variance of $X$.

```python
cov = (xc @ xc.T)/(n-1)
```

```python
cov
```

    array([[128.578,  -2.133,  -6.661, 127.395,  63.27 ,  42.369],
           [ -2.133,  95.051,   2.269,  -1.349,  44.885,  31.637],
           [ -6.661,   2.269,  94.934,  -5.048,  -1.141,  30.275],
           [127.395,  -1.349,  -5.048, 126.993,  63.196,  42.723],
           [ 63.27 ,  44.885,  -1.141,  63.196,  54.408,  36.838],
           [ 42.369,  31.637,  30.275,  42.723,  36.838,  36.563]])

#### Check

```python
np.cov(xs)
```

    array([[128.578,  -2.133,  -6.661, 127.395,  63.27 ,  42.369],
           [ -2.133,  95.051,   2.269,  -1.349,  44.885,  31.637],
           [ -6.661,   2.269,  94.934,  -5.048,  -1.141,  30.275],
           [127.395,  -1.349,  -5.048, 126.993,  63.196,  42.723],
           [ 63.27 ,  44.885,  -1.141,  63.196,  54.408,  36.838],
           [ 42.369,  31.637,  30.275,  42.723,  36.838,  36.563]])

#### Eigendecomposition

```python
e, v = la.eigh(cov)
```

```python
idx = np.argsort(e)[::-1]
```

```python
e = e[idx]
v = v[:, idx]
```

#### Explain the magnitude of the eigenvalues

Note that $x4, x5, x6$ are linear combinations of $x1, x2, x3$ with some added noise, and hence the last 3 eigenvalues are small.

```python
plt.stem(e)
pass
```

#### The eigenvalues and eigenvectors give a factorization of the covariance matrix

```python
v @ np.diag(e) @ v.T
```

    array([[128.578,  -2.133,  -6.661, 127.395,  63.27 ,  42.369],
           [ -2.133,  95.051,   2.269,  -1.349,  44.885,  31.637],
           [ -6.661,   2.269,  94.934,  -5.048,  -1.141,  30.275],
           [127.395,  -1.349,  -5.048, 126.993,  63.196,  42.723],
           [ 63.27 ,  44.885,  -1.141,  63.196,  54.408,  36.838],
           [ 42.369,  31.637,  30.275,  42.723,  36.838,  36.563]])

### Geometry of PCA

### Algebra of PCA

Note that $Q^T X$ results in a new data set that is uncorrelated.

```python
m = np.zeros(2)
s = np.array([[1, 0.8], [0.8, 1]])
x = np.random.multivariate_normal(m, s, n).T
```

```python
x.shape
```

    (2, 100)

#### Calculate covariance matrix from centered observations

```python
xc = (x - x.mean(1)[:, np.newaxis])
```

```python
cov = (xc @ xc.T)/(n-1)
```

```python
cov
```

    array([[1.014, 0.881],
           [0.881, 1.163]])

#### Find eigendecoposition

```python
e, v = la.eigh(cov)
idx = np.argsort(e)[::-1]
e = e[idx]
v = v[:, idx]
```

#### In original coordinates

```python
plt.scatter(x[0], x[1], alpha=0.5)
for e_, v_ in zip(e, v.T):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('x', fontsize=14)
plt.ylabel('y', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
pass
```

#### After change of basis

```python
yc = v.T @ xc
```

```python
plt.scatter(yc[0,:], yc[1,:], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
pass
```

#### Eigenvectors from PCA

```python
pca.components_
```

    array([[ 0.677,  0.736],
           [-0.736,  0.677]])

#### Eigenvalues from PCA

```python
pca.explained_variance_
```

    array([1.404, 0.452])

#### Explained variance

This is just a consequence of the invariance of the trace under change of basis. Since the original diagnonal entries in the covariance matrix are the variances of the featrues, the sum of the eigenvalues must also be the sum of the orignal varainces. In other words, the cumulateive proportion of the top $n$ eigenvaluee is the "explained variance" of the first $n$ principal components.

```python
e/e.sum()
```

    array([0.906, 0.094])

#### Check

```python
from sklearn.decomposition import PCA
```

```python
pca = PCA()
```

#### Note that the PCA from scikit-learn works with feature vectors, not observation vectors

```python
z = pca.fit_transform(x.T)
```

```python
pca.explained_variance_ratio_
```

    array([0.906, 0.094])

#### The principal components are identical to our home-brew version, up to a flip in direction of eigenvectors

```python
plt.scatter(z[:, 0], z[:, 1], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
pass
```

```python
plt.subplot(121)
plt.scatter(-z[:, 0], -z[:, 1], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
plt.title('Scikit-learn PCA (flipped)')

plt.subplot(122)
plt.scatter(yc[0,:], yc[1,:], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
plt.title('Homebrew PCA')
plt.tight_layout()
pass
```


# Latent Factor DBCM Analysis

This example highlights several advanced modeling strategies.

**Dynamic Binary Cascade Model**

The first is a Dynamic Binary Cascade Model (DBCM), which is the combination a Dynamic Count Mixture Model (DCMM) and a binary cascade (Berry et. al. 2020). When modeling sales in a retail setting, the DCMM (Berry and West, 2019) is used to model the number of transactions involving an item, and the binary cascade models the quantity purchased by each customer. The DCMM models the transactions as a mixture of a Bernoulli and Poisson DGLM:

\begin{equation} \label{eqn-dcmm}
z_t \sim Ber(\pi_t) \text{ and } y_t \mid z_t =
\begin{cases}
0, & \text{if } z_t = 0,\\
1 + x_t, \quad x_t \sim Po(\mu_t), & \text{if }z_t = 1
\end{cases}
\end{equation}

where $\pi_t$ and $\mu_t$ vary according to the dynamics of independent Bernoulli and Poisson DGLMs respectively:

\begin{equation*}
\text{logit}(\pi_t) = \mathbf{F}_{ber, t}^{'}\boldsymbol{\theta}_{ber, t} \qquad \text{and} \qquad \text{log}(\mu_t) = \mathbf{F}_{Po, t}^{'} \boldsymbol{\theta}_{Po, t}
\end{equation*}

A DCMM is useful to capture a higher prevalence of $0$ outcomes than in a standard Poisson DGLM.

Transactions are related to sales by modeling the number of units bought in each transaction. Recognizing that sales outliers often occur due to shoppers buying many units of an item, the probability of each quantity is modeled with a binary cascade. Let $n_{r,t}$ be the number of transactions with more than $r$ units. Then $n_{r,t} | n_{r-1, t} \sim Bin(n_{r-1,t}, \pi_{r,t})$ is defined by a binomial DGLM. This cascade of binomial DGLMs represents the sequence of conditional probabilities for purchasing $n$ units or greater, given that the shopper has bought $n-1$. This implies the following expression for sales $y_t$:

\begin{equation}\label{eqn-dbcm}
y_t =
\begin{cases}
0, & \text{if } z_t = 0,\\
\sum_{r=1:d} r(n_{r-1, t} - n_{r,t}) + e_t, & \text{if }z_t = 1,
\end{cases}
\end{equation}
where $d$ is the predefined length of the cascade and $e_t$ represents excess units greater than $d$. The cascade enables modeling of very small probabilities, in the rare cases when shoppers purchase large quantities of an item on a single grocery store trip.

**Latent Factors**

The second strategy in this example is the use of *latent factors* for multiscale modeling. This example features simulated data from a retail sales setting. There is data for an item, and also for total sales at the store. The total sales are smoother and more predictable than the item level sales. A model is fit to the total sales which includes a day-of-week seasonal effect. This seasonal effect is extracted from the model on total sales, and used as a predictor in the item level model.

Modeling effects at different levels of a hierarchy - aka multiscale modeling - can improve forecast accuracy, because the sales of an individual can be very noisy, making it difficult to learn accurate patterns.

**Copula Forecasting**

In this example we are focusing on forecasting retail sales *1* through *14* days into the future. To do this, we will simulate from the joint forecast distribution *1:14* days into the future. This is accomplished in an efficient manner through the use of a Copula, which accounts for dependence in the forecasts across days into the future.

```
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```
from pybats_nbdev.analysis import analysis, analysis_dbcm
from pybats_nbdev.latent_factor import seas_weekly_lf
from pybats_nbdev.shared import load_dbcm_latent_factor_example
from pybats_nbdev.plot import plot_data_forecast
from pybats_nbdev.point_forecast import median
from pybats_nbdev.loss_functions import MAD
```

We start by loading the simulated retail sales data.

The dataframe 'totaldata' has the total daily sales in a store, along with a predictor, which represents a measure of average price.

The dataframe 'data' is the sales of a single item, and has two predictors. 'X_transaction' is a measure of price, and will be used in as a predictor in the DCMM. 'X_cascade' is a binary indicator of whether the item is being promoted within the store, and is used as a predictor in the binary cascade within the DBCM.

```
data = load_dbcm_latent_factor_example()
totaldata, data = data.values()
totaldata['Y'] = np.log(totaldata['Y'] + 1)
```

```
totaldata.head()
```

      Y
      X

      2014-06-01
      5.746203
      0.793644

      2014-06-02
      5.375278
      -0.762627

      2014-06-03
      5.375278
      -0.752116

      2014-06-04
      5.613128
      -0.791505

      2014-06-05
      5.429346
      -0.496851

```
data.head()
```

      Sales
      Y_transaction
      X_transaction
      mt1
      mt2
      mt3
      mt4
      X_cascade
      excess

      2014-06-01
      7.0
      6.0
      0.376729
      1.0
      0.0
      0.0
      0.0
      0
      []

      2014-06-02
      3.0
      3.0
      1.797680
      0.0
      0.0
      0.0
      0.0
      1
      []

      2014-06-03
      9.0
      7.0
      -1.687661
      2.0
      0.0
      0.0
      0.0
      1
      []

      2014-06-04
      11.0
      10.0
      -0.188624
      1.0
      0.0
      0.0
      0.0
      0
      []

      2014-06-05
      5.0
      4.0
      0.684483
      1.0
      0.0
      0.0
      0.0
      1
      []

Here we define hyper parameters for the analysis:
-  *rho* is a discount factor which calibrates the forecast distribution. A value of *rho* less than $1$ increases the forecast variance.

- *k* is the number of days ahead to forecast.

- *nsamps* is the number of forecast samples to draw.

- *prior_length* is the number of days of data to use when defining the priors for model coefficients.

```
#Define hyper parameters
rho = .2  # rho must be greater than 0, and is typically < 1. A smaller rho widens the forecast distribution.
k = 14 # Number of days ahead that we will forecast
nsamps = 200
prior_length = 21
```

Next, we define the window of time that we want to forecast over. For each day within this time period, the model will sample from the path (joint) forecast distribution *1:k* days into the future.

```
# Define the forecast range
T = len(totaldata)
forecast_end_date = totaldata.index[-k]
forecast_start_date = forecast_end_date - pd.DateOffset(days=365)
```

The first analysis is run on the 'totaldata', which is used to learn a latent factor. In this case, we fit a normal DLM to the log of total sales in order to learn the weekly season effect, which is stored in the latent factor.

```
# Get multiscale signal (a latent factor) from higher level log-normal model
latent_factor = analysis(totaldata['Y'].values, totaldata['X'].values, k,
                         forecast_start_date, forecast_end_date, dates=totaldata.index,
                         seasPeriods=[7], seasHarmComponents=[[1,2,3]],
                         family="normal", ret=['new_latent_factors'], new_latent_factors= [seas_weekly_lf.copy()],
                         prior_length=prior_length)
```

    beginning forecasting

The second analysis is performed on 'data', which is the simulated sales of a single item. The latent factor is used to inform on the sales of this item. On each day, a copula is used to draw joint forecast samples from *1:k* days ahead.

```
# Update and forecast the model
mod, forecast_samples = analysis_dbcm(data['Y_transaction'].values.reshape(-1), data['X_transaction'].values.reshape(-1,1),
                                 data[['mt1', 'mt2', 'mt3', 'mt4']].values, data['X_cascade'].values.reshape(-1,1),
                                 data['excess'].values,
                                 forecast_start=forecast_start_date, forecast_end=forecast_end_date,
                                 prior_length=prior_length, k=k,
                                 nsamps=nsamps, rho=rho,
                                 latent_factor=latent_factor, dates = data.index,
                                 delregn_pois=.98)
```

    beginning forecasting

Finally, we can examine the results, first by plotting both the *1-* and *14-* step ahead forecasts:

```
forecast = median(forecast_samples)

horizon = 1
plot_length = 50
fig, ax = plt.subplots(figsize=(8,4))
start_date = forecast_end_date + pd.DateOffset(horizon - plot_length)
end_date = forecast_end_date + pd.DateOffset(horizon - 1)
ax = plot_data_forecast(fig, ax, data.loc[start_date:end_date].Sales,
                        forecast[-plot_length:,horizon - 1],
                        forecast_samples[:,-plot_length:,horizon - 1],
                        data.loc[start_date:end_date].index,
                        linewidth = 2)
```

```
horizon = 14
plot_length = 50
fig, ax = plt.subplots(figsize=(8,4))
start_date = forecast_end_date + pd.DateOffset(horizon - plot_length)
end_date = forecast_end_date + pd.DateOffset(horizon - 1)
ax = plot_data_forecast(fig, ax, data.loc[start_date:end_date].Sales,
                        forecast[-plot_length:,horizon - 1],
                        forecast_samples[:,-plot_length:,horizon - 1],
                        data.loc[start_date:end_date].index,
                        linewidth = 2)
```

As well as looking at the mean absolute deviation (MAD) between the forecast median and the observations over the forecast horizons. Interestingly, there is only a small increase in the MAD for longer forecast horizons.

```
# Mean absolute deviation at increasing forecast horizons
horizons = list(range(1, k+1))
list(map(lambda k: MAD(data.loc[forecast_start_date + pd.DateOffset(k-1):forecast_end_date + pd.DateOffset(k-1)].Sales,
                      forecast[:,k-1]),
         horizons))
```

    [3.122950819672131,
     3.0724043715846996,
     3.098360655737705,
     3.0778688524590163,
     3.0614754098360657,
     3.133879781420765,
     3.0655737704918034,
     3.121584699453552,
     3.1147540983606556,
     3.0969945355191255,
     3.127049180327869,
     3.1243169398907105,
     3.1174863387978142,
     3.1256830601092895]

```

```

examples/DBCM Latent Factor Example.ipynb


# Terrorism detection and estimations
Terrorism is another interesting application of Machine Learning for crime detection and prevention. The topic is really vast therefore we're going to talk about some techniques and give some references for the details.

We'll divide this notebook into 2 parts:

1. Definition of terrorism
2. **Estimating terrorism behaviour over time**
3. **Defeating terrorism**

# Introduction
**Terrorism** is, in the broadest sense, the use of intentionally indiscriminate violence as a means to create terror among masses of people; or fear to achieve a financial, political, religious or ideological aim. (Wikipedia)

Terrorist groups usually differ for these peculiarities:
- the organisation (solitaire, family-based, supported by a larger network etc.)
- ethnical composition
- strategies to conceal
- ideology (rebellion, politics, religion orthodoxy etc.)

Besides, collecting useful data to process may not be a trivial task: some terrorists are really good at hiding their tracks. Differently from other crimes, it is unacceptable not to prevent a terrorits attack in our modern, democratic society. Therefore the task is threefold complex.

## Red Brigade Movement and Volterra formula

According to Wikipedia, the **Lotka–Volterra equations**, also known as the **predator–prey equations**, are a pair of *first-order nonlinear differential equations*, frequently used to describe the dynamics of biological systems in which two species interact, one as a *predator* and the other as *prey*. (Bold and italics are ours).

The solution for the Lotka-Volterra equations is usually the so-called logistic curve. \begin{equation}
f(x) = \frac{L}{1 + e^{-k(x - x_0)}}
\end{equation}

Where:
* e = the natural logarithm base (also known as Euler's number),
* $x_0$ = the x-value of the sigmoid's midpoint,
* L = the curve's maximum value, and
* k = the steepness of the curve.

Basically it is nothing but a dilatated and translated sigmoid function: \begin{equation}g(x) = \frac{1}{1+e^{-x}} \end{equation}

Mr Marchetti noticed how this formula could work and fit fairly well even in radically differents contexts such as the artistic vérve of William Shakespeare and Sandro Botticelli, the rate of automobile population in Italy and the cumulative number of murderers done by Red Brigades members.

```python
%matplotlib inline

from __future__ import print_function

# for jupyter use mainly
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

from math import exp
import numpy as np
import matplotlib.pyplot as plt

# The purpuose is to return a function that is then applied to a dataset.
# Unfortunately sklearn's LogisticRegression is indeed a classifier

# xmin and xmax are the minimum and the maximum X value;
# ticks lets you add the ticks to the graph
# plot lets you choose whether to plot the graph or not
def logistic_function(L, k, x_0, xmin, xmax, ticks=False, plot=True):
    logit = np.vectorize(lambda x: L/(1+exp(-k*(x-x_0))))
    if plot:
        X_test = np.linspace(xmin, xmax)
        y = logit(X_test)
        if ticks:
            %matplotlib inline
            plt.xticks(np.unique(np.ceil(X_test)), rotation='vertical')
        plt.plot(X_test, y, color='red')
    return logit

interact(logistic_function, L = (0.5,5.), k = (0.05, 1), x_0 = (-5.,5.),
             xmin=-20, xmax = +20)
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    <function __main__.logistic_function>

We want to estimate the number of attacks made by red brigadists during the "lead years" in Italy during the period 1971-1975 and we want to predict the **cumulative** number of attacks and the end of the activity of the terrorist group. By cumulative we mean the sum of the attacks happened from the first year of the activity until the current time.

| Year | No of attacks (cumulative) |
|------|----------------------------|
| 1971 | 5                          |
| 1972 | 11                         |
| 1973 | 19                         |
| 1974 | 34                         |
| 1975 | 69

Since the solution is: \begin{equation} f(t) = \frac{250}{1+e^{-0.75(t - 1976.1)}} \end{equation}

we can try the following prediction

```python
%matplotlib inline

# fix the parameters and get the function
regression = logistic_function(L=250, k=0.75, x_0 = 1976.1,
                  xmin=1965, xmax = +1985,
                  ticks=True, plot=True)

# our dataset lacks of the year 1976 in order to get an accurate prediction
dataset = np.array([
    [1971, 5],
    [1972, 11],
    [1973, 19],
    [1974, 34],
    [1975, 69]
])

# numpy's ravel "flattens" a vector of vectors into a single vector
# otherwise we would have dataset[...,0] = [[1971], [1972], ...]
features = dataset[...,0].ravel()
targets = dataset[...,1].ravel()

# plot the graph, black dots for the true values, blue dots for the predicted values
plt.plot(features, targets, 'ko')
plt.plot(features, regression(features), 'bo')
np.trunc(regression(features)), np.trunc(targets)
```

Good! Now we can estimate the number of attacks in any year we like. Let's consider 1976

```python
# transform 1976 into [1976], apply the function and truncate
# the result to the nearest integer
np.trunc(regression(np.array(1976)))
```

    120.0

And actually the total number of assassination attempts in 1976 is 122. The prediction looked really accurate.

Now let's consider the *number of death per year* caused by the Red Brigades. It's simply $H(t) = f(t) - f(t-1) \approx \frac{f(t) - f(t-\delta)}{\delta} = f'(t)$, by letting $\delta \rightarrow 0$

That is, we consider the derivative of the logistic function.
So, **the derivative of this function corresponds to the number of victims per year**.
Its full expression is:

$$\frac{kLe^{-k(x+x_0)}}{(1+e^{-k(x-x_0)})^2}$$

When $x = x_0$ we see that $f'$ reaches its global maximum. In fact the assassination attempts reached the highest score during the period 1976-1977. We also see a horizontal asymptote when $x\geq1985$ , when the Red Brigades halted their criminal activity.

Why did they stop? It isn't ours to understand the underlying reasons behind this curve, what we only have to care know is that the logistic model turned out predicting fairly well.

If you are curious try it yourself with different datasets like Cosa Nostra's carnages. Other ideas are Ku Klux Clan, OLP, Jack the ripper.

# Terrorism detection

The mostly known and widespread system is mass surveillance. Computer vision, speech recognition, natural language processing and of course Internet tracking are just a few common buzzwords you'd have probably heard about. Many of them are "general-purpuose" i.e. are used to prevent a wide spectrum of crimes. Here we're going to see specific techniques for terrorism detection.

# Sentiment analysis

Although the field of Sentiment Analysis is extremely vast and complex \[2\], the common starting point is data mining. Almost any research paper dedicated to Sentiment Analysis makes use of Twitter to get an uninterrupted flow of messages (the current rates can be read in (Azizan et al., 2017)), thanks to both its set of API \[5\] and satellite "cleansing services" to strip @tags, \#hashtags etc. Broadly speaking tweets are difficult to analyse whereas Facebook posts can be easier to tackle, both for humans and machines.

The correlation between SA and terrorism is proved by terrorists' tendence to diffuse proselytism of false, violent beliefs, thus intrinsecally full of hate words and expression.

Things are not so easy.
* Negative reviews, jokes, sarchasm and complain result as negative as terrorist text messages.
* Outrageous posts may be isolated cases and have no link to previous nefarious messages.

Effectively defeating online terrorism is a hard task when it comes to analysing single posts which may seem ambiguous for humans too.

**N.B. We're not going to be exhaustive.**

## Naïve Bayes Classification (basically your spam detector)

Given a list of words we want to give the sentence a class like "\{happy, sad, angry,...\}"; terrorist messages may have a tendency to be particolarly fearsome and angry besides having some keyword such as caliph and Allah.

Let us introduce some notation:

$P(C_i)$ is the probability (for our purpuoses the relative frequency may suffice) of a certain message whereas $P(w_i)$ is the probability of a certain word $w_i$. In order to calculate $P(w_i)$ we need a good distribution model. A lookup frequency table may be fine however different distribution functions can work better in other contexts.

Let $W = \{w_1, w_2, ..., w_n\}$

Thanks to Bayes theorem we can say that

$$ P(C_i|W) = \frac{P(W|C_i)P(C_i)}{P(W)}$$

The "naive" assumption is that all the words used in this sentence are indipendent from each other so $P(W) = P(w_1)P(w_2)\cdots P(w_n)$ so the problem is a piece of cake:

$$P(C_i | W) = \frac{P(W|C_i)P(C_i)}{P(W)} = \frac{P(w_1|C_i)P(w_2|C_i)\cdots P(w_n|C_i)\cdot P(C_i)}{P(w_1)P(w_2)\cdots P(w_n)}$$

Then iterate for any class and take for good the class with the highest probability (*Maximum A Posteriori*)

That method has lots of drawbacks. Messages like "I feel bad when it rains" and "new therapy overcomes depression" may result in the same emotion score. Besides, there's no way to establish any relationships between messages and contextualize tweets. The accuracy is really low. Of course we're excluding the noise coming from tags and mispelled words.

An improvement suggested in \[3\] is collecting all the tweets from a single person or a group of interesting people, cleaning them (stripping tags, correcting mispelled words etc.) and classifying them together. The training step must respect the posts cronology.

## Artificial Neural Networks: A Highway to Hell

Neural networks are now able to process natural language (see NLP). Determine the structure of the sentence is a big improvement and could be used in pair with Bayesian score, especially when it comes to studying not only certain words but patterns like pairs, triples and in general the finding of useful n-grams. Example: *not* + adjective is its opposite. If a sentence ends with an exclamation point has higher probability to be sarchastic etc.

What neural networks models can be used?

* **Multi-layer perceptron** or **MLP**: it can be able to take multiple features in account at the same time. However the classification does not consider the "history" of the user. The accuracy is usually quite low. Besides, training may be really slow and require a huge dataset in the majority of cases so as to reach a discrete  performance. Besides *there's no memory* (i.e. provided the same input over time the classification will always be the same). Although this property is not really related to sentiment analysis it may be desirable for terrorism detection.

* **Recurrent neural network** or **RNN**: it is a class of artificial neural network where connections between nodes form a directed graph along a sequence. This allows it to exhibit dynamic temporal behavior for a time sequence (Wikipedia). The key idea is the ability to fully track recurrent traits and, to some extent, implement a rudimental memory. The main drawbacks however are the training period - usually longer than in MLPs - and the memory lenght, usually not as long as we may need

* **Long short-term memory network** or **LSTM**: can implement an arbitrary long memory. The core idea is to have some special cells for **input**, **output**, **storage** and **forget** stages or "gates". The layers of the LSTM are "conventional" neural networks, even RNNs.

The topic is discussed in detail in \[6\] and \[7\]

## Lexicon-based approach: be a bookworm
The lexicon-based approach involves calculating orientation for a document from the semantic orientation of words or phrases in the document (Turney 2002). It works by basically creating a dictionary where each word is given a score. The classification of the text is then given by the total score. It may sound drastically simple but in many cases (e.g. no sarcasm) it works. Generally the technique but be preceded by input cleaning like reducing long, exaggerated expressions like "NOOOOOOOOOOOO" to "NO", fixing typos, stemming and so on. The basics of this process are well-documented in \[8\]

## CAPPS

Computer-Assisted Passenger Prescreening System is a counter-terrorism system for risk prevention in airports. It is active mainly in USA from 1990 (CAPPS I)and strengthened after 9/11 (CAPPS II). However a lot of positive and negative cases continued to arose and privacy became a big concern among the passengers. This brought the project to its end in 2004.

The core idea behind CAPPS is that terrorists (in general, malicious people tend to share a similar behaviour when buying airplane tickets and that it usually differs from ordinary travelers.

CAPPS detects a number of "features":

1. Anagraphical information: does the passenger have "troublesome" origins?
2. Historical information: has the passenger been previously a suspect or has it been selected and screened? What was the outcome last time?
3. Behavioural information: is the route unusual for the passenger (like going to Syria without ever going there before)? Did the passenger buy a single or a return ticket?

These data are then collected and sent to an "intelligence-based system for unknown terrorists detection". Although everybody undergoes a pre-screening test passengers with a high score may be required to undergo more intensive pre-screening tests.

The chart below should be pretty eloquent.

The mechanism behind CAPPS II is strictly confidential but we may argue that a Machine-Learning algorithm was applied.

In 2009 Secure Flight was proposed by the American Government. Theorically that replacement should be less invasive and shouldn't implement any machine learning algorithm, but in practise... who knows? PRISM scandal turned our beliefs about the American Government upside down. For further reference check \[9\] .

# References

\[1\]: Marchetti, C., On Time and Crime: A Quantitative Analysis of the Time Pattern of Social and Criminal Activities, IIASA WP-85-084 (1985)

\[2\]: https://scholar.google.it/scholar?hl=en&as_sdt=0%2C5&q=sentiment+analysis+terrorism&btnG= : about 106,000 results so far (Jun 2018)

\[3\]: Azizan, S. A., Aziz, I. A.: Terrorism Detection Based on Sentiment Analysis Using Machine Learning, Universiti Teknologi Petronas Bandar Seri Iskandar, Perak, Malaysia (2017)

\[4\]: Hassan, A., Korashy, H., Medhat, W.: Sentiment analysis algorithms and applications - a survey. Ain Shams Eng. J. 5, 1093–1113 (2014)

\[5\]: https://developer.twitter.com/en/docs/api-reference-index.html

\[6\]: Kouloumpis, E., Wilson, T., Moore, J.: Twitter Sentiment analysis: The Good the Bad and the OMG!, Fifth International aaai (2009)

\[7\]: Chung, J., Gulcehre, C., Cho, K., Bengio, Y. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

\[8\]: Palanisamy, P., Yadav, V., Elchuri, H.: Serendio: Simple and Practical lexicon based approach to Sentiment Analysis, Serendio Software Pvt Ltd, Guindy, Chennai 600032, India (2013)

\[9\]: https://www.globalsecurity.org/security/systems/secure-flight.htm


```python

import math
import random
from array import *
from math import gcd as bltin_gcd
from fractions import Fraction
import matplotlib.pyplot as plt

import numpy as np

from functools import reduce
inf = float("inf")

import sympy
from sympy import *
from sympy import pprint
from sympy import symbols, var

from sympy.interactive.printing import init_printing
init_printing(use_unicode=False, wrap_line=False, no_global=True)
from sympy.matrices import *
```

```python
# --------------------------------------------------------------
#  The following functions are contained in the PL library
#  we include them here so the document is self-contained
# --------------------------------------------------------------

# calculates M(x+iy)
# assumes x,y are Fraction types
def mob_transf_int(M,x,y):
    a = M[0,0]
    b = M[0,1]
    c = M[1,0]
    d = M[1,1]

    den = (c*x+d)**2 + (y*c)**2

    return ( Fraction((a*x+b)*(c*x+d)+y*y*a*c,den) , Fraction(y*a*(c*x+d) - y*c*(a*x+b),den)    )

def mob_transf(M, a):
    # Mobius transofrmation associated to matrix M, where
    # M has all type Fraction entries (rational)
    # a must be Fraction or string INF
    # a is assumed to be rational on x-axis (imaginary coord =0)
    # returns a Fraction or string INF if it sends a to oo

    a1=M[0,0].numerator
    b1=M[0,0].denominator

    a3=M[1,0].numerator
    b3=M[1,0].denominator

    if( a == "INF"):
        if (a3 == 0):
            return "INF"
        else:
            return Fraction(a1*b3, a3*b1)

    x=a.numerator
    y=a.denominator

    a4=M[1,1].numerator
    b4=M[1,1].denominator

    if (a3*b4*x + a4*b3*y) ==0:
        return "INF"

    a2=M[0,1].numerator
    b2=M[0,1].denominator

    p=(b3*b4*y)*(a1*b2*x + a2*b1*y)
    q=(b1*b2*y)*(a3*b4*x + a4*b3*y)

    return Fraction(p,q)

def pi_rotation(x,y):
    # Returns the matrix representing pi rotation about (x,y)
    # This can be calculated to be:
    #  ( -x   y^2+x^2 )
    #  ( -1     x    )
    # Matrix is in GL(2,Q) (det may not be 1). that's ok since we only use it as a mob transf
    # Coordinates of matrix are type Fraction, so assumes inputs are integers

    Rotation = np.matrix( [ (Fraction(-x),Fraction(y*y+x*x))  , (Fraction(-1), Fraction(x))] )

    return Rotation

def printQmtx(M):
    print('(' , M[0,0],'   ', M[0,1], ')')
    print('(' , M[1,0],'   ', M[1,1], ')')
    return
```

```python
# ************************************************************************************
#      Calculating contraction constants for 4/3 in the (1,4) (n,m) case
# ************************************************************************************

print('Rotation around (2,2):')
print(' ')
R = pi_rotation(2,2)  # rotation around (2,2)
printQmtx(R)
print(' ')
print('  0 -> ', mob_transf(R, 0))
print('  4 -> ', mob_transf(R, 4))
print('  (2,2) -> ', mob_transf_int(R,2,2))
print('  (0,2) -> ', mob_transf_int(R,0,2))
print('  (4,2) -> ', mob_transf_int(R,4,2))
print('       ')

print('=> marked pts of triangle 0,2,4 are   (2,2), (1,1) and (3,1).')
print(' ')
print(' ')

# --------------------------------------

print('Rotation around (3,1):')
print(' ')
A = pi_rotation(3,1)    # rotation around (3,1)
printQmtx(A)
print(' ')
print('  0 -> ', mob_transf(A, 0))
print('  2 -> ', mob_transf(A, 2))
print('  4 -> ', mob_transf(A, 4))
print('  (3,1) -> ', mob_transf_int(A,3,1))
print('  (2,2) ->', mob_transf_int(A,2,2))
print('  (1,1) -> ', mob_transf_int(A,1,1))
print('       ')

print('=> marked pts of triangle 4,2,10/3 are   (3,1), (16/5,2/5) and (17/5,1/5).')
print(' ')
print(' ')

# --------------------------------------

print('Rotation around (16/5,2/5):')
print('  ')
A = pi_rotation(Fraction(16,5),Fraction(2,5))   # rotation around (16/5,2/5)
printQmtx(A)
print('  ')
print('  2 -> ', mob_transf(A, 2))
print('  10/3 -> ', mob_transf(A, Fraction(10,3)))
print('  (16/5,2/5) -> ', mob_transf_int(A, Fraction(16,5),Fraction(2,5)))
print(' ' )
print('  ** sends 4 to ', mob_transf(A, 4), '** ')

```

    Rotation around (2,2):

    ( -2     8 )
    ( -1     2 )

      0 ->  4
      4 ->  0
      (2,2) ->  (Fraction(2, 1), Fraction(2, 1))
      (0,2) ->  (Fraction(3, 1), Fraction(1, 1))
      (4,2) ->  (Fraction(1, 1), Fraction(1, 1))

    => marked pts of triangle 0,2,4 are   (2,2), (1,1) and (3,1).

    Rotation around (3,1):

    ( -3     10 )
    ( -1     3 )

      0 ->  10/3
      2 ->  4
      4 ->  2
      (3,1) ->  (Fraction(3, 1), Fraction(1, 1))
      (2,2) -> (Fraction(16, 5), Fraction(2, 5))
      (1,1) ->  (Fraction(17, 5), Fraction(1, 5))

    => marked pts of triangle 4,2,10/3 are   (3,1), (16/5,2/5) and (17/5,1/5).

    Rotation around (16/5,2/5):

    ( -16/5     52/5 )
    ( -1     16/5 )

      2 ->  10/3
      10/3 ->  2
      (16/5,2/5) ->  (Fraction(16, 5), Fraction(2, 5))

      ** sends 4 to  3 **

```python

```