Contenido bajo licencia Creative Commons BY 4.0 y código bajo licencia MIT. © Juan Gómez y Nicolás Guarín-Zapata 2020. Este material es parte del curso Modelación Computacional en el programa de Ingeniería Civil de la Universidad EAFIT.

# Interpolación de Lagrange en 1D

## Introducción

El problema de interpolación o de predecir funciones a partir de un número determinado de datos representativos de la función aparece con bastante frecuencia en la interpretación de datos experimentales. De otro lado, las técnicas de aproximación de funciones por medio de métodos de interpolación son la base para la formulación de los métodos más importantes de la mecánica computacional como lo son los elementos finitos y los elementos de frontera. En este Notebook se discutirán algunos aspectos básicos y fundamentales de la teoría de interpolación. El notebook se describe en términos del desarrollo completo de un problema ejemplo incluyendo su implementación en Python.

**Al completar este notebook usted debería estar en la capacidad de:**

* Reconocer el problema de interpolación de funciones como uno de aproximación de una función desconocida en términos de valores discretos de la misma función.

* Identificar las propiedades fundamentales de los polinomios de interpolación de Lagrange.

* Usar polinomios de interpolación de Lagrange para proponer aproximaciones a una función dado un conjunto de $N$ valores conocidos de la misma.

* Reconocer la diferencia entre variables primarias y secundarias en un esquema de interpolación.

## Interpolación de Lagrange

El problema de interpolación consiste en la determinación del valor de una función $f(x)$ en un punto arbitrario $x \in [x_1, x_n]$, dados valores conocidos de la función al interior de un dominio de solución. De acuerdo con el teorema de interpolación de Lagrange la aproximación $\hat f(x)$ a la función $f(x)$ se construye como:

\begin{equation}
\hat{f}(x) = \sum_{I=1}^n L_I(x) f_I
\end{equation}

donde $L_I$ es el $I-$ésimo polinomio de orden $n-1$ y $f_1, f_2, \cdots, f_n$ son los $n$ valores conocidos de la función. El $I-$ésimo polinomio de Lagrange se calcula siguiendo la siguiente expresión:

\begin{equation}
{L_I}(x) = \prod_{J=1, J \ne I}^{n} \frac{(x - x_J)}{(x_I - x_J)}\, .
\end{equation}

### Primera derivada

En varios problemas de ingeniería es necesario usar técnicas de interpolación para encontrar valores de las derivadas de la variable primaria o principal. Por ejemplo, en problemas de mecánica de sólidos y teoría de elasticidad, en los cuales la variable primaria es el campo de desplazamientos, uno puede estar interesado en determinar las deformaciones unitarias las cuales son derivadas espaciales de los desplazamientos. Considerando que solo se dispone de valores discretos de los desplazamientos se tiene que las derivadas que se requieren se pueden encontrar usando estos valores discretos. Lo anterior equivale a derivar $\hat{f}(x)$ directamente como sigue:

\begin{equation}
\frac{\mathrm{d}\hat{f}}{\mathrm{d}x}=\frac{\mathrm{d}L_I(x)}{\mathrm{d}x}f_I
\end{equation}

Haciendo,

$$B_I(x) = \frac{\mathrm{d}L_I(x)}{\mathrm{d}x}\, ,$$

podemos escribir el esquema de interpolación como:

\begin{equation}
\frac{\mathrm{d}\hat{f}}{\mathrm{d}x} = \sum_{I=1}^n B_I(x) f_I\, .
\end{equation}

## Ejemplo

Formule un esquema de interpolación para encontrar un valore de la función

\begin{equation}
f(x) = x^3 + 4x^2 - 10
\end{equation}

en un punto arbitrario $x$ en el intervalo $[ -1, 1]$ asumiendo que se conoce el valor exacto de la función en los puntos $x=-1.0$, $x=+1.0$ y $x=0.0.$

En este ejemplo se conoce la función y aparentemente no tiene mucho sentido buscar una aproximación de la misma resolviendo un problema de interpolación. Sin embargo es conveniente seleccionar una función conocida para poder asimilar el problema numérico y sus limitaciones. En este contexto asumiremos que en una aplicación real conocemos los valores de la función en un conjunto de puntos $x=-1.0$, $x=+1.0$ and $x=0.0$ los cuales se denominan puntos nodales o simplemente _nodos_.

El proceso de interpolación involucra 2 pasos fundamentales:

1. Determinar los polinomios de interpolación $L_I$ usando la productoria.

2. Usar la combinación lineal para construir el polinomio interpolante o la aproximación a la función $\hat f(x)$.

Veamos estos pasos entonces.

1. Considerando que tenemos 3 puntos _nodales_ necesitamos generar 3 polinomios de interpolación de segundo orden, cada uno de ellos asociado a cada punto nodal. Rotulemos los _nodos_ como $x_0 = -1.0$, $x_1 = +1.0$ y $x_2 = 0.0$. De acuerdo con esta denominación $L_0(x)$, $L_1(x)$ y $L_2(x)$ serán los polinomios de interpolación de segundo orden asociados a los puntos nodales $x_0 = -1.0$, $x_1 = +1.0$ y $x_2 = 0.0$. Usando la fórmula de la productoria tenemos:

\begin{align}
&L_0(x) = \frac{(x - x_1)(x - x_2)}{(x_0 - x_1)(x_0 - x_2)} \equiv \frac{1}{2}(x - 1.0)x\\
&L_1(x) = \frac{(x-x^0)(x-x^2)}{(x^1-x^0)(x^1-x^2)}\equiv\frac12(x+1.0)x\\
&L_2(x) = \frac{(x-x^0)(x-x^1)}{(x^2-x^0)(x^2-x^1)}\equiv-(x+1.0)(x-1.0)\, .
\end{align}

2. Para llegar a la aproximación final de la función realizamos la combinación lineal:

\begin{equation}
\hat{f}(x) = L_0(x)f_0 + L_1(x)f_1 + L_2(x)f_2
\end{equation}

**Preguntas**

- Verificar que los polinomios de interpolación $L_0(x)$, $L_1(x)$ y $L_2(x)$ satisfacen la propiedad

$$L_I(x_J)= \delta_{IJ} \equiv \begin{cases}
1\quad \text{si } I=J\\
0\quad \text{si } I\neq J
\end{cases}\, .$$

- Si la condición $L_I(x_J)=\delta_{IJ}$ no se satisface por una de las funciones de interpolación encontradas, ¿cuál sería el efecto resultante en la aproximación de la función?

### Algunas notas de interés

* En el método de los elementos finitos es común denominar a los polinomios de interpolación como _funciones de forma_.

* El dominio de calculo es el intervalo de tamaño 2 comprendido entre $x=-1.0$ y $x=+1.0$. Como se discutirá mas adelante, por facilidades en la programación en la implementación de métodos de elementos finitos los dominios de tamaño diferente son transformados al dominio de tamaño 2.

## Solución en Python

En los siguientes bloques de código mostramos el paso a paso en la construcción del polinomio de interpolación final $f(x)$ usando Python. Para seguir el Notebook se recomienda que de manera simultánea se implementen los bloques de código en un script independiente o que añada comentarios a las instrucciones más relevantes.

### Paso 1: Importación de módulos

En la escritura de scripts en Python es necesario importar **módulos** o bibliotecas (algunas personas usan la palabra _librería_ como mala traducción de _library_ del inglés) que contienen funciones de Python predefinidas. En este caso importaremos los **módulos**:

* `numpy` el cual es una biblioteca de funciones para realizar operaciones con matrices similar a Matlab.
* `matplotlib` el cual es una biblioteca de graficación.
* `scipy` el cual es una biblioteca fundamental para computación científica.
* `sympy` el cual es una biblioteca para realizar matemáticas simbólicas.

Python importa los módulos usando la palabra reservada `import` seguida del nombre del módulo y un nombre corto para ser usado como prefijo en referencias posteriores a las funciones contenidas en ese módulo.

```python
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import interpolate
import sympy as sym
```

```python
sym.init_printing()
```

### Paso 2: Definición de una función para determinar los polinomios de interpolación de Lagrange

En un programa de computador una **función** (o también llamada **subrutina**) es un bloque de código que realiza una tarea específica múltiples veces dentro de un programa o probablemente en diferentes programas. En Python estas funciones se definen por medio de la palabra clave `def` seguida del nombre dado a la función.

Conjuntamente con el nombre, encerrado entre paréntesis, la definición de la función debe incluir también una lista de parámetros (o argumentos) a ser usados por la función cuando esta realice tareas especificas.

En el ejemplo definiremos una función de Python para generar el polinomio de Lagrange usando la productoria definida previamente. Le daremos a esta función el nombre `lagrange_poly`. Sus parámetros de entrada serán la variable independiente $x$ a ser usada en el polinomio; el orden del polinomio definido por `order`; y el punto `i` asociado al polinomio. La función será usada posteriormente desde el programa principal.

```python
def lagrange_poly(x, order, i, xi=None):
    if xi == None:
        xi = sym.symbols('x:%d'%(order+1))
    index = list(range(order+1))
    index.pop(i)
    return sym.prod([(x - xi[j])/(xi[i] - xi[j]) for j in index])
```

Alternativamente a la definición de una función usando la palabra clave `def` Python también permite la definición de funciones en el sentido del Calculo, es decir definición de relaciones que permiten  transformar escalares o vectores en otros escalares o vectores. Esto es posible a través de la opción `lambda`. En este ejemplo usaremos la opción `lambda` para crear la función:

\begin{equation}
f(x)=x^3+4x^2-10.
\end{equation}

**Preguntas**

Use una terminal o un script independiente para probar el uso de la opción `lambda` en la definición de una función. Intente con diferentes funciones.

```python
fun = lambda x: x**3 + 4.0*x**2 - 10.0
```

Una vez creada la función $f(x)$ usando la opción `lambda` pasamos a definir un conjunto de puntos de evaluación. El numero de puntos se define por medio de la variable `npts` y usamos la función `linspace` del módulo `numpy` para crear una distribución equidistante de puntos entre $x = -1.0$ y $x = +1.0.$ Simultáneamente, el arreglo nulo `yy` almacenará los valores de la función en los puntos almacenados en `xx`.

Note que Python inicia desde la posición 0 el conteo de elementos en arreglos y otras estructuras de datos, de manera que empezamos a contar desde cero para mantener la consistencia con la implementación.

```python
npts = 200
x_pts = np.linspace(-1, 1, npts)
```

Con la función ahora disponible podemos calcular los valores (conocidos) listos para ser usados en el esquema de interpolación. Estos valores se almacenarán en el arreglo `fd()`. Para calcular cada valor de la función usamos el comando (ya disponible) `fx` correspondiente al nombre que usamos en la declaración de la función usando la opción `lambda`.

**Preguntas**

Intente usar nombres diferentes en la definición de la función usando la opción `lambda` y ensaye si su código aún funciona.

```python
pts = np.array([-1, 1, 0])
fd = fun(pts)
```

Ahora evalúe la función en los `npts` puntos y grafique la misma:

```python
plt.figure()
y_pts = fx(x_pts)
plt.plot(x_pts , y_pts)
plt.plot(pts, fd, 'ko')
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x7f982bdebbe0>]

### Paso 3: Encontrando los polinomios de interpolación de Lagrange

Calculemos ahora los polinomios de Lagrange invocando la función `lagrange_poly()`. Crearemos una lista vacía que llamaremos `pol` y cada que determinemos un nuevo polinomio lo adicionaremos a la lista usando para ello el método `.append()`. En ese momento los polinomios almacenados en la lista `pol=[]` existirán en notación simbólica (como si estuviéramos resolviendo el problema a mano) y estarán listos para ser evaluados en valores específicos de $x$.

```python
x = sym.symbols('x')
pol = []
pol.append(sym.simplify(lagrange_poly(x, 2, 0, [-1,1,0])))
pol.append(sym.simplify(lagrange_poly(x, 2, 1, [-1,1,0])))
pol.append(sym.simplify(lagrange_poly(x, 2, 2, [-1,1,0])))
pol
```

**Preguntas**

- Grafique los 3 polinomios de Lagrange de orden 2 y verifique que satisfacen la propiedad $L_I(x_J)=\delta_{IJ}$.

- Adicione más puntos al dominio de interpolación y grafique los polinomios resultantes. ¿Cuál es el efecto en los polinomios?

El método `subs` del módulo `sympy` substituye el valor de la variable independiente $x$ por el valor almacenado en $xx[i]$.

```python
plt.figure()

for k in range(3):
    for i in range(npts):
        yy[i] = pol[k].subs([(x, xx[i])])
    plt.plot(xx, yy)
```

    <IPython.core.display.Javascript object>

### Paso 4: Encontrando el polinomio de interpolación para aproximar la función  $f(x)$

Construyamos ahora el polinomio de aproximación completo $\hat f(x)$ de acuerdo con la superposición lineal

$$\hat{f}(x) = L_0(x)f(x_0) + L_1(x) f(x_1) +  L_2(x) f(x_2)\, ,$$

y utilizando la lista (ya disponible) `pol` que almacena los 3 polinomios generados. Solo para efectos de ilustrar primero lo mostramos con `display`. La versión evaluada de $\hat{f}(x)$ será almacenada en el arreglo `yy[i]`.

```python
display(pol[0]*fd[0] + pol[1]*fd[1] + pol[2]*fd[2])
```

```python
plt.figure()
for i in range(npts):
    yy[i] = fd[0]*pol[0].subs([(x, xx[i])]) + fd[1]*pol[1].subs([(x, xx[i])]) \
            + fd[2]*pol[2].subs([(x, xx[i])])

zz = fx(xx)
plt.plot([-1, 1, 0], fd, 'ko')
plt.plot(xx, yy , 'r--')
plt.plot(xx, zz)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x7f982bea99e8>]

Note que debido a la diferencia en orden entre la aproximación $\hat{f}(x)$ y la función $f(x)$  el polinomio de interpolación no coincide completamente con la función aunque si reproduce los valores correctos en los puntos nodales.

**Pregunta**

¿Cómo podemos mejorar la aproximación $\hat{f}(x)$  a la función $f(x)$?

### Paso 5: Encontrando las derivadas

Usemos la opción `lambda` una vez más para definir una nueva función `fdx`correspondiente a la primera derivada:

$$f'(x) = 3x^2 + 8x\, .$$

Las derivadas en los puntos nodales se almacenarán en el arreglo `fc`

```python
fdx = lambda x: 3*x**2+8.0*x
fc = np.array([fdx(-1.0), fdx(1.0) ,fdx(0.0)])
```

La interpolación de las derivadas se calculan de acuerdo con

\begin{equation}
\hat{f}'(x) = \frac{\mathrm{d}L_0(x)}{\mathrm{d}x} f_0 + \frac{\mathrm{d}L_1(x)}{\mathrm{d}x} f_1 + \frac{\mathrm{d}L_2(x)}{\mathrm{d}x} f_2\, .
\end{equation}

```python
dpol = []
dpol.append(sym.diff(pol[0],x))
dpol.append(sym.diff(pol[1],x))
dpol.append(sym.diff(pol[2],x))
display(dpol)
display(dpol[0]*fd[0] + dpol[1]*fd[1] + dpol[2]*fd[2])

plt.figure()
yy= fdx(xx)
plt.plot(xx, yy ,'r--')
plt.plot([-1, 1, 0], fc, 'ko')

for i in range(npts):
    yy[i] = fd[0]*dpol[0].subs([(x, xx[i])]) + fd[1]*dpol[1].subs([(x, xx[i])])\
            + fd[2]*dpol[2].subs([(x, xx[i])])
plt.plot(xx, yy)
```

## Glosario de términos

**Fórmula de productoria:** Expresión usada para calcular los polinomios de interpolación de Lagrange.

**Función de forma:** Nombre dado a un polinomio de interpolación en el contexto del método de los elementos finitos.

**Punto nodal:** (También nodo). Nombre dado a los puntos específicos donde se conoce el valor de una función y usado en la construcción del esquema de interpolación.

**Subrutina:** Bloque de código independiente que realiza una tarea de cómputo determinada dentro de un programa principal.

**Matriz de interpolación:** Arreglo unidimensional o bidimensional que almacena las funciones de forma en un esquema de interpolación determinado.

## Actividad para la Clase

El propósito de esta actividad es familiarizar al estudiante con el uso del método de interpolación de Lagrange en un contexto particular de la ingeniería, como el método de los elementos finitos. El método de elementos finitos utiliza dominios de interpolación de tamaño constante (llamados **elementos**) permitiendo el uso de funciones de interpolación fijos y al mismo tiempo favoreciendo la automatización en programas de computador. En una aplicación típica de elementos finitos los valores nodales de una función (por ejemplo el desplazamiento) son determinados tras resolver un sistema de ecuaciones algebraicas. Estos valores nodales son posteriormente usados para encontrar el desplazamiento a lo largo de todo el dominio del problema, y también para realizar cálculos posteriores y obtener información adicional del problema físico.

**Siga los pasos que se indican a continuación para implementar, en un notebook independiente, un esquema de interpolación típico del método de elementos finitos. El esquema de interpolación resultante será el correspondiente a un solo elemento**

* Asumiendo que un dominio de interpolación constante se define como $x \in [-1.0, +1.0]$ con puntos nodales correspondientes a $x_0 = -1.0$, $x_1 = +1.0$, $x_2 = 0.0$, $x_3 = -0.25$ y $x_4 = +0.25$ use la función `lagrange_poly()` para generar las funciones de interpolación asociadas a estos 5 puntos nodales. Presente los polinomios resultantes en una gráfica.

* Utilice las funciones de interpolación encontradas en el paso anterior, para definir una matriz $[N(x)]$, que se denominará en adelante **Matriz de interpolación**, de manera que la operación de interpolar se pueda realizar usando operaciones matriciales como:

  $$\hat{f}(x) = [N(x)] \{F\}\, ,$$

  y en la cual $F$ es un vector que almacena los valores nodales de la función.

* En su Notebook imprima la versión simbólica de la matriz de interpolación.

* En su Notebook grafique una función $f(x)$ (representando el desplazamiento de una barra) y su versión aproximada $\hat{f}(x)$  usando el esquema matricial desarrollado en el numeral anterior. El código también debe graficar la primera derivada de la función.

* Para realizar la interpolación en el Notebook programe una función o subrutina de Python que reciba como parámetros de entrada las coordenadas $x$ donde se desea calcular una aproximación de la función y el vector de valores nodales de la función de desplazamientos y entregue después de su ejecución el valor interpolado de la función.

## Formato del notebook

La siguiente celda cambia el formato del Notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


# Logit and Logistic of array values

This notebook illustrates the level of control and flexibility available in Julia functions.  The task is to evaluate the *logistic* function $(-\infty, \infty)\rightarrow(0,1)$

\begin{equation}
x \rightarrow \frac{1}{1 + e^{-x}}
\end{equation}

and its inverse, the *logit* or "log-odds" function $(0,1)\rightarrow(-\infty, \infty)$

\begin{equation}
p \rightarrow \log\left(\frac{p}{1-p}\right)
\end{equation}

on an array of numeric values.

The first priority is to evaluate these functions accurately and robustly.  This usually means watching for edge cases (e.g. very large positive or negative $x$ or values of $p$ that are close to zero or to one).

The second priority is evaluate them quickly and flexibly. When fitting a logistic regression model to very large data sets these functions may be evaluated hundreds of times on arrays with millions of elements.

In "vectorized" languages, such as [`R`](http://www.r-project.org) or [`Matlab/Octave`](http://octave.org) and, to some extent, [`Python`](http://python.org), the obvious choice is to work on vectors.  In fact, the language often hides the fact that vectorization is occuring.

## logit and logistic in R

The [`RCall`](https://github.com/JuliaStats/RCall.jl) for Julia starts an embedded R process and provides for two-way communication with it.  In a [`Jupyter`](http://jupyter.org) notebook like this a Julia string prepended with `R` is evaluated in the R process.  String delimiters are `"` or `"""`.  In the second case the string can span multiple lines and can contain `"` characters.

```julia
using RCall
```

```julia
R" logit <- function(p) log(p / (1 - p)) ";
```

Create a vector of 100,000,000 random values between 0 and 1 on which to evaluate `logit`.  This is done in `Julia` after setting the random number seed, to allow for reproducibility.

```julia
srand(1234321)
pvals = rand(100_000_000);
```

Copy the vector to the `R` process under the same name.

```julia
@rput pvals;
```

```julia
R""" print(system.time(xvals <- logit(pvals))) """;
```

       user  system elapsed
      5.412   0.164   5.576

The first few values of `pvals` and `xvals` are

```julia
R" print(str(list(pvals=pvals, xvals=xvals))) ";
```

    List of 2
     $ pvals: num [1:100000000] 0.0944 0.9366 0.2583 0.9309 0.5553 ...
     $ xvals: num [1:100000000] -2.261 2.693 -1.055 2.601 0.222 ...
    NULL

Similarly, a vectorized logistic function can be defined as

```julia
R"""
logistic <- function(x) 1 / (1 + exp(-x))
print(system.time(pvalsnew <- logistic(xvals)))
""";
```

       user  system elapsed
      3.472   0.156   3.631

```julia
R" all(pvals == pvalsnew) "; # check for 'round trip' identity
```

    RCall.RObject{RCall.LglSxp}
    [1] FALSE

The problem with the "round trip" check is that floating point arithmetic is not exact.  Numbers are represented in a finite precision.  `pvalsnew` is close to `pvals` but not exactly equal.

```julia
R" print(str(list(pvals=pvals, xvals=xvals, pvn=pvalsnew))) ";
```

    List of 3
     $ pvals: num [1:100000000] 0.0944 0.9366 0.2583 0.9309 0.5553 ...
     $ xvals: num [1:100000000] -2.261 2.693 -1.055 2.601 0.222 ...
     $ pvn  : num [1:100000000] 0.0944 0.9366 0.2583 0.9309 0.5553 ...
    NULL

`R` has an `all.equal` function that compares floating point values using a tolerance on the differences.

```julia
R" print(all.equal(pvals, pvalsnew)) ";
```

    [1] TRUE

## The problem with vectorization

Vectorized languages are wonderful environment when you begin programming because all the messy loop-related baggage is eliminated at the expense of some overhead.  The evaluation of `logistic(xvals)` is done in 5 stages

1. Allocate a vector, `t1`, of 100,000,000 doubles and loop over `x` writing `-x` into `t1`.
2. Allocate a vector, `t2`, of 100,000,000 doubles and loop over `t1` writing `exp(t1)` into `t2`.
3. Allocate a vector, `t3`, of 100,000,000 doubles and loop over `t2` writing `1 + t2` into `t3`.
4. Allocate a vector, `result`, of 100,000,000 doubles and loop over `t3` writing `1 / t3` into `result`.
5. Return `result`

Because R allows for missing data in any vector the scalar arithmetic is more complicated than just looping over the vector.  Every scalar addition in, e.g. `1 + t2` has a check on both addends to see if they are `NA`.  Furthermore, the "recycling rule" that cycles over the `1` operand while looping over the indices of `t2` is further logic implemented inside the loop.

Notice that there are 3 temporary vectors allocated and the `result` must also be allocated.  This storage must later be "garbage collected".

## Functional programming in Julia

The operations could be performed in exactly the same way in Julia. Currently some Julia arithmetic and math functions are vectorized and some aren't.  In future releases vectorization will need to be explicitly stated by appending a `.` to a function name or prepending a `.` to an operator.

```julia
vlogistic(x::Vector) = 1 ./ (1 .+ exp.(-x));
vlogit(p::Vector) = log.(p ./ (1 .- p))
xvals = vlogit(pvals)
show(xvals[1:5])
```

Check for approximate equality

```julia
pvals ≈ vlogistic(xvals)
```

The timings show that the Julia code is faster than the R functions but it still allocates a considerable amount of storage and uses time in garbage collection (gc).

```julia
@time vlogit(pvals);
```

```julia
@time vlogistic(xvals);
```

However, there is no need to allocate the intermediate values when operating on only one value.

```julia
logit(p) = log(p / (one(p) - p));
logistic(x) = inv(one(x) + exp(-x));
sxvals = logit.(pvals);
show(sxvals[1:5])
```

    [-2.2608,2.69298,-1.05468,2.60097,0.222041]

    WARNING: Method definition logit(Any) in module Main at In[1]:1 overwritten at In[3]:1.
    WARNING: Method definition logistic(Any) in module Main at In[1]:2 overwritten at In[3]:2.

```julia
@time logit.(pvals);
```

      2.740797 seconds (159 allocations: 762.948 MB, 2.27% gc time)

This type of evaluation is in the "functional programming" style where simple functions are mapped over arrays.  Julia allows for results to be pre-allocated as, e.g.

```julia
@time map!(logit, sxvals, pvals);
```

      2.631148 seconds (3.43 k allocations: 140.514 KB)

In this case there isn't much of a savings in time but there is a saving in the amount of storage allocated.  This becomes important when, e.g. fitting a generalized linear model or a generalized linear mixed model.

```julia
pvalsnew = similar(sxvals); @time map!(logistic, pvalsnew, sxvals);
```

      2.116498 seconds (4.67 k allocations: 197.003 KB)

helps with the amount of allocation but doesn't actually run substantially faster.

There is a way to make the evaluation of the `log` and `exp` functions slightly faster, which is to use the [`@fastmath`](http://docs.julialang.org/en/stable/manual/performance-tips.html?highlight=fastmath#performance-annotations) macro in the definitions of the scalar functions.

```julia
@fastmath flogit(p) = log(p / (one(p) + p))
function flogistic(x)
    expmx = @fastmath(exp(-x))
    inv(one(expmx) + expmx)
end
```

    flogistic (generic function with 1 method)

```julia
@time map!(flogit, sxvals, pvals);
```

      2.382810 seconds (5.07 k allocations: 220.258 KB)

```julia
@time map!(flogistic, pvalsnew, sxvals);
```

      1.978709 seconds (4.93 k allocations: 210.208 KB)

## Multiple threads

Some multithreading capability is available in v0.5.0 of Julia.  Later versions with enhance these capabilities.  Before starting this notebook I set the environment variable `JULIA_NUM_THREADS=4` as this is running on a 4-core processor.

It is easiest to use multiple threads on a simple loop.  Define a function that overwrites the values in one array with the logit or logistic of the values in another array.  By convention, a `!` is appended to the name of such a *mutating function*, which modifies one or more of its arguments.

```julia
function logit!(dest, src)
    length(dest) == length(src) || throw(DimensionMismatch())
    Threads.@threads for i in eachindex(dest, src)
        @inbounds dest[i] = flogit(src[i])
    end
    dest
end
```

    logit! (generic function with 1 method)

```julia
@time logit!(sxvals, pvals);
```

      2.281610 seconds (13.87 k allocations: 597.863 KB)

The scaling is very good with 4 threads being

```julia
1.66/0.571
```

times as fast as a single thread.


```python
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt
import numpy as np
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

# An Exercise:  2 x 3 x 2 Neural Network

For the purpose of mastering the concept of gradient descent and the backpropagation algorithm,  we will extend the previous examples by programming from scratch a neural network with 2 input layers, 3 hidden layers and 2 output layers. Again, we will be using simple variables to store the activations and derivatives instead of using vectors (next lecture). The purpose of writing the code in this simplified manner is to make the concepts as clear as possible, and show how they relate with the mathematical equations. __For that reason, code will deliberately be inefficient, and shouldn't be used for any serious application__.

The data and labels we will be using is the same as before:

```python
data = np.array([[ 1.2, 0.7],
                 [-0.3,-0.5],
                 [ 3.0, 0.1],
                 [-0.1,-1.0],
                 [-0.0, 1.1],
                 [ 2.1,-1.3],
                 [ 3.1,-1.8],
                 [ 1.1,-0.1],
                 [ 1.5,-2.2],
                 [ 4.0,-1.0]])
```

```python
labels = np.array([  1,
                    -1,
                     1,
                    -1,
                    -1,
                     1,
                    -1,
                     1,
                    -1,
                    -1])
```

Here is the illustration of the neural network we will be using:

Computing the forward pass of this network is very easy, and effectively not very different from the previous examples. This excercise will be reinforcing the skill of computing derivatives in a network in which there are more than one output. This brings slight differences in how we compute partial derivatives of the activations in the hidden layer.

### What changes with two output layers?

In the previous example, the last layer of the network comprised of a single neuron, feeding its activation into a cost function. The cost function computed the cost of a single data point by measuring the distance of the actual output of the network and the data point's label which was either `-1` or `1`.

Here we have two output neurons incorporating a sigmoid as an activation function whose outputs are $O_1$ and $O_2$. Due to this activation funtion, both outputs will output a value in range from 0 to 1.

Our activation function must be able to compute the cost from both outputs. Therefore, the loss/cost function interprets the outputs $O_1$. and $O_2$ as following:

* $O_1$ will output the probability that the label is `-1`
- $O_2$ will output the probability that the label is  `1`

Therefore, if a point has a label `-1`, the first output neuron should output a maximal probability with the value `1.0`, and the second output neuron should output a minimal probability with the value close to `0.0`. Respectivelly, if a point has a label `1`, the first neuron should output the minimal probability value `0.0`, and the second neuron should output the maximal probability with the value close to `1.0`. For this reason we will make a function that converts the real label to its probabilistic interpretation by means of two output neurons:

```python
def convert_label(label):
    if (label == -1):
        return (1,0)
    if (label == 1):
        return (0,1)
```

We can test this function:

```python
convert_label(-1)
```

    (1, 0)

```python
convert_label(1)
```

    (0, 1)

## Computing the backward pass

The first thing that we will compute are:
* The partial derivatives of the total cost in respect to the individual costs $C_1$ and $C_2$
* The partial derivatives of the individual costs $C_1$ and $C_2$ in respect to the outputs $O_1$ and $O_2$
* The partial derivatives of the outputs $O_1$ and $O_2$ in respect to the intermediate values $z_1$ and $z_2$.

\begin{align*}
\frac{\partial TC}{\partial C_1} &= 1; &\frac{\partial TC}{\partial C_2} &= 1; \\\\
\frac{\partial C_1}{\partial O_1} &= 2*(C_1-label[0]); &\frac{\partial C_2}{\partial O_2} &= 2*(C_2-label[1]); \\\\
\frac{\partial O_1}{\partial z_1} &= O_1*(1-O_1); &\frac{\partial O_2}{\partial z_2} &= O_2*(1-O_2);
\end{align*}

Once we have these partials, we can use chain rule to multiply them together. For that purpose we introduce new variables `dz1` and `dz2` to store the partial derivatives $\frac{\partial TC}{\partial z_1}$ and $\frac{\partial TC}{\partial z_2}$. This will help us to simplify the computation in the next steps.

\begin{align}
&dz_1 = \frac{\partial TC}{\partial z_1} = 2*(C_1-label[0])*O_1*(1-O_1); &dz_2 = \frac{\partial TC}{\partial z_2} = 2*(C_2-label[1]*O_2*(1-O_2); \\
\end{align}

Now we can just continue from there to find the weights, and biases of the hidden layer. Since we have already calculated the partial derivatives $\frac{\partial TC}{\partial z_1}$ and $\frac{\partial TC}{\partial z_2}$, and replaced it with `dz1` and `dz2`,  finding the partial derivatives of the total cost in respect to the biases $\frac{\partial TC}{\partial D_4}$ and $\frac{\partial TC}{\partial D_5}$ is trivial:

\begin{align*}
\frac{\partial TC}{\partial D_4} &= \frac{\partial TC}{\partial z_1}*\frac{\partial z_1}{\partial D_4}; &\frac{\partial TC}{\partial D_5} &= \frac{\partial TC}{\partial z_2}*\frac{\partial z_2}{\partial D_5}; \\\\
\frac{\partial TC}{\partial D_4} &= dz_1*1; &\frac{\partial TC}{\partial D_5} &= dz_2*1; \\\\
\end{align*}

It is also very easy to find the partial derivatives of the total cost in respect to the weights $A_4$,$B_4$,$C_4$,$A_5$,$B_5$,$C_5$:  $\frac{\partial TC}{\partial A_4}$, $\frac{\partial TC}{\partial B_4}$, $\frac{\partial TC}{\partial C_4}$, $\frac{\partial TC}{\partial A_5}$, $\frac{\partial TC}{\partial B_5}$, $\frac{\partial TC}{\partial C_5}$:

\begin{align*}
\frac{\partial TC}{\partial A_4} &= \frac{\partial TC}{\partial z_1}*\frac{\partial z_1}{\partial A_4}; &\frac{\partial TC}{\partial B_4} &= \frac{\partial TC}{\partial z_1}*\frac{\partial z_1}{\partial B_4};
&\frac{\partial TC}{\partial C_4} &= \frac{\partial TC}{\partial z_1}*\frac{\partial z_1}{\partial C_4};\\\\
\frac{\partial TC}{\partial A_4} &= dz_1*N_1;
&\frac{\partial TC}{\partial B_4} &= dz_1*N_2;
&\frac{\partial TC}{\partial C_4} &= dz_1*N_3;\\\\
\end{align*}

\begin{align*}
\frac{\partial TC}{\partial A_5} &= \frac{\partial TC}{\partial z_2}*\frac{\partial z_2}{\partial A_5}; &\frac{\partial TC}{\partial B_5} &= \frac{\partial TC}{\partial z_2}*\frac{\partial z_2}{\partial B_5};
&\frac{\partial TC}{\partial C_5} &= \frac{\partial TC}{\partial z_2}*\frac{\partial z_2}{\partial C_5};\\\\
\frac{\partial TC}{\partial A_5} &= dz_2*N_1;
&\frac{\partial TC}{\partial B_5} &= dz_2*N_2;
&\frac{\partial TC}{\partial C_5} &= dz_2*N_3;\\\\
\end{align*}

However, calculating the partial derivatives of the total cost in respect to the activations $N_1$ and $N_2$  warrants attention. Since the total cost of the neural network is influenced by the activation $N_1$ via __two paths__, going through neurons $o_1$ and $o_2$, the value of the partial derivative $\frac{\partial TC}{\partial N_1}$ is influenced by the total cost over the same 2 paths! This means, that to calculate $\frac{\partial TC}{\partial N_1}$, we need to add two partial derivatives on the two paths as shown below:

\begin{align*}
\frac{\partial TC}{\partial N_1}&=\frac{\partial TC}{\partial z_1}*\frac{\partial z_1}{\partial N_1}+
\frac{\partial TC}{\partial z_2}*\frac{\partial z_2}{\partial N_1}\\\\
\frac{\partial TC}{\partial N_1}&=dz_1*A_4+
dz_2*A_5\\\\
\end{align*}

The same holds for computing the partial derivatives of the total cost in respect to the activations $N_2$ and $N_3$:

\begin{align*}
&\frac{\partial TC}{\partial N_2}=dz_1*B_4+dz_2*B_5; &\frac{\partial TC}{\partial N_3}=dz_1*C_4+dz_2*C_5;
\end{align*}

To simplify the futher calculation, it is useful to calculate the partial derivatives of the total cost in respect to the intermediate values $p_1$, $p_2$, and $p_3$. They are simply the application of the derivative of the sigmoid function:

\begin{align*}
\frac{\partial TC}{\partial p_1}=\frac{\partial TC}{\partial N_1}*\sigma\prime(p_1);\;\;\;\frac{\partial TC}{\partial p_2}=\frac{\partial TC}{\partial N_2}*\sigma\prime(p_2);\;\;\; \frac{\partial TC}{\partial p_2}=\frac{\partial TC}{\partial N_3}*\sigma\prime(p_3);
\end{align*}

If we replace the value of $\frac{\partial TC}{\partial N_1}$, $\frac{\partial TC}{\partial N_2}$, and $\frac{\partial TC}{\partial N_3}$ by the variables `dN1`, `dN2` and `dN3`, we can easily compute these derivatives:

\begin{align*}
\frac{\partial TC}{\partial p_1}=dN_1*(N_1*(1-N_1));\;\;\;\frac{\partial TC}{\partial p_2}=dN_2*(N_2*(1-N_2));\;\;\; \frac{\partial TC}{\partial p_2}=dN_3*(N_3*(1-N_3));
\end{align*}

By following the same procedure as before, it is easy to compute the  partial derivatives in respect to the remaining weights and biases.

### The Algorithm

Let us first initialise the helper functions:

```python
def plot_data(data, labels):
    fig = plt.figure(figsize=(5,5))
    ax = fig.add_subplot(111)
    ax.scatter(data[:,0], data[:,1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)
    nudge = 0.08
    for i in range(data.shape[0]):
        d = data[i]
        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))
    ax.set_aspect('equal', 'datalim')
    plt.show()

def create_meshgrid(data):
    h = 0.02
    x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1
    y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    return (xx,yy,np.ones(xx.shape))

def eval_accuracy_neural(params, data, labels):
    A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5 = params
    num_correct = 0;

    for (X,Y),label_old in zip(data,labels):
        score = circuit_forward(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y)
        sc = np.array((int(score[0]>0.5),int(score[1]>0.5)))
        true_label = convert_label(label_old)
        if (sc[0]==true_label[0] and sc[1]==true_label[1]):
            num_correct += 1
    return num_correct / len(data)

def plot_neural_simple(params, grid,data, labels, iteration, accuracy):
    nudge = 0.06
    A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5 = params
    xx,yy,Z = grid

    for i in range(xx.shape[0]): # row
        for j in range(yy.shape[1]): #column
            X, Y = xx[i][j],yy[i][j]
            score = circuit_forward(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y)
            if (np.argmax(score)==0):
                score =  -1
            else:
                score =  1
            Z[i][j] = score
    fig = plt.figure(figsize=(5,5))
    ax = fig.add_subplot(111)
    plt.title(f'accuracy at the iteration {iteration}: {accuracy}')
    ax.contourf(xx, yy, Z, cmap=plt.cm.binary, alpha=0.1, zorder=15)
    ax.scatter(data[:, 0], data[:, 1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)
    ax.set_aspect('equal')
    for i in range(data.shape[0]):
        d = data[i]
        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.show()

def sigmoid(x):
    return 1/(1+np.exp(-x))

def circuit_forward(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y):
    N1 = sigmoid(A1*X + B1*Y + C1) # 1st neuron
    N2 = sigmoid(A2*X + B2*Y + C2) # 2nd neuron
    N3 = sigmoid(A3*X + B3*Y + C3) # 3rd neuron
    z1 = A4*N1 + B4*N2 + C4*N3 + D4 #temp 1
    z2 = A5*N1 + B5*N2 + C5*N3 + D5 #temp 2
    O1 = sigmoid(z1) # output 1
    O2 = sigmoid(z2) # output 2
    return (O1,O2)
```

```python
plot_data(data, labels)
```

Like before, the algorithm is extremely simplified and inefficiant in order to show clearly the internals of the calculation. Let's initialise the parameters of the network. This be done very efficiently with matrices, but here for the sake of clarifying the procedure we will initialise them manually with normally distributed random values between 0 and 1:

```python
def train_neural_network(data, labels, step_size, epochs, iter_info):
    # for visualisation purposes only
    grid = create_meshgrid(data)
    # first we initialise all the weights and biases as normally distributed random numbers
    rnd = np.random.normal(size=17)
    # hidden layer
    # neuron 1
    A1, B1, C1 = rnd[0], rnd[1], rnd[2]
    # neuron 2
    A2, B2, C2 = rnd[3], rnd[4], rnd[5]
    # neuron 3
    A3, B3, C3 = rnd[6], rnd[7], rnd[8]
    # output layer
    # neuron 1
    A4, B4, C4, D4 = rnd[9], rnd[10], rnd[11], rnd[12]
    # neuron 2
    A5, B5, C5, D5 = rnd[13], rnd[14], rnd[15], rnd[16]

    # in each epoch, we will go through all the data points once
    for i in range (epochs):
        # total cost must be restarted in every epoch
        total_cost = 0.0
        ddA1, ddA2, ddA3, ddA4, ddA5, ddB1, ddB2, ddB3, ddB4, ddB5, ddC1, ddC2, ddC3, ddC4, ddC5, ddD4, ddD5 = 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
        # here we loop trough all the data points, and get X, Y and the corresponding label
        for index, ((X,Y), label_old) in enumerate(zip(data, labels)):
            # forward pass
            N1 = sigmoid(A1*X + B1*Y + C1) # 1st neuron activation
            N2 = sigmoid(A2*X + B2*Y + C2) # 2nd neuron activation
            N3 = sigmoid(A3*X + B3*Y + C3) # 3rd neuron activation
            z1 = A4*N1 + B4*N2 + C4*N3 + D4 #intermediate step 1
            z2 = A5*N1 + B5*N2 + C5*N3 + D5 #intermediate step 1
            O1 = sigmoid(z1) # final activation 1
            O2 = sigmoid(z2) # final activation 1
            # convert the old labels -1 and 1 to tuples (1,0) and (0,1)
            label = convert_label(label_old)
            # This is the implementation of the quadratic cost for the single data point
            cost = (O1-label[0])**2 + (O2-label[1])**2
            # This value is then accumulated into total_cost variable, so that the cost is expressed
            # for all the data points
            total_cost += cost

            # backpropagating through the network
            # here we compute the derivative of the total cost TC in respect to the intermediate steps z1 and z2
            dz1 = 2*(O1-label[0])*O1*(1-O1)
            dz2 = 2*(O2-label[1])*O2*(1-O2)
            # computing the derivatives of the weights of the first neuron in the output layer
            dA4, dB4, dC4 = dz1*N1, dz1*N2, dz1*N3
            # computing the derivatives of the weights of the second neuron in the output layer
            dA5, dB5, dC5 = dz2*N1, dz2*N2, dz2*N3
            # computing the derivatives of the biases in both neurons of the output layer
            dD4, dD5 = dz1, dz2

            # computing the partials of the intermediate steps dz1, dz2, dz3 in respect to the activations N1, N2, N3
            dN1 = dz1*A4 + dz2*A5
            dN2 = dz1*B4 + dz2*B5
            dN3 = dz1*C4 + dz2*C5
            # computing the partials of the intermediate steps p1, p2 and p3
            dp1 = dN1*N1*(1-N1)
            dp2 = dN2*N2*(1-N2)
            dp3 = dN3*N3*(1-N3)
            # computing the partials of the total cost in respect to the biases C1, C2, C3
            dC1, dC2, dC3 = dp1, dp2, dp3
            # computing the partials of the total cost in respect to the weights A1, B1, A2, B2, C1, C2
            dA1, dA2, dA3 = dp1*X, dp2*X, dp3*X
            dB1, dB2, dB3 = dp1*Y, dp2*Y, dp3*Y

            # since we have multiple data points, and we can compute only a single partial derivative at the time
            # we accumulate their values in the variables ddA1...ddD4
            ddA1 += dA1
            ddA2 += dA2
            ddA3 += dA3
            ddA4 += dA4
            ddA5 += dA5
            ddB1 += dB1
            ddB2 += dB2
            ddB3 += dB3
            ddB4 += dB4
            ddB5 += dB5
            ddC1 += dC1
            ddC2 += dC2
            ddC3 += dC3
            ddC4 += dC4
            ddC5 += dC5
            ddD4 += dD4
            ddD5 += dD5

        # Now we make an average of all the accumulated derivatives by dividing with the number of data points
        datalen = len(data)
        ddA1 /= datalen
        ddA2 /= datalen
        ddA3 /= datalen
        ddA4 /= datalen
        ddA5 /= datalen
        ddB1 /= datalen
        ddB2 /= datalen
        ddB3 /= datalen
        ddB4 /= datalen
        ddB5 /= datalen
        ddC1 /= datalen
        ddC2 /= datalen
        ddC3 /= datalen
        ddC4 /= datalen
        ddC5 /= datalen
        ddD4 /= datalen
        ddD5 /= datalen

        # finally, we do the parameter update with the averaged values ddA1...ddD4
        A1 -= step_size * ddA1;
        B1 -= step_size * ddB1;
        C1 -= step_size * ddC1;
        A2 -= step_size * ddA2;
        B2 -= step_size * ddB2;
        C2 -= step_size * ddC2;
        A3 -= step_size * ddA3;
        B3 -= step_size * ddB3;
        C3 -= step_size * ddC3;
        A4 -= step_size * ddA4;
        B4 -= step_size * ddB4;
        C4 -= step_size * ddC4;
        A5 -= step_size * ddA5;
        B5 -= step_size * ddB5;
        C5 -= step_size * ddC5;
        D4 -= step_size * ddD4;
        D5 -= step_size * ddD5;

        if (i%iter_info==0):
            accuracy = eval_accuracy_neural((A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5),data,labels)
            plot_neural_simple((A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5),grid, data, labels, i, accuracy)
            print (f'total cost: {total_cost}\n')

    return (A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5)
```

```python
params = train_neural_network(data, labels, 3, 501, 100)
```

Inspect each data point's predicted vs true label

```python
A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5 = params
for i, ((X,Y),label_old) in enumerate(zip(data,labels)):
    score = circuit_forward(A1,A2,A3,A4,A5,B1,B2,B3,B4,B5,C1,C2,C3,C4,C5,D4,D5,X,Y)
    label = convert_label(label_old)
    sc = np.array((int(score[0]>0.5),int(score[1]>0.5)))
    print (f'data point {i}: real label : {label}, pred. label: {sc}, {(sc[0]==label[0] and sc[1]==label[1])}')
```

    data point 0: real label : (0, 1), pred. label: [0 1], True
    data point 1: real label : (1, 0), pred. label: [1 0], True
    data point 2: real label : (0, 1), pred. label: [0 1], True
    data point 3: real label : (1, 0), pred. label: [1 0], True
    data point 4: real label : (1, 0), pred. label: [1 0], True
    data point 5: real label : (0, 1), pred. label: [0 1], True
    data point 6: real label : (1, 0), pred. label: [1 0], True
    data point 7: real label : (0, 1), pred. label: [0 1], True
    data point 8: real label : (1, 0), pred. label: [1 0], True
    data point 9: real label : (1, 0), pred. label: [1 0], True

08. An Exercise - 2x3x2 Neural Network.ipynb


```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

# **Plot routine**

```python
def plot(type, xaxis, yaxis, zaxis, caxis, rot = [0, 0]):

 if type == 'data':
    fig = plt.figure(figsize=(8, 6))

    ax1 = fig.add_subplot(111)
    ax1.plot(caxis, xaxis, lw = 0.5, label = 'X data')
    ax1.set_xlabel("Time")
    ax1.set_ylabel("Axis data")

    ax1.plot(caxis, yaxis, lw = 0.5, label = 'Y data')

    ax1.plot(caxis, zaxis, lw = 0.5, label = 'Z data')

    plt.legend(loc = "upper center", ncol = 3)

    fig.tight_layout()

    f = plt.figure(figsize=(8, 6))

    ax4 = f.add_subplot(111)
    ax4 = f.gca(projection='3d')
    ax4.view_init(rot[0], rot[1])
    ax4.plot(xaxis, yaxis, zaxis, lw = 0.5)
    ax4.set_xlabel("X Axis")
    ax4.set_ylabel("Y Axis")
    ax4.set_zlabel("Z Axis")
    ax4.set_title("Lorenz Attractor")

    f.tight_layout()

    f2 = plt.figure(figsize=(8, 6))

    ax4 = f2.add_subplot(111)
    ax4 = f2.gca(projection='3d')
    ax4.view_init(rot[0], rot[1])
    ax4.scatter(xaxis, yaxis, zaxis, s = 0.1)
    ax4.set_xlabel("X Axis")
    ax4.set_ylabel("Y Axis")
    ax4.set_zlabel("Z Axis")
    ax4.set_title("Lorenz Attractor")

    f2.tight_layout()

    plt.show()
```

# **Lorenz Calc**

```python
def lorenz(x, y, z, s = 10, r = 28, b = 2.667):

    x_pt = s * (y - x)
    y_pt = r * x - y - x * z
    z_pt = x * y - b * z

    return x_pt, y_pt, z_pt
```

# **Set Parameters**

```python
dt = 0.01
num_steps = 10000

xs = np.empty(num_steps + 1)
ys = np.empty(num_steps + 1)
zs = np.empty(num_steps + 1)

#Set initial parameters
#Default: 0., 1., 1.05
xs[0], ys[0], zs[0] = (0, 1., 1.05)
```

# **Create data**

```python
for i in range(num_steps):
    x_pt, y_pt, z_pt = lorenz(xs[i], ys[i], zs[i])
    xs[i + 1] = xs[i] + (x_pt * dt)
    ys[i + 1] = ys[i] + (y_pt * dt)
    zs[i + 1] = zs[i] + (z_pt * dt)
```

\begin{equation}
        \frac{dx}{dt} = s (y - x)\\
        \frac{dy}{dt} = r x - y - x z\\
        \frac{dz}{dt} = x y - b z
    \end{equation}

```python
plot('data', xs, ys, zs, range(0, len(xs)), rot = [5, -90])
```

# **Pendulum**

## **Pendulum equation**

```python
def f(theta, omega, t):
    return -theta - k * omega + A * np.cos(phi * t)
```

## **Pendulum create**

```python
 def p_create(theta, omega, t, dt, nsteps, k, phi, A):

  t_list = [t]
  omega_list = [omega]
  theta_list = [theta]

  for step in range(0,10000):
      e = 0.07

      k1a = dt * omega
      k1b = dt * f(theta, omega, t)
      k2a = dt * (omega + k1b)
      k2b = dt * f(theta + k1a, omega + k1b, t + dt)

      theta = theta + (k1a + k2a) / 2
      omega = omega + (k1b + k2b) / 2
      t = t + dt

      t_list.append(t)

      theta_list.append(theta)
      omega_list.append(omega)

  return t_list,theta_list, omega_list
```

```python
theta = 0.2 #angle
omega = 0.0 #angular velocity
t = 0.0 #time
dt = 0.01 #increase
nsteps = 10000 #number of steps
k = 0.0 #damping coefficient
phi = 0.66667 #initial angular frequency
A = 0.0 #amplitude

time_pend, angles_pend, vel_pend = p_create(theta, omega, t, dt, nsteps, k, phi, A)

k = 0.15 #new damp

time_damp_pend, angles_damp_pend, vel_damp_pend = p_create(theta, omega, t, dt, \
                                                           nsteps, k, phi, A)

fig, axs = plt.subplots(2, 2, figsize=(12, 6))

axs[0, 0].set_title('Linear pendulum')
axs[0, 0].set_xlabel('time')
axs[0, 0].set_ylabel('Angle')
axs[0, 0].plot(time_pend, angles_pend)
axs[1, 0].set_title('Phase portrait')
axs[1, 0].set_xlabel('Angle')
axs[1, 0].set_ylabel('Velocity')
axs[1, 0].plot(angles_pend, vel_pend)
axs[0, 1].set_title('Damped pendulum')
axs[0, 1].set_xlabel('time')
axs[0, 1].set_ylabel('Angle')
axs[0, 1].plot(time_damp_pend, angles_damp_pend)
axs[1, 1].set_title('Phase portrait')
axs[1, 1].set_xlabel('Angle')
axs[1, 1].set_ylabel('Velocity')
axs[1, 1].plot(angles_damp_pend, vel_damp_pend)

fig.tight_layout()

plt.show()
```


```python
import scipy
from scipy import linalg, matrix
import numpy as np
```

```python
from sympy.solvers import solve
from sympy import Symbol
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

## Equation Solver

```python
x = Symbol('x')
solve(x**2 -1, x)
```

    [-1, 1]

## Eigenvelues with Equation

```python
A = matrix([[1,2],[3,-4]])
A
```

    matrix([[ 1,  2],
            [ 3, -4]])

```python
lam = Symbol('lam')
```

```python
A_lam = A - lam * np.asmatrix(np.identity(2))
A_lam
```

    matrix([[-1.0*lam + 1, 2],
            [3, -1.0*lam - 4]], dtype=object)

```python
equation = A_lam[0,0]*A_lam[1,1] - A_lam[0,1]*A_lam[1,0]
equation
```

    (-1.0*lam - 4)*(-1.0*lam + 1) - 6

```python
solve(equation,lam)
```

    [-5.00000000000000, 2.00000000000000]

## Eigenvalues and Eigenvecotrs with Package

```python
eigenvalue, eigenvector = linalg.eig(A)
```

```python
eigenvalue
```

    array([ 2.+0.j, -5.+0.j])

```python
eigenvector
```

    array([[ 0.89442719, -0.31622777],
           [ 0.4472136 ,  0.9486833 ]])

## Eigen Value Decomposition

```python
eigenvalue, eigenvector = linalg.eig(A)
```

```python
eigenvalue.shape[0]
```

    2

```python
L = np.identity(eigenvalue.shape[0])
for i in range(eigenvalue.shape[0]):
   L[i, i] = eigenvalue[i]
L
```

    C:\Users\admin\Anaconda3\lib\site-packages\ipykernel_launcher.py:3: ComplexWarning: Casting complex values to real discards the imaginary part
      This is separate from the ipykernel package so we can avoid doing imports until

    array([[ 2.,  0.],
           [ 0., -5.]])

```python
S = np.asmatrix(eigenvector)
S
```

    matrix([[ 0.89442719, -0.31622777],
            [ 0.4472136 ,  0.9486833 ]])

```python
A*S
```

    matrix([[ 1.78885438,  1.58113883],
            [ 0.89442719, -4.74341649]])

```python
S*L
```

    matrix([[ 1.78885438,  1.58113883],
            [ 0.89442719, -4.74341649]])

```python
A*S ==S*L
```

    matrix([[ True, False],
            [False,  True]], dtype=bool)

```python
np.allclose(A*S, S*L)
```

    True

## SVD

```python
A = matrix([[3, 1, 1], [-1, 3, 1]])
A
```

    matrix([[ 3,  1,  1],
            [-1,  3,  1]])

```python
U, s, V = linalg.svd(A, full_matrices=True)
```

```python
U = np.asmatrix(U)
U
```

    matrix([[-0.70710678, -0.70710678],
            [-0.70710678,  0.70710678]])

```python
s = np.asmatrix(s)
s
```

    matrix([[ 3.46410162,  3.16227766]])

```python
V = np.asmatrix(V)
V
```

    matrix([[ -4.08248290e-01,  -8.16496581e-01,  -4.08248290e-01],
            [ -8.94427191e-01,   4.47213595e-01,   5.27355937e-16],
            [ -1.82574186e-01,  -3.65148372e-01,   9.12870929e-01]])

```python
list(A.shape)
```

    [2, 3]

```python
np.min(list(A.shape))
```

    2

```python
S = np.zeros((A.shape))
for i in range(np.min(list(A.shape))):
    S[i, i] = s[0, i]
S
```

    array([[ 3.46410162,  0.        ,  0.        ],
           [ 0.        ,  3.16227766,  0.        ]])

```python
U*S*V
```

    matrix([[ 3.,  1.,  1.],
            [-1.,  3.,  1.]])

## Image Compression with SVD

https://github.com/rameshputalapattu/jupyterexplore/blob/master/jupyter_interactive_environment_exploration.ipynb

```python
import matplotlib.image as mpimg
img = mpimg.imread('sample.png')
```

```python
plt.imshow(img)
```

```python
from skimage.color import rgb2gray
from skimage import img_as_ubyte, img_as_float

gray_images = {
    "Pierrot": rgb2gray(img_as_float(img))
}
```

```python
def compress_svd(image, k):
    U, s, V = linalg.svd(image, full_matrices=False)
    reconst_matrix = np.dot(U[:, :k], np.dot(np.diag(s[:k]),V[:k,:]))

    return reconst_matrix, s
```

```python
reconst_matrix, s = compress_svd(rgb2gray(img_as_float(img)),50)
```

```python
s[:5]
```

    array([ 266.53154921,   68.28481384,   57.90747286,   39.15993506,
             36.53256079])

```python
plt.plot(s[:5])
```

```python
def compress_show_gray_images(img_name, k):

    image = gray_images[img_name]

    original_shape = image.shape
    reconst_img, s = compress_svd(image, k)

    fig, axes = plt.subplots(1,2, figsize=(8,5))

    axes[0].plot(s)

    compression_ratio = 100.0 *(k*(original_shape[0] + original_shape[1])+k)/(original_shape[0]*original_shape[1])

    axes[1].set_title("compression ratio={:.2f}".format(compression_ratio)+"%")
    axes[1].imshow(reconst_img, cmap='gray')
    axes[1].axis('off')

    fig.tight_layout()
```

```python
from ipywidgets import interact, interactive, interact_manual
interact(compress_show_gray_images, img_name=list(gray_images.keys()),k=(1,100));
```

```python
from sklearn import datasets
import pandas as pd
```

```python
iris = datasets.load_iris()

labels = pd.DataFrame(iris.target)
labels.columns = ['labels']

data = pd.DataFrame(iris.data)
data.columns=['Sepal length', 'Sepal width', 'Petal length', 'Petal width']

data = pd.concat([data, labels], axis=1)

data.head()
```

      Sepal length
      Sepal width
      Petal length
      Petal width
      labels

      0
      5.1
      3.5
      1.4
      0.2
      0

      1
      4.9
      3.0
      1.4
      0.2
      0

      2
      4.7
      3.2
      1.3
      0.2
      0

      3
      4.6
      3.1
      1.5
      0.2
      0

      4
      5.0
      3.6
      1.4
      0.2
      0

```python
feature = data[['Sepal length', 'Sepal width']]
feature.head()
```

      Sepal length
      Sepal width

      0
      5.1
      3.5

      1
      4.9
      3.0

      2
      4.7
      3.2

      3
      4.6
      3.1

      4
      5.0
      3.6

```python
from sklearn.cluster import SpectralClustering
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
model = SpectralClustering(n_clusters=3)
model.fit(feature)

predict = pd.DataFrame(model.fit_predict(feature))
predict.columns['predict']

r = pd.concat([feature, predict], axis=1)
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import interp1d
import emcee
```

# Fitting with model uncertainty

In fitting spectrum routine, given an observed spectrum and a computational model to simulate the spectrum, we are interested in determining the parameters (with its error) that gives the most similar spectrum to the observed one.

If we know the "true" underlying model of the spectrum (i.e. we can simulate the spectrum **exactly**, given a set of parameters), obtaining the error of the parameters are just like a linear fit. However, if we only have an approximate model, the error can be severely underdetermined.

In this notebook, I will show the problem of fitting with an approximate model and how to solve it (hopefully).

## The true model and the true data

In this notebook, we consider a simplified model of generating a spectrum. Let's say we have 4 parameters in which determining the values at \\(x = (0, 0.3, 0.7, 1.0)\\). All the other values are obtained using the cubic spline interpolation. We call this the "**cubic**" model which is the true model. Suppose in an experiment, we can observe 1500 data points linearly spaced from \\([0, 1.0]\\), with some noise.

```python
np.random.seed(123)
ndata = 1500
xparameters = [0, 0.3, 0.7, 1.0]
parameters = [0.1, 0.7, 0.2, 0.4]
sigma_noise = 0.05

finterp1 = interp1d(xparameters, parameters, kind="cubic")
xdata = np.linspace(0, 1, ndata)
ydata = finterp1(xdata) + np.random.randn(*xdata.shape) * sigma_noise

plt.plot(xdata, ydata, '.')
plt.show()
```

This is the data we observe (as an example).

## Using the true model and known noise level

Assume that we are in an ideal world where we know the true model and we also know the noise level. The Bayesian inference of the parameters, $\mathbf{p}$, knowing the data, $\mathcal{D}$, can be written as

$$\begin{equation}
P(\mathbf{p} | \mathcal{D}) \propto P(\mathcal{D} | \mathbf{p}) P(\mathbf{p}).
\end{equation}$$

Take a uniform prior for the parameters and the likelihood is just a Gaussian distribution. Therefore,

$$\begin{equation}
P(\mathbf{p} | \mathcal{D}) \propto \exp\left[-\sum_i\frac{\left(\mathbf{\hat{y_i}} - \mathbf{y_i(\mathbf{p})} \right)^2}{2\sigma^2}\right].
\end{equation}$$

We can use the `emcee` module to draw samples.

```python
def get_ymodel1(xparams, params, xdata):
    finterp1 = interp1d(xparams, params, kind="cubic")
    return finterp1(xdata)

def lnprob1(params, xparams, xdata, ydata):
    ymodel = get_ymodel1(xparams, params, xdata)
    sigma = 0.05 # known noise level
    return -np.sum((ydata - ymodel)**2) / (2 * sigma**2)
```

```python
ndim, nwalkers = 4, 100
p0 = np.random.random((nwalkers, ndim))
sampler1 = emcee.EnsembleSampler(nwalkers, ndim, lnprob1, args=[xparameters, xdata, ydata])
_ = sampler1.run_mcmc(p0, 1000)
```

```python
nburn = 10000
params1 = sampler1.flatchain[nburn:,:]
plt.figure(figsize=(12,4))
for i in range(ndim):
    plt.subplot(1, ndim, i+1)
    ax = plt.gca()
    plt.hist(params1[:,i], 50, normed=True, range=parameters[i] + np.array([-1, 1])*0.05)
    ylim = ax.get_ylim()
    plt.plot(np.zeros(2) + parameters[i], ylim, 'k--')
    ax.set_ylim(ylim)
    plt.title("parameter #%d" % (i+1))

plt.show()

for i in range(ndim):
    print("parameter #%d: %f +- %f (true: %f)" % \
         (i+1, np.median(params1[:,i]),
          np.percentile(params1[:,i], 84) - np.percentile(params1[:,i], 16),
          parameters[i]))
```

```python
plt.figure(figsize=(8,4))
plt.plot(xdata, ydata, '.')
plt.plot(xdata, get_ymodel1(xparameters, np.median(params1, 0), xdata), '-', linewidth=2)

# unseen on the graph because it is too narrow
plt.fill_between(xdata,
                 get_ymodel1(xparameters, np.percentile(params1, 16, axis=0), xdata),
                 get_ymodel1(xparameters, np.percentile(params1, 84, axis=0), xdata), facecolor="orange")
plt.show()
```

As we can see, the retrieved parameters (i.e. the median) are very close to the true values.

## Using an approximate model and known noise level

Now we are going to a slightly less ideal world where we used an approximate model with known noise level. The approximate model here uses the quadratic and linear spline interpolation (the "**quadratic**" and "**linear**" model) instead of the cubic spline interpolation (the "**cubic**" model).
The inference is still the same as before, we just change how we calculate the model data in the `lnprob1` function.

```python
def get_ymodel2(xparams, params, xdata):
    finterp1 = interp1d(xparams, params, kind="linear")
    return finterp1(xdata)

def get_ymodel3(xparams, params, xdata):
    finterp1 = interp1d(xparams, params, kind="quadratic")
    return finterp1(xdata)

def lnprob2(params, xparams, xdata, ydata):
    ymodel = get_ymodel2(xparams, params, xdata)
    sigma = 0.05 # known noise level
    return -np.sum((ydata - ymodel)**2) / (2 * sigma**2)

def lnprob3(params, xparams, xdata, ydata):
    ymodel = get_ymodel3(xparams, params, xdata)
    sigma = 0.05 # known noise level
    return -np.sum((ydata - ymodel)**2) / (2 * sigma**2)
```

```python
ndim, nwalkers = 4, 100
p0 = np.random.random((nwalkers, ndim))
sampler2 = emcee.EnsembleSampler(nwalkers, ndim, lnprob2, args=[xparameters, xdata, ydata])
_ = sampler2.run_mcmc(p0, 1000)
sampler3 = emcee.EnsembleSampler(nwalkers, ndim, lnprob3, args=[xparameters, xdata, ydata])
_ = sampler3.run_mcmc(p0, 1000)
```

```python
nburn = 10000
params2 = sampler2.flatchain[nburn:,:]
params3 = sampler3.flatchain[nburn:,:]
plt.figure(figsize=(16,6))
for i in range(ndim):
    for j in range(2):
        paramss = params2 if j == 0 else params3

        plt.subplot(2, ndim, j*ndim+i+1)
        ax = plt.gca()
        plt.hist(paramss[:,i], 50, normed=True, range=parameters[i] + np.array([-1, 1])*0.2)
        ylim = ax.get_ylim()
        plt.plot(np.zeros(2) + parameters[i], ylim, 'k--')
        ax.set_ylim(ylim)
        if j == 0:
            plt.title("parameter #%d" % (i+1))
        if i == 0:
            plt.ylabel("%s model" % ("linear" if j == 0 else "quadratic"))

plt.show()

for j in range(2):
    print("%s model" % ("linear" if j == 0 else "quadratic"))
    paramss = params2 if j == 0 else params3
    for i in range(ndim):
        print("parameter #%d: %f +- %f (true: %f)" % \
             (i+1, np.median(paramss[:,i]),
              np.percentile(paramss[:,i], 84) - np.percentile(paramss[:,i], 16),
              parameters[i]))
```

```python
plt.figure(figsize=(16,4))
plt.subplot(1,2,1)
plt.plot(xdata, ydata, '.')
plt.plot(xdata, get_ymodel2(xparameters, np.median(params2, 0), xdata), '-', linewidth=2)
# unseen on the graph because it is too narrow
plt.fill_between(xdata,
                 get_ymodel2(xparameters, np.percentile(params2, 16, axis=0), xdata),
                 get_ymodel2(xparameters, np.percentile(params2, 84, axis=0), xdata), facecolor="orange")
plt.subplot(1,2,2)
plt.plot(xdata, ydata, '.')
plt.plot(xdata, get_ymodel3(xparameters, np.median(params3, 0), xdata), '-', linewidth=2)
# unseen on the graph because it is too narrow
plt.fill_between(xdata,
                 get_ymodel3(xparameters, np.percentile(params3, 16, axis=0), xdata),
                 get_ymodel3(xparameters, np.percentile(params3, 84, axis=0), xdata), facecolor="orange")
plt.show()
```

As we can see here, if we are using approximate models, the retrieved values of the parameteres are off while the produced error is still small. In other words, it is confident, but it is wrong. If we are using less accurate model, the effect is even worse: the predicted values are off by much and the error remains small.

The problem with the inference we used in this case is that we only assume that the data differ only by the Gaussian noise. This is not true since the data also differ due to the model inadequacy.

#### Max deviation inadequacy

In my latest paper, I assume the model inadequacy as stochastic variables with distribution

$$\begin{equation}
P(\eta(x)) \propto \exp\left[-\frac{\max\left(\mathbf{\hat{y_i}} - \mathbf{y_i}\right)^2}{2d^2}\right]
\end{equation}$$

with value of $d$ approximately the values of the noise.

Unfortunately, I didn't take into account the effect of the Gaussian noise. So the distribution above becomes the likelihood of observing the data.

```python
def lnprob2_model1(params, xparams, xdata, ydata):
    ymodel = get_ymodel2(xparams, params, xdata)
    d = 0.05 # d approximately the noise level
    return -np.max((ydata - ymodel)**2) / (2 * d**2)

def lnprob3_model1(params, xparams, xdata, ydata):
    ymodel = get_ymodel3(xparams, params, xdata)
    d = 0.05 # d approximately the noise level
    return -np.max((ydata - ymodel)**2) / (2 * d**2)

```

```python
ndim, nwalkers = 4, 100
p0 = np.random.random((nwalkers, ndim))
sampler2_model1 = emcee.EnsembleSampler(nwalkers, ndim, lnprob2_model1, args=[xparameters, xdata, ydata])
_ = sampler2_model1.run_mcmc(p0, 1000)
sampler3_model1 = emcee.EnsembleSampler(nwalkers, ndim, lnprob3_model1, args=[xparameters, xdata, ydata])
_ = sampler3_model1.run_mcmc(p0, 1000)
```

```python
nburn = 10000
params2_model1 = sampler2_model1.flatchain[nburn:,:]
params3_model1 = sampler3_model1.flatchain[nburn:,:]
plt.figure(figsize=(16,6))
for i in range(ndim):
    for j in range(2):
        paramss = params2_model1 if j == 0 else params3_model1

        plt.subplot(2, ndim, j*ndim+i+1)
        ax = plt.gca()
        plt.hist(paramss[:,i], 50, normed=True, range=parameters[i] + np.array([-1, 1])*0.2)
        ylim = ax.get_ylim()
        plt.plot(np.zeros(2) + parameters[i], ylim, 'k--')
        ax.set_ylim(ylim)
        if j == 0:
            plt.title("parameter #%d" % (i+1))
        if i == 0:
            plt.ylabel("%s model" % ("linear" if j == 0 else "quadratic"))

plt.show()

for j in range(2):
    print("%s model" % ("linear" if j == 0 else "quadratic"))
    paramss = params2_model1 if j == 0 else params3_model1
    for i in range(ndim):
        print("parameter #%d: %f +- %f (true: %f)" % \
             (i+1, np.median(paramss[:,i]),
              np.percentile(paramss[:,i], 84) - np.percentile(paramss[:,i], 16),
              parameters[i]))
```

```python
plt.figure(figsize=(16,4))
plt.subplot(1,2,1)
plt.plot(xdata, ydata, '.', alpha=0.3)
plt.plot(xdata, get_ymodel2(xparameters, np.median(params2_model1, 0), xdata), '-', linewidth=2)
# unseen on the graph because it is too narrow
plt.fill_between(xdata,
                 get_ymodel2(xparameters, np.percentile(params2_model1, 16, axis=0), xdata),
                 get_ymodel2(xparameters, np.percentile(params2_model1, 84, axis=0), xdata),
                 facecolor="orange", alpha=0.5)
plt.subplot(1,2,2)
plt.plot(xdata, ydata, '.', alpha=0.3)
plt.plot(xdata, get_ymodel3(xparameters, np.median(params3_model1, 0), xdata), '-', linewidth=2)
# unseen on the graph because it is too narrow
plt.fill_between(xdata,
                 get_ymodel3(xparameters, np.percentile(params3_model1, 16, axis=0), xdata),
                 get_ymodel3(xparameters, np.percentile(params3_model1, 84, axis=0), xdata),
                 facecolor="orange", alpha=0.5)
plt.plot(xdata, get_ymodel1(xparameters, parameters, xdata), '-', linewidth=2)

nburn = 10000
params2 = sampler2.flatchain[nburn:,:]
params3 = sampler3.flatchain[nburn:,:]
plt.figure(figsize=(16,6))
for i in range(ndim):
    for j in range(2):
        paramss = params2 if j == 0 else params3

        plt.subplot(2, ndim, j*ndim+i+1)
        ax = plt.gca()
        plt.hist(paramss[:,i], 50, normed=True, range=parameters[i] + np.array([-1, 1])*0.2)
        ylim = ax.get_ylim()
        plt.plot(np.zeros(2) + parameters[i], ylim, 'k--')
        ax.set_ylim(ylim)
        if j == 0:
            plt.title("parameter #%d" % (i+1))
        if i == 0:
            plt.ylabel("%s model" % ("linear" if j == 0 else "quadratic"))

plt.show()

for j in range(2):
    print("%s model" % ("linear" if j == 0 else "quadratic"))
    paramss = params2 if j == 0 else params3
    for i in range(ndim):
        print("parameter #%d: %f +- %f (true: %f)" % \
             (i+1, np.median(paramss[:,i]),
              np.percentile(paramss[:,i], 84) - np.percentile(paramss[:,i], 16),
              parameters[i]))
plt.show()
```

From the results above, this model of the model inadequacy works well if the model is approximately accurate. If the model is quite far from the true model, then the result is quite far off (compared to its standard deviation). Moreover, this model requires us to make an assumption on how *right* (or *wrong*) the model is by setting the value of $d$ manually.

What we want is minimal input from users and the model is automatically adjust the error on the retrieved parameters. If the model is less accurate, then the error on the retrieved parameters should be larger. If the model is more accurate, then the error should be smaller.

#### Gaussian process model inadequacy

Another way to model the model inadequacy is using Gaussian Process.

```python
from scipy.stats import multivariate_normal

def lnprob2_model2(params, xparams, xdata, ydata, dist):
    d = params[-1]
    m = params[-2]
    sigma = 0.05
    if d > 1 or d < 0 or m < 0 or sigma < 0: return -np.inf

    ncut = 10
    cov = m*m * np.exp(-dist[::ncut,::ncut]**2/(2*d*d)) + np.eye(len(xdata[::ncut])) * (sigma*sigma + 1e-7)

    ymodel = get_ymodel2(xparams, params[:-3], xdata)
    obs = ymodel - ydata
    obs = obs[::ncut]
    try:
        return multivariate_normal.logpdf(obs, mean=np.zeros(obs.shape[0]), cov=cov) - np.log(sigma * m * d)
    except:
        return -np.inf

def lnprob3_model2(params, xparams, xdata, ydata, dist):
    d = params[-1]
    m = params[-2]
    sigma = 0.05
    if d > 1 or d < 0 or m < 0 or sigma < 0: return -np.inf

    ncut = 10
    cov = m*m * np.exp(-dist[::ncut,::ncut]**2/(2*d*d)) + np.eye(len(xdata[::ncut])) * (sigma*sigma + 1e-7)

    ymodel = get_ymodel3(xparams, params[:-3], xdata)
    obs = ymodel - ydata
    obs = obs[::ncut]
    try:
        return multivariate_normal.logpdf(obs, mean=np.zeros(obs.shape[0]), cov=cov) - np.log(sigma * m * d)
    except:
        return -np.inf

```

```python
import scipy.spatial.distance
dist = scipy.spatial.distance.squareform(scipy.spatial.distance.pdist(np.expand_dims(xdata, 1)))
```

```python
ndim, nwalkers = 4, 100
nhparams = 3
p0 = np.random.random((nwalkers, ndim+nhparams))
sampler2_model2 = emcee.EnsembleSampler(nwalkers, ndim+nhparams, lnprob2_model2, args=[xparameters, xdata, ydata, dist])
%time _ = sampler2_model2.run_mcmc(p0, 1000)
sampler3_model2 = emcee.EnsembleSampler(nwalkers, ndim+nhparams, lnprob3_model2, args=[xparameters, xdata, ydata, dist])
%time _ = sampler3_model2.run_mcmc(p0, 1000)
```

    CPU times: user 20min 17s, sys: 17min 33s, total: 37min 51s
    Wall time: 5min 14s
    CPU times: user 21min, sys: 18min 37s, total: 39min 37s
    Wall time: 5min 34s

```python
nburn = 10000
params2_model2 = sampler2_model2.flatchain[nburn:,:]
params3_model2 = sampler3_model2.flatchain[nburn:,:]
plt.figure(figsize=(16,6))
for i in range(ndim):
    for j in range(2):
        paramss = params2_model2 if j == 0 else params3_model2

        plt.subplot(2, ndim, j*ndim+i+1)
        ax = plt.gca()
        plt.hist(paramss[:,i], 50, normed=True, range=parameters[i] + np.array([-1, 1])*2)
        ylim = ax.get_ylim()
        plt.plot(np.zeros(2) + parameters[i], ylim, 'k--')
        ax.set_ylim(ylim)
        if j == 0:
            plt.title("parameter #%d" % (i+1))
        if i == 0:
            plt.ylabel("%s model" % ("linear" if j == 0 else "quadratic"))

plt.show()

for j in range(2):
    print("%s model" % ("linear" if j == 0 else "quadratic"))
    paramss = params2_model2 if j == 0 else params3_model2
    for i in range(ndim):
        print("parameter #%d: %f +- %f (true: %f)" % \
             (i+1, np.median(paramss[:,i]),
              np.percentile(paramss[:,i], 84) - np.percentile(paramss[:,i], 16),
              parameters[i]))
```

```python
plt.figure(figsize=(16,4))
plt.subplot(1,2,1)
plt.plot(xdata, ydata, '.', alpha=0.3)
plt.plot(xdata, get_ymodel2(xparameters, np.median(params2_model2[:,:-nhparams], 0), xdata), '-', linewidth=2)
# unseen on the graph because it is too narrow
plt.fill_between(xdata,
                 get_ymodel2(xparameters, np.percentile(params2_model2[:,:-nhparams], 16, axis=0), xdata),
                 get_ymodel2(xparameters, np.percentile(params2_model2[:,:-nhparams], 84, axis=0), xdata),
                 facecolor="orange", alpha=0.5)
plt.subplot(1,2,2)
plt.plot(xdata, ydata, '.', alpha=0.3)
plt.plot(xdata, get_ymodel3(xparameters, np.median(params3_model2[:,:-nhparams], 0), xdata), '-', linewidth=2)
# unseen on the graph because it is too narrow
plt.fill_between(xdata,
                 get_ymodel3(xparameters, np.percentile(params3_model2[:,:-nhparams], 16, axis=0), xdata),
                 get_ymodel3(xparameters, np.percentile(params3_model2[:,:-nhparams], 84, axis=0), xdata),
                 facecolor="orange", alpha=0.5)
plt.plot(xdata, get_ymodel1(xparameters, parameters, xdata), '-', linewidth=2)
plt.show()
```

```python

```


# Information Retrieval in High Dimensional Data

# Assignment #3, 09.01.2018

# Kernel PCA (kPCA) and Fisher LDA
Group Number: G10

Group Members:
- AboulAtta Moustafa
- Achtner Martin
- Arifi Ridon
- Ehrhardt Daniel
- Fichtner Lukas

## Kernel PCA (kPCA)

### Task 1.1 [5 points]:

Download the file task3_1_kpca_demo.py from the web page. Implement KPCA using a Gaussian kernel function

\begin{equation} k(\mathbf{x}_1, \mathbf{x}_2) = exp \left(- \frac{\|\mathbf{x}_1 - \mathbf{x}_2 \|^2}{2 \sigma^2}\right), \end{equation}

by filling in the missing lines.

Vary $ alpha $ and $ sigma $ and observe the generated plots. In each of the two plots all generated data points are plotted. The color in the first plot indicates the value of the respective point when projected onto the first PC. The color in the second plot indicates the value of the respective point when projected onto the second PC.

```python
# kpca_demo
import matplotlib.pyplot as plt
import numpy as np
import scipy.misc as scm
```

```python
# data generation
np.random.seed(1234)

# the following code generates 2 dimensional data.
# specifically, for each vector the first component is between 0 and alpha,
# while the second is between 0 and 1
n = 1000                       # number of data points
alpha = 2                      # length/width ratio
s = np.array([alpha,1])
X = np.diag(s).dot(np.random.rand(2,n))          # uniformly distributed points on a rectangle

H = np.eye(n) - np.ones((n,n))/n           # create centering matrix
```

```python
def custom_sdist(X):
    """
    Funktion that given a matrix X returns the squared pairwise distances
    of the column vectors in matrix form
    """
    XX = np.dot(X.T, X)
    pdists = np.outer(np.diag(XX), np.ones(XX.shape[1]).T) + np.outer(np.ones(XX.shape[0]), np.diag(XX).T) - 2*XX
    return pdists
```

```python
def K(X):
    sigma = 1
    denominator = 2*sigma**2
    K = np.exp(-(custom_sdist(X)/denominator))
    return K
```

```python
k = 2 # number of eigenvectors
K_centered = np.dot(H, np.dot(K(X), H))

_, s, Vt = np.linalg.svd(K_centered)
s = np.sqrt(s[:k])
Y = np.dot(np.diag(s), Vt[:k,:])
```

```python
fig, axs = plt.subplots(1,2,figsize=(15,6), facecolor='w', edgecolor='k')
fig.subplots_adjust(hspace=.5, wspace=.1)
axs = axs.ravel()
for ip in range(k):
    im = axs[ip].scatter(X[0,:], X[1,:], c=Y[ip,:])
    axs[ip].set_title('Color indicates value of PC {} at this point'.format(ip+1))

fig.colorbar(im)
plt.show()
```

For appropriate choices of $ alpha $ and $ sigma $ you see a horizontal separation in the first component and a vertical separation in the second component. (Note that horizontal separation means, that you can separate the different colors in the plot with vertical lines and vice versa for vertical separation.) However, if $ alpha $ becomes too large, the second component is no longer vertical. Test this behavior for $ alpha $ between 1 and 12 and determine the $ sigma $ that provides vertical separation in the second component for each $ alpha $ . Provide a 2-dimensional plot with the axis $ alpha $ and $ sigma $ to illustrate this behavior.

### Data obtained through empirical testing

$\begin{array}{c|c} \alpha  & \sigma \\ \hline\hline 1 & 0.5 \\
2 & 1 \\
3 & 2.5 \\
4 & 4.5 \\
5 & 8 \\
6 & 11 \\
7 & 14 \\
8 & 18 \\
9 & 23 \\
10 & 29 \\
11 & 34 \\
12 & 38 \\
\end{array}$

 (Note that for $ \alpha = 1 $ the orientation of separation is switched.)

```python
alpha = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
sigma = np.array([0.5, 1, 2.5, 4.5, 8, 11, 14, 18, 23, 29, 34, 38])
```

```python
plt.plot(alpha, sigma)
plt.title('Values of sigma depending on alpha')
plt.xlabel('alpha')
plt.ylabel('sigma')
plt.show()
```

## Task 1.2 [5 points]:

Download the python script task3_1_toy_data.py for generating a toy example. The produced dataset contains two groups. Implement kPCA and determine the kernel function that allows to linearly separate the data using the first principal component.

```python
# Generate Data
np.random.seed(1234)
m = 50
N = 200
X = np.random.randn(2, m)/10
```

```python
# Plot example data
for idx in range(N):
    t = np.random.randn(2, 2)
    tmp = t[:, 0:1]/np.linalg.norm(t[:, 0]) + t[:, 1]/np.linalg.norm(t[:, 1:2])/10
    X = np.hstack((X, tmp))

plt.figure()
plt.scatter(X[0, m:], X[1, m:])
plt.scatter(X[0, 1:m], X[1, 1:m], c='r')

plt.show()
```

```python
n = 450

H = np.eye(n) - np.ones((n,n))/n           # create centering matrix
```

```python
sigma = 0.1

def K(X):
    # Use "radial kernel"
    squares = np.sum(np.square(X), axis = 0)
    K = np.dot(np.expand_dims(squares, axis=1), np.expand_dims(squares, axis=0))
    return K
```

```python
k = 1 # number of eigenvectors

K_centered = np.dot(H, np.dot(K(X), H))

_, s, Vt = np.linalg.svd(K_centered)
s = np.sqrt(s[:k])
Y = np.dot(np.diag(s), Vt[:k,:])
```

```python
fig, axs = plt.subplots(1,2,figsize=(15,6), facecolor='w', edgecolor='k')
fig.subplots_adjust(hspace=.5, wspace=.1)
axs = axs.ravel()
for ip in range(k):
    axs[ip].scatter(X[0,:], X[1,:], c=Y[ip,:])
    axs[ip].set_title('Color indicates value of PC {} at this point'.format(ip+1))

    im = axs[1].scatter(Y[0,:], np.zeros(Y.shape[1]), c=Y[0,:])
    axs[1].set_title('Color and x component shows linear separation of PC1 \n with the above kernel')

fig.colorbar(im)
plt.show()
```

## Fisher LDA

### Task 2 [10 points]:

Refer to Lab Course 7. In 1.e, we implemented LDA by normalizing the between-scatter matrix $ \mathbf{S}_b $ with the within-scatter matrix $ \mathbf{S}_w $. Another way to approach LDA is by finding the projection space spanned by the columns of

\begin{equation} \hat{\mathbf{U}} = \arg\max\limits_{\mathbf{U}^T \mathbf{U} = \mathbf{I}_k} \, \text{tr}(\mathbf{U}^T \mathbf{S}_b \mathbf{U}) - \beta\, \text{tr}(\mathbf{U}^T \mathbf{S}_w \mathbf{U}), \end{equation}

and projecting the data onto that space. Here, $ \beta $ is a real positiv tuning parameter to be chosen by hand.

+ Explain, how you can solve this problem by EVD

$ \rightarrow $ Since the trace is a linear operation we can rewrite the equation as:

\begin{equation} \hat{\mathbf{U}} = \arg\max\limits_{\mathbf{U}^T \mathbf{U} = \mathbf{I}_k} \, \text{tr}(\mathbf{U}^T \mathbf{S}_b \mathbf{U} - \beta\,(\mathbf{U}^T \mathbf{S}_w \mathbf{U})) \end{equation}

after factoring out $ \mathbf{U}^T $ and $ \mathbf{U} $ the equation denotes to

\begin{equation} \hat{\mathbf{U}} = \arg\max\limits_{\mathbf{U}^T \mathbf{U} = \mathbf{I}_k} \, \text{tr}(\mathbf{U}^T( \mathbf{S}_b - \beta\,\mathbf{S}_w) \mathbf{U})) \end{equation}

Taking the EVD of the scatter matrices

\begin{equation} \mathbf{S}_b - \beta\,\mathbf{S}_w = \mathbf{V} \mathbf{\Lambda} \mathbf{V}^T \end{equation}

one can see that the equation is maximized by setting $ \mathbf{U} = \mathbf{V} $.
(Refer to PCA for further proof).

+ Provide PYTHON code that creates the plots in accordance wiith the lab course by performing the dimensionality reduction with the described approach. In particular, choose $ k = 2 $ for the subspace dimension.

+ Choose 1, 2 and 3 as the digit classes and 1 to 1000 as the training samples from each class.

```python
N=1000
digit_prefix=['d1','d2','d3']
X=np.zeros((784,3*N))
C=np.zeros((3*N,3))
for i,dp in enumerate(digit_prefix):
    val_rgb=np.zeros((3,))
    val_rgb[i]=1.0
    for j in range(N):
        X[:,i*N+j]=np.float64(scm.imread('mnist/'+dp+'/'+dp+'_'+'%04d.png'%(j+1)).ravel())
        C[i*N+j]=val_rgb

mu=np.mean(X,axis=1)
X_c=X-np.expand_dims(mu,axis=1) # expand dims to ensure correct broadcasting

[U,Sigma,VT]=np.linalg.svd(X_c,full_matrices=False)

Scores=np.expand_dims(Sigma,axis=1)*VT

#reorder indexes for visualization purposes
idx_new=((np.arange(3*N).reshape(3,N).T).reshape(N,3)).ravel()
plt.scatter(Scores[0,idx_new],Scores[1,idx_new],c=C[idx_new])
plt.show()
```

```python
X_1=X[:,:N]
X_2=X[:,N:2*N]
X_3=X[:,2*N:3*N]

X1c=X_1-np.expand_dims(np.mean(X_1,axis=1), 1)
X2c=X_2-np.expand_dims(np.mean(X_2,axis=1), 1)
X3c=X_3-np.expand_dims(np.mean(X_3,axis=1), 1)

S_w=(np.dot(X1c,X1c.T)+np.dot(X2c,X2c.T)+np.dot(X3c,X3c.T))
```

```python
mu = np.mean(X, axis=1)
mu1 = np.mean(X_1, axis=1) - mu
mu2 = np.mean(X_2, axis=1) - mu
mu3 = np.mean(X_3, axis=1) - mu

Mu1 = np.dot(np.expand_dims(mu1,1), np.expand_dims(mu1.T, 0))
Mu2 = np.dot(np.expand_dims(mu2,1), np.expand_dims(mu2.T, 0))
Mu3 = np.dot(np.expand_dims(mu3,1), np.expand_dims(mu3.T, 0))

S_b = 1000*(Mu1 + Mu2 + Mu3)
```

+ Create four plots for $ \beta = 2 $, $ \beta = 4 $, $ \beta = 6 $ and $ \beta = 8 $, respectively. Give an interpretation of the results.

```python
k = 2
betas = np.array([2, 4, 6, 8])
fig, axs = plt.subplots(nrows=2, ncols=2, figsize = (15,10))
axs = axs.ravel()
for idx, beta in enumerate(betas):
    eigenvalues, V = np.linalg.eigh(S_b - beta*S_w)
    V = np.flip(V, axis = 1)
    U = V[:, :k]
    S = np.dot(U.T, X_c)
    axs[idx].scatter(S[0, idx_new], S[1, idx_new], c = C[idx_new])

plt.show()
```

As you can see, with growing $ \beta $, the classes are separated better. (For $ \beta \in [2, 4, 6] $)
Furthermore, the absolute value of the reduced data decreases rapidly.
For $ \beta $ greater than 6, the first two positive eigenvalues become very small, and for $ \beta = 8 $ the positive eigenvalues are of the magnitude $ 10^{-7} $.
Since $ \beta $ is a tuning parameter, we would choose it to be 6, because then the centers of the projected data are furthest away from each other and we could still apply a classifiction algorithm.

```python
# Examine eigenvalues
eigenvalues, V = np.linalg.eigh(S_b - 8*S_w, UPLO='L')
eigenvalues = eigenvalues[::-1]
eigenvalues[:k]
```

    array([  3.70740948e-07,   3.55661168e-07])

```python

```


# **MIT 6.036 Spring 2020: Lab 11**

This colab notebook provides code and a framework for sections 2 and 3 of the lab.

## **Setup**

First, download the code distribution for this homework that contains test cases and helper functions.

Run the next code block to download and import the code for this lab.

```
!rm -rf code_for_lab11*
!rm -rf models
!rm -rf *.zip*
!rm -rf __*
!wget --quiet --no-check-certificate https://introml.odl.mit.edu/cat-soop/_static/6.036/labs/lab11/code_for_lab_11.zip
!unzip code_for_lab_11.zip
!mv ./code_for_lab11/* ./
```

```
from sm import *
from util import *
from code_for_lab11 import *
import numpy as np
import _pickle as cPickle
m = cPickle.load(open('models/food_rnn.p', 'rb'))
```

## **Section 2**

### Generating sequences

We are interested in building an RNN model that can predict the next
element in a sequence (this is sometimes referred to as a "language"
model).    The particular form of the model that we will look at is:

<displaymath>
\begin{align}
x_t &= \phi(c_{t-1})\\
s_t &= \tanh(W^{ss} s_{t-1} + W^{sx} x_t + W^{ss}_0)\\
p_t &= \text{softmax}(W^o s_t + W^o_0)
\end{align}
</displaymath>

**Please refer to the lab's prompt for more definitions and details about this section.**

As we just said, you can find more details in the lab's prompt, but here some extra details:

Training is as follows:

 For each sequence in the input data, it feeds in character <math>t-1</math> from
  the training data and predicts character $t$.

Generation is as follows:

 If interactive is False: We start with the `start` symbol ('.'),

If interactive is True: We start with the symbol you enter in (which must be present in the training word or be the start symbol), it predicts

The model then predicts a next
  character based on the softmax distribution in the trained model,
  then it feeds that character into the model and repeats until an
  end symbol ('\n') is generated.

For each of

*   "aaaaaaaaaa"
*   "aabaaabbaaaababaabaa"
*   "abcdefghijklmnopqrstuvwxyz"
*   "abcabcabcabcabc"

train an RNN and assess the difficulty of learning each string. You may control *num_hidden* and *num_steps* to facilitate your training in the code below.

**Note**: The output of `test_word` is 100 sequences produced by running the generation process on a trained model.

```
test_word(word="aaaaaaaaaa", interactive=False, num_hidden=5, num_steps=1000, step_size=0.005)
```

```
test_word(word="aabaaabbaaaababaabaa", interactive = False, num_hidden=9, num_steps=15000, step_size=0.005)
```

```
test_word(word="abcdefghijklmnopqrstuvwxyz", num_hidden=5, num_steps=10000, step_size=0.005)
```

```
test_word(word="abcabcabcabcabc", interactive = False, num_hidden=5, num_steps=10000, step_size=0.005)
```

## Problem 2D

The *test_company_names* function uses a file companies.txt of company names for training and generates new names.

Try it in interactive mode; it's more fun. If you have time, experiment with different values of `num_steps`; more steps gives better results. But **be careful not to use too many steps** (no more than the preloaded 20000 steps).

**In running `test_company_names`, you can try with `train=True` where you can train the RNN yourself with this option, but note that this will take a few minutes to run, and you may run out of memory. If this occurs, please restart the kernel under "Runtime"/"Restart runtime". The default is to run  `test_company_names` with `train=False`; which simply loads a pre-trained model.**

Please note that when we train our models, we need to decide what will be our vocabulary size (i.e. how many classes/characters/words our model will learn to predict). Hence, it is possible that when you input a character/word that is out of that vocabulary (depending on how the model is trained -- character-based or word-based), the model crashes (in more sophisticated solutions we could use a `<unknown>` token to represent **tokens out of vocabulary.**)

These models are trained by minimizing NLL.

```
train = False

test_company_names(interactive=False, interactive_top5=False, num_hidden=150,
                   num_steps=20000, step_size=0.001, train=train)

#Trained with num_hidden=150, num_steps=20000,
#             step_size=.001, split=0
```

## Problem 2E

The *test_food* function uses a file food.txt of recipe names. Try it in interactive mode; it's more fun. Experiment with different values of num_steps if you have time; more steps gives better results. Note the difference between starting with a capital letter versus lower case letter.

```
train = False

test_food(interactive = False, interactive_top5 = False, num_hidden=150,
          num_steps=20000, step_size=0.001, train=train)

#Trained with num_hidden = 150, num_steps = 20000,
#             step_size = .001, split=0
```

## Problem 2F

Now run one of test_food, test_company_names with interactive_top5=True; play with them for a while. Can you notice the mechanism of how these functions generate their output?

Use the cell below to run one of  test_food, test_company_names with interactive_top5=True!

- What is the mechanism by which these RNNs generate their output? More specifically, what character does the trained RNN seem to output at each location?
- Also, what would be a reasonable criterion for the above-trained RNNs to stop generating more characters? When does it seem to stop? Be prepared to explain this during checkoff.

```
train = False

#test_food(interactive = True, interactive_top5 = True, num_hidden=150,
#          num_steps=20000, step_size=0.001, train=train)

# Pre trained with split=0, num_hidden = 150,
#                   num_steps = 20000, step_size = .001,
```

```
train = False

#test_company_names(interactive = True, interactive_top5 = True, num_hidden=150,
#                  num_steps=20000, step_size=0.001,train=train)

# Pre trained with split=0, num_hidden = 150,
#                   num_steps = 20000, step_size = .001,
```


# 11 ODE integrators: Verlet (Students)
This notebook contains incomplete code. Complete it (and `integrators2.py`) during class. After the lesson, complete solutions will be posted.

```python
from importlib import reload

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')
```

## Velocity Verlet

Use expansion *forward* and *backward* (!) in time (Hamiltons (i.e. Newton without friction) equations are time symmetric)

\begin{align}
r(t + \Delta r) &\approx r(t) + \Delta t\, v(t) + \frac{1}{2m} \Delta t^2 F(t)\\
r(t) &\approx r(t + \Delta t) - \Delta t\, v(t + \Delta t) + \frac{1}{2m} \Delta t^2 F(t+\Delta t)
\end{align}

Solve for $v$:
\begin{align}
v(t+\Delta t) &\approx v(t) + \frac{1}{2m} \Delta t \big(F(t) + F(t+\Delta t)\big)
\end{align}

Complete **Velocity Verlet** integrator consists of the first and third equation.

In practice, split into three steps (calculate the velocity at the half time step):
\begin{align}
v(t+\frac{\Delta t}{2}) &= v(t) + \frac{\Delta t}{2} \frac{F(t)}{m} \\
r(t + \Delta r) &= r(t) + \Delta t\, v(t+\frac{\Delta t}{2})\\
v(t+\Delta t) &= v(t+\frac{\Delta t}{2}) + \frac{\Delta t}{2} \frac{F(t+\Delta t)}{m}
\end{align}

When writing production-level code, remember to re-use $F(t+\Delta t)$ als the "new" starting $F(t)$ in the next iteration (and don't recompute).

### Integration of planetary motion
Gravitational potential energy:
$$
U(r) = -\frac{GMm}{r}
$$
with $r$ the distance between the two masses $m$ and $M$.

#### Central forces
$$
U(\mathbf{r}) = f(r) = f(\sqrt{\mathbf{r}\cdot\mathbf{r}})\\
\mathbf{F} = -\nabla U(\mathbf{r}) = -\frac{\partial f(r)}{\partial r} \, \frac{\mathbf{r}}{r}
$$

#### Force of gravity
\begin{align}
\mathbf{F} &= -\frac{G m M}{r^2} \hat{\mathbf{r}}\\
\hat{\mathbf{r}} &= \frac{1}{\sqrt{x^2 + y^2}} \left(\begin{array}{c} x \\ y \end{array}\right)
\end{align}

#### Integrate simple planetary orbits
Set $M = 1$ (one solar mass) and $m = 3.003467×10^{-6}$ (one Earth mass in solar masses) and try initial conditions

\begin{alignat}{1}
x(0) &= 1,\quad  y(0) &= 0\\
v_x(0) &= 0,\quad  v_y(0) &= 6.179
\end{alignat}

Note that we use the following units:
* length in astronomical units (1 AU = 149,597,870,700 m )
* mass in solar masses (1 $M_☉ = 1.988435×10^{30}$ kg)
* time in years (1 year = 365.25 days, 1 day = 86400 seconds)

In these units, the gravitational constant is $G = 4\pi^2$ (in SI units $G = 6.674×10^{-11}\, \text{N}\cdot\text{m}^2\cdot\text{kg}^{-2}$).

```python
M_earth = 3.003467e-6
M_sun = 1.0
```

```python
G_grav = 4*np.pi**2

def F_gravity(r, m=M_earth, M=M_sun):
    # IMPLEMENT ME
    raise NotImplementedError

def U_gravity(r, m=M_earth, M=M_sun):
    return -G_grav*m*M/np.sqrt(np.sum(r*r))
```

Let's now integrate the equations of motions under gravity with the **Velocity Verlet** algorithm:

```python
x0 = np.array([1, 0])
v0 = np.array([0, 6.179])

# IMPLEMENT ME
# nsteps: number of integration steps
# r: array to hold (x,y) for all time steps
# v: array to hold (vx, vy) for all time steps
#
# integrate with Velocity Verlet
```

Plot the orbit, i.e., all points (x, y).

```python
# plot orbit
```

## Velocity Verlet vs RK4: Energy conservation
Assess the stability of `rk4` and `Velocity Verlet` by checking energy conservation over longer simulation times.

The file `integrators2.py` contains almost all code that you will need.

### Implement gravity force in `integrators2.py`
Add `F_gravity` to the `integrators2.py` module. Use the new function `unitvector()`.

### Planetary orbits with `integrators2.py`

```python
r0 = np.array([1, 0])
v0 = np.array([0, 6.179])
```

```python
import integrators2
from importlib import reload
reload(integrators2)
```

Use the new function `integrators2.integrate_newton_2d()` to integrate 2d coordinates.

#### RK4

```python
trk4, yrk4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=integrators2.F_gravity,
                                       integrator=integrators2.rk4)
```

```python
rxrk4, ryrk4 = yrk4[:, 0, 0], yrk4[:, 0, 1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxrk4, ryrk4)
```

```python
integrators2.analyze_energies(trk4, yrk4, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(trk4),
        integrators2.energy_conservation(trk4, yrk4, integrators2.U_gravity, m=M_earth)))
```

#### Euler

```python
te, ye = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                         h=0.01,
                            force=F_gravity,
                            integrator=integrators2.euler)
rex, rey = ye[:, 0].T
```

```python
ax = plt.subplot(1,1,1)
ax.plot(rx, ry, label="RK4")
ax.plot(rex, rey, label="Euler")
ax.legend(loc="best")
ax.set_aspect(1)
```

```python
integrators2.analyze_energies(te, ye, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Euler for {} steps: {}".format(
        len(te),
        integrators2.energy_conservation(te, ye, integrators2.U_gravity, m=M_earth)))
```

*Euler* is just awful... but we knew that already.

#### Velocity Verlet

```python
tv, yv = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
rxv, ryv = yv[:, 0].T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxv, ryv, label="velocity Verlet")
ax.plot(rxrk4, ryrk4, label="RK4")
ax.legend(loc="best")
```

```python
integrators2.analyze_energies(tv, yv, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv),
        integrators2.energy_conservation(tv, yv, integrators2.U_gravity, m=M_earth)))
```

*Velocity Verlet* only has moderate accuracy, especially when compared to *RK4*.

However, let's look at energy conservation over longer times:

#### Longer time scale stability
Run RK4 and Velocity Verlet for longer.

```python
tv2, yv2 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv2),
        integrators2.energy_conservation(tv2, yv2, integrators2.U_gravity, m=M_earth)))
```

```python
t4, y4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.rk4)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(t4),
        integrators2.energy_conservation(t4, y4, integrators2.U_gravity, m=M_earth)))
```

Velocity Verlet shows **good long-term stability** but relative low precision. On the other hand, RK4 has high precision but the accuracy decreases over time.

* Use a *Verlet* integrator when energy conservation is important and long term stability is required (e.g. molecular dynamics simulations). It is generally recommended to use an integrator that conserves some of the inherent symmetries and structures of the governing physical equations (e.g. for Hamilton's equations of motion, time reversal symmetry and the symplectic and area-preserving structure).
* Use *RK4* for high short-term accuracy (but may be difficult to know what "short term" should mean) or when solving general differential equations.

```python

```