# Symbolic Computation
Symbolic computation deals with symbols, representing them exactly, instead of numerical approximations (floating point).

We will start with the following [borrowed](https://docs.sympy.org/latest/tutorial/intro.html) tutorial to introduce the concepts of SymPy. Devito uses SymPy heavily and builds upon it in its DSL.

```python
import math

math.sqrt(3)
```

    1.7320508075688772

```python
math.sqrt(8)
```

    2.8284271247461903

$\sqrt(8) = 2\sqrt(2)$, but it's hard to see that here

```python
import sympy
sympy.sqrt(3)
```

$\displaystyle \sqrt{3}$

SymPy can even simplify symbolic computations

```python
sympy.sqrt(8)
```

$\displaystyle 2 \sqrt{2}$

```python
from sympy import symbols
x, y = symbols('x y')
expr = x + 2*y
expr

```

$\displaystyle x + 2 y$

Note that simply adding two symbols creates an expression. Now let's play around with it.

```python
expr + 1
```

$\displaystyle x + 2 y + 1$

```python
expr - x
```

$\displaystyle 2 y$

Note that `expr - x` was not `x + 2y -x`

```python
x*expr
```

$\displaystyle x \left(x + 2 y\right)$

```python
from sympy import expand, factor
expanded_expr = expand(x*expr)
expanded_expr
```

$\displaystyle x^{2} + 2 x y$

```python
factor(expanded_expr)
```

$\displaystyle x \left(x + 2 y\right)$

```python
from sympy import diff, sin, exp

diff(sin(x)*exp(x), x)
```

$\displaystyle e^{x} \sin{\left(x \right)} + e^{x} \cos{\left(x \right)}$

```python
from sympy import limit

limit(sin(x)/x, x, 0)
```

$\displaystyle 1$

### Exercise

Solve $x^2 - 2 = 0$ using sympy.solve

```python
# Type solution here
from sympy import solve
```

## Pretty printing

```python
from sympy import init_printing, Integral, sqrt

init_printing(use_latex=True)
```

```python
Integral(sqrt(1/x), x)
```

```python
from sympy import latex

latex(Integral(sqrt(1/x), x))
```

    '\\int \\sqrt{\\frac{1}{x}}\\, dx'

More symbols.
Exercise: fix the following piece of code

```python
# NBVAL_SKIP
# The following piece of code is supposed to fail as it is
# The exercise is to fix the code

expr2 = x + 2*y +3*z
```

### Exercise

Solve $x + 2*y + 3*z$ for $x$

```python
# Solution here
from sympy import solve
```

Difference between symbol name and python variable name

```python
x, y = symbols("y z")
```

```python
x
```

```python
y
```

```python
# NBVAL_SKIP
# The following code will error until the code in cell 16 above is
# fixed
z
```

Symbol names can be more than one character long

```python
crazy = symbols('unrelated')

crazy + 1
```

```python
x = symbols("x")
expr = x + 1
x = 2
```

What happens when I print expr now? Does it print 3?

```python
print(expr)
```

    x + 1

How do we get 3?

```python
x = symbols("x")
expr = x + 1
expr.subs(x, 2)
```

## Equalities

```python
x + 1 == 4
```

    False

```python
from sympy import Eq

Eq(x + 1, 4)
```

Suppose we want to ask whether $(x + 1)^2 = x^2 + 2x + 1$

```python
(x + 1)**2 == x**2 + 2*x + 1
```

    False

```python
from sympy import simplify

a = (x + 1)**2
b = x**2 + 2*x + 1

simplify(a-b)
```

### Exercise
Write a function that takes two expressions as input, and returns a tuple of two booleans. The first if they are equal symbolically, and the second if they are equal mathematically.

## More operations

```python
z = symbols("z")
expr = x**3 + 4*x*y - z
expr.subs([(x, 2), (y, 4), (z, 0)])
```

```python
from sympy import sympify

str_expr = "x**2 + 3*x - 1/2"
expr = sympify(str_expr)
expr
```

```python
expr.subs(x, 2)
```

```python
expr = sqrt(8)
```

```python
expr
```

```python
expr.evalf()
```

```python
from sympy import pi

pi.evalf(100)
```

```python
from sympy import cos

expr = cos(2*x)
expr.evalf(subs={x: 2.4})
```

### Exercise

```python
from IPython.core.display import Image
Image(filename='figures/comic.png')
```

Write a function that takes a symbolic expression (like pi), and determines the first place where 789 appears.
Tip: Use the string representation of the number. Python starts counting at 0, but the decimal point offsets this

## Solving an ODE

```python
from sympy import Function

f, g = symbols('f g', cls=Function)
f(x)
```

```python
f(x).diff()
```

```python
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x))
diffeq
```

```python
from sympy import dsolve

dsolve(diffeq, f(x))
```

## Finite Differences

```python

f = Function('f')
dfdx = f(x).diff(x)
dfdx.as_finite_difference()
```

```python
from sympy import Symbol

d2fdx2 = f(x).diff(x, 2)
h = Symbol('h')
d2fdx2.as_finite_difference(h)
```

Now that we have seen some relevant features of vanilla SymPy, let's move on to Devito, which could be seen as SymPy finite differences on steroids!


# Introducción a SymPy

__SymPy es una biblioteca de Python para matemática simbólica__. Apunta a convertirse en un sistema de algebra computacional (__CAS__) con todas sus prestaciones manteniendo el código tan simple como sea posible para manterlo comprensible y fácilmente extensible. SymPy está __escrito totalmente en Python y no requiere bibliotecas adicionales__. _Este proyecto comenzó en 2005, fue lanzado al público en 2007 y a él han contribuido durante estos años cientos de personas._

_ Otros CAS conocidos son Mathematica y Maple, sin embargo ambos son software privativo y de pago. [Aquí](https://github.com/sympy/sympy/wiki/SymPy-vs.-Maple) puedes encontrar una comparativa de SymPy con Maple. _

Hoy veremos cómo:

* Crear símbolos y expresiones.
* Manipular expresiones (simplificación, expansión...)
* Calcular derivadas e integrales.
* Límites y desarrollos en serie.
* Resolución de ecuaciones.
* Resolción de EDOs.
* Matrices

Sin embargo, SymPy no acaba aquí ni mucho menos...

## Documentación & SymPy Live Shell

```python
from IPython.display import HTML
HTML('')
```

## SymPy Gamma

```python
HTML('')
```

## Creación de símbolos

Lo primero, como siempre, es importar aquello que vayamos a necesitar. La manera usual de hacerlo con SymPy es importar la función `init_session`:
```
from sympy import init_session
init_session(use_latex=True)```

 Esta función ya se encarga de importar todas las funciones básicas y preparar las salidas gráficas. Sin embargo, en este momento, esta función se encuentra en mantenimiento para su uso dentro de los notebooks por lo que activaremos la salida gráfica e importaremos las funciones de la manera usual. Puedes consultar el estado de la corrección en: https://github.com/sympy/sympy/pull/13300 y https://github.com/sympy/sympy/issues/13319 .

El comando `init_session` llevaría a cabo algunas acciones por nostros:

* Gracias a `use_latex=True` obtenemos la salida en $\LaTeX$.
* __Crea una serie de variables__ para que podamos ponernos a trabajar en el momento.

Estas capacidades volverán a estar disponibles cuando el problema se corrija.

```python
from sympy import init_printing
```

```python
init_printing()
```

```python
# aeropython: preserve
from sympy import (symbols, pi, I, E, cos, sin, exp, tan, simplify, expand, factor, collect,
                   apart, cancel, expand_trig, diff, Derivative, Function, integrate, limit,
                   series, Eq, solve, dsolve, Matrix, N)
```

Nota:
En Python, no se declaran las variables, sin embargo, no puedes usar una hasta que no le hayas asignado un valor. Si ahora intentamos crear una variable `a` que sea `a = 2 * b`, veamos qué ocurre:

```python
# Intentamos usar un símbolo que no hemos creado
a = 2 * b
```

Como en `b` no había sido creada, Python no sabe qué es `b`.

Esto mismo nos ocurre con los símbolos de SymPy. __Antes de usar una variable, debo decir que es un símbolo y asignárselo:__

```python
# Creamos el símbolo a
a = symbols('a')
a
```

```python
# Número pi
(a + pi) ** 2
```

```python
# Unidad imaginaria
a + 2 * I
```

```python
# Número e
E
```

```python
# Vemos qué tipo de variable es a
type(a)
```

    sympy.core.symbol.Symbol

Ahora ya podría crear `b = 2 * a`:

```python
b = 2 * a
b
```

```python
type(b)
```

    sympy.core.mul.Mul

¿Qué está ocurriendo? Python detecta que a es una variable de tipo `Symbol` y al multiplicarla por `2` devuelve una variable de Sympy.

Como Python permite que el tipo de una variable cambie, __si ahora le asigno a `a` un valor float deja de ser un símbolo.__

```python
a = 2.26492
a
```

```python
type(a)
```

    float

---
__Las conclusiones son:__

* __Si quiero usar una variable como símbolo debo crearla previamente.__
* Las operaciones con símbolos devuelven símbolos.
* Si una varibale que almacenaba un símbolo recibe otra asignación, cambia de tipo.

---

__Las variables de tipo `Symbol` actúan como contenedores en los que no sabemos qué hay (un real, un complejo, una lista...)__. Hay que tener en cuenta que: __una cosa es el nombre de la variable y otra el símbolo con el que se representa__.

```python
#creación de símbolos
coef_traccion = symbols('c_T')
coef_traccion
```

Incluso puedo hacer cosas raras como:

```python
# Diferencia entre variable y símbolo
a = symbols('b')
a
```

Además, se pueden crear varos símbolos a la vez:

```python
x, y, z, t = symbols('x y z t')
```

y símbolos griegos:

```python
w = symbols('omega')
W = symbols('Omega')
w, W
```

_Fuente: Documentación oficial de SymPy_

__Por defecto, SymPy entiende que los símbolos son números complejos__. Esto puede producir resultados inesperados ante determinadas operaciones como, por ejemplo, lo logaritmos. __Podemos indicar que la variable es real, entera... en el momento de la creación__:

```python
# Creamos símbolos reales
x, y, z, t = symbols('x y z t', real=True)
```

```python
# Podemos ver las asunciones de un símbolo
x.assumptions0
```

    {'commutative': True,
     'complex': True,
     'hermitian': True,
     'imaginary': False,
     'real': True}

## Expresiones

Comencemos por crear una expresión como: $\cos(x)^2+\sin(x)^2$

```python
expr = cos(x)**2 + sin(x)**2
expr
```

### `simplify()`

Podemos pedirle que simplifique la expresión anterior:

```python
simplify(expr)
```

En este caso parece estar claro lo que quiere decir más simple, pero como en cualquier _CAS_ el comando `simplify` puede no devolvernos la expresión que nosotros queremos. Cuando esto ocurra necesitaremos usar otras instrucciones.

### `.subs()`

En algunas ocasiones necesitaremos sustituir una variable por otra, por otra expresión o por un valor.

```python
expr
```

```python
# Sustituimos x por y ** 2
expr.subs(x, y**2)
```

```python
# ¡Pero la expresión no cambia!
expr
```

```python
# Para que cambie
expr = expr.subs(x, y**2)
expr
```

Cambia el `sin(x)` por `exp(x)`

```python
expr.subs(sin(x), exp(x))
```

Particulariza la expresión $sin(x) + 3 x $ en $x = \pi$

```python
(sin(x) + 3 * x).subs(x, pi)
```

__Aunque si lo que queremos es obtener el valor numérico lo mejor es `.evalf()`__

```python
(sin(x) + 3 * x).subs(x, pi).evalf(25)
```

```python
#ver pi con 25 decimales
pi.evalf(25)
```

```python
#el mismo resultado se obtiene ocn la función N()
N(pi,25)
```

# Simplificación

SymPy ofrece numerosas funciones para __simplificar y manipular expresiones__. Entre otras, destacan:

* `expand()`
* `factor()`
* `collect()`
* `apart()`
* `cancel()`

Puedes consultar en la documentación de SymPy lo que hace cada una y algunos ejemplos. __Existen también funciones específicas de simplificación para funciones trigonométricas, potencias y logaritmos.__ Abre [esta documentación](http://docs.sympy.org/latest/tutorial/simplification.html) si lo necesitas.

##### ¡Te toca!

Pasaremos rápidamente por esta parte, para hacer cosas "más interesantes". Te proponemos algunos ejemplos para que te familiarices con el manejor de expresiones:

__Crea las expresiones de la izquierda y averigua qué función te hace obtener la de la derecha:__

expresión 1| expresión 2
:------:|:------:
$\left(x^{3} + 3 y + 2\right)^{2}$    |    $x^{6} + 6 x^{3} y + 4 x^{3} + 9 y^{2} + 12 y + 4$
$\frac{\left(3 x^{2} - 2 x + 1\right)}{\left(x - 1\right)^{2}} $ | $3 + \frac{4}{x - 1} + \frac{2}{\left(x - 1\right)^{2}}$
$x^{3} + 9 x^{2} + 27 x + 27$         |    $\left(x + 3\right)^{3}$
$\sin(x+2y)$                          |    $\left(2 \cos^{2}{\left (y \right )} - 1\right) \sin{\left (x \right )} + 2 \sin{\left (y \right )} \cos{\left (x \right )} \cos{\left (y \right )}$

```python
#1
expr1 = (x ** 3 + 3 * y + 2) ** 2
expr1
```

```python
expr1_exp = expr1.expand()
expr1_exp
```

```python
#2
expr2 = (3 * x ** 2 - 2 * x + 1) / (x - 1) ** 2
expr2
```

```python
expr2.apart()
```

```python
#3
expr3 = x ** 3 + 9 * x ** 2 + 27 * x + 27
expr3
```

```python
expr3.factor()
```

```python
#4
expr4 = sin(x + 2 * y)
expr4
```

```python
expand(expr4)
```

```python
expand_trig(expr4)
```

```python
expand(expr4, trig=True)
```

# Derivadas e integrales

Puedes derivar una expresion usando el método `.diff()` y la función `dif()`

```python
#creamos una expresión
expr = cos(x)

#obtenemos la derivada primera con funcion
diff(expr, x)
```

```python
#utilizando método
expr.diff(x)
```

__¿derivada tercera?__

```python
expr.diff(x, x, x)
```

```python
expr.diff(x, 3)
```

__¿varias variables?__

```python
expr_xy = y ** 3 * sin(x) ** 2 + x ** 2 * cos(y)
expr_xy
```

```python
diff(expr_xy, x, 2, y, 2)
```

__Queremos que la deje indicada__, usamos `Derivative()`

```python
Derivative(expr_xy, x, 2, y)
```

__¿Será capaz SymPy de aplicar la regla de la cadena?__

```python
# Creamos una función F
F = Function('F')
F(x)
```

```python
# Creamos una función G
G = Function('G')
G(x)
```

$$\frac{d}{d x} F{\left (G(x) \right )} $$

```python
# Derivamos la función compuesta F(G(x))
F(G(x)).diff(x)
```

$$\frac{d}{d x} G{\left (x \right )} \left. \frac{d}{d \xi_{1}} F{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=G{\left (x \right )} }}$$

En un caso en el que conocemos las funciones:

```python
# definimos una f
f = 2 * y * exp(x)
f
```

```python
# definimos una g(f)
g = f **2 * cos(x) + f
g
```

```python
#la derivamos
diff(g,x)
```

##### Te toca integrar

__Si te digo que se integra usando el método `.integrate()` o la función `integrate()`__. ¿Te atreves a integrar estas casi inmediatas...?:

$$\int{\cos(x)^2}dx$$
$$\int{\frac{dx}{\sin(x)}}$$
$$\int{\frac{dx}{(x^2+a^2)^2}}$$

```python
int1 = cos(x) ** 2
integrate(int1)
```

```python
int2 =  1 / sin(x)
integrate(int2)
```

```python
x, a = symbols('x a', real=True)

int3 = 1 / (x**2 + a**2)**2
integrate(int3, x)
```

# Límites

Calculemos este límite sacado del libro _Cálculo: definiciones, teoremas y resultados_, de Juan de Burgos:

$$\lim_{x \to 0} \left(\frac{x}{\tan{\left (x \right )}}\right)^{\frac{1}{x^{2}}}$$

Primero creamos la expresión:

```python
x = symbols('x', real=True)
expr = (x / tan(x)) ** (1 / x**2)
expr
```

Obtenemos el límite con la función `limit()` y si queremos dejarlo indicado, podemos usar `Limit()`:

```python
limit(expr, x, 0)
```

# Series

Los desarrollos en serie se pueden llevar a cabo con el método `.series()` o la función `series()`

```python
#creamos la expresión
expr = exp(x)
expr
```

```python
#la desarrollamos en serie
series(expr)
```

Se puede especificar el número de términos pasándole un argumento `n=...`. El número que le pasemos será el primer término que desprecie.

```python
# Indicando el número de términos
series(expr, n=10)
```

Si nos molesta el $\mathcal{O}(x^{10})$ lo podemos quitar con `removeO()`:

```python
series(expr, n=10).removeO()
```

```python
series(sin(x), n=8, x0=pi/3).removeO().subs(x, x-pi/3)
```

---

## Resolución de ecuaciones

Como se ha mencionado anteriormente las ecuaciones no se pueden crear con el `=`

```python
#creamos la ecuación
ecuacion = Eq(x ** 2 - x, 3)
ecuacion
```

```python
# También la podemos crear como
Eq(x ** 2 - x -3)
```

```python
#la resolvemos
solve(ecuacion)
```

Pero la gracia es resolver con símbolos, ¿no?
$$a e^{\frac{x}{t}} = C$$

```python
# Creamos los símbolos y la ecuación
a, x, t, C = symbols('a, x, t, C', real=True)
ecuacion = Eq(a * exp(x/t), C)
ecuacion
```

```python
# La resolvemos
solve(ecuacion ,x)
```

Si consultamos la ayuda, vemos que las posibilidades y el número de parámetros son muchos, no vamos a entrar ahora en ellos, pero ¿se ve la potencia?

## Ecuaciones diferenciales

Tratemos de resolver, por ejemplo:

$$y{\left (x \right )} + \frac{d}{d x} y{\left (x \right )} + \frac{d^{2}}{d x^{2}}  y{\left (x \right )} = \cos{\left (x \right )}$$

```python
x = symbols('x')
f = Function('y')
ecuacion_dif = Eq(y(x).diff(x,2) + y(x).diff(x) + y(x), cos(x))
ecuacion_dif
```

```python
#resolvemos
dsolve(ecuacion_dif, f(x))
```

# Matrices

```python
#creamos una matriz llena de símbolos
a, b, c, d = symbols('a b c d')
A = Matrix([
            [a, b],
            [c, d]
    ])
A
```

$$\left[\begin{matrix}a & b\\c & d\end{matrix}\right]$$

```python
#sacamos autovalores
A.eigenvals()
```

```python
#inversa
A.inv()
```

$$\left[\begin{matrix}\frac{d}{a d - b c} & - \frac{b}{a d - b c}\\- \frac{c}{a d - b c} & \frac{a}{a d - b c}\end{matrix}\right]$$

```python
#elevamos al cuadrado la matriz
A ** 2
```

$$\left[\begin{matrix}a^{2} + b c & a b + b d\\a c + c d & b c + d^{2}\end{matrix}\right]$$

---

_ Esto ha sido un rápido recorrido por algunas de las posibilidades que ofrece SymPy . El cálculo simbólico es un terreno díficil y este joven paquete avanza a pasos agigantados gracias a un grupo de desarrolladores siempre dispuestos a mejorar y escuchar sugerencias. Sus posibilidades no acaban aquí. En la siguiente clase presentaremos el paquete `mechanics`, pero además cuenta con herramientas para geometría, mecánica cuántica, teoría de números, combinatoria... Puedes echar un ojo [aquí](http://docs.sympy.org/latest/modules/index.html). _

Si te ha gustado esta clase:

Tweet

---

####
¡Síguenos en Twitter!

###### Follow @Pybonacci   Follow @Alex__S12   Follow @newlawrence

#####
Curso AeroPython por Juan Luis Cano Rodriguez y Alejandro Sáez Mollejo se distribuye bajo una Licencia Creative Commons Atribución 4.0 Internacional.

#####

---
_Las siguientes celdas contienen configuración del Notebook_

_Para visualizar y utlizar los enlaces a Twitter el notebook debe ejecutarse como [seguro](http://ipython.org/ipython-doc/dev/notebook/security.html)_

    File > Trusted Notebook

```python
%%html
<a href="https://twitter.com/Pybonacci" class="twitter-follow-button" data-show-count="false">Follow @Pybonacci</a>

```

Follow @Pybonacci

```python
# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = '../styles/aeropython.css'
HTML(open(css_file, "r").read())
```

/* This template is inspired in the one used by Lorena Barba
in the numerical-mooc repository: https://github.com/numerical-mooc/numerical-mooc
We thank her work and hope you also enjoy the look of the notobooks with this style */

<link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro|Josefin+Sans:400,700,400italic|Ubuntu+Condensed' rel='stylesheet' type='text/css'>

El estilo se ha aplicado =)


```python
%matplotlib inline
import numpy as np
from numpy import max, abs, dot, array
import pandas as pd
import matplotlib.pyplot as plt

```

### Multiplying by complex numbers is a rotation
This is a minor re-wording of the bread-and-butter knowledge in signal processing that (using $i=e^{j \frac {\pi} 2}$) multiplication of a signal with a complex exponential results in shifting the phase of the signal. If we view the signal as a vector in polar form in the complex plane, the multiplication by a complex exponential is equivalent to applying a rotation matrix constructed by the desired angle of the phase shift.

```python
complex_nums = array([
    -5+4j,
    -5+3j, -4+3j, -3+3j, -2+3j, -1+3j,
                         -2+2j,
])

def plot_axes(length):
    plt.plot([0, 0], [-length, length], color='k')
    plt.plot([-length, length], [0, 0], color='k')

def plot_complex(nums):
    plt.scatter(nums.real, nums.imag)
    lim = 1 + max([max(abs(nums.real)), max(abs(nums.imag))])
    plot_axes(lim)
    plt.axis([-lim, lim, -lim, lim])

# multiplying by i is a rotation counter-clockwise by 45 degrees
plt.figure()
inds = [1, 3, 4, 2]
for i in range(1, 5):
    plt.subplot(2, 2, inds[i-1])
    plot_complex(complex_nums)
    complex_nums *= 1j
```

### Complex Exponentials and Rotation Matrices
Let's bridge the gap that many face when asking signal processing folks and graphics folks about rotations. We can do this by showing how to rotate a vector $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$ in $\mathbb{C}$ by 45$^{\circ}$ by 1) multiplication by a complex exponential and 2) multiplication by a rotation matrix.

#### Complex Exponential (Signal Processing)
By Euler's method, we know that $i=e^{j \frac {\pi} 2}$. Let's also represent our vector $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$ in it's rectangular form as $1 + 0j$, and then convert it to its polar form as $e^{j0}$. Multiplying these two, we get

$$
\begin{align}
    e^{j \frac \pi 2}e^{j0} = e^{j \frac \pi 2 + 0}
            &= e^{j \frac \pi 2} \\
            &= 0 + 1j \\
            &= \begin{bmatrix}0 \\ 1 \end{bmatrix}
\end{align}
$$

#### Rotation Matrix
The canonical rotation matrix for a desired angle $\theta$ for a two-dimensional space is
$$
\begin{bmatrix}
cos \theta  & -sin \theta \\
sin \theta  & cos \theta
\end{bmatrix}
$$

So let's take $\mathbb{C}$ for our space. Since the current rotation is $\theta = 45^{\circ}$, the corresponding rotation matrix is
$$
\begin{bmatrix}
0  & -1 \\
1  & 0
\end{bmatrix}
$$

Applying the rotation matrix to the original vector, we have
$$
\begin{bmatrix}
0  & -1 \\
1  & 0
\end{bmatrix}
\begin{bmatrix}
1 \\
0
\end{bmatrix}
=
\begin{bmatrix}
0 \\
1
\end{bmatrix}
$$

#### The Bridge
We can start with the rotation matrix representation and bridge over to the phasor (complex exponential) representation. The rotation of angle $\theta$ on an arbitrary vector $c \in \mathbb{C}, c = c_r + j c_i$ is
$$
\begin{bmatrix}
cos \theta  & -sin \theta \\
sin \theta  & cos \theta
\end{bmatrix}
\begin{bmatrix}
c_r \\
c_i
\end{bmatrix}
=
\begin{bmatrix}
c_r cos \theta - c_i sin \theta \\
c_r sin \theta + c_i cos \theta
\end{bmatrix}
$$

Remember that this is a representation of a vector in $\mathbb{C}$, so $\begin{bmatrix}x \\ y \end{bmatrix} = x + j y$. Therefore,
$$
\begin{align}
    \begin{bmatrix}
    c_r cos \theta - c_i sin \theta \\
    c_r sin \theta + c_i cos \theta
    \end{bmatrix}
    &= c_r cos \theta - c_i sin \theta + j(c_r sin \theta + c_i cos \theta) \\
    &= c_r cos \theta + j^2 c_i sin \theta + j c_r sin \theta + j c_i cos \theta \\
    &= (c_r + j c_i)(cos \theta + j sin \theta) \\
    &= c e^{j\theta}
\end{align}
$$

So the real bridge here was that rotation via the complex exponential was accomplished using polar coordinates, while using the rotation matrix worked in rectangular coordinates. By converting the coordinate system of one, you arrive at the other.

```python

```


# Extended Likelihood

## Unbinned Extended Likelihood

Let $x$ be a random variable distributed according to a p.d.f. $~f\left(x\,\middle|\,\vec{\theta}\right)$,

\begin{equation*}
x \sim f\left(x\,\middle|\,\vec{\theta}\right),
\end{equation*}

with $n$ observations, $\vec{x} = \left(x_1, \cdots, x_n\right)$, and $m$ unkown parameters, $\vec{\theta} = \left(\theta_1, \cdots, \theta_m\right)$. The likelihood would normally then be

$$
L\left(\vec{\theta}\right) = \prod_{i=1}^{n} f\left(x_i; \vec{\theta}\right).
$$

However, if $n$ itself is a Poisson random variable with mean $\nu$,

$$
n \sim \text{Pois}\left(n \,\middle|\, \nu\right),
$$

then it follows that

\begin{align}
L\left(\nu; \vec{\theta}\right) &= \text{Pois}\left(n; \nu\right) \prod_{i=1}^{n} f\left(x_i; \vec{\theta}\right) \notag\\
    &= \frac{\nu^{n}\,e^{-\nu}}{n!} \prod_{i=1}^{n} f\left(x_i; \vec{\theta}\right) \notag\\
    &= \frac{e^{-\nu}}{n!} \prod_{i=1}^{n} \nu\, f\left(x_i; \vec{\theta}\right).
%\label{eq_extended-likelihood}
\end{align}

This equation is known as the "extended likelihood function", as we have "extended" the information encoded in the likelihood to include the expected number of events — a quantity of great importance to physicists. It can be see from inspection though that the extended likelihood still follows the form of a likelihood, so no different treatment is required in finding its MLE estimators.

### $\nu$ is dependent on $\vec{\theta}$

In the instance that $\nu$ is a function of $\vec{\theta}$, $\nu = \nu\left(\vec{\theta}\right)$, then

$$
L\left(\vec{\theta}\right) = \frac{e^{-\nu\left(\vec{\theta}\right)}}{n!} \prod_{i=1}^{n} \nu\left(\vec{\theta}\right)\, f\left(x_i; \vec{\theta}\right),
$$

such that

$$
\ln L\left(\vec{\theta}\right) = - \nu\left(\vec{\theta}\right) - \ln n! + \sum_{i=1}^{n} \ln\left(\nu\left(\vec{\theta}\right)\, f\left(x_i; \vec{\theta}\right)\right),
$$

where $n$ is a constant of the data, and so will have no effect on finding the estimators of any parameters, leading it to be safely ignored. Thus,

\begin{equation}
\boxed{-\ln L\left(\vec{\theta}\right) = \nu\left(\vec{\theta}\right) -\sum_{i=1}^{n} \ln\left(\nu\left(\vec{\theta}\right)\, f\left(x_i; \vec{\theta}\right)\right)}\,.
\end{equation}

Note that as the resultant estimators, $\hat{\vec{\theta}}$, exploit information from both $n$ and $x$ this should generally lead to smaller variations for $\hat{\vec{\theta}}$.

### $\nu$ is independent of $\vec{\theta}$

In the instance that $\nu$ is independent of $\vec{\theta}$,
$$
L\left(\nu; \vec{\theta}\right) = \frac{e^{-\nu}}{n!} \prod_{i=1}^{n} \nu\, f\left(x_i; \vec{\theta}\right),
$$
then

\begin{split}
\ln L\left(\nu; \vec{\theta}\right) &= - \nu - \ln n! + \sum_{i=1}^{n} \ln\left(\nu\, f\left(x_i; \vec{\theta}\right)\right)\\
    &= - \nu  + \sum_{i=1}^{n} \left(\ln\nu + \ln f\left(x_i; \vec{\theta}\right)\right) - \ln n! \\
    &= - \nu + n \ln\nu  + \sum_{i=1}^{n} \ln f\left(x_i; \vec{\theta}\right) - \ln n!\,,
\end{split}
such that
\begin{equation}
\boxed{-\ln L\left(\nu; \vec{\theta}\right) = \nu - n \ln\nu  - \sum_{i=1}^{n} \ln f\left(x_i; \vec{\theta}\right)}\,.
\end{equation}

As $L$ is maximized with respect to a variable $\alpha$ when $−\ln L$ is minimized,

$$
\frac{\partial \left(-\ln L\right)}{\partial \alpha} = 0,
$$

then it is seen from

$$
\frac{\partial \left(-\ln L\left(\nu; \vec{\theta}\right)\right)}{\partial \nu} = 1 - \frac{n}{\nu} = 0,
$$

that the maximum likelihood estimator for $\nu$ is
\begin{equation}
\hat{\nu} = n\,,
\end{equation}

and that
$$
\frac{\partial \left(-\ln L\left(\nu; \vec{\theta}\right)\right)}{\partial \theta_j} = 0,
$$
results in the the same estimators $\hat{\vec{\theta}}$ as in the "usual" maximum likelihood case.

If the p.d.f. is of the form of a mixture model,
$$
f\left(x; \vec{\theta}\right) = \sum_{i=1}^{m} \theta_i\, f_i\left(x\right),
$$
and an estimate of the weights is of interest, then as the parameters are not fully independent, given the constraint
$$
\sum_{i=1}^{m} \theta_i = 1,
$$
then one of the $m$ parameters can be replaced with
$$
1 - \sum_{i=1}^{m-1} \theta_i,
$$
so that the p.d.f. only constrains $m-1$ parameters. This then allows the the likelihood to be constructed that allows to find the estimator for the unconstrained parameter.

Equivalently, the extended likelihood function can be used, as

\begin{split}
\ln L\left(\nu; \vec{\theta}\right) &= - \nu + n \ln\nu  + \sum_{i=1}^{n} \ln f\left(x_i; \vec{\theta}\right) \\
    &= - \nu  + \sum_{i=1}^{n} \ln \left(\nu\,f\left(x_i; \vec{\theta}\right)\right)\\
    &= - \nu  + \sum_{i=1}^{n} \ln \left(\sum_{j=1}^{m} \nu\,\theta_j\, f_j\left(x_i\right)\right).
\end{split}

Letting $\mu_i$, the expected number of events of type $i$, be $\mu_i \equiv \theta_i \nu$, for $\vec{\mu} = \left(\mu_1, \cdots, \mu_m\right)$, then

$$
\ln L\left(\vec{\mu}\right) = - \sum_{j=1}^{m} \mu_j  + \sum_{i=1}^{n} \ln \left(\sum_{j=1}^{m} \mu_j\, f_j\left(x_i\right)\right).
$$

Here, $\vec{\mu}$ are unconstrained and all parameters are treated symetrically, such that $\hat{\mu_i}$ give the maximum likelihood estimator means of number of events of type $i$.

#### [Toy Example](http://www.physi.uni-heidelberg.de/~menzemer/Stat0708/statistik_vorlesung_7.pdf#page=10)

```python
import numpy as np
from scipy.optimize import minimize
```

```python
def NLL(x, n, S, B):
    nll = sum([(x[0] * S[meas] + B[meas]) - (n[meas] * np.log(x[0] * S[meas] + B[meas]))
               for meas in np.arange(0, len(n))])
    return nll
```

```python
n_observed = [6, 24]
f = np.array([1.])
S = [0.9, 4.0]
B = [0.2, 24.]

model = minimize(NLL, f, args=(n_observed, S, B), method='L-BFGS-B', bounds=[(0, 10)])
print("The MLE estimate for f: {f}".format(f=model.x[0]))
```

    The MLE estimate for f: 2.6155463202001847

#### HistFactory Example

Consider a single channel with one signal and one bakcagound contribution (and no systematics). For $n$ events, signal model $f_{S}(x_e)$, background model $f_{B}(x_e)$, $S$ expected signal events, $B$ expected backagound events, and signal fraciton $\mu$, a "marked Poisson model" [2] may be constructed, which treating the data as fixed results in the likelihood of

\begin{split}
L\left(\mu\right) &= \text{Pois}\left(n \,\middle|\, \mu S + B\right) \prod_{e=1}^{n} \frac{\mu S\, f_{S}\left(x_e\right) + B\, f_{B}\left(x_e\right)}{\mu S + B}\\
    &= \frac{\left(\mu S + B\right)^{n} e^{-\left(\mu S + B\right)}}{n!} \prod_{e=1}^{n} \frac{\mu S\, f_{S}\left(x_e\right) + B\, f_{B}\left(x_e\right)}{\mu S + B}\\
    &= \frac{e^{-\left(\mu S + B\right)}}{n!} \prod_{e=1}^{n} \left(\,\mu S\, f_{S}\left(x_e\right) + B\, f_{B}\left(x_e\right)\right),
\end{split}

and so

$$
-\ln L\left(\mu\right) = \left(\mu S + B\right) - \sum_{e=1}^{n} \ln \left(\,\mu S\, f_{S}\left(x_e\right) + B\, f_{B}\left(x_e\right)\right) + \underbrace{\ln n!}_{\text{constant}}.
$$

## Binned Extended Likelihood

## References and Acknowledgements
1. [Statistical Data Analysis](http://www.pp.rhul.ac.uk/~cowan/sda/), Glen Cowan, 1998
2. ROOT collaboration, K. Cranmer, G. Lewis, L. Moneta, A. Shibata and W. Verkerke, [_HistFactory: A tool for creating statistical models for use with RooFit and RooStats_](http://inspirehep.net/record/1236448), 2012.
3. [Vince Croft](https://www.nikhef.nl/~vcroft/), Discussions with the author at CERN, July 2017


# Numerical integrals and error

What about when we cannot integrate a function analytically? In other words, when there is no (obvious) closed-form solution. In these cases, we can use **numerical methods** to solve the problem.

Let's use this problem:
\begin{align}
\frac{dy}{dx} &= e^{-x^2} \\
y(x) &= \int e^{-x^2} dx + C
\end{align}

(You may recognize this as leading to the error function, $\text{erf}$:
$\frac{1}{2} \sqrt{\pi} \text{erf}(x) + C$,
so the exact solution to the integral over the range $[0,1]$ is 0.7468.)

```matlab
x = linspace(0, 1);
f = @(x) exp(-x.^2);
plot(x, f(x))
axis([0 1 0 1])
```

## Numerical integration: Trapezoidal rule

In such cases, we can find the integral by using the **trapezoidal rule**, which finds the area under the curve by creating trapezoids and summing their areas:
\begin{equation}
\text{area under curve} = \sum \left( \frac{f(x_{i+1}) + f(x_i)}{2} \right) \Delta x
\end{equation}

Let's see what this looks like with four trapezoids ($\Delta x = 0.25$):

```matlab
hold off
x = linspace(0, 1);
plot(x, f(x)); hold on
axis([0 1 0 1])

x = 0 : 0.25 : 1;

% plot the trapezoids
for i = 1 : length(x)-1
    xline = [x(i), x(i)];
    yline = [0, f(x(i))];
    line(xline, yline, 'Color','red','LineStyle','--')
    xline = [x(i+1), x(i+1)];
    yline = [0, f(x(i+1))];
    line(xline, yline, 'Color','red','LineStyle','--')
    xline = [x(i), x(i+1)];
    yline = [f(x(i)), f(x(i+1))];
    line(xline, yline, 'Color','red','LineStyle','--')
end
hold off
```

Now, let's integrate using the trapezoid formula given above:

```matlab
dx = 0.1;
x = 0.0 : dx : 1.0;

area = 0.0;
for i = 1 : length(x)-1
    area = area + (dx/2)*(f(x(i)) + f(x(i+1)));
end

fprintf('Numerical integral: %f\n', area)
exact = 0.5*sqrt(pi)*erf(1);
fprintf('Exact integral: %f\n', exact)
fprintf('Error: %f %%\n', 100.*abs(exact-area)/exact)
```

    Numerical integral: 0.746211
    Exact integral: 0.746824
    Error: 0.082126 %

We can see that using the trapezoidal rule, a numerical integration method, with an internal size of $\Delta x = 0.1$ leads to an approximation of the exact integral with an error of 0.08%.

You can make the trapezoidal rule more accurate by:

- using more segments (that is, a smaller value of $\Delta x$, or
- using higher-order polynomials (such as with Simpson's rules) over the simpler trapezoids.

First, how does reducing the segment size (step size) by a factor of 10 affect the error?

```matlab
dx = 0.01;
x = 0.0 : dx : 1.0;

area = 0.0;
for i = 1 : length(x)-1
    area = area + (dx/2)*(f(x(i)) + f(x(i+1)));
end

fprintf('Numerical integral: %f\n', area)
exact = 0.5*sqrt(pi)*erf(1);
fprintf('Exact integral: %f\n', exact)
fprintf('Error: %f %%\n', 100.*abs(exact-area)/exact)
```

    Numerical integral: 0.746818
    Exact integral: 0.746824
    Error: 0.000821 %

So, reducing our step size by a factor of 10 (using 100 segments instead of 10) reduced our error by a factor of 100!

## Numerical integration: Simpson's rule

We can increase the accuracy of our numerical integration approach by using a more sophisticated interpolation scheme with each segment. In other words, instead of using a straight line, we can use a polynomial. **Simpson's rule**, also known as Simpson's 1/3 rule, refers to using a quadratic polynomial to approximate the line in each segment.

Simpson's rule defines the definite integral for our function $f(x)$ from point $a$ to point $b$ as
\begin{equation}
\int_a^b f(x) \approx \frac{1}{6} \Delta x \left( f(a) + 4 f \left(\frac{a+b}{2}\right) + f(b) \right)
\end{equation}
where $\Delta x = b - a$.

That equation comes from interpolating between points $a$ and $b$ with a third-degree polynomial, then integrating by parts.

```matlab
hold off
x = linspace(0, 1);
plot(x, f(x)); hold on
axis([-0.1 1.1 0.2 1.1])

plot([0 1], [f(0) f(1)], 'Color','black','LineStyle',':');

% quadratic polynomial
a = 0; b = 1; m = (b-a)/2;
p = @(z) (f(a).*(z-m).*(z-b)/((a-m)*(a-b))+f(m).*(z-a).*(z-b)/((m-a)*(m-b))+f(b).*(z-a).*(z-m)/((b-a)*(b-m)));
plot(x, p(x), 'Color','red','LineStyle','--');

xp = [0 0.5 1];
yp = [f(0) f(m) f(1)];
plot(xp, yp, 'ok')
hold off
legend('exact', 'trapezoid fit', 'polynomial fit', 'points used')
```

We can see that the polynomial fit, used by Simpson's rule, does a better job of of approximating the exact function, and as a result Simpson's rule will be more accurate than the trapezoidal rule.

Next let's apply Simpson's rule to perform the same integration as above:

```matlab
dx = 0.1;
x = 0.0 : dx : 1.0;

area = 0.0;
for i = 1 : length(x)-1
    area = area + (dx/6.)*(f(x(i)) + 4*f((x(i)+x(i+1))/2.) + f(x(i+1)));
end

fprintf('Simpson rule integral: %f\n', area)
exact = 0.5*sqrt(pi)*erf(1);
fprintf('Exact integral: %f\n', exact)
fprintf('Error: %f %%\n', 100.*abs(exact-area)/exact)
```

    Simpson rule integral: 0.746824
    Exact integral: 0.746824
    Error: 0.000007 %

Simpson's rule is about three orders of magnitude (~1000x) more accurate than the trapezoidal rule.

In this case, using a more-accurate method allows us to significantly reduce the error while still using the same number of segments/steps.

## Error

Applying the trapezoidal rule and Simpson's rule introduces the concept of **error** in numerical solutions.

In our work so far, we have come across two obvious kinds of error, that we'll come back to later:

- **local truncation error**, which represents how "wrong" each interval/step is compared with the exact solution; and
- **global truncation error**, which is the sum of the truncation errors over the entire method.

In any numerical solution, there are five main sources of error:

1. Error in input data: this comes from measurements, and can be *systematic* (for example, due to uncertainty in measurement devices) or *random*.

2. Rounding errors: loss of significant digits. This comes from the fact that computers cannot represent real numbers exactly, and instead use a floating-point representation.

3. Truncation error: due to an infinite process being broken off. For example, an infinite series or sum ending after a finite number of terms, or discretization error by using a finite step size to approximate a continuous function.

4. Error due to simplifications in a mathematical model: *"All models are wrong, but some are useful"* (George E.P. Box) All models make some idealizations, or simplifying assumptions, which introduce some error with respect to reality. For example, we may assume gases are continuous, that a spring has zero mass, or that a process is frictionless.

5. Human error and machine error: there are many potential sources of error in any code. These can come from typos, human programming errors, errors in input data, or (more rarely) a pure machine error. Even textbooks, tables, and formulas may have errors.

### Absolute and relative error

We can also differentiate between **absolute** and **relative** error in a quantity. If $y$ is an exact value and $\tilde{y}$ is an approximation to that value, then we have

- absolute error: $\Delta y = | \tilde{y} - y |$
- relative error: $\frac{\Delta y}{y} = \left| \frac{\tilde{y} - y}{y} \right|$

If $y$ is a vector, then we can define error using the maximum of the elements:
\begin{equation}
\max_i \frac{ |y_i - \tilde{y}_{i} |}{|y_i|}
\end{equation}


# Supplementary Practice Problems

These are similar to programming problems you may encounter in the mid-terms. They are not graded but we will review them in lab sessions.

**1**. (10 points) The logistic map is defined by the following simple function

$$
f(x) = rx(1-x)
$$

For $x_0 = 0.1$ and $r = 4.0$, store the first 10 values of the iterated logistic map $x_{i+1} = rx_i(1-x_i)$ in a list. The first value in the list should be $x_0$.

```python
res = []
x = 0.1
r = 4.0
for i in range(10):
    res.append(x)
    x = r*x*(1-x)
print(res)
```

    [0.1, 0.36000000000000004, 0.9216, 0.28901376000000006, 0.8219392261226498, 0.5854205387341974, 0.970813326249438, 0.11333924730376121, 0.4019738492975123, 0.9615634951138128]

**2**. (10 points) Write a function to find the greatest common divisor (GCD) of 2 numbers using Euclid's algorithm.:

\begin{align}
\gcd(a,0) &= a \\
\gcd(a, b) &= \gcd(b, a \mod b)
\end{align}

Find the GCD of 5797 and 190978.

Now write a function to find the GCD given a collection of numbers.

Find the GCD of (24, 48, 60, 120, 8).

```python
def gcd(a,b):
    if b==0: return a
    else: return gcd(b,a%b)
gcd(5797,190978)
```

    17

```python
input = [24, 48, 60, 120, 8]
def gcd_collection(input):
    prev = gcd(input[0],input[1])
    for i in range(2,len(input)):
        curr = gcd(prev,input[i])
        prev = curr
    return curr
gcd_collection(input)
```

    4

**3**. (10 points) Find the least squares linear solution to the following data

```
y = [1,2,3,4]
x1 = [1,2,3,4]
x2 = [2,3,4,5]
```

That is, find the "best" intercept and slope for the variables `x1` and `x2`.

```python
import numpy as np
import scipy.linalg as la
y = np.array([1,2,3,4])
x1 = np.array([1,2,3,4]).reshape(4,1)
x2 = np.array([2,3,4,5]).reshape(4,1)
x0 = np.ones(4).reshape(4,1)
X = np.c_[x0,x1,x2]
la.lstsq(X,y)
```

    (array([-0.33333333,  0.66666667,  0.33333333]),
     array([], dtype=float64),
     2,
     array([9.34413269, 0.82896583, 0.        ]))

**4**. (10 points) Read the `mtcars` data frame from R to a `pandas` DataFrame. Find the mean `wt` and `mpg` for all cars grouped by the number of `gear`s.

```python
from rpy2.robjects import r, pandas2ri
pandas2ri.activate()
r.data('mtcars')
mtcars = r['mtcars']
mtcars.groupby('gear').mean()[['wt','mpg']]
```

      wt
      mpg

      gear

      3.0
      3.892600
      16.106667

      4.0
      2.616667
      24.533333

      5.0
      2.632600
      21.380000

**5**. (10 points) Read the `iris` data frame from R to a `pandas` DataFrame. Make a `seaborn` plot showing a linear regression of `Petal.Length` (y) against `Sepal.Length` (x). Make a separate regression line for each `Species`.

```python
import seaborn as sns
from rpy2.robjects import r, pandas2ri
pandas2ri.activate()
r.data('iris')
iris = r['iris']
g = sns.lmplot(x="Sepal.Length", y="Petal.Length", hue = 'Species', data = iris)
```

**6**. (10 points) Write a function that can flatten a nested list of arbitrary depth. Check that

```python
flatten([1,[2,3],[4,[5,[6,7],8],9],10,[11,12]])
```

returns

```python
[1,2,3,4,5,6,7,8,9,10,11,12]
```

For simplicity, assume that the only data structure you will encounter is a list. You can check if an item is a list by using

```python
isinstance(item, list)
```

```python
def flatten(l):
    if not isinstance(l,list):
        print('Not a list')
        return None
    else:

flatten([1,[2,3],[4,[5,[6,7],8],9],10,[11,12]])
```

    [1, [2, 3], [4, [5, [6, 7], 8], 9], 10, [11, 12]]

```python
def flatten(l):
    res = []
    for element in l:
        if isinstance(element, list):
            for item in flatten(element):
                res.append(item)
        else:
            res.append(element)
    return res
flatten([1,[2,3],[4,[5,[6,7],8],9],10,[11,12]])
```

    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

**7**. (10 points) Create the following table

```python
array([[  1,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0],
       [  1,   1,   0,   0,   0,   0,   0,   0,   0,   0,   0],
       [  1,   2,   1,   0,   0,   0,   0,   0,   0,   0,   0],
       [  1,   3,   3,   1,   0,   0,   0,   0,   0,   0,   0],
       [  1,   4,   6,   4,   1,   0,   0,   0,   0,   0,   0],
       [  1,   5,  10,  10,   5,   1,   0,   0,   0,   0,   0],
       [  1,   6,  15,  20,  15,   6,   1,   0,   0,   0,   0],
       [  1,   7,  21,  35,  35,  21,   7,   1,   0,   0,   0],
       [  1,   8,  28,  56,  70,  56,  28,   8,   1,   0,   0],
       [  1,   9,  36,  84, 126, 126,  84,  36,   9,   1,   0],
       [  1,  10,  45, 120, 210, 252, 210, 120,  45,  10,   1]])
```

Start with the first row

```
[  1,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0]
```

and build the subsequent rows using a simple rule that only depends on the previous row.

```python
matrix = np.fromfunction(lambda i,j: np.where((i==j) | (j==0), 1, 0), (11,11), dtype='int')
for i in range(2,11):
    for j in range(1,11):
        matrix[i,j] = matrix[i-1,j]+matrix[i-1,j-1]
matrix
```

    array([[  1,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0],
           [  1,   1,   0,   0,   0,   0,   0,   0,   0,   0,   0],
           [  1,   2,   1,   0,   0,   0,   0,   0,   0,   0,   0],
           [  1,   3,   3,   1,   0,   0,   0,   0,   0,   0,   0],
           [  1,   4,   6,   4,   1,   0,   0,   0,   0,   0,   0],
           [  1,   5,  10,  10,   5,   1,   0,   0,   0,   0,   0],
           [  1,   6,  15,  20,  15,   6,   1,   0,   0,   0,   0],
           [  1,   7,  21,  35,  35,  21,   7,   1,   0,   0,   0],
           [  1,   8,  28,  56,  70,  56,  28,   8,   1,   0,   0],
           [  1,   9,  36,  84, 126, 126,  84,  36,   9,   1,   0],
           [  1,  10,  45, 120, 210, 252, 210, 120,  45,  10,   1]])

**8**. (10 points) Read the following data sets into DataFrames.

- url1 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills.csv"
- url2 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills2000.csv"

Create a new DataFraem only containing the names present in both DataFrames. Drop the `timef` column and have a single column for `dist` , `climb` and `time` that shows the average value of the two DataFrames. The final DtataFrame will thus have 4 columns (name, dist, climb, time).

```python
import pandas as pd
url1 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills.csv"
url2 = "https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/hills2000.csv"
data1 = pd.read_csv(url1)
data2 = pd.read_csv(url2)
data = pd.merge(data1,data2,on='Unnamed: 0')
data.drop('timef',axis=1)
```

      Unnamed: 0
      dist_x
      climb_x
      time_x
      dist_y
      climb_y
      time_y

      0
      Craig Dunain
      6.0
      900
      33.650
      6.0
      900
      0.546111

      1
      Ben Lomond
      8.0
      3070
      62.267
      9.0
      3192
      1.037778

      2
      Goatfell
      8.0
      2866
      73.217
      8.0
      2866
      1.227778

      3
      Scolty
      5.0
      800
      29.750
      5.0
      800
      0.495833

      4
      Traprain
      6.0
      650
      39.750
      6.5
      650
      0.623889

      5
      Dollar
      5.0
      2000
      43.050
      6.0
      2000
      0.638333

      6
      Lomonds
      9.5
      2200
      65.000
      9.0
      2200
      1.053056

      7
      Black Hill
      4.5
      1000
      17.417
      4.0
      600
      0.447778

      8
      Meall Ant-Suidhe
      3.5
      1500
      27.900
      3.5
      1500
      0.465000

      9
      Creag Dubh
      4.0
      2000
      26.217
      3.0
      1223
      0.463889

      10
      Criffel
      6.5
      1750
      50.500
      7.0
      1800
      0.793333

      11
      Ben Nevis
      10.0
      4400
      85.583
      10.0
      4400
      1.426111

      12
      Knockfarrel
      6.0
      600
      32.383
      5.0
      1200
      0.623333

      13
      Two Breweries
      18.0
      5200
      170.250
      18.0
      4900
      2.565833


```python
from sympy import *
import mpmath
init_printing()

'''
r_GEO = 36000 + 6371 KM
r_LEO = 2000 + 6371 KM

G = 6.674e-11
Me = 5.972e24
'''

M, E = symbols("M E", Functions = True)
e_c, a, G, M_e, r, mu = symbols("e_c a G M_e r mu", Contstants = True)
T_circular, T_elliptical, T_GEO, T_GTO, T_LEO, r_LEO, r_GEO, T_tot = symbols("T_circular T_elliptical T_GEO T_GTO T_LEO r_LEO r_GEO T_tot", Constants = True)
t, x, y, Y = symbols("t x y Y", Variables = True)

mu_calculated = (6.674e-11 * 5.972e24)
```

The orbital period of a circular Orbit:

```python
Eq(T_circular, 2*pi*sqrt(r**3 / mu))
```

Where mu is:

```python
Eq(mu, G*M_e)
```

Then, the GEO's orbital period in hours is:

```python
r_GEO_Calculated = (36000 + 6371)*1000
T_GEO_Calculated = 2*pi*sqrt(r_GEO_Calculated**3 / mu_calculated)
Eq(T_GEO, T_GEO_Calculated.evalf()/60/60)
```

And the LEO's orbital period in hours is:

```python
r_LEO_Calculated = (2000 + 6371)*1000
T_LEO_Calculated = 2*pi*sqrt(r_LEO_Calculated**3 / mu_calculated)
Eq(T_LEO, T_LEO_Calculated.evalf()/60/60)
```

_____________________________________________
# Finding the GTO.
The goal is to get both 'e' (the eccentricity of our GTO) and 'a' (its semi-major axis). So, we need 2 eqns.

The equation of the GEO (A circle equation):

```python
geo = Eq(y**2, r**2-x**2)
geo
```

The equation of the GTO (An ellipse equation):

```python
gto = Eq(((x+a*e_c)**2/a**2)+(y**2/a**2*(1-e_c**2)), 1)
gto
```

We Wanna solve these two eqns to get the semi-major axis of our GTO and the raduis of our LEO.

first, substitute the GEO's eqn in the GTO's eqn.

```python
toSolve = gto.subs({y**2:geo.rhs})
toSolve
```

Now we can solve for x.

```python
solX = solveset(toSolve, x)
solX
```

Now we can calculate the y coordinate for each x.

```python
solY1 = solveset(Eq(geo.lhs, geo.rhs.subs({x:list(solX)[0]})), y)
solY1
```

```python
solY2 = solveset(Eq(geo.lhs, geo.rhs.subs({x:list(solX)[1]})), y)
solY2
```

We have 4 different possible points for the intersection between a circle and an ellipse, but the intersection between the GEO and the GTO is going to be at only one point with an x coordinate of '-r_GEO' (the radius of the GEO).

Now, we can get the first eqn.

```python
geoAndGtoIntersection = solveset(Eq(list(solX)[0], -r_GEO).subs({r:r_GEO}), a)
geoAndGtoIntersection
```

Surbrisingly, there are 2 possible values for a. But we're not interrested in the negative value. So our first eqn is:

```python
eqn1 = Eq(a, list(list(geoAndGtoIntersection.args)[2])[1])
eqn1
```

To get another eqn, we can do the same but this time with the LOE.

The intersection between our LEO and GTO is exactly at the x coordinate of 'r_LEO'.

```python
gtoAndLeoIntersection = solveset(Eq(list(solX)[1], r).subs({r:r_LEO}), a)
gtoAndLeoIntersection
```

Again, there are 2 possible values for 'r_LEO' but we need the positive one.

```python
eqn2 = Eq(a, list(list(gtoAndLeoIntersection.args)[2])[0])
eqn2
```

This is the positive because 0 < e_c < 1.

Now, we have 2 eqns and 2 variables. And we're ready to get 'a' and 'e_c'

```python
e_c_Exp = Eq(e_c, solveset(eqn1.subs({a:eqn2.rhs, r_GEO:r_GEO_Calculated, r_LEO:r_LEO_Calculated}), e_c).args[0])
e_c_Calculated = e_c_Exp.rhs
e_c_Exp
```

```python
s = solveset(eqn2.subs({r_LEO:r_LEO_Calculated, e_c:e_c_Calculated})).args[0]
a_Exp = Eq(a, s)
a_Calculated = a_Exp.rhs
a_Exp
```

There's another way for finding 'a'.

```python
p1 = plot(sqrt(r_GEO_Calculated**2-x**2), -sqrt(r_GEO_Calculated**2-x**2), sqrt(r_LEO_Calculated**2-x**2), -sqrt(r_LEO_Calculated**2-x**2), sqrt(a_Calculated**2*(1-e_c_Calculated**2)*(1-((x+a_Calculated*e_c_Calculated)**2/a_Calculated**2))), -sqrt(a_Calculated**2*(1-e_c_Calculated**2)*(1-((x+a_Calculated*e_c_Calculated)**2/a_Calculated**2))),(x, -5*10**7, 5*10**7),xlim = (-7.5*10**7, 7.5*10**7), ylim=((-5*10**7, 5*10**7)))
```

    <matplotlib.figure.Figure at 0x11423ef98>

From the geometry we can say that:

```python
Eq(a, (r_LEO + r_GEO)/2)
```

This could've saved us a lot of math work :)

__________________________________
# Now let's calculate the periods.

The orbital period of an elliptical orbit is:

```python
Eq(T_elliptical, 2*pi*sqrt(a**3 / mu))
```

```python
T_GTO_Calculated = 2*pi*sqrt(a_Calculated**3/mu_calculated)
Eq(T_GTO, T_GTO_Calculated.evalf()/60/60)
```

So, the total time required to put our satellite in a GEO using Hohmann transfer is:

```python
Eq(T_tot, T_GTO / 2 + T_LEO / 2)
```

The total time required to put our satellite in a GEO of a 36,000 Kilometers above sea level in hours is:

```python
Eq(T_tot, (T_GTO_Calculated / 2 + T_LEO_Calculated / 2).evalf()/60/60)
```


# Solutions to Project Euler #1-#10

```python
# This enables fetching of a PE problem by
#
#  %pe 32
#
%load_ext fetch_euler_problem
```

## Problem 1

[Multiples of 3 and 5](https://projecteuler.net/problem=1)

>
If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

>
Find the sum of all the multiples of 3 or 5 below 1000.

```python
def prob001(a: int = 3, b: int = 5, below: int = 1000) -> int:
    """
    >>> prob001(3, 5, 10)
    23
    """
    set1 = set(range(a, below, a))
    set2 = set(range(b, below, b))
    return sum(set1 | set2)

```

```python
prob001()
```

    233168

```python
%timeit prob001()
```

    20.6 µs ± 3.58 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

**Another Solution:** filtering might be nice.

```python
def prob001b(a: int = 3, b: int = 5, below: int = 1000) -> int:
    """
    >>> prob001b(3, 5, 10)
    23
    """
    return sum(i for i in range(below) if i % a == 0 or i % b == 0)

```

```python
prob001b()
```

    233168

```python
%timeit prob001b()
```

    120 µs ± 8.08 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

## Problem 2

[Even Fibonacci numbers](https://projecteuler.net/problem=2)

>
Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, the first 10 terms will be:

>
1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...

>
By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms.

Fibonacci sequence may be generated from repeated map of two-integer state. Or, in matrix form,

$$
\begin{align*}
    a_{n+1} &= b_n \\
    b_{n+1} &= a_n + b_n
\end{align*}
$$

with $a_0 = 0$ and  $b_0 = 1$.

Also note that an infinite series is nicely handled with the Python generator.

```python
import itertools as it

def prob002(maxval: int = 4000000) -> int:

    def fibs():
        i, j = 0, 1
        while True:
            (i, j) = (j, i + j)
            yield j

    fibseq = fibs()
    finite_fibs = it.takewhile(lambda x: x <= maxval, fibseq)
    return sum(n for n in finite_fibs if n % 2 == 0)

```

```python
prob002()
```

    4613732

```python
%timeit prob002()
```

    12.7 µs ± 2.87 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## Problem 3

[Largest prime factor](https://projecteuler.net/problem=3)

>
The prime factors of 13195 are 5, 7, 13 and 29.

>
What is the largest prime factor of the number 600851475143 ?

Quick (and cheating) way to factor integers is to use `factorint()` in `sympy.ntheory` module.

```python
import sympy.ntheory
sympy.ntheory.factorint(13195)
```

    {5: 1, 7: 1, 13: 1, 29: 1}

```python
def prob003_sympy(n: int = 600851475143) -> int:
    """
    >>> prob003_sympy(13195)
    29
    """
    factors = sympy.ntheory.factorint(n)
    return max(factors)
```

```python
prob003_sympy()
```

    6857

```python
%timeit prob003_sympy()
```

    710 µs ± 119 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Many nice algorithms for [integer factorization](https://en.wikipedia.org/wiki/Integer_factorization) exist (such as [Pollard's rho algorithm](https://en.wikipedia.org/wiki/Pollard's_rho_algorithm)), but a simple approach is good enough for this problem.

I prepared a generator giving psudo-prime sequence. I don't worry about composite numbers in the sequence because the target number $n$ is not divisible by them.

```python
from typing import Iterator

def psudo_primes() -> Iterator[int]:
    """
    Generate numbers n > 1 that is NOT multiple of 2 or 3.

    >>> import itertools
    >>> xs = tuple(itertools.takewhile(lambda x: x < 30, psudo_primes()))
    (2, 3, 5, 7, 11, 13, 17, 19, 23, 25, 29)
    """
    yield 2
    yield 3

    x = 5
    while True:
        yield x
        x += 2
        yield x
        x += 4

def prob003(n: int = 600851475143) -> int:
    """
    >>> prob003(13195)
    29
    """
    assert n > 1
    for p in psudo_primes():
        while n % p == 0:
            n //= p
            maxval = p
        if p * p > n:
            break
    if n > 1:
        maxval = n
    return maxval

```

```python
prob003()
```

    6857

```python
%timeit prob003()
```

    198 µs ± 21.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

## Problem 4

[Largest palindrome product](https://projecteuler.net/problem=4)

>
A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 × 99.

>
Find the largest palindrome made from the product of two 3-digit numbers.

$n$-digit number $x$ is equivalent to $10^{n-1} \leq x < 10^n$. Following brute force algorithm is quadratic time complexity.

**[FIXME]** It should be faster.

```python
def is_palindrome(number: int) -> bool:
    """
    Check if number is palindrome, the numbers identical to its reversed-direction digits.

    >>> is_palindrome(15651)
    True

    >>> is_palindrome(56)
    False
    """
    s = str(number)
    return s == "".join(reversed(s))

def prob004(digits: int = 3):
    """
    >>> prob004(digits=2)
    (9009, 91, 99)
    """
    lower_bound = 10 ** (digits - 1)
    upper_bound = 10 ** digits
    result = (0, 0, 0)
    for i in range(lower_bound, upper_bound):
        for j in range(i, upper_bound):
            x = i * j
            if is_palindrome(x) and result < (x, i, j):
                result = (x, i, j)

    return result

```

```python
prob004(digits=3)
```

    (906609, 913, 993)

```python
%timeit prob004()
```

    605 ms ± 217 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

## Problem 5

[Smallest multiple](https://projecteuler.net/problem=5)

>
2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder.

>
What is the smallest positive number that is <dfn title="divisible with no remainder">evenly divisible</dfn> by all of the numbers from 1 to 20?

The problem is to find the loweset common multiplier (LCM). `lcm` is available in `sympy`.

```python
import sympy
import functools

functools.reduce(sympy.lcm, range(1,11))
```

    2520

Greatest common divisor (GCD) is actually in the standard library `fractions`. And GCD and LCM are related by the identity

$$ \mathrm{LCM}(a, b) = \frac{a\, b}{\mathrm{GCD}(a, b)}. $$

```python
import math
import functools

def lcm(a: int, b: int) -> int:
    """
    Return the lowest common multiplier of a and b
    """
    return a // math.gcd(a, b) * b

def prob005(maxval: int = 20) -> int:
    """
    >>> prob005(10)
    2520
    """
    return functools.reduce(lcm, range(1, maxval + 1))

```

```python
prob005()
```

    232792560

```python
%timeit prob005()
```

    9.05 µs ± 1.32 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## Problem 6

[Sum square difference](https://projecteuler.net/problem=6)

>
The sum of the squares of the first ten natural numbers is,

>
12 + 22 + ... + 102 = 385

>
The square of the sum of the first ten natural numbers is,

>
(1 + 2 + ... + 10)2 = 552 = 3025

>
Hence the difference between the sum of the squares of the first ten natural numbers and the square of the sum is 3025 − 385 = 2640.

>
Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.

Knowing the sums,

\begin{align}
    \sum_{n=1}^{N} n   &= \frac{1}{2} N(N+1), \\
    \sum_{n=1}^{N} n^2 &= \frac{1}{6} N (N+1) (2N + 1).
\end{align}

we can calculate the difference between the sum of squares and the square of the sum.

$$ \left( \sum_{n=1}^{N} n \right)^2 - \sum_{n=1}^{N} n^2  = \frac{1}{12} n (3 n^3 + 2 n^2 - 3n -2). $$

`sympy` can reproduce the algebraic manipulations.

```python
import sympy
# sympy.init_printing()    # turn on sympy printing on Jupyter notebooks

def the_difference():
    i, n = sympy.symbols("i n", integer=True)
    simple_sum = sympy.summation(i, (i, 1, n))
    sum_of_squares = sympy.summation(i ** 2, (i, 1, n))
    formula = sympy.simplify(simple_sum ** 2 - sum_of_squares)
    return formula

def prob006_sympy(nval=100):
    """
    >>> prob006_sympy(10)
    2640
    """
    formula = the_difference()
    n = formula.free_symbols.pop()
    return int(formula.evalf(subs={n:nval}))

```

This problem only requires summation over first one hundred numbers so bruteforce approach works.

```python
def prob006(n: int = 100) -> int:
    """
    >>> prob006(10)
    2640
    """
    xs = range(1, n + 1)
    simple_sum = sum(xs)
    sum_of_squares = sum(x ** 2 for x in xs)
    return simple_sum ** 2 - sum_of_squares

```

Then all you need is to evaluate the formula with upper bound.

```python
prob006()
```

    25164150

```python
%timeit prob006()
```

    30.8 µs ± 2.64 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

## Problem 7

[10001st prime](https://projecteuler.net/problem=7)

>
By listing the first six prime numbers: 2, 3, 5, 7, 11, and 13, we can see that the 6th prime is 13.

>
What is the 10 001st prime number?

**Solution:** `sympy.ntheory` package has prime number generators.

```python
sympy.ntheory.prime(6)
```

    13

```python
sympy.ntheory.prime(10001)
```

    104743

s a good enough algorithm to generate prime numbers.

It is known [(wikipedia.org)](https://en.wikipedia.org/wiki/Prime_number_theorem#Approximations_for_the_nth_prime_number) that  $n$-th prime number $p_n$ is bounded by

$$ \log n + \log \log n - 1 < \frac{p_n}{n} < \log n + \log \log n  \ \ \ \textrm{ for $n \geq 6$}. $$

So, I'll select integers up to the upper bound with [Sieve of Eratosthenes](https://en.wikipedia.org/wiki/Sieve_of_Eratosthenes).

```python
import math
from typing import Iterator, List

# Reproduction from Problem #3
def psudo_primes() -> Iterator[int]:
    yield 2
    yield 3
    x = 5
    while True:
        yield x
        x += 2
        yield x
        x += 6

def sieve(n: int) -> List[int]:
    """
    Return all prime numbers below n

    >>> sieve(10)
    [2, 3, 5, 7]
    """
    assert n > 1
    remaining = [True] * n  # never use the first two elements (0th and 1st)
    for p in range(2, int(math.sqrt(n) + 1)):
        if not remaining[p]:
            continue
        for q in range(p * p, n, p):
            remaining[q] = False
    return [p for p in range(2, n) if remaining[p]]

def prime(n: int) -> List[int]:
    """
    Return first n prime numbers

    >>> prime(10)
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
    """
    log = math.log
    if n >= 6:
        upperbound = int(n * (log(n) + log(log(n))))
        out = sieve(upperbound)
    else:
        out = [2, 3, 5, 7, 11]
    return out[:n]

def prob007(n: int = 10001) -> int:
    """
    >>> prob007(6)
    13
    """
    return prime(n)[-1]

```

```python
prob007()
```

    104743

```python
import sympy
%timeit sympy.ntheory.prime(10001)
```

    17.3 ms ± 4.65 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%timeit prob007()
```

    16.7 ms ± 463 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

## Problem 8

## Problem 8

[Largest product in a series](https://projecteuler.net/problem=8)

>
The four adjacent digits in the 1000-digit number that have the greatest product are 9 × 9 × 8 × 9 = 5832.

>
73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450

>
Find the thirteen adjacent digits in the 1000-digit number that have the greatest product. What is the value of this product?

```python
import operator
import functools

def product(iterable):
    """
    >>> product(range(1, 6))
    120
    """
    return functools.reduce(operator.mul, iterable)

def prob008(s: str, seq_size: int = 5):

    def slice_N_digits(idx):
        for i in s[idx : idx + seq_size]:
            yield int(i)

    return max(product(slice_N_digits(idx)) for idx in range(len(s) - seq_size))

```

```python
s = """
73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450"""

xs = [int(c) for c in s.replace("\n", "")]
prob008(xs)

```

    40824

```python
%timeit prob008(xs)
```

    2.5 ms ± 358 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

## Problem 9

[Special Pythagorean triplet](https://projecteuler.net/problem=9)

>
A Pythagorean triplet is a set of three natural numbers, <var>a</var> < <var>b</var> < <var>c</var>, for which,

>
 <var>a</var>2 + <var>b</var>2 = <var>c</var>2

>
For example, 32 + 42 = 9 + 16 = 25 = 52.

>
There exists exactly one Pythagorean triplet for which <var>a</var> + <var>b</var> + <var>c</var> = 1000.
Find the product <var>abc</var>.

```python
from typing import Optional, Tuple

def prob009(total: int = 1000) -> Optional[Tuple[int, int, int, int]]:
    """
    >>> prob009(total=12)
    (3, 4, 5, 60)
    """
    ub = total // 2  # just an upper bound
    for a in range(1, ub):
        for b in range(a, ub):
            c = total - a - b
            if a ** 2 + b ** 2 == c ** 2:
                product = a * b * c
                return (a, b, c, product)
    else:
        return None
```

```python
prob009(12)
```

    (3, 4, 5, 60)

```python
%timeit prob009()
```

    66.8 ms ± 2.58 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

## Problem 10

[Summation of primes](https://projecteuler.net/problem=10)

>
The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17.

>
Find the sum of all the primes below two million.

`sympy` module has `primerange`.

```python
list(sympy.sieve.primerange(1, 10))
```

    [2, 3, 5, 7]

So, Just take sum of the prime numbers between 1 and 200000.

```python
import sympy

def prob010_sympy(n=2000000):
    """
    >>> prob010_sympy(10)
    17
    """
    return sum(sympy.sieve.primerange(1, n))
```

```python
%timeit prob010_sympy()
```

    34.2 ms ± 147 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

Use Eratosthenes sieve in Problem 7.

```python
from typing import List

# Reproduction from problem 7
def sieve(n: int) -> List[int]:
    assert n > 1
    remaining = [True] * (n + 1)  # never use the first two elements (0th and 1st)
    for p in range(2, int(math.sqrt(n) + 1)):
        if not remaining[p]:
            continue
        for q in range(p * p, n + 1, p):
            remaining[q] = False
    return [p for p in range(2, n + 1) if remaining[p]]

def prob010(n: int = 2000000) -> int:
    """
    >>> prob010(10)
    17
    """
    return sum(sieve(n))

```

```python
prob010()
```

    142913828922

```python
%timeit prob010()
```

    497 ms ± 128 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

## [Appendix] Run doctests

```python
import doctest
doctest.testmod()
```

    TestResults(failed=0, attempted=16)

```python

```


# Numerical methods for 2nd-order ODEs

We've gone over how to solve 1st-order ODEs using numerical methods, but what about 2nd-order or any higher-order ODEs? We can use the same methods we've already discussed by transforming our higher-order ODEs into a **system of first-order ODEs**.

Meaning, if we are given a 2nd-order ODE
\begin{equation}
\frac{d^2 y}{dx^2} = y^{\prime\prime} = f(x, y, y^{\prime})
\end{equation}
we can transform this into a system of **two 1st-order ODEs** that are coupled:
\begin{align}
\frac{dy}{dx} &= y^{\prime} = u \\
\frac{du}{dx} &= u^{\prime} = y^{\prime\prime} = f(x, y, u)
\end{align}
where $f(x, y, u)$ is the same as that given above for $\frac{d^2 y}{dx^2}$.

Thus, instead of a 2nd-order ODE to solve, we have two 1st-order ODEs:
\begin{align}
y^{\prime} &= u \\
u^{\prime} &= f(x, y, u)
\end{align}

So, we can use all of the methods we have talked about so far to solve 2nd-order ODEs by transforming the one equation into a system of two 1st-order equations.

## Higher-order ODEs

This works for higher-order ODEs too! For example, if we have a 3rd-order ODE, we can transform it into a system of three 1st-order ODEs:
\begin{align}
\frac{d^3 y}{dx^3} &= f(x, y, y^{\prime}, y^{\prime\prime}) \\
\rightarrow y^{\prime} &= u \\
u^{\prime} &= y^{\prime\prime} = w \\
w^{\prime} &= y^{\prime\prime\prime} = f(x, y, u, w)
\end{align}

## Example: mass-spring problem

For example, let's solve a forced damped mass-spring problem given by a 2nd-order ODE:
\begin{equation}
y^{\prime\prime} + 5y^{\prime} + 6y = 10 \sin \omega t
\end{equation}
with the initial conditions $y(0) = 0$ and $y^{\prime}(0) = 5$.

We start by transforming the equation into two 1st-order ODEs. Let's use the variables $z_1 = y$ and $z_2 = y^{\prime}$:
\begin{align}
\frac{dz_1}{dt} &= z_1^{\prime} = z_2 \\
\frac{dz_2}{dt} &= z_2^{\prime} = y^{\prime\prime} = 10 \sin \omega t - 5z_2 - 6z_1
\end{align}

### Forward Euler

Then, let's solve numerically using the forward Euler method. Recall that the recursion formula for forward Euler is:
\begin{equation}
y_{i+1} = y_i + \Delta x f(x_i, y_i)
\end{equation}
where $f(x,y) = \frac{dy}{dx}$.

Let's solve using $\omega = 1$ and with a step size of $\Delta t = 0.1$, over $0 \leq t \leq 3$.

We can compare this against the exact solution, obtainable using the method of undetermined coefficients:
\begin{equation}
y(t) = -6 e^{-3t} + 7 e^{-2t} + \sin t - \cos t
\end{equation}

```matlab
% plot exact solution first
t = linspace(0, 3);
y_exact = -6*exp(-3*t) + 7*exp(-2*t) + sin(t) - cos(t);
plot(t, y_exact); hold on

omega = 1;

dt = 0.1;
t = [0 : dt : 3];

f = @(t,z1,z2) 10*sin(omega*t) - 5*z2 - 6*z1;

z1 = zeros(length(t), 1);
z2 = zeros(length(t), 1);
z1(1) = 0;
z2(1) = 5;
for i = 1 : length(t)-1
    z1(i+1) = z1(i) + dt * z2(i);
    z2(i+1) = z2(i) + dt * f(t(i), z1(i), z2(i));
end

plot(t, z1, 'o--')
xlabel('time'); ylabel('displacement')
legend('Exact', 'Forward Euler', 'Location','southeast')
```

### Heun's Method

For schemes that involve more than one stage, like Heun's method, we'll need to implement both stages for each 1st-order ODE. For example:

```matlab
clear
% plot exact solution first
t = linspace(0, 3);
y_exact = -6*exp(-3*t) + 7*exp(-2*t) + sin(t) - cos(t);
plot(t, y_exact); hold on

omega = 1;

dt = 0.1;
t = [0 : dt : 3];

f = @(t,z1,z2) 10*sin(omega*t) - 5*z2 - 6*z1;

z1 = zeros(length(t), 1);
z2 = zeros(length(t), 1);
z1(1) = 0;
z2(1) = 5;
for i = 1 : length(t)-1
    % predictor
    z1p = z1(i) + z2(i)*dt;
    z2p = z2(i) + f(t(i), z1(i), z2(i))*dt;

    % corrector
    z1(i+1) = z1(i) + 0.5*dt*(z2(i) + z2p);
    z2(i+1) = z2(i) + 0.5*dt*(f(t(i), z1(i), z2(i)) + f(t(i+1), z1p, z2p));
end
plot(t, z1, 'o')
xlabel('time'); ylabel('displacement')
legend('Exact', 'Heuns', 'Location','southeast')
```

### Runge-Kutta: `ode45`

We can also solve using `ode45`, by providing a separate function file that defines the system of 1st-order ODEs. In this case, we'll need to use a single **array** variable, `Z`, to store $z_1$ and $z_2$. The first column of `Z` will store $z_1$ (`Z(:,1)`) and the second column will store $z_2$ (`Z(:,2)`).

```matlab
%%file mass_spring.m
function dzdt = mass_spring(t, z)
    omega = 1;
    dzdt = zeros(2,1);

    dzdt(1) = z(2);
    dzdt(2) = 10*sin(omega*t) - 6*z(1) - 5*z(2);
end
```

    Created file '/Users/kyle/projects/ME373/docs/mass_spring.m'.

```matlab
% plot exact solution first
t = linspace(0, 3);
y_exact = -6*exp(-3*t) + 7*exp(-2*t) + sin(t) - cos(t);
plot(t, y_exact); hold on

% solution via ode45:
[T, Z] = ode45('mass_spring', [0 3], [0 5]);

plot(T, Z(:,1), 'o')
xlabel('time'); ylabel('displacement')
legend('Exact', 'ode45', 'Location','southeast')
```

## Backward Euler for 2nd-order ODEs

We saw how to implement the Backward Euler method for a 1st-order ODE, but what about a 2nd-order ODE? (Or in general a system of 1st-order ODEs?)

The recursion formula is the same, except now our dependent variable is an array/vector:
\begin{equation}
\mathbf{y}_{i+1} = \mathbf{y}_i + \Delta t \, \mathbf{f} \left( t_{i+1} , \mathbf{y}_{i+1} \right)
\end{equation}
where the bolded $\mathbf{y}$ and $\mathbf{f}$ indicate array quantities (in other words, they hold more than one value).

In general, we can use Backward Euler to solve 2nd-order ODEs in a similar fashion as our other numerical methods:

1. Convert the 2nd-order ODE into a system of two 1st-order ODEs
2. Insert the ODEs into the Backward Euler recursion formula and solve for $\mathbf{y}_{i+1}$

The main difference is that we will now have a system of two equations and two unknowns: $y_{1, i+1}$ and $y_{2, i+1}$.

Let's demonstrate with an example:
\begin{equation}
y^{\prime\prime} + 6 y^{\prime} + 5y = 10 \quad y(0) = 0 \quad y^{\prime}(0) = 5
\end{equation}
where the exact solution is
\begin{equation}
y(t) = -\frac{3}{4} e^{-5t} - \frac{5}{4} e^{-t} + 2
\end{equation}

To solve numerically,

1. Convert the 2nd-order ODE into a system of two 1st-order ODEs:
\begin{gather}
y_1 = y \quad y_1(t=0) = 0 \\
y_2 = y^{\prime} \quad y_2 (t=0) = 5
\end{gather}
Then, for the derivatives of these variables:
\begin{align}
y_1^{\prime} &= y_2 \\
y_2^{\prime} &= 10 - 6 y_2 - 5 y_1
\end{align}

2. Then plug these derivatives into the Backward Euler recursion formulas and solve for $y_{1,i+1}$ and $y_{2,i+1}$:
\begin{align}
y_{1, i+1} &= y_{1, i} + \Delta t \, y_{2,i+1} \\
y_{2, i+1} &= y_{2, i} + \Delta t \left( 10 - 6 y_{2, i+1} - 5 y_{1,i+1} \right) \\
\\
y_{1, i+1} - \Delta t \, y_{2, i+1} &= y_{1,i} \\
5 \Delta t \, y_{1, i+1} + (1 + 6 \Delta t) y_{2, i+1} &= y_{2,i} + 10 \Delta t \\
\text{or} \quad
\begin{bmatrix} 1 & -\Delta t \\ 5 \Delta t & (1+6\Delta t)\end{bmatrix}
\begin{bmatrix} y_{1, i+1} \\ y_{2, i+1} \end{bmatrix} &=
\begin{bmatrix} y_{1,i} \\ y_{2,i} + 10 \Delta t \\ \end{bmatrix} \\
\mathbf{A} \mathbf{y}_{i+1} &= \mathbf{b}
\end{align}
To isolate $\mathbf{y}_{i+1}$ and get a usable recursion formula, we need to solve this system of two equations. We could solve this by hand using the substitution method, or we can use Cramer's rule:
\begin{align}
y_{1, i+1} &= \frac{ y_{1,i} (1 + 6 \Delta t) + \Delta t \left( y_{2,i} + 10 \Delta t \right)}{1 + 6 \Delta t + 5 \Delta t^2} \\
y_{2, i+1} &= \frac{ y_{2,i} + 10 \Delta t - 5 \Delta t y_{1,i}}{1 + 6 \Delta t + 5 \Delta t^2}
\end{align}

Let's confirm that this gives us a good, well-behaved numerical solution and compare with the Forward Euler method:

```matlab
clear

% Exact solution
t = linspace(0, 5);
y_exact = @(t) -(3/4)*exp(-5*t) - (5/4)*exp(-t) + 2;
plot(t, y_exact(t)); hold on

dt = 0.1;
t = 0 : dt : 5;

% Forward Euler
f = @(t, y1, y2) 10 - 6*y2 - 5*y1;
y1 = zeros(length(t), 1); y2 = zeros(length(t), 1);
y1(1) = 0; y2(1) = 5;
for i = 1 : length(t) - 1
    y1(i+1) = y1(i) + dt*y2(i);
    y2(i+1) = y2(i) + dt*f(t(i), y1(i), y2(i));
end
plot(t, y1, '+')

Y = zeros(length(t), 2);
Y(1,1) = 0;
Y(1,2) = 5;
for i = 1 : length(t) - 1
    D = 1 + 6*dt + 5*dt^2;
    Y(i+1, 1) = (Y(i,1)*(1 + 6*dt) + dt*(Y(i,2) + 10*dt)) / D;
    Y(i+1, 2) = (Y(i,2) + 10*dt - Y(i,1)*5*dt) / D;
end
plot(t, Y(:,1), 'o')
legend('Exact', 'Forward Euler', 'Backward Euler', 'location', 'southeast')

fprintf('Maximum error of Forward Euler: %5.3f\n', max(abs(y1(:) - y_exact(t)')));
fprintf('Maximum error of Backward Euler: %5.3f', max(abs(Y(:,1) - y_exact(t)')));
```

So, for $\Delta t = 0.1$, we see that the Forward and Backward Euler methods give an error $\mathcal{O}(\Delta t)$, as expected since both methods are *first-order* accurate.

Let's see how they compare for a larger step size:

```matlab
clear

% Exact solution
t = linspace(0, 5);
y_exact = @(t) -(3/4)*exp(-5*t) - (5/4)*exp(-t) + 2;
plot(t, y_exact(t)); hold on

dt = 0.5;
t = 0 : dt : 5;

% Forward Euler
f = @(t, y1, y2) 10 - 6*y2 - 5*y1;
y1 = zeros(length(t), 1); y2 = zeros(length(t), 1);
y1(1) = 0; y2(1) = 5;
for i = 1 : length(t) - 1
    y1(i+1) = y1(i) + dt*y2(i);
    y2(i+1) = y2(i) + dt*f(t(i), y1(i), y2(i));
end
plot(t, y1, 'o')

% Backward Euler

Y = zeros(length(t), 2);
Y(1,1) = 0;
Y(1,2) = 5;
for i = 1 : length(t) - 1
    D = 1 + 6*dt + 5*dt^2;
    Y(i+1, 1) = (Y(i,1)*(1 + 6*dt) + dt*(Y(i,2) + 10*dt)) / D;
    Y(i+1, 2) = (Y(i,2) + 10*dt - Y(i,1)*5*dt) / D;
end
plot(t, Y(:,1), 'o')
legend('Exact', 'Backward Euler', 'location', 'southeast')

%fprintf('Maximum error of Forward Euler: %5.3f\n', max(abs(y1(:) - y_exact(t)')));
%fprintf('Maximum error of Backward Euler: %5.3f', max(abs(Y(:,1) - y_exact(t)')));

fprintf('Maximum error of Forward Euler: %5.3f\n', max(abs(y1(:) - y_exact(t)')));
fprintf('Maximum error of Backward Euler: %5.3f', max(abs(Y(:,1) - y_exact(t)')));
```

Backward Euler, since it is unconditionally stable, remains well-behaved at this larger step size, while the Forward Euler method blows up.

One other thing: instead of using Cramer's rule to get expressions for $y_{1,i+1}$ and $y_{2,i+1}$, we could instead use Matlab to solve the linear system of equations at each time step. To do that, we could replace
```OCTAVE
A = [1  -dt; 5*dt  (1+6*dt)];
b = [Y(i,1); Y(i,2)+10*dt];
Y(i+1,:) = (A\b)';
```
where `A\b` is equivalent to `inv(A)*b`, but faster. Let's confirm that this gives the same answer:

```matlab
clear

% Exact solution
t = linspace(0, 5);
y_exact = @(t) -(3/4)*exp(-5*t) - (5/4)*exp(-t) + 2;
plot(t, y_exact(t)); hold on

dt = 0.1;
t = 0 : dt : 5;

Y = zeros(length(t), 2);
Y(1,1) = 0;
Y(1,2) = 5;
for i = 1 : length(t) - 1
    A = [1  -dt; 5*dt  (1+6*dt)];
    b = [Y(i,1); Y(i,2)+10*dt];
    Y(i+1,:) = (A\b)';
end
plot(t, Y(:,1), 'o')
legend('Exact', 'Backward Euler', 'location', 'southeast')
```

## Cramer's Rule

Cramer's Rule provides a solution method for a system of linear equations, where the number of equations equals the number of unknowns. It works for any number of equations/unknowns, but isn't really practical for more than two or three. We'll focus on using it for a system of two equations, with two unknowns $x_1$ and $x_2$:
\begin{gather}
a_{11} + x_1 + a_{12} x_2 = b_1 \\
a_{21} + x_1 + a_{22} x_2 = b_2 \\
\text{or } \mathbf{A} \mathbf{x} = \mathbf{b}
\end{gather}
where
\begin{gather}
\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} \\
\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \\
\mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \end{bmatrix}
\end{gather}

The solutions for the unknowns are then
\begin{align}
x_1 &= \frac{ \begin{vmatrix} b_1 & a_{12} \\ b_2 & a_{22} \end{vmatrix} }{D} = \frac{b_1 a_{22} - a_{12} b_2}{D} \\
x_2 &= \frac{ \begin{vmatrix} a_{11} & b_1 \\ a_{21} & b_2 \end{vmatrix} }{D} = \frac{a_{11} b_2 - b_1 a_{21}}{D}
\end{align}
where $D$ is the determinant of $\mathbf{A}$:
\begin{equation}
D = \det(\mathbf{A}) = \begin{vmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{vmatrix} = a_{11} a_{22} - a_{12} a_{21}
\end{equation}
This works as long as the determinant does not equal zero.


sergazy.nurbavliyev@gmail.com © 2021

## Can you form a triangle ?

Question: Assume we have a stick of length 1. Suppose you randomly break it into three parts. What is the probability that you can form a triangle using these three parts?

### Intuition

I always like to break into the cases. The idea is divide and conquer :)
Let us figure out what these cases are. First, assume we label these three pieces as $a,b$ and $c.$ Remember $a+b+c=1$. We know that if any of these pieces is bigger than $\frac{1}{2}$ than it is impossible to form a triangle. There are 3 such cases. Now the last case is if all of them are strictly smaller than  $\frac{1}{2}.$ In that case it is possible to form a triangle. You can now intuitively think that all of these outcomes have an equal probability. Thus the answer is 1/4. To make things clear I show these argument in a table.

| Cases:       | a              |b                |      c          | Triangle ?    |
|--------------| -------------- |-----------------| ----------------|---------------|
|Posssibility 1| a>1/2 |b<1/2| c<1/2|No |
|Posssibility 2|       a<1/2 |b>1/2| c<1/2|No |
|Posssibility 3|        a<1/2 |b<1/2| c>1/2|No |
|Posssibility 4|         a<1/2 |b<1/2| c<1/2|Yes |

```python

```

## Theoritical result

I would like to label the breaking points as $x$ and $y$. So there are two possibilities:
Case 1: $x < y$ and Case 2: $y < x$

Case 1:$x < y$: Length of pieces after choosing points x and y: would be
$x , (y-x) , (1-y)$

We showed the partition below.

0----- $x$-------$y$------1

There are 3 possible combination for satisfying triangle inequality.

\begin{equation}
x + (y-x)  > (1-y)\Rightarrow 2y > 1\Rightarrow  y > (1/2)
\end{equation}

\begin{equation}
 x + (1-1)  > (y-x)\Rightarrow  2x + 1 > 2y\Rightarrow  y < x + (1/2)
\end{equation}

\begin{equation}
(y-x) + (1-1)  > x\Rightarrow  2x < 1 \Rightarrow  x < 1/2
\end{equation}

I showed in a figure below the common area of these three region.

```python
from IPython.display import Image
Image(filename='triangle.jpeg')
```

Case 2:$y < x$: Length of pieces after choosing points x and y: would be
$y , (x-y) , (1-x)$. With the same logic, we would get  $\frac{1}{8}$

Now we can add the results to get the probability of forming triangle. That would be  $\frac{1}{8}+ \frac{1}{8}=\frac{1}{4}.$

## Python code for simulation

```python
import random
def forms_triangle(a, b, c):
    return a + b > c and a + c > b and b + c > a
```

```python
def counts(trials):
    num_tri = 0
    for i in range(trials):
        x, y = random.uniform(0, 1.0), random.uniform(0, 1.0)
        if x > y:
            x, y = y, x
            if forms_triangle(x, y - x, 1 - y):
                num_tri += 1
        else:
            x, y = x, y
            if forms_triangle(x, y - x, 1 - y):
                num_tri += 1

    return num_tri
```

```python
trials = 1000000
formed_triangles = counts(trials)
```

```python
formed_triangles/trials
```

    0.249959

```python

```

Can you form a triangle March 5, 2021.ipynb


# Neural Network Fundamentals

## Gradient Descent Introduction:
https://www.youtube.com/watch?v=IxBYhjS295w

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

np.random.seed(1)

%matplotlib inline
np.random.seed(1)
```

```python
N = 100
x = np.random.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 2*np.random.randn(N,1)
```

```python
plt.scatter(x,y_obs,label='Observations')
plt.plot(x,y,c='r',label='True function')
plt.legend()
plt.show()
```

## Gradient Descent

We are trying to minimise $\sum \xi_i^2$.

\begin{align}
\mathcal{L} & = \frac{1}{N}\sum_{i=1}^N (y_i-f(x_i,w,b))^2 \\
\frac{\delta\mathcal{L}}{\delta w} & = -\frac{1}{N}\sum_{i=1}^N 2(y_i-f(x_i,w,b))\frac{\delta f(x_i,w,b)}{\delta w} \\
& = -\frac{1}{N}\sum_{i=1}^N 2\xi_i\frac{\delta f(x_i,w,b)}{\delta w}
\end{align}
where $\xi_i$ is the error term $y_i-f(x_i,w,b)$ and
$$
\frac{\delta f(x_i,w,b)}{\delta w} = x_i
$$

Similar expression can be found for $\frac{\delta\mathcal{L}}{\delta b}$ (exercise).

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.

```python
# Helper functions
def f(w,b):
    return w*x+b

def loss_function(e):
    L = np.sum(np.square(e))/N
    return L

def dL_dw(e,w,b):
    return -2*np.sum(e*df_dw(w,b))/N

def df_dw(w,b):
    return x

def dL_db(e,w,b):
    return -2*np.sum(e*df_db(w,b))/N

def df_db(w,b):
    return np.ones(x.shape)
```

```python
# The Actual Gradient Descent
def gradient_descent(iter=100,gamma=0.1):
    # get starting conditions
    w = 10*np.random.randn()
    b = 10*np.random.randn()

    params = []
    loss = np.zeros((iter,1))
    for i in range(iter):
#         from IPython.core.debugger import Tracer; Tracer()()
        params.append([w,b])
        e = y_obs - f(w,b) # Really important that you use y_obs and not y (you do not have access to true y)
        loss[i] = loss_function(e)

        #update parameters
        w_new = w - gamma*dL_dw(e,w,b)
        b_new = b - gamma*dL_db(e,w,b)
        w = w_new
        b = b_new

    return params, loss

params, loss = gradient_descent()
```

```python
iter=100
gamma = 0.1
w = 10*np.random.randn()
b = 10*np.random.randn()

params = []
loss = np.zeros((iter,1))
for i in range(iter):
#         from IPython.core.debugger import Tracer; Tracer()()
    params.append([w,b])
    e = y_obs - f(w,b) # Really important that you use y_obs and not y (you do not have access to true y)
    loss[i] = loss_function(e)

    #update parameters
    w_new = w - gamma*dL_dw(e,w,b)
    b_new = b - gamma*dL_db(e,w,b)
    w = w_new
    b = b_new
```

```python
dL_dw(e,w,b)
```

    0.0078296405377948283

```python
plt.plot(loss)
```

```python
params = np.array(params)
plt.plot(params[:,0],params[:,1])
plt.title('Gradient descent')
plt.xlabel('w')
plt.ylabel('b')
plt.show()
```

```python
params[-1]
```

    array([ 4.98991104,  2.72258102])

## Multivariate case

We are trying to minimise $\sum \xi_i^2$. This time $ f = Xw$ where $w$ is Dx1 and $X$ is NxD.

\begin{align}
\mathcal{L} & = \frac{1}{N} (y-Xw)^T(y-Xw) \\
\frac{\delta\mathcal{L}}{\delta w} & = -\frac{1}{N} 2\left(\frac{\delta f(X,w)}{\delta w}\right)^T(y-Xw) \\
& = -\frac{2}{N} \left(\frac{\delta f(X,w)}{\delta w}\right)^T\xi
\end{align}
where $\xi_i$ is the error term $y_i-f(X,w)$ and
$$
\frac{\delta f(X,w)}{\delta w} = X
$$

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.

```python
N = 1000
D = 5
X = 5*np.random.randn(N,D)
w = np.random.randn(D,1)
y = X.dot(w)
y_obs = y + np.random.randn(N,1)
```

```python
w
```

    array([[ 0.93774813],
           [-2.62540124],
           [ 0.74616483],
           [ 0.67411002],
           [ 1.0142675 ]])

```python
X.shape
```

    (1000, 5)

```python
w.shape
```

    (5, 1)

```python
(X*w.T).shape
```

    (1000, 5)

```python
# Helper functions
def f(w):
    return X.dot(w)

def loss_function(e):
    L = e.T.dot(e)/N
    return L

def dL_dw(e,w):
    return -2*X.T.dot(e)/N
```

```python
def gradient_descent(iter=100,gamma=1e-3):
    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((iter,1))
    for i in range(iter):
        params.append(w)
        e = y_obs - f(w) # Really important that you use y_obs and not y (you do not have access to true y)
        loss[i] = loss_function(e)

        #update parameters
        w = w - gamma*dL_dw(e,w)

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
params[-1]
```

    array([[ 0.94792987],
           [-2.60989696],
           [ 0.72929842],
           [ 0.65272494],
           [ 1.01038855]])

```python
model = LinearRegression(fit_intercept=False)
model.fit(X,y)
model.coef_.T
```

    array([[ 0.93774813],
           [-2.62540124],
           [ 0.74616483],
           [ 0.67411002],
           [ 1.0142675 ]])

```python
# compare parameters side by side
np.hstack([params[-1],model.coef_.T])
```

    array([[ 0.94792987,  0.93774813],
           [-2.60989696, -2.62540124],
           [ 0.72929842,  0.74616483],
           [ 0.65272494,  0.67411002],
           [ 1.01038855,  1.0142675 ]])

## Stochastic Gradient Descent

```python
def dL_dw(X,e,w):
    return -2*X.T.dot(e)/len(X)

def gradient_descent(gamma=1e-3, n_epochs=100, batch_size=20, decay=0.9):
    epoch_run = int(len(X)/batch_size)

    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((n_epochs,1))
    for i in range(n_epochs):
        params.append(w)

        for j in range(epoch_run):
            idx = np.random.choice(len(X),batch_size,replace=False)
            e = y_obs[idx] - X[idx].dot(w) # Really important that you use y_obs and not y (you do not have access to true y)
            #update parameters
            w = w - gamma*dL_dw(X[idx],e,w)
        loss[i] = e.T.dot(e)/len(e)
        gamma = gamma*decay #decay the learning parameter

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
np.hstack([params[-1],model.coef_.T])
```

    array([[ 0.94494132,  0.93774813],
           [-2.6276984 , -2.62540124],
           [ 0.74654537,  0.74616483],
           [ 0.66766209,  0.67411002],
           [ 1.00760747,  1.0142675 ]])

```python

```

```python

```

```python
tested
```

tests/ml-books/Lesson 02 - GradientDescent.ipynb


## Chapter 5 - Support Vector Machines

### Nonlinear SVC

The support vector classifier is used when the boundary between the two classes are linear. However, in practice, we are sometimes faced with non-linear boundaries. In this case, we consider enlarging the feature space using the higher order features. E.g. rather than fitting a support vector classifier on $p$ features $\begin{pmatrix} X_1, \cdots, X_p\end{pmatrix}$, we add a polynomial (squared) feature and fit the support vector classifier on $2p$ features $\begin{pmatrix} X_1, X_1^2 \cdots, X_p, X_p^2\end{pmatrix}$. Now, the optimisation problem will be:

$$\underset{\beta_0, \beta_{11}, \beta_{12}, \cdots, \beta_{p1},\beta_{p2},\epsilon_1, \cdots, \epsilon_n}{\text{Maximise }}M \text{ s. t. }$$
$$\sum_{j=1}^p\sum_{k=1}^2 \beta_{jk}^2=1$$
$$y_i\begin{pmatrix}\beta_0 + \sum_{j=1}^p\beta_{j1}x_{ij} + \sum_{j=1}^p\beta_{j2}^2x^2_{ij} \end{pmatrix}\geq M(1-\epsilon_i)\,\,\forall i \in \{1,\cdots,n\}$$
$$\epsilon_i \geq 0\,\,\forall i \in \{1,\cdots,n\}\,\,, \sum_{i=1}^n \epsilon_i \leq C$$

In this enlarged feature space, the decision boundary is linear. However, in the original future space, the decision boundary is in the form $q(x)=0$ where $q$ is a quadratic polynomial, adn its solutions are generally non-linear. In extension, we can enlarge the feature space with higher polynomial terms or interaction terms.

```python
import warnings
warnings.filterwarnings("ignore")

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.datasets import (make_moons, load_iris)
from sklearn.preprocessing import (PolynomialFeatures, StandardScaler)
from sklearn.svm import LinearSVC, SVC, LinearSVR, SVR
from sklearn.model_selection import train_test_split
```

To achieve this in SKLearn, use `PolynomialFeatures` to transform before training.

```python
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
```

```python
# Transform to 3rd degree polynomial features
polyfeatures1 = PolynomialFeatures(degree=3)
scaler1 = StandardScaler()
X_expt1 = polyfeatures1.fit_transform(X_train)
X_expt1 = scaler1.fit_transform(X_expt1)

# Train on polynomial features
clf_expt1 = LinearSVC(C=10, loss='hinge', max_iter=1000000)
clf_expt1.fit(X_expt1, y_train)
```

    LinearSVC(C=10, class_weight=None, dual=True, fit_intercept=True,
              intercept_scaling=1, loss='hinge', max_iter=1000000,
              multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,
              verbose=0)

It is not hard to see that there are endless ways to enlarge the feature space and can come up with many features. This computationally becomes unmanageable. The support vector machine allows us to enlarge the feature space used by the support vector classifier in a way that leads to efficient computations.

### Nonlinear SVM Classification - Using Kernels

The Support Vector Machine extends the support vector classifier that results from enlarging the feature space using kernels. This results in a method that is more efficient computationally.

The following from SKLearn implements this using a 3rd degree polynomial kernel.

```python
# Use 3rd degree polynomial and then train SVC on it
clf_expt12= SVC(kernel='poly', degree=3, coef0=1, C=10)
clf_expt12.fit(X_train, y_train)
```

    SVC(C=10, break_ties=False, cache_size=200, class_weight=None, coef0=1,
        decision_function_shape='ovr', degree=3, gamma='scale', kernel='poly',
        max_iter=-1, probability=False, random_state=None, shrinking=True,
        tol=0.001, verbose=False)

The solution to the SVC problem involves only the inner products of the observations (as opposed to the observations themselves). The inner product of two vectors $\vec{a}$ and $\vec{b}$ of length $r$, $\langle\,\vec{a},\vec{b}\rangle$ is $\sum_{i=1}^ra_ib_i$

```python
a, b = np.array([3,4]), np.array([2,7])
print(np.dot(a,b))
```

    34

The inner product of two observations $x_i,x_{i'}$, denoted by $\langle x_{i},x_{i'}\rangle$ is hence
$$\langle x_{i},x_{i'}\rangle = \sum_{j=1}^p x_{ij}x_{i'j}$$

The linear SVC solution can then be represented as:

$$f(x) = \beta_0 + \sum_{i=1}^n \alpha_i \langle x,x_{i}\rangle$$

where there are $n$ parameters, $\alpha_i \forall i \in \{1,\cdots,n\}$

To estimate the parameters $\alpha_i, i \in \{1,\cdots,n\}$ and $\beta_0$, we need the inner products $\langle x,x_{i}\rangle$ of all the pairs of training observations.

It turns out that $\alpha_i$ is nonzero for only the support vectors in the solution. So if the training observation is not the support vector then $\alpha_i=0$. So if $S$ is the collection of indices of these support points, we can rewrite the solution function in the form:

$$f(x) = \beta_0 + \sum_{i\in S}^n \alpha_i \langle x,x_{i}\rangle$$ which involve far fewer terms.

To expand the solution, instead of using the inner product $\langle x,x_{i}\rangle$ we use a generalisation of the inner product in the form: $$K(x_i,x_{i'})$$

where $K$ is some function we refer to as a kernel. A kernel is a function that quantifies the similarity of two observations. If $K_{\text{linear}}(x_i,x_{i'}) = \sum_{j=1}^p x_{ij}x_{i'j}$ then $K_{\text{linear}}$ is a linear kernel and the solution returns the support vector classifier.

Instead, if the kernel is now $K_{\text{polynomial}}(x_i,x_{i'}) = (1 + \sum_{j=1}^p x_{ij}x_{i'j})^d$ where $d$ is a positive integer then we have $K_{\text{polynomial}}$ the polynomial kernel of degree $d$

Using a nonlinear kernel to perform classification results in the support vector machine. Now, the solution has the form:

$$\begin{align}f(x) &= \beta_0 + \sum_{i\in S}^n \alpha_i K_{\text{polynomial}}(x_i,x_{i'})\\&=\beta_0 + \sum_{i\in S}^n \alpha_i (1 + \sum_{j=1}^p x_{ij}x_{i'j})^d\end{align}$$

Another popular choice is the radial kernel, $K_{\text{radial}}(x_i,x_{i'}) = \exp(-\gamma \sum_{j=1}^p (x_{ij}-x_{i'j})^2)$ where $\gamma$ is a positive constant. Naturally, the solution to the radial kernel has the form:

$$\begin{align}f(x) &= \beta_0 + \sum_{i\in S}^n \alpha_i K_{\text{radial}}(x_i,x_{i'})\\&=\beta_0 + \exp(-\gamma \sum_{j=1}^p (x_{ij}-x_{i'j})^2)\end{align}$$

```python
# Feature Scaling
scaler2 = StandardScaler()
X_expt3 = scaler1.fit_transform(X_train)

# Train on RBF Kernel
clf_expt13 = SVC(kernel='rbf', gamma=5, C=0.001)
clf_expt13.fit(X_expt3, y_train)
```

    SVC(C=0.001, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
        decision_function_shape='ovr', degree=3, gamma=5, kernel='rbf', max_iter=-1,
        probability=False, random_state=None, shrinking=True, tol=0.001,
        verbose=False)

How does the radial kernel work? Given an unseen test observation $x^*$ is far from a training observation $x_i$ in terms of Euclidean distance, then $\sum_{j=1}^p (x^*_{j}-x_{ij})^2)$ is large and hence $K_{\text{radial}}(x^*,x_{i})$ is small. Then $x_i$ will play no role in $f(x^*)$. Training observations far from $x^*$ will play no role in the prediction for the class $x^*$

This means the radial kernel has very local behaviour, where only nearby training observations have an effect on the class label of the test observation.

The advantage of using kernels  is computational. When using kernels, we only compute $K$ for every pair of the observations. This can be done without working explicitly working on an enlarged feature space. In enlarged feature space solutions, the feature space could be so large the computations are intractable. In the case of the radial kernel, the feature space is implicit and infinite-dimensional, so we cannot perform the mathematical calculations anyway.

### SVM Regression (SVR)

The SVM algorithm is also versatile: It can support linear and nonlinear regression. The idea is to reverse the objective: instead of finding the largest possible street between two classes, we now find as many instances as possible on the street. The width of the street is controlled by a parameter $\epsilon$.

```python
# Linear SVR, no polynomial features
svr1 = LinearSVR(epsilon=1.5)
svr1.fit(X_train, y_train)
```

    LinearSVR(C=1.0, dual=True, epsilon=1.5, fit_intercept=True,
              intercept_scaling=1.0, loss='epsilon_insensitive', max_iter=1000,
              random_state=None, tol=0.0001, verbose=0)

Similarly, apply the polynomial kernel transformation if we want a SVM with a polynomial (not linear) solution.

```python
# SVR, with polynomial features
svr2 = SVR(kernel='poly', degree=2, C=100, epsilon=0.1)
svr2.fit(X_train, y_train)
```

    SVR(C=100, cache_size=200, coef0=0.0, degree=2, epsilon=0.1, gamma='scale',
        kernel='poly', max_iter=-1, shrinking=True, tol=0.001, verbose=False)