# Ejercicios tema 1
Blanca Cano Camarero

ANTES DE COMENZAR LAS PRÁCTCAS, LEED LE README

```python
# import básicos
from sympy import *
## declaraciones
x = symbols("x")
```

### Ejercicio 1
1.- Demuestre que la ecuación $x^3+4 x^2=10$ tiene una única raíz en el intervalo $[1,2]$. Aproxime dicha raíz con el método de bisección con al menos 3 cifras decimales exactas. ¿Cuántas iteraciones serán necesarias para conseguir 5 cifras decimales exactas? Aproxime también la raíz con el método de Newton-Raphson partiendo del extremo adecuado hasta que la diferencia en valor absoluto, entre dos aproximaciones consecutivas sea inferior a $10^{-3}$.

```python
# La siguiente función se anula en donde la ecuación tiene solución
f = x**4 + 4*x**2 - 10

print(f'en f(2)={f.evalf(subs={x:2})} y f(1)={f.evalf(subs={x:1})}')

# análisis de la derivada
df = diff(f,x)
print(f'Los puntos donde se anula l aderiva son {solve(df)}')
print(f'en df(-1)={df.evalf(subs={x:-1})} y df(1)={df.evalf(subs={x:1})}')

```

    en f(2)=22.0000000000000 y f(1)=-5.00000000000000
    Los puntos donde se anula l aderiva son [0, -sqrt(2)*I, sqrt(2)*I]
    en df(-1)=-12.0000000000000 y df(1)=12.0000000000000

Por ser f un polinomio (clase infinito) y  puesto que la derivada solo se anula en 0 y toma valores positivos en adelante; deducimos que la función $f$ es creciente a partir de 0.

Además gracias al teorema de Bolzano sabemos que se anulará en un punto por ser $f(1)<0<f(2)$ continua y creciente en ese intervalo.

#### Consideraciones bisección

```python
import biseccion as bis
# Para tres cifras decimales de precisión el error será (10**(-3))

n3 = bis.calcIterations(1,2,10**(-3))
n5 = bis.calcIterations(1,2,10**(-5))
print(f'El número mínimo de iteraciones para tres cifras es de {n3}')
print(f'El número mínimo de iteraciones para cinco cifras es de {n5}')

def f(x):
    return x**4+ 4*x**2 - 10

## Calculamos el valor de la raíz con bisección
raiz, ms_salida = bis.bisection(f,1,2,10**(-5))
print(f'La solución buscada es {raiz}')
```

    El número mínimo de iteraciones para tres cifras es de 10.0
    El número mínimo de iteraciones para cinco cifras es de 17.0
    La solución buscada es 1.3197250366210938

#### Consideraciones Newton-Raphson

```python
from newtonRaphson import newtonRaphsonSinInicio
f = x**4 + 4*x**2 - 10
raiz, ms_salida, num_iter = newtonRaphsonSinInicio(f,1,2,10**(-3))

print(f'La solución con newtonRapshon es {raiz}, que se ha alcanzado en {num_iter} iteraciones')
```

    La solución con newtonRapshon es 1.3197190191327715, que se ha alcanzado en 4 iteraciones

## Ejercicio 2

2.- Encuentre una aproximación de la raíz cúbica de 25 con dos decimales exactos, usando el algoritmo de bisección.

```python
#Esta función se anula en la raíz cúbica de 25
f = x**3 -25

print(f'en f(2)={f.evalf(subs={x:2})} y f(3)={f.evalf(subs={x:3})}')

# análisis de la derivada
df = diff(f,x)
print(f'Los puntos donde se anula la aderiva son {roots(df)}')
print(f'en df(-1)={df.evalf(subs={x:-1})} y df(1)={df.evalf(subs={x:1})}')

```

    en f(2)=-17.0000000000000 y f(3)=2.00000000000000
    Los puntos donde se anula la aderiva son {0: 2}
    en df(-1)=3.00000000000000 y df(1)=3.00000000000000

Por tanto (Como era completamente de esperar) la función es creciente a partir de 0 y gracias a bolzano en la semirecta real positiva existirá una única solución.

```python
#la calculamos por bisección
def f(x):
    return x**3 -25

## Calculamos el valor de la raíz con bisección
raiz, ms_salida = bis.bisection(f,2,3,10**(-2))
print(f'La solución buscada es {raiz} donde (25^(1/3)= {25**(1/3)}) ')
```

    La solución buscada es 2.9296875 donde (25^(1/3)= 2.924017738212866)

## Ejercicio 3

3.- Use el método de Newton-Raphson para aproximar las soluciones de las siguientes ecuaciones con precisión $10^{-5}$ , partiendo de
un valor adecuado, próximo a cada una de ellas en cada caso.

i) $x^3-x-1 = 0$  en $[1,2]$.

ii) $3x=2+x^2-e^x$.

iii) $x^2+10 cos x+x=0$.

#### i) $x^3-x-1 = 0$  en $[1,2]$.
$f(x) = x^3-x-1$
Su derivada es $x^2 -1$ positiva en tal intervalo y crecente, por tanto habrá una solucío a lo más.
$f(1)<0<f(2)$ que por continuidad y bolzano tenemos la certeza de que la solución será única.

La función netonRaphsonSinInicio directamente coloca la semilla en su sitio usando el criterio de la segunda derivada.

```python
from newtonRaphson import newtonRaphsonSinInicio
f = x**3-x-1
raiz, ms_salida, num_iter = newtonRaphsonSinInicio(f,1,2,10**(-5), 10**(-5))
print(f'La solución buscada es {raiz}')
```

    La solución buscada es 1.3247190494171253

#### ii) $3x=2+x^2-e^x$.

$x^2 -3x +2 =  e^x$

A ojo, casi sin querer se ve que $x^2 -3x +2 = (x-1)(x-2)$
por tanto sabemos que la parábola tomará valores negarivos entre $(1,2)$ y sabemos que la exponencial es estrictamente positiva y de crecimiento mucho más rápido,a partir de uno, antes de eso será aproximadamente 0.  así que de manera intuitiva habrá una solución en $(-\infty,1)$ cercana al uno.

Puede comprobarse la intuición si representamos la función.

```python
f = exp(x)-x**2 +  3*x -2
plot(f,ylim=(-10,10), xlim=(-3, 3))

```

Además la segunda derivada de $f=exp(x)-x**2 +  3*x -2$ será $f'' = exp(x) - 2$

Que en es positiva en 1 al igual que la función, así que aplicando el critero de los signos podemos partir de 1 que con el método de newton raphson será convergente.

Vamos a ellos:

```python
from newtonRaphson import newtonRaphson
f=exp(x)-x**2 +  3*x -2
raiz, ms_salida, num_iter = newtonRaphsonSinInicio(f,1,10**(-5), 10**(-5))
print(f'La solución buscada es {raiz}, se han realizado {num_iter} iteraciones')
```

    La solución buscada es 0.2575302854263488, se han realizado 4 iteraciones

#### iii) $x^2+10 cos x+x=0$.

En este caso tenemos que tener más cuidado, porque tenemos una parábola $x(x+1)$ y una función periódica, por tanto a lo sumo podemos tener dos soluciones y no queremos dejarnos ninguna.

Pero vamos a aprovechar las herraminetas gráficas para ver el número de solucionesy su intervalo.

```python
f = x**2 + 10*cos(x) + x
plot(f,ylim=(-10,10), xlim=(-5, 5))
```

Podemos ver que existen dos soluciones, cada respectivamente en $(-4,-3)$ y $(-2, -1)$, así que ya estamos preparados para utilizar nuestra implementación de Newton-Raphson.

```python
from newtonRaphson import newtonRaphsonSinInicio
f = x**2 + 10*cos(x) + x
raiz, ms_salida, num_iter = newtonRaphsonSinInicio(f,-2,-1,10**(-5), 10**(-5))
print(f'La solución entre (-2,-1)buscada es {raiz},que se ha alcanzado por {ms_salida}, en  {num_iter} iteraciones.')

raiz, ms_salida, num_iter = newtonRaphsonSinInicio(f,-4,-3,10**(-5), 10**(-10))
print(f'La solución entre (-4,-3)buscada es {raiz}  por {ms_salida}, en  {num_iter} iteraciones.')
```

    La solución entre (-2,-1)buscada es -1.6869398658093508,que se ha alcanzado por precision, en  4 iteraciones.
    La solución entre (-4,-3)buscada es -3.5623314972388327  por tolerancia, en  4 iteraciones.

## Ejercicio 4

4.- Para la función  $ f(x)= 3 x^2+e^x-1$,

i) encuentre, mediante el método de bisección una aproximación de la raíz en $[0,1]$ con, al menos, cuatro decimales exactos, y determine el número de iteraciones realizadas;

ii) encuentre, mediante el método de Newton-Raphson, una aproximación de la raíz en $[0,1]$ con una tolerancia de $10^{-4}$,
partiendo de $x_0=0$, y determine el número de iteraciones realizadas.

### Solución

```python

# BISECCIÓN
import biseccion as bis
# Para cuatro cifras decimales de precisión el error será (10**(-4))
n4 = bis.calcIterations(0,1,10**(-4))
print('Bisección')
print(f'El número mínimo de iteraciones en bisección para cuaro cifras será de {n3}')

def f(x):
    return 3* x**2 + exp(x)-1

## Calculamos el valor de la raíz con bisección
raiz, ms_salida = bis.bisection(f,0,1,10**(-4))
print(f'La solución a priori con bisección es {raiz}')

## NEWTON-RAPHSON
from newtonRaphson import newtonRaphson
f  = 3* x**2 + exp(x)-1
raiz, ms_salida, num_iter = newtonRaphson(f,0,10**(-4), 10**(-4))
print(f'NEWTON-RAPHSON: La solución buscada es {raiz} con {num_iter} iteraciones.')
```

    Bisección
    El número mínimo de iteraciones en bisección para cuaro cifras será de 10.0
    La solución a priori con bisección es 6.103515625e-05
    NEWTON-RAPHSON: La solución buscada es 0.0 con 1 iteraciones.

## Ejercicio 5
5.- Utilice las órdenes apropiadas de Python para aproximar todos los puntos donde se anulan las funciones siguientes (si es necesario, represéntelas gráficamente):

i) $f(x)=x^7-x^4+2$ ,

ii) $f(x)=x^7+\cos  x-3$.
### caso i

```python
fi = x**7 - x**4 +2

print(f'Las soluciones reales y complejas de {fi} son:')
complexN = solve(fi, x)
for i in complexN:
    print(N(i))

```

    Las soluciones reales y complejas de x**7 - x**4 + 2 son:
    -1.00000000000000
    -0.713102145851293 - 0.98823188827156*I
    -0.713102145851293 + 0.98823188827156*I
    1.06110281046036 - 0.376384046559108*I
    1.06110281046036 + 0.376384046559108*I
    0.151999335390936 - 1.01945594669063*I
    0.151999335390936 + 1.01945594669063*I

### caso ii
Hemos representado esta función en el intervalo $[-2.5,2.5]$ para valores fuera de ese intervalo no existe ninguna solución, pues el coseno tiene una amplitud de 1 y en para valores $N < -2.5$ se tiene que $N^7 -3 < -1 <-cos(N)$ y apara $N> 2.5$ se tiene que $N^7>4 > - cos(N) + 3$

Además la solución estará próxima a 1, un dato ideal para utilzar `nsolve`.

```python
# segunda
fii = x**7 + cos(x) -3
plot(fii, xlim=(-2.5,2.5), ylim=(-10, 10) )
# Por tanto calcula la única solución
sol = nsolve(fii, 1)

print(f'La raíz para {fii} es {sol}')
```

## Ejercicio 6
 6.- Aplicar los métodos de aceleración de la convergencia de Aitken y Steffensen a las sucesiones obtenidas para los distintos métodos
programados en esta práctica y comparar los resultados. Para aplicar el método de aceleración de Steffensen, recuerde que para transformar cualquier ecuación de la forma  $ f(x)=0 $  en un problema de puntos fijos  $ g(x)=x $, la forma más simple puede ser definir  $g(x)=x \pm f(x)$.

### Resolución

OJO CUIDADO, algunos resultados no convergen a la solución, y es que el método de aceleración de convergencia no te aseguran de ello, si no que el cocienente de la sucesión de errores se vaya a cero.

También influye cómo se haya obtenido $g$, es decir cómo se ha despejado la x para que quede sola y eso lo reflejo en que de una ecuación verá varios despejes distintos.

```python
# cada tupla contiene la función f de cada ejercicio, la semilla inicial
import aceleracionConvergencia as ac

#Dejo comentado todos las funciones posibles porque de otra forma esto es muy pesado de meter y yo diría que la idea se entiende.
#f=[(x**4 + 4*x**2 - 10,1),(x**3 -25,2),(x**3-x-1,1),(exp(x)-x**2 +  3*x -2,0),
 #(x**2 + 10*cos(x) + x,-2),(1,3* x**2 + exp(x)-1), (x**7 - x**4 +2,0),(x**7 + cos(x) -3,-1)]

f =[x**4 + 4*x**2 - 10,exp(x)-x**2 +  3*x -2, x**3 -25, x**3 -25, exp(x)-x**2 +  3*x -2,exp(x)-x**2 +  3*x -2, x**2 + 10*cos(x) + x ]

for i in range(len(f)):
    gaux=[(10/(x*(x+4)),1),((x**2+2-exp(x))/3,1.4),(x**3+x-25,2.93), (25/x**2, 2.9),
    (exp(x)-x**2 + x+ 3*x -2,0.25), (1/3*(-exp(x)+x**2 + x +2),0.25), (-(x**2 + 10*cos(x)),-3.55),
         ] #(3* x**2 + x+ exp(x)-1 ,0.1)]
    def g(y):
        return gaux[i][0].evalf(subs={x:y})
    print(f'La aceleración de {f[i]} con semilla {gaux[i][1]} en {gaux[i][0]}:')
    ac.tablasAceleracion(g,gaux[i][1])

```

    La aceleración de x**4 + 4*x**2 - 10 con semilla 1 en 10/(x*(x + 4)):
    Xn                        | Aitken                 |Stepheson
    1                         |                        |
    2.00000000000000          |                        |
    0.833333333333333         | 1.46153846153846       |
    2.48275862068966          | 1.51666666666667       |
    0.621306146572104         | 1.37815034468775       | 1.37815034468775
    3.48280861316797          | 1.34654507131928       | 1.34654507131928
    0.383712328149617         | 1.41935299784799       | 1.41935299784799
    5.94500467589520          | 1.29391242913432       | 1.29391242913432
    0.169138629713219         | 1.59554139423208       | 1.59554139423208
    14.1811305766207          | 1.04384454082576       | 1.04384454082576
    0.0387853985175662        | 2.35352259823629       | 2.35352259823629
    63.8382487052382          | -0.974754625444874     | -0.974754625444874
    0.00230910886112447       | 5.88011667047837       | 5.88011667047837
    La aceleración de -x**2 + 3*x + exp(x) - 2 con semilla 1.4 en x**2/3 - exp(x)/3 + 2/3:
    Xn                        | Aitken                 |Stepheson
    1.4                       |                        |
    -0.0317333222815582       |                        |
    0.344080702698839         | 0.265943783305879      |
    0.235899684121873         | 0.260079958147681      |
    0.263200453939663         | 0.256071638322854      | 0.256071638322854
    0.256062326008391         | 0.257530717992165      | 0.257530717992165
    0.257911558186928         | 0.257530268398944      | 0.257530268398944
    0.257431340610680         | 0.257530285669833      | 0.257530285669833
    0.257555968396257         | 0.257530285435571      | 0.257530285435571
    0.257523619331349         | 0.257530285439935      | 0.257530285439935
    0.257532015678931         | 0.257530285439859      | 0.257530285439859
    0.257529836344885         | 0.257530285439861      | 0.257530285439861
    0.257530402005542         | 0.257530285439861      | 0.257530285439861
    La aceleración de x**3 - 25 con semilla 2.93 en x**3 + x - 25:
    Xn                        | Aitken                 |Stepheson
    2.93                      |                        |
    3.08375700000000          |                        |
    7.40892069048313          | 2.92433255570526       |
    389.100179080503          | 3.03418434042021       |
    58909722.4089418          | 2.92150143189044       | 2.92150143189044
    2.04437672938503E+23      | 4.65717060612217       | 4.65717060612217
    8.54442390811763E+69      | 0                      | 0
    6.23804293586468E+209     | nan                    | nan
    2.42742085550551E+629     | nan                    | nan
    1.43032667048473E+1888    | nan                    | nan
    2.92621148325892E+5664    | nan                    | nan
    2.50563109951439E+16993   | nan                    | nan
    1.57308211140660E+50980   | nan                    | nan
    La aceleración de x**3 - 25 con semilla 2.9 en 25/x**2:
    Xn                        | Aitken                 |Stepheson
    2.9                       |                        |
    2.97265160523187          |                        |
    2.82912400000000          | 2.92441611166700       |
    3.12346067761173          | 2.92560466792560       |
    2.56252389048675          | 2.92402638779294       | 2.92402638779294
    3.80718659474958          | 2.92394574212361       | 2.92394574212361
    1.72477196110407          | 2.92455767849169       | 2.92455767849169
    8.40381806645469          | 2.91880646378207       | 2.91880646378207
    0.353986520684614         | 2.95840919148536       | 2.95840919148536
    199.510868259849          | 2.21686496195687       | 2.21686496195687
    0.000628068324895442      | 4.62750182697734       | 4.62750182697734
    63376204.4287542          | -1055.80776950707      | -1055.80776950707
    6.22425757889018E-15      | 99.8438526019454       | 99.8438526019454
    La aceleración de -x**2 + 3*x + exp(x) - 2 con semilla 0.25 en -x**2 + 4*x + exp(x) - 2:
    Xn                        | Aitken                 |Stepheson
    0.25                      |                        |
    0.221525416687741         |                        |
    0.0850071238216746        | 0.257504387790415      |
    -0.578472893261474        | 0.256892775322289      |
    -4.08776841696098         | 0.257526937369315      | 0.257526937369315
    -35.0441476682879         | 0.257732604822869      | 0.257732604822869
    -1370.26887646992         | 0.287528863010341      | 0.287528863010341
    -1883119.86932801         | 0.247735480396528      | 0.247735480396528
    -3546147974739.43         | -2.62847312700751      | -2.62847312700751
    -1.25751654587628E+25     | -34.0968756675720      | -34.0968756675720
    -1.58134786315260E+50     | 1073741481.68260       | 1073741481.68260
    -2.50066106429729E+100    | 0                      | 0
    -6.25330575849244E+200    | nan                    | nan
    La aceleración de -x**2 + 3*x + exp(x) - 2 con semilla 0.25 en 0.333333333333333*x**2 + 0.333333333333333*x - 0.333333333333333*exp(x) + 0.666666666666667:
    Xn                        | Aitken                 |Stepheson
    0.25                      |                        |
    0.342824861104086         |                        |
    0.350477254406297         | 0.351164790235221      |
    0.351188864597114         | 0.351261823088331      |
    0.351255617399848         | 0.351262532505594      | 0.351262532505594
    0.351261884184793         | 0.351262533522134      | 0.351262533522134
    0.351262472557723         | 0.351262533522904      | 0.351262533522904
    0.351262527798990         | 0.351262533522905      | 0.351262533522905
    0.351262532985496         | 0.351262533522905      | 0.351262533522905
    0.351262533472448         | 0.351262533522904      | 0.351262533522904
    0.351262533518167         | 0.351262533522904      | 0.351262533522904
    0.351262533522460         | 0.351262533522904      | 0.351262533522904
    0.351262533522863         | 0.351262533522904      | 0.351262533522904
    La aceleración de x**2 + x + 10*cos(x) con semilla -3.55 en -x**2 - 10*cos(x):
    Xn                        | Aitken                 |Stepheson
    -3.55                     |                        |
    -3.42495494033724         |                        |
    -2.12910813554661         | -3.56335517905669      |
    0.764449677143187         | -4.47597038254420      |
    -7.80201698008086         | -3.77208967876199      | -3.77208967876199
    -61.3908816588683         | -17.7134524671622      | -17.7134524671622
    -3770.13495617214         | -0.308949401778629     | -0.308949401778629
    -14213927.3384249         | 4.40737816363196       | 4.40737816363196
    -202035730382025          | -6.23603888153457      | -6.23603888153457
    -4.08184363509983E+28     | -1899.79022616796      | -1899.79022616796
    -1.66614474614050E+57     | 0                      | 0
    -2.77603831509158E+114    | nan                    | nan
    -7.70638872685652E+228    | nan                    | nan

## Ejercicio 7
 7.- Programar el método de Newton-Raphson acelerado para el caso de una raíz múltiple de una ecuación y comparar los resultados
con los que se obtienen mediante el empleo de los métodos de aceleración anteriores.

### Solución

Lo que relentiza el método de convergencia Newton-Raphson para $f(x)$ son sus raíces múltiples, pero la función $\mu(x) = f(x)/f'(x)$ tiene las misma raíces y ninguna múltiple,  así que será tan sencillo como aplicarle Newton-Raphson a esa función.

```python
import  newtonRaphson as nr

f = (x-2)*(x-2)

#método sin acelerar
raiz,mensaje, iteraciones = nr.newtonRaphson(f,100,max_iter=4)

print(f'Para el método tradicional la solución es {raiz}, tras {iteraciones} iteraciones')

df = 2*(x-2)
m = simplify(f/df)

raiz,mensaje, iteraciones = nr.newtonRaphson(m,100, max_iter=4)

print(f'Para el método acelerado la solución es {raiz}, tras {iteraciones} iteraciones')
```

    Para el método tradicional la solución es 14.25, tras 4 iteraciones
    Para el método acelerado la solución es 2.0, tras 4 iteraciones

## Ejercicio 8

 8.- Programar el conocido algoritmo de Horner para la evaluación de un polinomio y emplearlo de forma reiterativa para el cálculo del
desarrollo de Taylor de orden $ n$ de un polinomio cualquiera. Aprovecharlo también para programar una versión especial del método de Newton-Raphson para polinomios, evaluando tanto  $ p(x_k ) $ como $ p'(x_k)$ mediante el citado algoritmo.

### Solución

  El algoritmo de Horner se basa en sacar factor común $x$ para ahorrarse multiplicaciones, es decir si nuestro polinomio es de la forma $p = \sum_{i=0}^n \alpha_i x^i$ entonces el algoritmo de Horner sería, definimos $H(\alpha_i) = \alpha_i + x H(\alpha_{i+1})$, con $H(\alpha_n)=\alpha_n x$ sería $p =H(\alpha_0)$.

 Si ahora queremos calcular la derivada $d$ del polinomio expresado;

 para todo $i \geq d$
Definimos $H_d(\alpha_i) = \frac{i!}{(i-d)!} \alpha_i + x H_d(\alpha_{i+1})$,  con
 $H_d(\alpha_n)=\frac{n!}{(n-d)!}\alpha_n x$ y sería $p =H_d(\alpha_d)$.

```python
n=4
a = list(symbols('a0:'+ str(n)))
def horner(x,coef, d = 0):

    n = len(coef)
    salida = 0
    for i in range(n,d,-1):
        salida = factorial(i-1)/factorial(i-d-1)*coef[i-1] + x*salida

    return salida

for i in range(n+1):
    print(f' derivada {i}: {horner(x,a,i)}')

```

     derivada 0: a0 + x*(a1 + x*(a2 + a3*x))
     derivada 1: a1 + x*(2*a2 + 3*a3*x)
     derivada 2: 2*a2 + 6*a3*x
     derivada 3: 6*a3
     derivada 4: 0

#### Newton con horner
Puede encontrar la implementación al final del fichero `newtonRaphson.py`

```python

```


```python
import numpy as np
import sympy as sy
import matplotlib.pyplot as plt
from simtk import unit
```

# Harmonic Well Potential

The harmonic well potential is described by the following expression:

\begin{equation}
f(x)=\frac{1}{2} k \left(x^2 + y^2 + z^2 \right)
\end{equation}

Where $k$ is the only parameter and represents the stiffness of the harmonic potential -or the stiffness of the harmonic spring described by the hookes' law-. Notice that the potential for potential dimensions $Y$ and $Z$, has the same shape. In this way we have a three dimensional harmonic well. But let's see here only the proyection over a single dimension $X$ since $Y$ and $Z$ are decorrelated, and there by will behave as $X$.

```python
def harmonic_well(x,k):
    return 0.5*k*x**2
```

```python
k=5.0 * unit.kilocalories_per_mole/ unit.nanometers**2 # stiffness of the harmonic potential

x_serie = np.arange(-5., 5., 0.05) * unit.nanometers

plt.plot(x_serie, harmonic_well(x_serie, k), 'r-')
plt.ylim(-1,5)
plt.xlim(-2,2)
plt.grid()
plt.xlabel("X ({})".format(unit.nanometers))
plt.ylabel("Energy ({})".format(unit.kilocalories_per_mole))
plt.title("Harmonic Well")
plt.show()
```

Different values of $k$ can be tested to graphically see how this parameter accounts for the openness of the well's arms. Or, as it is described below, the period of oscillations of a particle of mass $m$ in a newtonian dynamics.

The hooks' law describes the force suffered by a mass attached to an ideal spring as:

\begin{equation}
F(x) = -k(x-x_{0})
\end{equation}

Where $k$ is the stiffness of the spring and $x_{0}$ is the equilibrium position. Now, since the force is minus the gradient of the potential energy $V(x)$,

\begin{equation}
F(x) = -\frac{d V(x)}{dx},
\end{equation}

we can proof that the spring force is the result of the first harmonic potential derivative:

\begin{equation}
V(x) = \frac{1}{2} k (x-x_{0})^{2}
\end{equation}

And the angular frequency of oscillations of a spring, or a particle goberned by the former potential, is:

\begin{equation}
\omega = \sqrt{\frac{k}{m}}
\end{equation}

Where $m$ is the mass of the particle. This way the potential can also be written as:

\begin{equation}
V(x) = \frac{1}{2} k (x-x_{0})^{2} = \frac{1}{2} m \omega^{2} (x-x_{0})^{2}
\end{equation}

Finnally, the time period of these oscillations are immediately computed from the mass of the particle, $m$, and the stiffness parameter $k$. Given that by definition:

\begin{equation}
T = 2\pi / \omega
\end{equation}

Then:

\begin{equation}
T = 2\pi \sqrt{\frac{m}{k}}
\end{equation}

## Working with this test system

This test system is fully documented in [HarmonicWell class API](../api/_autosummary/uibcdf_test_systems.HarmonicWell.html). Let's see an example of how to interact with it:

```python
from openmolecularsystems import HarmonicWell

open_molecular_system = HarmonicWell(n_particles = 1, mass = 32 * unit.amu,
                                     k=5.0 * unit.kilocalories_per_mole/unit.nanometers**2)
```

The potential expression and the value of the parameters are stored in `potential`:

```python
open_molecular_system.potential_expression
```

$\displaystyle 0.5 k \left(x^{2} + y^{2} + z^{2}\right)$

```python
open_molecular_system.parameters
```

    {'n_particles': 1,
     'mass': Quantity(value=32, unit=dalton),
     'k': Quantity(value=5.0, unit=kilocalorie/(nanometer**2*mole))}

```python
open_molecular_system.coordinates
```

    Quantity(value=array([[0., 0., 0.]], dtype=float32), unit=nanometer)

```python
open_molecular_system.topology
```

```python
open_molecular_system.system
```

    <simtk.openmm.openmm.System; proxy of <Swig Object of type 'OpenMM::System *' at 0x7f513ab0f300> >

There is a method to evaluate the potential at a given positions:

```python
open_molecular_system.evaluate_potential([-1.5, 0.0, 0.0] * unit.nanometers)
```

    Quantity(value=5.625, unit=kilocalorie/mole)

```python
position = np.zeros((200,3), dtype=float) * unit.nanometers
position[:,0] = np.linspace(-5., 5., 200) * unit.nanometers

plt.plot(position[:,0], open_molecular_system.evaluate_potential(position) , 'r-')
plt.ylim(-1,5)
plt.xlim(-2,2)
plt.grid()
plt.xlabel("X ({})".format(unit.nanometers))
plt.ylabel("Energy ({})".format(unit.kilocalories_per_mole))
plt.title("Harmonic Well")
plt.show()
```

```python
open_molecular_system.get_oscillations_time_period()
```

    Quantity(value=7.770948260727904, unit=picosecond)

```python
open_molecular_system.get_standard_deviation(300.0*unit.kelvin)
```

    Quantity(value=0.3453002396733165, unit=nanometer)

### Newtonian dynamics

```python
initial_positions = np.zeros([1, 3], np.float32) * unit.nanometers
initial_positions[0,0] = 1.0 * unit.nanometers

open_molecular_system.set_coordinates(initial_positions)
```

```python
from openmolecularsystems.tools.md import newtonian
traj_dict = newtonian(open_molecular_system, time=50.0*unit.picoseconds, saving_timestep=0.1*unit.picoseconds,
                      integration_timestep=0.02*unit.picoseconds)
```

    100%|██████████| 2500/2500 [00:00<00:00, 8947.70it/s]

We can now plot the trajectory of the x coordinate:

```python
plt.plot(traj_dict['time'], traj_dict['coordinates'][:,0,0])
plt.xlabel('time ({})'.format(traj_dict['time'].unit))
plt.ylabel('X ({})'.format(traj_dict['coordinates'].unit))
plt.show()
```

We can wonder now if the period of these oscillations is in agreement with the value calculated above.

```python
mass = 32 * unit.amu
T = 2*np.pi*np.sqrt(mass/k)

print('The period of the small oscillations around the minimum is',T)
```

    The period of the small oscillations around the minimum is 7.770948260727904 ps

```python
open_molecular_system.get_oscillations_time_period()
```

    Quantity(value=7.770948260727904, unit=picosecond)

```python
plt.plot(traj_dict['time'], traj_dict['coordinates'][:,0,0])
plt.axvline(T._value, color='gray', linestyle='--') # Period of the harmonic oscillations
plt.xlabel('time ({})'.format(traj_dict['time'].unit))
plt.ylabel('X ({})'.format(traj_dict['coordinates'].unit))
plt.show()
```

Remember that the integration timestep must be smaller than $\sim T/10.0$ to guarantee that no artifacts are introduced by the timestep size.

The newtonian dynamics can also include damping. This way we can simulate damped oscillations around the minimum.

```python
traj_dict = newtonian(open_molecular_system, time=50.0*unit.picoseconds, saving_timestep=0.1*unit.picoseconds,
                      integration_timestep=0.02*unit.picoseconds, friction=0.25/unit.picoseconds)
```

    100%|██████████| 2500/2500 [00:00<00:00, 8990.40it/s]

```python
plt.plot(traj_dict['time'], traj_dict['coordinates'][:,0,0])
plt.xlabel('time ({})'.format(traj_dict['time'].unit))
plt.ylabel('X ({})'.format(traj_dict['coordinates'].unit))
plt.show()
```

What would be the friction value needed to enter in the overdamped regime?

```python
traj_dict = newtonian(open_molecular_system, time=50.0*unit.picoseconds, saving_timestep=0.1*unit.picoseconds,
                      integration_timestep=0.02*unit.picoseconds, friction=5.0/unit.picoseconds)
```

    100%|██████████| 2500/2500 [00:00<00:00, 8904.40it/s]

```python
plt.plot(traj_dict['time'], traj_dict['coordinates'][:,0,0])
plt.xlabel('time ({})'.format(traj_dict['time'].unit))
plt.ylabel('X ({})'.format(traj_dict['coordinates'].unit))
plt.show()
```

### Stochastic Dynamics

```python
from openmolecularsystems.tools.md import langevin_NVT

traj_dict = langevin_NVT(open_molecular_system, time=0.5*unit.nanoseconds, saving_timestep=0.5*unit.picoseconds,
                         integration_timestep=0.02*unit.picoseconds,
                         friction=1.0/unit.picoseconds, temperature=300.0*unit.kelvin,
                         initial_velocities='boltzmann')
```

    100%|██████████| 25000/25000 [00:00<00:00, 30010.47it/s]

Let us see the time evolution of the coordinate $x$ of our single particle:

```python
plt.plot(traj_dict['time'], traj_dict['coordinates'][:,0,0])
plt.xlabel('time ({})'.format(traj_dict['time'].unit))
plt.ylabel('X ({})'.format(traj_dict['coordinates'].unit))
plt.show()
```

```python
open_molecular_system.get_standard_deviation(300.0*unit.kelvin)
```

    Quantity(value=0.3453002396733165, unit=nanometer)

```python
np.std(traj_dict['coordinates'][:,0,0])
```

    Quantity(value=0.36140549642467434, unit=nanometer)


# Least Squares Simplification

In this notebook, we explore the implications of zero-mean distributions on a least squares fit.

First, let $L$ be a block luma samples and $C$ the correspoding chroma samples

```python
from sympy import diff, Eq, expand, factor, Idx, IndexedBase, symbols, solveset, summation, init_printing
init_printing()

N = symbols('N', integer='True')
a, b = symbols('alpha, beta', integer='True')
i = symbols('i', cls=Idx)
L = IndexedBase('L', shape=N)
C = IndexedBase('C', shape=N)
```

Next, we define $e$ as the squarred prediction error from a linear model using zero-mean luma samples to predict chroma samples.

```python
e = summation((a*L[i] + b - C[i])**2, (i,0,N-1))
e
```

## Equation for $\alpha$
To minimize the error, we compute the derivative with respect to $\alpha$ and expand the result

```python
da = diff(e, a).expand()
da
```

Since $L$ is zero-mean, the sum Àof L is 0. As such, we can remove the middle term

```python
dazm = (factor(da.args[0]) + factor(da.args[1]) + factor(da.args[2])).subs(summation(L[i], (i,0,N-1)),0)
dazm
```

We solve for $\alpha$ with the derivative equal to 0

```python
solveset(Eq(dazm,0),a)
```

From this equation, we see that when $L$ is zero-mean, the equation of $\alpha$ is simplified.

## Equation for $\beta$
To minimize the error, we compute the derivative with respect to $\beta$ and expand the result

```python
db = diff(e, b).expand()
db
```

Since $L$ is zero-mean, the sum of L is 0. As such, we can remove the middle term

```python
dbzm = (factor(db.args[0]) + factor(db.args[1]) + factor(db.args[2])).subs(summation((L[i]), (i,0,N-1)), 0)
dbzm
```

We solve for $\beta$ with the derivative equal to 0

```python
solveset(Eq(dbzm,0),b)
```

From this equation, we see that when $L$ is zero-mean, the equation of $\beta$ is the average.


# Cauchy-Euler Homogeneus Second Order ODE - Theory

An Cauchy-Euler second order ODE has the following format:

$$x^2 \frac{d^2 y}{dx^2} + px \frac{dy}{dx} + qy(x)=0$$

## The Equation's Solution

The general solution $y(x)$ for this type of equation is a linear combination of two fundamental solutions $y_1(x)$ and $y_1(x)$:

$$y(x) = c_1y_1(x) + c_2y_2(x)$$

where $c_1$ and $c_2$ are constants that must be found by solving de initial value problem.

The simplest way of solving Cauchy-Euler second order ODE is assuming the trial solution $y(x)=x^m$. For that, we have $y'(x)=mx^{m-1}$ and $y''(x)=m(m-1)x^{m-2}$, where $m$ is a constant that we must find.

Substituting $y(x)$ and its derivatives in the equation, we have:

$$x^2[m(m-1)x^{m-2}] + px(mx^{m-1}) + qx^m = 0$$

Applying the distributive property, we have:

$$m^2 x^2 x^{m-2} - m x^2 x^{m-2} + px(mx^{m-1}) + qx^m = 0$$

Applying $x^{a-b} = x^a/x^b$, we have:

$$m^2 \frac{x^2 x^m}{x^2} - m \frac{x^2 x^m}{x^2} + pmx \frac{x^m}{x} + qx^m = 0$$

Simplifying:

$$m^2 x^m - m x^m + pm x^m + qx^m = 0$$

Putting $x^m$ in evidence:

$$x^m[m^2 + m(p-1) + q] = 0$$

Assuming that for $x \neq 0$ there is no $m \in \mathbb{C}$ which satisfacts $x^m = 0$, we have:

$$m^2 + m(p-1) + q = 0$$

Applying the quadratic formula, we have $m_1 = \frac{1 -p + \sqrt{\Delta}}{2}$ and $m_2 = \frac{1 -p - \sqrt{\Delta}}{2}$, where $\Delta = (p-1)^2 - 4q$.

### Fundamental and General Solutions

The fundamental solutions  $y_1(x)$  and  $y_2(x)$  need to be linearly independent. This linear independence can be verified by the Wronskian. The characteristc equation can have two real roots, one real root or two complex roots, and there are a pair of fundamentally linearly independent solutions for each of these cases.

#### Case $\Delta > 0$

In this case, $m_1, m_2 \in \mathbb{R}$ and $m_1 \neq m_2$.

The fundamental solutions are $y_1(x)=x^{m_1}$ and, $y_2(x)=x^{m_2}$.

The general solution is:

$$y(x) = c_1 x^{m_1} + c_2 x^{m_2}$$

where $c_1, c_2 \in \mathbb{R}$.

#### Case $\Delta = 0$

In this case, $m = m_1 = m_2$ and $m \in \mathbb{R}$.

The fundamental solutions are $y_1(x)=x^{m}\ln(x)$ and, $y_2(x)=x^{m}$.

The general solution is:

$$y(x) = c_1 x^{m} \ln(x) + c_2 x^{m}$$

where $c_1, c_2 \in \mathbb{R}$.

#### Case $\Delta < 0$

In this case, $m_1, m_2 \in \mathbb{C}$. These roots are $m_1 = \alpha + \beta i$ and $m_2 = \alpha - \beta i$.

The fundamental solutions are $y_1(x)=x^{m_1}$ and, $y_2(x)=x^{m_2}$. Applying $x^{a+b} = x^ax^b$, $e^{\ln x} = x$ and $(x^a)^b = x^{ab}$, we have:

$$y_1(x) = x^{m_1} = x^{\alpha + \beta i} = x^{\alpha}x^{\beta i} = x^{\alpha}(e^{\ln x})^{\beta i} = x^{\alpha} e^{i\beta \ln x }$$

and

$$y_2(x) = x^{m_2} = x^{\alpha - \beta i} = x^{\alpha}e^{-i\beta  \ln x}$$

Applying Euler's indentity ($e^{i\theta} = \cos\theta + i\sin\theta$), we have:

$$y_1(x) = x^{\alpha} [\cos(\beta \ln x) + i\sin(\beta \ln x)]$$

and

$$y_2(x) = x^{\alpha}[\cos(\beta  \ln x) - i \sin(\beta  \ln x)]$$

Finally, the general solution is:

$$y(x) = c_1 x^{m_1} + c_2 x^{m_2} = x^\alpha [k_1 \cos(\beta \ln x) + k_2 \sin(\beta \ln x)]$$

where $c_1, c_2, k_1, k_2$ are constants.

# Practice with Python

## Solving the Cauchy-Euler Homogeneus Second-Order ODE
In this practice, we will use the algebraic manipulation of the **SymPy** symbolic math package, numeric features of the **NumPy** package and the plotting from the **MatPlotLib** package. For first, we will import these packages.

```python
from sympy import *
from matplotlib import pyplot as plt
import numpy as np
```

Now, we gonna define a function to give the roots of the characteristic equation.

```python
def get_characteristic_eq_roots(p, q):
    m = Symbol('m') #Defines the unknown m

    c_eq = Eq(m**2 + m*(p-1) + q, 0) #Gets SymPy's characteristic equation

    c_eq_roots = solve(c_eq, m) #Gets the l roots of the c_eq

    return c_eq_roots
```

This function returns a list of symbolic roots, that can be two real roots, a single real root or two complex roots. Look that at the following example:

```python
print('Real roots (p=-3, q=2): ', get_characteristic_eq_roots(-3, 2))
print('Complex roots (p=1, q=1): ', get_characteristic_eq_roots(1, 1))
print('Single root (p=3, q=1): ', get_characteristic_eq_roots(3, 1))
```

    Real roots (p=-3, q=2):  [2 - sqrt(2), sqrt(2) + 2]
    Complex roots (p=1, q=1):  [-I, I]
    Single root (p=3, q=1):  [-1]

Now, we gonna define a function to give the fundamental solutions of the ODE.

```python
def get_ODE_fundamental_solutions(x, p, q):
    c_eq_roots = get_characteristic_eq_roots(p, q)

    if len(c_eq_roots) == 1:# Single real root
        m = c_eq_roots[0] #Root lambda

        y1 = x**m #fundamental solution y1(x)
        y2 = ln(x)*(x**m) #fundamental solution y2(x)

        return [y1, y2]

    elif len(c_eq_roots) == 2: #Two roots
        if c_eq_roots[0].is_real == True and c_eq_roots[1].is_real == True: # Real roots
            m1 = c_eq_roots[1] #Root lambda1
            m2 = c_eq_roots[0] #Root lambda2

            y1 = x**m1 #fundamental solution y1(x)
            y2 = x**m2 #fundamental solution y2(x)

            return [y1, y2]

        elif c_eq_roots[0].is_real == False and c_eq_roots[1].is_real == False: # Complex roots
            m1 = c_eq_roots[1] #Root lambda1
            m2 = c_eq_roots[0] #Root lambda2

            alpha = re(m1) #Real part of the roots
            beta = abs(im(m1)) #Absolute imaginary part of the roots

            y1 = (x**alpha)*( cos(beta*ln(x)) + I*sin(beta*ln(x))) #fundamental solution y1(x)
            y2 = (x**alpha)*( cos(beta*ln(x)) - I*sin(beta*ln(x))) #fundamental solution y2(x)

            return [y1, y2]

        else: #Error
            raise Exception('Unexpected answer from the get_characteristic_eq_roots function.')

    else: #Error
        raise Exception('Unexpected answer from the get_characteristic_eq_roots function.')
```

This function returns a list of symbolic fundamental solutions. Look that at the following examples:

```python
x = Symbol('x')

print('p=-3, q=2: ', get_ODE_fundamental_solutions(x, -3, 2))
print('p=1, q=1: ', get_ODE_fundamental_solutions(x, 1, 1))
print('p=3, q=1: ', get_ODE_fundamental_solutions(x, 2, 1))
```

    p=-3, q=2:  [x**(sqrt(2) + 2), x**(2 - sqrt(2))]
    p=1, q=1:  [I*sin(log(x)) + cos(log(x)), -I*sin(log(x)) + cos(log(x))]
    p=3, q=1:  [(I*sin(sqrt(3)*log(x)/2) + cos(sqrt(3)*log(x)/2))/sqrt(x), (-I*sin(sqrt(3)*log(x)/2) + cos(sqrt(3)*log(x)/2))/sqrt(x)]

Now, we gonna define a function to give the symbolic general solution.

```python
def get_ODE_solution(x, p, q, c1, c2):
    fundamental_solutions = get_ODE_fundamental_solutions(x, p, q)

    y1 = fundamental_solutions[0] #Fundamental solution y1(x)
    y2 = fundamental_solutions[1] #Fundamental solution y2(x)

    y = c1*y1 + c2*y2 #General solution y(x)

    return y
```

Testing...

```python
x, c1, c2 = Symbol('x'), Symbol('c1'), Symbol('c2')

print('p=-3, q=2: ', get_ODE_solution(x, -3, 2, c1, c2))
print('p=1, q=1: ', get_ODE_solution(x, 1, 1, c1, c2))
print('p=2, q=1: ', get_ODE_solution(x, 2, 1, c1, c2))
```

    p=-3, q=2:  c1*x**(sqrt(2) + 2) + c2*x**(2 - sqrt(2))
    p=1, q=1:  c1*(I*sin(log(x)) + cos(log(x))) + c2*(-I*sin(log(x)) + cos(log(x)))
    p=2, q=1:  c1*(I*sin(sqrt(3)*log(x)/2) + cos(sqrt(3)*log(x)/2))/sqrt(x) + c2*(-I*sin(sqrt(3)*log(x)/2) + cos(sqrt(3)*log(x)/2))/sqrt(x)

Finally, we will define a function to give the $y(x)$ values for any $x$ input in the ODE solution.

```python
def get_ODE_function(p, q, c1, c2):
    x = Symbol('x')

    symbolic_solution = get_ODE_solution(x, p, q, c1, c2)

    y_function = lambdify(x, symbolic_solution)

    return y_function
```

## Plotting $y(x)$
Plotting $y(x)$ from $y''(x) - 3y'(x) + 2y(x) = 0$ with $c_1 = c_2 = 1$:

```python
p, q, c1, c2 = (-3, 2, 1, 1)
y = get_ODE_function(p, q, c1, c2) #y(x) function
x = np.linspace(0,20,500) #500 dots in [0, 20]
plt.plot(x,y(x).real) #Plot the real part of y(x)
plt.show()
```

Plotting $y(x)$ from $y''(x) + 2y'(x) + 20y(x) = 0$ with $c_1 = c_2 = 1$:

```python
p, q, c1, c2 = (2, 20, 1, 1)
y = get_ODE_function(p, q, c1, c2) #y(x) function
x = np.linspace(0,20,500) #500 dots in [0, 20]
plt.plot(x,y(x).real) #Plot the real part of y(x)
plt.show()
```

Plotting $y(x)$ from $y''(x) + y'(x) + 10y(x) = 0$ with $c_1 = c_2 = 1$:

```python
p, q, c1, c2 = (1, 10, 1, 1)
y = get_ODE_function(p, q, c1, c2) #y(x) function
x = np.linspace(0,20,500) #500 dots in [0, 20]
plt.plot(x,y(x).real) #Plot the real part of y(x)
plt.show()
```


# Projection Experiment
In the open source implementation of [dehazing with color lines by Tomlk](https://github.com/Tomlk/Dehazing-with-Color-Lines), the distance metric that is used during ransac is different from the metric mentioned in [Fattal paper](https://www.cse.huji.ac.il/~raananf/projects/dehaze_cl/). We will explore the validty of this alternative form.

## Fattal's Formula
Fattal paper's defines the similarity metric (distance) between two  direction as follows (paraphrased):

$D = I(x_2) - I(x_1)$ - Direction vector of the color line.
$V = I(x_1)$ - Point on the color line.
$\Omega$ - The set of pixels in an image patch

Each line is associted with pixels $x \in \Omega$ that support it, i.e., pixels in which $I(x)$ is sufficiently close to the line. **This is measured by projecting $I(x) - V$ onto the plane perpendicular to $D$ and computing the norm of the projected vector.** In our implementation we associate a pixel with the line if the norm falls below $2 \times 10^{-2}$.

Let $\hat{D} = I(x) - V$.
Fattal's formula can be written as:

$$Distance = ||\hat{D}-\frac{\hat{D}\cdot D}{||D||^{2}}D||$$
The projection onto the plane is calculated as $\hat{D}$ minus its component that is not orthogonal to $D$.
$$ $$

## Tomlk's Formula
The formula used in Tomlk's implementation can be written as:

$$||\frac{D\times\hat{D}}{||D||}||$$

No further explanation is provided, so we will have to figure it out ourselves. Intuitively, the fraction makes sense as the norm of the cross product will increase when the angle between $D$ and $\hat{D}$ is larger (the norm of the cross product equals the [area of the parallelogram](https://en.wikipedia.org/wiki/Cross_product)). The denominator $||D||$ may possibly be a scalar that removes any effect the magnitude of $D$ may have (thereby treating it as a unit vector). Before proving the equality of these formulas, we can test them to rule out some obvious counter examples.

```python
import numpy as np
np.random.seed(2)

def explicit_project(vec1, vec2):
    """ Returns length of projection of vec2 onto a plane perpendicular to vec1.
        Fattal's Formula.
    """
    dot = np.dot(vec2, vec1)
    squared_norm = np.linalg.norm(vec1)**2
    proj_onto_vec1 = (dot / squared_norm) * vec1
    proj_onto_plane = vec2 - proj_onto_vec1
    result = np.linalg.norm(proj_onto_plane)
    result = abs(result)
    return result

def implicit_project(vec1, vec2):
    """ Returns the length of the cross project scaled to the lenght of vec1.
        Tomlk's Formula.
    """
    cross_product = np.cross(vec1, vec2)
    norm = np.linalg.norm(vec1)
    scaled_cross_product = cross_product / norm
    result = np.linalg.norm(scaled_cross_product)
    result = abs(result)
    return result

differences = list()
for i in range(1000):
    v1 = np.random.rand(3)
    v2 = np.random.rand(3)
    # Randomly scale random vectors
    v1 = v1 * np.random.randint(1000)
    v2 = v2 * np.random.randint(1000)
    explicit = explicit_project(v1, v2)
    implicit = implicit_project(v1, v2)
    difference = explicit - implicit
    differences.append(difference)

differences = np.array(differences)
print('The largest difference is equal to : {}'.format(differences.max()))
# The largest difference is equal to : 2.2737367544323206e-13
```

    The largest difference is equal to : 2.2737367544323206e-13

Such a small difference over so many samples is a good sign. Now let's prove the equality.

## Equality proof

$$||\hat{D}-\frac{\hat{D}\cdot D}{||D||^{2}}D|| = ||\frac{D\times\hat{D}}{||D||}||$$
$$||\hat{D}-proj_{D}\hat{D}|| = ||\frac{D\times\hat{D}}{||D||}|| $$

Given the propetry $||\frac{\vec{1}}{x}|| = \frac{||\vec{1}||}{x}$

$$ = \frac{||D \times \hat{D}||}{||D||} $$

Given the property $||a\times b|| = ||a||\hspace{0.1cm}||b||sin(\theta)$

$$ = \frac{||D||\hspace{0.1cm}||\hat{D}||sin(\theta)}{||D||}$$

$$= ||\hat{D}||sin(\theta)$$

Now if we draw $D$, $\hat{D}$, $P = proj_{D}\hat{D}$, and $W = \hat{D}-P$$ we have:

We see the vectors draw compose a right triangle :
With a hypotenuse of length $||\hat{D}||$
Two legs of length $||P|| = ||proj_{D}\hat{D}||$
and  $||W|| = ||\hat{D}-proj_{D}\hat{D}||$

Let us consider the trigonometric definition of sin.

This means that
\begin{align}
sin(\theta) &= \frac{||W||}{||\hat{D}||} \newline \newline
&= \frac{||\hat{D}-proj_{D}\hat{D}||}{||\hat{D}||}
\end{align}

Now we all necessary steps to complete the proof:

\begin{align}
||\hat{D}-proj_{D}\hat{D}|| &= ||\frac{D\times\hat{D}}{||D||}||\newline  \newline
 &= \frac{||D \times \hat{D}||}{||D||} \newline  \newline
 &= \frac{||D||\hspace{0.1cm}||\hat{D}||sin(\theta)}{||D||} \newline  \newline
 &= ||\hat{D}||sin(\theta) \newline \newline
 &= ||\hat{D}|| \frac{||\hat{D}-proj_{D}\hat{D}||}{||\hat{D}||} \newline \newline
 &= ||\hat{D}-proj_{D}\hat{D}|| \newline \newline
 ||\hat{D}-proj_{D}\hat{D}|| &= ||\hat{D}-proj_{D}\hat{D}||
\end{align}

$$ True $$


# Probability in Python

# Importing packages

You can ignore this part for now.

```python
import numpy as np
```

# Import statistics module
We will use scipy.stats, which has several functions for statistics and probability distributions.

```python
import scipy.stats as st
```

# Import pandas, matplotlib

```python
import pandas as pd
import matplotlib.pyplot as plt
```

# Function for uniform outcome

$n$: number of outcomes in the sample space

Output: $m$ outcomes selected uniformly at random from 1 to $n$

```python
def uniform(n, m):
  return np.random.randint(1, n+1, size = m)
```

# Toss a coin

Toss once, 10 times and 100 times

1: Heads and 2: Tails

```python
print(uniform(2, 1))
print(uniform(2, 10))
print(uniform(2,100))
```

    [1]
    [2 1 1 1 1 1 2 2 1 2]
    [1 1 2 2 1 1 1 1 1 1 1 2 2 1 1 1 1 2 2 1 1 1 1 2 2 2 1 1 1 2 2 2 1 2 1 1 2
     1 2 2 1 1 2 1 2 1 1 2 1 2 2 2 2 1 1 2 1 1 1 2 2 1 2 2 1 2 2 1 2 2 2 1 1 2
     1 2 1 1 2 1 1 2 1 2 1 1 2 1 2 2 1 2 1 2 1 1 1 1 1 2]

# Throw a die

Throw once, 10 times and 100 times

```python
print(uniform(6, 1))
print(uniform(6, 10))
print(uniform(6,100))
```

    [4]
    [6 2 2 5 3 2 3 1 1 3]
    [3 6 2 3 1 1 3 5 5 3 5 5 2 6 1 4 5 1 6 2 6 5 3 3 1 2 3 3 6 2 6 5 6 5 3 3 1
     2 3 3 1 6 1 5 6 5 5 2 4 3 4 6 2 5 6 2 2 3 3 5 2 1 1 4 3 2 1 1 4 6 1 4 2 6
     4 5 4 5 5 2 3 1 2 1 5 1 1 3 5 3 5 3 5 6 1 4 5 1 6 1]

# Estimating probability by simulation - Monte Carlo

The probability of an event $A$ can be estimated as follows. We can simulate the experiment repeatedly and independently, say $N$ times, and count the number of times the event occurred, say $N_A$.

A good estimate of $P(A)$ is the following:
$$P(A) \approx \frac{N_A}{N}$$
As $N$ grows larger and larger, the estimate becomes better and better. This method is generally termed as Monte Carlo simulation.

We will first evaluate probability of coin toss described above using Monte Carlo simulations. There are two steps: generate a large number of tosses and count the number of heads or tails. These two steps can be written in a single loop usually.

You should run the simulation multiple times to see what probability estimate is obtained each time. You will see that the estimate is close to 0.5.

```python
no_heads = 0   #variable for storing number of heads
for i in range(1000): #repeat 1000 times
  if uniform(2, 1) == 1: #check if coin toss is heads
    no_heads = no_heads + 1
print(no_heads/1000) #probability estimate by Monte Carlo
```

    0.492

# Probability of die showing a number

We will modify the Monte Carlo simulation above for finding the probability that a dies shows a number falling in an event $A$. You will see that the estimate is close to $P(A)$. If you change the loop iterations to 10000, the estimate will be much closer to $P(A)$ and more consistent as well.

```python
no = 0   #variable for storing number of event occurence
for i in range(10000): #repetitions
  die = uniform(6,1)  #experiment
  if die == 1 or die == 3: #Event
    no = no + 1
print(no/10000) #probability estimate by Monte Carlo
```

    0.3361

# Birthday problem

In a group of $n$ persons, what is the chance that some two have the same birthday? Assume birthday of a person is uniformly distributed in $\{1,2,\ldots,365\}$ and is independent of all other birthdays. Most people will think that you need at least 100 persons before you start seeing same birthdays. However, surprisingly perhaps, even with 23 persons there is a 50% chance of two sharing a birthday.

Event $A$: some two have same birthday

Event $A^c$: no two have same birthday

$A^c$: (Birthday 1 on any date $B_1$) and (Birthday 2 on any date other than $B_1$) and (Birthday 3 on any date other than $B_1$, $B_2$) and ... and (Birthday $n$ on any day other than $B_1,B_2,\ldots,B_{n-1}$)

$P(A^c)= 1 \cdot \left(1 - \frac{1}{365}\right)\left(1 - \frac{2}{365}\right)\cdots\left(1 - \frac{n-1}{365}\right)$

If $n=10$, what is the chance? If $n=30$, what is the chance?

We will do a Monte Carlo simulation to estimate the probability and compare with the calculation above.

```python
no = 0   #variable for storing number of event occurence
n = 60 #number of persons
print(1 - np.prod(1-np.arange(1,n)/365)) #probability from expression

for i in range(1000):
  B = np.zeros(366)   #array to keep track of birthdays seen
  for j in range(n): #generate birthdays for each person
    Bi = uniform(365, 1) #i-th birthday
    if B[Bi]  == 0:  #if Bi is seen for the first time
      B[Bi] = 1  #make note that Bi has been seen
    else:
      no = no + 1   #if Bi has been seen before, then two birthdays are same
      break   #we can stop generating more birthdays and exit loop early

print(no/1000) #probability estimate by Monte Carlo
```

    0.994122660865348
    0.996

# Monty Hall problem

Here is the problem taken from the [Wiki page](https://en.wikipedia.org/wiki/Monty_Hall_problem).

> Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

The assumptions (also taken from [Wiki](https://en.wikipedia.org/wiki/Monty_Hall_problem)) are as follows:
1. Car and goats are placed at random behind the doors.
2. Host always picks a door not chosen by contestant.
3. Host always reveals a goat and not a car.
4. Host always offers a choice to switch from the original door to the other closed door.

Under the above assumptions, here are the probabilities of winning.

P(win if contestant chooses to switch) = 2/3

P(win if contestant does not switch) = 1/3

You can see the Wiki page for the computation. Let us simulate and find the probability of winning under switch by Monte Carlo.

```python
no = 0   #variable for storing number of event occurence
for i in range(1000):
  car_loc = uniform(3, 1)
  if car_loc == 1:
    goat1_loc = 2; goat2_loc = 3
  elif car_loc == 2:
    goat1_loc = 1; goat2_loc = 3
  else:
    goat1_loc = 1; goat2_loc = 2

  contestant_orig = uniform(3, 1)
  if contestant_orig == goat1_loc:
    host_reveal_loc = goat2_loc; other_closed_door = car_loc
  elif contestant_orig == goat2_loc:
    host_reveal_loc = goat1_loc; other_closed_door = car_loc
  else:
    host_reveal_loc = goat1_loc; other_closed_door = goat2_loc
  if other_closed_door == car_loc:
    no = no + 1

print(no/1000) #probability estimate by Monte Carlo
```

    0.663

# Polya's urn scheme

Suppose an urn contains $r$ red and $b$ blue balls. The experiment proceeds in multiple steps, where Step $i$ is as follows:

Step $i$: Draw a ball at random, note down its colour and replace it in the urn. Add $c$ more balls of the same colour to the urn.

Let $R_i$ be the event that the $i$-th ball drawn is red. Let $B_i$ be the event that the $i$-th abll drawn is black.

Clearly, $P(R_1) = \frac{r}{r+b}$ and $P(B_1)=\frac{b}{r+b}$. It is perhaps surprising that, irrespective of $c$, we have, for all $i$,
$$P(R_i) = \frac{r}{r+b}, P(B_i) = \frac{b}{r+b}.$$
To prove the above, you can use induction. Assume that the above is true for $i$ and show it is true for $i+1$. Starting with $i=1$, by induction, the statement becomes true.

We will setup a Monte Carlo simulation for verifying $P(R_i)$ above for a few steps.

```python
no = 0   #variable for storing number of event occurence
r = 10; b = 5 #assume 1 to r is red and r+1 to r+b is blue
print(r/(r+b))
for i in range(1000):
  r = 10; b = 5
  c = 3
  for j in range(5): #do 5 steps
    if uniform(r+b, 1) <= r:
      r = r + c
    else:
      b = b + c
  if uniform(r+b, 1) <= r: #in the 6th step, count if red ball drawn
    no = no + 1
print(no/1000) #probability estimate by Monte Carlo
```

    0.6666666666666666
    0.66

# Gambler's ruin (simple random walk)

A gambler starting with $k$ units of money plays the following game at a casino:

* If he has $\ge 1$ units of money, a coin is tossed. If heads, the casino pays him 1 unit. If tails, he loses 1 unit to the casino.
* If he loses all money, he goes bankrupt and stops.
* If he gets $N$ units of money, he wins and stops playing.

If $p$ is the probability of heads and $q=1-p$, it can be shown that
$$\text{Pr}(\text{Bankruptcy})=\begin{cases}
1-k/N,&\text{ if }p=q=1/2,\\
\frac{\left(\dfrac{q}{p}\right)^k-\left(\dfrac{q}{p}\right)^N}{1-\left(\dfrac{q}{p}\right)^N}, &\text{ if }p\ne q.
\end{cases}$$
You can see some details of the proof of the above in the [Wiki page](https://en.wikipedia.org/wiki/Gambler%27s_ruin). Suppose $x_k$ denotes the probability of bankruptcy starting with $k$ units. The main idea is to condition on the first toss and derive the following recursive equation:
$$\begin{align}
x_k&=P(\text{Bankruptcy}\ |\ \text{first toss is head})\ p\ +\ P(\text{Bankruptcy}\ |\ \text{first toss is tail})\ q\\
&=x_{k+1}p+x_{k-1}q
\end{align}$$
with boundary conditions $x_0=1$ and $x_N=0$. Solution of the recursive equation results in the above closed form expression for $x_k$.

We are interested in Monte Carlo simulation of Gambler's ruin and verification of the formula for $x_k$. First, we consider the case $p=1/2$.

```python
no = 0   #variable for storing number of event occurence
k = 5; N = 10
print(1-k/N)
for i in range(1000):
  k = 5
  while k > 0 and k < N:
    if uniform(2, 1) == 1:
      k = k + 1
    else:
      k = k - 1
  if k == 0:
    no = no + 1
print(no/1000) #probability estimate by Monte Carlo
```

    0.5
    0.505

# Toss a biased coin

For $p\ne q$, we require a method to toss a biased coin. This is accomplished by the following function that generates $m$ coin tosses with probability of heads equal to $p$. Note that a value of 1 represents heads and 2 represents tails as before.

```python
def biased(p, m):
  return 2-(np.random.rand(m) < p)
```

```python
no_heads = 0   #variable for storing number of heads
p = 0.25
print(p)
for i in range(1000):
  if biased(p, 1) == 1:
    no_heads = no_heads + 1
print(no_heads/1000) #probability estimate by Monte Carlo
```

    0.25
    0.253

# Biased Gambler's ruin

We now simulate the biased version of Gambler's ruin.

```python
no = 0   #variable for storing number of event occurence
p = 0.35
qbyp = (1-p)/p
k = 5; N = 10
print((qbyp**k-qbyp**N)/(1-qbyp**N))
for i in range(1000):
  k = 5
  while k > 0 and k < N:
    if biased(p, 1) == 1:
      k = k + 1
    else:
      k = k - 1
  if k == 0:
    no = no + 1
print(no/1000) #probability estimate by Monte Carlo
```

    0.9566941509920124
    0.964

# Casino die game
Throw a pair of die. A player bets $k_1$ units of money on whether the sum of the two numbers is Under 7 or Over 7, and $k_2$ units on Equal to 7. For Under 7 and Over 7, the returns are $a$:1, while, for Equal to 7, the returns are $b$:1, if the player wins the bet. If the bet is lost, the unit of money goes to the casino.

The strategy for betting will be to independently and randomly select one of the 3 bets. The simulation will track the average return over a large number of trails.

```python
a = 1.0; b = 4.0
k1 = 1; k2 = 1
print((((a-1)*5-7)*k1+((b-1)-5)*k2)/6/3) #expected gain
avg_return = 0
for i in range(1000):
  bet = uniform(3,1) #1 - Under 7, 2 - Over 7, 3 - Equal to 7
  sum = uniform(6,1) + uniform(6,1)
  if ((bet == 1) and (sum < 7)) or ((bet == 2) and (sum > 7)): #win for Under 7 or Over 7 bet
    avg_return = avg_return + k1*(a-1)/1000
  if (bet == 3) and (sum == 7): #win for Equal to 7 bet
    avg_return = avg_return + k2*(b-1)/1000
  if ((bet == 1) and (sum >= 7)) or ((bet == 2) and (sum <= 7)): #loss for Under 7 or Over 7 bet
    avg_return = avg_return + (-k1)/1000
  if (bet == 3) and (sum != 7): #loss for Equal to 7 bet
    avg_return = avg_return + (-k2)/1000

print(avg_return) #simulated gain
```

    -0.5
    -0.5080000000000005

# Expected value of common distributions
The module has functions for generating binomial, geometric, Poisson and other distributions. We will generate a large number of samples and compute the average value and compare with the expected value.

```python
#binomial(20,0.3)
print(20*0.3) #expected value
x = st.binom.rvs(20,0.3,size=1000)
print(np.sum(x)/1000) #average value in simulation
```

    6.0
    6.062

```python
#geometric(0.3)
print(1/0.3) #expected value
x = st.geom.rvs(0.3,size=1000)
print(np.sum(x)/1000) #average value in simulation
```

    3.3333333333333335
    3.276

```python
#Poisson(6)
print(6) #expected value
x = st.poisson.rvs(6,size=1000)
print(np.sum(x)/1000) #average value in simulation
```

    6
    6.029

# Balls and bins
Suppose $m$ balls are thrown independently and uniformly at random into $n$ bins. We will compute the expected number of empty bins by simulation and compare with the theoretical value of $n(1-1/n)^m\approx ne^{-m/n}$.

```python
m = 10; n = 3
print(n*((1-1/n)**m)) #expected value
avg_empty_bins = 0
for i in range(1000):
  no_balls = np.zeros(n, dtype=int) #keep track of balls in bins
  for ball in range(m):
    bin = uniform(n, 1)
    no_balls[bin-1] += 1

  no_empty_bins = 0
  for bin in range(n):
    if no_balls[bin] == 0:
      no_empty_bins += 1

  avg_empty_bins += no_empty_bins/1000.0

print(avg_empty_bins) #average value in simulation
```

    0.0520245897474979
    0.05600000000000004

# Common continuous distributions and histograms
Scipy stats module can be used to generate samples from common continuous distributions. We will generate a number of such samples and plot their histogram to confirm that the samples follow the expected density function.

For histograms, we will use the hist() function from the matplotlib.pyplot module imported below.

## Uniform distribution
We will begin with the uniform distribution.

```python
x = st.uniform.rvs(0,3,size=10000)
plt.hist(x,bins=50,range=(0,3),density=True) #blue histogram
plt.plot([-0.2,0,0,3,3,3.2],[0,0,1.0/3,1.0/3,0,0],lw=2) #orange line. uniform[0,3] density
plt.show()
```

## From histogram to density
The code above generates 10000 samples that are supposed to be independent and uniformly distributed in $[0,3]$. The histogram, created using the plt.hist command, uses 100 bins of equal width in the range $[0,3]$. So, the bins are $[0,0.03),[0.03,0.06),\ldots,[2.97,3]$.

Suppose the number of samples that fall into the bin $[0,0.03]$ is $N_0$. Then, by Monte Carlo, we have that
$$P(0<X\le 0.03)\approx \frac{N_0}{10000},$$
where $X$ is a random variable with the sample distribution. Assuming that the density of $X$ satisfies $f_X(x)\approx f_X(0.015)$ over the bin ($0.015$ is the midpoint of the bin), we get
$$P(0<X\le 0.03)\approx 0.03f_X(0.015)\approx\frac{N_0}{10000}.$$
Using the above, we get
$$f_X(0.015)\approx\frac{N_0}{300}.$$
Similarly, if $N_i$ is the number of samples in the $i$-th bin with midpoint $x_i$, we have
$$f_X(x_i)\approx\frac{N_i}{300}.$$
The option density=True in the plt.hist command specifies that the above calculation is to be done.

The plt.plot command plots the expected PDF as a line plot. The parameter lw specifies the linewidth and 2 pts.

Try changing the bin size to see how the plot changes. Does 50 bins look better? Why?

## Exponential distribution
We will next repeat the same for $X\sim$ Exp$(\lambda)$. The PDF is
$$f_X(x)=\lambda\exp(-\lambda x),$$
where $\lambda$ is called the scale parameter. Try changing the various parameters below to see what happens.

```python
x = st.expon.rvs(scale=1,size=10000)
plt.hist(x,bins=50,range=(0,10),density=True) #blue histogram
xp = np.linspace(0,10,50)
plt.plot(xp,st.expon.pdf(xp,scale=1),lw=2) #orange line, exp(\lambda) density
plt.show()
```

## Normal distribution
We will repeat the same for $X\sim$ Normal$(\mu,\sigma^2)$. The PDF is
$$f_X(x)=\frac{1}{\sigma\sqrt{2\pi}}\exp(-(x-\mu)^2/2\sigma^2),$$
where the mean $\mu$ is called the location parameter `loc' and $\sigma$ is called the scale parameter.

```python
x = st.norm.rvs(loc=0, scale=1, size=10000)
plt.hist(x,bins=50,range=(-5,5),density=True) #blue histogram
xp = np.linspace(-5,5,50)
plt.plot(xp,st.norm.pdf(xp,loc=0,scale=1),lw=2) #orange line, normal pdf
plt.show()
```

```python
from sklearn.datasets import load_iris
```

```python
iris = load_iris()
print(iris.DESCR)
```

    .. _iris_dataset:

    Iris plants dataset
    --------------------

    **Data Set Characteristics:**

        :Number of Instances: 150 (50 in each of three classes)
        :Number of Attributes: 4 numeric, predictive attributes and the class
        :Attribute Information:
            - sepal length in cm
            - sepal width in cm
            - petal length in cm
            - petal width in cm
            - class:
                    - Iris-Setosa
                    - Iris-Versicolour
                    - Iris-Virginica

        :Summary Statistics:

        ============== ==== ==== ======= ===== ====================
                        Min  Max   Mean    SD   Class Correlation
        ============== ==== ==== ======= ===== ====================
        sepal length:   4.3  7.9   5.84   0.83    0.7826
        sepal width:    2.0  4.4   3.05   0.43   -0.4194
        petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
        petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
        ============== ==== ==== ======= ===== ====================

        :Missing Attribute Values: None
        :Class Distribution: 33.3% for each of 3 classes.
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
    from Fisher's paper. Note that it's the same as in R, but not as in the UCI
    Machine Learning Repository, which has two wrong data points.

    This is perhaps the best known database to be found in the
    pattern recognition literature.  Fisher's paper is a classic in the field and
    is referenced frequently to this day.  (See Duda & Hart, for example.)  The
    data set contains 3 classes of 50 instances each, where each class refers to a
    type of iris plant.  One class is linearly separable from the other 2; the
    latter are NOT linearly separable from each other.

    .. topic:: References

       - Fisher, R.A. "The use of multiple measurements in taxonomic problems"
         Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
         Mathematical Statistics" (John Wiley, NY, 1950).
       - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.
         (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
       - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
         Structure and Classification Rule for Recognition in Partially Exposed
         Environments".  IEEE Transactions on Pattern Analysis and Machine
         Intelligence, Vol. PAMI-2, No. 1, 67-71.
       - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
         on Information Theory, May 1972, 431-433.
       - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
         conceptual clustering system finds 3 classes in the data.
       - Many, many more ...

# Summarizing the data
iris.data: 2D array containing all data (size 150 x 4)
 - Class 0: Row 0 to Row 49 of iris.data array
 - Class 1: Row 50 to Row 99 of iris.data array
 - Class 2: Row 100 to Row 149 of iris.data array

From the module scipy.stats, we can use the 'describe' command to get summary statistics of an array of data.

```python
# all data
sum_stats = st.describe(iris.data)
print(sum_stats.minmax)
```

    (array([4.3, 2. , 1. , 0.1]), array([7.9, 4.4, 6.9, 2.5]))

```python
print(sum_stats.mean)
print(sum_stats.variance)
```

    [5.84333333 3.05733333 3.758      1.19933333]
    [0.68569351 0.18997942 3.11627785 0.58100626]

```python
#Class 0
sum_stats = st.describe(iris.data[:50,:])
print(sum_stats.minmax)
print(sum_stats.mean)
print(sum_stats.variance)
```

    (array([4.3, 2.3, 1. , 0.1]), array([5.8, 4.4, 1.9, 0.6]))
    [5.006 3.428 1.462 0.246]
    [0.12424898 0.1436898  0.03015918 0.01110612]

# Plotting histograms to get a sense of data
For every class, we can create histogram plots. The code below generates these histograms for Class 0.

```python
plt.subplot(221)
plt.hist(iris.data[:50,0])
plt.xlim([0,6])
plt.title('SL')
plt.subplot(222)
plt.hist(iris.data[:50,1])
plt.xlim([0,6])
plt.title('SW')
plt.subplot(223)
plt.hist(iris.data[:50,2])
plt.xlim([0,6])
plt.title('PL')
plt.subplot(224)
plt.hist(iris.data[:50,3])
plt.xlim([0,6])
plt.title('PW')
plt.suptitle('Class 0')
plt.tight_layout()
plt.show()
```

# 2D histograms
The following code shows how to plot 2D histograms. This is being provided as a reference for those who are interested.

```python
fig = plt.figure(figsize=(10,5), dpi = 150)
ax1 = fig.add_subplot(121, projection='3d')
ax2 = fig.add_subplot(122, projection='3d')

H, _x, _y = np.histogram2d(iris.data[:50,0], iris.data[:50,1])
_xx, _yy = np.meshgrid(_x[1:], _y[1:])
xx, yy, HH = _xx.ravel(), _yy.ravel(), H.ravel()

ax1.bar3d(xx, yy, np.zeros_like(HH), 0.13, 0.13, HH, shade=True, color='b')

ax1.set_xlabel('SL')
ax1.set_ylabel('SW')

H, _x, _y = np.histogram2d(iris.data[:50,2], iris.data[:50,3])
_xx, _yy = np.meshgrid(_x[1:], _y[1:])
xx, yy, HH = _xx.ravel(), _yy.ravel(), H.ravel()

ax2.bar3d(xx, yy, np.zeros_like(HH), 0.04, 0.04, HH, shade=True, color='g')

ax2.set_xlabel('PL')
ax2.set_ylabel('PW')
```

# Fitting a distribution and estimating parameters
Interarrival times in a Gamma ray experiment are given in the book "Mathematical Statistics and Data Analysis" by John Rice. The data is in the form of an excel file.

We will use the pandas module in python to read the excel file.

```python
#Make sure you upload the file gamma-arrivals.xls to the Colab runtime before running this...
df = pd.read_excel(r'/content/gamma-arrivals.xls')
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Col1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>38.026635</td>
    </tr>
    <tr>
      <th>1</th>
      <td>409.445210</td>
    </tr>
    <tr>
      <th>2</th>
      <td>272.463390</td>
    </tr>
    <tr>
      <th>3</th>
      <td>98.674526</td>
    </tr>
    <tr>
      <th>4</th>
      <td>87.858008</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
    </tr>
    <tr>
      <th>3930</th>
      <td>321.296880</td>
    </tr>
    <tr>
      <th>3931</th>
      <td>146.966750</td>
    </tr>
    <tr>
      <th>3932</th>
      <td>184.342160</td>
    </tr>
    <tr>
      <th>3933</th>
      <td>84.361201</td>
    </tr>
    <tr>
      <th>3934</th>
      <td>218.148090</td>
    </tr>
  </tbody>
</table>
<p>3935 rows × 1 columns</p>
</div>

We see that there are 3935 samples of data. To get a sense of the distribution, we should plot a histogram.

```python
plt.hist(df['Col1'],bins=50)
```

# Fitting a Gamma distribution
From the histogram, the distribution could be modelled as Gamma$(\alpha,\beta)$. The next step is to estimate $\alpha$ and $\beta$ from the given samples.

## Method of moments
Suppose $m_1$ and $m_2$ are the first and second moments of the samples. The method of moments estimates are obtained by solving
$$m_1=\frac{\alpha}{\beta},$$
$$m_2=\frac{\alpha^2}{\beta^2}+\frac{\alpha}{\beta^2}.$$
The solution results in
$$\hat{\alpha}_{MM}=\frac{m_1^2}{m_2-m_1^2}=\frac{m_1^2}{s^2},\hat{\beta}_{MM}=\frac{m_1}{m_2-m_1^2}=\frac{m_1}{s^2}.$$
We now compute the values of $m_1$ (sample mean) and $s^2=m_2-m_1^2$ (sample variance) from the data. After that, we can compute the estimates.

```python
x = np.array(df['Col1'])
m1 = np.average(x)
ss = np.var(x)
print(m1)
print(ss)

```

    79.93522014279034
    6311.67624499251

```python
alphaMM = m1*m1/ss
betaMM = m1/ss
print(alphaMM)
print(betaMM)
```

    1.0123522137792949
    0.012664657856335469

We can plot the density of the Gamma on top of the density histogram to check if the estimate gives a reasonable fit.

```python
fig,ax = plt.subplots(1,1)
ax.hist(x,density=True,bins=50)
xx = np.linspace(0,300,50)
ax.plot(xx, st.gamma.pdf(xx,alphaMM,scale=1/betaMM),label='gamma fit MM')
ax.legend(loc='best')
plt.show()
```

# Bootstrap
How do we find the bias and variance of the estimator? Theoretical derivations of the sampling distributions may be too cumbersome and difficult in most cases. Bootstrap is a Monte Carlo simulation method for computing metrics such as bias, variance and confidence intervals for estimators.

In the above example, we have found $\hat{\alpha}_{MM}=1.0123...$ and $\hat{\beta}_{MM}=0.01266...$. Using these values, we simulate $n=3935$ *iid* samples from Gamma$(1.0123...,0.0126...)$ and, using the simulated samples, we compute new estimates of $\alpha$ and $\beta$ and call them $\hat{\alpha}_{MM}(1)$ and $\hat{\beta}_{MM}(1)$. Now, repeat the simulation $N$ times to get estimates $\hat{\alpha}_{MM}(i)$ and $\hat{\beta}_{MM}(i)$, $i=1,2,\ldots,N$.

The sample variance of $\{\hat{\alpha}_{MM}(1), \hat{\alpha}_{MM}(2),\ldots,\hat{\alpha}_{MM}(N)\}$ is taken to be the bootstrap estimate for the variance of the estimator.

```python
N = 1000
n = 3935
alpha_hat = np.zeros(N)
beta_hat = np.zeros(N)
for i in np.arange(N):
  xi = st.gamma.rvs(alphaMM,scale=1/betaMM,size=n)
  m1i = np.average(xi); ssi = np.var(xi)
  alpha_hat[i] = m1i*m1i/ssi; beta_hat[i] = m1i/ssi
```

We can see the histograms of the estimates to get an idea of the spread of the values.

```python
ax1 = plt.subplot(121)
ax1.hist(alpha_hat,density=True)
ax2 = plt.subplot(122)
ax2.hist(beta_hat,density=True)
```

Notice how the histograms look roughly normal.

The sample standard deviations of the estimates is a bootstrap estimate for the standard error of the estimator.

```python
print(np.sqrt(np.var(alpha_hat)))
print(np.sqrt(np.var(beta_hat)))
```

    0.031822357311654136
    0.000449581400441715

## Confidence intervals
Suppose a parameter $\theta$ is estimated as $\hat{\theta}$, and suppose the distribution of $\hat{\theta}-\theta$ is known. Then, to obtain $(100(1-\alpha))$% confidence intervals (typical values are $\alpha=0.1$ for 90% confidence intervals and $\alpha=0.05$ for 95% confidence intervals), we use the CDF of $\hat{\theta}-\theta$ to obtain $\delta_1$ and $\delta_2$ such that
$$P(\hat{\theta}-\theta\le\delta_1)=1-\frac{\alpha}{2},$$
$$P(\hat{\theta}-\theta\le\delta_2)=\frac{\alpha}{2}.$$
Actually, the inverse of the CDF of $\hat{\theta}-\theta$ is used to find the above $\delta_1$ and $\delta_2$. From the above, we see that
$$P(\hat{\theta}-\theta \le \delta_1)-P(\hat{\theta}-\theta \le \delta_2)= P(\delta_2< \hat{\theta}-\theta \le \delta_1)=1-\frac{\alpha}{2}-\frac{\alpha}{2}=1-\alpha.$$
The above is rewritten as
$$P(\hat{\theta}-\delta_1\le\theta<\hat{\theta}-\delta_2)=1-\alpha,$$
and $[\hat{\theta}-\delta_1,\hat{\theta}-\delta_2]$ is interpreted as the $100(1-\alpha)$% confidence interval.

## Bootstrap confidence intervals
The CDF of $\hat{\theta}-\theta$ might be difficult to determine in many cases, and the bootstrap method is used often to estimate $\delta_1$ and $\delta_2$. We consider the list of numbers $\{\hat{\alpha}_{MM}(1)-1.0123...,\ldots,\hat{\alpha}_{MM}(N)-1.0123...\}$ and pick the $100(\alpha/2)$-th percentile and $100(1-\alpha/2)$-th percentile.

```python
del1 = np.percentile(alpha_hat - alphaMM, 97.5)
del2 = np.percentile(alpha_hat - alphaMM, 2.5)
print([del1,del2])
```

    [0.06149537903410332, -0.06043329601919404]

The 95% confidence interval for $\alpha$ using the method of moments estimator works out to $[1.0123-0.0615,1.0123-(-0.0604)]=[0.9508,1.0727]$.

## Maximum likelihood
We now turn to the maximum likelihood estimator for $\alpha$ and $\beta$. The likelihood $L(x_1,\ldots,x_n)$ can be written as
$$L = \frac{\beta^\alpha}{\Gamma(\alpha)}x_1^{\alpha-1}e^{-\beta x_1}\,\frac{\beta^\alpha}{\Gamma(\alpha)}x_2^{\alpha-1}e^{-\beta x_2}\cdots \frac{\beta^\alpha}{\Gamma(\alpha)}x_n^{\alpha-1}e^{-\beta x_n}= \frac{\beta^{n\alpha}}{\Gamma(\alpha)^n}(x_1\cdots x_n)^{\alpha-1}e^{-\beta(x_1+\cdots+x_n)},$$
$$\log L = n\alpha\log\beta-n\log\Gamma(\alpha)+(\alpha-1)\log(x_1\cdots x_n)-\beta(x_1+\cdots+x_n).$$
Differentiating $\log L$ with respect to $\beta$ and equating to zero, we get
$$n\alpha\frac{1}{\beta}-(x_1+\cdots+x_n)=0,\text{or }\alpha=\beta \frac{x_1+\cdots+x_n}{n}.$$
Differentiating $\log L$ with respect to $\alpha$ and equating to zero, we get
$$n\log\beta-n\frac{\Gamma'(\alpha)}{\Gamma(\alpha)}+\log(x_1\cdots x_n)=0.$$
So, we get two equations in the two variables $\alpha$ and $\beta$. However, the equations do not have a closed form solution, and we need to solve them numerically or approximately. From the first equation, we have $\log\beta=\log\alpha-\log\frac{x_1+\cdots+x_n}{n}$. Using this in the second equation, we get
$$\log\alpha - \frac{\Gamma'(\alpha)}{\Gamma(\alpha)}=\log\frac{x_1+\cdots+x_n}{n}-\frac{1}{n}\log(x_1\cdots x_n).$$
We will now solve the above equation to find the ML estimate of $\alpha$. This will be a numerical solution.

```python
lm1 = np.average(np.log(x))
#Write the equation as a function
#digamma is the function Gamma'/Gamma
from scipy.special import digamma
fML = lambda a: (np.log(a) - digamma(a) - np.log(m1)+lm1)
```

We can plot the above function to see how it looks.

```python
fig, ax = plt.subplots(1,1)

xx = np.linspace(0.1,2,50)
ax.plot(xx,fML(xx))
ax.grid(True)
plt.show()
```

```python
#For solving numerically, we will use scipy.optimize
import scipy.optimize as sopt
sol = sopt.root_scalar(fML, bracket=[0.1,2])
sol.root
```

    1.0263317358725452

```python
alphaML = sol.root
betaML = alphaML/m1
print([alphaML, betaML])
```

    [1.0263317358725452, 0.012839543495835532]

Let us check the fit with the histogram.

```python
fig,ax = plt.subplots(1,1)
ax.hist(x,density=True,bins=50)
xx = np.linspace(0,300,50)
ax.plot(xx, st.gamma.pdf(xx,alphaMM,scale=1/betaMM),lw='4',alpha=0.7,label='gamma fit MM')
ax.plot(xx, st.gamma.pdf(xx,alphaML,scale=1/betaML),lw='1',label='gamma fit ML')
ax.legend(loc='best')
plt.show()
```

Both the curves are literally on top of each other showing very good fit. Let us use the bootstrap method to find variance and confidence intervals for the ML estimator.

```python
N = 1000
n = 3935
alpha_hatML = np.zeros(N)
beta_hatML = np.zeros(N)
for i in np.arange(N):
  xi = st.gamma.rvs(alphaMM,scale=1/betaMM,size=n)
  m1i = np.average(xi); lm1i = np.average(np.log(xi))
  fMLi = lambda a: (np.log(a) - digamma(a) - np.log(m1i)+lm1i)
  soli = sopt.root_scalar(fMLi, bracket = [0.1,2])
  alpha_hatML[i] = soli.root; beta_hatML[i] = soli.root / m1i
```

```python
print(np.sqrt(np.var(alpha_hatML)))
print(np.sqrt(np.var(beta_hatML)))
```

    0.01988186514427374
    0.0003157550854393247

We see that the variance of the bootstrap ML estimator is lesser than that of bootstrap MM estimator.


# 2022-03-28 Gradient Descent

## Last time

* Assumptions of linear models
* Look at your data!
* Partial derivatives
* Loss functions

## Today

* Discuss projects
* Gradient-based optimization for linear models
* Nonlinear models

```julia
using LinearAlgebra
using Plots
default(linewidth=4, legendfontsize=12)

function vander(x, k=nothing)
    if isnothing(k)
        k = length(x)
    end
    m = length(x)
    V = ones(m, k)
    for j in 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function vander_chebyshev(x, n=nothing)
    if isnothing(n)
        n = length(x) # Square by default
    end
    m = length(x)
    T = ones(m, n)
    if n > 1
        T[:, 2] = x
    end
    for k in 3:n
        #T[:, k] = x .* T[:, k-1]
        T[:, k] = 2 * x .* T[:,k-1] - T[:, k-2]
    end
    T
end

function chebyshev_regress_eval(x, xx, n)
    V = vander_chebyshev(x, n)
    vander_chebyshev(xx, n) / V
end

runge(x) = 1 / (1 + 10*x^2)
runge_noisy(x, sigma) = runge.(x) + randn(size(x)) * sigma

CosRange(a, b, n) = (a + b)/2 .+ (b - a)/2 * cos.(LinRange(-pi, 0, n))

vcond(mat, points, nmax) = [cond(mat(points(-1, 1, n))) for n in 2:nmax]
```

    vcond (generic function with 1 method)

# Workshopping projects today

* Maybe you've started, maybe you're still looking for a good project
* What have you been thinking about?
* What do you need help on?

## Each breakout group will report out
1. One interesting thing about process
2. One question you have relevant to the projects (specific or general)
3. One question from content we've covered

# Variational notation for derivatives

It's convenient to express derivatives in terms of how they act on an infinitessimal perturbation. So we might write

$$ \delta f = \frac{\partial f}{\partial x} \delta x .$$

(It's common to use $\delta x$ or $dx$ for these infinitesimals.) This makes inner products look like a normal product rule

$$ \delta(\mathbf x^T \mathbf y) = \mathbf y^T (\delta \mathbf x) + \mathbf x^T (\delta \mathbf y). $$

A powerful example of variational notation is differentiating a matrix inverse

$$ 0 = \delta I = \delta(A^{-1} A) = (\delta A^{-1}) A + A^{-1} (\delta A) $$
and thus
$$ \delta A^{-1} = - A^{-1} (\delta A) A^{-1} $$

# Optimization for linear models
Given data $(x,y)$ and loss function $L(c; x,y)$, we wish to find the coefficients $c$ that minimize the loss, thus yielding the "best predictor" (in a sense that can be made statistically precise).  I.e.,
$$ \bar c = \arg\min_c L(c; x,y) . $$

It is usually desirable to design models such that the loss function is differentiable with respect to the coefficients $c$, because this allows the use of more efficient optimization methods.  Recall that our forward model is given in terms of the Vandermonde matrix,

$$ f(x, c) = V(x) c $$

and thus

$$ \frac{\partial f}{\partial c} = V(x) . $$

# Derivative of loss function

We now differentiate our loss function
$$ L(c; x, y) = \frac 1 2 \lVert f(x, c) - y \rVert^2 $$
using a more linear algebraic approach to write the same expression is
\begin{align} \nabla_c L(c; x,y) &= \big( f(x,c) - y \big)^T V(x) \\
&= \big(V(x) c - y \big)^T V(x) \\
&= V(x)^T \big( V(x) c - y \big) .
\end{align}
A necessary condition for the loss function to be minimized is that $\nabla_c L(c; x,y) = 0$.

* Is the condition sufficient for general $f(x, c)$?
* Is the condition sufficient for the linear model $f(x,c) = V(x) c$?
* Have we seen this sort of equation before?

## Gradient descent

Instead of solving the least squares problem using linear algebra (QR factorization), we could solve it using gradient descent.  That is, on each iteration, we'll take a step in the direction of the negative gradient.

```julia
function grad_descent(loss, grad, c0; gamma=1e-3, tol=1e-5)
    """Minimize loss(c) via gradient descent with initial guess c0
    using learning rate gamma.  Declares convergence when gradient
    is less than tol or after 500 steps.
    """
    c = copy(c0)
    chist = [copy(c)]
    lhist = [loss(c)]
    for it in 1:500
        g = grad(c)
        c -= gamma * g
        push!(chist, copy(c))
        push!(lhist, loss(c))
        if norm(g) < tol
            break
        end
    end
    (c, hcat(chist...), lhist)
end
```

    grad_descent (generic function with 1 method)

## Quadratic model

```julia
A = [1 1; 1 8]
@show cond(A)
loss(c) = .5 * c' * A * c
grad(c) = A * c

c, chist, lhist = grad_descent(loss, grad, [.9, .9],
    gamma=.22)
plot(lhist, yscale=:log10, xlims=(0, 80))
```

    cond(A) = 9.46578492882319

```julia
plot(chist[1, :], chist[2, :], marker=:circle)
x = LinRange(-1, 1, 30)
contour!(x, x, (x,y) -> loss([x, y]))
```

# Chebyshev regression via optimization

```julia
x = LinRange(-1, 1, 200)
sigma = 0.5; n = 8
y = runge_noisy(x, sigma)
V = vander(x, n)
function loss(c)
    r = V * c - y
    .5 * r' * r
end
function grad(c)
    r = V * c - y
    V' * r
end
c, _, lhist = grad_descent(loss, grad, ones(n),
    gamma=0.008)
c
```

    8-element Vector{Float64}:
      0.7891712932446856
      0.007025641791364225
     -1.957064629386119
     -0.36715187386667464
      0.9229823103908219
      0.3031710357963882
      0.4284481908323782
      0.27267026221101087

```julia
c0 = V \ y
l0 = 0.5 * norm(V * c0 - y)^2
@show cond(V' * V)
plot(lhist, yscale=:log10)
plot!(i -> l0, color=:black)
```

    cond(V' * V) = 52902.52994792479

# Why use QR vs gradient-based optimization?

# Nonlinear models

Instead of the linear model
$$ f(x,c) = V(x) c = c_0 + c_1 \underbrace{x}_{T_1(x)} + c_2 T_2(x) + \dotsb $$
let's consider a rational model with only three parameters
$$ f(x,c) = \frac{1}{c_1 + c_2 x + c_3 x^2} = (c_1 + c_2 x + c_3 x^2)^{-1} . $$
We'll use the same loss function
$$ L(c; x,y) = \frac 1 2 \lVert f(x,c) - y \rVert^2 . $$

We will also need the gradient
$$ \nabla_c L(c; x,y) = \big( f(x,c) - y \big)^T \nabla_c f(x,c) $$
where
\begin{align}
\frac{\partial f(x,c)}{\partial c_1} &= -(c_1 + c_2 x + c_3 x^2)^{-2} = - f(x,c)^2 \\
\frac{\partial f(x,c)}{\partial c_2} &= -(c_1 + c_2 x + c_3 x^2)^{-2} x = - f(x,c)^2 x \\
\frac{\partial f(x,c)}{\partial c_3} &= -(c_1 + c_2 x + c_3 x^2)^{-2} x^2 = - f(x,c)^2 x^2 .
\end{align}

# Fitting a rational function

```julia
f(x, c) = 1 ./ (c[1] .+ c[2].*x + c[3].*x.^2)
function gradf(x, c)
    f2 = f(x, c).^2
    [-f2 -f2.*x -f2.*x.^2]
end
function loss(c)
    r = f(x, c) - y
    0.5 * r' * r
end
function gradient(c)
    r = f(x, c) - y
    vec(r' * gradf(x, c))
end
```

    gradient (generic function with 1 method)

```julia
c, _, lhist = grad_descent(loss, gradient, ones(3), gamma=8e-2)
plot(lhist, yscale=:log10)
```

# Compare fits on noisy data

```julia
scatter(x, y)
V = vander_chebyshev(x, 7)
plot!(x -> runge(x), color=:black, label="Runge")
plot!(x, V * (V \ y), label="Chebyshev fit")
plot!(x -> f(x, c), label="Rational fit")
```


```python
# Intialization for notebook use
%pylab inline

import sympy as sp
sp.init_printing(use_latex=True)

import ipywidgets as widgets
```

    Populating the interactive namespace from numpy and matplotlib

```python
x_1, x_2 = sp.symbols('x_1 x_2') # indentify these as symbols
λ = sp.symbols('λ')
```

# Introduction

Consider the following 2nd order system (Mass-Damper-Spring):

$\ddot{x} + \dot{x} = 3x + x^2 = 0 \Longleftrightarrow \ddot{x} + c \dot{x} + f(x) = 0$

$c$ is the dampening force and $f(x)$ is the non-linear spring.

The system can be placed into the $\dot{\underline{x}} = \underline{f}(\underline{x})$ form:

$\begin{cases}
    \dot{x}_1 = x_2 \\
    \dot{x}_2 = -3x_1 - x_1^2 - x_2
\end{cases}$

# Analysis of the system: Manual

Analysis of the system $\Rightarrow$ Phase Plane $(x_1, x_2)$:

## Step 1: Find the Equillibrium Points:

$\begin{cases}
    f_1(x_1, x_2) = 0 \\
    f_2(x_2, x_2) = 0
\end{cases} \Rightarrow
\begin{cases}
    x_2 && = 0 \\
    -3x_1 - x_1^2 - x_2 && = 0 \, \Rightarrow \, -3x_1 - x_1^2 = 0
\end{cases}$

$\begin{cases}
    x_2 = 0 \\
    x_1(-3 - x_1) = 0
\end{cases} \Rightarrow$ 2 equillibrium points $\Rightarrow
\begin{cases}
    \underline{x}_1 = [0, 0]^T \\
    \underline{x}_2 = [-3, 0]^T
\end{cases}$

## Step 2: Linearize the System Around the Equillibrium Points:

$\displaystyle\dot{\underline{x}} = \underline{f}(\underline{x}) \Rightarrow \approx \underline{f}(\underline{x}^*) + \left . \frac{\partial \underline{f}}{\partial \underline{x}} \right |_{\underline{x}^*} (\underline{x} - \underline{x}^*)$

$\displaystyle A = \left . \frac{\partial\underline{f}}{\partial\underline{x}} \right |_{\underline{x}_{1, \, 2}^*} =
\begin{bmatrix}
    0 && 1 \\
    -3 - 2x_1 && -1
\end{bmatrix}_{\underline{x}_1^*, \, \underline{x}_2^*}$

### For $\underline{x} = \underline{x}_1$

We have the following system:

$\dot{\underline{x}} = A_1 \underline{x} \Rightarrow A_1 =
\begin{bmatrix}
    0 && 1 \\
    -3 && -1
\end{bmatrix}$

Look for the behavior of the system near $\underline{x}_1 = \underline{0}$.  e-value analysis:

$\det (A_1 - \lambda I) =
\begin{vmatrix}
    -\lambda && 1 \\
    -3 && -1 - \lambda
\end{vmatrix}
= \lambda \big(\lambda + 1\big) + 3 = 0$

$\Rightarrow \lambda^2 + \lambda + 3 = 0 \Rightarrow \lambda_{12} = -\frac{1}{2} \left( 1 \pm \sqrt{1 - 12}\right) = -\frac{1}{2} \left( 1 \pm \sqrt{11} j\right)$

### For $\underline{x} = \underline{x}_2$

We have the following system:

$\dot{\underline{x}} = A_2 \underline{x} \Rightarrow A_2 =
\begin{bmatrix}
    0 && 1 \\
    3 && -1
\end{bmatrix}$

Look for the behavior of the system near $\underline{x}_2 = \underline{0}$.  e-value analysis:

$\left|(A - \lambda I ) \right| =
\begin{vmatrix}
    -\lambda && 1 \\
    3 && -1 - \lambda
\end{vmatrix} = 0$

$\Rightarrow \lambda \left(\lambda + 1 \right) - 3 = 0 \Rightarrow \, \lambda^2 + \lambda - 3 = 0$

$\lambda = -\frac{1}{2} \left ( 1 \pm \sqrt{1 + 12} \right) = - \frac{1}{2} \pm \frac{1}{2} \sqrt{13}$

Note that both solutions of $\lambda$ are REAL!  Therefore:

$\begin{matrix}
    \lambda_1 = -\frac{1}{2} \left( 1 - \sqrt{13} \right ) > 0 \\
    \lambda_2 = -\frac{1}{2} \left( 1 + \sqrt{13} \right ) < 0
\end{matrix} \Rightarrow \text{ Unstable } \Rightarrow \underline{\text{Saddle Point}}$

Compute the e-vectors $\underline{v}_1,\, \underline{v}_2$ to determine the direction of convergence

## Step 3: Put Everything Together:

## !!!!! TODO: Insert Graphic !!!!!

# Analysis of the System: Python

Given that:

$\begin{cases}
    \dot{x}_1 = f_1(x_1, x_2) = x_2 \\
    \dot{x}_2 = f_2(x_2, x_2) = -3x_1 - x_1^2 - x_2
\end{cases}$

We can define two variables to these functions:

```python
f_1 = x_2
f_2 = -3 * x_1 - x_1 ** 2 - x_2
```

We can solve the system of equations using the `solve` function.

```python
slns = sp.solve([f_1, f_2])
slns
```

We get the same solutions that we found previously.

To linearize the system, we need to take the derivatives of each function with respect to $x_1$ and $x_2$.  We'll compile this into a matrix A_1 like so:

```python
A = sp.Matrix([[sp.diff(f_1, x_1), sp.diff(f_1, x_2)], [sp.diff(f_2, x_1), sp.diff(f_2, x_2)]])
A
```

Now we let $\dot{\underline{x}} = A_1 \, \underline{x}$.  Therefore `A_1` would be:

```python
A_1 = sp.Matrix([[0, 1], [-3, -1]])
A_1
```

Next we find the determinate of $A_1 - \lambda I$

```python
λ_fun = sp.det(A_1 - λ * sp.eye(2))
λ_fun
```

```python
sp.solve(λ_fun)
```

# Analysis of the System: Numpy Meshgrid

```python
import numpy as np
from matplotlib import pyplot as plt
x1, x2 = np.meshgrid(np.linspace(-.5, .5, 10), np.linspace(-.5, .5, 10))
x1dot = x2
x2dot = -3 * x1 - x1 ** 2 - x2
```

```python
plt.figure()
plt.quiver(x1, x2, x1dot, x2dot)
plt.show()
```

```python
@widgets.interact (
    x_start=(-10.0, 10.0, 0.1),
    x_stop=(-10.0, 10.0, 0.1),
    y_start=(-10.0, 10.0, 0.1),
    y_stop=(-10.0, 10.0, 0.1),
    space=(10, 50)
)
def inter_plot(x_start, x_stop, y_start, y_stop, space):
    x1, x2 = np.meshgrid(np.linspace(x_start, x_stop, space), np.linspace(y_start, y_stop, space))
    x1dot = x2
    x2dot = -3 * x1 - x1 ** 2 - x2
    plt.figure()
    plt.quiver(x1, x2, x1dot, x2dot)
    plt.show()
```

```python

```

01a - Example of Phase-Plane Analysis.ipynb


# Simplification

```python
# Load sympy module
from sympy import *
```

```python
# Define variables
x, y = symbols('x y')
```

## Simplification of $$\frac{x^2 + x\times y}{x + y} = x$$

```python
# Simplification
simplify((x**2 + x*y)/(x + y))
```

$\displaystyle x$

## Rational simplification of $$(x^2 + 1)(x + 2) = x^3 + 2x^2 + x + 2$$

```python
# Rational simplification
expand((x**2 + 1)*(x+2))
```

$\displaystyle x^{3} + 2 x^{2} + x + 2$

## Factor of $$x^2 + 2x + 1 = (x + 1)^2$$

```python
# Factor
factor(x**2 + 2*x + 1)
```

$\displaystyle \left(x + 1\right)^{2}$


# Lab Assignment 3

## Sam Dauncey, s2028017

We consider the system $$\frac{dx}{dt}=x(y-1),\quad \frac{dy}{dt}=4-y^2-x^2.$$

## Task 1 (2 marks)

Use `SymPy` to find the critical points of the system.

```python
import sympy as sym
sym.init_printing()
from IPython.display import display_latex
```

```python
# Define sympy symbols.
t = sym.symbols("t")
x = sym.Function("x")
y = sym.Function("y")

# Use these symbols to define the expressions for x' and y' given above.
x_prime = x(t)*(y(t) - 1)
y_prime = 4 - y(t)**2 - x(t)**2

deq_x = sym.Eq(x(t).diff(t), x_prime)
deq_y = sym.Eq(y(t).diff(t), y_prime)

# Symbolically solve for when (x', y') = (0, 0)
crit_point_dicts = sym.solve([x_prime, y_prime])

# Extract the critical points from the dictionaries given by sympy.
crit_points = [(point[x(t)], point[y(t)]) for point in crit_point_dicts]
crit_points
```

## Task 2 (4 marks)

Give your implementation of the `linearise` function from Lab 3.

Use this to find linear approximations of the system around the critical points with $x \geq 0$ and $y \geq 0$. Use the output to classify these critical points (use markdown cells and proper reasoning to explain the type of each critical point).

```python
# Define some variables to use in our linear system.
u = sym.Function("u")
v = sym.Function("v")

def lin_matrix(eqs, crit_point):
    """Returns the jacobian F(x, y) = (x', y') evaluated at the given critical point"""
    # Unpack the expressions for x' and y' and use them to calculate the Jacobian.
    eq1, eq2 = eqs
    FG = sym.Matrix([eq1.rhs, eq2.rhs])
    matJ = FG.jacobian([x(t), y(t)])

    # Evaluate the Jacobian at the given critical point.
    x0, y0 = crit_point
    lin_mat = matJ.subs({x(t):x0, y(t):y0})
    return lin_mat

def linearise(eqs, crit_point):
    """Returns a list of equations for the linearised system of eqs evaluated at the given critical point"""
    # Get the jacobian, J, at our critical point
    lin_mat = lin_matrix(eqs, crit_point)

    # Construct the system (u', v') = J (u, v) component-wise and return.
    uv_rhs = lin_mat * sym.Matrix([u(t),v(t)])
    u_eq = sym.Eq(u(t).diff(t), uv_rhs[0])
    v_eq = sym.Eq(v(t).diff(t), uv_rhs[1])
    return [u_eq, v_eq]

# Print info about the linear system at each of the critical points.
for point in crit_points:

    # If the x and y coords are non-negative, print information about the point.
    x0, y0 = point
    if x0 >= 0 and y0 >= 0:
        print("critical point:")
        display_latex((x0, y0))

        # Use lin_matrix() to get the matrix and eigenvalues of the linearised system
        linearised_matrix = lin_matrix([deq_x, deq_y], point)
        print("linearised matrix, eigenvalues")
        display_latex(linearised_matrix)
        display_latex(list(linearised_matrix.eigenvals().keys()))

        # Use linearise() to get a printable version of the linear system
        print("full linearised system:")
        display_latex(linearise([deq_x, deq_y], point))
        print()
        print()
```

    critical point:

$\displaystyle \left( 0, \  2\right)$

    linearised matrix, eigenvalues

$\displaystyle \left[\begin{matrix}1 & 0\\0 & -4\end{matrix}\right]$

$\displaystyle \left[ 1, \  -4\right]$

    full linearised system:

$\displaystyle \left[ \frac{d}{d t} u{\left(t \right)} = u{\left(t \right)}, \  \frac{d}{d t} v{\left(t \right)} = - 4 v{\left(t \right)}\right]$

    critical point:

$\displaystyle \left( \sqrt{3}, \  1\right)$

    linearised matrix, eigenvalues

$\displaystyle \left[\begin{matrix}0 & \sqrt{3}\\- 2 \sqrt{3} & -2\end{matrix}\right]$

$\displaystyle \left[ -1 - \sqrt{5} i, \  -1 + \sqrt{5} i\right]$

    full linearised system:

$\displaystyle \left[ \frac{d}{d t} u{\left(t \right)} = \sqrt{3} v{\left(t \right)}, \  \frac{d}{d t} v{\left(t \right)} = - 2 \sqrt{3} u{\left(t \right)} - 2 v{\left(t \right)}\right]$

We can see here that the point $(2, 0)$ will be unstable as the linearised system has a positive eigenvalue (namely $1$). In contrast, the eigenvalues for the linearised system at the critical point $(\sqrt{3}, 1)$ both have negative real parts so this critical point will be stable.

## Task 3 (4 marks)

Produce a phase portrait of the system, with trajectories showing the behaviour around all the critical points. A few trajectories are enough to show this behaviour. Use properly-sized arrows to diplay the vector field (the RHS of the ODE). There are some marks allocated to the quality of your figure in this part. Try to keep it illustrative yet not too cluttered.

```python

import numpy as np
from matplotlib import pyplot as plt
from scipy.integrate import odeint
%matplotlib inline

# Get figure and axes
fig, ax = plt.subplots(figsize=(12, 9))

# Define x and y derivatives (t variable for use with odeint)
def vector_field(xy, t):
    X, Y = xy
    return (X*(Y - 1), 4 - Y**2 - X**2)

# Get arrays for all the points with  -4 < x < 4, -3 < y < 3
X, Y = np.mgrid[-4:4:24j, -3:3: 18j]

# Evaluate the vector field and length of each vector at each point
X_prime, Y_prime = vector_field((X, Y), None)
Magnitude = np.hypot(X_prime, Y_prime)

# Plot arrows which are faded if they have large magnitute
ax.quiver(X, Y, X_prime, Y_prime, Magnitude,
          scale=200, pivot = 'mid', cmap = plt.cm.bone)

# Pick some initial conditions for phase portraits
ics  = [[0.2, 2.2], [-0.2, -1.8], [3, 2], [-1.5, 0.5]]
durations = [[0, 10], [0, 8], [0, 5], [0, 5]]

vcolors = plt.cm.autumn_r(np.linspace(0.5, 1., len(ics)))  # colors for each trajectory

# plot trajectories
for time_span, ic, color in zip(durations, ics, vcolors):
    t = np.linspace(*time_span, 100)
    sol = odeint(vector_field, ic, t)
    x_sol, y_sol = sol.T
    ax.plot(x_sol, y_sol, color=color, label=f"$(x_0, y_0)$ = {ic}")

def split_coords(tuple_list):
    """Helper function which takes [(a, b), (c, d), (e, f) ... ] and returns [[a, c, e .. ], [b, d, f ...]]"""
    return np.array(tuple_list).T

# Plot black and blue points for the critical points and initial conditions respectivelyl
ax.scatter(*split_coords(crit_points), color = "k", label="critical points")
ax.scatter(*split_coords(ics), color='b', label="initial conditions")

plt.xlabel('x')
plt.ylabel('y')
plt.legend()

plt.xlim(-4, 4)
plt.ylim(-3, 3)

plt.show()
```

```python

```

```python

```


# GDA

```{note}
Gaussian Discriminant Analysis(GDA) suppose that observations are presumed to come from one of several multivariate normal distributions.<br/>
It is
```

## Normal Distribution

Normal distribution is parameterized by a mean vector $\mu \in \mathbb{R}^{d}$ and a covariance matrix $\Sigma \in \mathbb{R}^{d \times d}$, where $\Sigma >= 0$ is symmetric and positive semi-definite, also written $\mathcal{N}(\mu,\Sigma)$, it's density is given by：

$$p(x;\mu,\Sigma)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }\exp\left ( -\frac{1}{2}(x-\mu)^{T}{\Sigma}^{-1}(x-\mu)\right )$$

unsurprisingly, for random variable $X \sim \mathcal{N}(\mu,\Sigma)$:

$$E[X] = \int_{x}xp(x;\mu, \Sigma)dx = \mu$$
$$Cov(X) = E[(X - E(X))(X - E(X))^{T}] = \Sigma$$

## Model

When we have a classification problem in which the input features $x$ are continous, we can use GDA, which model $p(x|y)$ using the multivariant normal distribution:

$$y\sim Bernoulli(\phi)$$
$$x | y=0 \sim \mathcal{N}(\mu_{0},\Sigma) $$
$$x | y=1 \sim \mathcal{N}(\mu_{1},\Sigma) $$

density:

$$p(y) = \phi^{y}(1 - \phi)^{1 - y}$$

$$p(x| y=0)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }\exp\left (-\frac{1}{2}(x-\mu_{0})^{T}{\Sigma}^{-1}(x-\mu_{0})\right )$$

$$p(x| y=1)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }\exp\left (-\frac{1}{2}(x-\mu_{1})^{T}{\Sigma}^{-1}(x-\mu_{1})\right )$$

Then the log-likelihood of the data is given by：

$$
\begin{equation}
\begin{split}
l(\phi,\mu_{0},\mu_{1},\Sigma) &= \log\prod_{i=1}^{n}p(x^{(i)},y^{(i)};\phi,\mu_{0},\mu_{1},\Sigma) \\
&= \log\prod_{i=1}^{n}p(x^{(i)}|y^{(i)};\mu_{0},\mu_{1},\Sigma)p(y^{(i)};\phi)
\end{split}
\end{equation}
$$

Maximum Likelihood Estimate result：

$$\phi = \frac{1}{n}\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}$$

$$\mu_{0} = \frac{\sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}x^{(i)}  }{\sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}} $$

$$\mu_{1} = \frac{\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}x^{(i)}  }{\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}} $$

$$\Sigma=\frac{1}{n}\sum_{i=1}^{n}(x^{(i)} - \mu_{y^{(i)}})(x^{(i)} - \mu_{y^{(i)}})^{T}$$

```python

```