# Clase Nro. 6: Álgebra Lineal

> El álgebra lineal es una rama de las matemáticas que estudia conceptos tales como vectores, matrices, espacio dual, sistemas de ecuaciones lineales y en su enfoque de manera más formal, espacios vectoriales y sus transformaciones lineales

np.linalg: paquete de álgebra lineal en NumPy

- Funciones básicas
    - vectores
    - operaciones con vectores
    - norma de un vector
    - operaciones con matrices
    - inversa de una matriz
    - determinante
- Resolución de sistemas

_Con el manejo básico de arrays en Python con NumPy, es hora de pasar a operaciones más interesantes como son las propias del Álgebra Lineal._

_Los productos escalares y las inversiones de matrices están por todas partes en los programas científicos e ingenieriles, así que vamos a estudiar cómo se realizan en Python._

Como sabemos, las operaciones del álgebra lineal aparecen con mucha frecuencia a la hora de resolver sistemas de ecuaciones en derivadas parciales y en general al linealizar problemas de todo tipo, y suele ser necesario resolver sistemas con un número enorme de ecuaciones e incógnitas. Gracias a los arrays de NumPy podemos abordar este tipo de cálculos en Python, ya que todas las funciones están escritas en C o Fortran y tenemos la opción de usar bibliotecas optimizadas al límite.

El paquete de álgebra lineal en NumPy se llama `linalg`, así que importando NumPy con la convención habitual podemos acceder a él escribiendo `np.linalg`.

**Pero ¿Porqué álgebra lineal?**

Si entendemos álgebra lineal podremos desarrollar una mejor intuición para el aprendizaje automático y algoritmos. Además, también seremos capaces de desarrollar algoritmos desde cero y hacer variaciones de ellos.

**¿Qué es un vector?**

Un vector tiene tanto magnitud como dirección. Utilizamos vectores para describir, por ejemplo, la velocidad de objetos en movimiento.

Por dirección, se refiere a dónde en el espacio apunta la “flecha”, y la magnitud te dice qué tan lejos debes ir en esa dirección. Si solo tienes magnitud, pero no dirección, entonces estás hablando de escalares. Una vez que le das al escalar alguna dirección, se convierte en un vector.

Un vector se representa con una letra minúscula y una flecha arriba, apuntando a la derecha

```python
import numpy as np
```

```python
V = np.array([3, 2])
V
```

    array([3, 2])

**Suma de vectores**

Para sumar los vectores (x₁,y₁) y (x₂,y₂), sumamos los componentes correspondientes de cada vector: (x₁+x₂,y₁+y₂)

```python
a = np.array([6, -2])
b = np.array([-4, 4])
a+b
```

    array([2, 2])

**Multiplicar un vector por un escalar**

```python
a = np.array([2, 1])
print(3 * a)
```

    [6 3]

**Producto punto o escalar**

Esto suena raro, pero no es más que la multiplicación de vectores cuyo resultado es un escalar. Para calcular el producto escalar de dos vectores, primero debemos multiplicar los elementos correspondientes y luego sumar los términos del producto.

En la siguiente fórmula lo vemos mucho más fácil

```python
a = np.array([3, 2])
b = np.array([2, 2])
print(a.dot(b))
```

    10

```python
np.dot(a, b)
```

    10

**Norma vectorial**

La norma es solo otro término para la magnitud de un vector y se denota con dos lineas dobles (||) en cada lado. Se define como una raíz cuadrada de la suma de cuadrados para cada componente de un vector

Pasos:
- Elevar al cuadrado cada componente
- Suma todos los cuadrados
- Toma la raíz cuadrada

Trabajemos ahora con la formula:

```python
a = np.array([3, 2, 6])
print(np.linalg.norm(a))
```

    7.0

**Vector unitario**

Los vectores unitarios son aquellos cuya magnitud es exactamente 1 unidad. Son muy útiles por diversas razones. Específicamente, los vectores unitarios [0,1] y [1,0] juntos pueden formar cualquier otro vector.

Un vector unitario se denota con mayor frecuencia con un símbolo de sombrero (^) y se determina calculando la norma y luego dividiendo cada componente del vector con la norma.

Suena complejo, pero vamos a hacer un ejercicio para poder ver que es más complejo de lo que imaginamos

```python
def unit_vector(a):
    return a/np.linalg.norm(a)

u = np.array([3, 6, 4])
print(unit_vector(u))
```

    [0.38411064 0.76822128 0.51214752]

## Operaciones con matrices

 ¿Qué es una matriz? Una matriz es simplemente un arreglo rectangular de números

En data science las usamos un montón, así no nos demos cuenta. Es muy importante de diferenciar un vector de una matriz. En pocas palabras un vector es una sola columna (atributo) en su conjunto de datos y una matriz es una colección de todas las columnas

**Definición de matrices**

$A = \begin{equation}
\begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 6\\
\end{bmatrix}
\end{equation}$

```python
A = np.array([
    [1, 2, 3],
    [4, 5, 6]
])
A
```

    array([[1, 2, 3],
           [4, 5, 6]])

$B = \begin{equation}
\begin{bmatrix}
5 & 5 & 5\\
5 & 5 & 5\\
\end{bmatrix}
\end{equation}$

```python
B = np.full((2, 3), 5)
B
```

    array([[5, 5, 5],
           [5, 5, 5]])

$v = \begin{equation}
\begin{bmatrix}
6\\
7\\
8
\end{bmatrix}
\end{equation}$

```python
v = np.array([6, 7, 8])
v
```

    array([6, 7, 8])

```python
v.reshape(3, 1)
```

    array([[6],
           [7],
           [8]])

```python
#Generación de matriz con números aleatorios
M = np.random.rand(10).reshape(2, 5)
M
```

    array([[0.26159254, 0.35669711, 0.70822817, 0.82603537, 0.12198188],
           [0.38912962, 0.13786627, 0.48969985, 0.14139557, 0.30644094]])

```python
M[0][0]
```

    0.1869780014242599

```python
M[0, 0]
```

    0.1869780014242599

```python
M[0]
```

    array([0.186978  , 0.78673675])

```python
M[1,-1]
```

    0.3064409440254877

```python
arreglo = np.arange(1,11).reshape(2, 5)
arreglo
```

    array([[ 1,  2,  3,  4,  5],
           [ 6,  7,  8,  9, 10]])

```python
np.arange(0, 11, 3)
```

    array([0, 3, 6, 9])

```python
np.linspace(0, 20, 10)
```

    array([ 0.        ,  2.22222222,  4.44444444,  6.66666667,  8.88888889,
           11.11111111, 13.33333333, 15.55555556, 17.77777778, 20.        ])

```python
np.logspace(0, 3, 100)
```

    array([   1.        ,    1.07226722,    1.149757  ,    1.23284674,
              1.32194115,    1.41747416,    1.51991108,    1.62975083,
              1.7475284 ,    1.87381742,    2.009233  ,    2.15443469,
              2.3101297 ,    2.47707636,    2.65608778,    2.84803587,
              3.05385551,    3.27454916,    3.51119173,    3.76493581,
              4.03701726,    4.32876128,    4.64158883,    4.97702356,
              5.33669923,    5.72236766,    6.13590727,    6.57933225,
              7.05480231,    7.56463328,    8.11130831,    8.69749003,
              9.32603347,   10.        ,   10.72267222,   11.49756995,
             12.32846739,   13.21941148,   14.17474163,   15.19911083,
             16.29750835,   17.475284  ,   18.73817423,   20.09233003,
             21.5443469 ,   23.101297  ,   24.77076356,   26.56087783,
             28.48035868,   30.53855509,   32.74549163,   35.11191734,
             37.64935807,   40.37017259,   43.28761281,   46.41588834,
             49.77023564,   53.36699231,   57.22367659,   61.35907273,
             65.79332247,   70.54802311,   75.64633276,   81.11308308,
             86.97490026,   93.26033469,  100.        ,  107.2267222 ,
            114.97569954,  123.28467394,  132.19411485,  141.74741629,
            151.9911083 ,  162.97508346,  174.75284   ,  187.38174229,
            200.92330026,  215.443469  ,  231.01297001,  247.7076356 ,
            265.60877829,  284.80358684,  305.38555088,  327.45491629,
            351.11917342,  376.49358068,  403.70172586,  432.87612811,
            464.15888336,  497.70235643,  533.66992312,  572.23676594,
            613.59072734,  657.93322466,  705.48023107,  756.46332755,
            811.13083079,  869.74900262,  932.60334688, 1000.        ])

```python
#np.logspace?
```

```python
np.random.randint(1, 100)
```

    41

```python
np.random.randint(1, 100, 20).reshape(5, 4)
```

    array([[69, 47, 80, 99],
           [92, 25, 88, 61],
           [43, 20, 64, 44],
           [89, 48, 51, 34],
           [ 9, 97, 74, 17]])

**Suma de matrices**

```python
A+B
```

    array([[ 6,  7,  8],
           [ 9, 10, 11]])

**Resta de matrices**

```python
A-B
```

**Multiplicación por un escalar**

```python
C = np.array([
    [3, 5],
    [1, 0]
])
C
```

    array([[3, 5],
           [1, 0]])

```python
print(2 * C)
```

    [[ 6 10]
     [ 2  0]]

**Matriz transpuesta**

```python
D = np.array([
    [3, 4],
    [1, 0]
])
print(D)
print(D.T)
```

    [[3 4]
     [1 0]]
    [[3 1]
     [4 0]]

**Matriz identidad**

```python
I = np.identity(5)
I
```

    array([[1., 0., 0., 0., 0.],
           [0., 1., 0., 0., 0.],
           [0., 0., 1., 0., 0.],
           [0., 0., 0., 1., 0.],
           [0., 0., 0., 0., 1.]])

```python
#help(np.linalg)
```

Recordemos que si queremos usar una función de un paquete pero no queremos escribir la "ruta" completa cada vez, podemos usar la sintaxis `from package import func`:

```python
from numpy.linalg import norm, det
norm
```

El producto matricial usual (no el que se hace elemento a elemento, sino el del álgebra lineal) se calcula con la misma función que el producto matriz-vector y el producto escalar vector-vector: con la función `dot`, que **no** está en el paquete `linalg` sino directamente en `numpy` y no hace falta importarlo.

```python
np.dot
```

Una consideración importante a tener en cuenta es que en NumPy no hace falta ser estricto a la hora de manejar vectores como si fueran matrices columna, siempre que la operación sea consistente. Un vector es una matriz con una sola dimensión: por eso si calculamos su traspuesta no funciona.

```python
M = np.array([
    [1, 2],
    [3, 4]
])
v = np.array([1, -1])
```

```python
v
```

    array([ 1, -1])

v

```python
M.T
```

    array([[1, 3],
           [2, 4]])

```python
v.T
```

    array([ 1, -1])

```python
v.reshape(2,1).T
```

    array([[ 1, -1]])

```python
u = np.dot(M, v)
u
```

    array([-1, -1])

**Multiplicación entre matrices**

```python
A = np.array([
    [3, 2],
    [1, 3]
])
print(a.shape)
B = np.array([
    [2, 2],
    [1, 2]
])
print(b.shape)

#Amxn
#Bcxd

#n == c

```

    (2, 2)
    (2, 2)

```python
# producto matricial
np.dot(A, B)
```

    array([[ 8, 10],
           [ 5,  8]])

### Ejemplo de vídeo

```python
A = np.array([
    [5, 3, -4, -2],
    [8, -1, 0, -3]

])
B = np.array([
    [1, 4, 0],
    [-5, 3, 7],
    [0, -9, 5],
    [5, 1, 4]
])

res = np.dot(A, B)
print(res)
```

    [[-20  63  -7]
     [ -2  26 -19]]

**Determinante de una matriz**

```python
# calcula el determinante de una matriz
A = np.array([
    [3, 2],
    [1, 3]
])
```

```python
np.linalg.det(A)
```

    7.000000000000001

**Inversa de una matriz**

Para calcular la matriz inversa en Python, use la función linalg.inv () del módulo numpy.

linalg.inv(x)

El parámetro x de la función es una matriz cuadrada invertible M definida con la función array() de numpy.

La función genera la matriz inversa M-1 de la matriz M.

> ¿Qué es la matriz inversa? La matriz inversa M-1 de una matriz cuadrada es una matriz tal que el producto M · M-1 es igual a una matriz de identidad I.

Encuentre la matriz inversa M-1 de la siguiente matriz invertible M.

```python
M=np.array([
    [3,4,-1],
    [2,0,1],
    [1,3,-2]
])
```

```python
# Calcula la matriz inversa con la función linalg.inv().
Minv = np.linalg.inv(M)
Minv
```

    array([[-0.6,  1. ,  0.8],
           [ 1. , -1. , -1. ],
           [ 1.2, -1. , -1.6]])

La matriz de salida inversa es también un objeto array(). Se puede leer como una lista anidada. Los elementos de la matriz inversa son números reales.

Verificación. El producto de la matriz M para la matriz M-1 es una matriz de identidad.

```python
# verificación
Mident = np.dot(M, Minv)
np.round(Mident)
```

    array([[ 1., -0., -0.],
           [ 0.,  1., -0.],
           [ 0., -0.,  1.]])

**Otro ejemplo**
Encontrar el determinante y la matriz inversa de D.

$ D = \begin{equation}
\begin{bmatrix}
1 & 2\\
3 & 4\\
\end{bmatrix}
\end{equation}$

```python
D = np.array([[1, 2], [3, 4]])
D
```

    array([[1, 2],
           [3, 4]])

```python
# cálculo del determinante
np.linalg.det(D)
```

    -2.0000000000000004

```python
# y el inverso
np.linalg.inv(D)
```

    array([[-2. ,  1. ],
           [ 1.5, -0.5]])

**Solución de ecuaciones con matrices**

1. Resolver el siguiente sistema de ecuaciones

$\begin{eqnarray}
3x + y  = 9 \\
x + 2y = 8 \\
\end{eqnarray}$

```python
coef = np.array([
    [3, 1],
    [1, 2]
])

result = np.array([9, 8])

sol = np.linalg.solve(coef, result)
sol

```

    array([2., 3.])

2. Al resolver un problema de circuito eléctrico, uno se encuentra rápidamente empantanado en
una gran cantidad de ecuaciones simultáneas. Por ejemplo, considere el circuito eléctrico que se
muestra en la figura 9.5. Contiene una sola fuente de voltaje y cinco reóstatos. Puede analizar este
circuito al dividirlo en partes más pequeñas y usar dos hechos básicos en torno a la electricidad

Sumatoria de voltaje alrededor de un circuito debe ser cero
Voltaje = corriente x resistencia (V = iR)

Seguir el lazo inferior izquierdo resulta en la primera ecuación:

$-V_1 +R_2(i_1-i_2)+R_4(i_1-i_3)=0$

Seguir el lazo superior resulta en la segunda ecuación:

$R_1i_2+R_3(i_2-i_3)+R_2(i_2-i_1)=0$

Finalmente, seguir el lazo inferior derecho resulta en la última ecuación:

$R_3(i_3-i_2)+R_5i_3+R_4(i_3-i_1)=0$

Dado que se conocen todas las resistencias (los valores R) y el voltaje, se tienen tres ecuaciones y tres incógnitas. Ahora se necesita reordenar las ecuaciones de modo que estén en una forma en la que se pueda aplicar una solución matricial. En otras palabras, se necesita aislar las íes del modo siguiente:

Cree un programa Python para resolver estas ecuaciones con el método de matriz inversa y empleando el método solve de linalg.

1. Establezca el problema.

Encontrar las tres corrientes para el circuito que se muestra.

2. Describa las entradas y salidas.

Entrada
 Cinco resistencias R1, R2, R3, R4, R5 y el voltaje V, proporcionados desde el teclado

Salida
 Tres valores de corrientes i1, i2, i3
3. Desarrolle una solución en Python

```python
import numpy as np

R = [float(sR) for sR in input("Ingrese 5 resistencias: ").split('')]
V = float(input("Ingrese el valor del voltaje: "))

R1 = R[0]
R2 = R[1]
R3 = R[2]
R4 = R[3]
R5 = R[4]

coef = np.array([
    [(R2+R4), -R2, -R4],
    [-R2, (R1+R2+R3), -R3],
    [-R4, -R3, (R3+R4+R5)]
])

result = np.array([V, 0, 0])

# Primer método
I = np.linalg.solve(coef, result)
print(I)

# Segundo método
I = np.dot(np.linalg.inv(coef), result)
print(I)
```

    Ingrese 5 resistencias:  2 4 6 8 10
    Ingrese el valor del voltaje:  10

    [1.69354839 0.96774194 0.80645161]
    [1.69354839 0.96774194 0.80645161]

#### Lectura de una matriz

```python
NUM_ROWS = 4
a = [[int(j) for j in input().split()] for i in range(NUM_ROWS)]
```

     1 2 3 4
     4 5 6 7
     8 9 10 11
     12 13 14 15

```python
a
```

    [[1, 2, 3, 4], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]]

### Ejercicios

1- Hallar el producto de estas dos matrices y su determinante:

$$\begin{pmatrix} 1 & 0 & 0 \\ 2 & 1 & 1 \\ -1 & 0 & 1 \end{pmatrix} \begin{pmatrix} 2 & 3 & -1 \\ 0 & -2 & 1 \\ 0 & 0 & 3 \end{pmatrix}$$

```python
from numpy.linalg import det
```

```python
A = np.array([
    [1, 0, 0],
    [2, 1, 1],
    [-1, 0, 1]
])
B = np.array([
    [2, 3, -1],
    [0, -2, 1],
    [0, 0, 3]
])
```

```python
C = np.dot(A, B)
```

```python
det = np.linalg.det(C)
det
```

    -12.0

3- Resuelva el siguiente sistema de ecuaciones

$\begin{eqnarray}
x + 2y  = 3 \\
3x + 4y = 5 \\
\end{eqnarray}$

```python
coef = np.array([
    [1, 2],
    [3, 4]
])

result = np.array([3, 5])

resp = np.linalg.solve(coef, result)
resp

```

    array([-1.,  2.])

4- Un ingeniereo escribe dos ecuaciones que describen el circuito, como sigue:

$\begin{gather}
300I_{1} + 500(I_{1}-I_{2})-20 = 0\\
200I_{2} + 500(I_{2}-I_{1}) +10 = 0\\
\end{gather}$

Coloque las dos ecuaciones en forma estándar y resuelva las dos ecuaciones:

```python
# 300i1 + 500i1 - 500i2 = 20
# 200i2 + 500i2 - 500i1 = -10

#  800 i1 - 500 i2  = 20
# -500 i1 + 700 i2  = -10

coef = np.array([
    [800, -500],
    [-500, 700]
])

result = np.array([20, -10])
resp = np.linalg.solve(coef, result)
print(resp)

# Resp: [0.029, 0.0064]
```

    [0.02903226 0.00645161]

5- Resolver el siguiente sistema de ecuaciones

```python
# 2000 (i1 - i2) + 4000 (i1 - i3) - 10      = 0
# 2000 (i2 - i1) + 8000 i2 + 5000 (i2 - i3) = 0
# 5000 (i3 - i2) + 3000 i3 + 4000 (i3 - i1) = 0

# 2i1 - 2i2 + 4i1 - 4i3       = 10/1000
# 2i2 - 2i1 + 8i2 + 5i2 - 5i3 = 0
# 5i3 - 5i2 + 3i3 + 4i3 - 4i1 = 0

#  6i1 -  2i2 - 4i3 = 10/1000
# -2i1 + 15i2 - 5i3 = 0
# -4i1 - 5i2  + 12i3 = 0

coef = np.array([
    [6, -2, -4],
    [-2, 15, -5],
    [-4, -5, 12]
])

result = np.array([10/1000, 0, 0])

I = np.linalg.solve(coef, result)
I
```

    array([0.00275801, 0.00078292, 0.00124555])

3- Resolver el siguiente sistema:

$$ \begin{pmatrix} 2 & 0 & 0 \\ -1 & 1 & 0 \\ 3 & 2 & -1 \end{pmatrix} \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} -1 \\ 3 \\ 0 \end{pmatrix} $$

```python
M = np.dot(
np.array([
    [2, 0, 0],
    [-1, 1, 0],
    [3, 2, -1]
]),
np.array([
    [1, 1, 1],
    [0, 1, 2],
    [0, 0, 1]
]))
M
```

    array([[ 2,  2,  2],
           [-1,  0,  1],
           [ 3,  5,  6]])

```python
x = np.linalg.solve(M, np.array([-1, 3, 0]))
x
```

    array([ 0.5, -4.5,  3.5])


```python
import numpy as np
```

```python
from sympy import *
```

```python
x = Symbol('x')
y = Symbol('y')
z = Symbol('z')
```

### Calculate the Jacobian of the function
- f(x,y,z) = (x^2)*cos(y) + exp(z)*sin(y)
- evaluate at the point (x,y,z)=(π,π,1).

```python
f = x**2*cos(y) + exp(z)*sin(y)
```

```python
J = np.array(
    [diff(f, x), diff(f, y), diff(f, z)])

print J
```

    [2*x*cos(y) -x**2*sin(y) + exp(z)*cos(y) exp(z)*sin(y)]

```python
print np.array([
    diff(f, x).subs({x:pi, y:pi, z:1}),
    diff(f, y).subs({x:pi, y:pi, z:1}),
    diff(f, z).subs({x:pi, y:pi, z:1})])
```

    [-2*pi -E 0]

### Calculate the Jacobian of the vector valued functions

- u(x,y) = (x^2)*y − cos(x)*sin(y) and v(x,y)=exp(x+y)
- evaluate at the point (0,π)

```python
u = x**2*y - cos(x)*sin(y)
v = exp(x+y)
```

```python
J = np.array([
    [diff(u, x), diff(u, y)],
    [diff(v, x), diff(v, y)]
])

print J
```

    [[2*x*y + sin(x)*sin(y) x**2 - cos(x)*cos(y)]
     [exp(x + y) exp(x + y)]]

```python
print np.array([
    [diff(u, x).subs({x:0, y:pi}), diff(u, y).subs({x:0, y:pi})],
    [diff(v, x).subs({x:0, y:pi}), diff(v, y).subs({x:0, y:pi})]
])
```

    [[0 1]
     [exp(pi) exp(pi)]]

### Calculate the Hessian for the function.
- f(x,y) = (x^3)*cos(y) − x*sin(y)

```python
f = x**3*cos(y) + x*sin(y)
```

```python
H = np.array([
    [diff(diff(f, x), x), diff(diff(f, x), y)],
    [diff(diff(f, y), x), diff(diff(f, y), y)]
])

print H
```

    [[6*x*cos(y) -3*x**2*sin(y) + cos(y)]
     [-3*x**2*sin(y) + cos(y) -x**3*cos(y) - x*sin(y)]]

### Calculate the Hessian for the function
- f(x,y,z) = xy + sin(y)sin(z) + (z^3)*exp(x)

```python
f = x*y + sin(y)*sin(z) + (z**3)*exp(x)
```

```python
H = np.array([
    [diff(diff(f, x), x), diff(diff(f, x), y), diff(diff(f, x), z)],
    [diff(diff(f, y), x), diff(diff(f, y), y), diff(diff(f, y), z)],
    [diff(diff(f, z), x), diff(diff(f, z), y), diff(diff(f, z), z)]
])

print H
```

    [[z**3*exp(x) 1 3*z**2*exp(x)]
     [1 -sin(y)*sin(z) cos(y)*cos(z)]
     [3*z**2*exp(x) cos(y)*cos(z) 6*z*exp(x) - sin(y)*sin(z)]]

### Calculate the Hessian for the function
- f(x,y,z) = xycos(z) − sin(x)*exp(y)*(z^3)
- evaluate at the point (x,y,z)=(0,0,0)

```python
f = x*y*cos(z) - sin(x)*exp(y)*(z**3)
```

```python
H = np.array([
    [diff(diff(f, x), x), diff(diff(f, x), y), diff(diff(f, x), z)],
    [diff(diff(f, y), x), diff(diff(f, y), y), diff(diff(f, y), z)],
    [diff(diff(f, z), x), diff(diff(f, z), y), diff(diff(f, z), z)]
])

print H
```

    [[z**3*exp(y)*sin(x) -z**3*exp(y)*cos(x) + cos(z)
      -y*sin(z) - 3*z**2*exp(y)*cos(x)]
     [-z**3*exp(y)*cos(x) + cos(z) -z**3*exp(y)*sin(x)
      -x*sin(z) - 3*z**2*exp(y)*sin(x)]
     [-y*sin(z) - 3*z**2*exp(y)*cos(x) -x*sin(z) - 3*z**2*exp(y)*sin(x)
      -x*y*cos(z) - 6*z*exp(y)*sin(x)]]

```python
print np.array([
    [diff(diff(f, x), x).subs({x:0, y:0, z:0}), diff(diff(f, x), y).subs({x:0, y:0, z:0}), diff(diff(f, x), z).subs({x:0, y:0, z:0})],
    [diff(diff(f, y), x).subs({x:0, y:0, z:0}), diff(diff(f, y), y).subs({x:0, y:0, z:0}), diff(diff(f, y), z).subs({x:0, y:0, z:0})],
    [diff(diff(f, z), x).subs({x:0, y:0, z:0}), diff(diff(f, z), y).subs({x:0, y:0, z:0}), diff(diff(f, z), z).subs({x:0, y:0, z:0})]
])
```

    [[0 1 0]
     [1 0 0]
     [0 0 0]]

Coursera_Mathematics_for_Machine Learning/2_Mathematics_for_Machine_Learning_Multivariate_Calculus/Week 2 - Jacobian - Hessian Matrix/Jacobian_Hessian.ipynb


# Sitzung 6

Diese Skripte sind ausschließlich als Zusatz-Material gedacht. Speziell für diejenigen unter Euch, die einen Einblick in das Programmieren gewinnen wollen. Wenn Du es also leid bist repetitive Tätigkeiten auszuführen und das lieber einer Maschine überlassen willst, bist Du hier genau richtig.

<span style="color:red"> Die Codes sind nicht für die Klausur relevant, genau genommen haben sie mit dem Lehrstuhl für Statistik __rein gar nichts__ zu tun. </span>

---

```python
import numpy as np
from scipy.special import binom
from matplotlib import pyplot as plt
from tqdm import trange
```

## Verteilungen diskreter Wartezeiten

Ist es möglich, die Argumentationsweise der Exponentialverteilung auf die Binomialverteilung und die Hypergeometrische Verteilung zu erweitern?

Sei $$T: \text{Wartezeit auf den ersten Erfolg}$$

Wobei $$X: \text{Anzahl der Erfolge}$$

Zur Erinnerung: warten heißt, dass bisher nichts passiert ist $X=0$.

Für $X \sim Pois(\lambda)$ gilt:

\begin{equation}
P(X=0) = \frac{\lambda^0}{0!}\exp(-\lambda) = \exp(-\lambda)
\end{equation}

Für unabhängige und identisch verteilte Zeiteinheiten gilt:

\begin{align}
P(X&=0 \text{ in 2 Zeiteinheiten}) \\
&= P(\{X=0\text{ in der ersten Zeiteinheit}\} , \{ X=0\text{ in der zweiten Zeiteinheit} \}) \\
&= P(\{X=0\text{ in der ersten Zeiteinheit}\}) \cdot P(\{ X=0\text{ in der zweiten Zeiteinheit} \}) \\
&= \exp(-\lambda) \cdot \exp(-\lambda) = \exp(-2\lambda)
\end{align}

Und allgemein:
$$P(X=0 \text{ in t Zeiteinheiten}) = \exp(-\lambda t) = P(T \geq t)$$

Damit können wir sagen:
\begin{equation}
P(T \leq t) = 1 - \exp(-\lambda t)
\end{equation}

---

## Erweiterung auf die Binomialverteilung

Für $X \sim Bin(n, p)$, mit $n \in \mathbb{N}, p \in \mathbb{R}_{+}$ gilt die obere Argumentation immer noch:

$$P_n(X=0)=\underbrace{{N \choose 0}}_{=1} \overbrace{p^0}^{=1} (1-p)^{n-0} = (1-p)^n$$

und

$$P(T \leq n) = 1 - P_n(X=0) = 1 - (1-p)^n$$

### Überprüfung:

Wartezeit auf die erste sechs beim Mensch-ärgere-dich-nicht

```python
trials = 1000000
n = np.arange(0, 100)
theoretical = 1 - (1-1/6)**n
samples = np.ones(trials)
for i in trange(trials):
    while np.random.randint(low=1, high=7) != 6:
        samples[i] += 1
values, counts = np.unique(samples, return_counts=True)
empirical = counts/trials
```

    100%|██████████| 1000000/1000000 [00:15<00:00, 66267.40it/s]

```python
plt.figure(figsize=(10, 5))
plt.step(n, theoretical, where='post', label='$P_{theoretisch}(X < x)$')
plt.step(values, empirical.cumsum(), where='post', label='$P_{empirisch}(X < x)$')
plt.legend()
plt.title("Vergleich/Überprüfung theoretischer Verteilungsfunktion")
plt.xlim([0, 40])
```

---
## Erweiterung auf die Hypergeometrische Verteilung

Für $X \sim Hyper(N, M, n)$

\begin{align}
P_n(X=0) &= \overbrace{\left(\frac{N-M}{N}\right) \cdot \left(\frac{N-M-1}{N-1}\right) \cdot \dots \cdot \left(\frac{N-M-(n-1)}{N-(n-1)}\right)}^{\textit{n Faktoren}} \\
&= \Large{\frac{\frac{(N-M)!}{(N-M-n)!}}{\frac{N!}{(N-n)!}}}
= \Large{\frac{\frac{(N-M)! \color{red}{n!}}{(N-M-n)!\color{red}{n!}}}{\frac{N!\color{red}{n!}}{(N-n)!\color{red}{n!}}}}\\
&= \Large{\frac{\color{red}{n!}{N-M \choose n}}{\color{red}{n!}{N \choose n}} = \frac{{N-M \choose n}}{{N \choose n}}}
\end{align}

und

$$P(T < n) = 1 - P_n(X=0) = 1 - \frac{{N-M \choose n}}{{N \choose n}}$$

### Überprüfung:

Wie wahrscheinlich ist es eine Partie russisches Roulette zu überleben?

```python
N = 6
M = 1
n = np.arange(0, 6)
theoretical = 1 - binom(N-M, n)/binom(N, n)

samples = np.zeros(trials)

for i in trange(trials):
    x = [1, 2, 3, 4, 5, 6]
    np.random.shuffle(x)
    didi_mao = None
    while didi_mao != 6:
        didi_mao = x.pop()
        samples[i] += 1

values, counts = np.unique(samples, return_counts=True)
empirical = counts/trials
```

    100%|██████████| 1000000/1000000 [00:03<00:00, 275661.38it/s]

```python
plt.figure(figsize=(10, 5))
plt.step(n, theoretical, where='post', label='$P_{theoretisch}(X < x)$')
plt.step(values, empirical.cumsum(), where='post', label='$P_{empirisch}(X < x)$')
plt.legend()
plt.title("Vergleich/Überprüfung theoretischer Verteilungsfunktion")
plt.xlim([0, 6])
```


# 8 Modeling a Drone Swinging in a Trifilar Pendulum

A trifilar pendulum is a common tool for determining the inertia of a rigid body. In the video below a small quadcopter drone is hung from a trifilar pendulum and set into an oscillation about the vertical axis. The frequency (or period) of this oscillation correlates with the inertia of the drone about the axis of rotation. In this notebook you will learn how to create a mathematical model of this system.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('HrG7xhrLbWo')
```

```python
import numpy as np
import sympy as sm
sm.init_printing()
```

```python
%matplotlib notebook
```

# System parameters

We introduced a number of variable to describe the geometry in the problem:

- $m$: total mass of the drone
- $l$: length of the very stiff strings
- $I$: rotational inertia of the drone about a vertical axis and with respect to the center of mass
- $g$: acceleration due to gravity
- $r$: the radial distance from the center of mass to the location of the string attachments

```python
m, I, g, l, r = sm.symbols('m, I, g, l, r', real=True, positive=True)
```

The goal is to write the equations of motion in terms of the generalized coordinate. In our case are going to use the rotational angle of the drone, $\theta$. This variable should be a function of time.

```python
t = sm.symbols('t')
```

```python
theta = sm.Function('theta')(t)
theta
```

With $\theta$ being a function of time the first and second derivatives can be easily taken with `.diff()`.

```python
theta.diff(t)
```

```python
theta.diff(t, 2)
```

# Getting the geometry and velocity right

The height of the drone from its equilibrium resting position is written in terms of $\theta$ below.

```python
h = l - sm.sqrt(l**2 - (2 * r * sm.sin(theta / 2))**2)
h
```

The time derivative of $h$ is also needed to compute the linear kinetic energy.

```python
v = h.diff(t)
v
```

This can be simplified a bit:

```python
v = sm.trigsimp(v)
v
```

# Forming the Lagrangian

Now the total kinetic energy can be formed by summing the linear and rotational kinetic energies:

```python
T = m * v**2 / 2 + I * theta.diff()**2 / 2
T
```

The total potential energy is:

```python
U = m * g * h
U
```

The Lagrangian:

```python
L = T - U
L
```

# Forming and evaluating Lagrange's Equation

Now Lagrange's equation of the second kind can be formed with respect to the single generalized coordinate.

$$\frac{d}{dt}\left(\frac{\partial L}{\partial \dot{\theta}}\right) - \frac{\partial L}{\partial \theta} = 0$$

```python
zero = L.diff(theta.diff(t)).diff(t) - L.diff(theta)
zero
```

Some trigonometric simplification can clean it up a bit:

```python
zero = sm.trigsimp(zero)
zero
```

# Putting the equations of motion into first order form

The equation of motion is linear with respect to $\ddot{\theta}$ and the coefficient can be extracted with:

```python
acc_coeff = zero.coeff(theta.diff(t, 2))
acc_coeff
```

This system can be put into first order form by introducing a new variable $\omega = \dot{\theta}$ (this relationship is also the first of the two ODEs). The second is then:

```python
omega = sm.Function('omega')(t)
omega_dot = sm.simplify(-(zero.subs({theta.diff(): omega}) - acc_coeff * omega.diff()) / acc_coeff)
sm.Eq(omega.diff(), omega_dot)
```

# Simulating the nonlinear equations of motion

```python
from resonance.nonlinear_systems import SingleDoFNonLinearSystem
```

```python
nl_sys = SingleDoFNonLinearSystem()
```

```python
nl_sys.constants['m'] = 1  # kg
nl_sys.constants['r'] = 0.3  # m
nl_sys.constants['l'] = 0.75  # m
nl_sys.constants['g'] = 9.81  # m/s**2
nl_sys.constants['I'] = 0.3**2  # kg m**2

nl_sys.coordinates['theta'] = np.deg2rad(15) # rad
nl_sys.speeds['omega'] = 0.0  # rad/s
```

The 2nd order differential equations for a non-linear system have to be specificied in explicit first order form for most numerical integration routines (which is what is used behind the scenes in resonance). This means we need to introduce a new variable, a generalized speed, that defines the first of the two first order equations. We also have to move everything to the right hand side of the equations except for terms with derivatives.:

$$
\dot{\theta} = \omega \\
\dot{\omega} = f(\omega, \theta, m, r, l, g, I, t)
$$

You have to create a function that computes the time derivatives of the coordinates and speeds (the states) given the current values of the coordinates and speeds. The function will have arguments that match your variable names defined above and it **must output the time derivatives that correspond to the order of `sys.states`.** See below how to translate this mathematical expression into a "right hand side" function.

Option 1) Copy code form SymPy output and modify by hand:

```python
str(omega_dot)
```

```python
def eval_rhs(theta, omega, I, m, r, l, g):
    theta_dot = omega
    omega_dot = (-m*r**2*(2*g*(l**2 + 2*r**2*np.cos(theta) -
                               2*r**2)**4*np.sin(theta) +
                 2*r**4*(l**2 + 2*r**2*np.cos(theta) -
                         2*r**2)**(5/2)*omega**2*np.sin(theta)**3 +
                 r**2*(l**2 + 2*r**2*np.cos(theta) -
                       2*r**2)**(7/2)*omega**2*np.sin(2*theta))/
                 (2*(I*(l**2 + 2*r**2*np.cos(theta) - 2*r**2) +
                     m*r**4*np.sin(theta)**2)*(l**2 + 2*r**2*np.cos(theta) -
                                               2*r**2)**(7/2)))
    return theta_dot, omega_dot

nl_sys.diff_eq_func = eval_rhs
```

Try out the function with some arbitrary values:

```python
eval_rhs(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7)
```

Option 2) Use lambdify to create the function with no manual copying.

```python
eval_omega_dot = sm.lambdify((theta, omega, I, m, r, l, g), omega_dot)
```

You can see that the function works the same as the above function:

```python
eval_omega_dot(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7)
```

```python
def eval_rhs(theta, omega, I, m, r, l, g):
    theta_dot = omega
    omega_dot = eval_omega_dot(theta, omega, I, m, r, l, g)
    return theta_dot, omega_dot

nl_sys.diff_eq_func = eval_rhs
```

```python
eval_rhs(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7)
```

```python
nl_traj = nl_sys.free_response(20.0)
```

```python
nl_traj.plot(subplots=True);
```

# Linearizing the Equations of Motion

Assume $\theta$ is small, therefore:

- $\sin\theta = \theta$
- $\cos(\theta) = 1$
- $\theta \cdot \theta = 0$
- $\dot{\theta} \cdot \dot{\theta} = 0$
- $\sin^2\theta = \theta^2 = 0$

```python
zero = zero.subs({theta.diff(): omega})
zero_lin = zero.subs({sm.sin(theta): theta,
                      sm.cos(theta): 1,
                      sm.sin(theta / 2): theta / 2,
                      sm.cos(theta / 2): 1,
                      theta**2: 0,
                      omega**2: 0,
                     }).subs(theta**2, 0)  # have to get this last one because order of subs matters
sm.Eq(zero_lin, 0)
```

# Simulating the linear equations of motion

```python
from resonance.linear_systems import SingleDoFLinearSystem
```

```python
sys = SingleDoFLinearSystem()
```

```python
sys.constants['m'] = 1  # kg
sys.constants['r'] = 0.3  # m
sys.constants['l'] = 0.7  # m
sys.constants['g'] = 9.81  # m/s**2
sys.constants['I'] = 0.3**2  # kg m**2
```

```python
sys.coordinates['theta'] = np.deg2rad(15)  # rad
sys.speeds['omega'] = 0.0  # rad/s
```

```python
def coeffs(m, r, l, g, I):
    k = g * m * r**2 / l
    return I, 0, k
```

```python
sys.canonical_coeffs_func = coeffs
```

```python
traj = sys.free_response(5.0)
```

```python
traj.plot(subplots=True);
```

# Compare the nonlinear and linear motion

```python
initial_angle = 35  # degrees

nl_sys.coordinates['theta'] = np.deg2rad(initial_angle)
sys.coordinates['theta'] = np.deg2rad(initial_angle)

nl_traj = nl_sys.free_response(10)
traj = sys.free_response(10)

axes = nl_traj.plot(subplots=True, color='blue')
traj.plot(subplots=True, ax=axes, color='red');
```


# $\chi^2_\nu$ (reduced chi-squared)
## Week 4, Intro-to-Astro 2020

This week we learned about the $\chi^2$ metric and how (under certain assumptions) minimizing $\chi^2$ with respect to a certain parameter of interest can produce the best-estimate of the true value for that parameter, based on our data.

As part of your assignments this week, we'll learn about the "reduced $\chi^2$" metric (written as $\chi^2_\nu$), which is used for another common problem in data analysis: *model selection*.

Complete this notebook by filling in the missing code snippets and answering questions when prompted.

**Assignment outline**

0. What's $\chi^2$ again?
1. What's $\chi^2_\nu$?
2. Goodness of fit with $\chi^2_\nu$
3. A cautionary tale

### Written by Joey Murphy, June 2020

### Adapted from lecture notes by Ian Czekala

## 0. $\chi^2$ refresher

In this week's tutorial on $\chi^2$ minimization, we estimated the true length of a house key based on a collection of measurements. We made assumptions about the data:

- the measurement uncertainty was the same for each data point
- the data was "drawn" from a Gaussian distribution
- each measurement was independent

In general, each of these assumptions can eventually be violated, e.g. each measurement could have a different measurement uncertainty, the data may not actually follow a Gaussian distribution (think photon counting), and data points may have some covariance (not independent measurements). For now we'll hold on to these assumptions to keep things simple, but it's always a good idea to be cognizant of the validity of the assumptions that go into your analysis.

**Question**:
What are some examples of how the three assumptions above could be violated when analyzing real astronomical data?

...

- Measurement Uncertainity can be different for different transit events discovered by TESS due to other factors such as Solar Spots, Gasses in the protoplanetary disk of the star surrounding the plannet, stellar dimming due to factors other than the planet transit (as seen in the case of Betelguese which, according to latest suggestions, accretes or barps out huge amount of gases which covers a part of the star, dimming its observable flux.
- Data may be drawn from other observations - Due to Red/Brown noise which occurs due to Brownian motion. Also, for a spatially and temporally coherent electromagnetic wave of a single frequency, the [photon counting](https://en.wikipedia.org/wiki/Photon_counting) also represents a Poisson distribution (Binomial distribution -> with probability of success, $p \rightarrow 0$ and $Np \rightarrow \mu$ (N being the number of total observations and $\mu$ being the mean), becomes the Poisson distribution, which, with a large $\mu$, becomes Gaussian distribution.
- _(From Key)_ Nearby data points in a spectrum may have some covariance, due to wavelength-dependent sensitivity in the CCD (which is the Charged Couple Device- the array of pMOS capacitors to store the incident light as pixels in digital imaging)

...

Summarizing the rest of what we did...

We calcualted the log-likelihood of our data

\begin{align}
\mathrm{ln} \mathcal{L} & \propto \Sigma_i^N \Big[-\frac{1}{2} \big(\frac{x_i-\mu}{\sigma}\big)^2  \Big],
\end{align}

and defined the $\chi^2$ statistic as

\begin{align}
\chi^2 & \equiv \Sigma_i^N \big(\frac{x_i - \mu}{\sigma}\big)^2,
\end{align}

giving us

\begin{align}
\mathrm{ln} \mathcal{L} & \propto -\frac{1}{2} \Sigma_i^N \big(\frac{x_i - \mu}{\sigma}\big)^2 \\
& \propto -\frac{1}{2} \chi^2.
\end{align}

We calculated our best estimate of the value of $\mu$ by minimizing the $\chi^2$ metric, finding

\begin{align}
\hat{\mu} = \frac{1}{N} \Sigma_i^N x_i.
\end{align}

As a bonus, if we examine the width of the likelihood function at $\mu = \hat{\mu}$, we can show that the *uncertainty* on our *estimate* of the true length of the key is

\begin{align}
\hat{\sigma} = \frac{\sigma}{\sqrt{N}},
\end{align}

where $\sigma$ is the measurement uncertainty for a single data point and $N$ is the number of data points in our sample. Our final estimate of the key's length is then

\begin{align}
\mu = \hat{\mu} \pm \frac{\sigma}{\sqrt{N}}.
\end{align}

*Upshot:* Our uncertainty on the estimate of the parameter's true value goes down as the square root of the number of observations. Want a more precise measurement? Get more data!

# 1. $\chi^2_\nu$ (reduced chi-squared)

What was the "model" that we used to describe our data in the previous example?

A very simple one! Basically, a flat line. That model had a single parameter that we "fit" to the data. In general, you'll run into real datasets that require more complicated models (i.e. they contain more parameters that need fitting).

These models could be

* a line: $\mu = m x + b$
* a quadratic function: $\mu = a x^2 + b x + c$
* a sinusoid: $\mu = A \sin(t) + B \cos(t)$
* a damped harmonic oscillator: $\mu = A e^{-t/10}\sin(5 t)$
* etc...

**Question**: What are some other physical models that you can think of?

...

- The curve of stellar flux of a planet (the transit phenomena in the lightcurve)
- The distribution of Mass and Radius of planets

...

For all of these models, good ol' $\chi^2$ still applies:

\begin{align}
\chi^2 & = \Sigma_i^N \big(\frac{x_i - \mu(\theta)}{\sigma_i}\big)^2,
\end{align}

where $\theta$ represents the model parameters.

_here $\sigma_i$ represents that there are different standard deviations/uncertainities for different observations_

We made no mention of whether or not the model that we chose was a good representation of the underlying physical process (in our case, it wasn't like the key we measured was growing or shrinking, so a constant value was actually a pretty good choice). In general, though, how do we determine which model to use to describe our data?

The $\chi^2_\nu$ statistic helps us with just that.

\begin{align}
\chi^2_\nu & \equiv \frac{\chi^2}{\nu},
\end{align}

where $\nu = N - m$ is referred to as the number of "degrees of freedom". $N$ is the number of data points and $m$ is the number of parameters being fit in your model. For our "model" of the length of the house key, there was a single parameter being fit: $\hat{\mu}$, our estimate of the key's true length.

**Question**: Why is $\nu = N - m$ called the number of "degrees of freedom" for our model? Give your best guess!

*Hint: If I fit a model with 5 parameters to a sample with only 5 data points, what happens?*

...

Because it defines the number of ways/dimansions in which the process that data is being retreived from can change.
If we fit a model with 5 params to a sample data with 5 data points, it can caoture all these individual degrees of freedom (N-m, N and m being 5)

...

## 2. Goodness of fit with $\chi^2_\nu$

So how do we actually use $\chi^2_\nu$ to assess whether or not the model we chose is good at describing our data?

### Goodness of fit rules of thumb

$\chi^2_\nu \gg 1 \rightarrow$ The model is inadequate (i.e. not enough parameters), or data uncertainties are underestimated

$\chi^2_\nu \approx 1 \rightarrow$ The model does a pretty good job of fitting the data

$\chi^2_\nu \ll 1 \rightarrow$ The model is too flexible (i.e. too many parameters), or data uncertainties are overestimated

**Question**: Can you try to explain why $\chi^2_\nu \gg 1$ suggests the model is too simplistic, and why $\chi^2_\nu \ll 1$ suggests the model is too complex? Again, give it your best guess!

*Hint: Look back at our definition of $\chi^2$. It's the sum of the squares of the differences between the data and the model, in units of the measurement uncertainty for each data point. Does $\chi^2_\nu$ look like an average of sorts, now?*

...

If $\chi^2_\nu$ is close to one, that means that the chi-squared (which is directly propotional to negative of log likelihood), is balanced. This is because $\chi_2$ is defined as the sum of square of differnces of data model from the mean divided by their variances, in other words, a value closer to 0 represents a better fir of the data model to the actual value. Since we are talking about average when we are talking about $\chi^2_\nu$, a value closer to 1 means a close fit of data model to the actual value (means) about the $\sigma^2$

$\chi^2_\nu \gg 1$ indicates that the data model is too far off the actual value as compared to the variance. It could be either due to underestimation of uncertainities of the data ($\sigma_i$2 being larger than what is estimated) or because the degrees of freedom of the data model are too low as compared to the actual data's degrees of freedom (due to less number of parameters being used to define the data model)

$\chi^2_\nu \ll 1$ indicates over fitting of the data. It may be because we have fed too many parameters in our data model and the actual degree of freedom of the data is smaller or because the $\sigma_i^2$ is estimated larger than it actually is
...

Now that we know how to use $\chi^2_\nu$ as a way to assess the "goodness of fit" for our model to data, let's revisit our experiment of measuring the length of the house key. After we create some "fake" data again, we'll compute $\chi^2_\nu$ and see how well we did with model selection.

```python
import numpy as np

# These lines copied from chi_squared_tutorial.ipynb

# Generate the measurement data
mu_key    = 6.0  # cm. The *actual* length of the key (unknown to us)
sigma_key = 0.05 # cm. Uncertainty on each measurement, based on the tick marks on our ruler
n_measure = 100  # Take 100 measurements
X = np.random.normal(mu_key, sigma_key, n_measure) # Generate the "measurements"- We assume a Gaussian distribution again
```

```python
# These lines copied from chi_squared_tutorial.ipynb

from scipy.optimize import minimize

def chi_squared(mu, sigma, data):
    '''
    Args
    ----------
    mu (float): Underlying mean of distribution (unknown to us).
    sigma (float): Measurement error.
    data (ndarray): Vector of observed data

    Returns
    ----------
    float: chi-squared value
    '''
    return np.sum(((data - mu)/sigma)**2)

mu_0 = 4.0 # cm. Our initial guess at the true length of the key, just from eye-balling it.
#print(chi_squared(mu_0, sigma_key, X))

min_results = minimize(chi_squared, mu_0, args=(sigma_key, X)) # Minimize chi-squared with respect to mu

print(f"Best-fitting value of mu_key from chi-squared minimization = {min_results.x[0]:.3f} cm")

##############################################
############ Fill in code here ###############
##############################################

# min_chi_squared_val = ... # Look at the documentation for scipy.optimize.minimize
#                           # (https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html)
#                           # to see how to retrieve the minimum chi-squared value from the min_results object.
#Way 1 to show everything
min_results = minimize(chi_squared, mu_0, args=(sigma_key, X), options={'disp':True})
#Way 2
min_chi_squared_val = min_results.fun
print(f"Minimum chi-squared = {min_chi_squared_val:.3f} ")
```

    Best-fitting value of mu_key from chi-squared minimization = 5.996 cm
    Optimization terminated successfully.
             Current function value: 101.461402
             Iterations: 3
             Function evaluations: 8
             Gradient evaluations: 4
    Minimum chi-squared = 101.461

```python
def reduced_chi_squared(min_chi_squared, N, m):
    '''
    Args
    ----------
    min_chi_squared (float): Minimum chi-squared value
    N (int): Number of data points
    m (int): Number of parameters fit by model.

    Returns
    ----------
    float: reduced chi-squared metric to assess goodness of fit.
    '''
    ##############################################
    ############ Fill in code here ###############
    ##############################################
    return min_chi_squared/(N-m)

reduced_chi_sq_val = reduced_chi_squared(min_chi_squared_val, n_measure, 1)
print(f"Reduced chi-squared = {reduced_chi_sq_val:.3f}")
```

    Reduced chi-squared = 1.025

**Question**: Did our model do a good job of describing the data? Why or why not?

*Enter your answer here*

Yes, because the reduced Chi-squared value, $\chi^2_\nu \approx 1$, which represents a value which is close to actual value

## 3. A cautionary tale

If you've made it this far, you've done *a lot* of reading about measuring the length of some silly key. Where's the astronomy?!

For the last part of this assignment, read [Gould 2013](https://arxiv.org/abs/1303.0834) (< 5 pages). Don't worry if you don't understand some bits of jargon here and there—just do your best! Regardless of the way in which this author went about making their point, they highlight a very important (and potentially costly) distinction between $\chi^2$ and $\chi^2_\nu$.

When you're done reading through the paper, come back to the assignment here.

Note that we didn't do any *minimizing* when it came to $\chi^2_\nu$, we just used it as a way to estimate whether or not our model was doing a good job at describing the data we observed. $\chi^2$ is the one that's actually related to the likelihood of the data, so it's the metric that we should be worried about when it comes to the optimization of our parameters.

**Question**: What did you think about the Gould paper? Can you write a one or two sentence summary of what it was about?

...

Gould's paper discusses the case of mass calculation of an Extra-galactic blackhole from David et al(2013), who used the minimization with dof (degrees of freedom) over $\chi^2$, or $\chi_\nu^2$, which resulted in a very large uncertainity in the output. Gould argues that the minimization should be over $\chi^2$ as the likelihood is dependent only on it and not on dofs. He further shows a 7 factor decrease of uncertainity in David et al. (2013)'s mass calculation, making it the most precise (more focus as compared to accuracy here) extra-galactic black holes

...


# Dicriminant Analysis

* So far, we have seen methods for regression, classification, and clustering. Today, let's look again at classification.
* The goal in *classification* is to take an input vector, **x**, and assign it to one of K discrete classes.
* Usually this is done by learning *decision boundaries* or *decision surfaces* that divide up the feature space and assign a class to each partition of the feature space created by the decision boundary.
* Data whose classes can be separated perfectly with a linear decision surface (i.e., a line in 2-D, a plane in 3-D, etc) are called *linearly separable.* These are the easiest classification problems to address.
* You can encode class labels in a number of ways: binary vectors, class number, etc.  Different encodings make sense in different situations.  But you want to select carefully - depending on the algorithm you use, it can make a difference.

## Fisher's Linear Discriminant

* We will begin with linear discriminant functions:
\begin{equation}
y(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + w_0
\end{equation}

* *Looks pretty familiar, right?* If you are on one side of the line, then you are in class 1.  If you are on the other side of the line, then you are in class 2.  So, the decision boundary is $y(\mathbf{x}) = 0$
* The distance of a point to the decision boundary is: $\frac{y(\mathbf{x})}{\left\| \mathbf{w} \right\|}$  See Figure 4.1 from textbook.

* We could use a least squares error function to solve for $\mathbf{w}$ and $w_0$ as we did in regression.  But, there are some issues.  *Can you think of any?*
* A more effective (and well-known) approach is *Fisher's Linear Discriminant*
* Given two classes, we can compute the mean of each class:
\begin{equation}
\mathbf{m}_1 = \frac{1}{{N}_1}\sum_{n\in C_1} \mathbf{x}_n, \quad \mathbf{m}_2 = \frac{1}{N_2}\sum_{n\in C_2} \mathbf{x}_n
\end{equation}
* So, we could try to maximize the separation of the means:
\begin{equation}
m_2 - m_1 = \mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)
\end{equation}
* This is like an optimal *dimensionality reduction*. $\mathbf{w}^T\mathbf{x}$ takes a $D$ dimensional data point and projects it down to 1-D with a weight sum of the original features.  We want to find a weighting that maximizes the separation of the class means.
* Not only do we want well separate means for each class, but we also want each class to be *compact* to minimize overlap between the classes.
* Consider the *within class variance:*

\begin{equation}
s_k^2 = \sum_{n \in C_k} (y_n - m_k)^2 = \sum_{n \in C_k} (\mathbf{w}^T\mathbf{x}_n - m_k)^2
\end{equation}
* So, we want to minimize within class variance and maximize between class separability.  How about the following objective function:

\begin{eqnarray}
J(\mathbf{w}) &=& \frac{(m_2 - m_1)^2}{s_1^2 + s_2^2}\\
      &=& \frac{\mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)(\mathbf{m}_2 - \mathbf{m}_1)^T\mathbf{w}}{\sum_{n \in C_1} (\mathbf{w}^T\mathbf{x}_n - m_1)^2 + \sum_{n \in C_2} (\mathbf{w}^T\mathbf{x}_n - m_2)^2}\\
	  &=& \frac{\mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)(\mathbf{m}_2 - \mathbf{m}_1)^T\mathbf{w}}{\mathbf{w}^T\left(\sum_{n \in C_1} (\mathbf{x}_n - \mathbf{m}_1)(\mathbf{x}_n - \mathbf{m}_1)^T + \sum_{n \in C_2} (\mathbf{x}_n - \mathbf{m}_2)(\mathbf{x}_n - \mathbf{m}_2)^T\right)\mathbf{w}}\\
	  &=& \frac{\mathbf{w}^T\mathbf{S}_B\mathbf{w}}{\mathbf{w}^T\mathbf{S}_W\mathbf{w}}
\end{eqnarray}

* Ok, so lets optimize

\begin{eqnarray}
\frac{\partial J(\mathbf{w})}{\partial \mathbf{w}} &=& \frac{2(\mathbf{w}^T\mathbf{S}_W\mathbf{w})\mathbf{S}_B\mathbf{w} - 2(\mathbf{w}^T\mathbf{S}_B\mathbf{w})\mathbf{S}_W\mathbf{w}}{(\mathbf{w}^T\mathbf{S}_W\mathbf{w})^2} = 0 \\
0 &=& \ \frac{\mathbf{S}_B\mathbf{w}}{(\mathbf{w}^T\mathbf{S}_W\mathbf{w})} -\frac{(\mathbf{w}^T\mathbf{S}_B\mathbf{w})\mathbf{S}_W\mathbf{w}}{(\mathbf{w}^T\mathbf{S}_W\mathbf{w})^2} \\
 (\mathbf{w}^T\mathbf{S}_W\mathbf{w})\mathbf{S}_B\mathbf{w} &=& (\mathbf{w}^T\mathbf{S}_B\mathbf{w})\mathbf{S}_W\mathbf{w} \\
 \mathbf{S}_W^{-1}\mathbf{S}_B\mathbf{w} &=& \lambda\mathbf{w}
\end{eqnarray}

where the scalar $\lambda = \frac{\mathbf{w}^T\mathbf{S}_B\mathbf{w}} { \mathbf{w}^T\mathbf{S}_W\mathbf{w}}$

* *Does this look familiar?* This is the eigenvalue/eigenvector equation.

## Eigenvalues and Eigenvectors

* For a square matrix $\mathbf{A}$ of size $M\times M$, the eigenvector equation is:
\begin{equation}
\mathbf{A}\mathbf{u}_i = \lambda\mathbf{u}_i
\end{equation}
* This can be viewed as a set of $M$ linear equations with the condition (i.e., the *characteristic equation*:
\begin{equation}
|\mathbf{A} - \lambda_i\mathbf{I}| = 0
\end{equation}
* Eigenvalues and eigenvectors play a big role in this course and this area.  We will return to this topic repeatedly.
* For now, we can recognize the equation and know that there are many solvers that will provide the eigenvalues and orthogonal eigenvectors for a matrix.

## Reading Assignment

* Section 4.1.7
* Section 5.1

```python
import numpy as np
import numpy.matlib
import matplotlib.pyplot as plt
import math

def fisherDiscriminant(data,t):
	# compute sb
	data1 = data[t==0,:]
	data2 = data[t==1,:]
	mean1 = np.atleast_2d(np.mean(data1,0))
	mean2 = np.atleast_2d(np.mean(data2,0))
	Sb  = (mean2 - mean1).T@(mean2 - mean1)
	Sw1 = np.dstack([(data1[i,:]-mean1).T@(data1[i,:]-mean1) for i in range(data1.shape[0])])
	Sw2 = np.dstack([(data2[i,:]-mean2).T@(data2[i,:]-mean2) for i in range(data2.shape[0])])
	Sw = np.sum(Sw1,2) + np.sum(Sw2,2)
	[l, v] = np.linalg.eig(np.linalg.inv(Sw)@Sb)
	data = v[:,l.argsort()[1]]@data.T
	return v[:,l.argsort()[1]],data

```

```python
# Fisher's Linear Discriminant
s     = .01
mean1 = [0, 0] #mean of class 1
mean0 = [1, 1] #mean of class 0
cov   = [1,0.01]*np.eye(2) #diagonal covariance
print(np.linalg.cond(cov))
numPnt1 = 100 #number of points for class1
numPnt0 = 100 #number of points for class0
data = np.random.multivariate_normal(mean1, cov, numPnt1) #generate points  for class 1
t      = np.ones([1,numPnt1]) #labels for class
data = np.vstack((data, np.random.multivariate_normal(mean0, cov, numPnt0))) #generate points for class 0
t    = np.hstack((t,np.zeros([1,numPnt0])))
v,data_t = fisherDiscriminant(data,t[0])
fig = plt.figure()
plt.scatter(data_t,np.zeros(data_t.shape),c=t[0,:],linewidth=0,s=100)

fig = plt.figure()
plt.scatter(data[:,0],data[:,1],c=t[0,:])
m = v[1]/v[0]
xx = np.arange(np.min(data[:,0])-.1,np.max(data[:,0])+.1,0.01)/m
yy = m*xx
plt.scatter(xx,yy,linewidth=0,s=5)
plt.plot([0,v[0]], [0,v[1]])
plt.show()
```

```python

```


# How to evaluate functions in sympy

Sympy knows functions. After

`from sympy import *`

which is usually ok in notebooks, i.e. `sin(x)`,  `cos(x)`, `exp(x)` or `ln(x)` are known functions.

Sympy also knows undefined functions, like

`f = Function('f')`

or

`g = Function('g')(x)`

Undefined functions can be used in differerential equations for example.

But sometimes it is useful to define a function from a given expression, like

$$
  f(x) = a\,x^2
$$

In this cases it is often recommended to use expressions:

`
a,x = symbols('a,x')
expr = a*x**2
`

Now, it is possible, to simulate the function $f(x)=a\,x^2$ using this expression, i.e.:

`
expr.subs(x,2) # to calculate f(2)
expr.diff(x)   # to calculate f'(x)
`

If you realy want to have a function, then it is often recommended to use `def` or `lambda` for this.

Let's try some examples:

```python
from sympy import *
init_printing()
```

```python
# some examples for the sin-function:
x = Symbol('x')
[sin(x), sin(0), sin(pi/2), sin(pi)]
```

```python
# Undefined functions:
f = Function('f')
f
```

    f

```python
f(x).diff()
```

```python
g = Function('g')(x)
g
```

```python
g.diff()
```

Now lets try to use the function

$$
  f(x) = a\,x^2
$$

Use expression to simulate the function:

```python
# Define symbols
a,x = symbols('a,x')

# define expression
expr = a*x**2

# output expression
expr
```

```python
# Evaluate f(2)
expr.subs(x,2)
```

```python
# Determine the expression that belongs to f'(x)
expr_1 = expr.diff(x)
expr_1
```

```python
# Evaluate f'(1)
expr_1.subs(x,1)
```

If we want to have a 'real' function, we could try to use `def`:

```python
def f(x):
    """
    define function
       a: global variable
       x: bound to the function
    """
    return a*x**2
```

```python
# Evaluate f(2)
f(2)
```

This seems to be ok, but it does not behave the same way as our previously defined expression does:

If, for example, at some point in the notebook, `a` is redefined, we get:

```python
a = 5

# the global variable a and the symbol a in expr are different:
expr.subs(x,2)
```

```python
# But here, a is the global variable of value 5
f(2)
```

This happens, because the function `f` uses the global variable `a`, which was redefined to be `a=5`.

But `expr` is defined with a Symbol `a` and this Symbol is not altered by changeing the global variable `a`.

If this is an unexpected behaviour, we could try to define the symbol `a` inside the function `f`:

```python
def f(x):
    """
    define function
       a: local variable, always a Symbol
       x: bound to the function
    """
    a = Symbol('a')
    return a*x**2
```

```python
a
```

```python
f(2)
```

How to define the derivative of `f`? The most straightforward way seems to be:

```python
def f_1(x):
    """
    define f'(x)
    for that, differentiate the expression f(x) wrt x
    and return the result.
    """
    return f(x).diff(x)

f_1(x)
```

This seems to work, but it fails:

```python
# This throws an ValueError
# The reason is, f_1 tries to diff f(2) wrt 2
f_1(2)
```

This means, we must be careful to define the derivative of the function `f`:

```python
def f_1(x):
    """
    define f'(x)
    for that, define a local variable v as Symbol
    and determine f'(v) wrt v.
    Substitute the symbol v by the bound variable x
    and return the result.
    """
    v = Symbol('v')
    return f(v).diff(v).subs(v,x)
```

```python
f_1(x)
```

```python
f_1(2)
```

Now assume, the value of the free symbol `a` has to be determined to satisfy

\begin{align*}
f(x) &= a\,x^2 \\
f(2) &= 8
\end{align*}

Using `expr` we might write

```python
eq = Eq(expr.subs(x,2),8)
eq
```

This gives the solution

```python
sol = solve(eq)
sol
```

With this, we can redefine `expr` as

```python
# We need to redefine a as Symbol
a = Symbol('a')
expr = expr.subs(a,sol[0])
expr
```

```python
# Test:
expr.subs(x,2)
```

Using `f(x)` this would read:

```python
eq = Eq(f(2),8)
eq
```

```python
sol = solve(eq)
sol
```

But now, $f$ needs to be rewritten in order to hold $f(x) = 2\,x^2$.

```python
def f(x):
    return 2*x**2
```

```python
f(2)
```


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also http://splines.readthedocs.io/.

# Derivation of Uniform Catmull-Rom Splines

tangent vectors:

\begin{equation}
\boldsymbol{\dot{x}}_i = \frac{\boldsymbol{x}_{i+1} - \boldsymbol{x}_{i-1}}{2}
\end{equation}

```python
%matplotlib inline
import sympy as sp
sp.init_printing()
```

```python
from utility import NamedExpression, NamedMatrix
```

Reminder: [Hermite splines](hermite-uniform.ipynb) use the start and end positions as well as the tangent vectors at start and end:

```python
control_values_H = sp.Matrix(sp.symbols('xbm:2 xdotbm:2'))
control_values_H
```

Catmull-Rom splines use 4 positions instead:
The start and end positions of the current segment ($\boldsymbol{x}_0$ and $\boldsymbol{x}_1$) plus the start position of the previous segment ($\boldsymbol{x}_{-1}$) and the end position of the following segment ($\boldsymbol{x}_2$).

TODO: figure? more explanations ...

```python
x_1, x0, x1, x2 = sp.symbols('xbm_-1 xbm:3')
```

```python
control_values_CR = sp.Matrix([x_1, x0, x1, x2])
control_values_CR
```

```python
xd0 = NamedExpression('xdotbm0', (x1 - x_1) / 2)
xd0
```

```python
xd1 = NamedExpression('xdotbm1', (x2 - x0) / 2)
xd1
```

So let's look for a way to transform Catmull-Rom control values to Hermite control values.
Since we already have $M_\text{H}$ from [the notebook about uniform Hermite splines](hermite-uniform.ipynb), we can use it to get $M_\text{CR}$:

```python
M_H = NamedMatrix(r'{M_\text{H}}', sp.Matrix([[2, -2, 1, 1], [-3, 3, -2, -1], [0, 0, 1, 0], [1, 0, 0, 0]]))
M_H
```

```python
M_CRtoH = NamedMatrix(r'{M_{\text{CR$\to$H}}}', 4, 4)
```

```python
M_CR = NamedMatrix(r'{M_\text{CR}}', M_H.name * M_CRtoH.name)
M_CR
```

```python
sp.Eq(control_values_H, M_CRtoH.name * control_values_CR)
```

If we substitute the above definitions of $\boldsymbol{\dot{x}}_0$ and $\boldsymbol{\dot{x}}_1$, we can directly read off the matrix elements:

```python
M_CRtoH.expr = sp.Matrix([[expr.coeff(cv) for cv in control_values_CR]
                          for expr in control_values_H.subs([xd0.args, xd1.args])])
M_CRtoH
```

```python
M_CRtoH.pull_out(sp.S.Half)
```

```python
print(_.expr)
```

```python
M_HtoCR = NamedMatrix(r'{M_{\text{H$\to$CR}}}', M_CRtoH.I.expr)
M_HtoCR
```

```python
print(_.expr)
```

```python
M_CR = M_CR.subs([M_H, M_CRtoH]).doit()
M_CR
```

```python
M_CR.pull_out(sp.S.Half)
```

```python
print(_.expr)
```

And for completeness' sake, its inverse:

```python
M_CR.I
```

```python
print(_.expr)
```

```python
t = sp.symbols('t')
```

```python
b_CR = NamedMatrix(r'{b_\text{CR}}', sp.Matrix([t**3, t**2, t, 1]).T * M_CR.expr)
b_CR.T
```

```python
sp.plot(*b_CR.expr, (t, 0, 1));
```

TODO: plot some example curves


# ORF307 Precept 9

## Optimality Conditions Review
primal and dual solutions are optimal if and only if the following are true:

1.   $x$ is primal feasible
2.   $y$ is dual feasible
3.   duality gap is zero

## Primal and Dual Simplex Takeaways

*   Primal simplex maintains primal feasibility and duality gap of zero, dual becomes feasible when converges
*   Dual simplex maintains dual feasibility and dualitgy gap of zero, primal becomes feasible when converges

## Sensitivity Analysis - Adding a New Variable

1.   $(x^*,0)$ is still primal feasible
2.   Dual problem has an extra constraint, is $A^T_{n+1}y^* + c_{n+1} \ge 0$ still dual feasible?
3.   If yes, $(x^*,0)$ is still optimal; If no, run primal simplex.

## Sensitivity Analysis - Adding a New Constraint

1.   Translates to having extra variable in dual problem, thus $(y^*,0)$ is still dual feasible
2.   Is $a^T_{m+1} x^* = b_{m+1}$ still primal feasible?
3.   If yes, $(y^*,0)$ is still optimal; If no, run dual simplex.

## Global/Local Sensitivity Analysis - Changing the Data

## Example 1

Consider the problem

$$
\begin{array}{ll}
\mbox{minimize} &-5x_1 - x_2 + 12x_3\\
\text{subject to} &3x_1 + 2x_2 + x_3 = 10\\
&5x_1 + 3x_2 + x_4 = 16\\
&x_1, \dots, x_4 \ge 0.
\end{array}
$$

An optimal solution to this problem is given by $x=(2,2,0,0)$. Suppose we change $a_{11}$ from 3 to $3 + \delta$. Keeping $x_1$ and $x_2$ as the basic variables, and let $A_B(\delta)$ be the corresponding basis matrix, as a function of $\delta$.

(a) Compute $A_B(\delta)^{-1}b$. For which values of $\delta$ is $A_B(\delta)$ a feasible basis?

(b) Compute $c_B^TA_B(\delta)^{-1}$. For which values of $\delta$ is $A_B(\delta)$ an optimal basis?

## Network Flows - Arc-Node Incidence Matrix

$m$ x $n$ matrix $A$ with entries

$$
\begin{align}
A_{ij} = \begin{cases}
1 &\mbox{if arc $j$ starts at node $i$}\\
-1 &\mbox{if arc $j$ starts at node $i$}\\
0 &\mbox{otherwise}
\end{cases}
\end{align}.
$$

Each column has one -1 and one 1

## Network Flows - Total Unimodularity

*   A matrix is totally unimodular if all its minors are -1, 0 or 1
*   Key property: the inverse of an nonsingular square submatrix of A has entires +1, -1, or 0
*   Above property, along with $b$ being an integer vector implies all the extreme points of the polyhedron $P = \{x \in \mathbf{R}^n \vert Ax=b, \mkern9mu x\ge0\}$ are integer vectors

## Example 2

A catering company must provide to a client $r_i$ tablecloths on each of $N$ consecutive days. The catering company can buy new tablecloths at a price of $p$ dollars each, or launder the used ones. Laundering can be done at a fast service facility that makes the tablecloths unavailable for the next $n$ days and cost $f$ dollars per tablecloth, or at a slower facility that makes tablecloths unavailable for the next $m$ days (with $m > n$) at a cost of $g$ dollars per tablecloth ($g<f$). The caterer's problem is to decide how to meet the client's demand at minimum cost, starting with no tablecloths and under the assumption that any leftover tablecloths have no value. Show that the problem can be formulated as a network flow problem.


# Particle in a box
*Roberto Di Remigio*, *Luca Frediani*

The [particle in a box] model is among the simplest, exactly solvable models in quantum mechanics.
In the one-dimensional case, we assume a particle of mass $m$ to be confined into a box of length $L$.
The confinement is achieved by means of a potential energy operator that is zero inside the box and infinite outside, as in the Figure below.

In practice, this means that the particle **cannot** escape the box: a result that we would have obtained also
from classical mechanics.
How does the quantum particle behave? We need to find the **eigenfunctions** and **eigenvalues** of the **Hamiltonian operator**, that is we have to solve the following ordinary differential equation:
\begin{equation}
-\frac{\hbar^2}{2m}\frac{\mathrm{d}^2}{\mathrm{d}x^2} \psi_n(x) = E_n\psi_n(x)
\end{equation}
with **boundary conditions**:
\begin{equation}
\begin{aligned}
  \psi_n(0) &= 0 \\
  \psi_n(L) &= 0
\end{aligned}
\end{equation}
Thus acceptable solutions are of the form:
\begin{equation}
\psi_n(x) = \sin(\frac{n\pi x}{L}) \quad\quad \forall n \neq 0
\end{equation}
with energies:
\begin{equation}
E_n = \frac{h^2n^2}{8mL^2}  \quad\quad \forall n \neq 0
\end{equation}

[particle in a box]: https://en.wikipedia.org/wiki/Particle_in_a_box

## Exercise 1: Normalization

The wavefunction(s) given above as solution to the particle in a box problem are not normalized:
\begin{equation}
\left\langle \psi_n | \psi_n \right\rangle = \int \mathrm{d}x \psi_n^*(x)\psi_n(x) = |A|^2 \neq 1
\end{equation}
Find the normalization constant.

## Exercise 2: Ground-state and probabilities

Given the **normalized** ground-state wavefunction for a particle confined to a one-dimensional box of length $L$, suppose the box is $10.0\,\,\mathrm{nm}$ long. What is the probability that the particle is:
1. Between $a = 4.95\,\,\mathrm{nm}$ and $b=5.05\,\,\mathrm{nm}$
2. Between $a = 1.95\,\,\mathrm{nm}$ and $b=2.05\,\,\mathrm{nm}$
3. Between $a = 9.90\,\,\mathrm{nm}$ and $b=10.0\,\,\mathrm{nm}$
4. In the right half of the box
5. In the central third of the box

How can we generalize to the excited states?

## Exercise 3: Eigenfunctions and probability distributions

Write a Python function to plot the eigenfunctions of the particle in a box model. The function should accept the quantum number $n$, the length of the box $L$ and a NumPy array of $x$ values as arguments:

```Python
def eigenfunction1D(n, L, x):
    """ Normalized eigenfunction for the 1D particle in a box.

    n -- the quantum number
    L -- the size of the box
    x -- the NumPy array with the x values
    """
```
Once this function is defined, we can obtain the respective probability distribution by taking its square:
```Python
x = np.linspace(0, 10.0, 1000)
eig = eigenfunction1D(1, 10.0, x)
prob = eigenfunction1D(1, 10.0, x)**2
```
and plot both of them with:
```Python
plt.plot(x, eig)
plt.plot(x, prob)
```

```python
import numpy as np
import matplotlib.pyplot as plt
# make sure we see it on this notebook
%matplotlib inline

def eigenfunction1D(n, L, x):
    """ Normalized eigenfunction for the 1D particle in a box.

    n -- the quantum number
    L -- the size of the box
    x -- the NumPy array with the x values
    """
    raise NotImplementedError('You need to write this function!')

x = np.linspace(0, 10.0, 1000)
eig = eigenfunction1D(1, 10.0, x)
prob = eigenfunction1D(1, 10.0, x)**2

plt.plot(x, eig)
plt.plot(x, prob)
```

## Exercise 4: Normalization of linear combinations of eigenfunctions

A linear combination is defined as:
\begin{equation}
\Psi(x) = \sum_{i=0}^{N} c_i \psi_i(x)
\end{equation}
where $c_i$ are the coefficients and $\psi_i(x)$ are the eigenfunctions of the particle in a box.
Is $\Psi(x)$ normalized? If not, how can we normalize it? Write a Python function that returns the normalization constant given a vector of coefficients $c_i$ in the linear combination.

```Python
def normalize(coeffs):
    """ Normalization constant for a linear combination of 1D particle in a box eigenfunctions

    coeffs -- a NumPy array with the coefficients
    """
```

```python
import numpy as np
import matplotlib.pyplot as plt
# make sure we see it on this notebook
%matplotlib inline

def normalize(coeffs):
    """ Normalization constant for a linear combination of 1D particle in a box eigenfunctions

    coeffs -- a NumPy array with the coefficients
    """
    raise NotImplementedError('You need to write this function!')

coeffs = np.array([0.5, 0.5])
normalize(coeffs)
```

## Exercise 5: Energy of linear combinations

The **expectation value** of an operator $O$ on the wavefunction $\Psi$ is defined as:
\begin{equation}
\bar{O} = \frac{\langle\Psi|O|\Psi\rangle}{\langle\Psi|\Psi\rangle} = \langle\Psi|O|\Psi\rangle
\end{equation}
where the last equality holds only if $\Psi$ is normalized.
The energy is the expectation value of the Hamiltonian operator:
\begin{equation}
\bar{H} = E = \langle\Psi|H|\Psi\rangle
\end{equation}

Given $\Psi$ a linear combination of particle in a box eigenfunctions, calculate its respective energy.
**Hint**: remember that $H\psi_i(x) = E_i\psi_i(x)$ and that the $\psi_i(x)$ are normalizd. You can also assume $\Psi$ to be already normalized.

Write a Python function to calculate the energy of such linear combinations. The function should accept the coefficients $c_i$ of the linear combination, the mass $M$ of the particle and the length $L$ of the box as arguments.

```Python
def energy(coeffs, M, L):
    """ Return energy of a linear combination of 1D particle in a box eigenfunctions.

    coeffs -- the coefficients of the linear combination
    M -- the mass of the particle
    L -- the size of the box
    """
```

```python
def energy(coeffs, M, L):
    """ Return energy of a linear combination of 1D particle in a box eigenfunctions.

    coeffs -- the coefficients of the linear combination
    M -- the mass of the particle
    L -- the size of the box
    """
    raise NotImplementedError('You have to write this function')

coeffs = np.array([0.5, 0.5])
energy(coeffs, 1.0, 10.0)
```

## Exercise 6: Linear combinations of eigenfunctions

We can now try to plot linear combinations of eigenfuntions and their corresponding probability densities.
Define the appropriate function to do so. Your function should use your previous implementation of `eigenfunction1D` to achieve its purpose.


[](https://colab.research.google.com/github/mravanba/comp551-notebooks/blob/master/EMforGaussianMixture.ipynb)

# Expectation Maximization for Gaussian Mixture Model
In K-means each datapoint is assigned to one cluster. An alternative is for each datapoint ($n$) to have a distribution over clusters -- that is $r_{n,k} \in [0,1]$ and $\sum_k r_{n,k} = 1$. To do this we can assume each cluster has a Gaussian distribution $\mathcal{N}(\mu_k, \Sigma_k)$. So we assume the data-distrubtion is a **mixture of Gaussians**
$$
p(x; \pi, \{\mu_k, \Sigma_k\}) = \sum_k \pi_k \mathcal{N}(x; \mu_k, \Sigma_k)
$$
where $\pi_k= p(z=k)$ with $\sum_k \pi_k = 1$ defining the weight of each Gaussian in the mixture. These weights should sum to one so that we have a valid pdf. To maximize the logarithm of the marginal-likelihood
$\ell(\pi, \{\mu_k, \Sigma_k\}) = \sum_n \log \left ( \sum_k \pi_k \mathcal{N}(x; \mu_k, \Sigma_k) \right)$, we set its partial derivative wrt various parameters to zero. This gives us the value of these parameters in terms of membership probabilities, aka *responsibilities*: $r_{n,k} = p(z=k|x^{(n)})= \frac{\pi_k \mathcal{N}(x; \mu_k, \Sigma_k)}{\sum_c \pi_c \mathcal{N}(x; \mu_c, \Sigma_c)}$. Since responsibilities are functions of model parmeters, we perform an iterative updating of these two values:
1. update responsibilites given the model parameters
2. given the responsibilities $r_{n,k}$, update the parameters $\mu_k, \Sigma_k$ and $\pi$. As we said these updates are given by taking the derivative of the log-likelihood:
    - New $\pi_k$ is easy to estimate, it is proportional to the $\sum_n r_{n,k}$. Since $\pi_k$ should sum to one we set
$$
\pi_k = \frac{\sum_n r_{n,k}}{\sum_{n,c} r_{n,c}}
$$
    - For $\mu_k$ and $\Sigma_k$ we need to estimate mean and covariance of a Gaussian using *weighted* samples, where the (unnormalized) weights for the $k^{th}$ Gaussian in the mixture are $r_{n,k} \forall n$.
\begin{align}
\mu_k &= \frac{\sum_n r_{n,k} x^{(n)}}{\sum_n r_{n,k}} \\
\Sigma_k &= \frac{ \sum_n r_{n,k} (x^{(n)} - \mu_k)(x^{(n)} - \mu_k)^\top }{\sum_n r_{n,k}}
\end{align}
The above gives us weighted mean and weighted covariance. We then repeat steps 1 and 2 similar to K-means.

Lets implement EM for Gaussian Mixture Model (GMM) below. We re-use our previous impelementation of multivariate Gaussian in GMM.

```python
import numpy as np
#%matplotlib notebook
%matplotlib inline
import matplotlib.pyplot as plt
from IPython.core.debugger import set_trace
import scipy as sp
np.random.seed(1234)
```

```python
#For more comments on multivariate Gaussian class refer to: https://github.com/mravanba/comp551-notebooks/blob/master/Gaussian.ipynb
class Gaussian():
    def __init__(self, mu=0, sigma=0):
        self.mu = np.atleast_1d(mu)
        if np.array(sigma).ndim == 0:
            self.Sigma = np.atleast_2d(sigma**2)
        else:
            self.Sigma = sigma

    def density(self, x):
        N,D = x.shape
        xm = (x-self.mu[None,:])
        normalization = (2*np.pi)**(-D/2) * np.linalg.det(self.Sigma)**(-1/2)
        quadratic = np.sum((xm @ np.linalg.inv(self.Sigma)) * xm,axis=1)
        return normalization * np.exp(-.5 * quadratic)

class GMM:
    def __init__(self, K=5, max_iters=200, epsilon=1e-5):
        self.K = K                            #Number of Gaussians
        self.max_iters = max_iters            #maximum number of iteration we want to run the mixture model
        self.epsilon = epsilon                #small value used as tolerance and for alleviating the singularity problem

    def fit(self, x):
        N,D = x.shape
        init_centers = np.random.choice(N, self.K)                               #generate K random values from [0,N-1]
        pi = np.ones(self.K)/self.K                                              #initialize the weight of the Gaussians
        mu = x[init_centers]                                                     #select K data points to initialize the mean parameter of K Gaussians shape:K X D
        sigma = np.tile(np.diag(np.var(x,axis=0))[None,:,:], (self.K, 1,1))      #initialize the sigma parameter by computing the variance of the data and making a diagonal matrix from it
        #Note that the tile function copies the sigma K times for all the Gausssians
        r = np.zeros((N,self.K))                                                 #initialize the responsibilities to zero
        ll = -np.inf                                                             #initialize the log likelihood to negative inifinity
        for t in range(self.max_iters):
            #update the responsibilities
            for i in range(self.K):
                r[:,i] = pi[i] * Gaussian(mu[i], sigma[i]).density(x)            #dimension N of the expression
            #normalize them over number of Gaussians
            r_norm = r / np.sum(r, axis=1, keepdims=True)                        #keepdims preserves the 2nd dimension for broadcasting during the division

            #update the parameters of the gaussian using MLE
            for i in range(self.K):
                mu[i,:] = np.average(x, axis=0, weights=r_norm[:,i])             #computes the weighted average where the weights are given by responsibilities of the i-th Gaussian
                sigma[i,:,:] = np.cov(x, aweights=r_norm[:,i], rowvar=False) + self.epsilon * np.eye(D)[None,:,:]
                #Note that we compute the weighted covariance natrix where the weights are given by responsibilities of the i-th Gaussian
                #We set rowvar False as our data is in the first axis with the feature variables in the second and we want to compute variance along columns
                #An epsilon variance is added in all the features to make sure it doesn't reach singularity where a Gaussian fit a single data point with zero variance in all the features
            #update the weight of Gaussians
            pi = np.sum(r_norm, axis=0)
            pi /= np.sum(pi)                                                     #normalize it
            #calculate the new log likelihood
            ll_new = np.mean(np.log(np.sum(r, axis=1)))
            #check if the log likelihood differenc is within the tolerance
            if np.abs(ll_new - ll) < self.epsilon:
                print(f'converged after {t} iterations, average log-likelihood {ll_new}')
                break
            ll = ll_new
        return mu, sigma, r
```

In the implementation above note that we add a diagonal matrix with small constant values ($\epsilon$) on the diagonal to the empirical covariance matrix. This is to avoid degeneracy. If the model puts one of the Gaussians centered on a data-points and make the covariance very small, it can arbitrarily increase the likelihood of the data (see Bishop p.434). This addition of epsilon to the diagonal prevents this degeneracy.

Now let's apply this to the Iris dataset

```python
from sklearn import datasets
dataset = datasets.load_iris()
x, y = dataset['data'][:,:2], dataset['target']

gmm = GMM(K=3)
mu, sigma, resp = gmm.fit(x)
resp /= np.sum(resp,axis=1,keepdims=True)

# plotting the result
fig, axes = plt.subplots(ncols=3, nrows=1, constrained_layout=True, figsize=(15, 5))
axes[0].scatter(x[:,0], x[:,1], c=resp, s=2)
axes[0].scatter(mu[:,0], mu[:,1], marker='x')

x0v = np.linspace(np.min(x[:,0]), np.max(x[:,0]), 200)
x1v = np.linspace(np.min(x[:,1]), np.max(x[:,1]), 200)
x0,x1 = np.meshgrid(x0v, x1v)
x_all = np.vstack((x0.ravel(),x1.ravel())).T

#Get the densities
p0 = Gaussian(mu[0], sigma[0]).density(x_all)
p1 = Gaussian(mu[1], sigma[1]).density(x_all)
p2 = Gaussian(mu[2], sigma[2]).density(x_all)
p = np.vstack([p0,p1,p2]).T
p /= np.sum(p,axis=1,keepdims=True)

#axes[0].contour(x0, x1, p0.reshape(x0.shape))
axes[0].tricontour(x_all[:,0], x_all[:,1], p0)
axes[0].tricontour(x_all[:,0], x_all[:,1], p1)
axes[0].tricontour(x_all[:,0], x_all[:,1], p2)
axes[0].set_title('Mixture of Gaussians found using EM')

axes[1].scatter(x[:,0], x[:,1], c=y, s=2)
axes[1].scatter(x_all[:,0], x_all[:,1], c=p, marker='.', alpha=.01)
axes[1].set_title('Cluster Responsibilities')

axes[2].scatter(x[:,0], x[:,1], c=y, s=2)
axes[2].set_title('class labels')
plt.show()
```

A mixture of Gaussians where the membership of each point is unobserved is an example of a **latent variable model**.
A latent variable model $p(x,z; \theta)$ assumes unobserved or latent variables $z$ that help explain the observations $x$. In this setup we still want to maximize the **marginal likelihood** of data ($z$ is marginalized out):
$$
\max_\theta \sum_n \log p(x^{(n)}; \theta) = \max_\theta \sum_n \log \sum_z p(x^{(n)}, z; \theta)
$$
EM can be used for learning with *latent variable models* or when we have *missing data*. The general approach is similar to what we saw here:
- computing the posterior $p(z | x^{(n)}; \theta) \forall n$ (Expectation or **E step**)
- maximizing the expected log-likelihood using this probabilistic *completion* of the data (Maximization or **M step**)


---
layout: post
title: Project Euler - Problem 5
post-order: 005
---

2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder.

What is the smallest positive number that is **evenly divisible** by all of the numbers from 1 to 20?

### Solution #1

Let's brute-force it and see what happens. One thing we know: the number we are trying to find **cannot be** smaller than the products of all primes up to $20$. And the reason for that is, the number we are trying to find has to be divisible by $1$ to $20$ and the only way that can be accomplished is if the prime factors (for example $13$) is in that product.

```python
import math
import time

# let's setle "floor" first: the products of all primes from 1 to 20
def primes_up_to(n):
    product = 2

    for candidate_for_prime in range (3,n+1):
        for i in range (2,candidate_for_prime):
            if (candidate_for_prime % i == 0):
                break
            elif ((candidate_for_prime % i != 0) & (i + 1 == candidate_for_prime)):
                product *= candidate_for_prime

    return product

start = time.time()

divisible_up_to = 20
floor = primes_up_to(divisible_up_to)

# the highest we can go is to n! (n factorial)
ceiling = math.factorial(divisible_up_to)

found = False

for ii in range (floor,ceiling):
    for i in range(2,divisible_up_to+1):
        if ii % i != 0:
            break
        elif i == divisible_up_to:
            print (ii)
            elapsed = time.time() - start
            print(elapsed)
            found = True
            break

    if found:
        break
```

    232792560
    162.78695583343506

As you may notice, the running time is extremely high (and it will get higher if we choose a greater `divisible_up_to`). We need to find a more efficient way to tacke this problem.

### Solution #2 - Greatest power of primes

The solution for this problem actually requires **no computation**. And the reason for that is, if we find the prime factorization of each number up to 20 and multiply the greatest power of each, we will find the correct solution.

$$
\begin{align}
2 &= 2^1\\
3 &= 3^1\\
4 &= 2^2\\
5 &= 5^1\\
\vdots\\
18 &= 2^1 \cdot 3^2\\
19 &= 19^1\\
20 &= 2^2 \cdot 5^1
\end{align}
$$

We can take advantage of this and build a list with the prime factors and a list with the prime factor's greatest power that shows up from 1 to `divisible_up_to`. After that, we evaluate the first item of the list of primes raised to the first item of the list of powers, multiplied by the second item of the list of primes raised to the second item of the list of powers and so on.

|LISTS|<td colspan=11>|
|----------------|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| list of primes | [ | 2 | 3 | 5 | 7 | 11| 13| 17| 19| ] |
| list of powers | [ |4  | 2 | 1 | 1 | 1 | 1 | 1 | 1 | ] |

Answer = $2^4 \cdot 3^2 \cdot 5^1 \cdot 7^1 \cdot 11^1 \cdot 13^1 \cdot 17^1 \cdot 19^1 = 232792560$

```python
def primes_list(n):
    list_of_primes = [1,2]

    for i in range (3,n+1):
        for ii in range (2,i):
            if (i % ii == 0):
                break
            elif ((i % ii != 0) & (ii + 1 == i)):
                list_of_primes.append(i)

    return list_of_primes

list_of_primes = primes_list(20)
print (list_of_primes)
```

    [1, 2, 3, 5, 7, 11, 13, 17, 19]

Let's build a new list, with the same number of elements, but now filled with zeros. It will receive the greatest power for each prime:

```python
list_of_powers = [1] * len(list_of_primes)
print(list_of_powers)
```

    [1, 1, 1, 1, 1, 1, 1, 1, 1]

Now we need to check what is the greatest power that shows up, for each of the primes, from 1 to 20.

```python
for i in list_of_primes:
    print (i)

# if it's prime, there's no need to check: the power will be 1
# do the prime factorization of all numbers up to 20
```

    1
    2
    3
    5
    7
    11
    13
    17
    19

Placing everything on a single code,

```python

```

The performance of this second algorithm is much better. We can even increase `divisible_up_to` without a substantial compromise on its performance:

```python

```


# Integrators

A dynamics simulation integrates the differential equations of motion.
We have,
\begin{align}
    \frac{dv}{dt} &= a \\
    \frac{dx}{dt} &= v
\end{align}
where $x$ is position, $v$ is velocity and $a$ is acceleration.
There is extensive research on integrators and their properties for dynamic
systems. Here, we focus on two commond ones: the simple Euler integrator and
the slightly more advanced Leapfrog integrator. The integrators are implemented
in separate classes and can be extended, e.g. with Runge-Kutta methods.

## Euler

The Euler integrator is a simple first-order method.

\begin{align}
    x_{t+1} &= x_t + v_t \Delta t \\
    v_{t+1} &= v_t + a_t \Delta t
\end{align}

## Leapfrog

Reference {cite}`birdsall2004plasma`.
The main integrator used here is the Leapfrog integrator. It has only a very
small computational overhead compared to the Euler method.

\begin{align}
    x_{t+1} &= x_t + v_t \Delta t + \frac{1}{2} a_t \Delta t^2 \\
    v_{t+1} &= v_t + \frac{1}{2}(a_t + a_{t+1}) \Delta t
\end{align}