```python
from sympy import *
from sympy.abc import *
from sympy.galgebra.ga import *
import numpy as np
from numpy import linalg as LA
from __future__ import print_function
init_printing()
```

## PDE
The acoustic wave equation for the square slowness m and a source q is given in 3D by :

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) =q  \\
 &u(.,0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

with the zero initial conditons to guaranty unicity of the solution

```python
p=Function('p')
m,s,h = symbols('m s h')
m=M(x,y,z)
q=Q(x,y,t)
d=D(x,y,t)
e=E(x,y)
```

### Time and space  discretization as a Taylor expansion.

The time discretization is define as a second order ( $ O (dt^2)) $) centered finite difference to get an explicit Euler scheme easy to solve by steping in time.

$ \frac{d^2 u(x,t)}{dt^2} \simeq \frac{u(x,t+dt) - 2 u(x,t) + u(x,t-dt)}{dt^2} + O(dt^2) $

And we define the space discretization also as a Taylor serie, with oder chosen by the user. This can either be a direct expansion of the second derivative bulding the laplacian, or a combination of first oder space derivative. The second option can be a better choice in case you would want to extand the method to more complex wave equations involving first order derivatives in chain only.

$ \frac{d^2 u(x,t)}{dt^2} \simeq \frac{1}{dx^2} \sum_k \alpha_k (u(x+k dx,t)+u(x-k dx,t)) + O(dx^k) $

```python
dtt=as_finite_diff(p(x,y,z,t).diff(t,t), [t-s,t, t+s])
dt=as_finite_diff(p(x,y,t).diff(t), [t-s, t+s])
# Spacial finite differences can easily be extended to higher order by increasing the list of sampling point in the next expression.
# Be sure to keep this stencil symmetric and everything else in the notebook will follow.
dxx=as_finite_diff(p(x,y,z,t).diff(x,x), [x-h,x, x+h])
dyy=as_finite_diff(p(x,y,z,t).diff(y,y), [y-h,y, y+h])
dzz=as_finite_diff(p(x,y,z,t).diff(z,z), [z-h,z, z+h])
dtt,dxx,dyy,dt
```

### Solve forward in time

The wave equation with absorbing boundary conditions writes

$ \eta \frac{d u(x,t)}{dt} + \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) =q  $

and the adjont wave equation

$ -\eta \frac{d u(x,t)}{dt} + \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) =q  $

where $ \eta$  is a damping factor equal to zero inside the physical domain and decreasing inside the absorbing layer from the pysical domain to the border

```python
# Forward wave equation
wave_equation = m*dtt- (dxx+dyy+dzz)
stencil = solve(wave_equation,p(x,y,z,t+s))[0]
ts=lambdify((p(x,y,t-s),p(x-h,y,t), p(x,y,t), p(x+h,y,t),p(x,y-h,t), p(x,y+h,t), q , m, s, h,e),stencil,"numpy")

eq=Eq(p(x,y,z,t+s),stencil)
eq
```

## Rewriting the discret PDE as part of an Inversion
### Accuracy and rigourousness of the dicretization

The above axpression are good for modelling. However, if you want to include a wave equation solver into an Inversion workflow, a more rigourous study of the discretization must be done. We can rewrite a single time step as follows

$ A_3  u(x,t+dt)  = A_1 u(x,t) + A_2 u(x,t-dt) +q(x,t)$

where $ A_1,A_2,A_3 $ are square, invertible matrices, and symetric without any boundary conditions. In more details we have :

\begin{align}
& A_1 = \frac{2}{dt^2 m} + \Delta \\
& A_2 = \frac{-1}{dt^2 m} \\
& A_3 = \frac{1}{dt^2 m}
\end{align}

We can the write the action of the adjoint wave equation operator. The adjoint wave equation is defined by
\begin{cases}
 &m \frac{d^2 v(x,t)}{dt^2} - \nabla^2 v(x,t) = \delta d  \\
 &v(.,T) = 0 \\
 &\frac{d v(x,t)}{dt}|_{t=T} = 0
\end{cases}

but by choosing to discretize first we will not discretize this equation. Instead we will take the adjoint of the forward wave equation operator and by testing that the operator is the true adjoint, we will guaranty solving the adjoint wave equation. We have the the single time step for the adjoint wavefield going backward in time in order to keep an explicit Euler scheme

$  A_2^T v(x,t-dt)  = A_1^T v(x,t) + A_3^T v(x,t+dt) + \delta d(x,t)$

and as $A_2$ and $A_3$ are diagonal matrices  there is no issue in inverting it. We can also see that choosing a asymetric stencil for the spacial derivative may lead to erro has the Laplacian would stop to be self-adjoint, and the actual adjoint finite difference scheme should be implemented.

```python
# Adjoint wave equation
wave_equationA = m*dtt- (dxx+dyy) - D(x,y,t) - e*dt
stencilA = solve(wave_equationA,p(x,y,t-s))[0]
tsA=lambdify((p(x,y,t+s),p(x-h,y,t), p(x,y,t), p(x+h,y,t),p(x,y-h,t), p(x,y+h,t), d , m, s, h,e),stencilA,"numpy")
stencilA
```

# Define the discrete model

```python
import matplotlib.pyplot as plt
from matplotlib import animation

hstep=25 #space increment d  = minv/(10*f0);
tstep=2 #time increment dt < .5 * hstep /maxv;
tmin=0.0 #initial time
tmax=300 #simulate until
xmin=-875.0 #left bound
xmax=875.0 #right bound...assume packet never reaches boundary
ymin=-875.0 #left bound
ymax=875.0 #right bound...assume packet never reaches boundary
f0=.010
t0=1/.010
nbpml=10
nx = int((xmax-xmin)/hstep) + 1 #number of points on x grid
ny = int((ymax-ymin)/hstep) + 1 #number of points on x grid
nt = int((tmax-tmin)/tstep) + 2 #number of points on t grid
xsrc=-400
ysrc=0.0
xrec = nbpml+4
#set source as Ricker wavelet for f0
def source(x,y,t):
    r = (np.pi*f0*(t-t0))
    val = (1-2.*r**2)*np.exp(-r**2)
    if abs(x-xsrc)<hstep/2 and abs(y-ysrc)<hstep/2:
        return val
    else:
        return 0.0

def dampx(x):
    dampcoeff=1.5*np.log(1.0/0.001)/(5.0*hstep);
    if x<nbpml:
        return dampcoeff*((nbpml-x)/nbpml)**2
    elif x>nx-nbpml-1:
        return dampcoeff*((x-nx+nbpml)/nbpml)**2
    else:
        return 0.0
def dampy(y):
    dampcoeff=1.5*np.log(1.0/0.001)/(5.0*hstep);
    if y<nbpml:
        return dampcoeff*((nbpml-y)/nbpml)**2
    elif y>ny-nbpml-1:
        return dampcoeff*((y-ny+nbpml)/nbpml)**2
    else:
        return 0.0
```

```python
# Velocity models
def smooth10(vel,nx,ny):
    out=np.ones((nx,ny))
    out[:,:]=vel[:,:]
    for a in range(5,nx-6):
        out[a,:]=np.sum(vel[a-5:a+5,:], axis=0) /10
    return out

# True velocity
vel=np.ones((nx,ny)) + 2.0
vel[floor(nx/2):nx,:]=4.5
mt=vel**-2
# Smooth velocity
v0=smooth10(vel,nx,ny)
m0=v0**-2

dm=m0-mt
```

# Create functions for the PDE

The Gradient/Born are here so that everything is at the correct place, it is described later

```python
def Forward(nt,nx,ny,m):
    u=np.zeros((nt,nx,ny))
    rec=np.zeros((nt,ny-2))
    for ti in range(0,nt):
        for a in range(1,nx-1):
            for b in range(1,ny-1):
                src = source(xmin+a*hstep,ymin+b*hstep,tstep*ti)
                damp=dampx(a)+dampy(b)
                if ti==0:
                    u[ti,a,b]=ts(0,0,0,0,0,0,src,m[a,b],tstep,hstep,damp)
                elif ti==1:
                    u[ti,a,b]=ts(0,u[ti-1,a-1,b],u[ti-1,a,b],u[ti-1,a+1,b],u[ti-1,a,b-1],u[ti-1,a,b+1],src,m[a,b],tstep,hstep,damp)
                else:
                    u[ti,a,b]=ts(u[ti-2,a,b],u[ti-1,a-1,b],u[ti-1,a,b],u[ti-1,a+1,b],u[ti-1,a,b-1],u[ti-1,a,b+1],src,m[a,b],tstep,hstep,damp)
                if a==xrec :
                    rec[ti,b-1]=u[ti,a,b]
    return rec,u

def Adjoint(nt,nx,ny,m,rec):
    v=np.zeros((nt,nx,ny))
    srca=np.zeros((nt))
    for ti in  range(nt-1, -1, -1):
        for a in range(1,nx-1):
            for b in range(1,ny-1):
                if a==xrec:
                    resid=rec[ti,b-1]
                else:
                    resid=0
                damp=dampx(a)+dampy(b)
                if ti==nt-1:
                    v[ti,a,b]=tsA(0,0,0,0,0,0,resid,m[a,b],tstep,hstep,damp)
                elif ti==nt-2:
                    v[ti,a,b]=tsA(0,v[ti+1,a-1,b],v[ti+1,a,b],v[ti+1,a+1,b],v[ti+1,a,b-1],v[ti+1,a,b+1],resid,m[a,b],tstep,hstep,damp)
                else:
                    v[ti,a,b]=tsA(v[ti+2,a,b],v[ti+1,a-1,b],v[ti+1,a,b],v[ti+1,a+1,b],v[ti+1,a,b-1],v[ti+1,a,b+1],resid,m[a,b],tstep,hstep,damp)
                if abs(xmin+a*hstep-xsrc)<hstep/2 and abs(ymin+b*hstep-ysrc)<hstep/2:
                    srca[ti]=v[ti,a,b]
    return srca,v

def Gradient(nt,nx,ny,m,rec,u):
    v1=np.zeros((nx,ny))
    v2=np.zeros((nx,ny))
    v3=np.zeros((nx,ny))
    grad=np.zeros((nx,ny))
    for ti in range(nt-1,-1,-1):
        for a in range(1,nx-1):
            for b in range(1,ny-1):
                if a==xrec:
                    resid=rec[ti,b-1]
                else:
                    resid=0
                damp=dampx(a)+dampy(b)
                v3[a,b]=tsA(v1[a,b],v2[a-1,b],v2[a,b],v2[a+1,b],v2[a,b-1],v2[a,b+1],resid,m[a,b],tstep,hstep,damp)
                grad[a,b]=grad[a,b]-(v3[a,b]-2*v2[a,b]+v1[a,b])*(u[ti,a,b])
        v1,v2,v3=v2,v3,v1
    return tstep**-2*grad

def Born(nt,nx,ny,m,dm):
    u1=np.zeros((nx,ny))
    U1=np.zeros((nx,ny))
    u2=np.zeros((nx,ny))
    U2=np.zeros((nx,ny))
    u3=np.zeros((nx,ny))
    U3=np.zeros((nx,ny))
    rec=np.zeros((nt,ny-2))
    src2=0
    for ti in range(0,nt):
        for a in range(1,nx-1):
            for b in range(1,ny-1):
                damp=dampx(a)+dampy(b)
                src = source(xmin+a*hstep,ymin+b*hstep,tstep*ti)
                u3[a,b]=ts(u1[a,b],u2[a-1,b],u2[a,b],u2[a+1,b],u2[a,b-1],u2[a,b+1],src,m[a,b],tstep,hstep,damp)
                src2 = -tstep**-2*(u3[a,b]-2*u2[a,b]+u1[a,b])*dm[a,b]
                U3[a,b]=ts(U1[a,b],U2[a-1,b],U2[a,b],U2[a+1,b],U2[a,b-1],U2[a,b+1],src2,m[a,b],tstep,hstep,damp)
                if a==xrec :
                    rec[ti,b-1]=U3[a,b]
        u1,u2,u3=u2,u3,u1
        U1,U2,U3=U2,U3,U1
    return rec

```

# A Forward propagation example

```python
(rect,ut)=Forward(nt,nx,ny,mt)
```

```python

fig = plt.figure()
plts = []             # get ready to populate this list the Line artists to be plotted
plt.hold("off")
for i in range(nt):
    r = plt.imshow(ut[i,:,:])   # this is how you'd plot a single line...
    plts.append( [r] )
ani = animation.ArtistAnimation(fig, plts, interval=50,  repeat = False)   # run the animation
plt.show()
```

```python
fig2 = plt.figure()
plt.hold("off")
shotrec = plt.imshow(rect)   # this is how you'd plot a single line...
#plt.show()
```

# Adjoint test

In ordr to guaranty we have the gradient we need to make sure that the solution of the adjoint wave equation is indeed the true adjoint. Tod os so one should check that

$ <Ax,y> - <x,A^Ty> = 0$

where $A$ is the wave_equation,  $A^T$ is wave_equationA and $x,y$ are any random vectors in the range of each operator. This can however be expensive as this two vector would be of size $N * n_t$. To test our operator we will the relax test by

$ <P_r A P_s^T x,y> - <x,P_SA^TP_r^Ty> = 0$

where $P_r , P_s^T$ are the source and recevier projection operator mapping the source and receiver locations and times onto the full domain. This allow to have only a random source of size $n_t$ at a random postion.

```python
(rec0,u0)=Forward(nt,nx,ny,m0)
(srca,v)=Adjoint(nt,nx,ny,m0,rec0)
```

```python
plts = []             # get ready to populate this list the Line artists to be plotted
plt.hold("off")
for i in range(0,nt):
    r = plt.imshow(v[i,:,:],vmin=-100, vmax=100)   # this is how you'd plot a single line...
    plts.append( [r] )
ani = animation.ArtistAnimation(fig, plts, interval=50,  repeat = False)   # run the animation
plt.show()
```

```python

shotrec = plt.plot(srca)   # this is how you'd plot a single line...
#plt.show()
```

```python
# Actual adjoint test
term1=0
for ti in range(0,nt):
    term1=term1+srca[ti]*source(xsrc,ysrc,(ti)*tstep)

term2=LA.norm(rec0)**2

term1,term2,term1-term2,term1/term2
```

# Least square objective Gradient

We will consider here the least square objective, as this is the one in need of an adjoint. The test that will follow are however necessary for any objective and associated gradient in a optimization framework. The objective function can be written

$ min_m \Phi(m) := \frac{1}{2} \| P_r A^{-1}(m) q - d\|_2^2$

And it's gradient becomes

$ \nabla_m \Phi(m) = - (\frac{dA(m)u}{dm})^T v $

where v is the soltuion if the adjoint wave equation. For the simple acoustic case the  gradient can be rewritten as

$ \nabla_m \Phi(m) = - \sum_{t=1}^{nt} \frac{d^2u(t)}{dt^2} v(t) $

```python
# Misfit

F0=.5*LA.norm(rec0-rect)**2
F0
```

```python
Im1=Gradient(nt,nx,ny,m0,rec0-rect,u0)
```

```python

shotrec = plt.imshow(rect,vmin=-1,vmax=1)   # this is how you'd plot a single line...

shotrec = plt.imshow(rec0,vmin=-1,vmax=1)   # this is how you'd plot a single line...

shotrec = plt.imshow(rec0-rect,vmin=-.1,vmax=.1)   # this is how you'd plot a single line...

shotrec = plt.imshow(Im1,vmin=-1,vmax=1)   # this is how you'd plot a single line...
#plt.show()
```

# Adjoint test for the gradient

The adjoint of the FWI Gradient is the Born modelling operator, implementing a double propagation forward in time with a wavefield scaled by the model perturbation for the second propagation

 $ J dm = - A^{-1}(\frac{d A^{-1}q}{dt^2}) dm $

```python
Im2=Gradient(nt,nx,ny,m0,rec0,u0)
```

```python
du1=Born(nt,nx,ny,m0,dm)
```

```python

term11=np.dot((rec0).reshape(-1),du1.reshape(-1))
term21=np.dot(Im2.reshape(-1),dm.reshape(-1))

term11,term21,term11-term21,term11/term21
```

# Jacobian test
The last part is to check that the operators are consistent with the problem. There is then two properties to be satisfied

$ U(m + hdm) = U(m) +  \mathcal{O} (h) \\
   U(m + h dm) = U(m) + h J[m]dm + \mathcal{O} (h^2) $

which are the linearization conditions for the objective. This is a bit slow to run here but here is the way to test it.

1 - Genrate data for the true model m
2 - Define a smooth initial model $m_0$ and comput the data $d_0$ for this model
3 - You now have $U(m_0)$
4 - Define $ dm = m-m_0$ and $ h = {1,.1,.01,.001,...}$
5 - For each $h$ compute $U(m_0 + h dm)$ by generating data for $m_0 + h dm$ and compute $(J[m_0 + h dm]^T\delta |d) $
6 - Plot in Loglog the two lines of equation above

```python
H=[1,0.1,0.01,.001,0.0001,0.00001,0.000001]
(D1,u0)=Forward(nt,nx,ny,m0)
dub=Born(nt,nx,ny,m0,dm)
error1=np.zeros((7))
error2=np.zeros((7))
for i in range(0,7):
    mloc=m0+H[i]*dm
    (d,u)=Forward(nt,nx,ny,mloc)
    error1[i]  = LA.norm(d - D1,ord=1)
    error2[i]  = LA.norm(d - D1 - H[i]*dub,ord=1)
```

```python
hh=np.zeros((7))
for i in range(0,7):
    hh[i]=H[i]*H[i]
shotrec = plt.loglog(H,error1,H,H)   # this is how you'd plot a single line...
plt.show()
shotrec = plt.loglog(H,error2,H,hh)   # this is howyou'd plot a single line...
plt.show()
```

# Gradient test
The last part is to check that the operators are consistent with the problem. There is then two properties to be satisfied

$ \Phi(m + hdm) = \Phi(m) +  \mathcal{O} (h) \\
   \Phi(m + h dm) = \Phi(m) + h (J[m]^T\delta |d)dm + \mathcal{O} (h^2) $

which are the linearization conditions for the objective. This is a bit slow to run here but here is the way to test it.

1 - Genrate data for the true model m
2 - Define a smooth initial model $m_0$ and comput the data $d_0$ for this model
3 - You now have $\Phi(m_0)$
4 - Define $ dm = m-m_0$ and $ h = {1,.1,.01,.001,...}$
5 - For each $h$ compute $\Phi(m_0 + h dm)$ by generating data for $m_0 + h dm$ and compute $(J[m_0 + h dm]^T\delta |d) $
6 - Plot in Loglog the two lines of equation above

```python
(DT,uT)=Forward(nt,nx,ny,mt)
(D1,u0)=Forward(nt,nx,ny,m0)
F0=.5*LA.norm(D1-DT)**2

g=Gradient(nt,nx,ny,m0,D1-DT,u0)
G=np.dot(g.reshape(-1),dm.reshape(-1));
error21=np.zeros((7))
error22=np.zeros((7))
for i in range(0,7):
    mloc=m0+H[i]*dm
    (D,u)=Forward(nt,nx,ny,mloc)
    error21[i]  = .5*LA.norm(D-DT)**2 -F0
    error22[i]  = .5*LA.norm(D-DT)**2 -F0 - H[i]*G
```

```python
shotrec = plt.loglog(H,error21,H,H)   # this is how you'd plot a single line...
plt.show()
shotrec = plt.loglog(H,error22,H,hh)   # this is how you'd plot a single line...
plt.show()
```

```python

```


# INF-285 / ILI-285
## Desafío 1, v1.01
### SCT 2020-1

# Introducción

En el siguiente desafío estudiaremos el comportamiento de $2$ algoritmos para obtener el punto fijo $r$ de funciones $g(x)$, es decir, $r=g(r)$.
Es importante destacar que el punto fijo de una función no es lo mismo que la raíz de una función, sin embargo sí están muy relacionados.
Solo a modo de recordatorio, la raíz de una función $f(x)$ es encontrar un $\hat{x}$ tal que $f(\hat{x})=0$.

## Iteración de Punto Fijo

El algoritmo llamado Iteración de Punto Fijo (IPF o *FPI*, *Fixed Point Iteration* del inglés) se define de la siguiente forma:
\begin{align*}
  x_0 &= \text{"Initial guess''},\\
  x_{i+1} &= g(x_i), \quad i\in {1,2,3,\dots}.
\end{align*}

El cual puede o no puede converger a su punto fijo $r=g(r)$ dependiendo del comportamiento de $g(x)$ entorno al punto fijo $r$.
En el caso de que la iteración de punto fijo diverja, uno debiera buscar otra forma de encontrar el punto fijo, la otra manera se explica a continuación.

## Método de la Bisección

En el caso de que la iteración de punto fijo diverja o simplemente converja muy lento, podemos usar convenientemente el Método de la Bisección.
Para poder utilizar el Método de la Bisección, debemos adaptarlo, dado que es un algoritmo diseñado para buscar raíces de una función, no puntos fijos de una función.
La adaptación consiste en escribir convenientemente la búsqueda de un punto fijo como la búsqueda de una raíz de la siguiente forma,
\begin{equation}
  f(x) = x - g(x),
\end{equation}

donde podemos comprobar que si evaluamos la función $f(x)$ en el punto fijo de $g(x)$ obtenemos la equivalencia,
\begin{equation}
  f(r) = r - g(r)=0.
\end{equation}

Por lo tanto, ¡hemos exitosamente conectado un problema de punto fijo con un problema de búsqueda de ceros!

**De esta forma ambos métodos podrían ser útiles si necesitamos encontrar puntos fijos de funciones**.

Comentario: ¿Puede visualizar ahora el como utilizar búsqueda de puntos fijos para encontrar raíces de funciones?

# Ejercicio

```python
# Bibliotecas necesarias
import numpy as np
import matplotlib.pyplot as plt
```

Se solicita implementar una rutina ```obtener_punto_fijo``` que reciba la función $g(x)$, un intervalo $[a, b]$ y un ```n_iter```, que indica el máximo número de iteraciones que pueden utilizar los métodos de bisección y punto fijo.
Notar que los métodos deben retornar la secuencia de soluciones obtenidas hasta que se logra la convergencia, es necesario que cuando se logre el punto fijo no se retorne una secuencia de valores repetidos, si no que se trunque el vector de salida hasta donde empezó a repetirse el valor respectivo, de otra forma se estará dividiendo por $0$ en la explicación incluida más adelante.

El retorno de la rutina debe ser la mejor solución aproximada ```x_sol```, y una estructura del tipo
```[('biseccion', tasa_bisección), ('punto fijo', tasa_punto_fijo)]```, donde se reporta el algoritmo (en el orden solicitado) y la tasa de convergencia respectiva.
Por lo tanto la firma de la función debería quedar como:
```python
  def obtener_punto_fijo(g, a, b, n_iter):
    # Su algoritmo...

    resultado = [('biseccion', tasa_biseccion), ('punto fijo', tasa_punto_fijo)]
    x_sol = ...
    return x_sol, resultado
```

La idea es que su algoritmo permita retornar la solución asociada al método con mejor *tasa de convergencia*.

Para que pueda calcular la *tasa de convergencia* se pone a disposición la función ```obtener_tasa(ratio)```, que recibe un arreglo con los cocientes de la estimación numérica de los errores en cada iteración. Los cuales deben ser obtenidos de la siguiente forma:
\begin{equation}
  ratio_i = \frac{|x_{i+1} - x_i|}{|x_i - x_{i-1}|}
\end{equation}

```python
def obtener_tasa(ratio):
  hist, bin_edges = np.histogram(ratio, bins=10000)
  k = np.argmax(hist)
  return np.round((bin_edges[k] + bin_edges[k+1]) / 2, 5)
```

Además, para que pueda probar el funcionamiento de su procedimiento, se ponen a disposición las siguientes funciones y los intevalos donde debe buscar el punto fijo:

```python
g1 = lambda x: np.cos(x) # Intervalo: [0, 1]
g2 = lambda x: 3 / (x-2) # Intervalo: [-3, 0]
g3 = lambda x: (x + 10.) ** (1 / 4) # Intervalo: [0, 2]
g4 = lambda x: 3 + 2 * np.sin(x) # Intervalo: [-5, 5]
g5 = lambda x: np.cos(x) / np.exp(x) # Intervalo: [0, 4]
g6 = lambda x: (np.exp(x) + x ** 3 + 4 * x ** 2 + 2 * x + 2) / (x ** 2 + 3 * x - 3) # Intervalo: [-1, 0]
g7 = lambda x: np.exp((np.exp(-x) / 3)) # Intervalo: [0, 2]
g8 = lambda x: -0.5 * x + 3 / 2 # Intervalo: [0, 1]
g9 = lambda x: (x ** 3 - 5) / 2 # Intervalo: [2, 3]
g10 = lambda x: -1 + 1.5 * x # Intervalo: [0,10]
g11 = lambda x: 0.7 + 1.7 * x # Intervalo: [-10,10]
```

Se incluye a continuación el enunciado de la función que usted debe entregar:

```python
def ratio(arreglo):
    x = arreglo[1:len(arreglo)-1]
    x_i_minus = arreglo[0:(len(arreglo)-2)]
    x_i_plus  = arreglo[2:]
    valores = abs(x_i_plus - x)/abs(x-x_i_minus)
    return valores

def bisection(f, a, b, n_iter):
  # CONSIDERE el caso que f(a) * f(b) sea positivo
  # RECUERDE no incluir valores repetidos al final de la secuencia del arreglo de salida para no tener errores igual a 0
    funcioncita = lambda x : x - f(x)
    if funcioncita(a)*funcioncita(b) > 0:
        return
    i = 0
    iteraciones = []
    while i < n_iter:
        c = (b+a)/2
        if funcioncita(c) == 0:
            iteraciones.append(c)
            ratios = ratio(np.array(iteraciones))
            return (c,ratios)
        else:
            if funcioncita(a)*funcioncita(c) < 0:
                b = c
            else:
                a = c
        if iteraciones == []:
            iteraciones.append(c)
        elif iteraciones[-1] != c:
            iteraciones.append(c)
        i+=1
    ratios = ratio(np.array(iteraciones))
    return (c,ratios)

```

```python
def fpi(g, x_0, n_iter):
  # CONSIDERE que el metodo puede no converger
  # RECUERDE no incluir valores repetidos al final de la secuencia del arreglo de salida para no tener errores igual a 0
    x =  x_0 # Calcular los valores de x para cada iteracion
    iteraciones=[]
    errores = []
    i = 0
    while i < n_iter:
        try:
            x_calc = g(x)
        except OverflowError:
            return None
        if x_calc == x and (x_calc not in iteraciones):
            iteraciones.append(x_calc)
        elif x_calc == x and (x_calc in iteraciones):
            i = n_iter
        else:
            iteraciones.append(x_calc)
        x = x_calc
        i+=1
    ratios = ratio(np.array(iteraciones))

    return (x_calc,ratios)

```

```python
def obtener_punto_fijo(g, a, b, n_iter):
    data_fpi = fpi(g,(a+b/2),100)
    data_bi = bisection(g,a,b,100)
    tasa_biseccion = obtener_tasa(data_bi[1]) # Debe obtenerla
    if data_fpi == None:
        tasa_punto_fijo = float("inf")
    else:
        tasa_punto_fijo = obtener_tasa(data_fpi[1]) # Debe obtenerla
    resultado = [('biseccion', tasa_biseccion), ('punto fijo', tasa_punto_fijo)]
    if tasa_biseccion < tasa_punto_fijo:
        x_sol = data_bi[0]
    elif tasa_biseccion > tasa_punto_fijo:
        x_sol = data_fpi[0]
    return x_sol, resultado

print(obtener_punto_fijo(g1,0,1,100))
print(obtener_punto_fijo(g2,-3,0,100))
print(obtener_punto_fijo(g3,0,2,100))
print(obtener_punto_fijo(g4,-5,5,100))
print(obtener_punto_fijo(g5,0,4,100))
print(obtener_punto_fijo(g6,-1,0,100))
print(obtener_punto_fijo(g7,0,2,100))
print(obtener_punto_fijo(g8,0,1,100))
print(obtener_punto_fijo(g9,2,3,100))
print(obtener_punto_fijo(g10,0,10,100))
print(obtener_punto_fijo(g11,-10,10,100))

```

    (0.7390851332151607, [('biseccion', 0.50005), ('punto fijo', 0.67361)])
    (-1.0, [('biseccion', 0.49999), ('punto fijo', 0.33333)])
    (1.8555845286409378, [('biseccion', 0.50005), ('punto fijo', 0.03913)])
    (3.0943834130492776, [('biseccion', 0.50005), ('punto fijo', 0.99999)])
    (0.5177573636824583, [('biseccion', 0.50005), ('punto fijo', 0.81266)])
    (-0.579158906050837, [('biseccion', 0.50005), ('punto fijo', 2e-05)])
    (1.1154480172165406, [('biseccion', 0.50005), ('punto fijo', 0.12187)])
    (1.0, [('biseccion', 0.50002), ('punto fijo', 0.49999)])
    (2.094551481542327, [('biseccion', 0.50002), ('punto fijo', inf)])
    (2.0, [('biseccion', 0.50001), ('punto fijo', 1.5)])
    (-1.0, [('biseccion', 0.50001), ('punto fijo', 1.7)])

```python

```


# Population coding (Pouget et al., 2010)

A response of a cell can be characterized by an "encoding model" of the stimulus ($s$):

\begin{align}
r_{i} = f_{i}(s) + n_{i}
\end{align}

in which $n$ represents a noise term assumed to follow a normal distribution with a variance proportional to the mean value, $f_{i}(s)$. When the model ("tuning function") is assumed to be gaussian (note to self: is this only for "circular properties" like orientation/motion direction?), it can be written as:

\begin{align}
f_{i}(s) = ke^{-(s - s_{i})^{2}/2\sigma^{2}}
\end{align}

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

def gaussian_model(s, si=100, k=1, sigma=5):
    """ Computes average activity of cell in response to stimulus `s`
    given the preferred value `si` and scaling faction `k` and width `sigma`.
    """
    return k * np.exp(-(s - si) ** 2 / (2 * sigma ** 2))

vals = gaussian_model(np.arange(200))
plt.figure(figsize=(8, 3))
plt.plot(vals)
plt.xlabel('Stimulus/feature value %s' % '$s_{i}$')
plt.ylabel('Activity (a.u.)')
sns.despine()
plt.show()
```

Now, suppose we have 64 cells that fire according to differently tuned tuning functions:

```python
true_si = np.linspace(-180, 180, 64)
true_activity = gaussian_model(s=40, si=true_si, sigma=20) + np.random.normal(0, 0.1, size=true_si.size)
plt.scatter(true_si, true_activity)
plt.title('Activity of 64 neurons')
sns.despine()
plt.show()
```

```python
np.exp(5)
```

    148.4131591025766


## Mixture Density Network

Mixture Density Networks (MDNs) [1] are mixture models in which the parameters such as means, covariances and mixture proportions are learned by a neural network. MDNs combine structured data representation (a density mixture) with unstructured parameter inference (MLP neural network). Mixture Density Network learn the mixture parameters by maximizing the log-likelihood or equivalently minimizing a negative log-likelihood loss.

Assuming a Gaussian Mixture Model (GMM) with $K$ components, we can write down the probability of a test data point $y_i$ conditioned on training data $x$ as follows:

\begin{equation}
    p(y_i|x) = \sum_{k=1}^{K}\pi_k(x) N\big(y_i| \mu_k(x), \Sigma_k(x)\big)
\end{equation}

where the parameters $\mu_k, \sigma_k, \pi_k$ are learned by a neural network (e.g. a Multi-Layer Perceptron (MLP)) parameterixed by $\theta$:

\begin{equation}
\mu_k, \sigma_k, \pi_k = \mathrm{NN}(x; \theta)
\end{equation}

As a result, the Neural Network (NN) is a multi-output model, subject to the following constraints on the ouput:

\begin{eqnarray}
    \forall k \sigma_{k}(x) &>& 0\\
    \sum_{k=1}^{K} \pi_k(x) &=& 1
\end{eqnarray}

The first constraint can be achieved by using exponential activations, while the second constrained can be achieved by using softmax activations. Finally, by making use of the iid assumption, we want to minimize the following loss function:

\begin{eqnarray}
    \min_{\theta} L(\theta) &=& \mathrm{NLLLoss}(\theta) = - \log \prod_{i=1}^{n} p(y_i|x) = -\sum_{i=1}^{n} \log p(y_i|x) \\
    &=& - \sum_{i=1}^{n}\log \bigg[\sum_{k=1}^{K} \pi_k(x_i, \theta) N(y_i|\mu_k(x_i,\theta), \Sigma_{k}(x_i,\theta)) \bigg]
\end{eqnarray}

In the example below, we assume an isotropic covariance $\Sigma_k = \sigma_k^{2} I$, thus we can write a $d$-dimensional Gaussian as a product:

\begin{equation}
    N(y_i | \mu_k, \Sigma_k) = \frac{1}{(2\pi)^{d/2}|\Sigma_k|^{1/2}} \exp \bigg(-\frac{1}{2}(y_i-\mu_k)^{T}\Sigma_{k}^{-1}(y_i-\mu_k) \bigg)
\end{equation}

\begin{equation}
     N(y_i | \mu_k, \Sigma_k) = \frac{1}{(2\pi \sigma_{k}^{2})^{d/2}} \exp \bigg[-\frac{1}{2\sigma_{k}^{2}} \sum_{d=1}^{D}(y_{i,d} - \mu_{k,d})^2\bigg]
\end{equation}

\begin{equation}
    N(y_i | \mu_k, \Sigma_k) = \prod_{d=1}^{D} \frac{1}{\sigma_k \sqrt{2\pi}} \exp \bigg[-\frac{1}{2\sigma_{k}^{2}}(y_{i,d}-\mu_{k,d})^{2} \bigg]
\end{equation}

Let's implement a Gaussian MDN using Keras [3]!

```python
%matplotlib inline
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.cm as cm
import matplotlib.pyplot as plt

import math
import tensorflow as tf

import keras
from keras import optimizers
from keras import backend as K
from keras import regularizers
from keras.models import Sequential, Model
from keras.layers import concatenate, Input
from keras.layers import Dense, Activation, Dropout, Flatten
from keras.layers import BatchNormalization

from keras.utils import np_utils
from keras.utils import plot_model
from keras.models import load_model

from keras.callbacks import ModelCheckpoint
from keras.callbacks import TensorBoard
from keras.callbacks import LearningRateScheduler
from keras.callbacks import EarlyStopping

from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score
from sklearn.metrics import normalized_mutual_info_score
from sklearn.model_selection import train_test_split

np.random.seed(0)
sns.set_style('whitegrid')
```

    Using TensorFlow backend.

    Couldn't import dot_parser, loading of dot files will not be possible.

We'll use a synthetic dataset of K=4, 2-D Gaussians for ease of visualization.

```python
def generate_data(N):
    pi = np.array([0.2, 0.4, 0.3, 0.1])
    mu = [[2,2], [-2,2], [-2,-2], [2,-2]]
    std = [[0.5,0.5], [1.0,1.0], [0.5,0.5], [1.0,1.0]]
    x = np.zeros((N,2), dtype=np.float32)
    y = np.zeros((N,2), dtype=np.float32)
    z = np.zeros((N,1), dtype=np.int32)
    for n in range(N):
        k = np.argmax(np.random.multinomial(1, pi))
        x[n,:] = np.random.multivariate_normal(mu[k], np.diag(std[k]))
        y[n,:] = mu[k]
        z[n,:] = k
    #end for
    z = z.flatten()
    return x, y, z, pi, mu, std

```

Let's define a function for computing a D-dimensional Gaussian with isotropic covariance:

```python
def tf_normal(y, mu, sigma):
    y_tile = K.stack([y]*num_clusters, axis=1) #[batch_size, K, D]
    result = y_tile - mu
    sigma_tile = K.stack([sigma]*data_dim, axis=-1) #[batch_size, K, D]
    result = result * 1.0/(sigma_tile+1e-8)
    result = -K.square(result)/2.0
    oneDivSqrtTwoPI = 1.0/math.sqrt(2*math.pi)
    result = K.exp(result) * (1.0/(sigma_tile + 1e-8))*oneDivSqrtTwoPI
    result = K.prod(result, axis=-1)    #[batch_size, K] iid Gaussians
    return result

```

Finally, we'll define the negative log-likelihood loss which takes in a $d$-dimensional $y_{true}$ vector and MDN output parameters $y_{pred}$:

```python
def NLLLoss(y_true, y_pred):
    out_mu = y_pred[:,:num_clusters*data_dim]
    out_sigma = y_pred[:,num_clusters*data_dim : num_clusters*(data_dim+1)]
    out_pi = y_pred[:,num_clusters*(data_dim+1):]

    out_mu = K.reshape(out_mu, [-1, num_clusters, data_dim])

    result = tf_normal(y_true, out_mu, out_sigma)
    result = result * out_pi
    result = K.sum(result, axis=1, keepdims=True)
    result = -K.log(result + 1e-8)
    result = K.mean(result)
    return tf.maximum(result, 0)

```

Let's generate and visualize training and test data:

```python
#generate data
X_data, y_data, z_data, pi_true, mu_true, sigma_true = generate_data(1024)

data_dim = X_data.shape[1]
num_clusters = len(mu_true)
#X_data, y_data = make_blobs(n_samples=1000, centers=num_clusters, n_features=data_dim, random_state=0)
#X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, random_state=0, test_size=0.7)

num_train = 512
X_train, X_test, y_train, y_test = X_data[:num_train,:], X_data[num_train:,:], y_data[:num_train,:], y_data[num_train:,:]
z_train, z_test = z_data[:num_train], z_data[num_train:]

#visualize data
plt.figure()
plt.scatter(X_train[:,0], X_train[:,1], c=z_train, cmap=cm.bwr)
plt.title('training data')
plt.savefig('./figures/mdn_training_data.png')

```

We can now define the training parameters as well as MDN parameters and architecture:

```python
#training params
batch_size = 128
num_epochs = 128

#model parameters
hidden_size = 32
weight_decay = 1e-4

#MDN architecture
input_data = Input(shape=(data_dim,))
x = Dense(32, activation='relu')(input_data)
x = Dropout(0.2)(x)
x = BatchNormalization()(x)
x = Dense(32, activation='relu')(x)
x = Dropout(0.2)(x)
x = BatchNormalization()(x)

mu = Dense(num_clusters * data_dim, activation='linear')(x) #cluster means
sigma = Dense(num_clusters, activation=K.exp)(x)            #diagonal cov
pi = Dense(num_clusters, activation='softmax')(x)           #mixture proportions
out = concatenate([mu, sigma, pi], axis=-1)

model = Model(input_data, out)
adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(loss=NLLLoss, optimizer=adam, metrics=['accuracy'])
model.summary()

```

    ____________________________________________________________________________________________________
    Layer (type)                     Output Shape          Param #     Connected to
    ====================================================================================================
    input_1 (InputLayer)             (None, 2)             0
    ____________________________________________________________________________________________________
    dense_1 (Dense)                  (None, 32)            96          input_1[0][0]
    ____________________________________________________________________________________________________
    dropout_1 (Dropout)              (None, 32)            0           dense_1[0][0]
    ____________________________________________________________________________________________________
    batch_normalization_1 (BatchNorm (None, 32)            128         dropout_1[0][0]
    ____________________________________________________________________________________________________
    dense_2 (Dense)                  (None, 32)            1056        batch_normalization_1[0][0]
    ____________________________________________________________________________________________________
    dropout_2 (Dropout)              (None, 32)            0           dense_2[0][0]
    ____________________________________________________________________________________________________
    batch_normalization_2 (BatchNorm (None, 32)            128         dropout_2[0][0]
    ____________________________________________________________________________________________________
    dense_3 (Dense)                  (None, 8)             264         batch_normalization_2[0][0]
    ____________________________________________________________________________________________________
    dense_4 (Dense)                  (None, 4)             132         batch_normalization_2[0][0]
    ____________________________________________________________________________________________________
    dense_5 (Dense)                  (None, 4)             132         batch_normalization_2[0][0]
    ____________________________________________________________________________________________________
    concatenate_1 (Concatenate)      (None, 16)            0           dense_3[0][0]
                                                                       dense_4[0][0]
                                                                       dense_5[0][0]
    ====================================================================================================
    Total params: 1,936
    Trainable params: 1,808
    Non-trainable params: 128
    ____________________________________________________________________________________________________

Let's train the model with early stopping:

```python
early_stopping = EarlyStopping(monitor='val_loss', min_delta=0.1, patience=32, verbose=1)
callbacks_list = [early_stopping]

#model training
hist = model.fit(X_train, y_train, batch_size=batch_size, epochs=num_epochs, callbacks=callbacks_list, validation_split=0.2, shuffle=True, verbose=2)
```

    Train on 409 samples, validate on 103 samples
    Epoch 1/128
    1s - loss: 5.9070 - acc: 0.0342 - val_loss: 6.0739 - val_acc: 0.0000e+00
    Epoch 2/128
    0s - loss: 5.5468 - acc: 0.0244 - val_loss: 5.9862 - val_acc: 0.0000e+00
    Epoch 3/128
    0s - loss: 5.5059 - acc: 0.0293 - val_loss: 5.8783 - val_acc: 0.0000e+00
    Epoch 4/128
    0s - loss: 5.2641 - acc: 0.0269 - val_loss: 5.7378 - val_acc: 0.0000e+00
    Epoch 5/128
    0s - loss: 5.0044 - acc: 0.0220 - val_loss: 5.5889 - val_acc: 0.0097
    Epoch 6/128
    0s - loss: 4.7956 - acc: 0.0196 - val_loss: 5.4150 - val_acc: 0.0194
    Epoch 7/128
    0s - loss: 4.7717 - acc: 0.0293 - val_loss: 5.2195 - val_acc: 0.0194
    Epoch 8/128
    0s - loss: 4.5822 - acc: 0.0293 - val_loss: 5.0472 - val_acc: 0.0291
    Epoch 9/128
    0s - loss: 4.4746 - acc: 0.0293 - val_loss: 4.9046 - val_acc: 0.0291
    Epoch 10/128
    0s - loss: 4.3982 - acc: 0.0513 - val_loss: 4.7775 - val_acc: 0.0388
    Epoch 11/128
    0s - loss: 4.3942 - acc: 0.0440 - val_loss: 4.6663 - val_acc: 0.0388
    Epoch 12/128
    0s - loss: 4.1395 - acc: 0.0611 - val_loss: 4.5731 - val_acc: 0.0388
    Epoch 13/128
    0s - loss: 4.0839 - acc: 0.0416 - val_loss: 4.4729 - val_acc: 0.0485
    Epoch 14/128
    0s - loss: 4.2564 - acc: 0.0367 - val_loss: 4.3763 - val_acc: 0.0583
    Epoch 15/128
    0s - loss: 4.0929 - acc: 0.0660 - val_loss: 4.2881 - val_acc: 0.0583
    Epoch 16/128
    0s - loss: 3.9556 - acc: 0.0440 - val_loss: 4.2074 - val_acc: 0.0583
    Epoch 17/128
    0s - loss: 4.0760 - acc: 0.0342 - val_loss: 4.1286 - val_acc: 0.0583
    Epoch 18/128
    0s - loss: 3.9408 - acc: 0.0367 - val_loss: 4.0593 - val_acc: 0.0777
    Epoch 19/128
    0s - loss: 3.8878 - acc: 0.0587 - val_loss: 3.9933 - val_acc: 0.0777
    Epoch 20/128
    0s - loss: 3.8631 - acc: 0.0489 - val_loss: 3.9211 - val_acc: 0.0777
    Epoch 21/128
    0s - loss: 3.7755 - acc: 0.0587 - val_loss: 3.8560 - val_acc: 0.1068
    Epoch 22/128
    0s - loss: 3.6820 - acc: 0.0611 - val_loss: 3.7960 - val_acc: 0.1068
    Epoch 23/128
    0s - loss: 3.6805 - acc: 0.0636 - val_loss: 3.7322 - val_acc: 0.1068
    Epoch 24/128
    0s - loss: 3.6072 - acc: 0.0562 - val_loss: 3.6619 - val_acc: 0.1165
    Epoch 25/128
    0s - loss: 3.5914 - acc: 0.0611 - val_loss: 3.5933 - val_acc: 0.1165
    Epoch 26/128
    0s - loss: 3.5474 - acc: 0.0660 - val_loss: 3.5266 - val_acc: 0.1165
    Epoch 27/128
    0s - loss: 3.6166 - acc: 0.0636 - val_loss: 3.4607 - val_acc: 0.1165
    Epoch 28/128
    0s - loss: 3.5039 - acc: 0.0709 - val_loss: 3.4030 - val_acc: 0.1165
    Epoch 29/128
    0s - loss: 3.4022 - acc: 0.0562 - val_loss: 3.3472 - val_acc: 0.1165
    Epoch 30/128
    0s - loss: 3.3929 - acc: 0.0685 - val_loss: 3.2874 - val_acc: 0.1165
    Epoch 31/128
    0s - loss: 3.2822 - acc: 0.0758 - val_loss: 3.2226 - val_acc: 0.1165
    Epoch 32/128
    0s - loss: 3.4340 - acc: 0.0391 - val_loss: 3.1615 - val_acc: 0.1165
    Epoch 33/128
    0s - loss: 3.2833 - acc: 0.0660 - val_loss: 3.0998 - val_acc: 0.1165
    Epoch 34/128
    0s - loss: 3.3057 - acc: 0.0611 - val_loss: 3.0442 - val_acc: 0.1165
    Epoch 35/128
    0s - loss: 3.2296 - acc: 0.0660 - val_loss: 2.9851 - val_acc: 0.1165
    Epoch 36/128
    0s - loss: 3.1976 - acc: 0.0709 - val_loss: 2.9262 - val_acc: 0.1165
    Epoch 37/128
    0s - loss: 3.3379 - acc: 0.0416 - val_loss: 2.8677 - val_acc: 0.1165
    Epoch 38/128
    0s - loss: 3.1349 - acc: 0.0587 - val_loss: 2.8123 - val_acc: 0.1165
    Epoch 39/128
    0s - loss: 3.2889 - acc: 0.0465 - val_loss: 2.7493 - val_acc: 0.1165
    Epoch 40/128
    0s - loss: 3.1267 - acc: 0.0440 - val_loss: 2.6958 - val_acc: 0.1165
    Epoch 41/128
    0s - loss: 3.1133 - acc: 0.0391 - val_loss: 2.6460 - val_acc: 0.1165
    Epoch 42/128
    0s - loss: 3.0921 - acc: 0.0807 - val_loss: 2.5883 - val_acc: 0.1165
    Epoch 43/128
    0s - loss: 2.9775 - acc: 0.0489 - val_loss: 2.5306 - val_acc: 0.1165
    Epoch 44/128
    0s - loss: 3.0426 - acc: 0.0562 - val_loss: 2.4894 - val_acc: 0.1165
    Epoch 45/128
    0s - loss: 2.9756 - acc: 0.0636 - val_loss: 2.4565 - val_acc: 0.1165
    Epoch 46/128
    0s - loss: 2.9206 - acc: 0.0538 - val_loss: 2.4081 - val_acc: 0.1165
    Epoch 47/128
    0s - loss: 2.8931 - acc: 0.0513 - val_loss: 2.3660 - val_acc: 0.1165
    Epoch 48/128
    0s - loss: 2.8887 - acc: 0.0587 - val_loss: 2.3295 - val_acc: 0.1165
    Epoch 49/128
    0s - loss: 2.7834 - acc: 0.0562 - val_loss: 2.2978 - val_acc: 0.1165
    Epoch 50/128
    0s - loss: 2.7840 - acc: 0.0489 - val_loss: 2.2597 - val_acc: 0.1165
    Epoch 51/128
    0s - loss: 2.7646 - acc: 0.0440 - val_loss: 2.2191 - val_acc: 0.0971
    Epoch 52/128
    0s - loss: 2.7916 - acc: 0.0391 - val_loss: 2.1742 - val_acc: 0.0874
    Epoch 53/128
    0s - loss: 2.6324 - acc: 0.0538 - val_loss: 2.1436 - val_acc: 0.0680
    Epoch 54/128
    0s - loss: 2.6104 - acc: 0.0709 - val_loss: 2.1323 - val_acc: 0.0583
    Epoch 55/128
    0s - loss: 2.5474 - acc: 0.0465 - val_loss: 2.0941 - val_acc: 0.0485
    Epoch 56/128
    0s - loss: 2.6980 - acc: 0.0416 - val_loss: 2.0505 - val_acc: 0.0291
    Epoch 57/128
    0s - loss: 2.5632 - acc: 0.0538 - val_loss: 2.0112 - val_acc: 0.0194
    Epoch 58/128
    0s - loss: 2.4390 - acc: 0.0342 - val_loss: 1.9608 - val_acc: 0.0000e+00
    Epoch 59/128
    0s - loss: 2.6240 - acc: 0.0489 - val_loss: 1.9134 - val_acc: 0.0000e+00
    Epoch 60/128
    0s - loss: 2.6062 - acc: 0.0465 - val_loss: 1.8807 - val_acc: 0.0000e+00
    Epoch 61/128
    0s - loss: 2.4623 - acc: 0.0318 - val_loss: 1.8648 - val_acc: 0.0000e+00
    Epoch 62/128
    0s - loss: 2.4274 - acc: 0.0293 - val_loss: 1.8435 - val_acc: 0.0000e+00
    Epoch 63/128
    0s - loss: 2.3820 - acc: 0.0416 - val_loss: 1.8149 - val_acc: 0.0000e+00
    Epoch 64/128
    0s - loss: 2.5219 - acc: 0.0293 - val_loss: 1.7766 - val_acc: 0.0000e+00
    Epoch 65/128
    0s - loss: 2.4784 - acc: 0.0318 - val_loss: 1.7594 - val_acc: 0.0000e+00
    Epoch 66/128
    0s - loss: 2.3320 - acc: 0.0562 - val_loss: 1.7520 - val_acc: 0.0000e+00
    Epoch 67/128
    0s - loss: 2.2655 - acc: 0.0269 - val_loss: 1.6908 - val_acc: 0.0000e+00
    Epoch 68/128
    0s - loss: 2.2435 - acc: 0.0367 - val_loss: 1.6086 - val_acc: 0.0000e+00
    Epoch 69/128
    0s - loss: 2.3730 - acc: 0.0367 - val_loss: 1.5474 - val_acc: 0.0000e+00
    Epoch 70/128
    0s - loss: 2.1912 - acc: 0.0293 - val_loss: 1.5112 - val_acc: 0.0000e+00
    Epoch 71/128
    0s - loss: 2.2566 - acc: 0.0538 - val_loss: 1.4697 - val_acc: 0.0000e+00
    Epoch 72/128
    0s - loss: 2.0497 - acc: 0.0293 - val_loss: 1.4552 - val_acc: 0.0000e+00
    Epoch 73/128
    0s - loss: 2.2115 - acc: 0.0367 - val_loss: 1.4480 - val_acc: 0.0000e+00
    Epoch 74/128
    0s - loss: 1.9978 - acc: 0.0244 - val_loss: 1.4215 - val_acc: 0.0000e+00
    Epoch 75/128
    0s - loss: 1.9611 - acc: 0.0220 - val_loss: 1.3279 - val_acc: 0.0000e+00
    Epoch 76/128
    0s - loss: 1.9493 - acc: 0.0244 - val_loss: 1.2173 - val_acc: 0.0000e+00
    Epoch 77/128
    0s - loss: 2.0401 - acc: 0.0220 - val_loss: 1.1278 - val_acc: 0.0000e+00
    Epoch 78/128
    0s - loss: 1.9477 - acc: 0.0293 - val_loss: 1.0995 - val_acc: 0.0000e+00
    Epoch 79/128
    0s - loss: 1.8232 - acc: 0.0342 - val_loss: 1.0629 - val_acc: 0.0000e+00
    Epoch 80/128
    0s - loss: 1.8441 - acc: 0.0342 - val_loss: 1.0179 - val_acc: 0.0000e+00
    Epoch 81/128
    0s - loss: 1.7945 - acc: 0.0196 - val_loss: 1.0220 - val_acc: 0.0000e+00
    Epoch 82/128
    0s - loss: 1.7765 - acc: 0.0196 - val_loss: 1.0123 - val_acc: 0.0000e+00
    Epoch 83/128
    0s - loss: 1.7779 - acc: 0.0196 - val_loss: 0.9859 - val_acc: 0.0000e+00
    Epoch 84/128
    0s - loss: 1.8818 - acc: 0.0220 - val_loss: 0.9265 - val_acc: 0.0000e+00
    Epoch 85/128
    0s - loss: 1.7221 - acc: 0.0269 - val_loss: 0.9135 - val_acc: 0.0000e+00
    Epoch 86/128
    0s - loss: 1.6196 - acc: 0.0244 - val_loss: 0.8858 - val_acc: 0.0000e+00
    Epoch 87/128
    0s - loss: 1.7099 - acc: 0.0196 - val_loss: 0.9017 - val_acc: 0.0194
    Epoch 88/128
    0s - loss: 1.7219 - acc: 0.0220 - val_loss: 0.9613 - val_acc: 0.0194
    Epoch 89/128
    0s - loss: 1.5476 - acc: 0.0293 - val_loss: 0.9461 - val_acc: 0.0097
    Epoch 90/128
    0s - loss: 1.5125 - acc: 0.0220 - val_loss: 0.8579 - val_acc: 0.0000e+00
    Epoch 91/128
    0s - loss: 1.4719 - acc: 0.0293 - val_loss: 0.7834 - val_acc: 0.0097
    Epoch 92/128
    0s - loss: 1.4103 - acc: 0.0147 - val_loss: 0.7143 - val_acc: 0.0097
    Epoch 93/128
    0s - loss: 1.6006 - acc: 0.0171 - val_loss: 0.7103 - val_acc: 0.0097
    Epoch 94/128
    0s - loss: 1.2985 - acc: 0.0122 - val_loss: 0.7105 - val_acc: 0.0097
    Epoch 95/128
    0s - loss: 1.3594 - acc: 0.0244 - val_loss: 0.6380 - val_acc: 0.0097
    Epoch 96/128
    0s - loss: 1.3300 - acc: 0.0122 - val_loss: 0.5237 - val_acc: 0.0000e+00
    Epoch 97/128
    0s - loss: 1.1956 - acc: 0.0147 - val_loss: 0.4397 - val_acc: 0.0000e+00
    Epoch 98/128
    0s - loss: 1.1126 - acc: 0.0049 - val_loss: 0.3710 - val_acc: 0.0000e+00
    Epoch 99/128
    0s - loss: 1.1429 - acc: 0.0244 - val_loss: 0.3235 - val_acc: 0.0000e+00
    Epoch 100/128
    0s - loss: 1.3316 - acc: 0.0244 - val_loss: 0.3533 - val_acc: 0.0000e+00
    Epoch 101/128
    0s - loss: 1.2698 - acc: 0.0171 - val_loss: 0.3225 - val_acc: 0.0000e+00
    Epoch 102/128
    0s - loss: 1.0162 - acc: 0.0318 - val_loss: 0.2567 - val_acc: 0.0000e+00
    Epoch 103/128
    0s - loss: 1.0019 - acc: 0.0244 - val_loss: 0.1909 - val_acc: 0.0000e+00
    Epoch 104/128
    0s - loss: 1.1213 - acc: 0.0073 - val_loss: 0.1690 - val_acc: 0.0000e+00
    Epoch 105/128
    0s - loss: 1.2172 - acc: 0.0220 - val_loss: 0.1825 - val_acc: 0.0000e+00
    Epoch 106/128
    0s - loss: 0.9417 - acc: 0.0293 - val_loss: 0.2106 - val_acc: 0.0000e+00
    Epoch 107/128
    0s - loss: 1.1023 - acc: 0.0098 - val_loss: 0.2005 - val_acc: 0.0097
    Epoch 108/128
    0s - loss: 0.9184 - acc: 0.0269 - val_loss: 0.1490 - val_acc: 0.0097
    Epoch 109/128
    0s - loss: 0.9715 - acc: 0.0293 - val_loss: 0.0899 - val_acc: 0.0097
    Epoch 110/128
    0s - loss: 0.8439 - acc: 0.0220 - val_loss: 0.0525 - val_acc: 0.0097
    Epoch 111/128
    0s - loss: 0.7072 - acc: 0.0147 - val_loss: 0.0605 - val_acc: 0.0000e+00
    Epoch 112/128
    0s - loss: 0.6803 - acc: 0.0098 - val_loss: 0.0146 - val_acc: 0.0000e+00
    Epoch 113/128
    0s - loss: 0.7736 - acc: 0.0171 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 114/128
    0s - loss: 0.5061 - acc: 0.0073 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 115/128
    0s - loss: 0.7100 - acc: 0.0122 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 116/128
    0s - loss: 0.6968 - acc: 0.0196 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 117/128
    0s - loss: 0.4926 - acc: 0.0269 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 118/128
    0s - loss: 0.5244 - acc: 0.0269 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 119/128
    0s - loss: 0.5382 - acc: 0.0196 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 120/128
    0s - loss: 0.6631 - acc: 0.0171 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 121/128
    0s - loss: 0.5660 - acc: 0.0171 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 122/128
    0s - loss: 0.6036 - acc: 0.0269 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 123/128
    0s - loss: 0.3026 - acc: 0.0220 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 124/128
    0s - loss: 0.3879 - acc: 0.0244 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 125/128
    0s - loss: 0.4804 - acc: 0.0318 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 126/128
    0s - loss: 0.4000 - acc: 0.0293 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 127/128
    0s - loss: 0.3020 - acc: 0.0342 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
    Epoch 128/128
    0s - loss: 0.3045 - acc: 0.0171 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00

We can now make predictions on test data:

```python
print "predicting on test data..."
y_pred = model.predict(X_test)

mu_pred = y_pred[:,:num_clusters*data_dim]
mu_pred = np.reshape(mu_pred, [-1, num_clusters, data_dim])
sigma_pred = y_pred[:,num_clusters*data_dim : num_clusters*(data_dim+1)]
pi_pred = y_pred[:,num_clusters*(data_dim+1):]
z_pred = np.argmax(pi_pred, axis=-1)
```

    predicting on test data...

We'll evaluate clustering performance by computing the adjusted rand score, normalized mutual information and by comparing true and predicted values for means and standard deviations:

```python
rand_score = adjusted_rand_score(z_test, z_pred)
print "adjusted rand score: ", rand_score

nmi_score = normalized_mutual_info_score(z_test, z_pred)
print "normalized MI score: ", nmi_score
```

    adjusted rand score:  0.842059401234
    normalized MI score:  0.840175516041

```python
mu_pred_list = []
sigma_pred_list = []
for label in np.unique(z_pred):
    z_idx = np.where(z_pred == label)[0]
    mu_pred_lbl = np.mean(mu_pred[z_idx,label,:], axis=0)
    mu_pred_list.append(mu_pred_lbl)

    sigma_pred_lbl = np.mean(sigma_pred[z_idx,label], axis=0)
    sigma_pred_list.append(sigma_pred_lbl)
#end for

print "true means: "
print np.array(mu_true)

print "predicted means: "
print np.array(mu_pred_list)

print "true sigmas: "
print np.array(sigma_true)

print "predicted sigmas: "
print np.array(sigma_pred_list)
```

    true means:
    [[ 2  2]
     [-2  2]
     [-2 -2]
     [ 2 -2]]
    predicted means:
    [[ 2.13656688 -1.97987711]
     [-1.47537231 -2.03052163]
     [ 1.95446241  2.02965617]
     [-1.944049    1.96065593]]
    true sigmas:
    [[ 0.5  0.5]
     [ 1.   1. ]
     [ 0.5  0.5]
     [ 1.   1. ]]
    predicted sigmas:
    [ 1.26810288  0.30546659  0.61887544  0.15010625]

Notice the clustering scores are quite high, the predicted means are also close to the true means, however, the predicted sigmas are slightly off, possibly due to a difference between training and test data.

```python
#generate plots
plt.figure()
plt.scatter(X_test[:,0], X_test[:,1], c=z_pred, cmap=cm.bwr)
plt.scatter(np.array(mu_pred_list)[:,0], np.array(mu_pred_list)[:,1], s=100, marker='x', lw=4.0, color='k')
plt.title('test data')
plt.savefig('./figures/mdn_test_data.png')

```

The black crosses in the figure above show the predicted means overlayed with test data. You can see visually, the means are close to cluster centers.

```python
plt.figure()
plt.plot(hist.history['loss'], c='b', lw=2.0, label='train')
plt.plot(hist.history['val_loss'], c='r', lw=2.0, label='val')
plt.title('Mixture Density Network')
plt.xlabel('Epochs')
plt.ylabel('Negative Log Likelihood Loss')
plt.legend(loc='upper left')
plt.savefig('./figures/mdn_loss.png')

```

We can see that both training and validation loss are decreasing with the number of epochs.

### References

[1] C. Bishop, "Pattern Recognition and Machine Learning", 2006
[2] EdwardLib, "Mixture Density Networks", http://edwardlib.org/tutorials/mixture-density-network
[3] F. Chollet, "Keras: The Python Deep Learning library", https://keras.io/


# COURSE: Master math by coding in Python
## SECTION: Linear algebra

#### https://www.udemy.com/course/math-with-python/?couponCode=MXC-DISC4ALL
#### INSTRUCTOR: sincxpress.com

Note about this code: Each video in this section of the course corresponds to a section of code below. Please note that this code roughly matches the code shown in the live recording, but is not exactly the same -- the variable names, order of lines, and parameters may be slightly different.

```python

```

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
from IPython.display import display,Math
```

# VIDEO: Vectors

```python
row_vec = np.array([4,2])
col_vec = np.array([ [-2], [3] ])
```

```python
display(Math(sym.latex(sym.sympify(row_vec))))
display(Math(sym.latex(sym.sympify(col_vec))))
```

```python
# check out the sizes
print(np.shape(row_vec))
print(np.shape(col_vec))
```

```python
plt.plot([0,row_vec[0]],[0,row_vec[1]],'r',label='row vector')
plt.plot([0,col_vec[0]],[0,col_vec[1]],'b',label='column vector')

plt.axis('square')
plt.axis([-5,5,-5,5])
plt.grid()
plt.show()
```

### Exercises

```python
# import 3d plotting tools
from mpl_toolkits.mplot3d import Axes3D

v3_1 = np.array([3,0,-4])
v3_2 = np.array([-1,1,3])

fig = plt.figure(figsize=plt.figaspect(1))
ax = fig.add_subplot(111, projection='3d')

# draw vectors
ax.plot([0,v3_1[0]],[0,v3_1[1]],[0,v3_1[2]],'b',linewidth=3)
ax.plot([0,v3_2[0]],[0,v3_2[1]],[0,v3_2[2]],'r',linewidth=3)

# guidelines
ax.plot([-5,5],[0,0],[0,0],'--',color=[.7,.7,.7])
ax.plot([0,0],[-5,5],[0,0],'--',color=[.7,.7,.7])
ax.plot([0,0],[0,0],[-5,5],'--',color=[.7,.7,.7])

ax.set_xlim3d(-5,5)
ax.set_ylim3d(-5,5)
ax.set_zlim3d(-5,5)
plt.show()
```

# VIDEO: Adding and scalar-multiplying vectors

```python
v = np.array([.5,1]) # vector
s = [1,-.5,2,.5] # scalars

for si in s:
    sv = si*v
    plt.plot([0,sv[0]],[0,sv[1]],'o-',linewidth=3,label='$\\lambda=%g$' %(si))

plt.axis('square')
plt.legend()
plt.axis([-3,3,-3,3])
plt.grid()
plt.show()
```

```python
# adding vectors
v1 = np.array([-1,2])
v2 = np.array([1,1])

v3a = v1+v2
v3b = np.add(v1,v2)

v3c = np.zeros(2)
for i in range(0,2):
    v3c[i] = v1[i] + v2[i]

print(v3a,v3b,v3c)
```

```python
plt.plot([0,v1[0]],[0,v1[1]],label='$v_1$',linewidth=3)
plt.plot([0,v2[0]],[0,v2[1]],label='$v_2$',linewidth=3)
plt.plot([0,v3[0]],[0,v3a[1]],label='$v_1+v_2$',linewidth=3)

plt.axis('square')
plt.legend()
plt.axis([-2,2,-2,4])
plt.grid()
plt.show()
```

### Exercise

```python
plt.plot([0,v1[0]],[0,v1[1]],label='$v_1$',linewidth=3)
plt.plot([0,v2[0]]+v1[0],[0,v2[1]]+v1[1],label='$v_2$',linewidth=3)
plt.plot([0,v3a[0]],[0,v3a[1]],label='$v_1+v_2$',linewidth=3)

str1 = sym.latex(sym.sympify(v1))
str2 = sym.latex(sym.sympify(v2))
str3 = sym.latex(sym.sympify(v3a))

display(Math('%s+%s=%s' %(str1,str2,str3)))

plt.axis('square')
plt.legend()
plt.axis([-2,2,-2,4])
plt.grid()
plt.show()
```

# VIDEO: The dot product

```python
v = np.arange(10,20,3)
w = np.arange(5,14,3)
print(w)
dp1 = 0

for i in range(0,len(v)):
    dp1 = dp1 + v[i]*w[i]

```

```python
dp2 = np.sum( np.multiply(v,w) )
dp3 = np.dot(v,w)
print(dp,dp2,dp3)
```

### Exercise

```python
# write a function that takes two vectors, tests whether the dot product is valid between them, and returns the dot product or gives an error message

def mydotproduct(v,w):

    # squeeze out singleton dimensions
    v = np.squeeze(v)
    w = np.squeeze(w)

    # check whether they are the same size
    if len(v)!=len(w):
        raise ValueError('Vectors must have the same length!')

    display(Math('v^Tw = %s' %np.dot(v,w)))

```

```python
v1 = np.random.randn(1,10)
v2 = np.random.randn(10,1)

# np.shape(np.squeeze(v2))

mydotproduct(v1,v2)
```

# VIDEO: Dot product application: Correlation coefficient

```python
# data
mikes_courses  = [4,5,8,2,9,9,1,3,7]
life_happiness = [6,7,9,3,9,3,1,6,7]

# mean-center
m = mikes_courses - np.mean(mikes_courses)
l = life_happiness - np.mean(life_happiness)

# compute correlation
numer = np.dot(m,l)
denom = np.sqrt( np.dot(m,m) ) * np.sqrt(np.dot(l,l))
r1 = numer/denom

# confirm with numpy function
r2 = np.corrcoef(m,l)[1][0]

# show the data!
plt.plot(mikes_courses,life_happiness,'ms',label='r=%s' %np.round(r1,2))

plt.axis([0,10,0,10])
plt.gca().set_aspect('equal')
plt.xlabel('Number of Mike\'s courses taken')
plt.ylabel('Life happiness')
plt.legend()
plt.show()

```

# VIDEO: The outer product

```python
v1 = np.random.randn(50)
v2 = np.random.randn(80)

np.dot(v1,v2)
```

```python
op = np.outer(v1,v2)
# display(Math(sym.latex(sym.sympify(op))))

plt.imshow(op)
plt.show()
```

### Exercise

```python
v = np.arange(1,10)
w = np.arange(1,5)

print(np.outer(v,w))
print(np.outer(w,v))
```

```python
# show that svw=vsw=vws
s = 4

left = s*np.outer(v,w)
middle = np.outer(v*s,w)
right = np.outer(v,w)*s

print(left-middle)
print(middle-right)
print(left-right)
```

# VIDEO: Matrix multiplication

```python
A = np.random.randn(14,10)
B = np.random.randn(14,10)
C = np.random.randn(10,14)

# print(A@B)
print(A@C)
print(C@A)
```

### Exercise

```python
def matrixMultiplication(A,B):

    # get matrix sizes
    szA = np.shape(A)
    szB = np.shape(B)

    # give error if sizes don't match
    if szA[1]!=szB[0]:
        raise ValueError('"Inner" dimensions don''t match!')

    # initialize
    C = np.zeros((szA[0],szB[1]))

    # compute!
    for i in range(szA[0]):
        for j in range(szB[1]):
            C[i,j] = np.dot(A[i,:],B[:,j])

    return C

```

```python
A = np.random.randn(4,6)
B = np.random.randn(5,2)

C1 = matrixMultiplication(A,B)
C2 = A@B

print(C1-C2)
```

# VIDEO: Transposing matrices and vectors

```python
r = np.random.randn(1,10)
c = np.random.randn(10,1)

print(np.shape(r))
print(np.shape(c))
```

```python
r_flip1 = np.transpose(r)
r_flip2 = r.T
print(np.shape(r_flip2))
```

```python
# matrix and image
mat = np.random.randn(8,4)
matT = mat.T

fig,ax = plt.subplots(1,2)
ax[0].imshow(mat)
ax[0].set_title('M')

ax[1].imshow(matT)
ax[1].set_title('M$^T$')

# set common properties
for i in ax:
    i.set_yticks([])
    i.set_xticks([])

plt.show()
```

### Exercise

```python
# show that a matrix times its transpose is square-symmetric

# sizes
m = 10
n = 14

amat = np.random.randn(m,n)

# palindromic variables! (literal and referent!)
amatama = amat@amat.T

# show it's square
print(np.shape(amatama))

# show it's symmetric
amatama - amatama.T

np.is
```

# VIDEO: Various special matrices

```python
# identity
I = np.eye(4)
print(I)
```

```python
# zeros
Z = np.zeros((5,5))
print(Z)
```

```python
# diagonal
D = np.diag([1,3,2,4,5])
print(D)
d = np.diagonal(D)
print(d)
```

```python
# triangular matrix
T = np.random.randn(15,15)

plt.imshow(np.triu(T))
plt.show()
```

### Exercise

```python
# matrix of random integers
intmat = np.random.randint(-4,5,(4,4))
I4 = np.eye(4)
upper4 = np.triu(np.random.randn(4,4))

print(intmat.dtype)
print('\n\n')
print((intmat@I4).dtype)
print('\n\n')
print(intmat@upper4)
```

```python
# figure out what a Toeplitz matrix is
from scipy.linalg import toeplitz

v = np.arange(1,6)
toe = toeplitz(v)

print(toe)

plt.imshow(toe)
plt.show()
```

# VIDEO: Matrix inverse

```python
# random matrices are nearly always invertible

m = 5

A = np.random.randint(-5,6,(m,m))
Ainv = np.linalg.inv(A)

fig,ax = plt.subplots(1,3)

ax[0].imshow(A)
ax[0].set_title('A')

ax[1].imshow(Ainv)
ax[1].set_title('A$^{-1}$')

ax[2].imshow(A@Ainv)
ax[2].set_title('AA$^{-1}$')

for i in ax:
    i.axis('off')

plt.show()
```

```python
# also no inverse for square matrix with repeated columns

A = np.random.randint(-5,6,(m,m))
A[:,0] = A[:,1]
print(A)

Ainv = np.linalg.inv(A)

```

### Exercise

```python
# pinv with repeated column in square matrix, and with non-square matrix
m = 5
A = np.random.randint(-5,6,(m,m))
A[:,0] = A[:,1]
print(A)

# A = np.random.randn(m,m+2)

# compute pseudo-inverse
Ainv = np.linalg.pinv(A)

fig,ax = plt.subplots(1,3)

ax[0].imshow(A)
ax[0].set_title('A')

ax[1].imshow(Ainv)
ax[1].set_title('A$^{*}$')

ax[2].imshow(A@Ainv)
ax[2].set_title('AA$^{*}$')

for i in ax:
    i.axis('off')

plt.show()
```

# VIDEO: Solving a system of equations

```python
X = np.array([ [2,1,-1],[3,4,2],[1,-5,-2] ])
y = np.array([1,13,0])

b = np.linalg.inv(X.T@X) @ (X.T@y)
b
```

```python
b = np.linalg.solve(X,y)
b
```

### Exercise

```python
# visualize the three possible outcomes of simultaneous equations

# 1) one solution
X1 = np.array([ [3,-1],[-1,1] ])
y1 = np.array([6,2])

# 2) infinite solutions
X2 = np.array([ [3,-1],[1.5,-.5] ])
y2 = np.array([6,3])

# 3) no solutions
X3 = np.array([ [3,-1],[1.5,-.5] ])
y3 = np.array([6,2])
```

```python
### generate the plot
xlim = np.array([0,10])

# 1) Case 1: one unique solution

# convert equations into y=mx+b form
yy1 = -X1[0,0]/X1[0,1]*xlim + y1[0]/X1[0,1]
yy2 = -X1[1,0]/X1[1,1]*xlim + y1[1]/X1[1,1]

# plot
plt.plot(xlim,yy1,'b',label='eq 1')
plt.plot(xlim,yy2,'r',label='eq 2')

# solve and plot
b1 = np.linalg.solve(X1,y1)
plt.plot(b1[0],b1[1],'ko',markersize=10,label='Solution')

plt.legend()
plt.grid()
plt.show()
```

```python
# Case 2: infinite solutions

# convert equations into y=mx+b form
yy1 = -X2[0,0]/X2[0,1]*xlim + y2[0]/X2[0,1]
yy2 = -X2[1,0]/X2[1,1]*xlim + y2[1]/X2[1,1]

# plot
plt.plot(xlim,yy1,'b',label='eq 1')
plt.plot(xlim,yy2,'r--',label='eq 2')

# solve and plot
b1 = np.linalg.solve(X2,y2)
plt.plot(b1[0],b1[1],'ko',markersize=10,label='Solution')

plt.legend()
plt.grid()
plt.show()
```

```python
# Case 3: no solutions

# convert equations into y=mx+b form
yy1 = -X3[0,0]/X3[0,1]*xlim + y3[0]/X3[0,1]
yy2 = -X3[1,0]/X3[1,1]*xlim + y3[1]/X3[1,1]

# plot
plt.plot(xlim,yy1,'b',label='eq 1')
plt.plot(xlim,yy2,'r--',label='eq 2')

# solve and plot
b1 = np.linalg.solve(X3,y3)
plt.plot(b1[0],b1[1],'ko',markersize=10,label='Solution')

plt.legend()
plt.grid()
plt.show()
```

# VIDEO: Visualizing matrix-vector multiplication

```python
M = np.array([ [.5,1],
               [1,.5] ])

v = np.array([1,2])
v = np.random.randn(2)
# v = np.array([1,1])
# v = np.array([1,-1])

Mv = M@v

plt.plot([0,v[0]],[0,v[1]],'b',label='v',linewidth=2)
plt.plot([0,Mv[0]],[0,Mv[1]],'r',label='Mv',linewidth=2)

plt.axis('square')
plt.axis([-5,5,-5,5])
plt.legend()
plt.grid()

plt.show()
```

### Exercise

```python
v1 = np.linspace(-2,2,40)

for i in v1:

    v = np.array([1,i])
    Mv = M@v
    plt.plot([0,v[0]],[0,v[1]],color=[1-abs(i)/4,.5,abs(i)/2],alpha=.8)
    plt.plot([0,Mv[0]],[0,Mv[1]],color=[1-abs(i)/2,abs(i)/4,.5])

plt.axis('square')
plt.axis('off')
plt.show()
```

# VIDEO: Eigenvalues and eigenvectors

```python
# the matrix
M = np.random.randint(-5,5,(5,5))
M = M@M.T

# its eigendecomposition
eigvals,eigvecs = np.linalg.eig(M)
print(eigvals)

# visualize the matrices
fig,ax = plt.subplots(1,3)

ax[0].imshow(M)
ax[0].set_title('The matrix')

ax[1].imshow(eigvecs)
ax[1].set_title('Eigenvectors')

ax[2].imshow(np.diag(eigvals))
ax[2].set_title('Eigenvalues')

for i in ax:
    i.axis('off')

plt.show()
```

```python
# but what do they mean??

Mv = M@eigvecs[:,0]
lv = eigvals[0] * eigvecs[:,0]

print(Mv)
print(lv)
```

    [-29.12339181  51.06094009  41.53707056  60.71971609   7.96930309]
    [-29.12339181  51.06094009  41.53707056  60.71971609   7.96930309]

### Exercise

```python
# part 1: 2x2 symmetric matrix. plot Mv and lv

M = np.array([ [1,2],[2,1] ])
d,V = np.linalg.eig(M)

lv = d[0]*V[:,0] # change to 1 without changing d
Mv = M@V[:,0]

plt.plot([0,lv[0]],[0,lv[1]],'b',label='v',linewidth=4)
plt.plot([0,Mv[0]],[0,Mv[1]],'r:',label='Mv',linewidth=3)

plt.axis('square')
plt.axis([-3,3,-3,3])
plt.legend()
plt.grid()

plt.show()
```

```python
# part 2: 6x6 symmetric matrix. compute all pairwise dot products of eigenvectors

# the matrix
m = 6
M = np.random.randn(m,m)
M = M@M.T

# its eigendecomposition
d,V = np.linalg.eig(M)

dps = np.zeros((m,m))

for i in range(0,m):
    for j in range(0,m):
        dps[i,j] = np.dot(V[:,i],V[:,j])

print(np.round(dps,2))
```

    [[1 0 0 0 0 0]
     [0 1 0 0 0 0]
     [0 0 0 0 0 0]
     [0 0 0 1 0 0]
     [0 0 0 0 1 0]
     [0 0 0 0 0 0]]

# VIDEO: Singular value decomposition

```python
# SVD of Einstein
from PIL import Image
ein = Image.open('einstein.jpg')

plt.imshow(ein)
plt.show()

# SVD is defined on 2D matrices
print( np.shape(ein) )
ein = np.mean(ein,2)
print( np.shape(ein) )
```

```python
# Take the SVD
U,s,V = np.linalg.svd(ein)

# image the results
fig,ax = plt.subplots(1,3)

ax[0].imshow(U)
ax[0].set_title('U')

ax[1].imshow(np.diag(np.log(s)))
ax[1].set_title('$\\Sigma$')

ax[2].imshow(V)
ax[2].set_title('V$^T$')
plt.show()
```

```python
# plot the eigenspectrum

plt.plot(s,'ks-',markerfacecolor='w')
plt.xlim([-1,50])
plt.xlabel('Component number')
plt.ylabel('Singular value')
plt.title('Eigenspectrum of Einstein')
plt.show()
```

### Exercise

```python
# reconstruct ein, then shuffle the singular values

# reconstruct S from s
S = np.zeros(np.shape(ein))
for i in range(0,len(s)):
    S[i,i] = s[i]

rein = U@S@V

plt.subplot(1,2,1)
plt.imshow(ein)
plt.axis('off')
plt.title('Original')

plt.subplot(1,2,2)
plt.imshow(rein)
plt.axis('off')
plt.title('$U\\Sigma V^T$')

plt.show()
```

```python

# reconstruct S from s
S = np.zeros(np.shape(ein))

randorder = np.random.permutation(len(s))

for i in range(0,len(s)):
    S[i,i] = s[randorder[i]]

# for i in range(5,len(s)): # 0-10 and 5-end
#     S[i,i] = s[i]

rein = U@S@V

plt.subplot(1,2,1)
plt.imshow(ein)
plt.axis('off')
plt.title('Original')

plt.subplot(1,2,2)
plt.imshow(rein)
plt.axis('off')
plt.title('$U\\Sigma V^T$ (shuffled $\\sigma$)')

plt.show()
```

# Linear algebra BUG HUNT!!

```python
# create a column vector
cv = np.array([ [-2], [3] ])

display(Math(sym.latex(sym.sympify(cv))))
```

$$\left[\begin{matrix}-2\\3\end{matrix}\right]$$

```python
# visualize scalar-vector multiplication

v = np.array([-2,2])
s = .7

sv = s*v
plt.plot([0,v[0]],[0,v[1]],'ro-',linewidth=3,label='v')
plt.plot([0,sv[0]],[0,sv[1]],'o-',linewidth=3,label='sv')

plt.axis('square')
plt.legend()
plt.axis([-3,3,-3,3])
plt.grid()
plt.show()
```

```python
# algorithm to compute the dot product
v = np.random.randn(7)
w = np.random.randn(7)

dp1 = 0
for i in range(0,len(v)):
    dp1 += v[i]*w[i]

dp2 = np.dot(v,w)

print(str(dp1) + '\t' + str(dp2))
```

    4.496205785840504	4.496205785840504

```python
# number of data points
n = 10

# data
data1 = np.arange(0,n) + np.random.randn(n)
data2 = np.arange(0,n) + np.random.randn(n)

# mean-center
data1 -= np.mean(data1)
data2 -= np.mean(data2)

# compute correlation
numer = np.dot(data1,data2)
denom = np.sqrt( np.dot(data1,data1) ) * np.sqrt(np.dot(data2,data2))
r1 = numer/denom

# confirm with numpy function
r2 = np.corrcoef(data1,data2)[1][0]

print(r1)
print(r2)
```

    0.9038359551350148
    0.903835955135015

```python
# outer product computation
o1 = np.random.randint(0,10,7)
o2 = np.random.randint(0,10,4)

outermat = np.zeros((len(o1),len(o2)))

for i in range(len(o1)):
    outermat[i,:] = o1[i]*o2

print(outermat-np.outer(o1,o2))
```

    [[0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]]

```python
# matrix multiplication
A = np.random.randn(5,5)
I = np.eye(5)

print(A)
print(' ')
print(A@I)
```

    [[ 1.41116507  0.11118908 -0.84036896  0.11313107  1.70270048]
     [-0.03428775 -1.04824032  1.01029019  0.05180005 -0.96271664]
     [ 0.09002964 -0.03697255 -0.76111096  0.13235124  0.6355435 ]
     [-0.99132593  0.08314975 -0.06567858  0.64614303 -0.57610782]
     [ 0.20196848 -2.21321209  2.28529331  0.59870048 -0.65780559]]

    [[ 1.41116507  0.11118908 -0.84036896  0.11313107  1.70270048]
     [-0.03428775 -1.04824032  1.01029019  0.05180005 -0.96271664]
     [ 0.09002964 -0.03697255 -0.76111096  0.13235124  0.6355435 ]
     [-0.99132593  0.08314975 -0.06567858  0.64614303 -0.57610782]
     [ 0.20196848 -2.21321209  2.28529331  0.59870048 -0.65780559]]

```python
# matrix multiplication
A = np.random.randn(8,5)
I = np.eye(8)

I@A
```

    array([[-0.10474819,  0.9019924 ,  0.91857271, -0.34820598, -1.85856762],
           [-1.24367987,  0.27385517,  0.97527506, -0.33152848, -1.27207881],
           [-0.04609427,  1.18712626, -0.5844609 ,  0.41414651,  0.67234351],
           [ 0.03110833, -1.29788706, -0.5543878 ,  1.09507752, -1.28611552],
           [-0.41642328,  0.13876337,  1.58639187, -0.00974641, -1.21843364],
           [-1.99781285,  1.1557947 , -0.77074796, -0.34969327,  0.01284731],
           [-1.02710798,  0.75087018, -0.03063462, -0.48408204, -0.55336977],
           [ 1.10811051, -0.89907204,  0.74964954,  0.21866847,  0.50128612]])

```python
# random matrices are invertible
A = np.random.randint(-5,6,(5,5))
Ainv = np.linalg.inv(A)

np.round(A@Ainv,4)
```

    array([[ 1.,  0., -0., -0., -0.],
           [ 0.,  1.,  0.,  0.,  0.],
           [ 0.,  0.,  1., -0.,  0.],
           [ 0., -0.,  0.,  1.,  0.],
           [ 0., -0.,  0.,  0.,  1.]])

```python
# plot the eigenspectrum
# the matrix
M = np.random.randint(-5,5,(5,5))
M = M@M.T

# its eigendecomposition
eigvals,eigvecs = np.linalg.eig(M)

plt.plot(np.matrix.flatten(eigvals),'s-')
plt.xlabel('Components')
plt.ylabel('Eigenvalues')

plt.show()
```

```python
# Reconstruct a matrix based on its SVD
A = np.random.randint(-10,11,(10,20))

U,s,V = np.linalg.svd(A)

# reconstruct S
# S = np.diag(s)
S = np.zeros(np.shape(A))
for i in range(len(s)):
    S[i,i] = s[i]

Arecon = U@S@V

fig,ax = plt.subplots(1,3)

ax[0].imshow(A,vmin=-10,vmax=10)
ax[0].set_title('A')

ax[1].imshow(Arecon,vmin=-10,vmax=10)
ax[1].set_title('Arecon')

ax[2].imshow(A-Arecon,vmin=-10,vmax=10)
ax[2].set_title('A-Arecon')

plt.show()
```


# Solving the Shortest Path Problem with GUROBI
# 1. Import Libraries
The first step is importing the libraries.

```python
import csv
import networkx as nx
import matplotlib.pyplot as plt
from gurobipy import *  # GRB_INIT
```

# 2. Given Data
## 2.1 Getting Link List and Link Costs

```python
links = tuplelist()
cost = {}
with open("links_shortest_path.csv", 'r') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)
    for row in csv_reader:
        if csv_reader.line_num > 1:
            from_node = int(row[0])
            end_node = int(row[1])
            cost[from_node, end_node] = float(row[2])
            links.append((from_node, end_node))
print(cost)
```

    {(0, 11): 30.0, (0, 12): 10.0, (0, 13): 20.0, (11, 12): 5.0, (11, 14): 20.0, (11, 15): 30.0, (12, 13): 50.0, (12, 14): 30.0, (12, 15): 30.0, (13, 14): 30.0, (13, 15): 20.0, (14, 20): 30.0, (15, 20): 30.0}

## 2.2 Nodes

```python
nodes = [x[0] for x in links] + [x[1] for x in links]
nodes = set(nodes)
nodes = list(nodes)
source_node = 0
destination_node = 20

```

# 3. Creating Model Object

```python
m = Model()  # GRB_INIT
```

# 3. Defining Decision Variables

```python
decision_l = m.addVars(links, vtype=GRB.BINARY)

```

# 4. Objective Function
\begin{equation}
\label{eq:obj}
\sum\limits_{(u,v)\in\mathcal{E}} l_{(u,v)} * cost_{(u,v)}
\end{equation}

```python
m.setObjective(quicksum(decision_l[u, v] * cost[u, v] for u, v in links), GRB.MINIMIZE)
```

# 5. Constraints
\begin{equation}
\label{eq:routing}
\sum\limits_{v\in\mathcal{V}} l_{(u,v)} - \sum\limits_{v\in\mathcal{V}} l_{(v,u)}  =
\begin{cases}
1, & \text{if } u = source\_node \\
-1,& \text{if } u = destination\_node \\
0, & \text{otherwise}
\end{cases}
\\ \forall u\in\mathcal{V}
\end{equation}

```python
for u in nodes:
    m.addConstr(quicksum(decision_l[u, v] for u, v in links.select(u, '*'))
                - quicksum(decision_l[v, u] for v, u in links.select('*', u)) ==
                (1 if u == source_node else -1 if u == destination_node else 0))
```

# 6. Running Solver

```python
m.optimize()
```

    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 7 rows, 13 columns and 26 nonzeros
    Model fingerprint: 0xf2e9cbcf
    Variable types: 0 continuous, 13 integer (13 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [5e+00, 5e+01]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 1e+00]
    Found heuristic solution: objective 80.0000000
    Presolve removed 7 rows and 13 columns
    Presolve time: 0.10s
    Presolve: All rows and columns removed

    Explored 0 nodes (0 simplex iterations) in 0.56 seconds
    Thread count was 1 (of 8 available processors)

    Solution count 2: 70

    Optimal solution found (tolerance 1.00e-04)
    Best objective 7.000000000000e+01, best bound 7.000000000000e+01, gap 0.0000%

# 7. Selected Links

```python
for u, v in links:
    if decision_l[u, v].x == 1:
        print("u:{} v:{} l:{}".format(u,v,decision_l[u, v].x * cost[u, v]))
```

    u:0 v:12 l:10.0
    u:12 v:15 l:30.0
    u:15 v:20 l:30.0

# 8. Plot Results

```python
G = nx.DiGraph()
G.add_nodes_from(nodes)
for u, v in links:
    G.add_edge(u, v)

# Adding the position attribute to each node
node_pos = {0: (0, 4),
            11: (2, 2), 12: (2, 4), 13: (2, 6), 14: (4, 3), 15: (4, 6),
            20: (6, 4)}

# Create a list of edges in shortest path
red_edges = [(i, j) for i, j in links if decision_l[i, j].x > 0]

# Create a list of nodes in shortest path
sp = [i for i, j in links if decision_l[i, j].x > 0]
sp.append(destination_node)

# If the node is in the shortest path, set it to red, else set it to white color
node_col = ['white' if not node in sp else 'red' for node in G.nodes()]
# If the edge is in the shortest path set it to red, else set it to white color
edge_col = ['black' if not edge in red_edges else 'red' for edge in G.edges()]
# Draw the nodes
nx.draw_networkx(G, node_pos, node_color=node_col, node_size=450)
# Draw the node labels
# nx.draw_networkx_labels(G1, node_pos,node_color= node_col)
# Draw the edges
nx.draw_networkx_edges(G, node_pos,edge_color=edge_col)
# Draw the edge labels
nx.draw_networkx_edge_labels(G, node_pos, edge_color=edge_col, edge_labels=cost)
# Remove the axis
plt.axis('off')

# Show the plot
plt.show()

```

```python

```

```python

```

```python

```

```python

```

```python

```


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import ipywidgets
```

```python
N = 1024
```

```python
T = np.logspace(1, 8, N)
```

```python
T_eV = T / 11605.
log_T_eV = np.log(T_eV)
```

```python
k1 = np.exp(-32.71396786375
          + 13.53655609057*log_T_eV
          - 5.739328757388*log_T_eV**2
          + 1.563154982022*log_T_eV**3
          - 0.2877056004391*log_T_eV**4
          + 0.03482559773736999*log_T_eV**5
          - 0.00263197617559*log_T_eV**6
          + 0.0001119543953861*log_T_eV**7
          - 2.039149852002e-6*log_T_eV**8)
k2 = 4.881357e-6*T**(-1.5)* (1.+1.14813e2 * T**(-0.407))**(-2.242)
```

```python
def sampling_rate(N):
    plt.loglog(T, k1, label='k1')
    plt.loglog(T[::N], k1[::N], label='k1s')
    plt.loglog(T, k2, label='k2')
    plt.loglog(T[::N], k2[::N], label='k2s')
    plt.ylim(1e-19, 1e-7)
    plt.legend(loc="lower left")
```

```python
sampling_rate(16)
```

$dS/dt$

So for atomic hydrogen, we have destruction from:

* H + de
* H + H
* H + He

For atomic hydrogen, we have formation from:

* H+ + de

```python
import sympy
```

```python
HI, HII, HeI, HeII, HeIII, de = sympy.sympify(
"HI, HII, HeI, HeII, HeIII, de")
k1, k2, k3, k4, k5, k6, k7, k8 = sympy.sympify(
"k1, k2, k3, k4, k5, k6, k7, k8")
```

```python
r1 = (HI + de), (HII + de + de), k1
r2 = (HII + de), (HI), k2
r3 = (HeI + de), (HeII + de + de), k3
r4 = (HeII + de), (HeI), k4
r5 = (HeII + de), (HeIII + de + de), k5
r6 = (HeIII + de), (HeII), k6
r7 = (HI + HI), (HII + de + HI), k7
r8 = (HI + HeI), (HII + de + HeI), k8
```

```python
all_reactions = [r1, r2, r3, r4, r5, r6, r7, r8]
all_reactions = [r1, r2]
```

```python
def find_formation(species):
    rxns = []
    for r in all_reactions:
        if species in r[1].atoms():
            rxns.append(r)
    return rxns

def find_destruction(species):
    rxns = []
    for r in all_reactions:
        if species in r[0].atoms():
            rxns.append(r)
    return rxns
```

```python
find_formation(HI)
```

    [(HII + de, HI, k2)]

```python
find_destruction(HI)
```

    [(HI + de, HII + 2*de, k1)]

```python

```

```python
def get_rhs(species):
    dSdt = 0
    for lhs, rhs, coeff in find_formation(species):
        term = coeff
        for atom in list(lhs.atoms()):
            term *= atom
        dSdt += term
    for lhs, rhs, coeff in find_destruction(species):
        term = -coeff
        for atom in list(lhs.atoms()):
            term *= atom
        dSdt += term
    return dSdt
```

```python
get_rhs(HI)
```

    -HI*de*k1 + HII*de*k2

```python
get_rhs(HII)
```

    HI*de*k1 - HII*de*k2

```python
sympy.solve(get_rhs(HI))
```

    [{HI: HII*k2/k1}, {de: 0}]

```python
plt.loglog(T, k1/k2, "-")
plt.ylim(1e-7, 10.0)
```

```python

```


# Linear 2D solution

## Init symbols for *sympy*

```python
from sympy import *
from geom_util import *
from sympy.vector import CoordSys3D
import matplotlib.pyplot as plt
import sys
sys.path.append("../")

%matplotlib inline

%reload_ext autoreload
%autoreload 2
%aimport geom_util
```

```python
# Any tweaks that normally go in .matplotlibrc, etc., should explicitly go here
%config InlineBackend.figure_format='retina'
plt.rcParams['figure.figsize'] = (12, 12)

plt.rc('text', usetex=True)

plt.rc('font', family='serif')

init_printing()
```

```python
N = CoordSys3D('N')
alpha1, alpha2, alpha3 = symbols("alpha_1 alpha_2 alpha_3", real = True, positive=True)
A,K,rho = symbols("A K rho")
```

```python
B=Matrix([[0, 1/(A*(K*alpha3 + 1)), 0, 0, 0, 0, 0, 0, K/(K*alpha3 + 1), 0, 0, 0], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1/(A*(K*alpha3 + 1)), 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0], [-K/(K*alpha3 + 1), 0, 0, 0, 0, 0, 0, 0, 0, 1/(A*(K*alpha3 + 1)), 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]])
B
```

```python
E=zeros(6,9)
E[0,0]=1
E[1,4]=1
E[2,8]=1
E[3,1]=1
E[3,3]=1
E[4,2]=1
E[4,6]=1
E[5,5]=1
E[5,7]=1
E
```

```python
mu = Symbol('mu')
la = Symbol('lambda')
C_tensor = getIsotropicStiffnessTensor(mu, la)
C = convertStiffnessTensorToMatrix(C_tensor)
C
```

```python
S=B.T*E.T*C*E*B*A*(1+alpha3*K)
S
```

```python
M=Matrix([[rho, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, rho, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, rho, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
M=M*A*(1+alpha3*K)
M
```

## Cartesian coordinates

```python
import fem.geometry as g
import fem.model as m
import fem.material as mat
import fem.solver as s
import fem.mesh as me
import plot

stiffness_matrix_func = lambdify([A, K, mu, la, alpha3], S, "numpy")
mass_matrix_func = lambdify([A, K, rho, alpha3], M, "numpy")

def stiffness_matrix(material, geometry, x1, x2, x3):
    A,K = geometry.get_A_and_K(x1,x2,x3)
    return stiffness_matrix_func(A, K, material.mu(), material.lam(), x3)

def mass_matrix(material, geometry, x1, x2, x3):
    A,K = geometry.get_A_and_K(x1,x2,x3)
    return mass_matrix_func(A, K, material.rho, x3)

def generate_layers(thickness, layers_count, material):
    layer_top = thickness / 2
    layer_thickness = thickness / layers_count
    layers = set()
    for i in range(layers_count):
        layer = m.Layer(layer_top - layer_thickness, layer_top, material, i)
        layers.add(layer)
        layer_top -= layer_thickness
    return layers

def solve(geometry, thickness, linear, N_width, N_height):
    layers_count = 1
    layers = generate_layers(thickness, layers_count, mat.IsotropicMaterial.steel())
    model = m.Model(geometry, layers, m.Model.FIXED_BOTTOM_LEFT_RIGHT_POINTS)
    mesh = me.Mesh.generate(width, layers, N_width, N_height, m.Model.FIXED_BOTTOM_LEFT_RIGHT_POINTS)
    lam, vec = s.solve(model, mesh, stiffness_matrix, mass_matrix)
    return lam, vec, mesh, geometry

width = 2
curvature = 0.8
thickness = 0.05

corrugation_amplitude = 0.05
corrugation_frequency = 20

geometry = g.General(width, curvature, corrugation_amplitude, corrugation_frequency)

N_width = 600
N_height = 2

lam, vec, mesh, geometry = solve(geometry, thickness, False, N_width, N_height)
results = s.convert_to_results(lam, vec, mesh, geometry)

results_index = 0

plot.plot_init_and_deformed_geometry_in_cartesian(results[results_index], 0, width, -thickness / 2, thickness / 2, 0, geometry.to_cartesian_coordinates)

to_print = 20
if (len(results) < to_print):
    to_print = len(results)

for i in range(to_print):
    print(results[i].rad_per_sec_to_Hz(results[i].freq))

```


# Introduction

Since Newton invented calculus, differentiating a function has been essential to the advancement of humanity. Calculating the derivative of a function is crucial to finding the extrema for a function and determining zeros for a function, two operations that are central to optimization (1). Often, we can find the symbolic/analytical solution to the derivative of a function, however this has become increasingly complex and computationally expensive as our functions/equations have grown in size and complexity. Numerically solving differential equations forms a cornerstone of modern science and engineering and is intimately linked with machine learning; however this method suffers from rounding errors and numerical instability. Many of these issues can be solved using Automatic Differentiation (AD) because AD can calculate the exact derivative up to machine precision (2). The logic and processes behind AD enables it to be implemented using computer code, making it easily accessible for use by scientists and mathematicians. This python package will implement the forward mode of AD.

# Background

The following mathematical concepts and background are required for understanding automatic differentiation:

### 1. Differential calculus

Differential calculus is a subfield of calculus concerned with the study of the rates at which quantities change (3).
Given the function:
\begin{align}
 f\left(x\right) &=  {x^{2}}
 \end{align}

 Increment x by h:
 \begin{align}
 f\left(x+h\right) &=  {(x+h)^{2}}
 \end{align}

 Apply the finite difference approximation to calculate the slope:
  \begin{align}
 \frac{f\left(x+h\right) - f\left(x\right) }{h}
 \end{align}

Simplify the equation:
  \begin{align}
 &= \frac{x^{2}+2xh+h^{2}-x^{2} }{h}\\
 &= \frac{2xh+h^{2}}{h}\\
 &=2x+h
 \end{align}

 Set $h = 0$:
   \begin{align}
 2x +0 &= 2x
  \end{align}

The derivative is then defined as:
\begin{align}
 \lim_{h\to0} \frac{f\left(x+h\right) - f\left(x\right) }{h}
 \end{align}

### 2. Elementary functions and their derivatives

|       Function $f(x)$                |       Derivative $f^{\prime}(x)$                |
| :-------------------:  | :------------------------------------------------------------------------------:  |
| ${c}$           | $0$         |
| ${x}$           | $1$         |
| ${x^{n}}$           | ${nx^{n-1}}$         |
| $\frac{1}{x}$     | $\frac{-1}{x^{2}}$     |
| $ln{x}$     | $\frac{1}{x}$     |
| $\sin(x)$           |   $\cos(x)$         |
| $\cos(x)$           |   $-\sin(x)$         |
| $\tan(x)$           |   $\dfrac{1}{\cos^2(x)}$         |
| $\exp(x)$           |   $\exp(x)$         |
| ${a^{x}}$           |   ${a^{x}\ln{a}}$         |

### 3. The chain rule$^{(1)}$

For a function $h(u(t))$, the derivative of $h$ with respect to $t$ can be expressed as:
$$\dfrac{\partial h}{\partial t} = \dfrac{\partial h}{\partial u}\dfrac{\partial u}{\partial t}.$$
If the function is expressed as a combination of multiple variables that are expressed in terms of t, i.e. $h(u(t), v(t))$, the the derivative of $h$ with respect to $t$ can be expressed as:
$$\frac{\partial h}{\partial t} = \frac{\partial h}{\partial u}\frac{\partial u}{\partial t} + \frac{\partial h}{\partial v}\frac{\partial v}{\partial t}$$

Note that we are only looking at scalar variables in this case, but this idea can be extended to vector variables as well.

For any $h = h\left(y\left(x\right)\right)$ where $y\in\mathbb{R}^{n}$ and $x\in\mathbb{R}^{m}$,

\begin{align}
  \nabla_{x}h = \sum_{i=1}^{n}{\frac{\partial h}{\partial y_{i}}\nabla y_{i}\left(x\right)}.
\end{align}

### 4. The graph structure of calculations and forward accumulation

Forward accumulation is computing the derivative using the chain rule starting from the inner most derivative to the outer most derivative, where we assume the most basic variables have seed values. Using a graph helps visualize forward accumulation (4). For example,

\begin{align}
 f\left(x,y\right) &= \frac{x}{y} +cos(x)sin(y)\\
 x &= y = 1
\end{align}

| Trace | Elementary Function | Current Value | Elementary Function Derivative |     $\nabla_{x}$ Value          |     $\nabla_{y}$ Value
| :---: | :-----------------: | :-----------: | :----------------------------: | :-----------------: | :-----------------: | :-----------------: |
| $w_{1}$ | $1$ | $1$ | $\dot{w_1}$ | $1$ | $0$ |
| $w_{2}$ | $1$ | $1$ | $\dot{w_2}$ | $0$ | $1$ |
| $w_{3}$ | $cos{(w_1})$ | $cos{(1)}$ | $-sin{(w_1)}\dot{w_1}$ | $-sin(1)$ | $0$ |
| $w_{4}$ | $sin{(w_2})$ | $sin{(1)}$ | $cos{(w_2)}\dot{w_2}$ | $0$ | $cos{(1)}$ |
| $w_{5}$ | $w_3\dot w_4$ | $sin{(1)}cos{(1)}$ | $w_4\dot{w_3} + w_3\dot{w_4}$ | $-sin^2{(1)}$ | $cos^2{(1)}$ |
| $w_{6}$ | $w_1 / w_2$ | $1$ | $\dot{w_1}/w_2 - w_1 \dot{w_2}/ w_2^2$ | $1$ | $-1$ |
| $w_{7}$ | $w_5 + w_6$ | $sin{(1)}cos{(1)} + 1$ | $\dot{w_5} + \dot{w_6}$ | $-sin^2{(1)} + 1$ | $cos^2{(1)}-1$ |

# How to Use *autodiff*

Ideally, a user should not be overwhelmed by our package. They would only need to import our package and instantiate the basic variables or vectors (i.e. $x_1, x_2, x_3,  \vec{x}$, etc.). Using the AD objects, the users will be able to easily call operations such as addition, subtraction, sine, exponential, etc. with both other AD objects and also normal numbers (*int* and *float*), forming the expressions that they want. The AD objects will feel perfectly integrated, and the users can get the value/derivatives of the expressions with basic get calls.

For example, in the case of scalar functions/values, the user will type something like this:

```python

import autodiff as ad

x = ad.Scalar('x', 2)
x.get_value()
>>>2
x.get_deriv()
>>>{'x': 1}
y = ad.Scalar('y', 5)
y.get_value()
>>>5
y.get_deriv()
>>>{'y': 1}
z  = x + y
z.get_value()
>>> 7
z.get_deriv()
>>> {'x': 1, 'y': 1}
z2 = x*y
z2.get_value()
>>>10
z2.get_deriv()
>>> {'x': 5 , 'y': 2 }
z3 = y ** 2
z3.get_value()
>>> 25
z3.get_deriv()
>>>{'y': 10}
z4 = ad.sin(ad.Scalar('x1',  0))
z4.get_value()
>>> 0
z4.get_deriv()
>>> {'x1': 1}
```

This idea can be extended to the cases of vectors.
```python
#do not need to follow this naming convention
x1 = ad.Scalar('x1', 10)
x2 = ad.Scalar('x2', 4)
x = ad.Vector([x1, x2, x1 + x2, x1 * x2])
x.get_values()
>>> np.array([10, 4, 14, 40])
#There can be a function to return it as matrix
x.get_derivs()
>>> [{'x1': 1, 'x2': 0}, {'x1': 0, 'x2': 1}, {'x1': 1, 'x2': 1}, {'x1': 4, 'x2': 10}]
```

# Software Organization

 * The directory structure would look like that:
 ```
 autodiff\
          autodiff\
                    __init__.py
                    variables.py (scalar and vector)
                    functions.py
                    test\
                         __init__.py
                         test_variables.py
                         test_functions.py
          README.md
          setup.py
          LICENSE

 ```

The plans on organizing our software package are:

* We are planning to use *numpy* to conduct most of our calculations, vector operations and definition of elementary functions.
* The test suite will live in the test directory shown above. We are using both `TravisCI` and `Coveralls`
* We want to release our package on `PyPI`

# Implementation

Our core data structure for implementing the forward mode of automatic differentiation will be based on the *Scalar* and *Vector* classes. To initialize a *Scalar* class object, the user will pass in a string that represents the variable (i.e. 'x', 'y', 'x1', etc.) and also the value of variable (the seed value). The *Scalar* class will hold two attributes: 1) the value of the variable `val` at the current step and 2) a dictionary `deriv` containing the derivative or partial derivatives (keys will be the names of the variables (i.e *x* and *y*) and the values will be the derivative value with respect to each variable). This allows us to easily compute derivatives with respect to a variable when we are performing operations with multiple variables since then we can update each partial derivative individually based on the variable. When a *Scalar* object is initialized, by default `deriv` will just be a dictionary with the only key being the string the user passes in with value 1. A user can access the value of a *Scalar* object using the *get_value()* method and access the derivative (or partial derivatives) for the object through the *get_derivs()* method. The dunder methods __add__, __sub__, __mul__,  __truediv__, __pow__, __iadd__, __isub__, __imul__, __itruediv__, __ipow__ (and the right equivalents for the ones that have one) will all be overwritten so that they return a new *Scalar* object with an updated value and derivatives. By overwriting these methods, we are implementing forward accumulation, as the orders of operation allows us to traverse the chainrule starting from the inside.

Another class called *Vector* will take in a list or array of *Scalar* objects. A *Vector* only has one attribute: a numpy array of *Scalar* objects, since each *Scalar* object will track its current value and derivative. The dunder methods __add__, __sub__, __mul__,  __truediv__, __pow__, __iadd__, __isub__, __imul__, __idiv__, __ipow__ (and the right equivalents) will all be overwritten so that they return a new array of *Scalar* objects with updates values and derivatives. Similar to numpy methods, the operations are conducted element-wise, i.e. In an addition operation between two *Vector* objects, the first row is added to the first row, second row is added to the second row, etc. As a result, one vector operation becomes multiple scalar operations. To access the values in the *Vector* object, the user can use the *get_values()* method, which returns a *numpy.array* of values. To access the derivatives in the *Vector* object, the user can use the *get_derivs()* method, which returns a list of dictionaries containing derivatives or partial derivatives for each *Scalar* object in the array. We can also add a function that returns this as a matrix, which is the Hessian. We can also add an optional argument to *get_derivs()* such that the user can just get the derivatives or partial derivatives with respect to the desired variables only (i.e. with respect to 'x', with respect to 'y', etc.). The user can obtain a copy of the *numpy.array* with *Scalar* objects using the *get_vector()* method, which will return a copy of the *numpy.array* to the user.

We will implement functions *sin, cos, tan, arcsin, arccos, arctan, exp* (e^x), *power* (a^x), and *abs* (absolute value). The functions will not be implemented in a specific class, similar to the implementation of *numpy* . These functions will be written such that if a *Scalar* object is passed in, then a new *Scalar* object with an updated value and derivative is returned, depending on the function being called. If a *Vector* object is passed in, then a new *Vector* object with updated values and derivatives is returned. If one of the functions is not differentiable at a given value, then we will throw an error for the user and explain that the function is not differentiable at this specific value.

The implementation of our classes and functions will rely heavily on *numpy*.

# Citations
1. Sondak, David. “Automatic Differentiation: The Basics.” CS207-Lecture9. Cambridge, MA. 2 October 2018.

2. Hoffman, Philipp H.W. “A Hitchhiker’s Guide to Automatic Differentiation.” *Numerical Algorithms*, 72, 24 October 2015, 775-811, *Springer Link*, DOI 10.1007/s11075-015-0067-6.

3. Calculus-ML Cheatsheet(2017). Retrieved October 17, 2018, from https://ml-cheatsheet.readthedocs.io/en/latest/calculus.html?fbclid=IwAR2vDAEHj1yy-4SSBTUH7Ki_D4S4uaDZJcgNXCtkVtzTrGqR-8dKmHg2L5s

4. Automatic Differentiation Or mathemagically finding derivatives. (2015, December 05). Retrieved November 16, 2018, from http://www.columbia.edu/~ahd2125/post/2015/12/5/

```python

```


Kod yazmak istediğinde ``+ Code`` tuşuna basman, metin girmek istediğinde ``+ Text`` düğmesine basman yeterli. Metin  yazarken Colab'a ait özel markdown kelime işlemcesini kullanacağız. Bu Github'ın Markdown  bazı bakımdan farklı komutlar içerecek.

Örnek olarak matematiksel formüller kollunacağız. Bir satır içeresinde matamatiksel ifade yazmak istersen, matematiksel ifadeyi iki dolar işareti arasına alman gerekir. Örnek olarak $x^2$ yazmak için yapmamız gereken \\$x^2\$ olarak yazman gerek.  Pisagor teoremi örnek olarak $a^2+b^2=c^2$ şeklinde verilir.  Eğer matematiği iki satır arasında göstermek istiyorsan,

\\$$

x^2+y^2=c^2

\$$

olarak yazabilirsin. Mesela yukarıdaki denklemi şu şekilde yazabiliriz.

$$
x^2+y^2=c^2
$$
Mesela matematiğin  en ünlü formülünü yazalım:

$$
e^{i\pi}+1=0
$$

Bu formülün niçin matematiğin en ünlü formülü olduğunu öğrenmek için [buraya](https://www.youtube.com/watch?v=IUTGFQpKaPU) bakabilirsin. Tabiki de daha güzel denklemleri de yazabilirsin. Bunun gibi:

\begin{align}
(a+b)^3 &= (a+b)^2(a+b)\\
&=(a^2+2ab+b^2)(a+b)\\
&=(a^3+2a^2b+ab^2) + (a^2b+2ab^2+b^3)\\
&=a^3+3a^2b+3ab^2+b^3
\end{align}

Matematik ve fizik formüllerini kullanırken LaTeX matematik yazım kurallarını kullanacağız. Daha fazla bilgi için [buraya](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference) bakabilirsin.

**Ödev 1** Yukarıda verilen bağlantıyı kullanırak LaTeX ile bir fizik veya matematik formülünü aşağıya yaz.

*Not:* Colab LaTeX'in bütün yazım kurallarını kapsamıyor. Onun için LaTeX'in bütün özelliklerini Colab'da kullanımıyabilirsin. Yukarıdaki bağlantıdaki bazı örnekleri yazamıyabilirsin.

1'den $n$'e kadar sayıların toplamını kısaca $\sum_{}{}$ işareti ile göstereceğim. Örnek olarak 1'den n'e kadar sayıların toplamı

$$
\sum_{i=0}^n i = \frac{n(n+1)}{2}
$$

formülü ile verilir.

Örnek olarak 1'den 100'ye kadar sayıların toplamını kısaca

$$
\sum_{i=0}^{100} i = \frac{100(100+1)}{2}
$$

verilir. Yukarıdaki toplamın cevabı 5050'dir. Yukarıdaki formülü ilkokulda Gauss'un bulduğu  söyleniyor. Bunu aşağıdaki kodu kullanarak ispatlıyalım.

```
sum = 0
for i in range(1,101):
  sum = sum +i
print (sum)
```

Şimdi de yukarıda yazdığımız kodu bir fonksiyonun içine koyalım.

```
def Gauss(n):
  sum = 0
  for i in range(1,101):
    sum = sum +i
  return sum

```

```
Gauss(100)
```

    5050

Şimdi de *Collatz Sanısı'na* (Collatz Conjecture) bakalım. Biz buna sanı diyeceğiz. Matematiksel olarak ispatlanmadı ama şimdiye kadar yanlışlanamadı da. Eğer doğru olduğunu ispatlarsan Matematikteki en büyük ödülü (Field Medal) alacağın kesindir. (Matematikte Nobel ödülü verilmiyor.)

**Collatz Sanısı:** Verilen bir $n$ ile diziye başlayın. Eğer $n$ çift ise ikiye bölün, eğer tek sayı ise bu sayıyı 3 ile çarpıp 1 ekleyin. Bu dizi 1'e ulaştığında dizi sonar erer.  Hangi $n$ ile başlarsak başlayalım, her zaman 1'e ulaşır. Bunu matematiksel ifade olarak yazarsak:

$$
f(n) =
\begin{cases}
n/2,  & \text{eğer $n$ çift ise } \\
3n+1, & \text{eğer $n$ is tek ise}
\end{cases}
$$

Yukarıdaki ifadeyi kodun içine koyalım:

```
def Collatz(n):
  while n>1:
    if n%2 == 0: # % operatörü bölümden kalan sayıyı verir. Kendin dene istersen
       n = n/2
    else:
      n= 3*n +1
    print(n, end=" ") # end' burada çıktıyı bir sıraya koymak için kullandım.

```

```
Collatz(19)
```

    58 29.0 88.0 44.0 22.0 11.0 34.0 17.0 52.0 26.0 13.0 40.0 20.0 10.0 5.0 16.0 8.0 4.0 2.0 1.0

Yukarıdaki fonksiyonu değiştirip, ulaştığı en yüksek sayıyı da bulan bir fonksiyon yazalım. Örnek olarak şöyle bir çıktı versin:

Collatz sanısı 19 için doğrulandı ve bu sınama sonunda ulaşılan en yüksek sayı 88 oldu.

```

```

```
def Collatz_max(n):
  n_init = n
  max_number = n
  while n>1:
    if n%2 == 0: # % operatörü bölümden kalan sayıyı verir. Kendin dene istersen
       n = n/2
    else:
      n= 3*n +1
    if (n > max_number):
      max_number = n
  print("Collatz sanısı " + str(n_init) + " için doğrulandı ve bu sınama sonunda ulaşılan en yüksek sayı " + str(max_number)+" oldu")

```

```
Collatz_max(19)
```

    Collatz sanısı 19 için doğrulandı ve bu sınama sonunda ulaşılan en yüksek sayı 88.0 oldu

**Ödev 2 (Çözümlü):** Matematikçi Leibniz aşağıdaki formülü vermiştir.

$$
\frac{\pi}{4}= \sum_{k=0}^{\infty}\frac{(-1)^k}{2k+1}
$$
Yukarıdaki formülün açılımı

$$
\frac{\pi}{4}= 1 - \frac{1}{3}+\frac{1}{5}-\frac{1}{7}+\frac{1}{9}-\ldots
$$
şeklinde yazabiliriz. ```Leibniz_pi(n)``` isimli bir fonksiyon yazınız ve fonksiyon ```n```'e kadar Leibniz formülünü kullansın. Sonuçta $\pi$'nin değerini ekrana bassın.

```
def Leibniz_pi(n):
  sum=0
  for i in range(0,n+1):
    sum= sum+(-1)**i/(2*i+1)
  result = sum*4
  print("Pi="+ str(result))

```

```
Leibniz_pi(1000)
```

    Pi=3.1425916543395442

**Ödev 3:**  3 ve 5'in katları

Eğer 10'dan düşük (10 dahil değil) sayılardan 3 veya 5'in katları olan sayıları dizersek, 3,5,6,9 sayılarını elde ederiz. Bu sayıların toplamı 23 dir.

1000'den düşük olan (1000 dahil değil) 3 veya 5'in katları olan sayıların toplamını yazan bir fonksiyon yazınız.

```
def multiples(n):
  # Burayı doldur.

```

**Ödev 4:** Çift Fibonacci Sayıları

Fibonaci dizisinde her yeni terim kendinden önceki iki terimin toplamından elde edilir. 1 ve 2 ile başlayarak, ilk 10 terimin Fibonacci  dizisi

$$
1,2,3,5,8,13,21,34,55,89,\ldots
$$
olur.

Yukarıdaki diziyi matematiksel olarak yazarsak

$$
F_n=
\begin{cases}
  0, & n=0\\
  1, & n=1 \\
  F_{n-1}+F_{n-2}, & n> 1
\end{cases}
$$

Fibonacci dizisinde değeri  4,000,000 (4 milyon dahil)'u geçmeyen sayıya kadar çift değerli Fibonacci terimlerinin toplamını bulun.

Yardım: Aşağıdaki ```Fibonacci_term(n)``` n'ninci Fibonacci terimini vermektedir. Bu fonksiyonu başka bir fonksiyon içinde kullanarak ```Fibonacci_evenSum``` çift sayıların toplamını yapın.

```
def Fibonacci_term(n):
  if (n == 1):
    return 1
  if (n == 2):
    return 2
  else:
    return Fibonacci_term(n-1) + Fibonacci_term(n-2)

```

```
Fibonacci_term(3)
```

    3

```
def Fibonacci_evenSum(nlast):
  # burayı doldur. nlast burada 4,000,000 olarak deneyeceksin.
```

```
Fibonacci_evenSum(10)
```

    44

```

```