# Cálculo diferencial en varias variables (II)

Unha das funcionalindades máis importantes do módulo **Sympy** é as ferramentas das que dispón para o cálculo de derivadas, xa se traten de derivadas ordinarias (para expresións que dependen dunha única variable) como de derivadas parciais (para expresións definidas a partir de varias variables). Nesta práctica, estudaremos como calcular derivadas de forma implícita, usar a regra da cadea tanto usando a matriz xacobiana ou directamente co comando `sp.diff` e a representación de planos tanxentes a un punto dunha superficie

### Obxectivos:

- Cálculo de derivadas parciais de forma implícita
- Uso da regra da cadea mediante o cálculo da matriz xacobiana
- Cálculo e representación gráfica do plano tanxente nun punto dunha superficie

Para dispoñer do módulo **Sympy** e importalo para o resto do guión de prácticas, usaremos:

```python
import sympy as sp
```

## Resolución de ecuacións en **Sympy**
Ata o de agora, aínda non precisamos calcular a solución de ecuacións (tanto lineais, polinómicas ou non lineais). Para facelo o módulo **Sympy** posúe a ferramenta `sp.solve`. Vexamos o uso desta función para resolver a ecuación $x^2+3x+a=0$ onde $x$ é a variable a calcular e $a$ é un parámetro constante asumido como coñecido:

```python
x, a = sp.symbols('x a', real=True) # define as variables simbólicas x e y
sol = sp.solve(sp.Eq(x**2+3*x+a,0),x)
display(sol)
```

    [-sqrt(9 - 4*a)/2 - 3/2, sqrt(9 - 4*a)/2 - 3/2]

A mesma función `sp.solve` tamén nos permite resolver sistemas de ecuacións (tanto lineais como non lineais). Por exemplo, se queremos resolver o sistema
$$
\left\{
\begin{array}
 33x^2+2y=1,\\
 x-y =2,
\end{array}
\right.
$$
empregaríase:

```python
x, y = sp.symbols('x y', real=True) # define as variables simbólicas x e y
sol = sp.solve((sp.Eq(3*x**2+2*y,1),sp.Eq(x-y,2)),(x,y))
display(sol)
```

    [(-5/3, -11/3), (1, -1)]

## Derivación implícita
Se queremos calcular as derivadas dunha función que depende dunha ou varias variables non temos porque coñecer a súa expresión de forma explícita, pode vir dada implicitamente. Por exemplo, a función $z=f(x,y)$ pode vir determinada pola expresión $z^2x+2yz+xy=3$. Neste caso, non se coñece de forma explícita cal é a expresión de $f$ en termos de $x$ e $y$. A pesar deste incoveniente é posible calcular  as derivadas parciais $\partial f/\partial x$ e $\partial f/\partial y$ como segue:

```python
x, y = sp.symbols('x y', real=True) # define as variables simbólicas x e y
z = sp.Function('f')(x,y)
eq = sp.Eq(z**2*x+2*y*z+x*y,3)
# Calculo da dereivada implicita con respecto a x
eq_dx = sp.Eq(sp.diff(eq.lhs, x),sp.diff(eq.rhs, x))
z_dx = sp.solve(eq_dx, sp.diff(z,x))
display(z_dx)
# Calculo da dereivada implicita con respecto a y
eq_dy = sp.Eq(sp.diff(eq.lhs, y),sp.diff(eq.rhs, y))
z_dy = sp.solve(eq_dy, sp.diff(z,y))
display(z_dy)
```

    [-(y + f(x, y)**2)/(2*x*f(x, y) + 2*y)]

    [-(x/2 + f(x, y))/(x*f(x, y) + y)]

### **Exercicio 7.1**
Calcula as derivadas parciais da función $u=f(x,y,z)$ tendo en conta que $x^2+y^2+z^2+u^2=1$.

```python
# O TEU CÓDIGO AQUÍ
```

## Matriz xacobiana e regra da cadea
A regra de derivación ordinaria para a composición de funcións viña dada pola chamda regra da cadea $(f\circ g)'(x)=f'(g(x))g'(x)$. No caso, de funcións de varias variables, formalmente tamén acudiremos á regra da cadea para cálculo multivariable. Por exemplo, se $F:\mathbb{R}^N\to \mathbb{R}$ e $G:\mathbb{R}^{M}\to\mathbb{R}^{N}$ entón, o cálculo do gradiente da composición $F\circ G$ resulta:
$$
\nabla(F\circ G)(x_{1},\ldots,x_{M})=(DG(x_{1},\ldots,x_{M}))^{t}(\nabla F)(G(x_{1},\ldots,x_{M}))
$$
onde $DG$ é a matriz xacobiana da función $G$. Vexamos como facer este cálculo nun caso particular onde
$$
F(x,y)=3x+2y^2\qquad e \qquad
G(s,t,r)=
\begin{pmatrix}
2s+t^2\\
r^2s
\end{pmatrix}.
$$
Para implementar este tipo de cálculos matriciais con **Sympy** será útil empregar a clase de obxectos `sp.Matrix` e tamén o uso do método `jacobian` para calcular a matriz xacobiana dunha expresión (ademais tamén teremos en conta que o gradiente dunha función escalar de varias variables é a matriz transposta da matriz xacobiana):

```python
x, y, r, s, t = sp.symbols('x y r s t', real=True) # define as variables simbólicas x e y
# Definición das expresións de F e G
F = sp.Matrix([3*x+2*y**2])
G = sp.Matrix([2*s+t**2, r**2*s])
# Cálculo do gradiente de F
gradient_F = F.jacobian([x,y]).transpose()
display(gradient_F)
# Cálculo da matriz xacobiana de G
DG = G.jacobian([s,t,r])
display(DG)
# Cálculo do gradiente de F(G)
DFoG = DG.transpose()*(gradient_F.subs({x:G[0], y:G[1]}))
display(DFoG)
```

$\displaystyle \left[\begin{matrix}3\\4 y\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}2 & 2 t & 0\\r^{2} & 0 & 2 r s\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}4 r^{4} s + 6\\6 t\\8 r^{3} s^{2}\end{matrix}\right]$

Para comprobar que os cálculo feitos anteriormente son correctos, tamén é posible calcular primeiro a composición de funcións e despois calcular as derivadas parciais usado a función `jacobian`. Tanto dunha forma como doutra o resultado debe ser idéntico. Nesta forma de proceder, usaremos a implementación do gradiente revisada na práctica anterior:

```python
def gradient(expr, varlist):
    return [sp.diff(expr,var) for var in varlist] # cálculo do gradiente almacenado nunha lista
```

E agora definimos dúas funcións con `sp.Lambda` e calculamos a composición para despois calcular o gradiente.
> ** Nota **: a composición de funcións debe facerse con coidado xa que a función `G` devolve unha lista e para extraer cada un dos elementos da lista e pasalos como argumento da función `F` débese empregar o operador `*`

```python
# Definición das funcións
F = sp.Lambda((x,y),3*x+2*y**2)
G = sp.Lambda((s,t,r),(2*s+t**2,r**2*s))
display('Avaliacion de G como unha lista de expresions:', G(s,t,r))
display('Avaliacion de G duas expresions independentes:', *G(s,t,r))
# Cálculo da composición
FoG = F(*G(s,t,r))
display('Composicion:', FoG)
# Cálculo do gradiente da composición
DFoG = gradient(FoG, (s,t,r))
display('Gradiente:',DFoG)
```

    'Avaliacion de G como unha lista de expresions:'

$\displaystyle \left( 2 s + t^{2}, \  r^{2} s\right)$

    'Avaliacion de G duas expresions independentes:'

$\displaystyle 2 s + t^{2}$

$\displaystyle r^{2} s$

    'Composicion:'

$\displaystyle 2 r^{4} s^{2} + 6 s + 3 t^{2}$

    'Gradiente:'

    [4*r**4*s + 6, 6*t, 8*r**3*s**2]

### **Exercicio 7.2**
Emprega a regra da cadea e a composicion de funcións para calcular o gradiente con respecto ás variables $(t,s)$ da función $w=xyz$ con $x=s+t$, $y=s-t$, e $z=st^2$.

```python
# O TEU CÓDIGO AQUÍ
```

## Recta tanxente e normal a unha curva
A interpretación xeométrica da derivada ordinaria $f'(x)$ dunha función dunha variable $f(x)$ é clara: $f(x_{0})$ é a pendente da recta tanxente á curva $y=f(x)$ no punto $(x_0,y_0)=(x_0,f(x_{0}))$. Do mesmo xeito, empregando as propiedades do gradiente, se definimos $F(x,y)=y-f(x)$, entón $\nabla F(x_0,y_0)$ é un vector perpendicular á gráfica $y=f(x)$ e polo tanto define o vector director da curva normal á gráfica nese punto.

Para a representación gráfica tanto de rectas como de planos, empregaremos **Matplotlib**

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

Vexamos estas propiedades co exemplo $f(x)=x^2+3$ considerendo o punto $x_0=1$:

```python
x, y = sp.symbols('x y', real=True) # define a variable simbólica x
f = sp.Lambda(x, x**2+3)
x0 = 1; y0=f(x0)
# Cálculo da ecuación da recta tanxente
df = sp.Lambda(x, sp.diff(f(x),x))
recta_tang = y0+df(x0)*(x-x0)
# Cálculo da ecuación da recta normal
grad_f = gradient(y-f(x),(x,y))
grad_fun = sp.Lambda((x,y),(grad_f[0],grad_f[1]))
u = grad_fun(x0,y0) # vector director da recta normal
recta_norm = sp.solve(sp.Eq(u[0]*(y-y0)-u[1]*(x-x0),0),y)[0]
```

Unha vez calculadas as rectas tanxentes e normais, trasladase a información do módulo **Sympy** a **Numpy**, para un uso posterior en **Matplotlib**:

```python
F = sp.lambdify(x,f(x),"numpy") # función numpy coa expresión de f(x,y)
Recta_tang = sp.lambdify(x,recta_tang,"numpy") # función numpy coa expresión vectorial de grad(f)
Recta_norm = sp.lambdify(x,recta_norm,"numpy") # función numpy coa expresión vectorial de grad(f)
```

Agora pódense usar as funcións de **Numpy** para a representación gráfica da función e das rectas tanxentes e normal:

```python
# Creación dos puntos onde se avalían as funcións
xvec = np.linspace(-2, 2, 100)

# Gráfica de sin(x) e cos(x), cunha etiqueta para cada unha
plt.plot(xvec, F(xvec), label='f(x)')
plt.plot(xvec, Recta_tang(xvec), label='tanxente')
plt.plot(xvec, Recta_norm(xvec), label='normal')

# Etiquetas dos eixes
plt.xlabel('x')
plt.ylabel('y')

# Engadir a lenda (mostrando as etiquetas dos "plot")
plt.legend()

# Definir os límites dos eixes x e y
plt.xlim(xvec[0], xvec[-1])
plt.ylim(2,6)
ax = plt.gca()
ax.set_aspect('equal')
plt.show()
```

### **Exercicio 7.3**
Calcula e representa graficamente as rectas tanxente e normal da gráfica da función $x-2y^2=1$ no punto $x_0=2$.

```python
# O TEU CÓDIGO AQUÍ
```

## Plano tanxente e recta normal a unha superficie
Como xa estudamos nas clases de pizarra, o gradiente dunha función $\nabla f$ é un campo de vectores que sempre é perpendicular ás curvas de nivel dunha función de dúas variables $f(x,y)$. Isto tamén é certo para funcións de tres variables $f(x,y,z)$, onde agora o gradiente $\nabla f(x,y,z)$ será perpendicular ás superficies de nivel. Isto nos permite calcular e representar graficamente tanto rectas normais como planos tanxentes a unha superficie dada.

Vexamos estas propiedades co exemplo da superficie $z=x^2+y^2+3$ considerendo o punto $(x_0,y_0)=(1,1/2)$:

```python
x, y, z, t = sp.symbols('x y z t', real=True) # define a variable simbólica x
f = sp.Lambda((x,y), x**2+y**2+3)
x0 = 1; y0=sp.Rational('1/2'); z0=f(x0,y0)
# Cálculo da ecuación do plano tanxente
grad_f = gradient(z-f(x,y),(x,y,z))
grad_fun = sp.Lambda((x,y,z),(grad_f[0],grad_f[1],grad_f[2]))
u = grad_fun(x0,y0,z0) # vector perpendicular a plano
plano_tanx = sp.solve(sp.Eq(u[0]*(x-x0)+u[1]*(y-y0)+u[2]*(z-z0),0),z)[0]
# Cálculo da ecuación vectorial da recta normal
recta_norm = sp.Lambda(t,(x0+t*u[0], y0+t*u[1], z0+t*u[2]))
```

Unha vez calculadas as rectas tanxentes e normais, trasladase a información do módulo **Sympy** a **Numpy**, para un uso posterior en **Matplotlib**:

```python
F = sp.lambdify((x,y),f(x,y),"numpy") # función numpy coa expresión de f(x,y)
Plano_tanx = sp.lambdify((x,y),plano_tanx,"numpy") # función numpy coa expresión vectorial de grad(f)
Recta_norm = sp.lambdify(t,recta_norm(t),"numpy") # función numpy coa expresión vectorial de grad(f)
```

Agora pódense usar as funcións de **Numpy** para a representación gráfica da función e das rectas tanxentes e normal:

```python
from mpl_toolkits.mplot3d import axes3d
# Nube de puntos para representar a superficie e o plano
N = 50
x = np.linspace(-2, 2, N)
y = np.linspace(-2, 2, N)
X, Y = np.meshgrid(x, y)
Fval = F(X,Y)
Pval = Plano_tanx(X,Y)

# Conxunto de puntos para representar a recta normal
tvec = np.linspace(-2, 2, 100)
Rval = Recta_norm(tvec)

# Creación da figura 3D
fig = plt.figure()
ax = plt.axes(projection="3d")
# Representación da superficie z=F(x,y)
surf_F = ax.plot_surface(X, Y, Fval)
surf_Plano = ax.plot_surface(X, Y, Pval)
recta = ax.plot3D(*Rval)
# Etiquetas dos eixes
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
# Punto de vista do observador: elevación, ángulo azimutal
ax.view_init(elev=10, azim=-100)
ax.auto_scale_xyz([-2,2],[-2,2],[-5,10])
plt.show()
```

### **Exercicio 7.4**
Calcula e representa graficamente o plano tanxente e a recta normal da gráfica da función $x^2+y^2+z^2=1$ no punto $(x_0,y_0,z_0)=(1,1,1)/\sqrt{3}$.

```python
# O TEU CÓDIGO AQUÍ
```


```
"""
Filename: mc_compute_stationary_mpmath.py

Author: Daisuke Oyama

"""
import sys
import numpy as np

try:
    try:
        import mpmath as mp
    except ImportError:
        import sympy.mpmath as mp
except ImportError:
    sys.stderr.write('Failed to import mpmath\n')
    sys.exit(1)

def mc_compute_stationary_mpmath(P, precision=17, irreducible=False, ltol=0, utol=None):
    """
    Computes the stationary distributions of Markov matrix P.

    Parameters
    ----------
    P : array_like(float, ndim=2)
        A discrete Markov transition matrix.

    precision : scalar(int), optional(default: 17)
        Decimal precision in float-point arithemetic with mpmath.
        mpmath.mp.dps is set to *precision*.

    irreducible : bool, optional(default: False)
        Set True if P is known a priori to be irreducible
        (for any i, j, (P^k)_{ij} > 0 for some k).
        If True, the eigenvector for the maximum eigenvalue is returned.

    ltol, utol: scalar(float), optional(default: ltol=0, utol=None)
        Lower and upper tolerance levels.
        Find eigenvectors for eigenvalues in [1-ltol, 1+utol]
        (where [1-ltol, 1+utol] = [1-ltol, +inf) when utol=None).

    Returns
    -------
    vecs : list of numpy.arrays of mpmath.ctx_mp_python.mpf
        A list of the eigenvectors of whose eigenvalues in [1-ltol, 1+utol].

    Notes
    -----
    mpmath 0.18 or above is required.

    References
    ----------

        http://mpmath.org/doc/current

    """
    LTOL = ltol  # Lower tolerance level
    if utol is None:  # Upper tolerance level
        UTOL = 'inf'
    else:
        UTOL = utol

    with mp.workdps(precision):  # Temporarily change the working precision
        E, EL = mp.eig(mp.matrix(P), left=True, right=False)
        # E  : a list of length n containing the eigenvalues of A
        # EL : a matrix whose rows contain the left eigenvectors of A
        # See: github.com/fredrik-johansson/mpmath/blob/master/mpmath/matrices/eigen.py
        E, EL = mp.eig_sort(E, EL)  # Sorted in a descending order

        if irreducible:
            num_eigval_one = 1
        else:
            num_eigval_one = sum(
                mp.mpf(1) - mp.mpf(LTOL) <= val <= mp.mpf(1) + mp.mpf(UTOL)
                for val in E
                )

        vecs = [np.array((EL[i, :]/sum(EL[i, :])).tolist()[0])
                for i in range(EL.rows-num_eigval_one, EL.rows)]

    return vecs

```

```
from __future__ import division
import numpy as np

def set_tran_matrix(N=6,p=1/3,epsilon=0.01):  #setting up the transition matrix A
	"""
	N :the number of players
	p : the turning point as to which equilibrium is prefered
	epsilon : tthe possibility that a man becomes "irrational"

	if the ratio of players playing 0 exceeds p, we end up the equilibrium(0,0) in an unperturbed game
	I think it's better if I can compute the value of p in this program.For now,we assume it's exogenous.
	"""
	A = np.zeros((N+1,N+1))
	for n in range(1, N):
		A[n, n-1] = \
	    	(n/N) * (epsilon * (1/2) +
                     (1 - epsilon) * (((n)/(N) < p) + ((n)/(N) == p) * (1/2))
                     )
		A[n, n+1] = \
            ((N-n)/N) * (epsilon * (1/2) +
                         (1 - epsilon) * ((n/(N) > p) + (n/(N) == p) * (1/2))
                         )
		A[n,n] = 1- A[n,n-1] - A[n,n+1]

	A[0,0] = 1 - epsilon/2
	A[0,1] = epsilon/2

	A[N,N-1] = epsilon/2
	A[N,N] = 1 - epsilon/2

	return A
```

```
P = set_tran_matrix()
```

```
mc_compute_stationary_mpmath(P)
```

    []

```
P = set_tran_matrix(N=26)
```

```
mc_compute_stationary_mpmath(P)
```

    [array([mpf('1.7946972729275572e-21'), mpf('2.3448306078450533e-22'),
            mpf('1.4728835476413624e-23'), mpf('5.9211398900153081e-25'),
            mpf('1.7108821290228975e-26'), mpf('3.7828549585157268e-28'),
            mpf('6.6532624792810135e-30'), mpf('9.5524218977083442e-32'),
            mpf('1.1400424773350582e-33'), mpf('2.2786936807187899e-33'),
            mpf('7.7088088919014376e-31'), mpf('2.2313497621135247e-28'),
            mpf('5.55048253316502e-26'), mpf('1.1895111028766731e-23'),
            mpf('2.1980465879585375e-21'), mpf('3.4992901680299916e-19'),
            mpf('4.7874663611360307e-17'), mpf('5.604151799212178e-15'),
            mpf('5.5761310402161177e-13'), mpf('4.6722108505389789e-11'),
            mpf('3.2541948574003983e-9'), mpf('1.850242218921941e-7'),
            mpf('8.3681409446696875e-6'), mpf('0.00028961044312856845'),
            mpf('0.0072040597728231406'), mpf('0.11468863158334436'),
            mpf('0.87780914173405714')], dtype=object)]

```
P
```

    array([[ 0.995     ,  0.005     ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.03826923,  0.95692308,  0.00480769,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.07653846,  0.91884615,  0.00461538,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.11480769,  0.88076923,  0.00442308,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.15307692,  0.84269231,
             0.00423077,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.19134615,
             0.80461538,  0.00403846,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.22961538,  0.76653846,  0.00384615,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.26788462,  0.72846154,  0.00365385,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.30615385,  0.69038462,  0.00346154,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.00173077,  0.34769231,
             0.65057692,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.00192308,
             0.38576923,  0.61230769,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.00211538,  0.42384615,  0.57403846,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.00230769,  0.46192308,  0.53576923,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.0025    ,  0.5       ,  0.4975    ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.00269231,  0.53807692,
             0.45923077,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.00288462,
             0.57615385,  0.42096154,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.00307692,  0.61423077,  0.38269231,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.00326923,  0.65230769,  0.34442308,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.00346154,  0.69038462,  0.30615385,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.00365385,  0.72846154,
             0.26788462,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.00384615,
             0.76653846,  0.22961538,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.00403846,  0.80461538,  0.19134615,  0.        ,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.00423077,  0.84269231,  0.15307692,  0.        ,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.00442308,  0.88076923,  0.11480769,
             0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.00461538,  0.91884615,
             0.07653846,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.00480769,
             0.95692308,  0.03826923],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.005     ,  0.995     ]])

```
P = set_tran_matrix()
```

```
mc_compute_stationary_mpmath(P,precision=11)
```

    []

```
mc_compute_stationary_mpmath(P,precision=8)
```

    []

```
mc_compute_stationary_mpmath(P,precision=13)
```

    []

```

```


```python
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    try:
        tmp = DX(str(float(obj)))
        obj = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(obj)):
            tmp = DX(str(float(obj[i])))
            obj[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return obj

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""
# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm, deg) = ( m/1000, m/100, pi/180)
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton

half = S(1)/2

# ---

def k(phi):
    """ element stiffness matrix """
    # phi: Angle between two rays 1 and 2:
    # 1: ray along global x axis and
    # 2: ray along 1-2-axis of rod
    # phi is counted positively about z.
    (c, s) = ( cos(phi), sin(phi) )
    (cc, ss, sc) = ( c*c, s*s, s*c)
    return Matrix(
        [
        [ cc,  sc, -cc, -sc],
        [ sc,  ss, -sc, -ss],
        [-cc, -sc,  cc,  sc],
        [-sc, -ss,  sc,  ss],
        ])

F, c = var("F, c")

# Stiffness Matrix:
k1 = c*k(135 *pi/180)
pprint("\n\nk1 / c: ")
pprint(k1/c)

# Linear System:
u, F1x, F2x, F2y = var("u, F1x, F2x, F2y")
f_ = Matrix([F1x, -F/2, F2x, F2y])
u_ = Matrix([0, -u, 0, 0])

sol = solve(Eq(k1*u_,f_), [u, F1x, F2x, F2y], dict=True)
pprint("\n\nSolution:")
pprint(sol[0])

# k1 / c:
# ⎡1/2   -1/2  -1/2  1/2 ⎤
# ⎢                      ⎥
# ⎢-1/2  1/2   1/2   -1/2⎥
# ⎢                      ⎥
# ⎢-1/2  1/2   1/2   -1/2⎥
# ⎢                      ⎥
# ⎣1/2   -1/2  -1/2  1/2 ⎦
#
#
# Solution:
# ⎧     F       -F        F     F⎫
# ⎨F1x: ─, F2x: ───, F2y: ─, u: ─⎬
# ⎩     2        2        2     c⎭

```


# DSCI 572 Lab 2

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

import time

import scipy.stats
import scipy.optimize as spo
import scipy.special
import sklearn.datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression, LinearRegression, SGDClassifier, SGDRegressor
from sklearn.preprocessing import scale, StandardScaler, minmax_scale, MinMaxScaler
from sklearn.feature_extraction.text import CountVectorizer
```

## Instructions
rubric={mechanics:20}

Follow the [general lab instructions](https://ubc-mds.github.io/resources_pages/general_lab_instructions/).

## Exercise 1: floating point errors

For each of the code snippets below, explain the result. The first three are answered for you, so you can get a sense of what types of explanations we're looking for. Note that in both the second and third case we get the "expected" result (zero), but the explanations are very different.

```python
0.3 - 0.2 - 0.1
```

    -2.7755575615628914e-17

**Sample solution**: The result is not zero because 0.3, 0.2, and 0.1 are not represented exactly as floating point numbers.

```python
0.5 - 0.25 - 0.125 - 0.125
```

    0.0

**Sample solution**: The result is zero because 0.5, 0.25, and 0.125 are powers of 2 and there they _are_ represented exactly as floating point numbers. There is no roundoff error present.

```python
0.4 - 0.2 - 0.2
```

    0.0

**Sample solution**: The result is correct (zero) despite the fact that 0.4 and 0.2 are not represented exactly as floating point numbers. This is a case of good luck: while 0.4 and 0.2 are not represented exactly, the roundoff errors happened to cancel out during the subtractions.

### snippet (a)
rubric={reasoning:5}

```python
30 - 20 - 10
```

### snippet (b)
rubric={reasoning:5}

```python
30.0 - 20.0 - 10.0
```

### snippet (c)
rubric={reasoning:5}

```python
(10.0**100 + 1) == 10.0**100
```

### snippet (d)
rubric={reasoning:5}

```python
(10.0**100000 + 1) - 10.0**100000
```

### snippet (e)
rubric={reasoning:5}

```python
np.exp(1000) - np.exp(1000)
```

### snippet (f)
rubric={reasoning:5}

```python
1/np.exp(1000) == np.exp(-1000)
```

### snippet (g)
rubric={reasoning:5}

```python
1/np.exp(100) == np.exp(-100)
```

### snippet (h)
rubric={reasoning:5}

```python
np.exp(1000)==np.exp(10000)
```

### snippet (i)
rubric={reasoning:5}

```python
sum(np.zeros(10)+0.1)
```

### snippet (j)
rubric={reasoning:5}

```python
np.sin(np.pi)
```

### (optional) snippet (k)
rubric={reasoning:1}

```python
x = np.ones(100000)
x[0] = 1e20

y = np.ones(100000)
y[-1] = 1e20

sum(x) == sum(y)
```

### (optional) snippet (l)
rubric={reasoning:1}

```python
x = np.zeros(10)+0.1
sum(x) == np.sum(x)
```

Hint for the above: see [Pairwise summation](https://en.wikipedia.org/wiki/Pairwise_summation).

## (optional) Exercise 2: floating point max/min
rubric={reasoning:2}

As discussed in lecture, IEEE double precision floating point numbers use 53 bits for the mantissa (one of which is a sign bit) and 11 bits for the exponent (again, one of which is a sign bit). Given thus, calculate the largest (furthest from zero) and smallest (closest to zero) possible representable floating point numbers. Then empirically check your results with Python. Are they what you expected? Discuss.

NOTE: Python has a special behaviour that we need to watch out for. If you do something like `10**1000` you will get a giant integer. That's because Python has a dynamically expanding integer type. This has nothing to do with floating point representations, which are the thing we really care about for scientific computation (not to mention that most other languages, including R, do not do this). So, when playing around, make sure you write `10**1000.0` or `10.0*1000` to ensure it's a floating point. Or `1e1000`... but that doesn't work for other bases besides 10.

(Also, and this is _REALLY_ out of scope but just FYI if anyone cares, in some languages `1eX` and `10^X` will return slightly different answers, if the language uses a special routine for `1eX` that is more optimized than the generral power function. I cannot imagine this ever mattering to any of us.)

## Exercise 3: logpdf
rubric={reasoning:10}

In most scientific computing environments, such as R or NumPy/SciPy, there are functions to compute the probability density function (pdf) of a probability distribution, and separate functions to compute the log of the pdf. For example, looking at the [R normal distribution documentation](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/Normal.html), you will see that you can set `log=TRUE` or `log=FALSE`. Likewise in Python we have `scipy.stats.norm.pdf` and `scipy.stats.norm.logpdf`. And yet, why bother creating these extra functions, when we can just take the log ourselves?! For example, we have NumPy functions for $\sin(x)$ and $\cos(x)$ but not for $\log\sin(x)$ and $\log \cos (x)$.

```python
scipy.stats.norm.logpdf(1) # is function this useless?
```

    -1.4189385332046727

```python
np.log(scipy.stats.norm.pdf(1)) # seems to do the same thing!
```

    -1.4189385332046727

### (optional) 3(b)
rubric={accuracy:1}

Write a numerically safe function `log_gaussian_pdf` that computes the log of the standard (zero mean, unit variance) Gaussian PDF. Your function should produce the same result as `scipy.stats.norm.logpdf` when evaluated at $x=100$.

## (optional) Exercise 4: $\log(1+\exp(z))$, continued
rubric={accuracy:1,reasoning:1}

In lecture we discussed computing $\log(1+\exp(z))$. We wrote a better version of the function for when $z\gg1$, reproduced below.

```python
def log_1_plus_exp(z):
    return np.log(1+np.exp(z))

def log_1_plus_exp_safe(z):
    if z > 100:
        return z
    else:
        return np.log(1+np.exp(z))
```

Now let's consider the case of $z\ll -1$, i.e. when $z$ is a large negative number. In that case, we get zero with both of the above implementations:

```python
print(log_1_plus_exp(-100))
```

    0.0

```python
print(log_1_plus_exp_safe(-100))
```

    0.0

Your tasks:

1. Investigate why this is happening. Is the problem that $\exp(-100)$ itself underflows?
2. Write a function `log_1_plus_exp_safer` that works well when $z\gg 1$ and $z\ll -1$.
3. For what range of values of $z$ does your `log_1_plus_exp_safer` function give reasonable results?
4. Your code presumably contains two thresholds, the upper and lower cutoffs for $z$ at which the approximations are invoked. Can you reason about the "optimal" or "best" values for these thresholds?

## Exercise 5: softmax logistic regression and log-sum-exp
rubric={reasoning:15,accuracy:5}

In the "multinomial" (aka softmax) approach to multi-class logistic regression, your loss ends up having one term for each class, so you get something of the form $\log \sum_{k=1}^c \exp(z_k)$, where $c$ is the number of classes, and $z_k$ is a quantity involving the weights and training data. For any choice of a constant $a$, we can rewrite this as follows:

$$\begin{align}\log \displaystyle\sum_{k=1}^c \exp(z_k) &= \log \displaystyle\sum_{k=1}^c \exp(z_k-a+a)\\ &= \log \left( \displaystyle\sum_{k=1}^c \exp(z_k-a)\exp(a) \right) \\ &= \log \left( \exp(a)\displaystyle\sum_{k=1}^c \exp(z_k-a)\right) \\ &= \log \left( \exp(a) \right) + \log\displaystyle\sum_{k=1}^c \exp(z_k-a) \\ &= a+ \log \displaystyle\sum_{k=1}^c \exp(z_k-a)\end{align}$$

Note: you only need to look at the first and last expression - the middle parts are only there to convince you they are indeed equivalent.

1. Explain why this final expression might be more numerically stable and why $a=\max \{z_1,z_2,\ldots,z_c\}$ is a sensible choice.
2. If $a=\max \{z_1,z_2,\ldots,z_c\}$, this trick seems to rely on the fact that overflow is more of a danger than underflow, because we may now compute $exp$ of some very large negative values if $a$ is large. Explain why overflow is more of a problem than underflow in this calculation.
3. Write a python function `log_sum_exp` that takes in an array `z` and computes the _original_ expression. Write another function `log_sum_exp_stable` that computes the final (more stable) expression using $a=\max \{z_1,z_2,\ldots,z_c\}$. Discuss the behaviour of the two functions. Also, compare your implementation to In fact, [`scipy.special.logsumexp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.logsumexp.html#scipy.special.logsumexp) for one or two cases.
4. (optional) Earlier, we used the approximation $\log(1+\exp(z))\approx z$ when $z\gg 1$. Is that just a specific case of what we have here? It seems that what we did earlier was an approximation, whereas what we did here is mathematically exact. Do you think there is any significance to this distinction, in practice?

(FYI: you'll see this trick in real implementations of ML algorithms! )

## Exercise 6: SGD implementation

Below is a Python function that performs gradient descent, much like the code you wrote in lab 1. I just made a few small changes:

- Removed the docstring and `print` statements for brevity.
- Renamed `x` to `w`.
- In this version, the user specifies a fixed number of iterations `n_iters` instead of a tolerance $\epsilon$.
- In this version, the user passes in `X` and `y` explicitly, and these are passed to `f_grad`.

```python
def gradient_descent(f, f_grad, w0, X, y, n_iters=1000, α=0.001):
    w = w0
    for t in range(n_iters):
        w = w - α*f_grad(w, X, y)
    return w
```

The code works (although is very slow as per usual for gradient descent). For example, here we do ordinary least squares linear regression:

```python
from sklearn.datasets import load_boston
```

```python
data = load_boston()
X = data['data']
y = data['target']
X = minmax_scale(X)
```

```python
def loss_ols(w,X,y): return 0.5*np.sum((X@w-y)**2)
def loss_ols_grad(w,X,y): return X.T@(X@w) - X.T@y

w0 = np.zeros(X.shape[1])
```

```python
w_gd = gradient_descent(loss_ols, loss_ols_grad, w0, X, y, n_iters=10**5)
```

```python
w_gd
```

    array([ -7.37446749,   4.79336123,   3.71903592,   2.67885881,
            -2.90900155,  36.50466055,   2.10874674,  -4.55318591,
             5.45719841,  -6.02418944,  -4.39530169,   9.88901723,
           -11.79090734])

```python
lr = LinearRegression(fit_intercept=False).fit(X, y)
```

```python
lr.coef_
```

    array([ -7.37446749,   4.79336123,   3.71903592,   2.67885881,
            -2.90900155,  36.50466055,   2.10874674,  -4.55318591,
             5.45719841,  -6.02418944,  -4.39530169,   9.88901723,
           -11.79090734])

As we can see, the coefficients obtained from gradient descent are very similar to those obtained by scikit-learn. All is well so far.

#### 6(a)
rubric={accuracy:20,quality:10}

Your task is to implement a function `stochastic_gradient_descent`, that performs SGD, _using_ the `gradient_descent` function provided above. You can have your function accept the same arguments as the `gradient_descent` function above, except:

- Change `n_iters` to `n_epochs`.
- Add an extra `batch_size` argument.

Note: even though this might not be a good idea in general, you can leave $\alpha$ constant in this implementation; that is, you can use the same $\alpha$ across all iterations.

Note: the pedagogical goal here is to help you see how SGD relates to regular gradient descent. In reality it would be fine to implement SGD "from scratch" without calling a GD function.

#### 6(b)
rubric={reasoning:5}

Show that when the batch size is set to the whole training set, you get exactly the same results as with gradient descent.

## Exercise 7: `SGDClassifier` and `SGDRegresor`

In this exercise we'll explore training a classifier with SGD on the [Sentiment140 dataset](http://help.sentiment140.com/home), which contains tweets labeled with sentiment associated with a brand, product, or topic. Please start by doing the following:

1. Download the corpus from [here](http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip).
2. Unzip.
3. Copy the file `training.1600000.processed.noemoticon.csv` into the current directory.
4. Create a `.gitignore` file so that you don't accidentally commit the dataset.

Once you're done the above, steps, run the starter code below:

```python
# Data loading and preprocessing
tweets_df = pd.read_csv('training.1600000.processed.noemoticon.csv',
                        encoding = "ISO-8859-1",
                        names=["label","id", "date", "no_query", "name", "text"])
tweets_df['label'] = tweets_df['label'].map({0: 'neg', 4: 'pos'})

tweets_df.head()

# only consider positive and negative reviews
tweets_pos_neg_df = tweets_df[tweets_df['label'].str.startswith(('pos','neg'))]
```

```python
tweets_pos_neg_df.head()
```

Now we split the data:

```python
tweets_df_train, tweets_df_test = train_test_split(tweets_pos_neg_df)
```

And then we encode it using `CountVectorizer`, which may take a minute or so:

```python
vec = CountVectorizer(stop_words='english')

X_train = vec.fit_transform(tweets_df_train['text'])
y_train = tweets_df_train['label']

# just use the already-fit vectorizer
X_test = vec.transform(tweets_df_test['text'])
y_test = tweets_df_test['label']
```

Note that our training data is rather large compared to datasets we've explored in the past:

```python
X_train.shape
```

```python
type(X_train)
```

Here is the fraction of elements that are nonzero. Having a sparse matrix really helps!!

```python
X_train.nnz/np.prod(X_train.shape)
```

Now let's train a classifier. I'll use `time` instead of `%timeit` because I want to keep the output, and it gets lost with `%timeit`.

```python
lr = LogisticRegression()
```

```python
t = time.time()

lr.fit(X_train, y_train)

print("Training took %.1f seconds" % (time.time() - t));
```

```python
lr.score(X_train, y_train)
```

```python
lr.score(X_test, y_test)
```

#### 7(a)
rubric={reasoning:10}

Train a logistic regression model on the same dataset using [`SGDClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html) instead of `LogisticRegression`. Compare the training time.

FYI: there's also a regression version, [`SGDRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.html), which does linear regression with SGD. However, we won't explore `SGDRegressor` here.

#### 7(b)
rubric={reasoning:15}

Discuss the training and test accuracies of the two approaches. Can you explain what you're seeing?

#### (optional) 7(c)
rubric={reasoning:2}

One possible explanation for `SGDClassifier`'s speed is that it's just doing fewer iterations/epochs. `SGDClassifier` and `LogisticRegression` have an `n_iter_` attribute which you can check after fitting. Compare these values and discuss them in the context of the above hypothesis. Then, using the `max_iter` parameter, do a "fair" experiment where `SGDClassifier` and `LogisticRegression` do the same number of passes through the dataset, and comment on the results.


# Expectation–maximization for Clustering

This code follows chapters 9 and 10 of the Bishop Book.

Important formulas:

$$
    \log q_j^\star(z_j) = \mathbb{E}_{i \neq j} \log p(X, Z) + \mathrm{const}
$$

$$
    P(X, Z) = P(X|\theta, Z) P(Z | \pi)
$$

```python
%matplotlib inline
```

```python
import math

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import scipy.special
import seaborn as sns
```

```python
class EMMixtureModel(object):
    def __init__(self, n_components, n_iter=100):
        self.n_components = n_components
        self.n_iter = n_iter

        self._mixture_weights = None
        self._params = None

    def fit(self, *x):
        params = self._initial_params()
        pi = np.ones(self.n_components) / self.n_components

        for _ in range(self.n_iter):
            gamma = self._responsibility(x, pi, params)
            params = self._sufficient_statistics(gamma, x)

            pi = gamma.sum(axis=1)
            pi /= pi.sum()

        self._mixture_weights = pi
        self._params = params
        return self

    def predict_proba(self, *x):
        return sum(
            self._mixture_weights[i] * self._component_density(x, self._mixture_params(i, self._params))
            for i in range(self.n_components)
        )

    def predict_responsibility(self, *x):
        return self._responsibility(x, self._mixture_weights, self._params)

    @staticmethod
    def _mixture_params(i, params):
        return [p[i] for p in params]

    def _responsibility(self, x, pi, parameters):
        result = [
            pi[i] * self._component_density(x, self._mixture_params(i, parameters))
            for i in range(self.n_components)
        ]

        result = np.stack(result)
        result /= result.sum(axis=0, keepdims=True)

        return result

    def _initial_params(self):
        raise NotImplementedError()

    def _component_density(self, ):
        raise NotImplementedError()

    def _sufficient_statistics(self, gamma, x):
        raise NotImplementedError()

class GaussianMixtureModel(EMMixtureModel):
    @staticmethod
    def _component_density(x, params):
        x, = x
        mu, sigma = params
        d = (x - mu) / sigma
        return np.exp(-d ** 2.0) / (np.sqrt(2.0 * np.pi) * sigma)

    @staticmethod
    def _sufficient_statistics(gamma, x):
        x, = x
        Nk = gamma.sum(axis=1)

        mu = (gamma * x).sum(axis=1) / Nk
        sigma = (gamma * (x - mu[..., None]) ** 2.0).sum(axis=1) / Nk
        sigma **= 0.5

        return mu, sigma

    @staticmethod
    def _initial_params():
        mu = np.random.uniform(-20, +20, 3)
        sigma = np.ones(3)

        return mu, sigma

class PoissonMixtureModel(EMMixtureModel):
    @staticmethod
    def _component_density(x, params):
        x, = x
        mu, = params

        # use numerically stable evaluation method, taken from wikipedia
        p = np.exp(x * np.log(mu) - mu - scipy.special.gammaln(x + 1))
        return p

    @staticmethod
    def _sufficient_statistics(gamma, x):
        x, = x
        Nk = gamma.sum(axis=1)
        mu = (gamma * x).sum(axis=1) / Nk
        return mu,

    def _initial_params(self):
        mu = np.random.gamma(1.0, 10.0, self.n_components)
        return mu,

class ExponentialMixtureModel(EMMixtureModel):
    @staticmethod
    def _component_density(x, params):
        x, = x
        tau, = params
        return np.exp(-x / tau) / tau

    @staticmethod
    def _sufficient_statistics(gamma, x):
        x, = x
        tau = (gamma * x).sum(axis=1) / gamma.sum(axis=1)
        return tau,

    def _initial_params(self):
        tau = np.random.gamma(1.0, 10.0, self.n_components)
        return tau,

class ExponentialSurvivalMixtureModel(EMMixtureModel):
    @staticmethod
    def _component_density(x, params):
        t, d = x
        tau, = params
        norm = 1.0 + d * (tau - 1)
        return np.exp(-t / tau) / norm

    @staticmethod
    def _sufficient_statistics(gamma, x):
        t, d = x

        T = (gamma * t).sum(axis=1)
        D = (gamma * d).sum(axis=1)
        return T / D,

    def _initial_params(self):
        tau = np.random.gamma(1.0, 10.0, self.n_components)
        return tau,

```

```python
color_cycle = mpl.rcParams['axes.prop_cycle'].by_key()['color']
```

```python
n_clusters = 3

np.random.seed(100)
cluster = np.random.choice(range(n_clusters), p=[0.2, 0.3, 0.5], size=1000)
centers = np.random.uniform(-20, 20, size=n_clusters)
x = np.random.normal(centers[cluster], 1.0)

model = GaussianMixtureModel(n_components=3)
model.fit(x)

xp = np.linspace(-20, 20, 100)
pp = model.predict_proba(xp)

plt.hist(x, range=(-30, 30), bins=50, normed=True)
for c in centers:
    plt.axvline(x=c)

plt.plot(xp, pp)

pass
```

# Poisson clustering

$$
    p(k; \mu) = \frac{\mu^k}{k!} e^{-\mu}
$$

```python
np.random.seed(42)
mu = np.random.gamma(1.0, 10.0, 3)
z = np.random.choice([0, 1, 2], p=[0.1, 0.3, 0.6], size=1000)
x = np.random.poisson(mu[z])

model = PoissonMixtureModel(n_components=3)
model.fit(x)

xp = np.arange(50)
pp = model.predict_proba(xp)

plt.hist(x, bins=np.arange(50) - 0.5, normed=True)
plt.plot(xp, pp, color=color_cycle[1])

for c in mu:
    plt.axvline(c, color=color_cycle[2])

for c in model._params[0]:
    plt.axvline(c, color=color_cycle[3], ls='--')

pass
```

# Exponential Distribution

Probability to observe an event at time $t$:

$$
    f(t;\tau) = \frac{e^{-t / \tau}}{\tau}
$$

$$
    \tau^\star = \frac{1}{N} \sum_i t_i
$$

```python
np.random.seed(70)
mu = np.asarray([5.0, 10.0, 30.0])
z = np.random.choice([0, 1, 2], p=[0.1, 0.3, 0.6], size=10000)
x = np.random.exponential(mu[z])

model = ExponentialMixtureModel(n_components=3)
model.fit(x)

xp = np.arange(50)
pp = model.predict_proba(xp)
g = model.predict_responsibility(x)

plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)

plt.hist(x, bins=np.arange(50) - 0.5, normed=True)
plt.plot(xp, pp, color=color_cycle[1])

for c in mu:
    plt.axvline(c, color=color_cycle[2])

for c in model._params[0]:
    plt.axvline(c, color=color_cycle[3], ls='--')

plt.subplot(1, 2, 2)
plt.hist([x] * model.n_components, weights=g, stacked=True, bins=np.arange(50) - 0.5)

pass
```

# Exponential Survival Models

Probability to observe an event at time $t$:

$$
    f(t;\tau) = \frac{e^{-t / \tau}}{\tau}
$$

For not yet observed events, we only know that the true time has to be larger than the observation time $t_f$. The likelihood for such an event is given by

$$
    S(t_f) = \int_{t_f}^\infty \mathrm{d}u\; f(u) = e^{-t_f / \tau}
$$

This yields the log-likelihood:

$$
    \mathcal{L} =
        -\sum_{i \in \text{obs.}} \frac{t_i}{\tau}
        -\sum_{i \in \text{not obs.}} \frac{t_f}{\tau}
        -\sum_{i \in \text{obs.}} \log \tau
$$

The optimimum can be found as:

$$
\begin{align}
    \tau^\star &= \frac{T}{D}
\\
    T &= \sum_i \begin{cases}
        t_i & \text{if event is observed} \\
        t_f & \text{if event is not observed}
    \end{cases}
\\
    D &= \sum_i \begin{cases}
        1 & \text{if event is observed} \\
        0 & \text{if event is not observed}
    \end{cases}
\end{align}
$$

For mixtures, the sums for both $T$ and and $D$ have to be weighted by $\gamma_i$.

```python
np.random.seed(70)
mu = np.asarray([5.0, 10.0, 30.0])
z = np.random.choice([0, 1, 2], p=[0.1, 0.3, 0.6], size=10000)
z = 2 + 0 * z
x = np.random.exponential(mu[z])

tc = 80
d = x <= tc
y = np.minimum(tc, x)

model = ExponentialSurvivalMixtureModel(n_components=3)
model.fit(y, d)

xp = np.arange(100)
pp = model.predict_proba(xp, np.ones_like(xp))
g = model.predict_responsibility(y, d)

plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)

plt.hist(x, bins=np.arange(100) - 0.5, normed=True)
plt.axvline(x=tc, color='k', ls="-")
plt.plot(xp, pp, color=color_cycle[1])

for c in mu:
    plt.axvline(c, color=color_cycle[2])

for c in model._params[0]:
    plt.axvline(c, color=color_cycle[3], ls='--')

plt.subplot(1, 2, 2)
plt.hist([x] * model.n_components, weights=g, stacked=True, bins=np.arange(50) - 0.5)

pass
```

```python

```


# Computational Astrophysics
## Partial Differential Equations. 05
## Non-Linear Hiperbolic PDE. Burguer's Equation

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the techniques used to solve the linear advection equation.

`A. Garcia. Numerical Methods for Physics. (1999). Chapter 6 - 7 `

---

##  The Linear 1-D Advection Equation

The linear advection equation is

\begin{equation}
\label{eq:advect}
\partial_t u + v \partial_x u = 0
\end{equation}

where $u(t,x)$ is some scalar quantity and $v$ is the constant velocity at
which it is advected ($v > 0$ advects to the right).

The solution to
this equation is to simply take the initial data, $u(t=0,x)$,
and displace it to the right at a speed $v$.  The shape of the initial
data is preserved in the advection. Direct substitution shows that $u(x - vt)$ is a solution to
advection equation for any choice of u. This means that
the solution is constant along the lines $x = v t$
(the curves along which the solution is constant are called the
characteristics).

Many hyperbolic systems of PDEs,
e.g. the equations of hydrodynamics, can be written in a form that
looks like a system of (nonlinear) advection equations, so the
advection equation provides important insight into the methods used
for these systems.

---

## Non-linear Hyperbolic PDE. Burguer's Equation

Burgers' equation is the simplest **nonlinear hyperbolic equation**,

\begin{equation}
\partial_t u + u \partial_x u = 0.
\end{equation}

It is almost identical to the advection equation treated before, but this time the wave speed is
**NOT a constant** $v$ but is
given by the field $u$ itself.

**Then, $u$ is both the quantity being advected and the speed at which
it is moving.**

---

### Shocks

For the linear advection equation, the solution was constant along lines $x = vt + x_0$, which are
parallel (because $v$ is spatially constant).

For Burgers' equation,
this is no longer the case, and the characteristic lines are now
given by $\frac{dx}{dt} = u$, with $x(0) = x_0$.
Since $u = u(t)$, we cannot integrate this directly.

If we take $u_0 = u(t=0)$, then we can look at how the characteristic
behave over a small time interval (before $u(x,t)$ changes
significantly).

We see that after a
short period of time, the characteristics intersect.  At the point,
$(x_s, t_s)$ where they intersect, there is no way to trace backwards
along the characteristics to find a unique initial state.

This
merging of the characteristics in the $x$-$t$ plane is a **shock**, and
represents just one way that nonlinear problems can differ from linear
ones.

---
### Rarefaction

Another type of wave not present in a linear system is a **rarefaction**.  Next figure shows initial
conditions of slower velocity to the left of faster velocity.

We see that the characteristics diverge in this case, and we will be left
with having to fill in the solution inbetween as some intermediate
state.


# The Capacitated Vehicle Routing Problem

This notebook look at solving the CVRP using the **exact methods implemented in Gurobi**

> Vehicle routing problems are inherently intractable: the length of time it takes to solve them grows exponentially with the size of the problem. For sufficiently large problems, it could take routing software years to find the optimal solution...

```python
import numpy as np
import pandas as pd
from numpy.random import RandomState
import matplotlib.pyplot as plt
from gurobipy import Model, GRB, quicksum
import seaborn as sns
sns.set()
```

## Helper functions

```python
def plot_patient_coordinates(x_coords, y_coords):
    '''
    Plot a simple scatter of coordinates.
    Depot is red, customers are blue

    Paramters:
    x_coords: array-like
        x coordinates for each patient. element 0 = depot.

    y_coords: array-like
        y coordinates for each patient. element 0 = depot
    '''
    plt.plot(x_coords[0], y_coords[0], c='r', marker='^')
    #plt.scatter(x_coords[1:], y_coords[1:], c='b')

    for i in range(1, len(x_coords[1:])):
        x = x_coords[i]
        y = y_coords[i]
        plt.plot(x, y, 'b*')
        plt.text(x * (1 + 0.01), y * (1 + 0.01) , i, fontsize=12)

    plt.xlim((0, 220))
    plt.ylim((0, 120))
    plt.show()
```

```python
def problem_instance(n_patients, vehicle_capacity, random_state=None):
    '''
    Create an instance of the CVRP in the formulation
    of a MIP.

    Parameters:
    ------
    n_patient: int
    vehicle_capacity: int
    random_state: int, optional (default=None)

    Returns:
    -----
        dict, keys = N, V, A, c, Q, q
    '''
    rand_gen = RandomState(seed=random_state)

    #n_patients = 1 so we include the depot
    x_coords = rand_gen.random(size=n_patients+1) * 200
    y_coords = rand_gen.random(size=n_patients+1) * 100

    N = np.arange(1, n_patients+1)
    V = np.arange(0, n_patients+1)
    A = [(i, j) for i in V for j in V if i != j]
    c = {(i, j): np.hypot(x_coords[i]-x_coords[j], y_coords[i]-y_coords[j])
     for i, j in A}
    Q = vehicle_capacity
    q = {i: 1 for i in N}
    return {'x_coords':x_coords, 'y_coords':y_coords,
            'N':N, 'V':V, 'A':A, 'c':c, 'Q':Q, 'q':q}
```

```python
def optimisation_model(instance):
    '''
    Create and return an istance of a gurobi optimisation
    model for solving the CVRP.

    Parameters:
    -------
    instance: dict
        problem instance for the CVRP
    '''
    mdl = Model('dialysis_transport')

    N = instance['N']
    V = instance['V']
    A = instance['A']
    c = instance['c']
    Q = instance['Q']
    q = instance['q']

    x = mdl.addVars(A, vtype=GRB.BINARY)
    u = mdl.addVars(N, vtype=GRB.CONTINUOUS)
    mdl.modelSense = GRB.MINIMIZE
    mdl.setObjective(quicksum(x[a]*instance['c'][a] for a in A))
    mdl.addConstrs(quicksum(x[i, j] for j in V if j != i) == 1 for i in N)
    mdl.addConstrs(quicksum(x[i, j] for i in V if i != j) == 1 for j in N)
    mdl.addConstrs((x[i, j] == 1) >> (u[i]+q[j] == u[j])
                   for i, j in A if i != 0 and j != 0)
    mdl.addConstrs(u[i] >= q[i] for i in N)
    mdl.addConstrs(u[i] <= Q for i in N)
    return mdl, x, u
```

```python
def plot_solution(x, instance):
    active_arcs = [a for a in instance['A'] if x[a].x > 0.99]
    xc = instance['x_coords']
    yc = instance['y_coords']

    for i, j in active_arcs:
        plt.plot([xc[i], xc[j]], [yc[i], yc[j]], c='green')
        plt.plot(xc[0], yc[0], c='r', marker='s')
        plt.scatter(xc[1:], yc[1:], c='b')
```

## Patient Geographic Location

```python
N_PATIENTS = 30
SEED = 101

#random number generator for synthetic data.
rand_gen = RandomState(seed=SEED)

#n_patients = 1 so we include the depot
x_coords = rand_gen.random(size=N_PATIENTS+1) * 200
y_coords = rand_gen.random(size=N_PATIENTS+1) * 100
```

```python
#visualise
plot_patient_coordinates(x_coords, y_coords)
```

## Approaches to 'solving' the CVRP

* Formulation as a **Mixed Integer Programme** (MIP) and use of a optimisation solver
* Heuristic solution.  E.g. using Iterated Local Search.

## Mixed Integer Programme (MIP) formulation

* The advantage of a mathematical forumulation of the problem is that we can use a modern solver like **Gurobi**.
* Gurobi is extremely powerful.  The downside is that a commerical tool and the source code is closed (but free to use in research).

$n$ is the number of patients

$N$ is the set of patients $\{1, 2, ..., n\}$

$V$ is the set of verticies (nodes) to visit. $V=\{0\}\cup N$

$A$ is the set of arcs (links) $A = \left\{ \left(i,j\right)  \in V^2 : i\neq j\right\}$

$c_{ij}$ is cost of travel over arc $(i,j)\in A$

$Q$ is the vehicle capacity (for ambulances this is an integer)

$q_i$ is the load represented by a patient (in our problem this always 1)

\begin{equation}
\begin{aligned}
       \text{min} &  \sum_{i=1}^{n} \sum_{j=1}^{n} c_{ij} x_{ij} \\
       \text{s.t.} & \sum_{i=1}^{n} x_{ij} = 1, & & j \in N, j \neq i \\
       &   \sum_{j=1}^{n} x_{ij} = 1, & & i \in N, i \neq j \\
       &  \text{if} \ x_{ij}=1 \ \Rightarrow \ u_i + q_j = u_j && i,j \in A : j\neq 0, i\neq 0 \\
       &  q_i \leq u_i \leq Q && i \in N \\
       &  x_{ij} \in \{0,1\} && i,j \in A
\end{aligned}
\end{equation}

## Model Execution

> Warning runtime can increase significantly for large problems!

Experiments:

* `N_PATIENTS = 10`
* `N_PATIENTS = 30`

```python
#model parameters
N_PATIENTS = 10
VEHICLE_CAPACITY = 4
TIME_LIMIT = -1
SEED = 101

#create a problem instance
instance = problem_instance(N_PATIENTS, VEHICLE_CAPACITY, random_state=SEED)

#create an instance of a gurobi optimisation model
model, x, u = optimisation_model(instance)

#limit run time?
if TIME_LIMIT > -1:
    model.Params.TimeLimit = TIME_LIMIT

#execute the optimisation procedure
model.optimize()
```

    Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)
    Optimize a model with 40 rows, 120 columns and 220 nonzeros
    Model fingerprint: 0x0169e2a8
    Model has 90 general constraints
    Variable types: 10 continuous, 110 integer (110 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [8e+00, 2e+02]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 4e+00]
    Presolve added 205 rows and 25 columns
    Presolve time: 0.00s
    Presolved: 245 rows, 145 columns, 1350 nonzeros
    Variable types: 55 continuous, 90 integer (90 binary)

    Root relaxation: objective 3.695996e+02, 53 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0  369.59959    0   22          -  369.59959      -     -    0s
    H    0     0                    1180.9647963  369.59959  68.7%     -    0s
    H    0     0                    1015.6786489  369.59959  63.6%     -    0s
    H    0     0                     580.6275380  369.59959  36.3%     -    0s
    H    0     0                     540.5513040  369.59959  31.6%     -    0s
    H    0     0                     529.6074856  382.24081  27.8%     -    0s
         0     0  382.24081    0   24  529.60749  382.24081  27.8%     -    0s
         0     0  397.86961    0   24  529.60749  397.86961  24.9%     -    0s
         0     0  397.86961    0   24  529.60749  397.86961  24.9%     -    0s
         0     0  434.90680    0   22  529.60749  434.90680  17.9%     -    0s
         0     0  443.15502    0   22  529.60749  443.15502  16.3%     -    0s
         0     0  455.53723    0   20  529.60749  455.53723  14.0%     -    0s
         0     0  455.53723    0   20  529.60749  455.53723  14.0%     -    0s
         0     2  455.53723    0   20  529.60749  455.53723  14.0%     -    0s

    Cutting planes:
      Learned: 1
      Gomory: 9
      Implied bound: 18
      MIR: 5
      Zero half: 3
      Relax-and-lift: 6

    Explored 1146 nodes (12744 simplex iterations) in 0.13 seconds
    Thread count was 16 (of 16 available processors)

    Solution count 5: 529.607 540.551 580.628 ... 1180.96

    Optimal solution found (tolerance 1.00e-04)
    Best objective 5.296074856061e+02, best bound 5.296074856061e+02, gap 0.0000%

```python
#show the routes
plot_solution(x, instance)
```

## Pre-saved results for 30 patients

* best in 2.5 hours = 1426.55594 with 39% optimality gap. (lower bound = 869)
*it took 930s (15.5 minutes) to reach this solution.

```python
active_arcs = pd.read_csv('best_solution_30_patients.csv').to_numpy()

xc = instance['x_coords']
yc = instance['y_coords']

for i, j in active_arcs:
    plt.plot([xc[i], xc[j]], [yc[i], yc[j]], c='green')
    plt.plot(xc[0], yc[0], c='r', marker='s')
    plt.scatter(xc[1:], yc[1:], c='b')
```


```python
import numpy as np
import matplotlib.pyplot as plt
from solver.interior_point import InteriorPointMethod
from solver.path_following import PathFollowing
import pandas as pd
from collections import OrderedDict
```

```python
def graph_2d(equations, x, x_iter, x_parallel=None, y_parallel=None, figsize=(10, 10), xlim=5, ylim=5):
    x1 = []
    y1 = []

    for x_ in x_iter:
        x1.append(x_[0])
        y1.append(x_[1])

    fig, ax = plt.subplots(figsize=(10, 10))
    ax.axhline(y=0, color='k')
    ax.axvline(x=0, color='k')

    if x_parallel is not None:
        for y_line in x_parallel:
            plt.axhline(y=y_line)
            plt.fill_between(0, y_line, x, alpha=0.5)

    if y_parallel is not None:
        for x_line in y_parallel:
            plt.axvline(x=x_line)
            plt.fill_between(x_line, 0, x, alpha=0.5)

    for (x, y) in equations:
        plt.plot(x, y, lw=1)
        plt.fill_between(x, 0, y, alpha=0.5)

    for (x_, y_) in zip(x1, y1):
        plt.plot(x_, y_, 'x', color='k')
    plt.plot(x1, y1, lw=1, label='Interior Point', color='k')

    # add non-negativity constraints
    plt.plot(np.zeros_like(x), x, lw=1, label='y non-negative')
    plt.plot(x, np.zeros_like(x), lw=1, label='x non-negative')

    # labels and stuff
    # ax.set_aspect('equal')
    ax.grid(True, which='both')

    plt.xlabel('x', fontsize=16)
    plt.ylabel('y', fontsize=16)
    plt.xlim(0, xlim)
    plt.ylim(0, ylim)

    handles, labels = ax.get_legend_handles_labels()

    plt.legend(handles=handles, fontsize=14)
    plt.show()
```

## Example 1

$
\begin{align}
    \max \quad & x_{1} + 2x_{2} \\
    \textrm{s.t.} \quad & x_{1} + s_{1} &= 2.3 \\
    \quad & 2x_{1} + 2x_{2} + s_{2} &= 10 \\
    \quad & 4x_{1} + x_{2} + s_{3} &= 10 \\
    \quad & 4x_{1} + 2x_{2} + s_{4} &= 12 \\
    \quad & x_{1} + 2.2x_{2} + s_{5} &= 10 \\
    \quad & x_{1}, x_{2} & \geq 0
\end{align}
$

```python
A = np.asarray(np.asmatrix('1, 0, 1, 0, 0, 0, 0; \
                         2, 2, 0, 1, 0, 0, 0; \
                         4, 1, 0, 0, 1, 0, 0; \
                         4, 2, 0, 0, 0, 1, 0; \
                         1, 2.2, 0, 0, 0, 0, 1'), dtype=np.float64)
b = np.asarray([2.3, 10, 10, 12, 10], dtype=np.float64)
c = np.asarray([-1, -2, 0, 0, 0, 0, 0], dtype=np.float64)

epsilon = 1e-3

ipm = InteriorPointMethod(A, b, c)
x_ipm, x_ipm_iter = ipm.solve(theta=0.95, gamma=0.01, epsilon=epsilon)
print(ipm.print_metric())
```

```python
x = np.linspace(0, 50)
eqs = []
eqs.append((x, 5-x))
eqs.append((x, 10 - 4*x))
eqs.append((x, 6 - 2*x))
eqs.append((x, (10 - x)/2.2))

y_parallel = [2.3]
graph_2d(eqs, x, x_ipm_iter, y_parallel=y_parallel, xlim=3, ylim=6)
```

## Example 2

$
\begin{align}
    \max \quad & 2x_{1} + 2x_{2} \\
    \textrm{s.t.} \quad & x_{1} + x_{2} + s_{1} &= 3 \\
    \quad & x_{1}, x_{2} & \geq 0
\end{align}
$

```python
A = np.asarray(np.matrix('1 1 1'), dtype=np.float64)
b = np.asarray([3], dtype=np.float64)
c = np.asarray([-2, -2, 0], dtype=np.float64)

epsilon = 1e-3
ipm = InteriorPointMethod(A, b, c)
x_ipm, x_ipm_iter = ipm.solve(theta=0.95, gamma=0.01, epsilon=epsilon)
print(ipm.print_metric())
```

```python
x = np.linspace(0, 50)
eqs = []

eqs.append((x, 3 - x))
graph_2d(eqs, x, x_ipm_iter, xlim=3, ylim=3)
```

## Example 3

$
\begin{align}
    \max \quad & 3x_{1} + 5x_{2} \\
    \textrm{s.t.} \quad & x_{1} + s_{1} &= 4 \\
    \quad & 2x_{2} + s_{2} &= 12 \\
    \quad & 3x_{1} + 2x_{2} + s_{3} &= 18 \\
    \quad & x_{1}, x_{2} & \geq 0
\end{align}
$

```python
A = np.asarray(np.matrix('1 0 1 0 0; 0 2 0 1 0; 3 2 0 0 1'), dtype=np.float64)
b = np.asarray([4, 12, 18], dtype=np.float64)
c = np.asarray([-3, -5, 0, 0, 0], dtype=np.float64)

epsilon = 1e-3
ipm = InteriorPointMethod(A, b, c)
x_ipm, x_ipm_iter = ipm.solve(theta=0.95, gamma=0.01, epsilon=epsilon)
print(ipm.print_metric())
```

```python
x = np.linspace(0, 50)
eqs = []
eqs.append((x, (18 - 3*x) / 2))

y_parallel = [4]
x_parallel = [6]
graph_2d(eqs, x, x_ipm_iter, y_parallel=y_parallel, xlim=7, ylim=10)
```

```python
A = np.asarray(np.matrix('1 1 1 0; 0 1 0 1'), dtype=np.float64)
b = np.asarray([2, 1], dtype=np.float64)
c = np.asarray([-2, -1, 0, 0], dtype=np.float64)

epsilon = 1e-3
ipm = InteriorPointMethod(A, b, c, init_val=1.0)
x_ipm, x_ipm_iter = ipm.solve(theta=0.9, gamma=0.1, epsilon=epsilon)
ipm.print_metric()
```

    =====================================================
    Iteration -> 0
    -----------------------------------------------------
    Delta Y -> 0
    -----------------------------------------------------
    Delta S -> 0
    -----------------------------------------------------
    Delta X -> 0
    -----------------------------------------------------
    Alpha k -> 1.0
    -----------------------------------------------------
    Ax - b = [1. 1.]
    -----------------------------------------------------
    Norm of Ax - b -> 1.4142135623730951
    -----------------------------------------------------
    Coordinates -> [1. 1. 1. 1.]
    -----------------------------------------------------
    Y Values -> [0. 0.]
    -----------------------------------------------------
    S Values -> [1. 1. 1. 1.]
    -----------------------------------------------------
    Functional value -> -3.0
    =====================================================
    =====================================================
    Iteration -> 1
    -----------------------------------------------------
    Delta Y -> [-1.28 -0.46]
    -----------------------------------------------------
    Delta S -> [-1.72 -0.26  0.28 -0.54]
    -----------------------------------------------------
    Delta X -> [ 0.82 -0.64 -1.18 -0.36]
    -----------------------------------------------------
    Alpha k -> 0.4709302325581396
    -----------------------------------------------------
    Ax - b = [0.23728814 0.23728814]
    -----------------------------------------------------
    Norm of Ax - b -> 0.335576099546158
    -----------------------------------------------------
    Coordinates -> [1.62542373 0.51186441 0.1        0.72542373]
    -----------------------------------------------------
    Y Values -> [-0.6027907  -0.21662791]
    -----------------------------------------------------
    S Values -> [0.1        0.86395349 1.14651163 0.71744186]
    -----------------------------------------------------
    Functional value -> -3.7627118644067794
    =====================================================
    =====================================================
    Iteration -> 2
    -----------------------------------------------------
    Delta Y -> [-1.39393482  0.38759214]
    -----------------------------------------------------
    Delta S -> [-0.10327448 -0.03819221  0.85021389 -0.88840609]
    -----------------------------------------------------
    Delta X -> [ 0.36319172 -0.45335897 -0.14712088  0.21607083]
    -----------------------------------------------------
    Alpha k -> 0.5505676565825153
    -----------------------------------------------------
    Ax - b = [0.09212905 0.09212905]
    -----------------------------------------------------
    Norm of Ax - b -> 0.13029015873256947
    -----------------------------------------------------
    Coordinates -> [1.8476033  0.23452576 0.01       0.8576033 ]
    -----------------------------------------------------
    Y Values -> [-1.37024612 -0.00323221]
    -----------------------------------------------------
    S Values -> [0.02493963 0.83619522 1.76445103 0.07174419]
    -----------------------------------------------------
    Functional value -> -3.9297323519036342
    =====================================================
    =====================================================
    Iteration -> 3
    -----------------------------------------------------
    Delta Y -> [-0.63161174  0.00873082]
    -----------------------------------------------------
    Delta S -> [-0.02308176  0.16016404  0.23740684 -0.0772428 ]
    -----------------------------------------------------
    Delta X -> [ 0.18450196 -0.26983878 -0.00679224  0.17770973]
    -----------------------------------------------------
    Alpha k -> 0.7039976337515753
    -----------------------------------------------------
    Ax - b = [0.0200639 0.0200639]
    -----------------------------------------------------
    Norm of Ax - b -> 0.028374643789401712
    -----------------------------------------------------
    Coordinates -> [1.99192435 0.02345258 0.00468698 0.99661133]
    -----------------------------------------------------
    Y Values -> [-1.8148993   0.00291427]
    -----------------------------------------------------
    S Values -> [0.00564483 0.97008154 1.96290712 0.00717442]
    -----------------------------------------------------
    Functional value -> -4.007301270963186
    =====================================================
    =====================================================
    Iteration -> 4
    -----------------------------------------------------
    Delta Y -> [-0.18571275 -0.00415659]
    -----------------------------------------------------
    Delta S -> [-0.00503279  0.03177283  0.03770492 -0.00593209]
    -----------------------------------------------------
    Delta X -> [ 0.00699516 -0.02292326 -0.0041358   0.00285936]
    -----------------------------------------------------
    Alpha k -> 0.81
    -----------------------------------------------------
    Ax - b = [0.00200639 0.00200639]
    -----------------------------------------------------
    Norm of Ax - b -> 0.002837464378942338
    -----------------------------------------------------
    Coordinates -> [1.99822000e+00 2.82163804e-03 9.64756641e-04 9.99184752e-01]
    -----------------------------------------------------
    Y Values -> [-1.96532662e+00 -4.52572659e-04]
    -----------------------------------------------------
    S Values -> [1.11532208e-03 9.98677086e-01 1.99684155e+00 1.83553496e-03]
    -----------------------------------------------------
    Functional value -> -3.99926162928753
    =====================================================
    =====================================================
    Iteration -> 5
    -----------------------------------------------------
    Delta Y -> [-0.03478275  0.00023332]
    -----------------------------------------------------
    Delta S -> [-0.00100595  0.00165154  0.00326782 -0.00161628]
    -----------------------------------------------------
    Delta X -> [ 0.00145552 -0.00260584 -0.00085607  0.00059945]
    -----------------------------------------------------
    Alpha k -> 0.81
    -----------------------------------------------------
    Ax - b = [0.00020064 0.00020064]
    -----------------------------------------------------
    Norm of Ax - b -> 0.00028374643790074965
    -----------------------------------------------------
    Coordinates -> [1.99952996e+00 4.76385740e-04 1.94290967e-04 9.99724253e-01]
    -----------------------------------------------------
    Y Values -> [-1.99350065e+00 -2.63585489e-04]
    -----------------------------------------------------
    S Values -> [2.09968824e-04 1.00016347e+00 1.99978259e+00 3.80883145e-04]
    -----------------------------------------------------
    Functional value -> -3.9995363103875485
    =====================================================
    =====================================================
    Iteration -> 6
    -----------------------------------------------------
    Delta Y -> [-0.00652013  0.00022202]
    -----------------------------------------------------
    Delta S -> [-0.00018919 -0.00010113  0.00023819 -0.00033932]
    -----------------------------------------------------
    Delta X -> [ 0.00040756 -0.0004347  -0.00017349  0.00023406]
    -----------------------------------------------------
    Alpha k -> 0.81
    -----------------------------------------------------
    Ax - b = [2.0063903e-05 2.0063903e-05]
    -----------------------------------------------------
    Norm of Ax - b -> 2.8374643794078703e-05
    -----------------------------------------------------
    Coordinates -> [1.99989676e+00 8.51522591e-05 3.81485356e-05 9.99934912e-01]
    -----------------------------------------------------
    Y Values -> [-1.99878196e+00 -8.37471263e-05]
    -----------------------------------------------------
    S Values -> [3.97009992e-05 1.00007246e+00 1.99999696e+00 7.54948516e-05]
    -----------------------------------------------------
    Functional value -> -3.9998786784757434
    =====================================================

    True

```python
ipm.get_metric()
```

    OrderedDict([(0,
                  {'alpha_k': 1.0,
                   'alpha_s': 1.0,
                   'alpha_x': 1.0,
                   'delta_s': 0,
                   'delta_x': 0,
                   'delta_y': 0,
                   'diff': array([1., 1.]),
                   'diff_norm': 1.4142135623730951,
                   'iter_num': 0,
                   'm': None,
                   'obj_val': -3.0,
                   'r': None,
                   's_val': array([1., 1., 1., 1.]),
                   'x_val': array([1., 1., 1., 1.]),
                   'y_val': array([0., 0.])}),
                 (1,
                  {'alpha_k': 0.4709302325581396,
                   'alpha_s': 0.5232558139534884,
                   'alpha_x': 0.7627118644067795,
                   'delta_s': array([-1.72, -0.26,  0.28, -0.54]),
                   'delta_x': array([ 0.82, -0.64, -1.18, -0.36]),
                   'delta_y': array([-1.28, -0.46]),
                   'diff': array([0.23728814, 0.23728814]),
                   'diff_norm': 0.335576099546158,
                   'iter_num': 1,
                   'm': array([[3., 1.],
                          [1., 2.]]),
                   'obj_val': -3.7627118644067794,
                   'r': array([-4.3, -2.2]),
                   's_val': array([0.1       , 0.86395349, 1.14651163, 0.71744186]),
                   'x_val': array([1.62542373, 0.51186441, 0.1       , 0.72542373]),
                   'y_val': array([-0.6027907 , -0.21662791])}),
                 (2,
                  {'alpha_k': 0.5505676565825153,
                   'alpha_s': 0.7268046439184257,
                   'alpha_x': 0.6117418406472391,
                   'delta_s': array([-0.10327448, -0.03819221,  0.85021389, -0.88840609]),
                   'delta_x': array([ 0.36319172, -0.45335897, -0.14712088,  0.21607083]),
                   'delta_y': array([-1.39393482,  0.38759214]),
                   'diff': array([0.09212905, 0.09212905]),
                   'diff_norm': 0.13029015873256947,
                   'iter_num': 2,
                   'm': array([[16.93392593,  0.59246755],
                          [ 0.59246755,  1.60359301]]),
                   'obj_val': -3.9297323519036342,
                   'r': array([-23.37515321,  -0.2043211 ]),
                   's_val': array([0.02493963, 0.83619522, 1.76445103, 0.07174419]),
                   'x_val': array([1.8476033 , 0.23452576, 0.01      , 0.8576033 ]),
                   'y_val': array([-1.37024612, -0.00323221])}),
                 (3,
                  {'alpha_k': 0.7039976337515753,
                   'alpha_s': 0.8359325076546203,
                   'alpha_x': 0.7822195930573059,
                   'delta_s': array([-0.02308176,  0.16016404,  0.23740684, -0.0772428 ]),
                   'delta_x': array([ 0.18450196, -0.26983878, -0.00679224,  0.17770973]),
                   'delta_y': array([-0.63161174,  0.00873082]),
                   'diff': array([0.0200639, 0.0200639]),
                   'diff_norm': 0.028374643789401712,
                   'iter_num': 3,
                   'm': array([[74.36917554,  0.28046771],
                          [ 0.28046771, 12.23409551]]),
                   'obj_val': -4.007301270963186,
                   'r': array([-46.96999583,  -0.07033296]),
                   's_val': array([0.00564483, 0.97008154, 1.96290712, 0.00717442]),
                   'x_val': array([1.99192435, 0.02345258, 0.00468698, 0.99661133]),
                   'y_val': array([-1.8148993 ,  0.00291427])}),
                 (4,
                  {'alpha_k': 0.81,
                   'alpha_s': 0.9,
                   'alpha_x': 0.9,
                   'delta_s': array([-0.00503279,  0.03177283,  0.03770492, -0.00593209]),
                   'delta_x': array([ 0.00699516, -0.02292326, -0.0041358 ,  0.00285936]),
                   'delta_y': array([-0.18571275, -0.00415659]),
                   'diff': array([0.00200639, 0.00200639]),
                   'diff_norm': 0.002837464378942338,
                   'iter_num': 4,
                   'm': array([[3.52902261e+02, 2.41758809e-02],
                          [2.41758809e-02, 1.38935965e+02]]),
                   'obj_val': -3.99926162928753,
                   'r': array([-65.53854866,  -0.58199028]),
                   's_val': array([1.11532208e-03, 9.98677086e-01, 1.99684155e+00, 1.83553496e-03]),
                   'x_val': array([1.99822000e+00, 2.82163804e-03, 9.64756641e-04, 9.99184752e-01]),
                   'y_val': array([-1.96532662e+00, -4.52572659e-04])}),
                 (5,
                  {'alpha_k': 0.81,
                   'alpha_s': 0.9,
                   'alpha_x': 0.9,
                   'delta_s': array([-0.00100595,  0.00165154,  0.00326782, -0.00161628]),
                   'delta_x': array([ 0.00145552, -0.00260584, -0.00085607,  0.00059945]),
                   'delta_y': array([-0.03478275,  0.00023332]),
                   'diff': array([0.00020064, 0.00020064]),
                   'diff_norm': 0.00028374643790074965,
                   'iter_num': 5,
                   'm': array([[1.79161134e+03, 2.82537577e-03],
                          [2.82537577e-03, 5.44358979e+02]]),
                   'obj_val': -3.9995363103875485,
                   'r': array([-62.31717358,   0.1269102 ]),
                   's_val': array([2.09968824e-04, 1.00016347e+00, 1.99978259e+00, 3.80883145e-04]),
                   'x_val': array([1.99952996e+00, 4.76385740e-04, 1.94290967e-04, 9.99724253e-01]),
                   'y_val': array([-1.99350065e+00, -2.63585489e-04])}),
                 (6,
                  {'alpha_k': 0.81,
                   'alpha_s': 0.9,
                   'alpha_x': 0.9,
                   'delta_s': array([-0.00018919, -0.00010113,  0.00023819, -0.00033932]),
                   'delta_x': array([ 0.00040756, -0.0004347 , -0.00017349,  0.00023406]),
                   'delta_y': array([-0.00652013,  0.00022202]),
                   'diff': array([2.0063903e-05, 2.0063903e-05]),
                   'diff_norm': 2.8374643794078703e-05,
                   'iter_num': 6,
                   'm': array([[9.52298558e+03, 4.76307876e-04],
                          [4.76307876e-04, 2.62475368e+03]]),
                   'obj_val': -3.9998786784757434,
                   'r': array([-62.09111687,   0.58275171]),
                   's_val': array([3.97009992e-05, 1.00007246e+00, 1.99999696e+00, 7.54948516e-05]),
                   'x_val': array([1.99989676e+00, 8.51522591e-05, 3.81485356e-05, 9.99934912e-01]),
                   'y_val': array([-1.99878196e+00, -8.37471263e-05])})])

```python
data_rows = []
for iter_iter in ipm.get_metric().values():
    new_dict = OrderedDict()
    new_dict['Iteration'] = iter_iter['iter_num']
    new_dict['x1'] = iter_iter['x_val'][0]
    new_dict['x2'] = iter_iter['x_val'][1]
    new_dict['x3'] = iter_iter['x_val'][2]
    new_dict['x4'] = iter_iter['x_val'][3]
    new_dict['xTs'] = np.dot(iter_iter['x_val'].T, iter_iter['s_val'])
    new_dict['s1'] = iter_iter['s_val'][0]
    new_dict['s2'] = iter_iter['s_val'][1]
    new_dict['s3'] = iter_iter['s_val'][2]
    new_dict['s4'] = iter_iter['s_val'][3]
    new_dict['y1'] = iter_iter['y_val'][0]
    new_dict['y2'] = iter_iter['y_val'][1]
    new_dict['alpha_k'] = iter_iter['alpha_k']
    new_dict['alpha_x'] = iter_iter['alpha_x']
    new_dict['alpha_s'] = iter_iter['alpha_s']
    new_dict['Objective Value'] = iter_iter['obj_val']
    data_rows.append(new_dict)
df = pd.DataFrame(data_rows)
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Iteration</th>
      <th>x1</th>
      <th>x2</th>
      <th>x3</th>
      <th>x4</th>
      <th>xTs</th>
      <th>s1</th>
      <th>s2</th>
      <th>s3</th>
      <th>s4</th>
      <th>y1</th>
      <th>y2</th>
      <th>alpha_k</th>
      <th>alpha_x</th>
      <th>alpha_s</th>
      <th>Objective Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>4.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>-3.000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>1.625424</td>
      <td>0.511864</td>
      <td>0.100000</td>
      <td>0.725424</td>
      <td>1.239870</td>
      <td>0.100000</td>
      <td>0.863953</td>
      <td>1.146512</td>
      <td>0.717442</td>
      <td>-0.602791</td>
      <td>-0.216628</td>
      <td>0.470930</td>
      <td>0.762712</td>
      <td>0.523256</td>
      <td>-3.762712</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>1.847603</td>
      <td>0.234526</td>
      <td>0.010000</td>
      <td>0.857603</td>
      <td>0.321360</td>
      <td>0.024940</td>
      <td>0.836195</td>
      <td>1.764451</td>
      <td>0.071744</td>
      <td>-1.370246</td>
      <td>-0.003232</td>
      <td>0.550568</td>
      <td>0.611742</td>
      <td>0.726805</td>
      <td>-3.929732</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3</td>
      <td>1.991924</td>
      <td>0.023453</td>
      <td>0.004687</td>
      <td>0.996611</td>
      <td>0.050345</td>
      <td>0.005645</td>
      <td>0.970082</td>
      <td>1.962907</td>
      <td>0.007174</td>
      <td>-1.814899</td>
      <td>0.002914</td>
      <td>0.703998</td>
      <td>0.782220</td>
      <td>0.835933</td>
      <td>-4.007301</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4</td>
      <td>1.998220</td>
      <td>0.002822</td>
      <td>0.000965</td>
      <td>0.999185</td>
      <td>0.008807</td>
      <td>0.001115</td>
      <td>0.998677</td>
      <td>1.996842</td>
      <td>0.001836</td>
      <td>-1.965327</td>
      <td>-0.000453</td>
      <td>0.810000</td>
      <td>0.900000</td>
      <td>0.900000</td>
      <td>-3.999262</td>
    </tr>
    <tr>
      <th>5</th>
      <td>5</td>
      <td>1.999530</td>
      <td>0.000476</td>
      <td>0.000194</td>
      <td>0.999724</td>
      <td>0.001666</td>
      <td>0.000210</td>
      <td>1.000163</td>
      <td>1.999783</td>
      <td>0.000381</td>
      <td>-1.993501</td>
      <td>-0.000264</td>
      <td>0.810000</td>
      <td>0.900000</td>
      <td>0.900000</td>
      <td>-3.999536</td>
    </tr>
    <tr>
      <th>6</th>
      <td>6</td>
      <td>1.999897</td>
      <td>0.000085</td>
      <td>0.000038</td>
      <td>0.999935</td>
      <td>0.000316</td>
      <td>0.000040</td>
      <td>1.000072</td>
      <td>1.999997</td>
      <td>0.000075</td>
      <td>-1.998782</td>
      <td>-0.000084</td>
      <td>0.810000</td>
      <td>0.900000</td>
      <td>0.900000</td>
      <td>-3.999879</td>
    </tr>
  </tbody>
</table>
</div>

```python
x = np.linspace(0, 50)
eqs = []
eqs.append((x, 2 - x))

x_parallel = [1]
graph_2d(eqs, x, x_ipm_iter, x_parallel=x_parallel, xlim=3, ylim=3)
```

```python
A = np.asarray(np.matrix('4 3 1 0 0; 4 1 0 1 0; 4 -1 0 0 1'), dtype=np.float64)
b = np.asarray([12, 8, 8], dtype=np.float64)
c = np.asarray([-3, -2, 0, 0, 0], dtype=np.float64)

epsilon = 1e-3
ipm = InteriorPointMethod(A, b, c)
x_ipm, x_ipm_iter = ipm.solve(theta=0.9, gamma=0.1, epsilon=epsilon)
ipm.print_metric()
```