# Internal stability of Runge-Kutta methods, revisited

We previously looked at internal stability of some classes of Runge--Kutta methods in [this notebook](https://github.com/ketch/nodepy/blob/master/examples/Internal_stability.ipynb), also posted on my blog [here](http://www.davidketcheson.info/2012/10/11/Internal_stability.html).  There I gave a formula for the internal stability polynomials based on the Butcher coefficients of the method.

It turns out that internal stability depends on the manner in which a Runge--Kutta method is implemented.  That is, two mathematically equivalent implementations of the same method can have quite different internal stability polynomials and internal amplification factors.  This is rather surprising, since the *stability polynomial* that we traditionally study doesn't depend on the particular implementation.

A general form for implementations of Runge--Kutta methods is the *modified Shu-Osher form*, which was introduced in order to facilitate analysis of strong stability preserving RK methods:

\begin{align}
y_1 & = u^n
\end{align}

```python
from nodepy import rk
import numpy as np
from matplotlib import pyplot as plt

rk4 = rk.loadRKM('RK44')
ssprk4 = rk.loadRKM('SSP104')
print(rk4)
ssprk4.print_shu_osher()
```

## Absolute stability regions

First we can use NodePy to plot the region of absolute stability for each method.  The absolute stability region is the set

$\\{ z \in C : |\phi (z)|\le 1 \\}$

where $\phi(z)$ is the *stability function* of the method:

$1 + z b^T (I-zA)^{-1}$

If we solve $u'(t) = \lambda u$ with a given method, then $z=\lambda \Delta t$ must lie inside this region or the computation will be unstable.

```python
p,q = rk4.stability_function()
print(p)
h1=rk4.plot_stability_region()
```

```python
p,q = ssprk4.stability_function()
print(p)
h2=ssprk4.plot_stability_region()
```

# Internal stability

The stability function tells us by how much errors from one step are amplified in the next one.  This is important since we introduce truncation errors at every step.  However, we also introduce roundoff errors at the each stage within a step.  Internal stability tells us about the growth of those.  Internal stability is typically less important than (step-by-step) absolute stability for two reasons:

 - Roundoff errors are typically much smaller than truncation errors, so moderate amplification of them typically is not significant
 - Although the propagation of stage errors within a step is governed by internal stability functions, in later steps these errors are propagated according to the (principal) stability function

Nevertheless, in methods with many stages, internal stability can play a key role.

Questions: *In the solution of PDEs, large spatial truncation errors enter at each stage.  Does this mean internal stability becomes more significant?  How does this relate to stiff accuracy analysis and order reduction?*

## Internal stability functions

We can write the equations of a Runge-Kutta method compactly as

\begin{align}
y & = u^n e + h A F(y) \\
u^{n+1} & = u^n + h b^T F(y),
\end{align}

where $y$ is the vector of stage values, $u^n$ is the previous step solution, $e$ is a vector with all entries equal to 1, $h$ is the step size, $A$ and $b$ are the coefficients in the Butcher tableau, and $F(y)$ is the vector of stage derivatives.  In floating point arithmetic, roundoff errors will be made at each stage.  Representing these errors by a vector $r$, we have

$y = u^n e + h A F(y) + r.$

Considering the test problem $F(y)=\lambda y$ and solving for $y$ gives

$y = u^n (I-zA)^{-1}e + (I-zA)^{-1}r,$

where $z=h\lambda$.  Substituting this result in the equation for $u^{n+1}$ gives

$u^{n+1} = u^n (1 + zb^T(I-zA)^{-1}e) + zb^T(I-zA)^{-1}r = \psi(z) u^n + \theta(z)^T r.$

Here $\psi(z)$ is the *stability function* of the method, that we already encountered above.  Meanwhile, the vector $\theta(z)$ contains the *internal stability functions* that govern the amplification of roundoff errors $r$ within a step:

$\theta(z) = z b^T (I-zA)^{-1}.$

Let's compute $\theta$ for the classical RK4 method:

```python
theta=rk4.internal_stability_polynomials()
theta
```

```python
for theta_j in theta:
    print(theta_j)
```

Thus the roundoff errors in the first stage are amplified by a factor $z^4/24 + z^3/12 + z^2/6 + z/6$, while the errors in the last stage are amplified by a factor $z/6$.

## Internal instability

Usually internal stability is unimportant since it relates to amplification of roundoff errors, which are very small.  Let's think about when things can go wrong in terms of internal instability.  If $|\theta(z)|$ is of the order $1/\epsilon_{machine}$, then roundoff errors could be amplified so much that they destroy the accuracy of the computation.  More specifically, we should be concerned if $|\theta(z)|$ is of the order $tol/\epsilon_{machine}$ where $tol$ is our desired error tolerance.  Of course, we only care about values of $z$ that lie inside the absolute stability region $S$, since internal stability won't matter if the computation is not absolutely stable.

We can get some idea about the amplification of stage errors by plotting the curves $|\theta(z)|=1$ along with the stability region.  Ideally these curves will all lie outside the stability region, so that all stage errors are damped.

```python
rk4.internal_stability_plot()
```

```python
ssprk4.internal_stability_plot()
```

For both methods, we see that some of the curves intersect the absolute stability region, so some stage errors are amplified.  But by how much?  We'd really like to know the maximum amplification of the stage errors under the condition of absolute stability.  We therefore define the *maximum internal amplification factor* $M$:

$M = \max_j \max_{z \in S} |\theta_j(z)|$

```python
print(rk4.maximum_internal_amplification())
print(ssprk4.maximum_internal_amplification())
```

We see that both methods have small internal amplification factors, so internal stability is not a concern in either case.  This is not surprising for the method with only four stages; it is a surprisingly good property of the method with ten stages.

Questions: *Do SSP RK methods always (necessarily) have small amplification factors?  Can we prove it?*

Now let's look at some methods with many stages.

## Runge-Kutta Chebyshev methods

The paper of Verwer, Hundsdorfer, and Sommeijer deals with RKC methods, which can have very many stages.  The construction of these methods is implemented in NodePy, so let's take a look at them.  The functions `RKC1(s)` and `RKC2(s)` construct RKC methods of order 1 and 2, respectively, with $s$ stages.

```python
s=4
rkc = rk.RKC1(s)
print(rkc)
```

```python
rkc.internal_stability_plot()
```

It looks like there could be some significant internal amplification here.  Let's see:

```python
rkc.maximum_internal_amplification()
```

Nothing catastrophic.  Let's try a larger value of $s$:

```python
s = 20
rkc = rk.RKC1(s)
rkc.maximum_internal_amplification(formula='pow')
```

As promised, these methods seem to have good internal stability properties.  What about the second-order methods?

```python
s = 20
rkc = rk.RKC2(s)
rkc.maximum_internal_amplification(formula='pow')
```

Again, nothing catastrophic.  We could take $s$ much larger than 20, but the calculations get to be rather slow (in Python) and since we're using floating point arithmetic, the accuracy deteriorates.

Remark: *we could do the calculations in exact arithmetic using Sympy, but things would get even slower.  Perhaps there are some optimizations that could be done to speed this up.  Or perhaps we should use Mathematica if we need to do this kind of thing.*

Remark 2: *of course, for the RKC methods the internal stability polynomials are shifted Chebyshev polynomials.  So we could evaluate them directly in a stable manner using the three-term recurrence (or perhaps scipy's special functions library).  This would also be a nice check on the calculations above.*

## Other methods with many stages

Three other classes of methods with many stages have been implemented in NodePy:

 - SSP families
 - Integral deferred correction (IDC) methods
 - Extrapolation methods

### SSP Families

```python
s = 20
ssprk = rk.SSPRK2(s)
ssprk.internal_stability_plot()
ssprk.maximum_internal_amplification(formula='pow')
```

```python
s = 25 # # of stages
ssprk = rk.SSPRK3(s)
ssprk.internal_stability_plot()
ssprk.maximum_internal_amplification(formula='pow')
```

The SSP methods seem to have excellent internal stability properties.

### IDC methods

```python
p = 6 #order
idc = rk.DC(p-1)
print(len(idc))
idc.internal_stability_plot()
idc.maximum_internal_amplification(formula='pow')
```

IDC methods also seem to have excellent internal stability.

### Extrapolation methods

```python
p = 6 #order
ex = rk.extrap(p)
print(len(ex))
ex.internal_stability_plot()
ex.maximum_internal_amplification()
```

Not so good.  Let's try a method with even more stages (this next computation will take a while; go stretch your legs).

```python
p = 10 #order
ex = rk.extrap(p)
print(len(ex))
ex.maximum_internal_amplification()
```

Now we're starting to see something that might cause trouble, especially since such high order extrapolation methods are usually used when extremely tight error tolerances are required.  Internal amplification will cause a loss of about 5 digits of accuracy here, so the best we can hope for is about 10 digits of accuracy in double precision.  Higher order extrapolation methods will make things even worse.  How large are their amplification factors?  (Really long calculation here...)

```python
pmax = 12
ampfac = np.zeros(pmax+1)
for p in range(1,pmax+1):
    ex = rk.extrap(p)
    ampfac[p] = ex.maximum_internal_amplification()[0]
    print(p, ampfac[p])
```

```python
plt.semilogy(ampfac,linewidth=3)
plt.xlabel(r"Order $p$")
plt.ylabel(r"Amplification factor")
```

We see roughly geometric growth of the internal amplification factor as a function of the order $p$.  It seems clear that very high order extrapolation methods applied to problems with high accuracy requirements will fall victim to internal stability issues.


# Ecuaciones Diferenciales Ordinarias

Como es sabido en la [historia de la computación](https://youtu.be/jWum9IJPyks), uno de los problemas que motivo más fuertemente la construcción de computadoras digitales programables fue la solucionar las ecuaciones diferenciales ordinarias que describen la trayectoria de misiles balísticos (esto, claro, durante la segunda guerra mundial).

Dichas ecuaciones, así como muchas otras ecuaciones diferenciales ordinarias, son imposibles de resolver analíticamente, por lo que la única manera de resolverlas es acudir a métodos numéricos que nos permitan obtener una solución aproximada al problema.

Existen [libros enteros](https://books.google.com.mx/books?id=lZysESfSWwgC&printsec=frontcover&dq=leveque+finite+differences&hl=en&sa=X&ved=2ahUKEwi9hJaQ-arrAhVN5awKHXjgDnkQ6AEwAHoECAUQAg#v=onepage&q=leveque%20finite%20differences&f=false), y cursos también, dedicados al estudio de la solución numérica de ecuaciones diferenciales tanto ordinarias como parciales.

## Preludio Computacional 1: alocación de memoria para arreglos

Durante el curso, hemos, en muchas ocasiones, construido arreglos que representan a los elementos de una sucesión, **iniciando con un arreglo que contiene un solo elemento y añadiéndole elementos uno a uno**. Esto se puede ya que, como ya mencionamos anteriormente, los arreglos en Julia son **dinámicos**.

En general, es mala idea tratar con arreglos dinámicos pues es computacionalmente costoso ir asignando o  **alocando** memoria a elementos nuevos. Lo más conveniente, de ser posible, es construir un arreglo de la longitud que necesitemos desde un inicio. Así, podemos tratar al arreglo de manera **estática**: solo cambiamos sus elementos a nuestro gusto pero no cambiamos su longitud

En Julia, existen varias funciones para crear arreglos:
* La función `zeros(n)` crea un arreglo 1D de longitud $n$ en la que cada elemento es un $0$.

* La función `ones(n)` crea un arreglo 1D de longitud $n$ en la que cada elemento es un $1$.

* La función `rand(n)` crea un arreglo 1D de longitud $n$ en la que cada elemento es un número aleatorio en el intervalo $[0,1]$

```julia
arr1 = zeros(10)
println("ceros")
println(arr1)
arr2 = ones(10)
println("unos")
println(arr2)
arr3 = rand(10)
println("aleatorios")
println(arr3)
```

    ceros
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
    unos
    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
    aleatorios
    [0.984644661709182, 0.9420071430380894, 0.6767320255672735, 0.15240316799622566, 0.49270156095619777, 0.9012042440681045, 0.5038629939061074, 0.2226911769835742, 0.8656047009619021, 0.9771549845598295]

Noten también que `rand()` te genera un solo número aleatorio

```julia
println(rand())
```

    0.5692904802083327

Por default, los arreglos creados con `zeros` y `ones` tienen elementos de tipo `Float64`. Sin embargo, podemos especificar que sean de otro tipo utilizando la sintaxis `zeros(T,n)` o `ones(T,n)` con `T` el tipo de interés

```julia
arr4 = zeros(5)
println(arr4)
println(typeof(arr4))
arr5 = zeros(Int32,5)
println(arr5)
println(typeof(arr5))
```

    [0.0, 0.0, 0.0, 0.0, 0.0]
    Array{Float64,1}
    Int32[0, 0, 0, 0, 0]
    Array{Int32,1}

Podemos usar la prealocación a la hora de construir sucesiones recursivas como las que usamos en la mayoría de métodos numéricos

```julia
# crea $n$ términos de la sucesión de fibonacci
function fibonacciPrealocado(n)
    # creamos arreglo para guardar los $n$ términos
    final = zeros(Int64,n)
    # definimos los primeros elementos
    final[1] = 1
    final[2] = 1
    # creamos los demás, iniciando en el tercero
    for i in 3:n
        final[i] = final[i-1] + final[i-2]
    end
    return final
end
```

    fibonacciPrealocado (generic function with 1 method)

```julia
fibonacciPrealocado(30)
```

    30-element Array{Int64,1}:
          1
          1
          2
          3
          5
          8
         13
         21
         34
         55
         89
        144
        233
          ⋮
       4181
       6765
      10946
      17711
      28657
      46368
      75025
     121393
     196418
     317811
     514229
     832040

## Preludio Computacional II: tuplas

Hasta ahora, hemos trabajado con objetos iterables como listas, arreglos y rangos, los cuales se pueden describir a grandes rasgos como sucesiones ordenadas de objetos. Sabemos que, en caso de las listas y los arreglos podemos cambiar sus elementos

```julia
nombres = ["aldo","david","crhistian"]
nombres[1] = "sayeg"
println(nombres)
```

    ["sayeg", "david", "crhistian"]

Podemos cambiar sus entradas por que las listas y los arreglos son **objetos mutables**, es decir, que pueden cambiar sus elementos. Existen objetos **no mutables** que también representan sucesiones de objetos pero que no podemos cambiar sus elementos. Dichos objetos son las **tuplas** (de tipo `Tuple` en Julia). Una tupla se define con la siguiente sintaxis

```julia
tup1 = (obj1,obj2,obj3)
```

```julia
tupNombres = ("aldo","david","crhistian")
println(tupNombres)
println(typeof(tupNombres))
```

    ("aldo", "david", "crhistian")
    Tuple{String,String,String}

```julia
# al intentar cambiar un elemento arrojará un error
tupNombres[1] = "sayeg"
```

`length` funciona de igual manera con las tuplas

```julia
println(length(tupNombres))
```

    3

Nuevamente, por ser inmutable, no podemos añadirle objetos y `push!` fracasará

```julia
push!(tupNombres,"miguel")
```

### Preludio Computacional III: productos cartesianos

En las siguientes clases trabajaremos con objetos como campos vectoriales y otros análogos, los cuales viven en espacios de varias dimensiones.

Muchas veces nos va a interesar poder obtener el producto cartesiano de de dos arreglos, es decir, un objeto de la forma

$$\{(x,y) \; \mid \; x \in xs \; , \; y \in ys\}$$

Podemos utilizar comprensión de arreglos en Julia para crear ese arreglo con la siguiente sintaxis

```julia
prodCart = [[x,y] for x in xs for y in ys]
```

```julia
# Ejemplo simple
xs = [1,2,3]
ys = [8,9]
# se empieza a iterar por el úlimo arreglo
prodCart = [[x,y] for x in xs for y in ys]
display(prodCart)
```

    6-element Array{Array{Int64,1},1}:
     [1, 8]
     [1, 9]
     [2, 8]
     [2, 9]
     [3, 8]
     [3, 9]

Podemos hacer esto mismo con tuplas:

```julia
nombres = ["aldo","david","miguel"]
apellidos = ["perez","lopez","medrano"]
# ahora tendremos tuplas en el producto cartesiano
prodCar = [(x,y) for x in nombres for y in apellidos]
display(prodCar)
```

    9-element Array{Tuple{String,String},1}:
     ("aldo", "perez")
     ("aldo", "lopez")
     ("aldo", "medrano")
     ("david", "perez")
     ("david", "lopez")
     ("david", "medrano")
     ("miguel", "perez")
     ("miguel", "lopez")
     ("miguel", "medrano")

## Preludio Matemático: Ecuciones Diferenciales I

Sea $x(t): I \subseteq \mathbb{R} \to \mathbb{R}$ una función de clase $C^{1}$, es decir, una función derivable en todo $I$ con derivada continua en $I$. Una ecuación diferencial de primer orden es una ecuación **funcional**  de la siguiente forma

$$
\dot{x}(t) = f(x(t),t)
$$

Donde $\dot{x}(t) = \frac{dx}{dt} (t)$ y $f$ una función continua de dos variables. Notemos que, a diferencia de una ecuación **algebraica** donde las variables representan números o miembros de algún otra estructura algebraica "simple" (vectores, matrices, etc) que solo se operan con operadores algebraicos como suma, producto etc, en una ecuación diferencial algunas variables representan **funciones**.

Más aún, las operaciones que se les realizan a nuestros objetos no son operaciones algebraicas simples si no operaciones más delicadas una derivada o la aplicación de una función arbitraria.

La técnica más estándar para resolver una ecuación diferencial consiste en utilizar el teorema fundamental del cálculo para integrar ambos lados de la ecuación respecto al tiempo. Sin embargo, a la hora de realizar dicha integración, es probable que surgan **constantes de integración**. Para poder determinar dichas constantes, será necesario tener una **condición** sobre la función $x$, es decir, conocer un punto en el que sepamos su valor.

$$
x(t_i) = C
$$

### Problemas de valores iniciales

Aunque el punto $t_i$ puede ser un punto arbitrario del dominio de la función, en general, lo que tenemos que es que ese tiempo es un tiempo que llamamos **inicial** y denotamos $t_0$, y la solución $x(t)$ nos interesa conocerla para tiempos $t \geq t_0$. Cuando ese es el caso, decimos que tenemos un **problema de valores iniciales** descrito por las dos ecuaciones

$$
\begin{cases}
\dot{x}(t) = f(x(t),t) \\
x(t_0) = x_0 \\
\end{cases}
$$

### Campo de direcciones o "retrato fase"

Antes de resolver una ecuación diferencial, podemos darnos una idea del comportamiento que tendrán las soluciones si graficamos el campo de direcciones o retrato fase de la curva: el campo vectorial de las pendientes de las funciones que son solución a la ecuación diferencial pero nonecesariamente a a la condición inicial. Dicho campo corresponde al campo vectorial $P:\mathbf{R}^2 \to \mathbf{R}^2$ definido por

$$
P(t,x) = (1,f(x,t))
$$

Recordando el tutorial de Plots para Julia, podemos graficar un campo vectorial utilizando la función `quiver` del paquete Plots. La sintaxis es

```julia
quiver(xs,ys,vectorfield=(us,vs))
```

La función hace una grafica consistente en flechas entre los puntos `(xs[i],ys[i])` y `(xs[i]+us[i],ys[i]+vs[i])`

### Ejemplo 1:

Sea $\dot{x}(t) = \alpha x(t) $. El campo vectorial entonces es

$$
P(t,x) = ( 1,\alpha x(t) )
$$

```julia
using Plots, LinearAlgebra
```

```julia
`
```

    campoDireccionesEjemplo1 (generic function with 1 method)

```julia
campoDireccionesEjemplo1(2)
```

## El método de Euler

El método más sencillo para resolver numéricamente un problema de valores iniciales es el **método de Euler**, que consiste en aproximar la derivada $\dot{x}$ con una diferencia finita hacia adelante. Así, obtenemos

$$
\frac{x(t+h) - x(t)}{h} = f(x(t),t)
$$

Despejando encontramos que

$$
x(t+h) = x(t) + h \; f(x(t),t)
$$

Es claro que, si conocemos $x(t_0) = x_0$ y $t_0$, esta regla nos permite calcular $x(t_0 +h)$ y, al tener $x(t_0+h)$ y también $t_0 + h$, posteriormente podemos calcular $x(t_0 + 2h)$ y así sucesivamente para obtener $x(t_0 + k h)$  para cualquier natural $k$. Planteandolo en términos de una sucesión recursiva, sea $t_k = t+ k h$ y entonces $x_{k} := x(t_k)$, entonces la sucesión recursiva del método de Euler es

$$
x_k = \begin{cases}
x_0 & k = 0\\
x_{k-1} + h f(x_{k-1},t_{k-1}) & k > 0\\
\end{cases}
$$

### Ejercicio 1

implementa una función `primerEuler(f,x_0,t_0,t_f,h)` con `f` la función **vectorial** tal que $\dot{x}(t) = f(x(t),t)$, `x_0`, `t_0` la condición inicial, `t_f` el tiempo máximo al que queremos obtener la solución y `h` el tamaño de paso. La función debe de implementar el método de euler y debe de regresar el arreglo de los tiempos `[t_0,t_1, t_2,....,t_n]` con $t_{n}< t_f < t_{n} + h$ y el de la solución a la ecuación diferencial `[x_0,x_1,...,x_n]`.

**Sugerencia:** utiliza una tupla para regresar ambos valores.

### Ejercicio 2

Utiliza tu función `primerEuler` para resolver las siguientes ecuaciones diferenciales (con las condiciones inciales y tiempo final de tu preferencia):

1. $\dot{x}(t) = 0.3 x(t) $

2. $\dot{x}(t) = 3t^2 - 5t $

3. $\dot{x}(t) = x(t)(1-x(t))$

Grafica el campo de direcciones para cada ecuación. Realiza una gráfica de la solución obtenida y compárala con la solución analítica en la misma gráfica

**Sugerencia**: ten cuidado pues no las condiciones iniciales que pueden utilizarse no son arbitrarias.

### Ejercicio 3

Podemos definir el error absoluto de la solución numérica como la diferencia entre el último valor de la función y su correspondiente analítico, es decir

$$
\text{error} = \mid x_n - x_{\text{analítica}}(t_n) \mid
$$

Para cada una de las funciones del ejercicio anterior, analiza el error absoluto como función del parámetro $h$. Realiza su gráfica y menciona algo sobre su comportamiento

## Ecuaciones de orden mayor: reducción

En la práctica del día a día, en particular en la física, por lo general tratamos con ecuaciones diferenciales de orden mayor a uno. En general,una ecuación diferencial de orden $l$ es una ecuación funcional de la forma

$$
y^{(l)} (t) = g(y^{(l-1)} (t),y^{(l-2)} (t), \ldots, y^{(1)}(t),t)
$$

Donde $y^{(l)} (t) = \frac{d^l y}{dt^l}$ es la $l$-ésima derivada de $y(t)$. Aunque podríamos desarrollar un método para atacar ecuaciones diferenciales de ese orden, existe un [**truco**](http://mathonline.wikidot.com/converting-nth-order-odes-to-systems-of-n-first-order-odes) para convertir una ecuación de orden $n$ en un sistema de ecuaciones de primer orden con $n$ variables. Definimos

$$
\begin{align}
 x_1(t) &= y(t) \\
 x_2(t) &= y^{(1)}(t) \\
 &\vdots \\
 x_i(t) &= y^{(i-1)}(t) \\
 &\vdots \\
 x_l(t) &= y^{(l-1)}(t)
\end{align}
$$

Entonces la ecuación diferencial se reduce al siguiente sistema

$$
\begin{align}
 \dot{x_1}(t) &= y^{(1)}(t) = x_2(t) \\
 \dot{x_2}(t) &= y^{(2)}(t) = x_3(t) \\
 &\vdots \\
 \dot{x_i}(t) &= y^{(i)}(t) = x_{i+1}(t) \\
 &\vdots \\
 \dot{x}_{l-1}(t) &= y^{(l-1)}(t) = x_{l}(t) \\
\dot{x_l}(t) &= y^{(l)}(t)
\end{align}
$$

Pero

$$
y^{(l)}(t) = g(y^{(l-1)} (t),y^{(l-2)} (t), \ldots, y^{(1)}(t),t) = g(x_{l-1}(t),x_{l-2}(t),\ldots,x_1(t),t)
$$

Por lo que el sistema se vuelve

$$
\begin{align}
 \dot{x_1}(t) &= x_2(t) \\
 \dot{x_2}(t) &= x_3(t) \\
 &\vdots \\
 \dot{x_i}(t) &= x_{i+1}(t) \\
 &\vdots \\
 \dot{x}_{l-1}(t) &= x_{l}(t) \\
\dot{x_l}(t) &= g(x_{l-1}(t),x_{l-2}(t),\ldots,x_1(t),t)
\end{align}
$$

### Ejercicio 4

Analíticamente, convierte las siguientes ecuaciones diferenciales de orden $n$ en sistemas de $n$ ecuaciones diferenciales:

1. $y''(t) = -\frac{10}{m}$

2. $y'''(t) = t + 2y(t) + 3y''(t)$

3. $y''(t) = -\frac{k}{m} y(t) - \frac{c}{m} y'(t)$

## Método de euler vectorial

Sea entonces **el vector de funciones**  $\mathbf{x}(t) : \mathbb{R} \to \mathbb{R}^{n}$

$$\mathbf{x}(t) := (x_1(t),\ldots,x_{l}(t))$$

y la función vectorial $\mathbf{f}(\mathbf{x}(t),t) : \mathbb{R}^{n+1} \to \mathbb{R}^{n}$

$$\mathbf{f}(\mathbf{x}(t),t) = (x_2(t),\ldots,x_{l}(t),g(x_{l-1}(t),x_{l-2}(t),\ldots,x_1(t),t))$$

Podemos escribir el sistema de ecuaciones diferenciales de forma vectorial como

$$
\dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t),t)
$$

Que nuevamente admite una solución mediante el método de euler al aproximar la derivada del vector $\mathbf{x}(t)$ con una diferencia hacia adelante:

$$
\mathbf{x}(t+h) = \mathbf{x}(t) + h \; \mathbf{f}(\mathbf{x}(t),t)
$$

### Ejercicio 5

Implementa una función `eulerVec(f,x_0,t_0,t_f,h)` con `f` la función **vectorial** tal que $\dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t),t)$, `x_0` (un vector de $n$ entradas) y `t_0` la condición inicial, `t_f` el tiempo máximo al que queremos obtener la solución y `h` el tamaño de paso. La función debe de implementar el método de euler y debe de regresar el arreglo de los tiempos `[t_0,t_1, t_2,....,t_n]` con $t_{n}< t_f < t_{n} + h$ y el de la solución a la ecuación diferencial `[x_0,x_1,...,x_n]`.

**Sugerencia:** utiliza una tupla para regresar ambos valores.

### Ejercicio 6

Utiliza tu función `eulerVectorial` para resolver los sistemas de ecuaciones obtenidos en el ejercicio 4. Nuevamente, puedes escoger arbitrariamente las condiciones iniciales, el tiempo final y las constantes de las ecucaiones ($m$,$k$,etc).

Grafica  el **espacio fase** de tus soluciones, es decir, el conjunto

$$\{ \mathbf{x}(t) \; \mid \; t_0 \leq t \leq t_f\} \subset \mathbb{R}^2, \mathbb{R}^3$$

Si es posible, compara con la solución analítica

### Ejercicio 7

El inciso 1 y 3 del ejercicio 4, la variable $y(t)$ corresponde a la posición como función del tiempo para un objeto en **caída libre** o en un **oscilador armónico amortiguado**. Para unas condiciones iniciales, tiempo final y valor de los parámetros, soluciona el sistema y realiza una **animación** del movimiento del objeto. Verifica que coincida con las soluciones que has visto en los cursos de mecánica.


```python
import numpy as np
import math
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
x_current = 1
f = 0
step = 0.01
x_start = 0
x_end = 1
t_start = 0
t_end = 1
u_dirichlet = 0
u0_func = lambda x: 1 * (0.3 <= x < 0.4)  # 1 * True = 1 / 1 * False = 0
```

```python
x = np.arange(x_start, x_end+step, step)
u0 = np.vectorize(u0_func)(x)
time = np.arange(t_start, t_end+step, step)
```

```python
def build_rhs(step, f, left_dirichlet, u_prev):
    dim = u_prev.size
    rhs = f * np.ones(dim)
    rhs[0] + left_dirichlet
    return rhs + step * u_prev
```

```python
def indexer(i, j):
    if i == j:
        return step * x_current + step
    elif i == j+1:
        return - step * x_current
    else:
        return 0

def build_mat(dim):
    return np.fromfunction(np.vectorize(indexer), (dim, dim))
```

```python
u_prev = u0
for t in time:
    rhs = build_rhs(step, f, u_dirichlet, u_prev)
    mat = build_mat(u_prev.size)
    u_prev = np.linalg.solve(mat, rhs)
    if t == 0.5 or t == 0.1:
        plt.plot(u_prev)
plt.plot(u_prev)
plt.show()
```

$$h b_{x} u_{t,1} - h b_{x} u_g(0, t) + h b_{y} u_{t,1} - h b_{y} u_{t-1,1} = f1 := \int_{T_{t,1}} f(x) dx$$
$$h b_{x} u_{t,j} - h b_{x} u_{t,j-1} + h b_{y} u_{t,j} - h b_{y} u_{t-1,j} = fj := \int_{T_{t,j}} f(x) dx$$

daraus folgt:

\begin{equation}

    \begin{pmatrix}
        h b_{x} + h b_{y} \\
        -h b_{x} & h b_{x} + h b_{y} \\
        & \ddots & \ddots \\
        & -h b_{x} & h b_{x} + h b_{y} \\
    \end{pmatrix}

    \begin{pmatrix}
    	u_{t,1} \\
        u_{t,2} \\
        \vdots  \\
        u_{t,j} \\
        \vdots  \\
        u_{t,n} \\
    \end{pmatrix}

    =

    \begin{pmatrix}
        f_{1} + h b_{y} u_{t-1,1} + h b_{x} u_g(0, t) \\
        f_{2} + h b_{y} u_{t-1,2} \\
        \vdots  \\
        f_{j} + h b_{y} u_{t-1,j} \\
        \vdots  \\
        f_{n} + h b_{y} u_{t-1,n} \\
    \end{pmatrix}
\end{equation}

```python

```


# Regression

### Preliminaries

- Goal
  - Introduction to Bayesian (Linear) Regression
- Materials
  - Mandatory
    - These lecture notes
  - Optional
    - Bishop pp. 152-158

### Regression - Illustration

Given a set of (noisy) data measurements, find the 'best' relation between an input variable $x \in \mathbb{R}^D$ and input-dependent outcomes $y \in \mathbb{R}$

### Regression vs Density Estimation

- Observe $N$ IID data **pairs** $D=\{(x_1,y_1),\dotsc,(x_N,y_N)\}$ with $x_n \in \mathbb{R}^D$ and $y_n \in \mathbb{R}$.

- Assume that we are interested in (a model for) the responses $y_n$ for **given inputs** $x_n$?, I.o.w. we are interested in building a model for the conditional distribution $p(y|x)$.

- Note that, since $p(x,y)=p(y|x)\, p(x)$, building a model $p(y|x)$ is similar to density estimation with the assumption that $x$ is drawn from a uniform distribution.

### Bayesian Linear Regression

- Next, we discuss (1) model specification, (2) Inference and (3) a prediction application for a Bayesian linear regression problem.

#####  1. Model Specification

- In a _regression_ model, we try to 'explain the data' by a purely deterministic term $f(x_n,w)$, plus a purely random term $\epsilon_n$ for 'unexplained noise',

    $$
    y_n  = f(x_n,w) + \epsilon_n
    $$

-  In a _linear regression_ model, i.e., linear w.r.t. the parameters $w$, we assume that
$$f(x_n,w)= \sum_{j=0}^{M-1} w_j \phi_j(x_n) = w^T \phi(x_n)$$
where $\phi_j(x)$ are called basis functions.
  - For notational simplicity, from now on we will assume $f(x_n,w) = w^T x_n$.

-  In _ordinary linear regression_ , the noise process $\epsilon_n$ is zero-mean Gaussian with constant variance, i.e.
$$
y_n  = w^T x_n  + \mathcal{N}(0,\beta^{-1}) \,.
$$

- Hence, given a data set  $D=\{(x_1,y_1),\dotsc,(x_N,y_N)\}$, the likelihood for an ordinary linear regression model is

$$\begin{align*}
p(y\,|\,\mathbf{X},w,\beta) &= \mathcal{N}(y\,|\,\mathbf{X} w,\beta^{-1} \mathbf{I}) \\
  &= \prod_n \mathcal{N}(y_n\,|\,w^T x_n,\beta^{-1}) \tag{B-3.10}
\end{align*}$$

where we defined $w = (w_0,w_1,\ldots,w_{D-1})^T$, the $(N\times D)$-dim matrix $\mathbf{X}  = \left( x_1 ,x_2 , \ldots ,x_n \right)^T$ and $y = (y_1,y_2,\dots,y_N)^T$.

- For full Bayesian learning we should also choose a prior $p(w)$:

$$\begin{equation}
p(w\,|\,\alpha) = \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \tag{B-3.52}
\end{equation}$$

- For simplicity, we will assume that $\alpha$ and $\beta$ are known.

##### 2. Inference

- We'll do Bayesian inference for the parameters $w$:
$$\begin{align*}
p(w|D) &\propto p(D|w)\cdot p(w) \\
   &= \mathcal{N}(y\,|\,\mathbf{X} w,\beta^{-1} \mathbf{I}) \cdot \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \\
   &\propto \exp \left\{ \frac{\beta}{2} \left( {y - \mathbf{X}w } \right)^T \left( {y - \mathbf{X}w } \right)  + \frac{\alpha}{2}w^T w \right\}\tag{B-3.55} \\
   &\propto \mathcal{N}\left(w\,|\,m_N,S_N \right) \tag{B-3.49}
\end{align*}$$
with
$$\begin{align*}
m_N &= \beta S_N \mathbf{X}^T y  \tag{B-3.53}\\
S_N &= \left(\alpha \mathbf{I} + \beta \mathbf{X}^T \mathbf{X}\right)^{-1} \tag{B-3.54}
\end{align*}$$

- Note that B-3.53 and B-3.54 combine to
$$
m_N = \left(\frac{\alpha}{\beta}\mathbf{I} + \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T y\,.
$$

 - (Bishop Fig.3.7) Illustration of sequential Bayesian learning for a simple linear model of the form $y(x, w) =
w_0 + w_1 x$. (Bishop Fig.3.7, detailed description at Bishop, pg.154.)

##### 3. Application: predictive distribution

- Assume we are interested in the distribution $p(y_\bullet \,|\, x_\bullet, D)$ for a new input $x_\bullet$. This can be worked out as (exercise B-3.10)

$$\begin{align*}
p(y_\bullet \,|\, x_\bullet, D) &= \int p(y_\bullet \,|\, x_\bullet, w) p(w\,|\,D)\,\mathrm{d}w \\
&= \int \mathcal{N}(y_\bullet \,|\, w^T x_\bullet, \beta^{-1}) \mathcal{N}(w\,|\,m_N,S_N)\,\mathrm{d}w \\
&= \mathcal{N}\left(y_\bullet\,|\, m_N^T x_\bullet, \sigma_N^2(x_\bullet)\right)
\end{align*}$$
with
$$\begin{align*}
\sigma_N^2(x_\bullet) = \beta^{-1} + x^T_\bullet S_N x_\bullet \tag{B-3.59}
\end{align*}$$

- (Bishop Fig.3.8). Examples of the predictive distribution (3.58) for a model consisting of 9 Gaussian basis functions, using a synthetic sinusoidal data set.

(Bishop Fig.3.9). Plots of draws of posteriors for the functions $f(x,w)$

###  Maximum Likelihood Estimation for Linear Regression Model

- Recall the posterior mean for the weight vector
$$
m_N = \left(\frac{\alpha}{\beta}\mathbf{I} + \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T y
$$
where $\alpha$ is the prior precision for the weights.

- The Maximum Likelihood solution for $w$ is obtained by letting $\alpha \rightarrow 0$, which leads to

$$\begin{equation*}
\hat w_{\text{ML}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T y
\end{equation*}$$

-  The matrix $\mathbf{X}^\dagger \equiv  (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T$ is also known as the **Moore-Penrose pseudo-inverse** (which is sort-of-an-inverse for non-square matrices).

-  Note that if we have fewer training samples than input dimensions, i.e., if $N<D$, then $\mathbf{X}^T \mathbf{X}$  will not be invertible and maximum likelihood blows up. The Bayesian solution does not suffer from this problem.

###  Least-Squares Regression

-  (You may say that) we don't need to work with probabilistic models. E.g., there's also the deterministic **least-squares** solution: minimize sum of squared errors,

$$\begin{align*} \hat w_{\text{LS}} &= \arg\min_{w} \sum_n {\left( {y_n  - w ^T x_n } \right)} ^2
  = \arg\min_{w} \left( {y - \mathbf{X}w } \right)^T \left( {y - \mathbf{X} w } \right)
\end{align*}$$

-  Setting the gradient
$ \frac{\partial \left( {y - \mathbf{X}w } \right)^T \left( {y - \mathbf{X}w } \right)}{\partial w} = -2 \mathbf{X}^T \left(y - \mathbf{X} w  \right)
$ to zero yields the so-called  **normal equations**
$\mathbf{X}^T\mathbf{X} \hat w_{\text{LS}} = \mathbf{X}^T y$  and consequently

$$
\hat w_{\text{LS}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T y
$$

which is the same answer as we got for the maximum likelihood weights $\hat w_{\text{ML}}$.

- $\Rightarrow$ Least-squares regression ($\hat w_{\text{LS}}$) corresponds to the (probabilistic) maximum likelihood solution ($\hat w_{\text{ML}}$) if the probabilistic model includes the following assumptions:
  1. The observations are independently and identically distributed (**IID**) (this determines how errors are combined), and
  1. The noise signal $\epsilon_n \sim \mathcal{N}(0,\,\beta^{-1})$ is **Gaussian** distributed (determines the error metric)

- If you use the Least-Squares method, you cannot see (nor modify) these assumptions. The probabilistic method forces you to state all assumptions explicitly!

###  Not Identically Distributed Data

-  Let's do an example regarding changing our assumptions. What if we assume that the variance of the measurement error varies with the sampling index,  $\epsilon_n \sim \mathcal{N}(0,\beta_n^{-1})$?

- The likelihood is now (using $\Lambda \triangleq \mathrm{diag}(\beta_n)$ )
$$
p(y\,|\,\mathbf{X},w,\Lambda) = \mathcal{N}(y\,|\,\mathbf{X} w,\Lambda^{-1} ) \,.
$$

- Combining this likelihood with the prior $p(w) = \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I})$ leads to a posterior

$$\begin{align*}
p(w|D) &\propto p(D|w)\cdot p(w) \\
   &= \mathcal{N}(y\,|\,\mathbf{X} w,\Lambda^{-1} \mathbf{I}) \cdot \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \\
   &\propto \exp \left\{ \frac{1}{2} \left( {y - \mathbf{X}w } \right)^T \Lambda \left( {y - \mathbf{X}w } \right)  + \frac{\alpha}{2}w^T w \right\} \\
   &\propto \mathcal{N}\left(w\,|\,m_N,S_N \right)
\end{align*}$$
with
$$\begin{align*}
m_N &=  S_N \mathbf{X}^T \Lambda y  \\
S_N &= \left(\alpha \mathbf{I} +  \mathbf{X}^T \Lambda \mathbf{X}\right)^{-1}
\end{align*}$$

- And maximum likelihood solution
$$
\hat{w}_{\text{ML}} = \left. m_N\right|_{\alpha \rightarrow 0} = \left(\mathbf{X}^T \Lambda \mathbf{X}\right)^{-1} \mathbf{X}^T \Lambda y
$$

-  This maximum likelihood solution is also called the **Weighted Least Squares** (WLS) solution. (Note that we just stumbled upon it, the crucial aspect is appropriate model specification!)

-  Note also that the dimension of $\Lambda$ grows with the number of data points. In general, models for which the number of parameters grow as the number of observations increase are called **non-parametric models**.

#### CODE EXAMPLE

We'll compare the Least Squares and Weighted Least Squares solutions for a simple linear regression model with input-dependent noise:

$$\begin{align*}
x &\sim \text{Unif}[0,1]\\
y|x &\sim \mathcal{N}(f(x), v(x))\\
f(x) &= 5x - 2\\
v(x) &= 10e^{2x^2}-9.5\\
\mathcal{D} &= \{(x_1,y_1),\ldots,(x_N,y_N)\}
\end{align*}$$

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate();
IJulia.clear_output();
```

```julia
using PyPlot, LinearAlgebra

# Model specification: y|x ~ 𝒩(f(x), v(x))
f(x) = 5*x .- 2
v(x) = 10*exp.(2*x.^2) .- 9.5 # input dependent noise variance
x_test = [0.0, 1.0]
plot(x_test, f(x_test), "k--") # plot f(x)

# Generate N samples (x,y), where x ~ Unif[0,1]
N = 50
x = rand(N)
y = f(x) + sqrt.(v(x)) .* randn(N)
plot(x, y, "kx"); xlabel("x"); ylabel("y") # Plot samples

# Add constant to input so we can estimate both the offset and the slope
_x = [x ones(N)]
_x_test = hcat(x_test, ones(2))

# LS regression
w_ls = pinv(_x) * y
plot(x_test, _x_test*w_ls, "b-") # plot LS solution

# Weighted LS regression
W = Diagonal(1 ./ v(x)) # weight matrix
w_wls = inv(_x'*W*_x) * _x' * W * y
plot(x_test, _x_test*w_wls, "r-") # plot WLS solution
ylim([-5,8]); legend(["f(x)", "D", "LS linear regr.", "WLS linear regr."],loc=2);
```

```julia
open("../../styles/aipstyle.html") do f
    display("text/html", read(f, String))
end
```

<!--
This HTML file contains custom styles and some javascript.
Include it a Jupyter notebook for improved rendering.
-->

<!-- Fonts -->
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

<!-- Custom style -->
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.container {
    min-width: 960px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:800px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

.boxed { /* draw a border around a piece of text */
  border: 1px solid blue ;
}

h4#CODE-EXAMPLE,
h4#END-OF-CODE-EXAMPLE {
    margin: 10px 0;
    padding: 10px;
    background-color: #d0f9ca !important;
    border-top: #849f81 1px solid;
    border-bottom: #849f81 1px solid;
}

.emphasis {
    color: red;
}

.exercise {
    color: green;
}

.proof {
    color: blue;
}

code {
  padding: 2px 4px !important;
  font-size: 90% !important;
  color: #222 !important;
  background-color: #efefef !important;
  border-radius: 2px !important;
}

/* This removes the actual style cells from the notebooks, but no in print mode
   as they will be removed through some other method */
@media not print {
  .cell:nth-last-child(-n+2) {
    display: none;
  }
}

footer.hidden-print {
    display: none !important;
}

</style>

<!-- MathJax styling -->

```julia

```


# 2018-09: Lattice Boltzmann solver - Ragnarok

Ragnarok is an open-source python library for solving lattice boltzmann method.

<p align="center">

</p>

Pip install (directly from git repo):

```
$ pip install git+https://github.com/lento234/ragnarok.git
```
---

## Doubly-periodic shear layer

A 2D NS problem with a doubly-period shear layer initial condition.

** Lattice Boltzmann method **

A simple lattice Boltzmann Bhatnagar-Gross-Krook (LBGK) model is used, given in discrete form as:
\begin{equation}
f_i \left(\mathbf{x} + \mathbf{c}_i, t+1\right) - f_i \left(\mathbf{x}, t\right) = \alpha \beta \left(f_i^{eq} - f_i\right)
\end{equation}

where $\alpha = 2$, $\beta = \left(2\nu/c_s^2 +1 \right)^{-1}$, $f_i$ is the population of discrete velocities $\mathbf{c}_i$ and $f_i^{eq}$ is the equilibrium population. The local macroscopic (i.e. density and momentum) is defined as:

\begin{equation}
\rho = \Sigma_i f_i
\end{equation}
and
\begin{equation}
\rho \mathbf{u} = \Sigma_i \mathbf{c}_i f_i
\end{equation}

The present LBM model uses D2Q9 lattice, discretizing the velocity spaces with 9 population function for simulating 2D-dimensional flow. The associated lattice velocites $[c_x, c_y]^T$ and lattice weights $W$ are:

\begin{equation}
\left[\begin{array}{c}
	c_x\\c_y
	\end{array} \right]
	=
	\left[
\begin{array}{ccccccccc}
	0 & 1 & 0 & -1 & 0 & 1 & -1 & -1 & 1\\
	0 & 0 & 1 & 0 & -1 & 1 & 1 & -1 & -1\\
	\end{array} \right]
\end{equation}
and
\begin{equation}
W =
\left[
\begin{array}{ccccccccc}
\frac{4}{9} & \frac{1}{9} & \frac{1}{9} & \frac{1}{9}& \frac{1}{9} & \frac{1}{36} & \frac{1}{36}  & \frac{1}{36}  & \frac{1}{36} \\
\end{array} \right]
\end{equation}

The equilibrium population $f_i^{eq}$ is determined from local conservations:
\begin{equation}
f_i^{eq} = \rho W_i \left(2 - \sqrt{1 + 3 u_x^2}\right) \left(2 - \sqrt{1 + 3 u_y^2}\right) \\
\left(\frac{2u_x- \sqrt{1 + 3 u_x^2}}{1 - u_x}\right)^{c_{x,i}} \left(\frac{2u_y- \sqrt{1 + 3 u_y^2}}{1 - u_y}\right)^{c_{y,i}}
\end{equation}

The discrete LBM is split into two operations: advection (streaming) and relaxtion (collision) step.

## Import Modules

**Standard**

```python
import time
import numpy as np
import math
```

**High performance computing**

```python
from numba import vectorize, jit
```

**Plotting**

```python
import matplotlib.pyplot as plt
import matplotlib.animation as animation
```

**Lattice boltzmann solver**

```python
import ragnarok
```

## Initial condition

```python
def doublyperiodicshearlayer():
    delta = 0.05
    kappa = 80.0
    u0 = 0.01
    ux = np.zeros(x.shape)
    ux[y<=Ny/2.0] = u0*np.tanh(kappa*(y[y<=Ny/2.0]/float(Ny) - 0.25))
    ux[y>Ny/2.0]  = u0*np.tanh(kappa*(0.75 - y[y>Ny/2.0]/float(Ny)))
    uy = delta*u0*np.sin(2*np.pi*(x/float(Nx) + 0.25))
    return ux, uy
```

## Plotting function

**GR version**

```python
def plot(t,zlim=(-0.25,0.25)):
    vortz = curl(solver.u)
    pygr.surface(vortz, rotation=0, tilt=90,colormap=34,
                 xlabel='x', ylabel='y', title='vortz - T = {}'.format(t),
                 zlim=zlim, accelerate=True)
```

** MPL version:**

```python
def plot_mpl(i):
    plt.figure('plot')
    plt.clf()
    vortz = curl(solver.u)
    levels = np.linspace(-0.25,0.25,26)
    plt.contourf(x,y,vortz,levels,cmap='RdBu_r',extend='both')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.title('T = %d' % i)
    plt.colorbar(ticks=levels[::5])
    plt.axis('scaled')
    plt.axis([0,Nx,0,Ny])
    plt.pause(0.1)
```

## Post-processing functions

**Velocity norm:**

```python
@vectorize(['float64(float64,float64)'],target='parallel')
def calcnorm(ux,uy):
    return math.sqrt(ux**2+uy**2)
```

**Velocity curl - vorticity:**

```python
@jit
def curl(u):
    dudx,dudy = np.gradient(u[0], 1.0/Nx,1.0/Ny)
    dvdx,dvdy = np.gradient(u[1], 1.0/Nx,1.0/Ny)
    return dvdx - dudy
```

## Settings

**Simulation parameters**

```python
Re = 30000 # Reynolds number
# T = 20000 # Number of time steps
T = 20000 # Number of time steps
U = 0.1 # Lattice velocity

Nx = 100 # Number of lattice in x-dir
Ny = 100 # Number of lattice in y-dir
```

**Postprocessing parameters**

```python
# Flags
apply_bc = True
plot_step = 100
plot_save = False
plot_flag = True
```

## Solver

Initialze the 2D NS solver:

```python
solver = ragnarok.NavierStokes2D(U=U, Re=Re, Nx=Nx, Ny=Ny)
```

    nu = 0.000333
    beta = 0.998004
    omega = 1.996008

Reference parameters:

```python
# Get parameters
L = solver.L
U = solver.U
nu = solver.nu
Nx = solver.Nx
Ny = solver.Ny
cs = solver.cs
x = solver.x[0]
y = solver.x[1]
```

## Initial conditions

The doubly periodic shear layer (DPSL) describes the case of roll-up of two anti-parallel shear layers within a periodic square domain. The initial condition is given as:

\begin{align}
u_x &= \left\{  \begin{array}{l}
u_0 \tanh \left[\kappa \left(\frac{y}{N} - \frac{1}{4}\right) \right], y \le N/2, \\
u_0 \tanh \left(\kappa \left(\frac{3}{4} - \frac{y}{N}\right) \right), y > N/2, \\
\end{array} \right.\\
\\
u_y &= \delta \sin \left[ 2 \pi  \left(\frac{x}{N} + \frac{1}{4} \right)\right].
\end{align}

**Assign the initial condition:**

```python
# Setup initial conditions (Doubly periodic shear layer)
ux, uy = doublyperiodicshearlayer()
```

**Initialize the population:**

```python
solver.initialize(ux=ux,uy=uy)
```

** Plot initial condition **

```python
plot_mpl(0)
```

## Time stepping

The simulation parameters are $\mathrm{Re} = (U L)/\nu = 30\ 000$, $\delta = 0.05$, $\kappa = 80$, $L = N = N_x = N_y = 100$ and $U=0.1$, providing a $\beta = 0.9980$. The vorticity of the flow is defined as $\omega = \nabla \times \mathbf{u}$ and in 2D with $\mathbf{u} = (u_x,u_y)^T$, it is given as:
\begin{equation}
\omega = \frac{\partial v}{\partial x} - \frac{\partial u}{\partial y}
\end{equation}

**Simulation Algorithm**:
 1. `stream`: Streaming / advection step: $f'_i(x) \leftarrow f^n_i(x-c_i)$
 2. `apply_periodic`: Apply periodic boundary condition
 3. `relax`: Relaxation / collision step: $f^{n+1}_i(x) \leftarrow f'_i  + \alpha\beta \left[f'^{eq}_i(x,t) - f'_i(x,t)\right]$

```python
fig, ax = plt.subplots()

ims = []
for t in range(T+1):
    if t==1: # for JIT
        startTime = time.time()

    # Plot
    if plot_flag and t % plot_step == 0:
        vortz = curl(solver.u)
        im = ax.imshow(vortz, vmin=-0.25, vmax=0.25, cmap='RdBu_r')
        ims.append([im])

    # Step 1: Streaming / advection step: f'_i(x) <- f^n_i(x-c_i)
    solver.stream()

    # Step 2: Apply boundary condition
    solver.apply_periodic()

    # Step 3: Relaxation / collision step: f^{n+1}_i(x) <- f'_i + \alpha\beta [f^{eq}'_i(x,t) - f'_i(x,t)]
    solver.relax()

    if solver.rho.min() <= 0.:
        print('Density is negative!')
        break

# Done
duration = time.time()-startTime
print('Total time: {:3g} sec for {:d} steps, {:3g} ms/step'.format(duration, T, duration/T*1000))
```

    Total time: 9.13418 sec for 20000 steps, 0.456709 ms/step

## Save animation

```python
ani = animation.ArtistAnimation(fig, ims, interval=10, repeat_delay=1000)
ani.save('animation.mp4')
```

```python

```


```python
%matplotlib inline
```

# Compute the explainable variance

Before fitting any voxelwise model to fMRI responses, it is good practice to
quantify the amount of signal in the test set that can be predicted by an
encoding model. This quantity is called the *explainable variance*.

The measured signal can be decomposed into a sum of two components: the
stimulus-dependent signal and noise. If we present the same stimulus multiple
times and we record brain activity for each repetition, the stimulus-dependent
signal will be the same across repetitions while the noise will vary across
repetitions. In voxelwise modeling, the features used to model brain activity
are the same for each repetition of the stimulus. Thus, encoding models will
predict only the repeatable stimulus-dependent signal.

The stimulus-dependent signal can be estimated by taking the mean of
brain responses over repeats of the same stimulus or experiment. The variance
of the estimated stimulus-dependent signal, which we call the explainable
variance, is proportional to the maximum prediction accuracy that can be
obtained by a voxelwise encoding model in the test set.

Mathematically, let $y_i, i = 1 \dots N$ be the measured signal in
a voxel for each of the $N$ repetitions of the same stimulus and
$\bar{y} = \frac{1}{N}\sum_{i=1}^Ny_i$ the average brain response
across repetitions. For each repeat, we define the residual timeseries
between brain response and average brain response as $r_i = y_i - \bar{y}$.
The explainable variance (EV) is estimated as

\begin{align}\text{EV} = \frac{1}{N}\sum_{i=1}^N\text{Var}(y_i) - \frac{N}{N-1}\sum_{i=1}^N\text{Var}(r_i)\end{align}

In the literature, the explainable
variance is also known as the *signal power*. For more information, see these
references [1]_ [2]_ [3]_.

```python

```

## Path of the data directory

```python
import os
from voxelwise_tutorials.io import get_data_home
directory = os.path.join(get_data_home(), "vim-5")
print(directory)
```

```python
# modify to use another subject
subject = "S01"
```

## Compute the explainable variance

```python
import numpy as np
from voxelwise_tutorials.io import load_hdf5_array
```

First, we load the fMRI responses on the test set, which contains brain
responses to ten (10) repeats of the same stimulus.

```python
file_name = os.path.join(directory, 'responses', f'{subject}_responses.hdf')
Y_test = load_hdf5_array(file_name, key="Y_test")
print("(n_repeats, n_samples_test, n_voxels) =", Y_test.shape)
```

Then, we compute the explainable variance for each voxel.

```python
from voxelwise_tutorials.utils import explainable_variance
ev = explainable_variance(Y_test)
print("(n_voxels,) =", ev.shape)
```

To better understand the concept of explainable variance, we can plot the
measured signal in a voxel with high explainable variance...

```python
import matplotlib.pyplot as plt

voxel_1 = np.argmax(ev)
time = np.arange(Y_test.shape[1]) * 2  # one time point every 2 seconds
plt.figure(figsize=(10, 3))
plt.plot(time, Y_test[:, :, voxel_1].T, color='C0', alpha=0.5)
plt.plot(time, Y_test[:, :, voxel_1].mean(0), color='C1', label='average')
plt.xlabel("Time (sec)")
plt.title("Voxel with large explainable variance (%.2f)" % ev[voxel_1])
plt.yticks([])
plt.legend()
plt.tight_layout()
plt.show()
```

... and in a voxel with low explainable variance.

```python
voxel_2 = np.argmin(ev)
plt.figure(figsize=(10, 3))
plt.plot(time, Y_test[:, :, voxel_2].T, color='C0', alpha=0.5)
plt.plot(time, Y_test[:, :, voxel_2].mean(0), color='C1', label='average')
plt.xlabel("Time (sec)")
plt.title("Voxel with low explainable variance (%.2f)" % ev[voxel_2])
plt.yticks([])
plt.legend()
plt.tight_layout()
plt.show()
```

We can also plot the distribution of explainable variance over voxels.

```python
plt.hist(ev, bins=np.linspace(0, 1, 100), log=True, histtype='step')
plt.xlabel("Explainable variance")
plt.ylabel("Number of voxels")
plt.title('Histogram of explainable variance')
plt.grid('on')
plt.show()
```

We see that many voxels have low explainable variance. This is
expected, since many voxels are not driven by a visual stimulus, and their
response changes over repeats of the same stimulus.
We also see that some voxels have high explainable variance (around 0.7). The
responses in these voxels are highly consistent across repetitions of the
same stimulus. Thus, they are good targets for encoding models.

## Map to subject flatmap

To better understand the distribution of explainable variance, we map the
values to the subject brain. This can be done with `pycortex
<https://gallantlab.github.io/pycortex/>`_, which can create interactive 3D
viewers to be displayed in any modern browser. ``pycortex`` can also display
flattened maps of the cortical surface to visualize the entire cortical
surface at once.

Here, we do not share the anatomical information of the subjects for privacy
concerns. Instead, we provide two mappers:

- to map the voxels to a (subject-specific) flatmap
- to map the voxels to the Freesurfer average cortical surface ("fsaverage")

The first mapper is 2D matrix of shape (n_pixels, n_voxels) that maps each
voxel to a set of pixel in a flatmap. The matrix is efficiently stored in a
``scipy`` sparse CSR matrix. The function ``plot_flatmap_from_mapper``
provides an example of how to use the mapper and visualize the flatmap.

```python
from voxelwise_tutorials.viz import plot_flatmap_from_mapper

mapper_file = os.path.join(directory, 'mappers', f'{subject}_mappers.hdf')
plot_flatmap_from_mapper(ev, mapper_file, vmin=0, vmax=0.7)
plt.show()
```

This figure is a flattened map of the cortical surface. A number of regions of
interest (ROIs) have been labeled to ease interpretation. If you have
never seen such a flatmap, we recommend taking a look at a `pycortex brain
viewer <https://www.gallantlab.org/brainviewer/Deniz2019>`_, which displays
the brain in 3D. In this viewer, press "I" to inflate the brain, "F" to
flatten the surface, and "R" to reset the view (or use the ``surface/unfold``
cursor on the right menu). Press "H" for a list of all keyboard shortcuts.
This viewer should help you understand the correspondance between the flatten
and the folded cortical surface of the brain.

On this flatmap, we can see that the explainable variance is mainly located
in the visual cortex, in early visual regions like V1, V2, V3, or in
higher-level regions like EBA, FFA or IPS. This is expected since this
dataset contains responses to a visual stimulus.

## Map to "fsaverage"

The second mapper we provide maps the voxel data to a Freesurfer
average surface ("fsaverage"), that can be used in ``pycortex``.

First, let's download the "fsaverage" surface.

```python
import cortex

surface = "fsaverage"

if not hasattr(cortex.db, surface):
    cortex.utils.download_subject(subject_id=surface,
                                  pycortex_store=cortex.db.filestore)
    cortex.db._subjects = None  # force filestore reload
    assert hasattr(cortex.db, surface)
```

Then, we load the "fsaverage" mapper. The mapper is a matrix of shape
(n_vertices, n_voxels), which maps each voxel to some vertices in the
fsaverage surface. It is stored as a sparse CSR matrix. The mapper is applied
with a dot product ``@`` (equivalent to ``np.dot``).

```python
from voxelwise_tutorials.io import load_hdf5_sparse_array
voxel_to_fsaverage = load_hdf5_sparse_array(mapper_file,
                                            key='voxel_to_fsaverage')
ev_projected = voxel_to_fsaverage @ ev
print("(n_vertices,) =", ev_projected.shape)
```

We can then create a ``Vertex`` object in ``pycortex``, containing the
projected data. This object can be used either in a ``pycortex`` interactive
3D viewer, or in a ``matplotlib`` figure showing only the flatmap.

```python
vertex = cortex.Vertex(ev_projected, surface, vmin=0, vmax=0.7, cmap='viridis')
```

To start an interactive 3D viewer in the browser, we can use the ``webshow``
function in pycortex. If you are running the notebook on Colab, you first
need to tunnel the pycortex application out of Colab. To do so, use the
following cell to start a tunnel with ``ngrok`` and to get an address where
the pycortex viewer will be made accessible.

```python
try:
    import google.colab  # noqa
    in_colab = True
except ImportError:
    in_colab = False
print(in_colab)

if in_colab:
    from IPython import get_ipython
    get_ipython().system_raw('./ngrok http 8050 &')
    plt.pause(1)

    command = """
        curl -s http://localhost:4040/api/tunnels | python3 -c \
        "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
        """
    result = get_ipython().getoutput(command, split=True)
    print("Use the following address to connect to the brain viewer:\n"
          f"{result}\n"
          "and not the one proposed by pycortex ('Open viewer: ...')\n")
```

Now you can start an interactive 3D viewer by changing ``run_webshow`` to
``True`` and running the following cell.  If you are using Colab, remember to
use the address returned by ngrok in the cell above rather than the address
returned by this cell.

```python
run_webshow = False
if run_webshow:
    cortex.webshow(vertex, open_browser=False, port=8050)
```

Alternatively, to plot a flatmap in a ``matplotlib`` figure, use the
`quickshow` function.

(This function requires Inkscape to be installed. The rest of the tutorial
does not use this function, so feel free to ignore.)

```python
from cortex.testing_utils import has_installed

fig = cortex.quickshow(vertex, colorbar_location='right',
                       with_rois=has_installed("inkscape"))
plt.show()
```

## References

.. [1] Sahani, M., & Linden, J. F. (2003). How linear are auditory cortical
   responses?. Advances in neural information processing systems, 125-132.

.. [2] Hsu, A., Borst, A., & Theunissen, F. E. (2004). Quantifying
   variability in neural responses and its application for the validation of
   model predictions. Network: Computation in Neural Systems, 15(2), 91-109.

.. [3] Schoppe, O., Harper, N. S., Willmore, B. D., King, A. J., & Schnupp,
       J. W. (2016). Measuring the performance of neural models. Frontiers in
       computational neuroscience, 10, 10.


# Modelado de sistemas eléctricos

## Un circuito sencillo

Empezaremos esta práctica modelando un sistema sencillo, un circuito con una resistencia y una capacitancia como el de la siguiente figura:

Este circuito tiene una ecuación diferencial:

$$
V_1 = I R_1 + \frac{1}{C_1} \int I dt
$$

en donde $R_1$ representa el valor de la resistencia ```R1```, y la caida de voltaje en esta resistencia es $I R_1$; $C_1$ representa el valor del capacitor ```C1```, y la caida de voltaje en este capacitor es $\frac{1}{C_1}\int I dt$; por lo que si sumamos las fuentes de voltaje y las igualamos a las caidas de voltaje (todos los elementos que consumen una parte de este voltaje) podremos obtener esta ecuación.

Si ahora hacemos increiblemente obvio cuales de estas cantidades son variables y cuales son constantes, llegaremos a escribir esta ecuación como:

$$
V_1(t) = I(t) R_1 + \frac{1}{C_1} \int I(t) dt
$$

En donde si considero que el voltaje suministrado por la fuente de alimentación puede variar con lo que pasa el tiempo, llego a la conclusión que el voltaje varia con el tiempo, es decir

$$
V_1 = V_1(t)
$$

Y que cuando este voltaje varíe, la corriente a traves del circuito cambiará, por lo que:

$$
I = I(t)
$$

La herramienta que utilizaremos para obtener una simulación de este sistema será la transformada de Laplace; sin embargo, para este tipo de sistemas podemos utilizar una simplificación de estas tomando en cuenta que solo servirán si mi sistema empieza con condiciones iniciales $0$ (en este caso significa que nuestro sistema empieza apagado, y en el tiempo $t=0$ lo enciendo):

$$
\begin{align}
\mathcal{L}\left\{ \int f(t) dt \right\} &= \frac{1}{s} F(s) \\
\mathcal{L}\left\{ f(t) \right\} &= F(s) \\
\mathcal{L}\left\{ f'(t) \right\} &= sF(s) \\
\mathcal{L}\left\{ f''(t) \right\} &= s^2F(s) \\
\end{align}
$$

Si aplicamos estas transformadas, obtendremos:

$$
V_1(s)  = I(s) R_1 + \frac{1}{C_1} \frac{1}{s} I(s)
$$

El siguiente paso será obtener la función de transferencia del sistema, para esto tenemos que pensar en el sistema como una caja negra, de la cual queremos obtener una relación:

Si pensamos en este sistema como una caja que multiplica su entrada por lo que sea que tenga adentro y como resultado de esta multiplicación obtenemos la salida, la ecuación de lo que tiene adentro tiene que ser:

$$
G(s) = \frac{Salida}{Entrada}
$$

por lo que nuestro objetivo tiene que ser obtener esta ecuación. Como dijimos anteriormente, la corriente $I(t)$ es el resultado de conectar nuestro circuito a un voltaje, por lo que nuestra función de transferencia tiene que ser:

$$
G(s) = \frac{I(s)}{V(s)}
$$

Si ahora despejamos $V(s)$ por un lado, y $I(s)$ del otro, obtendremos:

$$
V(s) = I(s)\left[ R_1 + \frac{1}{C_1s} \right]
$$

Obteniendo la división de $I(s)$ entre $V(s)$, obtendremos:

$$
\frac{I(s)}{V(s)} = \frac{1}{R_1 + \frac{1}{C_1s}}
$$

Si ahora multiplicamos tanto el numerador, como el denomidador por $s$, obtendremos:

$$
\frac{I(s)}{V(s)} = G_1(s) = \frac{s}{R_1 s + \frac{1}{C_1}}
$$

Con esta función de transferencia podremos simular nuestro sistema, para esto importamos las funciones ```tf``` y ```step_response``` de la librería de control, así como ```linspace``` de ```numpy```:

```python
from control import tf, step_response
from numpy import linspace
```

Para definir la función de transferencia debemos usar la función ```tf``` escribiendo el numerador de la función $G(s)$ primero y el denominador despues:

$s = s + 0\implies$ ```[1, 0]```

$R_1 s + \frac{1}{C_1} \implies$ ```[R1, 1/C1]```

```python
R1 = 220
C1 = 0.1e-3
G1 = tf([1, 0], [R1, 1/C1])
```

Usamos ```linspace``` para definir los limites de tiempo de nuestra simulación, así como el número de puntos a utilizar; en este caso simularemos nuestro sistema desde el tiempo $0s$ hasta el tiempo $0.4s$ con un total de $100$ puntos:

```python
ts = linspace(0, 0.4, 100)
```

Una vez que hemos definido todos estos parametros, los metemos a ```step_response``` para obtener nuestros datos:

```python
t, y = step_response(G1, ts)
```

Y graficarlos con la función ```plot``` de ```matplotlib```:

```python
from matplotlib.pyplot import plot
```

```python
plot(t, y);
```

Por lo que podemos ver que el comportamiento del sistema es tener un pico de corriente, en cuanto es conectado, de un poco mas de $4mA$, sin embargo este es el comportamiento del sistema cuando se le conecta un voltaje de $1V$, debido a que por default la función ```step_response``` simulará el comportamiento del sistema ante un escalon unitario.

Para poder simular el sistema ante un escalon de $5V$, tendremos que premultiplicar este sistema por una amplificación de $5$:

```python
A = 5
```

```python
t, y = step_response(A*G1, ts)
```

```python
plot(t, y);
```

Por lo que sabemos que al inicio de la simulación se tiene un pico de un poco mas de $20mA$.

Una vez que tenemos la simulación de la corriente del sistema, podemos obtener el voltaje en cualquier elemento, por ejemplo en el capacitor; para hacer esto necesitamos la función de transferencia del capacitor, la cual transforma una corriente $I(t)$ en un voltaje $V_C(t)$, si recordamos, ya tenemos esta ecuación:

$$
V_C = \frac{1}{C_1} \int I dt \implies \frac{V_C(s)}{I(t)} = G_2(s) = \frac{1}{C_1} \frac{1}{s} = \frac{1}{C_1s}
$$

por lo que definimos esta función de transferencia:

```python
G2 = tf([1], [C1, 0])
```

Y simulamos el comportamiento de los dos sistemas uno detras de otro:

```python
t, y = step_response(A*G1*G2, ts)
```

```python
plot(t, y);
```

En esta gráfica lo que podemos ver es el capacitor llenandose con el voltaje aplicado en aproximadamente una decima de segundo.

---
## Ejercicio

Tomando en cuenta el siguiente circuito:

con la siguiente ecuación diferencial:

$$
V_1 = L_1 \frac{dI}{dt} + I R_1 + \frac{1}{C_1} \int I dt
$$

* Define una función de transferencia $G_3$ y guardala en la variable ```G3```

> Nota: Recuerda eliminar la linea ```raise NotImplementedError``` cuando respondas cada ejercicio

```python
L1 = 0.01
# ESCRIBE TU CODIGO AQUI
raise NotImplementedError
```

```python
from numpy.testing import assert_allclose

assert_allclose(G3.pole(), [-21954.4511501,    -45.5488499])
assert_allclose(G3.zero(), [0])
```

* Simula el comportamiento de la corriente $I(t)$ cuando se le es aplicado un voltaje de $12V$, desde el tiempo $0s$ hasta el tiempo $1s$

```python
# ESCRIBE TU CODIGO AQUI
raise NotImplementedError
plot(t, y);
```

```python
from nose.tools import assert_almost_equal, assert_equal

assert_equal(ts[0], 0)
assert_equal(ts[-1], 1)
assert_almost_equal(max(y), 0.034573738498078065)
assert_almost_equal(y[-1], 0)
```

* Simula el comportamiento del voltaje en el inductor $V_L$

```python
# ESCRIBE TU CODIGO AQUI
raise NotImplementedError
plot(t, y);
```

```python
from nose.tools import assert_almost_equal, assert_equal

assert_equal(ts[0], 0)
assert_equal(ts[-1], 1)
assert_almost_equal(max(y), 12)
assert_almost_equal(y[1], -0.01574794025216697)
assert_almost_equal(y[-1], 0)
```

---

## Un circuito complejo

Si ahora queremos simular un sistema mas complejo, tenemos que utilizar un truco muy simple, pero antes analicemos como logramos simular el voltaje en el ejemplo pasado.

En el ejemplo anterior ya teniamos una función de transferencia para el sistema completo, pero queriamos especificamente el voltaje, no la corriente, por lo que creamos un sistema nuevo y los simulamos uno detras de otro, esto es equivalente al siguiente diagrama:

en donde podemos anotar las señales:

por lo que podemos pensar en la multiplicación de los dos sitemas como un sistema nuevo, podemos revisar esto si analizamos las variables guardadas:

```python
G1
```

```python
G2
```

```python
G1*G2
```

Nota que las funciones de transferencia se multiplican correctamente cuando se lo indicamos.

Si ahora revisamos nuestro circuito complejo:

podriamos utilizar Leyes de Kirchoff para obtener ecuaciones diferenciales de este sistema y sería completamente valido, sin embargo, podemos tambien dividir este problema en varios pequeños.

El lazo de la izquierda es equivalente a la función de transferencia $G_1$ que ya obtuvimos, mas aún, si ahora lo multiplicamos por la función de transferencia $G_2$ estaremos obteniendo la función de transferencia que haria esto equivalente a una fuente de alimentación de voltaje variable, es decir:

por lo que ahora solo tenemos que obtener otra función de transferencia para el sistema nuevo.

```python
R2 = 1000
C2 = 100e-9
L2 = 0.1
G1 = tf([1, 0], [L1, R1, 1/C1])
G2 = tf([1], [C1, 0])
G3 = tf([1, 0], [L2, R2, 1/C2])
```

y simularlo todo completo:

```python
t, y = step_response(5*G1*G2*G3, ts)
```

```python
plot(t, y);
```

y para obtener el voltaje en el capacitor $C_2$

```python
G4 = tf([1], [C2, 0])
```

```python
t, y = step_response(5*G1*G2*G3*G4, ts)
plot(t, y);
```

---
## Ejercicio

* Simule el comportamiento del voltaje en el capacitor $C_2$ del siguiente circuito del tiempo $0s$ a $0.2s$ con 1000 puntos:

> Nota: Utiliza las variables ```G1```, ```G2```, ```G3```, ```G4```, ```G5``` y ```G6```

```python
V1 = 12
L1 = 0.01
L2 = 0.001
R1 = 4700
R2 = 220
C1 = 0.47e-3
C2 = 100e-9

# ESCRIBE TU CODIGO AQUI
raise NotImplementedError
plot(t, y);
```

```python
from nose.tools import assert_almost_equal, assert_equal
from numpy.testing import assert_allclose

assert_equal(ts[0], 0)
assert_equal(ts[-1], 0.2)
assert_equal(len(ts), 1000)

assert_almost_equal(max(y), 24, 2)
assert_almost_equal(min(y), 0, 2)
assert_almost_equal(y[0], 0)

assert_allclose((V1*G1*G2*G3*G4*G5*G6).zero(), [0, 0])
assert_allclose((V1*G1*G2*G3*G4*G5*G6).pole(), [-4.7e+06+0.j, -4.5e+04+0.j, 1.5e-12+461.2j,
                                                1.5e-12-461.2j,0.0e0 +0.j, 0.0e0+0.j], 5)
```

* Defina una función de transferencia ```G7``` que sea capaz de simular el comportamiento de un amplificador operacional, el cual amplifica su señal de entrada $10$ veces.

```python
# ESCRIBE TU CODIGO AQUI
raise NotImplementedError
```

```python
from numpy.testing import assert_allclose

assert_allclose(G7.pole(), [])
assert_allclose(G7.zero(), [])
assert_allclose(G7.dcgain(), [10])
```

* Defina una función de transferencia ```G8``` que sea capaz de simular el comportamiento de un amplificador operacional configurado como un amplificador integrador, con una ganacia de $5$

```python
# ESCRIBE TU CODIGO AQUI
raise NotImplementedError
```

```python
from numpy.testing import assert_allclose
from numpy import inf

assert_allclose(G8.pole(), [0])
assert_allclose(G8.zero(), [])
assert_allclose(G8.dcgain(), [inf])
```

---


```python
from sympy import *
from sympy.matrices import matrix_multiply_elementwise as elementwise

from sympy import init_printing

from math import log10
from numpy import polyfit

init_printing()
```

```python
# R1,R2,C2 = symbols(['R1', 'R2', 'C2'])
Ri = symbols('R0:2', real=True, positive=True)
R0, R1 = Ri

C1 = symbols('C1', real=True, positive=True)

w = symbols('w')
s = symbols('s', imaginary=True)

ai = symbols('a0:3')
a0, a1, a2 = ai
taylor = Poly(ai[::-1],s)

x = symbols('x')
```

```python
Z = (R0+R1+C1*R0*R1*s)/(C1*R1*s+1)

numerator, denominator = fraction(Z)
print(numerator)
print(denominator)
```

    C1*R0*R1*s + R0 + R1
    C1*R1*s + 1

```python
pol = Poly(numerator.expand(), s)
coeffs = {s**m[0]:pol.coeff_monomial(m) for m in pol.monoms()}
```

```python
# partial fraction decomposition on a rational function
apart(Z,s)
```

```python
#linsolve(list(coeffs_s.values()), ai)
```

```python
mi = symbols('m0:4')
X = Function('X')
# Eq(X(s), mobius_direct(s, mi))
```

```python
mi = symbols('m0:4')
m0,m1,m2,m3 = mi
X = symbols('X')
# solveset(mobius_direct(s, mi)-X,s)
```

```python
pol = Poly(ai[::-1],x).as_expr()
pol
```

```python
def list_to_frac(l):

    expr = Integer(0)

    for i in reversed(l[1:]):

        expr += i

        expr = 1/expr

    return l[0] + expr

ai = symbols('a0:4')
a0, a1, a2, a3 = ai

frac = list_to_frac(ai)
frac

# reduce into standard rational function
frac = cancel(frac)
frac
```

```python
#
order = 3
ri = symbols('r0:2')
r0, r1 = ri
si = symbols('s0:2')
s0, s1 = ri

expr = sum([r*x**i for i,r in enumerate(ri)])/sum([s*x**i for i,s in enumerate(si)])
print(expr)

series(expr, x, n=order).removeO().coeff(x)
# !Does not always work!
def get_coefficients(expr, x):
   try:
       pol = Poly(expr.expand(), s)
   except:
       pol = expr

   coeffs = {x**m[0]:pol.coeff_monomial(m) for m in pol.monoms()}

   for k, v in coeffs.items():
       print('{}: {}'.format(k, v))

   return coeffs

expr = series(expr, x, n=order).removeO()
print(expr)

def from_rational_to_taylor(expr, x, order):
    # starting from zero order
    expr = series(expr, x, n=order).removeO()
    # this gives problems with the first coefficient (order 0)
    taylor_coeffs = [expr.coeff(x**i) for i in range(order)]
    taylor_coeffs[0] = (expr - sum([taylor_coeffs[i]*x**i for i in range(1,order)]))
    return taylor_coeffs

coeffs = from_rational_to_taylor(expr, x, order)

ai = symbols(['a{}'.format(i) for i in range(order)])
a0, a1, a2 = ai

# system of equations to solve
system = [a-b for a,b in zip(coeffs, ai)]

# now the coefficients of x in the rational function should be made of some other symbols
# to find those symbols
"""
solve_for = (R0,R1,C1) # tuple of symbols
sol = solve(system, solve_for, dict=True)
sol
"""
```

    (r0 + r1*x)/(s0 + s1*x)
    r0/s0 + x**2*(r0*s1**2/s0**3 - r1*s1/s0**2) + x*(-r0*s1/s0**2 + r1/s0)

    '\nsolve_for = (R0,R1,C1) # tuple of symbols\nsol = solve(system, solve_for, dict=True)\nsol\n'

```python
ai = symbols(['a{}'.format(i) for i in range(order)])
a0, a1, a2 = ai

system = [a-b for a,b in zip(coeffs, ai)]
system
solve_for = (R0,R1,C1)
sol = solve(system, solve_for, dict=True)
sol
```

```python
Poly(series(expr, x, n=3).removeO())
[]
#{x**m[0]:pol.coeff_monomial(m) for m in pol.monoms()}
```

```python

```


```python
import numpy as np
import scipy as sp
from scipy.stats import poisson
```

```python
import matplotlib.pyplot as plt

%matplotlib inline
```

##### Exercise 4.1

If $\pi$ is the equiprobable random policy, what is $Q^\pi(11, down)$, $Q^\pi(7, down)$?

We have $Q(s,a) = \sum_{s'} P^a_{ss'}[R^a_{ss'} + \gamma \sum_{a'} \pi(s', a') Q(s', a')]$, so

$$Q^\pi(11, down) = 1 [-1 + \gamma \cdot 1 \cdot 0] = -1$$

and

$$Q^\pi(7, down) = -1 + \gamma V^\pi(11) = -1 - 14 \cdot \gamma$$

##### Exercise 4.2

Suppose a new state 15 is added to the gridworld just below state 13, and its actions, left, up, right, and down, take the agent to states 12, 13, 14, and 15, respectively. Assume that the transitions from the original states are unchanged. What, then, is $V^\pi(15)$ for the equiprobable random policy?

$V^\pi(s) = \sum_a \pi(s,a) \sum_{s'} P^a_{ss'} [ R^a_{ss'} + \gamma V^\pi(s')]$

$V^\pi(15) = \frac{1}{4} [(-1 + \gamma V^\pi(12)) + (-1 + \gamma V^\pi(13)) + (-1 + \gamma V^\pi(14)) + (-1 + \gamma V^\pi(15))]$

$= \frac{1}{4}[ -4 + \gamma (V^\pi(12) + V^\pi(13) + V^\pi(14) + V^\pi(15))]$

so $V^\pi(15) = \frac{1}{1 - \frac{\gamma}{4}} [-1 + \frac{\gamma}{4} (V^\pi(12) + V^\pi(13) + V^\pi(14))]$

Now suppose the dynamics of state 13 are also changed, such that action down from state 13 takes the agent to the new state 15. What is $V^\pi(15)$ for the equiprobable random policy in this case?

It seems like we would need to do policy evaluation to find $V^\pi(15)$, since all value functions would change with the new state available? I'm not sure.

##### Exercise 4.3

What are the equations analogous to (4.3), (4.4), and (4.5) for the action-value function $Q^\pi$ and its successive approximation by a sequence of functions  $Q_0, Q_1, Q_2, ...$?

Using the Bellman Equation for $Q^\pi$,

$
\begin{equation}
\begin{split}
Q^\pi(s, a) =& E_\pi[\sum_{k=0}^{\infty} \gamma^k r_{t+k+1} | s_t = s, a_t = a]\\
=& E_\pi[r_{t+1} + \gamma \sum_{a'} \pi(s', a') Q^\pi(s', a')| s_t = s, a_t = a]\\
=& \sum_{s'} P^a_{ss'} [R^a_{ss'} + \gamma \sum_{a'} \pi(s', a') Q^\pi(s', a')]
\end{split}
\end{equation}
$

we obtain the following update equation:

$$Q_{k+1}(s, a) = \sum_{s'} P^a_{ss'} [R^a_{ss'} + \gamma \sum_{a'} \pi(s', a') Q_k(s', a')]$$

##### Exercise 4.3.5

In some undiscounted episodic tasks there may be policies for which eventual termination is not guaranteed. For example, in the grid problem above it is possible to go back and forth between two states forever. In a task that is otherwise perfectly sensible, $V^\pi(s)$ may be negative infinity for some policies and states, in which case the algorithm for iterative policy evaluation given in Figure  4.1 will not terminate. As a purely practical matter, how might we amend this algorithm to assure termination even in this case? Assume that eventual termination is guaranteed under the optimal policy.

We can threshold the min of $V^\pi(s)$ at a constant value so that $V^\pi(s)$ doesn't go to $-\infty$ since we know that states where $V^\pi(s) \rightarrow -\infty$ are actually terminal states that can't be escaped.

##### Exercise 4.4

https://webdocs.cs.ualberta.ca/~sutton/book/code/jacks.lisp

```python
# ;;; Jack's car rental problem.  The state is n1 and n2, the number of cars
# ;;; at each location a the end of the day, at most 20.  Actions are numbers of cars
# ;;; to switch from location 1 to location 2, a number between -5 and +5.

# ;;; P1(n1,new-n1) is a 26x21 array giving the probability that the number of cars at
# ;;; location 1 is new-n1, given that it starts the day at n1.  Similarly for P2

# ;;; R1(n1) is a 26 array giving the expected reward due to satisfied requests at
# ;;; location, given that the day starts with n1 cars at location 1.  SImilarly for R2.

```

The expected Rewards due to satisfied requests in a given state $n$ (state = number of cars available) at a parking lot are:

$$R_n = 10 \sum_{r=0}^{20} P(r) min(\{r, n\})$$

and the transition probability that the number of cars at a location is $n'$ after starting at $n$ is

$$P(n, n') = \sum_{r=0} P(r) \sum_{d=0} P(d) \delta(r = n, n' = min(\{20, n + d - req_{satisfied}\}))$$

for all requests $r$ and dropoffs $d$.

```python
class JacksCarRental(object):
    def __init__(self):

        self.lambda_requests1 = 3
        self.lambda_requests2 = 4
        self.lambda_dropoffs1 = 3
        self.lambda_dropoffs2 = 2

        self.gamma = 0.9            # discount factor
        self.theta = 0.0000001      # delta precision for policy evaluation

        # value function
        self.V = np.zeros((21, 21))
        self.Vs = [self.V]
        # policy
        self.PI = np.zeros((21, 21))
        self.PIs = [self.PI]

        # transition probabilities for each state
        self.P1 = np.zeros((26, 21))
        self.P2 = np.zeros((26, 21))

        # expected rewards in each state
        self.R1 = np.zeros(26)
        self.R2 = np.zeros(26)

        # calculate trans. probs and expected rewards
        self.P1, self.R1 = self.load_P_and_R(
            self.P1, self.R1,
            lambda_requests=self.lambda_requests1,
            lambda_dropoffs=self.lambda_dropoffs1
        )

        self.P2, self.R2 = self.load_P_and_R(
            self.P2, self.R2,
            lambda_requests=self.lambda_requests2,
            lambda_dropoffs=self.lambda_dropoffs2
        )

    def load_P_and_R(self, P, R, lambda_requests, lambda_dropoffs):
        # Get the transition probabilities and expected rewards
        requests = 0
        request_prob = poisson.pmf(requests, mu=lambda_requests)
        while request_prob >= .000001:
            # expected rewards
            for n in xrange(26):
                # rent out car for $10 each
                R[n] += 10 * request_prob * min([requests, n])

            # transition probabilities
            dropoffs = 0
            dropoff_prob = poisson.pmf(dropoffs, mu=lambda_dropoffs)
            while dropoff_prob >= .000001:
                for n in xrange(26):
                    satisfied_requests = min([requests, n])
                    new_n = min([20, n + dropoffs - satisfied_requests])
                    if new_n < 0:
                        print 'Warning negative new_n', new_n
                    P[n, new_n] += request_prob * dropoff_prob

                dropoffs += 1
                dropoff_prob = poisson.pmf(dropoffs, mu=lambda_dropoffs)

            requests += 1
            request_prob = poisson.pmf(requests, mu=lambda_requests)

        return P, R

    # 2. policy evaluation
    def backup_action(self, n1, n2, a):
        # number of cars to move from location 1 to 2, thresholded at 5 and -5 according to problem specs
        cars_to_move = max([min([n1, a]), -n2])
        cars_to_move = min([max([cars_to_move, -5]), 5])

        # costs $2 to move each cars
        cost_to_move = -2 * abs(cars_to_move)

        # do backup
        morning_n1 = n1 - cars_to_move
        morning_n2 = n2 + cars_to_move

        # sum over all possible next states
        newv = 0
        for newn1 in xrange(21):
            for newn2 in xrange(21):
                newv += self.P1[morning_n1, newn1] * self.P2[morning_n2, newn2] *\
                        (self.R1[morning_n1] + self.R2[morning_n2] +\
                         self.gamma * self.V[newn1, newn2])
        return newv + cost_to_move

    def policy_evaluation(self):
        delta = 1
        while delta > self.theta:
            delta = 0
            # Loop through all States
            for n1 in xrange(21):
                for n2 in xrange(21):
                    old_v = self.V[n1, n2]
                    action = self.PI[n1, n2]
                    # do a full backup for each state
                    self.V[n1, n2] = self.backup_action(n1, n2, action)
                    delta = max([delta, abs(old_v - self.V[n1, n2])])
#             print 'Policy evaluation delta: ', delta
        print 'Done with Policy Evaluation'
        return self.V

    # 3. Policy Improvement
    def get_best_policy(self, n1, n2):
        best_value = -1
        for a in range(max(-5, -n2), min(5, n1) + 1):
            this_action_value = self.backup_action(n1, n2, a)
            if this_action_value > best_value:
                best_value = this_action_value
                best_action = a

        return best_action

    def policy_improvement(self):
        self.V = self.policy_evaluation()
        policy_stable = False
        while policy_stable is False:
            policy_stable = True
            for n1 in xrange(21):
                for n2 in xrange(21):
                    b = self.PI[n1, n2]
                    self.PI[n1, n2] = self.get_best_policy(n1, n2)

                    if b != self.PI[n1, n2]:
                        policy_stable = False

            self.Vs.append(self.policy_evaluation())
            self.PIs.append(self.PI)

        return policy_stable
```

```python

```

```python
# Policy Iteration for Jack's Car Rental problem

jacks = JacksCarRental()
V = jacks.policy_improvement()
```

    /Applications/anaconda/envs/rl/lib/python2.7/site-packages/ipykernel/__main__.py:85: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future

    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation

```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

xv, yv = np.meshgrid(range(21), range(21))
ax.plot_surface(xv, yv, jacks.V)
ax.set_xlabel('Cars at 2nd location')
ax.set_ylabel('Cars at 1st location')

```

```python
plt.figure()
cs = plt.contour(xv, yv, jacks.PI)
plt.clabel(cs, inline=1, fontsize=10)
plt.xlabel('Cars at 2nd location')
plt.ylabel('Cars at 1st location')
```

One of Jack's employees at the first location rides a bus home each night and lives near the second location. She is happy to shuttle one car to the second location for free. Each additional car still costs \$2, as do all cars moved in the other direction. In addition, Jack has limited parking space at each location. If more than 10 cars are kept overnight at a location (after any moving of cars), then an additional cost of \$4 must be incurred to use a second parking lot (independent of how many cars are kept there).

```python
class JacksCarRental(object):
    def __init__(self):

        self.lambda_requests1 = 3
        self.lambda_requests2 = 4
        self.lambda_dropoffs1 = 3
        self.lambda_dropoffs2 = 2

        self.gamma = 0.9            # discount factor
        self.theta = 0.0000001      # delta precision for policy evaluation

        # value function
        self.V = np.zeros((21, 21))
        self.Vs = [self.V]
        # policy
        self.PI = np.zeros((21, 21))
        self.PIs = [self.PI]

        # transition probabilities for each state
        self.P1 = np.zeros((26, 21))
        self.P2 = np.zeros((26, 21))

        # expected rewards in each state
        self.R1 = np.zeros(26)
        self.R2 = np.zeros(26)

        # calculate trans. probs and expected rewards
        self.P1, self.R1 = self.load_P_and_R(
            self.P1, self.R1,
            lambda_requests=self.lambda_requests1,
            lambda_dropoffs=self.lambda_dropoffs1
        )

        self.P2, self.R2 = self.load_P_and_R(
            self.P2, self.R2,
            lambda_requests=self.lambda_requests2,
            lambda_dropoffs=self.lambda_dropoffs2
        )

    def load_P_and_R(self, P, R, lambda_requests, lambda_dropoffs):
        # Get the transition probabilities and expected rewards
        requests = 0
        request_prob = poisson.pmf(requests, mu=lambda_requests)
        while request_prob >= .000001:
            # expected rewards
            for n in xrange(26):
                # rent out car for $10 each
                R[n] += 10 * request_prob * min([requests, n])

            # transition probabilities
            dropoffs = 0
            dropoff_prob = poisson.pmf(dropoffs, mu=lambda_dropoffs)
            while dropoff_prob >= .000001:
                for n in xrange(26):
                    satisfied_requests = min([requests, n])
                    new_n = min([20, n + dropoffs - satisfied_requests])
                    if new_n < 0:
                        print 'Warning negative new_n', new_n
                    P[n, new_n] += request_prob * dropoff_prob

                dropoffs += 1
                dropoff_prob = poisson.pmf(dropoffs, mu=lambda_dropoffs)

            requests += 1
            request_prob = poisson.pmf(requests, mu=lambda_requests)

        return P, R

    # 2. policy evaluation
    def backup_action(self, n1, n2, a):
        # number of cars to move from location 1 to 2, thresholded at 5 and -5 according to problem specs
        cars_to_move = max([min([n1, a]), -n2])
        cars_to_move = min([max([cars_to_move, -5]), 5])

        # costs $2 to move each car,
        # but we get one car free if we move from n1 to n2!
        cost_to_move = -2 * abs(cars_to_move)

        if cars_to_move > 0:
            # 1 free one if we move n1 -> n2
            cost_to_move += 2

        # do backup
        morning_n1 = n1 - cars_to_move
        morning_n2 = n2 + cars_to_move

        # If more than 10 cars are kept overnight at a location (after any moving of cars),
        # then an additional cost of \$4 must be incurred
        extra_parking_cost = 0
        if morning_n1 > 10:
            extra_parking_cost -= 4
        if morning_n2 > 10:
            extra_parking_cost -= 4

        # sum over all possible next states
        newv = 0
        for newn1 in xrange(21):
            for newn2 in xrange(21):
                newv += self.P1[morning_n1, newn1] * self.P2[morning_n2, newn2] *\
                        (self.R1[morning_n1] + self.R2[morning_n2] +\
                         self.gamma * self.V[newn1, newn2])
        return newv + cost_to_move + extra_parking_cost

    def policy_evaluation(self):
        delta = 1
        while delta > self.theta:
            delta = 0
            # Loop through all States
            for n1 in xrange(21):
                for n2 in xrange(21):
                    old_v = self.V[n1, n2]
                    action = self.PI[n1, n2]
                    # do a full backup for each state
                    self.V[n1, n2] = self.backup_action(n1, n2, action)
                    delta = max([delta, abs(old_v - self.V[n1, n2])])
#             print 'Policy evaluation delta: ', delta
        print 'Done with Policy Evaluation'
        return self.V

    # 3. Policy Improvement
    def get_best_policy(self, n1, n2):
        best_value = -1
        for a in range(max(-5, -n2), min(5, n1) + 1):
            this_action_value = self.backup_action(n1, n2, a)
            if this_action_value > best_value:
                best_value = this_action_value
                best_action = a

        return best_action

    def policy_improvement(self):
        self.V = self.policy_evaluation()
        policy_stable = False
        while policy_stable is False:
            policy_stable = True
            for n1 in xrange(21):
                for n2 in xrange(21):
                    b = self.PI[n1, n2]
                    self.PI[n1, n2] = self.get_best_policy(n1, n2)

                    if b != self.PI[n1, n2]:
                        policy_stable = False

            self.Vs.append(np.copy(self.policy_evaluation()))
            self.PIs.append(np.copy(self.PI))

        return policy_stable
```

```python
jacks = JacksCarRental()
V = jacks.policy_improvement()
```

    /Applications/anaconda/envs/rl/lib/python2.7/site-packages/ipykernel/__main__.py:99: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future

    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation
    Done with Policy Evaluation

```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

xv, yv = np.meshgrid(range(21), range(21))
ax.plot_surface(xv, yv, jacks.V)
ax.set_xlabel('Cars at 2nd location')
ax.set_ylabel('Cars at 1st location')

```

```python
f, axx = plt.subplots(2, 3, sharex='col', sharey='row')

for i in xrange(len(jacks.PIs)):
    r = i / 3
    c = i % 3
    ax = axx[r, c]
    cs = ax.contour(xv, yv, jacks.PIs[i])
    ax.set_title(i)
```

```python
f, axx = plt.subplots(2, 3, sharex='col', sharey='row')

for i in xrange(len(jacks.Vs)):
    r = i / 3
    c = i % 3
    ax = axx[r, c]
    cs = ax.contour(xv, yv, jacks.Vs[i])
    ax.set_title(i)
#     ax.set_clabel(cs, inline=1, fontsize=10)
#     ax.set_xlabel('Cars at 2nd location')
#     ax.set_ylabel('Cars at 1st location')
```

##### 4.5
How would policy iteration be defined for action values? Give a complete algorithm for computing $Q^*$, analogous to Figure  4.3 for computing $V^*$. Please pay special attention to this exercise, because the ideas involved will be used throughout the rest of the book.

I'm not sure about step 3??, but I'm trying to use $\pi(s,a)$ as a deterministic probability of taking an action in a given state.

1. Initialization
    - $Q(s,a) \in \mathbb{R}$ and $\pi(s, a) \in \{0, 1\}$ arbitrarily for all $s \in S$ and $a \in A(s)$
2. Policy Evaluation
    - Repeat
        - $\Delta \leftarrow 0$
        - For each $s \in S$ and $a \in A(s)$
            - $q \leftarrow Q(s,a)$
            - $Q(s,a) \leftarrow \sum_{s'} P^a_{ss'}[R^a_{ss'} + \gamma \sum_{a'} \pi(s', a') Q(s', a')]$
            - $\Delta \leftarrow max(\Delta, |q - Q(s,a)|)$
    - Until $\Delta \lt \theta$

3. Policy Improvement
    - policy_stable = True
    - For each $s \in S$ and $a \in A(s)$
        - $b \leftarrow Q(s,a)$
        - $\pi(s,a) \leftarrow \sum_{s'} P^a_{ss'} [R^a_{ss'} + \gamma \cdot argmax_{a'} Q(s', a')] $
        - If $b \neq \pi(s, a)$, then policy_stable = False
    - For each $s \in S$ and $a \in A(s)$
        - $\pi(s,a) \leftarrow$ 1 if $\pi(s,a) = argmax_a \pi(s,a)$ else 0
    - If policy_stable, then stop, else go to 2

##### Exercise 4.6

Suppose you are restricted to considering only policies that are $\epsilon$-soft, meaning that the probability of selecting each action in each state, $s$, is at least $\epsilon / |A(s)|$. Describe qualitatively the changes that would be required in each of the steps 3, 2, and 1, in that order, of the policy iteration algorithm for  (Figure  4.3).

Step 3 would still take greedy updates to $\pi(s, a)$, but would have to make sure that $\pi(s,a)$ has at least a probability of $\epsilon / |A(s)|$ for each non-greedy action. $\pi(s,a)$ is the probability of taking action $a$ in state $s$, as opposed to $\pi(s)$ as used previously since we had deterministic policies. Step 2 would have to use the Bellman Equation for $V(s)$ with a non-deterministic policy evaluation, $V(s) \leftarrow \sum_a \pi(s,a) \sum_{s'} P^a_{ss'} [R^a_{ss'} + \gamma V(s')]$. Step 1 would have to initiliaze the $\pi(s,a)$ as random uniform.

##### Exercise 4.7

Why does the optimal policy for the gambler's problem have such a curious form? In particular, for capital of 50 it bets it all on one flip, but for capital of 51 it does not. Why is this a good policy?

If p >= .5, then you'd want to bet 1 at all states, since you are more likely to win than lose. If p < .5, then you are more likely to lose on any bet than to win. The reason that at p=.4, the policy has a weird shape, is because at state 2 for example, it is more likely that you get to state 4 by betting $2 than by betting $1 twice. The probability of going from state 2 to state 4 in one time-step by betting $1 twice is .4*.4=.16, which is less than if you bet $2 once.

The spikes on 12, 25, 50, and 75, seem empirically independent of p as long as p is greater than around .05. This suggests that the spikes around those values exist because the game terminates at 100. You are more likely to get to 100 if you bet values that are factors of 100. Since there is no state > 100, the optimal policy has this jagged form.

##### Exercise 4.8

 Implement value iteration for the gambler's problem and solve it for p=.25 and p=.55. In programming, you may find it convenient to introduce two dummy states corresponding to termination with capital of 0 and 100, giving them values of 0 and 1 respectively. Show your results graphically, as in Figure  4.6. Are your results stable as $\theta \rightarrow 0$?

```python
class GamblersProblem(object):
    def __init__(self, p=.45, gamma=1):
        self.PI = np.zeros(100)
        self.V = np.zeros(101)
        self.gamma = gamma
        self.p = p

    def backup_action(self, state, action):
        return self.p * self.gamma * self.V[state + action] +\
                (1 - self.p) * self.gamma * self.V[state - action]

    def value_iteration(self, epsilon=.0000000001):
        self.V = np.zeros(101)
        # You get a reward of 1 if you win $100
        self.V[100] = 1

        delta = 1
        while delta > epsilon:
            delta = 0
            for state in xrange(1, 100):
                old_v = self.V[state]
                # you can bet up to all your money as long as the winnings would be less than 100
                self.V[state] = np.max(
                    [self.backup_action(state, action + 1) \
                     for action in xrange(min(state, 100 - state))]
                )
                delta = max(delta, abs(self.V[state] - old_v))

    def get_deterministic_policy(self, epsilon=.0000000001):
        PI = np.zeros(100)
        for state in xrange(1, 100):
            values = [self.backup_action(state, action + 1)\
                               for action in xrange(min(state, 100 - state))]

            best_pi = 1
            best_value = -1
            for idx, v in enumerate(values):
                if v > best_value + epsilon:
                    best_value = v
                    best_pi = idx + 1

            PI[state] = best_pi

        self.PI = PI
        return PI

```

```python
gp = GamblersProblem(p=.25)
gp.value_iteration()
plt.plot(gp.V)
plt.show()
plt.plot(gp.get_deterministic_policy())
plt.show()
```

```python
gp = GamblersProblem(p=.55)
gp.value_iteration()
plt.plot(gp.V)
plt.show()
plt.plot(gp.get_deterministic_policy())
plt.show()
```

```python
gp = GamblersProblem(p=.001)
gp.value_iteration()
plt.plot(gp.V)
plt.show()
plt.plot(gp.get_deterministic_policy())
plt.show()
```

Results are not stable as $\theta \rightarrow 0$, I'm not really sure why.

##### Exercise 4.9

What is the analog of the value iteration backup (4.10) for action values, $Q_{k+1}(s,a)$?

- Initialize Q arbitrarily, e.g. $Q(s,a)=0$ for all $s \in S$ and $a \in A(s)$
- Repeat
    - $\Delta \leftarrow 0$
    - For each $s \in S$ and $a \in A(s)$
        - $q \leftarrow Q(s,a)$
        - $Q(s,a) = \sum_{s'} P^a_{ss'} [R^a_{ss'} + \gamma \cdot argmax_{a'} Q(s', a')]$
        - $\Delta \leftarrow max(\Delta, |q - Q(s,a)|)$
- until $\Delta \lt \theta$

- Output a deterministic policy $\pi$ such that
    - $\pi(s) = argmax_a Q(s,a)$

```python

```