# Introducción a Aprendizaje Reforzado

## Paradigmas de aprendizaje

**Aprendizaje Supervisado:** Tenemos **ejemplos etiquetados** $(x, y)$

> Buscamos un mapeo $f_\theta : x \to y$

el cual aprendemos optimizando una **función de costo**, *e.g.*

$$
\min_w \sum_{i=1}^N y_i \log (\hat y_i) \quad \hat y_i = f_\theta (x_i)
$$

**Aprendizaje No Supervisado:** Tenemos **ejemplos sin etiqueta** $(x)$

> Buscamos una representación de los datos

la cual aprendemos optimizando una **función de costo**, *e.g.*

$$
\min  \sum_{i=1}^N \sum_{k=1}^K w_{ik} \|x_i - \mu_k\|^2 \quad w_{ik} = \begin{cases} 1 & k = \text{arg}\min_j \| x_i - \mu_j\|^2 \\ 0 & \sim \end{cases}
$$

**Aprendizaje Reforzado (RL):** El problema fundamental de RL es

> **aprender** a tomar la mejor **decisión** en un **ambiente** cambiante

¿Quién toma las decisiones?

> El sistema que toma las decisiones e interactua con el ambiente se llama **agente**

¿Cómo se cual es la mejor decisión?

> La mejor decisión es aquella que obtiene mayor **recompensa**

Esto supone diferencias importantes con los paradigmas de Machine Learning anteriores. Definamos estos componentes resaltando las diferencias a continuación

## Componentes de Aprendizaje Reforzado

**Ambiente y Estado**

En lugar de ejemplos existe un **ambiente o mundo** el cual podemos observar

Nuestra perceción del ambiente no siempre es completa

El ambiente se representa por un vector denominado **estado**

**Acciones**

El agente no retorna predicciones sino que toma **decisiones**

En cada instante el agente escoge y realiza una **acción**

Existen **consecuencias**, las acciones realizadas pueden modificar el ambiente

**Recompensa**

La retroalimentación del agente no proviene de etiquetas sino de una **señal numérica escalar llamada recompensa**

La recompensa está asociada a uno o más estados

La recompensa puede ser positiva o negativa

**Diferencias clave**

- Supervisión: Al agente no se le dice que acción es buena, sino que estados son buenos
- Prueba y error: El agente debe descubrir que acción le entrega la mayor recompensa probándolas una a una
- Temporalidad: El entrenamiento y la ejecución son secuenciales, no se puede asumir iid
- Retraso en la retroalimentación: Las recompensas pueden demorar en llegar, las acciones pueden no traer recompensa inmediata pero si en el futuro

**Objetivo del agente**

> Seleccionar acciones para maximizar la **recompensa acumulada futura**

En ciertos casos podría ser mejor abandonar una recompensa intermedia en pos de obtener una mayor recompensa final

**Hipótesis de recompensa**

> Todo objetivo puede ser representado mediante la maximización de la recompensa acumulada esperada

**La vida de una gente**

> Un agente de RL debe tener objetivos, debe poder sentir su ambiente y debe ser capaz de actuar sobre el mismo

Tipicamente asumiremos que el tiempo es discreto $t=0, 1, 2, 3, ...$

En cada tiempo instante $t$ el agente
1. **recibe recompensa del ambiente:** $R_t$
1. **observa el ambiente:** $S_t$
1. **realiza una acción:** $A_t$

o en pseudo-código

    for t in 1, 2, 3, ...., N
        get Rt
        get St
        do At

o en diagrama (Sutton & Barto, Fig 3.1):

Luego la **historia** de un agente se puede definir como la siguiente trayectoria

$$
H_t = (S_0, A_0, R_1, S_1, A_1, \ldots, R_{t-1}, S_{t-1}, A_{t-1}, R_t, S_t, A_t)
$$

**Discusión**

¿Puedes reconocer los agentes, las acciones, el ambiente, la recompensa y demás elementos de RL en los siguientes ejemplos?

¿Se podría resolver este problema usando aprendizaje supervisado? ¿Cuáles serían las limitaciones?

```python
from IPython.display import YouTubeVideo
YouTubeVideo("60pwnLB0DqY", width=800, height=400)
```

```python
from IPython.display import YouTubeVideo
YouTubeVideo("Ev0wpVB7OEs", width=800, height=400)
```

```python
from IPython.display import YouTubeVideo
YouTubeVideo("lpi19vExbzc", width=800, height=400)
```

Ref: Sección 1.2, Sutton & Barto

> Phil prepares his breakfast. Closely examined, even this apparently mundane
activity reveals a complex web of conditional behavior and interlocking goal–subgoal
relationships: walking to the cupboard, opening it, selecting a cereal box, then
reaching for, grasping, and retrieving the box. Other complex, tuned, interactive
sequences of behavior are required to obtain a bowl, spoon, and milk carton. Each
step involves a series of eye movements to obtain information and to guide reaching
and locomotion. Rapid judgments are continually made about how to carry the
objects or whether it is better to ferry some of them to the dining table before
obtaining others. Each step is guided by goals, such as grasping a spoon or getting
to the refrigerator, and is in service of other goals, such as having the spoon to eat
with once the cereal is prepared and ultimately obtaining nourishment. Whether
he is aware of it or not, Phil is accessing information about the state of his body
that determines his nutritional needs, level of hunger, and food preferences.

(Relacionado: https://youtu.be/E2evC2xTNWg)

**Discusión**

Considere este [listado de ejemplos de aplicación de RL](https://towardsdatascience.com/applications-of-reinforcement-learning-in-real-world-1a94955bcd12)

¿Qué tienen estos ejemplos en común?

> Interacción entre un agente que toma decisiones y su ambiente

> El agente trata de cumplir una meta a pesar de la incerteza del ambiente

> El agente usa la experiencia que adquiere para mejorar su comportamiento

```python
from IPython.display import YouTubeVideo
YouTubeVideo("qBZPSTR96N4", width=800, height=400)
```

[Robot PR1](https://ieeexplore.ieee.org/document/4543527) haciendo todo tipo de tareas domésticas

El video es un montaje ya que PR1 está siendo operado remotamente por un humano

> La habilidad motriz ya es suficiente, el desafio pendiente está en los algoritmos

## El agente en acción

Asumamos que nuestro agente tiene un número limitado de acciones posibles

$$
\mathcal{A} = \{a_1, a_2, a_3, \ldots, a_N\}
$$

En cada instante $t$ el agente escoge una acción en base al estado

La decisión se hace según una función denominada **política** (*policy*)

$$
\pi : \mathcal{S} \to \mathcal{A}
$$

La política es un mapa entre el espacio de estados y el espacio de acciones

**Política determinista**

La política está representada por una función

$$
a = \pi (s)
$$

Ej: Si tengo hambre y el timbre no está sonando apreto el timbre

**Política estocástica**

La política está representada por una distribución

$$
a \sim \pi(A|S) = P(A=a|S=s)
$$

Ej: Si tengo hambre y el timbre no está sonando

- 9 de 10 veces apreto el timbre
- 1 de 10 veces aparece una mosca que me distrae y lo dejo en paz

En promedio apreto el timbre un 90% de las veces que me da hambre

## Procesos de decisión de Markov

El formalismo tras RL se basa en la teoría de **Sistemas Dinámicos** y **Control óptimo**

En particular se sustenta en los **procesos de decisión de Markov** (*Markov Decision Process*) que a su vez se basa en la **cadena de Markov** (*Markov Chain*)

En una cadena de Markov tenemos

- un conjunto de estados posibles $\mathcal{S}$
- un modelo de transición $T(s, s')$

El modelo representa la probablidad de pasar de un estado $s$ a otro $s'$

Además se cumple la **propiedad de Markov**

> El estado futuro es condicionalmente independiente del pasado dado presente

Que matematicamente, para un instante $t$ sería

$$
p(s_{t+1}| s_t, s_{t-1}, s_{t-2}, \ldots, s_2, s_1) = p(s_{t+1}| s_t)
$$

>El estado actual es un estadístico suficiente del futuro

**Ejemplo:** Queremos predecir el clima de Valdivia por medio de un modelo de cadena de Markov

Asumiremos que el clima de mañana es perfectamente predecible a partir del clima de hoy

Sean dos estados

- $s_A$ Luvioso
- $s_B$ Soleado

Con probabilidades condicionales $p(s_A|s_A) = 0.8$, $(s_B|s_A) = 0.2$, $P(s_A|s_B) = 0.5$ y $P(s_B|s_B) = 0.5$

La matriz de transición de la cadena es

$$
T = \begin{pmatrix} 0.8 & 0.5 \\ 0.2 & 0.5 \end{pmatrix}
$$

que también se puede visualizar como un mapa de transición

Si está lloviendo hoy, ¿Cuál es la probabilidad de que llueva mañana, en tres dias más y en una semana más?

```python
import numpy as np

s = np.array([1, 0])
T = np.array([[0.8, 0.5],[0.2, 0.5]])

np.dot(np.linalg.matrix_power(T, 2), s)
```

¿Puedes explicar de dónde sale este resultado?

**De Cadena de Markov a Proceso de Decisión de Markov (MDP)**

El MDP es una reinterpretación de la cadena de Markov

El MDP modela un agente que toma decisiones

En el MDP tenemos
- un conjunto de estados posibles $\mathcal{S}$
- un conjunto de acciones posibles $\mathcal{A}$
- una función de recompensa $R(s)$
- una modelo de transición de tres variables $T(s, a, s')$

Ahora el modelo se interpreta como la probabilidad de llegar a $s'$ si estaba en $s$ y ejecuté $a$

Dado el estado y la acción actual, el proximo estado no depende del pasado: **Propiedad de Markov**

La existencia de $R(s)$ nos dice que algunos estados son más deseables que otros

**Discusión: Aspiradora robot**

Sea una robo-aspiradora encargada de limpiar nuestra habitación

Nuestra habitación puede discretarse en $3\times 4 = 12$ bloques donde sólo 11 son accesibles por la aspiradora

- La aspiradora parte en el espacio $(1,1)$
- La estación de carga $(4,3)$ tiene una recompensa $+1$
- La escalera $(4, 2)$ tiene una recompensa $-1$
- Todos los demás cuadros tienen una recompensa $-0.04$
- La aspiradora rebota con las paredes y con el obstáculo
- El espacio tiene cuatro acciones, moverse al {N, S, E, O}

Si el ambiente es determinista

¿Cúal es la mejor política?

Asumamos que ahora existe un elemento estocástico en el ambiente

Cuando la aspiradora quiere avanzar en una dirección
- En un 80% lo logra
- En un 10% se resbala y avanza en la dirección más próxima contrareloj
- En un 10% la ataca el gato y avanza en la dirección más próxima en el sentido del reloj

Ej: quiero ir al norte, en 80% lo logro, en un 10% me voy al oeste y un 10% al este

¿Cúal es la mejor política?

¿Cómo cambia la politica según el nivel de carga del robot?

## Política basada en valor

> ¿Cómo se construye una política óptima?

Asumamos que estamos en un instante $t$ en particular

Podemos definir la **recompensa total futura** como

$$
R_t + R_{t+1} + R_{t+2} + R_{t+3} +\ldots
$$

y diseñar una política para maximizarla, pero

> Problema 1: la recompensa total futura podría diverger

> Problema 2: ambientes no estacionarios e incerteza sobre el futuro

Es más razonable usar una **recompensa total futura descontada**

$$
\begin{align}
G_t &= R_t + \gamma R_{t+1} + \gamma^2 R_{t+2} + \ldots \nonumber \\
& = \sum_{k=0}^\infty \gamma^k R_{t+k} \nonumber
\end{align}
$$

donde $\gamma \approx 0.99$ es el factor de descuento, que controla la escala de tiempo del aprendizaje del agente

#### Discusión

Asumiendo un mundo ideal donde la inflación no existe y somos seres responsables, ¿Qué es mejor?
- Aceptar una donación de 10.000 pesos ahora
- Aceptar una donación de 10.000 en un año más

### Función Q

En base a $G_t$ podemos escribir la **recompensa total futura esperada** o **ganancia esperada**

$$
Q(s,a) = \mathbb{E}[G_t|S_t=s, A_t=a]
$$

también llamada función **Q**

> La función Q nos dice cuanta recompensa futura podemos obtener si estamos en el estado $s$ y ejecutamos la acción $a$

Luego una  política determinista basada en Q sería

$$
\pi^*(s) = \text{arg} \max_{a\in \mathcal{A}} Q(s, a)
$$

> **Principio de máximas utilidades**: Un agente racional escogerá la acción que maximize su ganancia esperada

Si conocemos la función de transición del agente podemos descomponer la función **Q** como sigue

$$
\begin{align}
Q(s,a) &= \mathbb{E}[G_t|S_t=s, A_t=a] \nonumber \\
&= R(s,a) + \gamma  \sum_{s' \in \mathcal{S}} T(s, a, s') \max_{a' \in \mathcal{A}} Q(s', a') \nonumber
\end{align}
$$

> que se conoce como **Ecuación de Bellman**

Si el ambiente es determinista y la acción $a$ en $s$ me lleva a $s'$ entonces esto se simplifica como

$$
Q(s,a) = R(s,a) + \gamma  \max_{a' \in \mathcal{A}} Q(s', a')
$$

> El valor Q en el estado actual es igual a la recompensa percibida actualmente más el valor descontado del mejor Q alcanzanble en el nuevo estado

En base a esto veremos como encontrar **Q** usando Q-Learning

### Q-Learning

Sea un agente con un conjunto finito de acciones y un ambiente con un conjunto finito de estados

La función Q se representa como una tabla

| Q     | $a_1$ | $a_2$ | $a_3$ | $a_4$ |
|-------|-------|-------|-------|-------|
| $s_1$ | 0     | 0     | 0     | 0     |
| $s_2$ | 0     | 0     | 0     | 0     |
| $s_3$ | 0     | 0     | 0     | 0     |

Inicialmente la tabla Q está en cero y el objetivo es llenarla

La intuición es que si
- realizamos una acción en un estado determinado y nos da un resultado no deseado: evitar esa acción en ese estado
- realizamos una acción en un estado determinado y nos da un resultado deseado: preferir esa acción en ese estado
- todas las acciones en un cierto estado producen un resultado no deseado: evitar ese estado
- todas las acciones en un cierto estado producen un resultado deseado: buscar ese estado

> El agente se entrena por prueba y error hasta llenar la tabla  $Q(s,a)$

El algoritmo de Q-learning usa la siguiente regla de actualización al realizar la acción $a$ sobre un estado $s$ obteniendo una recompensa $r(s,a)$ y avanzando al estado $s'$

$$
\begin{align}
Q(s, a) \leftarrow& (1-\alpha)Q(s,a) + \alpha \left(R(s,a) + \gamma \max_{a' \in \mathcal{A}} Q(s', a')\right)  = \nonumber \\
& Q(s,a) + \alpha \left(R(s,a) + \gamma \max_{a' \in \mathcal{A}} Q(s', a') - Q(s,a)\right)
\end{align}
$$

donde $\alpha \in [0, 1]$ es la tasa de aprendizaje

## Nuestro primer agente RL en Python

Con lo visto hasta ahora tenemos lo necesario para entrenar nuestro primer agente de RL

Para simular el ambiente y el agente usaremos el *toolkit* [gym](https://gym.openai.com/) de OpenAI

- gym nos proporciona simuladores de varios ambientes en forma de juegos y agentes con acciones predefinidas
    - podemos seleccionar una accion y modificar el estado
    - podemos consultar el estado y pedir recompensa
- gym puede ser usado con cualquier librería numérica, *e.g* numpy, pytorch, tensorflow

Para instalar

    pip install 'gym[atari]'

Esto instalará los ambientes por defecto y adicionalmente los ambientes basados en juegos de ATARI

Los ambientes con su descripción puede consultarse [aquí](https://gym.openai.com/envs/#classic_control)

**Ejemplo:** [La laguna congelada](https://gym.openai.com/envs/FrozenLake-v0/)

> Usted y sus amigos juegan a lanzar el frisbee. El frisbee cae en una laguna que está parcialmente congelada. El objetivo es recuperar el frisbee sin caer donde se ha derretido el hielo

Con  esta premisa veamos como cargar este ambiente en *gym*

```python
import gym

# Genera el ambiente
env = gym.make("FrozenLake-v0")

# Resetea el ambiente
env.reset()

# Muestra el ambiente
env.render()
```

##### Espacio de estados

- El estado tiene 4x4 = 16 espacios que representan la laguna congelada
    - F (Frozen), es seguro caminar por ese bloque
    - H (Hole), te ahogas y mueres
    - S (Start), la posición inicial del agente
    - G (Goal), la posición del frisbee
- Tenemos **completa observabilidad de los estados**

Podemos consultar directamente sobre el tamaño del espacio de estados y estado actual

```python
# Espacio de estados
display(env.observation_space)
display(env.observation_space.n)

# Estado actual
display(env.s)
```

##### Espacio de acciones

Existen cuatro acciones
- 0: LEFT
- 1: DOWN
- 2: RIGHT
- 3: UP

Adicionalmente
- El agente rebota en las paredes
- El hielo es resbaladizo: **Las acciones tienen un componente estocástico**

Podemos consultar el tamaño del espacio de acciones con

```python
# Espacio de acciones
display(env.action_space)
display(env.action_space.n)
```

Podemos samplear una acción aleatoria con la propiedad `sample`

```python
env.action_space.sample()
```

La distribución es la acción aleatoria es uniforme

```python
import numpy as np
a = []
for i in range(1000):
    a.append(env.action_space.sample())

import matplotlib.pyplot as plt
plt.figure(figsize=(7, 3))
plt.hist(np.array(a));
```

**Ejecución de una acción**

Podemos ejectutar una acción con la función `step`

Esta función retorna el nuevo estado al que llegamos y la recompensa

Recordemos este escenario es **estocástico**: La acción que ejecutamos no es siempre la que se realiza

```python
env = gym.make("FrozenLake-v0")
env.reset()
env.render()

display("Estoy en el estado:", env.s)

a = env.action_space.sample()
display("Selecciono la acción:",a)

# Ejecuto la acción
s, r, end, info = env.step(a)

display("Llegué al estado:", s, env.s)

display("Obtuve la recompensa:", r)

env.render()
```

Por ejemplo la distribución para moverme a la derecha desde el cuadro inicial es

```python
env = gym.make("FrozenLake-v0")
arrival_state = []
for k in range(1000):
    env.reset()
    s, r, end, info = env.step(2)
    arrival_state.append(s)

import matplotlib.pyplot as plt
plt.figure(figsize=(7, 3))
plt.hist(np.array(arrival_state));
```

- 1/3 logró moverme a la derecha y llegar a 1
- 1/3 me muevo hacia abajo y llego a 4
- 1/3 me muevo hacia arriba y reboto para quedarme en 0

Esto también estaba indicado en `info`

```python
info
```

#### Recompensa y condición de término

Existen tres condiciones de término de episodio
- Llegar al frisbee: recompensa +1
- Caer en un agujero: recompensa +0
- Realizar 100 pasos: recompesa +0

¿Cómo le irá a un agente con politica aleatoria?

```python
from IPython.display import clear_output
from time import sleep
import gym

env = gym.make("FrozenLake-v0")
env.reset()
end = False
step = 0

while not end:
    a = env.action_space.sample()
    s, r, end, info = env.step(a)
    clear_output(wait=True)
    print(step)
    print(r)
    print(env.render(mode='ansi'))
    sleep(.25)
    step += 1
```

## Entrenamiento con Q-Learning

Entrenemos nuestro agente usando una tabla de Q-Learning inicialmente en cero

En cada iteración la actualizamos usando

$$
Q(s,a) \leftarrow Q(s,a) + \alpha \left(R(s,a) + \gamma \max_{a' \in \mathcal{A}} Q(s', a') - Q(s,a)\right)
$$

Como el ambiente es estocástico es preferible una tasa de aprendizaje baja

Estudiemos lo que ocurre

```python
import numpy as np
import gym
from tqdm.notebook import tqdm

env = gym.make("FrozenLake-v0")

rewards = []
alpha = 0.1
gamma = 0.99
Q = np.zeros(shape=(env.observation_space.n, env.action_space.n))

for episode in tqdm(range(10)):
    env.reset()

    end = False
    while not end:
        s_current = env.s
        # Escoger la mejor acción
        a = np.argmax(Q[s_current, :])
        # Ejecutarla
        s_future, r, end, info = env.step(a)
        # Actualizar Q
        Q[s_current, a] += alpha*(r + gamma*np.max(Q[s_future, :]) - Q[s_current, a])
    env.render()
    rewards.append(r)
```

#### Diagnósticos básicos

Revisamos la tabla Q y las recompensas ganadas

```python
display(Q)
from collections import Counter
display(Counter(rewards));
```

¿Qué paso aquí?

¿Puedes ver el problema de la linea 19?

### Dilema de exploración y explotación

Un problema del algoritmo que acabamos de ver es que es demasiado explotador

> Siempre escoge la acción que maximiza **Q**: es una *greedy policy*

Sin embargo, especialmente al inicio, es crítico explorar el espacio de acciones. Se puede hacer la siguiente corrección

Se muestrea una variable Bernoulli (binaria) con probabilidad $1-\epsilon$ de ser $1$
- Si resulta 1, entonces se sigue la política óptima (greedy)
- Si resulta 0, entonces se escoge una acción al azar

Tipicamente se inicia $\epsilon$ en un valor cercano a $1$ y se hace $\epsilon \to 0$ para $t \to \infty$

> Este tipo de estrategies se denominan $\epsilon$ greedy

### Parámetros de $\epsilon$ greedy Q-Learning

Los parámetros de Q-Learning convencional son

- La tasa de aprendizaje $\alpha$
    - Mientras más alta más rapidamente se adoptan los cambios en **Q**
    - Un valor muy alto puede ser perjudicial si el ambiente es estocástico
    - Un valor típico es $\alpha = 0.1$
- El factor de descuento $\gamma$
    - Si $\gamma = 0$ solo se enfoca en el presente
    - Si $\gamma = 1$ las recompensas de largo y corto plazo tienen igual valor y se tienen problemas de convergencia
    - Un valor típico es $\gamma = 0.9$

También se tiene

- El número de episodios: Se refiere a la cantidad de veces que se presenta el escenario al agente

Y ahora hemos agregado $\epsilon$ el cual podemos definir según

- El valor inicial y final para $\epsilon$
- La estrategia de disminución de $\epsilon$

Por ejemplo una estrategia exponencial con tasa de disminución $\tau_\epsilon>0$ sería

$$
\epsilon_k = \epsilon_T + (\epsilon_0 - \epsilon_T) e^{-\tau_\epsilon k}
$$

donde $\epsilon_0 >>> \epsilon_T$

Estos parámetros más la cantidad de episodios definen la transición entre las fases de exploración y explotación

```python
import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(figsize=(6, 4))
episodes = np.arange(10000)
epsilon_init = 1.0
epsilon_end = 0.01
epsilon_rate = 1e-3
ax.plot(episodes, epsilon_end + (epsilon_init - epsilon_end) * np.exp(-epsilon_rate*episodes));
ax.set_xlabel('Episodios')
ax.set_ylabel('Epsilon');
```

```python
import numpy as np
import gym
from tqdm.notebook import tqdm

#env = gym.make("FrozenLake-v0")
env = gym.make("FrozenLake-v0", is_slippery=False)
Q = np.zeros(shape=(env.observation_space.n,
                    env.action_space.n))

diagnostics = {'rewards': [], 'episode_length': []}
# Parametros
alpha = 0.1
gamma = 0.99
epsilon_init = 1.0 # NUEVO
epsilon_end = 0.01 # NUEVO
epsilon_rate = 1e-3 # NUEVO
epsilon = lambda episode : epsilon_end + (epsilon_init - epsilon_end) * np.exp(-epsilon_rate*episode) # NUEVO

for episode in tqdm(range(10000)):
    env.reset()
    end = False
    # Entrenamiento
    while not end:
        s_current = env.s
        # Seleccionar la acción
        if not np.random.binomial(1, p=1.-epsilon(episode)) and ~np.all(Q[s_current, :]==0): # NUEVO
            a = np.argmax(Q[s_current, :])
        else:
            a = env.action_space.sample() # NUEVO
        # Ejecutarla
        s_future, r, end, info = env.step(a)
        # Actualizar Q
        Q[s_current, a] += alpha*(r + gamma*np.max(Q[s_future, :]) - Q[s_current, a])

    # Prueba
    # Cada 100 epocas evaluamos nuestro agente
    if np.mod(episode, 100) == 0:
        diagnostics['rewards'].append(np.zeros(shape=(100,)))
        diagnostics['episode_length'].append(np.zeros(shape=(100,)))
        for k in range(100):
            env.reset()
            end = False
            episode_length = 0
            episode_reward = 0.0
            while not end:
                s_current = env.s
                a = np.argmax(Q[s_current, :])
                s_future, r, end, info = env.step(a)
                episode_length += 1
                episode_reward += r

            diagnostics['rewards'][-1][k] = episode_reward
            diagnostics['episode_length'][-1][k] = episode_length
```

### Diagnósticos: Debuggeando lo aprendido

Se puede verificar el aprendizaje del agente observando la tabla Q

¿Tiene sentido lo aprendido?

```python
np.set_printoptions(precision=3)
display(Q)
display(np.arange(16).reshape(4, 4))
movs = np.ones(9).reshape(3, 3)*-1
movs[0, 1] = 3
movs[1, 0] = 0
movs[1, 2] = 2
movs[2, 1] = 1
display(movs)
env.render()
```

El aprendizaje también se puede diagnosticar observando gráficas de la evolución de la recompensa y la cantidad de pasos promedio

```python
fig, ax = plt.subplots(2, figsize=(6, 5), sharex=True)
ax[0].errorbar(np.arange(10000//100)*100,
               np.array(diagnostics['rewards']).mean(axis=1),
               np.array(diagnostics['rewards']).std(axis=1));
ax[0].set_ylabel('Recompensa\npromedio');
ax[1].errorbar(np.arange(10000//100)*100,
               np.array(diagnostics['episode_length']).mean(axis=1),
               np.array(diagnostics['episode_length']).std(axis=1));
ax[1].set_xlabel('Episodios')
ax[1].set_ylabel('Largo promedio\nde los episodios');
```

### Probando nuestro agente en vivo

```python
from IPython.display import clear_output
from time import sleep

env.reset()
end = False

step = 0
while not end:
    s_current = env.s
    a = np.argmax(Q[s_current, :])
    s_future, r, end, info = env.step(a)

    clear_output(wait=True)
    print(step)
    print(r)
    print(env.render(mode='ansi'))
    sleep(.2)
    step += 1
```

##### ¿Qué ocurre si el ambiente no es estocástico?

Use

    env = gym.make("FrozenLake-v0", is_slippery=False)

para comprobarlo

# [Carro con péndulo](https://gym.openai.com/envs/CartPole-v0/)

En este ambiente existe un carro con un poste que se balancea sobre él

**Objetivo del ambiente**

> Aplicar fuerzas con tal de balancear el poste el mayor tiempo posible

Al inicio el poste está derecho

> Si el poste se inclina más de 15 grados en cualquier dirección pierdes

> Si el carro se aleja más de 2.4 unidades del centro pierdes

El escenario tiene un límite de 200 pasos

```python
import numpy as np
import gym

env = gym.make("CartPole-v0")
```

Ojo: El ambiente se renderea en una nueva ventana

```python
env.reset()
end = False
for k in range(200): # Maximo número de pasos
    env.render()
    a = env.action_space.sample()
    s_future, r, end, info = env.step(a)
    #if end:
    #    break
```

Podemos cerrar la ventana del ambiente con:

```python
env.close()
```

**Acciones**

Existen dos acciones:
- Aplicar una unidad de fuerza hacia la izquierda
- Aplicar una unidad de fuerza hacia derecha

Es decir el espacio de acciones es discreto

```python
# Tipo de las acciones
display(env.action_space)
# Cantidad de acciones
display(env.action_space.n)
# Muestreando una acción al azar
display(env.action_space.sample())
```

**Estados**

El espacio de estados tiene 4 variables
- Posición del carro
- Velocidad del carro
- Ángulo del poste (radianes)
- Velocidad angular del poste

El espacio de estado es continuo

Recuperamos el valor del estado con

    env.state

```python
display(env.state)
# Tipo del estado
display(env.observation_space)
```

Podemos también recuperar las cotas de las variables de estado

```python
# Si está acotado o no
display(env.observation_space.is_bounded())
# Cota inferior
display(env.observation_space.low)
# Cota superior
display(env.observation_space.high)
```

#### Discretización

¿Es posible utilizar Q-learning si el espacio es continuo?

> Si, pero necesitamos discretizar el estado para poder construir nuestra tabla Q

Debemos decidir
- el rango de las variables
- la resolución de las variables

En este caso particular teniamos que
- La posición del carro y el ángulo de la barra están acotadas
- La velocidad del carro y la velocidad angular de la barra no están acotadas

En base a esto creamos una función auxiliar que discretiza el espacio de estados

```python
import numpy as np

def create_bin_limits(bounds, n_bins):
    return np.linspace(bounds[0], bounds[1], n_bins-1)

n_bins = (100, 100, 100, 100) # Es adecuada esta resolución?
#n_bins = (10, 10, 10, 10)
n_bins = (1, 1, 10, 10)

# bins for x
x_limits = create_bin_limits((-2.4, 2.4), n_bins[0])
display(x_limits)
# bins for v
v_limits = create_bin_limits((-2.4, 2.4), n_bins[1])
display(v_limits)
# bins for t
t_limits = create_bin_limits((-15*np.pi/180, 15*np.pi/180), n_bins[2])
display(t_limits*180/np.pi)
# bins for w
w_limits = create_bin_limits((-50*np.pi/180, 50*np.pi/180), n_bins[3])
display(w_limits*180/np.pi)

display("Cantidad de estados", np.prod(np.array(n_bins)))

# Función para obtener estado discreto
def get_state(state):
    x, v, t, w = state
    x_state = np.searchsorted(x_limits, x)
    v_state = np.searchsorted(v_limits, v)
    t_state = np.searchsorted(t_limits, t)
    w_state = np.searchsorted(w_limits, w)
    return (x_state, v_state, t_state, w_state)

import gym
env = gym.make("CartPole-v0")
env.reset()
display(env.state)
display(get_state(env.state))
display(get_state([-10]*4), get_state([10.]*4))
```

Ojo:

- El tamaño del espacio de estados crece muy rápido con la cantidad de bines
- Las variables de estados no tienen que tener la misma resolución
- Los bines no tienen porque ser del mismo tamaño

Con esto ya podríamos entrenar nuestro agente usando Q-learning

```python
%matplotlib notebook
import matplotlib.pyplot as plt

fig, ax = plt.subplots(3, figsize=(6, 5), sharex=True, tight_layout=True)

def update_plot():
    for ax_ in ax:
        ax_.cla()

    episodes = np.arange((episode+1)//100)*100
    ax[0].errorbar(episodes,
                   np.array(diagnostics['rewards']).mean(axis=1),
                   np.array(diagnostics['rewards']).std(axis=1));
    ax[0].set_ylabel('Recompensa\npromedio');
    ax[1].errorbar(episodes,
                   np.array(diagnostics['episode_length']).mean(axis=1),
                   np.array(diagnostics['episode_length']).std(axis=1));
    ax[1].plot(episodes, [195]*len(episodes), 'k--')
    ax[1].set_ylabel('Largo promedio\nde los episodios');
    ax[2].plot(episodes, epsilon(episodes))
    ax[2].set_ylabel('Epsilon')
    ax[2].set_xlabel('Episodios')
    fig.canvas.draw()
```

```python
import numpy as np
import gym
from tqdm.notebook import tqdm

env = gym.make("CartPole-v0")

Q = np.zeros(shape=(n_bins[0], n_bins[1], n_bins[2], n_bins[3],  # NUEVO
                    env.action_space.n))

diagnostics = {'rewards': [], 'episode_length': []}
# Parametros
alpha = lambda episode: 0.1
#alpha = lambda episode: 0.01 + (1. - 0.01) * np.exp(-epsilon_rate*episode)
gamma = 0.999
epsilon_init = 1.0
epsilon_end = 0.01
epsilon_rate = 1e-3
epsilon = lambda episode : epsilon_end + (epsilon_init - epsilon_end) * np.exp(-epsilon_rate*episode)
for episode in tqdm(range(3000)):
    env.reset()
    end = False
    # Entrenamiento
    while not end:
        s_current = get_state(env.state) # NUEVO
        # Seleccionar la acción
        if not np.random.binomial(1, p=1.-epsilon(episode)) and ~np.all(Q[s_current]==0):
            a = np.argmax(Q[s_current])
        else:
            a = env.action_space.sample()
        # Ejecutarla
        s, r, end, info = env.step(a)
        s_future = get_state(s) # NUEVO
        # Actualizar Q
        Q[s_current][a] += alpha(episode)*(r + gamma*np.max(Q[s_future]) - Q[s_current][a])

    # Prueba
    # Cada 100 epocas evaluamos nuestro agente
    if np.mod(episode+1, 100) == 0:
        diagnostics['rewards'].append(np.zeros(shape=(10,)))
        diagnostics['episode_length'].append(np.zeros(shape=(10,)))
        for k in range(10):
            env.reset()
            end = False
            episode_length = 0
            episode_reward = 0.0
            while not end:
                s_current = get_state(env.state)
                a = np.argmax(Q[s_current])
                s_future, r, end, info = env.step(a)
                episode_length += 1
                episode_reward += r

            diagnostics['rewards'][-1][k] = episode_reward
            diagnostics['episode_length'][-1][k] = episode_length
        update_plot()
```

#### Estudiando la influencia de la resolución de los estados

- Vea que pasa usando 10 bines en lugar de 100
- Vea que pasa cuando se usa 1 sólo bin para la posición y para la velocidad

#### Visualización de la matriz Q

```python
fig, ax = plt.subplots(1, 3, figsize=(6, 2), tight_layout=True)

def update(x, v):
    for ax_ in ax:
        ax_.cla()
    ax[0].imshow(Q[x, v, :, :, 0], cmap=plt.cm.gray,
                 aspect='auto', extent=[-50, 50, -15, 15])
    ax[1].imshow(Q[x, v, :, :, 1], cmap=plt.cm.gray,
                 aspect='auto', extent=[-50, 50, -15, 15])
    ax[2].imshow(np.argmax(Q[x, v, :, :, :], axis=-1), cmap=plt.cm.RdBu,
                 aspect='auto', extent=[-50, 50, -15, 15]);

from ipywidgets import interact, IntSlider

interact(update, x=IntSlider(min=0, max=Q.shape[0]-1), v=IntSlider(min=0, max=Q.shape[1]-1));
# Horizontal: velocidad angular
# Vertical: angulo
```

##### Estudiando la influencia de $\gamma$
En este problema la recompensa es $+1$ por cada paso que equilibramos el péndulo

¿Cúanto pasos en el futuro es capaz de ver nuestro agente?

Es decir que la ganancia total descontada es (progresión geométrica)

$$
\sum_{t=0}^\infty R_t \gamma^t = \sum_{t=0}^\infty \gamma^t = \frac{1}{1-\gamma}
$$

¿Cuántos pasos estamos considerando si $\gamma = 0.9$? ¿Y $\gamma = 0.99$? ¿Y $\gamma = 0.999$?

#### Probando nuestro agente

```python
import numpy as np
import gym

env = gym.make("CartPole-v0")
```

```python
env.reset()
end = False
states = []
for k in range(500):
    env.render()
    states.append(env.state)
    s_current = get_state(env.state)
    a = np.argmax(Q[s_current])
    s_future, r, end, info = env.step(a)
    #if end:
    #    break
    if r == 0:
        display(k)
        break
display(end, k)
```

```python
env.close()
```

Seguimiento de los estados durante un episodio

```python
fig, ax = plt.subplots(4, tight_layout=True)
for i in range(4):
    if i > 2:
        ax[i].plot(np.array(states)[:, i]*180/np.pi)
    else:
        ax[i].plot(np.array(states)[:, i])

```

## Desafios en RL

- Formular adecuadamente los estados, acciones y recompesa
- Controlar el balance entre exploración y explotación
- Obtener datos que representan nuestra tarea (simulaciones?)

### Próximamente

- Deep Q-leaning: Aprender Q usando redes neuronales
- Policy gradients: Aprender la política directamente usando redes neuronales
    - Veremos también como extender a espacios de acción continuos

## Tarea: Taxi

En grupos de tres estudiantes resuelvan el ambiente ["Taxi-v3"](https://gym.openai.com/envs/Taxi-v3/)

Fecha de entrega: Lunes 30 de Marzo 23:59 al correo *phuijse at inf dot uach dot cl*

- Describa detalladamente el ambiente (estados posibles, acciones posibles, recompensas, etc)
- Describa el algoritmo de $\epsilon$ greedy Q-learning y sus parámetros
- Utilice $\epsilon$ greedy Q-learning para entrenar un agente que resuelva el problema, indique la mejor configuración de parámetros que encontró para este problema
- Muestre la tabla Q, escoja 4 estados y razone sobre los resultados obtenidos
- Muestre en una gráfica la evolución de la recompensa promedio de su agente y el número de pasos promedios
- Grabe un video o una animación de su agente ejecutando la tarea 5 veces

Cuide su salud, discutan el problema y la solución usando herramientas de teletrabajo (whereby.com, google talk, skype, slack)

```python

```


Table of Contents

1  Seq2Seq With Attention

1.1  Data Preparation

1.2  Model Implementation

1.2.1  Encoder

1.2.2  Attention

1.2.3  Decoder

1.2.4  Seq2Seq

1.3  Training Seq2Seq

1.4  Evaluating Seq2Seq

1.5  Summary

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(css_style='custom2.css', plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import os
import math
import time
import spacy
import random
import torch
import numpy as np
import torch.nn as nn
import torch.optim as optim
from torchtext.datasets import Multi30k
from torchtext.data import Field, BucketIterator

%watermark -a 'Ethen' -d -t -v -p numpy,torch,torchtext,spacy
```

    Ethen 2019-10-09 13:46:01

    CPython 3.6.4
    IPython 7.7.0

    numpy 1.16.5
    torch 1.1.0.post2
    torchtext 0.3.1
    spacy 2.1.6

# Seq2Seq With Attention

Seq2Seq framework involves a family of encoders and decoders, where the encoder encodes a source sequence into a fixed length vector from which the decoder picks up and aims to correctly generates the target sequence. The vanilla version of this type of architecture looks something along the lines of:

The RNN encoder has an input sequence $x_1, x_2, x_3, x_4$. We denote the encoder states by $c_1, c_2, c_3$. The encoder outputs a single output vector $c$ which is passed as input to the decoder. Like the encoder, the decoder is also a single-layered RNN, we denote the decoder states by $s_1, s_2, s_3$ and the network's output by $y_1, y_2, y_3, y_4$.
A problem with this vanilla architecture lies in the fact that the decoder needs to represent the entire input sequence $x_1, x_2, x_3, x_4$ as a single vector $c$, which can cause information loss. In other words, the fixed-length context vector is hypothesized to be the bottleneck in this framework.

The attention mechanism that we'll be introducing here extends the approach by allowing the model to soft search for parts of the source sequence that relevant to predicting the target sequence, which looks like the following:

The attention mechanism is located between the encoder and the decoder, its input is composed of the encoder's output vectors $h_1, h_2, h_3, h_4$ and the states of the decoder $s_0, s_1, s_2, s_3$, the attention's output is a sequence of vectors called context vectors denoted by $c_1, c_2, c_3, c_4$.

These context vectors enable the decoder to focus on certain parts of the input when predicting its output. Each context vector is a weighted sum of the encoder's output vectors $h_1, h_2, h_3, h_4$, where each vector $h_i$ contains information about the whole input sequence with a strong focus on the parts surrounding the i-th vector of the input sequence. The vectors $h_1, h_2, h_3, h_4$ are scaled by weights $\alpha_{ij}$ capturing the degree of relevance of input $x_j$ to output at time $i$, $y_i$. The context vectors $c_1, c_2, c_3, c_4$ are calculated by:

\begin{align}
c_i = \sum_{j=1}^4 a_{ij} h_j
\end{align}

The attention weights $a_{ij}$ are learned using an additional fully-connected network, denoted by $fc$, whose input consists of the decoder's hidden state $s_0, s_1, s_2, s_3$ and the encoder's output $h_1, h_2, h_3, h_4$. It's computation can be more formally defined by:

\begin{align}
a_{ij} = \frac{exp(e_{ij})}{\sum_{k=1}^4exp(e_{ik})}
\end{align}

Where:

\begin{align}
e_{ij} = fc(s_{i-1}, h_j)
\end{align}

As can be seen in the above image, the fully-connected network receives the concatenation of vectors $[s_{i-1}, h_i]$ as input at time step $i$. The network has a single fully-connected layer, the outputs of the layer, denoted by $e_{ij}$, are passed through a softmax function computing the attention weights, which lie in $[0,1]$.

Note that we are using the same fully-connected network for all the concatenated pairs $[s_{i-1},h_1], [s_{i-1},h_2], [s_{i-1},h_3], [s_{i-1},h_4]$, meaning there is a single network learning the attention weights.

To re-emphasize the attention weights $\alpha_{ij}$ reflects the importance of $h_j$ with respect to the previous hidden state $s_{i−1}$ in deciding the next state $s_i$ and generating $y_i$. A large $\alpha_{ij}$ attention weight causes the RNN to focus on input $x_j$ (represented by the encoder's output $h_j$), when predicting the output $y_i$.

We can talk through an iteration of the algorithm to see how it all ties together.

The first computation performed is the computation of vectors $h_1, h_2, h_3, h_4$ by the encoder. These are then used as inputs to the attention mechanism. This is where the decoder is first involved by inputting its initial state vector $s_0$ (note that for this initial state of the decoder, we often times use the hidden state from the encoder) and we have the first attention input sequence $[s_0, h_1], [s_0, h_2], [s_0, h_3], [s_0, h_4]$.

The attention mechanism picks up the inputs and computes the first set of attention weights $\alpha_{11}, \alpha_{12}, \alpha_{13}, \alpha_{14}$ enabling the computation of the first context vector $c_1$. The decoder now uses $[s_0,c_1]$ to generate the first output $y_1$. This process then repeats itself, until we've generated all the outputs.

## Data Preparation

This part is pretty much identical to that of the vanilla seq2seq, hence explanation is omitted.

```python
# !python -m spacy download de
# !python -m spacy download en
```

```python
SEED = 2222
random.seed(SEED)
torch.manual_seed(SEED)
```

    <torch._C.Generator at 0x11e3a86d0>

```python
# tokenize sentences into individual tokens
# https://spacy.io/usage/spacy-101#annotations-token
spacy_de = spacy.load('de_core_news_sm')
spacy_en = spacy.load('en_core_web_sm')

def tokenize_de(text):
    return [tok.text for tok in spacy_de.tokenizer(text)][::-1]

def tokenize_en(text):
    return [tok.text for tok in spacy_en.tokenizer(text)]
```

```python
source = Field(tokenize=tokenize_de, init_token='<sos>', eos_token='<eos>', lower=True)
target = Field(tokenize=tokenize_en, init_token='<sos>', eos_token='<eos>', lower=True)
```

```python
train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'), fields=(source, target))
print(f"Number of training examples: {len(train_data.examples)}")
print(f"Number of validation examples: {len(valid_data.examples)}")
print(f"Number of testing examples: {len(test_data.examples)}")
```

    Number of training examples: 29000
    Number of validation examples: 1014
    Number of testing examples: 1000

```python
train_data.examples[0].src
```

    ['.',
     'büsche',
     'vieler',
     'nähe',
     'der',
     'in',
     'freien',
     'im',
     'sind',
     'männer',
     'weiße',
     'junge',
     'zwei']

```python
train_data.examples[0].trg
```

    ['two',
     'young',
     ',',
     'white',
     'males',
     'are',
     'outside',
     'near',
     'many',
     'bushes',
     '.']

```python
source.build_vocab(train_data, min_freq=2)
target.build_vocab(train_data, min_freq=2)
print(f"Unique tokens in source (de) vocabulary: {len(source.vocab)}")
print(f"Unique tokens in target (en) vocabulary: {len(target.vocab)}")
```

    Unique tokens in source (de) vocabulary: 7855
    Unique tokens in target (en) vocabulary: 5893

```python
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

```python
BATCH_SIZE = 128

# create batches out of the dataset and sends them to the appropriate device
train_iterator, valid_iterator, test_iterator = BucketIterator.splits(
    (train_data, valid_data, test_data), batch_size=BATCH_SIZE, device=device)
```

```python
test_batch = next(iter(test_iterator))
test_batch
```

    [torchtext.data.batch.Batch of size 128 from MULTI30K]
    	[.src]:[torch.LongTensor of size 10x128]
    	[.trg]:[torch.LongTensor of size 14x128]

## Model Implementation

```python
# adjustable parameters
INPUT_DIM = len(source.vocab)
OUTPUT_DIM = len(target.vocab)
ENC_EMB_DIM = 256
DEC_EMB_DIM = 256
ENC_HID_DIM = 512
DEC_HID_DIM = 512
N_LAYERS = 1
ENC_DROPOUT = 0.5
DEC_DROPOUT = 0.5
```

The following sections are heavily "borrowed" from the wonderful tutorial on this topic listed below.

- [Jupyter Notebook: Neural Machine Translation by Jointly Learning to Align and Translate](https://nbviewer.jupyter.org/github/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb)

Some personal preference modifications have been made.

### Encoder

Like other seq2seq-like architectures, we first need to specify an encoder. Here we'll be using a bidirectional GRU layer. With a bidirectional layer, we have a forward layer scanning the sentence from left to right (shown below in green), and a backward layer scanning the sentence from right to left (yellow). From the coding perspective, we need to set the `bidirectional=True` for the GRU layer's argument.

More formally, we now have:

$$
\begin{align}
h_t^\rightarrow &= \text{EncoderGRU}^\rightarrow(x_t^\rightarrow,h_{t-1}^\rightarrow)\\
h_t^\leftarrow &= \text{EncoderGRU}^\leftarrow(x_t^\leftarrow,h_{t-1}^\leftarrow)
\end{align}
$$

Where $x_0^\rightarrow = \text{<sos>}, x_1^\rightarrow = \text{guten}$ and $x_0^\leftarrow = \text{<eos>}, x_1^\leftarrow = \text{morgen}$.

As before, we only pass an embedded input to our GRU layer. We'll get two context vectors, one from the forward layer after it has seen the final word in the sentence, $z^\rightarrow=h_T^\rightarrow$, and one from the backward layer after it has seen the first word in the sentence, $z^\leftarrow=h_T^\leftarrow$.

As we'll be using bidirectional layer, the next section is devoted to help us understand how the output looks like before we implement the actual encoder that we'll be using. The shape of the output is explicitly printed out to make it easier to comprehend. Here, we're using GRU layer, which can be replaced with a LSTM layer, which is similar, but return an additional cell state variable that has the same size as the hidden state.

```python
class Encoder(nn.Module):
    def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
        super().__init__()
        self.emb_dim = emb_dim
        self.hid_dim = hid_dim
        self.input_dim = input_dim
        self.n_layers = n_layers
        self.dropout = dropout

        self.embedding = nn.Embedding(input_dim, emb_dim)
        self.rnn = nn.GRU(emb_dim, hid_dim, n_layers, dropout=dropout,
                          bidirectional=True)

    def forward(self, src_batch):
        # src [sent len, batch size]
        embedded = self.embedding(src_batch) # [sent len, batch size, emb dim]
        outputs, hidden = self.rnn(embedded)  # [sent len, batch size, hidden dim]
        # outputs -> [sent len, batch size, hidden dim * n directions]
        # hidden -> [n layers * n directions, batch size, hidden dim]
        return outputs, hidden
```

```python
# first experiment with n_layers = 1
n_layers = 1
encoder = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, n_layers, ENC_DROPOUT).to(device)
outputs, hidden = encoder(test_batch.src)
outputs.shape, hidden.shape
```

    (torch.Size([10, 128, 1024]), torch.Size([2, 128, 512]))

Notice that output's last dimension is 1024, which is the hidden dimension (512) multiplied by the number of directions (2). Whereas the hidden's first dimension is 2, representing the number of directions (2).

- The returned outputs of bidirectional RNN at timestep $t$ is the output after feeding input to both the reverse and normal RNN unit at timestep $t$, where normal RNN has seen inputs $1...t$ and reverse RNN has seen inputs $t...n$, with $n$ being the length of the sequence).
- The returned hidden state of bidirectional RNN is the hidden state after the whole sequence is consume. For normal RNN it's after timestep $n$; for reverse RNN it's after timestep 1.

The following diagram can also come in handy when visualizing the difference between output and hidden.

In the diagram $n$ notes each timestep, and $w$ denotes the number of layer.

- output comprises all the hidden states in the last layer ("last" depth-wise, not time-wise).
- ($h_n$, $c_n$) comprise of the hidden states after the last timestep, $t = n$, so we could potentially feed them into another LSTM layer.

```python
# the outputs are concatenated at the last dimension
assert (outputs[-1, :, :ENC_HID_DIM] == hidden[0]).all()
assert (outputs[0, :, ENC_HID_DIM:] == hidden[1]).all()
```

```python
# experiment with n_layers = 2
n_layers = 2
encoder = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, n_layers, ENC_DROPOUT).to(device)
outputs, hidden = encoder(test_batch.src)
outputs.shape, hidden.shape
```

    (torch.Size([10, 128, 1024]), torch.Size([4, 128, 512]))

Notice now the first dimension of the hidden cell becomes 4, which represents the number of layers (2) multiplied by the number of directions (2). The order of the hidden state is stacked by [forward_1, backward_1, forward_2, backward_2, ...]

```python
assert (outputs[-1, :, :ENC_HID_DIM] == hidden[2]).all()
assert (outputs[0, :, ENC_HID_DIM:] == hidden[3]).all()
```

We'll need some final touches for our actual encoder. As our encoder's hidden state will be used as the decoder's initial hidden state, we need to make sure we make them the same shape. In our example, the decoder is not bidirectional, and only needs a single context vector, $z$, to use as its initial hidden state, $s_0$, and we currently have two, a forward and a backward one ($z^\rightarrow=h_T^\rightarrow$ and $z^\leftarrow=h_T^\leftarrow$, respectively). We solve this by concatenating the two context vectors together, passing them through a linear layer, $g$, and applying the $\tanh$ activation function.

$$
\begin{align}
z=\tanh(g(h_T^\rightarrow, h_T^\leftarrow)) = \tanh(g(z^\rightarrow, z^\leftarrow)) = s_0
\end{align}
$$

```python
class Encoder(nn.Module):
    def __init__(self, input_dim, emb_dim, enc_hid_dim, dec_hid_dim, n_layers, dropout):
        super().__init__()
        self.emb_dim = emb_dim
        self.enc_hid_dim = enc_hid_dim
        self.dec_hid_dim = dec_hid_dim
        self.input_dim = input_dim
        self.n_layers = n_layers
        self.dropout = dropout

        self.embedding = nn.Embedding(input_dim, emb_dim)
        self.rnn = nn.GRU(emb_dim, enc_hid_dim, n_layers, dropout=dropout,
                          bidirectional=True)
        self.fc = nn.Linear(enc_hid_dim * 2, dec_hid_dim)

    def forward(self, src_batch):
        # src [sent len, batch size]

        # [sent len, batch size, emb dim]
        embedded = self.embedding(src_batch)
        outputs, hidden = self.rnn(embedded)
        # outputs -> [sent len, batch size, hidden dim * n directions]
        # hidden -> [n layers * n directions, batch size, hidden dim]

        # initial decoder hidden is final hidden state of the forwards and
        # backwards encoder RNNs fed through a linear layer
        concated = torch.cat((hidden[-2, :, :], hidden[-1, :, :]), dim=1)
        hidden = torch.tanh(self.fc(concated))
        return outputs, hidden
```

```python
encoder = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, N_LAYERS, ENC_DROPOUT).to(device)
outputs, hidden = encoder(test_batch.src)
outputs.shape, hidden.shape
```

    (torch.Size([10, 128, 1024]), torch.Size([128, 512]))

### Attention

The next part is the hightlight. The attention layer will take in the previous hidden state of the decoder $s_{t-1}$, and all of the stacked forward and backward hidden state from the encoder $H$. The output will be an attention vector $a_t$, that is the length of the source sentece, each element of this vector will be a floating number between 0 and 1, and the entire vector sums up to 1.

Intuitively, this layer takes in what we've decoded so far $s_{t-1}$, and all of what have encoded $H$, to produce a vector $a_t$, that represents which word in the source sentence should we pay the most attention to in order to correctly predict the next thing in the target sequence $y_{t+1}$.

Graphically, this looks something like below. For the very first attention vector, where we use the encoder's hidden state as the initial hidden state from the decoder. The green/yellow blocks represent the hidden states from both the forward and backward RNNs, and the attention computation is all done within the pink block.

```python
class Attention(nn.Module):

    def __init__(self, enc_hid_dim, dec_hid_dim):
        super().__init__()
        self.enc_hid_dim = enc_hid_dim
        self.dec_hid_dim = dec_hid_dim

        self.fc1 = nn.Linear(enc_hid_dim * 2 + dec_hid_dim, dec_hid_dim)
        self.fc2 = nn.Parameter(torch.rand(dec_hid_dim))

    def forward(self, encoder_outputs, hidden):
        sent_len = encoder_outputs.shape[0]
        batch_size = encoder_outputs.shape[1]

        # the attention mechanism receives a concatenation of the hidden state
        # and the encoder output

        # repeat hidden state sent_len times [batch size, sent len, dec hid dim]
        hidden = hidden.unsqueeze(1).repeat(1, sent_len, 1)
        # reshape/permute the encoder output, so that the batch size coems first
        # [batch size, sent len, enc hid dim * 2], times 2 because of bidirectional
        outputs = encoder_outputs.permute(1, 0, 2)
        concat = torch.cat((hidden, outputs), dim=2)

        # fully connected layer to compute the energy and softmax layer to compute
        # the attention weight
        # [batch size, dec hid dim, sent len]
        fc1_output = torch.tanh(self.fc1(concat)).permute(0, 2, 1)

        # [batch size, 1, dec hid dim]
        fc2 = self.fc2.repeat(batch_size, 1).unsqueeze(1)

        # [batch size, sent len]
        energy = torch.bmm(fc2, fc1_output).squeeze(1)
        attention = torch.softmax(energy, dim=1)
        return attention
```

```python
attention = Attention(ENC_HID_DIM, DEC_HID_DIM).to(device)
attention_weight = attention(outputs, hidden)
attention_weight.shape
```

    torch.Size([128, 10])

### Decoder

Now comes the decoder, within the decoder, we first use the attention layer that we've created in the previous section to compute the attention weight, this gives us the weight for each source sentence that the model should pay attention to when generating the current target output in the sequence. Along with the output from the encoder, this gives us the context vector. Finally, the decoder takes the embedded input along with the context to generate the target output in the sequence.

```python
class Decoder(nn.Module):

    def __init__(self, output_dim, emb_dim, enc_hid_dim, dec_hid_dim, n_layers,
                 dropout, attention):
        super().__init__()
        self.emb_dim = emb_dim
        self.enc_hid_dim = enc_hid_dim
        self.dec_hid_dim = dec_hid_dim
        self.output_dim = output_dim
        self.n_layers = n_layers
        self.dropout = dropout
        self.attention = attention

        self.embedding = nn.Embedding(output_dim, emb_dim)
        self.rnn = nn.GRU(enc_hid_dim * 2 + emb_dim, dec_hid_dim, n_layers, dropout=dropout)
        self.linear = nn.Linear(dec_hid_dim, output_dim)

    def forward(self, trg, encoder_outputs, hidden):
        # trg [batch size]
        # outputs [src sen len, batch size, enc hid dim * 2], times 2 due to bidirectional
        # hidden [batch size, dec hid dim]

        # [batch size, 1, sent len]
        attention = self.attention(encoder_outputs, hidden).unsqueeze(1)

        # [batch size, sent len, enc hid dim * 2]
        outputs = encoder_outputs.permute(1, 0, 2)

        # [1, batch size, enc hid dim * 2]
        context = torch.bmm(attention, outputs).permute(1, 0, 2)

        # input sentence -> embedding
        # [1, batch size, emb dim]
        embedded = self.embedding(trg.unsqueeze(0))
        rnn_input = torch.cat((embedded, context), dim=2)

        outputs, hidden = self.rnn(rnn_input, hidden.unsqueeze(0))
        prediction = self.linear(outputs.squeeze(0))
        return prediction, hidden.squeeze(0)
```

```python
decoder = Decoder(OUTPUT_DIM, DEC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, N_LAYERS, DEC_DROPOUT, attention).to(device)
prediction, decoder_hidden = decoder(test_batch.trg[0], outputs, hidden)

# notice the decoder_hidden's shape should match the shape that's generated by
# the encoder
prediction.shape, decoder_hidden.shape
```

    (torch.Size([128, 5893]), torch.Size([128, 512]))

### Seq2Seq

This part is about putting the encoder and decoder together and is very much identical to the vanilla seq2seq framework, hence the explanation is omitted.

```python
class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder, device):
        super().__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.device = device

    def forward(self, src_batch, trg_batch, teacher_forcing_ratio=0.5):
        max_len, batch_size = trg_batch.shape
        trg_vocab_size = self.decoder.output_dim

        # tensor to store decoder's output
        outputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)

        # encoder_outputs : all hidden states of the input sequence (forward and backward)
        # hidden : final forward and backward hidden states, passed through a linear layer
        encoder_outputs, hidden = self.encoder(src_batch)

        trg = trg_batch[0]
        for i in range(1, max_len):
            prediction, hidden = self.decoder(trg, encoder_outputs, hidden)
            outputs[i] = prediction

            if random.random() < teacher_forcing_ratio:
                trg = trg_batch[i]
            else:
                trg = prediction.argmax(1)

        return outputs
```

```python
attention = Attention(ENC_HID_DIM, DEC_HID_DIM)
encoder = Encoder(INPUT_DIM, ENC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, N_LAYERS, ENC_DROPOUT)
decoder = Decoder(OUTPUT_DIM, DEC_EMB_DIM, ENC_HID_DIM, DEC_HID_DIM, N_LAYERS, DEC_DROPOUT, attention)
seq2seq = Seq2Seq(encoder, decoder, device).to(device)
seq2seq
```

    Seq2Seq(
      (encoder): Encoder(
        (embedding): Embedding(7855, 256)
        (rnn): GRU(256, 512, dropout=0.5, bidirectional=True)
        (fc): Linear(in_features=1024, out_features=512, bias=True)
      )
      (decoder): Decoder(
        (attention): Attention(
          (fc1): Linear(in_features=1536, out_features=512, bias=True)
        )
        (embedding): Embedding(5893, 256)
        (rnn): GRU(1280, 512, dropout=0.5)
        (linear): Linear(in_features=512, out_features=5893, bias=True)
      )
    )

```python
outputs = seq2seq(test_batch.src, test_batch.trg)
outputs.shape
```

    torch.Size([14, 128, 5893])

```python
def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(seq2seq):,} trainable parameters')
```

    The model has 12,975,877 trainable parameters

## Training Seq2Seq

We've done the hard work of defining our seq2seq module. The final touch is to specify the training/evaluation loop.

```python
optimizer = optim.Adam(seq2seq.parameters())

# ignore the padding index when calculating the loss
PAD_IDX = target.vocab.stoi['<pad>']
criterion = nn.CrossEntropyLoss(ignore_index=PAD_IDX)
```

```python
def train(seq2seq, iterator, optimizer, criterion):
    seq2seq.train()

    epoch_loss = 0
    for batch in iterator:
        optimizer.zero_grad()
        outputs = seq2seq(batch.src, batch.trg)

        # the loss function only works on 2d inputs
        # and 1d targets we need to flatten each of them
        outputs_flatten = outputs[1:].view(-1, outputs.shape[-1])
        trg_flatten = batch.trg[1:].view(-1)
        loss = criterion(outputs_flatten, trg_flatten)

        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()

    return epoch_loss / len(iterator)
```

```python
def evaluate(seq2seq, iterator, criterion):
    seq2seq.eval()

    epoch_loss = 0
    with torch.no_grad():
        for batch in iterator:
            # turn off teacher forcing
            outputs = seq2seq(batch.src, batch.trg, teacher_forcing_ratio=0)

            # trg = [trg sent len, batch size]
            # output = [trg sent len, batch size, output dim]
            outputs_flatten = outputs[1:].view(-1, outputs.shape[-1])
            trg_flatten = batch.trg[1:].view(-1)
            loss = criterion(outputs_flatten, trg_flatten)
            epoch_loss += loss.item()

    return epoch_loss / len(iterator)
```

```python
def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs
```

```python
N_EPOCHS = 10
best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss = train(seq2seq, train_iterator, optimizer, criterion)
    valid_loss = evaluate(seq2seq, valid_iterator, criterion)
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)

    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(seq2seq.state_dict(), 'tut2-model.pt')

    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')
```

    Epoch: 01 | Time: 2m 30s
    	Train Loss: 4.844 | Train PPL: 126.976
    	 Val. Loss: 4.691 |  Val. PPL: 108.948
    Epoch: 02 | Time: 2m 30s
    	Train Loss: 3.948 | Train PPL:  51.808
    	 Val. Loss: 4.004 |  Val. PPL:  54.793
    Epoch: 03 | Time: 2m 31s
    	Train Loss: 3.230 | Train PPL:  25.281
    	 Val. Loss: 3.498 |  Val. PPL:  33.059
    Epoch: 04 | Time: 2m 29s
    	Train Loss: 2.733 | Train PPL:  15.379
    	 Val. Loss: 3.413 |  Val. PPL:  30.360
    Epoch: 05 | Time: 2m 28s
    	Train Loss: 2.379 | Train PPL:  10.793
    	 Val. Loss: 3.269 |  Val. PPL:  26.285
    Epoch: 06 | Time: 2m 32s
    	Train Loss: 2.089 | Train PPL:   8.079
    	 Val. Loss: 3.228 |  Val. PPL:  25.229
    Epoch: 07 | Time: 2m 29s
    	Train Loss: 1.862 | Train PPL:   6.438
    	 Val. Loss: 3.201 |  Val. PPL:  24.561
    Epoch: 08 | Time: 2m 30s
    	Train Loss: 1.626 | Train PPL:   5.084
    	 Val. Loss: 3.297 |  Val. PPL:  27.044
    Epoch: 09 | Time: 2m 30s
    	Train Loss: 1.406 | Train PPL:   4.078
    	 Val. Loss: 3.312 |  Val. PPL:  27.451
    Epoch: 10 | Time: 2m 31s
    	Train Loss: 1.239 | Train PPL:   3.453
    	 Val. Loss: 3.467 |  Val. PPL:  32.050

## Evaluating Seq2Seq

```python
seq2seq.load_state_dict(torch.load('tut2-model.pt'))

test_loss = evaluate(seq2seq, test_iterator, criterion)
print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')
```

    | Test Loss: 3.237 | Test PPL:  25.467 |

Here, we pick a random example in our dataset, print out the original source and target sentence. Then takes a look at whether the "predicted" target sentence generated by the model.

```python
example_idx = 0
example = train_data.examples[example_idx]
print('source sentence: ', ' '.join(example.src))
print('target sentence: ', ' '.join(example.trg))
```

    source sentence:  . büsche vieler nähe der in freien im sind männer weiße junge zwei
    target sentence:  two young , white males are outside near many bushes .

```python
src_tensor = source.process([example.src]).to(device)
trg_tensor = target.process([example.trg]).to(device)
print(trg_tensor.shape)

seq2seq.eval()
with torch.no_grad():
    outputs = seq2seq(src_tensor, trg_tensor, teacher_forcing_ratio=0)

outputs.shape
```

    torch.Size([13, 1])

    torch.Size([13, 1, 5893])

```python
output_idx = outputs[1:].squeeze(1).argmax(1)
' '.join([target.vocab.itos[idx] for idx in output_idx])
```

    'two young white white males are outside near near some trees .'

## Summary

- Upon implementing the attention mechanism, we were able to achieve a better evaluation score on the test set, while even using less parameters. As mentioned in the original paper:

> We extended the basic encoder–decoder by letting a model (soft-)search for a set of input words. This frees the model from having to encode the whole source sentence into a fixed-length vector, and also lets the model focus only on information relevant to the generation of the next target word. This has a major positive impact on the ability of the neural machine translation system to yield good results on longer sentences.

- Note that another interesting thing that we're capable of doing but wasn't done here is to visualize the attention weight to see for a given translation, where the model is focusing on.

# Reference

- [Blog: Attention in RNNs](https://medium.com/datadriveninvestor/attention-in-rnns-321fbcd64f05)
- [Stackoverflow: What's the difference between “hidden” and “output” in PyTorch LSTM?](https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm)
- [Jupyter Notebook: Neural Machine Translation by Jointly Learning to Align and Translate](https://nbviewer.jupyter.org/github/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb)
- [Paper: D. Bahdanau, K. Cho, Y. Bengio (2014) - Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)


# Explicit Runge Kutta methods and their Butcher tables
## Authors: Brandon Clark & Zach Etienne

## This tutorial notebook stores known explicit Runge Kutta-like methods as Butcher tables in a Python dictionary format.

**Notebook Status:**  Validated

**Validation Notes:** This tutorial notebook has been confirmed to be **self-consistent with its corresponding NRPy+ module**, as documented [below](#code_validation). In addition, each of these Butcher tables has been verified to yield an RK method to the expected local truncation error in a challenging battery of ODE tests, in the [RK Butcher Table Validation tutorial notebook](Tutorial-RK_Butcher_Table_Validation.ipynb).

### NRPy+ Source Code for this module: [MoLtimestepping/RK_Butcher_Table_Dictionary.py](../edit/MoLtimestepping/RK_Butcher_Table_Dictionary.py)

## Introduction:

The family of explicit [Runge Kutta](https://en.wikipedia.org/w/index.php?title=Runge%E2%80%93Kutta_methods&oldid=898536315)-like methods are commonly used when numerically solving ordinary differential equation (ODE) initial value problems of the form

$$ y'(t) = f(y,t),\ \ \ y(t_0)=y_0.$$

These methods can be extended to solve time-dependent partial differential equations (PDEs) via the [Method of Lines](https://en.wikipedia.org/w/index.php?title=Method_of_lines&oldid=855390257). In the Method of Lines, the above ODE can be generalized to $N$ coupled ODEs, all written as first-order-in-time PDEs of the form

$$ \partial_{t}\mathbf{u}(t,x,y,u_1,u_2,u_3,...)=\mathbf{f}(t,x,y,...,u_1,u_{1,x},...),$$

where $\mathbf{u}$ and $\mathbf{f}$ are vectors. The spatial partial derivatives of components of $\mathbf{u}$, e.g., $u_{1,x}$, may be computed using approximate numerical differentiation, like finite differences.

As any explicit Runge-Kutta method has its own unique local truncation error, can in principle be used to solve time-dependent PDEs using the Method of Lines, and may be stable under different Courant-Friedrichs-Lewy (CFL) conditions, it is useful to have multiple methods at one's disposal. **This module provides a number of such methods.**

More details about the Method of Lines is discussed further in the [Tutorial-RK_Butcher_Table_Generating_C_Code](Tutorial-RK_Butcher_Table_Generating_C_Code.ipynb) module where we generate the C code to implement the Method of Lines, and additional description can be found in the [Numerically Solving the Scalar Wave Equation: A Complete C Code](Tutorial-Start_to_Finish-ScalarWave.ipynb) NRPy+ tutorial notebook.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Initialize needed Python modules
1. [Step 2](#introbutcher): The Family of Explicit Runge-Kutta-Like Schemes (Butcher Tables)
    1. [Step 2a](#codebutcher): Generating a Dictionary of Butcher Tables for Explicit Runge Kutta Techniques
        1. [Step 2.a.i](#euler): Euler's Method
        1. [Step 2.a.ii](#rktwoheun): RK2 Heun's Method
        1. [Step 2.a.iii](#rk2mp): RK2 Midpoint Method
        1. [Step 2.a.iv](#rk2ralston): RK2 Ralston's Method
        1. [Step 2.a.v](#rk3): Kutta's Third-order Method
        1. [Step 2.a.vi.](#rk3heun): RK3 Heun's Method
        1. [Step 2.a.vii](#rk3ralston): RK3 Ralston's Method
        1. [Step 2.a.viii](#ssprk3): Strong Stability Preserving Runge-Kutta (SSPRK3) Method
        1. [Step 2.a.ix](#rkfour): Classic RK4 Method
        1. [Step 2.a.x](#dp5): RK5 Dormand-Prince Method
        1. [Step 2.a.xi](#dp5alt): RK5 Dormand-Prince Method Alternative
        1. [Step 2.a.xii](#ck5): RK5 Cash-Karp Method
        1. [Step 2.a.xiii](#dp6): RK6 Dormand-Prince Method
        1. [Step 2.a.xiv](#l6): RK6 Luther Method
        1. [Step 2.a.xv](#dp8): RK8 Dormand-Prince Method
1. [Step 3](#code_validation): Code Validation against `MoLtimestepping.RK_Butcher_Table_Dictionary` NRPy+ module
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize needed Python modules [Back to [top](#toc)\]
$$\label{initializenrpy}$$

Let's start by importing all the needed modules from Python:

```python
# Step 1: Initialize needed Python modules
import sympy as sp
```

# Step 2: The Family of Explicit Runge-Kutta-Like Schemes (Butcher Tables) [Back to [top](#toc)\]
$$\label{introbutcher}$$

In general, a predictor-corrector method performs an estimate timestep from $n$ to $n+1$, using e.g., a Runge Kutta method, to get a prediction of the solution at timestep $n+1$. This is the "predictor" step. Then it uses this prediction to perform another, "corrector" step, designed to increase the accuracy of the solution.

Let us focus on the ordinary differential equation (ODE)

$$ y'(t) = f(y,t), $$

which acts as an analogue for a generic PDE $\partial_{t}u(t,x,y,...)=f(t,x,y,...,u,u_x,...)$.

The general family of Runge Kutta "explicit" timestepping methods are implemented using the following scheme:

$$y_{n+1} = y_n + \sum_{i=1}^s b_ik_i $$

where

\begin{align}
k_1 &= \Delta tf(y_n, t_n) \\
k_2 &= \Delta tf(y_n + [a_{21}k_1], t_n + c_2\Delta t) \\
k_3 &= \Delta tf(y_n +[a_{31}k_1 + a_{32}k_2], t_n + c_3\Delta t) \\
& \ \ \vdots \\
k_s &= \Delta tf(y_n +[a_{s1}k_1 + a_{s2}k_2 + \cdots + a_{s, s-1}k_{s-1}], t_n + c_s\Delta t)
\end{align}

Note $s$ is the number of right-hand side evaluations necessary for any given method, i.e., for RK2 $s=2$ and for RK4 $s=4$, and for RK6 $s=7$. These schemes are often written in the form of a so-called "Butcher tableau". or "Butcher table":

$$\begin{array}{c|ccccc}
    0 & \\
    c_2 & a_{21} & \\
    c_3 & a_{31} & a_{32} & \\
    \vdots & \vdots &  & \ddots \\
    c_s & a_{s_1} & a_{s2} & \cdots & a_{s,s-1} \\ \hline
     & b_1 & b_2 & \cdots & b_{s-1} & b_s
\end{array} $$

As an example, the "classic" fourth-order Runge Kutta (RK4) method obtains the solution $y(t)$ to the single-variable ODE $y'(t) = f(y(t),t)$ at time $t_{n+1}$ from $t_n$ via:

\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + \frac{1}{2}k_1, t_n + \frac{\Delta t}{2}), \\
k_3 &= \Delta tf(y_n + \frac{1}{2}k_2, t_n + \frac{\Delta t}{2}), \\
k_4 &= \Delta tf(y_n + k_3, t_n + \Delta t), \\
y_{n+1} &= y_n + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4) + \mathcal{O}\big((\Delta t)^5\big).
\end{align}

It's corresponding Butcher table is constructed as follows:

$$\begin{array}{c|cccc}
    0 & \\
    1/2 & 1/2 & \\
    1/2 & 0 & 1/2 & \\
    1 & 0 & 0 & 1 & \\ \hline
     & 1/6 & 1/3 & 1/3 & 1/6
\end{array} $$

This is one example of many explicit [Runge Kutta methods](https://en.wikipedia.org/w/index.php?title=List_of_Runge%E2%80%93Kutta_methods&oldid=896594269). Throughout the following sections we will highlight different Runge Kutta schemes and their Butcher tables from the first-order Euler's method up to and including an eighth-order method.

## Step 2.a: Generating a Dictionary of Butcher Tables for Explicit Runge Kutta Techniques   [Back to [top](#toc)\]
$$\label{codebutcher}$$

We can store all of the Butcher tables in Python's **Dictionary** format using the curly brackets {} and 'key':value pairs. The 'key' will be the *name* of the Runge Kutta method and the value will be the Butcher table itself stored as a list of lists. The convergence order for each Runge Kutta method is also stored. We will construct the dictionary `Butcher_dict` one Butcher table at a time in the following sections.

```python
# Step 2a: Generating a Dictionary of Butcher Tables for Explicit Runge Kutta Techniques

# Initialize the dictionary Butcher_dict
Butcher_dict = {}
```

### Step 2.a.i: Euler's Method  [Back to [top](#toc)\]
$$\label{euler}$$

[Forward Euler's method](https://en.wikipedia.org/w/index.php?title=Euler_method&oldid=896152463) is a first order Runge Kutta method. Euler's method obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:
$$y_{n+1} = y_{n} + \Delta tf(y_{n}, t_{n})$$
with the trivial corresponding Butcher table
$$\begin{array}{c|c}
0 & \\ \hline
 & 1
\end{array}$$

```python
# Step 2.a.i: Euler's Method

Butcher_dict['Euler'] = (
[[sp.sympify(0)],
["", sp.sympify(1)]]
, 1)
```

### Step 2.a.ii: RK2 Heun's Method [Back to [top](#toc)\]
$$\label{rktwoheun}$$

[Heun's method](https://en.wikipedia.org/w/index.php?title=Heun%27s_method&oldid=866896936) is a second-order RK method that obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:
\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + k_1, t_n + \Delta t), \\
y_{n+1} &= y_n + \frac{1}{2}(k_1 + k_2) + \mathcal{O}\big((\Delta t)^3\big).
\end{align}
with corresponding Butcher table
$$\begin{array}{c|cc}
    0 & \\
    1 & 1 & \\ \hline
     & 1/2 & 1/2
\end{array} $$

```python
# Step 2.a.ii: RK2 Heun's Method

Butcher_dict['RK2 Heun'] = (
[[sp.sympify(0)],
[sp.sympify(1), sp.sympify(1)],
["", sp.Rational(1,2), sp.Rational(1,2)]]
, 2)
```

### Step 2.a.iii: RK2 Midpoint Method [Back to [top](#toc)\]
$$\label{rk2mp}$$

[Midpoint method](https://en.wikipedia.org/w/index.php?title=Midpoint_method&oldid=886630580) is a second-order RK method that obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:
\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + \frac{2}{3}k_1, t_n + \frac{2}{3}\Delta t), \\
y_{n+1} &= y_n + \frac{1}{2}k_2 + \mathcal{O}\big((\Delta t)^3\big).
\end{align}
with corresponding Butcher table
$$\begin{array}{c|cc}
    0 & \\
    1/2 & 1/2 & \\ \hline
     & 0 & 1
\end{array} $$

```python
# Step 2.a.iii: RK2 Midpoint (MP) Method

Butcher_dict['RK2 MP'] = (
[[sp.sympify(0)],
[sp.Rational(1,2), sp.Rational(1,2)],
["", sp.sympify(0), sp.sympify(1)]]
, 2)
```

### Step 2.a.iv: RK2 Ralston's Method [Back to [top](#toc)\]
$$\label{rk2ralston}$$

Ralston's method (see [Ralston (1962)](https://www.ams.org/journals/mcom/1962-16-080/S0025-5718-1962-0150954-0/S0025-5718-1962-0150954-0.pdf), is a second-order RK method that obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:
\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + \frac{1}{2}k_1, t_n + \frac{1}{2}\Delta t), \\
y_{n+1} &= y_n + \frac{1}{4}k_1 + \frac{3}{4}k_2 + \mathcal{O}\big((\Delta t)^3\big).
\end{align}
with corresponding Butcher table
$$\begin{array}{c|cc}
    0 & \\
    2/3 & 2/3 & \\ \hline
     & 1/4 & 3/4
\end{array} $$

```python
# Step 2.a.iv: RK2 Ralston's Method

Butcher_dict['RK2 Ralston'] = (
[[sp.sympify(0)],
[sp.Rational(2,3), sp.Rational(2,3)],
["", sp.Rational(1,4), sp.Rational(3,4)]]
, 2)

```

### Step 2.a.v: Kutta's  Third-order Method [Back to [top](#toc)\]
$$\label{rk3}$$

[Kutta's third-order method](https://en.wikipedia.org/w/index.php?title=List_of_Runge%E2%80%93Kutta_methods&oldid=896594269) obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:
\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + \frac{1}{2}k_1, t_n + \frac{1}{2}\Delta t), \\
k_3 &= \Delta tf(y_n - k_1 + 2k_2, t_n + \Delta t) \\
y_{n+1} &= y_n + \frac{1}{6}k_1 + \frac{2}{3}k_2 + \frac{1}{6}k_3 + \mathcal{O}\big((\Delta t)^4\big).
\end{align}
with corresponding Butcher table
\begin{array}{c|ccc}
    0 & \\
    1/2 & 1/2 & \\
    1 & -1 & 2 & \\ \hline
     & 1/6 & 2/3 & 1/6
\end{array}

```python
# Step 2.a.v: Kutta's  Third-order Method

Butcher_dict['RK3'] = (
[[sp.sympify(0)],
[sp.Rational(1,2), sp.Rational(1,2)],
[sp.sympify(1), sp.sympify(-1), sp.sympify(2)],
["", sp.Rational(1,6), sp.Rational(2,3), sp.Rational(1,6)]]
, 3)
```

### Step 2.a.vi: RK3 Heun's Method [Back to [top](#toc)\]
$$\label{rk3heun}$$

[Heun's third-order method](https://en.wikipedia.org/w/index.php?title=List_of_Runge%E2%80%93Kutta_methods&oldid=896594269) obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:

\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + \frac{1}{3}k_1, t_n + \frac{1}{3}\Delta t), \\
k_3 &= \Delta tf(y_n + \frac{2}{3}k_2, t_n + \frac{2}{3}\Delta t) \\
y_{n+1} &= y_n + \frac{1}{4}k_1 + \frac{3}{4}k_3 + \mathcal{O}\big((\Delta t)^4\big).
\end{align}

with corresponding Butcher table

\begin{array}{c|ccc}
    0 & \\
    1/3 & 1/3 & \\
    2/3 & 0 & 2/3 & \\ \hline
     & 1/4 & 0 & 3/4
\end{array}

```python
# Step 2.a.vi: RK3 Heun's Method

Butcher_dict['RK3 Heun'] = (
[[sp.sympify(0)],
[sp.Rational(1,3), sp.Rational(1,3)],
[sp.Rational(2,3), sp.sympify(0), sp.Rational(2,3)],
["", sp.Rational(1,4), sp.sympify(0), sp.Rational(3,4)]]
, 3)
```

### Step 2.a.vii: RK3 Ralton's Method [Back to [top](#toc)\]
$$\label{rk3ralston}$$

Ralston's third-order method (see [Ralston (1962)](https://www.ams.org/journals/mcom/1962-16-080/S0025-5718-1962-0150954-0/S0025-5718-1962-0150954-0.pdf), obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:

\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + \frac{1}{2}k_1, t_n + \frac{1}{2}\Delta t), \\
k_3 &= \Delta tf(y_n + \frac{3}{4}k_2, t_n + \frac{3}{4}\Delta t) \\
y_{n+1} &= y_n + \frac{2}{9}k_1 + \frac{1}{3}k_2 + \frac{4}{9}k_3 + \mathcal{O}\big((\Delta t)^4\big).
\end{align}

with corresponding Butcher table

\begin{array}{c|ccc}
    0 & \\
    1/2 & 1/2 & \\
    3/4 & 0 & 3/4 & \\ \hline
     & 2/9 & 1/3 & 4/9
\end{array}

```python
# Step 2.a.vii: RK3 Ralton's Method

Butcher_dict['RK3 Ralston'] = (
[[0],
[sp.Rational(1,2), sp.Rational(1,2)],
[sp.Rational(3,4), sp.sympify(0), sp.Rational(3,4)],
["", sp.Rational(2,9), sp.Rational(1,3), sp.Rational(4,9)]]
, 3)
```

### Step 2.a.viii: Strong Stability Preserving Runge-Kutta (SSPRK3) Method [Back to [top](#toc)\]
$\label{ssprk3}$

The [Strong Stability Preserving Runge-Kutta (SSPRK3)](https://en.wikipedia.org/wiki/List_of_Runge%E2%80%93Kutta_methods#Kutta's_third-order_method) method obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:

\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + k_1, t_n + \Delta t), \\
k_3 &= \Delta tf(y_n + \frac{1}{4}k_1 + \frac{1}{4}k_2, t_n + \frac{1}{2}\Delta t) \\
y_{n+1} &= y_n + \frac{1}{6}k_1 + \frac{1}{6}k_2 + \frac{2}{3}k_3 + \mathcal{O}\big((\Delta t)^4\big).
\end{align}

with corresponding Butcher table

\begin{array}{c|ccc}
    0 & \\
    1 & 1 & \\
    1/2 & 1/4 & 1/4 & \\ \hline
     & 1/6 & 1/6 & 2/3
\end{array}

```python
# Step 2.a.viii: Strong Stability Preserving Runge-Kutta (SSPRK3) Method

Butcher_dict['SSPRK3'] = (
[[0],
[sp.sympify(1), sp.sympify(1)],
[sp.Rational(1,2), sp.Rational(1,4), sp.Rational(1,4)],
["", sp.Rational(1,6), sp.Rational(1,6), sp.Rational(2,3)]]
, 3)
```

### Step 2.a.ix: Classic RK4 Method [Back to [top](#toc)\]
$$\label{rkfour}$$

The [classic RK4 method](https://en.wikipedia.org/w/index.php?title=Runge%E2%80%93Kutta_methods&oldid=894771467) obtains the solution $y(t)$ at time $t_{n+1}$ from $t_n$ via:

\begin{align}
k_1 &= \Delta tf(y_n, t_n), \\
k_2 &= \Delta tf(y_n + \frac{1}{2}k_1, t_n + \frac{\Delta t}{2}), \\
k_3 &= \Delta tf(y_n + \frac{1}{2}k_2, t_n + \frac{\Delta t}{2}), \\
k_4 &= \Delta tf(y_n + k_3, t_n + \Delta t), \\
y_{n+1} &= y_n + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4) + \mathcal{O}\big((\Delta t)^5\big).
\end{align}

with corresponding Butcher table

$$\begin{array}{c|cccc}
    0 & \\
    1/2 & 1/2 & \\
    1/2 & 0 & 1/2 & \\
    1 & 0 & 0 & 1 & \\ \hline
     & 1/6 & 1/3 & 1/3 & 1/6
\end{array} $$

```python
# Step 2.a.vix: Classic RK4 Method

Butcher_dict['RK4'] = (
[[sp.sympify(0)],
[sp.Rational(1,2), sp.Rational(1,2)],
[sp.Rational(1,2), sp.sympify(0), sp.Rational(1,2)],
[sp.sympify(1), sp.sympify(0), sp.sympify(0), sp.sympify(1)],
["", sp.Rational(1,6), sp.Rational(1,3), sp.Rational(1,3), sp.Rational(1,6)]]
, 4)
```

### Step 2.a.x:  RK5 Dormand-Prince Method [Back to [top](#toc)\]
$$\label{dp5}$$

The fifth-order Dormand-Prince (DP) method from the RK5(4) family (see [Dormand, J. R.; Prince, P. J. (1980)](https://www.sciencedirect.com/science/article/pii/0771050X80900133?via%3Dihub)) Butcher table is:

$$\begin{array}{c|ccccccc}
    0 & \\
    \frac{1}{5} & \frac{1}{5} & \\
    \frac{3}{10} & \frac{3}{40} & \frac{9}{40} & \\
    \frac{4}{5} & \frac{44}{45} & \frac{-56}{15} & \frac{32}{9} & \\
    \frac{8}{9} & \frac{19372}{6561} & \frac{−25360}{2187} & \frac{64448}{6561} & \frac{−212}{729} & \\
    1 & \frac{9017}{3168} & \frac{−355}{33} & \frac{46732}{5247} & \frac{49}{176} & \frac{−5103}{18656} & \\
    1 & \frac{35}{384} & 0 & \frac{500}{1113} & \frac{125}{192} & \frac{−2187}{6784} & \frac{11}{84} & \\ \hline
     & \frac{35}{384} & 0 & \frac{500}{1113} & \frac{125}{192} & \frac{−2187}{6784} & \frac{11}{84} & 0
\end{array} $$

```python
# Step 2.a.x:  RK5 Dormand-Prince Method

Butcher_dict['DP5'] = (
[[0],
[sp.Rational(1,5), sp.Rational(1,5)],
[sp.Rational(3,10),sp.Rational(3,40), sp.Rational(9,40)],
[sp.Rational(4,5), sp.Rational(44,45), sp.Rational(-56,15), sp.Rational(32,9)],
[sp.Rational(8,9), sp.Rational(19372,6561), sp.Rational(-25360,2187), sp.Rational(64448,6561), sp.Rational(-212,729)],
[sp.sympify(1), sp.Rational(9017,3168), sp.Rational(-355,33), sp.Rational(46732,5247), sp.Rational(49,176), sp.Rational(-5103,18656)],
[sp.sympify(1), sp.Rational(35,384), sp.sympify(0), sp.Rational(500,1113), sp.Rational(125,192), sp.Rational(-2187,6784), sp.Rational(11,84)],
["", sp.Rational(35,384), sp.sympify(0), sp.Rational(500,1113), sp.Rational(125,192), sp.Rational(-2187,6784), sp.Rational(11,84), sp.sympify(0)]]
, 5)
```

### Step 2.a.xi:  RK5 Dormand-Prince Method Alternative [Back to [top](#toc)\]
$$\label{dp5alt}$$

The fifth-order Dormand-Prince (DP) method from the RK6(5) family (see [Dormand, J. R.; Prince, P. J. (1981)](https://www.sciencedirect.com/science/article/pii/0771050X81900103)) Butcher table is:

$$\begin{array}{c|ccccccc}
    0 & \\
    \frac{1}{10} & \frac{1}{10} & \\
    \frac{2}{9} & \frac{-2}{81} & \frac{20}{81} & \\
    \frac{3}{7} & \frac{615}{1372} & \frac{-270}{343} & \frac{1053}{1372} & \\
    \frac{3}{5} & \frac{3243}{5500} & \frac{-54}{55} & \frac{50949}{71500} & \frac{4998}{17875} & \\
    \frac{4}{5} & \frac{-26492}{37125} & \frac{72}{55} & \frac{2808}{23375} & \frac{-24206}{37125} & \frac{338}{459} & \\
    1 & \frac{5561}{2376} & \frac{-35}{11} & \frac{-24117}{31603} & \frac{899983}{200772} & \frac{-5225}{1836} & \frac{3925}{4056} & \\ \hline
     & \frac{821}{10800} & 0 & \frac{19683}{71825} & \frac{175273}{912600} & \frac{395}{3672} & \frac{785}{2704} & \frac{3}{50}
\end{array}$$

```python
# Step 2.a.xi:  RK5 Dormand-Prince Method Alternative

Butcher_dict['DP5alt'] = (
[[0],
[sp.Rational(1,10), sp.Rational(1,10)],
[sp.Rational(2,9), sp.Rational(-2, 81), sp.Rational(20, 81)],
[sp.Rational(3,7), sp.Rational(615, 1372), sp.Rational(-270, 343), sp.Rational(1053, 1372)],
[sp.Rational(3,5), sp.Rational(3243, 5500), sp.Rational(-54, 55), sp.Rational(50949, 71500), sp.Rational(4998, 17875)],
[sp.Rational(4, 5), sp.Rational(-26492, 37125), sp.Rational(72, 55), sp.Rational(2808, 23375), sp.Rational(-24206, 37125), sp.Rational(338, 459)],
[sp.sympify(1), sp.Rational(5561, 2376), sp.Rational(-35, 11), sp.Rational(-24117, 31603), sp.Rational(899983, 200772), sp.Rational(-5225, 1836), sp.Rational(3925, 4056)],
["", sp.Rational(821, 10800), sp.sympify(0), sp.Rational(19683, 71825), sp.Rational(175273, 912600), sp.Rational(395, 3672), sp.Rational(785, 2704), sp.Rational(3, 50)]]
, 5)
```

### Step 2.a.xii:  RK5 Cash-Karp Method [Back to [top](#toc)\]
$$\label{ck5}$$

The fifth-order Cash-Karp Method (see [J. R. Cash, A. H. Karp. (1980)](https://dl.acm.org/citation.cfm?doid=79505.79507)) Butcher table is:

$$\begin{array}{c|cccccc}
    0 & \\
	\frac{1}{5} & \frac{1}{5} & \\
	\frac{3}{10} & \frac{3}{40} & \frac{9}{40} & \\
	\frac{3}{5} & \frac{3}{10} & \frac{−9}{10} & \frac{6}{5} & \\
	1 & \frac{−11}{54} & \frac{5}{2} & \frac{−70}{27} & \frac{35}{27} & \\
	\frac{7}{8} & \frac{1631}{55296} & \frac{175}{512} & \frac{575}{13824} & \frac{44275}{110592} & \frac{253}{4096} & \\ \hline
	 & \frac{37}{378} & 0 & \frac{250}{621} & \frac{125}{594} & 0 & \frac{512}{1771}
\end{array}$$

```python
# Step 2.a.xii:  RK5 Cash-Karp Method

Butcher_dict['CK5'] = (
[[0],
[sp.Rational(1,5), sp.Rational(1,5)],
[sp.Rational(3,10),sp.Rational(3,40), sp.Rational(9,40)],
[sp.Rational(3,5), sp.Rational(3,10), sp.Rational(-9,10), sp.Rational(6,5)],
[sp.sympify(1), sp.Rational(-11,54), sp.Rational(5,2), sp.Rational(-70,27), sp.Rational(35,27)],
[sp.Rational(7,8), sp.Rational(1631,55296), sp.Rational(175,512), sp.Rational(575,13824), sp.Rational(44275,110592), sp.Rational(253,4096)],
["",sp.Rational(37,378), sp.sympify(0), sp.Rational(250,621), sp.Rational(125,594), sp.sympify(0), sp.Rational(512,1771)]]
, 5)
```

### Step 2.a.xiii:  RK6 Dormand-Prince Method [Back to [top](#toc)\]
$$\label{dp6}$$

The sixth-order Dormand-Prince method (see [Dormand, J. R.; Prince, P. J. (1981)](https://www.sciencedirect.com/science/article/pii/0771050X81900103)) Butcher Table is

$$\begin{array}{c|cccccccc}
    0 & \\
    \frac{1}{10} & \frac{1}{10} & \\
    \frac{2}{9} & \frac{-2}{81} & \frac{20}{81} & \\
    \frac{3}{7} & \frac{615}{1372} & \frac{-270}{343} & \frac{1053}{1372} & \\
    \frac{3}{5} & \frac{3243}{5500} & \frac{-54}{55} & \frac{50949}{71500} & \frac{4998}{17875} & \\
    \frac{4}{5} & \frac{-26492}{37125} & \frac{72}{55} & \frac{2808}{23375} & \frac{-24206}{37125} & \frac{338}{459} & \\
    1 & \frac{5561}{2376} & \frac{-35}{11} & \frac{-24117}{31603} & \frac{899983}{200772} & \frac{-5225}{1836} & \frac{3925}{4056} & \\
    1 & \frac{465467}{266112} & \frac{-2945}{1232} & \frac{-5610201}{14158144} & \frac{10513573}{3212352} & \frac{-424325}{205632} & \frac{376225}{454272} &  0 & \\ \hline
     & \frac{61}{864} & 0 & \frac{98415}{321776} & \frac{16807}{146016} & \frac{1375}{7344} & \frac{1375}{5408} & \frac{-37}{1120} & \frac{1}{10}
\end{array}$$

```python
# Step 2.a.xiii:  RK6 Dormand-Prince Method

Butcher_dict['DP6'] = (
[[0],
[sp.Rational(1,10), sp.Rational(1,10)],
[sp.Rational(2,9), sp.Rational(-2, 81), sp.Rational(20, 81)],
[sp.Rational(3,7), sp.Rational(615, 1372), sp.Rational(-270, 343), sp.Rational(1053, 1372)],
[sp.Rational(3,5), sp.Rational(3243, 5500), sp.Rational(-54, 55), sp.Rational(50949, 71500), sp.Rational(4998, 17875)],
[sp.Rational(4, 5), sp.Rational(-26492, 37125), sp.Rational(72, 55), sp.Rational(2808, 23375), sp.Rational(-24206, 37125), sp.Rational(338, 459)],
[sp.sympify(1), sp.Rational(5561, 2376), sp.Rational(-35, 11), sp.Rational(-24117, 31603), sp.Rational(899983, 200772), sp.Rational(-5225, 1836), sp.Rational(3925, 4056)],
[sp.sympify(1), sp.Rational(465467, 266112), sp.Rational(-2945, 1232), sp.Rational(-5610201, 14158144), sp.Rational(10513573, 3212352), sp.Rational(-424325, 205632), sp.Rational(376225, 454272), sp.sympify(0)],
["", sp.Rational(61, 864), sp.sympify(0), sp.Rational(98415, 321776), sp.Rational(16807, 146016), sp.Rational(1375, 7344), sp.Rational(1375, 5408), sp.Rational(-37, 1120), sp.Rational(1,10)]]
, 6)
```

### Step 2.a.xiv:  RK6 Luther's Method [Back to [top](#toc)\]
$$\label{l6}$$

Luther's sixth-order method (see [H. A. Luther (1968)](http://www.ams.org/journals/mcom/1968-22-102/S0025-5718-68-99876-1/S0025-5718-68-99876-1.pdf)) Butcher table is:
$$\begin{array}{c|ccccccc}
    0 & \\
    1 & 1 & \\
    \frac{1}{2} & \frac{3}{8} & \frac{1}{8} & \\
    \frac{2}{3} & \frac{8}{27} & \frac{2}{27} & \frac{8}{27} & \\
    \frac{(7-q)}{14} & \frac{(-21 + 9q)}{392} & \frac{(-56 + 8q)}{392} & \frac{(336 - 48q)}{392} & \frac{(-63 + 3q)}{392} & \\
    \frac{(7+q)}{14} & \frac{(-1155 - 255q)}{1960} & \frac{(-280 - 40q)}{1960} & \frac{320q}{1960} & \frac{(63 + 363q)}{1960} & \frac{(2352 + 392q)}{1960}     & \\
    1 & \frac{(330 + 105q)}{180} & \frac{2}{3} & \frac{(-200 + 280q)}{180} & \frac{(126 - 189q)}{180} & \frac{(-686 - 126q)}{180} & \frac{(490 - 70q)}{180} & \\ \hline
    & \frac{1}{20} & 0 & \frac{16}{45} & 0 & \frac{49}{180} & \frac{49}{180} & \frac{1}{20}
\end{array}$$

where $q = \sqrt{21}$.

```python
#  Step 2.a.xiv:  RK6 Luther's Method

q = sp.sqrt(21)
Butcher_dict['L6'] = (
[[0],
[sp.sympify(1), sp.sympify(1)],
[sp.Rational(1,2), sp.Rational(3,8), sp.Rational(1,8)],
[sp.Rational(2,3), sp.Rational(8,27), sp.Rational(2,27), sp.Rational(8,27)],
[(7 - q)/14, (-21 + 9*q)/392, (-56 + 8*q)/392, (336 -48*q)/392, (-63 + 3*q)/392],
[(7 + q)/14, (-1155 - 255*q)/1960, (-280 -  40*q)/1960, (-320*q)/1960, (63 + 363*q)/1960, (2352 + 392*q)/1960],
[sp.sympify(1), ( 330 + 105*q)/180, sp.Rational(2,3), (-200 + 280*q)/180, (126 - 189*q)/180, (-686 - 126*q)/180, (490 -  70*q)/180],
["", sp.Rational(1, 20), sp.sympify(0), sp.Rational(16, 45), sp.sympify(0), sp.Rational(49, 180), sp.Rational(49, 180), sp.Rational(1, 20)]]
, 6)
```

### Step 2.a.xv:  RK8 Dormand-Prince Method [Back to [top](#toc)\]
$$\label{dp8}$$

The eighth-order Dormand-Prince Method (see [Dormand, J. R.; Prince, P. J. (1981)](https://www.sciencedirect.com/science/article/pii/0771050X81900103)) Butcher table is:

$$\begin{array}{c|ccccccccc}
    0 & \\
    \frac{1}{18} & \frac{1}{18} & \\
    \frac{1}{12} & \frac{1}{48} & \frac{1}{16} & \\
    \frac{1}{8} & \frac{1}{32} & 0 & \frac{3}{32} & \\
    \frac{5}{16} & \frac{5}{16} & 0 & \frac{-75}{64} & \frac{75}{64} & \\
    \frac{3}{8} & \frac{3}{80} & 0 & 0 & \frac{3}{16} & \frac{3}{20} & \\
    \frac{59}{400} & \frac{29443841}{614563906} & 0 & 0 & \frac{77736538}{692538347} & \frac{-28693883}{1125000000} & \frac{23124283}{1800000000} & \\
    \frac{93}{200} & \frac{16016141}{946692911} & 0 & 0 & \frac{61564180}{158732637} & \frac{22789713}{633445777} & \frac{545815736}{2771057229} & \frac{-180193667}{1043307555} & \\
    \frac{5490023248}{9719169821} & \frac{39632708}{573591083} & 0 & 0 & \frac{-433636366}{683701615} & \frac{-421739975}{2616292301} & \frac{100302831}{723423059} & \frac{790204164}{839813087} & \frac{800635310}{3783071287} & \\
    \frac{13}{20} & \frac{246121993}{1340847787} & 0 & 0 & \frac{-37695042795}{15268766246} & \frac{-309121744}{1061227803} & \frac{-12992083}{490766935} & \frac{6005943493}{2108947869} & \frac{393006217}{1396673457} & \frac{123872331}{1001029789} & \\
    \frac{1201146811}{1299019798} & \frac{-1028468189}{846180014} & 0 & 0 & \frac{8478235783}{508512852} & \frac{1311729495}{1432422823} & \frac{-10304129995}{1701304382} & \frac{-48777925059}{3047939560} & \frac{15336726248}{1032824649} & \frac{-45442868181}{3398467696} & \frac{3065993473}{597172653} & \\
    1 & \frac{185892177}{718116043} & 0 & 0 & \frac{-3185094517}{667107341} & \frac{-477755414}{1098053517} & \frac{-703635378}{230739211} & \frac{5731566787}{1027545527} & \frac{5232866602}{850066563} & \frac{-4093664535}{808688257} & \frac{3962137247}{1805957418} & \frac{65686358}{487910083} & \\
    1 & \frac{403863854}{491063109} & 0 & 0 & \frac{-5068492393}{434740067} & \frac{-411421997}{543043805} & \frac{652783627}{914296604} & \frac{11173962825}{925320556} & \frac{-13158990841}{6184727034} & \frac{3936647629}{1978049680} & \frac{-160528059}{685178525} & \frac{248638103}{1413531060} & 0 & \\
    & \frac{14005451}{335480064} & 0 & 0 & 0 & 0 & \frac{-59238493}{1068277825} & \frac{181606767}{758867731} & \frac{561292985}{797845732} & \frac{-1041891430}{1371343529} & \frac{760417239}{1151165299} & \frac{118820643}{751138087} & \frac{-528747749}{2220607170} & \frac{1}{4}
\end{array}$$

```python
# Step 2.a.xv:  RK8 Dormand-Prince Method

Butcher_dict['DP8']=(
[[0],
[sp.Rational(1, 18), sp.Rational(1, 18)],
[sp.Rational(1, 12), sp.Rational(1, 48), sp.Rational(1, 16)],
[sp.Rational(1, 8), sp.Rational(1, 32), sp.sympify(0), sp.Rational(3, 32)],
[sp.Rational(5, 16), sp.Rational(5, 16), sp.sympify(0), sp.Rational(-75, 64), sp.Rational(75, 64)],
[sp.Rational(3, 8), sp.Rational(3, 80), sp.sympify(0), sp.sympify(0), sp.Rational(3, 16), sp.Rational(3, 20)],
[sp.Rational(59, 400), sp.Rational(29443841, 614563906), sp.sympify(0), sp.sympify(0), sp.Rational(77736538, 692538347), sp.Rational(-28693883, 1125000000), sp.Rational(23124283, 1800000000)],
[sp.Rational(93, 200), sp.Rational(16016141, 946692911), sp.sympify(0), sp.sympify(0), sp.Rational(61564180, 158732637), sp.Rational(22789713, 633445777), sp.Rational(545815736, 2771057229), sp.Rational(-180193667, 1043307555)],
[sp.Rational(5490023248, 9719169821), sp.Rational(39632708, 573591083), sp.sympify(0), sp.sympify(0), sp.Rational(-433636366, 683701615), sp.Rational(-421739975, 2616292301), sp.Rational(100302831, 723423059), sp.Rational(790204164, 839813087), sp.Rational(800635310, 3783071287)],
[sp.Rational(13, 20), sp.Rational(246121993, 1340847787), sp.sympify(0), sp.sympify(0), sp.Rational(-37695042795, 15268766246), sp.Rational(-309121744, 1061227803), sp.Rational(-12992083, 490766935), sp.Rational(6005943493, 2108947869), sp.Rational(393006217, 1396673457), sp.Rational(123872331, 1001029789)],
[sp.Rational(1201146811, 1299019798), sp.Rational(-1028468189, 846180014), sp.sympify(0), sp.sympify(0), sp.Rational(8478235783, 508512852), sp.Rational(1311729495, 1432422823), sp.Rational(-10304129995, 1701304382), sp.Rational(-48777925059, 3047939560), sp.Rational(15336726248, 1032824649), sp.Rational(-45442868181, 3398467696), sp.Rational(3065993473, 597172653)],
[sp.sympify(1), sp.Rational(185892177, 718116043), sp.sympify(0), sp.sympify(0), sp.Rational(-3185094517, 667107341), sp.Rational(-477755414, 1098053517), sp.Rational(-703635378, 230739211), sp.Rational(5731566787, 1027545527), sp.Rational(5232866602, 850066563), sp.Rational(-4093664535, 808688257), sp.Rational(3962137247, 1805957418), sp.Rational(65686358, 487910083)],
[sp.sympify(1), sp.Rational(403863854, 491063109), sp.sympify(0), sp.sympify(0), sp.Rational(-5068492393, 434740067), sp.Rational(-411421997, 543043805), sp.Rational(652783627, 914296604), sp.Rational(11173962825, 925320556), sp.Rational(-13158990841, 6184727034), sp.Rational(3936647629, 1978049680), sp.Rational(-160528059, 685178525), sp.Rational(248638103, 1413531060), sp.sympify(0)],
["", sp.Rational(14005451, 335480064), sp.sympify(0), sp.sympify(0), sp.sympify(0), sp.sympify(0), sp.Rational(-59238493, 1068277825), sp.Rational(181606767, 758867731), sp.Rational(561292985, 797845732), sp.Rational(-1041891430, 1371343529), sp.Rational(760417239, 1151165299), sp.Rational(118820643, 751138087), sp.Rational(-528747749, 2220607170), sp.Rational(1, 4)]]
, 8)
```

# Step 3: Code validation against `MoLtimestepping.RK_Butcher_Table_Dictionary` NRPy+ module [Back to [top](#toc)\]
$$\label{code_validation}$$

As a code validation check, we verify agreement in the dictionary of Butcher tables between
1. this tutorial and
2. the NRPy+ [MoLtimestepping.RK_Butcher_Table_Dictionary](../edit/MoLtimestepping/RK_Butcher_Table_Dictionary.py) module.

We analyze all key/value entries in the dictionary for consistency.

```python
# Step 3: Code validation against MoLtimestepping.RK_Butcher_Table_Dictionary NRPy+ module
import sys
from MoLtimestepping.RK_Butcher_Table_Dictionary import Butcher_dict as B_dict
valid = True
for key, value in Butcher_dict.items():
    if Butcher_dict[key] != B_dict[key]:
        valid = False
        print(key)
if valid == True and len(Butcher_dict.items()) == len(B_dict.items()):
    print("The dictionaries match!")
else:
    print("ERROR: Dictionaries don't match!")
    sys.exit(1)
```

    The dictionaries match!

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-RK_Butcher_Table_Dictionary.pdf](Tutorial-RK_Butcher_Table_Dictionary.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-RK_Butcher_Table_Dictionary.ipynb
!pdflatex -interaction=batchmode Tutorial-RK_Butcher_Table_Dictionary.tex
!pdflatex -interaction=batchmode Tutorial-RK_Butcher_Table_Dictionary.tex
!pdflatex -interaction=batchmode Tutorial-RK_Butcher_Table_Dictionary.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode