# Toy Examples of Format Features

This notebook demonstrates minimal examples of the following format features:

* [Time-dependent right hand sides](.time_dependent_rhs.yml)
* [Step functions in the right hand side](step_function.yml)
* [Function definition](function_in_rhs.yml)

## Time-dependent right hand side:

`time_dependent_rhs.yml` implements the ODE
\begin{align}
\dot{x} &= -\frac{x}{t+1} \\
x(0) &= 1
\end{align}
via
```yaml
time:
    variable: t

odes:
    - stateId: x
      rightHandSide: -x/(t+1)
      initialValue: 1
```
The time variable `t` needs to be defined.

```python
import yaml2sbml

yaml_file_basic = 'time_dependent_rhs.yml'
sbml_output_file = 'time_dependent_rhs.xml'

yaml2sbml.yaml2sbml(yaml_file_basic, sbml_output_file)
```

For the sake of brevity, simulation and plotting is done in a separate function. Details on these steps can be found on [other notebooks](../Lotka_Volterra_python/Lotka_Volterra.ipynb).

```python
%matplotlib inline

from simulation_and_plotting import simulate_AMICI, plot_AMICI
import amici.plotting

#simulate
amici_model, rdata = simulate_AMICI(sbml_output_file)
#plot
plot_AMICI(amici_model, rdata, 'Time-dependent right hand side')
```

## Step functions in the right hand side

`step_function.yml` implements the ODE

\begin{align*}
\dot{x} = \begin{cases}
      x , & t < 1 \\
      -x, & \text{otherwise}
    \end{cases}
\end{align*}
via
```yaml
time:
    variable: t

odes:
    - stateId: x
      rightHandSide: piecewise(x, t < 1, -x)
      initialValue: 1
```
More details on `piecewise` functions can be found in the [libsbml documentation](http://sbml.org/Special/Software/libSBML/docs/formatted/python-api/libsbml-math.html).

```python
import yaml2sbml

yaml_file_basic = 'step_function.yml'
sbml_output_file = 'step_function.xml'

# translate yaml file to sbml
yaml2sbml.yaml2sbml(yaml_file_basic, sbml_output_file)
```

As discussed above, we plot via:

```python
%matplotlib inline

#simulate
amici_model, rdata = simulate_AMICI(sbml_output_file)

#plot
plot_AMICI(amici_model, rdata, 'Step function')
```

## Function definition

`yaml2sbml` allows to define functions, that can then be called in other parts of the model. `functions_in_rhs.yaml` implements the ODE

\begin{align*}
\dot{x}_1 &= f(x_2, 1, 2) \\
\dot{x}_2 &= f(x_1, -1, 0),
\end{align*}

where $f(x, a, b) = a \cdot x + b$
via
```yaml
odes:
    - stateId: x_1
      rightHandSide: f(x_2, 1, 2)
      initialValue: 1

    - stateId: x_2
      rightHandSide: f(x_1, -1, 0)
      initialValue: 1

functions:
    - functionId: f
      arguments: x, a, b
      formula: a * x + b
```

```python
import yaml2sbml

yaml_file_basic = 'functions_in_rhs.yml'
sbml_output_file = 'functions_in_rhs.xml'

# translate yaml file to sbml
yaml2sbml.yaml2sbml(yaml_file_basic, sbml_output_file)
```

We now follow the usual work flow for simulation and plotting.

```python
%matplotlib inline

#simulate
amici_model, rdata = simulate_AMICI(sbml_output_file)

#plot
plot_AMICI(amici_model, rdata, 'Step function')
```


```python
import sympy as sp # charger la libraire sympy pour faire du calcul formel.
sp.init_printing()
```

# Calculer la fonction dérivée d'une fonction à une variable

Pour calculer une dérivée première, on utilise la fonction $diff()$ de la librairie Sympy.

Par exemple, pour calculer la dérivée de $\quad\sin(x)$ :

     sp.diff(sp.sin(x),x)

qui nous retourne bien : $\quad \quad \cos(x)$.

Il est n'est pas indispensable ici de spécifier que l'on dérive selon la variable $x$, mais il est préfréable de la faire pour respecter l'ordre des arguments, ce qui sera utile ci-dessous. Un exemple un peu plus compliqué :

     sp.diff(sp.sin(x)*sp.exp(x)/x)

qui donne $\quad \quad \frac{e^x\sin(x)}{x} + e^x\cos(x) −  \frac{e^x\sin(x)}{x^2}$.

Source du doc disponible sur :
[Cliquez ici](http://www.tangentex.com/CalculSymbolique.htm#Par6 "Source")

## Déninir le symbole, variable de votre fonction.

```python
x = sp.Symbol('x')
```

```python
sp.diff(sp.sin(x),x)
```

```python
sp.diff(sp.sin(x)*sp.exp(x)/x)
```

# Application 1 de notre cours : verifions ensemble ?

Attention : la fonction _diff()_ de sympy montre uniquement le résultat final, elle permet seulement de vérifier ses calculs !!!

```python
# la fonction f :
f = -7*x + 2
f
```

```python
sp.diff(f)
```

```python
# la onction g :
g = x**2 + sp.sqrt(x)
g
```

```python
sp.diff(g)
```

```python
# la fonction h :
h = -5/x
h
```

```python
sp.diff(h)
```

```python
# la fonction i :
i = x**3/6 -8*x + 1
i
```

```python
sp.diff(i)
```

```python
# la fonction j :
j = x**2 + 2/(3*x**2)
j
```

```python
sp.diff(j)
```

# Application 2 de notre cours : vérifions encore ?

l'exemple du produit vérifié ici :

$f(x) = 3x\sqrt{x}$

```python
sp.diff(3*x*sp.sqrt(x))
```

Application 2 est vérifié ici :

```python
sp.diff(x**2*sp.sqrt(x))
```

```python
sp.diff(5/(x**2 - x))
```

```python
sp.diff((7*x + 1)**3)
```

```python
sp.diff(sp.sqrt(2*x - 2))
```

## Travailler en autonomie sur plateforme `EULER-WIMS` de Versailles

```python
import IPython.display as ipd
```

```python
ipd.IFrame(src="http://acver.fr/gi8",width = 900,height = 400)
```


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

## Numerical Differentiation

The continuous derivative of an analytic function $f(x)$ is defined as the limit

$$f'(x)=\lim_{h\rightarrow 0}\frac{f(x+h)-f(x)}{h} \; , \tag{1}$$

where $h$ is in infinitesimal small quantity.

### Discretization

In the discussion of numeric precision (see [this notebook](https://github.com/cohmathonc/biosci670/blob/master/IntroductionComputationalMethods/02_IntroPythonForScientificComputing.ipynb)), we saw that *any*  digital number representation is discrete.
This implies that $h$ is finite and we cannot compute the limit $h\rightarrow 0$.

Often we have to resort to numerical differentiation in cases when the functional expression is unknown, which means that $f(x)$ cannot be evaluated at any arbitrary point $x$ or $x+h$.

The goal of numerical differentiation is to approximate the n-th derivative, given a sequence of values $f(x_0), f(x_1), \ldots f(x_N)$ of an unkown function $f(x)$ at known evaluation points $x_0, x_1, \ldots, x_N$.
Often the points at which values of a function are known are equally spaced. In that case, the spacing $h$ corresponds to the spacing between subsequent points, i.e. $h=x_{i+1} - x_i = \Delta x$.

```python
## domain
x_min = 0
x_max = 4*np.pi

# we can either

# (1) fix the number of points, or
# n_points = 100
# x = np.linspace(x_min, x_max, n_points)
# h = (x_max-x_min)/(n_points-1)

# (2) fix the grid spacing
delta_x = 0.1
x = np.arange(x_min, x_max, delta_x)
n_points = len(x)

## function values
y = np.cos(x)

plt.plot(x, y, label='cos(x)')
plt.legend()
```

**Discretization**

### Forward Difference

Identifying $h$ in Eq. (1) with $\Delta x$, we can approximate the derivative of the function at a specific point $x_i$ by **finite differences**:

$$f'(x_i)\approx \frac{f(x_i+\Delta x)-f(x_i)}{\Delta x} = \frac{f(x_{i+1})-f(x_i)}{\Delta x}\;. \tag{2}$$

The formula in Eq. (2) is called the **forward difference formula** because it estimates the function's derivative at $x_i$ using information about the function's value at the point $x_i$ and the 'next' point $x_{i+1}$.

---

**Exercise:**

- Approximate the derivative of $\cos(x)$ using the forward difference formula (2).
- Plot $\cos(x)$, $\cos'(x)=-\sin(x)$ and the numerical approximation.
- Examine the behavior of the numerical approximation for varying size of the grid spacing $\Delta x$.

---

```python
def derivative_forward(y, delta_x):
    """
    Computes forward difference on elements in 'y' array, assuming constant grid spacing 'delta_x'.

    Args:
    - y: an array of data points / function values
    - x: scalar, indicating the spacing between subsequent observation/evaluation points
    """
    n_eval_points = y.shape[0]
    derivative = np.ones(n_eval_points) * np.nan
    for i in range(n_eval_points-1):
        derivative[i] = (y[i+1]-y[i])/delta_x
    return derivative
```

This implementation
- creates a new array $y'$ (`derivative`)
- scans the input array `y`, and for each index $i$ of `y` computes the finite difference by accessing the $i$-th and $(i+1)$-th element of `y`.
- assigns the finite difference to the $i$-th element of vector $y'$ (`derivative`)

Note that the last element ($i=N$) of the array $y'$ (`derivative`) remains empty because the operation `y[i+1]-y[i]` can only be performed for $i\leq N-1$.

We have multiple options to handle this in our implementation. We could:
1. Create an array `derivative` that has one element less than the array `y`.
2. Create an array `derivative` of same size as array `y` and explicitly indicate that no specific value is assigned to the last element.

Above implementation uses approach (2) by initializing the array `derivative` with elements of the `np.nan` data type ([Not a Number](https://en.wikipedia.org/wiki/NaN)) instead of a specific value, such as ones or zeros.

```python
## THIS IS EQUIVALENT TO ABOVE BUT AVOIDS FOR LOOP BY VECTORIZATION

def derivative_forward_array(y, delta_x):
    """
    Computes forward difference on elements in 'y' array, assuming constant grid spacing 'delta_x'.

    Args:
    - y: an array of data points / function values
    - x: scalar, indicating the spacing between subsequent observation/evaluation points
    """
    # instead of iterating over each pair i, i+1, we create a new shifted array
    y_forward = np.roll(y, -1)     # i-th item in this array corresponds to value at i+1 in y array
    derivative = (y_forward - y)/delta_x
    derivative[-1] = np.nan
    return derivative
```

```python
## Domain
x_min = 0
x_max = 4*np.pi

## Evaluation Grid
delta_x = 0.5
x = np.arange(x_min, x_max, delta_x)

## function values
y = np.cos(x)
y_p = -np.sin(x)                                  # analytic derivative
y_p_approx = derivative_forward(y, delta_x)       # numeric approximation of derivative

# plot f, f', f'_approximated
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)
ax.plot(x, y,
        label='f(x) = cos(x)' )
ax.plot(x, y_p,
        label="exact f'(x) = -sin(x)")
ax.plot(x, y_p_approx,
        label="approx f'(x)", color='red', linestyle=':')

# change x axis ticks to multiples of pi
x_ticks_values = np.arange(0, 4*np.pi, np.pi/2 )
x_ticks_labels = np.arange(0, 4, 0.5)
ax.set_xticks(x_ticks_values)
ax.set_xticklabels(x_ticks_labels)
ax.set_xlabel("x in multiples of $\pi$")

ax.legend()
plt.show()

```

### Sources of Errors

Above example shows that the numerical values of the derivative computed by approximation (2) deviate from the analytic derivative with increasing $\Delta x$.

To estimate the error made by this approximation we focus on a specific point $x_0$ and expand the function $f(x)$ as a [*Taylor Series*](https://en.wikipedia.org/wiki/Taylor_series) around that point.
A Taylor series is a representation of a function at a single point $x_0$ as an infinite sum of terms that are calculated from the values of the function's derivatives at this point:

$$f(x)\big|_{x=x_0} = \sum_{n=0}^\infty\frac{f^{(n)}(x_0)}{n\,!}(x-x_0)^n \; . \tag{3}$$

Only taking into account the 0th ($n=0$), 1st ($n=1$) and 2nd ($n=2$) derivatives, we obtain the following approximation:
$$f(x)\big|_{x=x_0} \approx f(x_0) +(x-x_0)\, f'(x_0) + \frac{(x-x_0)^2}{2}f''(x_0) \;. \tag{4}$$

Where we neglected terms of third order and higher in $(x-x_0)$.

How does this help us in estimating the approximation error in the *forward difference formula* (2) introduced above?

Substituting $h=x-x_0$ in the truncated Taylor expansion (4) gives
$$f(x_0+h) \approx  f(x_0) + h\,f'(x_0) + \frac{h^2}{2}f''(x_0) ;.$$

We can now bring $f'(x_0)$ to the left-hand side and recover the *forward difference formula* from before (2):
    $$f'(x_0)\approx \frac{f(x_0+h)-f(x_0)}{h} -  \frac{h}{2}f''(x_0)\;. $$

Thus, the error made by this approximation is:
$$\underbrace{\frac{f(x_0+h)-f(x_0)}{h}}_{\text{forward difference approximation} } - f'(x_0)\approx   \underbrace{\frac{h}{2}f''(x_0)}_{\text{truncation error} } \;. \tag{5}$$

This error is called **truncation error** because the *forward difference formula* is obtained by truncating this term from the *actual* derivative, see Eq. (3).
For the *forward difference formula*, the truncation error  is proportional to $h=h^1$ ($\mathcal{O}(h)$)
The *forward difference formula* (2) is therefore called a **first order method**.

*Note 1*:
In step from equation (3) to (4), we have truncated multiple terms of higher order in $h$ (all terms of order $n\geq3$). Since we assume that $h$ is very small ($h<<1$), it follows that $h^n>h^{n+1}$. The most important error term is therefore the one of lowest order in $h$ and we can neglect all other terms.

*Note 2*:
The truncation error is proportional to the second derivative (and higher derivatives) of the function whose first derivative we aim to approximate.
This means that the forward difference formula is the *exact* derivative for linear functions.

---
**Exercise:**

Let's estimate the  error made by the  *forward difference* approximation (2) and compare it with the theoretical truncation error.

For this example we consider the function $f(x)=\cos(x)$ with first derivative $f'(x)=-\sin(x)$ and second derivative $f''(x)=-\cos(x)$.

We approximate its derivative $f'_{\mathrm{num}}(x)$ at the point $x_0=\pi/6$ and compute the following errors:

\begin{align}
\text{approximation error:}  \qquad\epsilon_{\mathrm{num}} &= \bigl| f'(x_0) - f'_{\mathrm{num}}(x_0) \bigr| \\
\text{truncation error:}  \qquad\epsilon_{\mathrm{trun}} &= -\frac{h}{2}\, f''(x_0) = \frac{h}{2}\cos(x_0)
\end{align}

We repeat this computation for multiple values of $h$, starting from $h=0.1$ and halving its value in each subsequent evaluation, so that $h_i =\frac{0.1}{2^i}$ for $i\in [0, 1 ...,40]$.

---

```python
## Functions and evaluation points
# f=cos(x_0), f'=sin(x_0); x_0 = pi/6
x_0 = np.pi/6
y   = np.cos(x_0)
y_p = -np.sin(x_0)

## Initalization
n_steps = 40
inital_h = 0.1

h_values = np.zeros(n_steps)                  # for h values
y_p_approx_num = np.zeros(n_steps)            # for numeric approximation of f'
eps_num = np.zeros(n_steps)                   # for approximation error
eps_trun = np.zeros(n_steps)                  # for truncation error

## Computations in loop
for i in range(n_steps):
    # compute current h
    h_values[i] = inital_h/(2**i)
    # compute discretization (we only consider x_0 and its neighbouring points)
    x = np.array([x_0-h_values[i], x_0, x_0+h_values[i]])           # x_0 at index 1
    y = np.cos(x)
    # compute numerical approximation of cos(x_0)' with current h
    y_p_approx_num_x0 = derivative_forward(y, h_values[i])[1]       # get approximated derivative at x_0
    y_p_approx_num[i] = y_p_approx_num_x0
    # compute error between analytic and numeric derivative
    eps_num[i] = np.abs(y_p - y_p_approx_num_x0)
    # compute the theoretical (truncation) error from taylor expansion
    eps_trun[i] = np.cos(x_0)*h_values[i]/2

## Print results
print("eps_num: ",eps_num)
print("eps_trun: ",eps_trun)

```

Now, we compare those two errors graphically by plotting the errors $\epsilon_{\mathrm{num}}$, $\epsilon_{\mathrm{trun}}$ against $h$. Since both quantities, errors and $h$, vary by multiple orders of magnitude, we use logarithmic axes.

```python
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)

ax.loglog(h_values, eps_num, marker='o',
          label='error of numerical approximation: $\epsilon_{\mathrm{num}}$' )
ax.loglog(h_values, eps_trun, marker='x',
          label='theoretical truncation error: $\epsilon_{\mathrm{trun}}$' )

ax.set_xlabel('$h$')
ax.set_ylabel("absolute error")
ax.set_title("Errors in forward-difference approximation of 1st " \
             "derivative of $\cos(\pi/6)$")
ax.legend()
plt.show()
```

For large values $h$, the error of our numerical approximation $\epsilon_{\mathrm{num}}$ agrees with the truncation error $\epsilon_{\mathrm{trun}}$ that is theoretically expected for this *first order* differentiation method.
However, with decreasing $h$, the numerical error *increases* and begins to deviate from the expected truncation error.

This increase in $\epsilon_{\mathrm{num}}$ is not explained by truncation but is caused by *rounding* due to the finite precision of standard numeric data types. This error is called **rounding error**.
**Rounding** and **truncation** are the *two primary sources of errors* when using numerical methods.

Such errors can have serious consequences. A few extreme cases where bad numerical computing practices lead to disasters are collected [here](http://www-users.math.umn.edu/~arnold/disasters/) and [here](https://www5.in.tum.de/persons/huckle/bugse.html).

### Convergence

We are concerned with numerical methods that approximate the solution $u$ of a continuous problem by computing the solution $\tilde{u}$ of a discretized problem.
In the previous section we have shown that the accuracy of this approximation depends on the granularity of the discretization, i.e. the spacing $h$. Therefore, we denote the numerical approximation by $\tilde{u}_h$.

An important characteristic of a numerical method is its convergence behavior, that is *whether* and *how quickly* the numerical estimate $\tilde{u}_h$ converges to the exact value $\tilde{u}$ with decreasing $h$.

In Eq. (5), we showed that the truncation error of the *forward difference method*, Eq  (2), is expected to be proportional to $h$ and therefore the method is said to be *of order 1*.
Decreasing $h$ by a factor of e.g. $2$ will decrease the error in the estimation $\tilde{u}_{h/2}$ by the same factor $2$.

In general, when a numerical method is said to be *of order $p$*,  this means that for sufficiently small $h$:
$$\left| \tilde{u}_h-u\right|\leq C\, h^p \tag{5}\; ,$$
where $C$ is a number independent of $h$ that typically depends on the exact solution.
The speed at which the approximation by this numerical method approaches the true solution is called the *convergence rate* of the method and is $h^p$.

If the order $p$ of a given numerical method is known, estimating $p$ from a sequence of approximations is a good way to check the implementation of this method.

If the exact value $u$ is known, we can compute
$$\log{\left| \tilde{u}_h-u\right|} \leq \log{\left|C\right|} + p\log{h}$$
for multiple different values $h$ and fit a linear function of $\log{h}$ to $\log{\left| \tilde{u}_h-u\right|}$ to approximate $p$.

The standard way to obtain precise estimates for $p$ is to halve the spacing $h$ and compare the ratios of the errors resulting from subsequent spacings:

$$ \frac{\left|\tilde{u}_{h}-u\right|}{\left|\tilde{u}_{h/2}-u\right|} \leq \frac{C\, h^p }{C\, (h/2)^p }=2^p\tag{6}$$
and thus
$$\log_2{\left|\frac{\tilde{u}_{h}-u}{\tilde{u}_{h/2}-u}\right| \leq p } \tag{7}\; .$$

Alternatively, if the the exact value $u$ is not known, comparing the ratios of differences between $\tilde{u}_{h}$ for different $h$ yields a similar result:

$$ \frac{\left|\tilde{u}_{h}-\tilde{u}_{h/2}\right|}{\left|\tilde{u}_{h/2}-\tilde{u}_{h/4}\right|} \leq \frac{1-2^{-p}}{2^{-p}-2^{-2p}}=2^p\tag{8}$$

---
**Exercise (2):**

In the previous exercise we chose $h_{i+1} = \frac{1}{2}h_i$, so that we can now apply formula (8) to verify whether our implementation of the *forward difference method* shows the expected convergence behavior.

---

```python
# compute ratio of errors between steps:
# ratio(n) = error(n-1)/error(n)
#          = error(h_n)/error(h_n/2)

eps_ratio = np.ones_like(eps_num)*np.nan

for i in range(1,eps_num.shape[0]):         # start at 1 to access index i-1
    eps_ratio[i] = eps_num[i-1]/eps_num[i]

convergence_order = np.log2(eps_ratio)

print("ratio of subsequent error: ", eps_ratio)
print("convergence order: ",convergence_order)
```

### Other Finite Difference Schemes

We will now apply the analysis methods developed above to various finite difference schemas.
To facilitate this analysis we define functions for recurring tasks.

```python

def compute_approx_different_h(function_num_deriv, function,
                               inital_h=0.1, x_0=0, n_steps=40):
    """
    Computes 'n_steps' approximations of derivative of 'function' around 'x_0'
    using 'function_num_deriv', halving spacing delta x in each iteration.

    Args:
    - function_num_deriv: function object that computes numerical derivative from
                          - array of function evaluations / data points and
                          - spacing of evaluation points delta_x
    - function: function object for function whose derivative is to be estimated
    - initial_h: float, initial spacing
    - x_0: float, point where derivative of 'function' is to be estimated
    - n_steps: integer number of different spacings steps
    - n: degree of derivative; only defined for first, second and third derivative.

    Returns:
    - array of spacing values, array of estimated derivatives at x_0 for each choosen spacing
    """
    # initialize arrays
    h_values = np.zeros(n_steps)                  # for h values
    y_p_approx_num = np.zeros(n_steps)            # for numeric approximation of f'
    # compute approximations for different h
    for i in range(n_steps):
        # compute current h
        h_values[i] = inital_h/(2**i)
        # compute discretization (we only consider x_0 and its neighbouring points)
        x = np.array([x_0-h_values[i], x_0, x_0+h_values[i]])      # x_0 at index 1
        y = function(x)
        # compute numerical approximation of cos(x_0)' with current h
        y_p_approx_num[i] = function_num_deriv(y, h_values[i])[1]  # get approximated derivative at x_0
    return h_values, y_p_approx_num

def compute_convergence_order(eps):
    """
    Estimates convergence order from ratio of errors between
    finite difference estimations with different spacings.
    Assumes that step size between subsequent estimations has been halved.
    """
    eps_shifted = np.roll(eps, shift=1)
    eps_shifted[0] = np.nan
    eps_ratio = eps_shifted / eps
    convergence_order = np.log2(eps_ratio)
    return convergence_order

def plot_errors(h_vals, eps_num, eps_trun=None, title=None):
    """
    Plots errors vs stepsize on LogLog axes.

    Args:
    - eps_num: array of errors of the numeric approximation
    - eps_trun: array of theoretic truncation errors
    - h_vals: array of h values

    All arrays are expeceted to be of equal length
    """
    # plot
    fig = plt.figure(figsize=plt.figaspect(0.5))
    ax = fig.add_subplot(111)
    ax.loglog(h_vals, eps_num, marker='o',
            label='error of numerical approximation: $\epsilon_{\mathrm{num}}$')
    if eps_trun is not None:
        ax.loglog(h_vals, eps_trun, marker='x',
              label='theoretical truncation error: $\epsilon_{\mathrm{trun}}$')
    ax.set_xlabel('spacing $h$')
    ax.set_ylabel("absolute error")
    if title:
        ax.set_title(title)
    ax.legend()
    plt.show()

```

#### Backward Difference

Similary to the *forward difference* (2), we can define the *backward difference*:

$$f'(x)\approx \frac{f(x)-f(x-h)}{h}\, \qquad \text{with}\qquad h>0 \;. \tag{9}$$

You can easily derive this formula from (2) by assuming a negative $h$, say $h=-\tilde{h}$ where $\tilde{h}>0$.

Like the *forward difference*, this is a *one-sided difference* first order method.

---
**Exercise (3):**

Implement the backward difference method.

---

```python
def derivative_backward(y, delta_x):
    """
    Computes backward difference on elements in 'y' array, assuming constant grid spacing 'delta_x'.

    Args:
    - y: an array of data points / function values
    - x: scalar, indicating the spacing between subsequent observation/evaluation points
    """
    n_eval_points = y.shape[0]
    derivative = np.ones(n_eval_points) * np.nan
    for i in range(1,n_eval_points):
        derivative[i] = (y[i]-y[i-1])/delta_x
    return derivative
```

```python
## THIS IS EQUIVALENT TO ABOVE BUT AVOIDS FOR LOOP BY VECTORIZATION

def derivative_backward_array(y, delta_x):
    """
    Computes backward difference on elements in 'y' array, assuming constant grid spacing 'delta_x'.

    Args:
    - y: an array of data points / function values
    - x: scalar, indicating the spacing between subsequent observation/evaluation points
    """
    # instead of iterating over each pair i, i+1, we create a new shifted array
    y_backward = np.roll(y, -1)     # i-th item in this array corresponds to value at i+1 in y array
    derivative = (y_backward - y)/delta_x
    derivative[0] = np.nan
    return derivative
```

```python
## Functions and evaluation points
x_0 = np.pi/6
function = np.cos
n_steps=40
inital_h=0.1

# Compute approximations
h_values, y_p_approx_num = compute_approx_different_h(function_num_deriv=derivative_backward,
                                                        function=function,
                                                        x_0=x_0, n_steps=n_steps, inital_h=inital_h)
# Compute numerical error
y_p = -np.sin(x_0)
eps_num = np.abs(y_p - y_p_approx_num)
# Compute convergence order
convergence_order  = compute_convergence_order(eps_num)
# compute theoretical (truncation) error from taylor expansion
eps_trun = np.cos(x_0)*h_values/2
# plot
plot_errors(h_values, eps_num, eps_trun,
            title="Errors of backward difference approximation of 1st derivative")

print("Convergence order: ",convergence_order)

```

#### Centered Difference

Forward and backward difference methods are both based on *one-sided differences*.
Could we use information from both sides of the current evaluation point $x_0$ to inform the numerical derivative?

From the Taylor expansion, equation (3), we can derive truncated approximations for the forward and backward difference:
\begin{align}
 f(x+h) &= f(x) + h\, f'(x) + \frac{1}{2}h^2\, f''(x) + \frac{1}{6}h^3\, f'''(x) \tag{10a}\\
 f(x-h) &= f(x) - h\, f'(x) + \frac{1}{2}h^2\, f''(x) - \frac{1}{6}h^3\, f'''(x) \tag{10b}
\end{align}
Substracting (10b) from (10a) gives:
$$f(x+h) - f(x-h) = 2h\, f'(x) + \frac{2}{6}h^3\, f'''(x)\,$$
and after rearranging:
$$f'(x) = \frac{f(x+h)-f(x-h)}{2h} - \frac{1}{6}h^2\, f'''(x)\,. \tag{11}$$

This approximation of the derivative is called **centered difference** formula. The highest order truncation error term is of *second order* in $h$ ($\mathcal{O}(h^2)$), which means that the *centered difference* approximation is a second order method.

Higher-order approximations can be derived in a similar manner, using Taylor expansions with more terms.

---
**Exercise (4):**

Implement the centered difference method for the first derivative (10) and verify that your implementation shows the expected convergence behavior.

---

```python

def derivative_centered(y, delta_x):
    """
    Computes centered difference on elements in 'y' array, assuming constant grid spacing 'delta_x'.
    Args:
    - y: an array of data points / function values
    - x: scalar, indicating the spacing between subsequent observation/evaluation points
    """
    n_eval_points = y.shape[0]
    derivative = np.ones(n_eval_points) * np.nan
    for i in range(1, n_eval_points-1):
      derivative[i] = (y[i+1]-y[i-1])/(2*delta_x)
    return derivative

```

```python
## THIS IS EQUIVALENT TO ABOVE BUT USES NUMPY'S VECTORIZATION INSTEAD OF FOR LOOP

def derivative_centered_array(y, delta_x):
    """
    Computes centered difference on elements in 'y' array, assuming constant grid spacing 'delta_x'.

    Args:
    - y: an array of data points / function values
    - x: scalar, indicating the spacing between subsequent observation/evaluation points
    """
    y_backward = np.roll(y, 1)     # i-th item corresponds to value at i-1
    y_forward = np.roll(y, -1)     # i-th item corresponds to value at i+1
    derivative = (y_forward-y_backward)/(2*delta_x)
    derivative[0] = np.nan
    derivative[-1] = np.nan
    return derivative
```

```python
## Implementation where you can provide either
#  - scalar value for constant grid spacing, or
#  - array of same size as y for actual evaluation points

def derivative_centered_alternative(y, delta_x):
    """
    Computes centered derivative of data at point `a` with.
    Warning: Derivative at boundary points is not computed

    Args:
    - data: an array of data points / function values
    - x:    can be a scalar, indicating the spacing between subsequent
            observation/evaluation points, or
            an array of same length as data, indicating the observation points themselves
    """
    n_eval_points = data.shape[0]
    derivative = np.ones(n_eval_points) * np.nan
    if isinstance(x, np.ndarray):      # x is array of coordinates
        for i in range(1, n_eval_points-1):
            derivative[i] = (data[i+1]-data[i-1])/(x[i+1]-x[i-1])
    elif (isinstance(x, float) or isinstance(x, int)):
        for i in range(1, n_eval_points-1): # x is spacing between points
            derivative[i] = (data[i+1]-data[i-1])/(2*x)
    return derivative
```

```python
## THIS IS EQUIVALENT TO ABOVE BUT USES NUMPY'S VECTORIZATION INSTEAD OF FOR LOOP

def derivative_centered_alternative_array(data, x):
    """
    Returns centered derivative of function `function` at point `a` with.
    Warning: Derivative at boundary points is not computed

    Args:
    - data: an array of data points
    - x:    can be a scalar, indicating the spacing between subsequent
            observation/evaluation points, or
            an array of same length as data, indicating the observation points
    """
    data_backward = np.roll(data, 1)     # i-th item corresponds to value at i-1
    data_forward = np.roll(data, -1)     # i-th item corresponds to value at i+1
    if isinstance(x, np.ndarray):
        x_backward = np.roll(x, 1)       # i-th item corresponds to value at i-1
        x_forward = np.roll(x, -1)       # i-th item corresponds to value at i+1
        derivative = (data_forward-data_backward)/(x_forward - x_backward)
    elif (isinstance(x, float) or isinstance(x, int)):
        derivative = (data_forward-data_backward)/(2*x)
    # above computations use 'periodic' boundary conditions, i.e. x_0 = x_{N+1}
    # but in general, we do not have sufficient information to approximate
    # f' at the boundary: x_0 and x_N
    derivative[:1] = np.nan
    derivative[-1:] = np.nan
    return derivative
```

```python
## Functions and evaluation points
x_0 = np.pi/6
function = np.cos
n_steps=40
inital_h=0.1

# Compute approximations
h_values, y_p_approx_num = compute_approx_different_h(function_num_deriv=derivative_centered_array,
                                                    function=function,
                                                    x_0=x_0, n_steps=n_steps, inital_h=inital_h)
# Compute numerical error
y_p = -np.sin(x_0)
eps_num = np.abs(y_p - y_p_approx_num)
# Compute convergence order
convergence_order  = compute_convergence_order(eps_num)
# compute theoretical (truncation) error from taylor expansion
eps_trun = np.sin(x_0)*h_values**2/6
# plot
plot_errors(h_values, eps_num, eps_trun,
            title="Errors of centered difference approximation of 1st derivative")

print("Convergence order: ",convergence_order)

```

#### Higher order Derivatives

A similar approach as for the *centered difference* scheme can be used to obtain finite approximations of higher derivatives, such as $f''(x)$, $f'''(x)$, etc.

Let's derive a second order scheme for the second derivative:

We add the fourth derivative term ($n=4$) to equations (10) and consider the sum of these extended (10a) and (10b):
$$f(x+h) + f(x-h) = 2\, f(x) + h^2\, f''(x) + \frac{2}{24}h^4\, f''''(x)\;.$$

Rearrangement yields a second order finite difference approximation of the second derivative:
$$f''(x) = \frac{f(x+h) - 2\,f(x) + f(x-h)}{h^2}-\frac{2}{24}h^2\, f''''(x) \tag{12}$$

---
**Exercise (5):**

Implement Eq. (12) and verify that your implementation shows the expected convergence behavior.

---

### Derivatives at Domain Boundaries

Note that our ability to approximate the derivative of a function at the domain boundary with a given finite difference scheme may be limited by the available information.
Derivatives at these points can typically be approximated by a combination of one-sided and centered finite difference schemes.

For example, the centered difference scheme (11) requires function values from two neighboring points $f(x_{i-1})$, $f(x_{i+1})$ to approximate $f'(x_i)$, and therefore cannot provide estimates for the derivative at points $x_0$ and $x_N$.

On the other hand, forward and backward difference formula only require information from the current point $x_i$ and the 'next' $x_{i+1}$ (forward) or  'previous' point $x_{i-1}$ (backward), respectively.
Hence, the first derivative on the entire grid $x_0, x_1, \ldots, x_N$ could be estimated using a combination of forward, centered and backward difference schemes.

### 'Exact' Numerical Derivative

When the truncation error of a finite difference method vanishes for a given function, the method computes the exact derivative of that function.

Consider the following polynomials:

- Linear: $f(x)=1+x$
- Quadratic: $f(x)=1+x+x^2$
- Cubic: $f(x)=1+x+x^2+x^3$

We now approximate the first derivative $f'(x)$ at $x=10$ for each of these Polynomials using the centered difference scheme and compute the approximation error in function of spacing $h$:

```python
# Functions and derivatives
def linear_function(x):
    return 1+x

def linear_function_deriv(x):
    return 1

def quad_function(x):
    return 1+x+x**2

def quad_function_deriv(x):
    return 1+2*x

def cube_function(x):
    return 1+x+x**2+x**3

def cube_function_deriv(x):
    return 1+2*x+3*x**2

## Evaluation points
x_0 = 10
n_steps=40
inital_h=0.1

# Linear
h_values_1, y_p_approx_num_1 = compute_approx_different_h(function_num_deriv=derivative_centered_array,
                                                        function=linear_function,
                                                        x_0=x_0, n_steps=n_steps, inital_h=inital_h)
y_p_1 = linear_function_deriv(x_0)
eps_num_1 = np.abs(y_p_1 - y_p_approx_num_1)

# Quadratic
h_values_2, y_p_approx_num_2 = compute_approx_different_h(function_num_deriv=derivative_centered_array,
                                                        function=quad_function,
                                                        x_0=x_0, n_steps=n_steps, inital_h=inital_h)
y_p_2 = quad_function_deriv(x_0)
eps_num_2 = np.abs(y_p_2 - y_p_approx_num_2)

# Cubic
h_values_3, y_p_approx_num_3 = compute_approx_different_h(function_num_deriv=derivative_centered_array,
                                                        function=cube_function,
                                                        x_0=x_0, n_steps=n_steps, inital_h=inital_h)
y_p_3 = cube_function_deriv(x_0)
eps_num_3 = np.abs(y_p_3 - y_p_approx_num_3)

# Plot
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)

ax.loglog(h_values_1, eps_num_1, marker='o',
        label='error of numerical approximation: Linear Function')
ax.loglog(h_values_2, eps_num_2, marker='o',
        label='error of numerical approximation: Quadratic Function')
ax.loglog(h_values_3, eps_num_3, marker='o',
        label='error of numerical approximation: Cubic Function')

ax.set_xlabel('spacing $h$')
ax.set_ylabel("absolute error")
ax.legend()
plt.show()
```

The truncation error of the centered difference scheme is proportional to the third derivative of the function and therefore vanishes for linear and quadratic functions.
The approximation error for those cases is dominated by numeric rounding errors for all choices of $h$ and increases monotonically with decreasing spacing $h$.
This is not the case for the cubic function where the truncation error does not vanish.

## Useful Python / Numpy Functions

```python
# np.gradient(); 1st derivative, 2nd order
# check help(np.gradient) for info

x_min = 0
x_max = 4*np.pi
n_points = 100

x = np.linspace(x_min, x_max, n_points)
y = np.sin(x)
y_p = np.gradient(y, x)

# plot f, f'_approximated
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)
ax.plot(x, y,
        label='f(x) = cos(x)' )
ax.plot(x, y_p,
        label="approx f'(x)", color='red', linestyle=':')

# change x axis ticks to multiples of pi
x_ticks_values = np.arange(0, 4*np.pi, np.pi/2 )
x_ticks_labels = np.arange(0, 4, 0.5)
ax.set_xticks(x_ticks_values)
ax.set_xticklabels(x_ticks_labels)
ax.set_xlabel("x in multiples of $\pi$")

ax.legend()
plt.show()
```

```python
# np.diff(x, n) computes the difference x_i+n - x_i and shortens the array by n

x = np.linspace(0, 4*np.pi, 40)
y = np.sin(x)

# using np.diff, the forward difference scheme becomes
y_p = np.diff(y, n=1) / np.diff(x, n=1)
x_shortened = x[1:]

# plot f, f'_approximated
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)
ax.plot(x, y,
        label='f(x) = cos(x)' )

# note that we removed x[0]
ax.plot(x_shortened, y_p,
        label="approx f'(x)", color='red', linestyle=':')

# change x axis ticks to multiples of pi
x_ticks_values = np.arange(0, 4*np.pi, np.pi/2 )
x_ticks_labels = np.arange(0, 4, 0.5)
ax.set_xticks(x_ticks_values)
ax.set_xticklabels(x_ticks_labels)
ax.set_xlabel("x in multiples of $\pi$")

ax.legend()
plt.show()
```

## Exercises

- In [this](https://github.com/cohmathonc/biosci670/blob/master/IntroductionComputationalMethods/exercises/04_NumericalDifferentiation.ipynb) exercise you extend the centered difference scheme to the domain boundary and approximate the first derivative of a sequence of given data points.

###### About
This notebook is part of the *biosci670* course on *Mathematical Modeling and Methods for Biomedical Science*.
See https://github.com/cohmathonc/biosci670coh for more information and material.


# The Minimax-Q Algorithm

For Each Agent $i\in\mathcal{I}$, given the states set $\underline{\textbf{S}}$, the own action set  $\underline{\textbf{A}}$, and the action set of the opponent $\underline{\textbf{O}}$:

### 1. Initialization

```
For all s in states set S, a in own-action set A, o in opponent-action set O:
    Let Q[s,a,o] := 1

For all s in S:
    Let V[s] := 1

For all s in S, a in A:
    Let pi[s, a] = 1 / |A|

Let alpha := 1.0

And other parameters:
1. the discount factor - gamma;
2. the exploration factor - explor;
3. the learning rate decay factor - decay;
```

### 2. Choose An Action

```
With probability explor, return an action uniformly at random.
Otherwise, if current state is s, return action a with probability pi[s, a]
```

### 3. Learning
```
After receiving reward rew for moving from state s to s' via own action a and opponent's action o
    Let Q[s, a, o] := ( 1 - alpha ) * Q[s, a, o] + alpha * ( rew + gamma * V[s'] )

    Using linear programming to find the optimal pi[s, .] such that:
        pi[s, .] := argmax{pi'[s, .], min{o', sum{a', pi[s, a'] * Q[s, a', o']}}}

    Let V[s] := min{o', sum{a', pi[s, a'] * Q[s, a', o']}}

    Let alpha := alpha * decay
```

### Linear Programming to find the optimal $PD(\underline{\textbf{A}}|s)$

According to the notion of optimality, the agent's minimum expected reward should be as large as possible.

Assumed that our unknown is $\underline{\textbf{X}} = {V, pi[s, a_0],...,p[s, a_n]}$, our optimization problem could be defined as the following:

\begin{equation}
  \max_{\pi_a} V\ \longrightarrow\ \min_{\pi_a} -V(its\ dual\ form)\ \ s.t.
  \begin{cases}
     V - pi[s, a_0] * Q[s, a_0, o_0] - ... - pi[s, a_n] * Q[s, a_n, o_0] \leq 0\\
     .\\
     .\\
     V - pi[s, a_0] * Q[s, a_0, o_n] - ... - pi[s, a_n] * Q[s, a_n, o_n] \leq 0\\
     pi[s, a_0] + ... + pi[s, a_n] = 1
  \end{cases}
\end{equation}

```
import numpy as np
from scipy.optimize import linprog
import matplotlib.pyplot as plt
import abc
```

```
class Agent(object):
  __metaclass__ = abc.ABCMeta

  '''
  Default attributes for each agent:
  1. _IsLearning controls whether agent is learning
  2. The size of state set, |S|
  3. The number of actions, |A|
  4. The number of opponent's actions, |O|(only for minimax-Q)
  5. The discount factor, gamma
  6. The learning rate factor, alpha
  7. The rate at which the learning rate decays, decay
  8. The probability to exploration, explor
  '''
  _IsLearning = True
  _Num_of_S = 0
  _Num_of_A = 0
  _Num_of_O = 0
  _Gamma = 0
  _Alpha = 0
  _Decay = 0
  _Explor = 0

  def __init__(self):
    super(Agent, self).__init__()

  '''
  Protected attributes' setters and getters
  '''
  @property
  def IsLearning(self):
    return self._IsLearning

  @IsLearning.setter
  def IsLearning(self, IsLearn):
    self._IsLearning = IsLearn

  @property
  def Num_of_S(self):
    return self._Num_of_S

  @Num_of_S.setter
  def Num_of_S(self, _val):
    self._Num_of_S = _val

  @property
  def Num_of_A(self):
    return self._Num_of_A

  @Num_of_A.setter
  def Num_of_A(self, _val):
    self._Num_of_A = _val

  @property
  def Num_of_O(self):
    return self._Num_of_O

  @Num_of_O.setter
  def Num_of_O(self, _val):
    self._Num_of_O = _val

  @property
  def gamma(self):
    return self._Gamma

  @gamma.setter
  def gamma(self, _val):
    self._Gamma = _val

  @property
  def alpha(self):
    return self._Alpha

  @alpha.setter
  def alpha(self, _val):
    self._Alpha = _val

  @property
  def decay(self):
    return self._Decay

  @decay.setter
  def decay(self, _val):
    self._Decay = _val

  @property
  def explor(self):
    return self._Explor

  @explor.setter
  def explor(self, _val):
    self._Explor = _val

  '''
  Initialize Q, V and Pi matrices
  '''
  @abc.abstractmethod
  def Init_Params(self):
    raise NotImplementedError

  '''
  Choose an action from the action set A/O.
  '''
  @abc.abstractmethod
  def Action(self, s):
    raise NotImplementedError

  '''
  Update:
  1. Q(s,a,..): The quality of a state-action pair (s,a) or (s,a,o);
  2. V(s): The value of state s;
  3. Pi(s,...): The agent’s current policy for state s;
  4. Alpha: The learning rate
  '''
  @abc.abstractmethod
  def Update(self, reward, old_s, new_s, a, o):
    raise NotImplementedError

  '''
  Using Linear programming solver to calculate:
  1. The maximum of V(s),
  2. The corresponding optimal policy Pi(s,...)
  '''
  @abc.abstractmethod
  def Optimization(self, old_s, stop=False):
    raise NotImplementedError
```

```
class Minimax_Q(Agent):

  def __init__(self, *args):
    '''
      Parameters:
      0. The size of state set |S|
      1. The number of the agent's actions |A|
      2. The number of the opponent's actions |O|
      3. The discount factor, gamma
      4. The probability for the agent to explore the entire state space, explor
      5. The rate at which the learning rate decays, decay
    '''
    super(Minimax_Q, self).__init__()
    self.Num_of_S = args[0]
    self.Num_of_A = args[1]
    self.Num_of_O = args[2]
    self.gamma = args[3]
    self.explor = args[4]
    self.decay = args[5]
    self.Init_Params()

  def Init_Params(self):
    self.Q = np.ones((self.Num_of_S, self.Num_of_A, self.Num_of_O))
    self.V = np.ones(self.Num_of_S)
    self.Pi = np.ones((self.Num_of_S, self.Num_of_A)) / self.Num_of_A
    self.alpha = 1.0

  def Action(self, s):
    if self.IsLearning and np.random.rand() < self.explor:
      return np.random.randint(self.Num_of_A)
    else:
      prob = np.random.rand()
      a = 0
      a += np.sum(prob > np.cumsum(self.Pi[s]))
      return a

  def Update(self, reward, old_s, new_s, a, o):
    if not self.IsLearning:
      return

    self.Q[old_s, a, o] = (1 - self.alpha) * self.Q[old_s, a, o]\
                            + self.alpha * (reward + self.gamma * self.V[new_s])
    self.V[old_s] = self.Optimization(s)
    self.alpha *= self.decay

  def Optimization(self, old_s, stop=False):
    # Coefficients of the objectvie function: -V
    Coeffs_Obj = np.zeros(self.Num_of_A + 1)
    Coeffs_Obj[0] = -1

    # Coefficients of the inequality constrains
    Coeffs_Inq = np.ones((self.Num_of_O, self.Num_of_A + 1))
    Coeffs_Inq[:,1:] = -self.Q[old_s,:,:].T
    UB_Inq = np.zeros(self.Num_of_O)

    # Coefficients of the equality constrains
    Coeffs_Eq = np.ones((1, self.Num_of_A + 1))
    Coeffs_Eq[0,0] = 0
    UB_Eq = [1]

    # Value range of each variable
    Bounds = ((None, None),)
    for i in np.arange(self.Num_of_A):
      Bounds += ((0, 1),)

    # Linear programming solver
    res = linprog(Coeffs_Obj, A_ub=Coeffs_Inq, b_ub=UB_Inq\
                 , A_eq=Coeffs_Eq, b_eq=UB_Eq, bounds=Bounds)

    # Check Solution
    if res.success:
      self.Pi[old_s] = res.x[1:]
    elif not stop:
      return Optimization(old_s, stop=True)
    else:
      return self.V[old_s]

    return res.x[0]

  def PlotPolicy(self, s, A):
        for a in range(self.Num_of_A):
            print("Action ID: {}, Name: {} : P={}".format(a, A[a], self.Pi[s, a]))
```

```
A = {0:"head", 1:"tail"}
agent = Minimax_Q(1, 2, 2, 1e-4, 0.2, 0.9)
agent.Q[0] = [[0, 1], [1, 0.5]]
V = agent.Optimization(0)
print(agent.Pi)
agent.PlotPolicy(0, A)
```

    [[0.33333333 0.66666667]]
    Action ID: 0, Name: head : P=0.33333333333333337
    Action ID: 1, Name: tail : P=0.6666666666666666

# The Q-Learning Algorithm

The difference between Q-learning and Minimax-Q is:

1.   It applies a “max” operator in place of the minimax to choose the best action with the highest Q value
2.   The Q-table in Q-learning does not keep information about the opponent’s action.

The other parameters are set identically to the minimax-Q case.

```
class Q(Agent):
  def __init__(self, *args):
    '''
      Parameters:
      0. The size of the state set |S|
      1. The number of the agent's actions |A|
      2. The discount factor, gamma
      3. The probability for the agent to explore the entire state space, explor
      4. The rate at which the learning rate decays, decay
    '''
    super(Q, self).__init__()
    self.Num_of_S = args[0]
    self.Num_of_A = args[1]
    self.gamma = args[2]
    self.explor = args[3]
    self.decay = args[4]

    self.Init_Params()

  def Init_Params(self):
    self.Q = np.ones((self.Num_of_S, self.Num_of_A))
    self.V = np.ones(self.Num_of_S)
    self.Pi = np.ones((self.Num_of_S, self.Num_of_A)) / self.Num_of_A
    self.alpha = 1.0

  def Action(self, s):
    if self.IsLearning and np.random.rand() < self.explor:
      return np.random.randint(self.Num_of_A)
    else:
      return np.argmax(self.Q[s])

  def Update(self, reward, old_s, new_s, a, o):
    if not self.IsLearning:
      return

    self.Q[old_s, a] = (1 - self.alpha) * self.Q[old_s, a] + self.alpha * \
                       (reward + self.gamma * self.V[new_s])
    self.Pi[old_s] = np.zeros(self.Num_of_A)
    self.Pi[old_s, np.argmax(self.Q[old_s])] = 1
    self.V[old_s] = self.Q[old_s, np.argmax(self.Q[old_s])]
    self.alpha *= self.decay

  def Optimization(self, old_s, stop=False):
    pass

  def PlotPolicy(self, s, A):
    for a in np.arange(self.Num_of_A):
      print("Action ID: {}, Name: {} : P={}".format(a, A[a], self.Pi[s, a]))
```

```
A = {0:"head", 1:"tail"}
agent = Q(1, 2, 1e-4, 0.2, 0.9)
agent.Q[0] = [1.0, 0.5]
agent.Update(1, 0, 0, 0, 0)
print(agent.Q)
agent.PlotPolicy(0, A)
```

    [[1.0001 0.5   ]]
    Action ID: 0, Name: head : P=1.0
    Action ID: 1, Name: tail : P=0.0

# Random Policy

Random Policy means that the agent chooses actions uniformly at random, and performs no any learning.

```
class Rand(Agent):
  def __init__(self, *args):
    '''
      Parameters:
      0. the number of the agent's actions |A|
    '''
    self.Num_of_A = args[0]

  def Init_Params(self):
    pass

  def Action(self, s):
    return np.random.randint(self.Num_of_A)

  def Update(self, reward, old_s, new_s, a, o):
    pass

  def Optimization(self, old_s, stop=False):
    pass
```

```
A = {0:"head", 1:"tail"}
agent = Rand(len(A))
print(A[agent.Action(0)])
```

    tail

# The "Rock, Paper, Scissors" Game

The "Rock, Paper, Scissors" Game is a normal strategic form game, it has a following game matrix, $R$, that is the instantaneous rewards matrix:

|||| Agent |
--- | --- | --- | ---
||| Rock | Paper | Scissors
|| Rock| 0 | 1 | -1
__Opponent__|| Paper | -1 | 0 | 1
|| Scissors | 1 | -1 | 0

where component $R_{i,j}$ is the reward to the agent for choosing action $j$ when its opponent chooses action $i$.

Moreover, the agent wants to maximise its own expected reward while its opponent tries to minimise it.

Any deterministic choice for the agent can be blocked indefinitely by a clever opponent. Only by choosing randomly amid its strategy set $\underline{\textbf{S}}$ can the agent guarantee an opening and therefore an opportunity to win the game.

Then, the agent's policy is a probability distribution over its actions $\pi\in PD(\underline{\textbf{A}})$. For "rock, paper, scissors", $\pi$ is made up of 3 components: $\pi_{rock}$, $\pi_{paper}$ and $\pi_{scissors}$.

According to optimality, the optimal agent's minimum expected reward should be as large as possible. Supposed that, there is a policy which can guarantee the agent to get an expected score of $V$ no matter which action the opponent acts. Hence, the linear constraints on the solution should be:

\begin{equation}
\begin{cases}
\pi_{paper} - \pi_{scissors} \geq V\ (vs.\ rock)\\
-\pi_{rock} +\pi_{scissors} \geq V\ (vs.\ paper)\\
\pi_{rock} -\pi_{paper} \geq V\ (vs.\ scissors)\\
\pi_{rock}+\pi_{paper}+\pi_{scissors} = 1
\end{cases}
\end{equation}

```
class Game(object):
  __metaclass__  = abc.ABCMeta

  def __init__(self):
    super(Game, self).__init__()

  @abc.abstractmethod
  def Play(self, a, o):
    '''
      Play this game once
    '''
    raise NotImplementedError
```

```
class MatrixGame(Game):
  def __init__(self):
    self.M = np.array([[0, 1, -1], [-1, 0, 1], [1, -1, 0]])

  def Play(self, a, o):
    if self.M[a, o] < 0:
      return -1 #agent loss
    elif self.M[a, o] > 0:
      return 1  #agent win
    else:
      return 0  #players draw
```

```
iterations = 50000
records = np.zeros(iterations)
A = {0:"rock", 1:"paper", 2:"scissors"}
O = {0:"rock", 1:"paper", 2:"scissors"}
S = {0:"null"}
G = MatrixGame()
Opt = np.array([1./3, 1./3, 1./3])

decay = 10**(np.log10(0.01)/iterations)
drawProbability = 0.01

agent = Minimax_Q(len(S), len(A), len(O), 1-drawProbability,  0.01, decay )
opponent = Rand(len(O))

History_A = []
History_O = []
R = []
P_Diff = []

for i in np.arange(iterations):
  if (i % (iterations / 10) == 0):
    print("%d%% finished." % (i * 100 / iterations))
  result = 0
  while result == 0:
    s = np.random.randint(0, len(S))
    a = agent.Action(s)
    o = opponent.Action(s)
    result = G.Play(a, o)

    History_A.extend([a])
    History_O.extend([o])
    P_Diff.extend([max(abs(agent.Pi[0] - Opt))])

    r = G.M[a, o]
    R.extend([r])

    agent.Update(r, s, s, a, o )
    opponent.Update(-r, s, s, a, o )
  records[i] = result
```

    0% finished.
    10% finished.
    20% finished.
    30% finished.
    40% finished.
    50% finished.
    60% finished.
    70% finished.
    80% finished.
    90% finished.

```
nx = 3
ny = 1

dxs = 7.5
dys = 5.0

fig, axes = plt.subplots(1, 3, figsize=(dxs*nx, dys*ny))
axes = axes.flatten()
for i_a in range(3):
  ax = axes[i_a]
  if i_a == 0:
    ax.plot((records == 1).cumsum())
    ax.plot((records == -1).cumsum())
    ax.legend(('Wins Agent', 'Wins Opponent'), loc=(0.7, 0.1))
    ax.set_title("Number of Times both Players Win")
  elif i_a == 1:
    ax.plot(np.array(R).cumsum())
    ax.plot(-np.array(R).cumsum())
    ax.legend(('Reward Agent', 'Reward Opponent'), loc=(0, 0.01))
    ax.set_title("Each Player's Reward over Iterations")
  elif i_a == 2:
    ax.plot(P_Diff[:100])
    ax.legend(('Distance from the Optimal Policy',), loc=(0.4, 0.8))
    ax.set_title("Speed of Convergence to the Optimal Policy")
plt.show()
fig.tight_layout()
plt.show()

print("\n==============")
print("A - action 0: %f" % (1. * sum(np.array(History_A) == 0) / len(History_A)))
print("A - action 1: %f" % (1. * sum(np.array(History_A) == 1) / len(History_A)))
print("A - action 2: %f" % (1. * sum(np.array(History_A) == 2) / len(History_A)))

print("\n==============")
print("B - action 0: %f" % (1. * sum(np.array(History_O) == 0) / len(History_O)))
print("B - action 1: %f" % (1. * sum(np.array(History_O) == 1) / len(History_O)))
print("B - action 2: %f" % (1. * sum(np.array(History_O) == 2) / len(History_O)))

print("\n==============")
print("Wins A : %d" % (records == 1).sum())
print("Wins B : %d" % (records == -1).sum())
print("Draws  : %d" % (records == 0).sum())

print("\n==============")
print("Reward A : %d" % sum(R))
print("Reward B : %d" % -sum(R))
```

```
iterations = 50000
records = np.zeros(iterations)
A = {0:"rock", 1:"paper", 2:"scissors"}
O = {0:"rock", 1:"paper", 2:"scissors"}
S = {0:"null"}
G = MatrixGame()
Opt = np.array([1./3, 1./3, 1./3])

decay = 10**(np.log10(0.01)/iterations)
drawProbability = 0.6

agent = Minimax_Q_Agent(S, A, O, gamma=1-drawProbability,  explor=0.01, decay=decay )
opponent = Rand_Agent(O)

History_A = []
History_O = []
R = []
P_Diff = []

for i in np.arange(iterations):
  if (i % (iterations / 10) == 0):
    print("%d%% finished." % (i * 100 / iterations))
  result = 0
  while result == 0:
    s = np.random.randint(0, len(S))
    a = agent.ChooseAction(s)
    o = opponent.ChooseAction(s)
    result = G.Play(a, o)

    History_A.extend([a])
    History_O.extend([o])
    P_Diff.extend([max(abs(agent.Pi[0] - Opt))])

    r = G.M[a, o]
    R.extend([r])

    agent.Learn(r, s, s, [a, o] )
    opponent.Learn(-r, s, s, [a, o] )
  records[i] = result
```

    0% finished.
    10% finished.
    20% finished.
    30% finished.
    40% finished.
    50% finished.
    60% finished.
    70% finished.
    80% finished.
    90% finished.

```
nx = 3
ny = 1

dxs = 7.5
dys = 5.0

fig, axes = plt.subplots(1, 3, figsize=(dxs*nx, dys*ny))
axes = axes.flatten()
for i_a in range(3):
  ax = axes[i_a]
  if i_a == 0:
    ax.plot((records == 1).cumsum())
    ax.plot((records == -1).cumsum())
    ax.legend(('Wins Agent', 'Wins Opponent'), loc=(0.7, 0.1))
    ax.set_title("Number of Times both Players Win")
  elif i_a == 1:
    ax.plot(np.array(R).cumsum())
    ax.plot(-np.array(R).cumsum())
    ax.legend(('Reward Agent', 'Reward Opponent'), loc=(0, 0.01))
    ax.set_title("Each Player's Reward over Iterations")
  elif i_a == 2:
    ax.plot(P_Diff[:100])
    ax.legend(('Distance from the Optimal Policy',), loc=(0.4, 0.8))
    ax.set_title("Speed of Convergence to the Optimal Policy")
plt.show()
fig.tight_layout()
plt.show()

print("\n==============")
print("A - action 0: %f" % (1. * sum(np.array(History_A) == 0) / len(History_A)))
print("A - action 1: %f" % (1. * sum(np.array(History_A) == 1) / len(History_A)))
print("A - action 2: %f" % (1. * sum(np.array(History_A) == 2) / len(History_A)))

print("\n==============")
print("B - action 0: %f" % (1. * sum(np.array(History_O) == 0) / len(History_O)))
print("B - action 1: %f" % (1. * sum(np.array(History_O) == 1) / len(History_O)))
print("B - action 2: %f" % (1. * sum(np.array(History_O) == 2) / len(History_O)))

print("\n==============")
print("Wins A : %d" % (records == 1).sum())
print("Wins B : %d" % (records == -1).sum())
print("Draws  : %d" % (records == 0).sum())

print("\n==============")
print("Reward A : %d" % sum(R))
print("Reward B : %d" % -sum(R))
```

```
iterations = 50000
records = np.zeros(iterations)
A = {0:"rock", 1:"paper", 2:"scissors"}
O = {0:"rock", 1:"paper", 2:"scissors"}
S = {0:"null"}
G = MatrixGame()
Opt = np.array([1./3, 1./3, 1./3])

decay = 0.5
drawProbability = 0.01

agent = Minimax_Q_Agent(S, A, O, gamma=1-drawProbability,  explor=0.01, decay=decay )
opponent = Rand_Agent(O)

History_A = []
History_O = []
R = []
P_Diff = []

for i in np.arange(iterations):
  if (i % (iterations / 10) == 0):
    print("%d%% finished." % (i * 100 / iterations))
  result = 0
  while result == 0:
    s = np.random.randint(0, len(S))
    a = agent.ChooseAction(s)
    o = opponent.ChooseAction(s)
    result = G.Play(a, o)

    History_A.extend([a])
    History_O.extend([o])
    P_Diff.extend([max(abs(agent.Pi[0] - Opt))])

    r = G.M[a, o]
    R.extend([r])

    agent.Learn(r, s, s, [a, o] )
    opponent.Learn(-r, s, s, [a, o] )
  records[i] = result
```

    0% finished.
    10% finished.
    20% finished.
    30% finished.
    40% finished.
    50% finished.
    60% finished.
    70% finished.
    80% finished.
    90% finished.

```
nx = 3
ny = 1

dxs = 7.5
dys = 5.0

fig, axes = plt.subplots(1, 3, figsize=(dxs*nx, dys*ny))
axes = axes.flatten()
for i_a in range(3):
  ax = axes[i_a]
  if i_a == 0:
    ax.plot((records == 1).cumsum())
    ax.plot((records == -1).cumsum())
    ax.legend(('Wins Agent', 'Wins Opponent'), loc=(0.7, 0.1))
    ax.set_title("Number of Times both Players Win")
  elif i_a == 1:
    ax.plot(np.array(R).cumsum())
    ax.plot(-np.array(R).cumsum())
    ax.legend(('Reward Agent', 'Reward Opponent'), loc=(0, 0.01))
    ax.set_title("Each Player's Reward over Iterations")
  elif i_a == 2:
    ax.plot(P_Diff[:100])
    ax.legend(('Distance from the Optimal Policy',), loc=(0.4, 0.8))
    ax.set_title("Speed of Convergence to the Optimal Policy")
plt.show()
fig.tight_layout()
plt.show()

print("\n==============")
print("A - action 0: %f" % (1. * sum(np.array(History_A) == 0) / len(History_A)))
print("A - action 1: %f" % (1. * sum(np.array(History_A) == 1) / len(History_A)))
print("A - action 2: %f" % (1. * sum(np.array(History_A) == 2) / len(History_A)))

print("\n==============")
print("B - action 0: %f" % (1. * sum(np.array(History_O) == 0) / len(History_O)))
print("B - action 1: %f" % (1. * sum(np.array(History_O) == 1) / len(History_O)))
print("B - action 2: %f" % (1. * sum(np.array(History_O) == 2) / len(History_O)))

print("\n==============")
print("Wins A : %d" % (records == 1).sum())
print("Wins B : %d" % (records == -1).sum())
print("Draws  : %d" % (records == 0).sum())

print("\n==============")
print("Reward A : %d" % sum(R))
print("Reward B : %d" % -sum(R))
```

# Soccer Game

__Game factors__:

*   A state set $\underline{\textbf{S}}$ with $4\times5$ elements, which represent the specific coordinates for all $4\times5$ grids
*   Two Players, A and B, which occupy distinct squares of the grid
*   Each player has $5$ actions on each turn: N, S, E, W, and stand

__Assumptions__:

*   The selected actions of both players are executed in random order
*   Possession of the ball goes to one or the other player at random

__Rule for the Game__:

*   When a player executes an action that would take it to the square occupied by the other player, possession of the ball goes to the stationary player and the move does not take place

__Goal__: The player with the ball(the circle) steps into the appropriate place: left for A, right for B. Then, player scores a point and the board is reset to the initial state.

```
class Soccer:
  def __init__(self, X_Lims = 5, Y_Lims = 4, Des_Y = [1, 2], Init_A = [3, 2], Init_O = [1,1], P_draw=0.1):
    # The Size of Play Ground
    self.X_Lims = X_Lims
    self.Y_Lims = Y_Lims

    # Y Coordinates of the Destination
    self.Des_Y = Des_Y

    self.Init_A = Init_A
    self.Init_O = Init_O

    self.Draw = P_draw

    self.init_params()

  # Reset this Game
  def init_params(self):
    # Action Set A and Player Set I
    self.A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
    self.I = {0:"Agent", 1:"Opponent"}

    # The initial Locations of both Players
    self.Init_Locs = np.array([self.Init_A, self.Init_O])

    # Current Locations of both Players
    self.Locations = self.Init_Locs.copy()

    # The Ownership of the Ball: A=0, B=1
    self.Ownership = np.random.randint(0, 2)

    # Offset for each Action
    self.Offsets = {
                      0: [-1, 0],
                      1: [0, 1],
                      2: [1, 0],
                      3: [0, -1],
                      4: [0, 0],
                   }

  # Validate the New Location
  def IsValid(self, new_loc):
    return (0 <= new_loc[0] < self.X_Lims and 0 <= new_loc[1] < self.Y_Lims)

  # Check whether the new Location is in the Destination
  def IsReach(self, new_loc):
    if (self.Des_Y[0] <= new_loc[1] <= self.Des_Y[1] ):
      if (new_loc[0] == -1 and self.Ownership == 0):
        return 1  # A won
      elif (new_loc[0] == self.X_Lims and self.Ownership == 1):
        return -1 # B won
    return 0

  # Execute player's action
  def Move(self, player, action):
    opponent = 1 - player

    # Update the location of the player
    New_Loc = self.Locations[player] + self.Offsets[action]

    # If new Location is occupied by opponent, ownership is changed
    if (New_Loc == self.Locations[opponent]).all():
      self.Ownership = opponent
    # If new Location is the destination
    elif self.Ownership == player and self.IsReach(New_Loc) != 0:
      return self.IsReach(New_Loc)
    # Coordinate is valid
    elif self.IsValid(New_Loc):
      self.Locations[player] = New_Loc

    return 0

  def Play(self, actions):
    # Players draw with some certainty
    if np.random.rand() < self.Draw:
      #print("Draw!")
      return -2

    # Moves are executed in random order
    first = np.random.randint(0,2)
    second = 1 - first

    # Is there someone reaching the desitination
    score = 0

    # The agent's turn
    score = self.Move(first, actions[first])
    if (score != 0):
      return score

    #The opponent's turn
    return self.Move(second, action[second])

  def Plot(self, positions=None, ballOwner=None):
    Locs = self.Locations if positions is None else np.array(positions)
    Owner = self.Ownership if ballOwner is None else ballOwner

    State = ''
    for y in range(self.Y_Lims)[::-1]:
      for x in range(self.X_Lims):
        if ([x, y] == Locs[0]).all():
          State += 'A' if Owner is 0 else 'a'
        elif ([x, y] == Locs[1]).all():
          State += 'B' if Owner is 1 else 'b'
        else:
          State += '-'
      State += '\n'

    print(State)
```

```
G = Soccer()
G.Plot()
#actions = [[0, 1], [0, 4], [1, 3], [1, 2], [1, 2], [4, 2], [4, 2]]
actions = [[0, 4], [0, 4], [0, 4], [0, 4], [0, 4]]
i = 1
for action in actions:
    print("iteration:{} - A:{}, B:{}".format(i, G.A[action[0]],G.A[action[1]]))
    print("The Result of Move: %d"% G.Play(action))
    G.Plot()
    i += 1
    print("\n=======================>")
```

    -----
    ---A-
    -b---
    -----

    iteration:1 - A:Left, B:No
    The Result of Move: 0
    -----
    --A--
    -b---
    -----

    =======================>
    iteration:2 - A:Left, B:No
    The Result of Move: 0
    -----
    -A---
    -b---
    -----

    =======================>
    iteration:3 - A:Left, B:No
    The Result of Move: 0
    -----
    A----
    -b---
    -----

    =======================>
    iteration:4 - A:Left, B:No
    The Result of Move: 1
    -----
    A----
    -b---
    -----

    =======================>
    iteration:5 - A:Left, B:No
    The Result of Move: 1
    -----
    A----
    -b---
    -----

    =======================>

```
def boardToState(G):
  x_A, y_A = G.Locations[0]
  x_B, y_B = G.Locations[1]
  s_A = y_A * G.X_Lims + x_A
  s_B = y_B * G.X_Lims + x_B
  s_B -= 1 if s_B > s_A else 0

  return ((s_A * (G.X_Lims * G.Y_Lims - 1) + s_B)\
          + (G.X_Lims * G.Y_Lims) * (G.X_Lims * G.Y_Lims - 1) * G.Ownership)

def stateToBoard(G, state):
  Ownership = state / ((G.X_Lims * G.Y_Lims) * (G.X_Lims * G.Y_Lims - 1))
  state = state % ((G.X_Lims * G.Y_Lims) * (G.X_Lims * G.Y_Lims - 1))

  s_A = state / (G.X_Lims * G.Y_Lims - 1)
  s_B = state % (G.X_Lims * G.Y_Lims - 1)
  s_B += 1 if s_B >= s_A else 0

  x_A = s_A % G.X_Lims
  y_A = s_A / G.X_Lims
  x_B = s_B % G.X_Lims
  y_B = s_B / G.X_Lims

  return [[[x_A, y_A], [x_B, y_B]], Ownership]

def plotResult(wins):
  sumWins = (wins == [[1], [-1], [-2]]).sum(1)
  lenWins = sumWins[0] + sumWins[1]
  print("Completed %d games" % (sumWins.sum()))
  print("Wins A : %d (%0.1f%%)" % (sumWins[0], (100. * sumWins[0] / lenWins)))
  print("Wins B : %d (%0.1f%%)" % (sumWins[1], (100. * sumWins[1] / lenWins)))
  print("Draws  : %d (%0.1f%%)" % (sumWins[2], (100. * sumWins[2] / sumWins.sum())))

  plt.plot((wins == 1).cumsum())
  plt.plot((wins == -1).cumsum())
  plt.legend(('WinsA', 'WinsB'), loc=(0.6, 0.8))
  plt.show()
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.01

MR = Minimax_Q_Agent(S, A, O, gamma=1-draw, explor=0.2, decay=decay )
R1 = Rand_Agent(A)

G = Soccer(P_draw=draw)

win_MR = np.zeros(iterations)
r = 0
for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = MR.ChooseAction(s)
    o = R1.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    MR.Learn(reward, s, new_s, [a, o] )
    R1.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      win_MR[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
plotResult(win_MR)
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.01

MM = Minimax_Q_Agent(S, A, O, gamma=1-draw, explor=0.2, decay=decay )
M1 = Minimax_Q_Agent(S, A, O, gamma=1-draw,  explor=0.2, decay=decay )

G = Soccer(P_draw=draw)

win_MM = np.zeros(iterations)
r = 0

for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = MM.ChooseAction(s)
    o = M1.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    MM.Learn(reward, s, new_s, [a, o] )
    M1.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      win_MM[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
 plotResult(win_MM)
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.01

QR = Q_Agent(S, A, gamma=1-draw, explor=0.2, decay=decay )
R2 = Rand_Agent(A)

G = Soccer(P_draw=draw)

win_QR = np.zeros(iterations)
r = 0
for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = QR.ChooseAction(s)
    o = R2.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    QR.Learn(reward, s, new_s, [a, o] )
    R2.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      win_QR[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
 plotResult(win_QR)
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.01

QQ = Q_Agent(S, A, gamma=1-draw, explor=0.2, decay=decay )
Q1 = Q_Agent(S, A, gamma=1-draw, explor=0.2, decay=decay )

G = Soccer(P_draw=draw)

win_QQ = np.zeros(iterations)
r = 0
for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = QQ.ChooseAction(s)
    o = Q1.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    QQ.Learn(reward, s, new_s, [a, o] )
    Q1.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      win_QQ[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
 plotResult(wins)
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.1

Rand = Rand_Agent(A)
MR.learning = False

G = Soccer(P_draw=draw)

wins_MR = np.zeros(iterations)
r = 0
for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = MR.ChooseAction(s)
    o = Rand.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    MR.Learn(reward, s, new_s, [a, o] )
    Rand.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      wins_MR[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
 plotResult(wins_MR)
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.1

MM.learning = False

G = Soccer(P_draw=draw)

wins_MM = np.zeros(iterations)
r = 0
for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = MM.ChooseAction(s)
    o = Rand.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    MM.Learn(reward, s, new_s, [a, o] )
    Rand.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      wins_MM[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
 plotResult(wins_MM)
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.1

QR.learning = False

G = Soccer(P_draw=draw)

wins_QR = np.zeros(iterations)
r = 0
for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = QR.ChooseAction(s)
    o = Rand.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    QR.Learn(reward, s, new_s, [a, o] )
    Rand.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      wins_QR[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
 plotResult(wins_QR)
```

```
iterations = 100000
Y_Lims = 4
X_Lims = 5
S = [ i for i in np.arange((X_Lims * Y_Lims) * (X_Lims * Y_Lims - 1) * 2)]
A = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}
O = {0:"Left", 1:"Up", 2:"Right", 3:"Down", 4:"No"}

decay = 10**(np.log10(0.01)/iterations)
draw = 0.1

QQ.learning = False

G = Soccer(P_draw=draw)

wins_QQ = np.zeros(iterations)
r = 0
for i in np.arange(iterations):
    if (i % (iterations / 10) == 0):
        print("%d%%" % (i * 100 / iterations))

    s = boardToState(G)
    a = QQ.ChooseAction(s)
    o = Rand.ChooseAction(s)
    r = G.Play([a, o])
    reward = 0 if r == -2 or r == 0 else r
    new_s = boardToState(G)

    QQ.Learn(reward, s, new_s, [a, o] )
    Rand.Learn(-reward, s, new_s, [a, o] )

    if r != 0:
      wins_QQ[i] = r
      G.init_params()
      r = 0
```

    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%

```
 plotResult(wins_QQ)
```

```
class Minimax_Q_Agent:
  def __init__(self, S, A, O, gamma, explor, decay):
    self.S = S
    self.A = A
    self.O = O
    self.Q = np.ones((len(S), len(A), len(O)), dtype=np.float32)
    self.V = np.ones( len(S), dtype=np.float32)
    self.Pi = np.ones((len(S), len(A)), dtype=np.float32) / len(A)
    self.alpha = 1.0
    self.gamma = gamma
    self.explor = explor
    self.decay = decay
    self.learning = True

  def ChooseAction(self, s):
    if self.learning and np.random.rand() < self.explor:
      return np.random.randint(len(self.A))
    else:
      p = np.random.rand()
      a = 0
      a += np.sum(p > np.cumsum(self.Pi[s]))
      return a

  def Learn(self, rew, s, s_, actions):
    if not self.learning:
      return

    a, o = actions
    self.Q[s, a, o] = (1-self.alpha) * self.Q[s, a, o] \
                          + self.alpha * (rew + self.gamma * self.V[s_])
    self.V[s] = self.UpdatePi(s)
    self.alpha *= self.decay

  def UpdatePi(self, s, stop=False):
    Coeffs_Obj = np.zeros(len(self.A) + 1)
    Coeffs_Obj[0] = -1

    Coeffs_Inq = np.ones((len(self.O), len(self.A) + 1))
    Coeffs_Inq[:,1:] = -self.Q[s,:,:].T
    UB_Inq = np.zeros(len(self.O))

    Coeffs_Eq = np.ones((1, len(self.A) + 1))
    Coeffs_Eq[0,0] = 0
    UB_Eq = [1]

    Bounds = ((None, None),)
    for i in np.arange(len(self.A)):
      Bounds += ((0, 1),)

    res = linprog(Coeffs_Obj, A_ub=Coeffs_Inq, b_ub=UB_Inq\
                    , A_eq=Coeffs_Eq, b_eq=UB_Eq, bounds=Bounds)

    if res.success:
      self.Pi[s] = res.x[1:]
    elif not stop:
      return self.UpdatePi(s, stop=True)
    else:
      #print("Alert : %s" % res.message)
      return self.V[s]

    return res.x[0]

  def PiPerState(self, s):
        for a in range(len(self.A)):
            print("Action ID: {}, Name: {} : P={}".format(a, self.A[a], self.Pi[s, a]))
```

```
agent = Minimax_Q_Agent({0:"null"}, {0:"head", 1:"tail"}, {0:"head",1:"tail"}, 1e-4, 0.2, 0.9)
agent.Q[0] = [[0, 1], [1, 0.5]]
V = agent.UpdatePi(0)
print(agent.Pi)
agent.PiPerState(0)
```

```
class Q_Agent():
  def __init__(self, S, A, gamma, explor, decay):
    self.S = S
    self.A = A
    self.Q = np.ones((len(S), len(A)))
    self.V = np.ones(len(S))
    self.Pi = np.ones((len(S), len(A))) / len(A)
    self.alpha = 1.0
    self.gamma = gamma
    self.explor = explor
    self.decay = decay
    self.learning = True

  def ChooseAction(self, s):
    if self.learning and np.random.rand() < self.explor:
      return np.random.randint(len(self.A))
    else:
      return np.argmax(self.Q[s])

  def Learn(self, rew, s, s_, actions):
    if not self.learning:
      return

    a, o = actions
    self.Q[s, a] = (1 - self.alpha) * self.Q[s, a] + self.alpha * (rew + self.gamma * self.V[s_])
    self.Pi[s] = np.zeros(len(self.A))
    self.Pi[s, np.argmax(self.Q[s])] = 1
    self.V[s] = self.Q[s, np.argmax(self.Q[s])]
    self.alpha *= self.decay

  def PiPerState(self, s):
    for a in np.arange(len(self.A)):
      print("Action ID: {}, Name: {} : P={}".format(a, self.A[a], self.Pi[s, a]))
```

```
agent = Q_Agent({0:"null"}, {0:"head", 1:"tail"}, 1e-4, 0.2, 0.9)
agent.Q[0] = [1.0, 0.5]
agent.Learn(1, 0, 0, [0, 0])
print(agent.Q)
agent.PiPerState(0)
```

```
class Rand_Agent():
  def __init__(self, A):
    self.A = A

  def ChooseAction(self, s):
    return np.random.randint(len(self.A))

  def Learn(self, rew, s, s_, actions):
    pass
```

```
agent = Rand_Agent({0:"head", 1:"tail"})
print(agent.A[agent.ChooseAction(0)])
```


# Basic stats using `Scipy`
In this example we will go over how to draw samples from various built in probability distributions and define your own custom distributions.

## Packages being used
+ `scipy`: has all the stats stuff
+ `numpy`: has all the array stuff

## Relevant documentation
+ `scipy.stats`: http://docs.scipy.org/doc/scipy/reference/tutorial/stats.html, http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_continuous.html#scipy.stats.rv_continuous, http://docs.scipy.org/doc/scipy/reference/stats.html#module-scipy.stats

```python
import numpy as np
import scipy.stats as st
# some special functions we will make use of later on
from scipy.special import erfc
from matplotlib import pyplot as plt
from astropy.visualization import hist
import mpl_style
%matplotlib notebook
plt.style.use(mpl_style.style1)
```

There are many probability distributions that are already available in `scipy`: http://docs.scipy.org/doc/scipy/reference/stats.html#module-scipy.stats.  These classes allow for the evaluations of PDFs, CDFs, PPFs, moments, random draws, and fitting.  As an example lets take a look at the normal distribution.

```python
norm = st.norm(loc=0, scale=1)
x = np.linspace(-5, 5, 1000)
plt.figure(1, figsize=(8, 10))
plt.subplot2grid((2, 2), (0, 0))
plt.plot(x, norm.pdf(x))
plt.xlabel('x')
plt.ylabel('PDF(x)')
plt.xlim(-5, 5)
plt.subplot2grid((2, 2), (0, 1))
plt.plot(x, norm.cdf(x))
plt.xlabel('x')
plt.ylabel('CDF(x)')
plt.xlim(-5, 5)
plt.subplot2grid((2, 2), (1, 0))
sample_norm = norm.rvs(size=100000)
hist(sample_norm, bins='knuth', histtype='step', lw=1.5, density=True)
plt.xlabel('x')
plt.ylabel('Random Sample')
plt.tight_layout()
```

You can calculate moments and fit data:

```python
for i in range(4):
    print('moment {0}: {1}'.format(i+1, norm.moment(i+1)))

print('best fit: {0}'.format(st.norm.fit(sample_norm)))
```

# Custom probability distributions
Sometimes you need to use obscure PDFs that are not already in `scipy` or `astropy`.  When this is the case you can make your own subclass of `st.rv_continuous` and overwrite the `_pdf` or `_cdf` methods.  This new sub class will act exactly like the built in distributions.

The methods you can override in the subclass are:

+ \_rvs: create a random sample drawn from the distribution
+ \_pdf: calculate the PDF at any point
+ \_cdf: calculate the CDF at any point
+ \_sf: survival function, a.k.a. 1-CDF(x)
+ \_ppf: percent point function, a.k.a. inverse CDF
+ \_isf: inverse survival function
+ \_stats: function that calculates the first 4 moments
+ \_munp: function that calculates the nth moment
+ \_entropy: differential entropy
+ \_argcheck: function to check the input arguments are valid (e.g. var>0)

You should override any method you have analytic functions for, otherwise (typically slow) numerical integration, differentiation, and function inversion are used to transform the ones that are specified.

## The exponentially modified Gaussian distribution
As and example lets create a class for the EMG distribution (https://en.wikipedia.org/wiki/Exponentially_modified_Gaussian_distribution).  This is the distributions resulting from the sum of a Gaussian random variable and an exponential random variable.  The PDF and CDF are:

\begin{align}
f(x;\mu,\sigma, \lambda) & = \frac{\lambda}{2} \exp{\left( \frac{\lambda}{2} \left[ 2\mu+\lambda\sigma^{2}-2x \right] \right)} \operatorname{erfc}{\left( \frac{\mu + \lambda\sigma^{2}-x}{\sigma\sqrt{2}} \right)} \\
F(x; \mu, \sigma, \lambda) & = \Phi(u, 0, v) - \Phi(u, v^2, v) \exp{\left( -u + \frac{v^2}{2} \right)} \\
\Phi(x, a, b) & = \frac{1}{2} \left[ 1 + \operatorname{erf}{\left( \frac{x - a}{b\sqrt{2}} \right)} \right] \\
u & = \lambda(x - \mu) \\
v & = \lambda\sigma
\end{align}

```python
# create a generating class
class EMG_gen1(st.rv_continuous):
    def _pdf(self, x, mu, sig, lam):
        u = 0.5 * lam * (2 * mu + lam * sig**2 - 2 * x)
        v = (mu + lam * sig**2 - x)/(sig * np.sqrt(2))
        return 0.5 * lam * np.exp(u) * erfc(v)
    def _cdf(self, x, mu, sig, lam):
        u = lam * (x - mu)
        v = lam * sig
        phi1 = st.norm.cdf(u, loc=0, scale=v)
        phi2 = st.norm.cdf(u, loc=v**2, scale=v)
        return phi1 - phi2 * np.exp(-u + 0.5 * v**2)
    def _stats(self, mu, sig, lam):
        # reutrn the mean, variance, skewness, and kurtosis
        mean = mu + 1 / lam
        var = sig**2 + 1 / lam**2
        sl = sig * lam
        u = 1 + 1 / sl**2
        skew = (2 / sl**3) * u**(-3 / 2)
        v = 3 * (1 + 2 / sl**2 + 3 / sl**4) / u**2
        kurt = v - 3
        return mean, var, skew, kurt
    def _argcheck(self, mu, sig, lam):
        return np.isfinite(mu) and (sig > 0) and (lam > 0)

class EMG_gen2(EMG_gen1):
    def _ppf(self, q, mu, sig, lam):
        # use linear interpolation to solve this faster (not exact, but much faster than the built in method)
        # pick range large enough to fit the full cdf
        var = sig**2 + 1 / lam**2
        x = np.arange(mu - 50 * np.sqrt(var), mu + 50 * np.sqrt(var), 0.01)
        y = self.cdf(x, mu, sig, lam)
        return np.interp(q, y, x)

class EMG_gen3(EMG_gen1):
    def _rvs(self, mu, sig, lam):
        # redefine the random sampler to sample based on a normal and exp dist
        return st.norm.rvs(loc=mu, scale=sig, size=self._size) + st.expon.rvs(loc=0, scale=1/lam, size=self._size)

# use generator to make the new class
EMG1 = EMG_gen1(name='EMG1')
EMG2 = EMG_gen2(name='EMG2')
EMG3 = EMG_gen3(name='EMG3')
```

Lets look at how long it takes to create readom samples for each of these version of the EMG:

```python
%time EMG1.rvs(0, 1, 0.5, size=1000)
print('=========')
%time EMG2.rvs(0, 1, 0.5, size=1000)
print('=========')
%time EMG3.rvs(0, 1, 0.5, size=1000)
print('=========')
```

As you can see, the numerical inversion of the CDF is very slow, the approximation to the inversion is much faster, and defining `_rvs` in terms of the `normal` and `exp` distributions is the fastest.

Lets take a look at the results for `EMG3`:

```python
dist = EMG3(0, 1, 0.5)
x = np.linspace(-5, 20, 1000)
plt.figure(2, figsize=(8, 10))
plt.subplot2grid((2, 2), (0, 0))
plt.plot(x, dist.pdf(x))
plt.xlabel('x')
plt.ylabel('PDF(x)')
plt.subplot2grid((2, 2), (0, 1))
plt.plot(x, dist.cdf(x))
plt.xlabel('x')
plt.ylabel('CDF(x)')
plt.subplot2grid((2, 2), (1, 0))
sample_emg = dist.rvs(size=10000)
hist(sample_emg, bins='knuth', histtype='step', lw=1.5, density=True)
plt.xlabel('x')
plt.ylabel('Random Sample')
plt.tight_layout()
```

As with the built in functions we can calculate moments and do fits to data.  **Note** Since we are not using the built in `loc` and `scale` params they are fixed to 0 and 1 in the fit below.

```python
for i in range(4):
    print('moment {0}: {1}'.format(i+1, dist.moment(i+1)))

print('best fit: {0}'.format(EMG3.fit(sample_emg, floc=0, fscale=1)))
```

For reference here is how `scipy` defines this distriubtion (found under the name `exponnorm`):

```python
import scipy.stats._continuous_distns as cd
np.source(cd.exponnorm_gen)
```

```python
%time st.exponnorm.rvs(0.5, size=1000)
```

```python

```


```python
import sympy as sm
import sympy.physics.mechanics as me
me.init_vprinting()
```

```python
m1, m2, m3, m4 = sm.symbols('m1, m2, m3, m4')
```

```python
N = me.ReferenceFrame('N')
```

```python
r1 = 3*N.x + 4*N.y
r2 = 3*N.y + 4*N.z
r3 = 4*N.x + 3*N.z
r4 = 4*N.y + 3*N.z
```

```python
first_moment = m1*r1 + m2*r2 + m3*r3 + m4*r4
```

```python
zeroth_moment = m1 + m2 + m3 + m4
```

```python
mass_center = first_moment/zeroth_moment
mass_center
```

```python
Ix = (
m1*me.cross(r1, me.cross(N.x, r1)) +
m2*me.cross(r2, me.cross(N.x, r2)) +
m3*me.cross(r3, me.cross(N.x, r3)) +
m4*me.cross(r4, me.cross(N.x, r4))
)
Ix
```

```python
na = (N.x + N.y + N.z).normalize()
```

```python
Ia = (
m1*me.cross(r1, me.cross(na, r1)) +
m2*me.cross(r2, me.cross(na, r2)) +
m3*me.cross(r3, me.cross(na, r3)) +
m4*me.cross(r4, me.cross(na, r4))
)
Ia
```

```python
sm.simplify(Ia.dot(na))
```

```python
Ia.dot(N.x)
```

```python
r_r = -3*N.y + 4*N.x
r_l =  3*N.y + 4*N.x

m = sm.symbols('m')

Ix = (
    m*me.cross(r_r, me.cross(N.x, r_r)) +
    m*me.cross(r_l, me.cross(N.x, r_l))
)
Ix
```

```python
m*me.cross(r_r, me.cross(N.x, r_r))
```

```python
m*me.cross(r_l, me.cross(N.x, r_l))

```

```python
Ix.dot(N.y)
```

```python

```


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.5.1 (Python 3.6.10-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.5.1/

```python
#init_printing?
```

### Parameters and two Gaussians

```python
a, b, c, a1, a2 = symbols('a b c a1 a2', positive=True, real=True)
```

```python
g1=exp(-a1*x**2)
g2=exp(-a2*x**2)
g1, g2
```

### Normalization constant

```python
N=integrate(g1*g1, (x, -oo, oo))
N
```

```python
1/sqrt(N)
```

```python
printing.sstrrepr(1/sqrt(N))
```

    '2**(1/4)*a1**(1/4)/pi**(1/4)'

### Overlap integral S

```python
S=integrate(g1*g2, (x, -oo, oo))
S
```

```python
S.simplify()
```

```python
printing.sstrrepr(S.simplify())
```

    'sqrt(pi)/sqrt(a1 + a2)'

### Kinetic energy $T = -\frac{\hbar^2}{2m} \frac{d^2}{dx^2} = \frac{1}{2m}\left(\frac{\hbar}{i}\frac{d}{dx} \right)^2$

```python
d1=diff(g1,x)
d2=diff(g2,x)
d1, d2
```

```python
T = 1/2 * integrate(d1*d2, (x, -oo, oo))
#T=T.simplify()
#T=T.factor()
T.factor()
```

```python
printing.sstrrepr(T.factor())
```

    '1.0*sqrt(pi)*a1*a2/(a1 + a2)**(3/2)'

### Potential $V(x) = (ax^2 - b)e^{-cx^2}$

```python
v=(a*x**2-b)*exp(-c*x**2)
v
```

```python
V = integrate(g1*v*g2, (x, -oo, oo))
V
```

```python
V.factor()
```

```python
printing.sstrrepr(V.factor())
```

    'sqrt(pi)*(a - 2*a1*b - 2*a2*b - 2*b*c)/(2*(a1 + a2 + c)**(3/2))'

```python

```


# Content:
1. [Simple example](#1.-Simple-example)
2. [Parametric equations](#2.-Parametric-equations)
3. [Polishing the plot](#3.-Polishing-the-plot)
4. [Contour plot](#4.-Contour-plot)
5. [Beginner-level animation](#5.-Beginner-level-animation)
6. [Intermediate-level animation](#6.-Intermediate-level-animation)

## 1. Simple example

```python
import numpy as np
import matplotlib.pyplot as plt

#=== x-range
x_min=-10
x_max=10
x_grids=501
x=np.linspace(x_min, x_max, x_grids)
#print(x.shape[0])

#=== variables to plot
y=x**2

#=== plot
plt.plot(x,y)
plt.title('Parabola')
plt.show()
```

You can change the default style such as line color, line width, etc. Bookmark [Matplotlib documentation](https://matplotlib.org/2.1.1/api/_as_gen/matplotlib.pyplot.plot.html) and refer this page for more details.

```python
#=== plot
plt.plot(x,y,color='r',linestyle='dashed',linewidth=4)
plt.title('Parabola')
plt.show()
```

## 2. Parametric equations

[Lissajous curves](https://en.wikipedia.org/wiki/Lissajous_curve) are defined by the parametric equations

$$
\begin{align}
x & = A\sin(at+\pi/2)\\
y & =  Bsin(bt)
\end{align}
$$

```python
import numpy as np
import matplotlib.pyplot as plt

#=== t-range
t_min=-np.pi
t_max=np.pi
t_grids=501
t=np.linspace(t_min, t_max, t_grids)

#=== Constants
A=1
B=1
a=10
b=12

#=== variables to plot
x=A*np.sin(a*t+np.pi/2)
y=B*np.sin(b*t)

#=== plot
plt.plot(x,y)
plt.title('Lissajous curves')
plt.grid()
plt.show()

#=== To see the current working directory, uncomment the following 2 lines
#import os
#os.getcwd()
```

## 3. Polishing the plot

```python
import numpy as np
import matplotlib.pyplot as plt

#=== t-range
t_min=-np.pi
t_max=np.pi
t_grids=501
t=np.linspace(t_min, t_max, t_grids)

#=== Constants
A=1
B=1
a=10
b=9

#=== variables to plot
x=A*np.sin(a*t+np.pi/2)
y=B*np.sin(b*t)

#=== make it a square plot
fig = plt.figure()                        # comment if square plot is not needed
ax = fig.add_subplot(111)                 # comment if square plot is not needed
plt.plot(x,y)
ax.set_aspect('equal', adjustable='box')  # comment if square plot is not needed

#=== labels, titles, grids
plt.xlabel("x")
plt.ylabel("y")
plt.title('Lissajous curves')
plt.grid()

#=== save in file, you can also use .pdf or .svg
plt.savefig('Lissajous.png')

#=== display
plt.show()
```

## 4. Contour plot

```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

#=== Define a 2D function
def f(x,y):
    #z=(x**2+y**2)
    z=(1-x**2+y**2)
    return z

x=np.arange(-2.0,2.0,0.1)
y=np.arange(-2.0,2.0,0.1)

X,Y=np.meshgrid(x,y)

Z=f(X,Y)

N_iso=np.arange(-2,2,0.5)

CS=plt.contour(Z,N_iso,linewidths=2,cmap=mpl.cm.jet)
plt.clabel(CS, inline=True, fmt='%1.1f', fontsize=10)
plt.colorbar(CS)
plt.title('A contour plot')
plt.show()
```

## 5. Beginner-level animation

```python
import os
import numpy as np
import matplotlib.pyplot as plt
import imageio
```

```python
x_min=-10
x_max=10
x_grids=501
x=np.linspace(x_min, x_max, x_grids)## ONE ##

def gauss(x,x0):
    f=np.exp(-(x-x0)**2)
    return f

#=== plot-1
x0=0
y=gauss(x,x0)
plt.plot(x,y)
plt.title('Moving Gaussian')
plt.savefig('_tmp_1.png')
plt.show()
#=== plot-2
x0=1
y=gauss(x,x0)
plt.plot(x,y)
plt.title('Moving Gaussian')
plt.savefig('_tmp_2.png')
plt.show()
#=== plot-3
x0=2
y=gauss(x,x0)
plt.plot(x,y)
plt.title('Moving Gaussian')
plt.savefig('_tmp_3.png')
plt.show()
#=== plot-4
x0=3
y=gauss(x,x0)
plt.plot(x,y)
plt.title('Moving Gaussian')
plt.savefig('_tmp_4.png')
plt.show()
```

```python
# Build GIF
with imageio.get_writer('mygif1.gif', mode='I') as writer:
    for filename in ['_tmp_1.png', '_tmp_2.png', '_tmp_3.png', '_tmp_4.png']:
        image = imageio.imread(filename)
        writer.append_data(image)
```

Double click the next image and then shift+enter

## 6. Intermediate-level animation

```python
import numpy as np
import matplotlib.pyplot as plt

#=== Particle-in-a-box solutions
hbar=1
mass=1
L = 1

#=== Eigenvalues
n=1
E1=n**2 * np.pi**2 * hbar**2 / (2.0 * mass * L**2)
n=2
E2=n**2 * np.pi**2 * hbar**2 / (2.0 * mass * L**2)

x=np.linspace(0,1,101)

def psi1(x):  # Ground state, n = 1
    n=1
    L=1
    val=np.sqrt(2.0/L)*np.sin(n*np.pi*x/L)
    return val

def psi2(x):  # First excited state, n = 2
    n=2
    L=1
    val=np.sqrt(2.0/L)*np.sin(n*np.pi*x/L)
    return val

c1=1.0/np.sqrt(2.0)
c2=1.0/np.sqrt(2.0)

filenames = []
t=0
it=0
dt=0.01
i=complex(0,1)

dx=0.1
#print(np.sqrt(np.dot(psi1(x)*dx,psi1(x)*dx))) # uncomment to check for Normalization

while it <= 100:
    psi=c1*psi1(x)*dx*np.exp(-i*E1*t/hbar) + c2*psi2(x)*dx*np.exp(-i*E2*t/hbar)

    plt.plot(x,np.real(psi)**2+np.imag(psi)**2)

    plt.xlim(0, 1)
    plt.ylim(0, 0.2)
    plt.xlabel("x")
    plt.ylabel("$|\psi|^2$(x)")   #NOTE: LaTex syntax for psi
    plt.title('Time evolution of $[\psi_1+\psi_2]/\sqrt{2}$')

    plt.text(0.2,0.15, r'$time=$ {0:10.3f} [au]'.format(t), fontsize=10)

    filename='_tmp_'+str(it).zfill(5)+'.png'
    filenames.append(filename)
    plt.savefig(filename)

    plt.close()
    t=t+dt
    it=it+1

# build gif
with imageio.get_writer('mygif2.gif', mode='I') as writer:
    for filename in filenames:
        image = imageio.imread(filename)
        writer.append_data(image)

# Remove files
for filename in set(filenames):
    os.remove(filename)
```

Double click the next image and then shift+enter