# Introduction to Partial Differential Equations
## (PDEs)

-----
Questions:
- Which physical systems can be described using a PDE?
- What is the Laplacian operator?
- When to I need boundary conditions and initial conditions?

------

------
Objectives:
- Recognise common classes of PDE: the diffusion equation, Poisson's equation and the wave equation
- Express the Laplacian as a differential operator
- Identify boundary value problems and initial value problems

-----

- In the previous section of the course we studied <bold>ordinary differential equations</bold>. ODEs have a single input (also known as independent variable) - for example, time.

- Partial differential equations (PDEs) have multiple inputs (independent variables). For example, think about a sheet of metal that has been heated unevenly across the surface. Over time, heat will diffuse through the 2-dimensional sheet. The temperature depends on both time *and* position - there are two inputs.

- Because PDEs have multiple inputs they are generally much more difficult to solve analytically than ODEs. However, there are a range of numerical methods that can be used to find approximate solutions.

### PDEs have application across a wide variety of topics

The same type of PDE often appears in different contexts. For example, the diffusion equation takes the form:

\begin{equation}
\nabla^2T = \alpha \frac{\partial T}{\partial t}
\end{equation}

When used to describe heat diffusion, this PDE is known as the heat equation. This same PDE however can be used to model other seemingly unrelated processes such as brownian motion, or used in financial modelling via the Black-Sholes equation.

Another type of PDE is known as Poisson's equation:

\begin{equation}
\nabla^2\phi = f(x,y,z)
\end{equation}

Poisson's equation can be used to describe electrostatic forces, where $\phi$ is the electric potential. It can also be applied to mechanics (where $\phi$ is the gravitational potential) or thermodynamics (where $\phi$ is the temperature). When $f(x,y,z)=0$ this equation is known as Laplace's equation.

The third common type of PDE is the wave equation:

\begin{equation}
\nabla^2r = \alpha \frac{\partial^2 r}{\partial t^2}
\end{equation}

This describes mechanical processes such as the vibration of a string or the motion of a pendulum. It can also be used in electrodynamics to describe the exchange of energy between the electric and magnetic fields.

In this course we will look at techniques for solving the diffusion equation and Poisson's equation, but many of the topics we will discuss - such as boundary conditions, and finite difference methods - can be transferred to PDEs more generally.

### The Laplacian operator corresponds to an average rate of change

*But what is the operator $\nabla^2$?*. This is the Laplacian operator. When applied to $\phi$ and written in full for a three dimensional cartesian coordinate system with dependent variables $x$, $y$ and $z$ it takes the following form:

\begin{equation}
\nabla^2\phi = \frac{\partial^2\phi}{\partial x^2} + \frac{\partial^2\phi}{\partial y^2} + \frac{\partial^2\phi}{\partial z^2}.
\end{equation}

We can think of the laplacian as encoding an average rate of change - the difference between a value at a point and the average of the values around that point.

```python
# https://gist.github.com/dm-wyncode/55823165c104717ca49863fc526d1354
"""Embed a YouTube video via its embed url into a notebook."""
from functools import partial

from IPython.display import display, IFrame

width, height = (560, 315, )

def _iframe_attrs(embed_url):
    """Get IFrame args."""
    return (
        ('src', 'width', 'height'),
        (embed_url, width, height, ),
    )

def _get_args(embed_url):
    """Get args for type to create a class."""
    iframe = dict(zip(*_iframe_attrs(embed_url)))
    attrs = {
        'display': partial(display, IFrame(**iframe)),
    }
    return ('YouTubeVideo', (object, ), attrs, )

def youtube_video(embed_url):
    """Embed YouTube video into a notebook.

    Place this module into the same directory as the notebook.

    >>> from embed import youtube_video
    >>> youtube_video(url).display()
    """
    YouTubeVideo = type(*_get_args(embed_url)) # make a class
    return YouTubeVideo() # return an object
```

```python
youtube_video("https://youtu.be/EW08rD-GFh0").display()
```

### Boundary value problems

Boundary value problems describe the behaviour of a variable in a space and we are given some constraints on the variable around the boundary of that space. For example, consider the 2-dimensional problem of a thin rectangular sheet with one side at voltage $V$ and all others at voltage zero.

The specification that one side is at voltage $V$ and all others are at voltage zero are the boundary conditions. We could then calculate the electrostatic potential $\phi$ at all points within the sheet using the two-dimensional Laplace's equation:

\begin{equation}
\nabla^2\phi = \frac{\partial^2\phi}{\partial x^2} + \frac{\partial^2\phi}{\partial y^2} = 0
\end{equation}

#### Initial value problems

Initial value problems are where the field - or other variable of interest - is varying in both space and time. We now require boundary conditions *and* initial values. This is a more difficult type of PDE to solve.

For example, consider heat diffusion in a two-dimensional sheet. Here we could specify that there is no heat flow in or out of the sheet - this is the boundary condition.

We could also specify that at time $t=0$ the centre of the sheet is at temperature $T_1$, whilst surrounding areas are at temperature  $T_0$. This is the initial condition. It differs from a boundary condition in that we are told what the temperature is at the start of our time grid (at $t=0$) but not at the end of our time grid (when the simulation finishes).

We could then calculate the temperature at time $t$ at all points $[x,y]$ within the sheet using the two-dimensional Diffusion equation:

\begin{equation}
\nabla^2T = \frac{\partial^2 T}{\partial x^2} + \frac{\partial^2 T}{\partial y^2}=\alpha \frac{\partial T}{\partial t}
\end{equation}

# Laplace's equation for electrostatics

-----
Questions:
- How do I use a finite difference method to calculate derivatives?
- How do I use the relaxation method to solve Laplace's equation?

------

### The method of finite differences

Consider the two-dimensional Laplace equation for the electric potential $\phi$ subject to appropriate boundary conditions:

\begin{equation}
\frac{\partial^2\phi}{\partial x^2} + \frac{\partial^2\phi}{\partial y^2} = 0
\end{equation}

The method of finite differences involves dividing the space into a grid of discrete points $[x,y]$ and calculating numerical derivatives or at each of these points.

 But how do we calculate these numerical derivatives?

Real physical problems are in three dimensions, but we can more easily visualise the method of finite differences - and the extension to three dimensions is straight forward.

### Calculating numerical derivatives

The standard definition of a derivative is

\begin{equation}
\frac{\mathrm{d} f}{\mathrm{d} x} = \lim_{h\to0}\frac{f(x+h)-f(x)}{h}.
\end{equation}

To calculate the derivative numerically we make $h$ very small and calculate

\begin{equation}
\frac{\mathrm{d} f}{\mathrm{d} x} \simeq \frac{f(x+h)-f(x)}{h}.
\end{equation}

This is the forward difference because it is measured in the forward direction from $x$.

The backward difference is measured in the backward direction from $x$:

\begin{equation}
\frac{\mathrm{d} f}{\mathrm{d} x} \simeq \frac{f(x)-f(x-h)}{h},
\end{equation}

and the central difference uses both the forwards and backwards directions around $x$:

\begin{equation}
\frac{\mathrm{d} f}{\mathrm{d} x} \simeq \frac{f(x+\frac{h}{h2})-f(x-\frac{h}{2})}{h},
\end{equation}

### Numerical second-order derivatives

The second derivative is a derivative of a derivative, and so we can calculate it be applying the first derivative formulas twice. The resulting expression (after application of central differences) is:

\begin{equation}
\frac{\mathrm{d} ^2f}{\mathrm{d} x^2} \simeq \frac{f(x+h)-2f(x)+f(x-h)}{h^2}.
\end{equation}

### Numerical partial derivatives

The extension to partial derivatives is straight-forward:

\begin{equation}
\frac{\mathrm{d} f}{\mathrm{d} x} \simeq \frac{f(x+\frac{h}{h2})-f(x-\frac{h}{2})}{h},
\end{equation}

\begin{equation}
\frac{\partial f}{\partial x} \simeq \frac{f(x+\frac{h}{2},y)-f(x-\frac{h}{2},y)}{h},
\end{equation}

\begin{equation}
\frac{\partial f}{\partial y} \simeq \frac{f(x,y+\frac{h}{2})-f(x,y-\frac{h}{2})}{h},
\end{equation}

\begin{equation}
\frac{\mathrm{d} ^2f}{\mathrm{d} x^2} \simeq \frac{f(x+h)-2f(x)+f(x-h)}{h^2}.
\end{equation}

\begin{equation}
\frac{\partial ^2f}{\partial x^2} \simeq \frac{f(x+h,y)-2f(x,y)+f(x-h,y)}{h^2},
\end{equation}

\begin{equation}
\frac{\partial ^2f}{\partial y^2} \simeq \frac{f(x,y+h)-2f(x,y)+f(x,y-h)}{h^2}.
\end{equation}

By adding the two equations above, the Laplacian in two dimensions is:

\begin{equation}
\frac{\partial ^2f}{\partial x^2} + \frac{\partial ^2f}{\partial y^2} \simeq \frac{f(x+h,y)+f(x-h,y)+f(x,y+h)+f(x,y-h)-4f(x,y)}{h^2},
\end{equation}

### PDEs --> linear simulatenous equations

Returning to our Laplace equation for for the electric potential $\phi$:

\begin{equation}
\frac{\partial^2\phi}{\partial x^2} + \frac{\partial^2\phi}{\partial y^2} = 0
\end{equation}

The numerical Laplacian can be substituted into the equation above, giving us a set of $n$ simulatenous equations for the $n$ grid points.

\begin{equation}
 \frac{\phi(x+h,y)+\phi(x-h,y)+\phi(x,y+h)+\phi(x,y-h)-4\phi(x,y)}{a^2} = 0,
\end{equation}

where $a$ is the distance between each grid point.

### To solve we use the relaxation method

\begin{equation}
 \frac{\phi(x+h,y)+\phi(x-h,y)+\phi(x,y+h)+\phi(x,y-h)-4\phi(x,y)}{a^2} = 0,
\end{equation}

\begin{equation}
 \phi(x,y)=\frac{1}{4}\left(\phi(x+h,y)+\phi(x-h,y)+\phi(x,y+h)+\phi(x,y-h)\right).
\end{equation}

This tells us that $\phi(x,y)$ is the average of the surrounding grid points, which can be represented visually as:

### To solve we use the relaxation method

#### Step one
Fix $\phi(x,y)$ at the boundaries using the boundary conditions.

#### Step two
Guess the initial values of the interior $\phi(x,y)$ points - our guesses do not need to be good, and can be zero.

#### Step three
Calculate new values of $\phi'(x,y)$ at all points in space using an iterative method:
\begin{equation}
 \phi'(x,y)=\frac{1}{4}\left(\phi(x+h,y)+\phi(x-h,y)+\phi(x,y+h)+\phi(x,y-h)\right).
\end{equation}

#### Step four
Repeat until the $\phi(x,y)$ values converge*, and that is our solution.

*Convergence can be tested by specifying what the maximum difference should be between iterations. For example, that $\phi'(x,y)-\phi(x,y)< 1e-5$ for all grid points.

# Heat Diffusion

----

Questions:

- How do I use the Forward-Time Centred-Space method (FTCS) to solve the diffusion equation?

---

### The diffusion equation is an initial value problem

An initial value problem is more complex than a boundary value problem as we are told the starting conditions and then have to predict future behaviour as a function of time.

#### Example

A 10cm rod of stainless steel initially at a uniform temperature of 20$^\mathrm{o}$ Celsius. The rod is dipped in a hot water bath at 90$^\mathrm{o}$ Celsius at one end, and held in someone's hand at the other. Assume that the hand is at constant body temperature throughout (27$^\mathrm{o}$ Celsius).

Assume:
- that the rod is perfectly insulated so that heat only moves horizontally --> a 1D problem
- neither the hot water bath or the hand change temperature

Thermal conduction is described by the diffusion equation:

\begin{equation}
 \frac{\partial \phi}{\partial t} = D\frac{\partial^2 \phi}{\partial x^2},
\end{equation}

where $D$ is the material dependent thermal diffusivity. For steel $D=4.25\times10^{-6}\mathrm{m}^2\mathrm{s}^{-1}$.

### Why can't we use the relaxation method?

We solved Laplace's equation and that had three variables ($x$,$y$,$z$) - why not do the same thing here?

The problem is that we  only have an *initial* condition in the time dimension - we know the value of $\phi(x,t)$ at $t=0$ but we do not typically know the value of $t$ at a later point. In the spatial dimensions we know the boundary conditions at either end of the grid.

Instead, we will use the Forward-Time Centred-Space method (FTCS).

### There are two steps to the Forward-Time Centred-Space method

#### Step one
Use the finite difference method to express the 1D Laplacian as a set of simulatenous equations:

\begin{equation}
\frac{\partial^2\phi}{\partial x^2} = \frac{\phi(x+a,t)+\phi(x-a,t) - 2\phi(x,t)}{a^2}
\end{equation}

where $a$ is the grid spacing.

Substitute this back into the diffusion equation:

\begin{equation}
 \frac{d \phi}{d t} = \frac{D}{a^2}(\phi(x+a,t)+\phi(x-a,t)-2\phi(x,t))
\end{equation}

We now have a set of simultaneous ODEs for $\phi(x,t)$.

#### Step two
So we can use Euler's method to evolve the system forward in time. Euler's method for solving an ODE of the form $\frac{d\phi}{dt} = f(\phi,t)$ has the general form:

\begin{equation}
\phi(t+h) \simeq \phi(t) + hf(\phi, t).
\end{equation}

Applying this to Equation 3 gives:

\begin{equation}
\phi(x,t+h) = \phi(x,t) + h\frac{D}{a^2}(\phi(x+a,t)+\phi(x-a,t)-2\phi(x,t))
\end{equation}

# Evaluating numerical errors and accuracy

-----
Questions:

- Which numerical errors are unavoidable in a Python programme?
- How do I choose the optimum step size $h$ when using the finite difference method?
- What do the terms first-order accurate and second-order accurate mean?
- How can I measure the speed of my code?
-----

###  Finite difference methods have two sources of error

- There are two sources of errors for finite difference methods:
    - the approximation that the step size $h$ is small but not zero.
    - the numerical rounding errors for floating point numbers
- If we decrease the step size $h$:
    - the finite-difference approximation will improve
    - the runtime of the programme will increase
    - counter-intuitively, the rounding error might *increase*.
- So it is possible that by decreasing $h$ we make our programme *less* accurate *and* it takes longer to run!

To demonstrate this, consider the Taylor expansion of $f(x)$ about $x$:

\begin{equation}
f(x+h) = f(x) + hf'(x) +\frac{1}{2}h^2f''(x) + \ldots
\end{equation}

Re-arrange the expression to get the expression for the forward difference method:

\begin{equation}
f'(x) = \frac{f(x+h)}{h} - \frac{1}{2}hf''(x)+\ldots
\end{equation}

A computer can typically store a number $f(x)$ to an accuracy of 16 significant figures, or $Cf(x)$ where $C=10^{-16}$. In the worst case, this makes the error $\epsilon$ on our derivative:

\begin{equation}
\epsilon = \frac{2C|f(x)|}{h} + \frac{1}{2}h|f''(x)|.
\end{equation}

We want to find the value of $h$ which minimises this error so we differentiate with respect to $h$ and set the result equal to zero.

\begin{equation}
-\frac{2C|f(x)|}{h^2} + h|f''(x)| = 0
\end{equation}

\begin{equation}
h = \sqrt{4C\lvert\frac{f(x)}{f''(x)}\rvert}
\end{equation}

If $f(x)$ and $f''(x)$ are order 1, then $h$ should be order $\sqrt{C}$, or $10^{-8}$.

Similar reasoning applied to the central difference formula suggests that the optimum step size for this method is $10^{-5}$.


$$
\newcommand{\wave}{{\cal W}}
\newcommand{\amdq}{{\cal A}^-\Delta Q}
\newcommand{\apdq}{{\cal A}^+\Delta Q}
$$

# An approximate solver for Burgers' equation

As a first example, we return to the inviscid Burgers' equation that we studied in [Burgers_equation.ipynb](Burgers_equation.ipynb):
\begin{align} \label{burgers}
q_t + \left(\frac{1}{2}q^2\right)_x & = 0.
\end{align}
Although it is easy to solve the Riemann problem for (\ref{burgers}) exactly, it is nevertheless interesting to consider approximate solvers because a numerical scheme does not make use of the full Riemann solution.  Furthermore, Burgers' equation provides a simple setting in which some of the intricacies of more complicated approximate solvers can be introduced.  Recall that we are interested in **approximate solutions that consist entirely of traveling discontinuities**.

## Shock wave solutions

Recall that the exact Riemann solution for (\ref{burgers}) consists of a single shock or rarefaction wave.  We have a shock if $q_l>q_r$ and we have a rarefaction if $q_l < q_r$.  In the case of a shock wave, we can simply use the exact solution as our approximation.  We have a single wave of strength $q_r-q_l$ traveling at speed $s=(q_r+q_l)/2$.

In terms of fluxes, the numerical flux is $F=f(q_l)$ if $s>0$ and $F=f(q_r)$ if $s<0$.  In the special case $s=0$ we have a stationary shock, and it must be that $f(q_l)=f(q_r) (=F)$.

## Rarefaction wave solutions

As discussed in [Approximate_solvers.ipynb](Approximate_solvers.ipynb), for numerical purposes it is convenient to approximate a rarefaction wave by a traveling discontinuity.  For Burgers' equation this may seem unnecessary, but for more complicated solvers for systems of equations it will be essential.

We will approximate the effect of the rarefaction wave by a fictitious shock:
$$\wave = q_r-q_l$$
whose speed is given by the Rankine-Hugoniot jump condition:
$$s = \frac{f(q_r)-f(q_l)}{q_r-q_l} = \frac{q_r + q_l}{2}.$$
Recall that this is indeed a weak solution of the Riemann problem.  This fictitious shock is not entropy-satisfying, but as long as it's traveling entirely to the left or entirely to the right, the effect on the numerical solution will be the same as if we used a (entropy-satisfying) rarefaction wave.  The numerical flux is again $F=f(q_l)$ if $s>0$ and $F=f(q_r)$ if $s<0$.

Because this is a scalar equation with convex flux, both the Roe and HLL approaches will simplify to what we have already described.  But we briefly discuss them here to further illustrate the main ideas.

## A Roe solver

Let us consider a linearized solver, in which we replace our nonlinear hyperbolic system with a linearization about some intermediate state $\hat{q}$.  For Burgers' equation, the quasilinear form is $q_t + q q_x = 0$ and the linearization gives
$$q_t + \hat{q}q_x = 0.$$
This is simply the advection equation with velocity $\hat{q}$.  The solution of the Riemann problem for this equation consists of a wave $\wave = q_r - q_l$ traveling at speed $\hat{q}$.  It remains only to determine the state $\hat{q}$, and thus the wave speed.

The defining feature of a Roe linearization is that it gives the exact solution in case the states $(q_r, q_l)$ lie on a single Hugoniot locus -- i.e., when the solution is a single shock.  We can achieve this by choosing
$$\hat{q} = \frac{q_r + q_l}{2}.$$
This is equivalent to using the approximate solver described already in the sections above.

### Examples
Below we show solutions for three examples; the first involves a shock, the second a (non-transonic) rarefaction, and the third a transonic rarefaction.  In the first row we plot the exact solution in terms of the waves in the x-t plane; in the second row we plot numerical solutions obtained by using a simple first-order method combined with the Riemann solver.

```python
%matplotlib inline
```

```python
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
from ipywidgets import interact
from ipywidgets import widgets
from exact_solvers import burgers
from utils import riemann_tools
```

```python
def setup(q_l, q_r, N=500, efix=False, solver_type='classic', order=1, lang='Python'):

    from clawpack import pyclaw
    from clawpack import riemann

    if lang == 'Python':
        rs = riemann.burgers_1D_py.burgers_1D
    else:
        rs = riemann.burgers_1D

    if solver_type == 'classic':
        solver = pyclaw.ClawSolver1D(rs)
        solver.order = order
    else:
        solver = pyclaw.SharpClawSolver1D(rs)

    solver.kernel_language = lang

    solver.bc_lower[0]=pyclaw.BC.extrap
    solver.bc_upper[0]=pyclaw.BC.extrap

    x = pyclaw.Dimension(-1.0,1.0,N,name='x')
    domain = pyclaw.Domain([x])
    state = pyclaw.State(domain,1)

    state.problem_data['efix'] = efix

    xc = state.grid.p_centers[0]

    state.q[0 ,:] = (xc<=0)*q_l + (xc>0)*q_r

    claw = pyclaw.Controller()
    claw.tfinal = 0.5
    claw.solution = pyclaw.Solution(state,domain)
    claw.solver = solver
    claw.num_output_times = 10
    claw.keep_copy = True
    claw.verbosity=0

    return claw
```

```python
shock = setup(2.,1.)
shock.run()
shocksol = burgers.exact_riemann_solution(2.,1.)

raref = setup(1.,2.)
raref.run()
rarefsol = burgers.exact_riemann_solution(1.,2.)

transonic = setup(-1.,2.)
transonic.run()
transonicsol = burgers.exact_riemann_solution(-1.,2.)

def plot_frame(i):
    fig, axes = plt.subplots(2,3,figsize=(8,4))
    for ax in axes[0]:
        ax.set_xlim((-1,1)); ax.set_ylim((-1.1,2.1))

    axes[1][0].plot(shock.frames[0].grid.x.centers, shock.frames[i].q[0,:],'-k',lw=2)
    axes[1][0].set_title('Shock')
    axes[1][1].plot(raref.frames[0].grid.x.centers, raref.frames[i].q[0,:],'-k',lw=2)
    axes[1][1].set_title('Rarefaction')
    axes[1][2].plot(transonic.frames[0].grid.x.centers, transonic.frames[i].q[0,:],'-k',lw=2)
    axes[1][2].set_title('Transonic rarefaction')
    t = i/10.
    riemann_tools.plot_waves(*shocksol,ax=axes[0][0],t=t)
    riemann_tools.plot_waves(*rarefsol,ax=axes[0][1],t=t)
    riemann_tools.plot_waves(*transonicsol,ax=axes[0][2],t=t)
    plt.tight_layout()
    plt.show()

interact(plot_frame, i=widgets.IntSlider(min=0, max=10, description='Frame'));
```

The solutions obtained for the shock wave and for the first rarefaction wave are good approximations of the true solution.  In the case of the transonic rarefaction, however, we see that part of the rarefaction has been replaced by an entropy-violating shock.  At the end of this chapter we will show how to apply an *entropy fix* so that the solver gives a good approximation also in the transonic case.

## Two-wave solvers

For Burgers' equation, the Riemann solution consists only of a single wave.  It is thus natural to modify the HLL approach by assuming that one of the waves vanishes, and denote the speed of the other wave simply by $s$.  Then the conservation condition discussed in [Approximate_solvers.ipynb](Approximate_solvers.ipynb#Two-wave-solvers) reduces to
$$f(q_r) - f(q_l) = s (q_r - q_l),$$
which is just the Rankine-Hugoniot condition and again leads to the speed discussed above.  Since the solution involves only one wave, that wave must carry the entire jump $q_r - q_l$, so this solver is entirely equivalent to that already described.

It is also possible to follow the full HLL approach, taking
\begin{align*}
s_1 & = \min f'(q) = \min(q_l, q_r) \\
s_2 & = \max f'(q) = \max(q_l, q_r).
\end{align*}
Regardless of the values of $q_l$ and $q_r$, this leads to
$$q_m = \frac{q_r + q_l}{2},$$
so that each of the two waves carries half of the jump.

*DK: Has anybody ever tried this solver?*

## Transonic rarefactions

In the approaches above, the solution was approximated by a single wave traveling either to the left or right.  For this scalar problem, this "approximation" is, in fact, an exact weak solution of the Riemann problem.  As discussed already, we do not typically need to worry about the fact that it may be entropy-violating, since the effect on the numerical solution (after averaging) is identical to that of the entropy-satisfying solution.

However, if $q_l < 0 < q_r$, then the true solution is a transonic rarefaction, in which part of the wave travels to the left and part travels to the right.  In this case, the true Riemann solution would lead to changes to both the left and right adjacent cells, whereas an approximate solution with a single wave will only modify one or the other.  This leads to an incorrect numerical solution (on the macroscopic level).  It is therefore necessary to modify the Riemann solver, imposing what is known as an *entropy fix* in the transonic case.

Specifically, we use a solution consisting of two waves, each of which captures the net effect of the corresponding rarefaction wave that appears in the exact solution:

\begin{align}
\wave_1 & = q_m - q_l, & s_1 = \frac{q_l + q_m}{2} \\
\wave_2 & = q_r - q_m, & s_2 = \frac{q_m + q_r}{2}.
\end{align}

For Burgers' equation, the value $q_s=0$ is the *sonic point* satisfying $f(q_s)=0$. A transonic rarefaction wave takes the value $q_s$ along $x/t = 0$ and so it makes sense to choose $q_m = 0$.  The formulas above then simplify to

\begin{align}
\wave_1 & = - q_l, & s_1 = \frac{q_l}{2} \\
\wave_2 & = q_r, & s_2 = \frac{q_r}{2}.
\end{align}

Note that this can also be viewed as an HLL solver, although not with the usual choice of wave speeds.  Choosing the waves speeds $s^1=q_l/2$ and $s^2=q_r/2$ and then solving for $q_m$ by requiring conservation gives $q_m=0$.

We now repeat the example given above, but with the entropy fix applied.

```python
shock_efix = setup(2.,1.,efix=True)
shock_efix.run()
shocksol = burgers.exact_riemann_solution(2.,1.)

raref_efix = setup(1.,2.,efix=True)
raref_efix.run()
rarefsol = burgers.exact_riemann_solution(1.,2.)

transonic_efix = setup(-1.,2.,efix=True)
transonic_efix.run()
transonicsol = burgers.exact_riemann_solution(-1.,2.)

def plot_frame(i):
    fig, axes = plt.subplots(2,3,figsize=(8,4))
    for ax in axes[0]:
        ax.set_xlim((-1,1)); ax.set_ylim((-1.1,2.1))

    axes[1][0].plot(shock_efix.frames[0].grid.x.centers, shock_efix.frames[i].q[0,:],'-k',lw=2)
    axes[1][0].set_title('Shock')
    axes[1][1].plot(raref_efix.frames[0].grid.x.centers, raref_efix.frames[i].q[0,:],'-k',lw=2)
    axes[1][1].set_title('Rarefaction')
    axes[1][2].plot(transonic_efix.frames[0].grid.x.centers, transonic_efix.frames[i].q[0,:],'-k',lw=2)
    axes[1][2].set_title('Transonic rarefaction')
    t = i/10.
    riemann_tools.plot_waves(*shocksol,ax=axes[0][0],t=t)
    riemann_tools.plot_waves(*rarefsol,ax=axes[0][1],t=t)
    riemann_tools.plot_waves(*transonicsol,ax=axes[0][2],t=t)
    plt.tight_layout()
    plt.show()

interact(plot_frame, i=widgets.IntSlider(min=0, max=10, description='Frame'));
```

The entropy fix has no effect on the first two solutions, since it is applied only in the case of a transonic rarefaction.  The third solution is greatly improved, and will converge to the correct weak solution as the grid is refined.


Copyright (c) 2015, 2016 [Sebastian Raschka](sebastianraschka.com)
[Li-Yi Wei](http://www.liyiwei.org)

https://github.com/1iyiwei/pyml

[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)

# Python Machine Learning - Code Examples

# Chapter 13 - Parallelizing Neural Network Training with Theano

We have seen how to write a multi-layer perceptron from scratch.

We can also just use existing libraries.
* Theano, Torch, TensorFlow, Caffe, etc.

Advantages of Theano
* Python interface
* Platform support
* GPU support for performance

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,matplotlib,theano,keras
```

    Using Theano backend.

    last updated: 2016-11-25

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.1
    matplotlib 1.5.1
    theano 0.8.2
    keras 1.1.0

*The use of `watermark` is optional. You can install this IPython extension via "`pip install watermark`". For more information, please see: https://github.com/rasbt/watermark.*

### Overview

- [Building, compiling, and running expressions with Theano](#Building,-compiling,-and-running-expressions-with-Theano)
  - [What is Theano?](#What-is-Theano?)
  - [First steps with Theano](#First-steps-with-Theano)
  - [Configuring Theano](#Configuring-Theano)
  - [Working with array structures](#Working-with-array-structures)
  - [Wrapping things up – a linear regression example](#Wrapping-things-up:-A--linear-regression-example)
- [Choosing activation functions for feedforward neural networks](#Choosing-activation-functions-for-feedforward-neural-networks)
  - [Logistic function recap](#Logistic-function-recap)
  - [Estimating probabilities in multi-class classification via the softmax function](#Estimating-probabilities-in-multi-class-classification-via-the-softmax-function)
  - [Broadening the output spectrum by using a hyperbolic tangent](#Broadening-the-output-spectrum-by-using-a-hyperbolic-tangent)
- [Training neural networks efficiently using Keras](#Training-neural-networks-efficiently-using-Keras)
- [Summary](#Summary)

```python
from IPython.display import Image

%matplotlib inline
```

# Building, compiling, and running expressions with Theano

Developed by the LISA lab lead by Joshua Bengio started in 2008.

Harness multi/many-core CPU/GPU without the burden of
* parallel computing code
* memory management across processors

## What is Theano?

A machine learning library with interface in Python.
* with more speed/memory optimization

Focus on tensors as the core data structure.

Tensors are multi-dimensional arrays.
* rank 0 for scalars
* rank 1 for vectors
* rank 2 for matrices

Symbolic manipulation
* build computation graphs
* automatic and symbolic differentiation
* send compiled expressions/graphs to CPU/GPU for execution

## First steps with Theano

http://deeplearning.net/software/theano/introduction.html

Depending on your system setup, it is typically sufficient to install Theano via

    pip install Theano

For more help with the installation, please see: http://deeplearning.net/software/theano/install.html

Introducing the TensorType variables. For a complete list, see http://deeplearning.net/software/theano/library/tensor/basic.html#all-fully-typed-constructors

```python
import theano
from theano import tensor as T
import numpy as np
```

```python
# define expression
# which can be visualized as a graph
x1 = T.scalar()
w1 = T.scalar()
w0 = T.scalar()
z1 = w1 * x1 + w0

# compile
net_input = theano.function(inputs=[w1, x1, w0], outputs=z1)

# execute
answer = net_input(2.0, 1.0, 0.5)
print(answer)
answer
```

    2.5

    array(2.5, dtype=float32)

```python
# define
b = T.scalar('b')
x = T.vector('x')
W = T.matrix('W')

y = x.dot(W.transpose())
z = W.dot(x) + b

# similar to python function
# theano function can return multiple outputs
f = theano.function(inputs = [x, W, b], outputs = [y, z])

output_y, output_z = f([1, 2], [[3, 4], [5, 6]], 1)
# output_y, output_z = f([[1, 2]], [[3, 4]], 1) # won't work as x is a vector not matrix
# output_y, output_z = f([1, 2], [3, 4], 1) # won't work as W is a matrix not vector
# output_y, output_z = f([1, 2], [[3, 4]], [1]) # won't work as b is a scalar not a vector/matrix

print(output_y)
print(output_z)
```

    [ 11.  17.]
    [ 12.  18.]

```python
# quadratic polynomial root example
# ax^2 + bx + c = 0
a = T.scalar('a')
b = T.scalar('b')
c = T.scalar('c')

core = b*b - 4*a*c
root_p = (-b + np.sqrt(core))/(2*a)
root_m = (-b - np.sqrt(core))/(2*a)

# compile
f = theano.function(inputs = [a, b, c], outputs = [root_p, root_m])
```

```python
# run
polys = [[1, 2, 1],
         [1, -7, 12],
         [1, 0, 1]
        ]

for poly in polys:
    a, b, c = poly
    root1, root2 = f(a, b, c)
    print(root1, root2)
```

    -1.0 -1.0
    4.0 3.0
    nan nan

Steps for using Theano:
* define symbols and functions
* compile the code
* execute the code

Each variable has a specific type (dtype)
* trade-off between accuracy and cost (speed and storage)
* we have to choose; good for control, bad as burden

## Configuring Theano

Processors:
- Modern CPUs support 64-bit memory address.
- GPUs (and old CPUs) remain in 32-bit.

Theano supports both 32 and 64 bits.

We can configure Theano to use either: float32 (for 32-bit processors) or float64 (for 64-bit processors).

For more options, see
- http://deeplearning.net/software/theano/library/config.html
- http://deeplearning.net/software/theano/library/floatX.html

```python
# default configuration
print(theano.config.floatX)
```

    float32

```python
# we can change it like this
theano.config.floatX = 'float32'
```

```python
print(theano.config.floatX)
```

    float32

To change the float type globally, execute

    export THEANO_FLAGS=floatX=float32

in your bash shell. Or execute Python script as

    THEANO_FLAGS=floatX=float32 python your_script.py

Running Theano on GPU(s). For prerequisites, please see: http://deeplearning.net/software/theano/tutorial/using_gpu.html

Note that `float32` is recommended for GPUs; `float64` on GPUs is currently still relatively slow.

```python
print(theano.config.device)
```

    cpu

You can run a Python script on CPU (e.g. for prototyping and debug) via:

    THEANO_FLAGS=device=cpu,floatX=float64 python your_script.py

or GPU (e.g. for real computation) via:

    THEANO_FLAGS=device=gpu,floatX=float32 python your_script.py

It may also be convenient to create a `.theanorc` file in your home directory to make those configurations permanent. For example, to always use `float32`, execute

    echo -e "\n[global]\nfloatX=float32\n" >> ~/.theanorc

Or, create a `.theanorc` file manually with the following contents

    [global]
    floatX = float32
    device = gpu

## Working with array structures

This is an example code to work with tensors.

Create a $2 \times 3$ tensor, and calculate its column sum.

```python
import numpy as np

# define
# if you are running Theano on 64 bit mode,
# you need to use dmatrix instead of fmatrix
x = T.matrix(name='x') # tensor with arbitrary shape
x_sum = T.sum(x, axis=0)

# compile
calc_sum = theano.function(inputs=[x], outputs=x_sum)

# execute (Python list)
ary = [[1, 2, 3], [1, 2, 3]]
print('Column sum:', calc_sum(ary))

# execute (NumPy array)
ary = np.array(ary, dtype=theano.config.floatX)
print('Column sum:', calc_sum(ary))
```

    Column sum: [ 2.  4.  6.]
    Column sum: [ 2.  4.  6.]

```python
# name can help debug
y = T.matrix(name='hello')
z = T.matrix()

print(y) # will print out variable name
print(z) # will print out variable type
print(y.type()) # will print out type

```

    hello
    <TensorType(float32, matrix)>
    <TensorType(float32, matrix)>

```python
# explicit type specification
wf = T.fmatrix(name='wfmatrix')
wd = T.dmatrix(name='wdmatrix')

print(wf.type())
print(wd.type())
```

    <TensorType(float32, matrix)>
    <TensorType(float64, matrix)>

## Memory management

### shared

Variable with storage that is shared between functions that it appears in.
* can have initial or constant values, e.g. weights of a neural network
* retain value across function calls
* cannot be used as input to a function
* can be updated after each function call

Can be more efficient than input variable
* update in place instead of transferring around
* Theano can then optimize the storage across CPUs and GPUs

More info about memory management in Theano can be found under:
* http://deeplearning.net/software/theano/tutorial/aliasing.html
* https://www.quora.com/What-is-the-meaning-and-benefit-of-shared-variables-in-Theano

```python
# initialize
x = T.matrix(name='x')
b = theano.shared(np.asarray([[1]], dtype=theano.config.floatX), name='b')
w = theano.shared(np.asarray([[0.0, 0.0, 0.0]],
                             dtype=theano.config.floatX))

# w = w + 1.0 # this will cause error
z = x.dot(w.T) + b

update = [[w, w + 1.0]] # update w after each function call

# compile
f = theano.function(inputs=[x],
                    updates=update,
                    outputs=z)

# won't compile as shared variable cannot be used as input
# g = theano.function(inputs=[x, b], outputs = z)

# execute
x_data = np.array([[1, 2, 3]], dtype=theano.config.floatX)
for i in range(5):
    print('z%d:' % i, f(x_data))

```

    z0: [[ 1.]]
    z1: [[ 7.]]
    z2: [[ 13.]]
    z3: [[ 19.]]
    z4: [[ 25.]]

### given

input: transfer from CPU to GPU multiple times
* e.g. multiple epochs

shared: retained values across functions, can be updated after each function call
* like static function variables
* not input for function call
* e.g. network weights

given: transfer from CPU to GPU once
* like constant variables
* shared between multiple function calls
* not input for function call
* values specified, for input or shared variables, during funcion compilation
* e.g. a mini-batch

If we use `inputs`, a datasets is transferred from the CPU to the GPU multiple times, for example, if we iterate over a dataset multiple times (epochs) during gradient descent.

We can use the `givens` variable to insert values into the graph before compiling it. Using this approach we can reduce the number of transfers from RAM (via CPUs) to GPUs to speed up learning with shared variables.

Via `givens`, we can keep the dataset on the GPU if it fits (e.g., a mini-batch).

```python
# define
num_samples = 10
samples = np.asarray([i for i in range(num_samples)],
                     dtype=theano.config.floatX)

# samples = theano.shared(samples)

x = T.lscalar(name='index')
#y = theano.shared(np.asscalar(np.array([1], dtype=theano.config.floatX)))
y = T.vector(name='samples')
w = theano.shared(np.asscalar(np.array([0], dtype=theano.config.floatX)))

z = y[x]*w

# compile
f = theano.function(inputs = [x],
                    updates = [[w, w+1]],
                    givens = {y: samples},
                    outputs = z)

# run
for i in range(np.prod(samples.shape)):
    print(f(i))
```

    0.0
    1.0
    4.0
    9.0
    16.0
    25.0
    36.0
    49.0
    64.0
    81.0

```python
# initialize
x_data = np.array([[1, 2, 3]], dtype=theano.config.floatX)

x = T.matrix(name='hi')
w = theano.shared(np.asarray([[0, 0, 0], [0, 0, 0], [0, 0, 0]], dtype=theano.config.floatX))

# an input variable can be given
b_data = np.array([[-1, 0, 1]], dtype=theano.config.floatX)
b = T.matrix(name='bias')

# a shared variable can be given
c_data = np.array([[4, 5, 6]], dtype=theano.config.floatX)
c = theano.shared(np.asarray([[0]], dtype=theano.config.floatX))

z = x.dot(w.T) + b + c

updates = [[w, w + 1.0]]
givens = {b: b_data, c: c_data}

# compile
net_input = theano.function(inputs=[x],
                            updates=updates,
                            givens=givens,
                            outputs=z)

# execute
for i in range(5):
    print('z:', net_input(x_data))
```

    z: [[ 3.  5.  7.]]
    z: [[  9.  11.  13.]]
    z: [[ 15.  17.  19.]]
    z: [[ 21.  23.  25.]]
    z: [[ 27.  29.  31.]]

## Wrapping things up: A  linear regression example

Model:
$
y = \sum_{i=0}^n w_i x_i = \mathbf{w}^T \mathbf{x}
$
with $x_0 = 1$.

Given a collection of sample data $\{\mathbf{x^{(i)}}, y^{(i)} \}$, find the line $\mathbf{w}$ that minimizes the regression error:
$$
\begin{align}
L(X, Y, \mathbf{w})
= \sum_i \left( y^{(i)} - \hat{y}^{(i)} \right)^2
= \sum_i \left( y^{(i)} - \mathbf{w}^T \mathbf{x}^{(i)} \right)^2
\end{align}
$$

### 2D case

$
y = w_0 + w_1 x
$

### Create some training data

```python
import numpy as np
X_train = np.asarray([[0.0], [1.0], [2.0], [3.0], [4.0],
                      [5.0], [6.0], [7.0], [8.0], [9.0]],
                     dtype=theano.config.floatX)

y_train = np.asarray([1.0, 1.3, 3.1, 2.0, 5.0,
                      6.3, 6.6, 7.4, 8.0, 9.0],
                     dtype=theano.config.floatX)
```

### Implement the training function

Notice:
* the symbolic differentiation for the gradient part
* how different variable types (input, shared, givens, output) are used

```python
import theano
from theano import tensor as T
import numpy as np

def train_linreg(X_train, y_train, eta, epochs):

    costs = []
    # Initialize arrays
    eta0 = T.scalar('eta0') # learning rate
    y = T.vector(name='y')
    X = T.matrix(name='X')
    w = theano.shared(np.zeros(
                      shape=(X_train.shape[1] + 1),
                      dtype=theano.config.floatX),
                      name='w')

    # calculate cost
    y_pred = T.dot(X, w[1:]) + w[0]
    errors = y - y_pred
    cost = T.sum(T.pow(errors, 2))

    # perform gradient update
    gradient = T.grad(cost, wrt=w) # symbolic differentialtion
    update = [(w, w - eta0 * gradient)]

    # compile model
    train = theano.function(inputs=[eta0],
                            outputs=cost,
                            updates=update,
                            givens={X: X_train,
                                    y: y_train})

    for _ in range(epochs):
        # since eta is input
        # we can gradually change the learning rate
        costs.append(train(eta))

    return costs, w
```

Plotting the sum of squared errors cost vs epochs.

```python
import matplotlib.pyplot as plt

costs, w = train_linreg(X_train, y_train, eta=0.001, epochs=10)

plt.plot(range(1, len(costs)+1), costs)

plt.tight_layout()
plt.xlabel('Epoch')
plt.ylabel('Cost')
plt.tight_layout()
# plt.savefig('./figures/cost_convergence.png', dpi=300)
plt.show()
```

### Make prediction

```python
def predict_linreg(X, w):
    Xt = T.matrix(name='X')
    y_pred = T.dot(Xt, w[1:]) + w[0]
    predict = theano.function(inputs=[Xt], givens={w: w}, outputs=y_pred)
    return predict(X)

plt.scatter(X_train, y_train, marker='s', s=50)
plt.plot(range(X_train.shape[0]),
         predict_linreg(X_train, w),
         color='gray',
         marker='o',
         markersize=4,
         linewidth=3)

plt.xlabel('x')
plt.ylabel('y')

plt.tight_layout()
# plt.savefig('./figures/linreg.png', dpi=300)
plt.show()
```

# Theano for neural networks

Also use Keras library

# Choosing activation functions for feedforward neural networks

There are various activation functions for a multi-layer neural networks.
* in theory we can use any differential function
* in practice we want (1) non-linearity and (2) goood convergence for gradient descent

Sigmoid is one we have seen.
* mimics biological neurons
* converge slowly for deep networks (vanishing gradients)

## Logistic function recap

The logistic function, often just called "sigmoid function" is in fact a special case of a sigmoid function.

Linear input $z$:
$$
\begin{align}
z &=  w_0x_{0} + \dots + w_mx_{m}
\\
&= \sum_{j=0}^{m} x_{j}w_{j}
\\
&= \mathbf{w}^T\mathbf{x}
\end{align}
$$
$w_0$ is the bias term, matching $x_0 = 1$

Logistic activation function:

$$\phi_{logistic}(z) = \frac{1}{1 +  e^{-z}}$$

Output range: (0, 1)
* probability for the positive class $z > 0$

### Concrete example

```python
# note that first element (X[0] = 1) to denote bias unit

X = np.array([[1, 1.4, 1.5]])
w = np.array([0.0, 0.2, 0.4])

def net_input(X, w):
    z = X.dot(w)
    return z

def logistic(z):
    return 1.0 / (1.0 + np.exp(-z))

def logistic_activation(X, w):
    z = net_input(X, w)
    return np.asscalar(logistic(z))

print('P(y=1|x) = %.3f' % logistic_activation(X, w))
```

    P(y=1|x) = 0.707

## Multiple outputs

One-hot encoding for multi-class classification.
* $K$ outputs for $K$ classes

Logistic activation outputs cannot be directly interpreted as probabilities.

### Example

A MLP perceptron with
* 3 hidden units + 1 bias unit in the hidden unit
* 3 output units

```python
# W : array, shape = [n_output_units, n_hidden_units+1]
#          Weight matrix for hidden layer -> output layer.
# note that first column (A[:][0] = 1) are the bias units
W = np.array([[1.1, 1.2, 1.3, 0.5],
              [0.1, 0.2, 0.4, 0.1],
              [0.2, 0.5, 2.1, 1.9]])

# A : array, shape = [n_hidden+1, n_samples]
#          Activation of hidden layer.
# note that first element (A[0][0] = 1) is for the bias units

A = np.array([[1.0],
              [0.1],
              [0.3],
              [0.7]])

# Z : array, shape = [n_output_units, n_samples]
#          Net input of output layer.

Z = W.dot(A)
y_probas = logistic(Z)
print('Probabilities:\n', y_probas)
```

    Probabilities:
     [[ 0.87653295]
     [ 0.57688526]
     [ 0.90114393]]

The outputs do not sum to 1 and thus are not probabilities.

Need normalization for probability
* divide all outputs by their summation
* softmax which should be applied to z (linear inputs) before logistic activation

OK for classification:

```python
y_class = np.argmax(Z, axis=0)
print('predicted class label: %d' % y_class[0])
```

    predicted class label: 2

## Estimating probabilities in multi-class classification via the softmax function

The softmax function
* is a generalization of the logistic function
* allows us to compute meaningful class probabilities in multi-class settings
* i.e. multinomial logistic regression

$z$: linear input as usual

$K$: number of classes

$$
\begin{align}
P(y=j|z) =\phi_{softmax}(z) = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}
\end{align}
$$

$P(y=j | z)$: probability of class $j$ for input $z$ in range $(0, 1)$

```python
def softmax(z):
    return np.exp(z) / np.sum(np.exp(z))

def softmax_activation(X, w):
    z = net_input(X, w)
    return softmax(z)
```

```python
y_probas = softmax(Z) # same Z computed above
print('Probabilities:\n', y_probas)
```

    Probabilities:
     [[ 0.40386493]
     [ 0.07756222]
     [ 0.51857284]]

```python
y_probas.sum()
```

    1.0

The class probabilities sum to 1.

The predicted class is the same as logistic regression.

```python
y_class = np.argmax(Z, axis=0)
y_class
```

    array([2], dtype=int64)

## Broadening the output spectrum using a hyperbolic tangent

Another special case of a sigmoid function, it can be interpreted as a rescaled version of the logistic function.

$$
\begin{align}
\phi_{tanh}(z) = \frac{e^{z}-e^{-z}}{e^{z}+e^{-z}}
\end{align}
$$

$\phi_{tanh}$ is a rescaled version of $\phi_{logistic}$:
$$
\begin{align}
\phi_{tanh}(z) = 2 \phi_{logistic}(z) - 1
\end{align}
$$

Output range: (-1, 1)

```python
def tanh(z):
    e_p = np.exp(z)
    e_m = np.exp(-z)
    return (e_p - e_m) / (e_p + e_m)
```

```python
import matplotlib.pyplot as plt

z = np.arange(-5, 5, 0.005)
log_act = logistic(z)
tanh_act = tanh(z)

# alternatives:
# from scipy.special import expit
# log_act = expit(z)
# tanh_act = np.tanh(z)

plt.ylim([-1.5, 1.5])
plt.xlabel('net input $z$')
plt.ylabel('activation $\phi(z)$')
plt.axhline(1, color='black', linestyle='--')
plt.axhline(0.5, color='black', linestyle='--')
plt.axhline(0, color='black', linestyle='--')
plt.axhline(-1, color='black', linestyle='--')

plt.plot(z, tanh_act,
         linewidth=2,
         color='black',
         label='tanh')
plt.plot(z, log_act,
         linewidth=2,
         color='lightgreen',
         label='logistic')

plt.legend(loc='lower right')
plt.tight_layout()
# plt.savefig('./figures/activation.png', dpi=300)
plt.show()
```

## Different activation functions

# Training neural networks efficiently using Keras

A library (stared in early 2015) to facilitate neural network training.
* built on top of Theano
* intuitive and popular API
* front-end for Theano and TensorFlow

Once you have Theano installed, [Keras](https://github.com/fchollet/keras) can be installed via

    pip install Keras

### Loading MNIST

1) Download the 4 MNIST datasets from http://yann.lecun.com/exdb/mnist/

- train-images-idx3-ubyte.gz:  training set images (9912422 bytes)
- train-labels-idx1-ubyte.gz:  training set labels (28881 bytes)
- t10k-images-idx3-ubyte.gz:   test set images (1648877 bytes)
- t10k-labels-idx1-ubyte.gz:   test set labels (4542 bytes)

2) Unzip those files

3) Copy the unzipped files to a directory `./mnist`

Li-Yi: I enhanced the functions below so that we can do everything (including downloading and decompression) inside ipynb.

```python
import os
import os.path
import struct
import gzip
import numpy as np

def open_mnist(full_path):
    if full_path.find('.gz') >= 0:
        return gzip.open(full_path, 'rb')
    else:
        return open(full_path, 'rb')

def pick_mnist(path, name, exts):
    for ext in exts:
        full_path = os.path.join(path, name + ext)
        if os.path.isfile(full_path):
            return full_path
    # none of the exts options works
    return None

def load_mnist(path, kind='train', exts=['', '.gz']):
    """Load MNIST data from `path`"""
    labels_path = pick_mnist(path, kind + '-labels-idx1-ubyte', exts)
    images_path = pick_mnist(path, kind + '-images-idx3-ubyte', exts)

    with open_mnist(labels_path) as lbpath:
        magic, n = struct.unpack('>II', lbpath.read(8))
        if(magic != 2049):
            raise IOError(str(magic) + ' != ' + str(2049))

        # np.fromfile does not work with gzip open
        # http://stackoverflow.com/questions/15966335/efficient-numpy-fromfile-on-zipped-files
        # labels = np.fromfile(lbpath, dtype=np.uint8)
        content = lbpath.read()
        labels = np.frombuffer(content, dtype=np.uint8)
        if(len(labels) != n):
            raise IOError(str(len(labels)) + ' != ' + str(n))

    with open_mnist(images_path) as imgpath:
        magic, num, rows, cols = struct.unpack(">IIII", imgpath.read(16))
        if(magic != 2051):
            raise IOError(str(magic) + ' != ' + str(2051))

        # images = np.fromfile(imgpath, dtype=np.uint8).reshape(num, rows*cols)
        content = imgpath.read()
        images = np.frombuffer(content, dtype=np.uint8).reshape(num, rows*cols)
        if(num != len(labels)):
            raise IOError(str(num) + ' != ' + str(len(labels)))

    return images, labels
```

```python
mnist_data_folder = os.path.join('..', 'datasets', 'mnist')
exts = ['', '.gz'] # for already gunzipped files and not yet gzipped files

X_train, y_train = load_mnist(mnist_data_folder, kind='train', exts=exts)
print('Rows: %d, columns: %d' % (X_train.shape[0], X_train.shape[1]))

X_test, y_test = load_mnist(mnist_data_folder, kind='t10k', exts=exts)
print('Rows: %d, columns: %d' % (X_test.shape[0], X_test.shape[1]))
```

    Rows: 60000, columns: 784
    Rows: 10000, columns: 784

### Multi-layer Perceptron in Keras

In order to run the following code via GPU, you can execute the Python script that was placed in this directory via

    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_keras_mlp.py

```python
import theano

theano.config.floatX = 'float32'
X_train = X_train.astype(theano.config.floatX)
X_test = X_test.astype(theano.config.floatX)
```

#### One-hot encoding of the class variable

```python
from keras.utils import np_utils

print('First 3 labels: ', y_train[:3])

y_train_ohe = np_utils.to_categorical(y_train)
print('\nFirst 3 labels (one-hot):\n', y_train_ohe[:3])
```

    First 3 labels:  [5 0 4]

    First 3 labels (one-hot):
     [[ 0.  0.  0.  0.  0.  1.  0.  0.  0.  0.]
     [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.]]

#### Implement a neural network

* fully connected with 2 hidden layers

* tanh for hidden layers

* softmax for output layer

* cross entropy loss function (to match softmax output)

* SGD optimization

```python
from keras.models import Sequential
from keras.layers.core import Dense
from keras.optimizers import SGD

np.random.seed(1)

model = Sequential()
model.add(Dense(input_dim=X_train.shape[1],
                output_dim=50,
                init='uniform',
                activation='tanh'))

model.add(Dense(output_dim=50,
                init='uniform',
                activation='tanh'))

model.add(Dense(output_dim=y_train_ohe.shape[1],
                init='uniform',
                activation='softmax'))

sgd = SGD(lr=0.001, decay=1e-7, momentum=.9)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=["accuracy"])

model.fit(X_train, y_train_ohe,
          nb_epoch=50,
          batch_size=300,
          verbose=1,
          validation_split=0.1 # 10% of training data for validation per epoch
         )
```

    Train on 54000 samples, validate on 6000 samples
    Epoch 1/50
    54000/54000 [==============================] - 2s - loss: 2.2290 - acc: 0.3594 - val_loss: 2.1094 - val_acc: 0.5338
    Epoch 2/50
    54000/54000 [==============================] - 2s - loss: 1.8844 - acc: 0.5290 - val_loss: 1.6073 - val_acc: 0.5585
    Epoch 3/50
    54000/54000 [==============================] - 2s - loss: 1.3925 - acc: 0.5863 - val_loss: 1.1676 - val_acc: 0.6633
    Epoch 4/50
    54000/54000 [==============================] - 2s - loss: 1.0641 - acc: 0.6919 - val_loss: 0.9085 - val_acc: 0.7692
    Epoch 5/50
    54000/54000 [==============================] - 2s - loss: 0.8601 - acc: 0.7694 - val_loss: 0.7421 - val_acc: 0.8262
    Epoch 6/50
    54000/54000 [==============================] - 2s - loss: 0.7228 - acc: 0.8204 - val_loss: 0.6176 - val_acc: 0.8608
    Epoch 7/50
    54000/54000 [==============================] - 2s - loss: 0.6251 - acc: 0.8448 - val_loss: 0.5409 - val_acc: 0.8802
    Epoch 8/50
    54000/54000 [==============================] - 2s - loss: 0.5589 - acc: 0.8632 - val_loss: 0.4755 - val_acc: 0.8908
    Epoch 9/50
    54000/54000 [==============================] - 2s - loss: 0.5019 - acc: 0.8743 - val_loss: 0.4247 - val_acc: 0.9005
    Epoch 10/50
    54000/54000 [==============================] - 2s - loss: 0.4520 - acc: 0.8843 - val_loss: 0.3786 - val_acc: 0.9080
    Epoch 11/50
    54000/54000 [==============================] - 2s - loss: 0.4181 - acc: 0.8935 - val_loss: 0.3548 - val_acc: 0.9140
    Epoch 12/50
    54000/54000 [==============================] - 2s - loss: 0.3936 - acc: 0.8970 - val_loss: 0.3394 - val_acc: 0.9170
    Epoch 13/50
    54000/54000 [==============================] - 2s - loss: 0.3720 - acc: 0.9012 - val_loss: 0.3212 - val_acc: 0.9177
    Epoch 14/50
    54000/54000 [==============================] - 2s - loss: 0.3590 - acc: 0.9030 - val_loss: 0.3136 - val_acc: 0.9160
    Epoch 15/50
    54000/54000 [==============================] - 2s - loss: 0.3462 - acc: 0.9074 - val_loss: 0.2912 - val_acc: 0.9245
    Epoch 16/50
    54000/54000 [==============================] - 2s - loss: 0.3318 - acc: 0.9106 - val_loss: 0.2791 - val_acc: 0.9262
    Epoch 17/50
    54000/54000 [==============================] - 2s - loss: 0.3214 - acc: 0.9117 - val_loss: 0.2741 - val_acc: 0.9285
    Epoch 18/50
    54000/54000 [==============================] - 2s - loss: 0.3166 - acc: 0.9125 - val_loss: 0.2625 - val_acc: 0.9310
    Epoch 19/50
    54000/54000 [==============================] - 3s - loss: 0.3088 - acc: 0.9135 - val_loss: 0.2570 - val_acc: 0.9272
    Epoch 20/50
    54000/54000 [==============================] - 4s - loss: 0.2950 - acc: 0.9173 - val_loss: 0.2464 - val_acc: 0.9358
    Epoch 21/50
    54000/54000 [==============================] - 6s - loss: 0.2941 - acc: 0.9160 - val_loss: 0.2582 - val_acc: 0.9275
    Epoch 22/50
    54000/54000 [==============================] - 4s - loss: 0.2906 - acc: 0.9173 - val_loss: 0.2493 - val_acc: 0.9352
    Epoch 23/50
    54000/54000 [==============================] - 3s - loss: 0.2869 - acc: 0.9201 - val_loss: 0.2425 - val_acc: 0.9322
    Epoch 24/50
    54000/54000 [==============================] - 4s - loss: 0.2782 - acc: 0.9211 - val_loss: 0.2445 - val_acc: 0.9345
    Epoch 25/50
    54000/54000 [==============================] - 4s - loss: 0.2722 - acc: 0.9217 - val_loss: 0.2345 - val_acc: 0.9382
    Epoch 26/50
    54000/54000 [==============================] - 3s - loss: 0.2666 - acc: 0.9240 - val_loss: 0.2260 - val_acc: 0.9370
    Epoch 27/50
    54000/54000 [==============================] - 3s - loss: 0.2635 - acc: 0.9253 - val_loss: 0.2239 - val_acc: 0.9400
    Epoch 28/50
    54000/54000 [==============================] - 4s - loss: 0.2629 - acc: 0.9255 - val_loss: 0.2250 - val_acc: 0.9398
    Epoch 29/50
    54000/54000 [==============================] - 3s - loss: 0.2584 - acc: 0.9261 - val_loss: 0.2291 - val_acc: 0.9345
    Epoch 30/50
    54000/54000 [==============================] - 4s - loss: 0.2478 - acc: 0.9299 - val_loss: 0.2076 - val_acc: 0.9433
    Epoch 31/50
    54000/54000 [==============================] - 4s - loss: 0.2464 - acc: 0.9281 - val_loss: 0.2230 - val_acc: 0.9363
    Epoch 32/50
    54000/54000 [==============================] - 4s - loss: 0.2518 - acc: 0.9267 - val_loss: 0.2201 - val_acc: 0.9368
    Epoch 33/50
    54000/54000 [==============================] - 3s - loss: 0.2455 - acc: 0.9271 - val_loss: 0.2071 - val_acc: 0.9390
    Epoch 34/50
    54000/54000 [==============================] - 3s - loss: 0.2371 - acc: 0.9302 - val_loss: 0.2148 - val_acc: 0.9413
    Epoch 35/50
    54000/54000 [==============================] - 6s - loss: 0.2442 - acc: 0.9292 - val_loss: 0.2132 - val_acc: 0.9395
    Epoch 36/50
    54000/54000 [==============================] - 3s - loss: 0.2362 - acc: 0.9310 - val_loss: 0.2127 - val_acc: 0.9392
    Epoch 37/50
    54000/54000 [==============================] - 2s - loss: 0.2361 - acc: 0.9315 - val_loss: 0.2021 - val_acc: 0.9438
    Epoch 38/50
    54000/54000 [==============================] - 4s - loss: 0.2241 - acc: 0.9353 - val_loss: 0.2006 - val_acc: 0.9432
    Epoch 39/50
    54000/54000 [==============================] - 3s - loss: 0.2323 - acc: 0.9324 - val_loss: 0.2181 - val_acc: 0.9397
    Epoch 40/50
    54000/54000 [==============================] - 2s - loss: 0.2219 - acc: 0.9351 - val_loss: 0.2040 - val_acc: 0.9422
    Epoch 41/50
    54000/54000 [==============================] - 2s - loss: 0.2254 - acc: 0.9337 - val_loss: 0.1984 - val_acc: 0.9443
    Epoch 42/50
    54000/54000 [==============================] - 2s - loss: 0.2157 - acc: 0.9369 - val_loss: 0.2002 - val_acc: 0.9432
    Epoch 43/50
    54000/54000 [==============================] - 2s - loss: 0.2190 - acc: 0.9370 - val_loss: 0.1963 - val_acc: 0.9453
    Epoch 44/50
    54000/54000 [==============================] - 2s - loss: 0.2146 - acc: 0.9368 - val_loss: 0.1797 - val_acc: 0.9495
    Epoch 45/50
    54000/54000 [==============================] - 2s - loss: 0.2128 - acc: 0.9383 - val_loss: 0.1973 - val_acc: 0.9423
    Epoch 46/50
    54000/54000 [==============================] - 2s - loss: 0.2114 - acc: 0.9378 - val_loss: 0.1850 - val_acc: 0.9492
    Epoch 47/50
    54000/54000 [==============================] - 2s - loss: 0.2095 - acc: 0.9376 - val_loss: 0.1918 - val_acc: 0.9468
    Epoch 48/50
    54000/54000 [==============================] - 2s - loss: 0.2076 - acc: 0.9394 - val_loss: 0.1878 - val_acc: 0.9497
    Epoch 49/50
    54000/54000 [==============================] - 2s - loss: 0.2084 - acc: 0.9385 - val_loss: 0.1831 - val_acc: 0.9467
    Epoch 50/50
    54000/54000 [==============================] - 3s - loss: 0.2023 - acc: 0.9404 - val_loss: 0.1907 - val_acc: 0.9470

    <keras.callbacks.History at 0x1f5d691b940>

```python
y_train_pred = model.predict_classes(X_train, verbose=0)
print('First 3 predictions: ', y_train_pred[:3])
```

    First 3 predictions:  [5 0 4]

```python
train_acc = np.sum(y_train == y_train_pred, axis=0) / X_train.shape[0]
print('Training accuracy: %.2f%%' % (train_acc * 100))
```

    Training accuracy: 94.08%

```python
y_test_pred = model.predict_classes(X_test, verbose=0)
test_acc = np.sum(y_test == y_test_pred, axis=0) / X_test.shape[0]
print('Test accuracy: %.2f%%' % (test_acc * 100))
```

    Test accuracy: 93.55%

# Summary

Plenty deep learning libraries to use so that we do not have to write code from scratch.
* Theano: Python interface, resource management (processors and memory)
* TensorFlow: similar flavor to Theano, developed by Google
* Keras: front-end for Theano and TensorFlow, easy and intuitive API
* Torch7: via lua language, used by DeepMind folks and Yann LeCun
* Caffe: c++ library
* etc.

Alternative libraries with Theano:
* Pylearn2
* Lasagne: more minimalistic and extensible

# Reading

* PML Chapter 13


# C33/C34 Beatmungsgeräte
> Manuel Vazquez
> Christian Spiller
> Michael Wild

Repository: https://github.com/DataHackDaysBE/C33_34_Ventilators
Binder: [](https://mybinder.org/v2/gh/DataHackDaysBE/C33_34_Ventilators.git/main)

## Imports

```python
import pandas as pd
import requests
from zipfile import ZipFile
import os, os.path as op
import tqdm.notebook as tqdm
import plotly.express as px, plotly.graph_objects as go
from ipywidgets import interact
```

## Load Data

```python
if not op.isdir('_data'):
    os.mkdir('_data')

def download_if_not_exists(fname, url):
    '''If the file `fname` does not exist in the '_data/' folder, download it from `url` and save it in `'_data/'+fname`.'''
    fpath = op.join('_data', fname)
    if not op.isfile(fpath):
        r = requests.get(url, allow_redirects=True)
        r.raise_for_status()
        with open(fpath, 'wb') as f:
            f.write(r.content)
    return fpath
```

### Ventilator Data

```python
# Get the ventilator occupancy time series
cantons = ['Kanton_' + k for k in 'AG AI AR BE BL BS FR GE GL GR JU LU NE NW OW SG SH SO SZ TG TI UR VD ZG ZH'.split()] + ['FL']
data = None
for c in tqdm.tqdm_notebook(cantons):
    fname = f'COVID19_Fallzahlen_{c}_total.csv'
    fpath = download_if_not_exists(fname, 'https://raw.githubusercontent.com/openZH/covid_19/master/fallzahlen_kanton_total_csv_v2/' + fname)
    df = pd.read_csv(fpath, parse_dates=[['date', 'time']])
    data = pd.concat((data, df)) if data is not None else df

data.set_index(['abbreviation_canton_and_fl', 'date_time'], inplace=True)
```

      0%|          | 0/26 [00:00<?, ?it/s]

### Incidence Data

```python
fname = 'sources-csv.zip'
fpath = download_if_not_exists(fname, 'https://www.covid19.admin.ch/api/data/20211112-b5cqnmme/downloads/' + fname)
inc_raw = None
with ZipFile(fpath) as zf:
    with zf.open(f'data/COVID19Cases_geoRegion_AKL10_w.csv') as f:
        inc_raw = pd.read_csv(f)
```

## Cleanup and Pre-Processing of Data

```python
inc = inc_raw[["datum", "entries", "altersklasse_covid19", "geoRegion"]]
inc = inc.rename(
    columns={
        "altersklasse_covid19": "age_group",
    }
)

# correct date
# -> split years
inc_2020 = inc.loc[inc["datum"] < 202100]
inc_2021 = inc.loc[inc["datum"] >= 202100]

# -> convert 2020xy to xy
inc_2020 = inc_2020.copy()
inc_2020["t"] = inc_2020["datum"] - 202000

# -> convert 2021xy to xy+53
inc_2021 = inc_2021.copy()
inc_2021["t"] = inc_2021["datum"] - 202100 + 53

# -> put together
inc = pd.concat([inc_2020, inc_2021], axis=0)
inc.drop(columns=["datum"], inplace=True)

# Set Index
inc = inc.copy()
inc.set_index(['t', 'age_group', 'geoRegion'], inplace=True)

# fill missing values
tmp = inc.reset_index().pivot(index=['t', 'geoRegion'], values='entries', columns=['age_group']).fillna(0).reset_index()
tmp.drop(columns='Unbekannt', inplace=True)
inc = tmp.melt(id_vars=['t', 'geoRegion'], value_name='entries').set_index(['t', 'age_group', 'geoRegion'])
```

```python
fig = px.line(inc.loc[:,:, 'BE'].reset_index(), x='t', y='entries', facet_col='geoRegion', facet_col_wrap=3, color='age_group', title='Incidence')
fig.show()
```

## Model

#### Total occupation at time $T$ is the sum of the occupations by cohorts (by age-group $i$)
\begin{equation}
    \Large V(T) = \sum_{i=1}^{N_i} V_{i}(T)
\end{equation}

#### The occupations by cohort $V_{i}$ are the cumulative sum of incremental changes
\begin{equation}
    \Large V_{i}(T) = \sum_{t=0}^T \Delta V_{i}(t)
\end{equation}

#### The incremental changes $\Delta V_{i}$ are the balance between entries $E_{i}$ and exits $A_{i}$
\begin{equation}
    \Large \Delta V_{i}(t) = E_{i}(t) - A_{i}(t)
\end{equation}

#### The entries $E_{i}$ at time $t$ are modelled as a fraction $\alpha_{i}$ of the incidence number $I_{i}$ at time $t - \tau_E$
\begin{equation}
    \Large E_{i}(t) = \alpha_{i} I_{i}(t-\tau_E)
\end{equation}

#### The exits $A_{i}$ at time $t$ are modelled as the entries at time $t - \tau_A$
\begin{equation}
    \Large A_{i}(t) = E_{i}(t - \tau_A)
\end{equation}

#### Putting things together, the model reads as
\begin{equation}
    \Large V(T) = \sum_{i=1}^{N_i} \sum_{t=0}^T \alpha_{i} \left(I_{i}(t - \tau_E) - I_{i}(t - \tau_A)\right)
\end{equation}

#### The parameters of the model are the _conversion factors_ $\alpha_{i}$ and the time offsets $\tau_E$ and $\tau_A$.

```python
def apply_model(inc, alpha, tau_e, tau_a):
    '''Model for the estimation of ventilator occupation given incidence time series

    Parameters
    ----------
    inc: pd.DataFrame
      DataFrame with incidence numbers with a multi-level index. Levels must be:
          * level=0: int, week numbers, named 't'
          * level=1: str, age-group, named 'age_group'
    alpha: dict
      Model parameters representing the _conversion factors_ from the incidence
      numbers to the number of patients requiring ventilation. The keys must
      correspond to the age-group labels of `inc` and the values must be floats
      in [0, 1].
    tau_e: int
      Model parameter describing the number of weeks that passes between
      infection and a patient being attached to a ventilator.
    tau_a: int
      Model parameter describing the number of weeks that passes between
      infection and a patient either no longer requires the ventilator
      or is deceased.

    Returns
    -------
    pd.DataFrame
      A time-series predicting the ventilator occupancy.
    '''

    # build grouping by age and vaccination status
    inc_grp = inc.groupby(level=['age_group'])
    # shift the groups and calculate the difference
    delta = inc_grp.shift(tau_e) - inc_grp.shift(tau_a)
    # for each group, multiply the difference with the conversion factor
    for i in inc.index.unique('age_group'):
        delta.loc[(slice(None), i), :] *= alpha[i]
    # for each group calculate the cumulative sum
    vi = delta.groupby(level=['age_group']).cumsum()
    # calculate total occupation by summing over the groups
    v = vi.groupby(level='t').sum()
    return v

```

```python
def do_plot(alpha, tau_E, tau_A, canton):
    a = {g: alpha/100.0 for g in inc.index.get_level_values(1)}
    pred = apply_model(inc.loc[:, :, canton].droplevel(2), a, tau_E, tau_A)
    pred.rename(columns={'entries': 'current_vent'}, inplace=True)
    f = px.line(pred.reset_index(), x='t', y='current_vent', title='Predicted ventilator occupation')
    f.show()
    tmp = data.loc[canton].reset_index()
    ff = px.line(tmp, x='date_time', y='current_vent', title='Real ventilator occupation')
    ff.show()
```

## Apply the Model

```python
interact(do_plot, alpha=0.35, tau_E=8, tau_A=15, canton=['BE', 'ZH', 'AG'])
```

    interactive(children=(FloatSlider(value=0.35, description='alpha', max=1.0499999999999998, min=-0.35), IntSlid…

    <function __main__.do_plot(alpha, tau_E, tau_A, canton)>

```python

```


```python
import os
import sys
jupyter_module_path = os.path.join( os.getcwd(), '..', '..', 'jupyter' )
sys.path.append( jupyter_module_path )

from ef_jupyter_visualize import *
```

In this example, an equation for a contour of an axially symmetric
beam propagating in free space is checked.

Suppose there is an axially symmetric charged particles beam propagating
in free space.
Under certain set of approximations, trajectory of a particle on the edge
of the beam can be described by the following expressions:

<!--  -->
\begin{align}
  & r = r(0) e^{u^2}
  \\
  & z =
  \left( \dfrac{m v^3}{q I} \right)^{1/2}
  r(0)
  \int_{0}^{u} e^{u^2} du
\end{align}

where $u$ is a parameter, $v$ is speed of particles along the $z$ axis,
$r(0)$ is initial radius and $I$ is full current of the beam.
Derivation of these relations can be found in supplementary notebook [link].

These equations define a contour of the beam $r(z)$. It is possible to perform
a numerical simulation of a beam propagating in free space and
plot particles coordinates on the X-Z or Y-Z plane.
This will allow to compare the numerical profile
with the analytical expression.

The beam will become $e = 2.71$ times wider than the initial at $u = 1$.
For this value of the parameter, the integral in the $z$ equation
[approximately equals to](https://www.wolframalpha.com/input/?i=integrate%28+e^u^2,+u,+0,+1+%29) $1.46$.
Since the speed in the $z$-direction is assumed constant, it is possible to estimate the time needed for
particles to pass this distance as

<!--  -->
\begin{align}
  t \approx
  1.5 \left( \dfrac{m v}{q I} \right)^{1/2}
  r(0)
\end{align}

Let's substitute some numerical values (see the cell below). Suppose there is an electron beam ($q = 4.8 \cdot 10^{-10} ~ [\mbox{cgs}], ~ m = 9.1 \cdot 10^{-28} ~ [\mbox{g}]$) with full current $I = 0.1 ~ [\mbox{A}] = 2.998 \cdot 10^{8} ~ [\mbox{cgs}]$ and initial radius $r(0) = 0.5 ~ [\mbox{cm}]$.
Energy of it's particles equals to 1 keV, so that speed is $v = 1.808 \cdot 10^{9} ~ [\mbox{cm/s}]$.
For this beam to get $2.71$ times wider (from $0.5 ~ [\mbox{cm}]$ to $1.36 ~ [\mbox{cm}]$), it takes $t = 2.63 \cdot 10^{-9} ~ [\mbox{s}]$ seconds.
This happens over a distance $z = 4.632 ~ [\mbox{cm}]$.

```python
from math import *

m = 9.8e-28
q = 4.8e-10
print( "q = {:.3e} [cgs]".format( q ) )
print( "m = {:.3e} [g]".format( m ) )

r_0 = 0.5
print( "beam_radius = {:.3e} [cm]".format( r_0 ) )

ampere_to_cgs = 2997924536.8431
I = 0.1 * ampere_to_cgs
print( "I = {:.3e} [A] = {:.3e} [cgs]".format( I / ampere_to_cgs, I ) )

ev_to_cgs = 1.60218e-12
E = 1000 * ev_to_cgs
v = sqrt( 2 * E / m )
print( "E = {:.3e} [eV] = {:.3e} [erg]".format( E / ev_to_cgs, E ) )
print( "v = {:.3e} [cm/s]".format( v ) )

r0_e_times_wider = e * r_0
z_e_times_wider = 1.46 * sqrt( m * v**3 / q / I ) * r_0
t_e_times_wider = 1.5 * sqrt( m * v / q / I ) * r_0
print( "r0_e_times_wider = {:.3e} [cm]".format( r0_e_times_wider ) )
print( "t_e_times_wider = {:.3e} [s]".format( t_e_times_wider ) )
print( "z_e_times_wider = {:.3e} [cm]".format( z_e_times_wider ) )
```

    q = 4.800e-10 [cgs]
    m = 9.800e-28 [g]
    beam_radius = 5.000e-01 [cm]
    I = 1.000e-01 [A] = 2.998e+08 [cgs]
    E = 1.000e+03 [eV] = 1.602e-09 [erg]
    v = 1.808e+09 [cm/s]
    r0_e_times_wider = 1.359e+00 [cm]
    t_e_times_wider = 2.632e-09 [s]
    z_e_times_wider = 4.632e+00 [cm]

These estimates can be used as a basis for config file parameters.

```python
axially_symmetric_beam_conf = EfConf()
```

Full simulation time is set to 3.0e-9 [s]. There are 100 time steps, with each 10th step written to file.

```python
sim_time = 3.0e-9
n_of_steps = 100
dt = sim_time / n_of_steps
save_each_step = 10
dt_save = dt * save_each_step
print( "simulation_time = {:.3e} [s]".format( sim_time ) )
print( "number_of_time_steps = {:d}".format( n_of_steps ) )
print( "time_step_size = {:.3e} [s]".format( dt ) )
print( "time_save_step = {:.3e} [s]".format( dt_save ))

axially_symmetric_beam_conf.time_grid.total_time = sim_time
axially_symmetric_beam_conf.time_grid.time_step_size = dt
axially_symmetric_beam_conf.time_grid.time_save_step = dt_save

print( axially_symmetric_beam_conf.time_grid.export() )
```

    simulation_time = 3.000e-09 [s]
    number_of_time_steps = 100
    time_step_size = 3.000e-11 [s]
    time_save_step = 3.000e-10 [s]
    {'Time grid': {'total_time': 3e-09, 'time_save_step': 3e-10, 'time_step_size': 3e-11}}

The simulation domain is defined as

```python
axially_symmetric_beam_conf.spatial_mesh = SpatialMesh(
    grid_x_size = 5.0,
    grid_x_step = 0.1,
    grid_y_size = 5.0,
    grid_y_step = 0.1,
    grid_z_size = 10.0,
    grid_z_step = 0.1 )
```

with z size approximately two times the estimated value, with 100 nodes in that direction. X and y sizes are 10 times the radius with 50 nodes in each direction.

The source is centered along the x and y axes and is close to origin along the z. For a time step dt = 3.00e-11 [s], to provide a current I = 0.1 [A] the source has to generate $n = I ~ dt ~/~ q = 1.87 \cdot 10^7$ particles each time step. This is not computationally feasible, so instead let's fix an amount of generated particles to 5000 at each step. The charge of macroparticles should be $Q = I ~ dt ~/~ n = 1.799 \cdot 10^{-6} ~ [\mbox{cgs}]$ To preserve charge-to-mass ratio, mass of the macroparticles should be set to $M = Q ~/~ q_e ~ m_e = 3.672 \cdot 10^{-24} ~ [\mbox{g}]$. To have the same initial velocity as electrons, mean momentum should be set to $p = M ~ v = 6.641 \cdot 10^{-15} ~ [\mbox{g * cm / s}]$.

```python
num_of_real_particles = I * dt / q
print( "num_of_real_particles = {:.3e}".format( num_of_real_particles ) )

num_of_macro_particles = 5000
macro_q = I * dt / num_of_macro_particles
macro_m = macro_q / q * m
macro_mean_momentum = macro_m * v
print( "num_of_macro_particles = {:d}".format( num_of_macro_particles ) )
print( "macro_q = {:.3e} [cgs]".format( macro_q ) )
print( "macro_m = {:.3e} [g]".format( macro_m ) )
print( "macro_mean_momentum = {:.3e} [g * cm / s]".format( macro_mean_momentum ) )

axially_symmetric_beam_conf.add_source( ParticleSourceCylinder(
    name = "cathode_emitter",
    initial_number_of_particles = num_of_macro_particles,
    particles_to_generate_each_step = num_of_macro_particles,
    cylinder_axis_start_x = axially_symmetric_beam_conf.spatial_mesh.grid_x_size / 2,
    cylinder_axis_start_y = axially_symmetric_beam_conf.spatial_mesh.grid_y_size / 2,
    cylinder_axis_start_z = 0.51,
    cylinder_axis_end_x = axially_symmetric_beam_conf.spatial_mesh.grid_x_size / 2,
    cylinder_axis_end_y = axially_symmetric_beam_conf.spatial_mesh.grid_y_size / 2,
    cylinder_axis_end_z = 0.52,
    cylinder_radius = 0.5,
    mean_momentum_x = 0,
    mean_momentum_y = 0,
    mean_momentum_z = macro_mean_momentum,
    temperature = 0.0,
    charge = -macro_q,
    mass = macro_m
))
```

    num_of_real_particles = 1.874e+07
    num_of_macro_particles = 5000
    macro_q = 1.799e-06 [cgs]
    macro_m = 3.672e-24 [g]
    macro_mean_momentum = 6.641e-15 [g * cm / s]

```python
axially_symmetric_beam_conf.visualize()
```

Potential is zero on each boundary of the domain.

```python
axially_symmetric_beam_conf.boundary_conditions = BoundaryConditions(
    boundary_phi_right = 0, boundary_phi_left = 0,
    boundary_phi_bottom = 0, boundary_phi_top = 0,
    boundary_phi_near = 0, boundary_phi_far = 0
)
```

Other config parameters:

```python
axially_symmetric_beam_conf.output_file = OutputFile(
    output_filename_prefix = "contour_jup_",
    output_filename_suffix = ".h5")
axially_symmetric_beam_conf.particle_interaction_model = ParticleInteractionModel(
    particle_interaction_model = "PIC" )
```

```python
axially_symmetric_beam_conf.print_config()
```

    [Particle interaction model]
    particle_interaction_model  =  PIC

    [Output filename]
    output_filename_prefix  =  contour_jup_
    output_filename_suffix  =  .h5

    [Boundary conditions]
    boundary_phi_top  =  0
    boundary_phi_bottom  =  0
    boundary_phi_near  =  0
    boundary_phi_left  =  0
    boundary_phi_far  =  0
    boundary_phi_right  =  0

    [Time grid]
    total_time  =  3e-09
    time_save_step  =  3e-10
    time_step_size  =  3e-11

    [Particle_source_cylinder.cathode_emitter]
    cylinder_axis_start_x  =  2.5
    charge  =  -1.7987547221058602e-06
    mean_momentum_y  =  0
    temperature  =  0.0
    cylinder_axis_start_z  =  0.51
    cylinder_axis_start_y  =  2.5
    mean_momentum_x  =  0
    particles_to_generate_each_step  =  5000
    cylinder_axis_end_y  =  2.5
    cylinder_axis_end_x  =  2.5
    mean_momentum_z  =  6.640708217582056e-15
    initial_number_of_particles  =  5000
    cylinder_radius  =  0.5
    cylinder_axis_end_z  =  0.52
    mass  =  3.672457557632798e-24

    [Spatial mesh]
    grid_x_size  =  5.0
    grid_y_size  =  5.0
    grid_y_step  =  0.1
    grid_z_size  =  10.0
    grid_z_step  =  0.1
    grid_x_step  =  0.1

```python
axially_symmetric_beam_conf.run(
    ef_command = "python3 ../../main.py",
    workdir = "./",
    save_config_as = None)
```

```python
import h5py
import numpy as np
import matplotlib.pyplot as plt
import scipy.integrate

filename = 'contour_jup_0000100.h5'
h5file = h5py.File( filename, mode = "r" )

def get_source_current( h5file ):
    time_step = h5file["/Time_grid"].attrs["time_step_size"]
    charge = h5file["/Particle_sources/cathode_emitter"].attrs["charge"]
    particles_per_step = h5file[
        "/Particle_sources/cathode_emitter"].attrs["particles_to_generate_each_step"]
    current = particles_per_step * charge / time_step
    return current

def get_source_geometry( h5file ):
    axis_start_x = \
        h5file["/Particle_sources/cathode_emitter"].attrs["cylinder_axis_start_x"]
    axis_start_z = \
        h5file["/Particle_sources/cathode_emitter"].attrs["cylinder_axis_start_z"]
    radius = h5file["/Particle_sources/cathode_emitter"].attrs["cylinder_radius"]
    return ( axis_start_x, axis_start_z, radius )

def get_source_particle_parameters( h5file ):
    mass = h5file["/Particle_sources/cathode_emitter"].attrs["mass"]
    charge = h5file["/Particle_sources/cathode_emitter"].attrs["charge"]
    momentum_z = h5file["/Particle_sources/cathode_emitter"].attrs["mean_momentum_z"]
    return ( mass, charge, momentum_z )

def beam_radius( u, r_0 ):
    return r_0 * np.exp( u ** 2 )

def beam_z( u, m, v, q, I, r_0 ):
    coeff = np.sqrt( m * v**3 / q / I ) * r_0
    subint = lambda t: np.exp( t * t )
    low_lim = 0
    up_lim = u
    integral_value = scipy.integrate.quad( subint, low_lim, up_lim )[0]
    return coeff * integral_value

beam_axis_x_pos, emitter_z_pos, r_0 = get_source_geometry( h5file )
I = get_source_current( h5file )
m, q, p = get_source_particle_parameters( h5file )
v = p / m

u_min = 0; u_max = 2; num_u_points = 100  # for u = 1, r = r(0) * 2.71812
u = np.linspace( u_min, u_max, num_u_points )
r_an = [ beam_radius( x, r_0 ) for x in u ]
r_an_upper = r_an + beam_axis_x_pos
r_an_lower = beam_axis_x_pos - r_an
z_an = [ beam_z( x, m = m, v = v, q = q, I = I, r_0 = r_0 ) for x in u ]
z_an = z_an + emitter_z_pos

r_num = h5file["/Particle_sources/cathode_emitter/position_x"]
z_num = h5file["/Particle_sources/cathode_emitter/position_z"]

z_volume_size = h5file["/Spatial_mesh"].attrs["z_volume_size"]
x_volume_size = h5file["/Spatial_mesh"].attrs["x_volume_size"]
plt.xlabel( "Z [cm]" )
plt.ylabel( "X [cm]" )
plt.ylim( 0, x_volume_size )
plt.xlim( 0, z_volume_size )
plt.plot( z_num, r_num, '.', label = "num" )
plt.plot( z_an, r_an_upper, label = "theory", color = "g" )
plt.plot( z_an, r_an_lower, color = "g" )
plt.legend()
h5file.close()

plt.savefig( "beam_contour_jup.png" )
plt.show()
```

```python

```


```python
# Give some definitions (There is no need to change the following definitions)
import sys
import re
from typing import List, Dict, Union

import sympy as sym
from IPython.display import display
from IPython.display import Math

def is_num(s: str) -> bool:
    try:
        float(s)
    except ValueError:
        return False
    else:
        return True

def set_symbol_from_text(text: str) -> List[sym.Symbol]:
    """Make list of sympy symbols from a text

    Parameters
    ----------
    text : str
        Comma separated words

    Returns
    -------
    symbol_list : List[sym.Symbol]
        List of replaced symbols

    Examples
    -------
    input_variables = r'x_{1}, x_{2}'

    x = set_symbol_from_text(input_variables)
    """
    symbol_list = []
    for term in re.split(',\s*', text):
        symbol_list.append(sym.Symbol(term))
    return symbol_list

def replace_text_with_symbol(
    text: str, c: List[sym.Symbol], x: List[sym.Symbol]
    ) -> str:
    """Make a replaced string with defined sympy symbols

    Parameters
    ----------
    text : str
        original string
    c : List[sym.Symbol]
        List of constants (sympy symbols)
    x: List[sym.symbol]
        List of variables (sympy symbols)

    Returns
    -------
    str
        replaced string
    """
    text = ' ' + text + ' '
    for i, v in enumerate(c):
        rep = r"\1c[{0}]\2".format(i)
        my_regex = r"([^a-zA-Z_0-9])" + re.escape(sym.latex(v)) + r"([^a-zA-Z_0-9])"
        while re.search(my_regex, text) != None:
            text = re.sub(my_regex, rep, text)
    for i, v in enumerate(x):
        rep = r"\1x[{0}]\2".format(i)
        my_regex = r"([^a-zA-Z_0-9])" + re.escape(sym.latex(v)) + r"([^a-zA-Z_0-9])"
        while re.search(my_regex, text) != None:
            text = re.sub(my_regex, rep, text)
    t = text.strip()
    text = t.replace(r"{", "").replace(r"}", "")
    return text
```

```python
# Start scripts
sym.init_printing(use_unicode=True)

# Set output file name
output_filename = "van_der_Pol_event.inp"
# Set constants
input_const = r'\epsilon, \nu_{11}, \nu_{22}'
c = set_symbol_from_text(input_const)
# Set variables
input_var = r"x_{1}, x_{2}"
x = set_symbol_from_text(input_var)
# Set variables for initial positions
input_var_c = r"x_{1}^{\mathrm{c}}, x_{2}^{\mathrm{c}}"
xc = set_symbol_from_text(input_var_c)
# Display inputs for check
display('Defined constants')
display(Math(sym.latex(c)))
display('Defined variables')
display(Math(sym.latex(x)))
display('Defined constants for the initial positions')
display(Math(sym.latex(xc)))
# Add the list for 'xc' to the list 'c'
c = c + xc
```

    'Defined constants'

$\displaystyle \left[ \epsilon, \  \nu_{11}, \  \nu_{22}\right]$

    'Defined variables'

$\displaystyle \left[ x_{1}, \  x_{2}\right]$

    'Defined constants for the initial positions'

$\displaystyle \left[ x_{1}^{\mathrm{c}}, \  x_{2}^{\mathrm{c}}\right]$

```python
# Set equations
# Variables, drift and diffusion coefficients must be enclosed in [].
Eqs = []
Eqs.append(r"d[x_{1}] = [x_{2}]*dt + [\nu_{11}]*d W_{1}")
Eqs.append(r"d[x_{2}] =" \
    + r"[\epsilon*(1-x_{1}**2)*x_{2} - x_{1}]*dt + [\nu_{22}]*d W_{2}")

# Extract strings for drift and diffusion
str_drift = []
str_diff = []
for eq in Eqs:
    result = re.findall(r'\[(.*?)\]', eq)
    if(len(result) != 3):
        print("The format of equation is not adequate: {0}".format(eq))
        sys.exit(1)
    str_drift.append(result[1])
    str_diff.append(result[2])

# Convert strings to sympy
drift = []
for ex in str_drift:
    drift.append(eval(replace_text_with_symbol(ex,c,x)))
drift = sym.Matrix(len(drift), 1, drift)
diff_vector = []
for ex in str_diff:
    diff_vector.append(eval(replace_text_with_symbol(ex,c,x)))
diff = []
for i, variable in enumerate(x):
    tmp_array = [0.0] * len(x)
    tmp_array[i] = diff_vector[i]
    diff.append(tmp_array)
diff = sym.Matrix(diff)

# Display input SDEs for check
latex_bold_x = sym.Symbol('\mathbf{x}')
latex_bold_W = sym.Symbol('\mathbf{W}')
display('Original stochastic differential equations')
for i in range(len(x)):
    latex_W = sym.Symbol('W_{0}'.format(i+1))
    print_sde = Math(r'd {0}(t) = \left({1}\right) dt + \left({2}\right) d {3}(t)'\
        .format(x[i], sym.latex(drift[i]), sym.latex(diff[i,i]), latex_W))
    display(print_sde)
```

    'Original stochastic differential equations'

$\displaystyle d x_{1}(t) = \left(x_{2}\right) dt + \left(\nu_{11}\right) d W_1(t)$

$\displaystyle d x_{2}(t) = \left(\epsilon x_{2} \left(1 - x_{1}^{2}\right) - x_{1}\right) dt + \left(\nu_{22}\right) d W_2(t)$

```python
# Make sympy symbols for partial derivatives
deriv = []
for tmp_x in x:
    deriv.append(sym.Symbol('\\partial_{{{0}}}'.format(tmp_x)))

# Display the derived SDEs
print_sde = Math(r'd{0}(t) = {1} dt + {2} d {3}(t)'\
    .format(latex_bold_x, sym.latex(drift),
    sym.latex(diff), latex_bold_W))
display('Extended stochastic differential equations')
display(print_sde)
```

    'Extended stochastic differential equations'

$\displaystyle d\mathbf{x}(t) = \left[\begin{matrix}x_{2}\\\epsilon x_{2} \left(1 - x_{1}^{2}\right) - x_{1}\end{matrix}\right] dt + \left[\begin{matrix}\nu_{11} & 0\\0 & \nu_{22}\end{matrix}\right] d \mathbf{W}(t)$

```python
# Make sympy symbols for partial derivatives
deriv = []
for tmp_x in x:
    deriv.append(sym.Symbol('\\partial_{{{0}}}'.format(tmp_x)))

# Derive the adjoint operator (backward Kolmogrov) and display it
adj_L = 0
B = diff * diff.transpose()

deriv = sym.Matrix([deriv])
latex_adj_L = sym.Symbol('\mathcal{L}^{\dagger}')
print_adj_L = ""
drift_terms = []
drift_derivs = []
for dri, der in zip(drift,deriv): # 1st order
    drift_terms.append(dri)
    drift_derivs.append(der)
    adj_L = adj_L + dri*der
    print_adj_L = print_adj_L \
        + '\\left({0}\\right) {1}'\
            .format(sym.latex(dri), sym.latex(der)) \
        + '+'
diff_terms = []
diff_derivs = []
for i in range(len(x)): # 2nd order
    for j in range(len(x)):
        if B[len(x)*i+j] != 0:
            diff_terms.append(0.5*B[len(x)*i+j])
            diff_derivs.append(deriv[i]*deriv[j])
            adj_L = adj_L + 0.5*B[len(x)*i+j]*deriv[i]*deriv[j]
            print_adj_L = print_adj_L \
             + '\\frac{{1}}{{2}}\\left({0}\\right) {1}{2}'\
                  .format(sym.latex(B[len(x)*i+j]), \
                          sym.latex(deriv[i]), \
                           sym.latex(deriv[j])
                          ) \
             + '+'
print_adj_L = print_adj_L[:-1]  # Remove the final plus sign
print_dual = Math(r'{0} = {1}'.format(latex_adj_L, print_adj_L))
display('Derived adjoint operator')
display(print_dual)

```

    'Derived adjoint operator'

$\displaystyle \mathcal{L}^{\dagger} = \left(x_{2}\right) \partial_{x_{1}}+\left(\epsilon x_{2} \left(1 - x_{1}^{2}\right) - x_{1}\right) \partial_{x_{2}}+\frac{1}{2}\left(\nu_{11}^{2}\right) \partial_{x_{1}}\partial_{x_{1}}+\frac{1}{2}\left(\nu_{22}^{2}\right) \partial_{x_{2}}\partial_{x_{2}}$

```python
# Apply variable transformations for coordinate shifts
for i, v in enumerate(x):
    for j, term in enumerate(drift_terms):
        drift_terms[j] = drift_terms[j].subs(v, v+xc[i])
    for j, term in enumerate(diff_terms):
        diff_terms[j] = diff_terms[j].subs(v, v+xc[i])

# Derive the adjoint operator (backward Kolmogrov) and display it
adj_L = 0
latex_adj_L = sym.Symbol('\mathcal{L}^{\dagger}')
print_adj_L = ""
for dri, der in zip(drift_terms,drift_derivs): # 1st order
    adj_L = adj_L + dri*der
    print_adj_L = print_adj_L \
        + '\\left({0}\\right) {1}'\
            .format(sym.latex(dri), sym.latex(der)) \
        + '+'
for diff, der in zip(diff_terms,diff_derivs): # 2nd order
    adj_L = adj_L + diff*der
    print_adj_L = print_adj_L \
        + '\\left({0}\\right) {1}'\
            .format(sym.latex(diff), sym.latex(der)) \
        + '+'

print_adj_L = print_adj_L[:-1]  # Remove the final plus sign
print_dual = Math(r'{0} = {1}'.format(latex_adj_L, print_adj_L))
display('Derived adjoint operator')
display(print_dual)
```

    'Derived adjoint operator'

$\displaystyle \mathcal{L}^{\dagger} = \left(x_{2} + x_{2}^{\mathrm{c}}\right) \partial_{x_{1}}+\left(\epsilon \left(1 - \left(x_{1} + x_{1}^{\mathrm{c}}\right)^{2}\right) \left(x_{2} + x_{2}^{\mathrm{c}}\right) - x_{1} - x_{1}^{\mathrm{c}}\right) \partial_{x_{2}}+\left(0.5 \nu_{11}^{2}\right) \partial_{x_{1}}^{2}+\left(0.5 \nu_{22}^{2}\right) \partial_{x_{2}}^{2}$

```python
# Display constants (again) for setting values
display('Defined constants (again)')
display(Math(sym.latex(c)))
```

    'Defined constants (again)'

$\displaystyle \left[ \epsilon, \  \nu_{11}, \  \nu_{22}, \  x_{1}^{\mathrm{c}}, \  x_{2}^{\mathrm{c}}\right]$

```python
# Print degrees for each term
adj_L = sym.expand(adj_L)
variables = []
variables.extend(x)
variables.extend(deriv)
variables.extend(c)
zero_list = [0]*len(x)

# Print terms (for check)
output = "# [State changes] / rate / [Indices for rate] / [Indices for constants] # term\n"
result = []
for t in adj_L.args:
    degree_list = list(sym.degree_list(t, gens=variables))
    change_list = []
    coeff_list = []
    const_list = []
    for i in range(len(x)):
        change_list.append(degree_list[i]-degree_list[i+len(x)])
        coeff_list.append(degree_list[i+len(x)])
    const_list = const_list + degree_list[2*len(x):]
    is_constant = all(elem == 0 for elem in degree_list)
    if is_constant == True:
        result.append([change_list, float(t), coeff_list, const_list, sym.latex(t)])
    else:
        result.append([change_list, float(sym.LC(t)), coeff_list, const_list, sym.latex(t)])
result = sorted(result)
for item in result:
    str0 = "{0}".format(item[0]).replace(' ', '')
    str1 = "{0}".format(item[1])
    str2 = "{0}".format(item[2]).replace(' ', '')
    str3 = "{0}".format(item[3]).replace(' ', '')
    str4 = "{0}".format(item[4])
    output = output + "{0} / {1} / {2} / {3} # {4}\n".format(str0, str1, str2, str3, str4)

state_change_tmp = [item[0] for item in result]
state_change = []
[x for x in state_change_tmp if x not in state_change and not state_change.append(x)]

# Add the basic information
output = "{0} {1} {2} {3}\n".format(len(adj_L.args), len(state_change), len(x), len(c)) + output
output = "# #terms #terms_without_duplicate #x #const / Constants: {0}\n".format(c) + output

with open(output_filename, mode='w') as f:
    f.write(output)

# For check
print(output)
```

    # #terms #terms_without_duplicate #x #const / Constants: [\epsilon, \nu_{11}, \nu_{22}, x_{1}^{\mathrm{c}}, x_{2}^{\mathrm{c}}]
    14 10 2 5
    # [State changes] / rate / [Indices for rate] / [Indices for constants] # term
    [-2,0] / 0.5 / [2,0] / [0,2,0,0,0] # 0.5 \nu_{11}^{2} \partial_{x_{1}}^{2}
    [-1,0] / 1.0 / [1,0] / [0,0,0,0,1] # \partial_{x_{1}} x_{2}^{\mathrm{c}}
    [-1,1] / 1.0 / [1,0] / [0,0,0,0,0] # \partial_{x_{1}} x_{2}
    [0,-2] / 0.5 / [0,2] / [0,0,2,0,0] # 0.5 \nu_{22}^{2} \partial_{x_{2}}^{2}
    [0,-1] / -1.0 / [0,1] / [0,0,0,1,0] # - \partial_{x_{2}} x_{1}^{\mathrm{c}}
    [0,-1] / -1.0 / [0,1] / [1,0,0,2,1] # - \epsilon \partial_{x_{2}} \left(x_{1}^{\mathrm{c}}\right)^{2} x_{2}^{\mathrm{c}}
    [0,-1] / 1.0 / [0,1] / [1,0,0,0,1] # \epsilon \partial_{x_{2}} x_{2}^{\mathrm{c}}
    [0,0] / -1.0 / [0,1] / [1,0,0,2,0] # - \epsilon \partial_{x_{2}} \left(x_{1}^{\mathrm{c}}\right)^{2} x_{2}
    [0,0] / 1.0 / [0,1] / [1,0,0,0,0] # \epsilon \partial_{x_{2}} x_{2}
    [1,-1] / -2.0 / [0,1] / [1,0,0,1,1] # - 2 \epsilon \partial_{x_{2}} x_{1} x_{1}^{\mathrm{c}} x_{2}^{\mathrm{c}}
    [1,-1] / -1.0 / [0,1] / [0,0,0,0,0] # - \partial_{x_{2}} x_{1}
    [1,0] / -2.0 / [0,1] / [1,0,0,1,0] # - 2 \epsilon \partial_{x_{2}} x_{1} x_{1}^{\mathrm{c}} x_{2}
    [2,-1] / -1.0 / [0,1] / [1,0,0,0,1] # - \epsilon \partial_{x_{2}} x_{1}^{2} x_{2}^{\mathrm{c}}
    [2,0] / -1.0 / [0,1] / [1,0,0,0,0] # - \epsilon \partial_{x_{2}} x_{1}^{2} x_{2}

```python

```