```python
import numpy as np
import matplotlib.pylab as plt
import pandas as pd
```

# Numerical Solution of ODEs

Consider a first order differential equation of the form

$$y'=\frac{dy}{dx}=f(x, y)\, , \qquad y(x=0)=y_0 \tag{1}$$

We want to find the function $y(x)$ that describes the evolution of this quantity in function of $x$.
Let's assume the value of $y(x_0)$ at a specific point $x=x_0$ is known;  $y(x=x_0)$ is called the *initial value*.

The task of finding function $y(x)$, given its derivative and initial value as in Eq. (1), is called an *initial value problem*.

Recall the definition of the continuous derivative of an analytic function $f(x)$:

$$y'(x_0)=\lim_{\Delta x\rightarrow 0}\frac{y(x_0+\Delta x)-y(x_0)}{\Delta x} =f(x_0, y)\; . \tag{2}$$

## Numerical Integration Methods

We approximate Eq (2) by finite differences:

$$y'(x)\approx \frac{y(x_0+\Delta x)-y(x_0)}{\Delta x} =f(x_0, y)\; , \tag{3}$$

which allows us to express $y$ at a neighbouring point $x=x_0+\Delta x$ in terms of its value at the current position $x_0$ and its derivative at this point:

$$y(x_0+\Delta x) = y(x_0) + y'(x)\, \Delta x = y(x_0) + f(x_0, y(x_0))\, \Delta x  \tag{4} \, .$$

We can make the following geometric analogy:

Consider the problem of calculating the shape of a curve $y(x)$ which starts at a given point $y(x_0)$and satisfies the differential equation (1).
The differential equation can be thought of as a formula that describes the slope of the tangent line to the curve $y$ at any point $x_i$, provided that the position of that point on the curve $y(x_i)$ is known.
Given a starting point $x_0$, we can compute the next point on the curve $y(x_0 + \Delta x)=y(x_1)$ using Eq. (4). This procedure can be repeated until the end of the domain is reached.

In this course we are mainly interested in the **time**-evolution of biological systems.
Therefore, for our purpose, the variable $x$ in (1) and (2) will be time $t$ so that (4) becomes:

$$y(t_0+\Delta t) = y(t_0) + y'(t)\, \Delta t = y(t_0) + f(t_0, y(t_0))\, \Delta t  \tag{5} \, .$$

### Euler Method (explicit)

Assume that we wish to approximate the unknown function $y(t)$ in the interval $[t_{\text{min}}, t_\text{max}]$ which is discretized into $N$ subintervals with $t_{\text{min}}=t_0 < t_1 < t_2 < \ldots <t_{N-1} < t_{N} =t_{\text{max}}$.
We define the *step size* between subsequent points as $\Delta_i = t_{i+1}-t_i$,and
write Eq. (4) as a *recursive* scheme:

$$ y_{i+1} = y_i + \Delta_i\, f(t_i, y_i) \tag{5}$$

This is called the **Euler method**.
The Euler method is an *explicit* integration method because any $y_{i+1}$ is an explicit function of the 'previous' $y_i$ for $i\leq N-1$:

\begin{align}
y_1 &= y_0 + \Delta_0 \, f(t_0, y_0) \\
y_2 &= y_1 + \Delta_1 \, f(t_1, y_1) \\
& \dots \\
y_{N-1} &= y_{N-2} + \Delta_{N-2} \, f(t_{N-2}, y_{N-2}) \\
y_{N} &= y_{N-1} + \Delta_{N-1} \, f(t_{N-1}, y_{N-1}) \\
\end{align}

If the evaluation points are spaced equally, $\Delta_i = \Delta = \left| t_{\text{max}}-t_\text{min}\right|/N$.

The following example illustrates how this method approximates $y(t)$.

```python
#=================================================
# Approximate function using Euler's method
#
# Explore effect of varying 'delta_t'
#=================================================
delta_t = 0.2
t_0 = 0
y_0 = 1

# function defs & preparation
from cycler import cycler
colors = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728',
          '#9467bd', '#8c564b', '#e377c2', '#7f7f7f',
          '#bcbd22', '#17becf']
color_cyler = cycler(color=colors)

def f_y(y_0, t):
    return y_0*np.exp(2*t)

def f_yp(y, t):
    return 2*y

def plot_deriv(ax, x, y, yp, delta_x=0.1, **kwargs):
    x_plot = np.array([x, x+delta_x])
    y_plot = np.array([y, y+yp*delta_x])
    ax.plot(x_plot[1], y_plot[1], '.k')
    ax.plot(x_plot, y_plot, linestyle = '--', **kwargs)

cm = plt.get_cmap('viridis_r')
n_steps = int(1/delta_t)
cgen = (cm(1.*i/n_steps) for i in range(n_steps))
color_cyler = cycler(color=cgen)

# plot (unknown) function
t = np.arange(t_0,1,0.05)
y = f_y(y_0, t)
fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111)
ax.plot(t, y, ':', label='y (unknown)')

# plot approximation
t_i = t_0
y_i = y_0
ax.plot(t_i, y_i, 'sk', label="y(t=%.2f)"%t_i)
for i, kwargs in zip(range(1,n_steps), color_cyler):
    plot_deriv(ax, t_i, y_i, f_yp(y_i, t_i), delta_x=delta_t, **kwargs)
    t_next = t_i + delta_t
    y_next = y_i + delta_t*f_yp(y_i, t_i)
    ax.plot(t_next, y_next, marker='s', **kwargs)
    t_i = t_next
    y_i = y_next
ax.legend()
```

---
**Exercise (1):**

1. Write a function that computes:
  $$\frac{d y(t)}{d t} = f(t,y)= y\, t\, .$$

2. Write a function that implements the Euler Method.
  This function should accept the following arguments:

  - *f* : a function that computes $\frac{dy}{dt}$ in function of $y$ and $t$, i.e. $f(t,y)$ above.
  - *t*: array of points (evaluation time points) at which y(t) should be estimated
  - *y_0*: initial value at $t=0$

---

```python
def dydt(t, y):
  """
  Args:
    - t: scalar, usually time
    - y: scalar, function value

  Returns:
    - scalar, derivative dy/dt of (unknown) function y(t)
  """
  return y*t

def solve_euler(f, t, y_0):
  """
  Uses explicit euler method to solve ODE: y'=f(t, y)
  with initial value y(t_0)=y_0.

  Args:
    - f: Function object that computes y', expected signature f(t, y),
         where t is evaluation point and y is function value
    - t: array of evaluation points
    - y_0: initial value, i.e. y(t[0])

  Returns:
    - array containing approximated function values y(t[i])
  """
  y = np.zeros_like(t)
  y[0] = y_0
  for i in range(0, len(t)-1):
    y[i+1] = y[i] + (t[i+1]-t[i]) * f(t[i],y[i])
  return y

```

---
**Exercise (2):**

1. Use this function to solve
$$\frac{dy}{dt}=r\, y \qquad y(t_0=0)=y_0=0$$
in the interval $t\in [0, 10]$, with constant $r=0.5$.

2. Compare your approximate solutions to the analytic solution for different choices of integration step size:
$$y(t) = y_0\,  e^{r\, t}$$

  What step size do you need for a good approximation?

3. Solve above equation for $r=-2.5$ and compare to analytic solution. What  do you observe at large step sizes $\Delta t \gtrapprox 0.5$.

---

*Note regarding function arguments*:

The ODE in above example describes an *autonomous system*:
$$\frac{d y }{dt} = f(y(t), t) = f(y(t))$$
that does not explicitly depend on the independent variable, time in this case.

The computational function `f()` for computing the value $\frac{dy}{dt}$ for a given $y$ therefore does not require $t$ as an argument.
However, the implementation of the ODE solver function should be sufficiently generic so that it can be applied to autonomous as well as non-autonomous (with time dependency) systems.

Standard ODE solvers therefore expect the function `f()` that computes the derivative  to accept two arguments, the current (estimated) value $y_i$ and the current time $t$.
Different conventions exist for the order in which those arguments are expected, either `f(y, t)` or `f(t, y)` respectively.
The order and type of the arguments of a function is called the *signature* of the function.
We use the second option `f(t, y)` throughout this notebook, i.e. a function signature that has time as first argument, and function values as second argument.
Also, we will include both arguments even when describing an autonomous system.

```python
# define functions
def dydt_1(t, y):
  return 0.5*y

# analytic:
def fun_an_1(t, y_0):
  return y_0*np.exp(0.5*t)

# define functions
def dydt_2(t, y):
  return -2.5*y

# analytic:
def fun_an_2(t, y_0):
  return y_0*np.exp(-2.5*t)

# domain bounds
t_0 = 0
t_N = 10

# initial value
y_0 = 1

# create time data for numeric approximation
n_steps = 15
t = np.linspace(t_0, t_N, n_steps)
step_size = (t_N-t_0)/n_steps
#print("step size: ",step_size)

# solve ODE
y_1 = solve_euler(f=dydt_1, t=t, y_0=y_0)
y_2 = solve_euler(f=dydt_2, t=t, y_0=y_0)

# create data for analytic function on fine grid
t_an = np.linspace(t_0, t_N, 100)
y_an_1 = fun_an_1(t_an, y_0)
y_an_2 = fun_an_2(t_an, y_0)

# plot
fig, axes= plt.subplots(1, 2, figsize=plt.figaspect(0.3))
axes[0].plot(t, y_1, label='numeric approximation of $y(t)$')
axes[0].plot(t_an, y_an_1, label='y(t)')
axes[0].set_ylabel("y(t)")
axes[0].set_xlabel("t")
axes[0].set_title("function (1): $r=0.5$; stepsize $\Delta t = %.2f$"%step_size)
axes[0].legend()

axes[1].plot(t, y_2, label='numeric approximation of $y(t)$')
axes[1].plot(t_an, y_an_2, label='y(t)')
axes[1].set_ylabel("y(t)")
axes[1].set_xlabel("t")
axes[1].set_title("function (2): $r=-2.5$; stepsize $\Delta t = %.2f$"%step_size)
axes[1].legend()

plt.show()
```

In the first of above examples ($r=0.5$), we noticed that the numeric approximation underestimates the values of the actual function $y(t)$ for stepsizes $\Delta t \gtrapprox 0.01$.
In the second example ($r=-2.5$), we saw that for step sizes $\Delta t \gtrapprox 0.5$, the numeric approximation may become *unstable*, assuming very large values or oscillating around the actual function value.

To ensure convergence with the explicit Euler method, very small stepsizes must be used. Oscillating function values can be numeric artefacts and a sign of numeric instability!

#### Error and Convergence

The Euler method is an iterative method. In each iteration step, it estimates the function value $y_i$ using a finite difference approximation, Eq. (4), that involves previous (estimated) values $y_{i-1}$.
As we have discussed in the notebook on [numeric derivatives](https://github.com/cohmathonc/biosci670/blob/master/IntroductionComputationalMethods/03_IntroCompMethods_NumericalDifferentiation.ipynb), this approximation involves a *truncation error* because it neglects contributions of higher function derivatives.

The truncation error that is made in iteration step $i+1$, when there was no error in step $i$, is called *local truncation error* (LTE).
To compute the LTE, we compare the Taylor expansion of $y(t+\Delta t)$ to the iterative formula of the Euler method, Eq (5):
\begin{align}
 y(t_i+\Delta t) &= y(t_i) + \Delta t\, y'(t_i) + \frac{1}{2}\Delta t^2\, y''(\tau) + \mathcal{O}(\Delta t^3) \tag{6a}\\
 y_{i+1} &= y_i + \Delta t \underbrace{f(t_i, y_i)}_{y'} \tag{6b}
\end{align}

where we assume that no error has been made in previous steps, i.e. that the estimate $y_i$ is identical to the actual function value $y(t_i)$. The difference between (6a) and (6b) corresponds to the LTE:
$$\text{LTE}:\qquad \epsilon^{\text{local}}_i = \frac{1}{2}\Delta t^2\, y''(\tau) + \mathcal{O}(\Delta t^3) \tag{7}$$

In an iterative method, errors accumulate with each additional step. The cumulative effect of these errors is called the *global truncation error* (GTE). The GTE is the error at a fixed time $t$ and after however many steps the method needs to take to reach this time from an initial time point $t_0$:
$$\text{GTE}:\qquad \epsilon^{\text{global}}_i = y(t_i) - y_i$$

For a given integration interval $[t_0, t_\text{max}]$, the number of steps is proportional to $\frac{1}{\Delta t}$, and it can be shown that the GTE is proportional to $\Delta t$. The Euler method is therefore a first order method.

---
**Exercise (3):**

Investigate the convergence behavior of the Euler method by computing the approximation error
$$\epsilon_i = y(t_i) - y_i$$
- at the very first iteration ($i=1$) step to obtain an estimate of the LTE (first step, no accumulation of errors) and
- at some later evaluation step to obtain an estimate of the GTE.

Repeat this computation for multiple values of stepsize $\Delta t$, halfing the stepsize each iteration step.
Plot the error terms in function of stepsize $\Delta t$ and compute the convergence order.

---

```python
# define functions for common analysis steps

def compute_error(f_yt, solver, solution, t_0, t_N, y_0):
  """
  Computes cumulative error between numeric approximation and analytic solution
  to ODE.

  Args:
    - f_yt: Function object that computes y', expected signature f(t, y),
            where t is evaluation point and y is function value
    - solver: ODE solver, expected signature: solver(f_yt, t, y_0)
            where t is array of time points.
    - solution: Function object that computes value y(t) for analytic solution.
            Expected signature: solution(t, y_0)
    - t_0, t_N: endpoints of integration interval
    - y_0: initial value, i.e. y(t[0])

  Returns:
    - pandas DataFrame containing 'lte' and 'gte' for different step sizes 'delta_t'
  """
  df = pd.DataFrame()
  for i in range(0,20):
      n_steps = 1 * (2**i)
      delta_t = (t_N - t_0)/n_steps
      t = np.linspace(t_0, t_N, n_steps)
      y_approx = solver(f_yt, t, y_0)
      y_exact  = solution(t, y_0)
      error_cum = np.sum(np.abs(y_exact - y_approx))/n_steps
      # keep record of parameters
      df.loc[i, 'n_steps'] = n_steps
      df.loc[i, 'delta_t'] = delta_t
      if n_steps > 2:
          ref_step = n_steps - 1
          error_gte = np.abs(y_exact[ref_step] - y_approx[ref_step])
          df.loc[i, 'gte'] = error_gte
      if n_steps > 1:
          error_lte = np.abs(y_exact[1] - y_approx[1])
          df.loc[i, 'lte'] = error_lte
  return df

def plot_error(df, method):
  """
  Plots cumulative error vs step size in log-log scale.

  Args:
    - df: pandas dataframe object with columns 'delta_t', 'lte', 'gte'
    - method: the name of the method that is being plotted
  """
  fig = plt.figure(figsize=plt.figaspect(0.5))
  ax = fig.add_subplot(111)

  ax.loglog(df.delta_t.values, df.gte.values, marker='o',
            linestyle='-', label='GTE')
  ax.loglog(df.delta_t.values, df.lte.values, marker='o',
            linestyle='-', label='LTE')
  ax.set_xlabel('$\Delta_i$')
  ax.set_ylabel("absolute error")
  ax.set_title('Approximation Error in %s'%method)
  ax.legend()
  plt.show()

def compute_convergence_order(eps):
  """
  Computes ratio of errors steps: ratio(n) = error(n-1)/error(n)
  """
  eps_shifted = np.roll(eps, shift=1)
  eps_shifted[0] = np.nan
  eps_ratio = eps_shifted / eps
  convergence_order = np.log2(eps_ratio)
  return convergence_order
```

```python
# ODE
def dydt(t, y):
  return y

# solution analytic
def fun_an(t, y_0):
  return y_0*np.exp(t)

# domain bounds
t_0 = 0
t_N = 10

# initial value
y_0 = 1

# compute error for different number of integration steps
df_euler = compute_error(dydt, solve_euler, fun_an, t_0, t_N, y_0)

# plot
plot_error(df_euler, 'euler method')

# convergence order
print("Convergence order of explicit Euler method (LTE) ",
     compute_convergence_order(df_euler.lte.values))
print("Convergence order of explicit Euler method (GTE) ",
     compute_convergence_order(df_euler.gte.values))
```

### Euler Method (implicit)

Instead of formulating the Euler method as an *explicit* iterative scheme, it can also be formulated as an *implicit* method:
$$ y_{i+1} = y_i + \Delta_i\, f(x_{i+1}, y_{i+1}) \tag{8}$$

 where the new appproximated value $y_{i+1}$ appears on both sides of the equation.

 This method is more stable than the explicit Euler method, but computationally more expensive because an algebraic equation needs to be solved in each time step for the unknown $y_{i+1}$.
 Like the explicit Euler method, the implicit method is of first order.

```python
def solve_euler_implicit(f, t, y_0):
  """
  Uses implicit euler method to solve ODE: y'=f(t, y)
  with initial value y(t_0)=y_0.

  Args:
    - f: Function object that computes y', expected signature f(t, y),
         where t is evaluation point and y is function value
    - t: array of evaluation points
    - y_0: initial value, i.e. y(t[0])

  Returns:
    - array containing approximated function values y(t[i])
  """
  y = np.zeros_like(t)
  y[0] = y_0
  for i in range(0, len(t)-1):
    delta_t = t[i+1] - t[i]
    yp = f(t[i+1], y[i]) / (1+delta_t)
    y[i+1] = y[i] + delta_t * yp
  return y
```

```python
# compute error for different number of integration steps
df_euler_implicit = compute_error(dydt, solve_euler_implicit,
                                  fun_an, t_0, t_N, y_0)

# plot
plot_error(df_euler_implicit, 'euler method (implicit)')

# convergence order
print("Convergence order of implicit Euler method (LTE) ",
     compute_convergence_order(df_euler_implicit.lte.values))
# convergence order
print("Convergence order of implicit Euler method (GTE): ",
     compute_convergence_order(df_euler_implicit.gte.values))
```

### Modified Euler Method

The Euler method tries to predict the next point on the curve $y(t_{i+1})$ from the slope of the curve at an earlier evaluation point. If the curve is convex, the estimated value $y(t_{i+1})$ will be lower than the actual value at $t_{i+1}$, if it is concave, the estimated value wil be higher than the actual value.

Instead of using the slope of $y(t_i)$ at $t_i$ or $t_{i+1}$, the *modified Euler methods* try to use better approximations of the slope.
The so-called *Heun's method* follows an approach similar to the *trapezoidal rule*:
It uses an 'optimal slope' that would connect the points $y(t_i)$ and $y(t_{i+1})$ and which equals the average of the slopes at $y(t_i)$ and $y(t_{i+1})$:
$$\text{average slope} = \frac{\Delta y}{\Delta t}=\frac{y_{i+1}-y_i}{t_{i+1}-t_i}=\frac{1}{2}\left(f(t_i, y_i) + f(t_{i+1}, y_{i+1})\right)$$

And therefore

$$ y_{i+1} = y_i + \text{average slope} \cdot \Delta t = y_i + \frac{1}{2}\Delta t\left(f(t_i, y_i) + f(t_{i+1}, \tilde{y}_{i+1})\right) \, ,$$

with the intermediate value

$$\tilde{y}_{i+1}= y_i + \Delta t\, f(t_i, y_i)$$

A similar approach, based on the *midpoint rule* exists.
Both methods are of second order accuracy.

```python
def solve_euler_modified(f, t, y_0):
  """
  Uses explicit euler method to solve ODE: y'=f(t, y)
  with initial value y(t_0)=y_0.

  Args:
    - f: Function object that computes y', expected signature f(t, y),
         where t is evaluation point and y is function value
    - t: array of evaluation points
    - y_0: initial value, i.e. y(t[0])

  Returns:
    - array containing approximated function values y(t[i])
  """

  y = np.zeros_like(t)
  y[0] = y_0
  for i in range(0, len(t)-1):
    y_int  = y[i] + (t[i+1]-t[i]) * f(t[i],y[i])
    y[i+1] = y[i] + 0.5*(t[i+1]-t[i]) * ( f(t[i],y[i]) + f(t[i+1],y_int) )
  return y
```

```python
# compute error for different number of integration steps
df_euler_mod = compute_error(dydt, solve_euler_modified,
                                  fun_an, t_0, t_N, y_0)

# plot
plot_error(df_euler_mod, 'modified euler method')

# convergence order
print("Convergence order of modified Euler method (GTE): ",
     compute_convergence_order(df_euler_mod.gte.values))
```

### Runge-Kutta Methods

The Euler and modified Euler methods above are specific cases of a  familiy of iterative methods used for approximating solutions to ODEs.
These methods are called [*Runge-Kutta*](https://en.wikipedia.org/wiki/Runge–Kutta_methods) (RK) methods.
Explicit and implicit Euler methods are *one stage* RK methods. The modified Euler methods are *two stage* RK methods.
In general, RK methods use the weighted average of function values evaluated at multiple increments (*stages*) to approximate the next value $y(t_{i+1})$.
The 'classical' Runge Kutta Method is RK4 which employs four stages for approximation.

\begin{align}
y_{i+1} &= y_i + \frac{1}{6} \left( k_1 + 2 k_2, +2k_3 + k_4\right)\\
t_{i+1} &= t_i + \Delta t
\end{align}

where

\begin{align}
k_1 &= \Delta t \, f(t_i, y_i)\\
k_2 &= \Delta t \, f(t_i+\frac{\Delta t}{2}, y_i + \frac{k_1}{2})\\
k_3 &= \Delta t \, f(t_i+\frac{\Delta t}{2}, y_i + \frac{k_2}{2})\\
k_4 &= \Delta t \, f(t_i+\Delta t, y_i + k_3)\\
\end{align}

```python
def solve_RK4(f, t, y_0):
  """
  Uses explicit euler method to solve ODE: y'=f(t, y)
  with initial value y(t_0)=y_0.

  Args:
    - f: Function object that computes y', expected signature f(t, y),
         where t is evaluation point and y is function value
    - t: array of evaluation points
    - y_0: initial value, i.e. y(t[0])

  Returns:
    - array containing approximated function values y(t[i])
  """

  y = np.zeros_like(t)
  y[0] = y_0
  for i in range(0, len(t)-1):
    delta_t = t[i+1]-t[i]
    k_1  = delta_t * f(t[i], y[i])
    k_2  = delta_t * f(t[i] + delta_t/2, y[i] + k_1/2)
    k_3  = delta_t * f(t[i] + delta_t/2, y[i] + k_2/2)
    k_4  = delta_t * f(t[i] + delta_t, y[i] + k_3)
    y[i+1] = y[i] + 1/6*(k_1 + 2*k_2 + 2*k_3 + k_4)
  return y
```

```python
# compute error for different number of integration steps
df_RK4 = compute_error(dydt, solve_RK4,
                                  fun_an, t_0, t_N, y_0)

# plot
plot_error(df_RK4, 'RK4 method')

# convergence order
print("Convergence order of RK4 method (GTE): ",
     compute_convergence_order(df_RK4.gte.values))
```

## Adaptive Stepsize

In all examples above, we used a constant step size $\Delta t$ for integrating $f(t,y)$.
However, this may be inefficient.

An *adaptive* integration method finds the appropriate $\Delta t$ to meet a given precision target. In Eq. (7) we introduced the local truncation error LTE. Suppose, you evaluate an estimate for $y_{i+1}$ twice, using a method with n-th order accuracy (LTE $\mathcal{O}(\Delta t^{n+1})$) yielding estimate $y_{i+1}^{(n)}$, and another method with $(n+1)$-th order accuracy (LTE $\mathcal{O}(\Delta t^{n+2})$) , yielding estimate $y_{i+1}^{(n+1)}$.
The error $\epsilon$ between $(n+1)$ and $n$ order methods can then be estimated as:
$$\epsilon = \left| y_{i+1}^{(n+1)} - y_{i+1}^{(n)}\right| = \mathcal{O}(\Delta t^{n+1})$$

Suppose we are working with 4th- and 5th- order Runge Kutta Methods, i.e. $n=4$ and would like to find the optimal stepsize $\Delta \tau$ so that  $\epsilon \leq \delta$ for some *desired accuracy* of $\delta$.
From above we see that the error $\epsilon$ scales with $\Delta t^5$.
Therefore
$$\frac{\delta}{\epsilon} = \left(\frac{\Delta\tau}{\Delta t}\right)^{5}$$
and solving for $\Delta\tau$
$$\Delta\tau = \Delta t\,  \left(\frac{\delta}{\epsilon}\right)^{1/5}$$

If $\epsilon < \delta$, we accept the current estimate $y_{i+1}^{(5)}$ and continue to the next time step, possibly increasing the step size for the next step.
If $\epsilon > \delta$, the current estimate is rejected and we recompute the failed time step using a smaller value of $\Delta \tau$.
You can read more details about this approach in the [numerical recipes](https://aip.scitation.org/doi/pdf/10.1063/1.4823060).

We will not implement this ourselves, but the ODE integration functions provided by `scipy.integrate` use adaptive stepsizes for integration. See examples below.

## Comparison: Runge Kutta 1, 2, 4 stage

Here we compare the different integration methods discussed previously by applying them to the ODEs of exercise (2):
- 1 stage Runge Kutta: explicit & implicit Euler
- 2 stage Runge Kutta: explicit modified Euler
- 4 stage Runge Kutta: 'RK4'

We also include two solvers of the `scipy.integrate` package in this comparison:
- [`scipy.integrate.odeint`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html#scipy.integrate.odeint), based on [ODEPACK LSODA](https://computation.llnl.gov/casc/odepack/) which automatically chooses between methods for *stiff* and *non-stiff* systems.
- [`scipy.integrate.RK45`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.RK45.html#scipy.integrate.RK45), an explicit *adaptive* Runge-Kutta method.

We call latter using the standard interface [`scipy.integrate.solve_ivp`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp) which provides access to multiple ode-integration methods.
This interface returns not only the estimated values of $y_i$ but a results *object* that contains various types of information. You can inspect the contents of this object with `dir()`.

Note that the interfaces `odeint` and `solve_ivp` expect the arguments of the ODE-defining function to be ordered differently.
All examples in this notebook use the function signature $f(t, y)$ as expected by `solve_ivp`.

```python
# we also include two solvers from the scipy package
from scipy.integrate import solve_ivp
from scipy.integrate import odeint

# Function (1)
def dydt_1(t, y):
  return 0.5*y

def fun_an_1(t, y_0):
  return y_0*np.exp(0.5*t)

# Function (2)
def dydt_2(t, y):
  return -2.5*y

def fun_an_2(t, y_0):
  return y_0*np.exp(-2.5*t)

# integration bounds
t_0 = 0
t_N = 10

# initial value
y_0 = 1

# create time data for numeric approximation
n_steps = 30
t = np.linspace(t_0, t_N, n_steps)
step_size = (t_N-t_0)/n_steps
#print("step size: ",step_size)

# create data for analytic function on fine grid
t_an = np.linspace(t_0, t_N, 100)
y_an_f1 = fun_an_1(t_an, y_0)
y_an_f2 = fun_an_2(t_an, y_0)

# solve ODEs
## f1
y_euler_f1 = solve_euler(f=dydt_1, t=t, y_0=y_0)
y_euler_imp_f1 = solve_euler_implicit(f=dydt_1, t=t, y_0=y_0)
y_euler_mod_f1 = solve_euler_modified(f=dydt_1, t=t, y_0=y_0)
y_RK4_f1 = solve_RK4(f=dydt_1, t=t, y_0=y_0)
### scipy solve_ivp
RK45_f1 = solve_ivp(dydt_1, [t_0, t_N], [y_0],
                        method='RK45')
y_RK45_f1 = RK45_f1.y.flatten()
t_RK45_f1 = RK45_f1.t
### scipy odeint
y_odeint_f1 = odeint(dydt_1, y_0, t, tfirst=True)
y_odeint_f1 = np.array(y_odeint_f1).flatten()
## f2
y_euler_f2 = solve_euler(f=dydt_2, t=t, y_0=y_0)
y_euler_imp_f2 = solve_euler_implicit(f=dydt_2, t=t, y_0=y_0)
y_euler_mod_f2 = solve_euler_modified(f=dydt_2, t=t, y_0=y_0)
y_RK4_f2 = solve_RK4(f=dydt_2, t=t, y_0=y_0)
### scipy solve_ivp
RK45_f2 = solve_ivp(dydt_2, [t_0, t_N], [y_0],
                        method='RK45')
y_RK45_f2 = RK45_f2.y.flatten()
t_RK45_f2 = RK45_f2.t
### scipy odeint
y_odeint_f2 = odeint(dydt_2, y_0, t, tfirst=True)
y_odeint_f2 = np.array(y_odeint_f2).flatten()

# plot
fig, axes= plt.subplots(1, 2, figsize=plt.figaspect(0.3))
axes[0].plot(t, y_euler_f1, label='Euler explicit')
axes[0].plot(t, y_euler_imp_f1, label='Euler implicit')
axes[0].plot(t, y_euler_mod_f1, label='Euler modified')
axes[0].plot(t, y_RK4_f1, label='RK4')
axes[0].plot(t_RK45_f1, y_RK45_f1, label='scipy RK45 -- adaptive')
axes[0].plot(t,y_odeint_f1, label='scipy odeint')
axes[0].plot(t_an, y_an_f1, label='analytic $y(t)=y_0\, e^{0.5\,t}$')
axes[0].set_title("function (1): $r=0.5$; stepsize $\Delta t = %.2f$"%step_size)
axes[0].set_ylabel("y(t)")
axes[0].set_xlabel("t")
axes[0].legend()
axes[1].plot(t, y_euler_f2, label='Euler explicit')
axes[1].plot(t, y_euler_imp_f2, label='Euler implicit')
axes[1].plot(t, y_euler_mod_f2, label='Euler modified')
axes[1].plot(t, y_RK4_f2, label='RK4')
axes[1].plot(t_RK45_f2,y_RK45_f2, label='scipy RK45 -- adaptive')
axes[1].plot(t,y_odeint_f2, label='scipy odeint')
axes[1].plot(t_an, y_an_f2, label='analytic $y(t)=y_0\, e^{-2.3\,t}$')
axes[1].set_title("function (2): $r=-2.5$; stepsize $\Delta t = %.2f$"%step_size)
axes[1].set_ylabel("y(t)")
axes[1].set_xlabel("t")
axes[1].legend()
plt.show()
```

## Solving Systems of ODEs

The same methods discussed before can also be used to solve systems of ODE's.

Consider the following example of a system of two ODEs:
\begin{align}
\frac{dx}{dt}&=-x+x^3 &=f(t, x) \\
\frac{dy}{dt}&=-2y & = g(t, y)
\end{align}

For numerical implementation, we first need to write a function that returns the values $f(t, x)$ and $g(t, y)$ for given $x, y, t$.
Ideally, we would like this function to have the same signature (i.e. type and order of arguments) as our previous functions, i.e. one argument for function values, and one for time.
This can be achieved by 'vectorizing' these functions:

Instead of accepting only a single scalar as input for the  current function value `y`, we permit an n-dimensional array as input argument.
Likewise, instead of returning only the derivative $y'$, we return an array of derivative values, one for each ODE in our system of ODEs.

---
**Exercise (4):**

1. 'Vectorize' the function `dydt` from the beginning of this notebook and implement above system of ODEs.

2. Modify the implementation of the explicit Euler method from before to work with this vectorized function.

3. Test your implementation using different combinations of initial values $x_0$, $y_0$.

---

```python
def dydt_ndim(t, y):
  """
  Args:
    - y: array of function values
    - t: scalar, time

  Returns:
    - array of derivatives dy/dt of (unknown) function y(t)
  """
  #print("in: ", y)
  y_out = np.zeros_like(y)
  y_out[0] = -y[0] + np.power(y[0],3)
  y_out[1] = -2*y[1]
  #print("out: ", y_out)
  return y_out

def solve_euler_ndim(f, t, y_0):
  """
  Uses explicit euler method to solve ODE: y'=f(y, t)
  with initial value y(t_0)=y_0.

  Args:
    - f: Function object that computes y', expected signature f(y, t),
         where t is evaluation point and y is function value
    - t: array of evaluation points
    - y_0: array of initial values, i.e. y(t[0])

  Returns:
    - array containing approximated function values y(t[i])
  """
  y = np.zeros((len(y_0),len(t)))
  y[:,0] = y_0
  for i in range(0, len(t)-1):
    y[:,i+1] = y[:,i] + (t[i+1]-t[i]) * f(t[i], y[:,i])
    #print(y[:,i+1])
  return y

```

```python
# domain bounds
t_0 = 0
t_N = 10

# initial value
y_0 = [0.5,1]

# create time data for numeric approximation
n_steps = 1000
t = np.linspace(t_0, t_N, n_steps)
step_size = (t_N-t_0)/n_steps
print("step size: ",step_size)

# solve ODE
y_euler_ndim = solve_euler_ndim(f=dydt_ndim, t=t, y_0=y_0)

# plot
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(111)
ax.plot(t, y_euler_ndim[0,:], label='x(t)')
ax.plot(t, y_euler_ndim[1,:], label='y(t)')
ax.set_xlabel("t")
plt.legend()
plt.show()
```

# Useful Python / Scipy Functions

```python
from scipy.integrate import odeint

# define functions
def dydt(y, t): # note that function signature differs from our convention
  return -2.3*y

# domain bounds
t_0 = 0
t_N = 10

# initial value
y_0 = 1

# create time data for numeric approximation
n_steps = 10
t = np.linspace(t_0, t_N, n_steps)

# expects dy/dx to have calling signature: fun(y, t) !
sol = odeint(dydt, y_0, t)
print("output: ", sol)
sol = np.array(sol).flatten()      # reshape output format from odeint
print("'flattened' output: ", sol)
```

```python
from scipy.integrate import solve_ivp

# define functions
def dydt(t, y):
  return -2.3*y

# domain bounds
t_0 = 0
t_N = 10

# initial value
y_0 = 1

# automatic time-step size: initial, min, max step size can be defined
# expects dy/dx to have calling signature: fun(t, y) !
sol = solve_ivp(dydt, [t_0, t_N], [y_0])   # supports different integration methods, default is 'RK45'
print("results object: ",sol)

```

## Exercises

- In [this](https://github.com/cohmathonc/biosci670/blob/master/IntroductionComputationalMethods/exercises/07_ODEs.ipynb) exercise you use the explicit Euler method (and optionally also other methods) to solve an initial value problem.

###### About
This notebook is part of the *biosci670* course on *Mathematical Modeling and Methods for Biomedical Science*.
See https://github.com/cohmathonc/biosci670 for more information and material.


[Home Page](../../Start_Here.ipynb)

[Previous Notebook](../introduction/Introductory_Notebook.ipynb)

[1](../introduction/Introductory_Notebook.ipynb)
[2]
[3](../spring_mass/Spring_Mass_Problem_Notebook.ipynb)
[4](../chip_2d/Challenge_CFD_Problem_Notebook.ipynb)

[Next Notebook](../spring_mass/Spring_Mass_Problem_Notebook.ipynb)

# Steady State 1D Diffusion in a Composite Bar using PINNs

This notebook give you a headstart in solving your own Partial Differential Equations (PDEs) using neural networks. Let's quickly recap the theory of PINNs before proceeding. We will embed the physics of the problem in the the neural networks and in such setting, we can use them to approximate the solution to a given differential equation and boundary condition without any training data from other solvers. More specifically, the neural network will be trained to minimize a loss function which is formed using the differential equation and the boundary conditions. If the network is able to minimize this loss then it will in effect solve the given differential equation. More information about the Physics Informed Neural Networks (PINNs) can be found in the [paper](https://www.sciencedirect.com/science/article/pii/S0021999118307125?casa_token=1CnSbVeDwJ0AAAAA:-8a6ZMjO7RgYjFBAxIoVU2sWAdsqFzLRTNHA1BkRryNPXx5Vjc8hCDCkS99gcZR0B1rpa33a30yG) published by Raissi et al.

In this notebook we will solve the steady 1 dimensional heat transfer in a composite bar. We will use NVIDIA's SimNet library to create the problem setup. You can refer to the *SimNet User Guide* for more examples on solving different types of PDEs using the SimNet library. Also, for more information about the SimNet APIs you can refer the *SimNet Source Code Documentation*.

### Learning Outcomes
1. How to use SimNet to simulate physics problems using PINNs
    1. How to write your own PDEs and formulate the different losses
    2. How to use the Constructive Solid Geometry (CSG) module
2. How to use SimNet to solve a parameterized PDEs

## Problem Description

Our aim is to obtain the temperature distribution inside the bar that is made up of two materials with different thermal conductivity. The geometry and the problem specification of the problem can be seen below

The composite bar extends from $x=0$ to $x=2$. The bar has material of conductivity $D_1=10$ from $x=0$ to $x=1$ and $D_2=0.1$ from $x=1$ to $x=2$. Both the ends of the bar, $x=0$ and $x=2$ are maintained at a constant temperatures of $0$ and $100$ respectively. For simplicity of modeling, we will treat the composite bar as two separate bars, bar 1 and bar 2, whose ends are joined together. We will treat the temperatures in the bar 1 as $U_1$ and the temperature in bar 2 as $U_2$.

The equations and boundary conditions governing the problem can be mathematically expressed as

One dimensional diffusion of temperature in bar 1 and 2:

$$
\begin{align}
\frac{d}{dx}\left( D_1\frac{dU_1}{dx} \right) = 0, && \text{when } 0<x<1 \\
\frac{d}{dx}\left( D_2\frac{dU_2}{dx} \right) = 0, && \text{when } 1<x<2 \\
\end{align}
$$

Flux and temperature continuity at interface $(x=1)$
$$
\begin{align}
D_1\frac{dU_1}{dx} = D_1\frac{dU_1}{dx}, && \text{when } x=1 \\
U_1 = U_2, && \text{when } x=1 \\
\end{align}
$$

## Case Setup

Now that we have our problem defined, let's take a look at the code required to solve it using SimNet's PINN library. SimNet has a variety of helper functions that will help us to set up the problem with ease. It has APIs to model geometry in a parameterized fashion using the Constructive Solid Geometry (CSG) module, write-up the required equations in a user-friendly symbolic format and comes with several advanced neural network architectures to choose for more complicated problems.

Now let's start the problem by importing the required libraries and packages

```python
# import SimNet library
from sympy import Symbol, Function, Number, sin, Eq, Abs, exp
import numpy as np
import tensorflow as tf

from simnet.solver import Solver
from simnet.dataset import TrainDomain, ValidationDomain, MonitorDomain
from simnet.data import Validation, Monitor
from simnet.sympy_utils.geometry_1d import Line1D
from simnet.controller import SimNetController
from simnet.node import Node
from simnet.pdes import PDES
from simnet.variables import Variables
```

The `Solver` class trains and evaluates the SimNet's neural network solver. The class `TrainDomain` is used to define the training data for the problem, while the other classes like `ValidationDomain`, `MonitorDomain`, etc. are used to create other data evaluations during the training.

The modules like `PDES` and `sympy_utils` contain predefined differential equations and geometries respectively that one can use to define the problem. We will describe each of them in detail as we move forward in the code. For more detailed information on all the different modules present in SimNet, we recommended you to refer the *SimNet Source Code Documentation*.

## Creating the geometry

In this problem, we will create the 1-dimensional geometry using the `Line1D` from the geometry module. The module also contains several 2d and 3d shapes like rectangle, circle, triangle, cuboids, sphere, torus, cones, tetrahedrons, etc. We will define the one dimensional line object using the two end-points. For composite bar, we will create two separate bars as defined in the problem statement

```python
# params for domain
L1 = Line1D(0,1)
L2 = Line1D(1,2)
```

Next we will define the properties for the problem which will later be used while making the equations, boundary conditions, etc. Also, for this problem, we can find the temperature at the interface analytically and we will use that as to form validation domain to compare our neural network results

```python
# defining the parameters for boundary conditions and equations of the problem
D1 = 1e1
D2 = 1e-1
Ta = 0
Tc = 100

# temperature at the interface from analytical solution
Tb = (Tc + (D1/D2)*Ta)/(1 + (D1/D2))
```

## Defining the differential equations for the problem

The `PDES` class allows us to write the equations symbolically in Sympy. This allows users to quickly write their equations in the most natural way possible. The Sympy equations are converted to TensorFlow expressions in the back-end and can also be printed to ensure correct implementation.

SimNet also comes with several common PDEs predefined for the user to choose from. Some of the PDEs that are already available in the PDEs module are: Navier Stokes, Linear Elasticity, Advection Diffusion, Wave Equations, etc.

Let's create the PDE to define the diffusion equation. We will define the equation in its most generic, transient 3-dimensional, form and then have an argument `dim` that can reduce it to lower dimensional forms.
$$\frac{\partial T}{\partial t}= \nabla\cdot \left( D \nabla T \right) + Q$$

Let's start defining the equation by inhereting from the `PDES` class. We will create the initialization method for this class that defines the equation(s) of interest. We will be defining the diffusion equation using the source(`Q`), diffusivity(`D`), symbol for diffusion(`T`). If `D` or `Q` is given as a string we will convert it to functional form. This will allow us to solve problems with spatially/temporally varying properties.

```python
class Diffusion(PDES):
  name = 'Diffusion'

  def __init__(self, T='T', D='D', Q=0, dim=3, time=True):
    # set params
    self.T = T
    self.dim = dim
    self.time = time

    # coordinates
    x, y, z = Symbol('x'), Symbol('y'), Symbol('z')

    # time
    t = Symbol('t')

    # make input variables
    input_variables = {'x':x,'y':y,'z':z,'t':t}
    if self.dim == 1:
      input_variables.pop('y')
      input_variables.pop('z')
    elif self.dim == 2:
      input_variables.pop('z')
    if not self.time:
      input_variables.pop('t')

    # Temperature
    assert type(T) == str, "T needs to be string"
    T = Function(T)(*input_variables)

    # Diffusivity
    if type(D) is str:
      D = Function(D)(*input_variables)
    elif type(D) in [float, int]:
      D = Number(D)

    # Source
    if type(Q) is str:
      Q = Function(Q)(*input_variables)
    elif type(Q) in [float, int]:
      Q = Number(Q)

    # set equations
    self.equations = Variables()
    self.equations['diffusion_'+self.T] = (T.diff(t)
                                          - (D*T.diff(x)).diff(x)
                                          - (D*T.diff(y)).diff(y)
                                          - (D*T.diff(z)).diff(z)
                                          - Q)
```

First we defined the input variables $x, y, z$ and $t$ with Sympy symbols. Then we defined the functions for $T$, $D$ and $Q$ that are dependent on the input variables $(x, y, z, t)$. Using these we can write out our simple equation $T_t = \nabla \cdot (D \nabla T) + Q$. We store this equation in the class by adding it to the dictionary of `equations`.

Note that we moved all the terms of the PDE either to LHS or RHS. This way, while using the equations in the `TrainDomain`, we
can assign a custom source function to the `’diffusion_T’` key instead of 0 to add more source terms to our PDE.

Great! We just wrote our own PDE in SimNet! Once you have understood the process to code a simple PDE, you can easily extend the procedure for different PDEs. You can also bundle multiple PDEs together in a same file by adding new keys to the equations dictionary. Below we show the code for the interface boundary condition where we need to maintain the field (dirichlet) and flux (neumann) continuity. *(More examples of coding your own PDE can be found in the SimNet User Guide Chapter 4)*.

**Note :** The field continuity condition is needed because we are solving for two different temperatures in the two bars.

```python
class DiffusionInterface(PDES):
  name = 'DiffusionInterface'

  def __init__(self, T_1, T_2, D_1, D_2, dim=3, time=True):
    # set params
    self.T_1 = T_1
    self.T_2 = T_2
    self.D_1 = D_1
    self.D_2 = D_2
    self.dim = dim
    self.time = time

    # coordinates
    x, y, z = Symbol('x'), Symbol('y'), Symbol('z')
    normal_x, normal_y, normal_z = Symbol('normal_x'), Symbol('normal_y'), Symbol('normal_z')

    # time
    t = Symbol('t')

    # make input variables
    input_variables = {'x':x,'y':y,'z':z,'t':t}
    if self.dim == 1:
      input_variables.pop('y')
      input_variables.pop('z')
    elif self.dim == 2:
      input_variables.pop('z')
    if not self.time:
      input_variables.pop('t')

    # variables to match the boundary conditions (example Temperature)
    T_1 = Function(T_1)(*input_variables)
    T_2 = Function(T_2)(*input_variables)

    # set equations
    self.equations = Variables()
    self.equations['diffusion_interface_dirichlet_'+self.T_1+'_'+self.T_2] = T_1 - T_2
    flux_1 = self.D_1 * (normal_x * T_1.diff(x) + normal_y * T_1.diff(y) + normal_z * T_1.diff(z))
    flux_2 = self.D_2 * (normal_x * T_2.diff(x) + normal_y * T_2.diff(y) + normal_z * T_2.diff(z))
    self.equations['diffusion_interface_neumann_'+self.T_1+'_'+self.T_2] = flux_1 - flux_2
```

## Creating Train Domain: Assigning the boundary conditions and equations to the geometry

As described earlier, we need to define a training domain for training our neural network. A loss function is then constructed which is a combination of contributions from the boundary conditions and equations that a neural network must satisfy at the end of the training. These training points (BCs and equations) are defined in a class that inherits from the `TrainDomain` parent class. The boundary conditions are implemented as soft constraints. These BCs along with the equations to be solved are used to formulate a composite loss that is minimized by the network during training.

$$L = L_{BC} + L_{Residual}$$

**Boundary conditions:** For generating a boundary condition, we need to sample the points on the required boundary/surface of the geometry and then assign them the desired values. We will use the method `boundary_bc` to sample the points on the boundary of the geometry we already created. `boundary_bc` will sample the entire boundary of the geometry, in this case, both the endpoints of the 1d line. A particular boundary of the geometry can be sub-sampled by using a particular criterion for the boundary_bc using the criteria parameter. For example, to sample the left end of `L1`, criteria is set to `Eq(x, 0)`.

The desired values for the boundary condition are listed as a dictionary in `outvar_sympy` parameter. In SimNet we define these variables as keys of this dictionary which are converted to appropriate nodes in the computational graph. For this problem,  we have `'u_1':0` at $x=0$ and `'u_2':100` at $x=2$. At $x=1$, we have the interface condition `'diffusion_interface_dirichlet_u_1_u_2':0` and `'diffusion_interface_neumann_u_1_u_2':0` that we defined earlier (i.e. $U_1=U_2$ and $D_1\frac{dU_1}{dx}=D_2\frac{dU_2}{dx}$).

The number of points to sample on each boundary are specified using the `batch_size_per_area` parameter. The
actual number of points sampled is then equal to the length/area of the geometry being sampled (boundary or interior)
times the batch_size_per_area.

In this case, since we only have 1 point on the boundary, we specify the `batch_size_per_area` as 1 for all the boundaries.

**Equations to solve:** The Diffusion PDE we defined is enforced on all the points in the
interior of both the bars, `L1` and `L2`. We will use `interior_bc` method to sample points in the interior of the geometry. Again, the equations to solve are specified as a dictionary input to `outvar_sympy` parameter. These dictionaries are then used when unrolling the computational graph for training.

For this problem we have the `'diffusion_u_1':0` and `'diffusion_u_2':0` for bars `L1` and `L2` respectively. The parameter `bounds`, determines the range for sampling the values for variables $x$ and $y$. The `lambda_sympy` parameter is used to determine the weights for different losses. In this problem, we weight each point equally and hence keep the variable to 1 for each key (default).

```python
class DiffusionTrain(TrainDomain):
  def __init__(self, **config):
    super(DiffusionTrain, self).__init__()
    # sympy variables
    x = Symbol('x')
    c = Symbol('c')

    # right hand side (x = 2) Pt c
    IC = L2.boundary_bc(outvar_sympy={'u_2': Tc},
                        batch_size_per_area=1,
                        criteria=Eq(x, 2))
    self.add(IC, name="RightHandSide")

    # left hand side (x = 0) Pt a
    IC = L1.boundary_bc(outvar_sympy={'u_1': Ta},
                        batch_size_per_area=1,
                        criteria=Eq(x, 0))
    self.add(IC, name="LeftHandSide")

    # interface 1-2
    IC = L1.boundary_bc(outvar_sympy={'diffusion_interface_dirichlet_u_1_u_2': 0,
                                      'diffusion_interface_neumann_u_1_u_2': 0},
                        lambda_sympy={'lambda_diffusion_interface_dirichlet_u_1_u_2': 1,
                                      'lambda_diffusion_interface_neumann_u_1_u_2': 1},
                        batch_size_per_area=1,
                        criteria=Eq(x, 1))
    self.add(IC, name="Interface1n2")

    # interior 1
    interior = L1.interior_bc(outvar_sympy={'diffusion_u_1': 0},
                              lambda_sympy={'lambda_diffusion_u_1': 1},
                              bounds={x: (0, 1)},
                              batch_size_per_area=200)
    self.add(interior, name="Interior1")

    # interior 2
    interior = L2.interior_bc(outvar_sympy={'diffusion_u_2': 0},
                              lambda_sympy={'lambda_diffusion_u_2': 1},
                              bounds={x: (1, 2)},
                              batch_size_per_area=200)
    self.add(interior, name="Interior2")
```

At this point you might be wondering where do we input the parameters of the equation, for eg. values for $D_1, D_2$, etc. Don't worry, we will discuss them while making the neural network solver. But before that, let's create the validation data to verify our simulation results against the analytical solution.

## Creating Validation Domain

For this 1d bar problem where the conductivity is constant in each bar, the temperature varies linearly with position inside the solid. The analytical solution can then be given as:

$$
\begin{align}
U_1 = xT_b + (1-x)T_a, && \text{when } 0 \leq x \leq 1 \\
U_2 = (x-1)T_c + (2-x)T_b, && \text{when } 1 \leq x \leq 2 \\
\end{align}
$$

where,
$$
\begin{align}
T_a = U_1|_{x=0}, && T_c = U_2|_{x=2}, && \frac{\left(T_c + \left( D_1/D_2 \right)T_a \right)}{1+ \left( D_1/D_2 \right)}\\
\end{align}
$$

Now let's create the validation domains. The validation domain is created by inheriting from the `ValidationDomain` parent class. We use numpy to solve for the `u_1` and `u_2` based on the analytical expressions we showed above. The dictionary of generated numpy arrays (`invar_numpy` and `outvar_numpy`)for input and output variables is used as an input to the class method `from_numpy`.

```python
class DiffusionVal(ValidationDomain):
  def __init__(self, **config):
    super(DiffusionVal, self).__init__()

    x = np.expand_dims(np.linspace(0, 1, 100), axis=-1)
    u_1 = x*Tb + (1-x)*Ta
    invar_numpy = {'x': x}
    outvar_numpy = {'u_1': u_1}
    val = Validation.from_numpy(invar_numpy, outvar_numpy)
    self.add(val, name='Val1')

    # make validation data line 2
    x = np.expand_dims(np.linspace(1, 2, 100), axis=-1)
    u_2 = (x-1)*Tc + (2-x)*Tb
    invar_numpy = {'x': x}
    outvar_numpy = {'u_2': u_2}
    val = Validation.from_numpy(invar_numpy, outvar_numpy)
    self.add(val, name='Val2')
```

## Creating Monitor Domain

SimNet library allows you to monitor desired quantities in Tensorboard as the simulation progresses and
assess the convergence. A `MonitorDomain` can be used to create such an feature. This a useful feature when we want
to monitor convergence based on a quantity of interest. Examples of such quantities can be point values of variables,
surface averages, volume averages or any other derived quantities. The variables are available as TensorFlow tensors. We can perform tensor operations available in TensorFlow to compute any desired derived quantity of our choice.

In the code below, we create monitors for flux at the interface. The variable `u_1__x` represents the derivative of `u_1` in x-direction (two underscores (`__`) and the variable (`x`)). The same notation is used while handling other derivatives using the SimNet library. (eg. a neumann boundary condition of $\frac{dU_1}{dx}=0$ can be assigned as `'u_1__x':0` in the train domain for solving the same problem with a adiabatic/fixed flux boundary condition).

The points to sample can be selected in a similar way as we did for specifying the Train domain. We create the monitors by inheriting from the `MonitorDoamin` parent class

```python
class DiffusionMonitor(MonitorDomain):
  def __init__(self, **config):
    super(DiffusionMonitor, self).__init__()
    x = Symbol('x')

    # flux in U1 at x = 1
    fluxU1 = Monitor(L1.sample_boundary(10, criteria=Eq(x, 1)),
                    {'flux_U1': lambda var: tf.reduce_mean(D1*var['u_1__x'])})
    self.add(fluxU1, 'FluxU1')

    # flux in U2 at x = 1
    fluxU2 = Monitor(L2.sample_boundary(10, criteria=Eq(x, 1)),
                    {'flux_U2': lambda var: tf.reduce_mean(D2*var['u_2__x'])})
    self.add(fluxU2, 'FluxU2')
```

## Creating the Neural Network Solver

Now that we have the train domain and other validation and monitor domains defined, we can prepare the neural network solver and run the problem. The solver is defined by inheriting the `Solver` parent class. The `train_domain`, `val_domain`, and `monitor_domains` are assigned. The equations to be solved are specified under `self.equations`. Here, we will call the `Diffusion` and `DiffusionInterface` classes we defined earlier to include the PDEs of the problem. Now we will pass the appropriate values for the parameters like the variable name (eg. `T='u_1'`) and also specify the dimensions of the problem (1d and steady).

The inputs and the outputs of the neural network are specified and the nodes of the architecture are made. The default
network architecture is a simple fully connected multi-layer perceptron architecture with *swish* activation function. The network consists of 6 hidden layers with 512 nodes in each layer. Here we are using two separate neural networks for each variable (`u_1` and `u_2`). All these values can be modified through `update_defaults` function. Also, the different architectures in SimNet library can be used (eg. Fourier Net architecture, Radial Basis Neural Network architecture, etc. More details can be found in *SimNet User Guide*). We use the default exponential learning rate decay, set the start learning rate and decay steps and also assign the `'max_steps'` to 5000.

```python
# Define neural network
class DiffusionSolver(Solver):
  train_domain = DiffusionTrain
  val_domain = DiffusionVal
  monitor_domain = DiffusionMonitor

  def __init__(self, **config):
    super(DiffusionSolver, self).__init__(**config)

    self.equations = (Diffusion(T='u_1', D=D1, dim=1, time=False).make_node()
                      + Diffusion(T='u_2', D=D2, dim=1, time=False).make_node()
                      + DiffusionInterface('u_1', 'u_2', D1, D2, dim=1, time=False).make_node())
    diff_net_u_1 = self.arch.make_node(name='diff_net_u_1',
                                   inputs=['x'],
                                   outputs=['u_1'])
    diff_net_u_2 = self.arch.make_node(name='diff_net_u_2',
                                   inputs=['x'],
                                   outputs=['u_2'])
    self.nets = [diff_net_u_1, diff_net_u_2]

  @classmethod
  def update_defaults(cls, defaults):
    defaults.update({
        'network_dir': './network_checkpoint_diff',
        'max_steps': 5000,
        'decay_steps': 100,
        'start_lr': 1e-4,
        #'end_lr': 1e-6,
        })
```

Awesome! We have just completed the file set up for the problem using the SimNet library. We are now ready to solve the PDEs using Neural Networks!

Before we can start training, we can make use of Tensorboard for visualizing the loss values and convergence of several other monitors we just created. This can be done inside the jupyter framework by selecting the directory in which the checkpoint will be stored by clicking on the small checkbox next to it. The option to launch a Tensorboard then shows up in that directory.

Also, SimNet is desinged such that it can accept command line arguments. This causes issues when the code is directly executed through the jupyter notebook. So as a workaround, we will save the code in form of a python script and execute that script inside the jupyter cell. This example is already saved for you and the code block below executes that script `diffusion_bar.py`. You are encouraged to open the script in a different window and go through the code once before executing. Also, feel free to edit the parameters of the model and see its effect on the results.

```python
import os
import sys

sys.path.append('../../source_code/diffusion_1d')

!python ../../source_code/diffusion_1d/diffusion_bar.py
```

## Visualizing the solution

SimNet saves the data in .vtu and .npz format by default. The .npz arrays can be plotted to visualize the output of the simulation. The .npz files that are created are found in the `network_checkpoint*` directory.

Now let's plot the temperature along the bar for the analytical and the neural network solution. A sample script to plot the results is shown below. If the training is complete, you should get the results like shown below. As we can see, our neural network solution and the analytical solution match almost exactly for this diffusion problem.

```python
%%capture
import sys
!{sys.executable} -m pip install ipympl
%matplotlib inline
import matplotlib.pyplot as plt

plt.figure()
network_dir = './network_checkpoint_diff/val_domain/results/'
u_1_pred = np.load(network_dir + 'Val1_pred.npz', allow_pickle=True)
u_2_pred = np.load(network_dir + 'Val2_pred.npz', allow_pickle=True)
u_1_pred = np.atleast_1d(u_1_pred.f.arr_0)[0]
u_2_pred = np.atleast_1d(u_2_pred.f.arr_0)[0]

plt.plot(u_1_pred['x'][:,0], u_1_pred['u_1'][:,0], '--', label='u_1_pred')
plt.plot(u_2_pred['x'][:,0], u_2_pred['u_2'][:,0], '--', label='u_2_pred')

u_1_true = np.load(network_dir + 'Val1_true.npz', allow_pickle=True)
u_2_true = np.load(network_dir + 'Val2_true.npz', allow_pickle=True)
u_1_true = np.atleast_1d(u_1_true.f.arr_0)[0]
u_2_true = np.atleast_1d(u_2_true.f.arr_0)[0]

plt.plot(u_1_true['x'][:,0], u_1_true['u_1'][:,0], label='u_1_true')
plt.plot(u_2_true['x'][:,0], u_2_true['u_2'][:,0], label='u_2_true')

plt.legend()

plt.savefig('image_diffusion_problem_bootcamp.png')
```

```python
from IPython.display import Image
Image(filename='image_diffusion_problem_bootcamp.png')
```

# Parameterizing the PDE

As we discussed in the introductory notebook, one important advantage of a PINN solver over traditional numerical methods is its ability to solve parameterized geometries and PDEs. This was initially proposed in the [paper](https://arxiv.org/abs/1906.02382) published by Sun et al. This allows us to significant computational advantage as one can now use PINNs to solve for multiple desigs/cases in a single training. Once the training is complete, it is possible to run inference on several geometry/physical parameter combinations as a post-processing step, without solving the forward problem again.

To demonstrate the concept, we will train the same 1d diffusion problem, but now by parameterizing the conductivity of the first bar in the range (5, 25). Once the training is complete, we can obtain the results for any conductivity value in that range saving us the time to train multiple models.

## Case Setup

The definition of equations remain the same for this part. Since earlier while defining the equations, we already defined the constants and coefficients of the PDE to be either numerical values or strings, this will allow us to paramterize `D1` by passing it as a string while calling the equations and making the neural network. Now let's start by creating the paramterized train domain. We will skip the parts that are common to the previous section and only discuss the changes. The complete script can be referred in `diffusion_bar_parameterized.py`

## Creating Train Domain

Before starting out to create the train domain using the `TrainDomain` class, we will create the symbolic variable for the $D_1$ and also specify the range of variation for the variable. While the simulation runs, we will validate it against the same diffusion coefficient that we solved earlier i.e. $D_1=10$. Once the sybolic variables and the ranges are described for sampling, these parameter ranges need to be inputted to the `param_ranges` attribute of each boundary and internal sampling (`boundary_bc` and `interior_bc`)

```python
# params for domain
L1 = Line1D(0,1)
L2 = Line1D(1,2)

D1 = Symbol('D1')
D1_range = {D1: (5, 25)}
D1_validation = 1e1

D2 = 1e-1

Tc = 100
Ta = 0
Tb = (Tc + (D1/D2)*Ta)/(1 + (D1/D2))

Tb_validation = float(Tb.evalf(subs={D1: 1e1}))

class DiffusionTrain(TrainDomain):
  def __init__(self, **config):
    super(DiffusionTrain, self).__init__()
    # sympy variables
    x = Symbol('x')
    c = Symbol('c')

    # right hand side (x = 2) Pt c
    IC = L2.boundary_bc(outvar_sympy={'u_2': Tc},
                        batch_size_per_area=10,
                        criteria=Eq(x, 2),
                        param_ranges=D1_range)
    self.add(IC, name="RightHandSide")

    # left hand side (x = 0) Pt a
    IC = L1.boundary_bc(outvar_sympy={'u_1': Ta},
                        batch_size_per_area=10,
                        criteria=Eq(x, 0),
                        param_ranges=D1_range)
    self.add(IC, name="LeftHandSide")

    # interface 1-2
    IC = L1.boundary_bc(outvar_sympy={'diffusion_interface_dirichlet_u_1_u_2': 0,
                                      'diffusion_interface_neumann_u_1_u_2': 0},
                        lambda_sympy={'lambda_diffusion_interface_dirichlet_u_1_u_2': 1,
                                      'lambda_diffusion_interface_neumann_u_1_u_2': 1},
                        batch_size_per_area=10,
                        criteria=Eq(x, 1),
                        param_ranges=D1_range)
    self.add(IC, name="Interface1n2")

    # interior 1
    interior = L1.interior_bc(outvar_sympy={'diffusion_u_1': 0},
                              lambda_sympy={'lambda_diffusion_u_1': 1},
                              bounds={x: (0, 1)},
                              batch_size_per_area=400,
                              param_ranges=D1_range)
    self.add(interior, name="Interior1")

    # interior 2
    interior = L2.interior_bc(outvar_sympy={'diffusion_u_2': 0},
                              lambda_sympy={'lambda_diffusion_u_2': 1},
                              bounds={x: (1, 2)},
                              batch_size_per_area=400,
                              param_ranges=D1_range)
    self.add(interior, name="Interior2")
```

## Creating Validation and Monitor Domains

The process to create these domains is again similar to the previous section. For validation data, we need to create an additional key for the string `'D1'` in the `invar_numpy`. The value for this string can be in the range we specified earlier and which we would like to validate against. It is possible to create multiple validations if required, eg. different $D_1$ values. For monitor domain, similar to `interior_bc` and `boundary_bc` in the train domain, we will supply the paramter ranges for monitoring in the `param_ranges` attribute of the `sample_boundary` method.

```python
class DiffusionVal(ValidationDomain):
  def __init__(self, **config):
    super(DiffusionVal, self).__init__()
    # make validation data line 1
    x = np.expand_dims(np.linspace(0, 1, 100), axis=-1)
    D1 = np.zeros_like(x) + D1_validation                      # For creating D1 input array
    u_1 = x*Tb_validation + (1-x)*Ta
    invar_numpy = {'x': x}                                    # Set the invars for the required D1
    invar_numpy.update({'D1': np.full_like(invar_numpy['x'], D1_validation)})
    outvar_numpy = {'u_1': u_1}
    val = Validation.from_numpy(invar_numpy, outvar_numpy)
    self.add(val, name='Val1')

    # make validation data line 2
    x = np.expand_dims(np.linspace(1, 2, 100), axis=-1)
    u_2 = (x-1)*Tc + (2-x)*Tb_validation
    invar_numpy = {'x': x}                           # Set the invars for the required D1
    invar_numpy.update({'D1': np.full_like(invar_numpy['x'], D1_validation)})
    outvar_numpy = {'u_2': u_2}
    val = Validation.from_numpy(invar_numpy, outvar_numpy)
    self.add(val, name='Val2')

class DiffusionMonitor(MonitorDomain):
  def __init__(self, **config):
    super(DiffusionMonitor, self).__init__()
    x = Symbol('x')

    # flux in U1 at x = 1
    fluxU1 = Monitor(L1.sample_boundary(10, criteria=Eq(x, 1), param_ranges={D1: D1_validation}),   # Set the parameter range for the required D1
                    {'flux_U1': lambda var: tf.reduce_mean(D1_validation*var['u_1__x'])})
    self.add(fluxU1, 'FluxU1')

    # flux in U2 at x = 1
    fluxU2 = Monitor(L2.sample_boundary(10, criteria=Eq(x, 1), param_ranges={D1: D1_validation}),   # Set the parameter range for the required D1
                    {'flux_U2': lambda var: tf.reduce_mean(D2*var['u_2__x'])})
    self.add(fluxU2, 'FluxU2')
```

## Creating the Neural Network Solver

Once all the parameterized domain definitions are completed, for training the parameterized model, we will have the symbolic parameters we defined earlier as inputs to both the neural networks in `diff_net_u_1` and `diff_net_u_2` viz. `'D1'` along with the usual x coordinate. The outputs remain the same as what we would have for any other non-parameterized simulation.

```python
# Define neural network
class DiffusionSolver(Solver):
  train_domain = DiffusionTrain
  val_domain = DiffusionVal
  monitor_domain = DiffusionMonitor

  def __init__(self, **config):
    super(DiffusionSolver, self).__init__(**config)

    self.equations = (Diffusion(T='u_1', D='D1', dim=1, time=False).make_node()         # Symbolic input to the equation
                      + Diffusion(T='u_2', D=D2, dim=1, time=False).make_node()
                      + DiffusionInterface('u_1', 'u_2', 'D1', D2, dim=1, time=False).make_node())
    diff_net_u_1 = self.arch.make_node(name='diff_net_u_1',
                                   inputs=['x', 'D1'],                                  # Add the parameters to the network
                                   outputs=['u_1'])
    diff_net_u_2 = self.arch.make_node(name='diff_net_u_2',
                                   inputs=['x', 'D1'],
                                   outputs=['u_2'])
    self.nets = [diff_net_u_1, diff_net_u_2]

  @classmethod # Explain This
  def update_defaults(cls, defaults):
    defaults.update({
        'network_dir': './network_checkpoint_diff_parameterized',
        'max_steps': 10000,
        'decay_steps': 200,
        'start_lr': 1e-4,
        'layer_size': 256,
        'xla': True,
        })
```

## Visualizing the solution

The .npz arrays can be plotted similar to previous section to visualize the output of the simulation. You can see that we get the same answer as the analytical solution. You can try to run the problem in `eval` mode by chaning the validation data and see how it performs for the other `D1` values as well. To run the model in evaluation mode (i.e. without training), you just need to add the `--run_mode=eval` flag while executing the script.

You can see that at a fractional increase in computational time, we solved the PDE for $D_1$ ranging from (5, 25). This concept can easily be extended to more complicated problems and this ability of solving parameterized problems comes very handy during desing optimization and exploring the desing space. For more examples of solving parameterized problems, please refer to *SimNet User Guide Chapter 13*

# Licensing
This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0)

[Home Page](../../Start_Here.ipynb)

[Previous Notebook](../introduction/Introductory_Notebook.ipynb)

[1](../introduction/Introductory_Notebook.ipynb)
[2]
[3](../spring_mass/Spring_Mass_Problem_Notebook.ipynb)
[4](../chip_2d/Challenge_CFD_Problem_Notebook.ipynb)

[Next Notebook](../spring_mass/Spring_Mass_Problem_Notebook.ipynb)


# The $\eta$ function

```python
import numpy as np
import matplotlib.pyplot as pl
%matplotlib notebook
from IPython.display import display, Math
import sympy
from sympy import *

# Initialize the session
init_session(quiet=True)

# Let's report what version of sympy this is
print("Using sympy version", sympy.__version__)
```

    Using sympy version 1.3

```python
eta, r, b, k0, k1, A, k2 = symbols("\eta r b kappa_0 kappa_1 A, k^2")
```

```python
k2 = (1 - r ** 2 - b ** 2 + 2 * b * r) / (4 * b * r)
A = Rational(1, 2) * sqrt((1 + (r + b)) * (b - (1 - r)) * (b + (1 - r)) * (1 + (r - b)))
k0 = atan2(2 * A, (r - 1) * (r + 1) + b ** 2)
k1 = atan2(2 * A, (1 - r) * (1 + r) + b ** 2)
```

The $\eta$ function in the paper is identical to $\eta_1$ (for $k^2 \le 1$) and $\eta_2$ (for $k^2 > 1$) in Equation (7) of Mandel & Agol (2002). Below we compute its derivatives with respect to $b$ and $r$.

## $k^2 \le$ 1

```python
eta = 1 / (2 * pi) * (k1 + r ** 2 * (r ** 2 + 2 * b ** 2) * k0 - Rational(1, 2) * (1 + 5 * r ** 2 + b ** 2) * A)
```

This is the derivative according to the paper:

```python
detadr_paper = (2 * r / pi) * ((r ** 2 + b ** 2) * k0 - 2 * A)
```

Is it correct?

```python
simplify(diff(eta, r) - detadr_paper) == 0
```

    True

**QED**

This is the derivative according to the paper:

```python
detadb_paper = (1 / (2 * pi * b)) * (4 * r ** 2 * b ** 2 * k0 - 2 * (1 + b ** 2 + r ** 2) * A)
```

Is it correct?

```python
simplify(diff(eta, b) - detadb_paper) == 0
```

    True

**QED**

## $k^2$ > 1

```python
eta = r ** 2 / 2 * (r ** 2 + 2 * b ** 2)
```

```python
simplify(diff(eta, r))
```

**QED**

```python
simplify(diff(eta, b))
```

**QED**


# Generalized linear models
## Marcel Nonnenmacher, Jan-Mathis Lückmann, Pedro Goncalves, Jakob Macke

> One of the central problems in systems neuroscience is that of characterizing the functional relationship between sensory stimuli and neural spike responses. Investigators call this the neural coding problem, because the spike trains of neurons can be considered a code by which the brain represents information about the state of the external world. One approach to understanding this code is to build mathematical models of the mapping between stimuli and spike responses; the code can then be interpreted by using the model to predict the neural response to a stimulus, or to decode the stimulus that gave rise to a particular response. [(Pillow, 2007)](http://pillowlab.princeton.edu/pubs/Pillow_BBchap07.pdf)

Here, we will build probabilistic models for the response of a single neuron, starting from a simple model, that we will then extend. Conditional on a stimulus $x$ and model parameters $\theta$ we will model the probability of a neural reponse $y$, i.e., $p(y|x, \theta)$. Our central goal will be to find model parameters $\theta$ such that the $p(y|x,\theta)$ is a good fit to a dataset of stimulus-response pairs we observed, $\mathcal{D} = \{ (x_k,y_k) \}_{k=1}^K$.

### Goals of these exercises

Central to inferring the best fitting parameters will be the likelihood function of the model. The simplest method goes by simply maximizing the likelihood using its gradient with respect to the model parameters (a technique called maximum likelihood estimation, MLE). You will learn to incorporate prior knowledge on the parameters, which leads to a method called maximum a posteriori (MAP). Finally, you will learn automatic differentiation (AD) which --- as the name suggests --- provides a automatic way to calcuate gradients of an objective function (here: the likelihood of parameters given the data). AD is a central ingredient to machine learning methods that are becoming increasingly popular.

### Assumptions and notation

Throughout this tutorial, we will adopt the following conventions:

- $T$ is the number of time bins within one trial; $t$ always specifies a time bin;
- $K$ is the number of trials in the experiment; $k$ always identifies a trial;
- to make the notation lighter, we will sometimes drop the subscript $k$;
- $\hat{\pi}(\cdot)$ indicates an unnormalized probability, and $\pi(\cdot)$ the same probability normalize to integrate to 1;
- $\mathcal{L}(\boldsymbol{\theta}) = p(\mathbf{y}\, |\, \boldsymbol{\theta})$ is the likelihood of the vector of parameters $\boldsymbol{\theta}$ for the (fixed) data $\mathbf{y}$.

For all models we consider, we assume that time is discretized in bins of size $\Delta$. Given $z_t$, the instantaneous *input rate* of a neuron at time $\Delta \cdot t$, the spike counts $y_t$ are assumed to be independent, and distributed according to

$\begin{equation}
    y_t \sim \mathrm{Poisson}\big(\eta(z_t)\big)
\end{equation}$

where  $\eta(\cdot)$ is corresponding canonical link function (here, we will always use $\eta(\cdot) = \exp(\cdot)$ for Poisson). We further assume that there is a linear dependence between $z_i$ and a set of external covariates $\mathbf{x}_t$ at time $t$, i.e. $z_t = \boldsymbol{\theta}^\top \mathbf{x}$, and $\boldsymbol{\theta}$ is a vector of parameters which fully characterizes the neuron.

Experiments are composed of $K$ trials, each subdivided into $T$ bins.

Note, and in contrast to the lectures, we assume that the rate $\mu_t$ is already 'per bin size', i.e. the expected number of spikes is $\mu$ (and not $\mu\Delta$, as we had in lectures).

For a Poisson neuron, the probability of producing $n$ spikes in an interval of size $\Delta$ is given by

$\begin{align}
P(y_t=n| \mu)= \frac{\mu^n e^{-\mu} }{n!}
\end{align}$

## Exercise 1 (from lectures)

Assume that you have spike counts $n_1$ to $n_K$ from $K$ trials, calculate the maximum likelihood estimate (MLE) of $\mu$.

## LNP model

The stimulus $\mathbf{u}_t$ is a white noise sequence, and the input rate is:

$\begin{equation}
    z_t = \mathbf{\beta}^\top \mathbf{u}_{t-\delta{}+1:t} + b = \boldsymbol{\theta}^\top \mathbf{x}
\end{equation}$,

i.e. $z_t$ is the result of a linear filter $\beta$ applied to the recent stimulus history $\mathbf{u}_{t-\delta{}+1:t}$, plus some offset $b$. This results in a vector of covariates at time $t$
$\mathbf{x}_{kt} = \big[1, \mathbf{u}_{kt-\delta{}+1},\ldots, \mathbf{u}_{kt} \big]^\top$ for temporal filter length $\delta \in \mathbb{N}$. Note that we can deal with any form of input in the second and third column of $\mathbf{x}_{kt}$, not just white noise.

The vector of parameters is $\boldsymbol{\theta} = \left[b, \beta^\top \right]^\top$.

### Simulating data from the model

Next, we  will want to generate data using this model. Execute the following code cell, which will load some functions you will need throughout the session.

```python
import numpy as np

%run -i helpers.ipynb
```

The following cell generates a matrix $\mathbf{x}$ as specified above.

```python
binsize = 0.001  # seconds
T = 10000        # 1s trials
K = 10  # number of trials
nbins = T*K

delta = 10       # length of temporal filter

# stimulus
U = np.random.normal(size=nbins)

def toyDesignMatrix(U=U, T=T, K=K):
    nbins = T*K
    X = np.zeros((delta+1, nbins))

    X[0,:] = 1.       # bias
    if delta > 0:
        X[1, :] = U   # instantaneous input
    for i in range(1,delta):
        X[i+1, i+1:] = U[:-(i+1)]

    return X

X = toyDesignMatrix()
```

Next, we define $\mathbf{\theta}$.

```python
# ground-truth vector of parameters
b = -6  # controls the offset and hence overall firing rate
beta = np.cos( np.linspace(0, PI, delta))
theta = np.hstack([b, beta])
```

Given `X` and `theta`, we want to generate sample spike trains. In the following cell, we do so by just using

```python
ys = []
for k in range(10):
    y, fr = toyModel(X, theta)  # spike train, firing rate
    ys.append(y)
```

... plotting spike rasters and PSTH:

```python
plt.subplot(311)
plt.plot(U[:200])
plt.subplot(312)
plt.imshow(np.asarray(ys)[:,:200], aspect=4, interpolation='None');
plt.subplot(313)
plt.plot(np.asarray(ys)[:, :200].mean(axis=0) / binsize);  # PSTH
plt.plot(fr[:200], linewidth=2);  # firing rate
```

### Optional: Try implementing the model yourself

(Optional): Above, you used an implementation of the model we provided. You can try implementing the model as stated above yourself. To do so, complete the following function template

```python
def toyModelExercise(X, theta):
    # TODO: given stimulus and theta, return spikes and firing rate
    return y, fr
```

To check whether this model is correct, reproduce the PSTHs for both models and compare.

### MLE inference

#### Likelihood ####

The likelihood defines a model that connects model parameters to the observed data:

$\begin{align}
\log \mathcal{L}(\boldsymbol{\theta}) &= \log p(\mathbf{y} | \boldsymbol{\theta}) = \log \bigg[ \prod_{k=1}^K \prod_{t=1}^T p(y_{kt} | b, \beta_1, \beta_2) \bigg] \\ &= \sum_{k=1}^K \sum_{t=1}^T \log p(y_{tk} | b, \beta_1, \beta_2) = \sum_{k=1}^K \sum_{t=1}^T \big[ z_{tk} y_{tk} - \mathrm{e}^{z_{tk}} \big],
\end{align}$

where as above $z_{tk} = \theta^\top \mathbf{x}_{tk} = b + \beta^\top \mathbf{u}_{tk-\delta{}+1:tk}$.

Large $\mathcal{L}$ for a given set of parameters $\theta$ indicates that the data is likely under that parameter set. We can iteratively find more likely parameters, starting from some initial guess $\theta_0$, by gradient ascent on $\mathcal{L}$.

For this model, the likelihood function has a unique maximum.

#### Gradients ####

**Exercise 1:** Using pen and paper, derive the gradient of the $\log \mathcal{L}$ with respect to $\mathbf{\theta}$.

### MLE parameter inference

We will now want to the use gradient you just derived to do parameter inference. For that, we will need to implement the functions `ll` and `dll` (the log-likelihood function and its derivative).

**Exercise 2.1: ** Implement `ll` and `dll` in the cell below.

```python
# say, we got a single spike train
y, fr = toyModel(X, theta)  # spike train, firing rate

def ll(theta):
    # TODO: implement log-likelihood function
    return NotImplemented

def dll(theta):
    # TODO: implement derivative of log-likelihood function wrt theta
    return NotImplemented
```

**Exercise 2.2**: Assume the true parameters that we used to generate the data, $\mathbf{\theta}^*$, were unknown. We want to recover $\mathbf{\theta}^*$ starting from an initial guess $\mathbf{\theta}_0$. Fill the gaps in the code block below. How good do you recover $\mathbf{\theta}^*$? What happens if you change `step_size`?

```python
theta_true = theta.copy()
theta_initial = np.random.randn(len(theta))

print('theta_star : {}'.format(theta_true))
print('theta_0 : {}'.format(theta_initial))

def gradientAscent(theta_initial, step_size=0.0001, num_iterations=1000):
    theta_hat = theta_initial.copy()

    for i in range(0, num_iterations):
        # TODO: fix the next lines
        log_likelihood = ...
        gradient = ...
        theta_hat = theta_hat + ...

    return theta_hat

theta_hat = gradientAscent(theta_initial)
print('theta_hat : {}'.format(theta_hat))
```

## Extending the model

Our simple model assumed independent firing in each time-bin that only depends on the stimulus. In reality, we know that the activity of neurons depends also on their recent firing history.

The GLM frameworks allows to flexibly extend our model simply by adding additional covariates and corresponding parameters, i.e. by adding columns to design matrix $\mathbf{X}$ and entries to parameter vector $\theta$.

Let us try introducing the recent spiking history $\mathbf{y}_{kt-\tau}, \ldots, \mathbf{y}_{kt-1}$ as additional covariates.

The vector of covariates at time $t$ becomes
$\mathbf{x}_{kt} = \big[1, \mathbf{u}_{kt-\delta+1 \ : \ tk}, \mathbf{y}_{kt-\tau \ : \ tk-1}\big]^\top$,

and we extend the vector of parameters as $\boldsymbol{\theta} = \left[b, \mathbf{\beta}^\top, \mathbf{\psi}^\top \right]^\top$, with history kernel $\mathbf{\psi} \in \mathbb{R}^\tau$ and history kernel length $\tau \in \mathbb{N}$.

**Question:** What other covariates could help improve our model?

### MLE Inference

**Exercise 3.1:** Write a function that implements the new design matrix $\mathbf{X}$ (now depends on data $\mathbf{y}$). Note that we provide a function `createDataset()` to generate data from the extended model with given parameter vector $\theta$.

```python
tau = 5  # length of history kernel (in bins)

psi = - 1.0 * np.arange(0, tau)[::-1]
theta_true = np.hstack((theta, psi))
y = createDataset(U, T, K, theta_true, delta)

def extendedDesignMatrix(y):
    # TODO: implement design matrix X with
    # X[kt,:] = [1, w*cos(t), w*sin(t), y_{kt-tau:kt-1}]
    return NotImplemented

X = extendedDesignMatrix(y) # you might have to re-run the cell defining ll() and dll()
                             # to update the used design matrix X and data y
```

**Exercise 3.2:** Write down the gradients for the extended model. What changes from our earlier simpler model?

```python

```

## MAP inference

The solution $\hat{\theta}$ obtained by gradient ascent on the log-likelihood depends on the data $\mathcal{D} = \{ (x_{tk}, y_{tk}) \}_{(t,k)}$. In particular for very short traces and few trials, this data only weakly constrains the solution.
We can often improve our obtained solutions by adding prior knowledge regarding what 'good' solutions should look like. In probabilistic modeling, this can be done by introducing prior distributions $p(\theta)$ on the model parameters, which together with the likelihood $\mathcal{L}$ define a posterior distribution over parameters given the data $p(\theta | \mathbf{y})$,

$$ \log p(\theta | \mathbf{y}) = \log p(\mathbf{y}|\theta) + \log p(\theta) - \log p(\mathbf{y}) = \mathcal{L}(\theta) + \log p(\theta) + const.$$

Maximum a-posterio (MAP) estimates parameters $\theta$ by gradient ascent on the (log-)posterior.

We will assume zero-mean Gaussian priors on $\beta, \psi$, i.e.
\begin{align}
p(\beta) &= \mathcal{N}(0, \Sigma_\beta) \\
p(\psi) &= \mathcal{N}(0, \Sigma_\psi).
\end{align}
We will not assume an explicit prior on $b$, which effectively assumes $b$ to be distributed 'uniformly' over $\mathbb{R}$.

### Gradients

Compared to maximum likelihood, MAP only requires adding the prior gradient:

\begin{align}
\frac{\partial}{\partial \theta} p(\theta|\mathbf{y}) =  \frac{\partial}{\partial \theta} \mathcal{L}(\theta) + \frac{\partial}{\partial \theta}p(\theta)
\end{align}

### Exercises

**Exercise 4: ** Derive the gradients for the prior. If you get stuck, or if you want to verify the solution, ask the tutors for help.

```python

```

**Exercise 5:** Fill gaps in codeblock below.

```python
## priors

# select prior covariance for input weights
Sig_beta = np.eye(delta)

# select prior covariance for history kernel
ir = np.atleast_2d(np.arange(tau, 0, -1))
Sig_psi = np.exp(- np.abs(ir.T - ir)/5)  # assuming smoothness

# convenience
P_beta, P_psi = np.linalg.inv(Sig_beta), np.linalg.inv(Sig_psi)

## functions and gradients

def po(theta):
    # TODO: implement log-posterior density function
    return NotImplemented

def dpo(theta):
    # TODO: implement derivative of log-posterior density function wrt theta
    return NotImplemented

# Hint: it can be helpful to first derive the functions for the prior below:

def pr(theta):
    # TODO: implement log-prior density function
    return NotImplemented

def dpr(theta):
    # TODO: implement derivative of log-prior density function wrt theta
    return NotImplemented

# leave as is
def ll(theta):
    z = np.dot(theta, X)
    return np.sum( y * z - link(z) )

# leave as is
def dll(theta):
    z = np.dot(theta, X)
    r = y - link(z)
    return np.dot(X, r)
```

**Exercise 6:** Numerical gradient checking -- use the code below to numerically ensure your gradients are correct

```python
from scipy import optimize

thrn = np.random.normal(size=theta_true.shape)
print(optimize.check_grad(ll, dll, thrn))
print(optimize.check_grad(pr, dpr, thrn))
print(optimize.check_grad(po, dpo, thrn))
```

**Exercise 7:** Do inference (WIP)

```python
data = createDataset(1000, 1, theta_true, omega)

# TODO: implement gradient ascent
```

## Automatic differentiation

Instead of calculating the gradients w.r.t. the model parameters by hand, we can calculate them automatically. Our objective function consists of many elementary functions, each of which is differentiable. [Automatic differentiation (AD)](https://en.wikipedia.org/wiki/Automatic_differentiation) applies the chain rule to the expression graph of our objective to find the gradient.

Here, we will use a Python library called `autograd` to find the gradient of our objective. AD is a central ingredient in libraries used for training artifical neural networks, including theano, TensorFlow and PyTorch.

### Installation

Install the [`autograd` package](https://github.com/HIPS/autograd) through your package manager.

Depending on how things are set up on your machine, install `autograd` by `pip3 install autograd --user` or by `pip install autograd --user`.

You might need to restart the notebook kernel in case the simple example which follows fails with an import error. If you restart the kernel, make sure to re-run the cells. You can do that by choosing `Kernel > Restart & Run All` from the menu.

### `autograd` by a simple example

```python
import autograd.numpy as np   # thinly-wrapped numpy
from autograd import grad     # the only autograd function you may ever need

def tanh(x):                  # Define a function
    y = np.exp(-x)
    return (1.0 - y) / (1.0 + y)

grad_tanh = grad(tanh)        # Obtain its gradient function

print('Gradient at x=1.0 (autograd) : {}'.format(grad_tanh(1.0)))

print('Gradient at x=1.0 (finite diff): {}'.format((tanh(1.0001) - tanh(0.9999)) / 0.0002))
```

```python
ipython nbconvert exercises.ipynb --to pdf
```

### Exercises

**Exercise 8**: Redo exercise 2 using `autograd`. To do so, go to the first code cell of the notebook. Instead of `import numpy as np` use `import autograd.numpy as np`. Restart the notebook kernel and run through the notebook from the beginning, till you get to the point where you implemented `dll`. Replace `dll` by a AD version that relies on the function `grad` (see simple example).

**Exercise 9:** Similar to the previous exercise, adapt the extended version of the model to rely on `autograd` for differentiation.

**Exercise 10 (open ended)** : Come up with your own extensions to one of the models.


# SymPy Tutorial

*Arthur Ryman* <br/>
*Last Updated: 2020-04-16*

This notebook contains the examples from the
[SymPy Tutorial](https://docs.sympy.org/latest/tutorial/index.html).

## Matrices

```python
from sympy import *
init_printing(use_unicode=True)
```

### simplify

```python
Matrix([[1, -1], [3, 4], [0, 2]])
```

```python
Matrix([1, 2, 3])
```

```python
M = Matrix([[1, 2, 3], [3, 2, 1]])
N = Matrix([0, 1, 1])
M*N
```

### Basic Operations

#### Shape

```python
M = Matrix([[1, 2, 3], [-2, 0, 4]])
M
```

```python
M.shape
```

#### Accessing Rows and Columns

```python
M.row(0)
```

```python
M.col(-1)
```

#### Deleting and Inserting Rows and Columns

```python
M.col_del(0)
M
```

```python
M.row_del(1)
M
```

```python
M
```

```python
M = M.row_insert(1, Matrix([[0, 4]]))
M
```

```python
M = M.col_insert(0, Matrix([1, -2]))
M
```

### Basic Methods

```python
M = Matrix([[1, 3], [-2, 3]])
N = Matrix([[0, 3], [0, 7]])
M + N
```

```python
M*N
```

```python
3*M
```

```python
M**2
```

```python
M**-1
```

```python
try:
    N**-1
except Exception as ex:
    print(type(ex))
    print(ex)
```

    <class 'sympy.matrices.common.NonInvertibleMatrixError'>
    Matrix det == 0; not invertible.

```python
M = Matrix([[1, 2, 3], [4, 5, 6]])
M
```

```python
M.T
```

### Matrix Constructors

```python
eye(3)
```

```python
eye(4)
```

```python
zeros(2, 3)
```

```python
ones(3, 2)
```

```python
diag(1, 2, 3)
```

```python
diag(-1, ones(2, 2), Matrix([5, 7, 5]))
```

### Advanced Methods

#### Determinant

```python
M = Matrix([[1, 0, 1], [2, -1, 3], [4, 3, 2]])
M
```

```python
M.det()
```

#### RREF

```python
M = Matrix([[1, 0, 1, 3], [2, 3, 4, 7], [-1, -3, -3, -4]])
M
```

```python
M.rref()
```

#### Nullspace

```python
M = Matrix([[1, 2, 3, 0, 0], [4, 10, 0, 0, 1]])
M
```

```python
M.nullspace()
```

#### Columnspace

```python
M = Matrix([[1, 1, 2], [2, 1, 3], [3, 1, 4]])
M
```

```python
M.columnspace()
```

#### Eigenvalues, Eigenvectors, and Diagonalization

```python
M = Matrix([\
           [3, -2, 4, -2],\
           [5, 3, -3, -2],\
           [5, -2, 2, -2],\
           [5, -2, -3, 3]])
M
```

```python
M.eigenvals()
```

```python
M.eigenvects()
```

```python
P, D = M.diagonalize()
P
```

```python
D
```

```python
P*D*P**-1 == M
```

    True

```python
lamda = symbols('lamda')
p = M.charpoly(lamda)
factor(p)
```

### Possible Issues

#### Zero Testing

```python
from sympy import *
q = Symbol('q', positive=True)
m = Matrix([
    [-2*cosh(q/3),      exp(-q),            1],
    [      exp(q), -2*cosh(q/3),            1],
    [           1,            1, -2*cosh(q/3)]])
m.nullspace()
```

```python
import warnings

def my_iszero(x):
    try:
        result = x.is_zero
    except AttributeError:
        result = None

    # Warnings if evaluated into None
    if result is None:
        warnings.warn('Zero testing of {} evaluated into None'.format(x))
    return result

m.nullspace(iszerofunc=my_iszero)
```

```python
def my_iszero(x):
    try:
        result = x.rewrite(exp).simplify().is_zero
    except AttributeError:
        result = None

    # Warnings of evaluated into None
    if result is None:
        warnings.warn('Zero testing of {} evaluated into None'.format(x))
    return result

m.nullspace(iszerofunc=my_iszero)
```