# Tutorial 3: Numerical Methods
**Week 0, Day 4: Calculus**

**By Neuromatch Academy**

__Content creators:__ John S Butler, Arvind Kumar

__Content reviewers:__

__Production editors:__

---
# Tutorial Objectives

While a great deal of neuroscience can be describe by mathematics a great deal of the mathematics used cannot be solved exactly. This might seem very odd that you can writing something in mathematics that cannot be immediately solved but that is the beauty and mystery of mathematics. To side step this issue we use Numerical Methods to estimate the solution.

In this tutorial, we will look at the Euler method to estimate the solution of a few different differential equations, the population equation, the Leaky Integrate and Fire model and a simplified version of the Wilson-Cowan model which is a system of differential equations.

**Steps:**
- Code the Euler estimate of the Population Equation;
- Investigate the impact of time step on the error of the numerical solution;
- Code the Euler estimate of the Learky Integrate and Fire model for a constant input;
- Visualize and listen to the response of the integrate for different inputs;
- Apply the Euler method to estimate the soluton of a system of differential equations.

```python
#@markdown Tutorial slides
# you should link the slides for all tutorial videos here (we will store pdfs on osf)

from IPython.display import HTML
HTML('')
```

```python
# Import libraries
import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import IPython.display as ipd
from matplotlib import gridspec

import ipywidgets as widgets  # interactive display
from ipywidgets import Label
%config InlineBackend.figure_format = 'retina'
# use NMA plot style
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
my_layout = widgets.Layout()
```

```python
# @title Plotting Functions

time = np.arange(0, 1, 0.01)

def plot_slope(dt):
  """
    Args:
      dt  : time-step
    Returns:
      A figure of an exponential, the slope of the exponential and the derivative exponential
  """

  t = np.arange(0, 5+0.1/2, 0.1)

  with plt.xkcd():

    fig = plt.figure(figsize=(6, 4))
    # Exponential
    p = np.exp(0.3*t)
    plt.plot(t, p, label='y')
    # slope
    plt.plot([1, 1+dt], [np.exp(0.3*1), np.exp(0.3*(1+dt))],':og',label=r'$\frac{y(1+\Delta t)-y(1)}{\Delta t}$')
    # derivative
    plt.plot([1, 1+dt], [np.exp(0.3*1), np.exp(0.3*(1))+dt*0.3*np.exp(0.3*(1))],'-k',label=r'$\frac{dy}{dt}$')
    plt.legend()
    plt.plot(1+dt, np.exp(0.3*(1+dt)), 'og')
    plt.ylabel('y')
    plt.xlabel('t')
    plt.show()

def plot_StepEuler(dt):
  """
    Args:
      dt  : time-step
    Returns:
      A figure of one step of the Euler method for an exponential growth function
  """

  t=np.arange(0, 1 + dt + 0.1 / 2, 0.1)

  with plt.xkcd():
    fig = plt.figure(figsize=(6,4))
    p=np.exp(0.3*t)
    plt.plot(t,p)
    plt.plot([1,],[np.exp(0.3*1)],'og',label='Known')
    plt.plot([1,1+dt],[np.exp(0.3*1),np.exp(0.3*(1))+dt*0.3*np.exp(0.3*1)],':g',label=r'Euler')
    plt.plot(1+dt,np.exp(0.3*(1))+dt*0.3*np.exp(0.3*1),'or',label=r'Estimate $p_1$')
    plt.plot(1+dt,p[-1],'bo',label=r'Exact $p(t_1)$')
    plt.vlines(1+dt,np.exp(0.3*(1))+dt*0.3*np.exp(0.3*1),p[-1],colors='r', linestyles='dashed',label=r'Error $e$')
    plt.legend()
    plt.ylabel('Population (millions)')
    plt.xlabel('time(years)')
    plt.show()

## LIF PLOT
def plot_IF(t, V, I, Spike_time):
  """
    Args:
      t  : time
      V  : membrane Voltage
      I  : Input
      Spike_time : Spike_times
    Returns:
      figure with three panels
      top panel: Input as a function of time
      middle panel: membrane potential as a function of time
      bottom panel: Raster plot
  """

  with plt.xkcd():
    fig = plt.figure(figsize=(12,4))
    gs = gridspec.GridSpec(3, 1,  height_ratios=[1, 4, 1])
    # PLOT OF INPUT
    plt.subplot(gs[0])
    plt.ylabel(r'$I_e(nA)$')
    plt.yticks(rotation=45)
    plt.plot(t,I,'g')
    #plt.ylim((2,4))
    plt.xlim((-50,1000))
    # PLOT OF ACTIVITY
    plt.subplot(gs[1])
    plt.plot(t,V,':')
    plt.xlim((-50,1000))
    plt.ylabel(r'$V(t)$(mV)')
    # PLOT OF SPIKES
    plt.subplot(gs[2])
    plt.ylabel(r'Spike')
    plt.yticks([])
    plt.scatter(Spike_time,1*np.ones(len(Spike_time)), color="grey", marker=".")
    plt.xlim((-50,1000))
    plt.xlabel('time(ms)')
    plt.show()

def plot_rErI(t, r_E, r_I):
  """
    Args:
      t   : time
      r_E : excitation rate
      r_I : inhibition rate

    Returns:
      figure of r_I and r_E as a function of time

  """
  with plt.xkcd():
    fig = plt.figure(figsize=(6,4))
    plt.plot(t,r_E,':',color='b',label=r'$r_E$')
    plt.plot(t,r_I,':',color='r',label=r'$r_I$')
    plt.xlabel('time(ms)')
    plt.legend()
    plt.ylabel('Firing Rate (Hz)')
    plt.show()

def plot_rErI_Simple(t, r_E, r_I):
  """
    Args:
      t   : time
      r_E : excitation rate
      r_I : inhibition rate

    Returns:
      figure with two panels
      left panel: r_I and r_E as a function of time
      right panel: r_I as a function of r_E with Nullclines

  """
  with plt.xkcd():
    fig = plt.figure(figsize=(12,4))
    gs = gridspec.GridSpec(1, 2)
    # LEFT PANEL
    plt.subplot(gs[0])
    plt.plot(t,r_E,':',color='b',label=r'$r_E$')
    plt.plot(t,r_I,':',color='r',label=r'$r_I$')
    plt.xlabel('time(ms)')
    plt.legend()
    plt.ylabel('Firing Rate (Hz)')
    # RIGHT PANEL
    plt.subplot(gs[1])
    plt.plot(r_E,r_I,'k:')
    plt.plot(r_E[0],r_I[0],'go')

    plt.hlines(0,np.min(r_E),np.max(r_E),linestyles="dashed",color='b',label=r'$\frac{d}{dt}r_E=0$')
    plt.vlines(0,np.min(r_I),np.max(r_I),linestyles="dashed",color='r',label=r'$\frac{d}{dt}r_I=0$')

    plt.legend(loc='upper left')

    plt.xlabel(r'$r_E$')
    plt.ylabel(r'$r_I$')
    plt.show()

def plot_rErI_Matrix(t, r_E, r_I, Null_rE, Null_rI):
  """
    Args:
      t   : time
      r_E : excitation firing rate
      r_I : inhibition firing rate
      Null_rE: Nullclines excitation firing rate
      Null_rI: Nullclines inhibition firing rate
    Returns:
      figure with two panels
      left panel: r_I and r_E as a function of time
      right panel: r_I as a function of r_E with Nullclines

  """

  with plt.xkcd():
    fig = plt.figure(figsize=(12,4))
    gs = gridspec.GridSpec(1, 2)
    plt.subplot(gs[0])
    plt.plot(t,r_E,':',color='b',label=r'$r_E$')
    plt.plot(t,r_I,':',color='r',label=r'$r_I$')
    plt.xlabel('time(ms)')
    plt.ylabel('Firing Rate (Hz)')
    plt.legend()
    plt.subplot(gs[1])
    plt.plot(r_E,r_I,'k:')
    plt.plot(r_E[0],r_I[0],'go')

    plt.plot(r_E,Null_rE,':',color='b',label=r'$\frac{d}{dt}r_E=0$')
    plt.plot(r_E,Null_rI,':',color='r',label=r'$\frac{d}{dt}r_I=0$')
    plt.legend(loc='best')
    plt.xlabel(r'$r_E$')
    plt.ylabel(r'$r_I$')
    plt.show()
```

---
# Section 1 Euler Method

```python
#@title Video 1: Intro to Numerical Methods and the Euler Method
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="ORbqe7TamVI", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

The Euler method is one of the most simple and elegant methods to approximate a differential. It was designed by [Leonhard Euler](https://en.wikipedia.org/wiki/Leonhard_Euler) (1707-1783).
Simply put we just replace the derivative in the differential equation by the formula for a line and re-arrange.

## Section 1.1: Slope of a line
The slope is the rate of change between two points. The formula for the slope of a line between the points $(t_0,y(t_0))$ and $(t_1,y(t_1))$ is given by:
$$ m=\frac{y(t_1)-y(t_0)}{t_1-t_0}, $$
which can be written as
$$ m=\frac{y_1-y_0}{t_1-t_0}, $$
or as
$$ m=\frac{\Delta y_0}{\Delta t_0}, $$
where $\Delta y_0=y_1-y_0$ and $\Delta t_0=t_1-t_0$ or in words as
$$ m=\frac{\text{ Change in y} }{\text{Change in t}}. $$
The slope can be used an approximation of the derivative such that
$$ \frac{d}{dt}y(t)\approx \frac{y(t_0+\Delta t)-y(t_0)}{t_0+\Delta t-t_0}=\frac{y(t_0+dt)-y(t_0)}{\Delta t}$$
where $\Delta t$ is a time-step.

### Interactive Demo 1.1: Slope of a Line
The plot below shows a function $y(t)$ in blue, its exact derivative $ \frac{d}{dt}y(t)$ at $t_0=1$ in black and the approximate derivative calculated using the slope formula $=\frac{y(1+\Delta t)-y(1)}{\Delta t}$ for different time-steps sizes $\Delta t$ in green.

Interact with the widget to see how time-step impacts the accuracy of the slope which is the estimate of the derivative.

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(

    dt=widgets.FloatSlider(1, min=0., max=4., step=.1,
                                layout=my_layout)

)

def Pop_widget(dt):
  plot_slope(dt)
  plt.show()
```

### Think! 1.1: How does the size of $dt$ affect the approximation?

```python
# to_remove explanation

"""
  The larger the time-step $dt$, the worse job the formula of the slope of a
  line does to approximate the derivative.
"""
```

---
# Section 2: Population Differential Equation

Linking with the differential equation tutorial, we will use the population to get an intuitive feel of Euler method to approximate the solution of a differential equation. We start with the population because it has an exact solution with no discontinuities unlike the Leaky Integrate and Fire model.

The population differential equation is
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= \alpha p(t)\\\\
p(0)&=p_0 \quad \text{Initial Condition}
\end{align*}

where $p(t)$ is the population of the world and $\alpha$ is a parameter representing birth rate, with
the exact solution
$$ p(t)=p_0e^{\alpha}.$$

To numerically approximate the population differential equation we replace the derivate with the slope of the line to get the discrete (not continuous) equation:

\begin{align*}
\\
\frac{p_1-p_0}{t_1-t_0} &= \alpha p_0\\\\
p(0)&=p_0 \quad \text{Initial Condition}
\end{align*}

where $p_1$ is the approximate of $p(t_1)$. Let $\Delta t=t_1-t_0$ be the time step and re-arrange the equation gives

\begin{align*}
\color{red}{p_1}&=\color{green}{p_0}+\color{blue}{\Delta t} (\color{blue}{\alpha} \color{green}{p_0})
\end{align*}

where $\color{red}{p_1}$ is the unknown future, $\color{green}{p_0}$ is the known current population, $\color{blue}{\Delta t}$ is the chosen time-step parameter and $\color{blue}{\alpha}$ is the given birth rate parameter.
Another way to read the re-agranged discrete equation is:

\begin{align*}
\color{red}{\text{"Future Population"}}&=\color{green}{\text{ "Current Population"}}+\color{blue}{\text{ time-step}} \times (\color{blue}{\text{ "Birth rate}}\text{ times } \color{green}{\text{ Current Population"}}).
\end{align*}

So pretty much we can use the current population and add a bit of the dynamics of the differential equation to predict the future. We will make millions, but wait there is always an error.

## Exercise 2: One step of Euler Method
Given the population differential equation
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= 0.3 p(t),\\\\
p(t_0=1)&=e^{0.3},\quad \text{Initial Condition}
\end{align*}

code the Euler method to estimate the population $p_1$ at $t_1=2$ given the time-step $\Delta t=1$ (for brevity, $\Delta t$ is `dt` in the code).

```python
dt = 1    # time-step
t_0 = 1   # initial time
p_0 = np.exp(0.3*t_0)    # initial population

########################################################################
## TODO for students: calculate the estimate solution of p at t_1
## Complete line of code and remove
raise NotImplementedError("Student exercise: calculate the estimate solution of p at t_1")
########################################################################

dp = ...
p_1 = ...
print(p_1)
```

```python
# to_remove solution
dt = 1    # time-step
t_0 = 1   # initial time
p_0 = np.exp(0.3*t_0)    # initial population

dp = dt * 0.3 * p_0
p_1 = p_0 + dp
print(p_1)
```

---
# Section 3: Euler Error

The solution of the Euler method $p_1$ is an estimate of the exact solution $p(t_1)$ at $t_1$ which means there is a bit of error $e_1$ which gives the equation
\begin{align*}
p(t_1)&=p_1+e_1\\\\
\text{Rearranging}\\\\
e_1&=p(t_1)-p_1.
\end{align*}

Most of the time we do not know the exact answer $p(t_1)$ but in the population equation we have the exact solution

$$ p(t)=p_0e^{\alpha}.$$

This means we can explore what the error looks like.

## Interactive Demo 3: Euler Error
Interact with the widget to see how the time-step $dt$ impacts the estimate $p_1$ and the error $e_1$ of the Euler method.

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
    dt=widgets.FloatSlider(.3, min=0., max=4., step=.1,
                                layout=my_layout)
)

def Pop_widget(dt):
  plot_StepEuler(dt)
  plt.show()
```

## Think! 3: Effect of $\Delta t$ on Euler Error

1. What happens to the estimate $p_1$ as the time-step $\Delta t$ increases?

2. Is there a relationship between the size of $\Delta t$ and $e_1$?

3. How would you improve the error $e_1$?

```python
# to_remove explanation

"""
  1. The larger the time-step $\Delta t$ the more the estimate $p_1$ deviates
  from the exact $p(t_1)$.

  2.

  3.
"""
```

The error $e_1$ from one timestep is known as the __local error__. For the Euler method the local error is linear, that means that the error decreases linearly as a function of timestep $O(\Delta t)$.

## Section 3.1: Time-Steps
In the above exercise we saw that by increasing the step size $dt$ the error between the estimate $p_1$ and the exact $p(t_1)$ increased with the largest error in the example above was for $\Delta t=4$ which gave the interval $[1, 5]$,
$$n=\frac{5-1}{\Delta t}=\frac{5-1}{4}=1,$$
hence only $1$ time-step of size $dt=4$, giving
$$ t_0=1,\ \text{ and } t_1=5. $$

To address decrease the error we divide the interval $[1, 5]$ into more steps using a smaller $\Delta t$.
In the plot below we use $\Delta t=1$, which divides the interval into four segments
$$n=\frac{5-1}{1}=4,$$
giving
$$ t_0=1, \ t_1=2, \ t_2=3, \ t_3=4 \ \text{ and } t_4=5. $$
This can be written as
$$ t_i=1+i\times1, \text{ for } i=0,\cdots 4, $$
and more generally as
$$ t_i=t_0+i\times dt, \text{ for } i=0,\cdots n. $$

```python
# @markdown *Execute this cell to visualize time steps*
dt  =1
t  =np.arange(1, 5+dt/2, dt) # Time from 1 to 5 years in 0.1 steps
with plt.xkcd():
    fig = plt.figure(figsize=(8, 2))
    plt.plot(t, 0*t, ':o')
    plt.plot(t[0] ,0*t[0],':go',label='Initial Condition')
    plt.text(t[0]-0.1, 0*t[0]-0.03, r'$t_0$')
    plt.text(t[1]-0.1, 0*t[0]-0.03, r'$t_1$')
    plt.text(t[2]-0.1, 0*t[0]-0.03, r'$t_2$')
    plt.text(t[3]-0.1, 0*t[0]-0.03, r'$t_3$')
    plt.text(t[4]-0.1, 0*t[0]-0.03,r'$t_4$')
    plt.text(t[0]+0.5, 0*t[0]+0.02, r'$\Delta t$')
    plt.text(t[1]+0.5, 0*t[0]+0.02, r'$\Delta t$')
    plt.text(t[2]+0.5, 0*t[0]+0.02, r'$\Delta t$')
    plt.text(t[3]+0.5, 0*t[0]+0.02, r'$\Delta t$')
    plt.yticks([])#plt.ylabel('Population (millions)')
    plt.xlabel('time(years)')
    plt.show()
```

## Section 3.2: The Difference Equation
Using the Euler method the differential equation is written as a series of difference equations of the form:
\begin{align*}
\color{red}{p_{i+1}}&=\color{green}{p_i}+\color{blue}{\Delta t} (\color{blue}{\alpha} \color{green}{p_i})\\
&\text{for } i=0,1,\cdots n-1
\end{align*}
where $\color{red}{p_{i+1}}$ is the unknown estimate of the future population at $t_{i+1}$, $\color{green}{p_i}$ is the known current population estimate at $t_i$, $\color{blue}{\Delta t}$ is the chosen time-step parameter and $\color{blue}{\alpha}$ is the given birth rate parameter.

### Coding Exercise 3.2: Step, step, step
Given the population differential equation
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= 0.3 p(t),\\
p(t_0=1)&=e^{0.3},\quad \text{Initial Condition,}
\end{align*}

code the difference equation

\begin{align*}
\color{red}{p_{i+1}}&=\color{green}{p_i}+\color{blue}{dt} (\color{blue}{0.3} \color{green}{p_i}),\\
\color{green}{p_0}&=e^{0.3},\quad \text{Initial Condition,}\\
&\text{for } i=0,1,\cdots 4,\\
\end{align*}

to estimate the population on the interval $[1,5]$ with a time-step $dt=1$.

```python
dt = 1
t = np.arange(1, 5+dt/2, dt) # Time from 0 to 10 years in 0.1 steps
n = len(t)
p = np.ones(n)
p[0] = np.exp(0.3*t[0]) # Initial Condition

fig = plt.figure(figsize=(6, 4))
plt.plot(t, np.exp(0.3*t), 'k', label='Exact Solution')

for i in range(n-1):
  ########################################################################
  ## TODO for students: calculate the population step for each time point
  ## Complete line of code and remove
  raise NotImplementedError("Student exercise: calculate the population step for each time point")
  ########################################################################

  dp = ...
  p[i+1] = ...

plt.plot(t, p,':o', label='Euler Estimate')
plt.vlines(t, p, np.exp(0.3*t),
           colors='r', linestyles='dashed', label=r'Error $e_i$')

plt.ylabel('Population (millions)')
plt.legend()
plt.xlabel('Time (years)')
plt.show()
```

```python
# to_remove solution
dt = 1
t = np.arange(1, 5+dt/2, dt) # Time from 0 to 10 years in 0.1 steps
n = len(t)
p = np.ones(n)
p[0] = np.exp(0.3*t[0]) # Initial Condition

with plt.xkcd():

  fig = plt.figure(figsize=(6, 4))
  plt.plot(t, np.exp(0.3*t), 'k', label='Exact Solution')

  for i in range(n-1):
    dp = dt * 0.3 * p[i]
    p[i+1] = p[i] + dp

  plt.plot(t, p,':o', label='Euler Estimate')
  plt.vlines(t, p, np.exp(0.3*t),
             colors='r', linestyles='dashed', label=r'Error $e_i$')

  plt.plot(t[0], p[0], 'go', label='Intial Condition')
  plt.ylabel('Population (millions)')
  plt.legend()
  plt.xlabel('Time (years)')
  plt.show()
```

The error is smaller for 4 time-step than taking one large time step from 1 to 5 but do note that the error is increasing for each step, this is known as __global error__ so the futher in time you want to predicted the larger the error.
You can read the theorems [here.](https://colab.research.google.com/github/john-s-butler-dit/Numerical-Analysis-Python/blob/master/Chapter%2001%20-%20Euler%20Methods/102_Euler_method_with_Theorems_nonlinear_Growth_function.ipynb)

---
# Section 4: Generalised Euler Method

The Euler method can be applied to all first order differential equations of the form
\begin{align*}
\frac{d}{dt}y(t)&=f(t,y(t)),\\
y(t_{0})&=y_0,\\
\end{align*}
on an interval $[a,b]$.

The differential equation can be written using the Euler method as
\begin{align*}
\frac{\color{red}{y_{i+1}}-\color{green}{y_i}}{\color{blue}{\Delta t}}&=f(\color{blue}{t_i},\color{green}{y_i}),\\
\color{red}{y_{i+1}}&=\color{green}{y_i}+\color{blue}{\Delta t}f(\color{blue}{t_i},\color{green}{y_i}),\\
&\text{ for } i=0, \cdots n-1,\\
y(\color{blue}{t_{0}})&=\color{green}{y_0},\\
\end{align*}
where  $\color{red}{y_{i+1}}$ is the unknown estimate at $t_{i+1}$, $\color{green}{y_i}$ is the known at $t_i$, $\color{blue}{\Delta t}$ is the chosen time-step parameter, $\color{blue}{t_i}$ is the time point and $f$ is the right handside of the differential equation.
The time discrete time stesp are:
\begin{align*}
\color{blue}{t_{i}}&=\color{blue}{t_0}+\color{blue}{i}\color{blue}{\Delta t},\\
n&=\frac{b-a}{\Delta t}\\
&\text{ for } i=0, \cdots n.\\
\end{align*}
Once again this can be simply put into words:
\begin{align*}
\color{red}{\text{ "Future" }}&=\color{green}{\text{ "Current Info" }}+\color{blue}{\text{ time-step } }\times\text{ "Dynamics of the system which is a function of } \color{blue}{ \text{ time }} \text{ and }\color{green}{ \text{ Current Info." }}
\end{align*}

---
# Section 5: Euler Method for the Leaky Integrate and Fire

```python
#@title Video 2: Leaky Integrate and Fire Model
video = YouTubeVideo(id="JWI2msDBZ7o", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

The Leaky Integrare and Fire differential equation is:

\begin{align}
\frac{dV}{dt} = \frac{-(V-E_L) + R_mI(t)}{\tau_m}\,
\end{align}

where $\tau_m$ is the time constant, $V$ is the membrane potential, $E_L$ is the resting potential, $R_m$ is leak resistance and $I(t)$ is the external input current.

The solution can be estimated by applyting the Euler method to give the difference equation:

\begin{align}
\frac{\color{red}{V_{i+1}}-\color{green}{V_{i}}}{\color{blue}{\Delta t}}=\big(\frac{-(\color{green}{V_i}-\color{blue}{E_L}) + \color{blue}{R_m}I_i}{\color{blue}{\tau_m}}\big),\\
\end{align}

Re-arranging gives:

\begin{align}
\color{red}{V_{i+1}}=\color{green}{V_{i}}+\color{blue}{\Delta t}\big(\frac{-(\color{green}{V_i}-\color{blue}{E_L}) + \color{blue}{R_m}I_i}{\color{blue}{\tau_m}}\big),\\
\text{for } i=0\cdots n-1,
\end{align}

where $\color{red}{V_{i+1}}$ is unknown, $\color{green}{V_{i}} $ is known, $\color{blue}{E_L}$, $\color{blue}{R_m}$ and $\color{blue}{\tau_m}$ are known parameters, $\color{blue}{\Delta t}$ is a chosen time-step and  $I(t_i)$ is a function for an external input current.

## Coding Exercise 5: LIF and Euler
Code the difference equation for the LIF:

\begin{align}
\color{red}{V_{i+1}}=\color{green}{V_{i}}+\color{blue}{\Delta t}\big(\frac{-(\color{green}{V_i}-\color{blue}{E_L}) + \color{blue}{R_m}I_i}{\color{blue}{\tau_m}}\big),\\
\text{for } i=0\cdots n-1,
\end{align}

with the given parameters set as:
* `V_reset = -75,`
* `E_L = -75,`
* `tau_m = 10,`
* `R_m = 10.`

The plot below is a figure with with three panels;
* the top panel is the sinusoidal input, $I$,
* the middle panel is the estimate membrane potential $V_i$ to illustrative the spike $V_i$ is set to $0$ and then reset,
* the bottom panel is the raster plot with each dot indicating a spike.

As electrophysiologists normally listen to spikes when conducting experiments,  listen to the music of different firing rates by changing the input value $I$.

```python
def Euler_Integrate_and_Fire(I, time, dt):
  """
  Args:
    I: Input
    time: time
    dt: time-step
  Returns:
    Spike: Spike count
    Spike_time: Spike times
    V: membrane potential esitmated by the Euler method
  """

  Spike = 0
  tau_m = 10
  R_m = 10
  t_isi = 0
  V_reset = E_L = -75
  n = len(time)
  V = V_reset * np.ones(n)
  V_th = -50
  Spike_time = []

  for i in range(n-1):
    #######################################################################
    ## TODO for students: calculate the estimate solution of V at t[i+1]
    ## Complete line of codes for dV and remove
    ## Run the code in Section 5.1 or 5.2 to see the output!
    raise NotImplementedError("Student exercise: calculate the estimate solution of V at t[i+1]")
    ########################################################################

    dV = ...
    V[i+1] = V[i] + dt*dV

    # Discontinuity for Spike
    if V[i] > V_th:
      V[i] = 0
      V[i+1] = V_reset
      t_isi = time[i]
      Spike = Spike + 1
      Spike_time = np.append(Spike_time, time[i])

  return Spike, Spike_time, V
```

```python
# to_remove solution

def Euler_Integrate_and_Fire(I, time, dt):
  """
  Args:
    I: Input
    time: time
    dt: time-step
  Returns:
    Spike: Spike count
    Spike_time: Spike times
    V: membrane potential esitmated by the Euler method
  """

  Spike = 0
  tau_m = 10
  R_m = 10
  t_isi = 0
  V_reset = E_L = -75
  n = len(time)
  V = V_reset * np.ones(n)
  V_th = -50
  Spike_time = []

  for i in range(n-1):
    dV = (-(V[i] - E_L) + R_m*I[i]) / tau_m
    V[i+1] = V[i] + dt*dV

    # Discontinuity for Spike
    if V[i] > V_th:
      V[i] = 0
      V[i+1] = V_reset
      t_isi = time[i]
      Spike = Spike + 1
      Spike_time = np.append(Spike_time, time[i])

  return Spike, Spike_time, V
```

## Section 5.1: Sinusoidal Input to LIF
The plot below shows the estimated solution of the LIF using the Euler method for a sinusoidal input.

```python
dt = 1
t = np.arange(0, 1000, dt)
I = np.sin(4 * 2 * np.pi * t/1000) + 2
Spike, Spike_time, V = Euler_Integrate_and_Fire(I, t, dt)

plot_IF(t, V,I,Spike_time)
ipd.Audio(V, rate=1000/dt)
plt.show()
```

### Section 5.2: Random Input to the LIF
The plot below shows a figure with with three panels;
* the top panel is the noisy input, $I_i$,
* the middle panel is the estimate membrane potential $V_i$ to illustrative the spike $V_i$ is set to $0$ and then reset,
* the bottom panel is the raster plot with each dot indicating a spike.

Listen to the music of different firing rates for a noisy input $I$.

```python
dt = 1
t = np.arange(0, 1000, dt)
I = np.random.randn(len(t)) + 2.5
Spike, Spike_time, V = Euler_Integrate_and_Fire(I, t, dt)

plot_IF(t, V, I, Spike_time)
ipd.Audio(V, rate=1000/dt)
plt.show()
```

---
# Section 6: Systems of Differential Equations

```python
#@title Video 3: Systems of Differential Equations
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="0rTYV7g0edM", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

To get to grips with solving a system of differential equations using the Euler method, we will simplify the Wilson Cowan model, a set of equations that will be exploted in more detail on W?D?. The next few sections will apply the Euler method to a system of equations and will also give you a feel of a how to investigate a system of differential equations.

We will start with a linear version of the Wilson Cowan model. Consider the equations,

\begin{align}
\tau_E \frac{dr_E}{dt} &= w_{EE}r_E+w_{EI}r_I \\
\tau_I \frac{dr_I}{dt} &= w_{IE}r_E+w_{II}r_I
\end{align}

$r_E(t)$ represents the average activation (or firing rate) of the excitatory population at time $t$, and $r_I(t)$ the activation (or firing rate) of the inhibitory population. The parameters $\tau_E$ and $\tau_I$ control the timescales of the dynamics of each population. Connection strengths are given by: $w_{EE}$ (E $\rightarrow$ E), $w_{EI}$ (I $\rightarrow$ E), $w_{IE}$ (E $\rightarrow$ I), and $w_{II}$ (I $\rightarrow$ I). The terms $w_{EI}$ and $w_{IE}$ represent connections from inhibitory to excitatory population and vice versa, respectively.

## Coding Exercise 6: Euler on a Simple System
We will further simplify the equations by setting $w_{EE}$ and $w_{II}$ to zero, we now have the equations

\begin{align}
\tau_E \frac{dr_E}{dt} &= w_{EI}r_I, \\
\tau_I \frac{dr_I}{dt} &= w_{IE}r_E, \qquad (1)
\end{align}

where $\tau_E=100$ and $\tau_I=120$, no internal connection $w_{EE}=w_{II}=0$, and $w_{EI}=-1$ and $w_{IE}=1$,
with the initial conditions

\begin{align}
r_E(0)=30, \\
r_I(0)=20.
\end{align}

The solution can be approximated using the Euler method such that we have the difference equations:

\begin{align*}
\frac{\color{red}{rE_{i+1}}-\color{green}{rE_i}}{\color{blue}{\Delta t}}&=\big(\frac{\color{blue}{w_{EI}}\color{green}{rI_i}}{\color{blue}{\tau_E}}\big),\\
\frac{\color{red}{rI_{i+1}}-\color{green}{rI_i}}{\color{blue}{\Delta t}}&=\big(\frac{\color{blue}{w_{IE}}\color{green}{rE_i}}{\color{blue}{\tau_I}}\big),\\
\end{align*}

Re-arranging:

\begin{align*}
\color{red}{rE_{i+1}}&=\color{green}{rE_i}+\color{blue}{\Delta t}\big(\frac{\color{blue}{w_{EI}}\color{green}{rI_i}}{\color{blue}{\tau_E}}\big),\\
\color{red}{rI_{i+1}}&=\color{green}{rI_i}+\color{blue}{\Delta t}\big(\frac{\color{blue}{w_{IE}}\color{green}{rE_i}}{\color{blue}{\tau_I}}\big),\\
&\text{ for  } i=0, \cdots , n-1,\\\\
rE_0&=30,\\
rI_0&=20.\\
\end{align*}

where $\color{red}{rE_{i+1}}$ and  $\color{red}{rI_{i+1}}$ are unknown, $\color{green}{rE_{i}} $ and  $\color{green}{rI_{i}} $ are known, $\color{blue}{w_{EI}}=-1$, $\color{blue}{w_{IE}}=1$ and $\color{blue}{\tau_E}=100$ and $\color{blue}{\tau_I}=120$  are known parameters and $\color{blue}{\Delta t}$ is a chosen time-step.

__Code the difference equations to estimate $r_{E}$ and $r_{I}$.__

Note that the equations have to estimated in the same `for` loop as they depend on each other.

```python
def Euler_Simple_Linear_System(t, dt):
  """
  Args:
    time: time
  dt  : time-step
  Returns:
    r_E : Excitation Firing Rate
    r_I : Inhibition Firing Rate

  """

  tau_E = 100
  tau_I = 120
  n = len(t)
  r_I = 20*np.ones(n)
  r_E = 30*np.ones(n)

  #######################################################################
  ## TODO for students: calculate the estimate solutions of r_E and r_I at t[i+1]
  ## Complete line of codes for dr_E and dr_I and remove
  raise NotImplementedError("Student exercise: calculate the estimate solutions of r_E and r_I at t[i+1]")
  ########################################################################

  for i in range(n-1):
    dr_E = ...
    r_E[i+1] = r_E[i] + dt*dr_E

    dr_I = ...
    r_I[i+1] = r_I[i] + dt*dr_I

  return r_E, r_I

dt = 0.1 # time-step
t = np.arange(0, 1000, dt)
r_E, r_I = Euler_Simple_Linear_System(t, dt)
plot_rErI(t, r_E, r_I)
```

```python
# to_remove solution

def Euler_Simple_Linear_System(t, dt):
  """
  Args:
    time: time
    dt: time-step
  Returns:
    r_E: Excitation Firing Rate
    r_I: Inhibition Firing Rate
  """

  tau_E = 100
  tau_I = 120
  n = len(t)
  r_I = 20*np.ones(n)
  r_E = 30*np.ones(n)

  for i in range(n-1):
    dr_E = -r_I[i]/tau_E
    r_E[i+1] = r_E[i] + dt*dr_E

    dr_I = r_E[i]/tau_I
    r_I[i+1] = r_I[i] + dt*dr_I

  return r_E, r_I

dt = 0.1 # time-step
t = np.arange(0, 1000, dt)
r_E, r_I = Euler_Simple_Linear_System(t, dt)
plot_rErI(t, r_E, r_I)
```

### Think! 6: Simple Euler Solution to the Wilson Cowan model

1. Is the simulation biologically plausible?
2. What is the effect of combined excitation and inhibition?

```python
#to_remove explanation

"""
  1. The simulation is not biologically plausible as there are negative firing
  rates but a mathematician could just say that it is firing rate relative to
  baseline.
  2. Excitation and inhibition creates an oscillation.
"""
```

## Section 6.1: Phase Plane Plot and Nullcline
When there are two differential equations describing the interaction of two processes like excitation $r_{E}$ and inhibition $r_{I}$ that are both dependent on each other one way of plotting them is as a function of each other which is called a phase plane plot. The phase plane plot can give insight give insight into the state of the system but more about that later in Neuromatch Academy.

An addition to the phase plane plot are the "nullcline". These lines plot when the rate of change $\frac{d}{dt}$ of the variables is equation to $0$. We saw a variation of this for a single differential equation in the differential equation tutorial.

As we have have two differential equations we set $\frac{dr_E}{dt}=0$ and $\frac{dr_I}{dt}=0$ which gives the equations,

\begin{align}
0&= w_{EI}r_I, \\
0&= w_{IE}r_E,\\
\end{align}

these are a unique example as they are a vertical and horizontal line. Where the lines cross is the stable point which the $r_E(t)$ excitatory population and $r_I(t)$ the inhibitory population orbit around.

```python
# @markdown Excute the code to plot the solution to the system
plot_rErI_Simple(t, r_E, r_I)
```

### Think! 6.1: Discuss the Plots

1. Which representation is more intuitive (and useful), the time plot or the phase plane plot?
2. Why do we only see one circle?
3. What do the quadrants represent?

```python
# to_remove explanation

"""
  1.
  2. The solution is stable.
  3.
"""
```

---
# Section 7: Linear System of Differential Equations

Building up the equations in the previous section we re-introduce internal connections $w_{EE}$, $w_{II}$. The two coupled differential equations, each representing the dynamics of the excitatory or inhibitory population are now:

\begin{align}
\tau_E \frac{dr_E}{dt} &=w_{EE}r_E +w_{EI}r_I, \\
\tau_I \frac{dr_I}{dt} &=w_{IE}r_E +w_{II}r_I ,
\end{align}

where $\tau_E=100$ and $\tau_I=120$, $w_{EE}=1$ and $w_{II}=-1$, and $w_{EI}=-5$ and $w_{IE}=0.6$,
with the initial conditions

\begin{align}
r_E(0)=30, \\
r_I(0)=20.
\end{align}

The solution can be approximated using the Euler method such that the equations become:

\begin{align*}
\frac{\color{red}{rE_{i+1}}-\color{green}{rE_i}}{\color{blue}{\Delta t}}&=\big(\frac{\color{blue}{w_{EE}}\color{green}{rE_i}+\color{blue}{w_{EI}}\color{green}{rI_i}}{\color{blue}{\tau_E}}\big),\\
\frac{\color{red}{rI_{i+1}}-\color{green}{rI_i}}{\color{blue}{\Delta t}}&=\big(\frac{\color{blue}{w_{II}}\color{green}{rI_i}+\color{blue}{w_{IE}}\color{green}{rE_i}}{\color{blue}{\tau_I}}\big),\\
\end{align*}

Re-arranging:

\begin{align*}
\color{red}{rE_{i+1}}&=\color{green}{rE_i}+\color{blue}{\Delta t}\big(\frac{\color{blue}{w_{EE}}\color{green}{rE_i}+\color{blue}{w_{EI}}\color{green}{rI_i}}{\color{blue}{\tau_E}}\big),\\
\color{red}{rI_{i+1}}&=\color{green}{rI_i}+\color{blue}{\Delta t}\big(\frac{\color{blue}{w_{II}}\color{green}{rI_i}+\color{blue}{w_{IE}}\color{green}{rE_i}}{\color{blue}{\tau_I}}\big),\\
&\text{ for } i=0, \cdots n-1,\\\\
rE_0&=30,\\
rI_0&=20.\\
\end{align*}

where $\color{red}{rE_{i+1}}$ and  $\color{red}{rI_{i+1}}$ are unknown, $\color{green}{rE_{i}} $ and  $\color{green}{rI_{i}} $ are known, $\color{blue}{w_{EI}}=-1$, $\color{blue}{w_{IE}}=1$ and $\color{blue}{\tau_E}=100$ and $\color{blue}{\tau_I}=120$  are known parameters and $\color{blue}{\Delta t}=0.1$ is a chosen time-step.

## Coding Exercise 7: Implement Euler for a System of Equations
Code the Euler method for the linear system,
\begin{align}
\tau_E \frac{dr_E}{dt} &=w_{EE}r_E +w_{EI}r_I, \\
\tau_I \frac{dr_I}{dt} &=w_{IE}r_E +w_{II}r_I,
\end{align}

where $\tau_E=100$ and $\tau_I=120$, $w_{EE}=1$ and $w_{II}=-1$, and $w_{EI}=-5$ and $w_{IE}=0.6$,
with the initial conditions,
\begin{align}
r_E(0)=30, \\
r_I(0)=20.
\end{align}
While this might seem very repititive to an exercise above, it just shows how straightforward it is to convert a system of differential equations into simple pluses and minuses and a bit of multiplication.

```python
def Euler_Linear_System_Matrix(t, dt, w_EE=1):
  """
  Args:
    time: time
    dt: time-step
    w_EE: Excitation to excitation weight
  Returns:
    r_E: Excitation Firing Rate
    r_I: Inhibition Firing Rate
    N_Er: Nullclines for drE/dt=0
    N_Ir: Nullclines for drI/dt=0
  """

  tau_E = 100
  tau_I = 120
  n = len(t)
  r_I = 20*np.ones(n)
  r_E = 30*np.ones(n)
  w_EI = -5
  w_IE = 0.6
  w_II = -1

  for i in range(n-1):

    ########################################################################
    # TODO for students: compute drE and drI and remove
    raise NotImplementedError("Student exercise: compute drE and drI")
    ########################################################################

    # Calculate the derivatives of the E and I populations
    drE = ...
    r_E[i+1] = r_E[i] + dt*drE

    drI = ...
    r_I[i+1] = r_I[i] + dt*drI

  N_Er = -w_EE / w_EI*r_E
  N_Ir = -w_IE / w_II*r_E

  return r_E, r_I, N_Er, N_Ir

dt = 0.1 # time-step
t = np.arange(0, 1000, dt)
r_E, r_I, _, _ = Euler_Linear_System_Matrix(t, dt)
plot_rErI(t, r_E, r_I)
```

```python
# to_remove solution

def Euler_Linear_System_Matrix(t, dt, w_EE=1):
  """
  Args:
    time: time
    dt: time-step
    w_EE: Excitation to excitation weight
  Returns:
    r_E: Excitation Firing Rate
    r_I: Inhibition Firing Rate
    N_Er: Nullclines for drE/dt=0
    N_Ir: Nullclines for drI/dt=0
  """

  tau_E = 100
  tau_I = 120
  n = len(t)
  r_I = 20*np.ones(n)
  r_E = 30*np.ones(n)
  w_EI = -5
  w_IE = 0.6
  w_II = -1

  for i in range(n-1):

    # Calculate the derivatives of the E and I populations
    drE = (w_EI*r_I[i] + w_EE*r_E[i]) / tau_E
    r_E[i+1] = r_E[i] + dt*drE

    drI = (w_II*r_I[i] + w_IE*r_E[i]) / tau_I
    r_I[i+1] = r_I[i] + dt*drI

  N_Er = -w_EE / w_EI*r_E
  N_Ir = -w_IE / w_II*r_E

  return r_E, r_I, N_Er, N_Ir

dt = 0.1 # time-step
t = np.arange(0, 1000, dt)
r_E, r_I, _, _ = Euler_Linear_System_Matrix(t, dt)
plot_rErI(t, r_E, r_I)
```

### Think! 7: Oscillations

1. What will happen to the oscillations if the time period is extended?
2. How would you control or predict the oscillations?

```python
# to_remove explanation

"""

"""
```

## Section 7.1: Phase Plane and Nullclines

Like before, we have two differential equations so we cane plot the results on a phase plane. We can also calculate the Nullclines when we set $\frac{dr_E}{dt}=0$ and $\frac{dr_I}{dt}=0$ which gives,
\begin{align}
0&= w_{EE}r_E+w_{EI}r_I, \\
0&= w_{IE}r_E+w_{II}r_I,
\end{align}
re-arranging as two lines
\begin{align}
r_I&= -\frac{w_{EE}}{w_{EI}}r_E, \\
r_I&= -\frac{w_{IE}}{w_{II}}r_E,
\end{align}
which crosses at the stable point.

The panel on the left shows excitation firing rate $r_E$ and inhibition firing rate  $r_I$ as a function of time. On the rightside the phase plane plot shows inhibition firing rate $r_I$ as a function of excitiation firing rate $r_E$ with the Nullclines for $\frac{dr_E}{dt}=0$ and $\frac{dr_I}{dt}=0.$

```python
# @markdown *Run this cell to visualize the phase plane*
dt = 0.1 # time-step
t = np.arange(0, 1000, dt)
r_E, r_I, N_Er, N_Ir = Euler_Linear_System_Matrix(t, dt)
plot_rErI_Matrix(t, r_E, r_I, N_Er, N_Ir)
```

### Interactive Demo 7.1: A small change changes everything
To illustrate that even changing one parameter in system of differential equations has a large impact on the solutions of the excitation firing rate $r_E$ and inhibition firing rate $r_I$.
Take note of:
1. How the solution changes for positive and negative of $w_{EE}$ pay close attention to the axis.

2. How would you maintain a stable oscillation.

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
    w_EE=widgets.FloatSlider(1, min=-1., max=2., step=.1,
                                layout=my_layout)
)

def Pop_widget(w_EE):
  dt = 0.1 # time-step
  t = np.arange(0,1000,dt)
  r_E, r_I, N_Er, N_Ir = Euler_Linear_System_Matrix(t, dt, w_EE)
  plot_rErI_Matrix(t, r_E, r_I, N_Er, N_Ir)
  plt.show()
```

---
# Take Home

```python
#@title Video 4: Numerical Methods - Take Home and Bonus Materials
video = YouTubeVideo(id="NDXcNZXeLsw", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

Using pretty much the formula for the slope of a line, the solution of differential equation can be estimated with reasonable accuracy.
This will be much more relevant when dealing with more complicated (non-linear) differential equations where there is no known exact solution.

---
# Links to Neuromatch Days

The Euler method is used to estimate differential equations in a number of different Neuromatch days:
* The LIF model is discussed in more details in Model Types (Week 1 Day 1) and Real Neurons (Week 2 Day 3).
* Drift Diffusion model which is a differential equation for decision making is discussed in Linear Systems (Week 2 Day 2).
* Phase-plane plots are discussed in Linear Systems (Week 2 Day 2) and Dynamic Networks (Week 2 Day 4).
* The Wilson-Cowan model is discussed in Dynamic Networks (Week 2 Day 4).

---
# References
1. Lotka, A. L, (1920)   Analytical note on certain rhythmic  relations inorganic  systems.Proceedings of the National Academy of Sciences,6(7):410–415,1920.

2. Brunel N, van Rossum MC. Lapicque's 1907 paper: from frogs to integrate-and-fire. Biol Cybern. 2007 Dec;97(5-6):337-9. doi: 10.1007/s00422-007-0190-0. Epub 2007 Oct 30. PMID: 17968583.

# Bibliography
1. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational Neuroscience Series.
2. Strogatz, S. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering (studies in nonlinearity), Westview Press; 2 edition (29 July 2014)

## Supplemental Popular Reading List
1. Lindsay, G. (2021). Models of the Mind: How Physics, Engineering and Mathematics Have Shaped Our Understanding of the Brain. Bloomsbury Publishing.
2. Strogatz, S. (2004). Sync: The emerging science of spontaneous order. Penguin UK.

## Popular Podcast
1. Strogatz, S. (Host). (2020-), Joy of X  https://www.quantamagazine.org/tag/the-joy-of-x/ Quanta Magazine

---
# Bonus 1: The 4th Order Runge Kutta
Another popular numerical method to estimate the solution of a differential equations of the general form:
$$ \frac{d}{dt}y=f(t,y)$$
is the 4th Order Runge Kutta:
$$k_1=f(t,y),$$
$$k_2=f(t+\frac{\Delta t}{2},y+\frac{\Delta t}{2}k_1),$$
$$k_3=f(t+\frac{\Delta t}{2},y+\frac{\Delta t}{2}k_2),$$
$$k_4=f(t+dt,y+\Delta tk_3),$$
$$y_{i+1}=y_{i}+\frac{\Delta t}{6}(k_1+2k_2+2k_3+k_4),$$
which is more accurate than the Euler method.

Given the population differential equation
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= 0.3 p(t)\\
p(t_0=1)&=e^{0.3}\quad \text{Initial Condition}
\end{align*}
the 4th Runge Kutta difference equation is
\begin{align*}
k_1&=0.3\color{green}{p_i},\\
k_2&=0.3(\color{green}{p_i}+\frac{\Delta t}{2}k_1),\\
k_3&=0.3(\color{green}{p_i}+\frac{\Delta t}{2}k_2),\\
k_4&=0.3(\color{green}{p_i}+k_3),\\
\color{red}{p_{i+1}}&=\color{green}{p_i}+\frac{\color{blue}{\Delta t}}{6}( \color{green}{k_1+2k_2+2k_3+k_4})\\
&\text{for } i=0,1,\cdots 4
\end{align*}
to estimate the population for $\Delta t=0.5$.

The code is implemented below. Note how much more accurate the 4th Order Runge Kutta (yellow) is compared to the Euler Method (blue).

```python
dt=0.5

t=np.arange(1, 5+dt/2, dt)

n = len(t)
p = np.ones(n)
pRK4 = np.ones(n)
p[0] = np.exp(0.3*t[0])
pRK4[0] = np.exp(0.3*t[0])# Initial Condition

with plt.xkcd():

  fig = plt.figure(figsize=(6, 4))
  plt.plot(t, np.exp(0.3*t), 'k', label='Exact Solution')

  for i in range(n-1):
    dp = dt*0.3*p[i]
    p[i+1] = p[i] + dp # Euler
    k1 = 0.3*(pRK4[i])
    k2 = 0.3*(pRK4[i] + dt/2*k1)
    k3 = 0.3*(pRK4[i] + dt/2*k2)
    k4 = 0.3*(pRK4[i] + dt*k3)
    pRK4[i+1] = pRK4[i] + dt/6 *(k1 + 2*k2 + 2*k3 + k4)

  plt.plot(t, p,':o', label='Euler Estimate')
  plt.plot(t, pRK4,':o', color='y', label='4th Order Runge Kutta Estimate')
  plt.vlines(t,p,np.exp(0.3*t),colors='r', linestyles='dashed',label=r'Error $e_i$')

  plt.plot(t[0], p[0], 'go', label='Intial Condition')
  plt.ylabel('Population (millions)')
  plt.legend()
  plt.xlabel('Time (years)')
  plt.show()
```

### Bonus Reference 1: A full course on numerical methods in Python

For a full course on Numerical Methods you can look here https://github.com/john-s-butler-dit/Numerical-Analysis-Python

# Bonus 2: Neural oscillations are a start toward understanding brain activity rather than the end
In an essay from May 2021 in PLOS Biology by Keith Dowling (a past Neuromatch TA) and M. Florencia Assaneo discussed a mathematical way of thinking about what should be defined as an oscillation.
This is an important question as a good deal of research has investigated oscillations of the brain, and as we saw earlier, a good deal of the equations we use to model neuronal activity result in an oscillation.

They motivated their approach using the Stuart–Landau equations which is a system of differential equations of the form

\begin{align}
 \frac{dx}{dt} &=\lambda x-\omega y -\gamma (x^2+y^2)x+s\\
 \frac{dy}{dt} &=\lambda y+\omega x -\gamma (x^2+y^2)y
\end{align}

where $s$ is input to the system, and $\lambda$, $\omega$ and $\gamma$ are parameters.
The Stuart–Landau equations are a well described system of differential equations that generates oscillations. For their purpose, they used the equations to illustrate what people should expect when conducting experiments looking for oscilations.
In their paper, using the Stuart–Landau equations, they outline
* "What is an oscillator?"
* "What an oscillator is not"
* "Not all that oscillates is oscillator"
* "Not all oscillators are alike."

The Euler form of the Stuart–Landau system of equations is:
\begin{align*}
\color{red}{x_{i+1}}&=\color{green}{x_i}+\color{blue}{\Delta t}\big(\lambda \color{green}{x_i}-\omega \color{green}{y_i} -\gamma (\color{green}{x_i}^2+\color{green}{y_i}^2)\color{green}{x_i}+s\big),\\
\color{red}{y_{i+1}}&=\color{green}{y_i}+\color{blue}{\Delta t}\big(\lambda \color{green}{y_i}+\omega \color{green}{x_i} -\gamma (\color{green}{x_i}^2+\color{green}{y_i}^2)\color{green}{y_i} \big),\\
&\text{ for } i=0, \cdots n-1,\\
x_0&=1,\\
y_0&=1,\\
\end{align*}
with $ \Delta t=0.1/1000$ ms.

```python
# @title Helper functions
def plot_Stuart_Landa(t, x, y, s):
  """
    Args:
      t  : time
      x  : x
      y  : y
      s : input
    Returns:
      figure with two panels
      top panel: Input as a function of time
      bottom panel: x
  """

  with plt.xkcd():
    fig = plt.figure(figsize=(14, 4))
    gs = gridspec.GridSpec(2, 2,  height_ratios=[1, 4], width_ratios=[4, 1])

    # PLOT OF INPUT
    plt.subplot(gs[0])
    plt.ylabel(r'$s$')
    plt.plot(t, s, 'g')
    #plt.ylim((2,4))

    # PLOT OF ACTIVITY
    plt.subplot(gs[2])
    plt.plot(t ,x)
    plt.ylabel(r'x')
    plt.xlabel(r't')
    plt.subplot(gs[3])
    plt.plot(x,y)
    plt.plot(x[0], y[0],'go')
    plt.xlabel(r'x')
    plt.ylabel(r'y')
    plt.show()

def Euler_Stuart_Landau(s,time,dt,lamba=0.1,gamma=1,k=25):
  """
  Args:
    I: Input
    time: time
    dt: time-step
  """

  n = len(time)
  omega = 4 * 2*np.pi
  x = np.zeros(n)
  y = np.zeros(n)
  x[0] = 1
  y[0] = 1

  for i in range(n-1):
    dx = lamba*x[i] - omega*y[i] - gamma*(x[i]*x[i] + y[i]*y[i])*x[i] + k*s[i]
    x[i+1] = x[i] + dt*dx
    dy = lamba*y[i] + omega*x[i] - gamma*(x[i]*x[i] + y[i]*y[i])*y[i]
    y[i+1] = y[i] + dt*dy

  return x, y
```

### Bonus 2.1: What is an Oscillator?
Doelling & Assaneo (2021), using the Stuart–Landau system, show different possible states of an oscilator by manipulating the $\lambda$ term in the equation.
From the paper: "this qualitative change in behavior takes place at λ = 0: For λ < 0, the system decays to a stable equilibrium, while for λ > 0, it keeps oscillating."

#### Interactive Demo Bonus 2.1
The plot below shows the estimated solution of the Stuart–Landau system with a base frequency $\omega$ set to $4\times 2\pi$, $\gamma=1$ and $k=25$ over 3 seconds. The input to the system $s(t)$ is plotted in the top panel, $x$ as a function of time in the the bottom panel and on the right the phase plane plot of $x$ and $y$. You can manipulate $\lambda$ to see how the oscilations change.

```python
# @markdown Make sure you execute this cell to enable the widget!
dt=0.1/1000
t=np.arange(0, 3, dt)

my_layout.width = '450px'
@widgets.interact(
    lamda=widgets.FloatSlider(1, min=-1., max=5., step=0.5,
                                layout=my_layout)
)

def Pop_widget(lamda):
    s=np.zeros(len(t))
    x,y=Euler_Stuart_Landau(s,t,dt,lamda)
    plot_Stuart_Landa(t, x, y, s)
    plt.show()
```

### Bonus 2.2 : Not all oscillators are alike

#### Interactive Demo Bonus 2: Stuart-Landau System
The plot below shows estimated solution of the Stuart–Landau system with a base frequency $\omega$ set to $4\times 2\pi$, $\lambda=1$, $\gamma=1$ and $k=50$ over 3 seconds the input to the system $s(t)=\sin(freq 2\pi t) $ in the top panel, $x$ as a function of time in the the bottom panel and on the right the phase plane plot of $x$ and $y$.

You can manipulate the frequency $freq$ of the input to see how the oscilations change and for frequencies $freq$ further and further from the base frequency of 4Hz the oscilations breaks down.

```python
# @markdown Make sure you execute this cell to enable the widget!
dt=0.1/1000
t=np.arange(0, 3, dt)

my_layout.width = '450px'
@widgets.interact(
    freq=widgets.FloatSlider(4, min=0.5, max=10., step=0.5,
                                layout=my_layout)
)

def Pop_widget(freq):
    s = np.sin(freq * 2*np.pi * t)

    x, y = Euler_Stuart_Landau(s, t, dt, lamba=1, gamma=.1, k=50)
    plot_Stuart_Landa(t, x, y, s)
```

### Bonus Reference 2
[Doelling, K. B., & Assaneo, M. F. (2021). Neural oscillations are a start toward understanding brain activity rather than the end. PLoS biology, 19(5), e3001234.](https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001234)


# Function Generation for the Training of λ-Nets

## Specitication of Experiment Settings

```python
#######################################################################################################################################
###################################################### CONFIG FILE ####################################################################
#######################################################################################################################################
sleep_time = 0 #minutes

config = {
    'data': {
        'd': 3, #degree
        'n': 5, #number of variables
        'monomial_vars': None, #int or None
        'laurent': False, #use Laurent polynomials (negative degree with up to -d)
        'neg_d': 0, #int or None
        'neg_d_prob': 0, #probability before other adjustments
        'sparsity': None,
        'sample_sparsity': 5,
        'x_max': 1,
        'x_min': 0,
        'x_distrib': 'uniform', #'normal', 'uniform',
        'a_max': 1,
        'a_min': -1,
        'polynomial_data_size': 50000, #number of polynomials to generate
        'noise': 0, #fraction of mean(abs(y_data))
        'noise_distrib': 'normal', #'normal', 'uniform'

        'function_generation_type': 'polynomial',# 'friedman1', 'polynomial'

        'shift_polynomial': False,

        'border_min': 0.2, # defines an intervall. Value is randomly chosen and defines the minimum gap between x_min / x_max and the outermost stationary points => two values (left and right gap will be generated per variable)
        'border_max': 0.4,
        'lower_degree_prob': 0.5, # probability that the degree of the whole polynomial will be reduced
        'a_random_prob': 0.5, # probability that a random generated function is used without adjustement

        'global_stationary_prob': 1, # probability that all variables are used for adjustement (0 recommended for higher number of variables)
        'bulge_min': 1, # bulge_min and bulge_max define an intervall of how much the function is bulged
        'bulge_max': 4,
        'min_variables_used': 2, # defines an Intervall of how many variables are used to get stationary points and therefore adjust the function
        'max_variables_used': 6,
        'max_monomials': 7, # maximum number of monomials, before adjusting the function (monomial of degree 0 is always defined, but is included in this number)
        'max_monomials_random': 10, #maximum number of monomials for random generated functions

        'same_training_all_lambda_nets': False,
    },
    'lambda_net': {
        'lambda_dataset_size': 5000, #number of samples per polynomial
    },
    'computation':{
        'n_jobs': 10,
        'use_gpu': False,
        'gpu_numbers': '0',
        'RANDOM_SEED': 1,
    }
}

```

```python
#######################################################################################################################################
########################################### IMPORT GLOBAL VARIABLES FROM CONFIG #######################################################
#######################################################################################################################################
globals().update(config['data'])
globals().update(config['lambda_net'])
globals().update(config['computation'])
```

## Imports

```python
import warnings
warnings.filterwarnings('ignore')
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

import tensorflow as tf
tf.get_logger().setLevel('WARNING')
tf.autograph.set_verbosity(2)

from itertools import product       # forms cartesian products
from more_itertools import random_product
from tqdm import tqdm_notebook as tqdm
import pickle

import numpy as np

import pandas as pd
from joblib import Parallel, delayed

import random
from random import sample

import os
import sys

from sklearn.datasets import make_regression
from sklearn.preprocessing import MinMaxScaler

from sympy import Symbol, sympify

import seaborn as sns

import random

import warnings

from time import perf_counter
```

```python
#######################################################################################################################################
###################################################### SET VARIABLES + DESIGN #########################################################
#######################################################################################################################################
variables = 'abcdefghijklmnopqrstuvwxyz'[:n]

os.environ['CUDA_VISIBLE_DEVICES'] = gpu_numbers if use_gpu else ''

sns.set_style("darkgrid")

random.seed(RANDOM_SEED)
np.random.seed(RANDOM_SEED)

pd.set_option('display.float_format', lambda x: '%.3f' % x)
pd.set_option('display.max_columns', 500)

warnings.filterwarnings('ignore')

#sys.path.append('..')
```

```python
from utilities.utility_functions import flatten, rec_gen, gen_monomial_identifier_list

list_of_monomial_identifiers_extended = []

if laurent:
    variable_sets = [list(flatten([[_d for _d in range(d+1)], [-_d for _d in range(1, neg_d+1)]])) for _ in range(n)]
    list_of_monomial_identifiers_extended = rec_gen(variable_sets)

    print('List length: ' + str(len(list_of_monomial_identifiers_extended)))
    #print('Number of monomials in a polynomial with ' + str(n) + ' variables and degree ' + str(d) + ': ' + str(sparsity))
    #print('Sparsity:' + str(sparsity))
    if len(list_of_monomial_identifiers_extended) < 500:
        print(list_of_monomial_identifiers_extended)

    list_of_monomial_identifiers = []
    for monomial_identifier in tqdm(list_of_monomial_identifiers_extended):
        if np.sum(monomial_identifier) <= d:
            if monomial_vars == None or len(list(filter(lambda x: x != 0, monomial_identifier))) <= monomial_vars:
                list_of_monomial_identifiers.append(monomial_identifier)
else:
    variable_list = ['x'+ str(i) for i in range(n)]
    list_of_monomial_identifiers = gen_monomial_identifier_list(variable_list, d, n)

print('List length: ' + str(len(list_of_monomial_identifiers)))
#print('Number of monomials in a polynomial with ' + str(n) + ' variables and degree ' + str(d) + ': ' + str(sparsity))
#print('Sparsity: ' + str(sparsity))
print(list_of_monomial_identifiers)

```

```python
from utilities.utility_functions import *
#######################################################################################################################################
####################################################### CONFIG ADJUSTMENTS ############################################################
#######################################################################################################################################
config['data']['sparsity'] = nCr(config['data']['n']+config['data']['d'], config['data']['d']) if not laurent else len(list_of_monomial_identifiers)
#nCr(config['data']['n']+config['data']['d'], config['data']['d']) if not laurent else (config['data']['d']*2+1)**config['data']['n'] - np.sum([i for i in range(1, config['data']['d']+1)])#- math.factorial(config['data']['d'])
#nCr(config['data']['n']+config['data']['d'], config['data']['d']) if not laurent else nCr(config['data']['n']+config['data']['d']*2, config['data']['d']*2)

config['data']['sample_sparsity'] = config['data']['sparsity'] if config['data']['sample_sparsity'] == None else config['data']['sample_sparsity']

#######################################################################################################################################
################################################## UPDATE VARIABLES ###################################################################
#######################################################################################################################################
globals().update(config['data'])
globals().update(config['lambda_net'])
globals().update(config['computation'])

initialize_utility_functions_config_from_curent_notebook(config)

#######################################################################################################################################
###################################################### PATH + FOLDER CREATION #########################################################
#######################################################################################################################################
globals().update(generate_paths(path_type='data_creation'))
generate_directory_structure()

#######################################################################################################################################
############################################################ SLEEP TIMER ##############################################################
#######################################################################################################################################
sleep_minutes(sleep_time)
```

```python
print(path_identifier_polynomial_data)

```

```python
##############DO NOT CHANGE###################
variables = 'abcdefghijklmnopqrstuvwxyz'[:n]

print('Variables: ' + str(n) + ' (' + variables + ')')
print('Degree: ' + str(d))
print('Sparsity: ' + str(sparsity))
print('Lambda-Net Dataset Size: ' + str(lambda_dataset_size))
print('I-Net Dataset Size: ' + str(polynomial_data_size))

print('Coefficient Range: ' + '[' + str(a_min) + ', ' + str(a_max) + ']')
print('Variable Range: ' + '[' + str(x_min) + ', ' + str(x_max) + ']')
```

# Function Generation

```python
if function_generation_type == 'polynomial':
    if shift_polynomial:
        all_sparsities = [nCr(n+i, i) for i in range(d+1)]
        start = perf_counter()
        parallel = Parallel(n_jobs=n_jobs, verbose=3, backend='multiprocessing')
        list_of_polynomials = parallel(delayed(get_polynomial_basic)(all_sparsities)for i in range(polynomial_data_size))
        list_of_polynomials = np.array(list_of_polynomials).astype('float64')
        end = perf_counter()
        print('Zeit zum generieren:', end-start, 'Sekunden')
        print('Zeit zum generieren pro Funktion:', (end-start)/polynomial_data_size)
    else:
        list_of_polynomials = np.random.uniform(low=a_min, high=a_max, size=(polynomial_data_size, sparsity))

        if sample_sparsity < sparsity:
            for polynomial in tqdm(list_of_polynomials):
                sparsity_indices = np.random.choice(nCr(n+d, d), nCr(n+d, d)-sample_sparsity, replace=False)
                for sparsity_index in sparsity_indices:
                    polynomial[sparsity_index] = 0

    list_of_monomial_identifiers_string = [''.join(str(e) for e in monomial_identifier) for monomial_identifier in list_of_monomial_identifiers] if n > 1 else [str(monomial_identifier[0]) for monomial_identifier in list_of_monomial_identifiers]
    polynomials_list_df = pd.DataFrame(data=list_of_polynomials, columns=list_of_monomial_identifiers_string).astype('float64')

    print(len(list_of_monomial_identifiers))
    print(polynomials_list_df.shape)
else:
    list_of_polynomials = np.zeros(shape=(polynomial_data_size, sparsity))

    list_of_monomial_identifiers_string = [''.join(str(e) for e in monomial_identifier) for monomial_identifier in list_of_monomial_identifiers] if n > 1 else [str(monomial_identifier[0]) for monomial_identifier in list_of_monomial_identifiers]
    polynomials_list_df = pd.DataFrame(data=list_of_polynomials, columns=list_of_monomial_identifiers_string).astype('float64')

```

```python
if same_training_all_lambda_nets:
    parallel = Parallel(n_jobs=n_jobs, verbose=3, backend='multiprocessing')
    result_list = parallel(delayed(gen_regression_symbolic)(polynomial_array=list_of_polynomials[i],
                                                               n_samples=lambda_dataset_size,
                                                               noise=noise,
                                                               noise_dist=noise_distrib,
                                                               seed=RANDOM_SEED,
                                                               sympy_calculation=False) for i in range(polynomial_data_size))
else:
    parallel = Parallel(n_jobs=n_jobs, verbose=3, backend='multiprocessing')
    result_list = parallel(delayed(gen_regression_symbolic)(polynomial_array=list_of_polynomials[i],
                                                               n_samples=lambda_dataset_size,
                                                               noise=noise,
                                                               noise_dist=noise_distrib,
                                                               seed=RANDOM_SEED+i,
                                                               sympy_calculation=False) for i in range(polynomial_data_size))

X_data_list = [[pd.Series(result[0],  index=list_of_monomial_identifiers_string), pd.DataFrame(result[1].astype(float), columns=list(variables[:n]))] for result in result_list]
y_data_list = [[pd.Series(result[0],  index=list_of_monomial_identifiers_string), pd.DataFrame(result[2].astype(float), columns=['result'])] for result in result_list]

```

```python
X_data_list[0][0].head(10)
```

```python
X_data_list[0][1].head(10)
```

```python
y_data_list[0][0].head(10)
```

```python
y_data_list[0][1].head(10)
```

```python
path_polynomials = './data/saved_polynomial_lists/polynomials_sample_' + path_identifier_polynomial_data + '.csv'
polynomials_list_df.to_csv(path_polynomials, index=False)

path_X_data = './data/saved_polynomial_lists/X_sample_' + path_identifier_polynomial_data + '.pkl'
with open(path_X_data, 'wb') as f:
    pickle.dump(X_data_list, f)#, protocol=2)

path_y_data = './data/saved_polynomial_lists/y_sample_' + path_identifier_polynomial_data + '.pkl'
with open(path_y_data, 'wb') as f:
    pickle.dump(y_data_list, f)#, protocol=2)

```

```python

```


$\newcommand{\mb}[1]{\mathbf{ #1 }}$
$\newcommand{\bb}[1]{\mathbb{ #1 }}$
$\newcommand{\bs}[1]{\boldsymbol{ #1 }}$

$\newcommand{\norm}[1]{\left\Vert #1 \right\Vert}$
$\newcommand{\der}[2]{\frac{ \mathrm{d} #1 }{ \mathrm{d} #2 }}$
$\newcommand{\derp}[2]{\frac{ \partial #1 }{ \partial #2 }}$

$\newcommand{\R}{\bb{R}}$

# Learning Dynamics

```python
from matplotlib.pyplot import show, subplots
```

## Robotic Systems

Let $\mathcal{Q} \subseteq \R^n$ be a configuration space. Consider a robotic system governed by:

\begin{equation}
    \mb{D}(\mb{q})\ddot{\mb{q}} + \mb{C}(\mb{q}, \dot{\mb{q}})\dot{\mb{q}} + \mb{G}(\mb{q}) = \mb{B}\mb{u},
\end{equation}

for generalized coordinates $\mb{q} \in \mathcal{Q}$, coordinate rates $\dot{\mb{q}} \in \R^n$, actions $\mb{u} \in \R^m$, inertia matrix function $\mb{D}: \mathcal{Q} \to \bb{S}^n_{++}$ (the space of $n \times n$ positive definite matrices), Coriolis terms $\mb{C}: \mathcal{Q} \times \R^n \to \R^{n \times n}$, potential terms $\mb{G}: \mathcal{Q} \to \R^n$, and static actuation matrix $\mb{B} \in \R^{n \times m}$. Assume $m \leq n$ and $\mb{B}$ is full rank.

### Inverted Pendulum

```python
from numpy import array, identity, linspace

from core.controllers import FBLinController, LQRController
```

```python
from core.systems import InvertedPendulum

ip = InvertedPendulum(m=0.25, l=0.5)

Q_ip = identity(2)
R_ip = identity(1)
lqr_ip = LQRController.build(ip, Q_ip, R_ip)
fb_lin_ip = FBLinController(ip, lqr_ip)

x_0_ip = array([1, 0])
ts_ip = linspace(0, 10, 1000 + 1)
xs_ip, _ = ip.simulate(x_0_ip, fb_lin_ip, ts_ip)

fig_ip, ax_ip = subplots(figsize=(6, 4))
ax_ip.plot(ts_ip, xs_ip[:, 0], linewidth=3, label='Oracle')
ax_ip.grid()
ax_ip.legend(fontsize=16)
ax_ip.set_title('Inverted Pendulum', fontsize=16)
ax_ip.set_xlabel('$t$ (sec)', fontsize=16)
ax_ip.set_ylabel('$\\theta$ (rad)', fontsize=16)
show()
```

### Double Inverted Pendulum

```python
from core.systems import DoubleInvertedPendulum

dip = DoubleInvertedPendulum(m_1=0.25, m_2=0.25, l_1=0.5, l_2=0.5)

Q_dip = identity(4)
R_dip = identity(2)
lqr_dip = LQRController.build(dip, Q_dip, R_dip)
fb_lin_dip = FBLinController(dip, lqr_dip)

x_0_dip = array([1, 0, 0, 0])
ts_dip = linspace(0, 10, 1000 + 1)
xs_dip, _ = dip.simulate(x_0_dip, fb_lin_dip, ts_dip)

fig_dip, (ax_dip_1, ax_dip_2) = subplots(2, figsize=(6, 8))
ax_dip_1.set_title('Double Inverted Pendulum', fontsize=16)

ax_dip_1.plot(ts_dip, xs_dip[:, 0], linewidth=3, label='Oracle')
ax_dip_1.grid()
ax_dip_1.legend(fontsize=16)
ax_dip_1.set_xlabel('$t$ (sec)', fontsize=16)
ax_dip_1.set_ylabel('$\\theta_1$ (rad)', fontsize=16)

ax_dip_2.plot(ts_dip, xs_dip[:, 1], linewidth=3, label='Oracle')
ax_dip_2.grid()
ax_dip_2.legend(fontsize=16)
ax_dip_2.set_xlabel('$t$ (sec)', fontsize=16)
ax_dip_2.set_ylabel('$\\theta_2$ (rad)', fontsize=16)

show()
```

## Uncertain Robotic Systems

Suppose $\mb{D}$, $\mb{C}$, $\mb{G}$, and $\mb{B}$ are unknown, and instead we have access to corresponding estimates $\hat{\mb{D}}$, $\hat{\mb{C}}$, $\hat{\mb{G}}$, and $\hat{\mb{B}}$ satisfying:

\begin{equation}
    \hat{\mb{D}}(\mb{q})\ddot{\mb{q}} + \hat{\mb{C}}(\mb{q}, \dot{\mb{q}})\dot{\mb{q}} + \hat{\mb{G}}(\mb{q}) = \hat{\mb{B}}\mb{u}.
\end{equation}

Assume that $\hat{\mb{B}}$ is also full rank.

The system dynamics can be in terms of the estimated terms as:

\begin{equation}
    \der{}{t} \begin{bmatrix} \mb{q} \\ \dot{\mb{q}} \end{bmatrix} = \begin{bmatrix} \dot{\mb{q}} \\ -\hat{\mb{D}}(\mb{q})^{-1}(\hat{\mb{C}}(\mb{q}, \dot{\mb{q}})\dot{\mb{q}} + \hat{\mb{G}}(\mb{q})) \end{bmatrix} + \begin{bmatrix} \mb{0}_{n \times m} \\ \hat{\mb{D}}(\mb{q})^{-1}\hat{\mb{B}} \end{bmatrix} \mb{u} + \begin{bmatrix} \mb{0}_n \\ \hat{\mb{D}}(\mb{q})^{-1}(\hat{\mb{C}}(\mb{q}, \dot{\mb{q}})\dot{\mb{q}} + \hat{\mb{G}}(\mb{q}))-\mb{D}(\mb{q})^{-1}(\mb{C}(\mb{q}, \dot{\mb{q}})\dot{\mb{q}} + \mb{G}(\mb{q})) \end{bmatrix} + \begin{bmatrix} \mb{0}_{n \times m} \\ \mb{D}(\mb{q})^{-1}\mb{B} - \hat{\mb{D}}(\mb{q})^{-1}\hat{\mb{B}} \end{bmatrix} \mb{u}
\end{equation}

### Inverted Pendulum

```python
ip_est = InvertedPendulum(m=0.24, l=0.48)

lqr_ip_est = LQRController.build(ip_est, Q_ip, R_ip)
fb_lin_ip_est = FBLinController(ip_est, lqr_ip_est)

xs_ip, us_ip = ip.simulate(x_0_ip, fb_lin_ip_est, ts_ip)

ax_ip.plot(ts_ip, xs_ip[:, 0], linewidth=3, label='Estimated')
ax_ip.legend(fontsize=16)
fig_ip
```

### Double Inverted Pendulum

```python
dip_est = DoubleInvertedPendulum(m_1=0.24, m_2=0.24, l_1=0.48, l_2=0.48)

lqr_dip_est = LQRController.build(dip_est, Q_dip, R_dip)
fb_lin_dip_est = FBLinController(dip_est, lqr_dip_est)

xs_dip, us_dip = dip.simulate(x_0_dip, fb_lin_dip_est, ts_dip)

ax_dip_1.plot(ts_dip, xs_dip[:, 0], linewidth=3, label='Esimated')
ax_dip_1.legend(fontsize=16)

ax_dip_2.plot(ts_dip, xs_dip[:, 1], linewidth=3, label='Esimated')
ax_dip_2.legend(fontsize=16)

fig_dip
```

## Learning Dynamics

```python
from tensorflow.logging import ERROR, set_verbosity
set_verbosity(ERROR)
```

### Inverted Pendulum

```python
from core.dynamics import LearnedFBLinDynamics
from core.learning.keras import KerasResidualAffineModel
```

    Using TensorFlow backend.

```python
d_drift_in_ip = 3
d_act_in_ip = 3
d_hidden_ip = 20
d_out_ip = 1

res_model_ip = KerasResidualAffineModel(d_drift_in_ip, d_act_in_ip, d_hidden_ip, 1, d_out_ip)
ip_learned = LearnedFBLinDynamics(ip_est, res_model_ip)
data = ip_learned.process_episode(xs_ip, us_ip, ts_ip)
ip_learned.fit(data, num_epochs=10, validation_split=0.1)
```

    Train on 898 samples, validate on 100 samples
    Epoch 1/10
    898/898 [==============================] - 1s 2ms/step - loss: 0.2794 - val_loss: 0.1563
    Epoch 2/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0626 - val_loss: 0.0301
    Epoch 3/10
    898/898 [==============================] - 1s 2ms/step - loss: 0.0385 - val_loss: 0.0207
    Epoch 4/10
    898/898 [==============================] - 1s 2ms/step - loss: 0.0358 - val_loss: 0.0482
    Epoch 5/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0352 - val_loss: 0.0293
    Epoch 6/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0339 - val_loss: 0.0289
    Epoch 7/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0343 - val_loss: 0.0443
    Epoch 8/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0331 - val_loss: 0.0380
    Epoch 9/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0325 - val_loss: 0.0279
    Epoch 10/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0310 - val_loss: 0.0286

```python
x_dots = array([ip.eval_dot(x, u, t) for x, u, t in zip(xs_ip, us_ip, ts_ip)])

_, (ax_ip_1, ax_ip_2) = subplots(2, figsize=(6, 8))
ax_ip_1.set_title('Inverted Pendulum', fontsize=16)

ax_ip_1.plot(ts_ip[:-1], x_dots[:, 0], linewidth=3, label='True')
ax_ip_1.grid()
ax_ip_1.set_xlabel('$t$ (sec)', fontsize=16)
ax_ip_1.set_ylabel('$\\dot{\\theta}$ (rad / sec)', fontsize=16)

ax_ip_2.plot(ts_ip[:-1], x_dots[:, 1], linewidth=3, label='True')
ax_ip_2.grid()
ax_ip_2.set_xlabel('$t$ (sec)', fontsize=16)
ax_ip_2.set_ylabel('$\\ddot{\\theta}$ (rad / sec$^2$)', fontsize=16)

x_dots = array([ip_learned.eval_dot(x, u, t) for x, u, t in zip(xs_ip, us_ip, ts_ip)])

ax_ip_1.plot(ts_ip[:-1], x_dots[:, 0], linewidth=3, label='Learned')
ax_ip_1.legend(fontsize=16)

ax_ip_2.plot(ts_ip[:-1], x_dots[:, 1], linewidth=3, label='Learned')
ax_ip_2.legend(fontsize=16)

show()
```

### Double Inverted Pendulum

```python
d_drift_in_dip = 5
d_act_in_dip = 5
d_hidden_dip = 40
d_out_dip = 2

res_model_dip = KerasResidualAffineModel(d_drift_in_dip, d_act_in_dip, d_hidden_dip, 2, d_out_dip)
dip_learned = LearnedFBLinDynamics(dip_est, res_model_dip)
data = dip_learned.process_episode(xs_dip, us_dip, ts_dip)
dip_learned.fit(data, num_epochs=10, validation_split=0.1)
```

    Train on 898 samples, validate on 100 samples
    Epoch 1/10
    898/898 [==============================] - 1s 2ms/step - loss: 0.8036 - val_loss: 0.7987
    Epoch 2/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.3102 - val_loss: 0.2532
    Epoch 3/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.1878 - val_loss: 0.1682
    Epoch 4/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.1601 - val_loss: 0.1361
    Epoch 5/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.1364 - val_loss: 0.1170
    Epoch 6/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.1188 - val_loss: 0.1140
    Epoch 7/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.1089 - val_loss: 0.1128
    Epoch 8/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.1025 - val_loss: 0.0734
    Epoch 9/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0946 - val_loss: 0.0686
    Epoch 10/10
    898/898 [==============================] - 1s 1ms/step - loss: 0.0913 - val_loss: 0.1110

```python
x_dots = array([dip.eval_dot(x, u, t) for x, u, t in zip(xs_dip, us_dip, ts_dip)])

_, axs_dip = subplots(4, figsize=(6, 16))
axs_dip[0].set_title('Double Inverted Pendulum', fontsize=16)
ylabels = ['$\\dot{\\theta}_1$ (rad / sec)', '$\\dot{\\theta}_2$ (rad / sec)', '$\\ddot{\\theta}_1$ (rad / sec$^2$)', '$\\ddot{\\theta}_2$ (rad / sec$^2$)']

for ax, data, ylabel in zip(axs_dip, x_dots.T, ylabels):
    ax.plot(ts_dip[:-1], data, linewidth=3, label='True')
    ax.grid()
    ax.set_xlabel('$t$ (sec)', fontsize=16)
    ax.set_ylabel(ylabel, fontsize=16)

x_dots = array([dip_learned.eval_dot(x, u, t) for x, u, t in zip(xs_dip, us_dip, ts_dip)])

for ax, data in zip(axs_dip, x_dots.T):
    ax.plot(ts_dip[:-1], data, linewidth=3, label='Learned')
    ax.legend(fontsize=16)

show()
```

## Overfitting

### Inverted Pendulum

```python
lqr_learned_ip = LQRController.build(ip_learned, Q_ip, R_ip)
fb_lin_learned_ip = FBLinController(ip_learned, lqr_learned_ip)

xs, _ = ip.simulate(x_0_ip, fb_lin_learned_ip, ts_ip)

_, ax = subplots(figsize=(6, 4))
ax.plot(ts_ip, xs[:, 0], linewidth=3)
ax.grid()
ax.set_title('Inverted Pendulum', fontsize=16)
ax.set_xlabel('$t$ (sec)', fontsize=16)
ax.set_ylabel('$\\theta$ (rad)', fontsize=16)

show()
```

### Double Inverted Pendulum

```python
lqr_learned_dip = LQRController.build(dip_learned, Q_dip, R_dip)
fb_lin_learned_dip = FBLinController(dip_learned, lqr_learned_dip)

xs, _ = dip.simulate(x_0_dip, fb_lin_learned_dip, ts_dip)

_, (ax_1, ax_2) = subplots(2, figsize=(6, 8))
ax_1.set_title('Double Inverted Pendulum', fontsize=16)

ax_1.plot(ts_dip, xs[:, 0], linewidth=3)
ax_1.grid()
ax_1.set_xlabel('$t$ (sec)', fontsize=16)
ax_1.set_ylabel('$\\theta_1$ (rad)', fontsize=16)

ax_2.plot(ts_dip, xs[:, 1], linewidth=3)
ax_2.grid()
ax_2.set_xlabel('$t$ (sec)', fontsize=16)
ax_2.set_ylabel('$\\theta_2$ (rad)', fontsize=16)

show()
```


# RHD Model Atmosphere Inner Boundary

Exploring the properties of RHD model atmosphere inner boundaries for AGB stars. Liljegren finds that some models fail to converge due to a temperature instability. Temperatures at the inner boundary increase rapidly before finally reaching a temperature that is outside of the pre-computed opacity grid.

The question becomes, is the temperature instability the result of missing physics near the inner boundary? We suspect that the lack of a description of convection for the inner layers of the atmosphere model are causing a false build-up of heat, leading to the runaway temperature instability. To explore whether this is the case, we'll need to first take a look at the thermodynamic state of the gas in the envelope.

## Thermodynamics

RHD models adopted by Liljegren are based on the RHD model atmosphere developed by Höfner et al. ([2003, A&A, 399, 589](http://adsabs.harvard.edu/abs/2003A%26A...399..589H)). For simplicity, and to allow for better comparison with previous studies, the RHD models adopt a perfect gas equation of state with $\gamma = 5/3$ and $\mu = 1.26$. The mean molecular weight is that for a non-interacting (i.e., perfect) gas without consideration of ionization states, which effectively equates the mean molecular weight with the mean atomic molecular weight.

Liljegren provided the following properties of the gas near the inner boundary prior to the temperature instability: $P_{\rm gas} = 1680\ {\rm Ba}$ with $T = 5600\ {\rm K}$. Using the perfect gas equation of state, one can derive a density for the gas
\begin{equation}
    \rho = \frac{\mu m_p}{k}\frac{P_{\rm gas}}{T_{\rm gas}}
\end{equation}
where $k$ is the Boltzmann constant and $m_p$ is the mass of a proton. We therefore find,

```python
rho = (1.26*1.6726219e-24/1.3806488e-16)*(1680./5600.)

print "Density of the gas [g/cm**3] = {:11.5e}.".format(rho)
```

    Density of the gas [g/cm**3] = 4.57938e-09.

We can also easily estimate the thermodynamic properties of the gas under the assumption of a perfect gas. We know that $\gamma = c_P / c_V = 5/3$. Of importance for our hypothesis is knowing the various thermodynamic temperature gradients, $\nabla_{\rm ad}$ and $\nabla_{\rm rad}$. The former can be expressed generally as
\begin{equation}
    \nabla_{\rm ad} = \frac{P_{\rm gas} \delta}{\rho T_{\rm gas} c_P},
\end{equation}
where $\delta = - (\partial\ln\rho / \partial\ln T_{\rm gas})_P$ (coefficient of thermal expansion) and $c_P$ is the specific heat at constant pressure. The latter can be evaluated using the general thermodynamic relation,
\begin{equation}
    c_P - c_V = \frac{P_{\rm gas} \delta^2}{\rho T_{\rm gas} \alpha},
\end{equation}
with $\alpha = (\partial\ln\rho / \partial\ln P_{\rm gas})_T$ (compressibility coefficient). Under the assumption that the gas particles are non-interacting and neglecting ionization, we have $\alpha = \delta = 1$. Therefore we can write
\begin{equation}
    c_P = \frac{5}{2} \frac{P_{\rm gas}}{\rho T_{\rm gas}},
\end{equation}
meaning
\begin{equation}
    \nabla_{\rm ad} = \frac{2}{5} = 0.4.
\end{equation}

## Convective Stability

Now, to understand whether convection may or may not be important under these conditions, we can consider a simple comparison of the adiabatic and radiative temperature gradients. Adiabatic temperature gradient is a limiting case for the temperature gradient if the gas is undergoing convection, but should provide an estimate of the super-adiabaticity of the temperature gradient.

The radiative temperature gradient is
\begin{equation}
    \nabla_{\rm rad} = \left(\frac{d\ln T_{\rm gas}}{d\ln P_{\rm gas}}\right)_{\rm rad} = \frac{3}{16\pi acG}\frac{\kappa P_{\rm gas}}{T_{\rm gas}^4}\frac{L_r}{M_r},
\end{equation}
where $X_r$ are quantities defined at a radius $r$. For our purposes, we can assume $L_r = L_{\star}$ and $M_r = M_{\star}$. This is justified as there are no energy generation sources in the outer envelope of an AGB star and the outer envelope contains a negliglbe fraction of the total mass. We can esitmate the Rosseland mean opacity from low temperature opacity sources (e.g., Ferguson et al. 2005), which yield an opacity with the range of $10^{-2}$ - $10^{-1}\ {\rm cm}^2\, {\rm g}^{-1}$. Thus,

```python
def del_rad(opac, P, T, L, M):
    return 7.62586e9*opac*P*L/(T**4*M)
```

```python
P = 1680.0  # Ba
T = 5600.0  # K
M = 1.5     # Msun
L = 6953.0  # Rstar = 412 Rsun; Teff = 2600 K

print "{:11.5e} < Del_rad < {:11.5e}".format(del_rad(1.0e-2, P, T, L, M), del_rad(1.0e-1, P, T, L, M))
```

    6.03847e-01 < Del_rad < 6.03847e+00

In the case where the radiative opacity is close to $10^{-2}\ {\rm cm}^2\, {\rm g}^{-1}$, the radiative gradient may exceed the adiabatic temperature gradient, suggesting that layers may be unstable to convective instabilities. It is therefore quite possible that thermodynamic conditions developing at the inner boundary of the RHD atmosphere models are ammenable to convective energy transport.

Since the inner boundary ignores convection and ionization, a temperature purturbation may lead to a growing instability in the temperature as radiation is becomes less able to transport the flux (at a given temperature) and ionization stages of hydrogen do not exist to alter the specific heat when the temperature begins to rapidly increase toward $8\,000 - 10\,000\ {\rm K}$.

A temperature instability is all inevitable once conditions become ammenable to convection (at least within the framework of adiabaic convection). Radiation will be unable to efficiently transport flux and the neglect of physics related to convective flux transport will force the local temperature gradient (and thus the deep interior temperature) to force the flux through the gas using only radiation.

We may be able to estimate what temperature is required, assuming the pressure is defined by the condition for hydrostatic equilibrium. The approximate conditions required for radiation to carry all of the flux through an ideal gas (neglecting ionization) is
\begin{equation}
    \nabla_{\rm rad} = \nabla_{\rm ad} = \frac{2}{5}.
\end{equation}
Therefore, the local temperature can be approximated at the inner boundary
\begin{equation}
    T_{\rm gas} = \left(\frac{15}{32\pi acG} \kappa P_{\rm gas} \frac{L_r}{M_r}\right )^{1/4}.
\end{equation}
For conditions at the inner boundary, the temperature must exceed

```python
def T_rad(opac, P, L, M):
    return (1.906465e10*opac*P*L/M)**0.25
```

```python
print "{:11.5e} < T < {:11.5e} K".format(T_rad(1.0e-2, 1680., L, M), T_rad(1.0e-1, 1680., L, M))
```

    6.20733e+03 < T < 1.10384e+04 K

which produces a density

```python
rho1 = (1.26*1.6726219e-24/1.3806488e-16)*(1680./6207.)
rho2 = (1.26*1.6726219e-24/1.3806488e-16)*(1680./11038.)

print "{:11.5e} < Density < {:11.5e} [g cm**-3].".format(rho1, rho2)
```

    4.13155e-09 < Density < 2.32329e-09 [g cm**-3].

Thus, the density could be forced to decrease by a factor of two compared to predictions from a non-ideal gas if the local conditions become favorable to convective instability, but the models neglect physics related to convective flux transport.

We have seen that increasing the local temperature will decrease the local density, if the layer is to maintain a constant pressure. Opacity in cool, low density environments is an increasing function of temperature and a decreasing function of density. Therefore, an increase in the local temperature and decrease in the local density will cause a dramatic increase in the opacity.


The purpose of this file is just to make sure that the environment is configured and functioning properly. Run some of these cells and make sure you can get outputs to appear.

```python
from sympy import *
x, y, z = symbols('x y z')
```

```python
diff(cos(x), x)
```

$\displaystyle - \sin{\left(x \right)}$

```python
diff(exp(x**2), x)
```

$\displaystyle 2 x e^{x^{2}}$

```python
p1 = plot(x**2, xlim=(0,2), ylim=(0,5))
```

```python
p2 = plot(exp(x**2), x)
```

suppose:
$L = \lim_{x\to 7} f(x)$