# Neuromatch Academy: Week 3, Day 1, Tutorial 1
# Real Neurons: The Leaky Integrate-and-Fire (LIF) Neuron Model

## Tutorial Objectives

This is Tutorial 1 of a series on implementing realistic neuron models. In this tutorial, we will build up a leaky integrate-and-fire (LIF) neuron model and study its dynamics in response to various types of inputs. In particular, we are going to write a few lines of code to:

- simulate the LIF neuron model

- drive the LIF neuron with external inputs, such as DC current, Gaussian white noise, and Poisson spike trains, etc.

- study how different inputs affect the  LIF neuron's output (firing rate and spike time irregularity)

Here we will especially emphasize on identifying conditions (input statistics) under which a neuron can spike at low firing rates and in a irregular manner. The reason for focusing on this is that in most cases neocortical neurons spike in an irregular manner.

# Leaky Integrate-and-Fire (LIF)

```python
#@title Video: LIF neuron
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='3cZjjU_HKJs', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=3cZjjU_HKJs

# Setup

```python
# Imports
import matplotlib.pyplot as plt    # import matplotlib
import numpy as np                 # import numpy
import time                        # import time
import ipywidgets as widgets       # interactive display
from scipy.stats import pearsonr   # import pearson correlation
```

```python
#@title Figure Settings
%matplotlib inline
fig_w, fig_h = 8, 6
my_fontsize = 18
my_params = {'axes.labelsize': my_fontsize,
          'axes.titlesize': my_fontsize,
          'figure.figsize': (fig_w, fig_h),
          'font.size': my_fontsize,
          'legend.fontsize': my_fontsize-4,
          'lines.markersize': 8.,
          'lines.linewidth': 2.,
          'xtick.labelsize': my_fontsize-2,
          'ytick.labelsize': my_fontsize-2}

plt.rcParams.update(my_params)
my_layout = widgets.Layout()
```

```python
#@title Helper functions
def plot_volt_trace(pars, v, sp):
  """
  Plot trajetory of membrane potential for a single neuron

  Expects:
  pars   : parameter dictionary
  v      : volt trajetory
  sp     : spike train

  Returns:
  figure of the membrane potential trajetory for a single neuron
  """

  V_th = pars['V_th']
  dt, range_t = pars['dt'], pars['range_t']
  if sp.size:
      sp_num = (sp/dt).astype(int)-1
      v[sp_num] += 10 # draw nicer spikes

  plt.plot(pars['range_t'], v, 'b')
  plt.axhline(V_th, 0, 1, color='k', ls='--')
  plt.xlabel('Time (ms)')
  plt.ylabel('V (mV)')
  plt.legend(['Membrane\npotential', r'Threshold V$_{\mathrm{th}}$'], loc=[1.05, 0.75])

def plot_GWN(pars, I_GWN):
  """
  Args:
    pars  : parameter dictionary
    I_GWN : Gaussian white noise input

  Returns:
    figure of the gaussian white noise input
  """

  plt.figure(figsize=(12, 4))
  plt.subplot(1,2,1)
  plt.plot(pars['range_t'][::3], I_GWN[::3], 'b')
  plt.xlabel('Time (ms)')
  plt.ylabel(r'$I_{GWN}$ (pA)')
  plt.subplot(1,2,2)
  plot_volt_trace(pars, v, sp)
  plt.tight_layout()

def my_hists(isi1, isi2, cv1, cv2):
  """
  Args:
    isi1 : vector with inter-spike intervals
    isi2 : vector with inter-spike intervals
    cv1  : coefficient of variation for isi1
    cv2  : coefficient of variation for isi2

  Returns:
    figure with two histograms, isi1, isi2

  """
  plt.figure(figsize=(11, 4))
  my_bins = np.linspace(10, 30, 20)
  plt.subplot(1, 2, 1)
  plt.hist(isi1, bins=my_bins, color='b', alpha=0.5);
  plt.xlabel('ISI (ms)')
  plt.ylabel('number')
  plt.title(r'$\sigma=$%.1f, CV$_{\mathrm{isi}}$=%.3f' % (sig_gwn1, cv1))

  plt.subplot(1, 2, 2)
  plt.hist(isi2, bins=my_bins, color='b', alpha=0.5);
  plt.xlabel('ISI (ms)')
  plt.ylabel('number')
  plt.title(r'$\sigma=$%.1f, CV$_{\mathrm{isi}}$=%.3f' % (sig_gwn2, cv2))
  plt.tight_layout()
  plt.show()
```

# Implementation of an LIF neuron model

Now, it's your turn to implement one of the simplest mathematical model of a neuron: the leaky integrate-and-fire (LIF) model. The basic idea of LIF neuron was proposed in 1907 by Louis Édouard Lapicque, long before we understood the electrophysiology of a neuron (see a translation of [Lapicque's paper](https://pubmed.ncbi.nlm.nih.gov/17968583/) ). More details of the model can be found in the book [**Theoretical neuroscience**](http://www.gatsby.ucl.ac.uk/~dayan/book/) by Peter Dayan and Laurence F. Abbott.

The subthreshold membrane potential dynamics of a LIF neuron is described by

\begin{eqnarray}
C_m\frac{dV}{dt} = -g_L(V-E_L) + I,\quad (1)
\end{eqnarray}

where $C_m$ is the membrane capacitance, $V$ is the membrane potential, $g_L$ is the leak conductance ($g_L = 1/R$, the inverse of the leak resistence $R$ mentioned in previous tutorials), $E_L$ is the resting potential, and $I$ is the external input current.

Dividing both sides of the above equation by $g_L$ gives

\begin{align}
\tau_m\frac{dV}{dt} = -(V-E_L) + \frac{I}{g_L}\,,\quad (2)
\end{align}

where the $\tau_m$ is membrane time constant and is defined as $\tau_m=C_m/g_L$.

You might wonder how come dividing capacitance by conductance gives units of time! Find out yourself why it is so.

Below we will use Eqn.(1) to simulate the LIF neuron dynamics.

If $I$ is sufficiently strong such that $V$ reaches a certain threshold value $V_{\rm th}$, $V$ is reset to a reset potential $V_{\rm reset}< V_{\rm th}$ and voltage is clamped to $V_{\rm reset}$ for $\tau_{\rm ref}$ ms mimicking the refractoriness of the neuron during an action potential, _i.e._,

\begin{eqnarray}
\mathrm{if}\quad V(t_{\text{sp}})\geq V_{\rm th}&:& V(t)=V_{\rm reset} \text{  for } t\in(t_{\text{sp}}, t_{\text{sp}} + \tau_{\text{ref}}]
\end{eqnarray}
where $t_{\rm sp}$ is the spike time when $V(t)$ just exceeded $V_{\rm th}$.

---
Note: in the lecture slides, $\theta$ corresponds to the threshold voltage $V_{th}$, and $\Delta$ corresponds to the refractory time $\tau_{\rm ref}$.

---

Thus, the LIF model captures the facts that a neuron
- performs spatial and temporal integration of synaptic inputs
- generates a spike when the voltage reaches a certain threshold
- goes refractory during the action potential
- membrane is leaky

The LIF model assumes that spatial and temporal integration of inputs is linear. Also, membrane potential dynamics close to the spike threshold is much slower in LIF neurons than in real neurons.

## Python code to simulate the LIF neuron

We now **write** Python code to calculate Eqn.(2) and simulate the LIF neuron dynamics.

The cell below initializes a dictionary that stores parameters of the LIF neuron model and the simulation scheme. You can use `pars=default_pars(T=simulation_time, dt=time_step)` to get the parameters (you can try to print the dictionary `pars`). Note that, `simulation_time` and `time_step` have the unit `ms`. In addition, you can add the value to a new parameter by `pars['New_param'] = value`.

```python
#@title Default value function: `default_pars( **kwargs)`
def default_pars(**kwargs):
    pars = {}

    ### typical neuron parameters###
    pars['V_th']    = -55. # spike threshold [mV]
    pars['V_reset'] = -75. #reset potential [mV]
    pars['tau_m']   = 10. # membrane time constant [ms]
    pars['g_L']     = 10. #leak conductance [nS]
    pars['V_init']  = -65. # initial potential [mV]
    pars['E_L']     = -75. #leak reversal potential [mV]
    pars['tref']    = 2. # refractory time (ms)

    ### simulation parameters ###
    pars['T'] = 400. # Total duration of simulation [ms]
    pars['dt'] = .1  # Simulation time step [ms]

    ### external parameters if any ###
    for k in kwargs:
        pars[k] = kwargs[k]

    pars['range_t'] = np.arange(0, pars['T'], pars['dt']) # Vector of discretized time points [ms]

    return pars
```

The cell below defines the function to simulate the LIF neuron when receiving external current inputs. You can use `v, sp = run_LIF(pars, I)` to get the membrane potential and spike train with give `pars` and input current `I`.

```python
#@title Function `run_LIF(pars, I)`

def run_LIF(pars, I):
    '''
    Simulate the LIF dynamics with external input current

    Expects:
    pars       : parameter dictionary
    I          : input current [pA]. The injected current here can be a value or an array

    Returns:
    rec_v      : mebrane potential
    rec_sp     : spike times
    '''

    # Set parameters
    V_th, V_reset = pars['V_th'], pars['V_reset']
    tau_m, g_L = pars['tau_m'], pars['g_L']
    V_init, E_L = pars['V_init'], pars['E_L']
    dt, range_t = pars['dt'], pars['range_t']
    Lt = range_t.size
    tref = pars['tref']
    # Initialize voltage and current
    v = np.zeros(Lt)
    v[0] = V_init
    I = I * np.ones(Lt)
    tr = 0. # the count for refractory duration
    # simulate the LIF dynamics
    rec_spikes = []               # record spike times
    for it in range(Lt-1):
      if tr >0: # check for refractoriness
            v[it] = V_reset
            tr = tr-1
      elif v[it] >= V_th:         #reset voltage and record spike event
            rec_spikes.append(it)
            v[it] = V_reset
            tr = tref/dt
      #calculate the increment of the membrane potential
      dv = (-(v[it]-E_L) + I[it]/g_L) * (dt/tau_m)

      #update the membrane potential
      v[it+1] = v[it] + dv

    rec_spikes = np.array(rec_spikes) * dt

    return v, rec_spikes

print(help(run_LIF))
```

    Help on function run_LIF in module __main__:

    run_LIF(pars, I)
        Simulate the LIF dynamics with external input current

        Expects:
        pars       : parameter dictionary
        I          : input current [pA]. The injected current here can be a value or an array

        Returns:
        rec_v      : mebrane potential
        rec_sp     : spike times

    None

## Response of an LIF model to different types of input currents
In the following we will learn how to inject direct current and white noise to study the response of an LIF neuron.

```python
#@title Video: Response of the LIF neuron to different inputs
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='VNAQbj6uNWo', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=VNAQbj6uNWo

### Exercise 1: LIF neuron driven by Constant current
Investigate the voltage response of the LIF neuron when receiving a DC current of 300 pA by `run_LIF` function.

```python
# Exercise 1
pars = default_pars(T=100.) # get the parameters

########################################################################
## TODO for students: compute the membrane potential v, spike train sp #
########################################################################

# Uncomment these lines after obtaining v and sp
# plot_volt_trace(pars, v, sp)
# plt.show()
```

```python
# to_remove solution
pars = default_pars(T=100.)
v, sp = run_LIF(pars, I = 300.)
with plt.xkcd():
  plot_volt_trace(pars, v, sp)
  plt.show()
```

In the plot above you see the membrane potential of an LIF neuron. You may notice that the neuron generates a spike. But this is just a cosmetic spike only for illustration purpose. In LIF neuron we only need to keep track of times when the neuron hit the threshold so the postsynaptic neurons can be informed of the spike.

### Parameter Exploration
Here's an interactive demo that shows how the LIF neuron behavior changes for  DC input with different amplitudes.

**Remember to enable the demo by running the cell.**

```python
#@title LIF neuron explorer for DC input
my_layout.width = '450px'
@widgets.interact(
    I_dc  = widgets.FloatSlider(200., min=100., max=300., step=5., layout=my_layout),
    tau_m = widgets.FloatSlider(10., min=2., max=20., step=2., layout=my_layout)
)

def diff_DC(I_dc=200., tau_m=10.):
  pars = default_pars(T=100.)
  pars['tau_m'] = tau_m
  v, sp = run_LIF(pars, I = I_dc)
  plot_volt_trace(pars, v, sp)
```

## Gaussian white noise (GWN) current

Given the noisy nature of neuronal activity _in vivo_, neurons usually receive complex, time-varying inputs.

To mimic this, we will now investigate the neuronal response when the LIF neuron receives Gaussian white noise $\xi(t)$ with mean

\begin{eqnarray}
E[\xi(t)]=\mu=0,
\end{eqnarray}

and autocovariance
\begin{eqnarray}
E[\xi(t)\xi(t+\tau)]=\sigma_\xi^2 \delta(\tau)
\end{eqnarray}

Note that the GWN has the zero mean, that is, it describes the fluctuations of the input received by a neuron. Thus, at the same time, we can use the DC current as the average input into the cell, which allows us to modify the definition of GWN by having a nonzero mean value $\mu$. The cell below defines the modified gaussian white noise currents with nonzero mean $\mu$.

```python
#@title Function to generate GWN: `my_GWN(pars, mu, sig, myseed=False)`
def my_GWN(pars, mu, sig, myseed=False):
  """
  Function that generates Gaussian white noise input

  Args:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    myseed     : random seed. int or boolean
                 the same seed will give the same
                 random number sequence

  Returns:
    I          : Gaussian white noise input
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  # you can fix the seed of the random number generator so that the results are reliable
  # however, when you want to generate multiple realization make sure that you change
  # the seed for each new realization
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate GWN
  # we divide here by 1000 to convert units to sec.
  I =  mu + sig * np.random.randn(Lt) / np.sqrt(dt/1000.)

  return I
```

#### Exercise 2: LIF neuron driven by GWN

You can gnerate a noisy input with `my_GWN(pars, mu, sig, myseed=False)`. Here, $\mu=250$ and $\sigma=5$. Note that fixing the value of the random seed (e.g., `myseed=2020`) will allow you to obtain the same result every time you run this.

```python
# Exercise
pars = default_pars(T=100.)
sig_gwn = 5.
mu_gwn  = 250.
#######################################################
## TODO for students: compute the noisy input I_GWN   #
## and then compute v, sp with inject I_GWN into LIF  #
#######################################################
#I_GWN = ...
#v, sp = ...

# Uncomment these lines after obtianing v, sp
# plot_GWN(pars, I_GWN)
# plt.show()

```

```python
# to_remove solution
pars = default_pars(T=100.)
sig_gwn = 5.
mu_gwn  = 250.
I_GWN = my_GWN(pars, mu = mu_gwn, sig=sig_gwn, myseed=2020)
v, sp = run_LIF(pars, I=I_GWN)

with plt.xkcd():
  plot_GWN(pars, I_GWN)
  plt.show()
```

#### Parameter Exploration
Here's an interactive demo that shows how the LIF neuron behavior changes for  noisy input with different amplitude and the mean $\mu$ and fluctuation $\sigma$.

**Remember to enable the demo by running the cell.**

```python
#@title LIF neuron Explorer for noisy input

my_layout.width = '450px'
@widgets.interact(
    mu_gwn  = widgets.FloatSlider(200., min=100., max=300., step=5., layout=my_layout),
    sig_gwn = widgets.FloatSlider(2.5, min=0., max=5., step=.5, layout=my_layout)
)

def diff_GWN_to_LIF(mu_gwn,sig_gwn):
    pars = default_pars(T=100.)
    I_GWN = my_GWN(pars, mu = mu_gwn, sig=sig_gwn)
    v, sp = run_LIF(pars, I =I_GWN)
    with plt.xkcd():
        plt.figure(figsize=(12, 4))
        plt.subplot(1,2,1)
        plt.plot(pars['range_t'][::3], I_GWN[::3], 'b')
        plt.xlabel('Time (ms)')
        plt.ylabel(r'$I_{GWN}$ (pA)')
        plt.subplot(1,2,2)
        plot_volt_trace(pars, v, sp)
        plt.tight_layout()
        plt.show()

#_ = widgets.interact(diff_GWN_to_LIF, mu_gwn = (-100., 400., 5.),sig_gwn = (0, 5., 0.5))
```

#### Think
- As we increase the input average ($\mu$) or the input fluctuation ($\sigma$), the spike count changes. How much can we increase the spike count and what might be the relationship between GWN mean/std or DC value and spike count?

- We have seen above that when we inject DC the neuron spikes in a regular manner (clock like) and this regularity is reduced when GWN is injected. The question is how much irregular can we make the neurons spiking by changing the parameters of the GWN?

## Firing rate and spike time irregularity
When we plot the output firing rate as a function of GWN mean or DC value, it is called the input-output transfer function of the neuron (so simply F-I curve).

Spike regularity can be quanitfied as **coefficient of variance (CV) of the inter-spike-interval (ISI)**:
\begin{align}
\text{CV}_{\text{ISI}} = \frac{std(\text{ISI})}{mean(\text{ISI})}
\end{align}

A Poisson train is an example of high irregularity, in which $\textbf{CV}_{\textbf{ISI}} \textbf{= 1}$. And for a clock like process (regular) we have $\textbf{CV}_{\textbf{ISI}} \textbf{= 0}$ because of **std(ISI)=0**.

#### Parameter Exploration
Here's an interactive demo that shows how the F-I curve of a LIF neuron changes for different level of fluctuation $\sigma$.

```python
#@title F-I Explorer for different `sig_gwn`

my_layout.width = '450px'
@widgets.interact(
    sig_gwn = widgets.FloatSlider(3.0, min=0., max=6., step=0.5, layout=my_layout)
)

def diff_std_affect_fI(sig_gwn):
  pars = default_pars(T=1000.)
  I_mean = np.arange(100.,400., 10.)
  spk_count = np.zeros(len(I_mean))
  spk_count_dc = np.zeros(len(I_mean))

  for idx in range(len(I_mean)):
      I_GWN = my_GWN(pars, mu=I_mean[idx], sig=sig_gwn, myseed=2020)
      v, rec_spikes = run_LIF(pars, I =I_GWN)
      v_dc, rec_sp_dc = run_LIF(pars, I =I_mean[idx])
      spk_count[idx] = len(rec_spikes)
      spk_count_dc[idx] = len(rec_sp_dc)

  # Plot the F-I curve i.e. Output firing rate as a function of input mean.
  plt.figure()
  plt.plot(I_mean, spk_count, 'k', label=r'$\sigma_{\mathrm{GWN}}=%.2f$' % sig_gwn)
  plt.plot(I_mean, spk_count_dc, 'k--', alpha=0.5, lw=4, dashes=(2,2), label='DC input')
  plt.ylabel('Spike count')
  plt.xlabel('Average injected current (pA)')
  plt.legend(loc='best')
  plt.show()
```

#### Exercise 3: Compute $CV_{ISI}$ values

As shown above, the F-I curve becomes smoother while increasing the amplitude of the fluctuation ($\sigma$). In addition, the fluctuation can also change the irregularity of the spikes. Let's investigate the effect of $\mu=250$, $\sigma=0.5$ and $\sigma=3$.

Fill in the code below to compute ISI, then plot the histogram of the ISI and compute the CV(ISI). Note that, you can use `np.diff(spike_time)` to calculate ISI.

```python
# Exercise

pars = default_pars(T=1000.) # run a little bit long time for sufficient data
mu_gwn  = 250.

sig_gwn1 = 0.5
I_GWN = my_GWN(pars, mu = mu_gwn, sig=sig_gwn1, myseed=2020)
v, sp1 = run_LIF(pars, I=I_GWN)

sig_gwn2 = 3.0
I_GWN = my_GWN(pars, mu = mu_gwn, sig=sig_gwn2, myseed=2020)
v, sp2 = run_LIF(pars, I=I_GWN)

##################################################################
## TODO for students: compute isi for sp1 and sp2,               #
##                    then compute their corresponding cv value  #
##################################################################
#isi = ...
#cv  = ...

# Uncomment these lines below after you have isi/cv
# my_hists(isi1, isi2, cv1, cv2)

```

```python
# to_remove solutions
pars = default_pars(T=1000.)
mu_gwn  = 250.

sig_gwn1 = 0.5
I_GWN = my_GWN(pars, mu = mu_gwn, sig=sig_gwn1, myseed=2020)
v, sp1 = run_LIF(pars, I=I_GWN)
isi1 = np.diff(sp1)
cv1 = isi1.std()/isi1.mean()

sig_gwn2 = 3.0
I_GWN = my_GWN(pars, mu = mu_gwn, sig=sig_gwn2, myseed=2020)
v, sp2 = run_LIF(pars, I=I_GWN)
isi2 = np.diff(sp2)
cv2 = isi2.std()/isi2.mean()

with plt.xkcd():
  my_hists(isi1, isi2, cv1, cv2)
```

#### Parameter Exploration
Here's an interactive demo that shows how the function of CVs on the mean current $\mu$ changes for different level of fluctuation $\sigma$.

```python
#@title Spike irregularity Explorer for different `sig_gwn`

my_layout.width = '450px'
@widgets.interact(
    sig_gwn = widgets.FloatSlider(0.0, min=0., max=5., step=0.5, layout=my_layout)
)

def diff_std_affect_fI(sig_gwn):
  pars = default_pars(T=1000.)
  I_mean = np.arange(100.,400.,20)
  spk_count = np.zeros(len(I_mean))
  cv_isi = np.empty(len(I_mean))

  for idx in range(len(I_mean)):
    I_GWN = my_GWN(pars, mu=I_mean[idx], sig=sig_gwn)
    v, rec_spikes = run_LIF(pars, I =I_GWN)
    spk_count[idx] = len(rec_spikes)
    if len(rec_spikes)>3:
      isi = np.diff(rec_spikes)
      cv_isi[idx] = np.std(isi)/np.mean(isi)

  # Plot the F-I curve i.e. Output firing rate as a function of input mean.
  plt.figure()
  plt.plot(I_mean[spk_count>5],cv_isi[spk_count>5], 'bo', alpha=0.5)
  plt.xlabel('Average injected current (pA)')
  plt.ylabel(r'Spike irregularity ($\mathrm{CV}_\mathrm{ISI}$)')
  plt.ylim(-0.1, 1.5)
  plt.grid(True)
  plt.show()
```

#### Try to answer the following
- Does standard deviation of the injected current affects the F-I curve in any qualitative manner?
- Why does increasing the mean of GWN reduce the $CV_{ISI}$?
- If you plot spike count (or rate) vs. $CV_{ISI}$, should there be a relationship between the two? Try out yourself.

## Generation of Poisson type spike trains

*In the next tutorials we will often use Poisson type spike train to explore properties of neurons and synapses. Therefore, it is good to know how to generate Poisson type spike trains.*

Mathematically a spike train is a Point Process. One of the simplest model of a sequence of presynaptic pulse inputs is the Poisson process. We know that given temporal integration and refractoriness, neurons cannot behave as a Poisson Process, and Gamma Process (*find out what might be the difference in the two processes*) gives a better approximation.
Here, however, we will assume that the incoming spike are following Poisson statistics. Question arises how to simulate a Poisson Process. The generation of the Poisson process can be realized by at least the following two ways:

- By definition, for a Poisson process with rate $\lambda$, the probability of finding one event in the time window with a sufficiently small length $\Delta t$ is $P(N = 1) = \lambda \Delta t$. Therefore, in each time window, we generate a uniformly distributed random variable $r \in [0,1]$ and generate a Poisson event when $r <\lambda \Delta t$. This method allows us to generate Poisson distributed spikes in an online manner.

- The interval $t_{k+1}-t_{k}$ between two Poisson events with rate $\lambda$ follows the exponential distribution, i.e., $P(t_{k+1}-t_{k}<t) = 1 - e^{\lambda t}$. Therefore, we only need to generate a set of exponentially distributed variables $\{s_k\}$ to obtain the timing of Poisson events $t_{k+1}=t_{k}+s_{k}$. In this method we need to generate all future spikes at once.

Below we use the first method and check the statistical property of the time interval between two Poisson events.

```python
#@title `Poisson_generator(pars, rate, n, myseed=False)`
def Poisson_generator(pars, rate, n, myseed=False):
  """
  Generates poisson trains

  Args:
    pars            : parameter dictionary
    rate            : noise amplitute [Hz]
    n               : number of Poisson trains
    myseed          : random seed. int or boolean

  Returns:
    pre_spike_train : spike train matrix, ith row represents whether
                      there is a spike in ith spike train over time
                      (1 if spike, 0 otherwise)
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate uniformly distributed random variables
  u_rand = np.random.rand(n, Lt)

  # generate Poisson train
  poisson_train = 1. * (u_rand<rate*dt/1000.)

  return poisson_train

# the function plot the raster of the Poisson spike train
def my_raster_Poisson(range_t, spike_train, n):
  """
  Generates poisson trains

  Args:
    range_t     : time sequence
    spike_train : binary spike trains, with shape (N, Lt)
    n           : number of Poisson trains plot

  Returns:
    Raster plot of the spike train
  """

  #find the number of all the spike trains
  N = spike_train.shape[0]

  # n should smaller than N:
  if n > N:
    print('The number n exceeds the size of spike trains')
    print('The number n is set to be the size of spike trains')
    n = N

  #plot rater
  i = 0
  while i< n:
    if spike_train[i, :].sum()>0.:
      t_sp = range_t[spike_train[i, :]>0.5] #spike times
      plt.plot(t_sp, i*np.ones(len(t_sp)), 'k|', ms=10, markeredgewidth=2)
    i += 1
  plt.xlim([range_t[0], range_t[-1]])
  plt.ylim([-0.5, n+0.5])
  plt.xlabel('Time (ms)', fontsize=12)
  plt.ylabel('Neuron ID', fontsize=12)
```

```python
# we can use Poisson_generator to mimic presynaptic spike trains
pars = default_pars()
pre_spike_train = Poisson_generator(pars, rate=10, n=100)
my_raster_Poisson(pars['range_t'], pre_spike_train, 100)

print(help(my_raster_Poisson)) # use `help` to learn about the function
```

How do we make sure that the above spike trains are following Poisson statistics?
A Poisson process must have the following properties:
- The ratio of the mean and variance of spike count is 1.
- Inter-spike-invervals are exponentially distributed
- Spike times are irregular i.e. CV$_{\rm ISI} = 1$
- Adjacent spike interval are independent of each other.

# Summary

Congratulations! You've just built a leaky integrate-and-fire (LIF) neuron model from scratch, and studied its dynamics in response to various types of inputs, having:

- simulated the LIF neuron model

- driven the LIF neuron with external inputs, such as DC current, Gaussian white noise, and Poisson spike trains, etc.

- studied how different inputs affect the  LIF neuron's output (firing rate and spike time irregularity),

with a special focus on low rate and irregular firing regime to mimc real cortical neurons. The next tutorial will look at how spiking statistics may be influenced by a neuron's input statistics.

However, if you have extra time, follow the section below to explore a different type of noise input.

# Bonus 1 - Orenstein-Uhlenbeck Process

When a neuron received spiking input the synaptic current is Shot Noise -- which is a kind of colored noise and the spectrum of the noise determined by the synaptic kernel time constant. That is, a neuron is driven by **colored noise** and not GWN.

We can model colored noise using the Ohrenstein-Uhlenbeck process - filtered white noise.

#### Ornstein-Uhlenbeck (OU) current

We next study if the input current is temporally correlated and is modeled as an Ornstein-Uhlenbeck process $\eta(t)$, i.e., low-pass filtered GWN with a time constant $\tau_{\eta}$:

$$\tau_\eta \frac{d}{dt}\eta(t) = \mu-\eta(t) + \sigma_\eta\sqrt{2\tau_\eta}\xi(t).$$

**Hint:** An OU process as defined above has

$$E[\eta(t)]=\mu$$

and autocovariance

$$[\eta(t)\eta(t+\tau)]=\sigma_\eta^2e^{-|t-\tau|/\tau_\eta},$$

which can be used to check your code.

```python
#@title 'my_OU(pars, mu, sig, myseed=False)'
def my_OU(pars, mu, sig, myseed=False):
  """
  Function that produces Ornstein-Uhlenbeck input

  Args:
    pars       : parameter dictionary
    sig        : noise amplitute
    myseed     : random seed. int or boolean

  Returns:
    I          : Ornstein-Uhlenbeck input current
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size
  tau_ou = pars['tau_ou'] # [ms]

  # set random seed
  if myseed:
    np.random.seed(seed=myseed)
  else:
    np.random.seed()

  # Initialize
  noise =  np.random.randn(Lt)
  I = np.zeros(Lt)
  I[0] = noise[0] * sig

  #generate OU
  for it in range(Lt-1):
    I[it+1] = I[it] + dt/tau_ou*(mu-I[it]) + np.sqrt(2.*dt/tau_ou) * sig * noise[it+1]

  return I
```

In the following we will check how a neuron responds to noisly current that follows the statistics of an OU process.

```python
#@title LIF Explorer with OU input

my_layout.width = '450px'
@widgets.interact(
  tau_ou = widgets.FloatSlider(10.0, min=5., max=20., step=2.5, layout=my_layout),
  sig_ou = widgets.FloatSlider(10.0, min=5., max=40., step=2.5, layout=my_layout),
  mu_ou  = widgets.FloatSlider(190.0, min=180., max=220., step=2.5, layout=my_layout)
)

def LIF_with_OU(tau_ou=10., sig_ou=40., mu_ou=200.):
  pars = default_pars(T = 1000.)
  pars['tau_ou'] = tau_ou #[ms]

  I_ou = my_OU(pars, mu_ou, sig_ou)

  v, sp = run_LIF(pars, I=I_ou)

  plt.figure(figsize=(12, 4))
  plt.subplot(1,2,1)
  plt.plot(pars['range_t'], I_ou, 'b', lw=1.0)
  plt.xlabel('Time (ms)')
  plt.ylabel(r'$I_{\mathrm{OU}}$ (pA)')
  plt.subplot(1,2,2)
  plot_volt_trace(pars, v, sp)
  plt.tight_layout()
  plt.show()
```

### Think!
- How does the OU type input change neuron responsiveness?
- What do you think will happen to the spike pattern and rate if you increased or decreased the time constant of the OU process?

# Bonus 2 - Generalized Integrate-and-Fire models

LIF model is not the only abstraction of real neurons. If you want to learn about more realistic types of neuronal models, watch the Bonus Video!

```python
#@title Video: Response of LIF neuron to different inputs
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='zwi6gc8Cbzc', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=zwi6gc8Cbzc


# **Combinatorial Promoter Example**

written by Andrey Shur, 5/4/2020

### CombinatorialPromoter Overview

The CombinatorialPromoter is supposed to represent the general case where a promoter is transcribable when bound by a specific combination of regulators.

In general, consider a set of $M$ regulators $R_1...R_M$ which bind to the promoter $P$ with cooperativities $n_1...n_M$ ie $n_i$ copies of $R_i$ bind to the promoter at once. The CombinatorialPromoter enumerates all possible bound Promoter-Regulator complexes which are returned as a list of ComplexSpecies using the Combinatorial_Cooperative_Binding Mechanism. These complexes will be default take the following form with the regulators in alphabetical order:

\begin{align}
\textrm{Species}=\{&P, P:n_1\textrm{x}R_1, &&..., P:n_M\textrm{x}R_M, \\
&P:n_1\textrm{x}R_1:n_2\textrm{x}R_2, &&..., P:n_{M-1}\textrm{x}R_{M-1}:n_M\textrm{x}R_M \\
&&. \\
&&. \\
&&. \\
&P:n_1\textrm{x}R_1:n_2\textrm{x}R_2: &&...:n_{M-1}\textrm{x}R_{M-1}:n_M\textrm{x}R_M \}
\end{align}

The Combinatorial_Cooperative_Binding Mechanism in Combinatorial Promoter returns all reactions where a Complex consisting of a promoter and a set of regulators bind to one additional regulator.
\begin{align}
\textrm{Reactions} =
\{&P + n_1 R_1 \leftrightarrow P:n_1\textrm{x}R_1, ...\\
 &..., P + n_M R_M \leftrightarrow P:n_M\textrm{x}R_M, ... \\
  &..., P:n_1\textrm{x}R_1 + n_2 R_2 \leftrightarrow P:n_1\textrm{x}R_1:n_2\textrm{x}R_2,... \\
  &..., P:n_{M-1}\textrm{x}R_{M-1} + n_M R_M \leftrightarrow P:n_{M-1}\textrm{x}R_{M-1}:n_M\textrm{x}R_M, ... \\
  &. \\
  &. \\
  &. \\
  &..., P:n_1\textrm{x}R_1:n_2\textrm{x}R_2:...:n_{M-1}\textrm{x}R_{M-1} + n_M R_M
  \leftrightarrow
  P:n_1\textrm{x}R_1:n_2\textrm{x}R_2:...:n_{M-1}\textrm{x}R_{M-1}:n_M\textrm{x}R_M
\end{align}

Arbitrary combinations of these regulators can be defined as transcribable using:

    CombinatorialPromoter(name, [Regulators], tx_capable_list = [[Transcribable Combination fo regulators 1], [Combination 2], ...])

Additionally, there is the option for leak reactions in which case the free promoter is transcribed at some leak rate.

    CombinatorialPromoter(name, [Regulators], leak = True/False )

### Below are imports required for these examples and plotting.

```python
#Using Bioscrape: Basic Imports
#from bioscrape.types import Model
#from bioscrape.simulator import py_simulate_model
#For arrays and plotting

try:
    import numpy as np
    import pylab as plt
    make_heatmap = True
    import numpy as np
    import pylab as plt
    import bioscrape
    import seaborn as sns
    import pandas as pd
except ModuleNotFoundError:
    make_heatmap = False
    print('please install the plotting libraries: pip install biocrnpyler[all]')

#this is for the network representation
try:
    import bokeh.plotting
    import bokeh.io
    bokeh.io.output_notebook()
    plotCRN = True
    from biocrnpyler import plotting
except (ModuleNotFoundError,ImportError) as e:
    plotCRN = False
    print('please install the plotting libraries: pip install biocrnpyler[all]')

#biocrnpyler
from biocrnpyler import *
```

    Loading BokehJS ...

## Example 1: Default Behavior to Produce AND logic with leak
In the following example, we consider a promoter phrpL which can be bound by the proteins hrpR and hrpS. When both proteins are present, transcription is activated, but when one or the other (or none) is present, transcription is not activated.

```python
#these are the additional parameters that you need:
parameters = {
    (None, 'phrpL_hrpR', 'cooperativity'): 1.0, #default is 2, but these proteins bind at cooperativity = 1
    (None, 'phrpL_hrpS', 'cooperativity'): 1.0,
    (None, 'phrpL_hrpR', 'ku'):50, #these are the regulator unbinding rates
    (None, 'phrpL_hrpS', 'ku'):50,
    (None, 'phrpL_leak', 'ktx'): 0.01,  #this is the leak rate
    (None, 'phrpL_leak', 'ku'): 100, #this is the leak polymerase unbinding rate

    ('translation_mm', 'B0030', 'ku'): 10.0, #Unbinding
    ('translation_mm', 'B0030', 'ktl'): 1.5, #Translation Rate

    "kb":100, "ku":10, "ktx":.05, "ktl":.2, "kdeg":2 #some default parameters
}
#above you can see the desired part_id is very simple. The general syntax is <promoter.name>_<regulator.name>.
#using this syntax you can define the strength of binding of each regulator to the promoter.
#cooperativity between different regulators is not supported.

#you can also define the transcription rate and binding and unbinding (kb or ku) of polymerase from a transcribable
#complex. A transcribable complex is a promoter with regulators bound that is capable of transcription.

#By default, all combinations of regulators bound will be capable of transcription. They will use the "ktx" parameter
#promoter with nothing bound will use the phrpL_leak parameter

#now we define the proteins. Their names should match what you put in the promoter, below
hrpR = Protein("hrpR")
hrpS = Protein("hrpS")

#Define the promoter

#this is an "AND gate" promoter. That means that it only leads to transcript if both regulators are bound
#binding of one or the other regulator should not lead to any transcription
#this is the network:
# hrpR + mydna <--> mydna:hrpR
# hrpS + mydna <--> mydna:hrpS
# hrpR + mydna:hrpS <--> mydna:hrpR:hrpS
# hrpS + mydna:hrpR <--> mydna:hrpR:hrpS

phrpL = CombinatorialPromoter("phrpL",["hrpR","hrpS"], leak=True)
#this defines a promoter which binds hrpR and hrpS, and it leaks.
#By default, leak only happens when there is nothing bound to the DNA.
#this also means that in this promoter, individually bound species (with only one regulator) will not lead to
#any transcription! This means they will not leak also.

#this is our DNA
dna = DNAassembly("mydna",promoter=phrpL,rbs="B0030",protein="GFP")

#TxTl Extract is a Mixture with more complex internal models
extract_1_TXTL = TxTlExtract(name = "e coli extract 1", components = [dna,hrpR,hrpS],
                             parameters=parameters, parameter_file = "default_parameters.txt",
                             overwrite_parameters = True) #Overwrite parameters will overwrite the parameters in the file with the dictionary
CRN_extract_1 = extract_1_TXTL.compile_crn()

print("The species and reactions in the CRN:\n", CRN_extract_1.pretty_print())

#below is CRN plotting code. set plotCRN to False to disable
if(plotCRN):
    DG, DGspec, DGrxn = generate_networkx_graph(CRN_extract_1)
    plot = Plot(plot_width=500, plot_height=500, x_range=Range1d(-500, 500), y_range=Range1d(-500, 500))
    graphPlot(DG,DGspec,DGrxn,plot)
    bokeh.io.show(plot)

    #this is just a normal time trace of the model, with both proteins present

    timepoints = np.linspace(0, 200, 1000)
    x0 = {"dna_mydna":5.0, "protein_hrpS":10, "protein_hrpR":7, "protein_RNAP":10., "protein_Ribo":50.,}
    Re1 = CRN_extract_1.simulate_with_bioscrape(timepoints, initial_condition_dict = x0)
    if Re1 is not None:
        plt.plot(timepoints,Re1["protein_GFP"], label = "protein_GFP")
        plt.plot(timepoints,Re1["protein_hrpR"], label = "protein_hrpR")
        plt.plot(timepoints,Re1["protein_hrpS"], label = "protein_hrpS")
        plt.plot(timepoints,Re1["rna_mydna"], label = "rna_mydna")
        plt.gca().set_yscale("log")
        plt.title("Time trace of AND gate promoter phrpL")
        plt.xlabel("time")
        plt.ylabel("protein")
        plt.legend()
    print(parameters)

```

# Vary hrpR and hrpS
To really test the combinatorial promoter we will vary the concentrations of hrpR and hrpS.

```python
if(make_heatmap):
    #aggregator data frame
    GFP_max = pd.DataFrame(columns = ['hrpS_conc', 'hrpR_conc', 'GFP_max'])
    #Different initial values of R and S
    conc_hrpR = np.linspace(0, 30, 9)
    conc_hrpS = np.linspace(0, 30, 9)

    for conc_R in conc_hrpR:
        x0["protein_hrpR"] = conc_R
        for conc_S in conc_hrpS:
            x0["protein_hrpS"] = conc_S #Change my initial condition dictionary
            Re1 = CRN_extract_1.simulate_with_bioscrape(timepoints, initial_condition_dict = x0)
            #now we are simulating over and over again, but only taking the final protein_GFP value
            if Re1 is not None:
                GFP_max = GFP_max.append({'hrpS_conc':conc_S,
                                'hrpR_conc':conc_R,
                                'GFP_max': Re1["protein_GFP"].values[-1]}, ignore_index=True)
    #now, you make a 2d plot with all the data
    if GFP_max is not None:
        data = pd.pivot_table(data = GFP_max, index = 'hrpS_conc',
                    columns = 'hrpR_conc',
                    values = 'GFP_max')

        ax = sns.heatmap(data,
                        cmap = 'viridis',
                        linewidths=.2)
        ax.set_ylim(9,-0.5) #not sure why this is needed

        ax.set(xlabel='hrpS', ylabel='hrpR')
        ax.set_title("hrpR, hrpS Combinatorial Promoter Heatmap")

```

### Improving the AND Logic Model using the tx_capable_list keyword and more detailed parameters

After careful inspection of the above heatmap, you might notice two important features:

1) The promoter is on at all times except when no regulators are present

2) the promoter is on very strongly for most possible concentrations

We will adjust this model by changing the parameters and adding additional transcribable states

1) increase the unbinding rate

2) define single regulator bound promoters as being "leak"

To do this, we will use the tx_capable_list keyword and modify the list of parameters

    phrpL = CombinatorialPromoter(... tx_capable_list = [["hrpR","hrpS"]] ...)

Notice that there are 3 transcribable states, each regulator bound individually and both regulators bound together. Each of these combination of regulators requires their own parameters which are seen below in the newparam dictionary.

```python
newparam = {
    (None, 'phrpL_hrpR', 'cooperativity'): 1.0,
    (None, 'phrpL_hrpS', 'cooperativity'): 1.0,
    (None, 'phrpL_hrpR','ku'):5, #unbinding rate of each regulator
    (None, 'phrpL_hrpS','ku'):5,
    (None, 'phrpL_hrpR_hrpS_RNAP', 'ktx'): 0.17025, #these are the rates for the correctly bound promoter
    (None, 'phrpL_hrpR_hrpS_RNAP', 'ku'): 11.01321586,
    (None, 'phrpL_leak','ktx'):0.01, #this is the leak transcription rate
    (None, 'phrpL_leak', 'ku'): 100.,#this unbinding rate is now needed because we changed the tx_capable_list. Transcription from singly bound promoter uses the default "ktx"
}

#WARNING!!!
#out of order regulators will not work! For example, phrpL_hrpR_hrpS_RNAP is correct, but phrpL_hrpS_hrpR_RNAP is not.

parameters.update(newparam)
#you can also define the ktx for the leak reaction. I don't think this is necessary
#    ('phrpL_hrpR_RNAP', 'ktx'): 0.17,
#    ('phrpL_hrpS_RNAP', 'ktx'): 0.17,

phrpL = CombinatorialPromoter("phrpL",["hrpR","hrpS"],
                              tx_capable_list = [[Species("hrpR",material_type="protein"),"hrpS"]], #species can also be used
                              leak=True)
#now we are saying that there are more species that are capable of transcribing. Specifically, that we
#can still transcribe when both are bound, but also when one or the other is bound.

#this is our DNA
dna = DNAassembly("mydna",promoter=phrpL,rbs="B0030",protein="GFP")

extract_1_TXTL = TxTlExtract(name = "e coli extract 1", components = [dna,hrpR,hrpS],
                             parameters=parameters, parameter_file = "default_parameters.txt",
                             overwrite_parameters = True) #Overwrite parameters will overwrite the parameters in the file with the dictionary
CRN_extract_1 = extract_1_TXTL.compile_crn()

if(make_heatmap):
    #aggregator data frame
    GFP_max = pd.DataFrame(columns = ['hrpS_conc', 'hrpR_conc', 'GFP_max'])
    #Different initial values of R and S
    conc_hrpR = np.linspace(0, 30, 9)
    conc_hrpS = np.linspace(0, 30, 9)

    for conc_R in conc_hrpR:
        x0["protein_hrpR"] = conc_R
        for conc_S in conc_hrpS:
            x0["protein_hrpS"] = conc_S #Change my initial condition dictionary
            Re1 = CRN_extract_1.simulate_with_bioscrape(timepoints, initial_condition_dict = x0)
            #now we are simulating over and over again, but only taking the final protein_GFP value
            GFP_max = GFP_max.append({'hrpS_conc':conc_S,
                            'hrpR_conc':conc_R,
                            'GFP_max': Re1["protein_GFP"].values[-1]}, ignore_index=True)

    #now, you make a 2d plot with all the data
    data = pd.pivot_table(data = GFP_max, index = 'hrpS_conc',
                columns = 'hrpR_conc',
                values = 'GFP_max')

    ax = sns.heatmap(data,
                    cmap = 'viridis',
                    linewidths=.2)
    ax.set_ylim(9,-0.5) #not sure why this is needed

    ax.set(xlabel='hrpS', ylabel='hrpR')
    ax.set_title("hrpR, hrpS Combinatorial Promoter Heatmap")
```

## Example 2: Using the tx_capable_list keyword to implement XOR logic
In the following example, we consider a promoter phrpL which can be bound by the proteins hrpR and hrpS. When either one  proteins or the other is present, but not both, transcription is activated. To do this, we pass in the following combinations to transcribable list:

    phrpL = CombinatorialPromoter(... tx_capable_list = [["hrpR"],["hrpS"]] ...)

This ensures that only when hrpR or hrpS is individually bound to the promoter can it be transcribed.

```python
newparam = {
    (None, 'phrpL_hrpR', 'cooperativity'): 1.0,
    (None, 'phrpL_hrpS', 'cooperativity'): 1.0,
    (None, 'phrpL_hrpR','ku'):3, #we have to massively decrease the unbinding rate to make this xor gate work
    (None, 'phrpL_hrpS','ku'):3,
    #(None, 'phrpL_hrpR_hrpS_RNAP', 'ku'): 100, #this time, this one is leak
    (None, 'phrpL_hrpR_RNAP', 'ktx'): 0.17, #these are now transcribing at high rates!
    (None, 'phrpL_hrpS_RNAP', 'ktx'): 0.17,
    (None, 'phrpL_hrpS_RNAP', 'ku'): 11.,
    (None, 'phrpL_hrpR_RNAP', 'ku'): 11,
    (None, 'phrpL','ku'):100,

}

parameters.update(newparam)
phrpL = CombinatorialPromoter("phrpL",["hrpR","hrpS"],
                              tx_capable_list = [["hrpR"],["hrpS"]],
                              leak=True)
#this is our DNA
dna = DNAassembly("mydna",promoter=phrpL,rbs="B0030",protein="GFP")

extract_1_TXTL = TxTlExtract(name = "e coli extract 1", components = [dna,hrpR,hrpS], \
                             parameters=parameters, parameter_file = "default_parameters.txt",
                             overwrite_parameters = True) #Overwrite parameters will overwrite the parameters in the file with the dictionary

CRN_extract_1 = extract_1_TXTL.compile_crn()

if(make_heatmap):
    #aggregator data frame
    GFP_max = pd.DataFrame(columns = ['hrpS_conc', 'hrpR_conc', 'GFP_max'])
    #Different initial values of R and S
    conc_hrpR = np.linspace(0,30, 9)
    conc_hrpS = np.linspace(0, 30, 9)
    for conc_R in conc_hrpR:
        x0["protein_hrpR"] = conc_R
        for conc_S in conc_hrpS:
            x0["protein_hrpS"] = conc_S #Change my initial condition dictionary
            Re1 = CRN_extract_1.simulate_with_bioscrape(timepoints, initial_condition_dict = x0)
            #now we are simulating over and over again, but only taking the final protein_GFP value
            GFP_max = GFP_max.append({'hrpS_conc':conc_S,
                            'hrpR_conc':conc_R,
                            'GFP_max': Re1["protein_GFP"].values[-1]}, ignore_index=True)

    #now, you make a 2d plot with all the data
    data = pd.pivot_table(data = GFP_max, index = 'hrpS_conc',
                columns = 'hrpR_conc',
                values = 'GFP_max')

    ax = sns.heatmap(data,
                    cmap = 'viridis',
                    linewidths=.2)
    ax.set_ylim(9,-0.5) #not sure why this is needed

    ax.set(xlabel='hrpS', ylabel='hrpR')
    ax.set_title("hrpR, hrpS Combinatorial Promoter Heatmap")
```

```python

```

examples/Specialized Component Tutorials/CombinatorialPromoter Details.ipynb


## Fully-connected Neural Networks with Keras

As part of this tutorial, we are going to revist the MNIST image classification use case using classical fully-connected (also dense) artificial neural networks (FC-NN, Dense-NN). In the basic form of such a neural network, each of the neurons in the network is a single logistic regression. We employ multiple of these units in order to be able to model highly complex, non-linear separation boundaries between individual classes (see also the XOR-problem).

Thereby, the individual neurons are usually arranged in so-called layers that connected with each other in a directed acyclic structure. There are usually at least two layers, the input and output layer, where the data samples are passed in, and the prediction read out respectively. Additionally, there may be a number of hidden layers, additional logistic regression units, allowing to model higher-order functions. Each of the neurons within a layer is connected to every unit of the previous and subsequent layer, implementing a layer-wise full (hence the name) connectivity. Having more than one hidden unit is bluntly put deep-learning.

© Medium.com [1]

### Setup

For a detailed explanation of the use modules, please refer to the respective sections in the [introductory notebook](0_MNIST_dataset.ipynb) and the one on [logistic regression](1_logistic_regression.ipynb).

```python
import h5py
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
plt.style.use('seaborn')

SEED = 42
np.random.seed(SEED)
tf.set_random_seed(SEED)
from keras.callbacks import Callback
from keras.models import Sequential
from keras.layers import Dense
from keras.utils import to_categorical
```

### Loading the Data

We will once again load the MNIST database from the HDF5 file. For details, please also see the [introductory notebook](MNIST dataset.ipynb).

```python
def load_data(path='mnist.h5'):
    """
    Loads a dataset and its supervisor labels from the HDF5 file specified by the path.
    It is assumed that the HDF5 dataset containing the data is called 'data' and the labels are called 'labels'.

    Parameters
    ----------
    path : str, optional
        The absolute or relative path to the HDF5 file, defaults to mnist.h5.

    Returns
    -------
    data_and_labels : tuple(np.array[samples, width, height], np.array[samples])
        a tuple with two numpy array containing the data and labels
    """
    with h5py.File(path, 'r') as handle:
        return np.array(handle['/data']), np.array(handle['/labels'])

data, labels = load_data()
data.shape, labels.shape
```

### Preprocessing the Data

Just like in the logistic regression tutorial, we consider each of the individual pixels of an image as one of the features of our classical problem. This means, we will again flatten the image into a long vector of 28 x 28 = 784 elements.

Additionally, we will from now on also only consider the multiclass classification problem, of identifying for all image the correct digit class from all of the available ten. Therefore, we need to think about how we encode our supervisor labels. The naïve approach would be to have a single output neuron and let it simply predict the number it thinks it depicted in the image. This approach, however, has two major downsides. First, the output neuron value range would be unbound, and a prediction of say *-1* or *11* could be reasonably expected. Second, in an output format like this some numbers would be illogically more similar to each other (due to their absolute difference) than they are in reality. The digit *7* for example would be very similar to *6* and *8* and not to *1*.

To counteract this effect, we are going to choose a different label encoding instead. For each samples, we will translate the label into a vector that is as long as the number of unique classes (here: 10). Each of the positions in the vector is uniquely associated with one of the individual classes and is set to 1 if the data sample is member of the respective class and 0 otherwise. Using this encoding, also called one-hot-encoding [2], each class has the same distance to every other class and we can additionally interpret the output again to be the probability of the data item belonging to each individual class.

\begin{align}
1 & \rightarrow [0, 1, 0, 0, 0, 0, 0, 0, 0, 0] \\
7 & \rightarrow [0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
\end{align}

One of the downsides of the proposed one-hot encoding method is a significantly increased memory footprint. In large-scale machine learning applications it might be necessary to use techniques like embedding or sparse matrix formats for the proper output encoding.

```python
def preprocess_data(data, labels):
    """
    Flattens the each image in the data to be a one-dimensional feature vector and encodes the labels in one-hot encoding.

    Parameters
    ----------
    data : np.array[samples, width, height]
        the image dataset
    labels : np.array[samples]
        the corresponding labels

    Returns
    -------
    data_and_labels : tuple(np.array[samples, width * height], np.array[samples, classes])
        a tuple with two numpy array containing the flattened data and one-hot encoded labels
    """
    return data.reshape(data.shape[0], -1), to_categorical(labels)

preprocessed_data, preprocessed_labels = preprocess_data(data, labels)
```

### Training and Test Dataset Partitioning

In order to be able to judge whether our machine learning algorithm actually learns a pattern and does not simply memorize the training data, we need to verify whether it is able to correctly predict unseen or "new" data. One way of emulating this behaviour, is to artificially split the whole input dataset into disjoint training and test data (sometimes additionally validation data for model hyperparameter optimization) and use the former for adjusting the internal model parameters and the latter for testing the out-of-sample performance.

```python
def partition(data, labels, train_fraction=0.8):
    """
    Flattens the each image in the data to be a one-dimensional feature vector and encodes the labels in one-hot encoding.

    Parameters
    ----------
    data : np.array[samples, width * height]
        the image dataset
    labels : np.array[samples, classes]
        the corresponding labels
    train_fraction : float, optional
        the fraction between in [0.0, 1.0] of data items beeing assigned to the training dataset, defaults to 0.8

    Returns
    -------
    train_and_test_data : tuple(
        np.array[samples, width * height], np.array[samples, classes]
        np.array[samples, width * height], np.array[samples, classes]
    )
        a tuple with four numpy arrays containing the training data and labels as well as the test data and labels
    """
    ##############
    return the training and test data sets
##############

train_data, train_labels, test_data, test_labels = partition(preprocessed_data, preprocessed_labels)
train_data.shape, train_labels.shape, test_data.shape, test_labels.shape
```

### Building a Fully-connected Neural Network

Using the previously introduced (see [logistic regression notebook](Logistic Regression.ipynb)) Keras API for implementing a logistic regression, we may also implement fully-connected neural networks by increasing the number of neurons in one layer and adding more layers. Using the "softmax" activation function, we can realize a multi-class classifier with a joint output probability of 1.

```python
def build_model(data, classes):
    """
    Constructs a fully-connected neural network model for the given data and number of classes

    Parameters
    ----------
    data : np.array[samples, width * height]
        the image dataset
    classes : int
        the number of unique classes in the dataset

    Returns
    -------
    model : keras.Model
        the fully-connected neural network
    """
    model = Sequential()

    ##############
    add all the layers
    ##############

    model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])

    return model

model = build_model(preprocessed_data, classes=preprocessed_labels.shape[1])
```

### Setup for Model Training Progress Monitoring

Keras allows basic training monitoring through the output on the standard output and some pre-implemented callback functions. We would like to have a more fine-grained tracking for the training history in order to be able closely observe the progress for each batch. To do so, we extend a default Keras Callback class with our custom implementation that track the history for training and test phase.

```python
class TrainingHistory(Callback):
    """
    Class for tracking the training progress/history of the neural network. Implements the keras.Callback interface.
    """
    def on_train_begin(self, logs):
        self.loss = []
        self.acc = []
        self.validation_loss = []
        self.validation_acc = []

    def on_batch_end(self, _, logs):
        """
        Callback invoked after each training batch.
        Should track the training loss and accuracy in the respective members.

        Parameters
        ----------
        _ : int
            unused, int corresponding to the batch number
        logs : dict{str -> float}
            a dictionary mapping from the observed quantity to the actual valu
        """
        if 'loss' in logs:
            self.loss.append(logs['loss'])
        if 'acc' in logs:
            self.acc.append(logs['acc'])

    def on_epoch_end(self, _, logs):
        if 'val_loss' in logs:
            self.validation_loss.append(logs['val_loss'])
        if 'val_acc' in logs:
            self.validation_acc.append(logs['val_acc'])
```

### Training a Model

Training a fully-connected neural network works analogous to [logistic regression](Logistic Regression.ipynb) using the Keras API.

```python
def train_model(model, train_data, train_labels, test_data, test_labels, epochs=50, batch_size=64):
    """
    Trains a fully-connected neural network given training and test data/labels.

    Parameters
    ----------
    model : keras.Model
        the fully-connected neural network
    train_data : np.array[samples, width * height]
        the training data
    train_labels : np.array[samples, classes]
        the one-hot encoded training labels
    test_data : np.array[samples, width * height]
        the test data
    test_labels : np.array[samples, classes]
        the one-hot encoded test labels
    epoch: positive int, optional
        the number of epochs for which the neural network is trained, defaults to 50
    batch_size: positive int, optional
        the size of the training batches, defaults to 64

    Returns
    -------
    history : TrainingHistory
        the tracked training and test history
    """
    history = TrainingHistory()
    ##############
    fit the model  # also pass validation_data=(test_data, test_labels,), shuffle=True, callbacks=[history]
    ##############
    return history

history = train_model(model, train_data, train_labels, test_data, test_labels)
```

### Visualization of the Training Progress

Using the previously created and filled TrainingHistory instance, we are able to plot the loss and accuracy of the training batches and test epochs.

```python
def plot_history(history):
    """
    Plots the training (batch-wise) and test (epoch-wise) loss and accuracy.

    Parameters
    ----------
    history : TrainingHistory
        an instance of TrainingHistory monitoring callback
    """
    figure, (batch_axis, epoch_axis) = plt.subplots(2, 1, figsize=(16, 9), sharey=True)

    # plot the training loss and accuracy
    batch_axis.set_xlabel('batch number')
    training_batches = np.arange(len(history.loss))

    batch_axis.grid(False)
    batch_axis.plot(training_batches, history.loss, color='C0', label='loss')
    batch_axis.set_ylabel('loss')

    batch_acc_axis = batch_axis.twinx()
    batch_acc_axis.grid(False)
    batch_acc_axis.set_ylabel('accuracy')
    batch_acc_axis.set_ylim(bottom=0.0)
    batch_acc_axis.plot(training_batches, history.acc, color='C4', label='accuracy')

    # plot the training loss and accuracy
    epoch_axis.set_xlabel('epoch number')
    validation_epochs = np.arange(len(history.validation_loss))

    epoch_axis.grid(False)
    epoch_axis.plot(validation_epochs, history.validation_loss, color='C0')
    epoch_axis.set_ylabel('validation loss')

    epoch_acc_axis = epoch_axis.twinx()
    epoch_acc_axis.grid(False)
    epoch_acc_axis.set_ylabel('validation accuracy')
    epoch_acc_axis.set_ylim(bottom=0.0)
    epoch_acc_axis.plot(validation_epochs, history.validation_acc, color='C4')

    # display a legend
    figure.legend(loc=8)
    plt.show()

plot_history(history)
```

### Prediction of "New Data"

Using a previously trained model, we are able to predict the classes of new "unseen" data. A prediction produces for each of the input data samples a 10-item long vector where each vector position contains the probability of the data item belonging to the respective class.

```python
def predict(model, test_data, batch_size=64):
    """
    Predicts for a given fully-connected neural network and a number of data samples the resulting classes.

    Parameters
    ----------
    model : keras.Model
        the fully-connected neural network
    test data : np.array[samples, width * height]
        the data to be predicted
    batch_size: positive int, optional
        the size of the training batches, defaults to 64

    Returns
    -------
    prediction : np.array[samples, classes]
        a class prediction for each class
    """
    ##############
    return all the predictions for the given data and parameters
##############

test_prediction = predict(model, test_data)
test_prediction.shape
```

### Let See Some Examples...

... in order to see how well our model works. Using the predictions on the test data, we can use a bar plot to visualize all the class membership probabilities of the respective images. Green bars indicate the original true class, while red bars indicate mispredictions.

```python
def visualize_prediction(data, labels, prediction, how_many=10):
    """
    Visualizes for a given number of test images the class membership prediction made by our neural network.

    Parameters
    ----------
    data : np.array[samples, width * height]
        the test image
    labels : np.array[samples, classes]
        the actual one-hot encoded labels for each image
    prediction: np.array[samples, classes]
        the one-hot encoded predictions for each image
    how_many : positive int, optional
        indicates how many samples shall be displayed starting from the first, defaults to 10
    """
    figure, axes = plt.subplots(how_many, 2, figsize=(12, 24))
    digits = np.arange(labels.shape[1])

    for i in range(how_many):
        left, right = axes[i]

        left.imshow(data[i].reshape(28, 28), cmap='gist_gray')
        left.grid(False)

        bars = right.bar(digits, prediction[i], color='C2')
        ##############
        color the correct bar in green ('C1')
        ##############
        right.grid(False)

        right.set_xlabel('digit')
        right.set_xticks(digits)

        right.set_ylim(0.0, 1.0)
        right.set_ylabel('predicted probability')

    plt.tight_layout()
    plt.show()

visualize_prediction(test_data, test_labels, test_prediction)
```

### Where Did Things Go Wrong - The Confusion Matrix

A confusion matrix is a visual way of comparing the overall predictions with the ground-truth labels. For this, a we create a class x class-sized matrix where we each position is equivalent to the relative frequency of the prediction given a particular class label. An optimal confusion matrix would result in a matrix with each of the diagonal elements equal to 1. In practice we will obviously not achieve this. Instead, with this mode of presentation we are able to judge what kind of class labels the prediction algorithm commonly confuses with other class and, potentially, take c

```python
def show_confusion_matrix(labels, prediction):
    """
    Computes and visualizes the confusion matrix for the test data.

    Parameters
    ----------
    labels : np.array[samples, classes]
        the actual labels of the data items
    prediction: np.array[samples, classes]
        the corresponding predictions of the neural network
    """
    classes = labels.shape[1]
    buckets = np.arange(classes)
    confusion = np.histogram2d(np.argmax(labels, axis=1), np.argmax(prediction, axis=1), bins=classes)[0]
    confusion /= confusion.sum(axis=0)

    figure, axis = plt.subplots(1, figsize=(12, 12))
    axis.grid(False)
    image = axis.imshow(confusion, cmap='viridis')
    axis.set_xlabel('actual')
    axis.set_xticks(buckets)
    axis.set_ylabel('prediction')
    axis.set_yticks(buckets)
    figure.colorbar(image, shrink=0.8)

    for y in range(classes):
        for x in range(classes):
            value = confusion[y, x]
            color = 'white' if value <= 0.5 else 'black'
            plt.annotate('{:.4f}'.format(float(value)), xy=(x, y), color=color, ha='center', va='center')

    plt.show()

show_confusion_matrix(test_labels, test_prediction)
```

### Bonus

Try to see what the effect of stacking multiple fully-connected layers is.

### References

[1] **Fully-connected Neural Network**, *Venelin Valkov*, (2017), [external link](https://medium.com/@curiousily/tensorflow-for-hackers-part-iv-neural-network-from-scratch-1a4f504dfa8).

[2] **One-hot**, *Wikipedia*, (2018), [external link](https://en.wikipedia.org/wiki/One-hot).


```python
%matplotlib inline
```

# Scaling the regularization parameter for SVCs

The following example illustrates the effect of scaling the
regularization parameter when using `svm` for
`classification <svm_classification>`.
For SVC classification, we are interested in a risk minimization for the
equation:

\begin{align}C \sum_{i=1, n} \mathcal{L} (f(x_i), y_i) + \Omega (w)\end{align}

where

    - $C$ is used to set the amount of regularization
    - $\mathcal{L}$ is a `loss` function of our samples
      and our model parameters.
    - $\Omega$ is a `penalty` function of our model parameters

If we consider the loss function to be the individual error per
sample, then the data-fit term, or the sum of the error for each sample, will
increase as we add more samples. The penalization term, however, will not
increase.

When using, for example, `cross validation <cross_validation>`, to
set the amount of regularization with `C`, there will be a
different amount of samples between the main problem and the smaller problems
within the folds of the cross validation.

Since our loss function is dependent on the amount of samples, the latter
will influence the selected value of `C`.
The question that arises is `How do we optimally adjust C to
account for the different amount of training samples?`

The figures below are used to illustrate the effect of scaling our
`C` to compensate for the change in the number of samples, in the
case of using an `l1` penalty, as well as the `l2` penalty.

l1-penalty case
-----------------
In the `l1` case, theory says that prediction consistency
(i.e. that under given hypothesis, the estimator
learned predicts as well as a model knowing the true distribution)
is not possible because of the bias of the `l1`. It does say, however,
that model consistency, in terms of finding the right set of non-zero
parameters as well as their signs, can be achieved by scaling
`C1`.

l2-penalty case
-----------------
The theory says that in order to achieve prediction consistency, the
penalty parameter should be kept constant
as the number of samples grow.

Simulations
------------

The two figures below plot the values of `C` on the `x-axis` and the
corresponding cross-validation scores on the `y-axis`, for several different
fractions of a generated data-set.

In the `l1` penalty case, the cross-validation-error correlates best with
the test-error, when scaling our `C` with the number of samples, `n`,
which can be seen in the first figure.

For the `l2` penalty case, the best result comes from the case where `C`
is not scaled.

.. topic:: Note:

    Two separate datasets are used for the two different plots. The reason
    behind this is the `l1` case works better on sparse data, while `l2`
    is better suited to the non-sparse case.

--

**NOTE:** This is sourced from ```scikit-learn``` learning module found here:
https://scikit-learn.org/stable/auto_examples/svm/plot_svm_scale_c.html#sphx-glr-auto-examples-svm-plot-svm-scale-c-py

--

```python
print(__doc__)

# Author: Andreas Mueller <amueller@ais.uni-bonn.de>
#         Jaques Grobler <jaques.grobler@inria.fr>
# License: BSD 3 clause

import numpy as np
import matplotlib.pyplot as plt

from sklearn.svm import LinearSVC
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import GridSearchCV
from sklearn.utils import check_random_state
from sklearn import datasets

rnd = check_random_state(1)

# set up dataset
n_samples = 100
n_features = 300

# l1 data (only 5 informative features)
X_1, y_1 = datasets.make_classification(n_samples=n_samples,
                                        n_features=n_features, n_informative=5,
                                        random_state=1)

# l2 data: non sparse, but less features
y_2 = np.sign(.5 - rnd.rand(n_samples))
X_2 = rnd.randn(n_samples, n_features // 5) + y_2[:, np.newaxis]
X_2 += 5 * rnd.randn(n_samples, n_features // 5)

clf_sets = [(LinearSVC(penalty='l1', loss='squared_hinge', dual=False,
                       tol=1e-3),
             np.logspace(-2.3, -1.3, 10), X_1, y_1),
            (LinearSVC(penalty='l2', loss='squared_hinge', dual=True,
                       tol=1e-4),
             np.logspace(-4.5, -2, 10), X_2, y_2)]

colors = ['navy', 'cyan', 'darkorange']
lw = 2

for clf, cs, X, y in clf_sets:
    # set up the plot for each regressor
    fig, axes = plt.subplots(nrows=2, sharey=True, figsize=(9, 10))

    for k, train_size in enumerate(np.linspace(0.3, 0.7, 3)[::-1]):
        param_grid = dict(C=cs)
        # To get nice curve, we need a large number of iterations to
        # reduce the variance
        grid = GridSearchCV(clf, refit=False, param_grid=param_grid,
                            cv=ShuffleSplit(train_size=train_size,
                                            test_size=.3,
                                            n_splits=250, random_state=1))
        grid.fit(X, y)
        scores = grid.cv_results_['mean_test_score']

        scales = [(1, 'No scaling'),
                  ((n_samples * train_size), '1/n_samples'),
                  ]

        for ax, (scaler, name) in zip(axes, scales):
            ax.set_xlabel('C')
            ax.set_ylabel('CV Score')
            grid_cs = cs * float(scaler)  # scale the C's
            ax.semilogx(grid_cs, scores, label="fraction %.2f" %
                        train_size, color=colors[k], lw=lw)
            ax.set_title('scaling=%s, penalty=%s, loss=%s' %
                         (name, clf.penalty, clf.loss))

    plt.legend(loc="best")
plt.show()
```

```python

```


# Pengenalan Notebook
Pertemuan minggu ke-14: pengenalan `jupyter notebook`

Iterasi `for` Python

```python
for i in range(0,11):
    print(i)
```

    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10

## Persamaan
misal

\begin{equation} \tag{1}
y = ax^2 +bx + c
\end{equation}

Merujuk persamaan (1), maka blabla.

## Plot Kurva
Digunakan `matplotlib`

```python
import numpy as np
import matplotlib.pyplot as plt
plt.ion()

x = [1,2,3,4,5]
y = [1,2,3,4,5]

fig, ax = plt.subplots()
ax.scatter(x,y)
ax.set_xlabel('x')
ax.set_ylabel('y')
```

## Plot dengan Fungsi
Fungsi:
\begin{equation}
g(x) = \sin{x}
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt
import math

plt.ion()

def g(x):
    y = 5 * math.sin(x)
    return y

x = []
y = []
for i in range(21):
    xx = 2 * math.pi * i/20
    yy = g(xx)

    x.append(xx)
    y.append(yy)

# plot
fig, ax = plt.subplots()
ax.scatter(x,y)
ax.set_xlabel('x')
ax.set_ylabel('y')
```

## Tabel
Pakai markdown

| Titik | $x$   | $y$   |$z$    |
|:-:|:-:|:-:|:-:|
| A | 1 | 2 | 3 |
| B | -1 | -2 | -3 |

menggunakan kode markdown berikut.
```md
| Titik | $x$   | $y$   |$z$    |
|:-:|:-:|:-:|:-:|
| A | 1 | 2 | 3 |
| B | -1 | -2 | -3 |
```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2018 parts of this notebook are from ([this Jupyter notebook](https://nbviewer.jupyter.org/github/krischer/seismo_live/blob/master/notebooks/Computational%20Seismology/The Finite-Difference Method/fd_first_derivative.ipynb)) by Kristina Garina and Heiner Igel ([@heinerigel](https://github.com/heinerigel)) which is a supplemenatry material to the book [Computational Seismology: A Practical Introduction](http://www.computational-seismology.org/),  additional modifications by D. Koehn, notebook style sheet by L.A. Barba, N.C. Clementi

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# The Finite Difference Method

In the last lecture I introduced a whole bunch of partial differential equations for the conservation of mass, momentum and energy. In order to solve these equations numerically, we will discuss a very simple approach called the finite-difference method.

## Approximation of first derivatives

The approximation of first derivatives using finite-differences is quite straightforward. Imagine that we have some time-dependent function $f(t)$, for which we want to calculate the first derivatives $\frac{\partial f(t)}{\partial t}$:

First, we discretize $f(t)$ at discrete times

\begin{equation}
t = i * dt, \nonumber
\end{equation}

where $dt$ denotes a constant temporal sample interval. By definition, the first derivative of $f(t)$ with respect to $t$ is:

\begin{equation}
\frac{\partial f(t)}{\partial t} = \lim_{dt \rightarrow 0} \frac{f(t+dt)-f(t)}{dt}, \nonumber
\end{equation}

In the **finite-difference (FD) approximation**, we neglect the limit $dt \rightarrow 0$:

\begin{equation}
\frac{\partial f(t)}{\partial t} \approx \frac{f(t+dt)-f(t)}{dt} \nonumber
\end{equation}

Because $dt$ remains finite, this approximation is called finite-differences. Furthermore, we can distinguish between different finite difference operators, depending on the points involved in the approximation. The above example is a **forward FD operator**

\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr)^+ \approx \frac{f(t+dt)-f(t)}{dt}. \nonumber
\end{equation}

Alternatively, we can also define a **backward FD operator**

\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr)^- \approx \frac{f(t)-f(t-dt)}{dt}. \nonumber
\end{equation}

By taking the arithmetic average of the forward and backward operator

\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr) \approx \frac{\left(\frac{\partial f(t)}{\partial t}\right)^- + \left(\frac{\partial f(t)}{\partial t}\right)^+}{2},\nonumber
\end{equation}

we get the **central FD operator**

\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr) \approx \frac{f(t+dt)-f(t-dt)}{2dt}. \nonumber
\end{equation}

##### Exercise 1

Prove for the quadratic function

\begin{equation}
g(t) = b t^2 \nonumber
\end{equation}

where $b$ is a constant time-independent parameter, that in the limit $dt \rightarrow 0$, the forward, backward and central FD operators lead to the correct temporal first derivative:

\begin{equation}
\frac{\partial g(t)}{\partial t} = 2 b t \nonumber
\end{equation}

```python
# Import Libraries
import numpy as np
import math
import matplotlib.pyplot as plt
from pylab import rcParams
```

As an example, let's try to calculate the first temporal derivative of the Gaussian function

\begin{equation}
f(t)=\dfrac{1}{\sqrt{2 \pi a}}e^{-\dfrac{(t-t_0)^2}{2a}},\nonumber
\end{equation}

where $a$ denotes the half-width of the Gaussian and $t_0$ a time-shift of the maximum.

```python
# Define figure size
rcParams['figure.figsize'] = 12, 5

# Initial parameters
tmax=10.0                     # maximum time
nt=1000                       # number of time sample
a=1                           # half-width
dt=tmax/nt                    # defining dt
t0 = tmax/2                   # defining time shift t0

time=np.linspace(0,tmax,nt)   # defining time

# Define gaussian function
f=(1./np.sqrt(2*np.pi*a))*np.exp(-(((time-t0)**2)/(2*a)))

# Plotting of gaussian
plt.plot(time, f)
plt.title('Gaussian function')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.xlim((0, tmax))
plt.grid()
plt.show()
```

Next, we calculate the numerical derivative using finite-differences with the forward operator:

\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr)^+ \approx \frac{f(t+dt)-f(t)}{dt}. \nonumber
\end{equation}

backward operator:

\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr)^- \approx \frac{f(t)-f(t-dt)}{dt}, \nonumber
\end{equation}

and central operator:

\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr) \approx \frac{f(t+dt)-f(t-dt)}{2dt}. \nonumber
\end{equation}

To test the accuarcy of these approaches, we compare them with the analytical derivative:

\begin{equation}
\frac{\partial f(t)}{\partial t}=-\dfrac{t-t_0}{a}\dfrac{1}{\sqrt{2\pi a}}e^{-\dfrac{(t-t_0)^2}{2a}} \nonumber
\end{equation}

```python
# First derivative of Gaussian function

# Initiation of numerical and analytical derivatives
nder_for=np.zeros(nt)      # forward FD operator
nder_back=np.zeros(nt)     # backward FD operator
nder_cent=np.zeros(nt)     # central FD operator
ader=np.zeros(nt)          # analytical derivative

# Numerical FD derivative of the Gaussian function
for it in range (1, nt-1):

    nder_for[it]=(f[it+1]-f[it])/dt        # forward operator
    nder_back[it]=(f[it]-f[it-1])/dt       # backward operator
    nder_cent[it]=(f[it+1]-f[it-1])/(2*dt) # central operator

# Analytical derivative of the Gaussian function
ader=(-(time-t0)/a)*(1/np.sqrt(2*np.pi*a))*np.exp(-(time-t0)**2/(2*a))

# Plot of the first derivative and analytical derivative
plt.plot(time, nder_for,label="FD forward operator", lw=2, color="y")
plt.plot(time, nder_back,label="FD backward operator", lw=2, color="b")
plt.plot(time, nder_cent,label="FD central operator", lw=2, color="g")
plt.plot(time, ader, label="Analytical derivative", ls="--",lw=2, color="red")
plt.title('First derivative')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()
```

The approximation of the first derivative of the Gaussian by all three FD operators seem to be very accurate. How large are actually the errors?

```python
# Erros of the FD approximation

# Plot of the errors of the first derivative FD approximations
plt.plot(time, 10*(nder_for-ader),label="FD forward operator error (10x)", lw=2, color="y")
plt.plot(time, 10*(nder_back-ader),label="FD backward operator error (10x)", lw=2, color="b")
plt.plot(time, 10*(nder_cent-ader),label="FD central operator error (10x)", lw=2, color="g")
plt.plot(time, ader, label="Analytical derivative",lw=2, color="red")
plt.title('First derivative errors')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()
```

The errors for all three operators are actually quite small, so we have to amplify them by a factor 10. Larger errors occur in areas with significant slope. Notice also, that the central operator has the smallest error. Based on the geometrical interpretation of the finite difference operators in the class, we concluded that the secant slope of the central operator is the most effective approximation to the tangent slope of the true first derivative $f(t)$. You can find a more detailed mathematical explanation based on Taylor series expansion in [this lecture](https://nbviewer.jupyter.org/github/daniel-koehn/Theory-of-seismic-waves-II/blob/master/04_FD_stability_dispersion/2_fd_accuracy.ipynb) of my seismic modelling course.

## FD approximation of the 2nd derivative

We derived FD approximations for the 1st derivative. However, to approximate the acoustic wave equation, we also need to know how to approximate a 2nd derivative.
To achieve this, we calculate the **Taylor expansion** of $f(t+dt)$ up to the second order term:

\begin{equation}
f(t+dt) \approx f(t) + \frac{\partial f(t)}{\partial t} dt + \frac{1}{2}\frac{\partial^2 f(t)}{\partial t^2} dt^2 + \mathcal{O}(dt^3)\nonumber
\end{equation}

We do the same for $f(t-dt)$:

\begin{equation}
f(t-dt) \approx f(t) - \frac{\partial f(t)}{\partial t} dt + \frac{1}{2}\frac{\partial^2 f(t)}{\partial t^2} dt^2 + \mathcal{O}(dt^3)\nonumber
\end{equation}

Adding the approximations $f(t+dt)$ and $f(t-dt)$ leads to

\begin{equation}
f(t-dt) + f(t+dt) = 2 f(t) + \frac{\partial^2 f(t)}{\partial t^2} dt^2\nonumber
\end{equation}

Finally, we can rearrange to $\frac{\partial^2 f(t)}{\partial t^2}$ and get the following **FD approximation for the 2nd derivative**

\begin{equation}
\frac{\partial^2 f(t)}{\partial t^2} \approx \frac{f(t-dt) + f(t+dt) - 2 f(t)}{dt^2}\nonumber
\end{equation}

##### Excercise 2

* Calculate the 2nd derivative $\frac{\partial^2 f(t)}{\partial t^2}$ of the Gaussian
\begin{equation}
f(t)=\dfrac{1}{\sqrt{2 \pi a}}e^{-\dfrac{(t-t_0)^2}{2a}},\nonumber
\end{equation}
analytically
* Compute and compare the numerical and analytical 2nd derivative of the Gaussian and the errors between both solutions

```python
# 2nd derivative of Gaussian function

# Initiation of numerical and analytical derivatives
nder2=np.zeros(nt)      # 2nd derivative FD operator
ader2=np.zeros(nt)      # analytical 2nd derivative

# Numerical FD derivative of the Gaussian function
for it in range (1, nt-1):

    # ADD YOUR 2ND DERIVATIVE OF THE GAUSSIAN HERE!
    # nder2[it]=        # 2nd derivative FD operator

# ADD ANALYTICAL 2ND DERIVATIVE OF THE GAUSSIAN HERE!
# ader2=

# Plot of the numerical and analytical second derivative of the Gaussian
#plt.plot(time, nder2,label="FD operator", lw=2, color="y")
#plt.plot(time, ader2, label="Analytical 2nd derivative", ls="--",lw=2, color="red")
plt.title('2nd derivative Gaussian')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()
```

```python
# Error of the 2nd derivative FD approximation

# Plot of the second derivative FD approximation error
#plt.plot(time, nder2-ader2,label="FD operator error", lw=2, color="violet")
#plt.plot(time, ader2,label="Analytical 2nd derivative", lw=2, color="red")
plt.title('2nd derivative errors')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()
```

## We learned:

* Approximation of numerical first derivative of a time dependent function by forward, backward and central FD operators
* Comparison with analytical solution to estimate the numerical errors
* Approximation of 2nd derivative of a time dependent function using Taylor series expansion


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""

# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
# https://colab.research.google.com/github/kassbohm/tm-snippets/blob/master/ipynb/TM_2/4_BB/2_BL/2.4.2.G-FEM_cc.ipynb

pprint("\nSolution 1: 2 Elements and using Symmetry:")

a, q, EI = var("l, q, EI")
# length of element:
l = a/4
l2 = l*l
l3 = l*l*l

K = EI/l3
K *= Matrix(
[
[  4*l2 ,  -6*l ,  2*l2 ,   6*l ,  0    ,   0    ],
[ -6*l  ,  12   , -6*l  , -12   ,  0    ,   0    ],
[  2*l2 ,  -6*l ,  8*l2 ,   0   ,  2*l2 ,   6*l  ],
[  6*l  , -12   ,  0    ,  24   , -6*l  , -12    ],
[  0    ,   0   ,  2*l2 ,  -6*l ,  4*l2 ,   6*l  ],
[  0    ,   0   ,  6*l  , -12   ,  6*l  ,  12    ],
]
)

w1,p2,w2,p3 = var("w1,p2,w2,p3")
M1,F3 = var("M1,F3")
u = Matrix([0,w1,p2,w2,p3,0])
f = Matrix([M1,0,0,0,0,F3]) + q * Matrix([-l2/12, l/2, 0, l,l2/12, l/2 ])

unknowns = [w1,p2,w2,p3,M1,F3]

eq = Eq(K*u , f)
sol = solve(eq, unknowns)
w1,p2,w2,p3 = sol[w1],sol[p2],sol[w2],sol[p3]

l, B = var("l, B")
sub_list = [
    (a, 4*l),
    (EI, B),
    ]

pprint("\n(w1, p2, w2, p3) / (l³ q / EI):")
for x in [w1,p2,w2,p3]:
    tmp = x.subs(sub_list)
    tmp /= l**3*q/B
    print(tmp)

pprint("\n(w1, ψ2, w2, ψ3) / ( q a³ / (EI) ):")
for x in [w1,p2,w2,p3]:
    tmp = x / (q*a**3 / EI)
    pprint(tmp)

pprint("\nSolution 2: 2 Elements + Symmetry, Stiffness matrix as in Klein:")

l = var("l")
l2 = l*l
l3 = l*l*l
# Klein:
sub_list_Klein=[
    (a, 2 *m),
    (q, 1 *Newton/m),
    (EI, 1 *Newton*m**2),
    ]

# Only partial matrix to find deformations only
K = Matrix(
    [
    [    12   ,  -6*l  ,  -12 ,  0      ],
    [  -6*l   ,  8*l2  ,    0 ,  2*l2   ],
    [   -12   ,  0     ,   24 ,  -6*l   ],
    [   0     ,  2*l2  ,  -6*l ,  4*l2  ],
    ])
K *= EI/l3

w1, p2, w2, p3 = var("w1, p2, w2, p3")
u = Matrix([w1, p2, w2, p3])
f = q * Matrix([l/2, 0, l, l2/12])

eq = Eq(K*u , f)
sol = solve(eq, [w1,p2,w2,p3])
w1, p2, w2, p3 = sol[w1], sol[p2], sol[w2], sol[p3]
pprint("\n(w1, ψ2, w2, ψ3) / ( q a³ / (EI) ):")
fac = a**3*q/EI

tmp = w1.subs(l, a/4)
pprint(tmp/fac)
w1 = tmp.subs(sub_list_Klein)

tmp = p2.subs(l, a/4)
pprint(tmp/fac)
p2 = tmp.subs(sub_list_Klein)

tmp = w2.subs(l, a/4)
pprint(tmp/fac)
w2 = tmp.subs(sub_list_Klein)

tmp = p3.subs(l, a/4)
pprint(tmp/fac)
p3 = tmp.subs(sub_list_Klein)

pprint("\n(w1 / m, ψ2 / rad, w2 / m, ψ3 / rad):")
w1 /= m
w1 = iso_round(w1, 0.001)
pprint(w1)

p2 = iso_round(p2, 0.001)
pprint(p2)

w2 /= m
w2 = iso_round(w2, 0.001)
pprint(w2)

p3 = iso_round(p3, 0.001)
pprint(p3)

pprint("\nSection loads:")
x, xi, l, B, q = var("x, xi, l, B, q")

N1 = -xi**3 + 2*xi**2 - xi
N2 = 2*xi**3 - 3*xi**2 + 1
N3 = -xi**3 + xi**2
N4 = -2*xi**3 + 3*xi**2
N5 = xi**4/24 - xi**3/12 + xi**2/24

# pprint("\ntmp:")
# tmp = N1.subs(xi,half)
# pprint(tmp)
# tmp = N3.subs(xi,half)
# pprint(tmp)
# exit()

N = Matrix([l*N1, N2, l*N3, N4, l**4/B * N5])

dNx = diff(N, xi) / l
d2Nx = diff(dNx, xi) / l
A = - B * d2Nx

fac = l**3*q/B
w1 = fac * 10*l/3
p2 = fac * S(11)/6
w2 = fac * 19*l/8
p3 = fac * S(8)/3

pprint("\n- B w'':")
u1 = Matrix([0, w1, p2, w2, q])
u2 = Matrix([p2, w2, p3, 0, q])
tmp = A.T*u1
tmp = tmp[0]
tmp = tmp.simplify()
pprint(tmp)
a = var("a")
tmp = tmp.subs(l, a/4)
pprint(tmp)

pprint("\nSolution 3: 1 element only, disregarding symmetry:")
# Using 1 element only:
p1, p2 = var("ψ₁, ψ₂")

fac = - q*l**3/B
eq1 = Eq(4*p1 + 2*p2, fac*S(1)/12)
eq2 = Eq(2*p1 + 4*p2, fac*S(-1)/12)

sol = solve([eq1, eq2], [p1, p2])
pprint(sol)

pprint("\nInterpolated displacement at x = 1/2 l:")
p1, p2 = sol[p1], sol[p2]
u = Matrix([p1, 0, p2, 0, q],)
tmp = N.dot(u)
tmp = tmp.subs(xi, S(1)/2)
pprint(tmp)

# Solution 1: 2 Elements + Symmetry:
#
# (w1, p2, w2, p3) / (l³ q / EI):
# 10*l/3
# 11/6
# 19*l/8
# 8/3
#
# (w1, ψ2, w2, ψ3) / ( q a³ / (EI) ):
# 5⋅l
# ───
# 384
#  11
# ───
# 384
# 19⋅l
# ────
# 2048
# 1/24
#
# Solution 2: 2 Elements + Symmetry, Stiffness matrix as in Klein:
#
# (w1, ψ2, w2, ψ3) / ( q a³ / (EI) ):
# 5⋅l
# ───
# 384
#  11
# ───
# 384
# 19⋅l
# ────
# 2048
# 1/24
#
# (w1 / m, ψ2 / rad, w2 / m, ψ3 / rad):
# 0.208
# 0.229
# 0.148
# 0.333
#
# Section loads:
#
# - B w'':
#  2   ⎛   2    ⎞
# l ⋅q⋅⎝- ξ  + 4⎠
# ───────────────
#        2
#  2   ⎛   2    ⎞
# a ⋅q⋅⎝- ξ  + 4⎠
# ───────────────
#        32
#
# Solution 3: 1 element only, disregarding symmetry:
# ⎧      3          3  ⎫
# ⎪    -l ⋅q       l ⋅q⎪
# ⎨ψ₁: ──────, ψ₂: ────⎬
# ⎪     24⋅B       24⋅B⎪
# ⎩                    ⎭
#
# Interpolated displacement at x = 1/2 l:
#    4
# 5⋅l ⋅q
# ──────
# 384⋅B

```


# 10. Stokes equation

In this tutorial, we consider the *2D* Stokes problem using $H^1(\Omega)$ discretization.

Let $\mathbf{u}$ and $p$ denote respectively the velocity and the pressure.

The Stokes problem writes

\begin{align*}
    - \nabla^2 \mathbf{u} + \nabla p &= 0 \\
    \nabla \cdot \mathbf{u} &= 0
\end{align*}

---
**note:**
The actual version of **GeLaTo** does not allow to mix scalar and vector fem spaces. We will have to use a component-wise version of the Stokes problem.

---

Let's define $\mathbf{u} = \begin{pmatrix} u_1 \\ u_2 \end{pmatrix}$

The Stokes equation writes

\begin{align*}
    - \nabla^2 u_1 + \partial_x p &= 0 \\
    - \nabla^2 u_2 + \partial_y p &= 0 \\
    \partial_x u_1 + \partial_y u_2 &= 0
\end{align*}

Let $v \in \mathcal{V}_h$ be a test function and we consider the following weak formulation

\begin{align*}
    \langle \nabla v, \nabla u_1 \rangle - \langle \partial_x  v, p \rangle &= 0 \\
    \langle \nabla v, \nabla u_2 \rangle - \langle \partial_y  v, p \rangle &= 0 \\
    \langle v, \partial_x u_1 + \partial_y u_2 \rangle &= 0
\end{align*}

```python
# needed imports
from numpy import linspace, meshgrid, pi, zeros, asarray
from scipy.linalg import eig

import matplotlib.pyplot as plt
%matplotlib inline

from IPython.display import Math

from sympy import symbols, Symbol, Lambda, IndexedBase, Tuple
from sympy import latex

from gelato.glt import glt_symbol
from gelato.printing.latex import glt_latex
from gelato.calculus   import (Dot, Cross, Grad, Curl, Rot, Div, dx, dy)
from gelato.calculus   import Constant
from gelato.fem.assembly import assemble_matrix
from gelato.fem.utils    import compile_kernel
from gelato.fem.utils    import compile_symbol

from spl.fem.splines import SplineSpace
from spl.fem.tensor  import TensorFemSpace
from spl.fem.vector  import VectorFemSpace
```

```python
x, y = symbols('x y')

u = Symbol('u')
v = Symbol('v')

epsilon = Constant('epsilon')

Laplace = lambda v,u: Dot(Grad(v), Grad(u))
Mass = lambda v,u: v*u
```

```python
u1,u2,p = symbols('u1 u2 p')
v1,v2,q = symbols('v1 v2 q')

a = Lambda((x,y,v1,v2,q,u1,u2,p),
             Laplace(v1,u1) - dx(v1) * p
           + Laplace(v2,u2) - dy(v2) * p
           + q * (dx(u1) + dy(u2))
           + epsilon * Mass(q,p))
```

```python
# ...  create a finite element space
p1  = 3 ; p2  = 3
ne1 = 16 ; ne2 = 16

grid_1 = linspace(0., 1., ne1+1)
grid_2 = linspace(0., 1., ne2+1)

V1 = SplineSpace(p1, grid=grid_1)
V2 = SplineSpace(p2, grid=grid_2)

V = TensorFemSpace(V1, V2)
# ...

# ... create a vector fem space
V = VectorFemSpace(V, V, V)
# ...
```

```python
symbol = glt_symbol(a, space=V)
```

```python
Math(latex(symbol))
```

$$\left[\begin{matrix}m_{1} s_{2} + m_{2} s_{1} & 0 & a_{1} m_{2}\\0 & m_{1} s_{2} + m_{2} s_{1} & a_{2} m_{1}\\a_{1} m_{2} & a_{2} m_{1} & \epsilon m_{1} m_{2}\end{matrix}\right]$$

```python
eigen = symbol.eigenvals()
eigen = list(eigen.keys())
```

```python
Math(latex(eigen))
```

$$\left [ \frac{m_{1} m_{2}}{2} \epsilon + \frac{m_{1} s_{2}}{2} + \frac{m_{2} s_{1}}{2} - \frac{1}{2} \sqrt{\epsilon^{2} m_{1}^{2} m_{2}^{2} - 2 \epsilon m_{1}^{2} m_{2} s_{2} - 2 \epsilon m_{1} m_{2}^{2} s_{1} + 4 a_{1}^{2} m_{2}^{2} + 4 a_{2}^{2} m_{1}^{2} + m_{1}^{2} s_{2}^{2} + 2 m_{1} m_{2} s_{1} s_{2} + m_{2}^{2} s_{1}^{2}}, \quad \frac{m_{1} m_{2}}{2} \epsilon + \frac{m_{1} s_{2}}{2} + \frac{m_{2} s_{1}}{2} + \frac{1}{2} \sqrt{\epsilon^{2} m_{1}^{2} m_{2}^{2} - 2 \epsilon m_{1}^{2} m_{2} s_{2} - 2 \epsilon m_{1} m_{2}^{2} s_{1} + 4 a_{1}^{2} m_{2}^{2} + 4 a_{2}^{2} m_{1}^{2} + m_{1}^{2} s_{2}^{2} + 2 m_{1} m_{2} s_{1} s_{2} + m_{2}^{2} s_{1}^{2}}, \quad m_{1} s_{2} + m_{2} s_{1}\right ]$$

```python
# TODO not working yet
d_types = {'epsilon': 'double'}
kernel_f90 = compile_kernel('kernel_2d', a, V,
                            d_args=d_types,
                            backend='fortran')
```

    pyccel: [semantic]
     |error: [43,29-48]| Incompatible types in assignment (|v_01| double <-> int)
     |error: [45,29-48]| Incompatible types in assignment (|v_10| double <-> int)

    pyccel: [codegen]
     |error: [43,29-48]| Incompatible types in assignment (|v_01| double <-> int)
     |error: [45,29-48]| Incompatible types in assignment (|v_10| double <-> int)

```python
M = assemble_matrix(V, kernel_f90,
                    args={'epsilon': 1.e-3})
```

```python
from utils import coo_from_blocks
```

```python
n_block_rows = 3 ; n_block_cols = 3
matrices = []
for ir in range(0, n_block_rows):
    line = []
    for ic in range(0, n_block_cols):
        line.append(M[ir][ic].tocoo())
    matrices.append(line)
_M = M
M = coo_from_blocks(matrices,
                    n_block_rows=n_block_rows,
                    n_block_cols=n_block_cols)
```

```python
# import eigenvalue solver
from scipy.linalg import eig

M = M.todense()
w, v = eig(M)
wr = w.real
wr.sort()
```

```python
# TODO not working yet => complex numbers
#symbol_f90 = compile_symbol('symbol_2d', a, V, backend='fortran')
```

```python
# TODO remove later
from gelato.glt import glt_lambdify
symbol = glt_symbol(a, space=V, evaluate=True)

symbol = symbol.subs({epsilon: 1.e-3})
f = glt_lambdify(symbol)
```

```python
n1 = V1.nbasis
n2 = V2.nbasis
t1 = linspace(-pi,pi, n1)
t2 = linspace(-pi,pi, n2)
x1 = linspace(0.,1., n1)
x2 = linspace(0.,1., n2)
```

```python
from numpy import imag
tol = 1.e-13

W = []
for i1 in range(0,n1):
    _x1 = x1[i1]
    _t1 = t1[i1]
    for i2 in range(0,n2):
        _x2 = x2[i2]
        _t2 = t2[i2]
        e = f(_x1,_x2,_t1,_t2)
        w,v = eig(e)
        assert([abs(imag(i)) < tol for i in w])
        W += list(w.real)
t = asarray(W)
t.sort()
```

```python
from numpy import where
xmins = where(t < 1.e-4)
print('> len(t) = ', len(t))
print('> len(xmins) = ', len(xmins[0]))
```

    > len(t) =  1083
    > len(xmins) =  131

```python
plt.plot(t, "+b", label="glt symbol")
plt.plot(wr, "xr", label="eigenvalues")
plt.legend(loc=2);
```

```python

```

```python

```

```python

```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```