# Tutorial 2: Differential Equations
**Week 0, Day 4: Calculus**

**By Neuromatch Academy**

__Content creators:__ John S Butler, Arvind Kumar

__Content reviewers:__

__Production editors:__

---
# Tutorial Objectives

A great deal of neuroscience can be modelled using differential equations, from gating channels to single neurons to a network of neurons to blood flow, to behaviour. A simple way to think about differential equations is they are equations that describe how something changes.

The most famous of these in neuroscience is the Nobel Prize winning Hodgkin Huxley equation, which describes a neuron by modelling the gating of each axon. But we will not start there; we will start a few steps back.

Differential Equations are mathematical equations that describe how something like population or a neuron changes over time. The reason why differential equations are so useful is they can generalise a process such that one equation can be used to describe many different outcomes.
The general form of a first order differential equation is:

\begin{align*}
\frac{d}{dt}y(t)&=f(t,y(t))\\
\end{align*}

which can be read as "the change in a process $y$ over time $t$ is a function $f$ of time $t$ and itself $y$". This might initially seem like a paradox as you are using a process $y$ you want to know about to describe itself, a bit like the MC Escher drawing of two hands painting [each other](https://en.wikipedia.org/wiki/Drawing_Hands). But that is the beauty of mathematics - this can be solved some of time, and when it cannot be solved exactly we can use numerical methods to estimate the answer (see next tutorial).

In this tutorial, we will see how __differential equations are motivated by observations of physical responses.__ We will break down the population differential equation, then the integrate and fire model, which leads nicely into raster plots and frequency-current curves to rate models.

**Steps:**
- Get an intuitive understanding of a linear population differential equation (humans, not neurons)
- Visualize the relationship between the change in population and the population
- Breakdown the Leaky Integrate and Fire (LIF) differential equation
- Code the exact solution of an LIF for a constant input
- Visualize and listen to the response of the LIF for different inputs

```python
#@markdown Tutorial Slides
# you should link the slides for all tutorial videos here (we will store pdfs on osf)

from IPython.display import HTML
HTML('')
```

```python
# Imports

import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import IPython.display as ipd
from matplotlib import gridspec

import ipywidgets as widgets  # interactive display
%config InlineBackend.figure_format = 'retina'

# use NMA plot style
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
my_layout = widgets.Layout()
```

```python
# @title Plotting Functions

def plot_dPdt(alpha=.3):
  """ Plots change in population over time
    Args:
      alpha: Birth Rate
    Returns:
      A figure two panel figure
      left panel: change in population as a function of population
      right panel: membrane potential as a function of time
  """

  with plt.xkcd():
    time=np.arange(0, 10 ,0.01)
    fig = plt.figure(figsize=(12,4))
    gs = gridspec.GridSpec(1, 2)

    ## dpdt as a fucntion of p
    plt.subplot(gs[0])
    plt.plot(np.exp(alpha*time), alpha*np.exp(alpha*time))
    plt.xlabel(r'Population $p(t)$ (millions)')
    plt.ylabel(r'$\frac{d}{dt}p(t)=\alpha p(t)$')

    ## p exact solution
    plt.subplot(gs[1])
    plt.plot(time, np.exp(alpha*time))
    plt.ylabel(r'Population $p(t)$ (millions)')
    plt.xlabel('time (years)')
    plt.show()

def plot_V_no_input(V_reset=-75):
  """
    Args:
      V_reset: Reset Potential
    Returns:
      A figure two panel figure
      left panel: change in membrane potential as a function of membrane potential
      right panel: membrane potential as a function of time
  """
  E_L=-75
  tau_m=10
  t=np.arange(0,100,0.01)
  V= E_L+(V_reset-E_L)*np.exp(-(t)/tau_m)
  V_range=np.arange(-90,0,1)
  dVdt=-(V_range-E_L)/tau_m

  with plt.xkcd():
    time=np.arange(0, 10, 0.01)
    fig = plt.figure(figsize=(12, 4))
    gs = gridspec.GridSpec(1, 2)

    plt.subplot(gs[0])
    plt.plot(V_range,dVdt)
    plt.hlines(0,min(V_range),max(V_range), colors='black', linestyles='dashed')
    plt.vlines(-75, min(dVdt), max(dVdt), colors='black', linestyles='dashed')
    plt.plot(V_reset,-(V_reset - E_L)/tau_m, 'o', label=r'$V_{reset}$')
    plt.text(-50, 1, 'Positive')
    plt.text(-50, -2, 'Negative')
    plt.text(E_L - 1, max(dVdt), r'$E_L$')
    plt.legend()
    plt.xlabel('Membrane Potential V (mV)')
    plt.ylabel(r'$\frac{dV}{dt}=\frac{-(V(t)-E_L)}{\tau_m}$')

    plt.subplot(gs[1])
    plt.plot(t,V)
    plt.plot(t[0],V_reset,'o')
    plt.ylabel(r'Membrane Potential $V(t)$ (mV)')
    plt.xlabel('time (ms)')
    plt.ylim([-95, -60])

    plt.show()

## LIF PLOT
def plot_IF(t, V,I,Spike_time):
  """
    Args:
      t  : time
      V  : membrane Voltage
      I  : Input
      Spike_time : Spike_times
    Returns:
      figure with three panels
      top panel: Input as a function of time
      middle panel: membrane potential as a function of time
      bottom panel: Raster plot
  """

  with plt.xkcd():
    fig = plt.figure(figsize=(12, 4))
    gs = gridspec.GridSpec(3, 1,  height_ratios=[1, 4, 1])

    # PLOT OF INPUT
    plt.subplot(gs[0])
    plt.ylabel(r'$I_e(nA)$')
    plt.yticks(rotation=45)
    plt.hlines(I,min(t),max(t),'g')
    plt.ylim((2, 4))
    plt.xlim((-50, 1000))

    # PLOT OF ACTIVITY
    plt.subplot(gs[1])
    plt.plot(t,V)
    plt.xlim((-50, 1000))
    plt.ylabel(r'$V(t)$(mV)')

    # PLOT OF SPIKES
    plt.subplot(gs[2])
    plt.ylabel(r'Spike')
    plt.yticks([])
    plt.scatter(Spike_time, 1 * np.ones(len(Spike_time)), color="grey", marker=".")
    plt.xlim((-50, 1000))
    plt.xlabel('time(ms)')
    plt.show()

##    Plotting the differential Equation
def plot_dVdt(I=0):
  """
    Args:
      I  : Input Current
    Returns:
      figure of change in membrane potential as a function of membrane potential
  """

  with plt.xkcd():
    E_L = -75
    tau_m = 10
    V = np.arange(-85, 0, 1)
    g_L = 10.
    fig = plt.figure(figsize=(6, 4))

    plt.plot(V,(-(V-E_L) + I*10) / tau_m)
    plt.hlines(0, min(V), max(V), colors='black', linestyles='dashed')
    plt.xlabel('V (mV)')
    plt.ylabel(r'$\frac{dV}{dt}$')
    plt.show()
```

```python
# @title Helper Functions

## EXACT SOLUTION OF LIF
def Exact_Integrate_and_Fire(I,t):
  """
    Args:
      I  : Input Current
      t : time
    Returns:
      Spike : Spike Count
      Spike_time : Spike time
      V_exact : Exact membrane potential
  """

  Spike = 0
  tau_m = 10
  R = 10
  t_isi = 0
  V_reset = E_L = -75
  V_exact = V_reset * np.ones(len(t))
  V_th = -50
  Spike_time = []

  for i in range(0, len(t)):

    V_exact[i] = E_L + R*I + (V_reset - E_L - R*I) * np.exp(-(t[i]-t_isi)/tau_m)

    # Threshold Reset
    if V_exact[i] > V_th:
        V_exact[i-1] = 0
        V_exact[i] = V_reset
        t_isi = t[i]
        Spike = Spike+1
        Spike_time = np.append(Spike_time, t[i])

  return Spike, Spike_time, V_exact
```

---
# Section 1: Population Differential Equation

```python
#@title Video 1: Differential Equations Introduction
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="nWKgnouNzGY", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

To get an intuitive feel of a differential equations, we will start with a population differential equation, which models the change in population [1], that is human population not neurons, we will get to neurons later. Mathematically it is written like:
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= \alpha p(t),\\
\end{align*}

where $p(t)$ is the population of the world and $\alpha$ is a parameter representing birth rate.

Another way of thinking about the models is that the equation
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= \alpha p(t),\\
\text{can be written as:}\\
\text{"Change in Population"} &= \text{ "Birth rate times Current population."}
\end{align*}

The equation is saying something reasonable maybe not the perfect model but a good start.

```python
#@title Video 2: The Population Equation
video = YouTubeVideo(id="yaQdEayD9Nw", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

## Coding Exercise 1: Plot the Change in Population

Let's investigate what the rate of change of population looks like as a function of the current population.

Write Python code to calculate the change in population `dpdt` with an `alpha=0.3` as a function of population `p`.  Uncomment the plot line and interpret the plot.

```python
p = np.arange(0, 100, 0.1)

########################################################################
## TODO for students: calculate the dp/dt
## Complete line of code
raise NotImplementedError("Calculate the dp/dt")
########################################################################

dpdt = ...

## Uncomment to plot answer
fig = plt.figure(figsize=(6, 4))
#plt.plot(p, dpdt)
plt.xlabel(r'Population $p(t)$ (millions)')
plt.ylabel(r'$\frac{d}{dt}p(t)=\alpha p(t)$')
plt.show()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_9953ed40.py)

*Example output:*

## Think! 1: Linear Differential Equations and the Population Equation

1. Why is the population differential equation is known as a linear differential equation?

2. How does population size affect the rate of change of the population?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_d65c50d6.py)

## Section 1.1: Exact Solution of the Population Equation

### Section 1.1.1: Initial Condition
The linear population differential equation is known as an initial value differential equation because we need an initial population value to solve it, so here we will set our initial population to:

\begin{align*}
&p(0)=1.\\
\end{align*}

Different initial conditions will lead to different answers, but they will not change the differential equation. This is one of the strengths of a differential equation.

###  Section 1.1.2: Exact Solution
To calculate the exact solution of a differential equation, we must integrate both sides and use some mathematical tricks - the harder the equation the more obscure the trick.

Given the linear population equation
\begin{align*}
\frac{d}{dt}\,p(t) &= \alpha p(t),\\\\
p(0)=1,\\
\end{align*}
has the exact solution:
\begin{align*}
p(t)&=e^{\alpha t}.\\
\end{align*}

The exact solution written in words is:

\begin{align*}
\text{"Population"}&=\text{"grows/declines exponentially as a function time and birth rate"}.\\
\end{align*}

Most differential equations do not have a known exact solution, so in the next tutorial on numerical methods we will show how the solution can be estimated.

A small aside: a good deal of progress in mathematics was due to mathematicians writing taunting letters to each other saying they had a trick that could solve something better than everyone else. So do not worry too much about the tricks.

####  Coding Exercise 1.1.2: Exact Solution of the Population Equation
Let's consider the population differential equation with a birth rate $\alpha=0.3$:

\begin{align*}
\frac{d}{dt}\,p(t) = 0.3 p(t),\\
\text{with the initial condition}\\
p(0)=1.\\
\end{align*}

It has an exact solution
\begin{align*}
\\
p(t)=e^{0.3 t}.
\end{align*}

Code and plot the exact solution of the population differential equation $p(t)$ for $\alpha=0.3$ for the time period $t=0$ to $t=10$.

Use the function `np.exp()` for the exponential.

```python
t = np.arange(0, 10, 0.1) # Time from 0 to 10 years in 0.1 steps

########################################################################
## TODO for students: calculate the exact solution of p
# Fill out function and remove
raise NotImplementedError("Calculate the exact solution of p")
########################################################################

p = ...

## Uncomment to plot answer
fig = plt.figure(figsize=(6, 4))
#plt.plot(t, p)
plt.ylabel('Population (millions)')
plt.xlabel('time (years)')
plt.show()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_359be293.py)

*Example output:*

## Section 1.2: Parameters (Birth Rate $\alpha$)
One of the goals when design a differential equation is to make it generalisable. For example, so that the differential equation will give reasonable solutions for different countries with different birth rates $\alpha$.

##  Interactive Demo 1.2:  Interactive Parameter Change
Play with the widget to see the relationship between $\alpha$ and the population differential equation as a function of population (left-hand side), and the population solution as a function of time(right-hand side). Pay close attention to the transision point from positive to negative.

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
    alpha=widgets.FloatSlider(.3, min=-1., max=1., step=.1, layout=my_layout)
)
def Pop_widget(alpha):
  plot_dPdt(alpha=alpha)
  plt.show()
```

### Think! 1.2: How do changing parameters of the Population Equation affect the outcome?

1. What happens when $\alpha < 0$?
2. What happens when $\alpha > 0$?
3. What happens when $\alpha = 0$?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_93be2531.py)

## Secton 1.3: Population Differential Equation Summary

The population differential equation is an over-simplification and has some very obvious limitations:
1. Population growth is not exponential as there are limited number of resources so the population will level out at some point.
2. It does not include any external factors on the populations like weather, predators and preys.

These kind of limitations can be addressed by extending the model.

While it might not seem that the population equation has direct relevance to neuroscience a similar equation is used to describe the accumulation of evidence for decision making known as the Drift Diffusion Model which you will see in more detail in the Linear System day in Neuromatch (W2D2).

Another differential equation that is similar to the population equation is the Leaky Integrate and Fire Model which we will explore below.

---
# Section 2: The Leaky Integrate and Fire Model

```python
#@title Video 3: Leaky Integrate and Fire Model
video = YouTubeVideo(id="htYJ84aVK3s", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

The Leaky Integrate and Fire Model is a linear differential equation that describes the membrane potential ($V$) of a single neuron which was proposed by Louis Édouard Lapicque in 1907 [2].

The subthreshold membrane potential dynamics of a LIF neuron is described by
\begin{align}
\tau_m\frac{dV}{dt} = -(V-E_L) + R_mI\,
\end{align}

where $\tau_m$ is the time constant, $V$ is the membrane potential,  $E_L$ is the resting potential, $R_m$ is membrance resistance, and $I$ is the external input current.

In the next few sections, we will break down the equation and then build it back up to get an intuitive feel of the different facets of the differential equation.

## Section 2.1: LIF without Input

First, we will simplify the equation by removing the input, which gives the equation

\begin{align}
\tau_m\frac{dV}{dt} &= -V+E_L,\\
\end{align}

which can be written in words as:

\begin{align}
\begin{matrix}\text{"Time constant multiplied by the} \\ \text{change in membrane potential"}\end{matrix}&=\begin{matrix}\text{"Minus Current} \\ \text{membrane potential"} \end{matrix}+
\begin{matrix}\text{"resting potential"}\end{matrix}.\\
\end{align}

The equation can be re-arranged to look even more like the population equation:

\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m}.\\
\end{align}

### Coding Exercise 2.1: LIF without Input
Code the right hand side `dVdt` of the differential equation,
\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m},\\
\end{align}
with the parameters set as:
* `E_L = -75`
* `V_reset = -50`
* `tau_m = 10`

and uncommment the code to plot for 'dVdt' as a function of $V$ between -90 and 0.

Take note of where the curve of `dVdt` cuts the dashed line indicating `dVdt = 0`.

```python
# Parameter definition
E_L = -75
tau_m = 10

# Range of Values of V
V = np.arange(-90, 0, 1)

########################################################################
## TODO for students: Complete line of code for the dVdt
# Fill out function and remove
raise NotImplementedError("Complete line of code for the dVdt")
########################################################################

dVdt = ...

# Uncomment code below to plot
fig = plt.figure(figsize=(6, 4))
# plt.plot(V, dVdt)
plt.vlines(-75, min(dVdt), max(dVdt), colors='black', linestyles='dashed')
plt.text(E_L, max(dVdt) + 1, r'$E_L$')

plt.hlines(0, min(V), max(V), colors='black', linestyles='dashed')
plt.text(-50, 1, 'Positive')
plt.text(-50, -2, 'Negative')
plt.xlabel(r'$V(t)$ (mV)')
plt.ylabel(r'$\frac{dV}{dt}=\frac{-(V-E_L)}{\tau_m}$')
plt.ylim(-8, 2)
plt.show()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_802f7e01.py)

*Example output:*

### Think! 2.1: Effect on Membrane Potential $V$ on the LIF Model

1. What is the effect on $\frac{dV}{dt}$ when $V>-75$ mV?
2. What is the effect on $\frac{dV}{dt}$ when $V<-75$ mV
3. What is the effect on $\frac{dV}{dt}$ when $V=-75$ mV?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_fe6aea6f.py)

### Section 2.1.1: Exact Solution of the LIF model without Input

Similar to the population equation, we need an initial membrane potential at time $0$ to solve the LIF model.

With this equation
\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m}\,\\
V(0)&=V_{reset},
\end{align}
where is $V_{reset}$ is called the reset potential.

The LIF model has the exact solution:
\begin{align*}
V(t)=&\ E_L+(V_{reset}-E_L)e^{\frac{-t}{\tau_m}}\\
\text{ which can be written as: }\\
\begin{matrix}\text{"Current membrane} \\ \text{potential}"\end{matrix}=&\text{"Resting potential"}+\begin{matrix}\text{"Reset potential minus resting potential} \\ \text{times exponential with rate one over time constant."}\end{matrix}\\
\end{align*}

#### Interactive Demo 2.1.1: Initial Condition $V_{reset}$
This exercise is to get an intuitive feel of how the different initial conditions $V_{reset}$ impact the differential equation of the LIF and the exact soluation for the equation:

\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m}\,\\
\end{align}
with the parameters set as:
* `E_L = -75,`
* `tau_m = 10.`

The panel on the left-hand side plot the change in membrane potential $\frac{dV}{dt}$ is a function of membrane potential $V$ and right-hand side panel plots the exact solution $V$ as a function of time $t,$ the green dot in both panels is the reset potential $V_{reset}$.

Pay close attention to when $V_{reset}=E_L=-75$mV.

```python
#@markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
  V_reset=widgets.FloatSlider(-77., min=-91., max=-61., step=2,
                              layout=my_layout)
)

def V_reset_widget(V_reset):
  plot_V_no_input(V_reset)
```

#### Think! 2.1.1: Effect of $V_{reset}$ on the solution

1. How does the solution look with initial values of $V_{reset} < -75$?
2. How does the solution look with initial values of $V_{reset} > -75$?
3. How does the solution look with initial values of $V_{reset} = -75$?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_b51f53ba.py)

## Section 2.2: LIF with Input
We will re-introduce the input $I$ and membrane resistance $R_m$ giving the original equation:

\begin{align}
\tau_m\frac{dV}{dt} = -(V-E_L) + \color{blue}{R_mI}\,
\end{align}

the input can be other neurons or sensory information.

### Interactive Demo 2.2: The Impact of Input
The interactive plot below mainpulates $I$ in the differential equation.
With increasing input what would you expect to happen to the solution.

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
  I=widgets.FloatSlider(3., min=0., max=20., step=2,
                                layout=my_layout)
)

def Pop_widget(I):
  plot_dVdt(I=I)
  plt.show()
```

### Think! 2.2: Effect of increasing Input

1. As $I$ increases, less and less of $\frac{dV}{dt}$ is below 0. How would this impact the solution?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_1cf3706d.py)

### Section 2.2.1: LIF Exact Solution

The LIF with a constant input is a linear differential equation has a known exact solution:
\begin{align*}
V(t)=&\ E_L+R_mI+(V_{reset}-E_L-R_mI)e^{\frac{-t}{\tau_m}}\\
\text{which is written as:}\\
\begin{matrix}\text{"Current membrane} \\ \text{potential"}\end{matrix}=&\text{"Resting potential"}+\begin{matrix}\text{"Reset potential minus resting potential} \\ \text{times exponential with rate one over time constant." }\end{matrix}\\
\end{align*}

#### Coding Exercise 2.2.1: LIF Exact Solution
Code the exact solution `V`:
$$
V(t)= E_L+RI+(V_{reset}-E_L-RI)e^{\frac{-t}{\tau_m}},\\
$$
with the parameters set as:
* `V_reset = -75,`
* `E_L = -75,`
* `tau_m = 10,`
* `R_m = 10,`
* `I = 10,`
of the LIF model and uncomment the code to plot the result.

Ask yourself, does the result make biological sense?
If not, what would you change?

```python
dt = 0.5
t_rest = 0

t = np.arange(0, 1000, dt)
tau_m = 10
R_m = 10
V_reset = E_L = -75

I = 10

########################################################################
## TODO for students: Complete line of code for the exact Solution V
# Fill out function and remove
raise NotImplementedError("Complete line of code for the exact Solution V")
########################################################################

V = ...

# Uncomment below to plot the result
fig = plt.figure(figsize=(6, 4))
#plt.plot(t,V)
plt.ylabel('V (mV)')
plt.xlabel('time (ms)')
plt.show()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_2172448c.py)

*Example output:*

## Section 2.3: Maths is one thing, but biology matters
While the mathematics of the exact solution is exact, it is not biologically valid as neurons spike and defintely do not plateau at a very positive value.

To model the firing of a spike, we must have a threshold voltage $V_{th}$ such that if the voltage $V(t)$ goes above it, the neuron spikes
$$V(t)>V_{th}.$$
We must record the time of spike $t_{isi}$  and count the number of spikes
$$t_{isi}=t, $$
$$𝑆𝑝𝑖𝑘𝑒=𝑆𝑝𝑖𝑘𝑒+1.$$
Then reset the membrane voltage $V(t)$
$$V(t_{isi} )=V_{Reset}.$$

To take into account the spike the exact solution becomes:
\begin{align*}
V(t)=&\ E_L+R_mI+(V_{reset}-E_L-R_mI)e^{\frac{-(t-t_{isi})}{\tau_m}},&\qquad V(t)<V_{th} \\
V(t)=&V_{reset},&\qquad V(t)>V_{th}\\
Spike=&Spike+1,&\\
t_{isi}=&t,\\
\end{align*}
while this does make the neuron spike, it introduces a discontinuity which is not as elegant mathematically as it could be, but it gets results so that is good.

### Interactive Demo 2.3.1: Listen to the Spikes
This exercise show the relationship between firing rate and the Input for exact solution `V` of the LIF:
$$
V(t)=\ E_L+R_mI+(V_{reset}-E_L-R_mI)e^{\frac{-(t-t_{isi})}{\tau_m}},
$$
with the parameters set as:
* `V_reset = -75,`
* `E_L = -75,`
* `tau_m = 10,`
* `R_m = 10.`

Below is a figure with with three panels;
* the top panel is the input, $I,$
* the middle panel is the membrane potential $V(t)$ to illustrative the spike $V(t)$ is set to $0$ and then reset to $-75$ mV,
* the bottom panel is the raster plot with each dot indicating a spike.

As electrophysiologist normally listen to spikes when conducting experiments, so listen to the music of different firing rates by changing the input value $I.$

```python
# @markdown Make sure you execute this cell to be able to hear the neuron
I = 3
t = np.arange(0, 1000, dt)
Spike, Spike_time, V = Exact_Integrate_and_Fire(I, t)

plot_IF(t, V, I, Spike_time)
ipd.Audio(V, rate=len(V))
```

### Interactive Demo 2.3.2: Input on Spikes
Manipulate the input into the LIF to see the impact of input on the firing pattern (rate).

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
  I=widgets.FloatSlider(3, min=2.0, max=4., step=.1,
                              layout=my_layout)
)

def Pop_widget(I):
  Spike, Spike_time, V = Exact_Integrate_and_Fire(I, t)
  plot_IF(t, V, I, Spike_time)
```

### Think! 2.3: Is the model biologically valid?

1. What is the effect of $I$ on spiking?
2. Is this biologically valid?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D4_Calculus/solutions/W0D4_Tutorial2_Solution_5101cefd.py)

## Section 2.4 Firing Rate as a function of Input

The firing frequency of a neuron plotted as a fucntion of current is called an input-output curve (F–I curve). It is also known as a transfer function, which you came accross in the previous tutorial. This function is one of the starting points for the rate model, which extends from modelling single neurons to collections of neurons.

By fitting this to a function, we can start to generalise the firing pattern of many neurons, which can be used to build rate models which is discussed later in Neuromatch.

```python
# @markdown *Execture this cell to visualize the FI curve*
I_range = np.arange(2.0, 4.0, 0.1)
Spike_rate = np.ones(len(I_range))

for i, I in enumerate(I_range):
  Spike_rate[i], _, _ = Exact_Integrate_and_Fire(I, t)

with plt.xkcd():
  fig = plt.figure(figsize=(6, 4))
  plt.plot(I_range,Spike_rate)
  plt.xlabel('Input Current (nA)')
  plt.ylabel('Spikes per Second (Hz)')
  plt.show()
```

---
## Summary of LIF model

The LIF model is a very nice differential equation to start with in computational neuroscience as it has been used as a building block for many papers that simulate neuronal response.

__Strenghts of LIF model:__
+ Has an exact solution;
+ Easy to interpret;
+ Great to build network of neurons.

__Weaknesses of the LIF model:__
- Spiking is a discontinuity;
- Abstraction from biology;
- Cannot generate different spiking patterns.

---
# Take Home Message

```python
#@title Video 4: Take Home Message and Outro
video = YouTubeVideo(id="TcKWdBD3jcA", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

In this tutorial, we have have seen two differential equations, the population differential equations and the leaky integrate and fire model.

We learned about:
* The motivation for differential equations.
* An intuitive relationship between the solution and the form of the differential equation.
* How different parameters of the differential equation impact the solution.
* The strengths and limitations of the simple differential equations.

---
# Links to Neuromatch Days

Differential equations turn up in a number of different Neuromatch days:
* The LIF model is discussed in more details in Model Types (Week 1 Day 1) and Real Neurons (Week 2 Day 3).
* Drift Diffusion model which is a differential equation for decision making is discussed in Linear Systems (Week 2 Day 2).
* Systems of differential equations are discussed in Linear Systems (Week 2 Day 2) and Dynamic Networks (Week 2 Day 4).

---
# References
1. Lotka, A. L, (1920)   Analytical note on certain rhythmic  relations inorganic  systems.Proceedings of the National Academy of Sciences,6(7):410–415,1920.

2. Brunel N, van Rossum MC. Lapicque's 1907 paper: from frogs to integrate-and-fire. Biol Cybern. 2007 Dec;97(5-6):337-9. doi: 10.1007/s00422-007-0190-0. Epub 2007 Oct 30. PMID: 17968583.

# Bibliography
1. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational Neuroscience Series.
2. Strogatz, S. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering (studies in nonlinearity), Westview Press; 2 edition (29 July 2014)

## Supplemental Popular Reading List
1. Lindsay, G. (2021). Models of the Mind: How Physics, Engineering and Mathematics Have Shaped Our Understanding of the Brain. Bloomsbury Publishing.
2. Strogatz, S. (2004). Sync: The emerging science of spontaneous order. Penguin UK.

## Popular Podcast
1. Strogatz, S. (Host). (2020-), Joy of X  https://www.quantamagazine.org/tag/the-joy-of-x/ Quanta Magazine


# Lost Secrets of the H-Bridge, Part I: Ripple Current in Inductive Loads #

IPython Notebook code — the [original article](http://www.embeddedrelated.com/showarticle/421.php) is located at
http://www.embeddedrelated.com/showarticle/421.php

© 2013 Jason M Sachs

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

```
import matplotlib.pyplot as plt
import numpy as np
import scipy.integrate

def ramp(t): return t % 1
def sawtooth(t): return 1-abs(2*(t % 1) - 1)

def pwm(t,D,centeralign = False):
  '''generate PWM signals with duty cycle D'''
  return ((sawtooth(t) if centeralign else ramp(t)) <= D) * 1.0

def digitalplotter(t,*signals):
  '''return a plotting function that takes an axis and plots
digital signals (or other signals in the 0-1 range)'''
  def f(ax):
    n = len(signals)
    for (i,sig) in enumerate(signals):
      ofs = (n-1-i)*1.1
      plotargs = []
      for y in sig[1:]:
        if isinstance(y,basestring):
          plotargs += [y]
        else:
          plotargs += [t,y+ofs]
      ax.plot(*plotargs)
    ax.set_yticks((n-1-np.arange(n))*1.1+0.55)
    ax.set_yticklabels([sig[0] for sig in signals])
    ax.set_ylim(-0.1,n*1.1)
  return f
```

```
t = np.arange(0,4,0.0005)
sigA = ('A',pwm(t,0.2,centeralign=True))
sigB = ('B',pwm(t,0.5,centeralign=False))
sigC = ('C',pwm(t,0.9,centeralign=False))

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
digitalplotter(t,
  sigA+(sawtooth(t),'k:'),
  sigB+(ramp(t),'k:'),
  sigC+(ramp(t),'k:'))(ax)
```

```
pwmA = pwm(t,0.5,centeralign=False)
pwmB = pwm(t,0.2,centeralign=False)

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
digitalplotter(t,
  ('pwmA',pwmA),
  ('pwmB',pwmB),
  ('pwmA-pwmB',pwmA-pwmB))(ax)
```

```
def extendrange(ra,rb):
  '''return a tuple (x1,x2) representing the interval from x1 to x2,
given two input ranges of the same form, or None (representing no input).'''
  if ra is None:
    return rb
  elif rb is None:
    return ra
  else:
    return (min(ra[0],rb[0]),max(ra[1],rb[1]))

def createLimits(margin, *args):
    '''add proportional margin to an interval:
createLimits(0.1, (1,3),(2,4),(0,2)) calculates the maximum extent
of the ranges provided, in this case (0,4), and adds another 0.1 (10%)
to the extent symmetrically, thus returning (-0.2, 4.2).'''
    r = None
    for x in args:
        r = extendrange(r, (np.min(x),np.max(x)))
    rmargin = (r[1]-r[0])*margin/2.0
    return (r[0]-rmargin,r[1]+rmargin)

def calcripple(t,y):
    ''' calculate ripple current by integrating the input,
then adjusting it by a linear function to put both endpoints
at the same value. The slope of the linear function is the
mean value of the input; the offset is chosen to make the mean value
of the output ripple current = 0.'''
    T = t[-1]-t[0]
    yint0 = np.append([0],scipy.integrate.cumtrapz(y,t))
    # cumtrapz produces a vector of length N-1
    # so we need to add one element back in at the beginning
    meanval0 = yint0[-1]/T
    yint1 = yint0 - (t-t[0])*meanval0
    meanval1 = scipy.integrate.trapz(yint1,t)/T
    return yint1 - meanval1

def showripple(fig,t,Va,Vb,titlestring):
    '''plot ripple current as well as phase duty cycles and load voltage'''
    axlist = []
    Iab = calcripple(t,Va-Vb)
    margin = 0.1
    ax = fig.add_subplot(3,1,1)
    digitalplotter(t,('Va',Va),('Vb',Vb))(ax)
    ax.set_ylabel('Phase duty cycles')
    axlist.append(ax)

    ax = fig.add_subplot(3,1,2)
    ax.plot(t,Va-Vb)
    ax.set_ylim(createLimits(margin,Va-Vb))
    ax.set_ylabel('Load voltage')
    axlist.append(ax)

    ax = fig.add_subplot(3,1,3)
    ax.plot(t,Iab)
    ax.set_ylim(createLimits(margin,Iab))
    ax.set_ylabel('Ripple current')
    axlist.append(ax)

    fig.suptitle(titlestring, fontsize=16)

    # annotate with peak values
    tlim = [min(t),max(t)]
    tannot0 = tlim[0] + (tlim[1]-tlim[0])*0.5
    tannot1 = tlim[0] + (tlim[1]-tlim[0])*0.6
    for y in [min(Iab),max(Iab)]:
      ax.plot(tlim,[y]*2,'k:')
      # see:
# http://matplotlib.org/api/axes_api.html#matplotlib.axes.Axes.annotate
      ax.annotate('%.5f' % y, xy=(tannot0,y), xytext=(tannot1,y*0.3),
        bbox=dict(boxstyle="round", fc="0.9"),
        arrowprops=dict(arrowstyle="->",
          connectionstyle="arc,angleA=0,armA=20,angleB=%d,armB=15,rad=7"
             % (-90 if y > 0 else 90))
      )
    return axlist

def showpwmripple(fig,t,Da,Db,centeralign=False,titlestring=''):
    return showripple(fig,t,
               pwm(t,Da,centeralign),
               pwm(t,Db,centeralign),
               titlestring='%s-aligned pwm, $D_a$=%.3f, $D_b$=%.3f' %
                 ('Center' if centeralign else 'Edge', Da, Db))
```

```
fig = plt.figure(figsize=(8, 6), dpi=80)
showpwmripple(fig,t,0.4,0.2,centeralign=False);
```

```
fig = plt.figure(figsize=(8, 6), dpi=80)
showpwmripple(fig,t,0.6,0.2,centeralign=True);
```

```
D = np.arange(-1,1,0.005)
plt.plot(D,abs(D)*(1-abs(D)))
plt.xlabel('net duty cycle $D = D_a - D_b$',fontsize=16)
plt.ylabel('ripple current / $I_{R0}$',fontsize=16)
```

```
fig = plt.figure(figsize=(8, 6), dpi=80)
showpwmripple(fig,t,0.6,0.1,centeralign=False);
```

```
tfine = np.arange(0,4,0.0001)
fig = plt.figure(figsize=(8, 6), dpi=80)
showpwmripple(fig,tfine,0.6,0.1,centeralign=False);
```

```
fig = plt.figure(figsize=(8, 6), dpi=80)
Da,Db = 0.6, 0.1
intcodes = 'ABCD'
axlist = showpwmripple(fig,t,Da,Db,centeralign=True)
for (i,tcenter) in enumerate([0.5, 1-(Da+Db)/4, 1, 1+(Da+Db)/4]):
    y = (i/2) * 1.1 + 0.55
    intcode = '[%s]' % intcodes[i]
    axlist[0].annotate(intcode,xy=(tcenter,y),
        horizontalalignment='center',
        verticalalignment='center')
    axlist[2].annotate(intcode,xy=(tcenter,0.05),
        horizontalalignment='center',
        verticalalignment='center')
```

```
def show1period(Da,Db):
  t1period = np.arange(-0.5,1.5,0.001)
  pwmA = pwm(t1period,Da,centeralign=True)
  pwmB = pwm(t1period,Db,centeralign=True)
  fig = plt.figure()
  ax = fig.add_subplot(1,1,1)
  Ir = calcripple(t1period,pwmA-pwmB)
  ax.plot(t1period,Ir)
  ax.set_xlim(-0.1,1.1)
  ax.set_ylim(min(Ir)*1.2,max(Ir)*1.2)
  # now annotate
  (D1,D2,V) = (Db,Da,1) if Da > Db else (Da,Db,-1)
  tpts = np.array([0, D1/2, D2/2, 0.5, 1-(D2/2), 1-(D1/2), 1])
  dtpts = np.append([0],np.diff(tpts))
  vpts = np.array([0, 0, V, 0, 0, V, 0]) - (Da-Db)
  ypts = np.cumsum(vpts*dtpts)
  ax.plot(tpts,ypts,'.',markersize=8)
  for i in range(7):
    ax.annotate('$t_%d$' % i,xy=(tpts[i]+0.01,ypts[i]), fontsize=16)

show1period(0.6,0.1)
show1period(0.9,0.8)
```

```
import sympy
%load_ext sympy.interactive.ipythonprinting
D, D0 = sympy.symbols("D D_0")
Da =  D/2 + D0
Db = -D/2 + D0
# now let's verify that we've defined D and D0 properly:
display('Da-Db=',Da-Db)
display('(Da+Db)/2=',(Da+Db)/2)
```

```
# average voltage at the load = D
I = [0,0,0,0,0,0,0]
I[1] = I[0] + Db/2*(0-D)
I[2] = I[1] + (Da-Db)/2*(1-D)
I[3] = I[2] + (1-Da)/2*(0-D)
I[4] = I[3] + (1-Da)/2*(0-D)
I[5] = I[4] + (Da-Db)/2*(1-D)
I[6] = I[5] + Db/2*(0-D)

# let's use pandas to display as a table
import pandas as pd
itable1 = pd.DataFrame([sympy.simplify(Ik) for Ik in I],
             index=['I[%d]' % d for d in range(7)])
itable1
```

      0

      I[0]
                          0

      I[1]
            D*(D - 2*D_0)/4

      I[2]
       D*(-D - 2*D_0 + 2)/4

      I[3]
                          0

      I[4]
        D*(D + 2*D_0 - 2)/4

      I[5]
           D*(-D + 2*D_0)/4

      I[6]
                          0

```
# average voltage at the load = D;
# load voltage alternates between 0 and -1
I = [0,0,0,0,0,0,0]
I[1] = I[0] + Da/2*(0-D)
I[2] = I[1] + (Db-Da)/2*(-1-D)
I[3] = I[2] + (1-Db)/2*(0-D)
I[4] = I[3] + (1-Db)/2*(0-D)
I[5] = I[4] + (Db-Da)/2*(-1-D)
I[6] = I[5] + Da/2*(0-D)
itable2 = pd.DataFrame([sympy.simplify(Ik) for Ik in I],
                index=['I[%d]' % d for d in range(7)])
itable2
```

      0

      I[0]
                          0

      I[1]
           D*(-D - 2*D_0)/4

      I[2]
        D*(D - 2*D_0 + 2)/4

      I[3]
                          0

      I[4]
       D*(-D + 2*D_0 - 2)/4

      I[5]
            D*(D + 2*D_0)/4

      I[6]
                          0

```
Da_vector = np.arange(0,1,0.005)
Db_vector = np.arange(0,1,0.005)
Da, Db = np.meshgrid(Da_vector, Db_vector)
D = Da-Db
D0 = (Da+Db)/2
Ipk = abs(D)*np.maximum(abs(abs(D)-2*D0),abs(2-abs(D)-2*D0))/4
cs = plt.contour(Da,Db,Ipk,numpy.arange(0,0.125,0.0125),colors='black')
plt.clabel(cs)
plt.xlabel('Da')
plt.ylabel('Db')
plt.title('Peak ripple current $I_{pk}/(V_{dc}T/L)$',fontsize=16)
```

```
cs = plt.contour(D,D0,Ipk,numpy.arange(0,0.125,0.0125),
             colors='black')
plt.clabel(cs)
plt.xlabel('D = Da - Db')
plt.xticks(np.arange(-1,1.25,0.25))
plt.yticks(np.arange(0,1.1,0.1))
plt.ylabel('common-mode duty cycle D0')
plt.title('Peak ripple current $I_{pk}/(V_{dc}T/L)$',
  fontsize=16)
```

```
fig = plt.figure(figsize=(8, 6), dpi=80)
Da,Db = 0.25, 0.75
axlist = showpwmripple(fig,t,Da,Db,centeralign=True)
```

```
fig = plt.figure(figsize=(8, 6), dpi=80)
Da,Db = 0.85, 0.15
axlist = showpwmripple(fig,t,Da,Db,centeralign=True)
```

```
t2 = np.arange(0,1,0.001)
D = 0.7
t0 = (1-D)/4
t1 = (1+D)/4
y1 = abs(D)*abs(1-D)/4
Vload = (pwm(t2,(1.0+D)/2,centeralign=True)
        -pwm(t2,(1.0-D)/2,centeralign=True))
I = calcripple(t2,Vload)
plt.plot(t2,I)
plt.ylim(-0.07,0.07)
plt.plot([t0,t1],[-y1,y1],'.',markersize=8)
plt.annotate('$(t_0,y_0)$',xy=(t0+0.01,-y1),fontsize=16)
plt.annotate('$(t_0+h,y_1)$',xy=(t1+0.01,y1),fontsize=16)
```

```
t,t0,y0,y1,h = sympy.symbols('t t0 y0 y1 h')
m = (y1-y0)/h
y = m*(t-t0)+y0
integral_ysquared = sympy.integrate(y**2,(t,t0,t0+h))
display('integral of y^2 = ',sympy.simplify(integral_ysquared))

```

```
Ipk,D,T = sympy.symbols('I_{pk} D T')
```

```
def f3(y0,y1,h):
    return h*(y0*y0 + y0*y1 + y1*y1)/3
integral_squared_ripple = (f3(-Ipk,Ipk,D*T/2) + f3(Ipk,-Ipk,(1-D)*T/2))
sympy.simplify(integral_squared_ripple/(T/2))
```

```
def piecewiseHarmonic(flist,tlist,k,T=1,t=None):
    ''' calculates piecewise integration of the
kth harmonic of a given function

flist is a list of N functions
tlist is a list of N+1 points in time
piecewiseHarmonic() calculates the sum of the
  integral of flist[i]*exp(2j*pi*k*t/T)
  between t=tlist[i] and tlist[i+1]
  iterating over i = 0:N-1
'''
    S = 0
    t = sympy.symbols('t') if t is None else t
    pi = sympy.pi
    for (i,f) in enumerate(flist):
        dt_range = (t,tlist[i],tlist[i+1])
        S += 2*sympy.integrate(f*sympy.cos(2*pi*k*t/T),dt_range)/T
        S += 2j*sympy.integrate(f*sympy.sin(2*pi*k*t/T),dt_range)/T
    return sympy.simplify(S)

t=sympy.symbols('t')
def cosk(k):
    return sympy.cos(2*sympy.pi*k*t)
def sink(k):
    return sympy.sin(2*sympy.pi*k*t)

display(piecewiseHarmonic([cosk(3)],[0,1],3))
display(piecewiseHarmonic([cosk(3)],[0,1],4))
display(piecewiseHarmonic([sink(3)],[0,1],3))
display(piecewiseHarmonic([sink(3)],[0,1],4))

```

```
def sawtooth_up(D,t):
    return (1-D)*(4*t-1)/4

def sawtooth_down(D,t):
    return (-D)*t

D = 0.7
t_down = numpy.arange(-(1-D)/2,(1-D)/2,0.001)
t_up = numpy.arange((1-D)/2,(1+D)/2,0.001)
plt.plot(t_down/2,sawtooth_down(D,t_down/2),
         t_up/2,sawtooth_up(D,t_up/2))
plt.grid('on')
```

```
D,t = sympy.symbols('D t')
for k in range(6):
    display('harmonic #%d' % k,piecewiseHarmonic(
            [sawtooth_down(D,t/2),
             sawtooth_up(D,t/2)],
            [-(1-D)/2,(1-D)/2,(1+D)/2],k))
```

```
def sawtoothApprox(D,kmax,printHarmonics=False):
    pi = numpy.pi
    def f(t):
        S = 0
        for k in range(1,kmax+1):
            a_k = (-1)**k / 2.0 / k / k
            A_k = sin(k*pi*D)*a_k/pi/pi
            if printHarmonics:
                print 'for k=%d, a_k=%9.6f, A_k=%9.6f' % (k,a_k,A_k)
            S += A_k*sin(2*pi*k*t)
        return S
    return f

t2 = np.arange(0,1,0.001)
D = 0.7
Vload = (pwm(t2,(1.0+D)/2,centeralign=True)
       - pwm(t2,(1.0-D)/2,centeralign=True))
I = calcripple(t2,Vload)
Iapprox = sawtoothApprox(D,6,printHarmonics=True)(2*t2)
# show exact sawtooth, approximation of 1st 6 harmonics,
# and then the approximation error
plt.plot(t2,I,t2,Iapprox,t2,(I-Iapprox),':k')
```

```
show1period(0.6,0.1)
```

```
D,D0,D0dev,T = sympy.symbols('D D_0 D_{0dev} T')
def f3(y0,y1,h):
    return h*(y0*y0 + y0*y1 + y1*y1)/3
I1 = D*(abs(D)-2*D0)/4
I2 = D*(2-abs(D)-2*D0)/4
integral_squared_ripple = (
   f3(I1,I2,abs(D)/2*T)
 + f3(I2,-I2,(1-D0-abs(D)/2)*T)
 + f3(-I2,-I1,abs(D)/2*T)
 + f3(-I1,I1,(D0-abs(D)/2))*T)
mean_squared_ripple = sympy.simplify(integral_squared_ripple/T)
mean_squared_ripple
```

$$\frac{1}{48} D^{2} \left(12 D_{0}^{2} - 12 D_{0} + \lvert{D}\rvert^{2} - 2 \lvert{D}\rvert + 4\right)$$

```
Int1 = sympy.Integer(1)
msr2 = sympy.simplify(mean_squared_ripple.subs(D0, D0dev + Int1/2)).subs(D0dev,D0-Int1/2)
display('mean squared ripple = ',msr2)
```

    mean squared ripple =

$$\frac{1}{48} D^{2} \left(12 \left(D_{0} - \frac{1}{2}\right)^{2} + \lvert{D}\rvert^{2} - 2 \lvert{D}\rvert + 1\right)$$

```
def check_jasons_algebra(D,D0):
  # here's what we know for certain
  I1 = D*(abs(D)-2*D0)/4
  I2 = D*(2-abs(D)-2*D0)/4
  # here's what I got with algebra by hand
  Ipk = abs(D)*(1-abs(D))/4 + abs(D)*abs(D0-0.5)/2
  print("D=%f,D0=%f,I1=%f,I2=%f" % (D,D0,I1,I2))
  print("max(abs(I1),abs(I2))=%f, Jason's calc=%f"
    % (max(abs(I1),abs(I2)), Ipk))

for (D,D0) in (
      (0.5,0.5),
      (0.5,0.4),
      (0.5,0.6),
      (0.1,0.5),
      (0.1,0.4),
      (0.1,0.6),
      (-0.5,0.5),
      (-0.5,0.4),
      (-0.5,0.6)
      ):
  check_jasons_algebra(D,D0)
```

    D=0.500000,D0=0.500000,I1=-0.062500,I2=0.062500
    max(abs(I1),abs(I2))=0.062500, Jason's calc=0.062500
    D=0.500000,D0=0.400000,I1=-0.037500,I2=0.087500
    max(abs(I1),abs(I2))=0.087500, Jason's calc=0.087500
    D=0.500000,D0=0.600000,I1=-0.087500,I2=0.037500
    max(abs(I1),abs(I2))=0.087500, Jason's calc=0.087500
    D=0.100000,D0=0.500000,I1=-0.022500,I2=0.022500
    max(abs(I1),abs(I2))=0.022500, Jason's calc=0.022500
    D=0.100000,D0=0.400000,I1=-0.017500,I2=0.027500
    max(abs(I1),abs(I2))=0.027500, Jason's calc=0.027500
    D=0.100000,D0=0.600000,I1=-0.027500,I2=0.017500
    max(abs(I1),abs(I2))=0.027500, Jason's calc=0.027500
    D=-0.500000,D0=0.500000,I1=0.062500,I2=-0.062500
    max(abs(I1),abs(I2))=0.062500, Jason's calc=0.062500
    D=-0.500000,D0=0.400000,I1=0.037500,I2=-0.087500
    max(abs(I1),abs(I2))=0.087500, Jason's calc=0.087500
    D=-0.500000,D0=0.600000,I1=0.087500,I2=-0.037500
    max(abs(I1),abs(I2))=0.087500, Jason's calc=0.087500


# Stochastic Differential Equations: Lab 2

```python
from IPython.core.display import HTML
css_file = 'https://raw.githubusercontent.com/ngcm/training-public/master/ipython_notebook_styles/ngcmstyle.css'
HTML(url=css_file)
```

<link href='http://fonts.googleapis.com/css?family=Open+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 1000px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1200px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Open Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:900px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Arvo', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Arvo', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Arvo', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Arvo', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Arvo', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>

This background for these exercises is article of D Higham, [*An Algorithmic Introduction to Numerical Simulation of Stochastic Differential Equations*, SIAM Review 43:525-546 (2001)](http://epubs.siam.org/doi/abs/10.1137/S0036144500378302).
Higham provides Matlab codes illustrating the basic ideas at <http://personal.strath.ac.uk/d.j.higham/algfiles.html>, which are also given in the paper.

```python
%matplotlib inline
import numpy
from matplotlib import pyplot
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
rcParams['figure.figsize'] = (12,6)
from scipy.integrate import quad
```

## Further Stochastic integrals

Quick recap: the key feature is the *Ito stochastic integral*

\begin{equation}
  \int_{t_0}^t G(t') \, \text{d}W(t') = \text{mean-square-}\lim_{n\to +\infty} \left\{ \sum_{i=1}^n G(t_{i-1}) (W_{t_i} - W_{t_{i-1}} ) \right\}
\end{equation}

where the key point for the Ito integral is that the first term in the sum is evaluated at the left end of the interval ($t_{i-1}$).

Now we use this to write down the SDE

\begin{equation}
  \text{d}X_t = f(X_t) \, \text{d}t + g(X_t) \, \text{d}W_t
\end{equation}

with formal solution

\begin{equation}
  X_t = X_0 + \int_0^t f(X_s) \, \text{d}s + \int_0^t g(X_s) \, \text{d}W_s.
\end{equation}

Using the Ito stochastic integral formula we get the Euler-Maruyama method

\begin{equation}
 X_{n+1} = X_n + \delta t \, f(X_n) + \sqrt{\delta t} \xi_n \, g(X_n)
\end{equation}

by applying the integral over the region $[t_n, t_{n+1} = t_n + \delta t]$. Here $\delta t$ is the width of the interval and $\xi_n$ is the normal random variable $\xi_n \sim N(0, 1)$.

### Normal chain rule

If

\begin{equation}
  \frac{\text{d}X}{\text{d}t} = f(X_t)
\end{equation}

and we want to find the differential equation satisfied by $h(X(t))$ (or $h(X_t)$), then we write

\begin{align}
  &&\frac{\text{d}}{\text{d}t} h(X_t) &= h \left( X(t) + \text{d}X(t) \right) - h(X(t)) \\
  &&&\simeq h(X(t)) + \text{d}X \, h'(X(t)) + \frac{1}{2} (\text{d}X)^2 \, h''(X(t)) + \dots - h(X(t)) \\
  &&&\simeq f(X) h'(X) \text{d}t + \frac{1}{2} (f(X))^2 h''(X) (\text{d}t)^2 + \dots \\
  \implies && \frac{\text{d} h(X)}{dt} &= f(X) h'(X).
\end{align}

### Stochastic chain rule

Now run through the same steps using the equation

\begin{equation}
  \text{d}X = f(X)\, \text{d}t + g(X) \, \text{d}W.
\end{equation}

We find

\begin{align}
  && \text{d}h &\simeq h'(X(t))\, \text{d}X + \frac{1}{2} h''(X(t)) (\text{d}X)^2 + \dots, \\
  &&&\simeq h'(X) f(X)\, \text{d}t + h'(X) g(X) ', \text{d}W + \frac{1}{2} \left( f(X) \text{d}t^2 + 2 f(x)g(x)\, \text{d}t dW + g^2(x) \text{d}W^2 \right) \\
  \implies && \text{d}h &= \left( f(X) h'(X) + \frac{1}{2} h''(X)g^2(X) \right) \, \text{d}t + h'(X) g(X) \, \text{d}W.
\end{align}

This additional $g^2$ term makes all the difference when deriving numerical methods, where the chain rule is repeatedly used.

### Using this result

Remember that

\begin{equation}
  \int_{t_0}^t W_s \, \text{d}W_s = \frac{1}{2} W^2_t - \frac{1}{2} W^2_{t_0} - \frac{1}{2} (t - t_0).
\end{equation}

From this we need to identify the stochastic differential equation, and also the function $h$, that will give us this result just from the chain rule.

The SDE is

\begin{equation}
  \text{d}X_t = \text{d}W_t, \quad f(X) = 0, \quad g(X) = 1.
\end{equation}

Writing the chain rule down in the form

\begin{equation}
  h(X_t) = h(X_0) + \int_0^t \left( f(X_s) h'(X_s) + \frac{1}{2} h''(X_s) g^2(X_s) \right) \, \text{d}t + \int_0^t h'(X_s) g(X_s) \, \text{d}W_s.
\end{equation}

Matching the final term (the integral over $\text{d}W_s$) we see that we need $h'$ to go like $X$, or

\begin{equation}
  h = X^2, \quad \text{d}X_t = \text{d}W_t, \quad f(X) = 0, \quad g(X) = 1.
\end{equation}

With $X_t = W_t$ we therefore have

\begin{align}
   W_t^2 &= W_0^2 + \int_{t_0}^t \frac{1}{2} 2 \, \text{d}s + \int_{t_0}^t 2 W_s \, \text{d}W_s
   &= W_0^2 + (t - t_0) + \int_{t_0}^t 2 W_s \, \text{d}W_s
\end{align}

as required.

## Milstein's method

Using our chain rule we can construct higher order methods for stochastic differential equations. Milstein's method, applied to the SDE
$$
  \text{d}X = f(X) \, \text{d}t + g(X) \,\text{d}W,
$$
is
$$
  X_{n+1} = X_n + h f_n + g_n \, \text{d}W_{n} + \tfrac{1}{2} g_n g'_n \left( \text{d}W_{n}^2 - h \right).
$$

### Tasks

Implement Milstein's method, applied to the problem in the previous lab:
$$
\begin{equation}
  \text{d}X(t) = \lambda X(t) \, \text{d}t + \mu X(t) \text{d}W(t), \qquad X(0) = X_0.
\end{equation}
$$

Choose any reasonable values of the free parameters $\lambda, \mu, X_0$.

The exact solution to this equation is $X(t) = X(0) \exp \left[ \left( \lambda - \tfrac{1}{2} \mu^2 \right) t + \mu W(t) \right]$. Fix the timetstep and compare your solution to the exact solution.

```python

```

Check the convergence again.

```python

```

Compare the *performance* of the Euler-Maruyama and Milstein method using eg `timeit`. At what point is one method better than the other?

```python

```

## Population problem

Apply the algorithms, convergence and performance tests to the SDE

$$
\begin{equation}
  \text{d}X(t) = r X(t) (K - X(t)) \, \text{d}t + \beta X(t) \,\text{d}W(t), \qquad X(0) = X_0.
\end{equation}
$$

Use the parameters $r = 2, K = 1, \beta = 0.25, X_0 = 0.5$.

```python
r = 2.0
K = 1.0
beta = 0.25
X0 = 0.5
T   = 1.0

```

```python

```

Investigate how the behaviour varies as you change the parameters $r, K, \beta$.

```python

```

FEEG6016 Simulation and Modelling/10-Stochastic-DEs-Lab-2.ipynb


# Overview

Then, the class itself. We kick things off with my short (and hilarous) intro to today, then cover a few more  administrative things.

It might seem boring to hear about admin, **but this is the stuff you need to know to do well in the class**. So I recommend watching and paying attention!

Then, we kick into gear and get started on the actual work. This week is about getting familiar with networks, and we'll focus on four main elements

* A short talk on the basic mathematical description of networks
* The `NetworkX` library
* Matplotlib, binning, and plotting degree distributions
* Random networks

(The work-load is still a little bit light, since we want to allow those who are not super-strong at programming a bit of time to catch up. If you're bored and in need of extra work, let me know and I'll point you in the right direction.)

## The Informal Intro

Ok. In the video below I discuss

 * How it's going with COVID-19
 * Some background and context on what we'll be learning about in class today
 * Where we are with the admin stuff (how to take the class)
 * **An overview of the rest of the semester**. This last part is especially important for you to know about.

```python
from IPython.display import YouTubeVideo
YouTubeVideo("XkDtCQOx8Xk",width=800, height=450)
```

# Part 0: The Admin Stuff
### (GitHub Page Overview Edition)

Last week, I gave you a lot of practical information about how to follow the class.

In addition to a lot of (boring) information about COVID and my unusual precautions, I discussed the principles of how I teach.
How each lecture is structured to maximize your learning according to state-of-the-art principles.

This week, the admin video talks about the structure of the Course Page, which actually contains a lot of useful information.

You'll notice that I've made it for the class that runs in the fall (Social Graphs and Interactions), but the web-page struture is similar, so everyhting should still work.

```python
YouTubeVideo("42KjmxgYYNI",width=800, height=450)
```

Next week, I'll talk about the final elements of admin we have to cover for now

* Forming groups
* Handing in assignments
* Peer Evaluations
* The structure of the class

# Part 1: Basic mathematical description of networks

This week, let's start with some lecturing. I love networks, so I'll take some time time today to tell you about them.

> **_Video Lecture_**. Then check out a few comments on "Network Notation" (don't worry it's short).

```python
YouTubeVideo("MMziC5xktHs",width=800, height=450)
```

Now for some reading

> _Reading_. We'll be reading the textbook _Network Science_ (NS) by Laszlo Barabasi. You can read the whole
> thing for free [**here**](http://barabasi.com/networksciencebook/).
>
> * Read chapter 2\.
>

And the associated exercises

> *Exercises*: Chapter 2 in *Network Science*.
>
> * Section 2.5 states that real networks are sparse. Can you think of a real network where each node has _many_ connections? Is that network still sparse? If yes, can you explain why?
>
> There are more questions on Chapter 2 below.
>

---

*The neurons of the brain are often organized as a densely connected network.*

http://www.network-science.org/highly-connected-society-dense-social-complex-networks.html

---

# Part 2: Exercises using the `NetworkX` library

Let's try to solve a few of the exercises from the book using `NetworkX`

> *Exercises*:
>
> * Go to NS Section 2.12: Homework, then
>   * Write the solution exercise **2.1** (the 'Königsberg Problem') from NS in your `notebook`.
>   * Solve exercise **2.3** ('Graph representation') from NS using `NetworkX` in your `notebook`. (You don't have to solve the last sub-question about *cycles of length 4* ... but I'll be impressed if you do it).
>   * Solve exercise **2.5** ('Bipartite Networks') from NS using `NetworkX` in your `notebook`. *Important note*: There is a a mistake in the book. When it says "Block diagonal", they mean "Anti-diagonal" (all elements are away from the diagonal blocks).
>     - *Note*: For those without the physical book (and therefore no exercise numbers), the part "Bipatite Networks - General Considerations" does not need to be solved (the two last questions do not make much sense to me).
>

## Exercises:

### Exercise 2.1 ('The 'Königsberg Problem')

All execpt B can. B cannot as it has two nodes with uneven degree. In fact four nodes have degree 3.

### Exercise 2.3 ('Graph representation')

```python
import networkx as nx
import numpy as np
adj_a = np.array([
    [0,1,1,1,0,1],
    [1,0,1,1,0,0],
    [1,1,0,0,0,1],
    [1,1,0,0,0,0],
    [0,0,0,0,0,0],
    [1,0,1,0,0,0],
])

adj_b = np.array([
    [0,1,0,0,0,0],
    [0,0,1,1,0,0],
    [1,1,0,0,0,0],
    [1,0,0,0,0,0],
    [0,0,0,0,0,0],
    [1,0,1,0,0,0],
])

ga = nx.from_numpy_matrix(adj_a)
gb = nx.from_numpy_matrix(adj_b)
```

The corresponding link lists is generated with `.edges` method note that Python is zero index rather than 1-index.

```python
print(ga.edges)
print(gb.edges)
```

    [(0, 1), (0, 2), (0, 3), (0, 5), (1, 2), (1, 3), (2, 5)]
    [(0, 1), (0, 2), (0, 3), (0, 5), (1, 2), (1, 3), (2, 5)]

#### Determine the average clustering coefficient of the network shown in Image 2.20a

$$ C_{i}=\frac{2 L_{i}}{k_{i}\left(k_{i}-1\right)} $$

```python
def avg_cluster_coef(adj):
    n, p = np.shape(adj)
    C = np.zeros(n)

    k = adj.sum(axis = 0)
    L = np.zeros(n)
    for i in range(n):

        neighbours = np.where(adj[i] == 1)[0]

        L[i] = adj[neighbours][:, neighbours].sum() / 2

        # incase no neighbours
        if k[i] <= 1:
            C[i] = 0
        else:
            C[i] = (2 * L[i])/(k[i]*(k[i] - 1))

    return C

```

```python
avg_cluster_coef(adj_a)
```

    array([0.5       , 0.66666667, 0.66666667, 1.        , 0.        ,
           1.        ])

```python
nx.average_clustering(ga)
```

    0.6388888888888888

#### If you switch the labels of nodes 5 and 6 in Image 2.20a, how does that move change the adjacency matrix? And the link list?
This can be achieved by switching between the two row of node 5 and 6. In the link list all the 5 and 6 have to swapped.

#### What kind of information can you not infer from the link list representation of the network that you can infer from the adjacency matrix?
If you have a dense network the link list quickly becomes very long, and thus the adjacency matrix might give a better overview. I myself like the adjacency matrix more, but it really is preference and you can always go from one to another.

#### In the (a) network, how many paths (with possible repetition of nodes and links) of length 3 exist starting from node 1 and ending at node 3? And in (b)?

```python
np.linalg.matrix_power(adj_a, 3)[0,2]
```

    7

```python
np.linalg.matrix_power(adj_b, 3)[0,2]
```

    0

#### With the help of a computer, count the number of cycles of length 4 in both networks.

If we allow for repetions

```python
np.trace(
    np.linalg.matrix_power(adj_a, 4)
)
```

    86

```python
np.trace(
    np.linalg.matrix_power(adj_b, 4)
)
```

    2

### Exercise 2.5 ('Bipartite Networks')

**A) Construct its adjacency matrix. Why is it a block-diagonal matrix?**

```python
adj_BN = np.array([
    [0,0,0,0,0,0,1,0,0,0,0],
    [0,0,0,0,0,0,0,0,1,0,0],
    [0,0,0,0,0,0,1,1,1,0,0],
    [0,0,0,0,0,0,0,0,1,1,0],
    [0,0,0,0,0,0,0,0,1,0,1],
    [0,0,0,0,0,0,0,0,0,0,1],
    [1,0,1,0,0,0,0,0,0,0,0],
    [0,0,1,0,0,0,0,0,0,0,0],
    [0,1,1,1,1,0,0,0,0,0,0],
    [0,0,0,1,0,0,0,0,0,0,0],
    [0,0,0,0,1,1,0,0,0,0,0],
])
```

The first nodes never visit each other. The same goes for the last nodes. This is what gives it the peculiar adjacency matrix.

**b) Construct the adjacency matrix of its two projections, on the purple and on the green nodes, respectively.**

```python
adj_purple = np.array([
    [0, 0, 1, 0, 0, 0],
    [0, 0, 1, 0, 1, 0],
    [1, 1, 0, 0, 1, 0],
    [0, 1, 1, 0, 0, 0],
    [0, 1, 1, 1, 0, 0],
    [0, 0, 0, 0, 1, 0],
])

adj_green = np.array([
    [0, 1, 1, 0, 0],
    [1, 0, 1, 0, 0],
    [1, 1, 0, 1, 1],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
])

g_purple = nx.from_numpy_matrix(adj_purple)
g_green = nx.from_numpy_matrix(adj_green)
```

**c) Calculate the average degree of the purple nodes and the average degree of the green nodes in the bipartite network.**

```python
print(adj_BN[6:, :].sum(axis =1).sum()/6)

print(adj_BN[:-5, :].sum(axis =1).sum()/5)
```

    1.6666666666666667
    2.0

**d) Calculate the average degree in each of the two network projections. Is it surprising that the values are different from those obtained in point (c)?**

```python
print( np.mean(list(dict(g_purple.degree).values())) )
print( np.mean(list(dict(g_green.degree).values())) )
```

    2.6666666666666665
    2.0

> **_Video Lecture_**: Once again, it's time to stop working for a couple of minutes to hear me talk about plotting with `NetworkX`.
>

```python
YouTubeVideo("iDlb9On_TDQ",width=800, height=450)
```

# Part 3: Plotting degree distributions

As always we'll learn about degree-distribution plotting by creating a `notebook` and trying it out.

As a little public service announcement about this exercise: I know that you've probably tried binning data before. And that there are many functions designed to make binning easy, so there is a good chance that you think that binning data into bins is something that you don't have to worry about.

**BUT** ... with this exercise, I aim to convince you that binning isn't always trivial. And that there are key things to think about every time you need to put data into buckets. So please read the exercise carefully. And follow my instructions carefully.

>
> *Exercises*:
>
> Begin by importing the right packages. Start by importing `matplotlib.pyplot` (for plotting), `numpy` (for binning and other stuff),
> `random` (for generating random numbers), and `networkx` (for generating networks.)
>
> * Binning real numbers
>   * Let's do a gentle start and use the `random` library generate 5000 data points from a Gaussian distribution with $\mu = 2$ and $\sigma = 0.125$.
>   * Now, let's use `numpy.histogram` to bin those number into 10 bins. What does the `numpy.histogram` function return? Do the two arrays have the same length?
>   * Then we use `matplotlib.pyplot.plot` to plot the binned data. You will have to deal with the fact that the counts- and bin-arrays have different lengths. Explain how you deal with this problem and why.
> * Binning integers
>   * But binning real numbers into a fixed number of bins is easy when `numpy.histogram` does all the work and finds the right bin boundaries for you.
> Now we'll generate a bunch of integers and set the bin boundaries manually. This time, let's grab data from a Poisson distribution. As it turns out
> `numpy` also has some convenient random number generators. Use `numpy.random.poisson` to generate 5000 numbers drawn from a Poisson distribution
> characterized by $\lambda = 10$. Find the maximum and minimum value of your 5000 random numbers.
>   * Instead of simplify specifying the number of bins for `numpy.histogram`, let's specify the bins we want using a vector.
> Create a vector $v$
> that results in a binning that puts each integer value in its own bin and where the first bin contains the minimum number you found above,
> and the last bin contains the maximum number.
> Use the vector by setting `numpy.histogram`'s `bin` parameter as `bin = ` $v$.
> What is the sum over bin counts? Explain how the binning-vectors first and last element
> relates to the min and max from the Poisson distribution.
>   * Now, use a bar chart (`matplotlib.pyplot.bar`) to plot the distribution
> * Binning and plotting degree distributions.
>   * Let's generate the Erdös-Renyi (ER) network which has a degree distribution that matches the Poisson distribution above.
> First we have to figure out which values the ER parameters (_N_ and _p_) should assume. It's easy to see that $N = 5000$, but how do you find $p$?
> **Hint**: The parameter $\lambda$ in the Poisson distribution corresponds to the average degree, so you have to find a $p$ that results in an average
> degree, $k = 10$. And you know that $\langle k \rangle = p (N-1)$, which will give you $p$.
>   * Now, use `networkx` to create the graph and extract the degree distribution.
>   * Finally, create a nice bar plot of the degree distribution, including axes labels and a plot title. Make sure that it looks like the Poisson distribution you plotted above.
>

```python
import matplotlib.pyplot as plt
import random
import numpy as np
import networkx as nx
np.random.seed(42)
```

### Binning real numbers

```python
r = np.random.normal(loc = 2, scale=0.125, size=5000)
hist, bin_edges = np.histogram(r, bins = 10)
```

```python
x = np.repeat(bin_edges, 2)[1:-1]
y = np.repeat(hist, 2)
```

```python
plt.plot(x,y)
```

### Binning integers

```python
r = np.random.poisson(lam = 10, size = 5000)
r_min = r.min()
r_max = r.max()
v = np.arange(r_min, r_max+2)
```

```python
hist, bin_edges = np.histogram(r, bins = v - 1/2)
```

```python
plt.bar(v[:-1], hist)
```

### Binning and plotting degree distributions

```python
N = 5000
k = 10
g_er = nx.erdos_renyi_graph(n = N, p = k/(N-1))
```

```python
edges = np.array([edges for node, edges in g_er.degree()])

g_min = edges.min()
g_max = edges.max()
v = np.arange(g_min, g_max+2)

hist, bin_edges = np.histogram(edges, bins = v - 1/2)
plt.bar(v[:-1], hist)
```

# Part 4: Random networks

> **_Video Lecture_**. Now it's time to relax and watch a few minutes of info on _Random Networks_.

```python
YouTubeVideo("c_SbQCzgqb0",width=800, height=450)
```

> _Reading_. Read section 3.1-3.7 (the most important part is 3.1-3.4) of Chapter 3 of _Network Science_. You can find the entire book [**here**](http://barabasi.com/networksciencebook/).

> _Exercises_ (should be completed in a `notebook`):
>
> * Work through NS exercise 3.1 ('Erdős-Rényi Networks'). The exercise can be found in Section 3.11: Homework.
> * Paths. Plot a random network with 200 nodes and an average degree of 1.5\. (I suggest using `networkx.draw` and reading [the documentation](https://networkx.github.io/documentation/stable/reference/drawing.html) carefully to get an overview of all the options and what they look like. For example, you may want to shrink the node size).
>   * Extract the Giant Connected Component, GCC. (Hint: You can use `networkx.connected_component_subgraphs` <- it works in ealier versions of NetworkX (<2.5) || Otherwise, you can use `nx.connected_components(G)` and `G.subgraph(c).copy()`)
>   * Choose a node at random from the GCC. (Hint: You may want to try `random.choice`.)
>   * Find all nodes that are precisely 2 steps away from that node. (Hint. I suggest `networkx.single_source_shortest_path_length`)
>   * Plot the GCC with the following choice of colors. Starting node _black_ (`"#000000"`). The nodes 2 steps away _red_ (`"#ff0000"`). All other nodes _blue_ (`"#A0CBE2"`). Again, I suggest using `networkx.draw()` and reading [the documentation](https://networkx.github.io/documentation/stable/reference/drawing.html) carefully find out how to color individual nodes.
>

By the way, the default plot coming out of `networkx` doesn't look like the one below. But by tweaking parameters, you should be able to create something that looks like it.

## Exercise 3.1 ('Erdős-Rényi Networks').
Consider an Erdős-Rényi network with $N = 3,000$ nodes, connected to each other with probability $p = 10–3$.

#### What is the expected number of links〈L〉
\begin{align}
    \langle L \rangle &=\frac{p N(N-1)}{2}
\end{align}

```python
p = 0.001
N = 3000
k = p*(N-1)

L = p*N*(N-1)/2
print(L)
print(k)
```

    4498.5
    2.999

#### In which regime is the network?
1. Subcritial: $\langle k \rangle < 1$
1. Supercritical: $\langle k \rangle \geq 1$
1. Fully Connected: $\langle k \rangle \geq \log N$

```python
if k < 1:
    print('Rubcritical Regime')
elif k >= 1:
    if k < np.log(n):
        print('Supercritical Regime:')
    else:
        print('Fully connected Regime:')
```

#### Calculate the probability $p_c$ so that the network is at the critical point
To be at the critical point $k=1$ thus:
\begin{align}
    k &= p \cdot (N-1) \\
    1 &= p \cdot (N-1) \\
      &\Updownarrow \\
    p  &= \frac{1}{(N-1)}
\end{align}

```python
print('p = {} in order to be at the critical point'.format( 1/(N-1) ))
```

    p = 0.00033344448149383126 in order to be at the critical point

#### Given the linking probability $p = 10–3$, calculate the number of nodes $N^{cr}$ so that the network has only one component.

In order to only have one component the network needs to be in fully connected domain which is achived at
$$\langle k \rangle  \; \geq \;  \log(N) \Leftrightarrow p \geq \frac{\log(N)}{N}$$

So we have to determine
\begin{align}
    P &= \frac{\log(N)}{N} \\
      &\Updownarrow \\
    N &= \exp(-W(-P))
\end{align}
Where $W$ is the product log function. Or it can be achived by simply seeing when  $p \geq \frac{\log(N)}{N}$

```python
n = np.arange(1,10000)
p = 0.001
val = np.log(n)/n

idx = np.where(0.001 > val)[0][0:2]
```

```python
plt.figure(figsize=(10,10))
plt.plot(val)
plt.hlines(0.001,1, 10000)
plt.scatter(n[idx[0]], p, s=100, label = "First solution: N={0}".format(n[idx[0]]))
plt.scatter(n[idx[1]], p, s=100, label = "Second solution: N={0}".format(n[idx[1]]))
plt.legend()
```

#### For the network in $(d)$, calculate the average degree $\langle k^{cr} \rangle$ and the average distance between two randomly chosen nodes $\langle d \rangle$

## Plot a random network with 200 nodes and an average degree of 1.5.

```python
N = 200
k = 1.5
g_er = nx.erdos_renyi_graph(n = N, p = k/(N-1))
```

```python
nx.draw_spring(
    G = g_er,
    node_size = 50,
    node_color = '#7FA8FF',
    node_shape = 'o',
    edge_color = 'k',
    with_labels = False,
    width = 1.3,
    edgecolors='k')
plt.draw()
```

#### Extract the Giant Connected Component, GCC

```python
cc = sorted(nx.connected_components(g_er), key = len, reverse = True)

gcc = g_er.subgraph(cc[0]).copy()

nx.draw_spring(
    G = gcc,
    node_size = 50,
    node_color = '#7FA8FF',
    node_shape = 'o',
    edge_color = 'k',
    with_labels = False,
    width = 1.3,
    edgecolors='k')
plt.draw()
```

#### Choose a node at random from the GCC.

```python
random_node = random.choice(list(gcc.nodes()))
```

```python
random_node
```

    95

#### Find all nodes that are precisely 2 steps away from that node.

```python
max_2_steps = nx.single_source_shortest_path_length(gcc, random_node, cutoff = 2)
two_steps = np.array([node for (node, steps) in max_2_steps.items() if steps == 2])
two_steps
```

    array([ 69,  45, 173, 176,  55])

```python
node_array
```

    array([  0,   1,   4,   6,   7,   8,  14,  16,  17,  18,  23,  26,  28,
            29,  31,  35,  36,  37,  38,  39,  40,  42,  43,  45,  46,  48,
            49,  53,  54,  55,  56,  58,  59,  60,  63,  64,  66,  68,  69,
            70,  71,  73,  74,  75,  76,  77,  78,  80,  81,  85,  89,  90,
            91,  94,  95,  96,  97,  98,  99, 102, 103, 104, 105, 108, 111,
           113, 116, 117, 119, 120, 121, 122, 125, 127, 130, 131, 132, 133,
           137, 138, 139, 140, 142, 143, 145, 146, 147, 148, 149, 150, 151,
           152, 153, 154, 155, 156, 158, 159, 160, 166, 168, 169, 170, 171,
           172, 173, 174, 176, 177, 178, 179, 180, 181, 183, 185, 186, 188,
           189, 190, 191, 192, 194, 195, 196, 197, 198, 199])

#### Plot the GCC with the following choice of colors. Starting node black ("#000000"). The nodes 2 steps away red ("#ff0000"). All other nodes blue ("#A0CBE2"). Again, I suggest using networkx.draw() and reading the documentation carefully find out how to color individual nodes.

```python
node_array = np.array(list(gcc.nodes))
idx_random_node = np.where(node_array == random_node)[0]
idx_two_step = np.array([np.where(node_array == two_steps[i])[0][0] for i in range(len(two_steps))] )
```

```python
node_colors = np.array(['#A0CBE2'] * len(gcc))
node_colors[idx_random_node] = '#000000'
node_colors[idx_two_step] = "#ff0000"
```

```python
nx.draw_spring(
    G = gcc,
    node_size = 50,
    node_color = node_colors,
    node_shape = 'o',
    edge_color = 'k',
    with_labels = False,
    width = 1.3,
    edgecolors='k')
plt.draw()
```


Table of Contents

1  Interactive Demo: Gaussian Distribution Explorer

2  Section 1.1: Probabilistic Models

3  Section 1.2: Likelihood Estimation

3.1  Exercise 1: Likelihood Function

4  Section 1.3: Finding the Maximum Likelihood Estimator

# Neuromatch Academy: Week 1, Day 3, Tutorial 2
# Model Fitting: Linear regression with MLE

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil  with help from Byron Galbraith

**Content reviewers**: Lina Teichmann, Madineh Sarvestani, Patrick Mineault, Ella Batty, Michael Waskom

---
#Tutorial Objectives

This is Tutorial 2 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will use a different approach to fit linear models that incorporates the random 'noise' in our data.
- Learn about probability distributions and probabilistic models
- Learn how to calculate the likelihood of our model parameters
- Learn how to implement the maximum likelihood estimator, to find the model parameter with the maximum likelihood

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper Functions
def plot_density_image(x, y, theta, sigma=1, ax=None):
  """ Plots probability distribution of y given x, theta, and sigma

  Args:

    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta (float): Slope parameter
    sigma (float): standard deviation of Gaussian noise

  """

  # plot the probability density of p(y|x,theta)
  if ax is None:
    fig, ax = plt.subplots()

  xmin, xmax = np.floor(np.min(x)), np.ceil(np.max(x))
  ymin, ymax = np.floor(np.min(y)), np.ceil(np.max(y))
  xx = np.linspace(xmin, xmax, 50)
  yy = np.linspace(ymin, ymax, 50)

  surface = np.zeros((len(yy), len(xx)))
  for i, x_i in enumerate(xx):
    surface[:, i] = stats.norm(theta * x_i, sigma).pdf(yy)

  ax.set(xlabel='x', ylabel='y')

  return ax.imshow(surface, origin='lower', aspect='auto', vmin=0, vmax=None,
            cmap=plt.get_cmap('Wistia'),
            extent=[xmin, xmax, ymin, ymax])

def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter.

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """
  theta_hat = (x.T @ y) / (x.T @ x)
  return theta_hat
```

---
# Section 1: Maximum Likelihood Estimation (MLE)

```python
#@title Video 1: Maximum Likelihood Estimation
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="8mpNmzLKNfU", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=8mpNmzLKNfU

In the previous tutorial we made the assumption that the data was drawn from a linear relationship with noise added, and found an effective approach for estimating model parameters based on minimizing the mean squared error.

In that case we treated the noise as simply a nuisance, but what if we factored it directly into our model?

Recall our linear model:

\begin{align}
y = \theta x + \epsilon.
\end{align}

The noise component $\epsilon$ is often modeled as a random variable drawn from a Gaussian distribution (also called the normal distribution).

The Gaussian distribution is described by its [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (pdf)
\begin{align}
\mathcal{N}(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(x-\mu)^2}
\end{align}

and is dependent on two parameters: the mean $\mu$ and the variance $\sigma^2$. We often consider the noise signal to be Gaussian "white noise", with zero mean and unit variance:

\begin{align}
\epsilon \sim \mathcal{N}(0, 1).
\end{align}

## Interactive Demo: Gaussian Distribution Explorer

Use the explorer widget below to see how varying the $\mu$ and $\sigma$ parameters change the location and shape of the samples.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(mu=widgets.FloatSlider(0.0, min=-2.0, max=2.0),
                  sigma=widgets.FloatSlider(1.0, min=0.5, max=2.0))
def plot_normal_dist(mu=0, sigma=1):

  # Generate pdf & samples from normal distribution with mu/sigma
  rv = stats.norm(mu, sigma)
  x = np.linspace(-5, 5, 100)
  y = rv.pdf(x)
  samples = rv.rvs(1000)

  # Plot
  fig, ax = plt.subplots()
  ax.hist(samples, 20, density=True, color='g', histtype='stepfilled', alpha=0.8,
          label='histogram')
  ax.plot(x, y, color='orange', linewidth=3, label='pdf')
  ax.vlines(mu, 0, rv.pdf(mu), color='y', linewidth=3, label='$\mu$')
  ax.vlines([mu-sigma, mu+sigma], 0, rv.pdf([mu-sigma, mu+sigma]), colors='red',
            color='b', linewidth=3, label='$\sigma$')
  ax.set(xlabel='x', ylabel='probability density', xlim=[-5, 5], ylim=[0, 1.0])
  ax.legend()
```

    interactive(children=(FloatSlider(value=0.0, description='mu', max=2.0, min=-2.0), FloatSlider(value=1.0, desc…

## Section 1.1: Probabilistic Models

Now that we have a model of our noise component $\epsilon$ as random variable, how do we incorporate this back into our original linear model from before? Consider again our simplified model $y = \theta x + \epsilon$ where the noise has zero mean and unit variance $\epsilon \sim \mathcal{N}(0, 1)$. We can now also treat $y$ as a random variable drawn from a Gaussian distribution where $\mu = \theta x$ and $\sigma^2 = 1$:

\begin{align}
y \sim \mathcal{N}(\theta x, 1)
\end{align}

which is to say that the probability of observing $y$ given $x$ and parameter $\theta$ is
\begin{align}
p(y|x,\theta) = \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(y-\theta x)^2}
\end{align}

---

Let's revisit our original sample dataset where the true underlying model has $\theta = 1.2$.

```python
# @title

# @markdown Execute this cell to generate some simulated data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence

np.random.seed(121)
theta = 1.2
n_samples = 30
x = 10 * np.random.rand(n_samples) # sample from a uniform distribution over [0,10)
noise = np.random.randn(n_samples) # sample from a standard normal distribution
y = theta * x + noise
```

This time we can plot the density of $p(y|x,\theta=1.2)$ and see how $p(y)$ changes for different values of $x$.

```python
#@title
#@markdown Execute this cell to visualize p(y|x, theta=1.2)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))

# Invokes helper function to generate density image plots from data and parameters
im = plot_density_image(x, y, 1.2, ax=ax1)
plt.colorbar(im, ax=ax1)
ax1.axvline(8, color='k')
ax1.set(title=r'p(y | x, $\theta$=1.2)')

# Plot pdf for given x
ylim = ax1.get_ylim()
yy = np.linspace(ylim[0], ylim[1], 50)
ax2.plot(yy, stats.norm(theta * 8, 1).pdf(yy), color='orange', linewidth=2)
ax2.set(
    title=r'p(y|x=8, $\theta$=1.2)',
    xlabel='y',
    ylabel='probability density');
```

## Section 1.2: Likelihood Estimation

Now that we have our probabilistic model, we turn back to our original challenge of finding a good estimate for $\theta$ that fits our data. Given the inherent uncertainty when dealing in probabilities, we talk about the [likelihood](https://en.wikipedia.org/wiki/Likelihood_function) that some estimate $\hat \theta$ fits our data. The likelihood function $\mathcal{L(\theta)}$ is equal to the probability density function parameterized by that $\theta$:

\begin{align}
\mathcal{L}(\theta|x,y) = p(y|x,\theta) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(y-\theta x)^2}
\end{align}

### Exercise 1: Likelihood Function

In this exercise you will implement the likelihood function $\mathcal{L}(\theta|x,y)$ for our linear model where $\sigma = 1$.

After implementing this function, we can produce probabilities that our estimate $\hat{\theta}$ generated the provided observations. We will try with one of the samples from our dataset.

TIP: Use `np.exp` and `np.sqrt` for the exponential and square root functions, respectively.

```python
def likelihood(theta_hat, x, y):
  """The likelihood function for a linear model with noise sampled from a
    Gaussian distribution with zero mean and unit variance.

  Args:
    theta_hat (float): An estimate of the slope parameter.
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    ndarray: the likelihood values for the theta_hat estimate
  """
  sigma = 1
  ##############################################################################
  ## TODO for students: implement the likelihood function
  # Fill out function and remove
#   raise NotImplementedError("Student exercise: implement the likelihood function")
  ##############################################################################

  # Compute Gaussian likelihood
  pdf = 1 / np.sqrt(2 * np.pi * sigma**2) * np.exp(-(y - theta_hat * x)**2 / (2 * sigma**2))

  return pdf

# Uncomment below to test your function
print(likelihood(1.0, x[1], y[1]))
```

    0.11344443599846923

```python
# to_remove solution
def likelihood(theta_hat, x, y):
  """The likelihood function for a linear model with noise sampled from a
    Gaussian distribution with zero mean and unit variance.

  Args:
    theta_hat (float): An estimate of the slope parameter.
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    float: the likelihood value for the theta_hat estimate
  """
  sigma = 1

  # Compute Gaussian likelihood
  pdf = 1 / np.sqrt(2 * np.pi * sigma**2) * np.exp(-(y - theta_hat * x)**2 / (2 * sigma**2))

  return pdf

print(likelihood(1.0, x[1], y[1]))
```

    0.11344443599846923

We should see that $\mathcal{L}(\theta=1.0|x=2.1,y=3.7) \approx 0.11$. So far so good, but how does this tell us how this estimate is better than any others?

When dealing with a set of data points, as we are with our dataset, we are concerned with their joint probability -- the likelihood that all data points are explained by our parameterization. Since we have assumed that the noise affects each output independently, we can factorize the likelihood, and write:

\begin{align}
\mathcal{L}(\theta|X,Y) = \prod_{i=1}^N \mathcal{L}(\theta|x_i,y_i),
\end{align}

where we have $N$ data points $X = \{x_1,...,x_N\}$ and $Y = \{y_1,...,y_N\}$.

In practice, such a product can be numerically unstable. Indeed multiplying small values together can lead to [underflow](https://en.wikipedia.org/wiki/Arithmetic_underflow), the situation in which the digital representation of floating point number reaches its limit. This problem can be circumvented by taking the logarithm of the likelihood because the logarithm transforms products into sums:

\begin{align}
\operatorname{log}\mathcal{L}(\theta|X,Y) = \sum_{i=1}^N \operatorname{log}\mathcal{L}(\theta|x_i,y_i)
\end{align}

We can take the sum of the log of the output of our `likelihood` method applied to the full dataset to get a better idea of how different $\hat\theta$ compare. We can also plot the different distribution densities over our dataset and see how they line up qualitatively.

```python
#@title
#@markdown Execute this cell to visualize different distribution densities
theta_hats = [0.5, 1.0, 2.2]
fig, axes = plt.subplots(ncols=3, figsize=(16, 4))
for theta_hat, ax in zip(theta_hats, axes):
  ll = np.sum(np.log(likelihood(theta_hat, x, y)))  # log likelihood
  im = plot_density_image(x, y, theta_hat, ax=ax)
  ax.scatter(x, y)
  ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat}, log likelihood: {ll:.2f}')
plt.colorbar(im, ax=ax);
```

Using the log likelihood calculation, we see that $\mathcal{L}(\theta=1.0) > \mathcal{L}(\theta=0.5) > \mathcal{L}(\theta=2.2)$.

This is great: now we have a way to compare estimators based on likelihood. But like with the MSE approach, we want an analytic solution to find the best estimator. In this case, we want to find the estimator that maximizes the likelihood.

## Section 1.3: Finding the Maximum Likelihood Estimator

We want to find the parameter value $\hat\theta$ that makes our data set most likely:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \mathcal{L}(\theta|X,Y)
\end{align}

We discussed how taking the logarithm of the likelihood helps with numerical stability, the good thing is that it does so without changing the parameter value that maximizes the likelihood. Indeed, the $\textrm{log}()$ function is *monotonically increasing*, which means that it preserves the order of its inputs. So we have:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \sum_{i=1}^m \textrm{log} \mathcal{L}(\theta|x_i,y_i)
\end{align}

Now substituting our specific likelihood function and taking its logarithm, we get:
\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} [-\frac{N}{2} \operatorname{log} 2\pi\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^N (y_i-\theta x_i)^2].
\end{align}

Note that maximizing the log likelihood is the same as minimizing the negative log likelihood (in practice optimization routines are developed to solve minimization not maximization problems). Because of the convexity of this objective function, we can take the derivative of our negative log likelihhood, set it to 0, and solve - just like our solution to minimizing MSE.

\begin{align}
\frac{\partial\operatorname{log}\mathcal{L}(\theta|x,y)}{\partial\theta}=\frac{1}{\sigma^2}\sum_{i=1}^N(y_i-\theta x_i)x_i = 0
\end{align}

This looks remarkably like the equation we had to solve for the optimal MSE estimator, and, in fact, we arrive to the exact same solution!

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \hat{\theta}_{\textrm{MSE}} = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}
\end{align}

```python
# Compute theta_hat_MLE
theta_hat_mle = (x @ y) / (x @ x)

```

```python
#@title
#@markdown Execute this cell to visualize density with theta_hat_mle

# Plot the resulting distribution density
fig, ax = plt.subplots()
ll = np.sum(np.log(likelihood(theta_hat_mle, x, y))) # log likelihood
im = plot_density_image(x, y, theta_hat_mle, ax=ax)
plt.colorbar(im, ax=ax);
ax.scatter(x, y)
ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat_mle:.2f}, log likelihood: {ll:.2f}');
```

---
# Summary

- Likelihood vs probability
    - $\mathcal{L}(\theta|x, y) = p(y|\theta, x)$
    - $p(y|\theta, x)$ -> "probability of observing the response $y$ given parameter $\theta$ and input $x$"
    - $\mathcal{L}(\theta|x, y)$ -> "likelihood model that parameters $\theta$ produced response $y$ from input $x$"
- Log-likelihood maximization
    - We take the $\textrm{log}$ of the likelihood function for computational convenience
    - The parameters $\theta$ that maximize $\textrm{log}\mathcal{L}(\theta|x, y)$ are the model parameters that maximize the probability of observing the data.
- **Key point**:
  - the log-likelihood is a flexible cost function, and is often used to find model parameters that best fit the data.

---
# Appendix

We can also see $\mathrm{p}(\mathrm{y} | \mathrm{x}, \theta)$ as a function of $x$. This is the stimulus likelihood function, and it is useful in case we want to decode the input $x$ from observed responses $y$. This is what is relevant from the point of view of a neuron that does not have access to the outside world and tries to infer what's out there from the responses of other neurons!