# Tutorial 3: Synaptic transmission - Models of static and dynamic synapses
**Week 2, Day 3: Real Neurons**

**By Neuromatch Academy**

__Content creators:__ Qinglong Gu, Songtin Li, John Murray, Richard Naud, Arvind Kumar

__Content reviewers:__  Maryam Vaziri-Pashkam, Ella Batty, Lorenzo Fontolan, Richard Gao, Matthew Krause, Spiros Chavlis, Michael Waskom

---
# Tutorial Objectives
Synapses connect neurons into neural networks or circuits. Specialized electrical synapses make direct, physical connections between neurons. In this tutorial, however, we will focus on **chemical synapses**, which are more common in the brain. These synapses do not physically join neurons. Instead, a spike in the presynaptic cell causes a chemical, or neurotransmitter, to be released into a small space between the neurons called the synaptic cleft. Once the chemical diffuses across that space, it changes the pearmeability of the postsynaptic membrane, which may result in a positive or negative change in the membrane voltage.

In this tutorial, we will model chemical synaptic transmission and study some interesting effects produced by **static synapses** and **dynamic synapses**.

First, we will start by writing code to simulate static synapses -- whose weight is always fixed.
Next, we will extend the model and model **dynamic synapses** -- whose synaptic strength is dependent on the recent spike history: synapses can either progressively increase or decrease the size of their effects on the post-synaptic neuron, based on the recent firing rate of its presynaptic partners. This feature of synapses in the brain is called **Short-Term Plasticity** and causes synapses to undergo *Facilitation* or *Depression*.

Our goals for this tutorial are to:

- simulate static synapses and study how excitation and inhibition affect the patterns in the neurons' spiking output
- define mean- or fluctuation-driven regimes
- simulate short-term dynamics of synapses (facilitation and depression)
- study how a change in pre-synaptic firing history affects the synaptic weights (i.e., PSP amplitude)

```python
#@title Video 1: Static and dynamic synapses
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='Hbz2lj2AO_0', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

---
# Setup

```python
# Import libraries
import matplotlib.pyplot as plt
import numpy as np
```

```python
# @title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format='retina'
# use NMA plot style
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
my_layout = widgets.Layout()
```

```python
# @title Helper functions

def my_GWN(pars, mu, sig, myseed=False):
  """
  Args:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    myseed     : random seed. int or boolean
                  the same seed will give the same random number sequence

  Returns:
    I          : Gaussian White Noise (GWN) input
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  # you can fix the seed of the random number generator so that the results
  # are reliable. However, when you want to generate multiple realizations
  # make sure that you change the seed for each new realization.
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate GWN
  # we divide here by 1000 to convert units to seconds.
  I_GWN = mu + sig * np.random.randn(Lt) / np.sqrt(dt / 1000.)

  return I_GWN

def Poisson_generator(pars, rate, n, myseed=False):
  """
  Generates poisson trains

  Args:
    pars       : parameter dictionary
    rate       : noise amplitute [Hz]
    n          : number of Poisson trains
    myseed     : random seed. int or boolean

  Returns:
    pre_spike_train : spike train matrix, ith row represents whether
                      there is a spike in ith spike train over time
                      (1 if spike, 0 otherwise)
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate uniformly distributed random variables
  u_rand = np.random.rand(n, Lt)

  # generate Poisson train
  poisson_train = 1. * (u_rand < rate * (dt / 1000.))

  return poisson_train

def default_pars(**kwargs):
  pars = {}

  ### typical neuron parameters###
  pars['V_th'] = -55.    # spike threshold [mV]
  pars['V_reset'] = -75. # reset potential [mV]
  pars['tau_m'] = 10.    # membrane time constant [ms]
  pars['g_L'] = 10.      # leak conductance [nS]
  pars['V_init'] = -65.  # initial potential [mV]
  pars['E_L'] = -75.     # leak reversal potential [mV]
  pars['tref'] = 2.      # refractory time (ms)

  ### simulation parameters ###
  pars['T'] = 400.  # Total duration of simulation [ms]
  pars['dt'] = .1   # Simulation time step [ms]

  ### external parameters if any ###
  for k in kwargs:
    pars[k] = kwargs[k]

  pars['range_t'] = np.arange(0, pars['T'], pars['dt'])  # Vector of discretized time points [ms]

  return pars

def my_illus_LIFSYN(pars, v_fmp, v):
  """
  Illustartion of FMP and membrane voltage

  Args:
    pars  : parameters dictionary
    v_fmp : free membrane potential, mV
    v     : membrane voltage, mV

  Returns:
    plot of membrane voltage and FMP, alongside with the spiking threshold
    and the mean FMP (dashed lines)
  """

  plt.figure(figsize=(14, 5))
  plt.plot(pars['range_t'], v_fmp, 'r', lw=1.,
           label='Free mem. pot.', zorder=2)
  plt.plot(pars['range_t'], v, 'b', lw=1.,
           label='True mem. pot', zorder=1, alpha=0.7)
  plt.axhline(-55, 0, 1, color='k', lw=2., ls='--',
              label='Spike Threshold', zorder=1)
  plt.axhline(np.mean(v_fmp), 0, 1, color='r', lw=2., ls='--',
              label='Mean Free Mem. Pot.', zorder=1)
  plt.xlabel('Time (ms)')
  plt.ylabel('V (mV)')
  plt.legend(loc=[1.02, 0.68])
  plt.show()

def my_illus_STD(Poi_or_reg=False, rate=20., U0=0.5,
                 tau_d=100., tau_f=50., plot_out=True):
  """
   Only for one presynaptic train

  Args:
    Poi_or_reg : Poisson or regular input spiking trains
    rate       : Rate of input spikes, Hz
    U0         : synaptic release probability at rest
    tau_d      : synaptic depression time constant of x [ms]
    tau_f      : synaptic facilitation time constantr of u [ms]
    plot_out   : whether ot not to plot, True or False

  Returns:
    Nothing.
  """

  T_simu = 10.0 * 1000 / (1.0 * rate)  # 10 spikes in the time window
  pars = default_pars(T=T_simu)
  dt = pars['dt']

  if Poi_or_reg:
    # Poisson type spike train
    pre_spike_train = Poisson_generator(pars, rate, n=1)
    pre_spike_train = pre_spike_train.sum(axis=0)
  else:
    # Regular firing rate
    isi_num = int((1e3/rate)/dt)  # number of dt
    pre_spike_train = np.zeros(len(pars['range_t']))
    pre_spike_train[::isi_num] = 1.

  u, R, g = dynamic_syn(g_bar=1.2, tau_syn=5., U0=U0,
                        tau_d=tau_d, tau_f=tau_f,
                        pre_spike_train=pre_spike_train,
                        dt=pars['dt'])

  if plot_out:
    plt.figure(figsize=(12, 6))

    plt.subplot(221)
    plt.plot(pars['range_t'], R, 'b', label='R')
    plt.plot(pars['range_t'], u, 'r', label='u')
    plt.legend(loc='best')
    plt.xlim((0, pars['T']))
    plt.ylabel(r'$R$ or $u$ (a.u)')
    plt.subplot(223)
    spT = pre_spike_train > 0
    t_sp = pars['range_t'][spT]  #spike times
    plt.plot(t_sp, 0. * np.ones(len(t_sp)), 'k|', ms=18, markeredgewidth=2)
    plt.xlabel('Time (ms)');
    plt.xlim((0, pars['T']))
    plt.yticks([])
    plt.title('Presynaptic spikes')

    plt.subplot(122)
    plt.plot(pars['range_t'], g, 'r', label='STP synapse')
    plt.xlabel('Time (ms)')
    plt.ylabel('g (nS)')
    plt.xlim((0, pars['T']))

    plt.tight_layout()

  if not Poi_or_reg:
    return g[isi_num], g[9*isi_num]

def plot_volt_trace(pars, v, sp):
  """
  Plot trajetory of membrane potential for a single neuron

  Args:
    pars   : parameter dictionary
    v      : volt trajetory
    sp     : spike train

  Returns:
    figure of the membrane potential trajetory for a single neuron
  """

  V_th = pars['V_th']
  dt = pars['dt']
  if sp.size:
      sp_num = (sp/dt).astype(int) - 1
      v[sp_num] += 10

  plt.plot(pars['range_t'], v, 'b')
  plt.axhline(V_th, 0, 1, color='k', ls='--', lw=1.)
  plt.xlabel('Time (ms)')
  plt.ylabel('V (mV)')
```

In the `Helper Function`:

- Gaussian white noise generator: `my_GWN(pars, mu, sig, myseed=False)`
- Poissonian spike train generator: `Poisson_generator(pars, rate, n, myseed=False)`
- default parameter function (as before) and other plotting utilities

---
# Section 1: Static synapses

## Section 1.1: Simulate synaptic conductance dynamics

Synaptic input _in vivo_ consists of a mixture of **excitatory** neurotransmitters, which depolarizes the cell and drives it towards spike threshold, and **inhibitory** neurotransmitters that hyperpolarize it, driving it away from spike threshold. These chemicals cause specific ion channels on the postsynaptic neuron to open, resulting in a change in that neuron's conductance and, therefore, the flow of current in or out of the cell.

This process can be modelled by assuming that the presynaptic neuron's spiking activity produces transient changes in the postsynaptic neuron's conductance ($g_{\rm syn}(t)$). Typically, the conductance transient is modeled as an exponential function.

Such conductance transients can be generated using a simple ordinary differential equation (ODE):

\\

\begin{eqnarray}
\frac{dg_{\rm syn}(t)}{dt} &=& \bar{g}_{\rm syn} \sum_k \delta(t-t_k) -g_{\rm syn}(t)/\tau_{\rm syn}
\end{eqnarray}

\\

where $\bar{g}_{\rm syn}$ (often referred to as synaptic weight) is the maximum conductance elicited by each incoming spike, and $\tau_{\rm syn}$ is the synaptic time constant. Note that the summation runs over all spikes received by the neuron at time $t_k$.

Ohm's law allows us to convert conductance changes to the current as:

\\

\begin{align}
I_{\rm syn}(t) = g_{\rm syn}(t)(V(t)-E_{\rm syn})  \\
\end{align}

\\

The reversal potential $E_{\rm syn}$ determines the direction of current flow and the excitatory or inhibitory nature of the synapse.

**Thus, incoming spikes are filtered by an exponential-shaped kernel, effectively low-pass filtering the input. In other words, synaptic input is not white noise, but it is, in fact, colored noise, where the color (spectrum) of the noise is determined by the synaptic time constants of both excitatory and inhibitory synapses.**

In a neuronal network, the total synaptic input current $I_{\rm syn}$ is the sum of both excitatory and inhibitory inputs. Assuming the total excitatory and inhibitory conductances received at time $t$ are $g_E(t)$ and $g_I(t)$, and their corresponding reversal potentials are $E_E$ and $E_I$, respectively, then the total synaptic current can be described as:

\\

\begin{align}
I_{\rm syn}(V(t),t) = -g_E(t) (V-E_E) - g_I(t) (V-E_I)
\end{align}

\\

Accordingly, the membrane potential dynamics of the LIF neuron under synaptic current drive become:

\\

\begin{eqnarray}
\tau_m\frac{dV(t)}{dt} = -(V(t)-E_L) - \frac{g_E(t)}{g_L} (V(t)-E_E) - \frac{g_I(t)}{g_L} (V(t)-E_I) + \frac{I_{\rm inj}}{g_L}\quad (2)
\end{eqnarray}

\\

$I_{\rm inj}$ is an external current injected in the neuron, which is under experimental control; it can be GWN, DC, or anything else.

We will use Eq. (2) to simulate the conductance-based LIF neuron model below.

In the previous tutorials, we saw how the output of a single neuron (spike count/rate and spike time irregularity) changes when we stimulate the neuron with DC and GWN, respectively. Now, we are in a position to study how the neuron behaves when it is bombarded with both excitatory and inhibitory spikes trains -- as happens *in vivo*.

What kind of input is a neuron receiving? When we do not know, we chose the simplest option. The simplest model of input spikes is given when every input spike arrives independently of other spikes, i.e., we assume that the input is Poissonian.

## Section 1.2: Simulate LIF neuron with conductance-based synapses

We are now ready to simulate a LIF neuron with conductance-based synaptic inputs! The following code defines the LIF neuron with synaptic input modeled as conductance transients.

```python
# @markdown Execute this cell to get a function for conductance-based LIF neuron (run_LIF_cond)

def run_LIF_cond(pars, I_inj, pre_spike_train_ex, pre_spike_train_in):
  """
  Conductance-based LIF dynamics

  Args:
    pars               : parameter dictionary
    I_inj              : injected current [pA]. The injected current here
                         can be a value or an array
    pre_spike_train_ex : spike train input from presynaptic excitatory neuron
    pre_spike_train_in : spike train input from presynaptic inhibitory neuron

  Returns:
    rec_spikes         : spike times
    rec_v              : mebrane potential
    gE                 : postsynaptic excitatory conductance
    gI                 : postsynaptic inhibitory conductance

  """

  # Retrieve parameters
  V_th, V_reset = pars['V_th'], pars['V_reset']
  tau_m, g_L = pars['tau_m'], pars['g_L']
  V_init, E_L = pars['V_init'], pars['E_L']
  gE_bar, gI_bar = pars['gE_bar'], pars['gI_bar']
  VE, VI = pars['VE'], pars['VI']
  tau_syn_E, tau_syn_I = pars['tau_syn_E'], pars['tau_syn_I']
  tref = pars['tref']
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # Initialize
  tr = 0.
  v = np.zeros(Lt)
  v[0] = V_init
  gE = np.zeros(Lt)
  gI = np.zeros(Lt)
  Iinj = I_inj * np.ones(Lt)  # ensure Iinj has length Lt

  if pre_spike_train_ex.max() == 0:
    pre_spike_train_ex_total = np.zeros(Lt)
  else:
    pre_spike_train_ex_total = pre_spike_train_ex.sum(axis=0) * np.ones(Lt)

  if pre_spike_train_in.max() == 0:
    pre_spike_train_in_total = np.zeros(Lt)
  else:
    pre_spike_train_in_total = pre_spike_train_in.sum(axis=0) * np.ones(Lt)

  # simulation
  rec_spikes = []  # recording spike times
  for it in range(Lt - 1):
    if tr > 0:
      v[it] = V_reset
      tr = tr - 1
    elif v[it] >= V_th:  # reset voltage and record spike event
      rec_spikes.append(it)
      v[it] = V_reset
      tr = tref / dt

    # update the synaptic conductance
    gE[it + 1] = gE[it] - (dt / tau_syn_E) * gE[it] + gE_bar * pre_spike_train_ex_total[it + 1]
    gI[it + 1] = gI[it] - (dt / tau_syn_I) * gI[it] + gI_bar * pre_spike_train_in_total[it + 1]

    # calculate the increment of the membrane potential
    dv = (dt / tau_m) * (-(v[it] - E_L) \
                         - (gE[it + 1] / g_L) * (v[it] - VE) \
                         - (gI[it + 1] / g_L) * (v[it] - VI) + Iinj[it] / g_L)

    # update membrane potential
    v[it+1] = v[it] + dv

  rec_spikes = np.array(rec_spikes) * dt

  return v, rec_spikes, gE, gI

print(help(run_LIF_cond))
```

### Exercise 1: Measure the mean free membrane potential

Let's simulate the conductance-based LIF neuron with presynaptic spike trains generated by a `Poisson_generator` with rate 10 Hz for both excitatory and inhibitory inputs. Here, we choose 80 excitatory presynaptic spike trains and 20 inhibitory ones.

Previously, we've already learned that $CV_{\rm ISI}$ can describe the irregularity of the output spike pattern. Now, we will introduce a new descriptor of the neuron membrane, i.e., the  **Free Membrane Potential (FMP)** -- the membrane potential of the neuron when its spike threshold is removed.

Although this is completely artificial, calculating this quantity allows us to get an idea of how strong the input is. We are mostly interested in knowing the mean and standard deviation (std.) of the FMP. In the exercise, you can visualize the FMP and membrane voltage with spike threshold.

```python
# To complete the exercise, uncomment the code and fill the missing parts (...)
pars = default_pars(T=1000.)
# Add parameters
pars['gE_bar'] = 2.4    # [nS]
pars['VE'] = 0.         # [mV] excitatory reversal potential
pars['tau_syn_E'] = 2.  # [ms]
pars['gI_bar'] = 2.4    # [nS]
pars['VI'] = -80.       # [mV] inhibitory reversal potential
pars['tau_syn_I'] = 5.  # [ms]

# generate presynaptic spike trains
pre_spike_train_ex = Poisson_generator(pars, rate=10, n=80)
pre_spike_train_in = Poisson_generator(pars, rate=10, n=20)

# simulate conductance-based LIF model
v, rec_spikes, gE, gI = run_LIF_cond(pars, 0, pre_spike_train_ex,
                                     pre_spike_train_in)
dt, range_t = pars['dt'], pars['range_t']
if rec_spikes.size:
  sp_num = (rec_spikes / dt).astype(int) - 1
  v[sp_num] = 10   # draw nicer spikes

####################################################################
## TODO for students: measure the free membrane potential
# In order to measure the free membrane potential, first,
# you should prevent the firing of the LIF neuron
# How to prevent a LIF neuron from firing? Increse the threshold pars['V_th'].
####################################################################
# Change the threshold
# pars['V_th'] = ...
# Calculate FMP
# v_fmp, _, _, _ = run_LIF_cond(pars, ..., ..., ...)

# uncomment when you have filled the exercise
# my_illus_LIFSYN(pars, v_fmp, v)
```

```python
# to_remove solution
pars = default_pars(T=1000.)
# Add parameters
pars['gE_bar'] = 2.4    # [nS]
pars['VE'] = 0.         # [mV] excitatory reversal potential
pars['tau_syn_E'] = 2.  # [ms]
pars['gI_bar'] = 2.4    # [nS]
pars['VI'] = -80.       # [mV] inhibitory reversal potential
pars['tau_syn_I'] = 5.  # [ms]

# generate presynaptic spike trains
pre_spike_train_ex = Poisson_generator(pars, rate=10, n=80)
pre_spike_train_in = Poisson_generator(pars, rate=10, n=20)

# simulate conductance-based LIF model
v, rec_spikes, gE, gI = run_LIF_cond(pars, 0, pre_spike_train_ex,
                                     pre_spike_train_in)
dt, range_t = pars['dt'], pars['range_t']
if rec_spikes.size:
  sp_num = (rec_spikes / dt).astype(int) - 1
  v[sp_num] = 10   # draw nicer spikes

# Change the threshold
pars['V_th'] = 1e3
# Calculate FMP
v_fmp, _, _, _ = run_LIF_cond(pars, 0, pre_spike_train_ex, pre_spike_train_in)

with plt.xkcd():
  my_illus_LIFSYN(pars, v_fmp, v)
```

### Interactive Demo: Conductance-based LIF Explorer with different E/I input

In the following, we can investigate how varying the ratio of excitatory to inhibitory inputs changes the firing rate and the spike time regularity (see the output text).

To change both the excitatory and inhibitory inputs, we will vary their firing rates. *However, if you wish, you can vary the strength and/or the number of these connections as well.*

Pay close attention to the mean free membrane potential (red dotted line) and its location with respect to the spike threshold (black dotted line). Try to develop a heuristic about the mean of the FMP and spike time irregularity ($CV_{\rm ISI}$)

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

my_layout.width = '450px'
@widgets.interact(
    inh_rate=widgets.FloatSlider(20., min=10., max=60., step=5.,
                                 layout=my_layout),
    exc_rate=widgets.FloatSlider(10., min=2., max=20., step=2.,
                                 layout=my_layout)
)

def EI_isi_regularity(exc_rate, inh_rate):

  pars = default_pars(T=1000.)
  # Add parameters
  pars['gE_bar'] = 3.     # [nS]
  pars['VE'] = 0.         # [mV] excitatory reversal potential
  pars['tau_syn_E'] = 2.  # [ms]
  pars['gI_bar'] = 3.     # [nS]
  pars['VI'] = -80.       # [mV] inhibitory reversal potential
  pars['tau_syn_I'] = 5.  # [ms]

  pre_spike_train_ex = Poisson_generator(pars, rate=exc_rate, n=80)
  pre_spike_train_in = Poisson_generator(pars, rate=inh_rate, n=20)  # 4:1

  # Lets first simulate a neuron with identical input but with no spike
  # threshold by setting the threshold to a very high value
  # so that we can look at the free membrane potential
  pars['V_th'] = 1e3
  v_fmp, rec_spikes, gE, gI = run_LIF_cond(pars, 0, pre_spike_train_ex,
                                           pre_spike_train_in)

  # Now simulate a LIP with a regular spike threshold
  pars['V_th'] = -55.
  v, rec_spikes, gE, gI = run_LIF_cond(pars, 0, pre_spike_train_ex,
                                       pre_spike_train_in)
  dt, range_t = pars['dt'], pars['range_t']
  if rec_spikes.size:
      sp_num = (rec_spikes / dt).astype(int) - 1
      v[sp_num] = 10  # draw nicer spikes

  spike_rate = 1e3 * len(rec_spikes) / pars['T']

  cv_isi = 0.
  if len(rec_spikes) > 3:
    isi = np.diff(rec_spikes)
    cv_isi = np.std(isi) / np.mean(isi)

  print('\n')
  plt.figure(figsize=(15, 10))
  plt.subplot(211)
  plt.text(500, -35, f'Spike rate = {spike_rate:.3f} (sp/s), Mean of Free Mem Pot = {np.mean(v_fmp):.3f}',
           fontsize=16, fontweight='bold', horizontalalignment='center',
           verticalalignment='bottom')
  plt.text(500, -38.5, f'CV ISI = {cv_isi:.3f}, STD of Free Mem Pot = {np.std(v_fmp):.3f}',
           fontsize=16, fontweight='bold', horizontalalignment='center',
           verticalalignment='bottom')

  plt.plot(pars['range_t'], v_fmp, 'r', lw=1.,
           label='Free mem. pot.', zorder=2)
  plt.plot(pars['range_t'], v, 'b', lw=1.,
           label='mem. pot with spk thr', zorder=1, alpha=0.7)
  plt.axhline(pars['V_th'], 0, 1, color='k', lw=1., ls='--',
              label='Spike Threshold', zorder=1)
  plt.axhline(np.mean(v_fmp),0, 1, color='r', lw=1., ls='--',
              label='Mean Free Mem. Pot.', zorder=1)
  plt.ylim(-76, -39)
  plt.xlabel('Time (ms)')
  plt.ylabel('V (mV)')
  plt.legend(loc=[1.02, 0.68])

  plt.subplot(223)
  plt.plot(pars['range_t'][::3], gE[::3], 'r', lw=1)
  plt.xlabel('Time (ms)')
  plt.ylabel(r'$g_E$ (nS)')

  plt.subplot(224)
  plt.plot(pars['range_t'][::3], gI[::3], 'b', lw=1)
  plt.xlabel('Time (ms)')
  plt.ylabel(r'$g_I$ (nS)')

  plt.tight_layout()
```

**Mean-driven and Fluctuation-driven regimes**

If we look at the figure above, we note that when the mean FMP is above spike threshold, the fluctuations in the FMP are rather small, and the neuron spikes in a fairly regular fashion. This regime, where the mean FMP is above the spike threshold, is called **mean-driven regime**.

When the mean FMP is below the spike threshold, the fluctuations in the FMP are large, and the neuron's spikes are driven by these fluctuations. As a consequence, the neuron spikes in more  Poisson-like fashion. This regime, where the mean FMP is below the spike threshold, and spikes are driven by the fluctuations, is called **fluctuation-driven regime**.

## Think!

- How much can you increase the spike pattern variability? Under what condition(s) might the neuron respond with Poisson-type spikes? Note that we injected Poisson-type spikes. (Think of the answer in terms of the ratio of the exc. and inh. input spike rates.)

- Link to the balance of excitation and inhibition: one of the definitions of excitation and inhibition balance is that mean free membrane potential remains constant as excitatory and inhibitory input rates are increased. What do you think happens to the neuron firing rate as we change excitatory and inhibitory rates while keeping the neuron in balance?  See [Kuhn, Aertsen, and Rotter (2004)](https://www.jneurosci.org/content/jneuro/24/10/2345.full.pdf) for much more on this.

```python
# to_remove explanation

"""
Discussion:

1. We can push the neuron to spike almost like a Poisson neuron. Of course given
that there is a refractoriness it will never spike completely like a Poisson process.
Poisson type spike irregularity will be achieved when mean is small (far from the
spike threshold) and fluctuations are large. This will achieved when excitatory
and inhibitory rates are balanced -- i.e. ratio of exc and inh. spike rate is
constant as you vary the inout rate.

2. Firing rate will increase because fluctuations will increase as we increase
exc. and inh. rates. But if synapses are modelled as conductance as opposed to
currents, fluctuations may start decrease at high input rates because neuron time
constant will drop.

""";
```

---
# Section 2: Short-term synaptic plasticity
Above, we modeled synapses with fixed weights. Now we will explore synapses whose weight change in some input conditions.

Short-term plasticity (STP) is a phenomenon in which synaptic efficacy changes over time in a way that reflects the history of presynaptic activity. Two types of STP, with opposite effects on synaptic efficacy, have been experimentally observed. They are known as Short-Term Depression (STD) and Short-Term Facilitation (STF).

The mathematical model (_for more information see [here](http://www.scholarpedia.org/article/Short-term_synaptic_plasticity)_) of STP is based on the concept of a limited pool of synaptic resources available for transmission ($R$), such as, for example, the overall amount of synaptic vesicles at the presynaptic terminals. The amount of presynaptic resource changes in a dynamic fashion depending on the recent history of spikes.

Following a presynaptic spike, (i) the fraction $u$ (release probability) of the available pool to be utilized increases due to spike-induced calcium influx to the presynaptic terminal, after which (ii) $u$ is consumed to increase the post-synaptic conductance. Between spikes, $u$ decays back to zero with time constant $\tau_f$ and $R$ recovers to 1 with time constant $\tau_d$. In summary, the dynamics of excitatory (subscript $E$) STP are given by:

\\

\begin{eqnarray}
&& \frac{du_E}{dt} &=& -\frac{u_E}{\tau_f} + U_0(1-u_E^-)\delta(t-t_{\rm sp}) \\[.5mm]
&& \frac{dR_E}{dt} &=& \frac{1-R_E}{\tau_d} - u_E^+ R_E^- \delta(t-t_{\rm sp}) \qquad (6) \\[.5mm]
&& \frac{dg_E(t)}{dt} &=& -\frac{g_E}{\tau_E} + \bar{g}_E u_E^+ R_E^- \delta(t-t_{\rm sp})
\end{eqnarray}

\\

where $U_0$ is a constant determining the increment of $u$ produced by a spike. $u_E^-$ and $R_E^-$ denote the corresponding values just before the spike arrives, whereas $u_E^+$ refers to the moment right after the spike. $\bar{g}_E$ denotes the maximum excitatory conductane, and $g_E(t)$ is calculated for all spiketimes $k$, and decays over time with a time constant $\tau_{E}$. Similarly, one can obtain the dynamics of inhibitory STP (i.e., by replacing the subscript $E$ with $I$).

The interplay between the dynamics of $u$ and $R$ determines whether the joint effect of $uR$ is dominated by *depression* or *facilitation*. In the parameter regime of $\tau_d \gg \tau_f$  and for large $U_0$, an initial spike incurs a large drop in $R$ that takes a long time to recover; therefore, the synapse is STD-dominated. In the regime of $\tau_d \ll \tau_f$ and for small $U_0$, the synaptic efficacy is increased gradually by spikes, and consequently, the synapse is STF-dominated. This phenomenological model successfully reproduces the kinetic dynamics of depressed and facilitated synapses observed in many cortical areas.

## Exercise 2: Compute $du$, $dR$ and $dg$

As we learned in several previous tutorials, the Euler numerical integration method involves the calculation of each derivative at step $n$:

\\

\begin{eqnarray}
du_E &=& -\frac{u_E[t]}{\tau_f} dt + U_0(1-u_E[t])\cdot \text{sp_or_not[t+dt]} \\
dR_E &=& \frac{1-R_E[t]}{\tau_d} dt - u_E[t+dt]R_E[t]\cdot \text{sp_or_not[t+dt]} \\
dg_E &=& -\frac{g_E[t]}{\tau_{E}} dt + \bar{g}_Eu_E[t+dt]R_E[t]\cdot \text{sp_or_not[t+dt]} \\
\end{eqnarray}

\\

where $\text{sp_or_not}=1$ if there's a spike in the time window $dt$, and $\text{sp_or_not}=0$  otherwise. In addition, note that any spike train generated by our `Poisson_generator` is binary. Then, the values are updated:

\\

\begin{eqnarray}
  u_E[t+dt] &=& u_E[t] + du_E \\
  R_E[t+dt] &=& R_E[t] + dR_E \\
  g_E[t+dt] &=& g_E[t] + dg_E \\
\end{eqnarray}

\\

Similarly, one can obtain the dynamics of inhibitory conductance.

```python
def dynamic_syn(g_bar, tau_syn, U0, tau_d, tau_f, pre_spike_train, dt):
  """
  Short-term synaptic plasticity

  Args:
    g_bar           : synaptic conductance strength
    tau_syn         : synaptic time constant [ms]
    U0              : synaptic release probability at rest
    tau_d           : synaptic depression time constant of x [ms]
    tau_f           : synaptic facilitation time constantr of u [ms]
    pre_spike_train : total spike train (number) input
                      from presynaptic neuron
    dt              : time step [ms]

  Returns:
    u               : usage of releasable neurotransmitter
    R               : fraction of synaptic neurotransmitter resources available
    g               : postsynaptic conductance

  """

  Lt = len(pre_spike_train)
  # Initialize
  u = np.zeros(Lt)
  R = np.zeros(Lt)
  R[0] = 1.
  g = np.zeros(Lt)

  # simulation
  for it in range(Lt - 1):

    #########################################################################
    ## TODO for students: compute du, dx and dg, remove NotImplementedError #
    # Note pre_spike_train[i] is binary, i.e., sp_or_not in the i-th timebin
    # Fill out function and remove
    raise NotImplementedError("Student excercise: compute the STP dynamics")
    #########################################################################
    # Compute du
    du = ...
    u[it + 1] = u[it] + du
    # Compute dR
    dR = ...
    R[it + 1] = R[it] + dR
    # Compute dg
    dg = ...
    g[it + 1] = g[it] + dg

  return u, R, g

# Uncomment this line after completing the dynamic_syn function
# _ = my_illus_STD(Poi_or_reg=False, rate=20., U0=0.5, tau_d=100., tau_f=50.)
```

```python
# to_remove solution
def dynamic_syn(g_bar, tau_syn, U0, tau_d, tau_f, pre_spike_train, dt):
  """
  Short-term synaptic plasticity

  Args:
    g_bar           : synaptic conductance strength
    tau_syn         : synaptic time constant [ms]
    U0              : synaptic release probability at rest
    tau_d           : synaptic depression time constant of x [ms]
    tau_f           : synaptic facilitation time constantr of u [ms]
    pre_spike_train : total spike train (number) input
                      from presynaptic neuron
    dt              : time step [ms]

  Returns:
    u               : usage of releasable neurotransmitter
    R               : fraction of synaptic neurotransmitter resources available
    g               : postsynaptic conductance

  """

  Lt = len(pre_spike_train)
  # Initialize
  u = np.zeros(Lt)
  R = np.zeros(Lt)
  R[0] = 1.
  g = np.zeros(Lt)

  # simulation
  for it in range(Lt - 1):
    # Compute du
    du = -(dt / tau_f) * u[it] + U0 * (1.0 - u[it]) * pre_spike_train[it + 1]
    u[it + 1] = u[it] + du
    # Compute dR
    dR = (dt / tau_d) * (1.0 - R[it]) - u[it + 1] * R[it] * pre_spike_train[it + 1]
    R[it + 1] = R[it] + dR
    # Compute dg
    dg = -(dt / tau_syn) * g[it] + g_bar * R[it] * u[it + 1] * pre_spike_train[it + 1]
    g[it + 1] = g[it] + dg

  return u, R, g

with plt.xkcd():
  _ = my_illus_STD(Poi_or_reg=False, rate=20., U0=0.5, tau_d=100., tau_f=50.)
```

## Section 2.1: Short-term syaptic depression (STD)

### Interactive Demo: STD Explorer with input rate
Below, an interactive demo that shows how Short-term synaptic depression (STD) changes for different firing rates of the presynaptic spike train and how the amplitude synaptic conductance $g$ changes with every incoming spike until it reaches its stationary state.

Does it matter if the neuron fires in a Poisson manner, rather than regularly?

**Note:** `Poi_or_Reg=1`: for *Posisson type* and `Poi_or_Reg=0`: for *regular* presynaptic spikes.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

def my_STD_diff_rate(rate, Poi_or_Reg):
      _ = my_illus_STD(Poi_or_reg=Poi_or_Reg, rate=rate)

_ = widgets.interact(my_STD_diff_rate, rate=(10., 100.1, 5.),
                     Poi_or_Reg=(0, 1, 1))
```

```python
# to_remove explanation

"""
Discussion:

Increasing the input rate, we decrease the synaptic efficacy, i.e., the synaptic
conductance decreases. This is the case for both Poisson or a regular spiking input.
In case of regular spiking, the synaptic conductance reaches a steady state. This
will not happen in the case of Poisson type spikes.
""";
```

### Synaptic depression and presynaptic firing rate
Once, I asked an experimentalist about the experimental values of the PSP amplitude produced by a connection between two neocortical excitatory neurons. She asked: "At what frequency?" I was confused, but you will understand her question, now that you know that PSP amplitude depends on the spike history, and therefore on the spike rate of the presynaptic neuron.

Here, we will study how the ratio of the synaptic conductance corresponding to the first and 10th spikes change as a function of the presynaptic firing rate (experimentalists often take the ratio of first and second PSPs).

For computational efficiency, we assume that the presynaptic spikes are regular. This assumption means that we do not have to run multiple trials.

```python
# @markdown STD conductance ratio with different input rate
# Regular firing rate
input_rate = np.arange(5., 40.1, 5.)
g_1 = np.zeros(len(input_rate))  # record the the PSP at 1st spike
g_2 = np.zeros(len(input_rate))  # record the the PSP at 10th spike

for ii in range(len(input_rate)):
   g_1[ii], g_2[ii] = my_illus_STD(Poi_or_reg=False, rate=input_rate[ii],
                                   plot_out=False, U0=0.5, tau_d=100., tau_f=50)

plt.figure(figsize=(11, 4.5))

plt.subplot(121)
plt.plot(input_rate, g_1, 'm-o', label='1st Spike')
plt.plot(input_rate, g_2, 'c-o', label='10th Spike')
plt.xlabel('Rate [Hz]')
plt.ylabel('Conductance [nS]')
plt.legend()

plt.subplot(122)
plt.plot(input_rate, g_2 / g_1, 'b-o')
plt.xlabel('Rate [Hz]')
plt.ylabel(r'Conductance ratio $g_{10}/g_{1}$')
plt.tight_layout()
```

As we increase the input rate the ratio of the first to tenth spike is increased, because the tenth spike conductance becomes smaller. This is a clear evidence of synaptic depression, as using the same amount of current has a smaller effect on the neuron.

## Section 2.2: Short-term synaptic facilitation (STF)

### Interactive Demo: STF explorer with input rate
Below, we see an illustration of a short-term facilitation example. Take note of the change in the synaptic variables: `U_0`, `tau_d`, and  `tau_f`.

- for STD, `U0=0.5, tau_d=100., tau_f=50.`

- for STP, `U0=0.2, tau_d=100., tau_f=750.`

How does the synaptic conductance change as we change the input rate? What do you observe in the case of a regular input and a Poisson type one?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def my_STD_diff_rate(rate, Poi_or_Reg):
  _ = my_illus_STD(Poi_or_reg=Poi_or_Reg, rate=rate, U0=0.2, tau_d=100., tau_f=750.)

_ = widgets.interact(my_STD_diff_rate, rate=(4., 40.1, 2.),
                     Poi_or_Reg=(0, 1, 1))
```

### Synaptic facilitation and presynaptic firing rate

Here, we will study how the ratio of the synaptic conductance corresponding to the $1^{st}$ and $10^{th}$ spike changes as a function of the presynaptic rate.

```python
# @title STF conductance ratio with different input rates
# Regular firing rate
input_rate = np.arange(2., 40.1, 2.)
g_1 = np.zeros(len(input_rate))  # record the the PSP at 1st spike
g_2 = np.zeros(len(input_rate))  # record the the PSP at 10th spike

for ii in range(len(input_rate)):
   g_1[ii], g_2[ii] = my_illus_STD(Poi_or_reg=False, rate=input_rate[ii],
                                   plot_out=False,
                                   U0=0.2, tau_d=100., tau_f=750.)

plt.figure(figsize=(11, 4.5))

plt.subplot(121)
plt.plot(input_rate, g_1, 'm-o', label='1st Spike')
plt.plot(input_rate, g_2, 'c-o', label='10th Spike')
plt.xlabel('Rate [Hz]')
plt.ylabel('Conductance [nS]')
plt.legend()

plt.subplot(122)
plt.plot(input_rate, g_2 / g_1, 'b-o',)
plt.xlabel('Rate [Hz]')
plt.ylabel(r'Conductance ratio $g_{10}/g_{1}$')
plt.tight_layout()
```

## Think!

Why does the ratio of the first and tenth spike conductance changes in a non-monotonic fashion for synapses with STF, even though it decreases monotonically for synapses with STD?

```python
# to_remove explanation

"""
Discussion:

Because we have a facilitatory synapses, as the input rate increases synaptic
resources released per spike also increase. Therefore, we expect that the synaptic
conductance will increase with input rate. However, total synaptic resources are
finite. And they recover in a finite time. Therefore, at high frequency inputs
synaptic resources are rapidly deleted at a higher rate than their recovery, so
after first few spikes, only a small number of synaptic resources are left. This
results in decrease in the steady-state synaptic conductance at high frequency inputs.
""";
```

---
# Summary

Congratulations! You have just finished the last tutorial of this day. Here, we saw how to model conductance-based synapses and also how to incorporate short-term dynamics in synaptic weights.

We covered the:

- static synapses and how excitation and inhibition affect the neuronal output
- mean- or fluctuation-driven regimes
- short-term dynamics of synapses (both facilitation and depression)

Finally, we incorporated all the aforementioned tools to study how a change in presynaptic firing history affects the synaptic weights!

There are many interesting things that you can try on your own to develop a deeper understanding of biological synapses. A couple of those are mentioned below in the optional boxes -- if you have time.

But now it is time to explore another important feature of biological synapses, i.e., spike timing dependent synaptic plasticity (go to the next tutorial).

---
# Bonus 1: Conductance-based LIF with STP

Previously, we looked only at how the presynaptic firing rate affects the presynaptic resource availability and thereby the synaptic conductance. It is straightforward to imagine that, while the synaptic conductances are changing, the output of the postsynaptic neuron will change as well.

So, let's put the STP on synapses impinging on an LIF neuron and see what happens.

```python
# @title Function for conductance-based LIF neuron with STP-synapses

def run_LIF_cond_STP(pars, I_inj, pre_spike_train_ex, pre_spike_train_in):
  """
  conductance-based LIF dynamics

  Args:
    pars               : parameter dictionary
    I_inj              : injected current [pA]
                         The injected current here can be a value or an array
    pre_spike_train_ex : spike train input from presynaptic excitatory neuron (binary)
    pre_spike_train_in : spike train input from presynaptic inhibitory neuron (binary)

  Returns:
    rec_spikes : spike times
    rec_v      : mebrane potential
    gE         : postsynaptic excitatory conductance
    gI         : postsynaptic inhibitory conductance

  """

  # Retrieve parameters
  V_th, V_reset = pars['V_th'], pars['V_reset']
  tau_m, g_L = pars['tau_m'], pars['g_L']
  V_init, V_L = pars['V_init'], pars['E_L']
  gE_bar, gI_bar = pars['gE_bar'], pars['gI_bar']
  U0E, tau_dE, tau_fE = pars['U0_E'], pars['tau_d_E'], pars['tau_f_E']
  U0I, tau_dI, tau_fI = pars['U0_I'], pars['tau_d_I'], pars['tau_f_I']
  VE, VI = pars['VE'], pars['VI']
  tau_syn_E, tau_syn_I = pars['tau_syn_E'], pars['tau_syn_I']
  tref = pars['tref']

  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  nE = pre_spike_train_ex.shape[0]
  nI = pre_spike_train_in.shape[0]

  # compute conductance Excitatory synapses
  uE = np.zeros((nE, Lt))
  RE = np.zeros((nE, Lt))
  gE = np.zeros((nE, Lt))
  for ie in range(nE):
    u, R, g = dynamic_syn(gE_bar, tau_syn_E, U0E, tau_dE, tau_fE,
                          pre_spike_train_ex[ie, :], dt)

    uE[ie, :], RE[ie, :], gE[ie, :] = u, R, g

  gE_total = gE.sum(axis=0)

  # compute conductance Inhibitory synapses
  uI = np.zeros((nI, Lt))
  RI = np.zeros((nI, Lt))
  gI = np.zeros((nI, Lt))
  for ii in range(nI):
    u, R, g = dynamic_syn(gI_bar, tau_syn_I, U0I, tau_dI, tau_fI,
                          pre_spike_train_in[ii, :], dt)

    uI[ii, :], RI[ii, :], gI[ii, :] = u, R, g

  gI_total = gI.sum(axis=0)

  # Initialize
  v = np.zeros(Lt)
  v[0] = V_init
  Iinj = I_inj * np.ones(Lt)  # ensure I has length Lt

  # simulation
  rec_spikes = []  # recording spike times
  tr = 0.
  for it in range(Lt - 1):
    if tr > 0:
      v[it] = V_reset
      tr = tr - 1
    elif v[it] >= V_th:  # reset voltage and record spike event
      rec_spikes.append(it)
      v[it] = V_reset
      tr = tref / dt

    # calculate the increment of the membrane potential
    dv = (dt / tau_m) * (-(v[it] - V_L) \
                         - (gE_total[it + 1] / g_L) * (v[it] - VE) \
                         - (gI_total[it + 1] / g_L) * (v[it] - VI) + Iinj[it] / g_L)

    # update membrane potential
    v[it+1] = v[it] + dv

  rec_spikes = np.array(rec_spikes) * dt

  return v, rec_spikes, uE, RE, gE, RI, RI, gI

print(help(run_LIF_cond_STP))
```

## Simulation of a postsynaptic neuron with STP synapses driven by Poisson type spike trains

Here we have assumed that both excitatory and inhibitory synapses show short-term depression. Change the nature of synapses and study how spike pattern variability changes.
In the interactive demo, `tau_d = 500*tau_ratio (ms)` and  `tau_f = 300*tau_ratio (ms)`.

You should compare the output of this neuron with what you observed in the previous tutorial when synapses were assumed to be static.

_Note: it will take slightly longer time to run each case_

### Interactive Demo: LIF with STP Explorer

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def LIF_STP(tau_ratio):
  pars = default_pars(T=1000)
  pars['gE_bar'] = 1.2 * 4  # [nS]
  pars['VE'] = 0.           # [mV]
  pars['tau_syn_E'] = 5.    # [ms]
  pars['gI_bar'] = 1.6 * 4  # [nS]
  pars['VI'] = -80.         # [ms]
  pars['tau_syn_I'] = 10.   # [ms]

  # here we assume that both Exc and Inh synapses have synaptic depression
  pars['U0_E'] = 0.45
  pars['tau_d_E'] = 500. * tau_ratio  # [ms]
  pars['tau_f_E'] = 300. * tau_ratio  # [ms]

  pars['U0_I'] = 0.45
  pars['tau_d_I'] = 500. * tau_ratio  # [ms]
  pars['tau_f_I'] = 300. * tau_ratio  # [ms]

  pre_spike_train_ex = Poisson_generator(pars, rate=15, n=80)
  pre_spike_train_in = Poisson_generator(pars, rate=15, n=20)  # 4:1

  v, rec_spikes, uE, RE, gE, uI, RI, gI = run_LIF_cond_STP(pars, 0,
                                                           pre_spike_train_ex,
                                                           pre_spike_train_in)

  t_plot_range = pars['range_t'] > 200

  plt.figure(figsize=(11, 7))
  plt.subplot(211)
  plot_volt_trace(pars, v, rec_spikes)

  plt.subplot(223)
  plt.plot(pars['range_t'][t_plot_range], gE.sum(axis=0)[t_plot_range], 'r')
  plt.xlabel('Time (ms)')
  plt.ylabel(r'$g_E$ (nS)')

  plt.subplot(224)
  plt.plot(pars['range_t'][t_plot_range], gI.sum(axis=0)[t_plot_range], 'b')
  plt.xlabel('Time (ms)')
  plt.ylabel(r'$g_I$ (nS)')

  plt.tight_layout()

_ = widgets.interact(LIF_STP, tau_ratio=(0.2, 1.1, 0.2))
```

When we vary the tau_ratio we are increasing `tau_f` and `tau_d` i.e. by increasing `tau_ratio` we are increasing the synaptic depression. The effect is same on both Exc and Inh conductances.
This is visible as a clear decrease in the firing rate of the neuron from 300-400ms onwards.

Not much happens in the beginning because synaptic depression takes some time to become visible.

It is curious that while both excitatory and inhibitory conductances have depressed but output firing rate has still decreased.

There are two explanations of this:
1. excitation has depressed more than the inhibition from their starting values.
2. because synaptic conductances have depressed, membrane fluctuation size has decreased.

Which is more likely reason? Think.

---
# Bonus 2: STP Synapse Parameter Exploration

Vary the parameters of the above simulation and observe the spiking pattern of the postsynaptic neuron.
Will the neuron show higher irregularity if the synapses have STP? If yes, what should be the nature of STP on static and dynamic synapses, respectively?

Calculate the $CV_{\rm ISI}$ for different `tau_ratio` after simulating the LIF neuron with STP (Hint:`run_LIF_cond_STP` help you understand the irregularity).

## Functional implications of short-term dynamics of synapses
As you have seen above, if the firing rate is stationary, the synaptic conductance quickly reaches a fixed point. On the other hand, if the firing rate transiently changes, synaptic conductance will vary -- even if the change is as short as a single inter-spike-interval. Such small changes can be observed in a single neuron when input spikes are regular and periodic. If the input spikes are Poissonian, then one may have to perform an average over several neurons.

_Come up with other functions that short-term dynamics of synapses can be used to implement and implement them._


# 第十九讲：微分动态规划及线性二次型高斯

在继续学习之前，我们先来回顾一下[上一讲](chapter18.ipynb)的内容：
* 其目标是$\displaystyle\max_a\mathrm E\left[R^{(0)}(s_0,a_0)+R^{(1)}(s_1,a_1)+\cdots+R^{(T)}(s_T,a_T)\right]$；
* 之后我们应用了动态规划算法：
    $$
    \begin{align}
    V_T^*(s)&=\max_{a_T}R(s_T,a_T)\tag{1}\\
    V_t^*(s)&=\max_a\left(R(s,a)+\sum_{s'}P_{sa}^{(t)}\left(s'\right)V_{t+1}^*\left(s'\right)\right)\tag{2}\\
    \pi_t^*(s)&=\arg\max_a\left(R(s,a)+\sum_{s'}P_{sa}^{(t)}\left(s'\right)V_{t+1}^*\left(s'\right)\right)\tag{3}
    \end{align}
    $$
    其中$(1)$式是有限时域MDP的最后一步；得出最后一步后就可以使用$(2)$式向后递推每一个$V_{T-1},\cdots,V_{0}$；最后，在使用$\arg\max$对每一步价值函数最大化的部分做运算，就可以得到每一步的最优策略。
* 接下来我们介绍了一个具体的LQR问题：
    * 状态为$S\in\mathbb R^n$；
    * 动作为$A\in\mathbb R^d$；
    * 将下一步作为关于当前状态及动作的函数$s_{t+1}=A_ts_t+B_ta_t+w_t$，此处的$w_t\sim\mathcal N(0,\varSigma_w)$是一个期望为零、协方差为$\varSigma_w$的高斯噪音项。
* 上面定义中有了关于当前状态及动作的函数$s_{t+1}=f(s_t,a_t)$，我们就可以选择一个系统长时间保持的状态（系统通常都会都处于这种状态）$(\bar s_t,\bar a_t)$，在该点使用线性函数近似非线性函数$s_{t+1}=f(s_t,a_t)$，进而得到$\displaystyle s_{t+1}\approx f(\bar s_t,\bar a_t)+(\nabla_sf(\bar s_t,\bar a_t))^T(s_t-\bar s_t)+(\nabla_af(\bar s_t,\bar a_t))^T(a_t-\bar a_t)$。最终能够在$(\bar s_t,\bar a_t)$附近得到一个线性函数$s_{t+1}=A_ts_t+B_ta_t$。
* LQR的奖励函数为$\displaystyle R^{(t)}(s_t,a_t)=-\left(s_t^TU_ts_t+a^TV_ta_t\right)$，其中矩阵$U,V$是半正定的，所以这个奖励函数总是非正数；
* 对LQR使用上面的动态规划得到求解方法：
    * 使用$\varPhi_t=-U_t,\varPsi_t=0$初始化$V_T^*$；
    * 向后递推计算$\varPhi_t,\varPsi_t$，使用离散时间Riccati方程，将其当做关于$\varPhi_{t+1},\varPsi_{t+1}$的线性函数；（依次求出$t=T-1,T-2,\cdots,0$。）
        $$
        \begin{align}
        \varPhi_t&=A_t^T\left(\varPhi_{t+1}-\varPhi_{t+1}B_t\left(B_t^T\varPhi_{t+1}B_t-V_t\right)^{-1}B_t\varPhi_{t+1}\right)A_t-U_t\\
        \varPsi_t&=-\mathrm{tr}\varSigma_w\varPhi_{t+1}+\varPsi_{t+1}
        \end{align}
        $$
    * 计算$\displaystyle L_t=\left(B_t^T\varPhi_{t+1}B_t-V_t\right)^{-1}B_t^T\varPhi_{t+1}A_t$，这是一个关于$\varPhi_{t+1},\varPsi_{t+1}$的函数；
    * 计算$\pi^*(s_t)=L_ts_t$，这是一个关于$s_t$的线性函数，系数是$L_t$。

  这个算法中有趣的一点是，$L_t$不依靠$\varPsi_t$，$\varPhi_t$也不依赖于$\varPsi_t$。所以，即使我们从不考虑$\varPsi$也不会影响最终的结果。另一个有趣的地方在于$\varSigma_w$只在$\varPsi_t$中出现，再看LQR定义中的$s_{t+1}=A_ts_t+B_ta_t+w_t$可以得出，即使在不知道噪音项协方差的前提下，也可以正确的求出最优策略（但不要忘了最优价值函数是受$\varPsi$影响的）。最后，这两点都是对这个特殊LQR模型（非线性动力系统）独有的，一旦我们对模型做出改变，则这两个特性将不复存在，也就是只有在这个例子中才有“最优策略不受噪音项协方差影响”。在后面讨论Kalman滤波的时候会用到这个性质。

## 8. 强化学习算法的调试

我们在[第十一讲](chapter11.ipynb)中，在关于机器学习算法的调试介绍过直升机模型提到：

1. 搭建一个遥控直升机模拟器，主要是对状态转换概率$P_{sa}$进行建模（比如通过学习物理知识，依照空气动力学原理编写模拟器；或者也可以收集大量试验数据，通过拟合这些数据一个线性或非线性模型，以得到一个用当前状态、当前操作表示下一步状态的函数）；
2. 选择奖励函数，比如$R(s)=\left\lVert s-s_\mathrm{desired}\right\rVert^2$（$s$代表直升机当前的位置，这里示例的函数表示直升机实际位置与期望位置的平方误差，我们在[上一讲](chapter18.ipynb)的LQR中已经使用过这个奖励函数了）；
3. 在模拟器中运行强化学习算法控制直升机并尝试最大化奖励函数，$\mathrm E\left[R(s_0)+R(s_1)+\cdots+R(s_T)\right]$，进而求得对应的策略$\pi_\mathrm{RL}$。

现在，假设我们已经按照上述步骤得到了控制模型，但是发现该模型比人类直接操作差很多，接下来应该怎么办呢？我们可能会：
* 改进模拟器（也许模拟器所用的模型不是线性或非线性的；也许需要收集更多数据来拟合模型；也许需要调整拟合模型时所使用的特征值）；
* 修改奖励函数$R$（也许它不只是一个二次函数）；
* 改进学习算法（也许RL并不适合这个问题；也许需要对状态进行更精细的描述；也许在价值函数近似过程中使用的特征值需要调整）；

对于调整算法这样的任务，我们有太多的选择方向，如果选择错误，则将会浪费大量的时间精力。

接下来我们假设：
1. 直升机模拟器足够精确；
2. RL算法能够在模拟器中正确的控制飞机，因此算法正确的最大化了预期总收益$V_t^{\pi_\mathrm{RL}}(s_0)=\mathrm E\left[R(s_0)+R(s_1)+\cdots+R(s_T)\mid\pi_\mathrm{RL},s_t=s_0\right]$；
3. 最大化预期总收益与模型正确的操作飞机强相关。

如果上述三条假设均正确，则可以推出$\pi_\mathrm{RL}$能够控制好真正的直升机。

于是，我们可以**依次**做出如下诊断：
1. 首先，如果$\pi_\mathrm{RL}$在模拟器中飞的很好，但是在实际飞机上却表现不好，则说明是模拟器的问题；
2. 然后，用$\pi_\mathrm{human}$表示人类操纵者在操纵直升机时使用的策略，比较人类和算法的策略所对应的价值函数，如果$V^{\pi_\mathrm{RL}}\lt V^{\pi_\mathrm{human}}$，则说明是RL算法的问题。（算法并没有成功的最大化预期总收益。）
3. 最后，如果经比较发现$V^{\pi_\mathrm{RL}}\gt V^{\pi_\mathrm{human}}$，则说明问题出在奖励函数上。（最大化这个奖励函数并没有使得飞行水平提高。）

以上仅是一个强化学习调试的例子，因为我们恰好找到了一个极优秀的直升机操纵者，如果没有的话则需要想出别的调试方法。在通常的问题中，我们都需要自己找出针对问题的有效的调试方法。

## 9. 微分动态规划（DDP: Differential Dynamic Programming）

继续使用遥控直升机的例子，假设我们已经有模拟器并可以通过模拟器知道$s_{t+1}=f(s_t,a_t)$（即下一个状态是一个关于当前状态及动作的函数），而且这个模拟器是非线性、确定性的，噪音项也比较小。我们现在想要让直升机飞出一些特定轨迹，于是：
1. 先写出这个轨迹：$(\bar s_0,\bar a_0),(\bar s_1,\bar a_1),\cdots,(\bar s_T,\bar a_T)$，这也称为**标准轨迹（nominal trajectory）**；（这个轨迹也可能来自一个很粗糙的控制算法，但是虽然粗糙，却也是描述的我们想要做出的动作，比如在空中做$90^\circ$转弯之类的动作。）
2. 然后，我们在这个轨迹附近线性化$f$得到：$\displaystyle s_{t+1}\approx f(\bar s_t,\bar a_t)+(\nabla_sf(\bar s_t,\bar a_t))^T(s_t-\bar s_t)+(\nabla_af(\bar s_t,\bar a_t))^T(a_t-\bar a_t)$，也就是说能够得到一个线性函数$s_{t+1}=A_ts_t+B_ta_t$。我们这是在课程中第一次使用LQR来处理有限时域上的非平稳的动态问题，尤其要注意的是这里的$A_t,B_t$是依赖于时间的（也就是这是一系列不相等的矩阵）；（即使标准轨迹来自一套很糟糕的控制，但我们仍然希望系统在$t$时刻的状态和动作能够近似于这个糟糕的轨迹。也许这个控制算法做的工作很马虎，但毕竟这个轨迹描述了我们想要得到的动作，也就是希望$(s_t,a_t)\approx(\bar s_t,\bar a_t)$。）
3. 有了线性模型，再使用LQR计算最优策略$\pi_t$，于是我们就会得到一个更好的策略；
4. 在模拟器中按照新的策略实现一个新的轨迹，也就是：
    * 用$\bar s_0$初始化模拟器；
    * 用刚学到的$\pi_t$控制每一步的动作$a_t=\pi_t(\bar s_t)$；
    * 最终得到一套新的轨迹$\bar s_{t+1}=f(\bar s_t,\bar a_t)$
5. 得到新的轨迹后，我们就可以继续重复上面的操作，不停的优化这个轨迹（也就是重复步骤2-5）。

在实践中能够知道，这是一个非常有效的算法，DDP实际上是一种局部搜索算法，在每次迭代中，找到一个稍好的点进行线性化，进而得到一个稍好的策略，重复这个动作最终就可以得到一个比较好的策略了。（“微分”指的就是每次迭代我们都会选择一个点并通过求导做线性化。）

## 10. 线性二次型高斯（LQG: Linear Quadratic Gaussian）

### 10.1 Kalman滤波（Kalman Filter）

现在介绍另一种类型的MDP，在这种MDP中，我们并不能直接观察到每一步的状态（在前面的MDP中我们都会假设系统的状态已知，于是可以计算出策略，它是一个关于状态的函数；在LQR中我们也是计算$L_ts_t$，状态都是已知的）。

我们暂时先不谈控制，来说说普通的不能观察到状态信息的动态系统。举个例子，我们对“使用雷达追踪飞行器”的过程进行极其简化的建模：线性模型$s_{t+1}=As_t+w_t$，其中$s_t=\begin{bmatrix}x_t\\\dot x_t\\y_t\\\dot y_t\end{bmatrix},\ A=\begin{bmatrix}1&1&0&0\\0&0.9&0&0\\0&0&1&1\\0&0&0&0.9\end{bmatrix}$（这是一个非常简化的模型，可以理解为一架在2D空间中移动的飞机）。

在这个简化模型中我们无法观察到每一步的状态，但假设我们可以观察到另一个量——$y_t=Cs_t+v_t,\ v_t\sim\mathcal N(0,\varSigma_v)$：比如$C=\begin{bmatrix}1&0&0&0\\0&0&1&0\end{bmatrix}$，则有$Cs_t=\begin{bmatrix}x_t\\y_t\end{bmatrix}$。这个量可能是雷达追踪到的飞行器的位置（只能得到位置相关的状态，而不能得到加速度相关的状态）。比如说在$\mathrm{xOy}$平面上，有一个雷达位于$x$轴某点，其视角沿$y$轴正方向大致对着飞行器，并依次观察到一系列飞行器移动过程中的不连续的点，因为雷达视角沿$y$轴正半轴方向，所以得到的点的坐标的纵坐标的噪音（方差）应该大于横坐标的方差，可以说这些点分别位于一个个小的高斯分布中，而这些高斯分布的协方差矩阵应该型为$\varSigma=\begin{bmatrix}a&0\\0&b\end{bmatrix},\ a\gt b$（即因为在$y$轴方向上的观测结果更容易出现偏差，所以等高线图中椭圆的长轴会沿着$y$轴方向。）而我们想要做的就是通过观测到的一系列坐标估计飞行器在每个观测点的状态$P(s_t\mid y_1,\cdots,y_t)$。

$s_0,\cdots,s_t;\ y_0,\cdots,y_t$组成了一个高斯分布的联合分布（多元正态分布），可以用$z=\begin{bmatrix}s_0\\\vdots\\s_t\\y_0\\\vdots\\y_t\end{bmatrix},\ z\sim\mathcal N(\mu,\varSigma)$表示，再结合[第十三讲](chapter13.ipynb)中了解的多元正态分布的边缘分布与条件分布的计算方法，我们就可以对$P(s_t\mid y_1,\cdots,y_t)$进行求解。虽然这个思路在概念上是可行的，但是在实际中跟踪一个飞行器会得到成千上万个位置，于是就会有一个巨大的协方差矩阵（期望向量和协方差矩阵都会随着时间、步数的增长呈线性增长，在计算过程中需要用到的边缘分布、条件分布都可能会使用协方差的逆，而计算协方差的逆的复杂度是矩阵规模的平方级别），所以对于计算来说可行性较低。

于是我们引入**Kalman滤波（Kalman Filter）**算法来解决这个效率问题。这个算法实际上是一个隐式马尔可夫模型，使用递推的方式计算，下面写出算法的步骤，算法的思路可以在[Hidden Markov Models](http://cs229.stanford.edu/section/cs229-hmm.pdf)中找到：
* 第一步称为**预测（predict）**步骤，使用$P(s_t\mid y_1,\cdots,y_t)$对$P(s_{t+1}\mid y_1,\cdots,y_t)$进行预测；
  * 有$s_t\mid y_0,\cdots,y_t\sim\mathcal N\left(s_{t\mid t},\varSigma_{t\mid t}\right)$；
  * 则有$s_{t+1}\mid y_0,\cdots,y_t\sim\mathcal N\left(s_{t+1\mid t},\varSigma_{t+1\mid t}\right)$；
  * 其中$\begin{cases}s_{t+1\mid t}&=As_{t\mid t}\\ \varSigma_{t+1\mid t}&=A\varSigma_{t\mid t}A^T+\varSigma_t\end{cases}$；
  * 需要说明的是，我们使用诸如$s_t,y_t$表示模型的真实状态（如为观测到的实际状态，已观测到的实际位置），而使用$s_{t\mid t},s_{t+1\mid t},\varSigma_{t\mid t}$表示由计算得到的值；
* 第二步称为**更新（update）**步骤，使用上一步预测的$P(s_{t+1}\mid y_1,\cdots,y_t)$更新$P(s_{t+1}\mid y_1,\cdots,y_{t+1})$，也就是有了预测结果之后再将对应的样本纳入模型：
  * 其中$s_{t+1\mid t+1}=s_{t+1\mid t}+K_{t+1}\cdot\left(y_{t+1}-Cs_{t+1\mid t}\right)$；
  * 而$K_{t+1}=\varSigma_{t+1\mid t}C^T\left(C\varSigma_{t+1\mid t}C^T+\varSigma_t\right)^{-1}$；
  * $\varSigma_{t+1\mid t+1}=\varSigma_{t+1\mid t}+\varSigma_{t+1\mid t}\cdot C^T\left(C\varSigma_{t+1\mid t}C^T+\varSigma_t\right)^{-1}C\varSigma_{t+1\mid t}$

  Kalman滤波算法的复杂度比计算协方差矩阵的逆低很多，因为采用了递推的计算过程，每当步骤增加，则只需多执行一步迭代，而且无需保留太多数据在内存中，相比于求一个巨大矩阵的逆而言，算法复杂度低了很多：
  $$
  \require{AMScd}
  \begin{CD}
  y_1 @. y_2 @. y_3 \\
  @VVV @VVV @VVV \\
  P\left(s_1\mid y_1\right) @>>> P\left(s_2\mid y_1,y_2\right) @>>> P\left(s_3\mid y_1,y_2,y_3\right) @>>> \cdots
  \end{CD}
  $$
  从这个草图可以看出来，当计算$P\left(s_3\mid y_1,y_2,y_3\right)$时，我们并不需要$y_1,y_2,P\left(s_1\mid y_1\right)$，每步计算只需要知道最新的观测值和上一步得出的概率估计即可。

### 10.2 LQG

将Kalman滤波与LQR结合起来就可以得到**线性二次高斯（LQG: Linear Quadratic Gaussian）**线性二次高速算法。在LQR问题中，我们将动作$a_t$加回到模型中：设非线性动力学系统为$s_{t+1}=As_t+Ba_t+w_t,\ w\sim\mathcal N(0,\varSigma_w)$，而且我们无法直接观测到状态，只能观测到$y_t=Cs_t+v_t,\ v\sim\mathcal N(0,\varSigma_v)$。现在使用Kalman滤波估计状态：
* 假设初始状态为$s_{0\mid0}=s_0,\varSigma_{0\mid0}=0$；（如果不知道确切初始状态，测可以大致估计初始状态$s_0\sim\mathcal N\left(s_{0\mid0},\varSigma_{0\mid0}\right)$，即使用初始状态的平均值和协方差的估计。）
* 预测步骤，$\begin{cases}s_{t+1\mid t}&=As_{t\mid t}+Ba_t\\\varSigma_{t+1\mid t}&=A\varSigma_{t\mid t}A^T+\varSigma_t\end{cases}$。如此，我们就“解决”为了状态观测不到的问题（这里的$A,B$都是已知项）；
* 控制步骤，按照LQR算法中的最优动作$a_t=L_ts_t$计算其中的$L_t$，并暂时忽略观测不到状态的问题；得到$L_t$后，选择$a_t=L_ts_{t\mid t}$作为最优动作（换句话说，因为不知道状态，所以我们对状态的最佳估计就是$s_{t\mid t}$）。

这个算法实际上是先设计了一个状态估计算法，然后再使用了一个控制算法，把两个算法结合起来就得到了LQG。


```python
%matplotlib inline
```

```python
!pip install pygments
import pygments
!pip install flit
import flit
!pip install scikit-learn
!pip install scikit-aero
from skaero.atmosphere import coesa
from skaero.gasdynamics import isentropic, shocks
!pip install numpy
import numpy as np
```

    Requirement already satisfied: pygments in /srv/conda/envs/notebook/lib/python3.6/site-packages (2.10.0)
    Requirement already satisfied: flit in /srv/conda/envs/notebook/lib/python3.6/site-packages (3.4.0)
    Requirement already satisfied: requests in /srv/conda/envs/notebook/lib/python3.6/site-packages (from flit) (2.26.0)
    Requirement already satisfied: tomli in /srv/conda/envs/notebook/lib/python3.6/site-packages (from flit) (1.2.2)
    Requirement already satisfied: flit_core>=3.4.0 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from flit) (3.4.0)
    Requirement already satisfied: docutils in /srv/conda/envs/notebook/lib/python3.6/site-packages (from flit) (0.18)
    Requirement already satisfied: tomli-w in /srv/conda/envs/notebook/lib/python3.6/site-packages (from flit) (0.4.0)
    Requirement already satisfied: charset-normalizer~=2.0.0 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from requests->flit) (2.0.0)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from requests->flit) (1.26.7)
    Requirement already satisfied: idna<4,>=2.5 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from requests->flit) (3.1)
    Requirement already satisfied: certifi>=2017.4.17 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from requests->flit) (2021.5.30)
    Requirement already satisfied: scikit-learn in /srv/conda/envs/notebook/lib/python3.6/site-packages (0.24.2)
    Requirement already satisfied: scipy>=0.19.1 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-learn) (1.5.3)
    Requirement already satisfied: joblib>=0.11 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-learn) (1.1.0)
    Requirement already satisfied: numpy>=1.13.3 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-learn) (1.19.5)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-learn) (3.0.0)
    Requirement already satisfied: scikit-aero in /srv/conda/envs/notebook/lib/python3.6/site-packages (0.1)
    Requirement already satisfied: numpy in /srv/conda/envs/notebook/lib/python3.6/site-packages (1.19.5)

```python
!pip install numba
```

    Requirement already satisfied: numba in /srv/conda/envs/notebook/lib/python3.6/site-packages (0.53.1)
    Requirement already satisfied: llvmlite<0.37,>=0.36.0rc1 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from numba) (0.36.0)
    Requirement already satisfied: numpy>=1.15 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from numba) (1.19.5)
    Requirement already satisfied: setuptools in /srv/conda/envs/notebook/lib/python3.6/site-packages (from numba) (58.0.4)

```python
A = np.array([1, 3, 7, 21])
output = np.std(A)
print(output)
print(output**4)
print(output/2)
print(output%2)
```

    7.810249675906654
    3720.999999999999
    3.905124837953327
    1.810249675906654

```python
import matplotlib.pyplot as plt
from sklearn.covariance import EmpiricalCovariance, MinCovDet
!pip install sims
import sims
```

    Requirement already satisfied: sims in /srv/conda/envs/notebook/lib/python3.6/site-packages (2.0.0)
    Requirement already satisfied: scipy in /srv/conda/envs/notebook/lib/python3.6/site-packages (from sims) (1.5.3)
    Requirement already satisfied: scikit-image in /srv/conda/envs/notebook/lib/python3.6/site-packages (from sims) (0.17.2)
    Requirement already satisfied: xarray in /srv/conda/envs/notebook/lib/python3.6/site-packages (from sims) (0.16.2)
    Requirement already satisfied: matplotlib in /srv/conda/envs/notebook/lib/python3.6/site-packages (from sims) (3.3.4)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from matplotlib->sims) (2.4.7)
    Requirement already satisfied: numpy>=1.15 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from matplotlib->sims) (1.19.5)
    Requirement already satisfied: python-dateutil>=2.1 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from matplotlib->sims) (2.8.2)
    Requirement already satisfied: kiwisolver>=1.0.1 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from matplotlib->sims) (1.3.1)
    Requirement already satisfied: cycler>=0.10 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from matplotlib->sims) (0.11.0)
    Requirement already satisfied: pillow>=6.2.0 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from matplotlib->sims) (8.3.2)
    Requirement already satisfied: six>=1.5 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from python-dateutil>=2.1->matplotlib->sims) (1.16.0)
    Requirement already satisfied: networkx>=2.0 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-image->sims) (2.6.3)
    Requirement already satisfied: imageio>=2.3.0 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-image->sims) (2.9.0)
    Requirement already satisfied: tifffile>=2019.7.26 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-image->sims) (2020.10.1)
    Requirement already satisfied: PyWavelets>=1.1.1 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from scikit-image->sims) (1.1.1)
    Requirement already satisfied: setuptools>=38.4 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from xarray->sims) (58.0.4)
    Requirement already satisfied: pandas>=0.25 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from xarray->sims) (1.1.5)
    Requirement already satisfied: pytz>=2017.2 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from pandas>=0.25->xarray->sims) (2021.3)

```python
!pip install geometer
from geometer import *
```

    Requirement already satisfied: geometer in /srv/conda/envs/notebook/lib/python3.6/site-packages (0.2.3)
    Requirement already satisfied: numpy<1.20,>=1.15 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from geometer) (1.19.5)
    Requirement already satisfied: sympy<=1.7,>=1.3 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from geometer) (1.7)
    Requirement already satisfied: mpmath>=0.19 in /srv/conda/envs/notebook/lib/python3.6/site-packages (from sympy<=1.7,>=1.3->geometer) (1.2.1)

```python
from numba import jit
import random
```


## 9-1. 古典エラー
データ蓄積の方法は時間ともに初期化されてしまう揮発性のものと、原則としてほぼ劣化しない不揮発性のものに分類される。ここでは揮発性のメモリの一種であるDRAMのメモリセルを考える。
現代の計算機は0,1のバイナリ情報を、コンデンサに蓄えらえれた電子の量が多いか少ないかというアナログな連続量で表現する。（通信ならある周波数帯や時間区間に含まれる光子の数で表現する。）
ここではエネルギーが蓄えられた状態を1、エネルギーが完全に放出された状態を0とする。

こうしたアナログな量は、外部との相互作用により常に一定値にいることはない。電子は時間とともにトランジスタのリーク電流などによってあるレートで放出される。
従って、1の状態はしばらくたつと0の状態に変化してしまう。この変化は現代のデバイスではおよそ秒未満の無視できないスケールで起きる。
これは「操作の有無に寄らず時間の変化によって一定レートで起きるエラー」である。
こうしたエラーは古典デバイスでは「0,1が判定不能になるより十分前に、一定時間おきに残電子量を読み出し、その数に応じて充電しなおす」ことで訂正されている。
この操作はリフレッシュと呼ばれ、ユーザが意図せずとも定期的に行われる。当然、電源が落ちるとリフレッシュは行われなくなる。このことを指しDRAMを揮発メモリと呼ぶ。

リフレッシュが行われリークによる影響が無視できるとき、次に重要となるエラーの要因は中性子線によるトランジスタの誤動作である。
中性子線は宇宙線の一種であり、金属などを貫通するために防護が困難である。この中性子線がトランジスタを通過するとき、生じた電荷がトランジスタを誤動作させることがある。
従って、中性子線エラーは「何らかの操作がトリガーとなり、単発的に起きるエラー」である。
読み出しタイミングでないときトランジスタが動作しゲートが解放されると、コンデンサの状態はbit lineの状態に合わせて変化し意図しないものになってしまう。
このエラーによってビットが反転される確率は一般のユーザには無視できるほど小さいが、規模が非常に大きな計算では無視できなくなる。
仮に単位ビット当たりに単位時間で中性子線が衝突し誤動作する確率を$p$とすると、$n$ビットのメモリに$t$秒間の間１ビットも変化しない確率は
$q = (1-p)^{nt}$である。$n=10^{12}$で$t=10^{3}$とし、$q=0.99$を実現するためには、$p\sim 10^{-18}$でなければならない。

中性子線エラーのリークとの大きな違いは、リークはすぐさま確認すれば0,1の中間的な状態にあるため復帰が可能なものの、中性子線は生じた後にはもとはどの状態であったか知ることができない点にある。
このため、中性子線エラーが問題となる領域での応用では、誤り訂正機能の付いたECCメモリが利用される。ECCメモリでは後述の誤り訂正を行い、1bitまでのエラーを小さなオーバーヘッドで訂正する。

### 古典エラー訂正: 多数決
最も単純な符号は多数決である。多数決では個々のビットを$d$倍にコピーする。この時、$k$ビットの情報を$n:=dk$ビットで表現することになる。多数決の引き分けを防ぐため、$d$は奇数とする。
現実に存在する$n$ビットのことを物理ビット、実態として表現している$k$ビットのことを論理ビットと呼ぶ。

多数決では以下のように誤りの検知と訂正を行う。個々の論理ビットについて複製した$d$ビットの情報を読み出し、0,1のどちらが多いかを数える。そして、頻度が高かった値がその論理ビットの値であったと判定する。初期の値がどうあれ、符号化された状態は全ての値が同じであるから、$d$ビットの値が一つでも一致していない場合、何らかのエラーが生じていることが確信できる。従って$n$ビットが丸ごと反転しない限り必ず誤りを検知することができる。

誤りを訂正したい場合は半数以上のビットが正しい値を保持していれば正しい結果を得ることができる。それぞれのビットに1より十分小さい確率$p$でエラーが生じるとすると、半数以上のビットがエラーを起こす確率はおよそ$p^{\lfloor d/2 \rfloor+1}$である。
従って、$d$を2増やすごとに、多数決が失敗する確率はオーダーで$p$倍小さくなることがわかる。
最も単純な$k=1$の場合をシミュレートするコードを書くと以下のようになる。

```python
import numpy as np

data = 1
d = 31
p = 0.01

print("original bit: {}".format(data))
state = np.repeat(data, d)
is_flipped = (np.random.rand(d)<p).astype(np.int)
state = (state + is_flipped)%2
if np.sum(state==0) < np.sum(state==1):
    majority = 1
else:
    majority = 0

print("decoded bit: {}".format(majority))
```

    original bit: 1
    decoded bit: 1

上記はランダムに成功不成功が変わるが、成功する確率は確率$p$に関する二項分布の累積として知ることができる。
横軸を各ビットのエラー確率、縦軸を復号の成功確率として、様々な奇数の$d$についてプロットすると以下のようになる。

```python
import numpy as np

d = 31
fail_prob_list = []
p_list = np.linspace(0,1,21)
for p in p_list:
    fail_prob = []
    binomial = [1]
    for m in np.arange(2,d+2):
        nbin = np.zeros(m)
        for i in range(m):
            if i!=0:
                nbin[i] += binomial[i-1]*p
            if i+1<m:
                nbin[i] += binomial[i]*(1-p)
        binomial = nbin
        if m%2==0:
            fail = np.sum(binomial[m//2:])
            fail_prob.append(fail)
    fail_prob_list.append(fail_prob)
fail_prob_list = np.array(fail_prob_list).T

import matplotlib.pyplot as plt
plt.figure(figsize=(12,8))
plt.subplot(1,2,1)
for index,line in enumerate(fail_prob_list[:6]):
    plt.plot(p_list, line, label = "n={}".format(2*index+1))
plt.ylim(0,1)
plt.xlim(0,1)
plt.xlabel("Error probability per each bit")
plt.ylabel("Failure probability")
plt.legend()

plt.subplot(1,2,2)
for index,line in enumerate(fail_prob_list.T[:7]):
    if index==0:
        continue
    distance_list = np.arange((d+1)//2)*2+1
    plt.plot(distance_list,line, label = "p={:.3}".format(p_list[index]))
plt.ylim(1e-5,0.5)
plt.yscale("log")
plt.xlabel("distance")
plt.ylabel("Failure probability")
plt.legend()
plt.show()
```

左のグラフから$p$がある一定値(今回は0.5)以下であれば、$d$を大きくすればするほど性能が良くなることがわかる。
一方、$p$がある値以上の場合、$d$を増やすとよくなるどころかむしろ性能が悪化してしまう。
このことから、多数決で誤り訂正を行うには、少なくとも各デバイスがある誤り率以下であることが必要条件となることがわかる。

このふるまいは量子符号を含むスケールする符号にたびたび見られる性質であり、このしきいとなるエラー確率のことをエラーしきい値(error threshold)と呼ぶ。一般の古典誤り訂正はエラーしきい値よりはるかに小さい値で行われるが、量子誤り訂正では現状のエラーがこのしきい値付近であるため、たびたび言及される。

右のグラフは$p$をしきい値以下の一定値に固定し、$d$を大きくしたときにどの程度復号の失敗確率が減少するかを指数プロットしたものである。$p$がしきい値より十分小さい場合、$d$を大きくすることで概ね指数的に復号の失敗確率が減少していくことがわかる。

### 線形符号
上記の多数決の枠組みは線形符号という符号の一種である。下記では線形符号の枠組みで多数決を解説する。
ある$k$ビットの情報$v$を$n=dk$ビットの情報$v'$に冗長化する操作は、$k \times n$の行列$G$を以下のように構成し、$v' = vG$という計算を行うと言い換えることができる。

```python
import numpy as np
k = 4
m = 3
n = k*m
v = np.random.randint(2,size=k)
print("original vector: v\n{}".format(v))

G = np.zeros((k,n),dtype=np.int)
for y in range(k):
    G[y,y*m:(y+1)*m]=1
print("generator matrix: G\n{}".format(G))

vd = (v@G)%2
print("encoded vector: v' = vG\n{}".format(vd))
```

    original vector: v
    [0 1 0 1]
    generator matrix: G
    [[1 1 1 0 0 0 0 0 0 0 0 0]
     [0 0 0 1 1 1 0 0 0 0 0 0]
     [0 0 0 0 0 0 1 1 1 0 0 0]
     [0 0 0 0 0 0 0 0 0 1 1 1]]
    encoded vector: v' = vG
    [0 0 0 1 1 1 0 0 0 1 1 1]

この行列$G$は生成行列という。

生成される$2^n$パターンの$n$ビット列のうち、$vG$の形で作られる高々$2^k$個の$n$ビット列を符号語と呼ぶ。符号語の集合を$W$としたとき、$w \neq w'$となる二つの符号語でハミング距離が短いもののハミング距離のことを符号の距離(distance)と呼ぶ。今の多数決の構成では、符号の距離は個々の論理ビットを複製する数$d$である。距離$d$以上の数のビットが一斉に反転してしまうとある符号語から別の符号語に変化してしまうため、現在の符号が正しく生成されたのか、他の符号にエラーが載ったものなのか区別がつかなくなる。このことから、エラーが生じるビット数が距離$d$未満であることがエラーが検知できる条件であることが分かる。

次に、mod 2のもとで$GH_c = 0$となり、かつ各行ベクトルが独立になるような$n \times (n-k)$行列$H_c$を考える。今回の場合、3ビットごとに二つの隣接したパリティを見る以下のような$H_c$が条件を満たす。

```python
Hc = np.zeros((n,n-k),dtype=np.int)
for x in range(n-k):
    Hc[x//2*m+x%2:x//2*m+x%2+2,x]=1
print("check matrix: H\n{}".format(Hc))

print("G Hc = \n{}".format( (G@Hc)%2))
```

    check matrix: H
    [[1 0 0 0 0 0 0 0]
     [1 1 0 0 0 0 0 0]
     [0 1 0 0 0 0 0 0]
     [0 0 1 0 0 0 0 0]
     [0 0 1 1 0 0 0 0]
     [0 0 0 1 0 0 0 0]
     [0 0 0 0 1 0 0 0]
     [0 0 0 0 1 1 0 0]
     [0 0 0 0 0 1 0 0]
     [0 0 0 0 0 0 1 0]
     [0 0 0 0 0 0 1 1]
     [0 0 0 0 0 0 0 1]]
    G Hc =
    [[0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0]]

$H_c$の列ベクトルは独立であるため、$2^n$パターンの$n$ビット列$w$のうち、$w H_c = 0$を満たすパターンは$2^n / 2^{n-k} = 2^k$個である。また、符号語の集合$W$に属する符号語$w$は、$w H_c = v G H_c = 0$である。したがって、符号語の集合$W$とは、$w H_c = 0$を満たすビット列の集合と言い換えることもできる。同時に$n$ビット列$w$が符号語ではない$(w \neq W)$ことは、$w H_c \neq 0$であることに等しい。この符号語に属している($w \in W$)ときかつその時に限り$0$になるベクトル$s = w H_c$のことを$w$のシンドロームと呼ぶ。

まとめると、符号語の集合$W$は二つの行列$G,H_c$のいずれでも特徴づけることができ、

\begin{align}
W &= \left\{w \in \{0,1\}^n \mathrel{}\middle|\mathrel{} \exists v \in \{0,1\}^k , w = vG \right\} \\
W &= \left\{w \in \{0,1\}^n \mathrel{}\middle|\mathrel{} w H_c = 0 \right\}
\end{align}

である。

符号化した後にエラー$e$が生じた場合、エラーが起きた後の符号の状態は$v'+e$となった状態である。
この時、$v'+e$のシンドローム値$s = (v'+e)H_c$がすべて0であるかどうかを調べることで、エラーの検出を行える。
シンドローム値が一つでも0でない場合、何らかのエラーが生じている。シンドローム値がすべて0の場合、エラーは生じていないか、検知不可能な$d$ビット以上のエラーが生じていることがわかる。

```python
p = 0.05
print("encoded vector: v'\n{}".format(vd))
e = np.random.choice([0,1],size=len(vd),p = [1-p,p])
print("error vector: e\n{}".format(e))
vde = (vd+e)%2
print("encoded vector with noise: v'+e\n{}".format(vde))
s = (vde@Hc)%2
print("syndrome values: s = (v'+e)Hc\n{}".format(s))
```

    encoded vector: v'
    [0 0 0 1 1 1 0 0 0 1 1 1]
    error vector: e
    [0 0 0 0 0 0 0 0 0 0 0 0]
    encoded vector with noise: v'+e
    [0 0 0 1 1 1 0 0 0 1 1 1]
    syndrome values: s = (v'+e)Hc
    [0 0 0 0 0 0 0 0]

$GH_c=0$という性質から、$s =(v'+e)H_c = vGH_c + eH_c = eH_c$となる。つまり、シンドローム値は初期状態のベクトル$v$と独立であることが分かる。

$n$ビットの多数決は$v'+e$を左から$m$個ずつ見て多数決を行えばよい。

```python
p = 0.1
print("encoded vector: v'\n{}".format(vd))
e = np.random.choice([0,1],size=len(vd),p = [1-p,p])
print("error vector: e\n{}".format(e))
vde = (vd+e)%2
print("encoded vector with noise: v'+e\n{}".format(vde))
s = (vde@Hc)%2
print("syndrome values: s = (v'+e)Hc\n{}".format(s))
def decode(vde,k,m):
    for x in range(k):
        subset = vde[x*m:(x+1)*m]
        val = np.argmax(np.bincount(subset))
        vde[x*m:(x+1)*m] = val
    return vde
vde_recovery = decode(vde,k,m)
print("recovered vector: \n{}".format(vde_recovery))
```

    encoded vector: v'
    [0 0 0 1 1 1 0 0 0 1 1 1]
    error vector: e
    [0 0 0 0 0 0 0 0 0 0 0 1]
    encoded vector with noise: v'+e
    [0 0 0 1 1 1 0 0 0 1 1 0]
    syndrome values: s = (v'+e)Hc
    [0 0 0 0 0 0 0 1]
    recovered vector:
    [0 0 0 1 1 1 0 0 0 1 1 1]

上記の復号はエラーを大きくしていくと、$m$の半数以上が間違えて訂正に失敗することがあることが分かる。

$e H_c=0$となる$0$でないベクトル$e$が生じた場合、そのようなエラーは訂正することができない。すなわち、符号の距離$d$は検査行列$H_c$にに対して

$$
d := \min w(e) \, \text{s.t.} \, H_e e = 0
$$

という値だと考えることもできる。

### 水平垂直パリティ検査符号
現実の誤り訂正では多数決の手法はあまり性能が良くないためそれほど使われない。ここでは現実の誤り訂正でよく使われるパリティ検査符号の一例として、水平垂直パリティ検査符号の枠組みを学ぶ。

パリティとは、与えられた複数のビットの和を2で割った余り、すなわち1の数の偶奇を意味する。(1,1,1)のパリティは1、(0,1,1)のパリティは0である。
パリティ検査符号は、元の情報のデータについてのいくつかのパリティを保持することで誤りの検知や訂正を可能にする。
元の情報やパリティにエラーが生じた場合、パリティの値と元の情報に整合性が保たれなくなるためエラーを検出することができる。

最もシンプルなパリティ検査符号はチェックサムである。チェックサムは小さなブロック$(x_1,\ldots, x_b)$ごとに、ブロックに含まれるビット列のパリティ$p= \sum_i x_i \bmod 2$をチェックサムとして記入しておく。
もし、伝送の途中でブロック中のビットやチェックサムのビットの記入を高々1bit誤った場合、ブロックに含まれるビット列のパリティを再計算すると保持するパリティ$p$と整合性が取れなくなる。
従って、ブロックかパリティのどこかにビットエラーが生じたことがわかる。ただし、この方法はどこでエラーが生じたかを知ることはできないし、2ビットエラーが生じた場合はエラーを検知することもできない。

```python
import numpy as np
length = 16
p = 0.05

# set random bitarray
bitstring = np.random.randint(2,size = length)
checksum = np.sum(bitstring)%2
print("original bitarray: {}". format(bitstring))
print("checksum: {}".format(checksum))

# error occurs
error = np.random.choice([0,1], size = length, p = [1-p, p])
print("error:             {}". format(error))

bitstring = (bitstring + error)%2
print("noisy bitarray:    {}". format(bitstring))
new_checksum = np.sum(bitstring)%2
print("checksum: {}".format(new_checksum))

# verify checksum
if checksum == new_checksum:
    print("No error detected")
else:
    print("Error detected!")
```

    original bitarray: [0 1 0 1 1 1 1 1 0 1 1 1 0 0 0 0]
    checksum: 1
    error:             [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
    noisy bitarray:    [0 1 0 1 1 1 1 1 0 1 1 1 0 0 0 1]
    checksum: 0
    Error detected!

1ビットのビットエラーの検知だけでなく訂正を行うには、どこでエラーが生じたかを分かるようにしなければいけない。
これを実現するナイーブな手法は、水平垂直パリティ検査という手法である。
保護する対象を$k=16$ビットをベクトルとし、これを$4*4$の行列に配置する。
ここで、各行のベクトルのパリティと各列のベクトルのパリティを$8$ビット保存しておく。
この場合、もし$k$ビットの中でエラーが生じると、一つの列と行のパリティと整合が取れなくなる。この場合、整合が取れない列と行に一致するデータビットが反転したと判定する。
もし、行のパリティビットにエラーが生じた場合、列のパリティは全て整合が取れているはずである。このように一方のパリティだけにエラーがある場合、パリティが反転したと結論付ける。列のエラーも同様である。
こうすることで、どの場所でビットエラーが生じてもそれが1ビットであれば原因を特定できる。このコードの距離は3ビットであり$[24,16,3]$である。一般には$[w^2+2w,w^2,3]$である。

```python
import numpy as np
width = 4
height = 4
size = width*height
p = 0.03

def show(bitmatrix, vertical_parity, horizontal_parity):
    for y in range(height):
        print("{} | {}".format(bitmatrix[y,:], horizontal_parity[y] ))
    print("-"*width*2)
    print("{}".format(vertical_parity ))
    print()

# set random bitarray
bitstring = np.random.randint(2,size=size)
print("original bitarray: {}". format(bitstring))

# check horizontal and vertical parity
bitmatrix = bitstring.reshape( (height,width) )
vertical_parity = np.sum(bitmatrix, axis = 0)%2
horizontal_parity = np.sum(bitmatrix, axis = 1)%2
print("stored as : ")
show(bitmatrix,vertical_parity, horizontal_parity)

# error occurs
encoded_size = size + width + height
error = np.random.choice([0,1], size = encoded_size, p = [1-p, p])
bitmatrix = (bitmatrix + error[:size].reshape((height,width)))%2
vertical_parity = (vertical_parity + error[size:size+width])%2
horziontal_parity = (vertical_parity + error[size+width:])%2

# result
print("result is : ")
show(bitmatrix,vertical_parity, horizontal_parity)

# verify checksum
result_vertical_parity = np.sum(bitmatrix, axis = 0)%2
result_horizontal_parity = np.sum(bitmatrix, axis = 1)%2
vertical_flip_count = np.sum((result_vertical_parity + vertical_parity)%2)
horizontal_flip_count = np.sum((result_horizontal_parity + horizontal_parity)%2)

if vertical_flip_count == 0 and horizontal_flip_count == 0:
    print("No error detected")
    print("decoded bitarray: ")
    show(bitmatrix, result_vertical_parity, result_horizontal_parity)
elif vertical_flip_count == 1 and horizontal_flip_count == 0:
    print("Error occurs on vertical parity")
    print("decoded bitarray: ")
    show(bitmatrix, result_vertical_parity, result_horizontal_parity)
elif vertical_flip_count == 0 and horizontal_flip_count == 1:
    print("Error occurs on horizontal parity")
    print("decoded bitarray: ")
    show(bitmatrix, result_vertical_parity, result_horizontal_parity)
elif vertical_flip_count == 1 and horizontal_flip_count == 1:
    print("Error occurs on data bit")
    print("decoded bitarray: ")
    x = np.argmax((result_vertical_parity + vertical_parity)%2)
    y = np.argmax((result_horizontal_parity + horizontal_parity)%2)
    bitmatrix[y,x] = (bitmatrix[y,x]+1)%2
    show(bitmatrix, vertical_parity, horizontal_parity)
else:
    print("Too many error occurs")
```

    original bitarray: [1 0 0 1 0 0 0 1 0 0 1 1 1 0 0 0]
    stored as :
    [1 0 0 1] | 0
    [0 0 0 1] | 1
    [0 0 1 1] | 0
    [1 0 0 0] | 1
    --------
    [0 0 1 1]

    result is :
    [1 1 0 1] | 0
    [0 0 0 1] | 1
    [0 1 1 1] | 0
    [1 0 0 0] | 1
    --------
    [0 0 1 1]

    Too many error occurs

水平垂直パリティ検査符号も多数決と同様に生成行列と検査行列を考えることができる。
簡単のために、$2 \times 3$の6ビットに対する水平垂直パリティ検査の場合を例に考える。
水平垂直パリティ検査で符号化するビットの横ベクトルを$v$とする。この時、$(1,2,3)$番目のパリティを観測する計算は、$1,2,3$番目のみが1で他が0となっているベクトル$w = (1,1,1,0,0,0)$を用意し、ベクトル間の内積$v w^{\rm T} \bmod 2$を計算することに等しい。従って、元のデータに追加されるパリティビットは以下のような行列$A$をmod 2のもとで作用することで生成できる。なお、以降のバイナリビット間の演算では特に断りが無い限りはmod 2を省略する。

```python
A = np.array([
    [1 , 0 , 1 , 0 , 0],
    [1 , 0 , 0 , 1 , 0],
    [1 , 0 , 0 , 0 , 1],
    [0 , 1 , 1 , 0 , 0],
    [0 , 1 , 0 , 1 , 0],
    [0 , 1 , 0 , 0 , 1]
])
print(A)
```

    [[1 0 1 0 0]
     [1 0 0 1 0]
     [1 0 0 0 1]
     [0 1 1 0 0]
     [0 1 0 1 0]
     [0 1 0 0 1]]

この時、パリティの値を並べたベクトル$p$は$p = vA$であり、符号化後のバイナリベクトルは$v' = (v,p)$である。従って生成行列として$k \times n$行列$G$を$G = (I, A)$と置けば、$v' = v G$という形にできる。

符号化したバイナリベクトルにはエラー$e$が生じる。この時、$e$のうちデータ部に生じたエラーを$e_v$、パリティ部に生じたエラーを$e_p$とする。エラーが生じた後のバイナリベクトルは$v'+e = (v+e_v, p+e_p)$となる。水平垂直パリティ検査では、エラーが生じた後にパリティ値が一致しているかどうかをもとにエラーが生じたかを判断する。このことを式で書くと、

$$
(v + e_v)A = p + e_p
$$

が成り立つか否かが、エラーを検知するかどうかの判定に等しい。$p=vA$を用いて上記の等式を整理すると、エラーが検知されない条件は

$$
e_v A + e_p = (e_v, e_p) \left(\begin{matrix} A \\ I \end{matrix}\right) = 0
$$

である。行列$H_c$を$H_c = \left( \begin{matrix} A \\ I \end{matrix} \right)$と置くと、

$$
G H_c = (I, A) \left(\begin{matrix} A \\ I \end{matrix}\right) = A + A = 0
$$

である。

```python
G = np.hstack( (np.eye(A.shape[0]), A) )
Hc = np.vstack( (A, np.eye(A.shape[1])) )
print("generator matrix: G\n{}\n".format(G))
print("check matrix: Hc \n{}\n".format(Hc))
print("GHc = \n{}\n".format( (G@Hc)%2 ))
```

    generator matrix: G
    [[1. 0. 0. 0. 0. 0. 1. 0. 1. 0. 0.]
     [0. 1. 0. 0. 0. 0. 1. 0. 0. 1. 0.]
     [0. 0. 1. 0. 0. 0. 1. 0. 0. 0. 1.]
     [0. 0. 0. 1. 0. 0. 0. 1. 1. 0. 0.]
     [0. 0. 0. 0. 1. 0. 0. 1. 0. 1. 0.]
     [0. 0. 0. 0. 0. 1. 0. 1. 0. 0. 1.]]

    check matrix: Hc
    [[1. 0. 1. 0. 0.]
     [1. 0. 0. 1. 0.]
     [1. 0. 0. 0. 1.]
     [0. 1. 1. 0. 0.]
     [0. 1. 0. 1. 0.]
     [0. 1. 0. 0. 1.]
     [1. 0. 0. 0. 0.]
     [0. 1. 0. 0. 0.]
     [0. 0. 1. 0. 0.]
     [0. 0. 0. 1. 0.]
     [0. 0. 0. 0. 1.]]

    GHc =
    [[0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0.]]

数値的にも確かに$GH_c = 0$となることが分かる。$H_c$は$n \times (n-k)$行列で、かつ列ベクトルは単位行列の部分行列を持つことから明らかに列ごとに独立であり、確かに検査行列の性質を満たしている。

このとき、シンドローム値$s$は$s = e H_c$となる。シンドローム値が全て0の時、符号はエラーが生じていないと判定する。もちろん、大きなエラーが生じている場合、エラーが生じているにもかかわらず$s=0$となりエラーが無いと判定されることもある。

$s \neq 0$である場合、何らかのエラーが起きていることはわかる。この状態でエラーを訂正するには、$(v'+e)$の値から符号化前の$v$を推測しなければならない。今回の場合、冒頭のコードで示したような方法でエラーが一つまでならデータビットとパリティビットのどこにエラーがあるのかを$O(1)$で特定することができる。

パリティ検査符号の枠組みをまとめると以下のようになる。

- 符号：生成行列$G$と検査行列$H_c$で特徴づけられる。これらは$GH_c = 0$を満たす。
- 符号化前：$k$-bitの横ベクトルデータ $v$ がある。
- 符号化: $k \times n$の生成行列$G$を用いて、符号化した状態$v'$は$v'=vG$となる。
- エラー: 符号化後の$v'$にランダムに$n$-bit列 $e$ が足される。
- エラー検査: $n \times (n-k)$の検査行列$H_c$を用いてシンドローム$s = e H_c$が得られる。$s \neq 0$ならばエラーがあると検知できる。$s=0$ならば、エラーが生じていないか、検査できる限界以上のエラーが生じている。
- エラー訂正: $(v'+e)$の値に従って、$k$-bitの横ベクトルデータ $v$ を推測する。$v'+e$から$v$を求めるアルゴリズムを復号アルゴリズムと呼ぶ。
- 距離: $e H_c = 0$となる$e$の最小weightが符号の距離$d$である。

パリティ検査符号は非常に広い枠組みであり、例えばECCメモリではハミング符号と呼ばれるパリティ検査行列の一種が用いられている。

### 一般の線形符号のケース

上記では多数決、チェックサム、水平垂直検査という特定の例を挙げたが、実際に使用する際には目的に対し最も性能の良い符号がほしい。しかし、一般に与えられた$(n,k,d)$に対して最適な線形符号$G,H_c$を求めることは組み合わせ論の問題となり、一般に困難である。

また、仮に最適な$(G,H_c)$という符号を与えられても復号に関して問題がある。ナイーブな復号アルゴリズムの例は、シンドローム$s$を観測した時、$s = e'H_c$を満たすような$e'$のうち、最も少ないビットにエラーを起こすものを起きたエラーとみなし、$(v' + e + e')$が正しい符号語になると期待して復元を試みる最小距離複合である。上記の最適化は

$$
\min_{e \in \{0,1\}^n} w(e') \, \text{s.t.} \, s = e' H_c
$$

という$n$-bitの0,1計画問題となるため、最小距離複合は一般には符号のサイズに対してNP困難である。しかも、この復号は実際には最適な復号とと一致するとは限らない。一方、符号や起きるエラーの種類を限定すると効率的に最小距離複合が実施でき、かつ多くの場合この復号方法は最適に近い複合である。例えば多数決は常に最小距離複合が可能であり、水平垂直パリティ検査符号では生じるエラーを $w(e)=1$ に限定すれば$O(1)$で最小距離複合が可能である。

### 線形符号上での演算
ノイジーな通信路に情報を通す場合はパリティ検査符号で符号化された情報を転送するだけである場合でよい。しかし、計算機自体がノイジーな場合、パリティ検査符号で符号化した状態で計算を行わなければならない。例えば、CPU自体がノイジーであり、符号化した状態のままで計算を行いたい場合は、計算のたびに復号してしまっては符号化の意味がない。

上記の水平垂直パリティ検査を用いる場合、行列の$(i,j)$要素に対するビット反転操作を行う場合、この操作がエラーとして検知されないようにするには関連するパリティの値も更新しなくてはいけない。つまり、$1$ビットを更新する目には、符号化されたビット列に対して$3$ビットの操作が必要となる。

より一般的には、$v$に対するビット反転操作$o$を考えると、$v \rightarrow v+o$とするには、符号化された空間で$oB$のビット反転が必要になる。操作のコストを考えると、ある論理ビットを操作するときに実際に操作せねばならないビットの数は少なければ少ないほど良い。
この時、論理ビットの反転のために実際に反転しなければいけないビットの数は高々$\max_i w(G_i)$である。ただし、$G_i$は$G$の$i$行ベクトルである。水平垂直パリティ検査では$G$の行のweightはすべて3である。

### 低密度パリティ検査符号
宇宙との通信など通信できる時間が限られていたり通信路が劣悪にならざるを得ない場合、できるだけ限界に近いレートで大量のデータを送りたい。このことが、符号化や復号にかかる計算コストは大きくとも良いので、符号化するビット数あたりの論理誤り率が最も小さくなる符号を作るモチベーションとなる。

通信路のノイズが既知である場合、ノイズのある通信路で送信可能な情報量のレートの限界はシャノン限界として計算できる。このレートはこれは構成することができる符号の性能の限界を意味する。低密度パリティ検査符号はこのシャノン限界に近い性能を実現する符号である。

低密度パリティ検査符号では、検査行列$H_c$が疎行列となるパリティ検査符号の総称である。ここでいう疎行列とは、各列に含まれる$1$の数が高々$O(k)$程度であることを意味する。すなわち、個々のシンドローム値は$O(k)$個のビットのエラーのパリティで表現できるということである。

低密度パリティ検査符号の利点はその性能がシャノン限界に近い点である。一方、低密度パリティ検査符号の欠点は、復号の計算量的な困難さである。この困難を回避するため、信頼伝播法といった近似アルゴリズムが復号のためにしばしば用いられる。

信頼伝播法では、次のようなグラフを考える。符号化後のビットをノードとみなす。あるパリティ値があるデータビットのエラーをパリティの一部に含むとき、対応するパリティビットのノードからデータビットのノードへ無向エッジを張る。こうして、$n$頂点が$nO(k)$個の辺で繋がれたの無向グラフ$G(V,E)$が完成する。信頼伝播法はこのグラフ上での逐次的な最適化を状態が符号後となるまで繰り返す。
信頼伝播法の性能はグラフ$G$における最小のループが大きければ大きいほど良いことが知られている。したがって、そのようなグラフを構築するべく、様々な低密度パリティ検査符号の構築方法が提案されている。


# Models and Pricing of Financial Derivativs HW_01

**11510691 程远星**

## Question 1

Explain carefully the difference between *selling* a *call* option and *buying* a *put* option.

$Answer$

>$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\EE}[2][\,\!]{\mathbb{E}_{#1}\left[#2\right]}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathrm{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}$Selling a call option: As the writer of the call option, I give the holder the right to buy an asset at a specified time $T$ for a specified price $K$. My payoff would be $-\max\P{S_T - K,0} = \min\P{K-S_T,0}$ for european call options. If I sold an american call option, the holder can exercise at any time before $T$.
>
>Buying a put option: As the holder of the put option, I actually was granted the right to sell an asset at a specified time $T$ for a specified price $K$. My payoff would be $\max\P{K - S_T, 0}$ Or before $T$ if what I bought is an american put option.

## Question 2

The current price of a stock is $\$94$, and $3$-month European call option with a strike price of $\$95$ currently sell for $\$4.70$. An investor who feels that the price of the stock will increase is trying to decide between buying $100$ shares and buying $2, 000$ call options ($20$ contracts). Both strategies involve an investment of $\$9, 400$. What advice would you give? How high does the stock price have to rise for the option strategy to be more profitable?

$Answer$

>We can write their profit function on the stock price $S_t$.
>
>- Stock: $100\P{S_t - 94}$
>- Option: $2000\big(\max\P{S_t - 95,0} - 4.7\big)$
>
>They intersect at two points, $\P{0,0}$ and $\P{100,600}$. It's generally acknowledge that it's of higher possibility that the stock price moves less than more, thus I personally think that holding the stocks rather than the options have a better chance to profit.
>
>As for the second question, since we've already acquire their intersection, we can say that when the stock price goes higher than $100$, options will win more.

## Question 3

A trader *buys* a *European call* option and *sells* a *European put* option. The options have the same underlying asset, strike price, and maturity. Describe the trader’s ***position***. Under what circumstances does the price of the call equal the price of the put?

$Answer$

>For the call, he has paid $c$ to buy the right that he can use $K$ to buy the underlying asset as time $T$. For the put, he has received $p$ and given the right to someone else selling him the asset at price $K$ at time $T$. The terminal payoff of the trader is
>
>$$\P{S_T - K}^+ - \P{K-S_T}^+ = S_T - K$$
>
>To let the prices equal, by the **Put-call parity** (or to avoid proving it, we replicate the strategy by buying an asset and borrowing $Ke^{-rT}$ at time $0$), we have
>
>$$c-p = S_0 - Ke^{-rT}$$
>
>Thus when $S_0 = Ke^{-rT}$, the price of the call $c$ equals the price of the put $p$.

## Question 4

The price of a stock is $\$40$. The price of a $1$-year *European put* option on the stock with a strike price of $\$30$ is quoted as $\$7$ and the price of a $1$-year *European call* option on the stock with a strike price of $\$50$ is quoted as $\$5$. Suppose that an investor *buys* $100$ shares, *shorts* $100$ *call* options, and *buys* $100$ put options. Draw a diagram illustrating how the investor’s ***profit or loss*** varies with the stock price over the next year. How does your answer change if the investor *buys* $100$ shares, *shorts* $200$ *call* options, and *buys* $200$ *put* options?

$Answer$

>We first write its payoff function on the stock price:
>
>$$\begin{align}
p &= 100S_T -100 \cdot\max\P{S_T - 50, 0} + 100\cdot \max\P{30 - S_T,0}\\
&= \begin{cases}
5000, &\text{if } S_T \geq 50 \\
100S_T, &\text{if } 50 \geq S_T \geq 30 \\
3000, &\text{if } 30 \geq S_T \geq 0
\end{cases}
\end{align}
$$
>
>
>
>After that, the payoff would change to:

>$$\begin{align}
p &= 100\P{S_T - 40} + 200\SB{5 - \max\P{S_T - 50, 0}} + 200\SB{\max\P{30 - S_T,0} - 7}\\
&= \begin{cases}
10000 - 100S_T, &\text{if } S_T \geq 50 \\
100S_T, &\text{if } 50 \geq S_T \geq 30 \\
6000 - 100S_T, &\text{if } 30 \geq S_T \geq 0
\end{cases}
\end{align}
$$
>
>
>
>Also note that the investor has paid $4200$ and $4400$ initially for the two portfolios, respectively.

## Question 5

What is a *lower bound* for the price of a $1$-month *European put* option on a non-dividend-paying stock when the stock price is $\$12$, the strike price is $\$15$, and the risk-free interest rate is $6\%$ per annum?

$Answer$

>The lower bound of the option can be obtained using the formula
>
>$\begin{align}
p &\geq K e^{-rT} - S_0 \\
&= 15 \cdot e^{-6\% \times 1/12} - 12 \\
&\approx 2.93
\end{align}$
>
>To prove this formula (***inevitable***!):
>
>- we construct a arbitrage portfolio: if not, then $p<Ke^{-rT}-S_0$. Thus we buy this put using $p$, buy an asset using $S_0$, sell $p+S_0$ much bonds. (**Zero Input**). At time $T$, we sell the asset using the put option and obtain $K$, pay the bonds $\P{p+S_0}e^{rT}$, and our profit is $K - \P{p+S_0}e^{rT} = e^{rT}\P{Ke^{-rT} - S_0 - p}>0$. (**Positive Output**)
>- consider two portolios at time $0$: $A$: a put and an asset $S_0$; $B$: cash $Ke^{-rT}$. At time $T$, $A$ worth $\P{K-S_T}^+ + S_T = \max\P{K,S_T}$ however $B$ worth $K\leq \max\P{K,S_T}$. Thus at time $0$ this relationship turns to $p \geq K e^{-rT} - S_0$

## Qustion 6
"The *early exercise* of an American *put* is a trade-off between the *time value of money* and the *insurance value of a put*." Explain this statement.

$Answer$

>The early exercise of an American put option is to sell the stock to the writer at the Strike price $K$ before the expiration date $T$. Suppose he exercised at time $t$ thus his time value of money is $Ke^{-r\P{T-t}}$. But then he can not sell the stock at $K$ at time $T$ any more. In return, he is now able to earn interest on it between the time of the early exercise and the expiration date.

## Question 7

The price of a non-dividend-paying stock is $19$ and the price of a $3$-month European call option on the stock with a strike price of $\$20$ is $\$1$. The risk-free rate is $4\%$ per annum. What is the price of a $3$-month European put option with a strike price of $\$20$?

$Answer$

>By the put-call parity, we have: $1 + 20 \times e^{-4\% \times 0.25} = p + 19$ thus $p = 1.80$
>
>To prove this parity (***inevitable***!), consider two portfolios at time $0$
>
>- a call and bond worth $c + Ke^{-rT}$
- a put and an asset worth $p + S_0$
>
>At time $T$ these two portfolios both worth $\max\P{S_T,K}$ thus at time $0$, we have $c + Ke^{-rT} = p + S_0$

## Question 8

A $1$-month European put option on a non-dividend-paying stock is currently selling for $\$2.50$. The stock price is $\$47$, the strike price is $\$50$, and the risk-free interest rate is $6\%$ per annum. What opportunities are there for an arbitrageur?

$Answer$

>Based on the put-call parity, $c + Ke^{-rT} = S_0 + p \Longrightarrow c + 49.75 = 47+2.5$. Thus there's always a chance for arbitraging. He can buy a stock and a put option using the borrowed money, $49.5$ with interest rate $6\%$.
>
>Then he can win $50 - 49.5e^{0.06\times 1/12} \approx 0.25 $ if the stock price goes lower than $50$ or more if the stock price goes higher than $50$.

## Question 9

$\P{1}$ The price of an American call on a non-dividend-paying stock is $\$4$. The stock price is $\$31$, the strike price is $\$30$, and the expiration date is in $3$ months. The risk-free interest rate is $8\%$. Derive upper and lower bounds for the price of an American put on the same stock with the same strike price and expiration date.

$Answer$

>$P \geq p = c + Ke^{-rT}-S_0 = C + Ke^{-rT} - S_0 = 4 + 30 e^{-8\%\times1/4} -31 \approx 2.41$
>
>And to find something that keeps over an American Put, we can use $K$ cash at the beginning, thus $c + K \geq P+ S_0$ always holds. Therefore, $P \leq c + K - S_0 = C + K - S_0 = 4 + 30 - 31 = 3$

$\P{2}$ Explain carefully the arbitrage opportunities in Problem $9(1)$ if the American put price is greater than the calculated upper bound.

$Answer$

>If the American Put price is greater than $3$ that is to say that $P \geq C + K - S_0$, then we write an American put option and sell it to somebody, then use the money to buy a American call option, to borrow a stock then sell it to gain $S_0$. Send $K$ cash to the bank. Then when the American put option holder want to exercise, we can instantly use $K = 30$ to buy the stock and return to the stock lender. Up to now, we start from nothing to a American call option and a positive payoff and some interest.

## Question 10

$\P{1}$ Suppose that $c_1$ , $c_2$ , and $c_3$ are the prices of European call options with strike prices $K_1$, $K_2$, and $K_3$, respectively, where $K_3 > K_2 > K_1$ and $K_3 − K_2 = K_2 − K_1$. All options have the same maturity. Show that $c_2 \leq 0.5\P{c_1 + c_3}$

$Answer$

>If not, then $2c_2 > c_1 + c_3$. So that we have the arbitrage chance. First write two call option with strike price $K_2$ and then use the money gained to buy two call option with strike price $K_1$ and $K_3$. We already have some money left now.
>
>Then we the exercise time comes, we can exercise all three at the same time since $2K_2 = K_1 + K_3$, meaning that we gain money from nothing. Thus $2c_2 \leq c_1 + c_3$.

$\P{2}$ What is the result corresponding to that in Problem $10(1)$ For European put options?

$Answer$

>$p_2 \leq 0.5\P{p_1 + p_3}$, the proof is similar to the preceding one.

```python
import math as m
import numpy as np
import matplotlib.pyplot as plt
```

```python
X = np.linspace(0,80,9)
Y = np.zeros(9)
Y[0:4] = 3000
Y[4:6]= 100 * X[4:6]
Y[6:9]= 5000
```

```python
fig = plt.figure()
fig.set_size_inches(10,5)
ax  = fig.add_subplot(111)
ax.plot(X,Y)
ax.set_xlabel('Stock Price')
ax.set_ylabel('Payoff')
```

```python
X = np.linspace(0,80,9)
Y = np.zeros(9)
Y[0:4] = 6000 - 100*X[0:4]
Y[4:6]= 100 * X[4:6]
Y[6:9]= 10000 - 100*X[6:9]
```

```python
fig = plt.figure()
fig.set_size_inches(10,5)
ax  = fig.add_subplot(111)
ax.plot(X,Y)
ax.set_xlabel('Stock Price')
ax.set_ylabel('Payoff')
```

```python
# another method
fig, axs = plt.subplots()
fig.set_size_inches(10,5)
axs.plot(X,Y)
axs.set_xlabel('Stock Price')
axs.set_ylabel('Payoff')
```

```python
15*m.exp(-0.005)-12
```

    2.9251871878902342

```python

```

FinMath/Models and Pricing of Financial Derivatives/HW/HW_01_fixed.ipynb


Based on
https://www.tensorflow.org/tutorials/images/classification

Unfortunately it proved out, that both the fruit and Pokemon datasets I tested would have been
too complex and would have contained too few images for any current quantum neural
network to classify them successfully.
Therefore this path was abandoned.

```python
%matplotlib inline

import collections
# import os.path

import cirq
from cirq.contrib.svg import SVGCircuit
import matplotlib.pyplot as plt
import sympy
import tensorflow as tf
from tensorflow.keras import layers
import tensorflow_quantum as tfq
```

Loading the dataset

```python
# data_dir = "./fruits360"
split_seed = 123
image_size = (8, 8)
validation_split=0.2
color_mode="grayscale"

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    "./fruits-360/Training",
    # validation_split=validation_split,
    # subset="training",
    seed=split_seed,
    image_size=image_size,
    color_mode=color_mode
)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    "./fruits-360/Test",
    # validation_split=validation_split,
    # subset="validation",
    seed=split_seed,
    image_size=image_size,
    color_mode=color_mode
)
class_names = train_ds.class_names
print(class_names)

plt.figure(figsize=(10, 10))
for images, labels in train_ds.take(1):
  for i in range(9):
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(images[i].numpy().astype("uint8"))
    plt.title(class_names[labels[i]])
    plt.axis("off")
```

```python
# Performance tuning
AUTOTUNE = tf.data.AUTOTUNE
train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)

classical_model = tf.keras.models.Sequential([
    layers.experimental.preprocessing.Rescaling(1/255, input_shape=image_size),
    layers.Conv2D(16, 3, padding="same", activation="relu"),
    layers.Flatten(),
    layers.Dense(8, activation="relu"),
    layers.Dense(2)
])

classical_model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

classical_model.summary()
```

```python
class CircuitLayerBuilder:
    def __init__(self, data_qubits, readout):
        self.data_qubits = data_qubits
        self.readout = readout

    def add_layer(self, circuit, gate, prefix):
        for i, qubit in enumerate(self.data_qubits):
            symbol = sympy.Symbol(prefix + '-' + str(i))
            circuit.append(gate(qubit, self.readout)**symbol)

def create_quantum_model():
    """Create a QNN model circuit and readout operation to go along with it."""
    data_qubits = cirq.GridQubit.rect(*image_size)
    readout = cirq.GridQubit.rect(8, 1)
    circuit = cirq.Circuit()

    # TODO

    # Prepare the readout qubit.
    circuit.append(cirq.X(readout))
    circuit.append(cirq.H(readout))

    builder = CircuitLayerBuilder(
        data_qubits = data_qubits,
        readout=readout)

    # Then add layers (experiment by adding more).
    builder.add_layer(circuit, cirq.XX, "xx1")
    builder.add_layer(circuit, cirq.ZZ, "zz1")

    # Finally, prepare the readout qubit.
    for qubit in readout:
        circuit.append(cirq.H(qubit))

    return circuit, cirq.Z(readout)

model_circuit, model_readout = create_quantum_model()

SVGCircuit(model_circuit)
```

```python
quantum_model = tf.keras.Sequential([
    # The input is the data-circuit, encoded as a tf.string
    tf.keras.layers.Input(shape=(), dtype=tf.string),
    # The PQC layer returns the expected value of the readout gate, range [-1,1].
    tfq.layers.PQC(model_circuit, model_readout),
])
```

```python
classical_model.fit(train_ds.)

# TODO
```


```python
__author__ = 'kjoseph'
from array import array

from deflection import *
from initialize_functions import *
from functions import *
import scipy.spatial
FUNDAMENTALS = ['ae','ap','aa','be','bp','ba','oe','op','oa']

EQUATION_FILE = "../data/sexes_avg_new.txt"

t,M = get_t_and_M_from_file(EQUATION_FILE, FUNDAMENTALS)
deflection_computation = get_coefficients(M,t)
H = np.dot(np.vstack((np.identity(9),-M)), np.hstack((np.identity(9),-M.transpose())))
S_a,g_a = compute_s_and_g(M,t,'a',EPA_LIST)
S_b,g_b = compute_s_and_g(M,t,'b',EPA_LIST)
S_o,g_o = compute_s_and_g(M,t,'o',EPA_LIST)
##TEST DEFLECTION

pm1 = ParameterStruct('t', 1,3,3,[0.556179039209, 1],[0.748935820403, 1],[0.608249328496,1],1,1)
pm2 = ParameterStruct('t', 1,3,3,[-0.42344161868, 1],[-0.659712135792, 1],[0.192399427834,1],1,1)
opt_v = get_optimal(M,t,H,S_b,g_b,['a','o'], pm1,pm2)

for v in [[-0.447532713413, -0.356447458267, 0.429316014051], [-0.069087468087,0.258912533522,0.0270347073674]]:
    print scipy.spatial.distance.cosine(opt_v,v)

deflection_computation = np.array(deflection_computation, dtype=np.object_)

vals_list = array('f',[2.45, 1.75, 0.29,1.85, 1.65, 0.30,1.49,.31,.75])
for x in compute_deflection_bad(vals_list,deflection_computation):
    print 'x: ', x

# np.random.shuffle(M)
# deflection_computation = get_coefficients(M, t)
# print 'shuffled'
# for x in compute_deflection_bad(vals_list,deflection_computation):
#     print x
#for fundamental_index in range(N_FUNDAMENTALS):
#    c0, c1, c2 = get_constants_for_fundamental(fundamental_index,deflection_computation,value_list)
#    print FUNDAMENTALS[fundamental_index], -c1/(2*c0), c1, c0
```

```python
def get_t_and_M_from_file(eq_filename, fundamentals,spl_char= "\t"):
    M = []
    t = []
    equation_file = open(eq_filename)
    i = 0
    for line in equation_file:
        t.append(set())
        line_spl = [l.strip() for l in line.split(spl_char)]
        M.append([float(x) for x in line_spl[1:]])

        coeff = line_spl[0].replace("Z","")
        for j in range(len(coeff)):
            if coeff[j] == '1':
                t[i].add(fundamentals[j])
        i+=1

    equation_file.close()
    return t, np.array(M)

FUNDAMENTALS = ['ae','ap','aa','be','bp','ba','oe','op','oa']

t, M = get_t_and_M_from_file("../data/sexes_avg_new.txt",FUNDAMENTALS,"\t")

fund_eq = [[] for i in range(len(FUNDAMENTALS))]
for j in range(len(FUNDAMENTALS)):
    for i,coef in enumerate(t):
        coef = "*".join(coef)
        l = M[i,j]
        app_str = ""
        if l > 0:
            app_str = "+"
        if l == 0:
            continue
        elif coef != '':
            fund_eq[j].append(app_str +str(l)+"*"+coef)
        else:
            fund_eq[j].append(app_str+str(l))

FUND_EQ_STRS = ["".join(x) for x in fund_eq]
print FUND_EQ_STRS
```

    ['-0.155+0.41*ae+0.42*be-0.02*bp-0.1*ba+0.03*oe+0.06*op+0.05*be*ae+0.03*ae*op+0.12*be*oe-0.05*be*op-0.05*oe*bp+0.03*be*oe*ae-0.02*be*ae*op', '-0.1+0.56*ap+0.06*aa-0.105*be+0.44*bp+0.04*oe-0.05*ap*bp+0.01*be*oe', '+0.14+0.05*ae+0.705*aa-0.06*be+0.29*ba-0.06*aa*ba', '-0.14+0.11*ae+0.555*be-0.12*ba+0.05*op+0.11*be*oe-0.05*be*op-0.02*oe*bp+0.02*be*oe*ae', '+0.06+0.16*ap-0.15*be+0.685*bp+0.03*oe-0.015*op+0.01*be*ae+0.02*be*oe', '+0.17+0.02*ae-0.06*ap+0.3*aa+0.04*be+0.64*ba', '+0.025+0.11*be+0.61*oe-0.01*oa+0.03*be*ae+0.04*be*oe-0.03*oe*bp', '-0.395+0.15*be-0.11*bp-0.115*oe+0.66*op+0.07*oa+0.03*be*oe+0.03*be*op-0.05*bp*op', '-0.035+0.02*be+0.03*oe-0.05*op+0.745*oa']

```python
from sympy.solvers import solve
from sympy import Symbol
from sympy import sympify
from sympy.polys import Poly
from math import sqrt

identities = [1, 2]
sentiment_words = [1,4]
constraints = [SocialEventConstraint(actor=1, behavior=1, object=2),
               EqualityConstraint(identity=1, sentiment_word=1)]

equation_str = ''

for starter, list_of_terms in  [['i',identities], ['z',sentiment_words]]:
    for term in list_of_terms:
        for epa in ['e','p','a']:
            id = starter+str(term)+epa
            symbols[id] = Symbol(id)

eq_constr = [constraint.get_constraint_string() for constraint in constraints]
equation_str = "+".join(eq_constr)
expr = sympify(equation_str)

dat = {"i1e": 1.39,
       "i1p": .88,
       "i1a": 0.96,
       "z1e": -1.92,
       "z1p": 1.00,
       "z1a": 1.62,
       "i2e": 1.49,
       "i2p": .31,
       "i2a": 0.75,
      }
substitutions = dat.items()
expr = expr.subs(substitutions).expand()
p = Poly(expr).coeffs()
print expr
print -p[1]/(2*p[0])
```

```python
class User:

    def __init__(self, n_identities, sentences):
        self.sentences = sentences
        self.sentences_for_identity = [list() * n_identities]
        for sent_it, sentence in enumerate(sentences):
            self.add_sentence(sentence,sent_it)

    def add_sentence(sentence,sent_it=None):
        self.sentences.append(sentence)
        if not sent_it:
            sent_it = len(sentence)
        for identity in sentence.identities_contained():
            self.sentences_for_identity[identity].append(sent_it)
```

```python
d = read_grouped_file

for g in d:
    construct_sentence
    add_to_user
```

```python

```

```python
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.corpus import wordnet as wn

_wnl = WordNetLemmatizer()
```

```python
sent_dict = {}
for x in codecs.open("../../../../thesis/thesis_work/lcss_study/data/all_epa_terms.txt","r","utf8"):
    x_spl = x.split("\t")
    sent_dict[x_spl[0]] = [float(z) for z in x_spl[1:]]

IDENTITY_LIST_FN = "../../../../thesis/thesis_work/lcss_study/data/identities_for_study.txt"
identity_set = {x.strip().lower() for x in open(IDENTITY_LIST_FN)}
full_set_of_interesting_terms = identity_set|set(sent_dict.keys())
```

```python
_wnl.lemmatize("stuck",wn.VERB)
```

    u'stick'

```python
sent_dict['deal with']
```

```python
identity_set = {i : x.strip().lower() for i,x in enumerate(open(IDENTITY_LIST_FN))}
```

```python
from gensim.models.word2vec import Word2Vec
model = Word2Vec.load_word2vec_format("../../../../thesis/thesis_work/identity_extraction/python/gensim_model/glove_twitter_50_raw_model.txt.gz", binary=False)
```

    WARNING:gensim.models.word2vec:consider setting layer size to a multiple of 4 for greater performance

```python
'miracle' in sent_dict
```

    True