# Tutorial 1: Sequential Probability Ratio Test
**Week 3, Day 2: Hidden Dynamics**

**By Neuromatch Academy**

__Content creators:__ Yicheng Fei and Xaq Pitkow

__Content reviewers:__ John Butler, Matt Krause, Spiros Chavlis, Melvin Selim Atay, Keith van Antwerp, Michael Waskom, Jesse Livezey, and Byron Galbraith

__Production Editor:__ Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

# Tutorial Objectives

*Estimated timing of tutorial: 45 minutes*

On Bayes Day, we learned how to combine the sensory measurement $m$ about a latent variable $s$ with and our prior knowledge, using Bayes' Theorem. This produced a posterior probability distribution $p(s|m)$. Today we will allow for _dynamic_ world states and measurements.

In Tutorial 1 we will assume that the world state is _binary_ ($\pm 1$) and _constant_ over time, but allow for multiple observations over time. We will use the *Sequential Probability Ratio Test* (SPRT) to infer which state is true. This leads to the *Drift Diffusion Model (DDM)* where evidence accumulates until reaching a stopping criterion.

By the end of this tutorial, you should be able to:
- Define and implement the Sequential Probability Ratio Test for a series of measurements
- Define what drift and diffusion mean in a drift-diffusion model
- Explain the speed-accuracy trade-off in a drift diffusion model

**Summary of Exercises**

0. Bonus (math): derive the Drift Diffusion Model mathematically from SPRT

1. Simulate the DDM
    1. _Code_: Accumulate evidence and make a decision (DDM)
    2. _Interactive_: Manipulate parameters and interpret

2. Analyze the DDM
    1. _Code_: Quantify speed-accuracy tradeoff
    2. _Interactive_: Manipulate parameters and interpret

```python
# @title Video 1: Overview of Tutorials on Hidden Dynamics
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Eh411r7hm", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="HH7HkQ1kv5M", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

---
# Setup

```python
# Imports

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
from scipy.special import erf
```

```python
# @title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/NMA2020/nma.mplstyle")
```

```python
# @title Helper Functions

def simulate_and_plot_SPRT_fixedtime(mu, sigma, stop_time, num_sample,
                                     verbose=True):
  """Simulate and plot a SPRT for a fixed amount of time given a std.

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation of the observations.
    stop_time (int): Number of steps to run before stopping.
    num_sample (int): The number of samples to plot.
    """

  evidence_history_list = []
  if verbose:
    print("#Trial\tTotal_Evidence\tDecision")
  for i in range(num_sample):
    evidence_history, decision, Mvec = simulate_SPRT_fixedtime(mu, sigma, stop_time)
    if verbose:
      print("{}\t{:f}\t{}".format(i, evidence_history[-1], decision))
    evidence_history_list.append(evidence_history)

  fig, ax = plt.subplots()
  maxlen_evidence = np.max(list(map(len,evidence_history_list)))
  ax.plot(np.zeros(maxlen_evidence), '--', c='red', alpha=1.0)
  for evidences in evidence_history_list:
    ax.plot(np.arange(len(evidences)), evidences)
    ax.set_xlabel("Time")
    ax.set_ylabel("Accumulated log likelihood ratio")
    ax.set_title("Log likelihood ratio trajectories under the fixed-time " +
                  "stopping rule")

  plt.show(fig)

def plot_accuracy_vs_stoptime(mu, sigma, stop_time_list, accuracy_analytical_list, accuracy_list=None):
  """Simulate and plot a SPRT for a fixed amount of times given a std.

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation of the observations.
    stop_time_list (int): List of number of steps to run before stopping.
    accuracy_analytical_list (int): List of analytical accuracies for each stop time
    accuracy_list (int (optional)): List of simulated accuracies for each stop time
  """
  T = stop_time_list[-1]
  fig, ax = plt.subplots(figsize=(12,8))
  ax.set_xlabel('Stop Time')
  ax.set_ylabel('Average Accuracy')
  ax.plot(stop_time_list, accuracy_analytical_list)
  if accuracy_list is not None:
    ax.plot(stop_time_list, accuracy_list)
  ax.legend(['analytical','simulated'], loc='upper center')

  # Show two gaussian
  stop_time_list_plot = [max(1,T//10), T*2//3]
  sigma_st_max = 2*mu*np.sqrt(stop_time_list_plot[-1])/sigma
  domain = np.linspace(-3*sigma_st_max,3*sigma_st_max,50)
  for stop_time in stop_time_list_plot:
    ins = ax.inset_axes([stop_time/T,0.05,0.2,0.3])
    for pos in ['right', 'top', 'bottom', 'left']:
      ins.spines[pos].set_visible(False)
    ins.axis('off')
    ins.set_title(f"stop_time={stop_time}")

    left = np.zeros_like(domain)
    mu_st = 4*mu*mu*stop_time/2/sigma**2
    sigma_st = 2*mu*np.sqrt(stop_time)/sigma
    for i, mu1 in enumerate([-mu_st,mu_st]):
      rv = stats.norm(mu1, sigma_st)
      offset = rv.pdf(domain)
      # lbl = "measurement distribution" if i==0 else ""
      lbl = "summed evidence" if i==1 else ""
      color = "crimson"
      ls = "solid" if i==1 else "dashed"
      ins.plot(domain, left+offset, label=lbl, color=color,ls=ls)

    rv = stats.norm(mu_st, sigma_st)
    domain0 = np.linspace(-3*sigma_st_max,0,50)
    offset = rv.pdf(domain0)
    ins.fill_between(domain0, np.zeros_like(domain0), offset, color="crimson", label="error")
    ins.legend(bbox_to_anchor=(1.05, 1.0), loc='upper left')

    # ins.legend(loc="upper right")

  plt.show(fig)

def simulate_and_plot_SPRT_fixedthreshold(mu, sigma, num_sample, alpha,
                                          verbose=True):
  """Simulate and plot a SPRT for a fixed amount of times given a std.

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation of the observations.
    num_sample (int): The number of samples to plot.
    alpha (float): Threshold for making a decision.
  """
  # calculate evidence threshold from error rate
  threshold = threshold_from_errorrate(alpha)

  # run simulation
  evidence_history_list = []
  if verbose:
    print("#Trial\tTime\tAccumulated Evidence\tDecision")
  for i in range(num_sample):
    evidence_history, decision, Mvec = simulate_SPRT_threshold(mu, sigma, threshold)
    if verbose:
      print("{}\t{}\t{:f}\t{}".format(i, len(Mvec), evidence_history[-1],
                                      decision))
    evidence_history_list.append(evidence_history)

  fig, ax = plt.subplots()
  maxlen_evidence = np.max(list(map(len,evidence_history_list)))
  ax.plot(np.repeat(threshold,maxlen_evidence + 1), c="red")
  ax.plot(-np.repeat(threshold,maxlen_evidence + 1), c="red")
  ax.plot(np.zeros(maxlen_evidence + 1), '--', c='red', alpha=0.5)

  for evidences in evidence_history_list:
      ax.plot(np.arange(len(evidences) + 1), np.concatenate([[0], evidences]))

  ax.set_xlabel("Time")
  ax.set_ylabel("Accumulated log likelihood ratio")
  ax.set_title("Log likelihood ratio trajectories under the threshold rule")

  plt.show(fig)

def simulate_and_plot_accuracy_vs_threshold(mu, sigma, threshold_list, num_sample):
  """Simulate and plot a SPRT for a set of thresholds given a std.

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation of the observations.
    alpha_list (float): List of thresholds for making a decision.
    num_sample (int): The number of samples to plot.
  """
  accuracies, decision_speeds = simulate_accuracy_vs_threshold(mu, sigma,
                                                               threshold_list,
                                                               num_sample)

  # Plotting
  fig, ax = plt.subplots()
  ax.plot(decision_speeds, accuracies, linestyle="--", marker="o")
  ax.plot([np.amin(decision_speeds), np.amax(decision_speeds)],
          [0.5, 0.5], c='red')
  ax.set_xlabel("Average Decision speed")
  ax.set_ylabel('Average Accuracy')
  ax.set_title("Speed/Accuracy Tradeoff")
  ax.set_ylim(0.45, 1.05)

  plt.show(fig)

def threshold_from_errorrate(alpha):
  """Calculate log likelihood ratio threshold from desired error rate `alpha`

  Args:
    alpha (float): in (0,1), the desired error rate

  Return:
    threshold: corresponding evidence threshold
  """
  threshold = np.log((1. - alpha) / alpha)
  return threshold
```

---

# Section 1: Sequential Probability Ratio Test as a Drift Diffusion Model

*Estimated timing to here from start of tutorial: 8 min*

```python
# @title Video 2: Sequential Probability Ratio Test
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Yo4y1D7Be", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="vv0yukRSTT0", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers the definition of and math behind the sequential probability ratio test (SPRT), and introduces the idea of the SPRT as a drift diffusion model.

Click here for text recap of video

**Sequential Probability Ratio Test**

The Sequential Probability Ratio Test is a likelihood ratio test for determining which of two hypotheses is more likely. It is appropriate for sequential independent and identially distributed (iid) data. iid means that the data comes from the same distribution.

Let's return to what we learned yesterday. We had probabilities of our measurement ($m$) given a state of the world ($s$). For example, we knew the probability of seeing someone catch a fish while fishing on the left side given that the fish were on the left side $P(m = \textrm{catch fish} | s = \textrm{left})$.

Now let's extend this slightly to assume we take a series of measurements, from time 1 up to time t ($m_{1:t}$), and that our state is either +1 or -1. We want to figure out what the state is, given our measurements. To do this, we can compare the total evidence up to time $t$ for our two hypotheses (that the state is +1 or that the state is -1). We do this by computing a likelihood ratio: the ratio of the likelihood of all these measurements given the state is +1, $p(m_{1:t}|s=+1)$, to the likelihood of the measurements given the state is -1, $p(m_{1:t}|s=-1)$. This is our likelihood ratio test. In fact, we want to take the log of this likelihood ratio to give us the log likelihood ratio $L_T$.

\begin{align*}
L_T &= log\frac{p(m_{1:t}|s=+1)}{p(m_{1:t}|s=-1)}
\end{align*}

Since our data is independent and identically distribution, the probability of all measurements given the state equals the product of the separate probabilities of each measurement given the state ($p(m_{1:t}|s) = \prod_{t=1}^T p(m_t | s) $). We can substitute this in and use log properties to convert to a sum.

\begin{align*}
L_T &= log\frac{p(m_{1:t}|s=+1)}{p(m_{1:t}|s=-1)}\\
&= log\frac{\prod_{t=1}^Tp(m_{t}|s=+1)}{\prod_{t=1}^Tp(m_{t}|s=-1)}\\
&= \sum_{t=1}^T log\frac{p(m_{t}|s=+1)}{p(m_{t}|s=-1)}\\
&= \sum_{t=1}^T \Delta_t
\end{align*}

In the last line, we have used $\Delta_t = log\frac{p(m_{t}|s=+1)}{p(m_{t}|s=-1)}$.

To get the full log likelihood ratio, we are summing up the log likelihood ratios at each time step. The log likelihood ratio at a time step ($L_T$) will equal the ratio at the previous time step ($L_{T-1}$) plus the ratio for the measurement at that time step, given by $\Delta_T$:

\begin{align*}
L_T =  L_{T-1} + \Delta_T
\end{align*}

The SPRT states that if $L_T$ is positive, then the state $s=+1$ is more likely than $s=-1$!

**Sequential Probability Ratio Test as a Drift Diffusion Model**

Let's assume that the probability of seeing a measurement given the state is a Gaussian (Normal) distribution where the mean ($\mu$) is different for the two states but the standard deviation ($\sigma$) is the same:

\begin{align*}
p(m_t | s = +1) &= \mathcal{N}(\mu, \sigma^2)\\
p(m_t | s = -1) &= \mathcal{N}(-\mu, \sigma^2)\\
\end{align*}

We can write the new evidence (the log likelihood ratio for the measurement at time $t$) as

$$\Delta_t=b+c\epsilon_t$$

The first term, $b$, is a consistant value and equals $b=2\mu^2/\sigma^2$. This term favors the actual hidden state. The second term, $c\epsilon_t$ where $\epsilon_t\sim\mathcal{N}(0,1)$, is a standard random variable which is scaled by the diffusion $c=2\mu/\sigma$. You can work through proving this in the bonus exercise 0 below if you wish!

The accumulation of evidence will thus "drift" toward one outcome, while "diffusing" in random directions, hence the term "drift-diffusion model" (DDM). The process is most likely (but not guaranteed) to reach the correct outcome eventually.

Bonus math exercise 0: derive Drift Diffusion Model from SPRT

We can do a little math to find the SPRT update $\Delta_t$ to the log-likelihood ratio. You can derive this yourself, filling in the steps below, or skip to the end result.

Assume measurements are Gaussian-distributed with different means depending on the discrete latent variable $s$:
$$p(m|s=\pm 1) = \mathcal{N}\left(\mu_\pm,\sigma^2\right)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(m-\mu_\pm)^2}{2\sigma^2}\right]}$$

In the log likelihood ratio for a single data point $m_i$, the normalizations cancel to give
$$\Delta_t=\log \frac{p(m_t|s=+1)}{p(m_t|s=-1)} = \frac{1}{2\sigma^2}\left[-\left(m_t-\mu_+\right)^2 + (m_t-\mu_-)^2\right] \tag{5}$$

It's convenient to rewrite $m=\mu_\pm + \sigma \epsilon$, where $\epsilon\sim \mathcal{N}(0,1)$ is a standard Gaussian variable with zero mean and unit variance. (Why does this give the correct probability for $m$?). The preceding formula can then be rewritten as
$$\Delta_t = \frac{1}{2\sigma^2}\left( -((\mu_\pm+\sigma\epsilon)-\mu_+)^2 + ((\mu_\pm+\sigma\epsilon)-\mu_-)^2\right) \tag{5}$$
Let's assume that $s=+1$ so $\mu_\pm=\mu_+$ (if $s=-1$ then the result is the same with a reversed sign). In that case, the means in the first term $m_t-\mu_+$ cancel, leaving
$$\Delta_t = \frac{\delta^2\mu^2}{2\sigma^2}+\frac{\delta\mu}{\sigma}\epsilon_t \tag{5}$$
where $\delta\mu=\mu_+-\mu_-$. If we take $\mu_\pm=\pm\mu$, then $\delta\mu=2\mu$, and
$$\Delta_t=2\frac{\mu^2}{\sigma^2}+2\frac{\mu}{\sigma}\epsilon_t$$

The first term is a constant *drift*, and the second term is a random *diffusion*.

The SPRT says that we should add up these evidences, $L_T=\sum_{t=1}^T \Delta_t$. Note that the $\Delta_t$ are independent. Recall that for independent random variables, the mean of a sum is the sum of the means. And the variance of a sum is the sum of the variances.

Adding these $\Delta_t$ over time gives
$$L_T\sim\mathcal{N}\left(2\frac{\mu^2}{\sigma^2}T,\ 4\frac{\mu^2}{\sigma^2}T\right)=\mathcal{N}(bT,c^2T)$$
as claimed. The log-likelihood ratio $L_t$ is a biased random walk --- normally distributed with a time-dependent mean and variance. This is the Drift Diffusion Model.

##  Coding Exercise 1.1: Simulating an SPRT model

Let's now generate simulated data with $s=+1$ and see if the SPRT can infer the state correctly.

We will implement a function `simulate_SPRT_fixedtime`, which will generate measurements based on $\mu$, $\sigma$, and the true state. It will then accumulate evidence over the time steps and output a decision on the state. The decision will be the state that is more likely according to the accumulated evidence. We will use the helper function `log_likelihood_ratio`, implemented in the next cell, which computes the log of the likelihood of the state being 1 divided by the likelihood of the state being -1.

**Your coding tasks are:**

**Step 1**: accumulate evidence.

**Step 2**: make a decision at the last time point.

We will then visualize 10 simulations of the DDM. In the next exercise you'll see how the parameters affect performance.

```python
# @markdown Execute this cell to enable the helper function `log_likelihood_ratio`

def log_likelihood_ratio(Mvec, p0, p1):
  """Given a sequence(vector) of observed data, calculate the log of
  likelihood ratio of p1 and p0

  Args:
    Mvec (numpy vector):           A vector of scalar measurements
    p0 (Gaussian random variable): A normal random variable with `logpdf'
                                    method
    p1 (Gaussian random variable): A normal random variable with `logpdf`
                                    method

  Returns:
    llvec: a vector of log likelihood ratios for each input data point
  """
  return p1.logpdf(Mvec) - p0.logpdf(Mvec)
```

```python
def simulate_SPRT_fixedtime(mu, sigma, stop_time, true_dist = 1):
  """Simulate a Sequential Probability Ratio Test with fixed time stopping
  rule. Two observation models are 1D Gaussian distributions N(1,sigma^2) and
  N(-1,sigma^2).

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation of observation models
    stop_time (int): Number of samples to take before stopping
    true_dist (1 or -1): Which state is the true state.

  Returns:
    evidence_history (numpy vector): the history of cumulated evidence given
                                      generated data
    decision (int): 1 for s = 1, -1 for s = -1
    Mvec (numpy vector): the generated sequences of measurement data in this trial
  """

  #################################################
  ## TODO for students ##
  # Fill out function and remove
  raise NotImplementedError("Student exercise: complete simulate_SPRT_fixedtime")
  #################################################

  # Set means of observation distributions
  assert mu > 0, "Mu should be > 0"
  mu_pos = mu
  mu_neg = -mu

  # Make observation distributions
  p_pos = stats.norm(loc = mu_pos, scale = sigma)
  p_neg = stats.norm(loc = mu_neg, scale = sigma)

  # Generate a random sequence of measurements
  if true_dist == 1:
    Mvec = p_pos.rvs(size = stop_time)
  else:
    Mvec = p_neg.rvs(size = stop_time)

  # Calculate log likelihood ratio for each measurement (delta_t)
  ll_ratio_vec = log_likelihood_ratio(Mvec, p_neg, p_pos)

  # STEP 1: Calculate accumulated evidence (S) given a time series of evidence (hint: np.cumsum)
  evidence_history = ...

  # STEP 2: Make decision based on the sign of the evidence at the final time.
  decision = ...

  return evidence_history, decision, Mvec

# Set random seed
np.random.seed(100)

# Set model parameters
mu = .2
sigma = 3.5  # standard deviation for p+ and p-
num_sample = 10  # number of simulations to run
stop_time = 150 # number of steps before stopping

# Simulate and visualize
simulate_and_plot_SPRT_fixedtime(mu, sigma, stop_time, num_sample)
```

```python
# to_remove solution
def simulate_SPRT_fixedtime(mu, sigma, stop_time, true_dist = 1):
  """Simulate a Sequential Probability Ratio Test with fixed time stopping
  rule. Two observation models are 1D Gaussian distributions N(1,sigma^2) and
  N(-1,sigma^2).

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation of observation models
    stop_time (int): Number of samples to take before stopping
    true_dist (1 or -1): Which state is the true state.

  Returns:
    evidence_history (numpy vector): the history of cumulated evidence given
                                      generated data
    decision (int): 1 for s = 1, -1 for s = -1
    Mvec (numpy vector): the generated sequences of measurement data in this trial
  """

  # Set means of observation distributions
  assert mu > 0, "Mu should be > 0"
  mu_pos = mu
  mu_neg = -mu

  # Make observation distributions
  p_pos = stats.norm(loc = mu_pos, scale = sigma)
  p_neg = stats.norm(loc = mu_neg, scale = sigma)

  # Generate a random sequence of measurements
  if true_dist == 1:
    Mvec = p_pos.rvs(size = stop_time)
  else:
    Mvec = p_neg.rvs(size = stop_time)

  # Calculate log likelihood ratio for each measurement (delta_t)
  ll_ratio_vec = log_likelihood_ratio(Mvec, p_neg, p_pos)

  # STEP 1: Calculate accumulated evidence (S) given a time series of evidence (hint: np.cumsum)
  evidence_history = np.cumsum(ll_ratio_vec)

  # STEP 2: Make decision based on the sign of the evidence at the final time.
  decision = np.sign(evidence_history[-1])

  return evidence_history, decision, Mvec

# Set random seed
np.random.seed(100)

# Set model parameters
mu = .2
sigma = 3.5  # standard deviation for p+ and p-
num_sample = 10  # number of simulations to run
stop_time = 150 # number of steps before stopping

# Simulate and visualize
with plt.xkcd():
  simulate_and_plot_SPRT_fixedtime(mu, sigma, stop_time, num_sample)
```

## Interactive Demo 1.2: Trajectories under the fixed-time stopping rule

In the following demo, you can change the drift level (mu), noise level (sigma) in the observation model and the number of time steps before stopping (stop_time) using the sliders. You will then observe 10 simulations with those parameters. As in the previous exercise, the true state is +1.

1.   Are you more likely to make the wrong decision (choose the incorrect state) with high or low noise?
2. What happens when sigma is very small? Why?
3.   Are you more likely to make the wrong decision (choose the incorrect state) with fewer or more time steps before stopping?

```python
# @markdown Make sure you execute this cell to enable the widget!

def simulate_SPRT_fixedtime(mu, sigma, stop_time, true_dist = 1):
  """Simulate a Sequential Probability Ratio Test with fixed time stopping
  rule. Two observation models are 1D Gaussian distributions N(1,sigma^2) and
  N(-1,sigma^2).

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation of observation models
    stop_time (int): Number of samples to take before stopping
    true_dist (1 or -1): Which state is the true state.

  Returns:
    evidence_history (numpy vector): the history of cumulated evidence given
                                      generated data
    decision (int): 1 for s = 1, -1 for s = -1
    Mvec (numpy vector): the generated sequences of measurement data in this trial
  """

  # Set means of observation distributions
  assert mu > 0, "Mu should be >0"
  mu_pos = mu
  mu_neg = -mu

  # Make observation distributions
  p_pos = stats.norm(loc = mu_pos, scale = sigma)
  p_neg = stats.norm(loc = mu_neg, scale = sigma)

  # Generate a random sequence of measurements
  if true_dist == 1:
    Mvec = p_pos.rvs(size = stop_time)
  else:
    Mvec = p_neg.rvs(size = stop_time)

  # Calculate log likelihood ratio for each measurement (delta_t)
  ll_ratio_vec = log_likelihood_ratio(Mvec, p_neg, p_pos)

  # STEP 1: Calculate accumulated evidence (S) given a time series of evidence (hint: np.cumsum)
  evidence_history = np.cumsum(ll_ratio_vec)

  # STEP 2: Make decision based on the sign of the evidence at the final time.
  decision = np.sign(evidence_history[-1])

  return evidence_history, decision, Mvec

np.random.seed(100)
num_sample = 10

@widgets.interact(mu=widgets.FloatSlider(min=0.1, max=5.0, step=0.1, value=0.5), sigma=(0.05, 10.0, 0.05), stop_time=(5, 500, 1))
def plot(mu, sigma, stop_time):
  simulate_and_plot_SPRT_fixedtime(mu, sigma, stop_time, num_sample, verbose=False)
```

```python
# to_remove explanation

"""

1) Higher noise, or higher sigma, means that the evidence accumulation varies up
   and down more. You are more likely to make a wrong decision with high noise,
   since the accumulated log likelihood ratio is more likely to be negative at the end
   despite the true distribution being s = +1.

2) When sigma is very small, the cumulated log likelihood ratios are basically a linear
   diagonal line. This is because each new measurement will be very similar (since they are
   being drawn from a Gaussian with a tiny standard deviation)

3) You are more likely to be wrong with a small number of time steps before decision. There is
   more change that the noise will affect the decision. We will explore this in the next section.

"""
```

```python
# @title Video 3: Section 1 Exercises Discussion
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1h54y1E7UC", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="P6xuOS5TB7Q", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

---
# Section 2: Analyzing the DDM: accuracy vs stopping time

*Estimated timing to here from start of tutorial: 28 min*

```python
# @title Video 4: Speed vs Accuracy Tradeoff
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1s54y1E7yT", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="Hc3uXQiKvZA", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

If you make a hasty decision (e.g., after only seeing 2 samples), or if observation noise buries the signal, you may see a negative accumulated log likelihood ratio and thus make a wrong decision. Let's plot how decision accuracy varies with the number of samples. Accuracy is the proportion of correct trials across our repeated simulations: $\frac{\# \textrm{ correct decisions}}{\# \textrm{ total decisions}}$.

### Coding Exercise 2.1: The Speed/Accuracy Tradeoff

We will fix our observation noise level. In this exercise you will implement a function to run many simulations for a certain stopping time, and calculate the _average decision accuracy_. We will then visualize the relation between average decision accuracy and stopping time.

```python
def simulate_accuracy_vs_stoptime(mu, sigma, stop_time_list, num_sample, no_numerical=False):
  """Calculate the average decision accuracy vs. stopping time by running
  repeated SPRT simulations for each stop time.

  Args:
      mu (float): absolute mean value of the symmetric observation distributions
      sigma (float): standard deviation for observation model
      stop_list_list (list-like object): a list of stopping times to run over
      num_sample (int): number of simulations to run per stopping time
      no_numerical (bool): flag that indicates the function to return analytical values only

  Returns:
      accuracy_list: a list of average accuracies corresponding to input
                      `stop_time_list`
      decisions_list: a list of decisions made in all trials
  """

  #################################################
  ## TODO for students##
  # Fill out function and remove
  raise NotImplementedError("Student exercise: complete simulate_accuracy_vs_stoptime")
  #################################################

  # Determine true state (1 or -1)
  true_dist = 1

  # Set up tracker of accuracy and decisions
  accuracies = np.zeros(len(stop_time_list),)
  accuracies_analytical = np.zeros(len(stop_time_list),)
  decisions_list = []

  # Loop over stop times
  for i_stop_time, stop_time in enumerate(stop_time_list):

    if not no_numerical:
      # Set up tracker of decisions for this stop time
      decisions = np.zeros((num_sample,))

      # Loop over samples
      for i in range(num_sample):

        # STEP 1: Simulate run for this stop time (hint: use output from last exercise)
        _, decision, _= ...

        # Log decision
        decisions[i] = decision

      # STEP 2: Calculate accuracy by averaging over trials
      accuracies[i_stop_time] = ...

      # Log decision
      decisions_list.append(decisions)

    # Calculate analytical accuracy
    sigma_sum_gaussian = sigma / np.sqrt(stop_time)
    accuracies_analytical[i_stop_time] = 0.5 + 0.5 * erf(mu / np.sqrt(2) / sigma_sum_gaussian)

  return accuracies, accuracies_analytical, decisions_list

# Set random seed
np.random.seed(100)

# Set parameters of model
mu = 0.5
sigma = 4.65  # standard deviation for observation noise
num_sample = 100  # number of simulations to run for each stopping time
stop_time_list = np.arange(1, 150, 10) # Array of stopping times to use

# Calculate accuracies for each stop time
accuracies, accuracies_analytical, _ = simulate_accuracy_vs_stoptime(mu, sigma, stop_time_list,
                                                   num_sample)

# Visualize
plot_accuracy_vs_stoptime(mu, sigma, stop_time_list, accuracies_analytical, accuracies)
```

```python
# to_remove solution

def simulate_accuracy_vs_stoptime(mu, sigma, stop_time_list, num_sample, no_numerical=False):
  """Calculate the average decision accuracy vs. stopping time by running
  repeated SPRT simulations for each stop time.

  Args:
      mu (float): absolute mean value of the symmetric observation distributions
      sigma (float): standard deviation for observation model
      stop_list_list (list-like object): a list of stopping times to run over
      num_sample (int): number of simulations to run per stopping time
      no_numerical (bool): flag that indicates the function to return analytical values only

  Returns:
      accuracy_list: a list of average accuracies corresponding to input
                      `stop_time_list`
      decisions_list: a list of decisions made in all trials
  """

  # Determine true state (1 or -1)
  true_dist = 1

  # Set up tracker of accuracy and decisions
  accuracies = np.zeros(len(stop_time_list),)
  accuracies_analytical = np.zeros(len(stop_time_list),)
  decisions_list = []

  # Loop over stop times
  for i_stop_time, stop_time in enumerate(stop_time_list):

    if not no_numerical:
      # Set up tracker of decisions for this stop time
      decisions = np.zeros((num_sample,))

      # Loop over samples
      for i in range(num_sample):

        # STEP 1: Simulate run for this stop time (hint: use output from last exercise)
        _, decision, _= simulate_SPRT_fixedtime(mu, sigma, stop_time, true_dist)

        # Log decision
        decisions[i] = decision

      # STEP 2: Calculate accuracy by averaging over trials
      accuracies[i_stop_time] = np.sum(decisions == true_dist) / decisions.shape[0]

      # Store the decisions
      decisions_list.append(decisions)

    # Calculate analytical accuracy
    # S_t is a normal variable with SNR scale as sqrt(stop_time)
    sigma_sum_gaussian = sigma / np.sqrt(stop_time)
    accuracies_analytical[i_stop_time] = 0.5 + 0.5 * erf(mu / np.sqrt(2) / sigma_sum_gaussian)

  return accuracies, accuracies_analytical, decisions_list

# Set random seed
np.random.seed(100)

# Set parameters of model
mu = 0.5
sigma = 4.65  # standard deviation for observation noise
num_sample = 100  # number of simulations to run for each stopping time
stop_time_list = np.arange(1, 150, 10) # Array of stopping times to use

# Calculate accuracies for each stop time
accuracies, accuracies_analytical, _ = simulate_accuracy_vs_stoptime(mu, sigma, stop_time_list,
                                                   num_sample)

# Visualize
with plt.xkcd():
  plot_accuracy_vs_stoptime(mu, sigma, stop_time_list, accuracies_analytical, accuracies)
```

In the figure above, we are plotting the simulated accuracies in orange. We can actually find an analytical equation for the average accuracy in this specific case, which we plot in blue. We will not dive into this analytical solution here but you can imagine that if you ran a bunch of different simulations and had the equivalent number of orange lines, the average of those would resemble the blue line.

In the insets, we are showing the evidence distributions for the two states at a certain time point. Recall from Section 1 that the likelihood ratio at time $T$ for state of +1 is: $$L_T\sim\mathcal{N}\left(2\frac{\mu^2}{\sigma^2}T,\ 4\frac{\mu^2}{\sigma^2}T\right)=\mathcal{N}(bT,c^2T)$$

If the state is -1, the mean is the reverse sign. We are plotting this Gaussian distribution for the state equaling -1 (dashed line) and the state equaling +1 (solid line). The area in red reflects the error rate - this region corresponds to $L_T$ being below 0 even though the true state is +1 so you would decide on the wrong state. As more time goes by, these distributions separate more and the error is lower.

### Interactive Demo 2.2: Accuracy versus stop-time

For this same visualization, now vary the mean $\mu$ and standard deviation `sigma` of the evidence. What do you predict will the accuracy vs stopping time plot look like for low noise and high noise?

```python
#@markdown Make sure you execute this cell to enable the widget!
def simulate_accuracy_vs_stoptime(mu, sigma, stop_time_list, num_sample, no_numerical=False):
  """Calculate the average decision accuracy vs. stopping time by running
  repeated SPRT simulations for each stop time.

  Args:
      mu (float): absolute mean value of the symmetric observation distributions
      sigma (float): standard deviation for observation model
      stop_list_list (list-like object): a list of stopping times to run over
      num_sample (int): number of simulations to run per stopping time
      no_numerical (bool): flag that indicates the function to return analytical values only

  Returns:
      accuracy_list: a list of average accuracies corresponding to input
                      `stop_time_list`
      decisions_list: a list of decisions made in all trials
  """

  # Determine true state (1 or -1)
  true_dist = 1

  # Set up tracker of accuracy and decisions
  accuracies = np.zeros(len(stop_time_list),)
  accuracies_analytical = np.zeros(len(stop_time_list),)
  decisions_list = []

  # Loop over stop times
  for i_stop_time, stop_time in enumerate(stop_time_list):

    if not no_numerical:
      # Set up tracker of decisions for this stop time
      decisions = np.zeros((num_sample,))

      # Loop over samples
      for i in range(num_sample):

        # Simulate run for this stop time (hint: last exercise)
        _, decision, _= simulate_SPRT_fixedtime(mu, sigma, stop_time, true_dist)

        # Log decision
        decisions[i] = decision

      # Calculate accuracy
      accuracies[i_stop_time] = np.sum(decisions == true_dist) / decisions.shape[0]
      # Log decisions
      decisions_list.append(decisions)

    # Calculate analytical accuracy
    sigma_sum_gaussian = sigma / np.sqrt(stop_time)
    accuracies_analytical[i_stop_time] = 0.5 + 0.5 * erf(mu / np.sqrt(2) / sigma_sum_gaussian)

  return accuracies, accuracies_analytical, decisions_list

np.random.seed(100)
num_sample = 100
stop_time_list = np.arange(1, 100, 1)

@widgets.interact
def plot(mu=widgets.FloatSlider(min=0.1, max=5.0, step=0.1, value=1.0), sigma=(0.05, 10.0, 0.05)):
 # Calculate accuracies for each stop time
  _, accuracies_analytical, _ = simulate_accuracy_vs_stoptime(mu, sigma, stop_time_list, num_sample, no_numerical=True)

  # Visualize
  plot_accuracy_vs_stoptime(mu, sigma, stop_time_list, accuracies_analytical)
```

```python
# to_remove explanation

"""

1) Low levels of noise results in higher accuracies generally, especially
   at early stop times.

2) High levels of noise results in lower accuracies generally.

"""
```

```python
# @title Video 5: Section 2 Exercises Discussion
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV11g411M7Lm", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="OBDv6nB6a2g", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

**Application**

We have looked at the drift diffusion model of decisions in the context of the fishing problem. There are lots of uses of this in neuroscience! As one example, a classic experimental task in neuroscience is the random dot kinematogram ([Newsome, Britten, Movshon 1989](https://www.nature.com/articles/341052a0.pdf)), in which a pattern of moving dots are moving in random directions but with some weak coherence that favors a net rightward or leftward motion. The observer must guess the direction. Neurons in the brain are informative about this task, and have responses that correlate with the choice, as predicted by the Drift Diffusion Model (Huk and Shadlen 2005).

Below is a video by Pamela Reinagle of a rat guessing the direction of motion in such a task.

```python
# @markdown Rat performing random dot motion task
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="oDxcyTn-0os", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

After you finish the other tutorials, come back to see Bonus material to learn about a different stopping rule for DDMs: a fixed threshold on confidence.

---
# Summary

*Estimated timing of tutorial: 45 minutes*

Good job! By simulating Drift Diffusion Models, you have learnt how to:

* Calculate individual sample evidence as the log likelihood ratio of two candidate models
* Accumulate evidence from new data points, and compute posterior using recursive formula
* Run repeated simulations to get an estimate of decision accuracies
* Measure the speed-accuracy tradeoff

---
# Bonus

---
## Bonus Section 1: DDM with fixed thresholds on confidence

```python
# @title Video 6: Fixed threshold on confidence

from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Ya4y1a7c1", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="E8lvgFeIGQM", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

The next exercises consider a variant of the DDM with fixed confidence thresholds instead of fixed decision time. This may be a better description of neural integration. Please complete this material after you have finished the main content of all tutorials, if you would like extra information about this topic.

### Bonus Coding Exercise 1.1, Coding: Simulating the DDM with fixed confidence thresholds

*Referred to as exercise 3 in video*

In this exercise, we will use thresholding as our stopping rule and observe the behavior of the DDM.

With thresholding stopping rule, we define a desired error rate and will continue making measurements until that error rate is reached. Experimental evidence suggested that evidence accumulation and thresholding stopping strategy happens at neuronal level (see [this article](https://www.annualreviews.org/doi/full/10.1146/annurev.neuro.29.051605.113038) for further reading).

* Complete the function `threshold_from_errorrate` to calculate the evidence threshold from desired error rate $\alpha$ as described in the formulas below. The evidence thresholds $th_1$ and $th_0$ for $p_+$ and $p_-$ are opposite of each other as shown below, so you can just return the absolute value.
$$
\begin{align}
 th_{L} &= \log \frac{\alpha}{1-\alpha} &= -th_{R} \\
 th_{R} &= \log \frac{1-\alpha}{\alpha} &= -th{_1}\\
 \end{align}
 $$

* Complete the function `simulate_SPRT_threshold` to simulate an SPRT with thresholding stopping rule given noise level and desired threshold

* Run repeated simulations for a given noise level and a desired error rate visualize the DDM traces using our provided code

```python
def simulate_SPRT_threshold(mu, sigma, threshold , true_dist=1):
  """Simulate a Sequential Probability Ratio Test with thresholding stopping
  rule. Two observation models are 1D Gaussian distributions N(1,sigma^2) and
  N(-1,sigma^2).

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation
    threshold (float): Desired log likelihood ratio threshold to achieve
                        before making decision

  Returns:
    evidence_history (numpy vector): the history of cumulated evidence given
                                      generated data
    decision (int): 1 for pR, 0 for pL
    data (numpy vector): the generated sequences of data in this trial
  """
  assert mu > 0, "Mu should be > 0"
  muL = -mu
  muR = mu

  pL = stats.norm(muL, sigma)
  pR = stats.norm(muR, sigma)

  has_enough_data = False

  data_history = []
  evidence_history = []
  current_evidence = 0.0

  # Keep sampling data until threshold is crossed
  while not has_enough_data:
    if true_dist == 1:
      Mvec = pR.rvs()
    else:
      Mvec = pL.rvs()

    ########################################################################
    # Insert your code here to:
    #      * Calculate the log-likelihood ratio for the new sample
    #      * Update the accumulated evidence
    raise NotImplementedError("`simulate_SPRT_threshold` is incomplete")
    ########################################################################

    # STEP 1: individual log likelihood ratios
    ll_ratio = log_likelihood_ratio(...)

    # STEP 2: accumulated evidence for this chunk
    evidence_history.append(...)

    # update the collection of all data
    data_history.append(Mvec)
    current_evidence = evidence_history[-1]

    # check if we've got enough data
    if abs(current_evidence) > threshold:
      has_enough_data = True

  data_history = np.array(data_history)
  evidence_history = np.array(evidence_history)

  # Make decision
  if evidence_history[-1] >= 0:
    decision = 1
  elif evidence_history[-1] < 0:
    decision = 0

  return evidence_history, decision, data_history

# Set parameters
np.random.seed(100)
mu = 1.0
sigma = 2.8
num_sample = 10
log10_alpha = -3 # log10(alpha)
alpha = np.power(10.0, log10_alpha)

# Simulate and visualize
simulate_and_plot_SPRT_fixedthreshold(mu, sigma, num_sample, alpha)
```

```python
# to_remove solution
def simulate_SPRT_threshold(mu, sigma, threshold , true_dist=1):
  """Simulate a Sequential Probability Ratio Test with thresholding stopping
  rule. Two observation models are 1D Gaussian distributions N(1,sigma^2) and
  N(-1,sigma^2).

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation
    threshold (float): Desired log likelihood ratio threshold to achieve
                        before making decision

  Returns:
    evidence_history (numpy vector): the history of cumulated evidence given
                                      generated data
    decision (int): 1 for pR, 0 for pL
    data (numpy vector): the generated sequences of data in this trial
  """
  assert mu > 0, "Mu should be > 0"
  muL = -mu
  muR = mu

  pL = stats.norm(muL, sigma)
  pR = stats.norm(muR, sigma)

  has_enough_data = False

  data_history = []
  evidence_history = []
  current_evidence = 0.0

  # Keep sampling data until threshold is crossed
  while not has_enough_data:
    if true_dist == 1:
      Mvec = pR.rvs()
    else:
      Mvec = pL.rvs()

    # STEP 1: individual log likelihood ratios
    ll_ratio = log_likelihood_ratio(Mvec, pL, pR)

    # STEP 2: accumulated evidence for this chunk
    evidence_history.append(ll_ratio + current_evidence)

    # update the collection of all data
    data_history.append(Mvec)
    current_evidence = evidence_history[-1]

    # check if we've got enough data
    if abs(current_evidence) > threshold:
      has_enough_data = True

  data_history = np.array(data_history)
  evidence_history = np.array(evidence_history)

  # Make decision
  if evidence_history[-1] >= 0:
    decision = 1
  elif evidence_history[-1] < 0:
    decision = 0

  return evidence_history, decision, data_history

# Set parameters
np.random.seed(100)
mu = 1.0
sigma = 2.8
num_sample = 10
log10_alpha = -3 # log10(alpha)
alpha = np.power(10.0, log10_alpha)

# Simulate and visualize
with plt.xkcd():
  simulate_and_plot_SPRT_fixedthreshold(mu, sigma, num_sample, alpha)
```

### Bonus Interactive Demo 1.2: DDM with fixed confidence threshold

Play with difference values of `alpha` and `sigma` and observe how that affects the dynamics of Drift-Diffusion Model.

```python
# @markdown Make sure you execute this cell to enable the widget!
def simulate_SPRT_threshold(mu, sigma, threshold , true_dist=1):
  """Simulate a Sequential Probability Ratio Test with thresholding stopping
  rule. Two observation models are 1D Gaussian distributions N(1,sigma^2) and
  N(-1,sigma^2).

  Args:
    mu (float): absolute mean value of the symmetric observation distributions
    sigma (float): Standard deviation
    threshold (float): Desired log likelihood ratio threshold to achieve
                        before making decision

  Returns:
    evidence_history (numpy vector): the history of cumulated evidence given
                                      generated data
    decision (int): 1 for pR, 0 for pL
    data (numpy vector): the generated sequences of data in this trial
  """
  assert mu > 0, "Mu should be > 0"
  muL = -mu
  muR = mu

  pL = stats.norm(muL, sigma)
  pR = stats.norm(muR, sigma)

  has_enough_data = False

  data_history = []
  evidence_history = []
  current_evidence = 0.0

  # Keep sampling data until threshold is crossed
  while not has_enough_data:
    if true_dist == 1:
      Mvec = pR.rvs()
    else:
      Mvec = pL.rvs()

    # STEP 1: individual log likelihood ratios
    ll_ratio = log_likelihood_ratio(Mvec, pL, pR)

    # STEP 2: accumulated evidence for this chunk
    evidence_history.append(ll_ratio + current_evidence)

    # update the collection of all data
    data_history.append(Mvec)
    current_evidence = evidence_history[-1]

    # check if we've got enough data
    if abs(current_evidence) > threshold:
      has_enough_data = True

  data_history = np.array(data_history)
  evidence_history = np.array(evidence_history)

  # Make decision
  if evidence_history[-1] >= 0:
    decision = 1
  elif evidence_history[-1] < 0:
    decision = 0

  return evidence_history, decision, data_history

np.random.seed(100)
num_sample = 10

@widgets.interact
def plot(mu=(0.1,5.0,0.1), sigma=(0.05, 10.0, 0.05), log10_alpha=(-8, -1, .1)):
  alpha = np.power(10.0, log10_alpha)
  simulate_and_plot_SPRT_fixedthreshold(mu, sigma, num_sample, alpha, verbose=False)
```

### Bonus Coding Exercise 1.3: Speed/Accuracy Tradeoff Revisited

The faster you make a decision, the lower your accuracy often is. This phenomenon is known as the **speed/accuracy tradeoff**. Humans can make this tradeoff in a wide range of situations, and many animal species, including ants, bees, rodents, and monkeys also show similar effects.

To illustrate the speed/accuracy tradeoff under thresholding stopping rule, let's run some simulations under different thresholds and look at how average decision "speed" (1/length) changes with average decision accuracy. We use speed rather than accuracy because in real experiments, subjects can be incentivized to respond faster or slower; it's much harder to precisely control their decision time or error threshold.

* Complete the function `simulate_accuracy_vs_threshold` to simulate and compute average accuracies vs. average decision lengths for a list of error thresholds. You will need to supply code to calculate average decision 'speed' from the lengths of trials. You should also calculate the overall accuracy across these trials.

* We've set up a list of error thresholds. Run repeated simulations and collect average accuracy with average length for each error rate in this list, and use our provided code to visualize the speed/accuracy tradeoff. You should see a positive correlation between length and accuracy.

```python
def simulate_accuracy_vs_threshold(mu, sigma, threshold_list, num_sample):
  """Calculate the average decision accuracy vs. average decision length by
  running repeated SPRT simulations with thresholding stopping rule for each
  threshold.

  Args:
      mu (float): absolute mean value of the symmetric observation distributions
      sigma (float): standard deviation for observation model
      threshold_list (list-like object): a list of evidence thresholds to run
                                          over
      num_sample (int): number of simulations to run per stopping time

  Returns:
      accuracy_list: a list of average accuracies corresponding to input
                      `threshold_list`
      decision_speed_list: a list of average decision speeds
  """
  decision_speed_list = []
  accuracy_list = []
  for threshold in threshold_list:
    decision_time_list = []
    decision_list = []
    for i in range(num_sample):
      # run simulation and get decision of current simulation
      _, decision, Mvec = simulate_SPRT_threshold(mu, sigma, threshold)
      decision_time = len(Mvec)
      decision_list.append(decision)
      decision_time_list.append(decision_time)

    ########################################################################
    # Insert your code here to:
    #      * Calculate mean decision speed given a list of decision times
    #      * Hint: Think about speed as being inversely proportional
    #        to decision_length. If it takes 10 seconds to make one decision,
    #        our "decision speed" is 0.1 decisions per second.
    #      * Calculate the decision accuracy
    raise NotImplementedError("`simulate_accuracy_vs_threshold` is incomplete")
    ########################################################################
    # Calculate and store average decision speed and accuracy
    decision_speed = ...
    decision_accuracy = ...
    decision_speed_list.append(decision_speed)
    accuracy_list.append(decision_accuracy)

  return accuracy_list, decision_speed_list

# Set parameters
np.random.seed(100)
mu = 1.0
sigma = 3.75
num_sample = 200
alpha_list = np.logspace(-2, -0.1, 8)
threshold_list = threshold_from_errorrate(alpha_list)

# Simulate and visualize
simulate_and_plot_accuracy_vs_threshold(mu, sigma, threshold_list, num_sample)
```

```python
# to_remove solution
def simulate_accuracy_vs_threshold(mu, sigma, threshold_list, num_sample):
  """Calculate the average decision accuracy vs. average decision speed by
  running repeated SPRT simulations with thresholding stopping rule for each
  threshold.

  Args:
      mu (float): absolute mean value of the symmetric observation distributions
      sigma (float): standard deviation for observation model
      threshold_list (list-like object): a list of evidence thresholds to run
                                          over
      num_sample (int): number of simulations to run per stopping time

  Returns:
      accuracy_list: a list of average accuracies corresponding to input
                      `threshold_list`
      decision_speed_list: a list of average decision speeds
  """
  decision_speed_list = []
  accuracy_list = []
  for threshold in threshold_list:
    decision_time_list = []
    decision_list = []
    for i in range(num_sample):
      # run simulation and get decision of current simulation
      _, decision, Mvec = simulate_SPRT_threshold(mu, sigma, threshold)
      decision_time = len(Mvec)
      decision_list.append(decision)
      decision_time_list.append(decision_time)

    # Calculate and store average decision speed and accuracy
    decision_speed = np.mean(1. / np.array(decision_time_list))
    decision_accuracy = sum(decision_list) / len(decision_list)
    decision_speed_list.append(decision_speed)
    accuracy_list.append(decision_accuracy)

  return accuracy_list, decision_speed_list

# Set parameters
np.random.seed(100)
mu = 1.0
sigma = 3.75
num_sample = 200
alpha_list = np.logspace(-2, -0.1, 8)
threshold_list = threshold_from_errorrate(alpha_list)

# Simulate and visualize
with plt.xkcd():
  simulate_and_plot_accuracy_vs_threshold(mu, sigma, threshold_list, num_sample)
```

### Bonus Interactive demo 1.4: Speed/Accuracy with a threshold rule

Manipulate the noise level `sigma` and observe how that affects the speed/accuracy tradeoff.

```python
# @markdown Make sure you execute this cell to enable the widget!
def simulate_accuracy_vs_threshold(mu, sigma, threshold_list, num_sample):
  """Calculate the average decision accuracy vs. average decision speed by
  running repeated SPRT simulations with thresholding stopping rule for each
  threshold.

  Args:
      mu (float): absolute mean value of the symmetric observation distributions
      sigma (float): standard deviation for observation model
      threshold_list (list-like object): a list of evidence thresholds to run
                                          over
      num_sample (int): number of simulations to run per stopping time

  Returns:
      accuracy_list: a list of average accuracies corresponding to input
                      `threshold_list`
      decision_speed_list: a list of average decision speeds
  """
  decision_speed_list = []
  accuracy_list = []
  for threshold in threshold_list:
    decision_time_list = []
    decision_list = []
    for i in range(num_sample):
      # run simulation and get decision of current simulation
      _, decision, Mvec = simulate_SPRT_threshold(mu, sigma, threshold)
      decision_time = len(Mvec)
      decision_list.append(decision)
      decision_time_list.append(decision_time)

    # Calculate and store average decision speed and accuracy
    decision_speed = np.mean(1. / np.array(decision_time_list))
    decision_accuracy = sum(decision_list) / len(decision_list)
    decision_speed_list.append(decision_speed)
    accuracy_list.append(decision_accuracy)

  return accuracy_list, decision_speed_list

np.random.seed(100)
num_sample = 100
alpha_list = np.logspace(-2, -0.1, 8)
threshold_list = threshold_from_errorrate(alpha_list)

@widgets.interact
def plot(mu=(0.1, 5.0, 0.1), sigma=(0.05, 10.0, 0.05)):
  alpha = np.power(10.0, log10_alpha)
  simulate_and_plot_accuracy_vs_threshold(mu, sigma, threshold_list, num_sample)
```


# Detector simulation
In this notebook we load a track dataset generated by `edep-sim` and we calculate the ADC counts corresponding to each pixel. The result is exported to a HDF5 file.

```python
# This is need so you can import larndsim without doing python setup.py install
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
```

```python
from math import ceil
from time import time

import matplotlib.pyplot as plt
from matplotlib import cm, colors
import mpl_toolkits.mplot3d.art3d as art3d

import numpy as np
import cupy as cp
import fire
import h5py

from numba import cuda
from numba.cuda.random import create_xoroshiro128p_states

import matplotlib as mpl
mpl.rcParams['figure.dpi'] = 100

from tqdm.notebook import tqdm
```

```python
from larndsim import consts
import importlib
importlib.reload(consts)
consts.load_detector_properties("../larndsim/detector_properties/module0.yaml","../larndsim/pixel_layouts/multi_tile_layout-2.2.16.yaml")

from larndsim import quenching, drifting, detsim, pixels_from_track, fee
```

```python
response = cp.load('../larndsim/response.npy')
```

### Dataset import
First of all we load the `edep-sim` output. For this sample we need to invert $z$ and $y$ axes.

```python
with h5py.File('module0_corsika.h5', 'r') as f:
    tracks = np.array(f['segments'])

x_start = np.copy(tracks['x_start'] )
x_end = np.copy(tracks['x_end'])
x = np.copy(tracks['x'])

tracks['x_start'] = np.copy(tracks['z_start'])
tracks['x_end'] = np.copy(tracks['z_end'])
tracks['x'] = np.copy(tracks['z'])

tracks['z_start'] = x_start
tracks['z_end'] = x_end
tracks['z'] = x

selected_tracks = tracks[:5]
```

### Quenching and drifting
We calculate the number of electrons after recombination (`quenching` module) and the position and number of electrons after drifting (`drifting` module).

```python
TPB = 256
BPG = ceil(selected_tracks.shape[0] / TPB)
quenching.quench[BPG,TPB](selected_tracks, consts.birks)
drifting.drift[BPG,TPB](selected_tracks)
```

We find the pixels intersected by the projection of the tracks on the anode plane using the Bresenham's algorithm. We also take into account the neighboring pixels, due to the transverse diffusion of the charges.

```python
# Here we build a map between tracks and event IDs
unique_eventIDs = cp.unique(selected_tracks['eventID'])
event_id_map = cp.searchsorted(unique_eventIDs,cp.asarray(selected_tracks['eventID']))

longest_pix = ceil(max(selected_tracks["dx"])/consts.pixel_pitch)
max_radius = ceil(max(selected_tracks["tran_diff"])*5/consts.pixel_pitch)

MAX_PIXELS = int((longest_pix*4+6)*max_radius*1.5)
MAX_ACTIVE_PIXELS = int(longest_pix*1.5)

active_pixels = cp.full((selected_tracks.shape[0], MAX_ACTIVE_PIXELS, 2), -1, dtype=np.int32)
neighboring_pixels = cp.full((selected_tracks.shape[0], MAX_PIXELS, 2), -1, dtype=np.int32)
n_pixels_list = cp.zeros(shape=(selected_tracks.shape[0]))
TPB = 128
BPG = ceil(selected_tracks.shape[0] / TPB)
pixels_from_track.get_pixels[BPG,TPB](selected_tracks,
                                      active_pixels,
                                      neighboring_pixels,
                                      n_pixels_list,
                                      max_radius+1)
```

```python
def cupy_unique_axis0(array):
    # axis is still not supported for cupy.unique, this
    # is a workaround
    if len(array.shape) != 2:
        raise ValueError("Input array must be 2D.")
    sortarr     = array[cp.lexsort(array.T[::-1])]
    mask        = cp.empty(array.shape[0], dtype=cp.bool_)
    mask[0]     = True
    mask[1:]    = cp.any(sortarr[1:] != sortarr[:-1], axis=1)
    return sortarr[mask]

shapes = neighboring_pixels.shape
joined = neighboring_pixels.reshape(shapes[0]*shapes[1],2)
unique_pix = cupy_unique_axis0(joined)
unique_pix = unique_pix[(unique_pix[:,0] != -1) & (unique_pix[:,1] != -1),:]
```

### Charge distribution calculation
Here we calculate the current induced by each track on the pixels, taking into account longitudinal and transverse diffusion. The track segment is parametrized as:
\begin{align}
x'(r') &=x_s + \frac{\Delta x}{\Delta r}r'\\
y'(r') &=y_s + \frac{\Delta y}{\Delta r}r'\\
z'(r') &=z_s + \frac{\Delta z}{\Delta r}r',
\end{align}
where $\Delta r$ is the segment length. Here we assume $z$ as the drift direction.
The diffused charge distribution is calculated with the following integral:
\begin{equation}
\rho(x,y,z) = \frac{Q}{\sqrt{(2\pi)^3}\sigma_x\sigma_y\sigma_z\Delta r}\exp\left[-\frac{(x-x_s)^2}{2\sigma_x^2}-\frac{(y-y_s)^2}{2\sigma_y^2}-\frac{(z-z_s)^2}{2\sigma_z^2}\right]\int^{r'=\Delta r}_{r'=0}dr'\exp[-(ar'^2+br')],
\end{equation}
where
\begin{align}
a &= \left[\left(\frac{\Delta x}{\Delta r}\right)^2\frac{1}{2\sigma_x^2} + \left(\frac{\Delta y}{\Delta r}\right)^2\frac{1}{2\sigma_y^2} + \left(\frac{\Delta z}{\Delta r}\right)^2\frac{1}{2\sigma_z^2} \right]\\
b &= -\left[\frac{(x-x_s)}{\sigma_x^2}\frac{\Delta x}{\Delta r}+
\frac{(y-y_s)}{\sigma_y^2}\frac{\Delta y}{\Delta r}+
\frac{(z-z_s)}{\sigma_z^2}\frac{\Delta z}{\Delta r}\right].
\end{align}

The simmetry of the transverse diffusion along the track allows to take a slice on the $xy$ plane and solve the integral once at a fixed $z$ coordinate (e.g. at $z_{m} = (z_s+z_e)/2$) and re-use it at other $z$ coordinates away from the endpoints (where $\rho(x,y,z)$ varies along $z$ so must be calculated at each $z$).

```python
max_length = cp.array([0])
track_starts = cp.empty(selected_tracks.shape[0])
threadsperblock = 128
blockspergrid = ceil(selected_tracks.shape[0] / threadsperblock)
detsim.time_intervals[blockspergrid,threadsperblock](track_starts, max_length,  event_id_map, selected_tracks)

signals = cp.zeros((selected_tracks.shape[0],
                    neighboring_pixels.shape[1],
                    cp.asnumpy(max_length)[0]), dtype=np.float32)
threadsperblock = (1,1,64)
importlib.reload(detsim)
blockspergrid_x = ceil(signals.shape[0] / threadsperblock[0])
blockspergrid_y = ceil(signals.shape[1] / threadsperblock[1])
blockspergrid_z = ceil(signals.shape[2] / threadsperblock[2])
blockspergrid = (blockspergrid_x, blockspergrid_y, blockspergrid_z)
detsim.tracks_current[blockspergrid,threadsperblock](signals,
                                                     neighboring_pixels,
                                                     selected_tracks,
                                                     response)
```

Here we create a map that associates each track to the pixels where it induces a current

```python
pixel_index_map = cp.full((selected_tracks.shape[0], neighboring_pixels.shape[1]), -1)

try:
    compare = neighboring_pixels[..., np.newaxis, :] == unique_pix
    indices = cp.where(cp.logical_and(compare[..., 0], compare[..., 1]))
except cp.cuda.memory.OutOfMemoryError:
    print("out of memory")

pixel_index_map[indices[0], indices[1]] = indices[2]
```

The three-dimensional array can contain more than one signal for each pixel at different times. If we want to plot the full induced signal on the pixel, we need to join the signals corresponding to the same pixel. First, we find the start time of each signal with `time_intervals`:

```python
# Mapping between unique pixel array and track array index
track_pixel_map = cp.full((unique_pix.shape[0], detsim.MAX_TRACKS_PER_PIXEL), -1)
TPB = 32
BPG = ceil(unique_pix.shape[0] / TPB)
detsim.get_track_pixel_map[BPG, TPB](track_pixel_map, unique_pix, neighboring_pixels)

threadsperblock = (1,1,64)
blockspergrid_x = ceil(signals.shape[0] / threadsperblock[0])
blockspergrid_y = ceil(signals.shape[1] / threadsperblock[1])
blockspergrid_z = ceil(signals.shape[2] / threadsperblock[2])
blockspergrid = (blockspergrid_x, blockspergrid_y, blockspergrid_z)
pixels_signals = cp.zeros((len(unique_pix), len(consts.time_ticks)*3))
pixels_tracks_signals = cp.zeros((len(unique_pix),len(consts.time_ticks)*3,track_pixel_map.shape[1]))

detsim.sum_pixel_signals[blockspergrid,threadsperblock](pixels_signals,
                                                        signals,
                                                        track_starts,
                                                        pixel_index_map,
                                                        track_pixel_map,
                                                        pixels_tracks_signals)
currents = np.sum(pixels_signals,axis=1)*consts.t_sampling/consts.e_charge
```

Now, let's verify that the integral of our induced currents

```python
print("Ratio: ", sum(currents)/sum(selected_tracks['n_electrons']))
```

    Ratio:  1.0194242566692833

### 3D event display

```python
cmap = cm.Spectral_r
from matplotlib import colors
norm = colors.Normalize(vmin=0, vmax=256)
m = cm.ScalarMappable(norm=norm, cmap=cmap)

cmap = cm.viridis
norm_curr = colors.LogNorm(vmin=min(currents[currents>0]), vmax=max(currents))
m_curr = cm.ScalarMappable(norm=norm_curr, cmap=cmap)
fig = plt.figure(figsize=(7,7))
ax = fig.add_subplot(111, projection='3d')

for it,t in enumerate(selected_tracks):
    if it == 0:
        ax.plot((t["x_start"], t["x_end"]),
            (t["z_start"], t["z_end"]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1,
            alpha=1,
            zorder=10,
            label='Geant4 detector segment')
    else:
        ax.plot((t["x_start"], t["x_end"]),
                (t["z_start"], t["z_end"]),
                (t["y_start"], t["y_end"]),
                c='r',
                lw=1,
                alpha=1,
                zorder=9999)

    ax.plot((t["x_start"], t["x_end"]),
            (consts.tpc_borders[t['pixel_plane']][2][0], consts.tpc_borders[t['pixel_plane']][2][0]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1,
            ls=':',
            alpha=1,
            zorder=9999)

for ip, p in enumerate(unique_pix):
    x_rect, y_rect = detsim.get_pixel_coordinates(p.get())
    pixel_plane = int(p[0] // consts.n_pixels[0])
    row = int(pixel_plane // 4)
    column = int(pixel_plane % 4)
    if currents[ip] > 0:
        rect = plt.Rectangle((x_rect, y_rect),
                             consts.pixel_pitch, consts.pixel_pitch,
                             linewidth=0.1, fc=m_curr.to_rgba(currents[ip].get()),
                             edgecolor='gray')
        ax.add_patch(rect)
        art3d.pathpatch_2d_to_3d(rect, z=consts.tpc_borders[pixel_plane][2][0], zdir="y")

anode1 = plt.Rectangle((consts.tpc_borders[0][0][0], consts.tpc_borders[0][1][0]),
                        consts.tpc_borders[0][0][1]-consts.tpc_borders[0][0][0],
                        consts.tpc_borders[0][1][1]-consts.tpc_borders[0][1][0],
                       linewidth=1, fc='none',
                       edgecolor='gray', label=('Pixel' if ip == 5 else ''))
ax.add_patch(anode1)
art3d.pathpatch_2d_to_3d(anode1, z=consts.tpc_borders[0][2][0], zdir="y")

anode2 = plt.Rectangle((consts.tpc_borders[0][0][0], consts.tpc_borders[0][1][0]),
                        consts.tpc_borders[0][0][1]-consts.tpc_borders[0][0][0],
                        consts.tpc_borders[0][1][1]-consts.tpc_borders[0][1][0],
                       linewidth=1, fc='none',
                       edgecolor='gray', label=('Pixel' if ip == 5 else ''))
ax.add_patch(anode2)
art3d.pathpatch_2d_to_3d(anode2, z=consts.tpc_borders[1][2][0], zdir="y")

cathode = plt.Rectangle((consts.tpc_borders[0][0][0], consts.tpc_borders[0][1][0]),
                        consts.tpc_borders[0][0][1]-consts.tpc_borders[0][0][0],
                        consts.tpc_borders[0][1][1]-consts.tpc_borders[0][1][0],
                        linewidth=1, fc='gray', alpha=0.5,
                        edgecolor='gray', label=('Pixel' if ip == 5 else ''))
ax.add_patch(cathode)
art3d.pathpatch_2d_to_3d(cathode, z=0, zdir="y")

ax.plot((consts.tpc_borders[0][0][0],consts.tpc_borders[0][0][0]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][0],consts.tpc_borders[0][1][0]), lw=1,color='gray')

ax.plot((consts.tpc_borders[0][0][0],consts.tpc_borders[0][0][0]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][1],consts.tpc_borders[0][1][1]), lw=1,color='gray')

ax.plot((consts.tpc_borders[0][0][1],consts.tpc_borders[0][0][1]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][0],consts.tpc_borders[0][1][0]), lw=1,color='gray')

ax.plot((consts.tpc_borders[0][0][1],consts.tpc_borders[0][0][1]),(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0]),
        (consts.tpc_borders[0][1][1],consts.tpc_borders[0][1][1]), lw=1,color='gray')

# ax.set_ylim(consts.module_borders[pixel_plane][2][0],50)
ax.set_xlim(consts.tpc_borders[0][0][0],consts.tpc_borders[1][0][1])
ax.set_ylim(consts.tpc_borders[0][2][0],consts.tpc_borders[1][2][0])
ax.set_zlim(consts.tpc_borders[0][1][0],consts.tpc_borders[1][1][1])

ax.set_box_aspect((2,2,4))
ax.grid(False)
ax.xaxis.set_major_locator(plt.MaxNLocator(3))
ax.yaxis.set_major_locator(plt.MaxNLocator(3))
# ax.set_axis_off()
ax.w_xaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))
ax.w_yaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))
ax.w_zaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))

ax.view_init(azim=20)
# def rotate(angle):
#     ax.view_init(azim=angle)

# from matplotlib import animation

# rot_animation = animation.FuncAnimation(fig, rotate, frames=np.arange(0,362,2),interval=50)

ax.set_ylabel("z [cm]")
ax.set_xlabel("x [cm]")
ax.set_zlabel("y [cm]")
_ = plt.colorbar(m_curr,fraction=0.035, pad=0.05,label='Induced current integral [# electrons]')

```

```python
# rot_animation.save('animation.gif', writer='imagemagick', fps=10, bitrate=10, dpi=100)
```

### Electronics response and digitization
Here we simulate the electronics response (the self-triggering cycle) and the signal digitization.

```python
time_ticks = cp.linspace(0, len(unique_eventIDs)*consts.time_interval[1]*3, pixels_signals.shape[1]+1)
integral_list = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES))
adc_ticks_list = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES))
TPB = 128
BPG = ceil(pixels_signals.shape[0] / TPB)
rng_states = create_xoroshiro128p_states(TPB * BPG, seed=2)
integrate = cp.zeros((pixels_signals.shape[0], time_ticks.shape[0]))
backtracked_ids = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES, track_pixel_map.shape[1]))
importlib.reload(fee)
fee.get_adc_values[BPG,TPB](pixels_signals,
                            pixels_tracks_signals,
                            time_ticks,
                            integral_list,
                            adc_ticks_list,
                            0,
                            rng_states,
                            backtracked_ids)
adc_list = fee.digitize(integral_list)
```

As an example, we plot the pixel with the largest amount of deposited charge

```python
largest_pix = np.argmax(pixels_signals.sum(axis=1)).get()
```

```python
fig,ax=plt.subplots(1,1)
current = pixels_signals[largest_pix][:2001].get()
ax.plot(consts.time_ticks,pixels_tracks_signals[largest_pix][:2001].get())
ax.plot(consts.time_ticks,current,label='Total track contribution',c='k',lw=2)
ax.plot(consts.time_ticks,np.sum(pixels_tracks_signals[largest_pix][:2001].get(),axis=1),c='r',ls='--',label='Sum of each track contribution')
start_time = np.nonzero(current)[0][0]
ax.set_xlim(consts.time_ticks[start_time]-10,consts.time_ticks[start_time]+30)
ax.set_xlabel(r"Time [$\mathrm{\mu}$s]")
ax.set_ylabel(r"Current [A]")
ax.set_ylim(-0.1e-14, max(current)*1.3)
_=ax.legend()
```

### 2D event display with induced current and ADC counts

```python
from matplotlib import colors
norm = colors.Normalize(vmin=0, vmax=256)
m = cm.ScalarMappable(norm=norm, cmap=cmap)

cmap = cm.viridis
norm_curr = colors.LogNorm(vmin=min(currents[currents>0]), vmax=max(currents))
m_curr = cm.ScalarMappable(norm=norm_curr, cmap=cmap)

fig,ax = plt.subplots(1,2,figsize=(10,4.5),sharex=True,sharey=True)
importlib.reload(detsim)
for ip, p in enumerate(unique_pix):
    x_rect, y_rect = detsim.get_pixel_coordinates(p.get())
    pixel_plane = int(p[0] // consts.n_pixels[0])
    c = currents[ip].get()
    if c >= 1:
        rect = plt.Rectangle((x_rect, y_rect+21.8236),
                             consts.pixel_pitch, consts.pixel_pitch,
                             linewidth=0.2, fc=m_curr.to_rgba(c),
                             edgecolor='grey')
        ax[0].add_patch(rect)

    a = adc_list[ip][adc_list[ip]>fee.digitize(0)]
    if len(a):
        rect = plt.Rectangle((x_rect, y_rect+21.8236),
                             consts.pixel_pitch, consts.pixel_pitch,
                             linewidth=0.2, fc=m.to_rgba(np.sum(a.get())),
                             edgecolor='grey')
        ax[1].add_patch(rect)

for it,t in enumerate(selected_tracks):
    ax[0].plot((t["x_start"], t["x_end"]),
            (t["y_start"]+21.8236, t["y_end"]+21.8236),
            c='r',
            lw=1.25,
            ls=':',
            alpha=1,
            zorder=10)
    ax[1].plot((t["x_start"], t["x_end"]),
            (t["y_start"]+21.8236, t["y_end"]+21.8236),
            c='r',
            lw=1.25,
            ls=':',
            alpha=1,
            zorder=100)
    ax[0].scatter((t["x_start"], t["x_end"]),
                  (t["y_start"]+21.8236, t["y_end"]+21.8236),
                  c='r', s=1, zorder=99999)
#     ax[1].scatter((t["x_start"], t["x_end"]),
#                   (t["y_start"], t["y_end"]),
#                   c='r', s=1, zorder=99999)

ax[0].set_aspect("equal")
ax[1].set_aspect("equal")
ax[0].set_xlabel("x [cm]")
ax[1].set_xlabel("x [cm]")
ax[0].set_ylabel("y [cm]")
from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable

divider0 = make_axes_locatable(ax[1])
cax0 = divider0.append_axes("right", size="7%", pad=0.07)
fig.colorbar(m, ax=ax[1], cax=cax0, label='ADC counts sum')

divider1 = make_axes_locatable(ax[0])
cax1 = divider1.append_axes("right", size="7%", pad=0.07)
fig.colorbar(m_curr, ax=ax[0], cax=cax1, label='Induced current integral [# electrons]')

plt.subplots_adjust(hspace=0.5)
fig.savefig("currentadc.pdf")
```

### Export result
As a last step we backtrack the ADC counts to the Geant4 tracks and we export the result in a HDF5 file.

```python
!rm test.h5
track_pixel_map=cp.repeat(track_pixel_map[:, cp.newaxis], 10, axis=1)
pc = fee.export_to_hdf5(adc_list.get(),
                        adc_ticks_list.get(),
                        unique_pix.get(),
                        backtracked_ids.get(),
                        track_pixel_map.get(),
                        "test.h5",
                        bad_channels='../larndsim/detector_properties/module0_bad_channels.json')
```

    Writing to HDF5...: 100%|██████████| 301/301 [00:00<00:00, 34739.98it/s]

```python
with h5py.File('test.h5', 'a') as f:
    f.create_dataset("tracks", data=selected_tracks)
```

## Analyze the result
Here we read back the file we just produced and compare it with the _true_ tracks. So first we create a dictionary that associates the chip key to a $(x,y)$ position:

```python
from collections import defaultdict
import yaml

geometry_yaml = yaml.load(open("/global/u1/s/soleti/larnd-sim/larndsim/pixel_layouts/multi_tile_layout-2.2.16.yaml"), Loader=yaml.FullLoader)

pixel_pitch = geometry_yaml['pixel_pitch']
is_multi_tile = True
chip_channel_to_position = geometry_yaml['chip_channel_to_position']
tile_orientations = geometry_yaml['tile_orientations']
tile_positions = geometry_yaml['tile_positions']
tpc_centers = geometry_yaml['tpc_centers']
tile_indeces = geometry_yaml['tile_indeces']
xs = np.array(list(chip_channel_to_position.values()))[:, 0] * pixel_pitch
ys = np.array(list(chip_channel_to_position.values()))[:, 1] * pixel_pitch
x_size = max(xs)-min(xs)+pixel_pitch
y_size = max(ys)-min(ys)+pixel_pitch

def _rotate_pixel(pixel_pos, tile_orientation):
    return pixel_pos[0]*tile_orientation[2], pixel_pos[1]*tile_orientation[1]

tile_geometry = defaultdict(int)
geometry = defaultdict(lambda: (0,0))
io_group_io_channel_to_tile = {}

for tile in geometry_yaml['tile_chip_to_io']:
    tile_orientation = tile_orientations[tile]
    tile_geometry[tile] = tile_positions[tile], tile_orientations[tile]

    for chip in geometry_yaml['tile_chip_to_io'][tile]:
        io_group_io_channel = geometry_yaml['tile_chip_to_io'][tile][chip]
        io_group = io_group_io_channel//1000
        io_channel = io_group_io_channel % 1000
        io_group_io_channel_to_tile[(
            io_group, io_channel)] = tile
    for chip_channel in geometry_yaml['chip_channel_to_position']:
        chip = chip_channel // 1000
        channel = chip_channel % 1000
        try:
            io_group_io_channel = geometry_yaml['tile_chip_to_io'][tile][chip]
        except KeyError:
            #print("Chip %i on tile %i not present in network" % (chip,tile))
            continue

        io_group = io_group_io_channel // 1000
        io_channel = io_group_io_channel % 1000
        x = chip_channel_to_position[chip_channel][0] * \
            pixel_pitch - x_size / 2 + pixel_pitch / 2
        y = chip_channel_to_position[chip_channel][1] * \
            pixel_pitch - y_size / 2 + pixel_pitch / 2

        x, y = _rotate_pixel((x, y), tile_orientation)
        x += tile_positions[tile][2] + \
            tpc_centers[tile_indeces[tile][1]][0]
        y += tile_positions[tile][1] + \
            tpc_centers[tile_indeces[tile][1]][1]

        geometry[(io_group, io_channel,
                       chip, channel)] = x, y
```

Then, we can compare the digitized charge associated to packets with the number of true electrons that reach the anode

```python
f = h5py.File("test.h5", "r")
datawords = f['packets']['dataword'][f['packets']['packet_type']==0]
print("Digitized charge/true charge:", sum(datawords/fee.ADC_COUNTS*(fee.V_REF-fee.V_CM)+fee.V_CM-fee.V_PEDESTAL)/fee.GAIN/sum(f['tracks']['n_electrons']))
```

    Digitized charge/true charge: 0.8882622159358068

Here we verify the association between the _true_ track and the hit. The color of the pixel corresponds to the color of the first associated track (a hit can be associated to more tracks, not shown here).

```python
fig,ax = plt.subplots(1,1,constrained_layout=True)
plotted_tracks = []
colors = ['tab:blue','tab:orange','tab:green','tab:red','tab:purple','tab:brown']

for ip,packet in enumerate(f['packets']):
    if packet['packet_type'] == 0:
        io_group, io_channel, chip, channel = packet['io_group'], packet['io_channel'], packet['chip_id'], packet['channel_id']
        x,y = geometry[(io_group, io_channel, chip, channel)]
        track_ids = f['mc_packets_assn']['track_ids'][ip]
        fractions = f['mc_packets_assn']['fraction'][ip]

        rect = plt.Rectangle((x-consts.pixel_pitch*5, y-consts.pixel_pitch*5),
                     consts.pixel_pitch*10, consts.pixel_pitch*10,
                     linewidth=0.5, fc=colors[track_ids[np.argmax(fractions)]%len(colors)],
                     edgecolor='k',alpha=0.25)

        ax.add_patch(rect)

        for trackid in track_ids:
            if trackid >= 0 and trackid not in plotted_tracks:
                plotted_tracks.append(trackid)
                ax.plot((f['tracks'][trackid]['x_start']*10,f['tracks'][trackid]['x_end']*10),
                        (f['tracks'][trackid]['y_start']*10+218.236,f['tracks'][trackid]['y_end']*10+218.236),c=colors[trackid%len(colors)])

ax.set_aspect("equal")
ax.set_xlabel("x [mm]")
ax.set_ylabel("y [mm]")

fig.savefig("mc.pdf")
```


# Alpha and Smart Beta

```python
from IPython.display import HTML

HTML('''
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')
```

<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>

```python
import pandas as pd
import numpy as np
import scipy.optimize as sco
import matplotlib.pyplot as plt
import datetime as dt

from matplotlib import style
import matplotlib as mpl
import matplotlib.font_manager as fm

style.use('ggplot')
font = fm.FontProperties(fname='Font/helvetica.ttf')
mpl.rcParams['font.family'] = font.get_name()

import warnings
warnings.filterwarnings('ignore')

```

## CAPM

This blog post is an extension of the previous one [INSERT LINK HERE] .

We looked at the Efficient Frontier, and caclulated the Tengency Portfolio and constructed the Security Market Line (SML).

It is easy to see that the SML can be described by a linear equation:

$$
\begin{align}
\\R_p &= R_f + \beta (R_m - R_f) + \epsilon,\quad where
\\
\\R_p &- expected\space return\space of\space the\space portfolio\space or\space stock
\\R_f &- risk\space free\space rate\space
\\R_m &- expected\space return\space of\space the\space market\space portfolio\space
\end{align}
$$

In the case of the SML the market portfolio was represented by the Tangency Portfolio.

In fact, this equation represents the key equation of the Capital Asset Pricing Model (CAPM). This model was built upon Harry Markowitz (1952) work, and suggested by a number of scholars working independently, but most often attributed to William F. Sharpe (1964).

In practice, when analyzing stock returns, analysts usually use a broad market index, i.e. S&P500, as a proxy of the market return ($R_m$).

This framework allows us to look at the stock/portfolio return from the point of view of the risk premium. We can say that the expected return is determined by:
- the risk-free rate ($R_f$),
- the Equity Premium ($R_m - R_f$),
- the measure of the correlation/exposure of the portfolio to the market ($\beta$),
- and, finally, by an idiosyncratic risk of a certain stock/portfolio ($\epsilon$).

The indtoduction of the CAPM made the life of asset managers a bit harder. From the analytical framework it is clear, that 'beating the market' can be achived in three ways:

1) Increasing the exposure to market ($\beta > 1 $ ), if the market grows. But by doing so, a manager increases the risk of the downfall if the market suddenly declines.

2) Timing the market, i.e. increasing or decreasing $\beta$, based on market expectations.

3) Finding idiosyncratic risks ($\epsilon$) that pay off.

We can transform the CAPM equation into an equation that describes the sources of the portfolio income as:

$$
\begin{align}
\\R_p - R_f &= \alpha + \beta (R_m - R_f)
\end{align}
$$

Here, $\alpha$ is a measure of a manager's skill to either time the market or to select companies, based on their idiosyncratic risks. Clearly, the managers that just use high market exposure (high $\beta$), should not be rewarded extra for basically providing a market exposure. Therefore, for some time the term 'beta' became almost derogatory term in asset management circles.

## Multi-factor models (Smart-beta)

The attitude changed somewhat after Fama and French (1992) observed that the idiosyncratic risk ($\epsilon$) can be further dissected into other types of risks inherent for stocks. This way (INSERT EQUATION NUMBER) is transormed into:

$$
\begin{align}
\\R_p - R_f &= \alpha + \beta (R_m - R_f) + \sum{\beta R_{factors}},\quad where\\
\\ R_{factors} &- expected\space returns\space of\space risk\space factors
\end{align}
$$

Fama and French created risk factors as long/short portfolios of stocks exhibiting certain charecteristics. The simple way to think about it is that to create the portfolios we can divide the universe of stocks in two halves based on charecterictics (e.g. small stocks vs. large stocks) and create a certain factor by substracting returns of one portfolio from another. In practice, Fama and French had a more complicated way of the portfolio composition (see http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/Data_Library/f-f_bench_factor.html for more information).

In thier original paper (1992) they suggested two additional factors:

- High [book-to-market] minus Low (HML), and
- Small [market capitalization] minus Big (SMB).

Later, Carhart (1997) proposed another factor (UMD - Up minus Down) to capture the tendency of well performing stock to continue good performance by inertia, as well as badly performing stocks to keep performing badly.

Recently, Fama and French (2015) added two more factors:

- Robust (operational income) minus Weak (RMW), and
- Conservative (capital expenditure) minus Aggressive (CMA).

These are just examples of the factor approach to investing. There are other frameworks that use the similar idea, e.g. Fung and Hsieh (2001) seven-factor model.

The take-away here, is that the stock return has many determinants besides the broad market dynamics. However, a prudent investor can evaluate and manage his/her portfolio based on exposures to various risk factors.

This approach rehabilitated the notion of 'beta'. Now some managers refer to the exposure to other than the market risk factors as a 'smart beta', implying that the skill of the asset manager can manifest itself in choosing risk exposure to various factors that pay off.

## Value investing via risk factors

Another important outcome is that by constructing factor portfolios, we can express our investment philosophy through a diversified portfolio of stocks.

For example, HML factor can be viewed as a classic Value vs. Growth investment approach. Value investing was suggested by Graham and Dodd (1934), and came to fame due to the success of Warran Buffet. In its simple form, Value investing implies acquiring stocks of companies with low Price-to-Book ratio. In academia, the numerator and the denominator are usually flipped, so academica talk about high Book-to-Price as an example of Value investing.

Let's see how the Fama and French Value portfolio performed historically.

```python
# Load factors
df = pd.read_csv('Data/ff_5factors.CSV', index_col=0 ,parse_dates=True)

# Plot HML compunded
hml = (1+ df['HML']/100).cumprod() - 1
_ = plt.plot(hml, color='maroon')
_ = plt.xlim('1963','2018')
_ = plt.suptitle('Historic compound performance HML factor', fontsize=18)
_ = plt.title('(1963 - 2018)', fontsize= 12)

# Save the plot
plt.savefig('Graphs\HML_performance.png', dpi=300)
```

Looks like the Value factors performed fairly well histrorically. Now, we will look at it as it pertains to market cyclicity. First we will analyze compounded performance in growing and declining Value markets.

To do it we adopt the approach to finding market peaks and troughs similar to Pagan et al (2003), but with slightly different time frames:

- We find local mimima and maxima in 8 months windows both ways (prior and past the date of the observation).
- We enforce the change of cycles (minima after maxima and vice versa)
- We exclude cucles of less than 16 months and phases of less than 8 months unless the threshold of price increase or drop is met. We set the threshold at 20%.

```python
def find_ip(data, window, min_cycle, min_phase, threshold):
    '''
    Finds inflection points in a dataset.
    Inflection point determination is inspired by Pagan et al, 2003

    INPUTS:
        data - DataFrame with timeseries of stock/index closing price

    OUTPUTS:
        cycles - data frame with inflection points containing
                'ip'= 1/ -1 if the point is Max or Min in the cycle
                'price' - the price of the asset

        timeseries - data frame of of all prices. Contains 'price' and 'ip'

    '''

    tmp_df = pd.DataFrame({'price':data, 'ip':np.zeros(len(data))},
                           index = data.index)

    # Find local maxima and minima

    for i in range(window, len(data)-window):
        local_max = data.iloc[i-window : i+window].max()
        local_min = data.iloc[i-window : i+window].min()

        if tmp_df['price'].iloc[i] == local_max:
            tmp_df['ip'].iloc[i] = 1
        elif tmp_df['price'].iloc[i] == local_min:
            tmp_df['ip'].iloc[i] = -1
        else:
            tmp_df['ip'].iloc[i] = 0

    cycles = tmp_df[tmp_df['ip']!=0]

    # remove short cycles and phases:

    cycles['date'] = pd.to_datetime(cycles.index)
    cycles['time_diff'] = cycles['date'] - cycles['date'].shift(1)
    cycles['time_diff'] = cycles['time_diff'].astype(dt.timedelta).map(lambda x: np.nan if pd.isnull(x) else x.days)

    for i in range(len(cycles)-2):
        #Calculate cycle and phases length
        cycle_length = cycles['time_diff'].iloc[i+1] + cycles['time_diff'].iloc[i+2]
        phase_length = cycles['time_diff'].iloc[i+1]

        #Calculate price change
        price_chng = np.abs((cycles['price'].iloc[i+1] - cycles['price'].iloc[i]) / cycles['price'].iloc[i])

        #Remove short cycles and phases
        if price_chng > threshold:  #Keep cycles and phases with large swings despite length
            next
        else:
            if cycle_length < min_cycle or phase_length < min_phase:
                cycles['ip'].iloc[i-1] = 0
                cycles['ip'].iloc[i] = 0

    #Enforce alteration

    for k in range(2):
        for i in range(len(cycles)-1):
            if cycles['ip'].iloc[i] == cycles['ip'].iloc[i+1] and cycles['ip'].iloc[i]==1:
                if cycles['price'].iloc[i] > cycles['price'].iloc[i+1]:
                    cycles['ip'].iloc[i] = 1
                    cycles['ip'].iloc[i+1] = 0
                else:
                    cycles['ip'].iloc[i] = 0
                    cycles['ip'].iloc[i+1] = 1

            if cycles['ip'].iloc[i] == cycles['ip'].iloc[i+1] and cycles['ip'].iloc[i]==-1:
                if cycles['price'].iloc[i] < cycles['price'].iloc[i+1]:
                    cycles['ip'].iloc[i] = -1
                    cycles['ip'].iloc[i+1] = 0
                else:
                    cycles['ip'].iloc[i] = 0
                    cycles['ip'].iloc[i+1] = -1

        tmp_df.update(cycles)
        cycles = tmp_df[tmp_df['ip']!=0]

    tmp_df.update(cycles[['ip']])

    cycles = tmp_df[tmp_df['ip']!=0]

    # Return dataframes
    timeseries = tmp_df

    return cycles, timeseries

```

We mark peaks with green arrows and troughs with red.

```python
# Calculate peaks and troughs
cycles, timeseries = find_ip(hml, window = 21*8, min_cycle = 21*16, min_phase = 21*8, threshold = 0.2)

```

```python
# Print a graph with peaks and troughs
peak = timeseries.loc[timeseries['ip']==1, 'ip'] * timeseries.loc[timeseries['ip']==1, 'price']
trough = (-1) * timeseries.loc[timeseries['ip']==-1, 'ip'] * timeseries.loc[timeseries['ip']==-1, 'price']

_ = plt.plot(timeseries['price'], color='maroon')
_ = plt.plot(peak, 'v',color = 'green')
_ = plt.plot(trough, '^',color = 'red')
_ = plt.xlim('1963','2018')
_ = plt.suptitle('Peaks and Troughs of HML factor', fontsize=18)
_ = plt.title('(1963 - 2018)', fontsize= 12)

# Save the plot
plt.savefig('Graphs\HML_peaks_troughs.png', dpi=300)
```

To better visualize it let's look at compund return in each phase of the Value cycle.

```python
df_tmp = timeseries.copy()

df_tmp['ret'] = df['HML']/100

df_tmp.loc[df_tmp['ip']==0,'ip'] = np.nan
df_tmp.loc[np.isfinite(df_tmp['ip']),'ip'] = 0
df_tmp.loc[np.isnan(df_tmp['ip']),'ip'] = 1

buy_peak = pd.DataFrame(index=df_tmp.index) #Place holder for the dataframe

comp_ret=np.array([0.])
for i in range(1, len(df_tmp)):
    comp_ret_new = ((1 + comp_ret[i-1]) * (1 + df_tmp['ret'].iloc[i]) -1) * df_tmp['ip'].iloc[i]
    comp_ret = np.append(comp_ret, comp_ret_new)

buy_peak['B_o_P'] = comp_ret

x = buy_peak.index
y = buy_peak['B_o_P']

_ = plt.fill_between(x, y, where= y >= 0, facecolor='gray')
_ = plt.fill_between(x, y, where= y < 0, facecolor='maroon')
_ = plt.xlim(('1963', '2018'))
_ = plt.suptitle('Compound return of HML in various phases', fontsize=18 )
_ = plt.title('(1963 - 2018)')

# Add annotation
# Divide into periods
df_tmp = cycles.copy()
df_tmp = df_tmp.join(buy_peak['B_o_P'].shift(1), how='left')
df_tmp.loc[df.index[0],'B_o_P'] = 0
df_tmp.loc[df.index[-1],'B_o_P'] = buy_peak['B_o_P'].iloc[-1]
df_tmp.sort_index(inplace=True)
df_tmp.reset_index(inplace=True)

# Calculate perdiod length
df_tmp['period'] = df_tmp['index'] - df_tmp['index'].shift(1)
df_tmp.set_index('index', inplace=True)
df_tmp = df_tmp.iloc[1:,]
df_tmp['period'] = (df_tmp['period']/ np.timedelta64(1, 'D')).astype(int)

# Calculate average period and return
period = df_tmp.groupby('ip')['period'].mean()
ret = df_tmp.groupby('ip')['B_o_P'].mean()

hml_stats = pd.DataFrame([period, ret]).T
hml_stats.index = ['Down', 'Up']
hml_stats.columns = ['Avg. period (days)', 'Avg. compound return']
hml_stats.iloc[:,0] = hml_stats.iloc[:,0].astype('int')
hml_stats.iloc[:,1] = hml_stats.iloc[:,1].round(4)

up = 'The \'up\' market is on average ' + str(hml_stats.iloc[1,0]) + ' days long\nand yields ' + str(round(hml_stats.iloc[1,1]*100, 2)) +'% compound return'
down = 'The \'down\' market is on average ' + str(hml_stats.iloc[0,0]) +' days long \nand yields ' + str(round(hml_stats.iloc[0,1]*100,2)) +'% compound return'

_ = plt.text('1988', 1.20, up, size=9)
_ = plt.text('1964', -0.35, down, size=9, color='maroon')

```

We can see that historically the Value cycles were longer and more profitable than the Growth cycles. However, it the recent decade things have changed. In my next blog post I will present my view on the possible explanation of thia phenomenon and on its potential impact on the stock market in the near future.

## Conclusion

We can create factor portfolios that provide risk/return exposure to certain desirable charecteristics of stocks, e.g. Value investing, robust profitability, etc. These factor portfolios, aka 'smart beta', can be the basis for the investment strategy implemented by an asset manager.

Our quick analysis of the Value (HML) factor, shows that historically Value cycles have been longer and yielded higher compound returns than Growth cycles.

## References

*1. Markowitz, Harry. "Portfolio selection." The journal of finance 7.1 (1952): 77-91.*

*2. Sharpe, William F. "Capital asset prices: A theory of market equilibrium under conditions of risk." The journal of finance 19.3 (1964): 425-442.*

*3. Fama, Eugene F., and Kenneth R. French. "The cross‐section of expected stock returns." the Journal of Finance 47.2 (1992): 427-465.*

*4. Fama, Eugene F., and Kenneth R. French. "Incremental variables and the investment opportunity set." Journal of Financial Economics 117.3 (2015): 470-488.*

*5. Carhart, Mark M. "On persistence in mutual fund performance." The Journal of finance 52.1 (1997): 57-82.*

*6. Fung, William, and David A. Hsieh. "The risk in hedge fund strategies: Theory and evidence from trend followers." The Review of Financial Studies 14.2 (2001): 313-341.*

*7. Pagan, Adrian R., and Kirill A. Sossounov. "A simple framework for analysing bull and bear markets." Journal of Applied Econometrics 18.1 (2003): 23-46.*

### Reminder

**If you choose to use this publication for your own work, please include a reference to this publication and a link leading to this document. **

### Disclaimer

**Not an Offer or Inducement**

*This publication has been prepared for informational purposes only and does not constitute an offer to sell, a solicitation of an offer to buy, or a recommendation of any security or any other product or service by QFL Capital LLC., QFL Holdings Inc., authors of the publication, or any other third party regardless of whether such security, product or service is referenced in this publication. None of the information	or analyses presented are intended to form the basis for any investment decision, and no specific recommendations are intended.*

*Accordingly this publication does not constitute investment advice or counsel or solicitation for investment in any security. This publication does not constitute or form part of, and should not be construed as, any offer for sale or subscription of, or any invitation to offer to buy or subscribe for, any securities, nor should it or any part of it form the basis of, or be relied on in any connection with, any contract or commitment whatsoever.*

*QFL Capital LLC., QFL Holdings Inc., and authors of the publication expressly disclaim any	and all responsibility for any direct or consequential loss or damage of any kind whatsoever arising directly or indirectly from: (i) the use of this site, (ii) reliance on any information contained herein, (iii) any error, omission or inaccuracy in any such information or (iv) any action resulting therefrom. *

**Investment Performance**

*Investors should seek financial advice regarding the appropriateness of investing in any securities or investment strategies discussed or recommended in this documentation and should understand that statements regarding future prospects may not be realized. Investors should note that income from such securities may fluctuate and that each security's price or value may rise or fall. Accordingly, investors may receive back less than originally invested.*

*Past performance is not necessary a guide to future performance. In consideration of being permitted access to the publication, the viewer hereby acknowledges having read and understood this agreement and agrees to be bound by its terms and conditions.*

**No Warranties**

*QFL Capital LLC., QFL Holdings Inc., and authors of the publication take care to ensure that the information provided is accurate and up to date. However this documentation is provided without warranties of any kind, either expressed or implied, including but not limited to warranties of title or implied warranties of merchantability or fitness for a particular purpose. Opinions expressed herein are subject to change without notice. All information and content on this publication is furnished "as is," without warranty of any kind, express or implied, including but not limited to implied warranties of merchantability, fitness for a particular purpose, or non-infringement. *