# Using Python to Reverse-Engineer Guitar Effects
Author: Dan Lynch\
Date: November 21, 2020

---

This notebook will demonstrate some concepts in signal processing and some signal processing/analysis tools available in Python, primarily SciPy.

We'll use a "simple" example: distortion, in the style of the infamous Boss MT-2 Metal Zone pedal.
Although nonlinear, distortion is a little easier to demonstrate than delay-based effects such as echo, reverb, chorus, flanger, and phaser.

There are a number of Python modules for working with audio files; see [here](https://www.audiolabs-erlangen.de/resources/MIR/FMP/B/B_PythonAudio.html) for a helpful list. You'll see SciPy in that list, but the authors note that it doesn't always work well, and I have found that too. Instead, I'm using a module named [SoundFile](https://pysoundfile.readthedocs.io/en/0.9.0/#) that has a clean API. First we need to install it:

```python
!pip install soundfile
```

## Load Audio Files
This tutorial starts with two WAV files:
* `riff_clean.wav`: an unprocessed signal, recorded from my guitar direct to my computer
* `riff_dist.wav`: a heavily-distorted signal, recorded from my guitar passed through a Digitech RP-70 modeling pedal. This pedal is an all-in-one device that emulates a rack-full of guitar effects, including various amplifiers, compressors, EQs, chorus/flanger/phaser, and echo/reverb.

Here's the signal chain:

```python
from IPython.display import Audio

# get the clean riff WAV:
riff_clean_url = 'https://raw.githubusercontent.com/dlynch7/reverse_engineer_guitar_fx/main/distortion/riff_clean.wav'

Audio(url=riff_clean_url)
```

```python
# get the distorted riff WAV:
riff_dist_url = 'https://raw.githubusercontent.com/dlynch7/reverse_engineer_guitar_fx/main/distortion/riff_dist.wav'

Audio(url=riff_dist_url) # it's a lot louder than the clean recording, so don't destroy your ears!
```

We're going to be doing a lot with these audio clips, so let's download them, temporarily, to our local workspace.

```python
import requests

r = requests.get(riff_clean_url, allow_redirects=True)
open('riff_clean.wav', 'wb').write(r.content) # 'wb' because we are writing a *binary* file

r = requests.get(riff_dist_url, allow_redirects=True)
open('riff_dist.wav', 'wb').write(r.content) # 'wb' because we are writing a *binary* file
```

Open your Colab notebook's filesystem by clicking on the folder icon to the left. You should see these files now. To see their sizes, mouse over them: both should be around 2 MB.

These files are "large" because they are WAV files, and WAV is one of the _**uncompressed**_ audio formats. In contrast, MP3s are smaller because they are compressed audio formats. Compression removes some audio information, and we want all the information available as we proceed to analyze and modify these signals, so that's why we're using WAV instead of MP3.

One more thing about the format: I recorded to 16-bit WAV files; 16-bit is typical CD-quality audio. You can record to 24-bit and 32-bit too (audio professionals use these resolutions), but those files are bigger, and it's hard to hear the difference. 16-bit is good enough for our purposes. The important thing is that the audio data is uncompressed.

Google Colab wasn't designed with audio processing in mind, and I'm unable to autosave my notebook once I download those two audio files. To get around this, execute the following cells in order.

```python
from scipy.io import wavfile
import scipy.io
import soundfile as sf

riff_clean_data, riff_clean_samplerate = sf.read('riff_clean.wav')
riff_clean_dict = {'wav_data':riff_clean_data,
                   'samplerate':riff_clean_samplerate,
                   'wav_fname':'riff_clean.wav'}
print(riff_clean_dict)

riff_dist_data, riff_dist_samplerate = sf.read('riff_dist.wav')
riff_dist_dict = {'wav_data':riff_dist_data,
                  'samplerate':riff_dist_samplerate,
                  'wav_fname':'riff_dist.wav'}
print(riff_dist_dict)
```

```python
# !rm riff_clean.wav
```

```python
# !rm riff_dist.wav
```

## Visualize each waveform

```python
# code based on
# 1) https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html
# 2) https://matplotlib.org/3.3.3/gallery/lines_bars_and_markers/cohere.html#sphx-glr-gallery-lines-bars-and-markers-cohere-py

import matplotlib.pyplot as plt
import matplotlib
import numpy as np

matplotlib.rcParams['font.size'] = 16.0

def plot_wave(wav_dict):
  wav_data = wav_dict['wav_data']
  samplerate = wav_dict['samplerate']

  # print(f"number of channels = {wav_data.shape[1]}")
  length = wav_data.shape[0] / samplerate
  # print(f"length = {length}s")
  time = np.linspace(0., length, wav_data.shape[0])

  fig, axs = plt.subplots(wav_data.shape[1],1,figsize=(20, 10))
  for channel in range(wav_data.shape[1]):
    axs[channel].plot(time, wav_data[:, channel])
    axs[channel].set_ylabel("Channel %d Amplitude" % channel)
    axs[channel].grid(True)
  axs[-1].set_xlabel("Time [s]")
  axs[0].set_title(wav_dict['wav_fname'])
  plt.show()
  return

plot_wave(riff_clean_dict)
plot_wave(riff_dist_dict)
```

Interesting! We can make an interesting observation already:

Notice that the distorted riff has near-constant amplitude, while the clean riff only has loud transients corresponding to each note I picked on the guitar.
So why the difference in perceived loudness? It's in our heads (a psychoacoustic effect): our ears/brains don't perceive instantaneous loudness but rather average loudness over some time window.
This explains why the distorted riff sounds much louder than the clean riff, even though they have almost identical peak amplitudes.

As an aside, this psychoacoustic effect led to widespread use (some might say abuse) of an audio effect called "limiting" which makes quiet parts louder. This effect was a key player in the "[Loudness War](https://en.wikipedia.org/wiki/Loudness_war)" of the 2000s.

## How does distortion work?
Let's zoom in near the end of the riff and compare the clean and distorted signals.

```python
def overlay_wavs(wav_dict_list,t_start,t_end):
  if not type(wav_dict_list) is list:
    raise TypeError('type(wav_dict_list) must be list.')
  if wav_dict_list == []:
    raise ValueError('wav_dict_list must have at least one element.')
  if not type(wav_dict_list[0]) is dict:
    raise TypeError('elements of wav_dict_list must be of type dict.') # could be more robust than it is
  if not (type(t_start) is float or type(t_start) is int):
    raise TypeError('type(t_start) must be float or int.')
  if not (type(t_end) is float or type(t_end) is int):
    raise TypeError('type(t_end) must be float or int.')

  if not t_start > 0:
    raise ValueError('t_start must be > 0')
  if not t_end > t_start:
    raise ValueError('t_start must be < t_end')

  # create a local dictionary to contain additional info:
  wavs_data = []
  for i in range(len(wav_dict_list)):
    length = wav_dict_list[i]['wav_data'].shape[0] / wav_dict_list[i]['samplerate']
    if not t_end < length:
      raise ValueError('t_end must be < %f' % length)
    time = np.linspace(0., length, wav_dict_list[i]['wav_data'].shape[0])
    wavs_data.append({'fname':      wav_dict_list[i]['wav_fname'],
                      'samplerate': wav_dict_list[i]['samplerate'],
                      'data':       wav_dict_list[i]['wav_data'],
                      'length':     length,
                      'time':       time})

  fig, axs = plt.subplots(len(wav_dict_list),1,figsize=(20, 10))
  for i in range(len(wavs_data)):
    n_start = int(t_start*wavs_data[i]['samplerate'])
    n_end = int(t_end*wavs_data[i]['samplerate'])
    t_range = np.linspace(t_start, t_end, n_end - n_start)
    axs[i].plot(t_range, wavs_data[i]['data'][n_start:n_end, 0]) # looking at channel 0, i.e., the left channel
    axs[i].set_ylabel("Channel 0 Amplitude, %s" % wavs_data[i]['fname'])
    axs[i].grid(True)
  axs[-1].set_xlabel("Time [s]")
  axs[0].set_title('Comparison of signals for %f <= time <= %f' % (t_start,t_end))
  plt.show()
  return

overlay_wavs([riff_clean_dict,riff_dist_dict],10.85,12.3)
overlay_wavs([riff_clean_dict,riff_dist_dict],10.85,11.0)
overlay_wavs([riff_clean_dict,riff_dist_dict],10.85,10.9)
```

Bear in mind that the upper signal is clean and the lower signal is distorted.
Let's take stock of what we can see after three rounds of zooming in on the signal:
* the distorted signal is louder;
* the distorted signal has more high-frequency content than the clean signal. You can see this pretty clearly around 10.86 seconds;
* the amplitude of the clean signal appears to exponentially decay after each transient, and if you imagine a vibrating string (and, indeed, the partial differential equation that models a vibrating string - the "wave equation"), this makes physical sense;
* the decay envelope of the distorted signal is "concave down", whereas the decay envelope of the clean signal is "concave up". This is noticeable at 10.85s, 10.865s, 10.88s, etc. This doesn't make physical sense!

### Overdrive and clipping
What's going on with the distorted signal might make more sense when you think about how guitar distortion works: _**clipping**_.

Most distortion effects are based on two stages:
* a gain stage (often achieved with a vacuum tube, a transistor, or an op-amp) that amplifies an incoming signal _a lot_.
* a clipping stage (sometimes built into the gain stage, other times achieved separately with a pair of diodes) that doesn't let the amplitude exceed some threshold.

Together, they act on an incoming signal thusly:

The graph above comes from a [teardown and analysis of the ProCo RAT pedal](https://www.electrosmash.com/proco-rat) that does a great job explaining how this effect can be achieved using discrete electronic components.

### What does clipping *sound* like?
We just saw what clipping *looks* like, when applied to a pure sine wave, but what does it sound like? To answer that, let's mock up a clipping stage in Python, and let's implement it as a function that acts on NumPy arrays, since we can convert WAV files to NumPy arrays.

### Software clipping, attempt 1: hard (square) clipping
This function will need to take at least two inputs:
* an incoming signal, `sig_in`, which will be a NumPy array
* a threshold amplitude, `clip_thresh`, above which clipping will happen. Let's use a float between 0 and 1.

We'll rescale the clipping threshold to 15-bit range (0 to 2^15 - 1) inside the function. We're working with 16-bit audio, and audio is an AC signal, with positive and negative amplitude, so the maximum amplitude is 2^15 - 1 and the minimum amplitude is -2^15.

```python
def hard_clipper(sig_in,clip_thresh):
  if not type(sig_in) is np.ndarray:
    raise TypeError('type(sig_in) must be numpy.ndarray')
  if not (type(clip_thresh) is float or type(clip_thresh) is int):
    raise TypeError('type(clip_thresh) must be float or int')
  if (clip_thresh > 1 or clip_thresh < 0):
    raise ValueError('clip_thresh must be between 0 and 1.')

  sig_out = sig_in.copy()
  pos_thresh = clip_thresh
  neg_thresh = -clip_thresh

  # clip the positive part of sig_in:
  pos_clip_indices = sig_in > pos_thresh
  sig_out[pos_clip_indices] = pos_thresh

  # clip the negative part of sig_in:
  neg_clip_indices = sig_in < neg_thresh
  sig_out[neg_clip_indices] = neg_thresh

  return sig_out

samplerate = 44100; fs = 5 # too low to hear, but easy to see
t = np.linspace(0., 1., samplerate)
amplitude = 1
sine_data = amplitude * np.sin(2. * np.pi * fs * t)

hard_clip_75 = hard_clipper(sine_data,0.75)
hard_clip_50 = hard_clipper(sine_data,0.50)
hard_clip_25 = hard_clipper(sine_data,0.25)

fig, ax = plt.subplots(1,1,figsize=(20, 10))
ax.plot(t, sine_data,label='thresh = 1.0')
ax.plot(t, hard_clip_75,label='thresh = 0.75')
ax.plot(t, hard_clip_50,label='thresh = 0.50')
ax.plot(t, hard_clip_25,label='thresh = 0.25')

ax.set_ylabel("Amplitude")
ax.grid(True)
ax.legend()
ax.set_xlabel("Time [s]")
plt.show()
```

### Making up lost amplitude
Clipping is an effective way to introduce distortion, but it also makes everything quieter, and that isn't very fun. To fix that, most distortion effects have another gain stage after the clipping stage.

In Python, it's even easier: if we know our clipping threshold, we also know the amount of make-up gain we need to apply:
```python
makeup_gain = 1/clip_thresh
```
* if `clip_thresh` = 1, `makeup_gain` = 1
* if `clip_thresh` = 0.5, `makeup_gaint` = 2
* if `clip_thresh` = 0.1, `makeup_gain` = 10

Let's include this makeup gain in our hard-clipping function:

```python
def hard_clipper_makeup(sig_in,clip_thresh):
  if not type(sig_in) is np.ndarray:
    raise TypeError('type(sig_in) must be numpy.ndarray')
  if not (type(clip_thresh) is float or type(clip_thresh) is int):
    raise TypeError('type(clip_thresh) must be float or int')
  if (clip_thresh > 1 or clip_thresh < 0):
    raise ValueError('clip_thresh must be between 0 and 1.')

  sig_out = sig_in.copy()

  # clip the positive part of sig_in:
  pos_clip_indices = sig_in > clip_thresh
  sig_out[pos_clip_indices] = clip_thresh

  # clip the negative part of sig_in:
  neg_clip_indices = sig_in < -clip_thresh
  sig_out[neg_clip_indices] = -clip_thresh

  # make up attenuation due to clipping. This is a little more robust than
  # the method described in the text above, but functionally equivalent:
  sig_out = (sig_out / np.max(np.abs(sig_out))) # a little less than 32768.

  return sig_out

samplerate = 44100; fs = 100
t = np.linspace(0., 1, samplerate)
amplitude = 1
sine_data = amplitude * np.sin(2. * np.pi * fs * t)

hard_clip_mu_75 = hard_clipper_makeup(sine_data,0.75)
hard_clip_mu_50 = hard_clipper_makeup(sine_data,0.50)
hard_clip_mu_25 = hard_clipper_makeup(sine_data,0.25)

fig, ax = plt.subplots(1,1,figsize=(20, 10))
T = 1000
ax.plot(t[0:T], sine_data[0:T],label='thresh = 1.0')
ax.plot(t[0:T], hard_clip_mu_75[0:T],label='thresh = 0.75')
ax.plot(t[0:T], hard_clip_mu_50[0:T],label='thresh = 0.50')
ax.plot(t[0:T], hard_clip_mu_25[0:T],label='thresh = 0.25')

ax.set_ylabel("Amplitude")
ax.grid(True)
ax.legend()
ax.set_xlabel("Time [s]")
plt.show()
```

### Visualizing hard-clipping in the frequency domain
Although clipping alone is not soley responsible for the original distorted guitar sound, it does explain one thing we saw earlier: the high frequency content in the distorted guitar recording.

Let's explore that a little more closely, using a signal analysis tool called the [Fast Fourier Transform](https://en.wikipedia.org/wiki/Fast_Fourier_transform) (FFT). This is a numerical algorithm for finding the frequency content of a signal, and it is based on the idea that an arbitrary periodic signal can be approximated by an infinite sum of sines and cosines.

First we'll use SciPy to take the FFT of our sine wave and the various hard-clipped versions:

```python
# based on https://realpython.com/python-scipy-fft/

from scipy.fft import rfft, rfftfreq

# Number of samples:
N = int(samplerate * t[-1]) # cast to int b/c required by rfftfreq.

fmax = int(1e3)
xf = [[],[],[],[]]
yf = [[],[],[],[]]

# take the FFT of the unclipped sine wave:
yf[0] = rfft(sine_data)
xf[0] = rfftfreq(N, 1/samplerate)

# FFT of lightly-clipped sine wave:
yf[1] = rfft(hard_clip_mu_75)
xf[1] = rfftfreq(N, 1/samplerate)

# FFT of moderately-clipped sine wave:
yf[2] = rfft(hard_clip_mu_50)
xf[2] = rfftfreq(N, 1/samplerate)

# FFT of heavily-clipped sine wave:
yf[3] = rfft(hard_clip_mu_25)
xf[3] = rfftfreq(N, 1/samplerate)

fig, axs = plt.subplots(2,2,figsize=(20,20))
axs[0,0].plot(xf[0][0:fmax], np.abs(yf[0][0:fmax])*(2/N))
axs[0,0].set_xlabel('Frequency [Hz]')
axs[0,0].set_ylabel('Amplitude')
axs[0,0].set_title('FFT, unclipped signal')

axs[0,1].plot(xf[1][0:fmax], np.abs(yf[1][0:fmax])*(2/N))
axs[0,1].set_xlabel('Frequency [Hz]')
axs[0,1].set_ylabel('Amplitude')
axs[0,1].set_title('FFT, lightly-clipped signal')

axs[1,0].plot(xf[2][0:fmax], np.abs(yf[2][0:fmax])*(2/N))
axs[1,0].set_xlabel('Frequency [Hz]')
axs[1,0].set_ylabel('Amplitude')
axs[1,0].set_title('FFT, moderately-clipped signal')

axs[1,1].plot(xf[3][0:fmax], np.abs(yf[3][0:fmax])*(2/N))
axs[1,1].set_xlabel('Frequency [Hz]')
axs[1,1].set_ylabel('Amplitude')
axs[1,1].set_title('FFT, heavily-clipped signal')

plt.show()
```

Let's try our hard-clipper (with gain makeup) on `riff_clean.wav` and see how it sounds. Don't get your hopes up!

```python
riff_hc75_data = hard_clipper_makeup(riff_clean_dict['wav_data'],0.75) # sounds almost like the clean signal
sf.write('riff_hc75.wav',riff_hc75_data,44100)
riff_hc75_dict = {'wav_data':riff_hc75_data,
                  'samplerate':44100,
                  'wav_fname':'riff_hc75.wav'}
plot_wave(riff_hc75_dict)

riff_hc50_data = hard_clipper_makeup(riff_clean_dict['wav_data'],0.50) # crunch is barely perceptible
sf.write('riff_hc50.wav',riff_hc50_data,44100)
riff_hc50_dict = {'wav_data':riff_hc50_data,
                  'samplerate':44100,
                  'wav_fname':'riff_hc50.wav'}
plot_wave(riff_hc50_dict)

riff_hc25_data = hard_clipper_makeup(riff_clean_dict['wav_data'],0.25) # crunch begins to be noticeable, still pretty clean
sf.write('riff_hc25.wav',riff_hc25_data,44100)
riff_hc25_dict = {'wav_data':riff_hc25_data,
                  'samplerate':44100,
                  'wav_fname':'riff_hc25.wav'}
plot_wave(riff_hc25_dict)

riff_hc10_data = hard_clipper_makeup(riff_clean_dict['wav_data'],0.10) # now it's starting to really distort
sf.write('riff_hc10.wav',riff_hc10_data,44100)
riff_hc10_dict = {'wav_data':riff_hc10_data,
                  'samplerate':44100,
                  'wav_fname':'riff_hc10.wav'}
plot_wave(riff_hc10_dict)

riff_hc05_data = hard_clipper_makeup(riff_clean_dict['wav_data'],0.05) # now it's starting to really distort
sf.write('riff_hc05.wav',riff_hc05_data,44100)
riff_hc05_dict = {'wav_data':riff_hc05_data,
                  'samplerate':44100,
                  'wav_fname':'riff_hc05.wav'}
plot_wave(riff_hc05_dict)
```

```python
Audio(filename='riff_hc75.wav')
```

```python
Audio(filename='riff_hc50.wav')
```

```python
Audio(filename='riff_hc25.wav')
```

```python
Audio(filename='riff_hc10.wav')
```

```python
Audio(filename='riff_hc05.wav')
```

## Bandpass filtering, before clipping stage
We're making good progress toward reverse-engineering the original distorted guitar sound. For comparison, here it is again:

```python
Audio(filename='riff_dist.wav')
```

Oh. Hm. We're not so close after all. If it's any comfort, I faced this same exact problem when I was building a hardware distortion pedal based on the ProCo RAT circuit: it distorted just fine, but it sounded "fuzzy" and not, well, metal.

Here's the current problem:
* in the absence of any filtering before the clipping stage, the low-frequency signal content from my guitar dominates the clipping stage and makes the clipper's output sound "fuzzy", like a blown speaker.
* I can't keep adding more distortion (cranking up the gain on the input to the clipper, or equivalently, reducing the clipping threshold and then adding makeup gain).
* in other words, I want more distortion, but not at the low frequencies

How did I achieve that? Filtering! Specifically, bandpass filtering. Here's why that helps:
* First, I filter out low-frequency content before the clipping stage
* Then, I really crank up the distortion at the clipping stage, by reducing the clipping threshold
* As a result, the mid-range frequencies distort while the bottom-end is attenuated

I didn't exactly come up with this myself. The Boss Metal Zone pedal [does this exact same thing](https://electricdruid.net/boss-mt-2-metal-zone-pedal-analysis/), albeit much more cleverly.

### Complementary Filters
So, now we come to a question: how does one implement a filter in Python? This is potentially a bottomless rabbit hole, but before diving in, let's look at a really simple approach to digital filters: the _**complementary filter**_. This is basically a weighted average.

Consider an input signal $x$ and an output signal $y$. These are discrete-time signals, so they can be written as $x[i], y[i],\  i = 1,2,\ldots,n$. It often makes sense to think of $x[n]$ and $y[n]$ as the "current" input and output, $x[n-1]$ and $y[n-1]$ as the "previous" input and output, and so on.

The next two equations use scalar gains $a$ and $b$ which are required to be between 0 and 1.

There are two basic kinds of complementary filters. They are recursive, because they process an input signal as well as their past outputs. These filters belong to a larger family of filters called "infinite-impulse-response" (IIR) filters. As you might guess, there is a separate family of "finite-impulse-response" (FIR) filters. Each filter architecture has pros and cons.

The next two examples demonstrate two complementary filter equations.

#### Complementary lowpass filter
The block diagram of a complementary lowpass filter is

If you haven't seen these kinds of block diagrams before, don't worry. They're pretty straightforward. The one symbol you might find unfamiliar is the triangle with numbers/letters inside it: that represents a "gain" stage, where the output signal's amplitude is some multiple or fraction of the input signal's amplitude, depending on the value of the number/letter inside the triangle.

The equation for a complementary highpass filter is
\begin{equation}
y[n]=(1-a)x[n] + a y[n-1]
\end{equation}

Think of this as a weighted average between the current input and the previous output. Consider two limits:
* As $a\rightarrow 0$, $y[n] \rightarrow x[n]$, in which case the output is simply the input, and no filtering occurs; all input signal content is passed through.
* As $a\rightarrow 1$, $y[n] \rightarrow y[n-1]$, in which case the output doesn't change in response to input; the signal is maximally lowpass filtered because only the DC component remains.

```python
def lowpass_iir(sig_in,filt_coef):
    if not type(sig_in) is np.ndarray:
      raise TypeError('type(sig_in) must be numpy.ndarray')
    if not (type(filt_coef) is float or type(filt_coef) is int):
      raise TypeError('type(filt_coef) must be float or int')
    if (filt_coef > 1 or filt_coef < 0):
      raise ValueError('filt_coef must be between 0 and 1.')

    sig_out = np.zeros_like(sig_in)
#     print(sig_out.shape)

    for n in range(1,sig_in.shape[0]): # notice that we're starting at "1" instead of "0"
        sig_out[n] = (1-filt_coef)*sig_in[n] + filt_coef*sig_out[n-1]

    return sig_out

# To test our lowpass filter, create a sinusoidal signal and then hard-clip it:
samplerate = 44100; fs = 100
t = np.linspace(0., 1, samplerate)
amplitude = 1
sine_data = amplitude * np.sin(2. * np.pi * fs * t)
hard_clip_mu_10 = hard_clipper_makeup(sine_data,0.10) # this will look like a square wave: lots of harmonic content!

# Now let's lowpass-filter our quasi-square wave:
lpf_square_wave_50 = lowpass_iir(hard_clip_mu_10,0.5) # set the filter coefficient to 0.5
lpf_square_wave_75 = lowpass_iir(hard_clip_mu_10,0.75) # set the filter coefficient to 0.75
lpf_square_wave_90 = lowpass_iir(hard_clip_mu_10,0.90) # set the filter coefficient to 0.90
lpf_square_wave_95 = lowpass_iir(hard_clip_mu_10,0.95) # set the filter coefficient to 0.90

fig, ax = plt.subplots(1,1,figsize=(20, 10))
T = 1000
ax.plot(t[0:T], sine_data[0:T],label='pure sine')
ax.plot(t[0:T], hard_clip_mu_10[0:T],label='in: hard-clipped sine, thresh = 0.10')
ax.plot(t[0:T], lpf_square_wave_50[0:T],label="out: LPF'd sine, coeff = 0.5")
ax.plot(t[0:T], lpf_square_wave_75[0:T],label="out: LPF'd sine, coeff = 0.75")
ax.plot(t[0:T], lpf_square_wave_90[0:T],label="out: LPF'd sine, coeff = 0.90")
ax.plot(t[0:T], lpf_square_wave_95[0:T],label="out: LPF'd sine, coeff = 0.95")

ax.set_ylabel("Amplitude")
ax.grid(True)
ax.legend()
ax.set_xlabel("Time [s]")
plt.show()
```

We just _saw_ what a complementary _**lowpass**_ filter does to a signal. Now let's _hear_ what it does to our heavily-clipped guitar recording.

Before lowpass filtering:

```python
Audio(filename='riff_hc05.wav')
```

After lowpass filtering:

```python
sf.write('riff_hc05_lpf.wav',
         lowpass_iir(riff_hc05_data,0.75),
         44100)

Audio(filename='riff_hc05_lpf.wav')
```

Sounds muffled, right? That means our lowpass filter is working!

Now let's move on and implement a complementary _**highpass**_ filter.

#### Complementary highpass filter
The block diagram of a complementary highpass filter is

The equation for a complementary highpass filter is
\begin{equation}
y[n]=(1-b) x[n] + b(x[n]-x[n-1])
\end{equation}

Again, think of this as a weighted average between the current input and the previous output, and again consider two limits:
* As $b\rightarrow 0$, $y[n] \rightarrow x[n]$, in which case the output is simply the input, and no filtering occurs; all input signal content is passed through.
* As $b\rightarrow 1$, $y[n] \rightarrow x[n]-x[n-1]$, in which case the output is maximally highpass filtered, as it tracks the _difference_ between the current input and the previous input, thereby approximating the continuous-time derivative of the signal.

```python
def highpass_iir(sig_in,filt_coef):
    if not type(sig_in) is np.ndarray:
      raise TypeError('type(sig_in) must be numpy.ndarray')
    if not (type(filt_coef) is float or type(filt_coef) is int):
      raise TypeError('type(filt_coef) must be float or int')
    if (filt_coef > 1 or filt_coef < 0):
      raise ValueError('filt_coef must be between 0 and 1.')

    sig_out = np.zeros_like(sig_in)
#     print(sig_out.shape)

    for n in range(1,sig_in.shape[0]): # notice that we're starting at "1" instead of "0"
        sig_out[n] = (1-filt_coef)*sig_in[n] + filt_coef*(sig_in[n] - sig_in[n-1])

    return sig_out

# To test our highpass filter, create a sinusoidal signal and then hard-clip it:
samplerate = 44100; fs = 100
t = np.linspace(0., 1, samplerate)
amplitude = 1
sine_data = amplitude * np.sin(2. * np.pi * fs * t)
hard_clip_mu_10 = hard_clipper_makeup(sine_data,0.10) # this will look like a square wave: lots of harmonic content!

# Now let's highpass-filter our quasi-square wave:
hpf_square_wave_50 = highpass_iir(hard_clip_mu_10,0.5) # set the filter coefficient to 0.5
hpf_square_wave_75 = highpass_iir(hard_clip_mu_10,0.75) # set the filter coefficient to 0.75
hpf_square_wave_90 = highpass_iir(hard_clip_mu_10,0.90) # set the filter coefficient to 0.90
hpf_square_wave_95 = highpass_iir(hard_clip_mu_10,0.95) # set the filter coefficient to 0.90

fig, ax = plt.subplots(1,1,figsize=(20, 10))
T = 1000
ax.plot(t[0:T], sine_data[0:T],label='pure sine')
ax.plot(t[0:T], hard_clip_mu_10[0:T],label='in: hard-clipped sine, thresh = 0.10')
ax.plot(t[0:T], hpf_square_wave_50[0:T],label="out: HPF'd sine, coeff = 0.5")
ax.plot(t[0:T], hpf_square_wave_75[0:T],label="out: HPF'd sine, coeff = 0.75")
ax.plot(t[0:T], hpf_square_wave_90[0:T],label="out: HPF'd sine, coeff = 0.90")
ax.plot(t[0:T], hpf_square_wave_95[0:T],label="out: HPF'd sine, coeff = 0.95")

ax.set_ylabel("Amplitude")
ax.grid(True)
ax.legend()
ax.set_xlabel("Time [s]")
plt.show()
```

Now let's hear what it sounds like, again applied to our hard-clipped guitar signal. First, here's the un-filtered recording:

```python
Audio(filename='riff_hc05.wav')
```

And now, the highpass-filtered version:

```python
sf.write('riff_hc05_hpf.wav',
         highpass_iir(riff_hc05_data,0.90),
         44100)
Audio(filename='riff_hc05_hpf.wav')
```

Sounds tinnier, right? That means our complementary highpass filter is working.

#### Complementary bandpass filter
We can simply cascade the lowpass and highpass filters:

Mathematically,
\begin{align}
w[n]=(1-a)x[n] + a w[n-1],\\
y[n]=(1-b)w[n] + b(w[n]-w[n-1]),
\end{align}

Again, consider the limit cases:
* As $a,b\rightarrow 0$, $y[n] \rightarrow x[n]$ (pass-through);
* Hold $a=0$; as $b\rightarrow 1$, $y[n] \rightarrow x[n]-x[n-1]$ (highpass);
* Hold $b=0$; as $a\rightarrow 1$, $y[n] \rightarrow y[n-1]$ (lowpass);
* As $a,b\rightarrow 1$; the cascaded filters work as a bandpass filter, letting mid-frequency content pass through while attenuating low-frequency and high-frequency content. If you let $a < b$, some frequency range of the input signal will be unattenuated.

Now, let's implement this in a function. If we were writing a real-time audio application, we would have to write this in a way that works with streaming input and output, but instead, we'll write our function to post-process an existing signal:

```python
def bandpass_iir(sig_in,lpf_coef,hpf_coef):
    # this function just calls other functions we wrote,
    # and those functions check their inputs, so we don't need to do that here.

    # first, lowpass-filter the input signal
    sig_med = lowpass_iir(sig_in,lpf_coef)

    # second, highpass-filter the lowpass-filtered signal
    sig_out = highpass_iir(sig_med,hpf_coef)

    return sig_out

# To test our bandpass filter, create a sinusoidal signal and then hard-clip it:
samplerate = 44100; fs = 100
t = np.linspace(0., 1, samplerate)
amplitude = 1
sine_data = amplitude * np.sin(2. * np.pi * fs * t)
hard_clip_mu_10 = hard_clipper_makeup(sine_data,0.10) # this will look like a square wave: lots of harmonic content!

# Now let's highpass-filter our quasi-square wave:
bpf_square_wave_10_10 = bandpass_iir(hard_clip_mu_10,0.1,0.1) # LPF coefficient = HPF coefficient = 0.1
bpf_square_wave_10_20 = bandpass_iir(hard_clip_mu_10,0.1,0.2) # LPF coef = 0.1, HPF coef = 0.2
bpf_square_wave_20_50 = bandpass_iir(hard_clip_mu_10,0.2,0.5) # LPF coef = 0.2, HPF coef = 0.5
bpf_square_wave_50_90 = bandpass_iir(hard_clip_mu_10,0.5,0.9) # LPF coef = 0.5, HPF coef = 0.9

fig, ax = plt.subplots(1,1,figsize=(20, 10))
T = 1000
ax.plot(t[0:T], sine_data[0:T],label='pure sine')
ax.plot(t[0:T], hard_clip_mu_10[0:T],label='in: hard-clipped sine, thresh = 0.10')
ax.plot(t[0:T], bpf_square_wave_10_10[0:T],label="out: BPF'd sine, coeffs = (0.1,0.1)")
ax.plot(t[0:T], bpf_square_wave_10_20[0:T],label="out: BPF'd sine, coeffs = (0.1,0.2)")
ax.plot(t[0:T], bpf_square_wave_20_50[0:T],label="out: BPF'd sine, coeffs = (0.2,0.5)")
ax.plot(t[0:T], bpf_square_wave_50_90[0:T],label="out: BPF'd sine, coeffs = (0.5,0.9)")

ax.set_ylabel("Amplitude")
ax.grid(True)
ax.legend()
ax.set_xlabel("Time [s]")
plt.show()
```

As usual, let's test it out on our guitar recording. Here's the unfiltered recording:

```python
Audio(filename='riff_hc05.wav')
```

And here's a bandpass-filtered version:

```python
sf.write('riff_hc05_bpf.wav',
         bandpass_iir(riff_hc05_data,0.75,0.9),
         44100)

Audio(filename='riff_hc05_bpf.wav')
```

For comparison, here's the lowpass-filtered version:

```python
Audio(filename='riff_hc05_lpf.wav')
```

And also for comparison, here's the highpass-filtered version:

```python
Audio(filename='riff_hc05_hpf.wav')
```

## Cascade # 1: bandpass filter, then clip
Now let's start putting things together.
1. First, we'll bandpass-filter the incoming signal;
2. Then, we'll send the filtered signal through our clipping stage (with makeup gain).

Remember why we're going this:
* by filtering out some of the low-frequency content from the clean guitar signal, we can apply more distortion via clipping before the output starts to sound bad
* there isn't much frequency content above a few kHz coming from the guitar, so we might as well filter out what little is there.

```python
def bpf_and_clip(sig_in,lpf_coef,hpf_coef,clip_thresh):
    # this function just calls the other functions we wrote,
    # and they check their inputs, so we don't need to do that here.

    # First, bandpass-filter the input signal:
    sig_med = bandpass_iir(sig_in,lpf_coef,hpf_coef)

    # Second, hard-clip the bandpass-filtered signal:
    sig_out = hard_clipper_makeup(sig_med,clip_thresh)

    return sig_out

# test it out on audio directly:
lpf_coef = 0.75
hpf_coef = 0.99
clip_thresh = 0.001
samplerate = 44100

sf.write('riff_bpf_hc.wav',
         bpf_and_clip(riff_clean_data,lpf_coef,hpf_coef,clip_thresh),
         samplerate)

Audio(filename='riff_bpf_hc.wav')
```

As usual, here's our target sound, for comparison:

```python
Audio(filename='riff_dist.wav')
```

## Cascade #2: BPF, then clip, then BPF
Now it's getting pretty close, but our reverse-engineered signal chain still sounds harsh on the ears, and I think some additional filtering after the clipping stage will help with this.

This is also what is done in the ProCo RAT, the Boss MT-2 Metal Zone, and many other distortion units.

The overarching goal is to pack as much distortion into our effects as possible without it sounding "crappy", and smoothing out the clipper's output will help with that. Let's give it a shot!

```python
def bpf_clip_bpf(sig_in,lpf_coef1,hpf_coef1,clip_thresh,lpf_coef2,hpf_coef2):
    # this function just calls the other functions we wrote,
    # and they check their inputs, so we don't need to do that here.

    # First, bandpass-filter the input signal:
    sig_1 = bandpass_iir(sig_in,lpf_coef,hpf_coef)

    # Second, hard-clip the bandpass-filtered signal:
    sig_2 = hard_clipper_makeup(sig_1,clip_thresh)

    # Third, lowpass-filter the clipped signal:
    sig_out = bandpass_iir(sig_2,lpf_coef2,hpf_coef2)

    return sig_out
```

Here's the clean guitar signal, which we're going to distort the heck out of:

```python
Audio(filename='riff_clean.wav')
```

Now let's use the function we just wrote to process the clean signal:

```python
lpf_coef1 = 0.5
hpf_coef1 = 0.999
clip_thresh = 0.0001
lpf_coef2 = 0.875
hpf_coef2 = 0.6
samplerate = 44100

sf.write('riff_bpf_hc_bpf.wav',
         bpf_clip_bpf(riff_clean_data,lpf_coef1,hpf_coef1,clip_thresh,lpf_coef2,hpf_coef2),
         samplerate)

Audio(filename='riff_bpf_hc_bpf.wav')
```

As usual, here's our target sound, for comparison:

```python
Audio(filename='riff_dist.wav')
```

### FFTs of the guitar signal at each stage in the effects chain

```python
# based on https://realpython.com/python-scipy-fft/

from scipy.fft import rfft, rfftfreq

lpf_coef1 = 0.5
hpf_coef1 = 0.999
clip_thresh = 0.0001
lpf_coef2 = 0.9 # 0.85
hpf_coef2 = 0.7

# Number of samples:
samplerate = 44100
N = samplerate # N is the number of samples, and (below) we're going to restrict ourselves to 1 second of data

fmax = int(2e4)
xf = [[],[],[],[]]
yf = [[],[],[],[]]
yf_db = [[],[],[],[]]

# take the FFT of the clean guitar signal:
yf[0] = rfft(riff_clean_data[0:N][:,0])
yf_db[0] = 20 * np.log10(np.abs(yf[0] / yf[0][0]))
xf[0] = rfftfreq(N, 1/samplerate)

# FFT of bandpass-filtered guitar signal:
bpf_data = bpf_clip_bpf(riff_clean_data,lpf_coef1,hpf_coef1,1,0,0)
yf[1] = rfft(bpf_data[0:N][:,0])
yf_db[1] = 20 * np.log10(np.abs(yf[1] / yf[0][0]))
xf[1] = rfftfreq(N, 1/samplerate)

# FFT of bandpass-filtered and hard-clipped guitar signal:
bpf_hc_data = bpf_clip_bpf(riff_clean_data,lpf_coef1,hpf_coef1,clip_thresh,0,0)
yf[2] = rfft(bpf_hc_data[0:N][:,0])
yf_db[2] = 20 * np.log10(np.abs(yf[2] / yf[0][0]))
xf[2] = rfftfreq(N, 1/samplerate)

# FFT of bandpass-filtered, hard-clipped, then re-bandpass-filtered guitar signal:
bpf_hc_bpf_data = bpf_clip_bpf(riff_clean_data,lpf_coef1,hpf_coef1,clip_thresh,lpf_coef2,hpf_coef2)
yf[3] = rfft(bpf_hc_bpf_data[0:N][:,0])
yf_db[3] = 20 * np.log10(np.abs(yf[3] / yf[0][0]))
xf[3] = rfftfreq(N, 1/samplerate)

fig, axs = plt.subplots(2,2,figsize=(20,20))
axs[0,0].plot(xf[0][0:fmax], yf_db[0][0:fmax])
axs[0,0].set_xlabel('Frequency [Hz]')
axs[0,0].set_xscale('log')
axs[0,0].set_ylabel('Amplitude [dB]')
axs[0,0].set_title('FFT, clean signal')

axs[0,1].plot(xf[1][0:fmax], yf_db[1][0:fmax])
axs[0,1].set_xlabel('Frequency [Hz]')
axs[0,1].set_xscale('log')
axs[0,1].set_ylabel('Amplitude [dB]')
axs[0,1].set_title("FFT, BPF'd signal")

axs[1,0].plot(xf[2][0:fmax], yf_db[2][0:fmax])
axs[1,0].set_xlabel('Frequency [Hz]')
axs[1,0].set_xscale('log')
axs[1,0].set_ylabel('Amplitude [dB]')
axs[1,0].set_title("FFT, BPF'd & hard-clipped signal")

axs[1,1].plot(xf[3][0:fmax], yf_db[3][0:fmax])
axs[1,1].set_xlabel('Frequency [Hz]')
axs[1,1].set_xscale('log')
axs[1,1].set_ylabel('Amplitude [dB]')
axs[1,1].set_title("FFT, BPF'd, hard-clipped, re-BPF'd signal")

plt.show()
```

## Conclusion
We managed to get a decent approximation of the distorted guitar recording, using a cascade of pretty simple effects units:
* bandpass filter (itself implemented as a cascade: a lowpass complementary filter followed by a highpass complementary filter)
* hard-clipping followed by makeup gain
* another bandpass filter to smooth the sonic rough edges introduced by hard-clipping

Still, our reverse-engineered effects chain doesn't quite sound like the original recording, and there are a number of things that could be done to get closer. Three extensions that stand out are
* apply some "compression" (aka dynamic gain control) to the clean input before filtering it, clipping it, etc. This will provide a more uniform signal to the effects later in the signal chain, and that could be advantageous;
* replace the hard-clipping effect with a soft-clipping effect: instead of simply saturating the input signal, a smooth function like a logistic curve or the arctangent could be used to achieve the desired distortion with less of a "bitcrushed" sound;
* put more effort into filter design, starting with replacing the complementary filters with finite-impulse-response filters, which introduce fewer audio artifacts and whose frequency-domain properties can be designed using SciPy tools.

```python

```


```python
# imports from the pyfoomb package
import pyfoomb
print(f'Current package version of {pyfoomb.__name__}: {pyfoomb.__version__}')

from pyfoomb import BioprocessModel
from pyfoomb import Caretaker
from pyfoomb import Visualization
```

    C:\Users\hemmerich\AppData\Local\Continuum\miniconda3\envs\pyfoomb-dev\lib\site-packages\assimulo\solvers\odepack.py:701: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      N.array([[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.],  # 5th order
    Could not find GLIMDA.

    Current package version of pyfoomb: 2.17.7

# Modelling

In general, it is advised to write down the model before implementing is in code. In juypter notebooks, this can be nicely done using the TeX syntax.

### Sequential kinetic model

$$A \xrightarrow[]{k_1} B \xrightarrow[]{k_2} C$$

#### Derivatives
$$
\begin{align}
    \frac{dA}{dt} &= -k_1 \cdot A & A\left(t_0\right) = A_0\\
    \frac{dB}{dt} &= k_1 \cdot A - k_2 \cdot B & B\left(t_0\right) = B_0\\
    \frac{dC}{dt} &= k_2 \cdot B & C\left(t_0\right) = C_0\\
\end{align}
$$

#### Parametrization
$$
\begin{align}
    \mathbf{\theta_x} &= \left[ k_1 \; k_2\right]^T\\
    \mathbf{x}_0 &= \left[ A_0 \; B_0 \; C_0\right]^T\\
\end{align}
$$

# Implementation

Afterwards, the model class is created. this new class (here, `SequentialKinetic`) subclasses `BioprocessModel`. Thereby, the new class will inherit all relevant methods and properties needed to work with the model in the `pyFOOMB` package.

It is mandatory that the new class `SequentialKinetics` implements the method `rhs` (for right hand side), as seen below. This method describes the state derivatives, which are used by the integrator. In case the new class does not implement the method `rhs`, an error is raised.

There are two more methods that can be implemented by the new class in case one seeks to implement a model with events. These methods are `state_events` with signature `(self, t, y, sw)` and `change_states` with signature `(self, y, sw, state_event_info)`. When implementing a model with event, the signature of method `rhs` in needed to look like `(self, t, y, sw)`. How to work on models with events is shown in the seconds example notebook.

```python
# Defines the model class
class SequentialKinetic(BioprocessModel):

    def rhs(self, t, y):
        # Unpacks the state vector. The states are alphabetically ordered.
        A, B, C = y

        # Unpacks the model parameters.
        k1 = self.model_parameters['k1']
        k2 = self.model_parameters['k2']

        # Defines the derivatives.
        dAdt = -k1 * A
        dBdt = k1 * A - k2*B
        dCdt = k2*B

        # Returns the derivative as list (or numpy array).
        # The order corresponds to the state vector.
        return [dAdt, dBdt, dCdt]

# Defines a dictionary for the initial values.
# The keys corresponds to the model states, extended by a 0 (zero).
initial_values = {
    'A0' : 50.0,
    'B0' : 0.0,
    'C0' : 0.0,
}

# Defines a dictionary for the model parameters
# The keys match those variable names used in the model class
model_parameters = {
    'k1' : 0.2,
    'k2' : 0.1,
}
```

# The Caretaker

After defining a model class and corresponding dictionaries describing its parametrization, a `caretaker` object is creates from the `Caretaker` class, provided by the `pyFOOMB` package. The caretaker manages all the methods that are needed to work with your bioprocess model.

```python
# Instantiate a Caretaker, which manages the model and its parametrization.
caretaker = Caretaker(
    bioprocess_model_class=SequentialKinetic,
    model_parameters=model_parameters,
    initial_values=initial_values,
)
```

```python
# Runs a forward simulation.
simulation = caretaker.simulate(t=24)
```

```python
# Visualize the resulting trajectories
_ = Visualization.show_kinetic_data(simulation)
```

```python
# One can run a forward simulation also for different, specific parameter values by using the `parameters` keyword.
simulation = caretaker.simulate(t=24, parameters={'B0' : 5.0, 'k1' : 0.05})
_ = Visualization.show_kinetic_data(simulation)
```

```python
# This is easily verified by looking at the parameter mapping property of the caretaker
caretaker.parameter_mapping
```

      local_name
      value

      global_name
      replicate_id

      A0
      NaN
      A0
      50.0

      B0
      NaN
      B0
      0.0

      C0
      NaN
      C0
      0.0

      k1
      NaN
      k1
      0.2

      k2
      NaN
      k2
      0.1

# Model replicates

In this parameter mapping, there are entries named `global_name`, `replicate_id`, and `local_name`. This is because the Caretaker class can handle several replicates / instances of a model, which all share the same strukture. On the other hand it is allowed that certain variables can be share by a subset of the model replicates. This is indicated by the same `local_name`. The next section will show how to create such a caretaker and how to handle the multiple model replicates.

```python
# We'll overwrite the current caretaker.
# Note the use of the keyword `replicate_ids`
caretaker = Caretaker(
    bioprocess_model_class=SequentialKinetic,
    model_parameters=model_parameters,
    initial_values=initial_values,
    replicate_ids=['1st', '2nd'],
)
```

```python
# Like before, runs a forward simulation and visualize the result
simulation = caretaker.simulate(t=24)
_ = Visualization.show_kinetic_data(simulation)
```

We obtain the same trajectories for both replicate_ids. This is becuase they share all the same parameters by default.
To allow `replicate_id` specific parameter values, one need to apply parameter mappings.

```python
from pyfoomb import ParameterMapper
```

```python
# Creates a list of ParameterMapper objects that will be applied to the caretaker
# NOTE: One can use also just tuples with entries (replicate_id, global_id, local_name, value)
# NOTE: The `local_name` parameter can also be left empty.
#       Then, the local parameter name will be constructed automatically from the `replicate_id` and `global_name`
mappings = [
    ParameterMapper(replicate_id='1st', global_name='A0', value=50.0),
    ParameterMapper(replicate_id='2nd', global_name='A0', value=25.0),
    ParameterMapper(replicate_id='2nd', global_name='B0', local_name='B0_second', value=5.0),
]

# Applies the mappings
caretaker.apply_mappings(mappings)

# Show the result
caretaker.parameter_mapping
```

      local_name
      value

      global_name
      replicate_id

      A0
      1st
      A0_1st
      50.0

      2nd
      A0_2nd
      25.0

      B0
      1st
      B0
      0.0

      2nd
      B0_second
      5.0

      C0
      1st
      C0
      0.0

      2nd
      C0
      0.0

      k1
      1st
      k1
      0.2

      2nd
      k1
      0.2

      k2
      1st
      k2
      0.1

      2nd
      k2
      0.1

```python
# Runs a forward simulation and visualize the result
simulation = caretaker.simulate(t=24)
_ = Visualization.show_kinetic_data(simulation)
```

```python
# Of course, one can add another replicate to the caretaker
caretaker.add_replicate(replicate_id='3rd')

# The new replicate will be assigned the default parameter names and values
caretaker.parameter_mapping
```

    c:\users\hemmerich\source\repos\pyfoomb\pyfoomb\parameter.py:164: UserWarning: Detected unknown parameters, which are ignored: {'A0_1st', 'B0_second', 'A0_2nd'}
      warnings.warn(f'Detected unknown parameters, which are ignored: {unknown_parameters}', UserWarning)

      local_name
      value

      global_name
      replicate_id

      A0
      1st
      A0_1st
      50.0

      2nd
      A0_2nd
      25.0

      3rd
      A0
      50.0

      B0
      1st
      B0
      0.0

      2nd
      B0_second
      5.0

      3rd
      B0
      0.0

      C0
      1st
      C0
      0.0

      2nd
      C0
      0.0

      3rd
      C0
      0.0

      k1
      1st
      k1
      0.2

      2nd
      k1
      0.2

      3rd
      k1
      0.2

      k2
      1st
      k2
      0.1

      2nd
      k2
      0.1

      3rd
      k2
      0.1

```python
# Now modify a model parameter for the newly added replicate
# NOTE: The mappings bust be list of ParameterMapper objects
# NOTE: When the `value` argument is not set, the value of the global parameter is used.
mappings = [ParameterMapper(replicate_id='3rd', global_name='k1')]
caretaker.apply_mappings(mappings)
caretaker.parameter_mapping
```

      local_name
      value

      global_name
      replicate_id

      A0
      1st
      A0_1st
      50.0

      2nd
      A0_2nd
      25.0

      3rd
      A0
      50.0

      B0
      1st
      B0
      0.0

      2nd
      B0_second
      5.0

      3rd
      B0
      0.0

      C0
      1st
      C0
      0.0

      2nd
      C0
      0.0

      3rd
      C0
      0.0

      k1
      1st
      k1
      0.2

      2nd
      k1
      0.2

      3rd
      k1_3rd
      0.2

      k2
      1st
      k2
      0.1

      2nd
      k2
      0.1

      3rd
      k2
      0.1

```python
simulation = caretaker.simulate(t=24)
_ = Visualization.show_kinetic_data(simulation)
```

```python

```


# Neuromatch Academy: Week 3, Day 1, Tutorial 2
# Real Neurons: Effects of Input Correlation
__Content creators:__ Qinglong Gu, Songtin Li, John Murray, Richard Naud, Arvind Kumar

__Content reviewers:__ Maryam Vaziri-Pashkam, Ella Batty, Lorenzo Fontolan, Richard Gao, Matthew Krause, Spiros Chavlis, Michael Waskom

---
# Tutorial Objectives
In this tutorial, we will use the leaky integrate-and-fire (LIF) neuron model (see Tutorial 1) to study how they transform input correlations to output properties (transfer of correlations). In particular, we are going to write a few lines of code to:

- inject correlated GWN in a pair of neurons

- measure correlations between the spiking activity of the two neurons

- study how the transfer of correlation depends on the statistics of the input, i.e. mean and standard deviation.

---
# Setup

```python
# Import libraries
import matplotlib.pyplot as plt
import numpy as np
import time
```

```python
# @title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
# use NMA plot style
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
my_layout = widgets.Layout()
```

```python
# @title Helper functions
def default_pars(**kwargs):
  pars = {}

  ### typical neuron parameters###
  pars['V_th'] = -55.     # spike threshold [mV]
  pars['V_reset'] = -75.  # reset potential [mV]
  pars['tau_m'] = 10.     # membrane time constant [ms]
  pars['g_L'] = 10.       # leak conductance [nS]
  pars['V_init'] = -75.   # initial potential [mV]
  pars['V_L'] = -75.      # leak reversal potential [mV]
  pars['tref'] = 2.       # refractory time (ms)

  ### simulation parameters ###
  pars['T'] = 400. # Total duration of simulation [ms]
  pars['dt'] = .1  # Simulation time step [ms]

  ### external parameters if any ###
  for k in kwargs:
    pars[k] = kwargs[k]

  pars['range_t'] = np.arange(0, pars['T'], pars['dt'])  # Vector of discretized
                                                         # time points [ms]
  return pars

def run_LIF(pars, Iinj):
  """
  Simulate the LIF dynamics with external input current

  Args:
    pars       : parameter dictionary
    Iinj       : input current [pA]. The injected current here can be a value or an array

  Returns:
    rec_spikes : spike times
    rec_v      : mebrane potential
  """

  # Set parameters
  V_th, V_reset = pars['V_th'], pars['V_reset']
  tau_m, g_L = pars['tau_m'], pars['g_L']
  V_init, V_L = pars['V_init'], pars['V_L']
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size
  tref = pars['tref']

  # Initialize voltage and current
  v = np.zeros(Lt)
  v[0] = V_init
  Iinj = Iinj * np.ones(Lt)
  tr = 0.

  # simulate the LIF dynamics
  rec_spikes = []   # record spike times
  for it in range(Lt - 1):
    if tr > 0:
      v[it] = V_reset
      tr = tr - 1
    elif v[it] >= V_th:  # reset voltage and record spike event
      rec_spikes.append(it)
      v[it] = V_reset
      tr = tref / dt

    # calculate the increment of the membrane potential
    dv = (-(v[it] - V_L) + Iinj[it] / g_L) * (dt / tau_m)

    # update the membrane potential
    v[it + 1] = v[it] + dv

  rec_spikes = np.array(rec_spikes) * dt

  return v, rec_spikes

def my_GWN(pars, sig, myseed=False):
  """
  Function that calculates Gaussian white noise inputs

  Args:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    myseed     : random seed. int or boolean
                 the same seed will give the same random number sequence

  Returns:
    I          : Gaussian white noise input
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # Set random seed. You can fix the seed of the random number generator so
  # that the results are reliable however, when you want to generate multiple
  # realization make sure that you change the seed for each new realization
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate GWN
  # we divide here by 1000 to convert units to sec.
  I_GWN = sig * np.random.randn(Lt) * np.sqrt(pars['tau_m'] / dt)

  return I_GWN

def Poisson_generator(pars, rate, n, myseed=False):
  """
  Generates poisson trains

  Args:
    pars       : parameter dictionary
    rate       : noise amplitute [Hz]
    n          : number of Poisson trains
    myseed     : random seed. int or boolean

  Returns:
    pre_spike_train : spike train matrix, ith row represents whether
                      there is a spike in ith spike train over time
                      (1 if spike, 0 otherwise)
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate uniformly distributed random variables
  u_rand = np.random.rand(n, Lt)

  # generate Poisson train
  poisson_train = 1. * (u_rand < rate * (dt / 1000.))

  return poisson_train

def example_plot_myCC():
  pars = default_pars(T=50000, dt=.1)

  c = np.arange(10) * 0.1
  r12 = np.zeros(10)
  for i in range(10):
    I1gL, I2gL = correlate_input(pars, mu=20.0, sig=7.5, c=c[i])
    r12[i] = my_CC(I1gL, I2gL)

  plt.figure()
  plt.plot(c, r12, 'bo', alpha=0.7, label='Simulation', zorder=2)
  plt.plot([-0.05, 0.95], [-0.05, 0.95], 'k--', label='y=x',
           dashes=(2, 2), zorder=1)
  plt.xlabel('True CC')
  plt.ylabel('Sample CC')
  plt.legend(loc='best')

def LIF_output_cc(pars, mu, sig, c, bin_size, n_trials=20):
  """ Simulates two LIF neurons with correlated input and computes output correlation

  Args:
  pars       : parameter dictionary
  mu         : noise baseline (mean)
  sig        : noise amplitute (standard deviation)
  c          : correlation coefficient ~[0, 1]
  bin_size   : bin size used for time series
  n_trials   : total simulation trials

  Returns:
  r          : output corr. coe.
  sp_rate    : spike rate
  sp1        : spike times of neuron 1 in the last trial
  sp2        : spike times of neuron 2 in the last trial
  """

  r12 = np.zeros(n_trials)
  sp_rate = np.zeros(n_trials)
  for i_trial in range(n_trials):
    I1gL, I2gL = correlate_input(pars, mu, sig, c)
    _, sp1 = run_LIF(pars, pars['g_L'] * I1gL)
    _, sp2 = run_LIF(pars, pars['g_L'] * I2gL)

    my_bin = np.arange(0, pars['T'], bin_size)

    sp1_count, _ = np.histogram(sp1, bins=my_bin)
    sp2_count, _ = np.histogram(sp2, bins=my_bin)

    r12[i_trial] = my_CC(sp1_count[::20], sp2_count[::20])
    sp_rate[i_trial] = len(sp1) / pars['T'] * 1000.

  return r12.mean(), sp_rate.mean(), sp1, sp2

def plot_c_r_LIF(c, r, mycolor, mylabel):
  z = np.polyfit(c, r, deg=1)
  c_range = np.array([c.min() - 0.05, c.max() + 0.05])
  plt.plot(c, r, 'o', color=mycolor, alpha=0.7, label=mylabel, zorder=2)
  plt.plot(c_range, z[0] * c_range + z[1], color=mycolor, zorder=1)

```

The helper function contains the:

- Parameter dictionary: `default_pars( **kwargs)`
- LIF simulator: `run_LIF`
- Gaussian white noise generator: `my_GWN(pars, sig, myseed=False)`
- Poisson type spike train generator: `Poisson_generator(pars, rate, n, myseed=False)`
- Two LIF neurons with correlated inputs simulator: `LIF_output_cc(pars, mu, sig, c, bin_size, n_trials=20)`
- Some additional plotting utilities

---
# Section 1: Correlations (Synchrony)
Correlation or synchrony in neuronal activity can be described for any readout of brain activity. Here, we are concerned with the spiking activity of neurons.

In the simplest way, correlation/synchrony refers to coincident spiking of neurons, i.e., when two neurons spike together, they are firing in **synchrony** or are **correlated**. Neurons can be synchronous in their instantaneous activity, i.e., they spike together with some probability. However, it is also possible that spiking of a neuron at time $t$ is correlated with the spikes of another neuron with a delay (time-delayed synchrony).

## Origin of synchronous neuronal activity:
- Common inputs, i.e., two neurons are receiving input from the same sources. The degree of correlation of the shared inputs is proportional to their output correlation.
- Pooling from the same sources. Neurons do not share the same input neurons but are receiving inputs from neurons which themselves are correlated.
- Neurons are connected to each other (uni- or bi-directionally): This will only give rise to time-delayed synchrony. Neurons could also be connected via gap-junctions.
- Neurons have similar parameters and initial conditions.

## Implications of synchrony
When neurons spike together, they can have a stronger impact on downstream neurons. Synapses in the brain are sensitive to the temporal correlations (i.e., delay) between pre- and postsynaptic activity, and this, in turn, can lead to the formation of functional neuronal networks - the basis of unsupervised learning (we will study some of these concepts in a forthcoming tutorial).

Synchrony implies a reduction in the dimensionality of the system. In addition, correlations, in many cases, can impair the decoding of neuronal activity.

```python
# @title Video 1: Input & output correlations
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="nsAYFBcAkes", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=nsAYFBcAkes

## How to study the emergence of correlations

A simple model to study the emergence of correlations is to inject common inputs to a pair of neurons and measure the output correlation as a function of the fraction of common inputs.

Here, we are going to investigate the transfer of correlations by computing the correlation coefficient of spike trains recorded from two unconnected LIF neurons, which received correlated inputs.

The input current to LIF neuron $i$ $(i=1,2)$ is:

\begin{equation}
\frac{I_i}{g_L} =\mu_i + \sigma_i (\sqrt{1-c}\xi_i + \sqrt{c}\xi_c) \quad (1)
\end{equation}

where $\mu_i$ is the temporal average of the current. The Gaussian white noise $\xi_i$ is independent for each neuron, while $\xi_c$ is common to all neurons. The variable $c$ ($0\le c\le1$) controls the fraction of common and independent inputs. $\sigma_i$ shows the variance of the total input.

So, first, we will generate correlated inputs.

```python
# @title
#@markdown Execute this cell to get a function for generating correlated GWN inputs
def correlate_input(pars, mu=20., sig=7.5, c=0.3):
  """
  Args:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    c.         : correlation coefficient ~[0, 1]

  Returns:
    I1gL, I2gL : two correlated inputs with corr. coe. c
  """

  # generate Gaussian whute noise xi_1, xi_2, xi_c
  xi_1 = my_GWN(pars, sig)
  xi_2 = my_GWN(pars, sig)
  xi_c = my_GWN(pars, sig)

  # Generate two correlated inputs by Equation. (1)
  I1gL = mu + np.sqrt(1. - c) * xi_1 + np.sqrt(c) * xi_c
  I2gL = mu + np.sqrt(1. - c) * xi_2 + np.sqrt(c) * xi_c

  return I1gL, I2gL

print(help(correlate_input))
```

    Help on function correlate_input in module __main__:

    correlate_input(pars, mu=20.0, sig=7.5, c=0.3)
        Args:
          pars       : parameter dictionary
          mu         : noise baseline (mean)
          sig        : noise amplitute (standard deviation)
          c.         : correlation coefficient ~[0, 1]

        Returns:
          I1gL, I2gL : two correlated inputs with corr. coe. c

    None

### Exercise 1: Compute the correlation

The _sample correlation coefficient_ between two input currents $I_i$ and $I_j$ is defined as the sample covariance of $I_i$ and $I_j$ divided by the square root of the sample variance of $I_i$ multiplied with the square root of the sample variance of $I_j$. In equation form:

\begin{align}
r_{ij} &= \frac{cov(I_i, I_j)}{\sqrt{var(I_i)} \sqrt{var(I_j)}}\\
cov(I_i, I_j) &= \sum_{k=1}^L (I_i^k -\bar{I}_i)(I_j^k -\bar{I}_j) \\
var(I_i) &= \sum_{k=1}^L (I_i^k -\bar{I}_i)^2
\end{align}

where $\bar{I}_i$ is the sample mean, k is the time bin, and L is the length of $I$.  This means that $I_i^k$ is current i at time $k\cdot dt$. Note that the equations above are not accurate for sample covariances and variances as they should be additionally divided by L-1 - we have dropped this term because it cancels out in the sample correlation coefficient formula.

The _sample correlation coefficient_ may also be referred to as the _sample Pearson correlation coefficient_. Here, is a beautiful paper that explains multiple ways to calculate and understand correlations [Rodgers and Nicewander 1988](https://www.stat.berkeley.edu/~rabbee/correlation.pdf).

In this exercise, we will create a function, `my_CC` to compute the sample correlation coefficient between two time series. Note that while we introduced this computation here in the context of input currents, the sample correlation coefficient is used to compute the correlation between any two time series - we will use it later on binned spike trains.

```python
def my_CC(i, j):
  """
  Args:
    i, j  : two time series with the same length

  Returns:
    rij   : correlation coefficient
  """

  # Calculate the covariance of i and j
  cov = ((i - i.mean()) * (j - j.mean())).sum()
  # Calculate the variance of i
  var_i = ((i - i.mean()) * (i - i.mean())).sum()
  # Calculate the variance of j
  var_j = ((j - j.mean()) * (j - j.mean())).sum()
  # Calculate the correlation coefficient
  rij = cov / np.sqrt(var_i*var_j)

  return rij

with plt.xkcd():
  example_plot_myCC()
```

### Exercise 2: Measure the correlation between spike trains

After recording the spike times of the two neurons, how can we estimate their correlation coefficient?

In order to find this, we need to bin the spike times and obtain two time series. Each data point in the time series is the number of spikes in the corresponding time bin. You can use `np.histogram()` to bin the spike times.

Complete the code below to bin the spike times and calculate the correlation coefficient for two Poisson spike trains. Note that `c` here is the ground-truth correlation coefficient that we define.

```python
# @title

# @markdown Execute this cell to get a function for generating correlated Poisson inputs (generate_corr_Poisson)

def generate_corr_Poisson(pars, poi_rate, c, myseed=False):
  """
  function to generate correlated Poisson type spike trains
  Args:
    pars       : parameter dictionary
    poi_rate   : rate of the Poisson train
    c.         : correlation coefficient ~[0, 1]

  Returns:
    sp1, sp2   : two correlated spike time trains with corr. coe. c
  """

  range_t = pars['range_t']

  mother_rate = poi_rate / c
  mother_spike_train = Poisson_generator(pars, rate=mother_rate,
                                         n=1, myseed=myseed)[0]
  sp_mother = range_t[mother_spike_train > 0]

  L_sp_mother = len(sp_mother)
  sp_mother_id = np.arange(L_sp_mother)
  L_sp_corr = int(L_sp_mother * c)

  np.random.shuffle(sp_mother_id)
  sp1 = np.sort(sp_mother[sp_mother_id[:L_sp_corr]])

  np.random.shuffle(sp_mother_id)
  sp2 = np.sort(sp_mother[sp_mother_id[:L_sp_corr]])

  return sp1, sp2

print(help(generate_corr_Poisson))
```

    Help on function generate_corr_Poisson in module __main__:

    generate_corr_Poisson(pars, poi_rate, c, myseed=False)
        function to generate correlated Poisson type spike trains
        Args:
          pars       : parameter dictionary
          poi_rate   : rate of the Poisson train
          c.         : correlation coefficient ~[0, 1]

        Returns:
          sp1, sp2   : two correlated spike time trains with corr. coe. c

    None

Sample output

```
True corr coe = 0.200
Simu corr coe = 0.197
```

```python
def corr_coeff_pairs(pars, rate, c, trials, bins):
  """
  Calculate the correlation coefficient of two spike trains, for different
  realizations

  Args:
      pars   : parameter dictionary
      rate   : rate of poisson inputs
      c      : correlation coefficient ~ [0, 1]
      trials  : number of realizations
      bins   : vector with bins for time discretization

  Returns:
    r12      : correlation coefficient of a pair of inputs
  """

  r12 = np.zeros(n_trials)

  for i in range(n_trials):
    # Generate correlated Poisson inputs
    sp1, sp2 = generate_corr_Poisson(pars, poi_rate, c, myseed=2020+i)

    # Bin the spike times of the first input
    sp1_count, _ = np.histogram(sp1, bins=bins)

    # Bin the spike times of the second input
    sp2_count, _ = np.histogram(sp2, bins=bins)

    # Calculate the correlation coefficient
    r12[i] = my_CC(sp1_count, sp2_count)

  return r12

poi_rate = 20.
c = 0.2  # set true correlation
pars = default_pars(T=10000)
# bin the spike time
bin_size = 20  # [ms]
my_bin = np.arange(0, pars['T'], bin_size)
n_trials = 100  # 100 realizations

r12 = corr_coeff_pairs(pars, rate=poi_rate, c=c, trials=n_trials, bins=my_bin)
print(f'True corr coe = {c:.3f}')
print(f'Simu corr coe = {r12.mean():.3f}')
```

    True corr coe = 0.200
    Simu corr coe = 0.197

---
# Section 2: Investigate the effect of input correlation on the output correlation

Now let's combine the aforementioned two procedures. We first generate the correlated inputs by Equation (1). Then we inject the correlated inputs $I_1, I_2$ into a pair of neurons and record their output spike times. We continue measuring the correlation between the output and
investigate the relationship between the input correlation and the output correlation.

## Drive a neuron with correlated inputs and visualize its output
In the following, you will inject correlated GWN in two neurons. You need to define the mean (`gwn_mean`), standard deviation (`gwn_std`), and input correlations (`c_in`).

We will simulate $10$ trials to get a better estimate of the output correlation. Change the values in the following cell for the above variables (and then run the next cell) to explore how they impact the output correlation.

```python
# Play around with these parameters

pars = default_pars(T=80000, dt=1.)  # get the parameters
c_in = 0.1  # set input correlation value
gwn_mean = 5.
gwn_std = 10.

```

```python
# @title

# @markdown Do not forget to execute this cell to simulate the LIF

bin_size = 10.  # ms

starttime = time.perf_counter()  # time clock

r12_ss, sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=gwn_mean, sig=gwn_std, c=c_in,
                                        bin_size=bin_size, n_trials=10)

# just the time counter
endtime = time.perf_counter()
timecost = (endtime - starttime) / 60.
print(f"Simulation time = {timecost:.2f} min")

print(f"Input correlation = {c_in}")
print(f"Output correlation = {r12_ss}")

plt.figure(figsize=(12, 6))
plt.plot(sp1, np.ones(len(sp1)) * 1, '|', ms=20, label='neuron 1')
plt.plot(sp2, np.ones(len(sp2)) * 1.1, '|', ms=20, label='neuron 2')
plt.xlabel('time (ms)')
plt.ylabel('neuron id.')
plt.xlim(1000, 8000)
plt.ylim(0.9, 1.2)
plt.legend()
plt.show()
```

## Think!
- Is the output correlation always smaller than the input correlation? If yes, why?
- Should there be a systematic relationship between input and output correlations?

You will explore these questions in the next figure but try to develop your own intuitions first!

Lets vary `c_in` and plot the relationship between the `c_in` and output correlation. This might take some time depending on the number of trials.

```python
#@title

#@markdown Don't forget to execute this cell!

pars = default_pars(T=80000, dt=1.)  # get the parameters
bin_size = 10.
c_in = np.arange(0, 1.0, 0.1)  # set the range for input CC
r12_ss = np.zeros(len(c_in))  # small mu, small sigma

starttime = time.perf_counter() # time clock
for ic in range(len(c_in)):
  r12_ss[ic], sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=10.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=10)

endtime = time.perf_counter()
timecost = (endtime - starttime) / 60.
print(f"Simulation time = {timecost:.2f} min")

plt.figure(figsize=(7, 6))
plot_c_r_LIF(c_in, r12_ss, mycolor='b', mylabel='Output CC')
plt.plot([c_in.min() - 0.05, c_in.max() + 0.05],
         [c_in.min() - 0.05, c_in.max() + 0.05],
         'k--', dashes=(2, 2), label='y=x')

plt.xlabel('Input CC')
plt.ylabel('Output CC')
plt.legend(loc='best', fontsize=16)
plt.show()
```

```python
"""
Discussion: The results above show that
- output correlation is smaller than input correlation
- output correlation varies linearly as a function of input correlation.

While the general result holds, this relationship might change depending on the neuron type.
""";
```

---
# Section 3: Correlation transfer function
The above plot of input correlation vs. output correlation is called the __correlation transfer function__ of the neurons.

## Section 3.1: How do the mean and standard deviation of the GWN affect the correlation transfer function?

The correlations transfer function appears to be linear. The above can be taken as the input/output transfer function of LIF neurons for correlations, instead of the transfer function for input/output firing rates as we had discussed in the previous tutorial (i.e., F-I curve).

What would you expect to happen to the slope of the correlation transfer function if you vary the mean and/or the standard deviation of the GWN?

```python
#@markdown Execute this cell to visualize correlation transfer functions

pars = default_pars(T=80000, dt=1.) # get the parameters
no_trial = 10
bin_size = 10.
c_in = np.arange(0., 1., 0.2)  # set the range for input CC
r12_ss = np.zeros(len(c_in))   # small mu, small sigma
r12_ls = np.zeros(len(c_in))   # large mu, small sigma
r12_sl = np.zeros(len(c_in))   # small mu, large sigma

starttime = time.perf_counter()  # time clock
for ic in range(len(c_in)):
  r12_ss[ic], sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=10.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=no_trial)
  r12_ls[ic], sp_ls, sp1, sp2 = LIF_output_cc(pars, mu=50.0, sig=10.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=no_trial)
  r12_sl[ic], sp_sl, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=20.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=no_trial)
endtime = time.perf_counter()
timecost = (endtime - starttime) / 60.
print(f"Simulation time = {timecost:.2f} min")

plt.figure(figsize=(7, 6))
plot_c_r_LIF(c_in, r12_ss, mycolor='b', mylabel=r'Small $\mu$, small $\sigma$')
plot_c_r_LIF(c_in, r12_ls, mycolor='y', mylabel=r'Large $\mu$, small $\sigma$')
plot_c_r_LIF(c_in, r12_sl, mycolor='r', mylabel=r'Small $\mu$, large $\sigma$')
plt.plot([c_in.min() - 0.05, c_in.max() + 0.05],
         [c_in.min() - 0.05, c_in.max() + 0.05],
         'k--', dashes=(2, 2), label='y=x')
plt.xlabel('Input CC')
plt.ylabel('Output CC')
plt.legend(loc='best', fontsize=14)
plt.show()
```

### Think!
Why do both the mean and the standard deviation of the GWN affect the slope of the correlation transfer function?

```python
# to_remove explanation

"""
Discussion: This has got to do with which part of the input current distribution
is transferred to the spiking activity.

Intuitive understanding is difficult but this relationship arises due to non-linearities
in the neuron F-I curve.  When F-I curve is linear, output correlation is independent
of the mean and standard deviation. But this relationship arises even in neurons with
threshold-linear F-I curve.

Please see:
De La Rocha J, Doiron B, Shea-Brown E, Josić K, Reyes A. Correlation between
neural spike trains increases with firing rate. Nature. 2007 Aug;448(7155):802-6.
""";
```

## Section 3.2: What is the rationale behind varying $\mu$ and $\sigma$?
The mean and the variance of the synaptic current depends on the spike rate of a Poisson process. We can use [Campbell's theorem](https://en.wikipedia.org/wiki/Campbell%27s_theorem_(probability)) to estimate the mean and the variance of the synaptic current:

\begin{align}
\mu_{\rm syn} = \lambda J \int P(t) \\
\sigma_{\rm syn} = \lambda J \int P(t)^2 dt\\
\end{align}

where $\lambda$ is the firing rate of the Poisson input, $J$ the amplitude of the postsynaptic current and $P(t)$ is the shape of the postsynaptic current as a function of time.

Therefore, when we varied $\mu$ and/or $\sigma$ of the GWN, we mimicked a change in the input firing rate. Note that, if we change the firing rate, both $\mu$ and $\sigma$ will change simultaneously, not independently.

Here, since we observe an effect of $\mu$ and $\sigma$ on correlation transfer, this implies that the input rate has an impact on the correlation transfer function.

### Think!

- What are the factors that would make output correlations smaller than input correlations? (Notice that the colored lines are below the black dashed line)
- What does it mean for the correlation in the network?
- Here we have studied the transfer of correlations by injecting GWN. But in the previous tutorial, we mentioned that GWN is unphysiological. Indeed, neurons receive colored noise (i.e., Shot noise or OU process). How do these results obtained from injection of GWN apply to the case where correlated spiking inputs are injected in the two LIFs? Will the results be the same or different?

Reference
- De La Rocha, Jaime, et al. "Correlation between neural spike trains increases with firing rate." Nature (2007) (https://www.nature.com/articles/nature06028/)

- Bujan AF, Aertsen A, Kumar A. Role of input correlations in shaping the variability and noise correlations of evoked activity in the neocortex. Journal of Neuroscience. 2015 Jun 3;35(22):8611-25. (https://www.jneurosci.org/content/35/22/8611)

```python
# to_remove explanation

"""
Discussion:

1. Anything that tries to reduce the mean or variance of the input e.g. mean can
be reduced by inhibition, sigma can be reduced by the membrane time constant.
Obviously, if the two neurons have different parameters that will decorrelate them.
But more importantly, it is the slope of neuron transfer function that will affect the
output correlation.

2. These observations pose an interesting problem at the network level. If the
output correlation are smaller than the input correlation, then the network activity
should eventually converge to zero correlation. But that does not happen. So there
is something missing in this model to understand origin of synchrony in the network.

3. For spike trains, if we do not have explicit control over mu and sigma.
And these two variables will be tied to the firing rate of the inputs. So the
results will be qualitatively similar. But when we think of multiple spike inputs
two different types of correlations arise (see Bujan et al. 2015 for more info)
""";
```

---
# Summary

In this tutorial, we studied how the input correlation of two LIF neurons is mapped to their output correlation. Specifically, we:

- injected correlated GWN in a pair of neurons,

- measured correlations between the spiking activity of the two neurons, and

- studied how the transfer of correlation depends on the statistics of the input, i.e., mean and standard deviation.

Here, we were concerned with zero time lag correlation. For this reason, we restricted estimation of correlation to instantaneous correlations. If you are interested in time-lagged correlation, then we should estimate the cross-correlogram of the spike trains and find out the dominant peak and area under the peak to get an estimate of output correlations.

We leave this as a future to-do for you if you are interested.

---
# Bonus 1: Example of a conductance-based LIF model
Above, we have written code to generate correlated Poisson spike trains. You can write code to stimulate the LIF neuron with such correlated spike trains and study the correlation transfer function for spiking input and compare it to the correlation transfer function obtained by injecting correlated GWNs.

```python
# @title Function to simulate conductance-based LIF

def run_LIF_cond(pars, I_inj, pre_spike_train_ex, pre_spike_train_in):
  """
  conductance-based LIF dynamics

  Args:
    pars               : parameter dictionary
    I_inj              : injected current [pA]. The injected current here can
                         be a value or an array
    pre_spike_train_ex : spike train input from presynaptic excitatory neuron
    pre_spike_train_in : spike train input from presynaptic inhibitory neuron

  Returns:
    rec_spikes         : spike times
    rec_v              : mebrane potential
    gE                 : postsynaptic excitatory conductance
    gI                 : postsynaptic inhibitory conductance
  """

  # Retrieve parameters
  V_th, V_reset = pars['V_th'], pars['V_reset']
  tau_m, g_L = pars['tau_m'], pars['g_L']
  V_init, E_L = pars['V_init'], pars['E_L']
  gE_bar, gI_bar = pars['gE_bar'], pars['gI_bar']
  VE, VI = pars['VE'], pars['VI']
  tau_syn_E, tau_syn_I = pars['tau_syn_E'], pars['tau_syn_I']
  tref = pars['tref']
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # Initialize
  tr = 0.
  v = np.zeros(Lt)
  v[0] = V_init
  gE = np.zeros(Lt)
  gI = np.zeros(Lt)
  Iinj = I_inj * np.ones(Lt)  # ensure I has length Lt

  if pre_spike_train_ex.max() == 0:
    pre_spike_train_ex_total = np.zeros(Lt)
  else:
    pre_spike_train_ex_total = pre_spike_train_ex * np.ones(Lt)

  if pre_spike_train_in.max() == 0:
    pre_spike_train_in_total = np.zeros(Lt)
  else:
    pre_spike_train_in_total = pre_spike_train_in * np.ones(Lt)

  # simulation
  rec_spikes = []  # recording spike times
  for it in range(Lt - 1):
    if tr > 0:
      v[it] = V_reset
      tr = tr - 1
    elif v[it] >= V_th:   # reset voltage and record spike event
      rec_spikes.append(it)
      v[it] = V_reset
      tr = tref / dt
    # update the synaptic conductance
    # input change synaptic conductance
    gE[it+1] = gE[it] - (dt / tau_syn_E) * gE[it] + gE_bar * pre_spike_train_ex_total[it + 1]
    gI[it+1] = gI[it] - (dt / tau_syn_I) * gI[it] + gI_bar * pre_spike_train_in_total[it + 1]

    # calculate the increment of the membrane potential
    # VE: excitatory synapse conductance; VI: inhibitory synapse conductance
    dv = (-(v[it] - E_L) - (gE[it + 1] / g_L) * (v[it] - VE) - \
            (gI[it + 1] / g_L) * (v[it] - VI) + Iinj[it] / g_L) * (dt / tau_m)

    # update membrane potential
    v[it + 1] = v[it] + dv

  rec_spikes = np.array(rec_spikes) * dt

  return v, rec_spikes, gE, gI

print(help(run_LIF_cond))
```

    Help on function run_LIF_cond in module __main__:

    run_LIF_cond(pars, I_inj, pre_spike_train_ex, pre_spike_train_in)
        conductance-based LIF dynamics

        Args:
          pars               : parameter dictionary
          I_inj              : injected current [pA]. The injected current here can
                               be a value or an array
          pre_spike_train_ex : spike train input from presynaptic excitatory neuron
          pre_spike_train_in : spike train input from presynaptic inhibitory neuron

        Returns:
          rec_spikes         : spike times
          rec_v              : mebrane potential
          gE                 : postsynaptic excitatory conductance
          gI                 : postsynaptic inhibitory conductance

    None

## Interactive Demo: Correlated spike input to an LIF neuron

In the following you can explore what happens when the neurons receive correlated spiking input.

You can vary the correlation between excitatory input spike trains. For simplicity, the correlation between inhibitory spike trains is set to 0.01.

Vary both excitatory rate and correlation and see how the output correlation changes. Check if the results are qualitatively similar to what you observed previously when you varied the $\mu$ and $\sigma$.

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

my_layout.width = '450px'
@widgets.interact(
  pwc_ee=widgets.FloatSlider(0.3, min=0.05, max=0.99, step=0.01,
                             layout=my_layout),
  exc_rate=widgets.FloatSlider(1e3, min=500., max=5e3, step=50.,
                               layout=my_layout),
  inh_rate=widgets.FloatSlider(500., min=300., max=5e3, step=5.,
                               layout=my_layout),
)

def EI_isi_regularity(pwc_ee, exc_rate, inh_rate):
  pars = default_pars(T=1000.)
  # Add parameters
  pars['V_th'] = -55.     # spike threshold [mV]
  pars['V_reset'] = -75.  # reset potential [mV]
  pars['tau_m'] = 10.     # membrane time constant [ms]
  pars['g_L'] = 10.       # leak conductance [nS]
  pars['V_init'] = -65.   # initial potential [mV]
  pars['E_L'] = -75.      # leak reversal potential [mV]
  pars['tref'] = 2.       # refractory time (ms)

  pars['gE_bar'] = 4.0    # [nS]
  pars['VE'] = 0.         # [mV] excitatory reversal potential
  pars['tau_syn_E'] = 2.  # [ms]
  pars['gI_bar'] = 2.4    # [nS]
  pars['VI'] = -80.       # [mV] inhibitory reversal potential
  pars['tau_syn_I'] = 5.  # [ms]

  my_bin = np.arange(0, pars['T']+pars['dt'], .1)  # 20 [ms] bin-size

  # exc_rate = 1e3
  # inh_rate = 0.4e3
  # pwc_ee = 0.3
  pwc_ii = 0.01
  # generate two correlated spike trains for excitatory input
  sp1e, sp2e = generate_corr_Poisson(pars, exc_rate, pwc_ee)

  sp1_spike_train_ex, _ = np.histogram(sp1e, bins=my_bin)
  sp2_spike_train_ex, _ = np.histogram(sp2e, bins=my_bin)

  # generate two uncorrelated spike trains for inhibitory input
  sp1i, sp2i = generate_corr_Poisson(pars, inh_rate, pwc_ii)
  sp1_spike_train_in, _ = np.histogram(sp1i, bins=my_bin)
  sp2_spike_train_in, _ = np.histogram(sp2i, bins=my_bin)

  v1, rec_spikes1, gE, gI = run_LIF_cond(pars, 0, sp1_spike_train_ex, sp1_spike_train_in)
  v2, rec_spikes2, gE, gI = run_LIF_cond(pars, 0, sp2_spike_train_ex, sp2_spike_train_in)

  # bin the spike time
  bin_size = 20  # [ms]
  my_bin = np.arange(0, pars['T'], bin_size)
  spk_1, _ = np.histogram(rec_spikes1, bins=my_bin)
  spk_2, _ = np.histogram(rec_spikes2, bins=my_bin)

  r12 = my_CC(spk_1, spk_2)
  print(f"Input correlation = {pwc_ee}")
  print(f"Output correlation = {r12}")

  plt.figure(figsize=(14, 7))
  plt.subplot(211)
  plt.plot(sp1e, np.ones(len(sp1e)) * 1, '|', ms=20,
           label='Exc. input 1')
  plt.plot(sp2e, np.ones(len(sp2e)) * 1.1, '|', ms=20,
           label='Exc. input 2')
  plt.plot(sp1i, np.ones(len(sp1i)) * 1.3, '|k', ms=20,
           label='Inh. input 1')
  plt.plot(sp2i, np.ones(len(sp2i)) * 1.4, '|k', ms=20,
           label='Inh. input 2')

  plt.ylim(0.9, 1.5)
  plt.legend()
  plt.ylabel('neuron id.')

  plt.subplot(212)
  plt.plot(pars['range_t'], v1, label='neuron 1')
  plt.plot(pars['range_t'], v2, label='neuron 2')
  plt.xlabel('time (ms)')
  plt.ylabel('membrane voltage $V_{m}$')
  plt.tight_layout()
  plt.show()
```

    interactive(children=(FloatSlider(value=0.3, description='pwc_ee', layout=Layout(width='450px'), max=0.99, min…

Above, we are estimating the output correlation for one trial. You can modify the code to get a trial average of output correlations.

---
# Bonus 2: Ensemble Response

Finally, there is a short BONUS lecture video on the firing response of an ensemble of neurons to time-varying input. There are no associated coding exercises - just enjoy.

```python
#@title Video 2 (Bonus): Response of ensemble of neurons to time-varying input
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="78_dWa4VOIo", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=78_dWa4VOIo


**This notebook is an exercise in the [Computer Vision](https://www.kaggle.com/learn/computer-vision) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/convolution-and-relu).**

---

# Introduction #

In this exercise, you'll work on building some intuition around feature extraction. First, we'll walk through the example we did in the tutorial again, but this time, with a kernel you choose yourself. We've mostly been working with images in this course, but what's behind all of the operations we're learning about is mathematics. So, we'll also take a look at how these feature maps can be represented instead as arrays of numbers and what effect convolution with a kernel will have on them.

Run the cell below to get started!

```python
# Setup feedback system
from learntools.core import binder
binder.bind(globals())
from learntools.computer_vision.ex2 import *

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

plt.rc('figure', autolayout=True)
plt.rc('axes', labelweight='bold', labelsize='large',
       titleweight='bold', titlesize=18, titlepad=10)
plt.rc('image', cmap='magma')

tf.config.run_functions_eagerly(True)
```

# Apply Transformations #

The next few exercises walk through feature extraction just like the example in the tutorial. Run the following cell to load an image we'll use for the next few exercises.

```python
image_path = '../input/computer-vision-resources/car_illus.jpg'
image = tf.io.read_file(image_path)
image = tf.io.decode_jpeg(image, channels=1)
image = tf.image.resize(image, size=[400, 400])

img = tf.squeeze(image).numpy()
plt.figure(figsize=(6, 6))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.show();
```

You can run this cell to see some standard kernels used in image processing.

```python
import learntools.computer_vision.visiontools as visiontools
from learntools.computer_vision.visiontools import edge, bottom_sobel, emboss, sharpen

kernels = [edge, bottom_sobel, emboss, sharpen]
names = ["Edge Detect", "Bottom Sobel", "Emboss", "Sharpen"]

plt.figure(figsize=(12, 12))
for i, (kernel, name) in enumerate(zip(kernels, names)):
    plt.subplot(1, 4, i+1)
    visiontools.show_kernel(kernel)
    plt.title(name)
plt.tight_layout()
```

# 1) Define Kernel #

Use the next code cell to define a kernel. You have your choice of what kind of kernel to apply. One thing to keep in mind is that the *sum* of the numbers in the kernel determines how bright the final image is. Generally, you should try to keep the sum of the numbers between 0 and 1 (though that's not required for a correct answer).

In general, a kernel can have any number of rows and columns. For this exercise, let's use a $3 \times 3$ kernel, which often gives the best results. Define a kernel with `tf.constant`.

```python
# YOUR CODE HERE: Define a kernel with 3 rows and 3 columns.
kernel = tf.constant([
    [0, 1, 2],
    [1, 0, -1],
    [-2, -1, 0]
])
# Uncomment to view kernel
visiontools.show_kernel(kernel)

# Check your answer
q_1.check()
```

```python
# Lines below will give you a hint or solution code
#q_1.hint()
#q_1.solution()
```

Now we'll do the first step of feature extraction, the filtering step. First run this cell to do some reformatting for TensorFlow.

```python
# Reformat for batch compatibility.
image = tf.image.convert_image_dtype(image, dtype=tf.float32)
image = tf.expand_dims(image, axis=0)
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

# 2) Apply Convolution #

Now we'll apply the kernel to the image by a convolution. The *layer* in Keras that does this is `layers.Conv2D`. What is the *backend function* in TensorFlow that performs the same operation?

```python
# YOUR CODE HERE: Give the TensorFlow convolution function (without arguments)
conv_fn = tf.nn.conv2d

# Check your answer
q_2.check()
```

    <IPython.core.display.Javascript object>

Correct

```python
# Lines below will give you a hint or solution code
#q_2.hint()
#q_2.solution()
```

Once you've got the correct answer, run this next cell to execute the convolution and see the result!

```python
image_filter = conv_fn(
    input=image,
    filters=kernel,
    strides=1, # or (1, 1)
    padding='SAME',
)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_filter)
)
plt.axis('off')
plt.show();
```

Can you see how the kernel you chose relates to the feature map it produced?

# 3) Apply ReLU #

Now detect the feature with the ReLU function. In Keras, you'll usually use this as the activation function in a `Conv2D` layer. What is the *backend function* in TensorFlow that does the same thing?

```python
# YOUR CODE HERE: Give the TensorFlow ReLU function (without arguments)
relu_fn = tf.nn.relu

# Check your answer
q_3.check()
```

    <IPython.core.display.Javascript object>

Correct

```python
# Lines below will give you a hint or solution code
#q_3.hint()
#q_3.solution()
```

Once you've got the solution, run this cell to detect the feature with ReLU and see the result!

The image you see below is the feature map produced by the kernel you chose. If you like, experiment with some of the other suggested kernels above, or, try to invent one that will extract a certain kind of feature.

```python
image_detect = relu_fn(image_filter)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_detect)
)
plt.axis('off')
plt.show();
```

In the tutorial, our discussion of kernels and feature maps was mainly visual. We saw the effect of `Conv2D` and `ReLU` by observing how they transformed some example images.

But the operations in a convolutional network (like in all neural networks) are usually defined through mathematical functions, through a computation on numbers. In the next exercise, we'll take a moment to explore this point of view.

Let's start by defining a simple array to act as an image, and another array to act as the kernel. Run the following cell to see these arrays.

```python
# Sympy is a python library for symbolic mathematics. It has a nice
# pretty printer for matrices, which is all we'll use it for.
import sympy
sympy.init_printing()
from IPython.display import display

image = np.array([
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 1, 1, 1],
    [0, 1, 0, 0, 0, 0],
])

kernel = np.array([
    [1, -1],
    [1, -1],
])

display(sympy.Matrix(image))
display(sympy.Matrix(kernel))
# Reformat for Tensorflow
image = tf.cast(image, dtype=tf.float32)
image = tf.reshape(image, [1, *image.shape, 1])
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

$\displaystyle \left[\begin{matrix}0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 1 & 1 & 1\\0 & 1 & 0 & 0 & 0 & 0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & -1\\1 & -1\end{matrix}\right]$

# 4) Observe Convolution on a Numerical Matrix #

What do you see? The image is simply a long vertical line on the left and a short horizontal line on the lower right. What about the kernel? What effect do you think it will have on this image? After you've thought about it, run the next cell for the answer.

```python
# View the solution (Run this code cell to receive credit!)
q_4.check()
```

    <IPython.core.display.Javascript object>

Correct:

In the tutorial, we talked about how the pattern of positive numbers will tell you the kind of features the kernel will extract. This kernel has a vertical column of 1's, and so we would expect it to return features of vertical lines.

Now let's try it out. Run the next cell to apply convolution and ReLU to the image and display the result.

```python
image_filter = tf.nn.conv2d(
    input=image,
    filters=kernel,
    strides=1,
    padding='VALID',
)
image_detect = tf.nn.relu(image_filter)

# The first matrix is the image after convolution, and the second is
# the image after ReLU.
display(sympy.Matrix(tf.squeeze(image_filter).numpy()))
display(sympy.Matrix(tf.squeeze(image_detect).numpy()))
```

$\displaystyle \left[\begin{matrix}-2.0 & 2.0 & 0 & 0 & 0\\-2.0 & 2.0 & 0 & 0 & 0\\-2.0 & 2.0 & 0 & 0 & 0\\-2.0 & 2.0 & -1.0 & 0 & 0\\-2.0 & 2.0 & -1.0 & 0 & 0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}0 & 2.0 & 0 & 0 & 0\\0 & 2.0 & 0 & 0 & 0\\0 & 2.0 & 0 & 0 & 0\\0 & 2.0 & 0 & 0 & 0\\0 & 2.0 & 0 & 0 & 0\end{matrix}\right]$

Is the result what you expected?

# Conclusion #

In this lesson, you learned about the first two operations a convolutional classifier uses for feature extraction: **filtering** an image with a **convolution** and **detecting** the feature with the **rectified linear unit**.

# Keep Going #

Move on to [**Lesson 3**](https://www.kaggle.com/ryanholbrook/maximum-pooling) to learn the final operation: **condensing** the feature map with **maximum pooling**!

---

*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/computer-vision/discussion) to chat with other learners.*