```
!pip install coremltools
```

    Collecting coremltools
      Downloading https://files.pythonhosted.org/packages/32/b0/14c37edf39a9b32c2c9c7aa3e27ece4ef4f5b2dd2c950102661a106520f1/coremltools-4.1-cp37-none-manylinux1_x86_64.whl (3.4MB)
         |████████████████████████████████| 3.4MB 7.9MB/s
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from coremltools) (1.15.0)
    Collecting attr
      Downloading https://files.pythonhosted.org/packages/de/be/ddc7f84d4e087144472a38a373d3e319f51a6faf6e5fc1ae897173675f21/attr-0.3.1.tar.gz
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from coremltools) (20.9)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from coremltools) (4.41.1)
    Requirement already satisfied: attrs in /usr/local/lib/python3.7/dist-packages (from coremltools) (20.3.0)
    Requirement already satisfied: numpy<1.20,>=1.14.5 in /usr/local/lib/python3.7/dist-packages (from coremltools) (1.19.5)
    Requirement already satisfied: protobuf>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from coremltools) (3.12.4)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from coremltools) (1.4.1)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from coremltools) (1.1.1)
    Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->coremltools) (2.4.7)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf>=3.1.0->coremltools) (53.0.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->coremltools) (1.2.1)
    Building wheels for collected packages: attr
      Building wheel for attr (setup.py) ... done
      Created wheel for attr: filename=attr-0.3.1-cp37-none-any.whl size=2458 sha256=47dd1b2f754ba622fc5fae7903f2dd448b0471cf02a309da3b6db064b3032c1c
      Stored in directory: /root/.cache/pip/wheels/f0/96/9b/1f8892a707d17095b5a6eab0275da9d39e68e03a26aee2e726
    Successfully built attr
    Installing collected packages: attr, coremltools
    Successfully installed attr-0.3.1 coremltools-4.1

```
google_drive_proj_dir = '/content/drive/MyDrive/Vienna University of Technology/mnist-sample'
```

```
import coremltools as ct
import matplotlib.cm as cm
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import os
import tensorflow as tf

from PIL import Image
from tensorflow.keras import backend as K
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Conv2D, Dense, Dropout, Flatten, Input, MaxPooling2D
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical
```

    WARNING:root:scikit-learn version 0.22.2.post1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
    WARNING:root:TensorFlow version 2.4.1 detected. Last version known to be fully compatible is 2.3.1 .
    WARNING:root:Keras version 2.4.3 detected. Last version known to be fully compatible of Keras is 2.2.4 .

```
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
```

    Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
    11493376/11490434 [==============================] - 0s 0us/step

```
print(X_train.shape)
print(Y_train.shape)

print(X_test.shape)
print(Y_test.shape)
```

    (60000, 28, 28)
    (60000,)
    (10000, 28, 28)
    (10000,)

```
X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)
X_train = X_train.astype('float32')/255
Y_train = to_categorical(Y_train, 10)

X_test = X_test.reshape(X_test.shape[0], 28, 28, 1)
X_test = X_test.astype('float32')/255
Y_test = to_categorical(Y_test, 10)
```

```
fig = plt.figure(figsize=(20, 20))
for i in range(6):
    ax = fig.add_subplot(1, 6, i+1,
                         xticks=[],
                         yticks=[])
    ax.imshow(np.squeeze(X_train[i], axis=-1), cmap='gray')
    ax.set_title(str(Y_train[i]))
```

```
model = Sequential()

model.add(Input(shape=(28, 28, 1)))
model.add(Conv2D(filters=32, kernel_size=(3, 3), padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
# model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
# model.add(Dropout(0.4))
model.add(Dense(10, activation='softmax'))
```

```
model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv2d (Conv2D)              (None, 28, 28, 32)        320
    _________________________________________________________________
    max_pooling2d (MaxPooling2D) (None, 14, 14, 32)        0
    _________________________________________________________________
    conv2d_1 (Conv2D)            (None, 14, 14, 32)        4128
    _________________________________________________________________
    max_pooling2d_1 (MaxPooling2 (None, 7, 7, 32)          0
    _________________________________________________________________
    conv2d_2 (Conv2D)            (None, 7, 7, 32)          4128
    _________________________________________________________________
    max_pooling2d_2 (MaxPooling2 (None, 3, 3, 32)          0
    _________________________________________________________________
    flatten (Flatten)            (None, 288)               0
    _________________________________________________________________
    dense (Dense)                (None, 512)               147968
    _________________________________________________________________
    dense_1 (Dense)              (None, 10)                5130
    =================================================================
    Total params: 161,674
    Trainable params: 161,674
    Non-trainable params: 0
    _________________________________________________________________

```
model.compile(loss='categorical_crossentropy',
              optimizer=Adam(),
              metrics=['accuracy'])

model.fit(X_train, Y_train, batch_size=512, epochs=16, verbose=1)

score = model.evaluate(X_test, Y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

    Epoch 1/16
    118/118 [==============================] - 8s 8ms/step - loss: 1.1657 - accuracy: 0.6685
    Epoch 2/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.1270 - accuracy: 0.9619
    Epoch 3/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0829 - accuracy: 0.9740
    Epoch 4/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0576 - accuracy: 0.9830
    Epoch 5/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0509 - accuracy: 0.9836
    Epoch 6/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0418 - accuracy: 0.9869
    Epoch 7/16
    118/118 [==============================] - 1s 8ms/step - loss: 0.0352 - accuracy: 0.9893
    Epoch 8/16
    118/118 [==============================] - 1s 8ms/step - loss: 0.0309 - accuracy: 0.9902
    Epoch 9/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0273 - accuracy: 0.9917
    Epoch 10/16
    118/118 [==============================] - 1s 8ms/step - loss: 0.0281 - accuracy: 0.9901
    Epoch 11/16
    118/118 [==============================] - 1s 8ms/step - loss: 0.0224 - accuracy: 0.9929
    Epoch 12/16
    118/118 [==============================] - 1s 8ms/step - loss: 0.0208 - accuracy: 0.9932
    Epoch 13/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0161 - accuracy: 0.9949
    Epoch 14/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0173 - accuracy: 0.9939
    Epoch 15/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0148 - accuracy: 0.9955
    Epoch 16/16
    118/118 [==============================] - 1s 7ms/step - loss: 0.0155 - accuracy: 0.9951
    Test loss: 0.02640668675303459
    Test accuracy: 0.991599977016449

```
# Define the input type as image
image_input = ct.ImageType(shape=(1, 28, 28, 1), scale=1/255)

# set class labels
classifier_config = ct.ClassifierConfig(
    class_labels=['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'])

# Convert the model using the Unified Conversion API
model = ct.convert(
    model, inputs=[image_input], classifier_config=classifier_config)
```

    Running TensorFlow Graph Passes: 100%|██████████| 5/5 [00:00<00:00, 27.75 passes/s]
    Converting Frontend ==> MIL Ops: 100%|██████████| 32/32 [00:00<00:00, 609.45 ops/s]
    Running MIL optimization passes: 100%|██████████| 18/18 [00:00<00:00, 355.31 passes/s]
    Translating MIL ==> MLModel Ops: 100%|██████████| 55/55 [00:00<00:00, 1581.42 ops/s]

```
# Save model
model.save(os.path.join(google_drive_proj_dir, 'MNISTSample.mlmodel'))
```


$
\begin{align}
a_1&=b_1+c_1 \tag{1}\\
a_2&=b_2+c_2+d_2 \tag{2}\\
a_3&=b_3+c_3 \tag{3}
\end{align}
$

[Euler](https://krasjet.github.io/quaternion/bonus_gimbal_lock.pdf)

[Quaternion](https://krasjet.github.io/quaternion/bonus_gimbal_lock.pdf)

[Source](https://github.com/Krasjet/quaternion)

```python

```

```python

```


```python
from sympy import init_printing; init_printing();
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
```

```python
from silkpy.geometry_map.coord_transform import CoordTransform
from silkpy.geometry_map.coord_transform.predefined import Cylindrical2Cartesian, Cartesian2Cylindrical

Cartesian2Cartesian = Cartesian2Cylindrical() | Cylindrical2Cartesian()
Cylindrical2Cylindrical = Cylindrical2Cartesian() | Cartesian2Cylindrical()

```

```python
Cartesian2Cartesian.expr()
Cartesian2Cartesian.sym()
```

```python
Cylindrical2Cylindrical.expr().simplify()
Cylindrical2Cylindrical.sym()
```

```python

```

```python

```


#  Neuromatch Academy: Week 1, Day 1, Tutorial 3
# Model Types: "Why" models
__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom

We would like to acknowledge [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x) for sharing their data, a subset of which is used here.

___
# Tutorial Objectives
This is tutorial 3 of a 3-part series on different flavors of models used to understand neural data. In parts 1 and 2 we explored mechanisms that would produce the data. In this tutorial we will explore models and techniques that can potentially explain *why* the spiking data we have observed is produced the way it is.

To understand why different spiking behaviors may be beneficial, we will learn about the concept of entropy. Specifically, we will:

- Write code to compute formula for entropy, a measure of information
- Compute the entropy of a number of toy distributions
- Compute the entropy of spiking activity from the Steinmetz dataset

```python
#@title Video 1: “Why” models
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV16t4y1Q7DR', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video

```

    Video available at https://www.bilibili.com/video/BV16t4y1Q7DR

# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")

```

```python
#@title Helper Functions

def plot_pmf(pmf,isi_range):
  """Plot the probability mass function."""
  ymax = max(0.2, 1.05 * np.max(pmf))
  pmf_ = np.insert(pmf, 0, pmf[0])
  plt.plot(bins, pmf_, drawstyle="steps")
  plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
  plt.title(f"Neuron {neuron_idx}")
  plt.xlabel("Inter-spike interval (s)")
  plt.ylabel("Probability mass")
  plt.xlim(isi_range);
  plt.ylim([0, ymax])
```

```python
#@title Download Data
import io
import requests
r = requests.get('https://osf.io/sy5xt/download')
if r.status_code != 200:
  print('Could not download data')
else:
  steinmetz_spikes = np.load(io.BytesIO(r.content), allow_pickle=True)['spike_times']
```

# Section 1: Optimization and Information

Neurons can only fire so often in a fixed period of time, as the act of emitting a spike consumes energy that is depleted and must eventually be replenished. To communicate effectively for downstream computation, the neuron would need to make good use of its limited spiking capability. This becomes an optimization problem:

What is the optimal way for a neuron to fire in order to maximize its ability to communicate information?

In order to explore this question, we first need to have a quantifiable measure for information. Shannon introduced the concept of entropy to do just that, and defined it as

\begin{align}
  H_b(X) &= -\sum_{x\in X} p(x) \log_b p(x)
\end{align}

where $H$ is entropy measured in units of base $b$ and $p(x)$ is the probability of observing the event $x$ from the set of all possible events in $X$. See the Appendix for a more detailed look at how this equation was derived.

The most common base of measuring entropy is $b=2$, so we often talk about *bits* of information, though other bases are used as well e.g. when $b=e$ we call the units *nats*.

First, let's explore how entropy changes between some simple discrete probability distributions. In the rest of this tutorial we will refer to these as probability mass functions (PMF), where $p(x_i)$ equals the $i^{th}$ value in an array, and mass refers to how much of the distribution is contained at that value.

For our first PMF, we will choose one where all of the probability mass is located in the middle of the distribution.

```python
n_bins = 50  # number of points supporting the distribution
x_range = (0, 1)  # will be subdivided evenly into bins corresponding to points

bins = np.linspace(*x_range, n_bins + 1)  # bin edges

pmf = np.zeros(n_bins)
pmf[len(pmf) // 2] = 1.0  # middle point has all the mass

# Since we already have a PMF, rather than un-binned samples, `plt.hist` is not
# suitable. Instead, we directly plot the PMF as a step function to visualize
# the histogram:
pmf_ = np.insert(pmf, 0, pmf[0])  # this is necessary to align plot steps with bin edges
plt.plot(bins, pmf_, drawstyle="steps")
# `fill_between` provides area shading
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

If we were to draw a sample from this distribution, we know exactly what we would get every time. Distributions where all the mass is concentrated on a single event are known as *deterministic*.

How much entropy is contained in a deterministic distribution?

## Exercise 1: Computing Entropy

Your first exercise is to implement a method that computes the entropy of a discrete probability distribution, given its mass function. Remember that we are interested in entropy in units of _bits_, so be sure to use the correct log function.

Recall that $\log(0)$ is undefined. When evaluated at $0$, NumPy log functions (such as `np.log2`) return `np.nan` ("Not a Number"). By convention, these undefined terms— which correspond to points in the distribution with zero mass—are excluded from the sum that computes the entropy.

```python
def entropy(pmf):
  """Given a discrete distribution, return the Shannon entropy in bits.

  This is a measure of information in the distribution. For a totally
  deterministic distribution, where samples are always found in the same bin,
  then samples from the distribution give no more information and the entropy
  is 0.

  For now this assumes `pmf` arrives as a well-formed distribution (that is,
  `np.sum(pmf)==1` and `not np.any(pmf < 0)`)

  Args:
    pmf (np.ndarray): The probability mass function for a discrete distribution
      represented as an array of probabilities.
  Returns:
    h (number): The entropy of the distribution in `pmf`.

  """
  ############################################################################
  # Exercise for students: compute the entropy of the provided PMF
  #   1. Exclude the points in the distribution with no mass (where `pmf==0`).
  #      Hint: this is equivalent to including only the points with `pmf>0`.
  #   2. Implement the equation for Shannon entropy (in bits).
  #  When ready to test, comment or remove the next line
  raise NotImplementedError("Excercise: implement the equation for entropy")
  ############################################################################

  # reduce to non-zero entries to avoid an error from log2(0)
  pmf = ...

  # implement the equation for Shannon entropy (in bits)
  h = ...

  # return the absolute value (avoids getting a -0 result)
  return np.abs(h)

# Uncomment to test your entropy function
# print(f"{entropy(pmf):.2f} bits")
```

```python
# to_remove solution
def entropy(pmf):
  """Given a discrete distribution, return the Shannon entropy in bits.

  This is a measure of information in the distribution. For a totally
  deterministic distribution, where samples are always found in the same bin,
  then samples from the distribution give no more information and the entropy
  is 0.

  For now this assumes `pmf` arrives as a well-formed distribution (that is,
  `np.sum(pmf)==1` and `not np.any(pmf < 0)`)

  Args:
    pmf (np.ndarray): The probability mass function for a discrete distribution
      represented as an array of probabilities.
  Returns:
    h (number): The entropy of the distribution in `pmf`.
  """
  # reduce to non-zero entries to avoid an error from log2(0)
  pmf = pmf[pmf > 0]
  # implement the equation for shannon entropy (in bits)
  h = -np.sum(pmf * np.log2(pmf))
  # return the absolute value (avoids getting a -0 result)
  return np.abs(h)

print(f"{entropy(pmf):.2f} bits")
```

    0.00 bits

We expect zero surprise from a deterministic distribution. If we had done this calculation by hand, it would simply be

$-1\log_2 1 = -0=0$

Note that changing the location of the peak (i.e. the point and bin on which all the mass rests) doesn't alter the entropy. The entropy is about how predictable a sample is with respect to a distribution. A single peak is deterministic regardless of which point it sits on.

```python
pmf = np.zeros(n_bins)
pmf[2] = 1.0  # arbitrary point has all the mass

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

What about a distribution with mass split equally between two points?

```python
pmf = np.zeros(n_bins)
pmf[len(pmf) // 3] = 0.5
pmf[2 * len(pmf) // 3] = 0.5

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

Here, the entropy calculation is

$-(0.5 \log_2 0.5 + 0.5\log_2 0.5)=1$

There is 1 bit of entropy. This means that before we take a random sample, there is 1 bit of uncertainty about which point in the distribution the sample will fall on: it will either be the first peak or the second one.

Likewise, if we make one of the peaks taller (i.e. its point holds more of the probability mass) and the other one shorter, the entropy will decrease because of the increased certainty that the sample will fall on one point and not the other:

$-(0.2 \log_2 0.2 + 0.8\log_2 0.8)\approx 0.72$

Try changing the definition of the number and weighting of peaks, and see how the entropy varies.

If we split the probability mass among even more points, the entropy continues to increase. Let's derive the general form for $N$ points of equal mass, where $p_i=p=1/N$:

\begin{align}
 -\sum_i p_i \log_b p_i&= -\sum_i^N \frac{1}{N} \log_b \frac{1}{N}\\
                       &= -\log_b \frac{1}{N} \\
                       &= \log_b N
\end{align}
$$$$

If we have $N$ discrete points, the _uniform distribution_ (where all points have equal mass) is the distribution with the highest entropy: $\log_b N$. This upper bound on entropy is useful when considering binning strategies, as any estimate of entropy over $N$ discrete points (or bins) must be in the interval $[0, \log_b N]$.

```python
pmf = np.ones(n_bins) / n_bins  # [1/N] * N

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range);
plt.ylim(0, 1);
```

Here, there are 50 points and the entropy of the uniform distribution is $\log_2 50\approx 5.64$. If we construct _any_ discrete distribution $X$ over 50 points (or bins) and calculate an entropy of $H_2(X)>\log_2 50$, something must be wrong with our implementation of the discrete entropy computation.

# Section 2: Information, neurons, and spikes

```python
#@title Video 2: Entropy of different distributions
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1df4y1976g', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV1df4y1976g

Recall the discussion of spike times and inter-spike intervals (ISIs) from Tutorial 1. What does the information content (or distributional entropy) of these measures say about our theory of nervous systems?

We'll consider three hypothetical neurons that all have the same mean ISI, but with different distributions:

1. Deterministic
2. Uniform
3. Exponential

Fixing the mean of the ISI distribution is equivalent to fixing its inverse: the neuron's mean firing rate. If a neuron has a fixed energy budget and each of its spikes has the same energy cost, then by fixing the mean firing rate, we are normalizing for energy expenditure. This provides a basis for comparing the entropy of different ISI distributions. In other words: if our neuron has a fixed budget, what ISI distribution should it express (all else being equal) to maximize the information content of its outputs?

Let's construct our three distributions and see how their entropies differ.

```python
n_bins = 50
mean_isi = 0.025
isi_range = (0, 0.25)

bins = np.linspace(*isi_range, n_bins + 1)
mean_idx = np.searchsorted(bins, mean_isi)

# 1. all mass concentrated on the ISI mean
pmf_single = np.zeros(n_bins)
pmf_single[mean_idx] = 1.0

# 2. mass uniformly distributed about the ISI mean
pmf_uniform = np.zeros(n_bins)
pmf_uniform[0:2*mean_idx] = 1 / (2 * mean_idx)

# 3. mass exponentially distributed about the ISI mean
pmf_exp = stats.expon.pdf(bins[1:], scale=mean_isi)
pmf_exp /= np.sum(pmf_exp)
```

```python
#@title
#@markdown Run this cell to plot the three PMFs
fig, axes = plt.subplots(ncols=3, figsize=(18, 5))

dists =  [# (subplot title, pmf, ylim)
          ("(1) Deterministic", pmf_single, (0, 1.05)),
          ("(1) Uniform", pmf_uniform, (0, 1.05)),
          ("(1) Exponential", pmf_exp, (0, 1.05))]

for ax, (label, pmf_, ylim) in zip(axes, dists):
  pmf_ = np.insert(pmf_, 0, pmf_[0])
  ax.plot(bins, pmf_, drawstyle="steps")
  ax.fill_between(bins, pmf_, step="pre", alpha=0.4)
  ax.set_title(label)
  ax.set_xlabel("Inter-spike interval (s)")
  ax.set_ylabel("Probability mass")
  ax.set_xlim(isi_range);
  ax.set_ylim(ylim);
```

```python
print(
  f"Deterministic: {entropy(pmf_single):.2f} bits",
  f"Uniform: {entropy(pmf_uniform):.2f} bits",
  f"Exponential: {entropy(pmf_exp):.2f} bits",
  sep="\n",
)
```

    Deterministic: 0.00 bits
    Uniform: 3.32 bits
    Exponential: 3.77 bits

```python
#@title Video 3: Probabilities from histogram
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1Jk4y1B7cz', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV1Jk4y1B7cz

In the previous example we created the PMFs by hand to illustrate idealized scenarios. How would we compute them from data recorded from actual neurons?

One way is to convert the ISI histograms we've previously computed into discrete probability distributions using the following equation:

\begin{align}
p_i = \frac{n_i}{\sum\nolimits_{i}n_i}
\end{align}

where $p_i$ is the probability of an ISI falling within a particular interval $i$ and $n_i$ is the count of how many ISIs were observed in that interval.

### Exercise 2: Probabilty Mass Function

Your second exercise is to implement a method that will produce a probability mass function from an array of ISI bin counts.

To verify your solution, we will compute the probability distribution of ISIs from real neural data taken from the Steinmetz dataset.

```python
neuron_idx = 283

isi = np.diff(steinmetz_spikes[neuron_idx])
bins = np.linspace(*isi_range, n_bins + 1)
counts, _ = np.histogram(isi, bins)
```

```python
def pmf_from_counts(counts):
  """Given counts, normalize by the total to estimate probabilities."""
  ###########################################################################
  # Exercise: Compute the PMF. Remove the next line to test your function
  raise NotImplementedError("Student excercise: compute the PMF from ISI counts")
  ###########################################################################

  pmf = ...

  return pmf

# Uncomment when ready to test your function
# pmf = pmf_from_counts(counts)
# plot_pmf(pmf,isi_range)
```

```python
# to_remove solution
def pmf_from_counts(counts):
  """Given counts, normalize by the total to estimate probabilities."""
  pmf = counts / np.sum(counts)
  return pmf

pmf = pmf_from_counts(counts)
with plt.xkcd():
  plot_pmf(pmf,isi_range)
```

# Section 3: Calculate entropy from a PMF

```python
#@title Video 4: Calculating entropy from pmf
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1vA411e7Cd', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV1vA411e7Cd

Now that we have the probability distribution for the actual neuron spiking activity, we can calculate its entropy.

```python
print(f"Entropy for Neuron {neuron_idx}: {entropy(pmf):.2f} bits")
```

    Entropy for Neuron 283: 3.36 bits

## Interactive Demo: Entropy of neurons

We can combine the above distribution plot and entropy calculation with an interactive widget to explore how the different neurons in the dataset vary in spiking activity and relative information. Note that the mean firing rate across neurons is not fixed, so some neurons with a uniform ISI distribution may have higher entropy than neurons with a more exponential distribution.

```python
#@title
#@markdown **Run the cell** to enable the sliders.

def _pmf_from_counts(counts):
  """Given counts, normalize by the total to estimate probabilities."""
  pmf = counts / np.sum(counts)
  return pmf

def _entropy(pmf):
  """Given a discrete distribution, return the Shannon entropy in bits."""
  # remove non-zero entries to avoid an error from log2(0)
  pmf = pmf[pmf > 0]
  h = -np.sum(pmf * np.log2(pmf))
  # absolute value applied to avoid getting a -0 result
  return np.abs(h)

@widgets.interact(neuron=widgets.IntSlider(0, min=0, max=(len(steinmetz_spikes)-1)))
def steinmetz_pmf(neuron):
  """ Given a neuron from the Steinmetz data, compute its PMF and entropy """
  isi = np.diff(steinmetz_spikes[neuron])
  bins = np.linspace(*isi_range, n_bins + 1)
  counts, _ = np.histogram(isi, bins)
  pmf = _pmf_from_counts(counts)

  plot_pmf(pmf,isi_range)
  plt.title(f"Neuron {neuron}: H = {_entropy(pmf):.2f} bits")

```

---
# Summary

```python
#@title Video 5: Summary of model types
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1F5411e7ww', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV1F5411e7ww

Congratulations! You've finished your first NMA tutorial. In this 3 part tutorial series, we used different types of models to understand the spiking behavior of neurons recorded in the Steinmetz data set.

 - We used "what" models to discover that the ISI distribution of real neurons is closest to an exponential distribution
 - We used "how" models to discover that balanced excitatory and inbhitiory inputs, coupled with a leaky membrane, can give rise to neuronal spiking with exhibiting such an exponential ISI distribution
 - We used "why" models to discover that exponential ISI distributions contain the most information when the mean spiking is constrained

# Bonus

### The foundations for Entropy

In his foundational [1948 paper](https://en.wikipedia.org/wiki/A_Mathematical_Theory_of_Communication) on information theory, Claude Shannon began with three criteria for a function $H$ defining the entropy of a discrete distribution of probability masses $p_i\in p(X)$ over the points $x_i\in X$:
1. $H$ should be continuous in the $p_i$.
  - That is, $H$ should change smoothly in response to smooth changes to the mass $p_i$ on each point $x_i$.
2. If all the points have equal shares of the probability mass, $p_i=1/N$, $H$ should be a non-decreasing function of $N$.
  - That is, if $X_N$ is the support with $N$ discrete points and $p(x\in X_N)$ assigns constant mass to each point, then $H(X_1) < H(X_2) < H(X_3) < \dots$
3. $H$ should be preserved by (invariant to) the equivalent (de)composition of distributions.
  - For example (from Shannon's paper) if we have a discrete distribution over three points with masses $(\frac{1}{2},\frac{1}{3},\frac{1}{6})$, then their entropy can be represented in terms of a direct choice between the three and calculated $H(\frac{1}{2},\frac{1}{3},\frac{1}{6})$. However, it could also be represented in terms of a series of two choices:
    1. either we sample the point with mass $1/2$ or not (_not_ is the other $1/2$, whose subdivisions are not given in the first choice),
    2. if (with probability $1/2$) we _don't_ sample the first point, we sample one of the two remaining points, masses $1/3$ and $1/6$.

    Thus in this case we require that $H(\frac{1}{2},\frac{1}{3},\frac{1}{6})=H(\frac{1}{2},\frac{1}{2}) + \frac{1}{2}H(\frac{1}{3}, \frac{1}{6})$

There is a unique function (up to a linear scaling factor) which satisfies these 3 requirements:

\begin{align}
  H_b(X) &= -\sum_{x\in X} p(x) \log_b p(x)
\end{align}

Where the base of the logarithm $b>1$ controls the units of entropy. The two most common cases are $b=2$ for units of _bits_, and $b=e$ for _nats_.

We can view this function as the expectation of the self-information over a distribution:

$$H_b(X) = \mathbb{E}_{x\in X} \left[I_b(x)\right]$$

$$I_b(x)=-\log_b p(x)$$

Self-information is just the negative logarithm of probability, and is a measure of how surprising an event sampled from the distribution would be. Events with $p(x)=1$ are certain to occur, and their self-information is zero (as is the entropy of the distribution they compose) meaning they are totally unsurprising. The smaller the probability of an event, the higher its self-information, and the more surprising the event would be to observe.


```python
from commpy.sequences  import pnsequence as bits
from commpy.modulation import QAMModem
from commpy.utilities  import upsample
from commpy.utilities  import signal_power
from commpy.utilities  import bitarray2dec, dec2bitarray
from commpy.filters    import rrcosfilter
from commpy.channels   import awgn
from scipy.signal      import lfilter as firFilter
from scipy.signal      import decimate
from scipy.signal      import dlti
from sympy.combinatorics.graycode import GrayCode
from numpy.matlib import repmat
from numpy.random import randint
from numpy.random import randn
import matplotlib.pyplot as plt
import numpy as np
from scipy.interpolate import interp1d
from scipy import interpolate
import itertools
import pandas as pd
import math
import scipy
from scipy import signal
from scipy.special import erfc
from scipy import special as sp

# import functions
# %run -i '/home/weverton_mm/Documentos/TCC_github/TCC-OFDM-system/functions.py'

```

```python
def SP(bits):
    return bits.reshape(len(dataCarriers), mu)

def mapping_tab(mu):
    if mu == 2:
        mapping_table = {
            (0,0) : -1 - 1j,     (0,1) : -1 + 1j,      (1,1) : 1 + 1j,     (1,0) : 1 - 1j
        }
    elif mu == 4:
        mapping_table = {
            (0,0,0,0) : -3-3j,    (0,0,0,1) : -3-1j,    (0,0,1,0) : -3+3j,    (0,0,1,1) : -3+1j,    (0,1,0,0) : -1-3j,
            (0,1,0,1) : -1-1j,    (0,1,1,0) : -1+3j,    (0,1,1,1) : -1+1j,    (1,0,0,0) :  3-3j,    (1,0,0,1) :  3-1j,
            (1,0,1,0) :  3+3j,    (1,0,1,1) :  3+1j,    (1,1,0,0) :  1-3j,    (1,1,0,1) :  1-1j,    (1,1,1,0) :  1+3j,
            (1,1,1,1) :  1+1j
        }
    elif mu == 6:
        mapping_table = {
            (0,0,0,0,0,0) : 7 + 7j,    (0,0,0,0,0,1) : 7 + 5j,    (0,0,0,0,1,0) : 5 + 7j,    (0,0,0,0,1,1) : 5 + 5j,
            (0,0,0,1,0,0) : 7 + 1j,    (0,0,0,1,0,1) : 7 + 3j,    (0,0,0,1,1,0) : 5 + 1j,    (0,0,0,1,1,1) : 5 + 3j,
            (0,0,1,0,0,0) : 1 + 7j,    (0,0,1,0,0,1) : 1 + 5j,    (0,0,1,0,1,0) : 3 + 7j,    (0,0,1,0,1,1) : 3 + 5j,
            (0,0,1,1,0,0) : 1 + 1j,    (0,0,1,1,0,1) : 1 + 3j,    (0,0,1,1,1,0) : 3 + 1j,    (0,0,1,1,1,1) : 3 + 3j,
            (0,1,0,0,0,0) : 7 - 7j,    (0,1,0,0,0,1) : 7 - 5j,    (0,1,0,0,1,0) : 5 - 7j,    (0,1,0,0,1,1) : 5 - 5j,
            (0,1,0,1,0,0) : 7 - 1j,    (0,1,0,1,0,1) : 7 - 3j,    (0,1,0,1,1,0) : 5 - 1j,    (0,1,0,1,1,1) : 5 - 3j,
            (0,1,1,0,0,0) : 1 - 7j,    (0,1,1,0,0,1) : 1 - 5j,    (0,1,1,0,1,0) : 3 - 7j,    (0,1,1,0,1,1) : 3 - 5j,
            (0,1,1,1,0,0) : 1 - 1j,    (0,1,1,1,0,1) : 1 - 3j,    (0,1,1,1,1,0) : 3 - 1j,    (0,1,1,1,1,1) : 3 - 3j,
            (1,0,0,0,0,0) : -7 + 7j,    (1,0,0,0,0,1) : -7 + 5j,    (1,0,0,0,1,0) : -5 + 7j,    (1,0,0,0,1,1) : -5 + 5j,
            (1,0,0,1,0,0) : -7 + 1j,    (1,0,0,1,0,1) : -7 + 3j,    (1,0,0,1,1,0) : -5 + 1j,    (1,0,0,1,1,1) : -5 + 3j,
            (1,0,1,0,0,0) : -1 + 7j,    (1,0,1,0,0,1) : -1 + 5j,    (1,0,1,0,1,0) : -3 + 7j,    (1,0,1,0,1,1) : -3 + 5j,
            (1,0,1,1,0,0) : -1 + 1j,    (1,0,1,1,0,1) : -1 + 3j,    (1,0,1,1,1,0) : -3 + 1j,    (1,0,1,1,1,1) : -3 + 3j,
            (1,1,0,0,0,0) : -7 - 7j,    (1,1,0,0,0,1) : -7 - 5j,    (1,1,0,0,1,0) : -5 - 7j,    (1,1,0,0,1,1) : -5 - 5j,
            (1,1,0,1,0,0) : -7 - 1j,    (1,1,0,1,0,1) : -7 - 3j,    (1,1,0,1,1,0) : -5 - 1j,    (1,1,0,1,1,1) : -5 - 3j,
            (1,1,1,0,0,0) : -1 - 7j,    (1,1,1,0,0,1) : -1 - 5j,    (1,1,1,0,1,0) : -3 - 7j,    (1,1,1,0,1,1) : -3 - 5j,
            (1,1,1,1,0,0) : -1 - 1j,    (1,1,1,1,0,1) : -1 - 3j,    (1,1,1,1,1,0) : -3 - 1j,    (1,1,1,1,1,1) : -3 - 3j,

        }

    return mapping_table

def Mapping(bits):
    return np.array([mapping_table[tuple(b)] for b in bits])

def OFDM_symbol(QAM_payload):
    symbol = np.zeros(K, dtype = complex) # todas as K subportadoras
    symbol[pilotCarriers] = pilotValue  # aloca as subportadoras piloto
    symbol[dataCarriers] = QAM_payload  # aloca as subportadoras de dados
    return symbol

def IDFT(OFDM_data):
    return np.fft.ifft(np.sqrt(len(OFDM_data))*OFDM_data)

def addCP(OFDM_time, CP):
    cp = OFDM_time[-CP:]               # pega as últimas amostras do CP ...
    return np.hstack([cp, OFDM_time])  # ... e as adiciona no início dos dados

def channel(signal, type_channel, snr, M):
#     WIRELESS
    if type_channel == 'wireless':
        convolved = np.convolve(signal, channelResponse)
        signal_power = np.mean(abs(convolved ** 2))
        sigma2 = signal_power * 10 **(-snr/10)  # potência do ruído

#         print ("RX Potência do sinal: %.8f. Potência do ruído: %.8f" % (signal_power, sigma2))

        noise = np.sqrt(sigma2/2) * (np.random.randn(*convolved.shape) + 1j * np.random.randn(*convolved.shape))

        return convolved + noise

    elif type_channel == 'awgn':

        snrdB    = snr + 10*np.log10(np.log2(M))
        avg_energy = sum(abs(signal)**2) / len(signal)

        N0       = avg_energy/(10**(snrdB/10))
        σ        = np.sqrt(N0*SpS)

        noise    = np.random.normal(0, σ, len(signal)) + 1j*np.random.normal(0, σ, len(signal))
        noise    = (1/np.sqrt(np.pi)) * noise
        return signal + noise

def removeCP(signal):
    return signal[CP:len(signal)]

def DFT(OFDM_RX):
    return np.fft.fft(np.sqrt(len(OFDM_RX))*OFDM_RX)

def channelEstimate(OFDM_demod):
    pilots = OFDM_demod[pilotCarriers]  #  extrair os valores piloto do sinal RX
    Hest_at_pilots = pilots / pilotValue # divide pelo valor do piloto transmitido; pilotValue = 3 + 3j

    # Interpolação entre as portadoras piloto para realizar a estimação do canal nas portadoras de dados
    Hest_abs = interp1d(pilotCarriers, abs(Hest_at_pilots), kind = 'linear')(allCarriers)
    Hest_phase = interp1d(pilotCarriers, np.angle(Hest_at_pilots), kind = 'linear')(allCarriers)
    Hest = Hest_abs * np.exp(1j * Hest_phase)

#     plt.plot(allCarriers, abs(H_exact), label = 'Canal correto')
#     plt.stem(pilotCarriers, abs(Hest_at_pilots), label = 'Estimativas piloto')
#     plt.plot(allCarriers, abs(Hest), label = 'Canal estimado via interpolação')
#     plt.grid(True); plt.xlabel('índice da portadora'); plt.ylabel('$|H(f)|$'); plt.legend(fontsize=10)
#     plt.ylim(0,2)

    return Hest

def equalize(OFDM_demod, Hest):
    return OFDM_demod / Hest

def get_payload(equalized): # apenas os símbolos OFDM de portadoras de dados
    return equalized[dataCarriers]

def Demapping(QAM_est):
    # pontos da constelação
    constellation = np.array([x for x in demapping_table.keys()])

    # calcula a distância entre cada ponto recebido
    dists = abs(QAM_est.reshape((-1,1)) - constellation.reshape((1,-1)))

    # para cada elemento no QAM, escolher o índice na constelação
    # que pertence ao ponto de constelação mais próximo
    const_index = dists.argmin(axis=1) # argmin(axis = 1) porque vai pegar o índice do menor valor em cada linha

    # obter de volta o ponto real da constelação
    hardDecision = constellation[const_index]

    # converte o símbolo da constelação em sequência de bits
    return np.vstack([demapping_table[C] for C in hardDecision]), hardDecision

def PS(bits):
    return bits.reshape((-1,))

def Qfunc(x):
    return 0.5-0.5*sp.erf(x/np.sqrt(2))

def theoryBER_MQAM(M, EbN0):

    EbN0lin = 10**(EbN0/10)
    L  = np.sqrt(M)
    Pb = 2*(1-1/L)/np.log2(L)*Qfunc(np.sqrt(3*np.log2(L)/(L**2-1)*(2*EbN0lin)))

    return Pb
```

```python
n = 14
mu = 6
M = 2**mu
SpS = 1
payloadBits_per_OFDM = mu * 2**n
K = payloadBits_per_OFDM // 2**(n-5)
bits_per_ofdm = payloadBits_per_OFDM // K
CP = K // 4
P = 8 # number of pilot carriers per OFDM block
pilotValue = 7+7j # The known value each pilot transmits

allCarriers = np.arange(K)  # indices of all subcarriers ([0, 1, ... K-1])
pilotCarriers = allCarriers[::K//P] # Pilots is every (K/P)th carrier.
        # For convenience of channel estimation, let's make the last carriers also be a pilot
pilotCarriers = np.hstack([pilotCarriers, np.array([allCarriers[-1]])])
P = P+1
dataCarriers = np.delete(allCarriers, pilotCarriers) # data carriers are all remaining carriers

mapping_table = mapping_tab(mu)
demapping_table = {v : k for k, v in mapping_table.items()}

channelResponse = np.array([1, 0, 0.3+0.3j])  # the impulse response of the wireless channel
H_exact = np.fft.fft(channelResponse, K)

EbN0dB_ = np.arange(1,20.5,0.5)+5
BER     = np.zeros(EbN0dB_.shape)

for subcarrier in range(K):
    for indSNR in range(EbN0dB_.size):

        EbN0dB = EbN0dB_[indSNR]

        bits = np.random.binomial(n = 1, p = 0.5, size = (len(dataCarriers) * mu, ))
        bits_SP = SP(bits)

        QAM = Mapping(bits_SP)
        OFDM_data = OFDM_symbol(QAM)
#         print("OFDM_data DOWN", len(OFDM_data))

#         symbolsUp = upsample(OFDM_data, SpS) # Aumenta a taxa de amostragem do sinal para SpS amostras/símbolo
#         print("OFDM_data UP", len(symbolsUp))

        OFDM_time = IDFT(OFDM_data)

        OFDM_withCP = addCP(OFDM_time, CP)

        OFDM_TX = OFDM_withCP

        OFDM_RX = channel(OFDM_TX, 'awgn', EbN0dB, M = 64)
        OFDM_RX_noCP = removeCP(OFDM_RX)
        OFDM_demod = DFT(OFDM_RX_noCP)

#         plt.figure(2)
#         plt.plot(OFDM_demod.real, OFDM_demod.imag,'.',color = 'red');
#         plt.xlabel('real');
#         plt.ylabel('imag');

        Hest = channelEstimate(OFDM_demod)
        equalized_Hest = equalize(OFDM_demod, Hest)
        QAM_est = get_payload(equalized_Hest)

        plt.figure(3)
        plt.plot(QAM_est.real, QAM_est.imag,'.',color = 'red');
        plt.xlabel('real');
        plt.ylabel('imag');

        PS_est, hardDecision = Demapping(QAM_est)

        bits_est = PS(PS_est)

        BER[indSNR] = BER[indSNR] + (np.sum(abs(bits-bits_est))/len(bits))/K

```

```python
BERtheory = theoryBER_MQAM(M, EbN0dB_)

plt.plot(EbN0dB_,np.log10(BER),'o');
plt.plot(EbN0dB_, np.log10(BERtheory))
plt.legend(['Monte Carlo sim', "Teoria "+ str(M) + "-QAM"])
plt.xlabel('EbN0 (dB)');
plt.ylabel('BER');
plt.ylim(-8)
plt.grid()
```

```python

```


# Python Programming

**Chapter 5 : Computing with Python**

Python is a fun language to learn, and really easy to pick up even if you are new to programming. In fact, quite often, Python is easier to pick up if you do not have any programming experience whatsoever. Python is high level programming language, targeted at students and professionals from diverse backgrounds.

In this chapter, we will cover
- File Handling
- Numeric Computation
- Image Handling
- Symbolic Computation

**License Declaration** : Following the lead from the inspirations for this material, and the *spirit* of Python education and development, all modules of this work are licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/.

---

## File Handling

Handling files is a pre-requisite for a number of Python computing applications. Hence we learn that first.

```python
# Handling Files in Python
file = open("files/poemRobertFrost.txt", "r", encoding = "utf8")
print("Name of File : ", file.name)
print("File Closed? : ", file.closed)
print("Opening Mode : ", file.mode)
file.close()  # closing a file is crucial
```

```python
# Reading Lines from a File
with open("files/poemRobertFrost.txt", "r", encoding = "utf8") as file:
    line = file.readline()
    line_count = 1
    while line:
        print("Line {0:2d} \t {1:s}".format(line_count, line.strip()))
        line = file.readline()
        line_count += 1
```

```python
# Reading Lines from a File
with open("files/poemRobertFrost.txt", "r", encoding = "utf8") as file:
    for line_count, line in enumerate(file):
        print("Line {0:2d} \t {1:s}".format(line_count, line.strip()))
```

```python
# Reading Words from a File
with open("files/poemRobertFrost.txt", "r", encoding = "utf8") as file:
    for line_count, line in enumerate(file):
        print("Line {0:2d} \t".format(line_count), end = "")
        for word in line.split(" "):
            print(word.strip(), end = "\t")
        print()
```

#### Quick Tasks

- Create a Frequency Table or Dictionary `{Words : Count}` for all distinct words in the file `poemRobertFrost.txt`.

```python

```

```python

```

---

## Numeric Computation

`NumPy` is one of the most versatile modules in Python. It helps in Numeric and Scientific computations. This library does not come with the standard Python installation. If you want to use `NumPy`, install the package using your Python package manager of choice. I prefer Conda : `conda install numpy`.

```python
# Import NumPy
import numpy as np
```

```python
# Create an Array
x = np.array([1, 2, 3, 4, 5, 6])
print(x, "is of type", x.dtype, "and of size", x.size, "with shape", x.shape)
```

```python
# Reshape Matrix
x = np.array([1, 2, 3, 4, 5, 6])
x = x.reshape(2,3)
print(x, "is of type", x.dtype, "and of size", x.size, "with shape", x.shape)
```

```python
# Array Arithmetic
x = np.arange(1,7).reshape(2,3)
y = np.random.random((2,3))

print(x)
print()
print(y)
print()
print(x + y)
print()
print(x - y)
```

```python
# Array Product
x = np.arange(1,7).reshape(2,3)
y = np.random.random((3,2))

print(x)
print()
print(y)
print()
print(x.dot(y))
```

#### Quick Tasks

- Find the Eigenvalues and Eigenvectors of the `3 x 3` random matrix `np.random.random((3,3))`.

```python

```

```python

```

---

## Image Handling

`NumPy` and `MatPlotLib` libraries are capable of providing basic functionalities of image handling and processing.

```python
# Import MatPlotLib and Read Image
import matplotlib.pyplot as plt
po = plt.imread("files/poDumpling.jpg")
print(type(po), "of size", po.shape)
```

```python
# Show the Image
fig, axs = plt.subplots(nrows=1, ncols=1, figsize=(8,8))
axs.imshow(po)
```

```python
# Slice the Image
fig, axs = plt.subplots(nrows=1, ncols=1, figsize=(8,8))
axs.imshow(po[64:256,192:384,:])
```

Convert the image to its Grayscale version, with conversion formula `0.2989 R + 0.5870 G + 0.1140 B`.

```python
# Covnert to Grayscale
po_gray = np.dot(po[:,:,0:3], [0.299, 0.587, 0.114])

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(16,8))
axs[0].imshow(po)
axs[1].imshow(po_gray, cmap=plt.get_cmap('gray'))
```

#### Quick Tasks

- Find the Singular Value Decomposition (SVD) of the `512 x 512` grayscale image `po_gray`.

```python

```

```python

```

---

## Symbolic Computation

`SymPy` is the Python library for Symbolic Computation. You may install the `SymPy` module using Conda : `conda install sympy`.

```python
# Import SymPy
import sympy as sp
```

```python
# Create Symbol and Expression
x = sp.symbols('x')
ep = sp.sin(x) * sp.exp(x)
print(ep, "is of type", type(ep))
```

```python
# Symbolic Derivative
df = sp.diff(ep, x)
print("The derivative of", ep, "is", df)
```

```python
# Symbolic Integration
it = sp.integrate(df, x)
print("The integral of", df, "is", it)
```

```python
# Differential Equation
t = sp.symbols('t')
y = sp.Function('y')
sp.dsolve(sp.Eq(y(t).diff(t, t) - y(t), sp.exp(t)), y(t))
```

```python
# Pretty Print in LaTeX
z = sp.dsolve(sp.Eq(y(t).diff(t, t) - y(t), sp.exp(t)), y(t))
print(sp.latex(z))
```

#### Quick Tasks

- Find a Function (or a family of functions) whose derivative is a constant multiple of the Function itself.

```python

```

```python

```


# The gradient regularized asymptotic correction.
## I. The ionization potenital issue

Often with density functional theory one would like to do something interesting with the resulting wavefunction; however a number of issues exsist with conventional DFT computations.

Let us first compute Ionization Potential at several different levels of the theory, and then compare the results to the NIST database (http://physics.nist.gov).

```julia
using PyCall: pyimport
psi4 = pyimport("psi4")
np   = pyimport("numpy") # used only to cast to Psi4 arrays
using LinearAlgebra: dot
import Formatting: printfmt
include("ks_helper.jl")
import PyPlot: plt, matplotlib
matplotlib.use(backend="MacOSX")

build_superfunctional = nothing
if VersionNumber(psi4.__version__) >= v"1.3a1"
    build_superfunctional = psi4.driver.dft.build_superfunctional
else
    build_superfunctional = psi4.driver.dft_funcs.build_superfunctional
end

mol = psi4.geometry("""
Ne
symmetry c1
""")
options = (Dict("BASIS" => "aug-cc-pvdz",
                "NAT_ORBS" => true,
                "NUM_ROOTS" => 2,
                "QC_MODULE" => "DETCI",
                "DFT_SPHERICAL_POINTS" => 110,
                "DFT_RADIAL_POINTS" => 20))
psi4.set_options(options)

scf_e, scf_wfn = psi4.energy("SCF",  return_wfn=true)
scf_e, dft_wfn = psi4.energy("PBE0", return_wfn=true)
scf_e, ci_wfn =  psi4.energy("CISD", return_wfn=true)
```

    (-128.7034071556303, PyObject <psi4.core.CIWavefunction object at 0x1538b6bf0>)

Now that we have the wavefunctions we can compute the ionization potenital. For SCF theories this is the energy of the HOMO while for more advanced theories, it is the differente between the ground and excited states. We can obtain the IP for each level of theory in the following way:

```julia
scf_ip = -1 * scf_wfn.epsilon_a().np[scf_wfn.nalpha()] * psi4.constants.hartree2ev
dft_ip = -1 * dft_wfn.epsilon_a().np[dft_wfn.nalpha()] * psi4.constants.hartree2ev
ci_ip = -1 * (psi4.core.variable("CI ROOT 0 TOTAL ENERGY") -
              psi4.core.variable("CI ROOT 1 TOTAL ENERGY")) * psi4.constants.hartree2ev

println("Neon Ionization Potential (eV)")
printfmt("SCF     {:10.4f}\n", scf_ip)
printfmt("PBE0    {:10.4f}\n", dft_ip)
printfmt("CISD    {:10.4f}\n", ci_ip)
printfmt("NIST    {:10.4f}\n", 21.5645)

```

    Neon Ionization Potential (eV)
    SCF        23.2125
    PBE0       16.0362
    CISD       22.2776
    NIST       21.5645

## I. Density shapes
If we think carefully about the ionization potential and what this really means, it can be summed up susinctly as "How hard is it for the electron to leave the system" or "How deep is our potenital well"? A easy way to view the shape of the potenital for a given method is to inspect the density as a function of spatial extent. A weaker potential would mean that the density extends further than it should, and our IP would be too small. Whereas a stronger potenital would mean that the density is more concentrated and the IP is too large. As we only have a single atom we can project our density into a spatial grid and then sort the results as a function of distance from the nucleus.

```julia
function one_electron_radial(matrices, Vpot)
    """
    For every matrix in matrices, computes the radial density using the incoming Vpot object.
    """

    results = [[] for x in matrices]
    Rlist = []

    points_func = Vpot.properties()[1]
    superfunc = Vpot.functional()

    xc_e = 0.0
    vv10_e = 0.0

    # First loop over the outer set of blocks
    for block in 1:Vpot.nblocks()

        # Obtain general grid information
        grid = Vpot.get_block(block-1)
        x = np.array(grid.x())
        y = np.array(grid.y())
        z = np.array(grid.z())
        npoints = size(z,1)

        # Get radial information
        R = @. (x^2 + y^2 + z^2)^0.5
        push!(Rlist, R)

        # Loop over and compute the density for each matrix
        for (num, mat) in enumerate(matrices)
            points_func.set_pointers(mat)
            points_func.compute_points(grid)
            push!(results[num], np.array(points_func.point_values()["RHO_A"])[1:npoints])
        end
    end

    # Stack R
    Rlist = vcat(Rlist...)

    # Get the R unique and sort
    #uidx = findfirst.(isequal.(unique(Rlist)), [Rlist])
    uidx = indexin(sort(unique(Rlist)), Rlist)
    Rlist = Rlist[uidx]

    # Julia does magic, it would be worthwhile to consider what this does exactly
    results = [vcat(x...)[uidx] for x in results]

    Rlist, results
end

# Compute the radial densities
sup = build_superfunctional("SVWN", true)[1]
Vpot = psi4.core.VBase.build(scf_wfn.basisset(), sup, "RV")
Vpot.initialize()

R, data = one_electron_radial([scf_wfn.Da(), dft_wfn.Da(), ci_wfn.Da()], Vpot);
```

Now that we have the radial densities, we build  a function that will plot the density as a ratio between
the method and a benchmark. Here we will select the CISD densities as the benchmark and plot the SCF and DFT quantities against them.

```julia
function plot_scatter_ratio(dictionary, bench, x_values)
    fig1 = plt.figure()
    ax1 = fig1.add_subplot(111)
    for (k, v) in dictionary
        ax1.plot(x_values, v ./ bench, label=k)
    end

    ax1.legend(loc="upper left")
    ax1.set_xlim(0, 7)
    ax1.set_xlabel("R")
    ax1.set_ylabel(raw"ρ(r) / ρ(r)_bench")

    plt.plot([0, 7], [1.0, 1.0], "k-")
end
```

    plot_scatter_ratio (generic function with 1 method)

Plotting this data we note that the SCF density falls off too rapidly, meaning that potential is too deep while the DFT density falls off far too slowly indicating the potenital is too shallow. For example many theories examine the overlap between densities. From this chart we can surmise issues that would arise with both DFT and HF theories.

```julia
data_dict = Dict("SCF" => data[1], "DFT" => data[2])
plot_scatter_ratio(data_dict, data[3], R)
```

## 2. The GRAC Correction
A question then arises: what if we could correct the DFT potential asymptote so that it recovers the correct density and therefore the correction ionization potenital? Effectively, we want to accomplish:

$$v_{\rm{xc}}({\bf r}) = -\frac{1}{r} + v_{\rm{xc}}(\infty).$$

However, having a correct asympote would require knowledge of the exact $v_{\rm{xc}}$ potential. As we operate completely in the Density Funcitonal Approximation domain, this is not a possible task. Instead, the GRAC correction requires that we know our Ionization Potential and we can replace the complex DFT term with the following:

$$v_{\rm{xc}}({\bf r}) = -\frac{1}{r} + (I_{\bf P} + \epsilon_N)$$

Where $\epsilon_N$ is our HOMO energy. This is quite clever as we now only need a functional that correctly goes to the $\frac{1}{r}$ asymptotic. As no general functional accomplishes this task a special functional, "LB94", was built with exactly this property. The downside is that the characteristics of this functional are quite poor in the "bulk" region, or the region near the atom itself. To generalize this a switching function is introduced:

$$
\begin{align}
v_{\rm{xc}}^{GRAC}(\rho({\bf r})) &= [1 - f[x({\bf r})]]v_{\rm{XC}}^{\rm bulk}+ f[x({\bf r})]v_{\rm{XC}}^{\rm Asym.}  \\ \\
f[x({\bf r})] &= \frac{1}{1 + e^{-\alpha[x({\bf r}) - \beta]}}\\\\
x({\bf r}) &= \frac{| \nabla \rho({\bf r}) | }{\rho^\frac{4}{3}({\bf r})}
\end{align}
$$

We seamlessly switch between the bulk and asymptotic potenitals. In addition, such a formulation allows us to apply the GRAC correction to any underlying functional in a general way.

```julia
# Build a LB 94 functional
lb94_func = psi4.core.SuperFunctional.blank()
lb94_func.set_max_points(500000)
lb94_func.set_deriv(1)

# Add member functionals
tmp_func = psi4.core.LibXCFunctional("XC_GGA_X_LB", true)
# We need to scale this for the amount of exact exchange is in the functional
tmp_func.set_alpha(0.75)
lb94_func.add_x_functional(tmp_func)

# Dont forget VWN correlation!
tmp_func = psi4.core.LibXCFunctional("XC_LDA_C_VWN", true)
lb94_func.add_c_functional(tmp_func)

# Allocate the functional
lb94_func.allocate()

grac_shift = (21.5645 - dft_ip) /  psi4.constants.hartree2ev

"""
Computes the XC energy and the V_xc Fock term
"""
function compute_V_GRAC(D, Vpot)

    grac_alpha = 0.5
    grac_beta = 40.0

    nbf = D.shape[1]
    Varr = zeros(nbf, nbf)

    points_func = Vpot.properties()[1]
    superfunc   = Vpot.functional()

    total_e = 0.0

    for x in 1:Vpot.nblocks()

        # Grid data
        grid = Vpot.get_block(x-1)
        w = np.array(grid.w())
        npoints = size(w,1)

        points_func.compute_points(grid)
        lpos = np.array(grid.functions_local_to_global()) .+ 1
        nfunctions = size(lpos,1)

        phi = np.array(points_func.basis_values()["PHI"])[1:npoints, 1:nfunctions]

        rho = np.array(points_func.point_values()["RHO_A"])[1:npoints]
        gamma = np.array(points_func.point_values()["GAMMA_AA"])[1:npoints]

        grac_x = @. (gamma^0.5) / (rho^(4.0/3.0))
        grac_fx = @. 1.0 / (1.0 + exp(-grac_alpha * (grac_x - grac_beta)))

        ret = superfunc.compute_functional(points_func.point_values(), -1)
        lbret = lb94_func.compute_functional(points_func.point_values(), -1)

        total_e += dot(np.array(ret["V"])[1:npoints], w)

        v_rho_a  = @. ( 1 - grac_fx ) * (np.array(ret["V_RHO_A"])[1:npoints] - grac_shift)
        v_rho_a += @. grac_fx * np.array(lbret["V_RHO_A"])[1:npoints]
        v_rho_a .*= 0.5 .* w

        # Vtmp[ab] = ϕ[pb] Vρ_a[p] ϕ[pa]
        Vtmp = phi' * (v_rho_a .* phi)

        # GGA
        rho_x = np.array(points_func.point_values()["RHO_AX"])[1:npoints]
        rho_y = np.array(points_func.point_values()["RHO_AY"])[1:npoints]
        rho_z = np.array(points_func.point_values()["RHO_AZ"])[1:npoints]

        phi_x = np.array(points_func.basis_values()["PHI_X"])[1:npoints, 1:nfunctions]
        phi_y = np.array(points_func.basis_values()["PHI_Y"])[1:npoints, 1:nfunctions]
        phi_z = np.array(points_func.basis_values()["PHI_Z"])[1:npoints, 1:nfunctions]

        v_gamma  = @. (1 - grac_fx) * np.array(ret["V_GAMMA_AA"])[1:npoints]
        v_gamma .*= 2.0 .* w
        # Vtmp[ab] += ϕ[pb] v_gamma[p] ∇ₓρ[p] ∇ₓϕ[pa] ∀ x,y,z
        Vtmp += phi' * (v_gamma .* rho_x .* phi_x)
        Vtmp += phi' * (v_gamma .* rho_y .* phi_y)
        Vtmp += phi' * (v_gamma .* rho_z .* phi_z)

        Varr[lpos, lpos] += Vtmp + Vtmp'

    end
    total_e, Varr
end

# Test our function the the handy ks solver.
grac_e, grac_data = ks_solver("PBE0", mol, options, compute_V_GRAC)
```

    Number of occupied orbitals: 5
    Number of basis functions:   23

    Starting SCF iterations:

        Iter            Energy             XC E         Delta E        D RMS

    SCF Iter  1: -122.77932612141758   -11.5580423   -1.22779E+02   1.55466E-01
    SCF Iter  2: -117.35576551166125    -6.4504083   5.42356E+00   1.89487E-01
    SCF Iter  3: -128.68766707945363    -9.7183972   -1.13319E+01   2.91504E-02
    SCF Iter  4: -128.80683742064258    -9.4890898   -1.19170E-01   1.05122E-02
    SCF Iter  5: -128.82176520990393    -9.3901802   -1.49278E-02   1.39167E-03
    SCF Iter  6: -128.82218546709365    -9.3740660   -4.20257E-04   3.33912E-05
    SCF Iter  7: -128.82219002585748    -9.3743046   -4.55876E-06   1.71356E-06
    SCF Iter  8: -128.82218988268599    -9.3743231   1.43171E-07   3.24981E-08

    Final SCF energy: -128.82218988 hartree

    (-128.822189882686, Dict{Any,Any}("eigenvalues" => PyObject <psi4.core.Vector object at 0x1545abb90>,"Da" => PyObject <psi4.core.Matrix object at 0x15996e1d0>,"Ca" => PyObject <psi4.core.Matrix object at 0x1545abb30>))

We can now check against the Psi4 reference implementation. Note that we set the GRAC shift on either side of the computation so that we do not contaminate other computations.

```julia
psi4.set_options(Dict("dft_grac_shift" => grac_shift))
println("Psi4 PBE0 energy ", psi4.energy("PBE0"))
psi4.set_options(Dict("dft_grac_shift" => 0.0))
```

    Psi4 PBE0 energy -128.82218989858404

## 3. Inspecting the outcome
Now that we applied the GRAC shift we can observe the density and see how well we do.

```julia
grac_ip = -1 * grac_data["eigenvalues"].np[dft_wfn.nalpha()] * psi4.constants.hartree2ev

println("Neon Ionization Potential (eV)")
printfmt("SCF     {:10.4f}\n", scf_ip)
printfmt("PBE0    {:10.4f}\n", dft_ip)
printfmt("PBE0-AC {:10.4f}\n", grac_ip)
printfmt("CISD    {:10.4f}\n", ci_ip)
printfmt("NIST    {:10.4f}\n", 21.5645)
```

    Neon Ionization Potential (eV)
    SCF        23.2125
    PBE0       16.0362
    PBE0-AC    21.4275
    CISD       22.2776
    NIST       21.5645

We observe that our ionization potenital is now much closer to the correct ionization potenital and while the density is not perfect, it is much closer to the CISD density.

```julia
R, data = one_electron_radial([scf_wfn.Da(), dft_wfn.Da(), grac_data["Da"], ci_wfn.Da()], Vpot)
data_dict = Dict("SCF"=> data[1],
                 "DFT"=> data[2],
                 "DFT-AC"=> data[3])
plot_scatter_ratio(data_dict, data[4], R)
```

Refs:
 - M. Gruning, O. V. Gritsenko, S. J. A. van Gisbergen, and E. J. BaerendsJ. Chem. Phys., **2001**, *114*, 652


# Unit conversion for the valve coefficient

## Friction losses in energy balance

The contribution of friction losses is considered as a head loss in the enrgy balance.

EB   $0~m=\frac{\Delta p}{\rho g}+\Delta z+\frac{\Delta w^2}{2 g}+\Delta H_{v}+\frac{\dot Q}{\rho g \dot V}+\frac{C_v\Delta T}{g}+\frac{-\dot W_{Welle}}{\rho g \dot V}$

Anlagenkennlinie $H_{Anlage}=\underbrace{\Delta z+\frac{p_{aus}-p_{ein}}{\rho g}}_{\text{stat. Förderhöhe, }f(\Delta z)}+\underbrace{\frac{w_{aus}^2-w_{ein}^2}{2g}+\sum H_{Verluste}}_{\text{dyn. Förderhöhe, }f(\dot V)}$

Pumpenkennlinie $H_{Pumpe}=H(\dot V)$

Where $p$ is pressure, $\rho$ is density, $g$ is the local acceleration due to gravy, $z$ is elevation, $w$ is linear velocity, $H_{Verluste}$ are head losses due to friction, $\dot Q$ is heat flow into the system, $\dot V$ is volume flow, $C_v\Delta T$ is change in internal energy due solely to temperature change of incompressible fluid, $\dot W_{Welle}$ is power given by shaft work. $H$ are pressure heads as heights.

The pump curve gives a decreasing dynamic head $H_{Pumpe}$ with increasing volume flow $\dot V$. The plant curve $H_{Anlage}$ consists of a static head dependent on plant conditions like height, inlet and outlet pressure (independent of flow), and a dynamic head incresing with volume flow $\dot V$. Pump must be able to meet the dynamic + static head of the plant at a given flow.

## Definition of valve flow coefficients

See [1] and [2]. Note that below, as in ref. [1] volume flow is $Q$.

### Resistance coefficient $\zeta$ - dimensionless

Friction loss attributable to a pipe, valve, fitting, etc. in the pipeline, in terms of velocity head or velocity pressure. aka Widerstandsbeiwert.

$\Delta H=\zeta \frac{w^2}{2g}$

Expressed as pressure loss, $\Delta H=\frac{\Delta p_{Verlust}}{\rho g}$, or $\Delta p_{Verlust}=\zeta \frac{\rho w^2}{2}$.

The total head losses are composed of the sum of all resistances given by pipes and fittings:

Losses $\sum H_{Verlust}=\zeta_{ges}\cdot\frac{w^2}{2g}$

$\zeta_{ges}$ is the total resistance coefficient.

$\zeta_{ges}=\zeta_{Rohr}+\zeta_{Ventile}+\zeta_{Einbauten}+...$

For a pipe, the resistance coefficient is related to the length (L) to diameter (d) ratio, to give the *Widerstandszahl* (aka Darcy-Weisbach friction factor $f_D=\lambda$).

$\lambda=\zeta\frac{d}{L}=\frac{\Delta p}{\rho g}\frac{2g}{w^2}\frac{d}{L}$

More precisely, the pressure loss through the pipe is proportional to the length and inversely proportional to the diameter, by the Darcy-Weisbach friction factor.

$\frac{\Delta p}{L}=\lambda\frac{\rho}{d}\frac{w^2}{2}$

The Nikuradse-diagram (Nikuradse-Colebrook-Moody) shows $\lambda$ as a function of the Reynolds-number $Re=\frac{w d_{i}\rho}{\eta}=\frac{w d_i}{\nu}$ and the surface relative roughness $k_s/d_i$ (or $\epsilon/d_i$). For laminar flow  ($Re<3000$), the friction factor converges to the Poiseuille analytical solution of the momentum balance.

$\lambda_{laminar}=\frac{64}{Re}$

In turbulent flow, an approximate calculation is given by Blasius [2], not considering roughness.

$\lambda_{turb}\approx(100 Re)^{-1/4} \qquad 3000<Re<10^5$

or by Prandtl /. Kasman considering roughness [2,3].

$\frac{1}{\sqrt{\lambda}}=2\cdot ln\left(\frac{d_i}{k}\right)+1,14$

### *US* Flow coefficient $C_v$ - dimensions $\frac{US_{gal}}{min}$

$C_v=Q\sqrt{\frac{\Delta p_0}{\Delta p}\frac{\rho}{\rho_0}}$

Q    flow in US gal/min

$\Delta p_0$   reference differential pressure = 1lb/in²

$\Delta p_0$   operating differential pressure in lb/in²

$\rho_0$     density of reference fluid water = 62.4 lg/ft³

$\rho$     density of operating fluid in lb/ft³

### *Metric* flow coefficient $K_v$ - dimensions $\frac{m^3}{h}$

$K_v=Q\sqrt{\frac{\Delta p_0}{\Delta p}\frac{\rho}{\rho_0}}$

Q    flow in m³/h

$\Delta p_0$   reference differential pressure = 1bar

$\Delta p_0$   operating differential pressure in bar

$\rho_0$     density of reference fluid water = 1000 kg/m³

$\rho$     density of operating fluid in kg/m³

### *Metric* flow coefficient Av - dimensions $m^2$

$A_v=Q\sqrt{\frac{\rho}{\Delta p}}$

Q    flow in m³/s

$\Delta p_0$   operating differential pressure in Pa

$\rho$     density of Newtonian liquid in kg/m³

This coefficient is derived from the dimensionless resistance  coefficient $\zeta$, following from its definition in pressure loss form:

$\Delta p_{Verlust}=\zeta \frac{\rho w^2}{2g}$

$\Rightarrow Q=A\cdot w=\underbrace{A\cdot \sqrt{\frac{2}{\zeta}}}_{A_v}\cdot\sqrt{\frac{\Delta p}{\rho}}$

The definition of $A_v$ then follows, as a coefficient proportional to the orifice diameter cross section at complete opening $A$, by the factor $\left(\frac{2}{\zeta}\right)^{1/2}$.

## Interconversions

Due to prevailing historic use of the *imperial* unit system [4], it is unavoidable to carry out units intervonversions. In order to build the relationship between flow coefficients with units, it is necessary to do this in dimensionless form, as follows:

$\begin{align}
\frac{C_v/(US_{gal}/min)}{K_v/(m^3/h)}&=\frac{\left(\frac{Q}{US_{gal}/min}\right)}{\left(\frac{Q}{m^3/h}\right)}\cdot\frac{\sqrt{\frac{1\frac{lb_f}{in^2}}{\Delta p}\cdot\frac{\rho}{62.4\frac{lb_m}{ft^3}}}}{\sqrt{\frac{1~bar}{\Delta p}\cdot\frac{\rho}{1000\frac{kg}{m^3}}}}\\
&=\frac{m^3/h}{US_{gal}/min}\cdot\sqrt{\frac{1\frac{lb_f}{in^2}}{1~bar}\cdot\frac{1000\frac{kg}{m^3}}{62.4\frac{lb_m}{ft^3}}}\cdot\frac{\frac{1~h}{60~min}}{3.7854\cdot 10^{-3}\frac{m^3}{US_{gal}}}\\
&=4.4029\cdot\sqrt{\frac{1\frac{lb_f}{in^2}\cdot\frac{4.4482~N}{lb_f}\cdot\left(\frac{1~in}{2.54\cdot 10^{-2}~m}\right)^2}{1~bar\cdot\frac{10^5~N/m^2}{bar}}\cdot\frac{1000\frac{kg}{m^3}}{62.4\frac{lb_m}{ft^3}\cdot\frac{1~kg}{2.2042~lb_m}\cdot\left(\frac{1~ft}{30.48\cdot 10^{-2}~m}\right)^3}}\\
&=4.4029\cdot\sqrt{\frac{6894.72}{10^5}\cdot \frac{1000}{999.74}}\\
&=1.1562
\end{align}$

This result is equal to $1/(865\cdot10^{-3})$, which is listed in ref. [1] as "$K_v/C_v=865 \times 10^{-3}$".

Similarly, the relationship of $C_v$ to $A_v$ is built in dimensionless form.

$\begin{align}
\frac{A_v/(m^2)}{C_v/(US_{gal}/min)}&=\frac{\left(\frac{Q}{m^3/s}\right)}{\left(\frac{Q}{US_{gal}/min}\right)}\cdot\frac{\sqrt{\frac{\rho}{\Delta p}}}{\sqrt{\frac{1\frac{lb_f}{in^2}}{\Delta p}\cdot\frac{\rho}{62.4\frac{lb_m}{ft^3}}}}\\
&=\frac{US_{gal}/min}{m^3/s}\cdot\sqrt{\frac{62.4\frac{lb_m}{ft^3}}{1\frac{lb_f}{in^2}}}\cdot 3.7854\cdot 10^{-3}\frac{m^3}{US_{gal}}\frac{1~min}{60~s}\\
&=6.309\cdot 10^{-5}\sqrt{\frac{62.4\frac{lb_m}{ft^3}}{1\frac{lb_f}{in^2}}\cdot\frac{\frac{1~kg/m^3}{2.2042\cdot(30.48\cdot 10^{-2})^3 lb_m/ft^3}}{\frac{\frac{4.4482}{(2.54\cdot 10^{-2})^2}\frac{kg m}{s^2 m^2}}{lb_f/in^2}}}\\
&=6.309\cdot 10^{-5}\cdot\sqrt{\frac{999.744}{6894.72}}\frac{s}{m}\\
&=24.027\cdot 10^{-6}\frac{s}{m}
\end{align}$

This is listed in ref. [1] as $A_v/C_v=23.8\times 10^{-6}$.

From these two, the relationship from $A_v$ to $K_v$ is:

$\frac{A_v/(m^2)}{K_v/(m^3/h)}=\frac{A_v/(m^2)}{C_v/(US_{gal}/min)}\cdot\frac{C_v/(US_{gal}/min)}{K_v/(m^3/h)}=1.1562\cdot 24.027\cdot 10^{-6}\frac{s}{m}=27.78\cdot 10^{-6}\frac{s}{m}$

References:
1. Smith, P., & Zappe, R. W. (2004). Valve selection handbook: engineering fundamentals for selecting the right valve design for every industrial flow application. Elsevier.
2. PAT notes
3. Stephan, P. u.a. (2013). Vdi-Wärmeatlas. 11., bearb. und erw. Aufl. Berlin, Heidelberg: Springer-Verlag.
4. Christen, D. S. (2009). Praxiswissen der chemischen verfahrenstechnik: handbuch für chemiker und verfahrensingenieure. Springer-Verlag.


# Neuromatch Academy: Week 3, Day 1, Tutorial 2
# Real Neurons: Effects of Input Correlation

## Tutorial Objectives
In this tutorial, we will use the leaky integrate-and-fire (LIF) neuron model (see Tutorial 1) to study how they transform input correlations to output properties (transfer of correlations). In particular, we are going to write a few lines of code to:

- inject correlated GWN in a pair of neurons,

- measure correlations between the spiking activity of the two neurons, and

- study how transfer of correlation depends on the statistics of the input, i.e., mean and standard deviation.

# Setup

```python
# Imports
import matplotlib.pyplot as plt    # import matplotlib
import numpy as np                 # import numpy
import time                        # import time
import ipywidgets as widgets       # interactive display
from scipy.stats import pearsonr   # import pearson correlation coefficient
```

```python
#@title Figure Settings
fig_w, fig_h = (8, 6)
my_fontsize = 18
my_params = {'axes.labelsize': my_fontsize,
          'axes.titlesize': my_fontsize,
          'figure.figsize': [fig_w, fig_h],
          'font.size': my_fontsize,
          'legend.fontsize': my_fontsize-3,
          'lines.markersize': 8.,
          'lines.linewidth': 2.,
          'xtick.labelsize': my_fontsize-2,
          'ytick.labelsize': my_fontsize-2}

plt.rcParams.update(my_params)
my_layout = widgets.Layout()
%matplotlib inline
```

```python
# @title Helper functions
def default_pars( **kwargs):
  pars = {}

  ### typical neuron parameters###
  pars['V_th']    = -55. # spike threshold [mV]
  pars['V_reset'] = -75. #reset potential [mV]
  pars['tau_m']   = 10. # membrane time constant [ms]
  pars['g_L']     = 10. #leak conductance [nS]
  pars['V_init']  = -65. # initial potential [mV]
  pars['V_L']     = -75. #leak reversal potential [mV]
  pars['tref']    = 2. # refractory time (ms)

  ### simulation parameters ###
  pars['T'] = 400. # Total duration of simulation [ms]
  pars['dt'] = .1  # Simulation time step [ms]

  ### external parameters if any ###
  for k in kwargs:
    pars[k] = kwargs[k]

  pars['range_t'] = np.arange(0, pars['T'], pars['dt']) # Vector of discretized time points [ms]

  return pars

def run_LIF(pars, I):
  """
  Simulate the LIF dynamics with external input current

  Args:
    pars       : parameter dictionary
    I          : input current [pA]. The injected current here can be a value or an array

  Returns:
    rec_spikes : spike times
    rec_v      : mebrane potential
  """

  # Set parameters
  V_th, V_reset = pars['V_th'], pars['V_reset']
  tau_m, g_L = pars['tau_m'], pars['g_L']
  V_init, V_L = pars['V_init'], pars['V_L']
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size
  tref = pars['tref']
  # Initialize voltage and current
  v = np.zeros(Lt)
  v[0] = V_init
  I = I * np.ones(Lt)
  tr = 0.
  # simulate the LIF dynamics
  rec_spikes = []               # record spike times
  for it in range(Lt-1):
    if tr >0:
      v[it] = V_reset
      tr = tr-1
    elif v[it] >= V_th:         #reset voltage and record spike event
      rec_spikes.append(it)
      v[it] = V_reset
      tr = tref/dt
    #calculate the increment of the membrane potential
    dv = (-(v[it]-V_L) + I[it]/g_L) * (dt/tau_m)

    #update the membrane potential
    v[it+1] = v[it] + dv

  rec_spikes = np.array(rec_spikes) * dt

  return v, rec_spikes

def my_GWN(pars, sig, myseed=False):
  """
  Function that calculates Gaussian white noise inputs

  Args:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    myseed     : random seed. int or boolean
                 the same seed will give the same random number sequence

  Returns:
    I          : Gaussian white noise input
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  # you can fix the seed of the random number generator so that the results are reliable
  # however, when you want to generate multiple realization make sure that you change
  # the seed for each new realization
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  #generate GWN
  # we divide here by 1000 to convert units to sec.
  I =  sig * np.random.randn(Lt) * np.sqrt(pars['tau_m']/dt)

  return I

def Poisson_generator(pars, rate, n, myseed=False):
  """
  Generates poisson trains

  Args:
    pars       : parameter dictionary
    rate       : noise amplitute [Hz]
    n          : number of Poisson trains
    myseed     : random seed. int or boolean

  Returns:
    pre_spike_train : spike train matrix, ith row represents whether
                      there is a spike in ith spike train over time
                      (1 if spike, 0 otherwise)
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate uniformly distributed random variables
  u_rand = np.random.rand(n, Lt)

  # generate Poisson train
  poisson_train = 1. * (u_rand<rate*dt/1000.)

  return poisson_train
```

The helper function contains a:

- Parameter dictionary: `default_pars( **kwargs)`
- LIF simulator: `run_LIF`
- Gaussian white noise generator: `my_GWN(pars, sig, myseed=False)`
- Poisson type spike train genertor: `Poisson_generator(pars, rate, n, myseed=False)`

---
# Correlations (Synchrony)
Correlation, or synchrony, in neuronal activity can be described for any readout of brain activity. Here, we are concerned with the spiking activity of neurons.

In the simplest way, correlation/synchrony refers to coincident spiking of neurons, i.e., when two neurons spike together, they are firing in synchrony or are correlated. Neurons can be synchronous in their instantaneous activity, i.e., they spike together with some probability. However, it is also possible that spiking of a neuron at time $t$ is correlated with the spikes of another neuron with a delay (time-delayed synchrony).

### Origin of synchronous neuronal activity:
- Common inputs, i.e., two neurons are receiving input from the same sources. The degree of correlation of the shared inputs is proportional to their output correlation.
- Neurons do not share the same input neurons, but are receiving inputs from neurons which themselves are correlated.
- Neurons are connected to each other (uni- or bi-directionally). This will only give rise to time-delayed synchrony. Neurons could also be connected via gap-junctions.
- Neurons have similar parameters and initial conditions.

### Implications of synchrony
When neurons spike together, they can have a stronger impact on downstream neurons. Synapses in the brain are sensitive to the temporal correlations (i.e., delay) between pre- and postsynaptic activity, and this, in turm, can lead to the formation of functional neuronal networks - the basis of unsupervised learning (we will study some of these concepts in a forthcoming tutorial).

Synchrony implies a reduction in the dimensionality of the system. In addition, correlations in many cases can impair the decoding of neuronal activity.

```python
#@title Video: Correlations
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="3Sd9WlvNegM", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=3Sd9WlvNegM

## Correlated Input

Correlated firing is often observed in neuronal circuits, and has been linked to stimulus encoding, attention, stimulus discrimination, and
motor behaviour. The origin of correlation in neuronal activity is usually attributed to common inputs. Correlation may arise when neurons sample their activity from the same source of correlated spike train, even though they are not directly connected to the same inputs.

A simple model to study the emergence of correlation is to inject common inputs to a pair of neurons and measure the output correlation as a function of the fraction of common inputs.

Here, we are going to investigate the transfer of correlations by computing the spike train correlation coefficient of unconnected pairs of LIF neurons receiving correlated inputs

The input current to LIF neuron $i$ $(i=1,2)$ is:

\begin{equation}
\frac{I_i}{g_L} =\mu_i + \sigma_i (\sqrt{1-c}\xi_i + \sqrt{c}\xi_c) \quad (1)
\end{equation}

where $\mu_i$ is the temporal average of the current. The Gaussian white noise $\xi_i$ is independent for each neuron, while $\xi_c$ is common to all the neurons. The variable $c$ ($0\le c\le1$) controls the fraction of common and independent inputs. $\sigma_i$ shows the variance of the total input.

```python
#@title Correlated inputs generator
def correlate_input(pars, mu=20., sig=7.5, c=0.3):
    '''
    Expects:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    c.         : correlation coefficient ~[0, 1]

    Returns:
    I1gL, I2gL : two correlated inputs with corr. coe. c
    '''
    # generate Gaussian whute noise xi_1, xi_2, xi_c
    xi_1 = my_GWN(pars, sig)
    xi_2 = my_GWN(pars, sig)
    xi_c = my_GWN(pars, sig)

    # Generate two correlated inputs by Equation. (1)
    I1gL = mu + np.sqrt(1.-c)*xi_1 + np.sqrt(c)*xi_c
    I2gL = mu + np.sqrt(1.-c)*xi_2 + np.sqrt(c)*xi_c

    return I1gL, I2gL

print(help(correlate_input))

def example_plot_myCC():
  pars = default_pars(T=50000, dt=.1)

  c = np.arange(10) * 0.1
  r12 = np.zeros(10)
  for i in range(10):
    I1gL, I2gL = correlate_input(pars, mu=20.0, sig=7.5, c=c[i])
    r12[i] = my_CC(I1gL, I2gL)

  plt.figure()
  plt.plot(c, r12, 'bo', alpha=0.7, label='Simulation', zorder=2)
  plt.plot([-0.05,0.95], [-0.05,0.95], 'k--', label='y=x', dashes=(2, 2), zorder=1)
  plt.xlabel('True CC')
  plt.ylabel('Sample CC')
  plt.legend(loc='best')
```

    Help on function correlate_input in module __main__:

    correlate_input(pars, mu=20.0, sig=7.5, c=0.3)
        Expects:
        pars       : parameter dictionary
        mu         : noise baseline (mean)
        sig        : noise amplitute (standard deviation)
        c.         : correlation coefficient ~[0, 1]

        Returns:
        I1gL, I2gL : two correlated inputs with corr. coe. c

    None

### Exercise 1: Compute the correlation

From the inputs $I_i(t)$ ($i=1,\ 2$), we can obtain $\{ I_i^1, ..., I_i^L \}$ on the time-grid of of stepsize $dt$, where $I_i^k = I_i(k\cdot dt)$. The _sample correlation coefficient_:

$$r_{ij} = \frac{\displaystyle{\sum_{k=1}^L{(I_i^k-\bar{I}_i)}{(I^k_j-\bar{I}_j)}}}{\sqrt{\displaystyle{\sum_{k=1}^L{(I_i^k-\bar{I}_i)^2}}} \sqrt{\displaystyle{\sum_{k=1}^L{(I_j^k-\bar{I}_j)^2}}}} \quad (2)$$
where $\bar{I}_i$ is the sample mean. The _sample correlation coefficient_ may also be referred as the _sample Pearson correlation coefficient_.

Here is a beautiful paper that explains multiple ways to calculate and understand correlations [Rodgers and Nicewander 1988](https://www.stat.berkeley.edu/~rabbee/correlation.pdf)

```python
# Exercise
def my_CC(x, y):
  """
  Args:
    x, y  : two time serieses with same length

  Returns:
    rxy   : correlation coefficient
  """
  ########################################################################
  ## TODO for students: compute rxy, then remove the NotImplementedError #
  ########################################################################
  # xy  = ...
  # xx  = ... #Tip1: array([a1, a2, a3])*array([b1, b2, b3]) = array([a1*b1, a2*b2, a3*b3])
  # yy  = ... #Tip2: np.sum(array([a1, a2, a3])) = a1+a2+a3
  # rxy = ... #Tip3: square root, np.sqrt()

  raise NotImplementedError("Student excercise: compute the sample correlation coefficient")

  return rxy

# Uncomment the line after completing the my_CC function
# example_plot_myCC()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W3D1_RealNeurons/solutions/W3D1_Tutorial2_Solution_64c2de3e.py)

*Example output:*

### Exercise 2: Measure the correlation of spike trains

After recording the spike times of the two neurons, how can we estimate their correlation coefficient?

In order to find this, we need to bin the spike times and obtain two time series. Each data point in the time series is the number of spikes in the corresponding time bin. You can use `numpy.histogram()` to bin the spike times.

Complete the code below to bin the spike times and calculate the correlation coefficient for two Poisson spike trains. Note that `c` here is the ground-truth correlation coefficient that we define.

Note: please execute the below cell first.

```python
#@title Helper function for generating poisson spike trains.
def generate_corr_Poisson(pars, poi_rate, c):
  """
  function to generate correlated Poisson type spike trains
  Args:
    pars       : parameter dictionary
    poi_rate   : rate of the Poisson train
    c.         : correlation coefficient ~[0, 1]

  Returns:
    sp1, sp2   : two correlated spike time trains with corr. coe. c
  """

  dt, range_t = pars['dt'], pars['range_t']
  Lt = len(range_t)

  mother_rate = poi_rate/c
  mother_spike_train = Poisson_generator(pars, rate=mother_rate, n=1)[0]
  sp_mother = range_t[mother_spike_train>0]

  L_sp_mother = len(sp_mother)
  sp_mother_id = np.arange(L_sp_mother)
  L_sp_corr = int(L_sp_mother*c)

  np.random.shuffle(sp_mother_id)
  sp1 = np.sort(sp_mother[sp_mother_id[:L_sp_corr]])

  np.random.shuffle(sp_mother_id)
  sp2 = np.sort(sp_mother[sp_mother_id[:L_sp_corr]])

  return sp1, sp2
```

```python
# Exercise
# generate correlated Poisson spike time
poi_rate = 20.
c = 0.2 # set true correlation
pars = default_pars(T=10000)
sp1, sp2 = generate_corr_Poisson(pars, poi_rate, c)

#bin the spike time
bin_size = 20 # [ms]
my_bin = np.arange(0, pars['T'], bin_size)
##############################################################
## TODO for students: Use np.histogram to bin the spike time #
## Use my_CC() compute corr coe, compare with c              #
##############################################################
# e.g., sp1_count, _= np.histogram(...)

# Note that you can run multiple realizations and compute their r_12(diff_trials)
# with the defined function above. The average r_12 over trials can get close to c.
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W3D1_RealNeurons/solutions/W3D1_Tutorial2_Solution_703c3a41.py)

## Investigate the effect of input correlation on the output correlation

Now let's combine the aforementioned two precudures. We first generate the correlated inputs by Equation (1) (i.e., Exercise 1). Then we inject the correlated inputs $I_1, I_2$ into a pair of neurons and record their output spike times. We continue meansuring the correlation between the output, and
investigate the relationship between the input correlation and the output correlation.

```python
#@title Helper function for LIF neurons with correlated input

def LIF_output_cc(pars, mu, sig, c, bin_size, n_trials=20):
      '''
      Expects:
      pars       : parameter dictionary
      mu         : noise baseline (mean)
      sig        : noise amplitute (standard deviation)
      c          : correlation coefficient ~[0, 1]
      bin_size   : bin size used for time series
      n_trials   : total simulation trials

      Returns:
      r          : output corr. coe.
      sp_rate    : spike rate
      sp1        : spike times of neuron 1 in the last trial
      sp2        : spike times of neuron 2 in the last trial
      '''

      r12 = np.zeros(n_trials)
      sp_rate = np.zeros(n_trials)
      for i_trial in range(n_trials):
          I1gL, I2gL = correlate_input(pars, mu, sig, c)
          _, sp1 = run_LIF(pars, pars['g_L']*I1gL)
          _, sp2 = run_LIF(pars, pars['g_L']*I2gL)

          #bin_size = bin_size #int(pars['T']/len(sp1))
          my_bin = np.arange(0, pars['T'], bin_size)

          sp1_count, _ = np.histogram(sp1, bins=my_bin)
          sp2_count, _ = np.histogram(sp2, bins=my_bin)

          r12[i_trial] = my_CC(sp1_count[::20], sp2_count[::20])
          sp_rate[i_trial] = len(sp1) / pars['T'] * 1000.

      return  r12.mean(), sp_rate.mean(), sp1, sp2

def plot_c_r_LIF(c, r, mycolor, mylabel):
    z = np.polyfit(c, r, deg=1)
    c_range = np.array([ c.min()-0.05, c.max()+0.05])
    plt.plot(c, r, 'o', color=mycolor, alpha=0.7, label=mylabel, zorder=2)
    plt.plot(c_range, z[0]*c_range+z[1], color=mycolor, zorder=1)
```

### Drive a neuron with correlated inputs and visualize its output
In the following, you will inject correlated GWN in two neurons. You need to define the mean (`gwn_mean`), standard deviation (`gwn_std`) and input correlations (`c_in`).

We will simulate $10$ trials to get a better estimate of the output correlation. Change the values for the above variables to explore how they impact the output correlation.

```python
pars = default_pars(T=80000, dt=1.) # get the parameters
c_in = 0.3 # set input correlation value
gwn_mean = 10.
gwn_std = 10.

bin_size = 10. # ms

starttime = time.perf_counter() # time clock

r12_ss, sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=gwn_mean, sig=gwn_std, c=c_in, bin_size = bin_size, n_trials=10)

# just the time counter
endtime = time.perf_counter()
timecost = (endtime - starttime)/60.
print('Simulation time = %.2f min' % timecost)
```

    Simulation time = 0.04 min

```python
print('Input correlation = '+str(c_in))
print('Output correlation = '+str(r12_ss))

plt.figure(figsize=(12, 6))
plt.plot(sp1,np.ones(len(sp1))*1,'|',ms = 20,label = 'neuron 1')
plt.plot(sp2,np.ones(len(sp2))*1.1,'|',ms = 20,label = 'neuron 2')
plt.ylim(0.9, 1.2)
plt.legend()
plt.xlabel('time (ms)')
plt.ylabel('neuron id.')
plt.xlim(1000,8000)
```

### Think!
- Is the output correlation always smaller than the input correlation? If yes, why?
- Should there be a systematic relationship between input and output correlations?

Lets vary `c_in` and plot the relationship between the `c_in` and output correlation. This might take some time depending on the number of trials.

```python
pars = default_pars(T=80000, dt=1.) # get the parameters
bin_size = 10.
c_in = np.arange(0,1.0,0.1) # set the range for input CC
r12_ss = np.zeros(len(c_in)) # small mu, small sigma

starttime = time.perf_counter() # time clock
for ic in range(len(c_in)):
  r12_ss[ic], sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=10., c=c_in[ic], bin_size = bin_size, n_trials=10)
endtime = time.perf_counter()
timecost = (endtime - starttime)/60.
print('Simulation time = %.2f min' % timecost)
```

    Simulation time = 0.39 min

```python
plt.figure(figsize=(7, 6))
plot_c_r_LIF(c_in, r12_ss, mycolor='b',mylabel='Output CC')
plt.plot([ c_in.min()-0.05, c_in.max()+0.05], [ c_in.min()-0.05, c_in.max()+0.05], 'k--', dashes=(2,2), label='y=x')
plt.xlabel('Input CC')
plt.ylabel('Output CC')
plt.legend(loc='best', fontsize=16)
```

## Correlation transfer function
The above plot of input correlation vs. output correlation is called the __correlation transfer function__ of the neurons.

### How do mean and standard deviation of the GWN affect the correlation transfer function?
The correlations transfer function appears to be linear. The above can be taken as the input/output transfer function of LIF neurons for correlations, instead of the transfer function for input/output firing rates as we had discussed in the previous tutorial (i.e., F-I curve).

What would you expect to happen to the slope of the correlation transfer function if you vary the mean and/or the standard deviation of the GWN?

```python
pars = default_pars(T=80000, dt=1.) # get the parameters
no_trial = 10
bin_size = 10.
c_in = np.arange(0.,1., 0.2) # set the range for input CC
r12_ss = np.zeros(len(c_in)) # small mu, small sigma
r12_ls = np.zeros(len(c_in)) # large mu, small sigma
r12_sl = np.zeros(len(c_in)) # small mu, large sigma

starttime = time.perf_counter() # time clock
for ic in range(len(c_in)):
  r12_ss[ic], sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=10., c=c_in[ic], bin_size = bin_size, n_trials=no_trial)
  r12_ls[ic], sp_ls, sp1, sp2 = LIF_output_cc(pars, mu=18.0, sig=10., c=c_in[ic], bin_size = bin_size, n_trials=no_trial)
  r12_sl[ic], sp_sl, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=20., c=c_in[ic], bin_size = bin_size, n_trials=no_trial)
endtime = time.perf_counter()
timecost = (endtime - starttime)/60.
print('Simulation time = %.2f min' % timecost)
```

    Simulation time = 0.59 min

```python
plt.figure(figsize=(7, 6))
plot_c_r_LIF(c_in, r12_ss, mycolor='b', mylabel=r'Small $\mu$, small $\sigma$')
plot_c_r_LIF(c_in, r12_ls, mycolor='y', mylabel=r'Large $\mu$, small $\sigma$')
plot_c_r_LIF(c_in, r12_sl, mycolor='r', mylabel=r'Small $\mu$, large $\sigma$')
plt.plot([ c_in.min()-0.05, c_in.max()+0.05], [ c_in.min()-0.05, c_in.max()+0.05], 'k--', dashes=(2,2), label='y=x')
plt.xlabel('Input CC')
plt.ylabel('Output CC')
plt.legend(loc='best', fontsize=14)
```

### Think!
Why do both the mean and the standard deviation of the GWN affect the slope of correlation transfer function?

### What is the rationale behind varying $\mu$ and $\sigma$?
The mean and the variance of the synaptic current depends on the spike rate of a Poisson process. We can use the [Campbell's theorem](https://en.wikipedia.org/wiki/Campbell%27s_theorem_(probability)) to estimate the mean and the variance of the synaptic current:

\begin{align}
\mu_{\rm syn} = \lambda J \int P(t) \\
\sigma_{\rm syn} = \lambda J \int P(t)^2 dt\\
\end{align}

where $\lambda$ is the firing rate of the Poisson input, $J$ the amplitude of the postsynaptic current, and $P(t)$ is the shape of the postsynaptic current as a function of time.

Therefore, when we varied $\mu$ and/or $\sigma$ of the GWN, we mimicked a change in the input firing rate. Note that, if we changed the firing rate, both $\mu$ and $\sigma$ will change simultaneously, not independently.

Here, since we observe an effect of $\mu$ and $\sigma$ on correlation transfer, this implies that the input rate has an impact on the correlation transfer function.

### Think!
- What are factors that would make output correlations smaller than input correlations? (Notice that the colored lines are below the black dashed line)
- What does it mean for the correlation in the network?
- What do you think will happen if you use a wider range of input correlations?
- Here we have studied the transfer of correlations by injecting GWN. But in the previous tutorial we mentioned that GWN is unphysiological. Indeed, neurons receive colored noise (i.e., Shot noise or OU process). How do these results obtained from injection of GWN apply to the case where correlated spiking inputs are injected in the two LIFs? Will the results be the same or different?

_Here is a reference [De La Rocha, Jaime, et al. "Correlation between neural spike trains increases with firing rate." Nature  (2007)](https://www.nature.com/articles/nature06028/)_

## Example of a conductance-based LIF model
Above, we have written code to generate correlated Poisson spike trains. You can write code to stimulate the LIF neuron with such correlated spike trains and study the correlation transfer function for spiking input and compare it to the correlation transfer function obtained by injecting correlated GWNs.

```python
#@title Conductance-based LIF: `run_LIF_cond`

def run_LIF_cond(pars, I_inj, pre_spike_train_ex, pre_spike_train_in):
    '''
    conductance-based LIF dynamics

    Expects:
    pars               : parameter dictionary
    I_inj              : injected current [pA]. The injected current here can be a value or an array
    pre_spike_train_ex : spike train input from presynaptic excitatory neuron
    pre_spike_train_in : spike train input from presynaptic inhibitory neuron

    Returns:
    rec_spikes : spike times
    rec_v      : mebrane potential
    gE         : postsynaptic excitatory conductance
    gI         : postsynaptic inhibitory conductance
    '''

    # Retrieve parameters
    V_th, V_reset = pars['V_th'], pars['V_reset']
    tau_m, g_L = pars['tau_m'], pars['g_L']
    V_init, E_L = pars['V_init'], pars['E_L']
    gE_bar, gI_bar = pars['gE_bar'], pars['gI_bar']
    VE, VI = pars['VE'], pars['VI']
    tau_syn_E, tau_syn_I = pars['tau_syn_E'], pars['tau_syn_I']
    tref = pars['tref']
    dt, range_t = pars['dt'], pars['range_t']
    Lt = range_t.size

    # Initialize
    tr = 0.
    v = np.zeros(Lt)
    v[0] = V_init
    gE = np.zeros(Lt)
    gI = np.zeros(Lt)
    I = I_inj * np.ones(Lt) #ensure I has length Lt

    if pre_spike_train_ex.max() == 0:
      pre_spike_train_ex_total = np.zeros(Lt)
    else:
      pre_spike_train_ex_total = pre_spike_train_ex * np.ones(Lt)

    if pre_spike_train_in.max() == 0:
      pre_spike_train_in_total = np.zeros(Lt)
    else:
      pre_spike_train_in_total = pre_spike_train_in * np.ones(Lt)

    # simulation
    rec_spikes = [] # recording spike times
    for it in range(Lt-1):
      if tr >0:
            v[it] = V_reset
            tr = tr-1
      elif v[it] >= V_th:         #reset voltage and record spike event
            rec_spikes.append(it)
            v[it] = V_reset
            tr = tref/dt
      #update the synaptic conductance
      gE[it+1] = gE[it] - (dt/tau_syn_E)*gE[it] + gE_bar*pre_spike_train_ex_total[it+1]
      gI[it+1] = gI[it] - (dt/tau_syn_I)*gI[it] + gI_bar*pre_spike_train_in_total[it+1]

      #calculate the increment of the membrane potential
      dv = (-(v[it]-E_L) - (gE[it+1]/g_L)*(v[it]-VE) - \
              (gI[it+1]/g_L)*(v[it]-VI) + I[it]/g_L) * (dt/tau_m)

      #update membrane potential
      v[it+1] = v[it] + dv

    rec_spikes = np.array(rec_spikes) * dt

    return v, rec_spikes, gE, gI
```

```python
#@title Interactive Demo: Correlated spike input to an LIF neuron

my_layout.width = '450px'
@widgets.interact(
  pwc_ee = widgets.FloatSlider(0.3, min=0.05, max=0.99, step=0.01, layout=my_layout),
  exc_rate = widgets.FloatSlider(1e3, min=500., max=5e3, step=50., layout=my_layout),
  inh_rate = widgets.FloatSlider(500., min=300., max=5e3, step=5., layout=my_layout),
)

def EI_isi_regularity(pwc_ee, exc_rate, inh_rate):

  pars = default_pars(T=1000.)
  #Add parameters
  pars['V_th']    = -55. # spike threshold [mV]
  pars['V_reset'] = -75. #reset potential [mV]
  pars['tau_m']   = 10. # membrane time constant [ms]
  pars['g_L']     = 10. #leak conductance [nS]
  pars['V_init']  = -65. # initial potential [mV]
  pars['E_L']     = -75. #leak reversal potential [mV]
  pars['tref']    = 2. # refractory time (ms)

  pars['gE_bar']    = 4.0  # [nS]
  pars['VE']        = 0.   # [mV] excitatory reversal potential
  pars['tau_syn_E'] = 2.   # [ms]
  pars['gI_bar']    = 2.4  # [nS]
  pars['VI']        = -80. # [mV] inhibitory reversal potential
  pars['tau_syn_I'] = 5.   # [ms]

  my_bin = np.arange(0, pars['T']+pars['dt'], .1) # 20 [ms] bin-size

  exc_rate = 1e3;
  inh_rate = 0.4e3
  pwc_ee = 0.3
  pwc_ii = 0.01
  # generate two correlated spike trains for excitatory input
  sp1e, sp2e = generate_corr_Poisson(pars, exc_rate, pwc_ee)

  sp1_spike_train_ex, _ = np.histogram(sp1e, bins=my_bin)
  sp2_spike_train_ex, _ = np.histogram(sp2e, bins=my_bin)

  # generate two uncorrelated spike trains for inhibitory input
  sp1i, sp2i = generate_corr_Poisson(pars, inh_rate, pwc_ii)
  sp1_spike_train_in, _ = np.histogram(sp1e, bins=my_bin)
  sp2_spike_train_in, _ = np.histogram(sp2e, bins=my_bin)

  v1, rec_spikes1, gE, gI = run_LIF_cond(pars, 0, sp1_spike_train_ex, sp1_spike_train_in)
  v2, rec_spikes2, gE, gI = run_LIF_cond(pars, 0, sp2_spike_train_ex, sp2_spike_train_in)

  #bin the spike time
  bin_size = 20 # [ms]
  my_bin = np.arange(0, pars['T'], bin_size)
  spk_1, _ = np.histogram(rec_spikes1, bins=my_bin)
  spk_2, _ = np.histogram(rec_spikes2, bins=my_bin)

  r12 = my_CC(spk_1, spk_2)
  print('Input correlation = '+str(pwc_ee))
  print('Output correlation = '+str(r12))

  plt.figure(figsize=(14, 7))
  plt.subplot(2,1,1)
  plt.plot(sp1e,np.ones(len(sp1e))*1,'|',ms = 20,label = 'Exc. input 1')
  plt.plot(sp2e,np.ones(len(sp2e))*1.1,'|',ms = 20,label = 'Exc. input 2')
  plt.plot(sp1e,np.ones(len(sp1e))*1.3,'|k',ms = 20,label = 'Inh. input 1')
  plt.plot(sp2e,np.ones(len(sp2e))*1.4,'|k',ms = 20,label = 'Inh. input 2')

  plt.ylim(0.9, 1.5)
  plt.legend()
  plt.xlabel('time (ms)')
  plt.ylabel('neuron id.')

  plt.subplot(2,1,2)
  plt.plot(pars['range_t'],v1,label = 'neuron 1')
  plt.plot(pars['range_t'],v2,label = 'neuron 2')
```

In the above, we are estimating output correlation for one trial. You can modify the code to get a trial average of output correlations.

---
# Summary

In this tutorial, we studied how the input correlation of two LIF neurons is mapped to their output correlation. Specifically, we:

- injected correlated GWN in a pair of neurons,

- measured correlations between the spiking activity of the two neurons, and

- studied how transfer of correlation depends on the statistics of the input, i.e., mean and standard deviation.

Here, we were concerned with zero-time lag correlation. For this reason we restricted estimation of correlation to instanteneous correlations. If you are interested in time-lagged correlation, then we should estimate the cross-correlogram of the spike trains and find out the dominant peak and area under the peak to get an estimate of output correlations.

We leave this as a future to-do for you, if you are interested.

# Bonus

Finally, there is a short BONUS lecture video on the firing response of an ensemble of neurons to time-varying input. There is no associated coding exercises - just enjoy.

```python
#@title Video: Response of ensemble of neurons to time-varying input
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="4DDWrmVy_0o", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=4DDWrmVy_0o