## Theorem 0.0.2 (Approximate Caratheodory's Theorem)
In this worksheet we run through the proof of approximate Caratheodory, keeping an example to work with as we go. Please fill in code where indicated. Here is the theorem (slightly generalized) for reference:

**Theorem 0.0.2** (Generalized)**.** *Consider a set $T \subset \mathbb{R}^n$. Then, for every point $x \in \text{conv}(T)$ and every integer $k$, one can find points $x_1,\ldots,x_k \in T$ such that*
$$ \left\lVert x - \frac{1}{k}\sum_{j=1}^k x_j \right\rVert \leq \frac{\text{diam}(T)}{\sqrt k}$$

```python
# Some useful imports
import matplotlib.pyplot as plt
import numpy as np
import random as rand
import math
```

For the purposes of the code, let us fix a set $T \subset \mathbb R^n$ and an integer $k$ to work with.

```python
# Choose a dimension to work in
n = 3

# Take some set of points with the points listed as np arrays of length n.
T = [np.zeros(3),np.zeros(3)] # TODO: write this!

# Pick a k, any k
k = 5

```

Let $m = |T|$.

```python
m = 2 # TODO: Set m to the size of T (ideally, don't hard-code this in)
```

Our task is to find some $x \in \text{conv}(T)$ and express it as a convex combination of vectors in $T$. Recalling the definition of the convex hull, this amounts to finding an assignment of coefficients $\lambda_1, \ldots, \lambda_m$, so that $\lambda_1 + \cdots + \lambda_m = 1$ and $\lambda_1,\ldots,\lambda_m \geq 0$.

Below we present one way of doing this, but it would be nice to be able to sample more uniformly from $\text{conv}(T)$. See if you can improve it!

```python
# Find random coefficients lambda_1,...,lambda_m as necessary.
# To recap, these satisfy:
# 1. lambda_1 + ... + lambda_m = 1
# 2. lambda_1, ... , lambda_m >= 0

coefficients = [0]*m # initialize list of coefficients
perm = np.random.permutation(m) # get a random permutation of 0,...,m-1
accum = 1.0
for i in perm[:-1]:
    # sample from viable range
    coefficients[i] = np.random.uniform(0,accum)
    accum = accum - coefficients[i]
coefficients[perm[-1]] = accum

coefficients
```

    [0.01639505649315731, 0.9836049435068427]

We can now get $x \in \text{conv}(T)$ with $x = \lambda_1 z_1 + \cdots + \lambda_m z_m$ where $z_1, \ldots, z_m$ are the elements of $T$.

```python
# Construct x
x = np.zeros(n) # TODO: Write this!

x
```

    array([0., 0., 0.])

Now, we interpret the definition of convex combination probabilistically, with $\lambda_i$ taking the roles of probabilities. Specifically, we can define a random vector $Z$ that takes values $z_i$ with probabilities $\lambda_i$:
$$
\mathbb P \{Z = z_i\} = \lambda_i, \ \ \ i = 1, \ldots, m.
$$

```python
# We can sample from the distribution of this random variable as follows:
Z_sample = rand.choices(T, weights = coefficients)
print(Z_sample)
```

    [array([0., 0., 0.])]

Consider independent copies $Z_1, \ldots, Z_k$ of $Z$. Then we are interested in the random variable
$$\left\lVert x - \frac{1}{k} \sum_{j=1}^k Z_j \right\rVert_2^2$$.

```python
# Sample from the distribution of the random variable specified above, and return
# the sampled value along with the assignments to Z_1,...,Z_k
def sample_from_distribution():
    vectors = rand.choices(T, weights = coefficients, k = k)
    vec_sum = np.zeros(len(T[0]))
    for vec in vectors:
        vec_sum = vec_sum + vec
    distance = np.sum((x - vec_sum/k)**2)
    return distance, vectors
```

In the proof of the theorem it is shown that
\begin{equation}
\mathbb E \left\lVert x - \frac{1}{k} \sum_{j=1}^k Z_j \right\rVert_2^2 \leq \frac{\text{diam}^2(T)}{k} \tag{1}
\end{equation}

Prove that this generalization to an arbitrary set is true.

We now find $\text{diam}(T)$. Recall $\text{diam}(T) = \sup_{x,y \in T} \lvert x - y \rvert $

```python
# Compute diam(T)
diamT = 0 # TODO: Write this!
```

We can now verify inequality $(1)$ for our set $T$.

```python
numSamples = 100000
samples = []

bestVal = float("inf")
best = None
for i in range(numSamples):
    sample, vectors = sample_from_distribution()
    if sample < bestVal:
        best = vectors
        bestVal = sample
    samples.append(sample)
```

```python
# plotting the example
num_bins = 10
plt.hist(samples, num_bins, facecolor='blue', alpha=0.5)
plt.axvline((diamT**2)/k, color='k', linestyle='dashed', linewidth=1.5, label = "$diam^2(T)/k$")
plt.axvline(np.mean(samples), color='r', linestyle='dashed', linewidth=1.5, label = "mean" )
plt.legend()
plt.show()
print("Best assignment (with value",bestVal, "): ")
print(best)
```

Copyright (c) 2020 TRIPODS/GradStemForAll 2020 Team


# 02. Neuralnet Q Learning example

Neuralnet Q Learning을 실습해봅니다.
- 신경망의 parameter(weight)를 업데이트 함에 따라 state에 대한 Q value가 변화합니다.
- Q Learning의 TD error를 loss function으로 하여 학습합니다.

## Colab 용 package 설치 코드

```python
!pip install gym
```

### Package import

```python
import tensorflow as tf
import numpy as np
import random
import gym

import matplotlib.pyplot as plt
%matplotlib inline

np.random.seed(777)
tf.set_random_seed(777)

print("tensorflow version: ", tf.__version__)
print("gym version: ", gym.__version__)
```

## Frozen Lake

**[state]**

        SFFF
        FHFH
        FFFH
        HFFG

    S : starting point, safe
    F : frozen surface, safe
    H : hole, fall to your doom
    G : goal, where the frisbee is located

**[action]**

    LEFT = 0
    DOWN = 1
    RIGHT = 2
    UP = 3

### Frozen Lake (not Slippery)

```python
def register_frozen_lake_not_slippery(name):
    from gym.envs.registration import register
    register(
        id=name,
        entry_point='gym.envs.toy_text:FrozenLakeEnv',
        kwargs={'map_name' : '4x4', 'is_slippery': False},
        max_episode_steps=100,
        reward_threshold=0.78, # optimum = .8196
    )

register_frozen_lake_not_slippery('FrozenLakeNotSlippery-v0')
```

```python
# Load Environment
env = gym.make('FrozenLakeNotSlippery-v0')
state_size = env.observation_space.n
action_size = env.action_space.n
print("State_size : ", state_size)
print("Action_size: ",action_size)
```

## Q-Learning using Neural Network
**Update 식**

$J(w) = \mathbb{E}_{\pi}[(target - \hat q(S,A,w))^2]$

$ \begin{align} \Delta w & = - \frac{1}{2} \alpha \nabla_w J(w)
\\ & = \alpha (R_{t+1} + \gamma max(\hat q(S_{t+1},a ,w)) - \hat q(S_{t},A_{t},w))\nabla_w \hat q(S_{t},A_{t},w) \end{align}$

### 학습 순서
1. 초기 state 받음 (env.reset())
2. action 선택 (e - greedy policy)
3. 선택한 action으로 다음 state로 이동 (env.step())
4. 다음 state와 reward를 이용해 update식 작성
5. 신경망 업데이트
6. 반복

## Tensorflow 코드 흐름
1. 각 연산자에 대한 그래프를 구성한다.
2. 실제 데이터를 그래프에 할당하면서 전체 그래프를 실행한다.

### Build graph

```python
# placeholder 선언
# state
inputs = tf.placeholder(shape=[1], dtype=tf.int64)
# state에 대한 action
input_action = tf.placeholder(shape=[1], dtype=tf.int64)
# Loss 식의 target
target = tf.placeholder(shape=[1], dtype=tf.float32)

layers = tf.contrib.layers

# 신경망 구성 함수
# one-hot vector : 입력 1을 단순한 숫자로 받는 것보다 [1, 0, 0, 0] 처럼 encoding된 값으로 바꾸어 받는 것이
#                       학습에 유리하다. 모든 입력이 크기에 관계없이 동등해지게 된다.
# tf.one_hot( 입력, one-hot size )
def build_network(inputs):
    with tf.variable_scope('q_net'):
        # 빈칸 {} 을 지우고 채워주세요.
        input_onehot = tf.one_hot({}, {}, dtype=tf.float32)
        fc1 = layers.fully_connected(inputs={},
                                     num_outputs={},
                                     activation_fn=None)
    return fc1

# 신경망 구성
q_value = build_network(inputs)

# 현재 action에 대한 Q_value 구하는 연산
# q_value = [1, 2, 3, 4] curr_action = [0, 1, 0, 0] --(원소 곱)--> [0, 2, 0, 0] --(sum)--> [2]
curr_action = tf.one_hot({}, {})
curr_q_value = tf.reduce_sum(tf.multiply({}, {}))

# Loss 함수 구성
# 직접 구현해보세요. ( learning_rate = 0.1 )
# 참고) 제곱 : tf.square()
#          optimizer : tf.train.GradientDescentOptimizer( learning_rate )
loss_op =
opt = tf.train.GradientDescentOptimizer(learning_rate=0.1)
train_op = opt.minimize(loss_op)
```

### Executing a graph in a tf.Session

```python
# Session 열기
tf.reset_default_graph()
sess = tf.Session()

# 변수 초기화
sess.run(tf.global_variables_initializer())

# 변수(파라미터) 확인
for var in tf.trainable_variables('q_net'):
    print(var)
    print(sess.run(var))
```

### Action select using Q value

```python
# 초기 state
state = env.reset()
state = np.reshape(state, [1])
print("Current state:", state)

# 현재 state에 대한 Q-value
# 빈칸 {}을 채워보세요.
# 참고) sess.run( "Q-value를 구하는 신경망 그래프", feed_dict={inputs: "신경망 입력"} )
curr_q = sess.run({}, feed_dict={inputs: {} })
print("Q value of current state:", curr_q)
```

```python
# action 선택 ( greedy )
action = sess.run({}, feed_dict={})
# 직접 구현해보세요. (hint. np.argmax)

print("Choose action: ", action)
```

### 선택한 Action으로 다음 State, Reward 받기

```python
# action을 이용해 env.step하기
# 빈칸 {} 을 채워보세요.
next_state, reward, done, _ = env.step({})
next_state = np.reshape(next_state, [1])
print("next_state", next_state)
```

### update를 위한 (미래)보상 값(target) 계산

```python
gamma = 0.9
if done: # terminal state 이면 reward가 곧 next_q_value
    next_q_value =
else:
    # 직접 작성해보세요.
    # 위 수식 참고.
    # 참고) R + gamma * next state의 q-value 중 max
    next_q_value =
print("next_q_value", next_q_value)
```

### Update Neural Net

```python
action = np.reshape(action, [1])

# train_op를 sess.run 하여 학습 실행.
# 빈칸 {} 을 채워보세요.
loss, _ = sess.run([loss_op, train_op], feed_dict={inputs: {}, target: {}, input_action: {}})
print("loss", loss)

for var in tf.trainable_variables('q_net'):
    print(var)
    print(sess.run(var))
```

### 학습 시작

```python
rlist = []
slist = []
epsilon_list = []

EPISODE = 2000
gamma = 0.99

update_count = 0
loss_list = []

# Episode 수만큼 반복
for step in range(EPISODE):
    # step마다 epsilon 감소
    epsilon = 1. / ((step/50)+10)
    epsilon_list.append(epsilon)

    # 초기 state
    state = env.reset()
    state = np.reshape(state, [1])
    print("[Episode {}]".format(step))
    total_reward = 0
    limit = 0
    done = False

    while not done and limit < 99:
        # 위에서 했던 코드를 참조하여 아래 학습 코드를 작성해보세요.

        # e-greedy policy로 action 선택
        if epsilon > np.random.random():
            # random
            action = env.action_space.sample()
        else:
            # greedy
            # 현재 state의 Q value의 argmax값을 sess.run().
            curr_q =
            action =

        # 선택한 action으로 env.step 하기
        next_state, reward, done, _ =
        next_state = np.reshape(next_state, [1])

        if reward == 1.0:
            print("GOAL")

        # 업데이트를 위한 (미래)보상값 반환
        # episode가 끝났다면
        if done:
            next_q_value =
        # 끝나지 않았다면
        else:
            next_q_value =

        # Q update
        action = np.reshape(action, [1])
        loss, _ =

        loss_list.append(loss)
        update_count += 1

        slist.append(state.item())
        state = next_state
        total_reward += reward
        limit += 1

    print(slist)
    slist = []
    print("total reward: ", total_reward)
    rlist.append(total_reward)

print("Success Prob" + str(sum(rlist) / EPISODE) + "%")
```

```python
for var in tf.trainable_variables('q_net'):
    print(var)
    print(sess.run(var))
```

```python
# epsilon 변화 그래프
steps = np.arange(EPISODE)
plt.title('Epsilon values')
plt.xlabel('Timestep')
plt.ylabel('$\\epsilon$')
plt.plot(steps, epsilon_list)
```

```python
# loss 변화 그래프
update_count = np.arange(update_count)
plt.title('Loss values')
plt.xlabel('Update Count')
plt.ylabel('Loss')
plt.plot(update_count, loss_list)
```

### Test agent

```python
state = env.reset()
state = np.reshape(state, [1])
done = False
limit = 0

epsilon = 0.0
while not done and limit < 30:
    # 학습된 신경망을 테스트하는 코드를 작성해보세요.
    curr_q =
    action =
    next_state, reward, done, _ =
    next_state = np.reshape(next_state, [1])

    env.render()
    state = next_state
    limit += 1
```

```python

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

    c:\users\darky\appdata\local\programs\python\python36\lib\site-packages\IPython\core\magics\pylab.py:160: UserWarning: pylab import has clobbered these variables: ['solve']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

```python
from sympy import symbols, sympify, latex, integrate, solve, solveset, Matrix, expand, factor, primitive, simplify, factor_list
from sympy.parsing.sympy_parser import parse_expr
```

```python
M = np.array(Matrix(np.random.normal(0, 3, 500))).astype('float')
N = np.array(Matrix(np.random.normal(10, 3, 500))).astype('float')
```

```python
fig = plt.figure(figsize=(8,8))
ax = plt.axes()
plt.tight_layout()
boxplot(M, positions=[1], boxprops={'color': 'purple', 'linewidth': 2}, medianprops={'color': 'purple', 'linewidth': 2}, widths=0.6)
boxplot(N, positions=[3], medianprops={'color': 'purple', 'linewidth': 2}, widths=0.6)
```

```python
plt.figure(figsize=(8,8))
hist(M, fc='purple', alpha=0.5, bins=30)
hist(N, fc='salmon', alpha=0.5, bins=30)
#grid()
```

```python
np.random.normal(0, 1, 10)
```

    array([-0.34418066,  0.48756486, -0.51190391,  0.32712884,  1.3299908 ,
            0.85261622, -0.63709424, -0.18202305,  1.35780287,  0.64367802])

```python
plt.figure(figsize=(8,8))
scatter(N,M,c='purple',lw=1)
axhline(np.mean(M),ls='dashed', c='gray', lw=2, alpha=0.5)
axvline(np.mean(N), ls='dashed', c='gray', lw=2, alpha=0.5)
title('Scatterplot of N~M')
```

```python

```


#  Neuromatch Academy: Week 1, Day 1, Tutorial 3
# Model Types: "Why" models
__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom

We would like to acknowledge [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x) for sharing their data, a subset of which is used here.

___
# Tutorial Objectives
This is tutorial 3 of a 3-part series on different flavors of models used to understand neural data. In parts 1 and 2 we explored mechanisms that would produce the data. In this tutorial we will explore models and techniques that can potentially explain *why* the spiking data we have observed is produced the way it is.

To understand why different spiking behaviors may be beneficial, we will learn about the concept of entropy. Specifically, we will:

- Write code to compute formula for entropy, a measure of information
- Compute the entropy of a number of toy distributions
- Compute the entropy of spiking activity from the Steinmetz dataset

```python
#@title Video 1: “Why” models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='OOIDEr1e5Gg', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper Functions

def plot_pmf(pmf, isi_range):
    """Plot the probability mass function."""
    ymax = max(0.2, 1.05 * np.max(pmf))
    pmf_ = np.insert(pmf, 0, pmf[0])
    plt.plot(bins, pmf_, drawstyle="steps")
    plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
    plt.title(f"Neuron {neuron_idx}")
    plt.xlabel("Inter-spike interval (s)")
    plt.ylabel("Probability mass")
    plt.xlim(isi_range)
    plt.ylim([0, ymax])
```

```python
# @title Download Data
import io
import requests
r = requests.get('https://osf.io/sy5xt/download')
if r.status_code != 200:
    print('Could not download data')
else:
    steinmetz_spikes = np.load(io.BytesIO(r.content),
                               allow_pickle=True)['spike_times']
```

# Section 1: Optimization and Information

Neurons can only fire so often in a fixed period of time, as the act of emitting a spike consumes energy that is depleted and must eventually be replenished. To communicate effectively for downstream computation, the neuron would need to make good use of its limited spiking capability. This becomes an optimization problem:

What is the optimal way for a neuron to fire in order to maximize its ability to communicate information?

In order to explore this question, we first need to have a quantifiable measure for information. Shannon introduced the concept of entropy to do just that, and defined it as

\begin{align}
  H_b(X) &= -\sum_{x\in X} p(x) \log_b p(x)
\end{align}

where $H$ is entropy measured in units of base $b$ and $p(x)$ is the probability of observing the event $x$ from the set of all possible events in $X$. See the Appendix for a more detailed look at how this equation was derived.

The most common base of measuring entropy is $b=2$, so we often talk about *bits* of information, though other bases are used as well e.g. when $b=e$ we call the units *nats*.

First, let's explore how entropy changes between some simple discrete probability distributions. In the rest of this tutorial we will refer to these as probability mass functions (PMF), where $p(x_i)$ equals the $i^{th}$ value in an array, and mass refers to how much of the distribution is contained at that value.

For our first PMF, we will choose one where all of the probability mass is located in the middle of the distribution.

```python
n_bins = 50  # number of points supporting the distribution
x_range = (0, 1)  # will be subdivided evenly into bins corresponding to points

bins = np.linspace(*x_range, n_bins + 1)  # bin edges

pmf = np.zeros(n_bins)
pmf[len(pmf) // 2] = 1.0  # middle point has all the mass

# Since we already have a PMF, rather than un-binned samples, `plt.hist` is not
# suitable. Instead, we directly plot the PMF as a step function to visualize
# the histogram:
pmf_ = np.insert(pmf, 0, pmf[0])  # this is necessary to align plot steps with bin edges
plt.plot(bins, pmf_, drawstyle="steps")
# `fill_between` provides area shading
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

If we were to draw a sample from this distribution, we know exactly what we would get every time. Distributions where all the mass is concentrated on a single event are known as *deterministic*.

How much entropy is contained in a deterministic distribution?

## Exercise 1: Computing Entropy

Your first exercise is to implement a method that computes the entropy of a discrete probability distribution, given its mass function. Remember that we are interested in entropy in units of _bits_, so be sure to use the correct log function.

Recall that $\log(0)$ is undefined. When evaluated at $0$, NumPy log functions (such as `np.log2`) return `np.nan` ("Not a Number"). By convention, these undefined terms— which correspond to points in the distribution with zero mass—are excluded from the sum that computes the entropy.

```python
def entropy(pmf):
    """Given a discrete distribution, return the Shannon entropy in bits.

    This is a measure of information in the distribution. For a totally
    deterministic distribution, where samples are always found in the same bin,
    then samples from the distribution give no more information and the entropy
    is 0.

    For now this assumes `pmf` arrives as a well-formed distribution (that is,
    `np.sum(pmf)==1` and `not np.any(pmf < 0)`)

    Args:
      pmf (np.ndarray): The probability mass function for a discrete distribution
        represented as an array of probabilities.
    Returns:
      h (number): The entropy of the distribution in `pmf`.

    """
    # reduce to non-zero entries to avoid an error from log2(0)
    pmf = pmf[pmf != 0]

    # implement the equation for Shannon entropy (in bits)
    h = -np.sum(pmf * np.log2(pmf))

    # return the absolute value (avoids getting a -0 result)
    return np.abs(h)

# Uncomment to test your entropy function
print(f"{entropy(pmf):.2f} bits")
```

    0.00 bits

```python
# to_remove solution
def entropy(pmf):
  """Given a discrete distribution, return the Shannon entropy in bits.

  This is a measure of information in the distribution. For a totally
  deterministic distribution, where samples are always found in the same bin,
  then samples from the distribution give no more information and the entropy
  is 0.

  For now this assumes `pmf` arrives as a well-formed distribution (that is,
  `np.sum(pmf)==1` and `not np.any(pmf < 0)`)

  Args:
    pmf (np.ndarray): The probability mass function for a discrete distribution
      represented as an array of probabilities.
  Returns:
    h (number): The entropy of the distribution in `pmf`.
  """
  # reduce to non-zero entries to avoid an error from log2(0)
  pmf = pmf[pmf > 0]

  # implement the equation for Shannon entropy (in bits)
  h = -np.sum(pmf * np.log2(pmf))

  # return the absolute value (avoids getting a -0 result)
  return np.abs(h)

print(f"{entropy(pmf):.2f} bits")
```

We expect zero surprise from a deterministic distribution. If we had done this calculation by hand, it would simply be

$-1\log_2 1 = -0=0$

Note that changing the location of the peak (i.e. the point and bin on which all the mass rests) doesn't alter the entropy. The entropy is about how predictable a sample is with respect to a distribution. A single peak is deterministic regardless of which point it sits on.

```python
pmf = np.zeros(n_bins)
pmf[2] = 1.0  # arbitrary point has all the mass

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

What about a distribution with mass split equally between two points?

```python
pmf = np.zeros(n_bins)
pmf[len(pmf) // 3] = 0.5
pmf[2 * len(pmf) // 3] = 0.5

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);

print(f"{entropy(pmf):.2f} bits")
```

Here, the entropy calculation is

$-(0.5 \log_2 0.5 + 0.5\log_2 0.5)=1$

There is 1 bit of entropy. This means that before we take a random sample, there is 1 bit of uncertainty about which point in the distribution the sample will fall on: it will either be the first peak or the second one.

Likewise, if we make one of the peaks taller (i.e. its point holds more of the probability mass) and the other one shorter, the entropy will decrease because of the increased certainty that the sample will fall on one point and not the other:

$-(0.2 \log_2 0.2 + 0.8\log_2 0.8)\approx 0.72$

Try changing the definition of the number and weighting of peaks, and see how the entropy varies.

If we split the probability mass among even more points, the entropy continues to increase. Let's derive the general form for $N$ points of equal mass, where $p_i=p=1/N$:

\begin{align}
 -\sum_i p_i \log_b p_i&= -\sum_i^N \frac{1}{N} \log_b \frac{1}{N}\\
                       &= -\log_b \frac{1}{N} \\
                       &= \log_b N
\end{align}
$$$$

If we have $N$ discrete points, the _uniform distribution_ (where all points have equal mass) is the distribution with the highest entropy: $\log_b N$. This upper bound on entropy is useful when considering binning strategies, as any estimate of entropy over $N$ discrete points (or bins) must be in the interval $[0, \log_b N]$.

```python
pmf = np.ones(n_bins) / n_bins  # [1/N] * N

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range);
plt.ylim(0, 1);

print(f"{entropy(pmf):.2f} bits")
```

Here, there are 50 points and the entropy of the uniform distribution is $\log_2 50\approx 5.64$. If we construct _any_ discrete distribution $X$ over 50 points (or bins) and calculate an entropy of $H_2(X)>\log_2 50$, something must be wrong with our implementation of the discrete entropy computation.

# Section 2: Information, neurons, and spikes

```python
#@title Video 2: Entropy of different distributions
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='o6nyrx3KH20', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

Recall the discussion of spike times and inter-spike intervals (ISIs) from Tutorial 1. What does the information content (or distributional entropy) of these measures say about our theory of nervous systems?

We'll consider three hypothetical neurons that all have the same mean ISI, but with different distributions:

1. Deterministic
2. Uniform
3. Exponential

Fixing the mean of the ISI distribution is equivalent to fixing its inverse: the neuron's mean firing rate. If a neuron has a fixed energy budget and each of its spikes has the same energy cost, then by fixing the mean firing rate, we are normalizing for energy expenditure. This provides a basis for comparing the entropy of different ISI distributions. In other words: if our neuron has a fixed budget, what ISI distribution should it express (all else being equal) to maximize the information content of its outputs?

Let's construct our three distributions and see how their entropies differ.

```python
n_bins = 50
# the mean isi is matched for each distribution!
mean_isi = 0.025
isi_range = (0, 0.25)

bins = np.linspace(*isi_range, n_bins + 1)
mean_idx = np.searchsorted(bins, mean_isi)

# 1. all mass concentrated on the ISI mean
pmf_single = np.zeros(n_bins)
pmf_single[mean_idx] = 1.0

# 2. mass uniformly distributed about the ISI mean
pmf_uniform = np.zeros(n_bins)
# divide by the index range, chosen to be centered on mean_idx
# if this range is larger, it increases the entropy
# however this also increases the mean_isi
pmf_uniform[0:2*mean_idx] = 1 / (2 * mean_idx)

# 3. mass exponentially distributed about the ISI mean
pmf_exp = stats.expon.pdf(bins[1:], scale=mean_isi)
# normalize
pmf_exp /= np.sum(pmf_exp)
```

## Regarding the width of the uniform distribution increasing the entropy

Fixing the firing rate is equivalent to fixing the spike rate, which does define the energy budget needed. However you could argue that widening this distribution (in the uniform case) simply reduces the probability of short isi's and allows for longer and longer isi's. *Intuitively, this would reduce the energy cost, while increasing the entropy*. However, this would have a trade off with time taken per isi, perhaps reducing information sent per time unit? The exponential however, prefers short isi's, and has fewer long isi's. This could potentially provide a rate advantage?

### Noticed the source of confusion above:
The uniform distribution chosen has the same average firing rate as the other two distributions, so for a fixed firing rate, the entropy is higher!
Two constraints, time and entropy, doesn't make sense to have an infinitely long uniform distribution! This increase the length of the isi.

```python
# @title
# @markdown Run this cell to plot the three PMFs
fig, axes = plt.subplots(ncols=3, figsize=(18, 5))

dists = [  # (subplot title, pmf, ylim)
    ("(1) Deterministic", pmf_single, (0, 1.05)),
    ("(1) Uniform", pmf_uniform, (0, 1.05)),
    ("(1) Exponential", pmf_exp, (0, 1.05))
]

for ax, (label, pmf_, ylim) in zip(axes, dists):
    pmf_ = np.insert(pmf_, 0, pmf_[0])
    ax.plot(bins, pmf_, drawstyle="steps")
    ax.fill_between(bins, pmf_, step="pre", alpha=0.4)
    ax.set_title(label)
    ax.set_xlabel("Inter-spike interval (s)")
    ax.set_ylabel("Probability mass")
    ax.set_xlim(isi_range)
    ax.set_ylim(ylim)
```

```python
print(
  f"Deterministic: {entropy(pmf_single):.2f} bits",
  f"Uniform: {entropy(pmf_uniform):.2f} bits",
  f"Exponential: {entropy(pmf_exp):.2f} bits",
  sep="\n",
)
```

    Deterministic: 0.00 bits
    Uniform: 3.32 bits
    Exponential: 3.77 bits

```python
#@title Video 3: Probabilities from histogram
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='e2U_-07O9jo', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

In the previous example we created the PMFs by hand to illustrate idealized scenarios. How would we compute them from data recorded from actual neurons?

One way is to convert the ISI histograms we've previously computed into discrete probability distributions using the following equation:

\begin{align}
p_i = \frac{n_i}{\sum\nolimits_{i}n_i}
\end{align}

where $p_i$ is the probability of an ISI falling within a particular interval $i$ and $n_i$ is the count of how many ISIs were observed in that interval.

### Exercise 2: Probabilty Mass Function

Your second exercise is to implement a method that will produce a probability mass function from an array of ISI bin counts.

To verify your solution, we will compute the probability distribution of ISIs from real neural data taken from the Steinmetz dataset.

```python
neuron_idx = 283

isi = np.diff(steinmetz_spikes[neuron_idx])
bins = np.linspace(*isi_range, n_bins + 1)
counts, _ = np.histogram(isi, bins)
```

```python
def pmf_from_counts(counts):
    """Given counts, normalize by the total to estimate probabilities."""

    pmf = counts/np.sum(counts)

    return pmf

# Uncomment when ready to test your function
pmf = pmf_from_counts(counts)
plot_pmf(pmf,isi_range)
```

```python
# to_remove solution
def pmf_from_counts(counts):
    """Given counts, normalize by the total to estimate probabilities."""
    pmf = counts / np.sum(counts)
    return pmf

pmf = pmf_from_counts(counts)
with plt.xkcd():
    plot_pmf(pmf, isi_range)
```

# Section 3: Calculate entropy from a PMF

```python
#@title Video 4: Calculating entropy from pmf
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='Xjy-jj-6Oz0', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

Now that we have the probability distribution for the actual neuron spiking activity, we can calculate its entropy.

```python
print(f"Entropy for Neuron {neuron_idx}: {entropy(pmf):.2f} bits")
```

## Interactive Demo: Entropy of neurons

We can combine the above distribution plot and entropy calculation with an interactive widget to explore how the different neurons in the dataset vary in spiking activity and relative information. Note that the mean firing rate across neurons is not fixed, so some neurons with a uniform ISI distribution may have higher entropy than neurons with a more exponential distribution.

```python
# @title
# @markdown **Run the cell** to enable the sliders.

def _pmf_from_counts(counts):
    """Given counts, normalize by the total to estimate probabilities."""
    pmf = counts / np.sum(counts)
    return pmf

def _entropy(pmf):
    """Given a discrete distribution, return the Shannon entropy in bits."""
    # remove non-zero entries to avoid an error from log2(0)
    pmf = pmf[pmf > 0]
    h = -np.sum(pmf * np.log2(pmf))
    # absolute value applied to avoid getting a -0 result
    return np.abs(h)

@widgets.interact(neuron=widgets.IntSlider(0,
                                           min=0,
                                           max=(len(steinmetz_spikes) - 1)))
def steinmetz_pmf(neuron):
    """ Given a neuron from the Steinmetz data, compute its PMF and entropy """
    isi = np.diff(steinmetz_spikes[neuron])
    bins = np.linspace(*isi_range, n_bins + 1)
    counts, _ = np.histogram(isi, bins)
    pmf = _pmf_from_counts(counts)

    plot_pmf(pmf, isi_range)
    plt.title(f"Neuron {neuron}: H = {_entropy(pmf):.2f} bits")
```

    interactive(children=(IntSlider(value=0, description='neuron', max=733), Output()), _dom_classes=('widget-inte…

---
# Summary

```python
#@title Video 5: Summary of model types
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='X4K2RR5qBK8', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

Congratulations! You've finished your first NMA tutorial. In this 3 part tutorial series, we used different types of models to understand the spiking behavior of neurons recorded in the Steinmetz data set.

 - We used "what" models to discover that the ISI distribution of real neurons is closest to an exponential distribution
 - We used "how" models to discover that balanced excitatory and inbhitiory inputs, coupled with a leaky membrane, can give rise to neuronal spiking with exhibiting such an exponential ISI distribution
 - We used "why" models to discover that exponential ISI distributions contain the most information when the mean spiking is constrained

# Bonus

### The foundations for Entropy

In his foundational [1948 paper](https://en.wikipedia.org/wiki/A_Mathematical_Theory_of_Communication) on information theory, Claude Shannon began with three criteria for a function $H$ defining the entropy of a discrete distribution of probability masses $p_i\in p(X)$ over the points $x_i\in X$:
1. $H$ should be continuous in the $p_i$.
  - That is, $H$ should change smoothly in response to smooth changes to the mass $p_i$ on each point $x_i$.
2. If all the points have equal shares of the probability mass, $p_i=1/N$, $H$ should be a non-decreasing function of $N$.
  - That is, if $X_N$ is the support with $N$ discrete points and $p(x\in X_N)$ assigns constant mass to each point, then $H(X_1) < H(X_2) < H(X_3) < \dots$
3. $H$ should be preserved by (invariant to) the equivalent (de)composition of distributions.
  - For example (from Shannon's paper) if we have a discrete distribution over three points with masses $(\frac{1}{2},\frac{1}{3},\frac{1}{6})$, then their entropy can be represented in terms of a direct choice between the three and calculated $H(\frac{1}{2},\frac{1}{3},\frac{1}{6})$. However, it could also be represented in terms of a series of two choices:
    1. either we sample the point with mass $1/2$ or not (_not_ is the other $1/2$, whose subdivisions are not given in the first choice),
    2. if (with probability $1/2$) we _don't_ sample the first point, we sample one of the two remaining points, masses $1/3$ and $1/6$.

    Thus in this case we require that $H(\frac{1}{2},\frac{1}{3},\frac{1}{6})=H(\frac{1}{2},\frac{1}{2}) + \frac{1}{2}H(\frac{1}{3}, \frac{1}{6})$

There is a unique function (up to a linear scaling factor) which satisfies these 3 requirements:

\begin{align}
  H_b(X) &= -\sum_{x\in X} p(x) \log_b p(x)
\end{align}

Where the base of the logarithm $b>1$ controls the units of entropy. The two most common cases are $b=2$ for units of _bits_, and $b=e$ for _nats_.

We can view this function as the expectation of the self-information over a distribution:

$$H_b(X) = \mathbb{E}_{x\in X} \left[I_b(x)\right]$$

$$I_b(x)=-\log_b p(x)$$

Self-information is just the negative logarithm of probability, and is a measure of how surprising an event sampled from the distribution would be. Events with $p(x)=1$ are certain to occur, and their self-information is zero (as is the entropy of the distribution they compose) meaning they are totally unsurprising. The smaller the probability of an event, the higher its self-information, and the more surprising the event would be to observe.