# 3. Imagined movement

In this tutorial we will look at imagined movement. Our movement is controlled in the motor cortex where there is an increased level of mu activity (8–12 Hz) when we perform movements. This is accompanied by a reduction of this mu activity in specific regions that deal with the limb that is currently moving. This decrease is called Event Related Desynchronization (ERD). By measuring the amount of mu activity at different locations on the motor cortex, we can determine which limb the subject is moving. Through mirror neurons, this effect also occurs when the subject is not actually moving his limbs, but merely imagining it.

## Credits
The CSP code was originally written by Boris Reuderink of the Donders
Institute for Brain, Cognition and Behavior. It is part of his Python EEG
toolbox: https://github.com/breuderink/eegtools

Inspiration for this tutorial also came from the excellent code example
given in the book chapter:

Arnaud Delorme, Christian Kothe, Andrey Vankov, Nima Bigdely-Shamlo,
Robert Oostenveld, Thorsten Zander, and Scott Makeig. MATLAB-Based Tools
for BCI Research, In _(B+H)CI: The Human in Brain-Computer Interfaces and
the Brain in Human-Computer Interaction._ Desney S. Tan and Anton Nijholt
(eds.), 2009, 241-259, http://dx.doi.org/10.1007/978-1-84996-272-8

## About the data
The dataset for this tutorial is provided by the fourth BCI competition.
Before you continue with this tutorial, please, go to http://www.bbci.de/competition/iv/#download
and fill in your name and email address.
You don't need to actually download the data, since it is installed in the virtual server already, but this way, the wonderful organizers of the competition get notified that their data is still being used and by whom.

[Description of the data](http://bbci.de/competition/iv/desc_1.html)

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
!dir data\
```

     Volume in drive C is OS
     Volume Serial Number is A0B9-791B

     Directory of C:\Users\wmvan\projects\neuroscience_tutorials\eeg-bci\data

    19/08/2019  18:57    <DIR>          .
    19/08/2019  18:57    <DIR>          ..
    19/08/2019  18:57        22.482.408 BCICIV_calib_ds1d.mat
    19/08/2019  18:14        15.491.346 tutorial1-01.mat
    19/08/2019  18:17           837.635 tutorial2-01.mat
    19/08/2019  18:20        66.913.197 tutorial4-01.mat
                   4 File(s)    105.724.586 bytes
                   2 Dir(s)  338.756.145.152 bytes free

Executing the code cell below will load the data:

```python
import numpy as np
import scipy.io

m = scipy.io.loadmat('data/BCICIV_calib_ds1d.mat', struct_as_record=True)

# SciPy.io.loadmat does not deal well with Matlab structures, resulting in lots of
# extra dimensions in the arrays. This makes the code a bit more cluttered

sample_rate = m['nfo']['fs'][0][0][0][0]
EEG = m['cnt'].T
nchannels, nsamples = EEG.shape

channel_names = [s[0] for s in m['nfo']['clab'][0][0][0]]
event_onsets = m['mrk'][0][0][0]
event_codes = m['mrk'][0][0][1]
labels = np.zeros((1, nsamples), int)
labels[0, event_onsets] = event_codes

cl_lab = [s[0] for s in m['nfo']['classes'][0][0][0]]
cl1 = cl_lab[0]
cl2 = cl_lab[1]
nclasses = len(cl_lab)
nevents = len(event_onsets)
```

Now we have the data in the following python variables:

```python
# Print some information
print('Shape of EEG:', EEG.shape)
print('Sample rate:', sample_rate)
print('Number of channels:', nchannels)
print('Channel names:', channel_names)
print('Number of events:', len(event_onsets))
print('Event codes:', np.unique(event_codes))
print('Class labels:', cl_lab)
print('Number of classes:', nclasses)
```

    Shape of EEG: (59, 190473)
    Sample rate: 100
    Number of channels: 59
    Channel names: ['AF3', 'AF4', 'F5', 'F3', 'F1', 'Fz', 'F2', 'F4', 'F6', 'FC5', 'FC3', 'FC1', 'FCz', 'FC2', 'FC4', 'FC6', 'CFC7', 'CFC5', 'CFC3', 'CFC1', 'CFC2', 'CFC4', 'CFC6', 'CFC8', 'T7', 'C5', 'C3', 'C1', 'Cz', 'C2', 'C4', 'C6', 'T8', 'CCP7', 'CCP5', 'CCP3', 'CCP1', 'CCP2', 'CCP4', 'CCP6', 'CCP8', 'CP5', 'CP3', 'CP1', 'CPz', 'CP2', 'CP4', 'CP6', 'P5', 'P3', 'P1', 'Pz', 'P2', 'P4', 'P6', 'PO1', 'PO2', 'O1', 'O2']
    Number of events: 1
    Event codes: [-1  1]
    Class labels: ['left', 'right']
    Number of classes: 2

This is a large recording: 59 electrodes where used, spread across the entire scalp. The subject was given a cue and then imagined either right hand movement or the movement of his feet. As can be seen from the [Homunculus](http://en.wikipedia.org/wiki/Cortical_homunculus), foot movement is controlled at the center of the motor cortex (which makes it hard to distinguish left from right foot), while hand movement is controlled more lateral.

## Plotting the data

The code below cuts trials for the two classes and should look familiar if you've completed the previous tutorials. Trials are cut in the interval [0.5–2.5 s] after the onset of the cue.

```python
# Dictionary to store the trials in, each class gets an entry
trials = {}

# The time window (in samples) to extract for each trial, here 0.5 -- 2.5 seconds
win = np.arange(int(0.5*sample_rate), int(2.5*sample_rate))

# Length of the time window
nsamples = len(win)

# Loop over the classes (right, foot)
for cl, code in zip(cl_lab, np.unique(event_codes)):

    # Extract the onsets for the class
    cl_onsets = event_onsets[event_codes == code]

    # Allocate memory for the trials
    trials[cl] = np.zeros((nchannels, nsamples, len(cl_onsets)))

    # Extract each trial
    for i, onset in enumerate(cl_onsets):
        trials[cl][:,:,i] = EEG[:, win+onset]

# Some information about the dimensionality of the data (channels x time x trials)
print('Shape of trials[cl1]:', trials[cl1].shape)
print('Shape of trials[cl2]:', trials[cl2].shape)
```

    Shape of trials[cl1]: (59, 200, 100)
    Shape of trials[cl2]: (59, 200, 100)

Since the feature we're looking for (a decrease in $\mu$-activity) is a frequency feature, lets plot the PSD of the trials in a similar manner as with the SSVEP data. The code below defines a function that computes the PSD for each trial (we're going to need it again later on):

```python
from matplotlib import mlab

def psd(trials):
    '''
    Calculates for each trial the Power Spectral Density (PSD).

    Parameters
    ----------
    trials : 3d-array (channels x samples x trials)
        The EEG signal

    Returns
    -------
    trial_PSD : 3d-array (channels x PSD x trials)
        the PSD for each trial.
    freqs : list of floats
        Yhe frequencies for which the PSD was computed (useful for plotting later)
    '''

    ntrials = trials.shape[2]
    trials_PSD = np.zeros((nchannels, 101, ntrials))

    # Iterate over trials and channels
    for trial in range(ntrials):
        for ch in range(nchannels):
            # Calculate the PSD
            (PSD, freqs) = mlab.psd(trials[ch,:,trial], NFFT=int(nsamples), Fs=sample_rate)
            trials_PSD[ch, :, trial] = PSD.ravel()

    return trials_PSD, freqs
```

```python
# Apply the function
psd_r, freqs = psd(trials[cl1])
psd_f, freqs = psd(trials[cl2])
trials_PSD = {cl1: psd_r, cl2: psd_f}
```

The function below plots the PSDs that are calculated with the above function. Since plotting it for 118 channels will clutter the display, it takes the indices of the desired channels as input, as well as some metadata to decorate the plot.

```python
import matplotlib.pyplot as plt

def plot_psd(trials_PSD, freqs, chan_ind, chan_lab=None, maxy=None):
    '''
    Plots PSD data calculated with psd().

    Parameters
    ----------
    trials : 3d-array
        The PSD data, as returned by psd()
    freqs : list of floats
        The frequencies for which the PSD is defined, as returned by psd()
    chan_ind : list of integers
        The indices of the channels to plot
    chan_lab : list of strings
        (optional) List of names for each channel
    maxy : float
        (optional) Limit the y-axis to this value
    '''
    plt.figure(figsize=(12,5))

    nchans = len(chan_ind)

    # Maximum of 3 plots per row
    nrows = np.ceil(nchans / 3)
    ncols = min(3, nchans)

    # Enumerate over the channels
    for i,ch in enumerate(chan_ind):
        # Figure out which subplot to draw to
        plt.subplot(nrows,ncols,i+1)

        # Plot the PSD for each class
        for cl in trials.keys():
            plt.plot(freqs, np.mean(trials_PSD[cl][ch,:,:], axis=1), label=cl)

        # All plot decoration below...

        plt.xlim(1,30)

        if maxy != None:
            plt.ylim(0,maxy)

        plt.grid()

        plt.xlabel('Frequency (Hz)')

        if chan_lab == None:
            plt.title('Channel %d' % (ch+1))
        else:
            plt.title(chan_lab[i])

        plt.legend()

    plt.tight_layout()
```

Lets put the `plot_psd()` function to use and plot three channels:

 1. C3: Central, left
 2. Cz: Central, central
 3. C4: Central, right

```python
plot_psd(
    trials_PSD,
    freqs,
    [channel_names.index(ch) for ch in ['C3', 'Cz', 'C4']],
    chan_lab=['left', 'center', 'right'],
    maxy=500
)
```

A spike of mu activity can be seen on each channel for both classes. At the right hemisphere, the mu for the left hand movement is lower than for the right hand movement due to the ERD. At the left electrode, the mu for the right hand movement is reduced and at the central electrode the mu activity is about equal for both classes. This is in line with the theory that the left hand is controlled by the right hemisphere and the feet are controlled centrally.

## Classifying the data

We will use a machine learning algorithm to construct a model that can distinguish between the right hand and foot movement of this subject. In order to do this we need to:

 1. find a way to quantify the amount of mu activity present in a trial
 2. make a model that describes expected values of mu activity for each class
 3. finally test this model on some unseen data to see if it can predict the correct class label

We will follow a classic BCI design by Blankertz et al. [1] where they use the logarithm of the variance of the signal in a certain frequency band as a feature for the classifier.

[1] Blankertz, B., Dornhege, G., Krauledat, M., Müller, K.-R., & Curio, G. (2007). The non-invasive Berlin Brain-Computer Interface: fast acquisition of effective performance in untrained subjects. *NeuroImage*, 37(2), 539–550. doi:10.1016/j.neuroimage.2007.01.051

The script below designs a band pass filter using [`scipy.signal.irrfilter`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.iirfilter.html) that will strip away frequencies outside the 8--15Hz window. The filter is applied to all trials:

```python
import scipy.signal

def bandpass(trials, lo, hi, sample_rate):
    '''
    Designs and applies a bandpass filter to the signal.

    Parameters
    ----------
    trials : 3d-array (channels x samples x trials)
        The EEGsignal
    lo : float
        Lower frequency bound (in Hz)
    hi : float
        Upper frequency bound (in Hz)
    sample_rate : float
        Sample rate of the signal (in Hz)

    Returns
    -------
    trials_filt : 3d-array (channels x samples x trials)
        The bandpassed signal
    '''

    # The iirfilter() function takes the filter order: higher numbers mean a sharper frequency cutoff,
    # but the resulting signal might be shifted in time, lower numbers mean a soft frequency cutoff,
    # but the resulting signal less distorted in time. It also takes the lower and upper frequency bounds
    # to pass, divided by the niquist frequency, which is the sample rate divided by 2:
    a, b = scipy.signal.iirfilter(6, [lo/(sample_rate/2.0), hi/(sample_rate/2.0)])

    # Applying the filter to each trial
    ntrials = trials.shape[2]
    trials_filt = np.zeros((nchannels, nsamples, ntrials))
    for i in range(ntrials):
        trials_filt[:,:,i] = scipy.signal.filtfilt(a, b, trials[:,:,i], axis=1)

    return trials_filt
```

```python
# Apply the function
trials_filt = {cl1: bandpass(trials[cl1], 8, 15, sample_rate),
               cl2: bandpass(trials[cl2], 8, 15, sample_rate)}
```

Plotting the PSD of the resulting `trials_filt` shows the suppression of frequencies outside the passband of the filter:

```python
psd_r, freqs = psd(trials_filt[cl1])
psd_f, freqs = psd(trials_filt[cl2])
trials_PSD = {cl1: psd_r, cl2: psd_f}

plot_psd(
    trials_PSD,
    freqs,
    [channel_names.index(ch) for ch in ['C3', 'Cz', 'C4']],
    chan_lab=['left', 'center', 'right'],
    maxy=300
)
```

As a feature for the classifier, we will use the logarithm of the variance of each channel. The function below calculates this:

```python
# Calculate the log(var) of the trials
def logvar(trials):
    '''
    Calculate the log-var of each channel.

    Parameters
    ----------
    trials : 3d-array (channels x samples x trials)
        The EEG signal.

    Returns
    -------
    logvar - 2d-array (channels x trials)
        For each channel the logvar of the signal
    '''
    return np.log(np.var(trials, axis=1))
```

```python
# Apply the function
trials_logvar = {cl1: logvar(trials_filt[cl1]),
                 cl2: logvar(trials_filt[cl2])}
```

Below is a function to visualize the logvar of each channel as a bar chart:

```python
def plot_logvar(trials):
    '''
    Plots the log-var of each channel/component.
    arguments:
        trials - Dictionary containing the trials (log-vars x trials) for 2 classes.
    '''
    plt.figure(figsize=(12,5))

    x0 = np.arange(nchannels)
    x1 = np.arange(nchannels) + 0.4

    y0 = np.mean(trials[cl1], axis=1)
    y1 = np.mean(trials[cl2], axis=1)

    plt.bar(x0, y0, width=0.5, color='b')
    plt.bar(x1, y1, width=0.4, color='r')

    plt.xlim(-0.5, nchannels+0.5)

    plt.gca().yaxis.grid(True)
    plt.title('log-var of each channel/component')
    plt.xlabel('channels/components')
    plt.ylabel('log-var')
    plt.legend(cl_lab)
```

```python
# Plot the log-vars
plot_logvar(trials_logvar)
```

We see that most channels show a small difference in the log-var of the signal between the two classes. The next step is to go from 118 channels to only a few channel mixtures. The CSP algorithm calculates mixtures of channels that are designed to maximize the difference in variation between two classes. These mixures are called spatial filters.

```python
from numpy import linalg

def cov(trials):
    ''' Calculate the covariance for each trial and return their average '''
    ntrials = trials.shape[2]
    covs = [ trials[:,:,i].dot(trials[:,:,i].T) / nsamples for i in range(ntrials) ]
    return np.mean(covs, axis=0)

def whitening(sigma):
    ''' Calculate a whitening matrix for covariance matrix sigma. '''
    U, l, _ = linalg.svd(sigma)
    return U.dot( np.diag(l ** -0.5) )

def csp(trials_r, trials_f):
    '''
    Calculate the CSP transformation matrix W.
    arguments:
        trials_r - Array (channels x samples x trials) containing right hand movement trials
        trials_f - Array (channels x samples x trials) containing foot movement trials
    returns:
        Mixing matrix W
    '''
    cov_r = cov(trials_r)
    cov_f = cov(trials_f)
    P = whitening(cov_r + cov_f)
    B, _, _ = linalg.svd( P.T.dot(cov_f).dot(P) )
    W = P.dot(B)
    return W

def apply_mix(W, trials):
    ''' Apply a mixing matrix to each trial (basically multiply W with the EEG signal matrix)'''
    ntrials = trials.shape[2]
    trials_csp = np.zeros((nchannels, nsamples, ntrials))
    for i in range(ntrials):
        trials_csp[:,:,i] = W.T.dot(trials[:,:,i])
    return trials_csp
```

```python
# Apply the functions
W = csp(trials_filt[cl1], trials_filt[cl2])
trials_csp = {cl1: apply_mix(W, trials_filt[cl1]),
              cl2: apply_mix(W, trials_filt[cl2])}
```

To see the result of the CSP algorithm, we plot the log-var like we did before:

```python
trials_logvar = {cl1: logvar(trials_csp[cl1]),
                 cl2: logvar(trials_csp[cl2])}
plot_logvar(trials_logvar)
```

Instead of 118 channels, we now have 118 mixtures of channels, called components. They are the result of 118 spatial filters applied to the data.

The first filters maximize the variation of the first class, while minimizing the variation of the second. The last filters maximize the variation of the second class, while minimizing the variation of the first.

This is also visible in a PSD plot. The code below plots the PSD for the first and last components as well as one in the middle:

```python
psd_r, freqs = psd(trials_csp[cl1])
psd_f, freqs = psd(trials_csp[cl2])
trials_PSD = {cl1: psd_r, cl2: psd_f}

plot_psd(trials_PSD, freqs, [0,28,-1], chan_lab=['first component', 'middle component', 'last component'], maxy=0.75 )
```

In order to see how well we can differentiate between the two classes, a scatter plot is a useful tool. Here both classes are plotted on a 2-dimensional plane: the x-axis is the first CSP component, the y-axis is the last.

```python
def plot_scatter(left, foot):
    plt.figure()
    plt.scatter(left[0,:], left[-1,:], color='b')
    plt.scatter(foot[0,:], foot[-1,:], color='r')
    plt.xlabel('Last component')
    plt.ylabel('First component')
    plt.legend(cl_lab)
```

```python
plot_scatter(trials_logvar[cl1], trials_logvar[cl2])
```

We will apply a linear classifier to this data. A linear classifier can be thought of as drawing a line in the above plot to separate the two classes. To determine the class for a new trial, we just check on which side of the line the trial would be if plotted as above.

The data is split into a train and a test set. The classifier will fit a model (in this case, a straight line) on the training set and use this model to make predictions about the test set (see on which side of the line each trial in the test set falls). Note that the CSP algorithm is part of the model, so for fairness sake it should be calculated using only the training data.

```python
# Percentage of trials to use for training (50-50 split here)
train_percentage = 0.5

# Calculate the number of trials for each class the above percentage boils down to
ntrain_r = int(trials_filt[cl1].shape[2] * train_percentage)
ntrain_f = int(trials_filt[cl2].shape[2] * train_percentage)
ntest_r = trials_filt[cl1].shape[2] - ntrain_r
ntest_f = trials_filt[cl2].shape[2] - ntrain_f

# Splitting the frequency filtered signal into a train and test set
train = {cl1: trials_filt[cl1][:,:,:ntrain_r],
         cl2: trials_filt[cl2][:,:,:ntrain_f]}

test = {cl1: trials_filt[cl1][:,:,ntrain_r:],
        cl2: trials_filt[cl2][:,:,ntrain_f:]}

# Train the CSP on the training set only
W = csp(train[cl1], train[cl2])

# Apply the CSP on both the training and test set
train[cl1] = apply_mix(W, train[cl1])
train[cl2] = apply_mix(W, train[cl2])
test[cl1] = apply_mix(W, test[cl1])
test[cl2] = apply_mix(W, test[cl2])

# Select only the first and last components for classification
comp = np.array([0,-1])
train[cl1] = train[cl1][comp,:,:]
train[cl2] = train[cl2][comp,:,:]
test[cl1] = test[cl1][comp,:,:]
test[cl2] = test[cl2][comp,:,:]

# Calculate the log-var
train[cl1] = logvar(train[cl1])
train[cl2] = logvar(train[cl2])
test[cl1] = logvar(test[cl1])
test[cl2] = logvar(test[cl2])
```

For a classifier the Linear Discriminant Analysis (LDA) algorithm will be used. It fits a gaussian distribution to each class, characterized by the mean and covariance, and determines an optimal separating plane to divide the two. This plane is defined as $r = W_0 \cdot X_0 + W_1 \cdot X_1 + \ldots + W_n \cdot X_n - b$, where $r$ is the classifier output, $W$ are called the feature weights, $X$ are the features of the trial, $n$ is the dimensionality of the data and $b$ is called the offset.

In our case we have 2 dimensional data, so the separating plane will be a line: $r = W_0 \cdot X_0 + W_1 \cdot X_1 - b$. To determine a class label for an unseen trial, we can calculate whether the result is positive or negative.

```python
def train_lda(class1, class2):
    '''
    Trains the LDA algorithm.
    arguments:
        class1 - An array (observations x features) for class 1
        class2 - An array (observations x features) for class 2
    returns:
        The projection matrix W
        The offset b
    '''
    nclasses = 2

    nclass1 = class1.shape[0]
    nclass2 = class2.shape[0]

    # Class priors: in this case, we have an equal number of training
    # examples for each class, so both priors are 0.5
    prior1 = nclass1 / float(nclass1 + nclass2)
    prior2 = nclass2 / float(nclass1 + nclass1)

    mean1 = np.mean(class1, axis=0)
    mean2 = np.mean(class2, axis=0)

    class1_centered = class1 - mean1
    class2_centered = class2 - mean2

    # Calculate the covariance between the features
    cov1 = class1_centered.T.dot(class1_centered) / (nclass1 - nclasses)
    cov2 = class2_centered.T.dot(class2_centered) / (nclass2 - nclasses)

    W = (mean2 - mean1).dot(np.linalg.pinv(prior1*cov1 + prior2*cov2))
    b = (prior1*mean1 + prior2*mean2).dot(W)

    return (W,b)

def apply_lda(test, W, b):
    '''
    Applies a previously trained LDA to new data.
    arguments:
        test - An array (features x trials) containing the data
        W    - The project matrix W as calculated by train_lda()
        b    - The offsets b as calculated by train_lda()
    returns:
        A list containing a classlabel for each trial
    '''
    ntrials = test.shape[1]

    prediction = []
    for i in range(ntrials):
        # The line below is a generalization for:
        # result = W[0] * test[0,i] + W[1] * test[1,i] - b
        result = W.dot(test[:,i]) - b
        if result <= 0:
            prediction.append(1)
        else:
            prediction.append(2)

    return np.array(prediction)
```

Training the LDA using the training data gives us $W$ and $b$:

```python
W,b = train_lda(train[cl1].T, train[cl2].T)

print('W:', W)
print('b:', b)
```

    W: [ 5.31347949 -5.52963938]
    b: 0.3802472099471048

It can be informative to recreate the scatter plot and overlay the decision boundary as determined by the LDA classifier. The decision boundary is the line for which the classifier output is exactly 0. The scatterplot used $X_0$ as $x$-axis and $X_1$ as $y$-axis. To find the function $y = f(x)$ describing the decision boundary, we set $r$ to 0 and solve for $y$ in the equation of the separating plane:

<div style="width:600px">
$$\begin{align}
W_0 \cdot X_0 + W_1 \cdot X_1 - b &= r &&\text{the original equation} \\\
W_0 \cdot x + W_1 \cdot y - b &= 0     &&\text{filling in $X_0=x$, $X_1=y$ and $r=0$} \\\
W_0 \cdot x + W_1 \cdot y &= b         &&\text{solving for $y$}\\\
W_1 \cdot y &= b - W_0 \cdot x \\\
\\\
y &= \frac{b - W_0 \cdot x}{W_1}
\end{align}$$
</div>

We first plot the decision boundary with the training data used to calculate it:

```python
# Scatterplot like before
plot_scatter(train[cl1], train[cl2])
title('Training data')

# Calculate decision boundary (x,y)
x = np.arange(-5, 1, 0.1)
y = (b - W[0]*x) / W[1]

# Plot the decision boundary
plt.plot(x,y, linestyle='--', linewidth=2, color='k')
plt.xlim(-5, 1)
plt.ylim(-2.2, 1)
```

The code below plots the boundary with the test data on which we will apply the classifier. You will see the classifier is going to make some mistakes.

```python
plot_scatter(test[cl1], test[cl2])
title('Test data')
plt.plot(x,y, linestyle='--', linewidth=2, color='k')
plt.xlim(-5, 1)
plt.ylim(-2.2, 1)
```

Now the LDA is constructed and fitted to the training data. We can now apply it to the test data. The results are presented as a confusion matrix:

<table>
    <tr><td></td><td colspan='2' style="font-weight:bold">True labels →</td></tr>
    <tr><td style="font-weight:bold">↓ Predicted labels</td><td>Right</td><td>Foot</td></tr>
    <tr><td>Right</td><td></td><td></td></tr>
    <tr><td>Foot</td><td></td><td></td></tr>
</table>

The number at the diagonal will be trials that were correctly classified, any trials incorrectly classified (either a false positive or false negative) will be in the corners.

```python
# Print confusion matrix
conf = np.array([
    [(apply_lda(test[cl1], W, b) == 1).sum(), (apply_lda(test[cl2], W, b) == 1).sum()],
    [(apply_lda(test[cl1], W, b) == 2).sum(), (apply_lda(test[cl2], W, b) == 2).sum()],
])

print('Confusion matrix:')
print(conf)
print()
print('Accuracy: %.3f' % (np.sum(np.diag(conf)) / float(np.sum(conf))))
```

    Confusion matrix:
    [[45  4]
     [ 5 46]]

    Accuracy: 0.910

The confusion matrix shows that 4 out of the 50 trials with foot movement were incorrectly classified as right hand movement and 5 out of the 50 trials with right hand movement were incorrectly classified as foot movement. In total, 91% of the trials were correctly classified, not a bad score!

If you want, you can continue with the next tutorial:
[4. Classifiying the P300](4.%20Classifying%20the%20P300.ipynb)


# Big Panda Dataframe

Panda knows how to do lots of things with big data sets, so figure out how to create a panda dataframe.

```python
%%capture
%matplotlib inline
import numpy as np
import sympy as sp
import pandas as pd
import matplotlib.pyplot as plt

# To get equations the look like, well, equations, use the following.
from sympy.interactive import printing
printing.init_printing(use_latex=True)
from IPython.display import display

# Tools for manipulating quaternions.
import Q_tools as qt;
```

The function range is a generator.

```python
qha = qt.QHArray()

qDf = pd.DataFrame(qha.range(q_start=qt.QH([0, 0, 0, 0]), q_delta=qt.QH([1, 0.1, 0.2, 0.3]), n_steps=1000))
qDf.tail(3)
```

      0

      998
      (998, 99.7999999999986, 199.5999999999972, 299...

      999
      (999, 99.8999999999986, 199.7999999999972, 299...

      1000
      (1000, 99.9999999999986, 199.9999999999972, 30...

Need to check if Panda wants array format for data

```python
qha = qt.QHaArray()

qDf = pd.DataFrame(qha.range(q_start=qt.QHa([0, 0, 0, 0]), q_delta=qt.QHa([1, 0.1, 0.2, 0.3]), n_steps=1000))
qDf.tail(3)
```

      0

      998
      (998.0, 99.7999999999986, 199.5999999999972, 2...

      999
      (999.0, 99.8999999999986, 199.7999999999972, 2...

      1000
      (1000.0, 99.9999999999986, 199.9999999999972, ...

```python

```


```python
j
```

```python
import sympy as sp
import numpy as np
import altair as alt
```

# Solving steady state expression for λ

```python
# Define the efficiencies
gamma_max = sp.Symbol('{{\gamma^{(max)}}}')
nu_max = sp.Symbol(r'{{\nu^{(max)}}}')
phi_R = sp.Symbol('{{\phi_R}}')
Kd = sp.Symbol('{{K_D^{(c_{AA})}}}')
mu = sp.Symbol('\mu')
cAA_symb = sp.Symbol('{{c_{AA}}}')
Gamma = gamma_max * phi_R
Nu = nu_max * (1  - phi_R)
cAA = -1 + (nu_max * (1 - phi_R)) / (mu)
eq = gamma_max * phi_R * cAA / (cAA + Kd)
mu_soln = sp.solve(eq - mu, mu)[1]
dmu_dphir = sp.diff(mu_soln, phi_R)
# phiR_soln = sp.solve(dmu_dphir, phi_R)[0]
cAA_eq = (nu_max * (1 - phi_R) / (gamma_max * phi_R)) * (1 + Kd / cAA_symb) - 1
cAA_soln = sp.solve(cAA_eq - cAA_symb, cAA_symb)[0]
# cAA_soln = ((Nu - Gamma) + sp.sqrt((Gamma - Nu)**2 + 4 * Gamma * Nu * Kd))/(2 * Gamma)
# mu_soln = (Gamma + Nu - sp.sqrt((Gamma + Nu)**2 - 4 * (1 - Kd) * Gamma * Nu)) / (2 * (1 - Kd))
# mu_phiR = Gamma * (1 + (Kd * mu) / (Nu - mu))**-1
# phiR_mu = sp.solve(mu_phiR - mu, phi_R)
# phiR_mu[0]
opt_phiR_soln = sp.solve(dmu_dphiR, phi_R)[0]
```

# Modeling ppGpp activity

```python

```

```python
gamma_max = sp.Symbol('{{\gamma_{max}}}')
nu_max = sp.Symbol('{{\nu_{max}}}')
Taa = sp.Symbol('{{T_{AA}}}')
Taa_star = sp.Symbol('{{T_{AA}^*}}')
Kd_Taa = sp.Symbol('{{K_D^{T_{AA}}}}')
Kd_Taa_star = sp.Symbol('{{K_D^{T_{AA}^*}}}')
tau = sp.Symbol('{{\tau}}')
phiR = sp.Symbol('{{\phi_R}}')
mu = sp.Symbol('{{\mu}}')
Taa_exp_a = 1
Taa_exp_b = 1 + Kd_Taa - (nu_max * (1 - phiR) / mu)
Taa_exp_c = Kd_Taa
Taa_exp = (-Taa_exp_b + sp.sqrt(Taa_exp_b**2 - 4 * Taa_exp_a * Taa_exp_c)) / (2 * Taa_exp_a)
Taa_star_exp = ((nu_max * (1 - phi_R) * Taa_exp) / (mu * (Taa_exp + Kd_Taa))) - 1
gamma = gamma_max * (Taa_star_exp / (Taa_star_exp + Kd_Taa_star))
tau_exp = Taa_star_exp / Taa_exp
fa = tau_exp / (tau_exp + tau)
mu_exp = gamma * fa * phiR
soln = sp.solve(mu - mu_exp, mu)
```

```python
mu_exp
```

$\displaystyle \frac{{{\gamma_{max}}} {{\phi_R}} \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right) \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} - \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}\right)}{{{\mu}} \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} + \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}\right)} - 1\right)^{2}}{\left({{	au}} + \frac{\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right) \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} - \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}\right)}{{{\mu}} \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} + \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}\right)} - 1}{\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} - \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}}\right) \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right) \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} - \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}\right)}{{{\mu}} \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} + \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}\right)} + {{K_D^{T_{AA}^*}}} - 1\right) \left(\frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{2 {{\mu}}} - \frac{{{K_D^{T_{AA}}}}}{2} + \frac{\sqrt{- 4 {{K_D^{T_{AA}}}} + \left(- \frac{{{
u_{max}}} \left(1 - {{\phi_R}}\right)}{{{\mu}}} + {{K_D^{T_{AA}}}} + 1\right)^{2}}}{2} - \frac{1}{2}\right)}$

```python
mu_exp.simplify()
```

```python

```


Markdown and $\LaTeX$
=====================

Markdown
========

## Jupyter Notebooks use the [Markdown](https://en.wikipedia.org/wiki/Markdown) language to render text in cells
Paragraphs are separated
by a blank line.

Two spaces at the end of a line
produces a line break.

> Markdown uses email-style
> characters for blockquoting.
Paragraphs are separated
by a blank line.

Two spaces at the end of a line
produces a line break.

> Markdown uses email-style
> characters for blockquoting.
# Main Heading
# Main Heading
## Sub-heading
### Sub-sub-heading
#### Sub-sub-sub-heading
##### Sub-sub-sub-sub-heading
## Sub-heading
### Sub-sub-heading
#### Sub-sub-sub-heading
##### Sub-sub-sub-sub-heading
Horizontal rule:

---
---
Text attributes: _italic_ , **bold**, `monospace`, and ~~strikethrough~~.
Text attributes: _italic_ , **bold**, `monospace`, and ~~strikethrough~~.
Bullet list:

  * john
  * paul
  * george
    * ringo
* john
* paul
* george
  * ringo
Numbered list (you need the "." after the number):

  1. john
  2. paul
  3. george
    1. ringo
Numbered list (you need the "." after the number):

  1. john
  2. paul
  3. george
    1. ringo
A web link:

[Words](url)

This is pretty much just taken from the [Wiki entry for Markdown](https://en.wikipedia.org/wiki/Markdown)
### This is all pretty much just taken from the [Wiki entry for Markdown](https://en.wikipedia.org/wiki/Markdown)

---

# ♡ $\LaTeX$ ♡

* $\LaTeX$ is based on the $\TeX$ typesetting language. The [$\TeX$](https://en.wikipedia.org/wiki/TeX) language has a long and deep history (1978).
* $\LaTeX$ is widely used in academia, especially in Astronomy.

<p align="center">

</p>

## Jupyter Notebooks use the $\LaTeX$ language to typeset equations.

* Actually, they use a subset of $\LaTeX$ called [MathJax](https://en.wikipedia.org/wiki/MathJax).
Most LaTeX commands are prefixed with a "\". For example \pi is the
command to produce the lower case Greek letter pi.

The characters  # $ % & ~ _ ^ \ { } are special characters in LaTeX. If
you want to typeset them you need to put a \ in front of them. For
example \$ will typeset the symbol $

Special case: if you want the "\" character use \backslash
----
$ - in-line math mode

The circumfrence of a circle is $2 \pi r$
The circumfrence of a circle is $2 \pi r$
$$ - display mode

$$
2 \pi r
$$
$$
2 \pi r
$$
Do not put any bank lines around the equation. This will not work:

$$

2 \pi r
$$
$$

2 \pi r
$$
% - LaTeX comment character

$$
% - nothing here
2 \pi r
% - and nothing here
$$
$$
% - nothing here
2 \pi r
% - and nothing here
$$
Superscript: ^

$$
\pi r^{2}
$$
$$
\pi r^{2}
$$
Subscript: _

$$
x_{2}
$$
$$
x_{2}
$$
Fractions: \frac{}{}

$$
\frac{2}{\pi}
$$
$$
\frac{2}{\pi}
$$
Nice looking delimiters

Do not use () or []

$$
( \frac{2}{\pi} )
$$
$$
( \frac{2}{\pi} )
$$
Use '\left(' and '\right)' or '\left[' '\right]'

$$
\left( \frac{2}{\pi} \right)
$$
$$
\left( \frac{2}{\pi} \right)
$$
Font size (from small to big): \tiny \Tiny \small \normalsize \large \Large \LARGE \huge \Huge$$
\huge
\left( \frac{2}{\pi} \right)
$$
$$
\huge
\left( \frac{2}{\pi} \right)
$$
$$
\large
x_{2}
$$
$$
\large
x_{2}
$$
$$
\large
x_{\small 2}
$$
$$
\large
x_{\small 2}
$$
Spacing can make equations look nicer

\, - really, really small space
\: - really small space
\; - small space
\  - text space (space after backslash!) - depends on fontsize
\quad - big space
\qquad - really big space
\! - (negative) really, really, small space$$
f(x)\! =\! x^{2}\! +\! 2 \\
f(x) = x^{2}+2 \\
f(x)\, =\, x^{2}\,+\,2 \\
f(x)\: =\: x^{2}\:+\:2 \\
f(x)\; =\; x^{2}\;+\;2 \\
f(x)\ =\ x^{2}\ +\ 2 \\
f(x)\quad =\quad x^{2}\quad +\quad 2 \\
f(x)\qquad =\qquad x^{2}\qquad +\qquad 2 \\
$$
$$
f(x)\! =\! x^{2}\! +\! 2 \\
f(x) = x^{2}+2 \\
f(x)\, =\, x^{2}\,+\,2 \\
f(x)\: =\: x^{2}\:+\:2 \\
f(x)\; =\; x^{2}\;+\;2 \\
f(x)\ =\ x^{2}\ +\ 2 \\
f(x)\quad =\quad x^{2}\quad +\quad 2 \\
f(x)\qquad =\qquad x^{2}\qquad +\qquad 2 \\
$$
$$
\large
\sqrt{2x}
$$
$$
\large
f(x)=\sqrt{2x}
$$
$$
\large
f(x) = \sqrt{\,2x\,}
$$
$$
\large
f(x) = \sqrt{\,2x\,}
$$
Aligned Equations

Use \\ to start a new line
Use & to mark where you want the equations to line up

\begin{align}
a_1 & = b_1 + c_1 \\
a_2 & = b_2 + c_2 + d_2
\end{align}
\begin{align}
a_1 & = b_1 + c_1 \\
a_2 & = b_2 + c_2 + d_2
\end{align}

### Matrices and Brackets

* The ampersand `&` is used to separate columns
* Use `\\` to end a row (except for the last row)
$$
\begin{matrix}
a & b \\
c & d
\end{matrix}
$$
$$
\begin{matrix}
a & b \\
c & d
\end{matrix}
$$
$$
\begin{pmatrix}
a & b \\
c & d
\end{matrix}
$$
$$
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix}
$$
$$
\begin{bmatrix}
a & b \\
c & d
\end{bmatrix}
$$
$$
\begin{bmatrix}
a & b \\
c & d
\end{bmatrix}
$$

---
### These few commands, plus a list of [Latex Symbols](http://oeis.org/wiki/List_of_LaTeX_mathematical_symbols), are all you need for most $\LaTeX$ equations

### If you cannot find a symbol, you can try the [Detexify Latex](http://detexify.kirelabs.org/classify.html) drawing tool

### Note: Jupyter notebooks only support a [small subset of the full $\LaTeX$ language](https://docs.mathjax.org/en/latest/input/tex/differences.html).

* Mostly just the Math stuff (`mathmode`)
* Only symbols that **do not** require external packages `\usepackage{}`
* Here is the [Full list of supported commands](http://www.onemathematicalcat.org/MathJaxDocumentation/TeXSyntax.htm)
* You will definitely find some symbols that will not render in a notebook!

----

## Writing $\LaTeX$ equations from scratch can be a pain. There are some ways to make it easier:

### Use can use `SymPy` to make $\LaTeX$ equations for you!

```python
import sympy as sp
```

```python
z = sp.symbols('z')
```

```python
my_equation =  1/( ( z + 2 ) * ( z + 1 ) )

print(sp.latex(my_equation))
```

    \frac{1}{\left(z + 1\right) \left(z + 2\right)}

$$
\frac{1}{\left(z + 1\right) \left(z + 2\right)}
$$
$$
\frac{1}{\left(z + 1\right) \left(z + 2\right)}
$$

----
## My favorite trick is the Wikipedia $\LaTeX$ equation cheat:

* #### Navigate over to: [Plank's Law](https://en.wikipedia.org/wiki/Planck%27s_law)

### Double-click on this cell

$$

$$

# Magic!

----

### There are also a bunch of on-line $\LaTeX$ equation editors:

* https://www.tutorialspoint.com/latex_equation_editor.htm
* https://www.latex4technics.com/
* https://www.codecogs.com/latex/eqneditor.php
* http://atomurl.net/math/
* http://www.sciweavers.org/free-online-latex-equation-editor

----

## Use can use $\LaTeX$ in plots:

* You need to use the `raw` strings: (`r`)
* For example: `r"$e^{-\sqrt{x}}$"`

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
plt.style.use('ggplot')

my_x = np.linspace(0,2*np.pi,100)
my_y = np.sin(5*my_x) * np.exp(-np.sqrt(my_x))
```

```python
fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,4)

fig.tight_layout()

ax.plot(my_x, my_y, color='r', marker='None', linestyle='-');

ax.set_xlabel(r"This is in units of 2$\pi$", fontsize=16)
ax.set_ylabel(r"This is Y", fontsize=16)

my_text = r"$y\ =\, \sin(5x)\, e^{-\sqrt{x}}$"

ax.text(2.0, 0.4, my_text, color='green', fontsize=36);
```

----

Your assignment
===============

# Part I

* ### Go to the [Overleaf](https://www.overleaf.com?r=f33161ec&rm=d&rs=b) site and create an account using your `uw.edu` address.
* ### It is important that you use your `uw.edu` address!

# Part II

  * ### Create a butterfly plot: http://en.wikipedia.org/wiki/Butterfly_curve_(transcendental)
  * ### Plot X vs. Y
  * ### Feel free to change the parameters to get a different butterfly
  * ### If you don't like the butterfly, you can do a fancier [parametic equation](https://en.wikipedia.org/wiki/Parametric_equation)
  * ### Make gratuitous use of $\LaTeX$ labels
  * ### Save the plot to a png file
  * ### Upload the PNG file to Canvas (Assignment: CoolPlot)
  * ### I have started it for you below

  ----

```python
t = np.linspace(0, 12*np.pi, 3000)
```

```python

```

```python
fig,ax = plt.subplots(1,1)                    # One window
fig.set_size_inches(11,8.5)                   # (width,height) - letter paper landscape

fig.tight_layout()                            # Make better use of space on plot

```

```python
fig.savefig('MyCoolPlot.png', bbox_inches='tight')
```

## Due Mon Dec 7 - 1 pm

* upload `png` files to `CoolPlot` assignment


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from __future__ import division
import pandas as pd
```

```python
V = np.linspace(0,1000,1000)
plt.plot(V, 6.43 - 5e-14*(np.exp(V/2.6) - 1)) #in V and A
plt.ylim(0,10)
plt.xlim(0,100)
```

```python
V = np.linspace(0,100,1000)
I_o = 5e-14 #A
I_L = 6.43 #A
R_s = 0 #ohm
R_sh = 1e6 #ohm
plt.plot(V, I_L - I_o*(np.exp(V/2.6) - 1)) #in V and A
plt.ylim(0,10)
plt.xlim(0,100)
```

```python
I = 100
plt.plot(I - I_L - I_o*(np.exp((V+I*R_s)/2.6) - 1)) #this
```

```python
from sympy import solve, Symbol, exp
x = Symbol('x')
y = Symbol('y')
solve([x + 5*y - 2, -3*x + 6*y - 15], [x, y])
```

    {x: -3, y: 1}

```python
V = Symbol('V')
I = Symbol('I')
I_o = 5e-14 #A
I_L = 6.43 #A
R_s = 10 #mohm
R_sh = 1e5 #mohm
n = 1

#solve(I - I_L - I_o*(exp( (V + I * R_s) /(26*n)) - 1) , I)
```

```python
from scipy.optimize import fsolve
import math

def equations(p):
    x, y = p
    return (x+y**2-4, math.exp(x) + x*y - 3)

x, y =  fsolve(equations, (1, 1))

print equations((x, y))
```

    (4.4508396968012676e-11, -1.0512035686360832e-11)

```python
import scipy.optimize as optimize
from math import sqrt

def f(c):
    return sqrt(c[0]**2 + c[1]**2 + (c[2]-2)**2)

result = optimize.minimize(f, [1,1,1])
print result.values()[6]
```

    [ -7.29867252e-09  -7.29867252e-09   1.99999999e+00]

```python
import scipy.optimize as optimize
from math import sqrt

# I, c[0]
I_L = 6.43 #A
# I_o = 5e-14 #A,  c[2]
# n = 1, c[2]
V = 1 #mV
# R_s = 1 #mohm, c[3]
# R_sh = 1e5 #mohm c[4]

def f(c):

    I - I_L - I_o*(exp( (V + I * R_s) /(26*n)) - 1)
    return I

result = optimize.minimize(f, [1,1,1])
print result.values()[6]
```

    [ 1.  1.  1.]

```python
irrad_df = pd.read_csv('data/ASTMG173.csv')
```

```python
irrad_df.head()
```

      wavelength
      etr
      globaltilt
      directcircum

      0
      280.0
      0.082
      4.730000e-23
      2.540000e-26

      1
      280.5
      0.099
      1.230000e-21
      1.090000e-24

      2
      281.0
      0.150
      5.690000e-21
      6.130000e-24

      3
      281.5
      0.212
      1.570000e-19
      2.750000e-22

      4
      282.0
      0.267
      1.190000e-18
      2.830000e-21

```python
irrad_df['globaltilt'].plot()
```

```python
eqe_df = pd.read_csv('data/eqe_sunpower_25.csv')
```

```python
eqe_df.head()
```

      wavelength
      percent

      0
      299.768250
      64.683427

      1
      303.939745
      67.215777

      2
      308.806489
      70.355838

      3
      322.016223
      73.598574

      4
      331.054461
      76.840606

```python
eqe_df['percent'].values
```

    array([ 64.68342696,  67.21577658,  70.35583848,  73.5985743 ,
            76.84060607,  79.57642607,  86.66984467,  93.45970049,
            95.99603972,  97.11454009,  98.13494287,  98.5429397 ,
            98.44707159,  98.75345058,  98.95762501,  99.26916702,
            98.7698784 ,  98.77257726,  98.97862916,  98.58530003,
            98.79158661,  98.79510685,  98.79804039,  98.80038723,
            99.00585241,  99.00972469,  98.81118266,  98.51125746,
            97.70465113,  96.39218505,  93.15578568,  89.20935213,
            86.47705238,  75.84824794,  69.976473  ,  66.02839667,
            39.80782375,  33.63236869,  24.21722868,   8.42457134,   1.44074981])

```python
from scipy import interpolate
x = eqe_df['wavelength'].values
y = eqe_df['percent'].values
f = interpolate.interp1d(x, y)

wav_new = np.arange(300,1180, 0.5)
eqe_new = f(xnew)   # use interpolation function returned by `interp1d`
plt.plot(x, y, 'o', wav_new, eqe_new, '-')
plt.show()
```

```python
irrad_df[irrad_df['wavelength']==300]
```

      wavelength
      etr
      globaltilt
      directcircum

      40
      300
      0.458
      0.00102
      0.000456

```python
irrad_df[irrad_df['wavelength']==1180]
```

      wavelength
      etr
      globaltilt
      directcircum

      1020
      1180
      0.522
      0.441
      0.421

```python

```

```python
from scipy import interpolate

x = irrad_df['wavelength'][40:1021].values
irrad_global = irrad_df['globaltilt'][40:1021].values #AM1.5 spectrum

f = interpolate.interp1d(x, irrad_global)
wav_new = np.arange(300,1180, 0.5) #300 nm to 1180 nm with 0.5 nm spacing
irrad_new = f(xnew)   #recreate AM1.5 with 0.5 nm spacing

plt.plot(x, irrad_global, 'o', wav_new, irrad_new, '-')
plt.show()
```

```python
plt.plot(wav_new,eqe_new*irrad_new*wav_new)
```

```python
(1/1240)*sum(eqe_new*irrad_new*wav_new)*.5/1e3 #mA/cm^2
```

    41.796068304420977

```python
iv_df = pd.read_csv('data/i_v_sunpower_25.csv')
```

```python
plt.plot(iv_df.voltage,iv_df.current, 'r--')
I_o = 3.6e-10 #mA/cm^2
I_L = 41.74 #mA/cm^2
plt.plot(iv_df.voltage, I_L - I_o*(np.exp(iv_df.voltage/.0283) - 1)) #in V and A
plt.ylim(0,50)
```

```python

```

```python

```

```python

```


---
title: "Cross entropy — Part 1: Introduction and definitions"
date: 2021-02-23
categories: [statistics, entropy]
tags: [statistics, entropy, cross entropy]
preview_image: /enrightward.github.io/assets/img/cross-entropy/blue-white-paint.jpg
---

## 1. Introduction

In this post, I discuss the _cross entropy_ $H(p, q)$ of two discrete PDFs $p$ and $q$. It is a loose measure of similarity of $p$ and $q$, and so is used in machine learning to define objective functions for tasks where the goal is to learn a PDF $p$ implicit in training data by updating the internal parameters of a learnt PDF $q$. After explaining this application in more detail, I:

* Recall the definitions of _entropy_ and _cross entropy_ ;
* Use numpy to do some example computations using cross entropy, and;
* Write down a proof that for fixed but arbitrary $p$, the function $q \mapsto H(p, q)$ obtains a global minimum at $q = p$.

This last point both suggests that cross entropy measures similarity.

## 2. Why is cross entropy useful in machine learning?

The above-mentioned global minimum $q = p$ of $q \mapsto H(p, q)$ suggests that cross entropy is a measure of closeness between $p$ and $q$. This is not true in the strict sense of being a [metric à la pure mathematics](https://en.wikipedia.org/wiki/Metric_(mathematics)#Definition) (in general, $H(p, p)$ is not zero, and
$H(p, q)$ and $H(q, p)$ are different), but it is true enough to be useful in machine learning. Before jumping into details, let me give an example of this usefulness.

In language modelling, the goal is to train the machine to predict the next word in a sentence, given the preceding ones. The sentence might be:

> "the cat sat on the \[mat\]".

The machine would have to predict "mat", given the other words. Cross entropy enters the picture when one formalises this task, by defining:

* The PDF $p$ to be the mythical PDF that always predicts the right answer — formally, it is the "one-hot" PDF $p$ over all words in the English language defined by $p(\textrm{mat}) = 1$ and $p(w) = 0$ for all other words $w$ — and
* The PDF $q$ to be the machine's best guess about the next word, based on the text it has seen so far.

Under the hood, $q$ often depends on many, many parameters, which the machine can alter. The machine's learning task is to update these parameters to make $q$ as "close" to $p$ as possible. Here, people can and often do define "close" to mean the cross entropy $H(p, q)$ is as small as possible.

## 3. TL;DR:
Cross entropy is often used in machine learning to define an objective function that measures the difference between a mythical, always-correct PDF $p$ (which in practice is approximated from the data), and the machine's best-guess PDF $q$.

## 4. A quick introduction to entropy

Suppose $p$ is a discrete PDF with support ("outcome space") $S = \{ x_{1}, \ldots, x_{n} \}$. The _entropy_ of $p$ is defined to be:

\begin{equation}
H(p) := \sum_{i=1}^{n} p(x_{i}) \log \left( \frac{1}{p(x_{i})} \right) = -\sum_{i=1}^{n} p(x_{i}) \log(p(x_{i})),
\end{equation}

provided none of the $p(x_{i})$ is zero. If $p(x_{i}) = 0$ for some $i$, then we declare by fiat that the summand $p(x_{i}) \log(p(x_{i}))$ is zero. One justification for this is that:

\begin{equation}
\lim_{x \rightarrow 0^{+}} x \log x = 0.
\end{equation}

The quantity $s(x_{i}) := \log(1/p(x_{i}))$ is called the _surprisal_ of the outcome $x_{i}$. It quantifies the "information" associated to $x_{i}$ — lower probability events are more surprising, hence their occurrence more informative. [Shannon's Source Coding Theorem](https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem#:~:text=In%20information%20theory%2C%20Shannon's%20source,meaning%20of%20the%20Shannon%20entropy.&text=However%20it%20is%20possible%20to,with%20negligible%20probability%20of%20loss.) states that, if the logarithm is base 2, then $s(x_{i})$ approximates the length in bits of an optimal, lossless binary encoding for a message written in the outcome space alphabet $x_{i}$ and whose characters appear in this message with probabilities defined by $p$.

## 5. Cross entropy

I won't talk more about entropy in this blog — instead I discuss _cross entropy_. Let $q$ be a second PDF with the same support $S$. Then the _cross entropy_ of the pair $(p, q)$ is defined to be:

\begin{equation}
H(p, q) := -\sum_{i=1}^{n} p(x_{i}) \log(q(x_{i})).
\end{equation}

This is not a symmetric quantity: $H(p, q)$ and $H(q, p)$ are different in general. We'll see this by computing examples. To do this, we first define two python functions.

```python
import numpy as np
import matplotlib.pyplot as plt

def cross_entropy(p, q):
    """Require that p and q are numpy arrays"""
    result = -p.dot(np.log(q))
    return result

def random_discrete_pdf(length):
    """Compute a random multinomial
    PDF with `length` entries"""
    # Generate `length` random numbers between 0 and 1.
    pdf = np.random.rand(length)
    # normalise, so it's a pdf
    pdf = pdf/pdf.sum()
    return pdf
```

Taking $p$ and $q$ to be two randomly-generated PDFs of length five, we have:

```python
length = 5
p = random_discrete_pdf(length)
q = random_discrete_pdf(length)
print('p =', p)
print('q =', q)
print()
print('H(p, q) = {:0.3f}'.format(cross_entropy(p, q)))
print('H(q, p) = {:0.3f}'.format(cross_entropy(q, p)))
```

    p = [0.08669807 0.24134156 0.32611669 0.16300244 0.18284124]
    q = [0.17582746 0.02024274 0.29378282 0.14374014 0.36640684]

    H(p, q) = 1.991
    H(q, p) = 1.671

Now let's keep $p$ fixed, and compute $H(p, q)$ for randomly-generated $q$:

```python
N = 1000000
hpqs = []

for _ in range(N):
    q = random_discrete_pdf(length)
    hpq = cross_entropy(p, q)
    hpqs.append(hpq)

hpqs = np.array(hpqs)
```

We plot the $H(p, q)$'s in a histogram, to get a sense of the distribution.

```python
num_bins = 1000

fig, ax = plt.subplots(figsize=(15, 8))
n, bins, patches = plt.hist(hpqs, num_bins, density=1, facecolor='blue', alpha=0.5)

colour = 'white'

ax.spines['bottom'].set_color(colour)
ax.spines['top'].set_color(colour)
ax.spines['left'].set_color(colour)
ax.spines['right'].set_color(colour)
ax.xaxis.label.set_color(colour)
ax.yaxis.label.set_color(colour)
ax.title.set_color(colour)
ax.tick_params(axis='x', colors=colour)
ax.tick_params(axis='y', colors=colour)

plt.xlabel('H(p, q)')
plt.ylabel('Frequency')
plt.title(r'Distribution of H(p, q)')
plt.show()
```

There appears to be a hard cut-off around $1 \le H(p, q) \le 2$, below which there are no samples. What's going on? It turns out this lower bound is $H(p, p)$. Empirically, this looks correct:

```python
print('H(p, p) = {:0.3f}'.format(cross_entropy(p, p)))
```

    H(p, p) = 1.527

## 6. Proof that $q = p$ minimises $p \mapsto H(p, q)$

The goal is to show that for a fixed discrete PDF $p$, $H(p, p) \le H(p, q)$ for all PDFs $q$ with the same support as $p$. We use the following fact: Since $y = x - 1$ is the tangent at $x = 1$ to the convex function $y = \log(x)$, we have:

\begin{equation}
\log(x) \le x - 1,
\end{equation}

for all $x \ge 0$. To ease notation, write $p_{i} := p(x_{i})$ for all $1 \le i \le n$, and define $I := \{ 1 \le i \le n \mid p_{i} \neq 0 \}$. Then:

\begin{equation}
\sum_{i \in I} p_{i} \log \left( \frac{q_{i}}{p_{i}} \right) \le^{\textrm{(a)}}
\sum_{i \in I} p_{i} \left(\frac{q_{i}}{p_{i}} - 1 \right) = \\
\sum_{i \in I} q_{i} - \sum_{i \in I} p_{i} =^{\textrm{(b)}}
\left( \sum_{i \in I} q_{i} \right) - 1 \le^{\textrm{(c)}} 0.
\end{equation}

Here, the inequality (a) follows from the log inequality above, and the equality (b) holds because the sum of $p_{i}$ over $I$ is 1. Since the sum of $q_{i}$ over $I$ may be less than 1, inequality (c) holds. So we've shown that:

\begin{equation}
0 \ge \sum_{i \in I} p_{i} \log \left( \frac{q_{i}}{p_{i}} \right) =
\sum_{i \in I} p_{i} \log (q_{i}) - \sum_{i \in I} p_{i} \log (p_{i}),
\end{equation}

but this implies that:

\begin{equation}
H(p, p)  = - \sum_{i \in I} p_{i} \log (p_{i}) \le -\sum_{i \in I} p_{i} \log (q_{i}) = H(p, q).
\end{equation}

## 7. Roundup

We introduced the notion of cross entropy, and noted its usefulness in machine learning as an objective function measuring the difference between an ideal PDF $p$ implicit in a data set, and a learnt, parametrised PDF $q$. This has particular applications in language modelling. We then wrote down a proof that for fixed but arbitrary $p$, the function $q \mapsto H(p, q)$ obtains a global minimum at $q = p$. In the [next post](https://enrightward.github.io/enrightward.github.io/posts/cross-entropy-part-2/), we will show, with the help of the softmax parametrisation, that this function is convex, so can be optimised via a gradient descent algorithm.