# Week 4: Linear Algebra IV

# Tutorial 1

# [insert your name]

**Important reminders**: Before starting, click "File -> Save a copy in Drive". Produce a pdf for submission by "File -> Print" and then choose "Save to PDF".

To complete this tutorial, you should have watched Videos 4.1, 4.2, 4.3, and 4.4.

**Credits**: Video 4.3 is the Week 1 Day 5 Intro Video from NMA (Neuromatch Academy) (https://github.com/NeuromatchAcademy/course-content). Exercise 2/3 in this tutorial are modified from content in NMA W1D5 tutorials.

We are again using code for visualizing linear transformations from https://openedx.seas.gwu.edu/courses/course-v1:GW+EngComp4+2019/about. In particular, we are using their `plot_linear_transformation` and `plot_linear_transformations` functions.

The interactive demo in Exercise 2A is based on matlab code from: https://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues

```
# @markdown Imports

# Imports
import numpy as np
import matplotlib.pyplot as plt
import ipywidgets as widgets  # interactive display

```

```
# @markdown Plotting functions
import numpy
from numpy.linalg import inv, eig
from math import ceil
from matplotlib import pyplot, ticker, get_backend, rc
from mpl_toolkits.mplot3d import Axes3D
from itertools import cycle

%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")

classic = 'k'

_int_backends = ['GTK3Agg', 'GTK3Cairo', 'MacOSX', 'nbAgg',
                 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo',
                 'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo']
_backend = get_backend()   # get current backend name

# shrink figsize and fontsize when using %matplotlib notebook
if _backend in _int_backends:
    fontsize = 4
    fig_scale = 0.75
else:
    fontsize = 5
    fig_scale = 1

grey = '#808080'
gold = '#cab18c'   # x-axis grid
lightblue = '#0096d6'  # y-axis grid
green = '#008367'  # x-axis basis vector
red = '#E31937'    # y-axis basis vector
darkblue = '#004065'

pink, yellow, orange, purple, brown = '#ef7b9d', '#fbd349', '#ffa500', '#a35cff', '#731d1d'

quiver_params = {'angles': 'xy',
                 'scale_units': 'xy',
                 'scale': 1,
                 'width': 0.012}

grid_params = {'linewidth': 0.5,
               'alpha': 0.8}
def plot_sample_images(X):
  """
  Plots 9 images from the data.

  Args:
     X (numpy array of floats) : Data matrix each column corresponds to a
                                 different random variable

  Returns:
    Nothing.

  """
  im_size = int(np.sqrt(X.shape[1]))
  fig, ax = plt.subplots()
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(X[k, :], (im_size, im_size)),
                 extent=[(k1 + 1) * im_size, k1 * im_size, (k2+1) * im_size, k2 * im_size],
                 vmin=0, vmax=255, cmap='gray')
  plt.xlim((3 * im_size, 0))
  plt.ylim((3 * im_size, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  plt.clim([0, 250])
  ax.set_xticks([])
  ax.set_yticks([])
  plt.show()

def plot_variance_explained(variance_explained):
  """
  Plots eigenvalues.

  Args:
    variance_explained (numpy array of floats) : Vector of variance explained
                                                 for each PC

  Returns:
    Nothing.

  """

  plt.figure()
  plt.plot(np.arange(1, len(variance_explained) + 1), variance_explained,
           '--k')
  plt.xlabel('Number of components')
  plt.ylabel('Variance explained')
  plt.show()

def plot_reconstructions(X, X_reconstructed):
  """
  Plots 9 images in the dataset side-by-side with the reconstructed
  images.

  Args:
    X (numpy array of floats)               : Data matrix each column
                                              corresponds to a different
                                              random variable
    X_reconstructed (numpy array of floats) : Data matrix each column
                                              corresponds to a different
                                              random variable

  Returns:
    Nothing.
  """

  im_size = int(np.sqrt(X.shape[1]))

  plt.figure()
  ax = plt.subplot(121)
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(X[k, :], (im_size, im_size)),
                 extent=[(k1 + 1) * im_size, k1 * im_size, (k2 + 1) * im_size, k2 * im_size],
                 vmin=0, vmax=255, cmap='gray')
  plt.xlim((3 * im_size, 0))
  plt.ylim((3 * im_size, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.title('Data')
  plt.clim([0, 250])
  ax = plt.subplot(122)
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(np.real(X_reconstructed[k, :]), (im_size, im_size)),
                 extent=[(k1 + 1) * im_size, k1 * im_size, (k2 + 1) * im_size, k2 * im_size],
                 vmin=0, vmax=255, cmap='gray')
  plt.xlim((3 * im_size, 0))
  plt.ylim((3 * im_size, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.clim([0, 250])
  plt.title('Reconstructed')
  plt.tight_layout()

def plot_principal_components(weights):
  """
  Visualize PCA basis vector weights. Red = positive weights,
  blue = negative weights, white = zero weight.

  Args:
     weights (numpy array of floats) : PCA basis vector

  Returns:
     Nothing.
  """
  im_size = int(np.sqrt(X.shape[1]))

  fig, ax = plt.subplots()
  cmap = plt.cm.get_cmap('seismic')
  plt.imshow(np.real(np.reshape(weights, (im_size, im_size))), cmap=cmap)
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  plt.clim(-.15, .15)
  plt.colorbar(ticks=[-.15, -.1, -.05, 0, .05, .1, .15])
  ax.set_xticks([])
  ax.set_yticks([])
  plt.show()

def plot_pca_transformation(data, transformed_data):

    fig, axes = plt.subplots(1, 2)
    axes[0].scatter(data[:,0], data[:, 1], s=1, c='#63BA79');
    for j in range(2):
      axes[j].spines['right'].set_visible(False)
      axes[j].spines['top'].set_visible(False)

    orig_correlation = round(np.corrcoef(data[:, 0], data[:, 1])[0, 1], 2)
    axes[0].set(title='Data in original coordinates \n Correlation = ' + str(orig_correlation), xlabel='Neuron 1 activity', ylabel='Neuron 2 activity', xlim=[-5, 15], ylim=[-5, 15]);

    axes[1].scatter(transformed_data[:,0], transformed_data[:, 1], s=1, c='#63BA79');
    pca_correlation = round(np.corrcoef(transformed_data[:, 0], transformed_data[:, 1])[0, 1], 2)
    axes[1].set(title='Data in PC coordinates  \n Correlation = ' + str(pca_correlation), xlabel='PC 1', ylabel='PC 2');

    plt.tight_layout()

def plot_data_and_PCs(X, W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable
    W (numpy array of floats) : Square matrix representing new orthonormal
                                basis each column represents a basis vector

  Returns:
    Nothing.
  """

  plt.figure()
  plt.scatter(X[:, 0], X[:, 1], s=1, color='#63BA79')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
  plt.plot([0, W[0, 0]], [0, W[1, 0]], color=colors[4], linewidth=1,
           label='Component 1')
  plt.plot([0, W[0, 1]], [0, W[1, 1]], color=colors[3], linewidth=1,
           label='Component 2')
  plt.legend()
  plt.show()

def plot_vector(vectors, tails=None):
    ''' Draw 2d vectors based on the values of the vectors and the position of their tails.

    Parameters
    ----------
    vectors : list.
        List of 2-element array-like structures, each represents a 2d vector.

    tails : list, optional.
        List of 2-element array-like structures, each represents the coordinates of the tail
        of the corresponding vector in vectors. If None (default), all tails are set at the
        origin (0,0). If len(tails) is 1, all tails are set at the same position. Otherwise,
        vectors and tails must have the same length.

    Examples
    --------
    >>> v = [(1, 3), (3, 3), (4, 6)]
    >>> plot_vector(v)      # draw 3 vectors with their tails at origin
    >>> t = [numpy.array((2, 2))]
    >>> plot_vector(v, t)   # draw 3 vectors with their tails at (2,2)
    >>> t = [[3, 2], [-1, -2], [3, 5]]
    >>> plot_vector(v, t)   # draw 3 vectors with 3 different tails

    '''
    vectors = numpy.array(vectors)
    assert vectors.shape[1] == 2, "Each vector should have 2 elements."
    if tails is not None:
        tails = numpy.array(tails)
        assert tails.shape[1] == 2, "Each tail should have 2 elements."
    else:
        tails = numpy.zeros_like(vectors)

    # tile vectors or tails array if needed
    nvectors = vectors.shape[0]
    ntails = tails.shape[0]
    if nvectors == 1 and ntails > 1:
        vectors = numpy.tile(vectors, (ntails, 1))
    elif ntails == 1 and nvectors > 1:
        tails = numpy.tile(tails, (nvectors, 1))
    else:
        assert tails.shape == vectors.shape, "vectors and tail must have a same shape"

    # calculate xlimit & ylimit
    heads = tails + vectors
    limit = numpy.max(numpy.abs(numpy.hstack((tails, heads))))
    limit = numpy.ceil(limit * 1.2)   # add some margins

    figsize = numpy.array([2,2]) * fig_scale
    figure, axis = pyplot.subplots(figsize=figsize)
    axis.quiver(tails[:,0], tails[:,1], vectors[:,0], vectors[:,1], color=darkblue,
                  angles='xy', scale_units='xy', scale=1)
    axis.set_xlim([-limit, limit])
    axis.set_ylim([-limit, limit])
    axis.set_aspect('equal')

    # if xticks and yticks of grid do not match, choose the finer one
    xticks = axis.get_xticks()
    yticks = axis.get_yticks()
    dx = xticks[1] - xticks[0]
    dy = yticks[1] - yticks[0]
    base = max(int(min(dx, dy)), 1)   # grid interval is always an integer
    loc = ticker.MultipleLocator(base=base)
    axis.xaxis.set_major_locator(loc)
    axis.yaxis.set_major_locator(loc)
    axis.grid(True, **grid_params)

    # show x-y axis in the center, hide frames
    axis.spines['left'].set_position('center')
    axis.spines['bottom'].set_position('center')
    axis.spines['right'].set_color('none')
    axis.spines['top'].set_color('none')

def plot_transformation_helper(axis, matrix, *vectors, unit_vector=True, unit_circle=False, title=None):
    """ A helper function to plot the linear transformation defined by a 2x2 matrix.

    Parameters
    ----------
    axis : class matplotlib.axes.Axes.
        The axes to plot on.

    matrix : class numpy.ndarray.
        The 2x2 matrix to visualize.

    *vectors : class numpy.ndarray.
        The vector(s) to plot along with the linear transformation. Each array denotes a vector's
        coordinates before the transformation and must have a shape of (2,). Accept any number of vectors.

    unit_vector : bool, optional.
        Whether to plot unit vectors of the standard basis, default to True.

    unit_circle: bool, optional.
        Whether to plot unit circle, default to False.

    title: str, optional.
        Title of the plot.

    """
    assert matrix.shape == (2,2), "the input matrix must have a shape of (2,2)"
    grid_range = 20
    x = numpy.arange(-grid_range, grid_range+1)
    X_, Y_ = numpy.meshgrid(x,x)
    I = matrix[:,0]
    J = matrix[:,1]
    X = I[0]*X_ + J[0]*Y_
    Y = I[1]*X_ + J[1]*Y_
    origin = numpy.zeros(1)

    # draw grid lines
    for i in range(x.size):
        axis.plot(X[i,:], Y[i,:], c=gold, **grid_params)
        axis.plot(X[:,i], Y[:,i], c=lightblue, **grid_params)

    # draw (transformed) unit vectors
    if unit_vector:
        axis.quiver(origin, origin, [I[0]], [I[1]], color=green, **quiver_params)
        axis.quiver(origin, origin, [J[0]], [J[1]], color=red, **quiver_params)

    # draw optional vectors
    color_cycle = cycle([pink, darkblue, orange, purple, brown])
    if vectors:
        for vector in vectors:
            color = next(color_cycle)
            vector_ = matrix @ vector.reshape(-1,1)
            axis.quiver(origin, origin, [vector_[0]], [vector_[1]], color=color, **quiver_params)

    # draw optional unit circle
    if unit_circle:
        alpha =  numpy.linspace(0, 2*numpy.pi, 41)
        circle = numpy.vstack((numpy.cos(alpha), numpy.sin(alpha)))
        circle_trans = matrix @ circle
        axis.plot(circle_trans[0], circle_trans[1], color=red, lw=0.8)

    # hide frames, set xlimit & ylimit, set title
    limit = 4
    axis.spines['left'].set_position('center')
    axis.spines['bottom'].set_position('center')
    axis.spines['left'].set_linewidth(0.3)
    axis.spines['bottom'].set_linewidth(0.3)
    axis.spines['right'].set_color('none')
    axis.spines['top'].set_color('none')
    axis.set_xlim([-limit, limit])
    axis.set_ylim([-limit, limit])
    if title is not None:
        axis.set_title(title)

def plot_linear_transformation(matrix, *vectors, unit_vector=True, unit_circle=False):
    """ Plot the linear transformation defined by a 2x2 matrix using the helper
    function plot_transformation_helper(). It will create 2 subplots to visualize some
    vectors before and after the transformation.

    Parameters
    ----------
    matrix : class numpy.ndarray.
        The 2x2 matrix to visualize.

    *vectors : class numpy.ndarray.
        The vector(s) to plot along with the linear transformation. Each array denotes a vector's
        coordinates before the transformation and must have a shape of (2,). Accept any number of vectors.

    unit_vector : bool, optional.
        Whether to plot unit vectors of the standard basis, default to True.

    unit_circle: bool, optional.
        Whether to plot unit circle, default to False.

    """
    with plt.rc_context({"figure.dpi": 200, 'font.family':'serif', 'axes.axisbelow':True, 'font.size':fontsize, "axes.titlesize":5, "lines.linewidth":1}):
      figsize = numpy.array([4,2]) * fig_scale
      figure, (axis1, axis2) = pyplot.subplots(1, 2, figsize=figsize)
      plot_transformation_helper(axis1, numpy.identity(2), *vectors, unit_vector=unit_vector, unit_circle=unit_circle, title='Before transformation')
      plot_transformation_helper(axis2, matrix, *vectors, unit_vector=unit_vector, unit_circle=unit_circle, title='After transformation')

def plot_linear_transformations(*matrices, unit_vector=True, unit_circle=False):
    """ Plot the linear transformation defined by a sequence of n 2x2 matrices using the helper
    function plot_transformation_helper(). It will create n+1 subplots to visualize some
    vectors before and after each transformation.

    Parameters
    ----------
    *matrices : class numpy.ndarray.
        The 2x2 matrices to visualize. Accept any number of matrices.

    unit_vector : bool, optional.
        Whether to plot unit vectors of the standard basis, default to True.

    unit_circle: bool, optional.
        Whether to plot unit circle, default to False.

    """
    nplots = len(matrices) + 1
    nx = 2
    ny = ceil(nplots/nx)
    with plt.rc_context({"figure.dpi": 200, 'font.family':'serif', 'axes.axisbelow':True, 'font.size':fontsize, "axes.titlesize":5, "lines.linewidth":1}):
      figsize = numpy.array([2*nx, 2*ny]) * fig_scale
      figure, axes = pyplot.subplots(nx, ny, figsize=figsize)

      for i in range(nplots):  # fig_idx
          if i == 0:
              matrix_trans = numpy.identity(2)
              title = 'Before transformation'
          else:
              matrix_trans = matrices[i-1] @ matrix_trans
              if i == 1:
                  title = 'After {} transformation'.format(i)
              else:
                  title = 'After {} transformations'.format(i)
          plot_transformation_helper(axes[i//nx, i%nx], matrix_trans, unit_vector=unit_vector, unit_circle=unit_circle, title=title)
      # hide axes of the extra subplot (only when nplots is an odd number)
      if nx*ny > nplots:
          axes[-1,-1].axis('off')
```

```
# @markdown Helper functions
def sort_evals_descending(evals, evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats)    : Vector of eigenvalues
    evectors (numpy array of floats) : Corresponding matrix of eigenvectors
                                        each column corresponds to a different
                                        eigenvalue

  Returns:
    (numpy array of floats)          : Vector of eigenvalues after sorting
    (numpy array of floats)          : Matrix of eigenvectors after sorting
  """

  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:, index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:, 0],
                           1 / np.sqrt(2) * np.array([1, 1]))) > np.pi / 2:
      evectors[:, 0] = -evectors[:, 0]
    if np.arccos(np.matmul(evectors[:, 1],
                           1 / np.sqrt(2) * np.array([-1, 1]))) > np.pi / 2:
      evectors[:, 1] = -evectors[:, 1]
  return evals, evectors

def calculate_cov_matrix(var_1, var_2, corr_coef):
  """
  Calculates the covariance matrix based on the variances and
  correlation coefficient.

  Args:
    var_1 (scalar)         :  variance of the first random variable
    var_2 (scalar)         :  variance of the second random variable
    corr_coef (scalar)     :  correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  cov = corr_coef * np.sqrt(var_1 * var_2)
  cov_matrix = np.array([[var_1, cov], [cov, var_2]])
  return cov_matrix

def get_variance_explained(evals):
  """
  Plots eigenvalues.

  Args:
    (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """

  # cumulatively sum the eigenvalues
  csum = np.cumsum(evals)
  # normalize by the sum of eigenvalues
  variance_explained = csum / np.sum(evals)

  return variance_explained

def add_noise(X, frac_noisy_pixels):
  """
  Randomly corrupts a fraction of the pixels by setting them to random values.

  Args:
     X (numpy array of floats)  : Data matrix
     frac_noisy_pixels (scalar) : Fraction of noisy pixels

  Returns:
     (numpy array of floats)    : Data matrix + noise

  """

  X_noisy = np.reshape(X, (X.shape[0] * X.shape[1]))
  N_noise_ixs = int(X_noisy.shape[0] * frac_noisy_pixels)
  noise_ixs = np.random.choice(X_noisy.shape[0], size=N_noise_ixs,
                               replace=False)
  X_noisy[noise_ixs] = np.random.uniform(0, 255, noise_ixs.shape)
  X_noisy = np.reshape(X_noisy, (X.shape[0], X.shape[1]))

  return X_noisy

```

# Key concept review & coding tips

## Special matrices

### Diagonal matrices
*   Have only nonzero entries on the diagonal
*   Can be rectangular
*   Scales space
*   Inverse is diagonal matrix with inverse entries

### Orthogonal matrices
*  Square matrix where every column is a unit vector and every pair of columns is orthogonal
*  Rotates space
*  Its inverse is its transpose

### Symmetric matrices
*  Square matrix where $a_{ij} = a_{ji}$
*  Equals its own transpose
*  Eigenvalues are always real (not complex)
*  Eigenvectors associated with different eigenvalues are orthogonal

## Matrix decomposition

* Factorization of a matrix into a product of matrices
* Product matrices might be more compact/ordered, could make computations easier, could shed light on matrix structure

### Eigendecomposition
*  $A = VDV^{-1}$ where V has eigenvectors as columns and D is diagonal matrix with eigenvalues on the diagonal
*  Can only do this if A is square and if eigenvectors of A form a basis for space
*  $A^n = VD^nV^{-1}$
*  $A^{-1} = VD^{-1}V^{-1}$

### Singular value decomposition
*  `np.linalg.svd`
*  $A = USV^T$ where U/V are orthogonal matrices and S is diagonal
*  Can decompose any matrix this way
*  Diagonal entries of S are singular values, columns of U are left singular values, columns of V are right singular values
* Decomposes transformation that matrix enacts into a rotation, then a scaling, then a rotation
* Columns of V associated with zero or non-existant singular values form an orthogonal basis for the nullspace of A
* Columns of U associated with non-zero singular values form an orthogonal basis for the column space of A
* rank(A) = number of non-zero singular values
* SVD factorizes A into sum of outer products with decreasing influence, can use first K sums to form rank K approximation of A

## Dimensionality Reduction
* Transform data from high D to low D while keeping as much information as possible about the data (finding new representation for the data)
* Can help with data visualization, noise reduction, data preprocessing for further analyses, and scientific findings, among other things

### Principal components analysis
* `sklearn.decomposition.pca` for PCA, see info at the end of this tutorial
*  Find axes in space that maximize the variance of the data (and minimize the residuals) while being orthongal to each other. Project data onto these axes and keep first K components
* Can think of PCA as a change of basis where the new basis vectors are the principal components
* $U = XV$ where U is the transformed data (# data points x reduced dim), X is the data matrix (# data points x orig dim), and V is the components matrix (orig dim x reduced dim)
* Always center your data first!
* Can find principal components as the eigenvectors of the covariance matrix ($\frac{1}{n}X^TX$), eigenvalues tell you the variance explained by that component (plot these to make a scree plot)
* Could also use SVD to find PCA - the columns of V are the eigenvectors of the covariance matrix and the squared singular values over N are the eigenvalues

# Exercise 1: Delving into SVD

We'll explore SVD by hand in this problem to help solidify our understanding of how the matrices interact with each other. Let $$A = \begin{bmatrix}
2 & -4 \\
3 & 1 \\
\end{bmatrix}, \bar{v} = \begin{bmatrix}
2 \\
1 \\
\end{bmatrix}$$

In the following plot, we'll see the transformation that this matrix enacts.

```
A = np.array([[2, -4], [3, 1]])
plot_linear_transformation(A)
```

## A) Computing the SVD

Use `np.linalg.svd` to get the SVD of the matrix A. Note that the outputs are not quite the U, S, V we've been discussing. This function outputs $V^T$ directly. Get S/V from the outputs.

```
U, s, VT = ...
S = ...
V = ...
```

## B) SVD step by step transformations

Multiply out the operations of $V^T$, S, and U with vector $\bar{v}$ one at a time. In other words, get $V^T\bar{v}$, then $SV^T\bar{v}$, then $USV^t\bar{v}$. You do not need to do this by hand - use code - but make sure you understand the matrix vector multiplication!

Make sure $USV^t\bar{v}$ = $A\bar{v}$.

Execute the following cell to visualize the vectors.

```
v = ...
VTv = ...
SVTv = ...
USVTv = ...
Av = ...
print(USVTv)
print(Av)
```

```
# @title
# @markdown Execute to visualize vector transforms
vec_names = [r'$\bar{v}$', r'$SV^T\bar{v}$', r'$V^T\bar{v}$', r'A$\bar{v}$']
vecs = np.array([v,
                 SVTv,
                 VTv,
                 USVTv])

fig, axes = plt.subplots(1, 1)
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

axes.set(xlim=[-8, 8], ylim=[-8, 8])
axes.axis('Off')

for i_vec, vec in enumerate(vecs):
  axes.arrow(0, 0, vec[0], vec[1], head_width=.2, facecolor=colors[i_vec], edgecolor=colors[i_vec], length_includes_head=True);
  axes.annotate(vec_names[i_vec], xy=(vec[0]+np.sign(vec[0])*.15, vec[1]+np.sign(vec[1])*.15), color=colors[i_vec]);

axes.plot([0, 0], [-8, 8], classic, alpha=.4);
axes.plot([-8, 8], [0, 0], classic, alpha=.4);

```

What transformation is happening to $\bar{v}$ at each step?

**Your text answer**

## C) Low rank approximation

We'll explore successful low rank approximations of receptive fields in Tutorial 2 and this will make the concept much more intuitive - the goal of this problem is just to understand the computation involved and what a rank 1 approximation means.

Calculate a rank 1 approximation of A by hand. Specifically, compute:

$$\text{Rank 1 approx } = B = s_1\bar{u}_1\bar{v}_1^T $$

where $s_1$ is the first (highest) singular value and $\bar{u}_1$ and $\bar{v}_1$ are the corresponding columns of U and V.

Show your work for the computation! You should round to 2 places after the decimal.

**Your math answer** show your work!

Compare B to the original matrix A. What does a rank 1 approximation mean? What is the computation "trying to do"? What is happening with the columns/rows of B?

**Your text answer here**

Note that the rank 1 approximation here is not great because our matrix is not anywhere close to rank 1! We would fully recover our matrix with a rank 2 approximation - $ A = s_1\bar{u}_1\bar{v}_1^T + s_2\bar{u}_2\bar{v}_2^T$ - since A is 2 x 2 and has maximum rank of 2.

## (Optional) Extra info: Orthogonal matrices can also reflect

Execute the next cell to visualize the transformation at each step of SVD (by $V^T$, then $S$, then $U$). You will notice that it isn't simply rotation, then scaling, then a rotation. Both $V^T$ and $U$ enact a reflection in addition to a rotation. Orthogonal matrices can reflect in addition to rotating space.

We could get an SVD without reflection if we hadn't ordered our columns by the size of the singular values.  If we switched the columns in U, S, and V, we would see just a rotation, then scaling, then another rotation (show below).

```
# @markdown Execute this cell to visualize transformations

plot_linear_transformations(VT, S, U, unit_vector=True, unit_circle=False)

```

```
# @markdown Execute this cell to visualize transformations with permuted columns

plot_linear_transformations(V[:, [1, 0]].T, np.diag(s[::-1]), U[:, [1, 0]], unit_vector=True, unit_circle=False)

```

# Exercise 2: PCA implementation and correlation exploration

### Modified from NMA W1D5 T2

In this exercise, you will implement PCA, apply it to 2 dimensional data, and examine the effects of correlations between dimensions.

```
# @markdown Execute this cell to generate some data (X)
np.random.seed(123)
variance_1 = 1
variance_2 = 1
corr_coef = 0.8
cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
X =  np.random.multivariate_normal([5, 10], cov_matrix, size=1000)

fig, ax = plt.subplots()
ax.scatter(X[:,0], X[:, 1], s=1, color='#63BA79');
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

ax.set(xlabel='Neuron 1 activity', ylabel='Neuron 2 activity', xlim=[-5, 15], ylim=[-5, 15]);
```

## A) Interactive Demo: Identifying first principal component

Let's take a subset of our data as shown below and mean subtract it. Play with the interactive demo. About which value of theta represents the first principal component? Why?

```
# @markdown Make sure you execute this cell to enable the widget!

def plot_potential_component(theta=180):
  n_points = 30

  mean_subtracted_X = X - np.mean(X, 0)

  fig, ax = plt.subplots()
  ax.spines['right'].set_visible(False)
  ax.spines['top'].set_visible(False)

  ax.set(xlabel='Neuron 1 activity', ylabel='Neuron 2 activity', xlim=[-5, 5], ylim=[-5, 5]);

  w = np.asarray([np.cos(theta*np.pi/180), np.sin(theta*np.pi/180)])[None, :];
  z = mean_subtracted_X[:n_points, :] @ w.T @ w;
  for i in range(n_points):
    ax.plot([mean_subtracted_X[i,0], z[i,0]], [mean_subtracted_X[i,1], z[i,1]], 'r')

  ax.plot(w[0, 0]*5*np.array([-1, 1]), w[0, 1]*5*np.array([-1, 1]), 'k')
  ax.scatter(z[:,0], z[:, 1],  color='r')
  ax.scatter(mean_subtracted_X[:n_points,0], mean_subtracted_X[:n_points, 1], color='#63BA79');

_ = widgets.interact(plot_potential_component, theta = (0, 180, 1), fontsize=60)
```

**Your text answer**

## B) Implement PCA

Let's first implement PCA! We will build a function that takes in data and returns the transformed data, the principal components, and the variance explained by each component.

We will use an implementation involving the eigenvectors/eigenvalues of the covariance matrix (as opposed to using SVD).

```
def pca(X):
  """
  Sorts eigenvalues and eigenvectors in decreasing order.

  Args:
    X (numpy array of floats): Data matrix each column corresponds to a
                               different random variable

  Returns:
    (numpy array of floats)  : Data projected onto the new basis
    (numpy array of floats)  : Vector of eigenvalues
    (numpy array of floats)  : Corresponding matrix of eigenvectors

  """

  # Subtract the mean of X
  X = ...

  # Calculate the sample covariance matrix
  cov_matrix = ... # hint: covariance matrix = (1/n)X^TX

  # Calculate the eigenvalues and eigenvectors
  evals, evectors = ... # hint: use np.linalg.eig

  # Sort the eigenvalues in descending order using a helper function
  evals, evectors = sort_evals_descending(evals, evectors)

  # Project the data onto the new eigenvector basis
  transformed_data = ... # hint: remember U = XV

  return transformed_data, evectors, evals

# Uncomment below once you have filled in the above function

# Perform PCA on the data matrix X
#X_pca, evectors, evals = pca(X)

# Plot the data projected into the new basis
#plot_pca_transformation(X, X_pca)
```

Note that the correlation between dimensions goes to 0 after the transformation to the principal components basis! This is a property of PCA: it decorrelates the data.

## C) Visualize variance explained

We want to create a plot telling us the percent of variance explained by each principal component (here we have just two). Determine what information you need for this (the inputs) and complete the function below.

```
def plot_variances(...):

   percent_explained_variance = ...

   fig, ax = plt.subplots()
   colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
   ax.plot(percent_explained_variance, '-o', color=colors[4])
   ax.set(ylim=[0, 1], ylabel='% Explained Variance', xlabel='Component number', xticks=np.arange(len(percent_explained_variance)))

plot_variances(...)
```

## D) Interactive Demo: Exploration of the correlation coefficient

Run the following cell and use the slider to change the correlation coefficient of the data. This will update a plot of the data with the principal components overlaid and a plot of percentage of explained variance.

**Questions:**
* Can you find a correlation coefficient value for which the components have equal explained variance?
* Can you find a value for which only only one component explains any variance?

```
# @markdown Make sure you execute this cell to enable the widget!

def refresh(corr_coef=.8):
  cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
  X = X =  np.random.multivariate_normal([0, 0], cov_matrix, size=1000)
  score, evectors, evals = pca(X)
  plot_data_and_PCs(X, evectors)
  plot_variances(evals)

_ = widgets.interact(refresh, corr_coef=(-1, 1, .1), fontsize=60)
```

**Your text answer**

## Optional advanced challenge: PCA implementation with SVD

Take the PCA function from part A and implement with SVD instead of with the eigenvectors of the covariance matrix.

```
def pca_with_SVD(X):

    ...
```

# Exercise 3: PCA of images
## Modified from NMA W1D5 T3

In this exercise, we will look at the PCA of images. We will use images from the MNIST dataset, which is a dataset of handdrawn numbers (0-9). We're using this data instead of more neuroscience related data because it's a small dataset that's easy to interpret. Everything we will learn here could be applied to, for example, the frames of a video of a mouse performing a task.

The MNIST dataset consists of a 70,000 images of individual handwritten digits. Each image is a 28x28 pixel grayscale image. For convenience, each 28x28 pixel image is often unravelled into a single 784 (=28*28) element vector (a process called flattening the images), so that the whole dataset is represented as a 70,000 x 784 matrix. Each row represents a different image, and each column represents a different pixel.

Execute the following cell to load the MNIST dataset and plot the first nine images.

```
from sklearn.datasets import fetch_openml
mnist = fetch_openml(name='mnist_784')
X = mnist.data
plot_sample_images(X)
```

## A) Explained variances

We will first perform PCA and plot the cumulative percentage explained variance over components. Note that this is related to our earlier plots but now we are plotting the percentage of explained variance **cumulatively**. Execute the next cell to do this.

- How many principal components are required to explain 90% of the variance?
- How does the intrinsic dimensionality of this dataset compare to its extrinsic dimensionality?

```
transformed_data, evectors, evals = pca(X)
variance_explained = get_variance_explained(evals)
plot_variance_explained(variance_explained)
```

**Your text answer**

## B) PCA Reconstruction

Let's try projecting down onto our reduced dimensionality PCA space and then **reconstructing** our images from the low-D space.

To see this, recall that to perform PCA we projected the data $\bf X$ onto the eigenvectors of the covariance matrix:
\begin{equation}
\bf U = X V
\end{equation}
where $U$ is the transformed data, $X$ is the data matrix, and $V$ is the components matrix.

Since $\bf V$ is an orthogonal matrix, ${\bf V}^{-1} = {\bf V}^T$. We can reconstruct by:

\begin{equation}
\bf X = U V^T
\end{equation}

To reconstruct the data from a low-dimensional approximation, we just have to truncate these matrices.  Let's call ${\bf U}_{1:K}$ and ${\bf V}_{1:K}$ as keeping only the first $K$ columns of this matrix. Then our reconstruction is:
\begin{equation}
{\bf \hat X = U}_{1:K} ({\bf V}_{1:K})^T.
\end{equation}

Complete the following function to reconstruct the images from the top K components.

```
def reconstruct_data(transformed_data, evectors, X_mean, K):
  """
  Reconstruct the data based on the top K components.

  Args:
    transformed_data (numpy array of floats)    : data projected onto PCA basis
    evectors (numpy array of floats) : Matrix of eigenvectors
    X_mean (numpy array of floats)   : Vector corresponding to data mean
    K (scalar)                       : Number of components to include

  Returns:
    (numpy array of floats)          : Matrix of reconstructed data

  """

  #################################################
  ## TO DO for students: Reconstruct the original data in X_reconstructed
  # Comment once you've filled in the function
  raise NotImplementedError("Student exercise: reconstructing data function!")
  #################################################

  # Reconstruct the data from the score and eigenvectors
  # Don't forget to add the mean!!
  X_reconstructed =  ...

  return X_reconstructed

K = 100

# Fill in below then uncomment the last line

# Reconstruct the data based on all components
X_mean = ...
X_reconstructed = ...

# Plot the data and reconstruction
# plot_reconstructions(X, X_reconstructed)
```

## C) Interactive Demo: Reconstruct the data matrix using different numbers of PCs

Now run the code below and experiment with the slider to reconstruct the data matrix using different numbers of principal components.

-  How many principal components are necessary to reconstruct the numbers (by eye)? How does this relate to the intrinsic dimensionality of the data?
- Do you see any information in the data with only a single principal component?

```
# @markdown Make sure you execute this cell to enable the widget!

def refresh(K=100):
  X_reconstructed = reconstruct_data(transformed_data, evectors, X_mean, K)
  plot_reconstructions(X, X_reconstructed)
  plt.title('Reconstructed, K={}'.format(K))

_ = widgets.interact(refresh, K=(1, 784, 10))
```

**Your text answer**

## D) Visualization of the principal components

We can visualize the principal components as images by reversing the flattening. Here we plot using a differenet colormap than black & white as it highlights more structure.

* What structure do you see in the first principal component? What kinds of images would this basis vector differentiate?
* Try visualizing the second and third basis vectors. Do you see any structure? What about the 100th basis vector? 500th? 700th?

```
  plot_principal_components(evectors[:, 0])
```

**Your text answer**

## (Read only) E) Denoising with PCA

We will add some noise to our images to see how PCA reconstructions can be helpful for reducing noise. In this case, we will set 20% of the pixels to random values. We will then visualize some of the noisy images and the resulting cumulative variance explained plot.

In the next cell, we will project the images onto the original PCA space (from the clean, not noisy, data) and then reconstruct from the top 50 components. Observe that this removes the noise quite effectively!

```
# @markdown Execute this cell to visualize noisy data
np.random.seed(2020)  # set random seed
X_noisy = add_noise(X, .2)
score_noisy, evectors_noisy, evals_noisy = pca(X_noisy)
variance_explained_noisy = get_variance_explained(evals_noisy)

plot_sample_images(X_noisy)
plot_variance_explained(variance_explained_noisy)
```

```
# @markdown Execute to visualize denoised reconstructions
X_noisy_mean = np.mean(X_noisy, 0)
projX_noisy = np.matmul(X_noisy - X_noisy_mean, evectors)
X_reconstructed = reconstruct_data(projX_noisy, evectors, X_noisy_mean, 50)

plot_reconstructions(X_noisy, X_reconstructed)
```

# Extra info: PCA & Sklearn

In this tutorial, we created our own functions to compute PCA and reconstruct images so we could better understand the algorithms. Usually though, you would  use `sklearn.decomposition.pca` to perform PCA. Sklearn is a class based package - I have a video explaining the basics of class based programming (object oriented programming) and a video on sklearn as part of my Pandemic Python for Neuroscientists course so check that out if interested.
I'll demonstrate the basics here using some data with 3 features (X).

See docs here: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

```
#  @markdown Execute to generate some data X
np.random.seed(123)
variance_1 = 1
variance_2 = 1
corr_coef = 0.8
cov_matrix = np.array([[1, .2, .3], [.2, 1, .3], [.3, .3, 1]])
X =  np.random.multivariate_normal([5, 10, 2], cov_matrix, size=1000)

```

```
# Import PCA class
from sklearn.decomposition import PCA

# Set up model, tell it the number of components you'll want to keep (if not set, all components will be kept)
pca = PCA(n_components=2)

# Fit the model to your data, aka find the principal components
pca.fit(X)

# Now you can access the principal components
print(pca.components_)

# And the % of explained variance for each component
print(pca.explained_variance_ratio_)

# You can transform your data now
transformed_data = pca.transform(X)

# You could have fit and transformed at the same time if you had wanted
transformed_data = pca.fit_transform(X)

# You can also reconstruct into the original space
reconstruction = pca.inverse_transform(transformed_data)
```


```python
import snap
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import networkx as nx

from matplotlib.pyplot import loglog

%matplotlib inline

```

```python
df = pd.read_csv('/home/merchantsameer2014/project/dnc-temporalGraph/out.dnc-temporalGraph', sep='\t', header=None)
```

```python
df.head(2)
```

      0
      1
      2
      3

      0
      419
      465
      1
      1463507482

      1
      869
      453
      1
      1462337903

```python
df.columns = ['src', 'dst', 'weight', 'timestamp']
```

```python
df.head(2)
```

      src
      dst
      weight
      timestamp

      0
      419
      465
      1
      1463507482

      1
      869
      453
      1
      1462337903

```python
def ProcessEdgeRow(row, Graph):
    src = row['src']
    dst = row['dst']
    timestamp = row['timestamp']

    if not Graph.IsNode(src):
        Graph.AddNode(src)

    if not Graph.IsNode(dst):
        Graph.AddNode(dst)

    if not Graph.IsEdge(src, dst):
        EId = Graph.AddEdge(src, dst)
    else:
        EId = Graph.GetEI(src, dst)

    Graph.AddIntAttrDatE(EId, timestamp, 'timestamp')

    return

```

```python
def GenerateDirectedGraph(df):
    '''
    Returns a TNEANet Graph object
    '''
    Graph = snap.TNEANet.New()
    df.apply(ProcessEdgeRow, axis=1, Graph=Graph)

    return Graph
```

```python
G = GenerateDirectedGraph(df)
```

```python
print "Nodes: ", G.GetNodes()
print "Edges: ", G.GetEdges()
```

    Nodes:  1891
    Edges:  5598

## Get Raw Statistics

    1. Degree Centrality
    2. Clustering Coefficient
    3. Mean shortest path from each node
    4. Betweeness Centralitily
    5. Hub and Authority Score

```python
NodeAttributes = dict()
```

#### Step 1: Degree Centrality

```python
#
# 1. Degree Centrality
#    Get In Degree and Out Degree for each node
#
InDegV = snap.TIntPrV()
OutDegV = snap.TIntPrV()

snap.GetNodeOutDegV(G, OutDegV)
snap.GetNodeInDegV(G, InDegV)

InDegreeList = [ (item.GetVal1(), item.GetVal2()) for item in InDegV ]
OutDegreeList = [ (item.GetVal1(), item.GetVal2()) for item in OutDegV ]
```

```python
InDegreeList.sort(key=lambda x: x[1], reverse=True)
OutDegreeList.sort(key=lambda x: x[1], reverse=True)

minOutDegree = min(OutDegreeList, key = lambda x: x[1])[1]
maxOutDegree = max(OutDegreeList, key = lambda x: x[1])[1]
minInDegree = min(InDegreeList, key = lambda x: x[1])[1]
maxInDegree = max(InDegreeList, key = lambda x: x[1])[1]
```

```python
#
# Sanity Check
print maxOutDegree, minOutDegree, maxInDegree, minInDegree
print InDegreeList[0], InDegreeList[-1]
```

    331 0 249 0
    (1874, 249) (893, 0)

```python
for (nodeId, Degree) in InDegreeList:
    if not NodeAttributes.get(nodeId, None):
        NodeAttributes[nodeId] = dict()
    NodeAttributes[nodeId]['InDegree'] = Degree
    normalizedDegree = (float(Degree) - float(minInDegree))/(float(maxInDegree - float(minInDegree)))
    NodeAttributes[nodeId]['NormInDegree'] = normalizedDegree

for (nodeId, Degree) in OutDegreeList:
    NodeAttributes[nodeId]['OutDegree'] = Degree
    normalizedDegree = (float(Degree) - float(minOutDegree))/(float(maxOutDegree - float(minOutDegree)))
    NodeAttributes[nodeId]['NormOutDegree'] = normalizedDegree
```

```python
#
# Sanity Check
#
print NodeAttributes[1874]
print NodeAttributes[893]
```

    {'InDegree': 249, 'NormInDegree': 1.0, 'NormOutDegree': 1.0, 'OutDegree': 331}
    {'InDegree': 0, 'NormInDegree': 0.0, 'NormOutDegree': 0.0030211480362537764, 'OutDegree': 1}

```python

#
# Get Degree Distribution
#
OutDegToCntV = snap.TIntPrV()
snap.GetOutDegCnt(G, OutDegToCntV)
count = 0
nodeList = []
degreeList = []
for item in OutDegToCntV:
    (n, d) = (item.GetVal2(), item.GetVal1())
    nodeList.append(n)
    degreeList.append(d)
x = np.array( [ np.log10(item.GetVal1()) for itemm in OutDegToCntV if item.GetVal1() > 0 ] )
y = np.array( [ np.log10(item.GetVal2()) for item in OutDegToCntV if item.GetVal2() > 0 ] )
#
# Plot Degree Distribution
#
plt.figure(figsize=(15,15))
loglog(degreeList, nodeList, 'bo')
#plt.plot(x_plot, 10**b*x_plot**a, 'r-')
plt.title("LogLog plot of out-degree distribution")
plt.show()
```

#### Step 2: Clustering Coefficient

```python
NIdCCfH = snap.TIntFltH()
snap.GetNodeClustCf(G, NIdCCfH)
ClusterCoeffList = list()
for nodeId in NIdCCfH:
    NodeAttributes[nodeId]['ClusterCoeff'] = NIdCCfH[nodeId]
    ClusterCoeffList.append((nodeId, NIdCCfH[nodeId]))
```

```python
ClusterCoeffList.sort(key=lambda x: x[1], reverse=True)
minClusterCoeff = min(ClusterCoeffList, key=lambda x: x[1])[1]
maxClusterCoeff = max(ClusterCoeffList, key=lambda x: x[1])[1]
```

```python
#
# Sanity Check
#
print ClusterCoeffList[1], maxClusterCoeff, ClusterCoeffList[-1], minClusterCoeff
```

    (1753, 1.0) 1.0 (1381, 0.0) 0.0

```python
NIdCCfH = snap.TIntFltH()
snap.GetNodeClustCf(G, NIdCCfH)
ClusterCoeffList = list()
for nodeId in NIdCCfH:
    clusterCoeff = NIdCCfH[nodeId]
    normClusterCoeff = (clusterCoeff - minClusterCoeff)/(maxClusterCoeff - minClusterCoeff)
    NodeAttributes[nodeId]['NormClusterCoeff'] = normClusterCoeff
```

```python
print NodeAttributes[2012]
```

    {'NormOutDegree': 0.0, 'ClusterCoeff': 0.6666666666666666, 'InDegree': 3, 'OutDegree': 0, 'NormInDegree': 0.012048192771084338, 'NormClusterCoeff': 0.6666666666666666}

#### Step 3: Avergate shortest path length

```python
nodeCount = float(G.GetNodes() - 1)
avgShortPathLenList = list()
for src in G.Nodes():
    srcId = src.GetId()
    totalShortPathLength = 0

    for dst in G.Nodes():
        dstId = dst.GetId()
        #
        # Skip Self Edges
        #
        if srcId == dstId:
            continue

        #
        # Compute Shortest Path
        #
        l = snap.GetShortPath(G, srcId, dstId, True)

        #
        # Skip nodes that cannot be reached
        #
        if l < 0:
            continue

        totalShortPathLength += float(l)
    NodeAttributes[srcId]['AvgShortPathLen'] = totalShortPathLength/nodeCount
    avgShortPathLenList.append((srcId, totalShortPathLength/nodeCount))
```

```python
minAvgShortPathLength = min(avgShortPathLenList, key=lambda x: x[1])[1]
maxAvgShortPathLength = max(avgShortPathLenList, key=lambda x: x[1])[1]
```

```python
for (node, spLen) in avgShortPathLenList:
    normAvgShortPath = (spLen - minAvgShortPathLength)/(maxAvgShortPathLength - minAvgShortPathLength)
    NodeAttributes[node]['normAvgShortPathLen'] = normAvgShortPath
```

```python
print NodeAttributes[480]
```

    {'NormOutDegree': 0.0030211480362537764, 'ClusterCoeff': 0.3333333333333333, 'normAvgShortPathLen': 1.0, 'InDegree': 2, 'OutDegree': 1, 'NormInDegree': 0.008032128514056224, 'NormClusterCoeff': 0.3333333333333333, 'AvgShortPathLen': 4.321164021164021}

#### Step 4: Betweeness Centrality

```python
Nodes = snap.TIntFltH()
Edges = snap.TIntPrFltH()
BetweenessNodeList = list()
BetweenessEdgeList = list()

snap.GetBetweennessCentr(G, Nodes, Edges, 1.0)
for node in Nodes:
    NodeAttributes[node]['Betweeness'] = Nodes[node]
    BetweenessNodeList.append((node, Nodes[node]))

for edge in Edges:
    #print "edge: (%d, %d) centrality: %f" % (edge.GetVal1(), edge.GetVal2(), Edges[edge])
    BetweenessEdgeList.append((edge.GetVal1(), edge.GetVal2(), Edges[edge]))

BetweenessNodeList.sort(key=lambda x: x[1], reverse=True)
BetweenessEdgeList.sort(key=lambda x: x[2], reverse=True)
```

```python
print BetweenessNodeList[0], BetweenessNodeList[-1]
```

    (1669, 556845.3564806876) (884, 0.0)

```python
minBetweeness = BetweenessNodeList[-1][1]
maxBetweeness = BetweenessNodeList[0][1]
for (node, betweeness) in BetweenessNodeList:
    normBetweeness = (betweeness - minBetweeness)/(maxBetweeness - minBetweeness)
    NodeAttributes[node]['normBetweeness'] = normBetweeness

```

```python
print NodeAttributes[1669]
print NodeAttributes[884]
```

    {'NormOutDegree': 0.7311178247734139, 'ClusterCoeff': 0.0007700181235972993, 'normAvgShortPathLen': 0.33304763070895066, 'InDegree': 222, 'normBetweeness': 1.0, 'OutDegree': 242, 'NormInDegree': 0.891566265060241, 'NormClusterCoeff': 0.0007700181235972993, 'Betweeness': 556845.3564806876, 'AvgShortPathLen': 1.439153439153439}
    {'NormOutDegree': 0.0, 'ClusterCoeff': 0.0, 'normAvgShortPathLen': 0.0, 'InDegree': 1, 'normBetweeness': 0.0, 'OutDegree': 0, 'NormInDegree': 0.004016064257028112, 'NormClusterCoeff': 0.0, 'Betweeness': 0.0, 'AvgShortPathLen': 0.0}

#### Step 5: Get Auth and Hub Score

```python
NIdHubH = snap.TIntFltH()
NIdAuthH = snap.TIntFltH()
snap.GetHits(G, NIdHubH, NIdAuthH)
HubNodes = []
for nodeId in  NIdHubH:
    HubNodes.append((nodeId,  NIdHubH[nodeId]))
    NodeAttributes[nodeId]['HubScore'] = NIdHubH[nodeId]

HubNodes.sort(key = lambda x: x[1], reverse=True)

AuthNodes = []
for nodeId in  NIdAuthH:
    AuthNodes.append((nodeId,  NIdAuthH[nodeId]))
    NodeAttributes[nodeId]['AuthScore'] = NIdAuthH[nodeId]
AuthNodes.sort(key = lambda x: x[1], reverse=True)

```

```python
print AuthNodes[0], AuthNodes[-1]
print HubNodes[0], HubNodes[-1]
```

    (1874, 0.30510831720503034) (893, 0.0)
    (1874, 0.38410704002084095) (884, 0.0)

```python
minAuthNodes = AuthNodes[-1][1]
maxAuthNodes = AuthNodes[0][1]
minHubNodes = HubNodes[-1][1]
maxHubNodes = HubNodes[0][1]
```

```python
for (node, hubScore) in HubNodes:
    normHubScore = (hubScore - minHubNodes)/(maxHubNodes - minHubNodes)
    NodeAttributes[node]['normHubScore'] = normHubScore

for (node, authScore) in AuthNodes:
    normAuthScore = (authScore - minAuthNodes)/(maxAuthNodes - minAuthNodes)
    NodeAttributes[node]['normAuthScore'] = normAuthScore
```

```python
print NodeAttributes[1874]
print NodeAttributes[893]
```

    {'NormOutDegree': 1.0, 'ClusterCoeff': 0.0014452513597956258, 'normAvgShortPathLen': 0.33696583812905595, 'InDegree': 249, 'normBetweeness': 0.8890896997683746, 'normAuthScore': 1.0, 'OutDegree': 331, 'AuthScore': 0.30510831720503034, 'NormInDegree': 1.0, 'normHubScore': 1.0, 'NormClusterCoeff': 0.0014452513597956258, 'HubScore': 0.38410704002084095, 'Betweeness': 495085.4708108281, 'AvgShortPathLen': 1.456084656084656}
    {'NormOutDegree': 0.0030211480362537764, 'ClusterCoeff': 0.0, 'normAvgShortPathLen': 0.0001224439818782907, 'InDegree': 0, 'normBetweeness': 0.0, 'normAuthScore': 0.0, 'OutDegree': 1, 'AuthScore': 0.0, 'NormInDegree': 0.0, 'normHubScore': 0.00048173348087729835, 'NormClusterCoeff': 0.0, 'HubScore': 0.00018503722141871546, 'Betweeness': 0.0, 'AvgShortPathLen': 0.0005291005291005291}

## Automatic Social Hierarchy Detection From Email Network

### Steps

    1. Compute node's importance on response time for email
    2. Get all Cliques (Algorithm 457)
    3. Number of clique each node is part of
    4. Raw Clique Score computed using

 $$R*2^{n-1}$$

    5. Weighted Clique Score (Based on importance of a person)

 $$W = t*2^{n-1}$$

### Use networkx library for Clique analysis

    1. Build a multigraph with edges for each (src, dst, timestamp) entry in the email data set
    2. Build undirected graph with edges between nodes only when email count exceeds N=4 between two those nodes

```python
def ProcessNxEdgeRow(row, Graph):
    src = row['src']
    dst = row['dst']
    timestamp = row['timestamp']

    Graph.add_node(src)
    Graph.add_node(dst)
    Graph.add_edge(src, dst, timestamp=timestamp)

    return

```

```python
def GenerateDirectedNxGraph(df):
    '''
    Returns a TNEANet Graph object
    '''
    Graph = nx.MultiDiGraph(name="DNC Email Network")
    df.apply(ProcessNxEdgeRow, axis=1, Graph=Graph)

    return Graph
```

```python
GNx = GenerateDirectedNxGraph(df)
```

```python
print "Networkx Nodes: ", GNx.number_of_nodes()
print "Networkx Edges: ", GNx.number_of_edges()
```

    Networkx Nodes:  1891
    Networkx Edges:  39264

#### Prune all edges with N <=4 emails exchanged
As per the paper - consider edges between nodes only if the nodes have exchanged > N messages
N is a tunable parameter

DNC email has a median of message exchanged = 2

```python
edgeCount = dict()
for edge in GNx.edges():
    if not edgeCount.get(edge, None):
        edgeCount[edge] = 0
    edgeCount[edge] += 1
```

```python
emailDist= [ v for k, v in edgeCount.iteritems() ]
emailDist.sort(reverse=True)
```

```python
print emailDist[0], emailDist[-1], np.median(emailDist), len(emailDist)
```

    731 1 2.0 5598

```python
N = 4
pruneList = [ k for k, v in edgeCount.iteritems() if v <= N ]
prunedEdgeList = [ k for k, v in edgeCount.iteritems() if v > N ]
```

### Generate undirected graph with edges between nodes with email count > N

```python
uGNx = nx.Graph()

for edge in prunedEdgeList:
    (src, dst) = edge
    uGNx.add_edge(src, dst)

print "Networkx Nodes: ", uGNx.number_of_nodes()
print "Networkx Edges: ", uGNx.number_of_edges()
```

    Networkx Nodes:  452
    Networkx Edges:  1139

### Step 1: Compute Nodes importance on email response time

### Approach 1: Based on Rowe et. al. paper

Use response time to measure importance of a node

    1. For each node get all the out bound email timestamps
    2. For each email sent - check the response time from the receiver
    3. Consider responses within a day for computing time score.
    4. Consider response time for nodes that have exchanged at least 100 emails
       (fewer emails with high response time can falsely promote node based on time score)

```python
temporalMap = dict()
for n, nbrs in GNx.adjacency():
    temporalMap[n] = dict()
    for nbr, edict in nbrs.items():
        t1 = [d['timestamp'] for d in edict.values()]
        t1.sort()
        temporalMap[n][nbr] = t1
```

```python
avgNodeResponse = list()

for src, destinations in temporalMap.iteritems():
    totalRequestResp = 0
    totalResponseTime = 0.0

    for dst, reqTimestamps in destinations.iteritems():
        responseTime = None
        #
        # Walk through ALl requests sent to a destination
        #
        for req in reqTimestamps:
            reqTime = datetime.fromtimestamp(req)

            #
            # Look for response time from the destinaton to ths source
            #
            for resp in temporalMap[dst].get(src, list()):
                respTime = datetime.fromtimestamp(resp)

                if resp < req:
                    #
                    # Look for first response after the request time
                    #
                    continue

                deltaTime = respTime - reqTime

                if deltaTime.total_seconds() > 86400:
                    #
                    # If response time exceeds a day don't
                    # consider it as a response
                    break

                #print "Found Response time: src: %d, dst: %d req: %s, resp: %s" % (src, dst, reqTime, respTime)

                totalRequestResp += 1
                totalResponseTime += deltaTime.total_seconds()
                break
    #
    # Compute average across all dst response times
    #
    if totalRequestResp > 0 and totalResponseTime > 0:
        avgResponse = totalResponseTime/totalRequestResp
    else:
        #
        # Set default response to 7 day
        #
        avgResponse = float(7*86400)
    #
    # Lower response Higher the timeScore
    #
    timeScore = 1/avgResponse
    avgNodeResponse.append((src, timeScore, totalResponseTime, totalRequestResp ))

```

```python
avgNodeTimeScore = list()

#
# Ignore response time for email exchanges fewer than 10
#
for x in avgNodeResponse:
    (src, timeScore, totalResponseTime, totalRequestResp) = x

    if totalRequestResp <= 200:
        timeScore = 1.0/float(7*86400)

    avgNodeTimeScore.append((src, timeScore, totalResponseTime, totalRequestResp))

avgNodeTimeScore.sort(key=lambda x: x[1], reverse=True)

```

```python
print avgNodeTimeScore[-1]
print avgNodeTimeScore[:2]
```

    (2029, 1.6534391534391535e-06, 0.0, 0)
    [(1625, 0.0020221266801294163, 368424.0, 745), (895, 0.00011885108532413246, 1918367.0, 228)]

```python
print NodeAttributes[1625]
```

    {'NormOutDegree': 0.030211480362537766, 'ClusterCoeff': 0.05714285714285714, 'normAvgShortPathLen': 0.4491245255295702, 'InDegree': 7, 'normBetweeness': 0.007319923714214854, 'normAuthScore': 0.12096933772423019, 'OutDegree': 10, 'AuthScore': 0.036908751066446865, 'NormInDegree': 0.028112449799196786, 'normHubScore': 0.09633108380044303, 'NormClusterCoeff': 0.05714285714285714, 'HubScore': 0.037001447460587755, 'Betweeness': 4076.0655300534095, 'AvgShortPathLen': 1.9407407407407407}

```python
normTimeScore = dict()
minAvgTimeScore = avgNodeTimeScore[-1][1]
maxAvgTimeScore = avgNodeTimeScore[0][1]

for (node, timeScore, _, _) in avgNodeTimeScore:
    normTimeScore[node] = (timeScore - minAvgTimeScore)/(maxAvgTimeScore - minAvgTimeScore)
```

```python
normTimeScore[1625]
```

    1.0

```python
nodeCliqueCount = dict()
rawCliqueScore = dict()
weightedCliqueScore = dict()

#
# Find all maximal cliques
#
for clique in nx.find_cliques(uGNx):
    for node in clique:

        if not nodeCliqueCount.get(node, None):
            nodeCliqueCount[node] = 0

        if not rawCliqueScore.get(node, None):
            rawCliqueScore[node] = 0

        if not weightedCliqueScore.get(node, None):
            weightedCliqueScore[node] = 0

        nodeCliqueCount[node] += 1

        n = len(clique)

        rawCliqueScore[node] += 2**n

        weightedCliqueScore[node] += (2**n)*normTimeScore[node]
```

```python
#
# Get a sorted list of nodes based on their clique count
#
nodeCliqueList = [ (k, v) for k, v in nodeCliqueCount.iteritems() ]
nodeCliqueList.sort(key=lambda x: x[1], reverse=True )
```

```python
rawCliqueList = [ (k, v) for k, v in rawCliqueScore.iteritems() ]
rawCliqueList.sort(key=lambda x: x[1], reverse=True )
```

```python
weightedCliqueList = [ (k, v) for k, v in weightedCliqueScore.iteritems() ]
weightedCliqueList.sort(key=lambda x: x[1], reverse=True )
```

```python
print nodeCliqueList[:10]
print rawCliqueList[:10]
print weightedCliqueList[:10]
```

    [(1874, 157), (1839, 123), (1669, 104), (999, 79), (1258, 79), (1369, 76), (453, 75), (585, 66), (1144, 57), (1159, 46)]
    [(1874, 29604), (999, 27636), (1258, 27616), (511, 22656), (1369, 21044), (1974, 18560), (1998, 18416), (585, 17972), (1144, 12248), (1899, 10304)]
    [(1258, 1133.2113553681547), (1874, 1079.971912453243), (999, 887.6525051646298), (511, 671.973655462151), (1998, 455.752744276308), (585, 335.72489139325376), (1669, 246.35443082589936), (1440, 223.6014465859277), (1287, 134.0397246718862), (453, 98.89859633344226)]

```python
minNodeCliqueCount = nodeCliqueList[-1][1]
maxNodeCliqueCount = nodeCliqueList[0][1]

minRawCliqueScore = rawCliqueList[-1][1]
maxRawCliqueScore = rawCliqueList[0][1]

minWeightedCliqueScore = weightedCliqueList[-1][1]
maxWeightedCliqueScore = weightedCliqueList[0][1]
```

```python
for node, score in nodeCliqueList:
    NodeAttributes[node]['nodeClique'] = score
    NodeAttributes[node]['normNodeClique'] = float(score - minNodeCliqueCount)/float(maxNodeCliqueCount - minNodeCliqueCount)
```

```python
for node, score in rawCliqueList:
    NodeAttributes[node]['rawClique'] = score
    NodeAttributes[node]['normRawClique'] = float(score - minRawCliqueScore)/float(maxRawCliqueScore - minRawCliqueScore)
```

```python
for node, score in weightedCliqueList:
    NodeAttributes[node]['weigthedClique'] = score
    NodeAttributes[node]['normWeightedClique'] = float(score - minWeightedCliqueScore)/float(maxWeightedCliqueScore - minWeightedCliqueScore)
```

```python
NodeAttributes[1874]
```

    {'AuthScore': 0.30510831720503034,
     'AvgShortPathLen': 1.456084656084656,
     'Betweeness': 495085.4708108281,
     'ClusterCoeff': 0.0014452513597956258,
     'HubScore': 0.38410704002084095,
     'InDegree': 249,
     'NormClusterCoeff': 0.0014452513597956258,
     'NormInDegree': 1.0,
     'NormOutDegree': 1.0,
     'OutDegree': 331,
     'nodeClique': 157,
     'normAuthScore': 1.0,
     'normAvgShortPathLen': 0.33696583812905595,
     'normBetweeness': 0.8890896997683746,
     'normHubScore': 1.0,
     'normNodeClique': 1.0,
     'normRawClique': 1.0,
     'normWeightedClique': 0.9530189645005671,
     'rawClique': 29604,
     'weigthedClique': 1079.971912453243}

## Compute Weighted Social Score

```python
#
# Weights:
#
# W = [ Weighted Clique, RawClique, NumClique, OutDegree, InDegree, Cluster Coeff, Betweeness, Avg ShortPath, Auth, Hub]
#
w_weightedClique = 0.9
w_rawClique = 0.8
w_numClique = 0.7
w_outDegree = 0.6
w_inDegree = 0.6
w_clusterCoeff = 0.3
w_betweeness = 0.3
w_shortpath = 0.5
w_auth = 0.2
w_hub = 0.1

w = np.array([w_weightedClique,\
             w_rawClique, \
             w_numClique, \
             w_outDegree, \
             w_inDegree, \
             w_clusterCoeff, \
             w_betweeness, \
             w_shortpath, \
             w_auth, \
             w_hub])

w_sum = np.sum(w)
```

### Social Score is computed as follows

\begin{align}
w_x * C_x & = w_x * 100 * \left[ \frac{x_i - infx}{sup\ x - inf\ x} \right] \\
\\
score & = \frac{\Sigma_{all\ x} w * C_x}{\Sigma_{all\ x} w}
\end{align}

```python
socialScore = list()

for node, attributes in NodeAttributes.iteritems():
    weightedClique = attributes.get('normWeightedClique', 0.0)
    rawClique = attributes.get('normRawClique', 0.0)
    numClique = attributes.get('normNodeClique', 0.0)
    outDeg =  attributes.get('NormOutDegree', 0.0)
    inDeg =  attributes.get('NormInDegree', 0.0)
    clusterCoeff =  attributes.get('NormClusterCoeff', 0.0)
    betweeness = attributes.get('normBetweeness', 0.0)
    shortestPath = attributes.get('normAvgShortPathLen', 0.0)
    authScore = attributes.get('normAuthScore', 0.0)
    hubSccore = attributes.get('normHubScore', 0.0)

    C_x = np.array([weightedClique, \
                   rawClique, \
                   numClique, \
                   outDeg, \
                   inDeg, \
                   clusterCoeff, \
                   betweeness, \
                   shortestPath, \
                   authScore, \
                   hubSccore])

    score = 100.0 * np.dot(w, C_x)/w_sum
    socialScore.append((node, score))

```

```python
socialScore.sort(key=lambda x: x[1], reverse=True)
```

```python
print socialScore[:10]
```

    [(1874, 85.86720944906979), (1258, 55.9778907924103), (999, 50.61051396942534), (1669, 47.27542700847994), (511, 40.31705218340864), (585, 33.064089627423826), (1998, 32.41884981873541), (1839, 28.873591736745578), (1369, 28.7430390082418), (453, 28.201871267917863)]

```python
with open('socialScore.txt', 'w') as fd:
    for (node, score) in socialScore:
        fd.write("%r  %r\n" % (node, score))
```


# Neuromatch Academy: Week 2, Day 5, Tutorial 1
# Learning to Predict

__Content creators:__ Marcelo Mattar and Eric DeWitt with help from Matt Krause

__Content reviewers:__ Byron Galbraith and Michael Waskom

---

# Tutorial objectives

In this tutorial, we will learn how to estimate state-value functions in a classical conditioning paradigm using Temporal Difference (TD) learning and examine TD-errors at the presentation of the conditioned and unconditioned stimulus (CS and US) under different CS-US contingencies. These exercises will provide you with an understanding of both how reward prediction errors (RPEs) behave in classical conditioning and what we should expect to see if Dopamine represents a "canonical" model-free RPE.

At the end of this tutorial:
* You will learn to use the standard tapped delay line conditioning model
* You will understand how RPEs move to CS
* You will understand how variability in reward size effects RPEs
* You will understand how differences in US-CS timing effect RPEs

```python
# Imports
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper functions
from matplotlib import ticker

def plot_value_function(V, ax=None, show=True):
  """Plot V(s), the value function"""
  if not ax:
    fig, ax = plt.subplots()

  ax.stem(V, use_line_collection=True)
  ax.set_ylabel('Value')
  ax.set_xlabel('State')
  ax.set_title("Value function: $V(s)$")

  if show:
    plt.show()

def plot_tde_trace(TDE, ax=None, show=True, skip=400):
  """Plot the TD Error across trials"""
  if not ax:
    fig, ax = plt.subplots()

  indx = np.arange(0, TDE.shape[1], skip)
  im = ax.imshow(TDE[:,indx])
  positions = ax.get_xticks()
  # Avoid warning when setting string tick labels
  ax.xaxis.set_major_locator(ticker.FixedLocator(positions))
  ax.set_xticklabels([f"{int(skip * x)}" for x in positions])
  ax.set_title('TD-error over learning')
  ax.set_ylabel('State')
  ax.set_xlabel('Iterations')
  ax.figure.colorbar(im)
  if show:
    plt.show()

def learning_summary_plot(V, TDE):
  """Summary plot for Ex1"""
  fig, (ax1, ax2) = plt.subplots(nrows = 2, gridspec_kw={'height_ratios': [1, 2]})

  plot_value_function(V, ax=ax1, show=False)
  plot_tde_trace(TDE, ax=ax2, show=False)
  plt.tight_layout()

def reward_guesser_title_hint(r1, r2):
  """"Provide a mildly obfuscated hint for a demo."""
  if (r1==14 and r2==6) or (r1==6 and r2==14):
    return "Technically correct...(the best kind of correct)"

  if  ~(~(r1+r2) ^ 11) - 1 == (6 | 24): # Don't spoil the fun :-)
    return "Congratulations! You solved it!"

  return "Keep trying...."

#@title Default title text
class ClassicalConditioning:

    def __init__(self, n_steps, reward_magnitude, reward_time):

        # Task variables
        self.n_steps = n_steps
        self.n_actions = 0
        self.cs_time = int(n_steps/4) - 1

        # Reward variables
        self.reward_state = [0,0]
        self.reward_magnitude = None
        self.reward_probability = None
        self.reward_time = None

        self.set_reward(reward_magnitude, reward_time)

        # Time step at which the conditioned stimulus is presented

        # Create a state dictionary
        self._create_state_dictionary()

    def set_reward(self, reward_magnitude, reward_time):

        """
        Determine reward state and magnitude of reward
        """
        if reward_time >= self.n_steps - self.cs_time:
            self.reward_magnitude = 0

        else:
            self.reward_magnitude = reward_magnitude
            self.reward_state = [1, reward_time]

    def get_outcome(self, current_state):

        """
        Determine next state and reward
        """
        # Update state
        if current_state < self.n_steps - 1:
            next_state = current_state + 1
        else:
            next_state = 0

        # Check for reward
        if self.reward_state == self.state_dict[current_state]:
            reward = self.reward_magnitude
        else:
            reward = 0

        return next_state, reward

    def _create_state_dictionary(self):

        """
        This dictionary maps number of time steps/ state identities
        in each episode to some useful state attributes:

        state      - 0 1 2 3 4 5 (cs) 6 7 8 9 10 11 12 ...
        is_delay   - 0 0 0 0 0 0 (cs) 1 1 1 1  1  1  1 ...
        t_in_delay - 0 0 0 0 0 0 (cs) 1 2 3 4  5  6  7 ...
        """
        d = 0

        self.state_dict = {}
        for s in range(self.n_steps):
            if s <= self.cs_time:
                self.state_dict[s] = [0,0]
            else:
                d += 1 # Time in delay
                self.state_dict[s] = [1,d]

class MultiRewardCC(ClassicalConditioning):
  """Classical conditioning paradigm, except that one randomly selected reward,
    magnitude, from a list, is delivered of a single fixed reward."""
  def __init__(self, n_steps, reward_magnitudes, reward_time=None):
    """"Build a multi-reward classical conditioning environment
      Args:
        - nsteps: Maximum number of steps
        - reward_magnitudes: LIST of possible reward magnitudes.
        - reward_time: Single fixed reward time
      Uses numpy global random state.
      """
    super().__init__(n_steps, 1, reward_time)
    self.reward_magnitudes = reward_magnitudes

  def get_outcome(self, current_state):
    next_state, reward = super().get_outcome(current_state)
    if reward:
      reward=np.random.choice(self.reward_magnitudes)
    return next_state, reward

class ProbabilisticCC(ClassicalConditioning):
  """Classical conditioning paradigm, except that rewards are stochastically omitted."""
  def __init__(self, n_steps, reward_magnitude, reward_time=None, p_reward=0.75):
    """"Build a multi-reward classical conditioning environment
      Args:
        - nsteps: Maximum number of steps
        - reward_magnitudes: Reward magnitudes.
        - reward_time: Single fixed reward time.
        - p_reward: probability that reward is actually delivered in rewarding state
      Uses numpy global random state.
      """
    super().__init__(n_steps, reward_magnitude, reward_time)
    self.p_reward = p_reward

  def get_outcome(self, current_state):
    next_state, reward = super().get_outcome(current_state)
    if reward:
      reward*= int(np.random.uniform(size=1)[0] < self.p_reward)
    return next_state, reward

```

---
# Section 1: TD-learning

```python
#@title Video 1: Introduction
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV13f4y1d7om', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV13f4y1d7om

__Environment:__

- The agent experiences the environment in episodes or trials.
- Episodes terminate by transitioning to the inter-trial-interval (ITI) state and they are initiated from the ITI state as well. We clamp the value of the terminal/ITI states to zero.
- The classical conditioning environment is composed of a sequence of states that the agent deterministically transitions through. Starting at State 0, the agent moves to State 1 in the first step, from State 1 to State 2 in the second, and so on.  These states represent time in the tapped delay line representation
- Within each episode, the agent is presented a CS and US (reward).
- The CS is always presented at 1/4 of the total duration of the trial. The US (reward) is then delivered after the CS. The interval between the CS and US is specified by `reward_time`.
- The agent's goal is to learn to predict expected rewards from each state in the trial.

**General concepts**

* Return $G_{t}$: future cumulative reward, which can be written in arecursive form
\begin{align}
G_{t} &= \sum \limits_{k = 0}^{\infty} \gamma^{k} r_{t+k+1} \\
&= r_{t+1} + \gamma G_{t+1}
\end{align}
where $\gamma$ is discount factor that controls the importance of future rewards, and $\gamma \in [0, 1]$. $\gamma$ may also be interpreted as probability of continuing the trajectory.
* Value funtion $V_{\pi}(s_t=s)$: expecation of the return
\begin{align}
V_{\pi}(s_t=s) &= \mathbb{E} [ G_{t}\; | \; s_t=s, a_{t:\infty}\sim\pi] \\
& = \mathbb{E} [ r_{t+1} + \gamma G_{t+1}\; | \; s_t=s, a_{t:\infty}\sim\pi]
\end{align}
With an assumption of **Markov process**, we thus have:
\begin{align}
V_{\pi}(s_t=s) &= \mathbb{E} [ r_{t+1} + \gamma V_{\pi}(s_{t+1})\; | \; s_t=s, a_{t:\infty}\sim\pi] \\
&= \sum_a \pi(a|s) \sum_{r, s'}p(s', r)(r + V_{\pi}(s_{t+1}=s'))
\end{align}

**Temporal difference (TD) learning**

* With a Markovian assumption, we can use $V(s_{t+1})$ as an imperfect proxy for the true value $G_{t+1}$ (Monte Carlo bootstrapping), and thus obtain the generalised equation to calculate TD-error:
\begin{align}
\delta_{t} = r_{t+1} + \gamma V(s_{t+1}) - V(s_{t})
\end{align}

* Value updated by using the learning rate constant $\alpha$:
\begin{align}
V(s_{t}) \leftarrow V(s_{t}) + \alpha \delta_{t}
\end{align}

  (Reference: https://web.stanford.edu/group/pdplab/pdphandbook/handbookch10.html)

__Definitions:__

* TD-error:
\begin{align}
\delta_{t} = r_{t+1} + \gamma V(s_{t+1}) - V(s_{t})
\end{align}

* Value updates:
\begin{align}
V(s_{t}) \leftarrow V(s_{t}) + \alpha \delta_{t}
\end{align}

## Exercise 1: TD-learning with guaranteed rewards

Implement TD-learning to estimate the state-value function in the classical-conditioning world with guaranteed rewards, with a fixed magnitude, at a fixed delay after the conditioned stimulus, CS. Save TD-errors over learning (i.e., over trials) so we can visualize them afterwards.

In order to simulate the effect of the CS, you should only update $V(s_{t})$ during the delay period after CS. This period is indicated by the boolean variable `is_delay`. This can be implemented by multiplying the expression for updating the value function by `is_delay`.

Use the provided code to estimate the value function.

```python
def td_learner(env, n_trials, gamma=0.98, alpha=0.001):
  """ Temporal Difference learning

  Args:
    env (object): the environment to be learned
    n_trials (int): the number of trials to run
    gamma (float): temporal discount factor
    alpha (float): learning rate

  Returns:
    ndarray, ndarray: the value function and temporal difference error arrays
  """
  V = np.zeros(env.n_steps) # Array to store values over states (time)
  TDE = np.zeros((env.n_steps, n_trials)) # Array to store TD errors

  for n in range(n_trials):
    state = 0 # Initial state
    for t in range(env.n_steps):
      # Get next state and next reward
      next_state, reward = env.get_outcome(state)
      # Is the current state in the delay period (after CS)?
      is_delay = env.state_dict[state][0]

      ########################################################################
      ## TODO for students: implement TD error and value function update
      # Fill out function and remove
      raise NotImplementedError("Student excercise: implement TD error and value function update")
      #################################################################################
      # Write an expression to compute the TD-error
      TDE[state, n] = ...

      # Write an expression to update the value function
      V[state] += ...

      # Update state
      state = next_state

  return V, TDE

# Uncomment once the td_learner function is complete
# env = ClassicalConditioning(n_steps=40, reward_magnitude=10, reward_time=10)
# V, TDE = td_learner(env, n_trials=20000)
# learning_summary_plot(V, TDE)
```

```python
#to_remove solution
def td_learner(env, n_trials, gamma=0.98, alpha=0.001):
  """ Temporal Difference learning

  Args:
    env (object): the environment to be learned
    n_trials (int): the number of trials to run
    gamma (float): temporal discount factor
    alpha (float): learning rate

  Returns:
    ndarray, ndarray: the value function and temporal difference error arrays
  """
  V = np.zeros(env.n_steps) # Array to store values over states (time)
  TDE = np.zeros((env.n_steps, n_trials)) # Array to store TD errors

  for n in range(n_trials):
    state = 0 # Initial state
    for t in range(env.n_steps):
      # Get next state and next reward
      next_state, reward = env.get_outcome(state)
      # Is the current state in the delay period (after CS)?
      is_delay = env.state_dict[state][0]

      # Write an expression to compute the TD-error
      TDE[state, n] = (reward + gamma * V[next_state] - V[state])

      # Write an expression to update the value function
      V[state] += alpha * TDE[state, n] * is_delay

      # Update state
      state = next_state

  return V, TDE

env = ClassicalConditioning(n_steps=40, reward_magnitude=10, reward_time=10)
V, TDE = td_learner(env, n_trials=20000)
with plt.xkcd():
  learning_summary_plot(V, TDE)
```

## Interactive Demo 1: US to CS Transfer

During classical conditioning, the subject's behavioral response (e.g., salivating) transfers from the unconditioned stimulus (US; like the smell of tasty food) to the conditioned stimulus (CS; like Pavlov ringing his bell) that predicts it. Reward prediction errors play an important role in this process by adjusting the value of states according to their expected, discounted return.

Use the widget below to examine how reward prediction errors change over time. Before training (orange line), only the reward state has high reward prediction error. As training progresses (blue line, slider), the reward prediction errors shift to the conditioned stimulus, where they end up when the trial is complete (green line).

Dopamine neurons, which are thought to carry reward prediction errors _in vivo_, show exactly the same behavior!

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

n_trials = 20000

@widgets.interact
def plot_tde_by_trial(trial = widgets.IntSlider(value=5000, min=0, max=n_trials-1 , step=1, description="Trial #")):
  if 'TDE' not in globals():
    print("Complete Exercise 1 to enable this interactive demo!")
  else:

    fig, ax = plt.subplots()
    ax.axhline(0, color='k') # Use this + basefmt=' ' to keep the legend clean.
    ax.stem(TDE[:, 0], linefmt='C1-', markerfmt='C1d', basefmt=' ',
            label="Before Learning (Trial 0)",
            use_line_collection=True)
    ax.stem(TDE[:, -1], linefmt='C2-', markerfmt='C2s', basefmt=' ',
            label="After Learning (Trial $\infty$)",
            use_line_collection=True)
    ax.stem(TDE[:, trial], linefmt='C0-', markerfmt='C0o', basefmt=' ',
            label=f"Trial {trial}",
            use_line_collection=True)

    ax.set_xlabel("State in trial")
    ax.set_ylabel("TD Error")
    ax.set_title("Temporal Difference Error by Trial")
    ax.legend()
```

## Interactive Demo 2: Learning Rates and Discount Factors

Our TD-learning agent has two parameters that control how it learns: $\alpha$, the learning rate, and $\gamma$, the discount factor. In Exercise 1, we set these parameters to $\alpha=0.001$ and $\gamma=0.98$ for you. Here, you'll investigate how changing these parameters alters the model that TD-learning learns.

Before enabling the interactive demo below, take a moment to think about the functions of these two parameters. $\alpha$ controls the size of the Value function updates produced by each TD-error. In our simple, deterministic world, will this affect the final model we learn? Is a larger $\alpha$ necessarily better in more complex, realistic environments?

The discount rate $\gamma$ applies an exponentially-decaying weight to returns occuring in the future, rather than the present timestep. How does this affect the model we learn? What happens when $\gamma=0$ or $\gamma \geq 1$?

Use the widget to test your hypotheses.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact
def plot_summary_alpha_gamma(alpha = widgets.FloatSlider(value=0.0001, min=0.001, max=0.1, step=0.0001, description="alpha"),
                             gamma = widgets.FloatSlider(value=0.980, min=0, max=1.1, step=0.010, description="gamma")):
  env = ClassicalConditioning(n_steps=40, reward_magnitude=10, reward_time=10)
  try:
    V_params, TDE_params = td_learner(env, n_trials=20000, gamma=gamma, alpha=alpha)
  except NotImplementedError:
    print("Finish Exercise 1 to enable this interactive demo")

  learning_summary_plot(V_params,TDE_params)

```

```python
#to_remove explanation
"""
Alpha determines how fast the model learns. In the simple, deterministic world
we're using here, this allows the moodel to quickly converge onto the "true"
model that heavily values the conditioned stimulus. In more complex environments,
however, excessively large values of alpha can slow, or even prevent, learning,
as we'll see later.

Gamma effectively controls how much the model cares about the future: larger values of
gamma cause the model to weight future rewards nearly as much as present ones. At gamma=1,
the model weights all rewards, regardless of when they occur, equally and when greater than one, it
starts to *prefer* rewards in the future, rather than the present (this is rarely good).
When gamma=0, however, the model becomes greedy and only considers rewards that
can be obtained immediately.
 """;
```

---
# Section 2: TD-learning with varying reward magnitudes

In the previous exercise, the environment was as simple as possible. On every trial, the CS predicted the same reward, at the same time, with 100% certainty. In the next few exercises, we will make the environment more progressively more complicated and examine the TD-learner's behavior.

## Interactive Demo 3: Match the Value Functions

First, will replace the environment with one that dispenses one of several rewards, chosen at random. Shown below is the final value function $V$ for a TD learner that was trained in an enviroment where the CS predicted a reward of 6 or 14 units; both rewards were equally likely).

Can you find another pair of rewards that cause the agent to learn the same value function? Assume each reward will be dispensed 50% of the time.

Hints:
* Carefully consider the definition of the value function $V$. This can be solved analytically.
* There is no need to change $\alpha$ or $\gamma$.
* Due to the randomness, there may be a small amount of variation.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

n_trials = 20000
np.random.seed(2020)
rng_state = np.random.get_state()
env = MultiRewardCC(40, [6, 14], reward_time=10)
V_multi, TDE_multi = td_learner(env, n_trials, gamma=0.98, alpha=0.001)

@widgets.interact
def reward_guesser_interaction(r1 = widgets.IntText(value=0, min=0, max=50, description="Reward 1"),
                               r2 = widgets.IntText(value=0, min=0, max=50, description="Reward 2")):
  try:
    env2 = MultiRewardCC(40, [r1, r2], reward_time=10)
    V_guess, _ = td_learner(env2, n_trials, gamma=0.98, alpha=0.001)
    fig, ax = plt.subplots()
    m, l, _ = ax.stem(V_multi, linefmt='y-', markerfmt='yo', basefmt=' ', label="Target",
            use_line_collection=True)
    m.set_markersize(15)
    m.set_markerfacecolor('none')
    l.set_linewidth(4)
    m, _, _ = ax.stem(V_guess, linefmt='r', markerfmt='rx', basefmt=' ', label="Guess",
                      use_line_collection=True)
    m.set_markersize(15)

    ax.set_xlabel("State")
    ax.set_ylabel("Value")
    ax.set_title("Guess V(s)\n" + reward_guesser_title_hint(r1, r2))
    ax.legend()
  except NotImplementedError:
    print("Please finish Exercise 1 first!")
```

## Section 2.1 Examining the TD Error

Run the cell below to plot the TD errors from our multi-reward environment. A new feature appears in this plot? What is it? Why does it happen?

```python
plot_tde_trace(TDE_multi)
```

```python
#to_remove explanation
"""
The TD trace now takes on negative values because the reward delivered is
sometimes larger than the expected reward and sometimes smaller.
""";
```

---
# Section 3: TD-learning with probabilistic rewards

In this environment, we'll return to delivering a single reward of ten units. However, it will be delivered intermittently: on 20 percent of trials, the CS will be shown but the agent will not receive the usual reward; the remaining 80% will proceed as usual.

 Run the cell below to simulate. How does this compare with the previous experiment?

Earlier in the notebook, we saw that changing $\alpha$ had little effect on learning in a deterministic environment. What happens if you set it to an large value, like 1, in this noisier scenario? Does it seem like it will _ever_ converge?

```python
np.random.set_state(rng_state) # Resynchronize everyone's notebooks
n_trials = 20000
try:
  env = ProbabilisticCC(n_steps=40, reward_magnitude=10, reward_time=10,
                        p_reward=0.8)
  V_stochastic, TDE_stochastic = td_learner(env, n_trials*2, alpha=1)
  learning_summary_plot(V_stochastic, TDE_stochastic)
except NotImplementedError:
  print("Please finish Exercise 1 first")
```

```python
#to_remove explanation
"""
The multi-reward and probabilistic reward enviroments are the same. You
could simulate a probabilistic reward of 10 units, delivered 50% of the time,
by having a mixture of 10 and 0 unit rewards, or vice versa. The takehome message
from these last three exercises is that the *average* or expected reward is
what matters during TD learning.

Large values of alpha prevent the TD Learner from converging. As a result, the
value function seems implausible: one state may have an extremely low value while
the neighboring ones remain high. This pattern persists even if training continues
for hundreds of thousands of trials.
""";
```

---
# Summary

In this notebook, we have developed a simple TD Learner and examined how its state representations and reward prediction errors evolve during training. By manipualting its environment and parameters ($\alpha$, $\gamma$), you developed an intuition for how it behaves.

This simple model closely resembles the behavior of subjects undergoing classical conditioning tasks and the dopamine neurons that may underlie that behavior. You may have implemented TD-reset or used the model to recreate a common experimental error. The update rule used here has been extensively studied for [more than 70 years](https://www.pnas.org/content/108/Supplement_3/15647) as a possible explanation for artificial and biological learning.

However, you may have noticed that something is missing from this notebook. We carefully calculated the value of each state, but did not use it to actually do anything. Using values to plan _**Actions**_ is coming up next!

# Bonus

## Exercise 2: Removing the CS

In Exercise 1, you (should have) included a term that depends on the conditioned stimulus. Remove it and see what happens. Do you understand why?
This phenomena often fools people attempting to train animals--beware!

```python
#to_remove explanation

"""
You should only be updating the V[state] once the conditioned stimulus appears.

If you remove this term the Value Function becomes periodic, dropping towards zero
right after the reward and gradually rising towards the end of the trial. This
behavior is actually correct, because the model is learning the time until the
*next* reward, and State 37 is closer to a reward than State 21 or 22.

In an actual experiment, the animal often just wants rewards; it doesn't care about
/your/ experiment or trial structure!
""";
```


```python
%matplotlib inline
```

序列模型和 LSTM 网络（长短记忆网络）
===================================================

之前我们已经学过了许多的前馈网络. 所谓前馈网络, 就是网络中不会保存状态. 然而有时
这并不是我们想要的效果. 在自然语言处理 (NLP, Natural Language Processing)
中, 序列模型是一个核心的概念. 所谓序列模型, 即输入依赖于时间信息的模型. 一个典型
的序列模型是隐马尔科夫模型 (HMM, Hidden Markov Model). 另一个序列模型的例子
是条件随机场 (CRF, Conditional Random Field).

递归神经网络是指可以保存某种状态的神经网络. 比如说, 网络上个时刻的输出可以作为下个
时刻的输入, 这样信息就可以通过序列在网络中一直往后传递. 对于LSTM (Long-Short
Term Memory) 来说, 序列中的每个元素都有一个相应的隐状态 $h_t$, 该隐状态
原则上可以包含序列当前结点之前的任一节点的信息. 我们可以使用隐藏状态来预测语言模型
中的单词, 词性标签以及其他各种各样的东西.

Pytorch 中的 LSTM
~~~~~~~~~~~~~~~~~~

开始例子之前,有几个点说明一下. Pytorch 中, LSTM 的所有的形式固定为3D 的 tensor.
每个维度有固定的语义含义, 不能乱掉. 其中第一维是序列本身, 第二维以 mini-batch 形式
来索引实例, 而第三维则索引输入的元素. 因为我们没有讨论过 mini-batch, 所以在这里我们
假设第二维的维度总是1. 如果我们想在句子 "The cow jumped" 上运行一个序列模型, 模型
的输入类似这样:

\begin{align}\begin{bmatrix}
   \overbrace{q_\text{The}}^\text{row vector} \\
   q_\text{cow} \\
   q_\text{jumped}
   \end{bmatrix}\end{align}

除了有一个额外的大小为1的第二维度.

此外, 你还可以向网络逐个输入序列, 在这种情况下, 第一个轴的大小也是1.

来看一个简单的例子.

```python
# 作者: Robert Guthrie

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(1)
```

    <torch._C.Generator at 0x114b251f0>

```python
lstm = nn.LSTM(3, 3)  # 输入维度是3, 输出维度也是3
inputs = [autograd.Variable(torch.randn((1, 3)))
          for _ in range(5)]  # 构造一个长度为5的序列

# 初始化隐藏状态
hidden = (autograd.Variable(torch.randn(1, 1, 3)),
          autograd.Variable(torch.randn((1, 1, 3))))
for i in inputs:
    # 将序列的元素逐个输入到LSTM
    # 经过每步操作,hidden 的值包含了隐藏状态的信息
    out, hidden = lstm(i.view(1, 1, -1), hidden)

# 另外, 我们还可以一次对整个序列进行训练. LSTM 返回的第一个值表示所有时刻的隐状态值,
# 第二个值表示最近的隐状态值 (因此下面的 "out"的最后一个值和 "hidden" 的值是一样的).
# 之所以这样设计, 是为了通过 "out" 的值来获取所有的隐状态值, 而用 "hidden" 的值来
# 进行序列的反向传播运算, 具体方式就是将它作为参数传入后面的 LSTM 网络.

# 增加额外的第二个维度
inputs = torch.cat(inputs).view(len(inputs), 1, -1)
hidden = (autograd.Variable(torch.randn(1, 1, 3)), autograd.Variable(
    torch.randn((1, 1, 3))))  # 清空输出隐状态
out, hidden = lstm(inputs, hidden)
print(out)
print(hidden)
```

    tensor([[[-0.0187,  0.1713, -0.2944]],

            [[-0.3521,  0.1026, -0.2971]],

            [[-0.3191,  0.0781, -0.1957]],

            [[-0.1634,  0.0941, -0.1637]],

            [[-0.3368,  0.0959, -0.0538]]], grad_fn=<CatBackward>)
    (tensor([[[-0.3368,  0.0959, -0.0538]]], grad_fn=<ViewBackward>), tensor([[[-0.9825,  0.4715, -0.0633]]], grad_fn=<ViewBackward>))

例子: 用 LSTM 来进行词性标注
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

在这部分, 我们将会使用一个 LSTM 网络来进行词性标注. 在这里我们不会用到维特比算法,
前向后向算法或者任何类似的算法, 而是将这部分内容作为一个 (有挑战) 的练习留给读者,
希望读者在了解了这部分的内容后能够实现如何将维特比算法应用到 LSTM 网络中来.

整个模型的参数定义如下: 输入的句子定义为 $w_1, \dots, w_M$, 其中动词定义
为 $w_1, \dots, w_M$, 标签集合定义为 $T$, 单词 $w_i$ 的实际
标签为 $y_i$. 定义单词 $w_i$ 的预测标签为 $\hat{y}_i$.

这是一个结构预测模型, 我们的输出是一个序列 $\hat{y}_1, \dots, \hat{y}_M$,
其中 $\hat{y}_i \in T$.

在进行预测时, 需将句子每个词输入到一个 LSTM 网络中. 将时刻 $i$ 的隐状态标记
为 $h_i$. 同样地, 对每个标签赋一个独一无二的索引 (类似 word embeddings 部分
word\_to\_ix 的设置). 然后就得到了 $\hat{y}_i$ 的预测规则:

\begin{align}\hat{y}_i = \text{argmax}_j \  (\log \text{Softmax}(Ah_i + b))_j\end{align}

即先对隐状态进行一个仿射变换, 然后计算一个对数 softmax, 最后得到的预测标签即为对数
softmax 中最大的值对应的标签. 注意, 这也意味着 $A$ 空间的维度是 $|T|$.

准备数据:

```python
def prepare_sequence(seq, to_ix):
    idxs = [to_ix[w] for w in seq]
    tensor = torch.LongTensor(idxs)
    return autograd.Variable(tensor)

training_data = [
    ("The dog ate the apple".split(), ["DET", "NN", "V", "DET", "NN"]),
    ("Everybody read that book".split(), ["NN", "V", "DET", "NN"])
]
word_to_ix = {}
for sent, tags in training_data:
    for word in sent:
        if word not in word_to_ix:
            word_to_ix[word] = len(word_to_ix)
print(word_to_ix)
tag_to_ix = {"DET": 0, "NN": 1, "V": 2}

# 实际中通常使用更大的维度如32维, 64维.
# 这里我们使用小的维度, 为了方便查看训练过程中权重的变化.
EMBEDDING_DIM = 6
HIDDEN_DIM = 6
```

    {'The': 0, 'dog': 1, 'ate': 2, 'the': 3, 'apple': 4, 'Everybody': 5, 'read': 6, 'that': 7, 'book': 8}

构造模型:

```python
class LSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

        #  LSTM 以 word_embeddings 作为输入, 输出维度为 hidden_dim 的隐状态值
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # 线性层将隐状态空间映射到标注空间
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        # 开始时刻, 没有隐状态
        # 关于维度设置的详情,请参考 Pytorch 文档
        # 各个维度的含义是 (num_layers, minibatch_size, hidden_dim)
        return (autograd.Variable(torch.zeros(1, 1, self.hidden_dim)),
                autograd.Variable(torch.zeros(1, 1, self.hidden_dim)))

    def forward(self, sentence):
        embeds = self.word_embeddings(sentence)
        lstm_out, self.hidden = self.lstm(
            embeds.view(len(sentence), 1, -1), self.hidden)
        tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))
        tag_scores = F.log_softmax(tag_space, dim=1)
        return tag_scores
```

训练模型:

```python
model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix))
loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# 查看下训练前得分的值
# 注意: 输出的 i,j 元素的值表示单词 i 的 j 标签的得分
inputs = prepare_sequence(training_data[0][0], word_to_ix)
tag_scores = model(inputs)
print(tag_scores)

for epoch in range(300):  # 再次说明下, 实际情况下你不会训练300个周期, 此例中我们只是构造了一些假数据
    for sentence, tags in training_data:
        # Step 1. 请记住 Pytorch 会累加梯度
        # 每次训练前需要清空梯度值
        model.zero_grad()

        # 此外还需要清空 LSTM 的隐状态
        # 将其从上个实例的历史中分离出来
        model.hidden = model.init_hidden()

        # Step 2. 准备网络输入, 将其变为词索引的 Variables 类型数据
        sentence_in = prepare_sequence(sentence, word_to_ix)
        targets = prepare_sequence(tags, tag_to_ix)

        # Step 3. 前向传播
        tag_scores = model(sentence_in)

        # Step 4. 计算损失和梯度值, 通过调用 optimizer.step() 来更新梯度
        loss = loss_function(tag_scores, targets)
        loss.backward()
        optimizer.step()

# 查看训练后得分的值
inputs = prepare_sequence(training_data[0][0], word_to_ix)
tag_scores = model(inputs)
# 句子是 "the dog ate the apple", i,j 表示对于单词 i, 标签 j 的得分.
# 我们采用得分最高的标签作为预测的标签. 从下面的输出我们可以看到, 预测得
# 到的结果是0 1 2 0 1. 因为 索引是从0开始的, 因此第一个值0表示第一行的
# 最大值, 第二个值1表示第二行的最大值, 以此类推. 所以最后的结果是 DET
# NOUN VERB DET NOUN, 整个序列都是正确的!
print(tag_scores)
```

    tensor([[-1.1389, -1.2024, -0.9693],
            [-1.1065, -1.2200, -0.9834],
            [-1.1286, -1.2093, -0.9726],
            [-1.1190, -1.1960, -0.9916],
            [-1.0137, -1.2642, -1.0366]], grad_fn=<LogSoftmaxBackward>)
    tensor([[-0.0858, -2.9355, -3.5374],
            [-5.2313, -0.0234, -4.0314],
            [-3.9098, -4.1279, -0.0368],
            [-0.0187, -4.7809, -4.5960],
            [-5.8170, -0.0183, -4.1879]], grad_fn=<LogSoftmaxBackward>)

练习: 使用字符级特征来增强 LSTM 词性标注器
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

在上面的例子中, 每个词都有一个词嵌入, 作为序列模型的输入. 接下来让我们使用每个的单词的
字符级别的表达来增强词嵌入. 我们期望这个操作对结果能有显著提升, 因为像词缀这样的字符级
信息对于词性有很大的影响. 比如说, 像包含词缀 *-ly* 的单词基本上都是被标注为副词.

具体操作如下. 用 $c_w$ 来表示单词 $w$ 的字符级表达, 同之前一样, 我们使
用 $x_w$ 来表示词嵌入. 序列模型的输入就变成了 $x_w$ 和 $c_w$
的拼接. 因此, 如果 $x_w$ 的维度是5, $c_w$ 的维度是3, 那么我们的 LSTM
网络的输入维度大小就是8.

为了得到字符级别的表达, 将单词的每个字符输入一个 LSTM 网络, 而 $c_w$ 则为这个
LSTM 网络最后的隐状态. 一些提示:

* 新模型中需要两个 LSTM, 一个跟之前一样, 用来输出词性标注的得分, 另外一个新增加的用来
  获取每个单词的字符级别表达.
* 为了在字符级别上运行序列模型, 你需要用嵌入的字符来作为字符 LSTM 的输入.

```python

```

```python

```