# Neuromatch Academy: Week 3, Day 4, Tutorial 1
# Deep Learning: Decoding Neural Responses

**Content creators**: Jorge A. Menendez, Carsen Stringer

**Content reviewers**: Roozbeh Farhoodi,  Madineh Sarvestani, Kshitij Dwivedi, Spiros Chavlis, Ella Batty, Michael Waskom

---
# Tutorial Objectives
In this tutorial, we'll use deep learning to decode stimulus information from the responses of sensory neurons. Specifically, we'll look at the activity of ~20,000 neurons in mouse primary visual cortex responding to oriented gratings recorded in [this study](https://www.biorxiv.org/content/10.1101/679324v2.abstract). Our task will be to decode the orientation of the presented stimulus from the responses of the whole population of neurons. We could do this in a number of ways, but here we'll use deep learning. Deep learning is particularly well-suited to this problem for a number of reasons:
* The data are very high-dimensional: the neural response to a stimulus is a ~20,000 dimensional vector. Many machine learning techniques fail in such high dimensions, but deep learning actually thrives in this regime, as long as you have enough data (which we do here!).
* As you'll be able to see below, different neurons can respond quite differently to stimuli. This complex pattern of responses will, therefore, require non-linear methods to be decoded, which we can easily do with non-linear activation functions in deep networks.
* Deep learning architectures are highly flexible, meaning we can easily adapt the architecture of our decoding model to optimize decoding. Here, we'll focus on a single architecture, but you'll see that it can easily be modified with few changes to the code.

More concretely, our goal will be learn how to:
* Build a deep feed-forward network using PyTorch
* Evaluate the network's outputs using PyTorch built-in loss functions
* Compute gradients of the loss with respect to each parameter of the network using automatic differentiation
* Implement gradient descent to optimize the network's parameters

This tutorial will take up the first full session (equivalent to two tutorials on other days).

```python
#@title Video 1: Decoding from neural data using feed-forward networks in pytorch
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1Xa4y1a7Jz', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV1Xa4y1a7Jz

---
# Setup

```python
import os
import numpy as np

import torch
from torch import nn
from torch import optim

import matplotlib as mpl
from matplotlib import pyplot as plt
```

```python
#@title Data retrieval and loading
import hashlib
import requests

fname = "W3D4_stringer_oribinned1.npz"
url = "https://osf.io/683xc/download"
expected_md5 = "436599dfd8ebe6019f066c38aed20580"

if not os.path.isfile(fname):
  try:
    r = requests.get(url)
  except requests.ConnectionError:
    print("!!! Failed to download data !!!")
  else:
    if r.status_code != requests.codes.ok:
      print("!!! Failed to download data !!!")
    elif hashlib.md5(r.content).hexdigest() != expected_md5:
      print("!!! Data download appears corrupted !!!")
    else:
      with open(fname, "wb") as fid:
        fid.write(r.content)
```

```python
#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper Functions

def load_data(data_name=fname, bin_width=1):
  """Load mouse V1 data from Stringer et al. (2019)

  Data from study reported in this preprint:
  https://www.biorxiv.org/content/10.1101/679324v2.abstract

  These data comprise time-averaged responses of ~20,000 neurons
  to ~4,000 stimulus gratings of different orientations, recorded
  through Calcium imaginge. The responses have been normalized by
  spontanous levels of activity and then z-scored over stimuli, so
  expect negative numbers. They have also been binned and averaged
  to each degree of orientation.

  This function returns the relevant data (neural responses and
  stimulus orientations) in a torch.Tensor of data type torch.float32
  in order to match the default data type for nn.Parameters in
  Google Colab.

  This function will actually average responses to stimuli with orientations
  falling within bins specified by the bin_width argument. This helps
  produce individual neural "responses" with smoother and more
  interpretable tuning curves.

  Args:
    bin_width (float): size of stimulus bins over which to average neural
      responses

  Returns:
    resp (torch.Tensor): n_stimuli x n_neurons matrix of neural responses,
        each row contains the responses of each neuron to a given stimulus.
        As mentioned above, neural "response" is actually an average over
        responses to stimuli with similar angles falling within specified bins.
    stimuli: (torch.Tensor): n_stimuli x 1 column vector with orientation
        of each stimulus, in degrees. This is actually the mean orientation
        of all stimuli in each bin.

  """
  with np.load(data_name) as dobj:
    data = dict(**dobj)
  resp = data['resp']
  stimuli = data['stimuli']

  if bin_width > 1:
    # Bin neural responses and stimuli
    bins = np.digitize(stimuli, np.arange(0, 360 + bin_width, bin_width))
    stimuli_binned = np.array([stimuli[bins == i].mean() for i in np.unique(bins)])
    resp_binned = np.array([resp[bins == i, :].mean(0) for i in np.unique(bins)])
  else:
    resp_binned = resp
    stimuli_binned = stimuli

  # Return as torch.Tensor
  resp_tensor = torch.tensor(resp_binned, dtype=torch.float32)
  stimuli_tensor = torch.tensor(stimuli_binned, dtype=torch.float32).unsqueeze(1)  # add singleton dimension to make a column vector

  return resp_tensor, stimuli_tensor

def plot_data_matrix(X, ax):
  """Visualize data matrix of neural responses using a heatmap

  Args:
    X (torch.Tensor or np.ndarray): matrix of neural responses to visualize
        with a heatmap
    ax (matplotlib axes): where to plot

  """

  cax = ax.imshow(X, cmap=mpl.cm.pink, vmin=np.percentile(X, 1), vmax=np.percentile(X, 99))
  cbar = plt.colorbar(cax, ax=ax, label='normalized neural response')

  ax.set_aspect('auto')
  ax.set_xticks([])
  ax.set_yticks([])

def identityLine():
  """
  Plot the identity line y=x
  """
  ax = plt.gca()
  lims = np.array([ax.get_xlim(), ax.get_ylim()])
  minval = lims[:, 0].min()
  maxval = lims[:, 1].max()
  equal_lims = [minval, maxval]
  ax.set_xlim(equal_lims)
  ax.set_ylim(equal_lims)
  line = ax.plot([minval, maxval], [minval, maxval], color="0.7")
  line[0].set_zorder(-1)

def get_data(n_stim, train_data, train_labels):
  """ Return n_stim randomly drawn stimuli/resp pairs

  Args:
    n_stim (scalar): number of stimuli to draw
    resp (torch.Tensor):
    train_data (torch.Tensor): n_train x n_neurons tensor with neural
      responses to train on
    train_labels (torch.Tensor): n_train x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data, in radians

  Returns:
    (torch.Tensor, torch.Tensor): n_stim x n_neurons tensor of neural responses and n_stim x 1 of orientations respectively
  """
  n_stimuli = train_labels.shape[0]
  istim = np.random.choice(n_stimuli, n_stim)
  r = train_data[istim]  # neural responses to this stimulus
  ori = train_labels[istim]  # true stimulus orientation

  return r, ori

def stimulus_class(ori, n_classes):
  """Get stimulus class from stimulus orientation

  Args:
    ori (torch.Tensor): orientations of stimuli to return classes for
    n_classes (int): total number of classes

  Returns:
    torch.Tensor: 1D tensor with the classes for each stimulus

  """
  bins = np.linspace(0, 360, n_classes + 1)
  return torch.tensor(np.digitize(ori.squeeze(), bins)) - 1  # minus 1 to accomodate Python indexing

def plot_decoded_results(train_loss, test_labels, predicted_test_labels):
  """ Plot decoding results in the form of network training loss and test predictions

  Args:
    train_loss (list): training error over iterations
    test_labels (torch.Tensor): n_test x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data, in radians
    predicted_test_labels (torch.Tensor): n_test x 1 tensor with predicted orientations of the
      stimuli from decoding neural network

  """

  # Plot results
  fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))

  # Plot the training loss over iterations of GD
  ax1.plot(train_loss)

  # Plot true stimulus orientation vs. predicted class
  ax2.plot(stimuli_test.squeeze(), predicted_test_labels, '.')

  ax1.set_xlim([0, None])
  ax1.set_ylim([0, None])
  ax1.set_xlabel('iterations of gradient descent')
  ax1.set_ylabel('negative log likelihood')
  ax2.set_xlabel('true stimulus orientation ($^o$)')
  ax2.set_ylabel('decoded orientation bin')
  ax2.set_xticks(np.linspace(0, 360, n_classes + 1))
  ax2.set_yticks(np.arange(n_classes))
  class_bins = [f'{i * 360 / n_classes: .0f}$^o$ - {(i + 1) * 360 / n_classes: .0f}$^o$' for i in range(n_classes)]
  ax2.set_yticklabels(class_bins);

  # Draw bin edges as vertical lines
  ax2.set_ylim(ax2.get_ylim())  # fix y-axis limits
  for i in range(n_classes):
    lower = i * 360 / n_classes
    upper = (i + 1) * 360 / n_classes
    ax2.plot([lower, lower], ax2.get_ylim(), '-', color="0.7", linewidth=1, zorder=-1)
    ax2.plot([upper, upper], ax2.get_ylim(), '-', color="0.7", linewidth=1, zorder=-1)

  plt.tight_layout()
```

---
# Section 1: Load and visualize data

In the next cell, we have provided code to load the data and plot the matrix of neural responses.

Next to it, we plot the tuning curves of three randomly selected neurons.

```python
#@title

#@markdown Execute this cell to load and visualize data

# Load data
resp_all, stimuli_all = load_data()  # argument to this function specifies bin width
n_stimuli, n_neurons = resp_all.shape

print(f'{n_neurons} neurons in response to {n_stimuli} stimuli')

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(2 * 6, 5))

# Visualize data matrix
plot_data_matrix(resp_all[:100, :].T, ax1)  # plot responses of first 100 neurons
ax1.set_xlabel('stimulus')
ax1.set_ylabel('neuron')

# Plot tuning curves of three random neurons
ineurons = np.random.choice(n_neurons, 3, replace=False)  # pick three random neurons
ax2.plot(stimuli_all, resp_all[:, ineurons])
ax2.set_xlabel('stimulus orientation ($^o$)')
ax2.set_ylabel('neural response')
ax2.set_xticks(np.linspace(0, 360, 5))

plt.tight_layout()
```

We will split our data into a training set and test set. In particular, we will have a training set of orientations (`stimuli_train`) and the corresponding responses (`resp_train`). Our testing set will have held-out orientations (`stimuli_test`) and the corresponding responses (`resp_test`).

```python
#@title
#@markdown Execute this cell to split into training and test sets

# Set random seeds for reproducibility
np.random.seed(4)
torch.manual_seed(4)

# Split data into training set and testing set
n_train = int(0.6 * n_stimuli)  # use 60% of all data for training set
ishuffle = torch.randperm(n_stimuli)
itrain = ishuffle[:n_train]  # indices of data samples to include in training set
itest = ishuffle[n_train:]  # indices of data samples to include in testing set
stimuli_test = stimuli_all[itest]
resp_test = resp_all[itest]
stimuli_train = stimuli_all[itrain]
resp_train = resp_all[itrain]
```

---
# Section 2: Deep feed-forward networks in *pytorch*

We'll now build a simple deep neural network that takes as input a vector of neural responses and outputs a single number representing the decoded stimulus orientation.

To keep things simple, we'll build a deep network with **one** hidden layer. See the appendix for a deeper discussion of what this choice entails, and when one might want to use deeper/shallower and wider/narrower architectures.

Let $\mathbf{r}^{(n)} = \begin{bmatrix} r_1^{(n)} & r_2^{(n)} & \ldots & r_N^{(n)} \end{bmatrix}^T$ denote the vector of neural responses (of neurons $1, \ldots, N$) to the $n$th stimulus. The network we will use is described by the following set of equations:
\begin{align}
    \mathbf{h}^{(n)} &= \mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in}, && [\mathbf{W}^{in}: M \times N], \\
    y^{(n)} &= \mathbf{W}^{out} \mathbf{h}^{(n)} + \mathbf{b}^{out},  && [\mathbf{W}^{out}: 1 \times M],
\end{align}
where $y^{(n)}$ denotes the scalar output of the network: the decoded orientation of the $n$th stimulus.

The $M$-dimensional vector $\mathbf{h}^{(n)}$ denotes the activations of the **hidden layer** of the network.

The blue components of this diagram denote the **parameters** of the network, which we will later optimize with gradient descent. These include all the weights and biases $\mathbf{W}^{in}, \mathbf{b}^{in}, \mathbf{W}^{out}, \mathbf{b}^{out}$.

### Section 2.1: Introduction to PyTorch

Here, we'll use the **PyTorch** package to build, run, and train deep networks of this form in Python. There are two core components to the PyTorch package:

1. The first is the `torch.Tensor` data type used in PyTorch. `torch.Tensor`'s are effectively just like a `numpy` arrays, except that they have some important attributes and methods needed for automatic differentiation (to be discussed below). They also come along with infrastructure for easily storing and computing with them on GPU's, a capability we won't touch on here but which can be really useful in practice.

2. The second core ingredient is the PyTorch `nn.Module` class. This is the class we'll use for constructing deep networks, so that we can then easily train them using built-in PyTorch functions. Keep in my mind that `nn.Module` classes can actually be used to build, run, and train any model -- not just deep networks!

  The next cell contains code for building the deep network we defined above using the `nn.Module` class. It contains three key ingredients:

  * `__init__()` method to initialize its parameters, like in any other Python class. In this case, it takes two arguments:
    * `n_inputs`: the number of input units. This should always be set to the number of neurons whose activities are being decoded (i.e. the dimensionality of the input to the network).
    * `n_hidden`: the number of hidden units. This is a parameter that we are free to vary in deciding how to build our network. See the appendix for a discussion of how this architectural choice affects the computations the network can perform.

  * `nn.Linear` modules, which are built-in PyTorch classes containing all the weights and biases for a given network layer (documentation [here](https://pytorch.org/docs/master/generated/torch.nn.Linear.html)). This class takes two arguments to initialize:
    * \# of inputs to that layer
    * \# of outputs from that layer

    For the input layer, for example, we have:
    * \# of inputs = \# of neurons whose responses are to be decoded ($N$, specified by `n_inputs`)
    * \# of outputs = \# of hidden layer units ($M$, specified by `n_hidden`)

    PyTorch will initialize all weights and biases randomly.

  * `forward()` method, which takes as argument an input to the network and returns the network output. In our case, this comprises computing the output $y$ from a given input $\mathbf{r}$ using the above two equations. See the next cell for code implementing this computation using the built-in PyTorch `nn.Linear` classes.

```python
class DeepNet(nn.Module):
  """Deep Network with one hidden layer

  Args:
    n_inputs (int): number of input units
    n_hidden (int): number of units in hidden layer

  Attributes:
    in_layer (nn.Linear): weights and biases of input layer
    out_layer (nn.Linear): weights and biases of output layer

  """

  def __init__(self, n_inputs, n_hidden):
    super().__init__()  # needed to invoke the properties of the parent class nn.Module
    self.in_layer = nn.Linear(n_inputs, n_hidden) # neural activity --> hidden units
    self.out_layer = nn.Linear(n_hidden, 1) # hidden units --> output

  def forward(self, r):
    """Decode stimulus orientation from neural responses

    Args:
      r (torch.Tensor): vector of neural responses to decode, must be of
        length n_inputs. Can also be a tensor of shape n_stimuli x n_inputs,
        containing n_stimuli vectors of neural responses

    Returns:
      torch.Tensor: network outputs for each input provided in r. If
        r is a vector, then y is a 1D tensor of length 1. If r is a 2D
        tensor then y is a 2D tensor of shape n_stimuli x 1.

    """
    h = self.in_layer(r)  # hidden representation
    y = self.out_layer(h)
    return y
```

The next cell contains code for initializing and running this network. We use it to decode stimulus orientation from a vector of neural responses to the very first stimulus. Note that when the initialized network class is called as a function on an input (e.g. `net(r)`), its `.forward()` method is called. This is a special property of the `nn.Module` class.

Note that the decoded orientations at this point will be nonsense, since the network has been initialized with random weights. Below, we'll learn how to optimize these weights for good stimulus decoding.

```python
# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

# Initialize a deep network with M=200 hidden units
net = DeepNet(n_neurons, 200)

# Get neural responses (r) to and orientation (ori) to one stimulus in dataset
r, ori = get_data(1, resp_train, stimuli_train)  # using helper function get_data

# Decode orientation from these neural responses using initialized network
out = net(r)  # compute output from network, equivalent to net.forward(r)

print('decoded orientation: %.2f degrees' % out)
print('true orientation: %.2f degrees' % ori)
```

    decoded orientation: 0.08 degrees
    true orientation: 139.00 degrees

---
### Section 2.2: Activation functions

```python
#@title Video 2: Nonlinear activation functions
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1m5411h7V5', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV1m5411h7V5

Note that the deep network we constructed above comprises solely **linear** operations on each layer: each layer is just a weighted sum of the elements in the previous layer. It turns out that linear hidden layers like this aren't particularly useful, since a sequence of linear transformations is actually essentially the same as a single linear transformation. We can see this from the above equations by plugging in the first one into the second one to obtain
\begin{equation}
    y^{(n)} = \mathbf{W}^{out} \left( \mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in} \right) + \mathbf{b}^{out} = \mathbf{W}^{out}\mathbf{W}^{in} \mathbf{r}^{(n)} + \left( \mathbf{W}^{out}\mathbf{b}^{in} + \mathbf{b}^{out} \right)
\end{equation}
In other words, the output is still just a weighted sum of elements in the input -- the hidden layer has done nothing to change this.

To extend the set of computable input/output transformations to more than just weighted sums, we'll incorporate a **non-linear activation function** in the hidden units. This is done by simply modifying the equation for the hidden layer activations to be
\begin{equation}
    \mathbf{h}^{(n)} = \phi(\mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in})
\end{equation}
where $\phi$ is referred to as the activation function. Using a non-linear activation function will ensure that the hidden layer performs a non-linear transformation of the input, which will make our network much more powerful (or *expressive*, cf. appendix). In practice, deep networks *always* use non-linear activation functions.

#### Exercise 1: Nonlinear Activations

Create a new class `DeepNetReLU` by modifying our above deep network model to use a non-linear activation function. We'll use the linear rectification function:
\begin{equation}
  \phi(x) =
  \begin{cases}
    x & \text{if } x > 0 \\
    0 & \text{else}
  \end{cases}
\end{equation}
which can be implemented in PyTorch using `torch.relu()`. Hidden layers with this activation function are typically referred to as "**Re**ctified **L**inear **U**nits", or **ReLU**'s.

Initialize this network with 20 hidden units and run on an example stimulus.

**Hint**: you only need to modify the `forward()` method of the above `DeepNet()` class.

```python
class DeepNetReLU(nn.Module):

  def __init__(self, n_inputs, n_hidden):
    super().__init__()  # needed to invoke the properties of the parent class nn.Module
    self.in_layer = nn.Linear(n_inputs, n_hidden) # neural activity --> hidden units
    self.out_layer = nn.Linear(n_hidden, 1) # hidden units --> output

  def forward(self, r):

    ############################################################################
    ## TO DO for students: write code for computing network output using a
    ## rectified linear activation function for the hidden units
    # Fill out function and remove
    raise NotImplementedError("Student exercise: complete DeepNetReLU forward")
    ############################################################################

    h = ...
    y = ...

    return y

# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

# Get neural responses (r) to and orientation (ori) to one stimulus in dataset
r, ori = get_data(1, resp_train, stimuli_train)

# Uncomment to test your class

# Initialize deep network with M=20 hidden units and uncomment lines below
# net = DeepNetReLU(...)

# Decode orientation from these neural responses using initialized network
# net(r) is equivalent to net.forward(r)
# out = net(r)

# print('decoded orientation: %.2f degrees' % out)
# print('true orientation: %.2f degrees' % ori)
```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W3D4_DeepLearning1/solutions/W3D4_Tutorial1_Solution_5bdc2033.py)

You should see that the decoded orientation is 0.13 $^{\circ}$ while the true orientation is 139.00 $^{\circ}$.

---
# Section 3: Loss functions and gradient descent

```python
#@title Video 3: Loss functions & gradient descent
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV19k4y1271n', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV19k4y1271n

### Section 3.1: Loss functions

Because the weights of the network are currently randomly chosen, the outputs of the network are nonsense: the decoded stimulus orientation is nowhere close to the true stimulus orientation. We'll shortly write some code to change these weights so that the network does a better job of decoding.

But to do so, we first need to define what we mean by "better". One simple way of defining this is to use the squared error
\begin{equation}
    L = (y - \tilde{y})^2
\end{equation}
where $y$ is the network output and $\tilde{y}$ is the true stimulus orientation. When the decoded stimulus orientation is far from the true stimulus orientation, $L$ will be large. We thus refer to $L$ as the **loss function**, as it quantifies how *bad* the network is at decoding stimulus orientation.

PyTorch actually carries with it a number of built-in loss functions. The one corresponding to the squared error is called `nn.MSELoss()`. This will take as arguments a **batch** of network outputs $y_1, y_2, \ldots, y_P$ and corresponding target outputs $\tilde{y}_1, \tilde{y}_2, \ldots, \tilde{y}_P$, and compute the **mean squared error (MSE)**
\begin{equation}
    L = \frac{1}{P}\sum_{n=1}^P \left(y^{(n)} - \tilde{y}^{(n)}\right)^2
\end{equation}

#### Exercise 2: Computing MSE

Evaluate the mean squared error for a deep network with $M=20$ rectified linear units, on the decoded orientations from neural responses to 20 random stimuli.

```python
# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

# Initialize a deep network with M=20 hidden units
net = DeepNetReLU(n_neurons, 20)

# Get neural responses to first 20 stimuli in the data set
r, ori = get_data(20, resp_train, stimuli_train)

# Decode orientation from these neural responses
out = net(r)

###################################################
## TO DO for students: evaluate mean squared error
###################################################

# Initialize PyTorch mean squared error loss function (Hint: look at nn.MSELoss)
loss_fn = ...

# Evaluate mean squared error
loss = ...

# Uncomment once above is filled in
# print('mean squared error: %.2f' % loss)
```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W3D4_DeepLearning1/solutions/W3D4_Tutorial1_Solution_0e539ef5.py)

You should see a mean squared error of 42943.75.

---
### Section 3.2: Optimization with gradient descent

Our goal is now to modify the weights to make the mean squared error loss $L$ as small as possible over the whole data set. To do this, we'll use the **gradient descent (GD)** algorithm, which consists of iterating three steps:
1. **Evaluate the loss** on the training data,
```
out = net(train_data)
loss = loss_fn(out, train_labels)
```
where `train_data` are the network inputs in the training data (in our case, neural responses), and `train_labels` are the target outputs for each input (in our case, true stimulus orientations).
2. **Compute the gradient of the loss** with respect to each of the network weights. In PyTorch, we can do this with one line of code:
```
loss.backward()
```
This command tells PyTorch to compute the gradients of the quantity stored in the variable `loss` with respect to each network parameter using [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation). These gradients are then stored behind the scenes (see appendix for more details).
3. **Update the network weights** by descending the gradient. In Pytorch, we can do this using built-in optimizers. We'll use the `optim.SGD` optimizer (documentation [here](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD)) which updates parameters along the negative gradient, scaled by a learning rate (see appendix for details). To initialize this optimizer, we have to tell it
  * which parameters to update, and
  * what learning rate to use

  For example, to optimize *all* the parameters of a network `net` using a learning rate of .001, the optimizer would be initialized as follows
  ```
  optimizer = optim.SGD(net.parameters(), lr=.001)
  ```
  where `.parameters()` is a method of the `nn.Module` class that returns a [Python generator object](https://wiki.python.org/moin/Generators) over all the parameters of that `nn.Module` class (in our case, $\mathbf{W}^{in}, \mathbf{b}^{in}, \mathbf{W}^{out}, \mathbf{b}^{out}$).

  After computing all the parameter gradients in step 2, we can then update each of these parameters using the `.step()` method of this optimizer,
  ```
  optimizer.step()
  ```
  This single line of code will extract all the gradients computed with `.backward()` and execute the SGD updates for each parameter given to the optimizer. Note that this is true no matter how big/small the network is, allowing us to use the same two lines of code to perform the gradient descent updates for any deep network model built using PyTorch.

Finally, an important detail to remember is that the gradients of each parameter need to be cleared before calling `.backward()`, or else PyTorch will try to accumulate gradients across iterations. This can again be done using built-in optimizers via the method `zero_grad()`, as follows:
```
optimizer.zero_grad()
```

Putting all this together, each iteration of the GD algorith will contain a block of code that looks something like this:
```
Get outputs from network
Evaluate loss

# Compute gradients
optimizer.zero_grad()  # clear gradients
loss.backward()

# Update weights
optimizer.step()
```

In the next exercise, we'll give you a code skeleton for implementing the GD algorithm. Your job will be to fill in the blanks.

For the mathematical details of the GD algorithm, see the appendix. Note, in particular, that here we using the gradient descent algorithm, rather than the more commonly used *stochastic* gradient descent algorithm. See the appendix for a more detailed discussion of how these differ and when one might need to use the stochastic variant.

#### Exercise 3: Gradient descent in PyTorch

Complete the function `train()` that uses the gradient descent algorithm to optimize the weights of a given network. This function takes as input arguments
* `net`: the PyTorch network whose weights to optimize
* `loss_fn`: the PyTorch loss function to use to evaluate the loss
* `train_data`: the training data to evaluate the loss on (i.e. neural responses to decode)
* `train_labels`: the target outputs for each data point in `train_data` (i.e. true stimulus orientations)

We will then train a neural network on our data and plot the loss (mean squared error) over time. When we run this function, behind the scenes PyTorch is actually changing the parameters inside this network to make the network better at decoding, so its weights will now be different than they were at initialization.

**Hint:** all the code you need for doing this is provided in the above description of the GD algorithm.

```python
def train(net, loss_fn, train_data, train_labels, n_iter=50, learning_rate=1e-4):
  """Run gradient descent to opimize parameters of a given network

  Args:
    net (nn.Module): PyTorch network whose parameters to optimize
    loss_fn: built-in PyTorch loss function to minimize
    train_data (torch.Tensor): n_train x n_neurons tensor with neural
      responses to train on
    train_labels (torch.Tensor): n_train x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data, in radians
    n_iter (int): number of iterations of gradient descent to run
    learning_rate (float): learning rate to use for gradient descent

  Returns:
    (list): training loss over iterations

  """

  # Initialize PyTorch SGD optimizer
  optimizer = optim.SGD(net.parameters(), lr=learning_rate)

  # Placeholder to save the loss at each iteration
  track_loss = []

  # Loop over epochs (cf. appendix)
  for i in range(n_iter):

    ######################################################################
    ## TO DO for students: fill in missing code for GD iteration
    raise NotImplementedError("Student exercise: write code for GD iterations")
    ######################################################################

    # Evaluate loss using loss_fn
    out = ...  # compute network output from inputs in train_data
    loss = ...  # evaluate loss function

    # Compute gradients
    ...

    # Update weights
    ...

    # Store current value of loss
    track_loss.append(loss.item())  # .item() needed to transform the tensor output of loss_fn to a scalar

    # Track progress
    if (i + 1) % (n_iter // 5) == 0:
      print(f'iteration {i + 1}/{n_iter} | loss: {loss.item():.3f}')

  return track_loss

# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

# Initialize network
net = DeepNetReLU(n_neurons, 20)

# Initialize built-in PyTorch MSE loss function
loss_fn = nn.MSELoss()

# Run GD on data
#train_loss = train(net, loss_fn, resp_train, stimuli_train)

# Plot the training loss over iterations of GD
#plt.plot(train_loss)
plt.xlim([0, None])
plt.ylim([0, None])
plt.xlabel('iterations of gradient descent')
plt.ylabel('mean squared error')
plt.show()
```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W3D4_DeepLearning1/solutions/W3D4_Tutorial1_Solution_8f827dbe.py)

*Example output:*

---
# Section 4: Evaluating model performance

## Section 4.1: Generalization performance with test data

Note that gradient descent is essentially an algorithm for fitting the network's parameters to a given set of training data. Selecting this training data is thus crucial for ensuring that the optimized parameters **generalize** to unseen data they weren't trained on. In our case, for example, we want to make sure that our trained network is good at decoding stimulus orientations from neural responses to any orientation, not just those in our data set.

To ensure this, we have split up the full data set into a **training set** and a **testing set**. In Exercise 3, we trained a deep network by optimizing the parameters on a training set. We will now evaluate how good the optimized parameters are by using the trained network to decode stimulus orientations from neural responses in the testing set. Good decoding performance on this testing set should then be indicative of good decoding performance on the neurons' responses to any other stimulus orientation. This procedure is commonly used in machine learning (not just in deep learning)and is typically referred to as **cross-validation**.

We will compute the MSE on the test data and plot the decoded stimulus orientations as a function of the true stimulus.

```python
#@title
#@markdown Execute this cell to evaluate and plot test error

out = net(resp_test)  # decode stimulus orientation for neural responses in testing set
ori = stimuli_test  # true stimulus orientations
test_loss = loss_fn(out, ori)  # MSE on testing set (Hint: use loss_fn initialized in previous exercise)

plt.plot(ori, out.detach(), '.')  # N.B. need to use .detach() to pass network output into plt.plot()
identityLine()  # draw the identity line y=x; deviations from this indicate bad decoding!
plt.title('MSE on testing set: %.2f' % test_loss.item())  # N.B. need to use .item() to turn test_loss into a scalar
plt.xlabel('true stimulus orientation ($^o$)')
plt.ylabel('decoded stimulus orientation ($^o$)')
axticks = np.linspace(0, 360, 5)
plt.xticks(axticks)
plt.yticks(axticks)
plt.show()
```

**PyTorch Note**:

An important thing to note in the code snippet for plotting the decoded orientations is the `.detach()` method. The PyTorch `nn.Module` class is special in that, behind the scenes, each of the variables inside it are linked to each other in a computational graph, for the purposes of automatic differentiation (the algorithm used in `.backward()` to compute gradients). As a result, if you want to do anything that is not a `torch` operation to the parameters or outputs of an `nn.Module` class, you'll need to first "detach" it from its computational graph. This is what the `.detach()` method does. In this hidden code above, we need to call it on the outputs of the network so that we can plot them with the `plt.plot()` function.

---
## (Bonus) Section 4.2: Model criticism

Please move to the Summary and visit this section only if you have time after completing all non-bonus material!

Let's now take a step back and think about how our model is succeeding/failing and how to improve it.

```python
#@title
#@markdown Execute this cell to plot decoding error

out = net(resp_test)  # decode stimulus orientation for neural responses in testing set
ori = stimuli_test  # true stimulus orientations
error = out - ori  # decoding error

plt.plot(ori, error.detach(), '.')   # plot decoding error as a function of true orientation (make sure all arguments to plt.plot() have been detached from PyTorch network!)

# Plotting
plt.xlabel('true stimulus orientation ($^o$)')
plt.ylabel('decoding error ($^o$)')
plt.xticks(np.linspace(0, 360, 5))
plt.yticks(np.linspace(-360, 360, 9))
plt.show()
```

### Think

In the cell below, we will plot the *decoding error* for each neural response in the testing set. The decoding error is defined as the decoded stimulus orientation minus true stimulus orientation
\begin{equation}
  \text{decoding error} = y^{(n)} - \tilde{y}^{(n)}
\end{equation}

In particular, we plot decoding error as a function of the true stimulus orientation.

  * Are some stimulus orientations harder to decode than others?
  * If so, in what sense? Are the decoded orientations for these stimuli more variable and/or are they biased?
  * Can you explain this variability/bias? What makes these stimulus orientations different from the others?
  * (Will be addressed in next exercise) Can you think of a way to modify the deep network in order to avoid this?

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W3D4_DeepLearning1/solutions/W3D4_Tutorial1_Solution_3ccf8501.py)

### (Advanced Bonus) Exercise 4: Improving the loss function
As illustrated in the previous exercise, the squared error is not a good loss function for circular quantities like angles, since two angles that are very close (e.g. $1^o$ and $359^o$) might actually have a very large squared error.

Here, we'll avoid this problem by changing our loss function to treat our decoding problem as a **classification problem**. Rather than estimating the *exact* angle of the stimulus, we'll now aim to construct a decoder that classifies the stimulus into one of $C$ classes, corresponding to different bins of angles of width $b = \frac{360}{C}$. The true class $\tilde{y}^{(n)}$ of stimulus $i$ is now given by
\begin{equation}
  \tilde{y}^{(n)} =
  \begin{cases}
    1 &\text{if angle of stimulus $n$ is in the range } [0, b] \\
    2 &\text{if angle of stimulus $n$ is in the range } [b, 2b] \\
    3 &\text{if angle of stimulus $n$ is in the range } [2b, 3b] \\
    \vdots \\
    C &\text{if angle of stimulus $n$ is in the range } [(C-1)b, 360]
  \end{cases}
\end{equation}

We have a helper function `stimulus_class` that will extract `n_classes` stimulus classes for us from the stimulus orientations.

To decode the stimulus class from neural responses, we'll use a deep network that outputs a $C$-dimensional vector of probabilities $\mathbf{p} = \begin{bmatrix} p_1, p_2, \ldots, p_C \end{bmatrix}^T$, corresponding to the estimated probabilities of the stimulus belonging to each class $1, 2, \ldots, C$.

To ensure the network's outputs are indeed probabilities (i.e. they are positive numbers between 0 and 1, and sum to 1), we'll use a [softmax function](https://en.wikipedia.org/wiki/Softmax_function) to transform the real-valued outputs from the hidden layer into probabilities. Letting $\sigma(\cdot)$ denote this softmax function, the equations describing our network are
\begin{align}
    \mathbf{h}^{(n)} &= \phi(\mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in}), && [\mathbf{W}^{in}: M \times N], \\
    \mathbf{p}^{(n)} &= \sigma(\mathbf{W}^{out} \mathbf{h}^{(n)} + \mathbf{b}^{out}),  && [\mathbf{W}^{out}: C \times M],
\end{align}
The decoded stimulus class is then given by that assigned the highest probability by the network:
\begin{equation}
  y^{(n)} = \underset{i}{\arg\max} \,\, p_i
\end{equation}
The softmax function can be implemented in PyTorch simply using `torch.softmax()`.

Often *log* probabilities are easier to work with than actual probabilities, because probabilities tend to be very small numbers that computers have trouble representing. We'll therefore actually use the logarithm of the softmax as the output of our network,
\begin{equation}
    \mathbf{l}^{(n)} = \log \left( \mathbf{p}^{(n)} \right)
\end{equation}
which can implemented in PyTorch together with the softmax via an `nn.LogSoftmax` layer. The nice thing about the logarithmic function is that it's *monotonic*, so if one probability is larger/smaller than another, then its logarithm is also larger/smaller than the other's. We therefore have that
\begin{equation}
  y^{(n)} = \underset{i}{\arg\max} \,\, p_i^{(n)} = \underset{i}{\arg\max} \, \log p_i^{(n)} = \underset{i}{\arg\max} \,\, l_i^{(n)}
\end{equation}

See the next cell for code for constructing a deep network with one hidden layer that of ReLU's that outputs a vector of log probabilities.

```python
# Deep network for classification
class DeepNetSoftmax(nn.Module):
  """Deep Network with one hidden layer, for classification

  Args:
    n_inputs (int): number of input units
    n_hidden (int): number of units in hidden layer
    n_classes (int): number of outputs, i.e. number of classes to output
      probabilities for

  Attributes:
    in_layer (nn.Linear): weights and biases of input layer
    out_layer (nn.Linear): weights and biases of output layer

  """

  def __init__(self, n_inputs, n_hidden, n_classes):
    super().__init__()  # needed to invoke the properties of the parent class nn.Module
    self.in_layer = nn.Linear(n_inputs, n_hidden)  # neural activity --> hidden units
    self.out_layer = nn.Linear(n_hidden, n_classes)  # hidden units --> outputs
    self.logprob = nn.LogSoftmax(dim=1)  # probabilities across columns should sum to 1 (each output row corresponds to a different input)

  def forward(self, r):
    """Predict stimulus orientation bin from neural responses

    Args:
      r (torch.Tensor): n_stimuli x n_inputs tensor with neural responses to n_stimuli

    Returns:
      torch.Tensor: n_stimuli x n_classes tensor with predicted class probabilities

    """
    h = torch.relu(self.in_layer(r))
    logp = self.logprob(self.out_layer(h))
    return logp
```

What should our loss function now be? Ideally, we want the probabilities outputted by our network to be such that the probability of the true stimulus class is high. One way to formalize this is to say that we want to maximize the *log* probability of the true stimulus class $\tilde{y}^{(n)}$ under the class probabilities predicted by the network,
\begin{equation}
  \log \left( \text{predicted probability of stimulus } n \text{ being of class } \tilde{y}^{(n)} \right) = \log p^{(n)}_{\tilde{y}^{(n)}} = l^{(n)}_{\tilde{y}^{(n)}}
\end{equation}
To turn this into a loss function to be *minimized*, we can then simply multiply it by -1: maximizing the log probability is the same as minimizing the *negative* log probability. Summing over a batch of $P$ inputs, our loss function is then given by
\begin{equation}
  L = -\sum_{n=1}^P \log p^{(n)}_{\tilde{y}^{(n)}} = -\sum_{n=1}^P l^{(n)}_{\tilde{y}^{(n)}}
\end{equation}
In the deep learning community, this loss function is typically referred to as the **cross-entropy**, or **negative log likelihood**. The corresponding built-in loss function in PyTorch is `nn.NLLLoss()` (documentation [here](https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html)).

In the next cell, we've provided most of the code to train and test a network to decode stimulus orientations via classification, by minimizing the negative log likelihood. Fill in the missing pieces.

Once you've done this, have a look at the plotted results. Does changing the loss function from mean squared error to a classification loss solve our problems? Note that errors may still occur -- but are these errors as bad as the ones that our network above was making?

```python
def decode_orientation(n_classes, train_data, train_labels, test_data, test_labels):
  """ Initialize, train, and test deep network to decode binned orientation from neural responses

  Args:
    n_classes (scalar): number of classes in which to bin orientation
    train_data (torch.Tensor): n_train x n_neurons tensor with neural
      responses to train on
    train_labels (torch.Tensor): n_train x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data, in radians
    test_data (torch.Tensor): n_test x n_neurons tensor with neural
      responses to train on
    test_labels (torch.Tensor): n_test x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data, in radians

  Returns:
    (list, torch.Tensor): training loss over iterations, n_test x 1 tensor with predicted orientations of the
      stimuli from decoding neural network
  """

  # Bin stimulus orientations in training set
  train_binned_labels = stimulus_class(train_labels, n_classes)

  ##############################################################################
  ## TODO for students: fill out missing pieces below to initialize, train, and
  # test network
  # Fill out function and remove
  raise NotImplementedError("Student exercise: complete decode_orientation function")
  ##############################################################################

  # Initialize network
  net = ...  # use M=20 hidden units

  # Initialize built-in PyTorch MSE loss function
  loss_fn = nn.NLLLoss()

  # Run GD on training set data, using learning rate of 0.1
  train_loss = ...

  # Decode neural responses in testing set data
  out = ...
  out_labels = np.argmax(out.detach(), axis=1)  # predicted classes

  return train_loss, out_labels

# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

n_classes = 12  # start with 12, then (bonus) try making this as big as possible! does decoding get worse?

# Uncomment below to test your function

# Initialize, train, and test network
#train_loss, predicted_test_labels = decode_orientation(n_classes, resp_train, stimuli_train, resp_test, stimuli_test)

# Plot results
#plot_decoded_results(train_loss, stimuli_test, predicted_test_labels)
```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W3D4_DeepLearning1/solutions/W3D4_Tutorial1_Solution_e2e84634.py)

*Example output:*

---
# Summary

We have now covered a number of common and powerful techniques for applying deep learning to decoding from neural data, some of which are common to almost any machine learning problem:
* Building and training deep networks using the **PyTorch** `nn.Module` class and built-in **optimizers**
* Choosing and evaluating **loss functions**
* Testing a trained model on unseen data via **cross-validation**, by splitting the data into a **training set and testing set**

An important aspect of this tutorial was the `train()` function we wrote in exercise 6. Note that it can be used to train *any* network to minimize *any* loss function (cf. advanced exercise) on *any* training data. This is the power of using PyTorch to train neural networks and, for that matter, **any other model**! There is nothing in the `nn.Module` class that forces us to use `nn.Linear` layers that implement neural network operations. You can actually put anything you want inside the `.__init__()` and `.forward()` methods of this class. As long as its parameters and computations involve only `torch.Tensor`'s, and the model is differentiable, you'll then be able to optimize the parameters of this model in exactly the same way we optimized the deep networks here.

What kinds of conclusions can we draw from these sorts of analyses? If we can decode the stimulus well from visual cortex activity, that means that there is information about this stimulus available in visual cortex. Whether or not the animal uses that information to make decisions is not determined from an analysis like this. In fact mice perform poorly in orientation discrimination tasks compared to monkeys and humans, even though they have information about these stimuli in their visual cortex. Why do you think they perform poorly in orientation discrimination tasks?

See this paper for some potential hypotheses (https://www.biorxiv.org/content/10.1101/679324v2), but this is totally an open question!

---
# Appendix

## Neural network *depth*, *width* and *expressivity*

Two important architectural choices that always have to be made when constructing deep feed-forward networks like those used here are
* the number of hidden layers, or the network's *depth*
* the number of units in each layer, or the layer *widths*

Here, we restricted ourselves to networks with a single hidden layer with a width of $M$ units, but it is easy to see how this code could be adapted to arbitrary depths. Adding another hidden layer simply requires adding another `nn.Linear` module to the `__init__()` method and incorporating it into the `.forward()` method.

The depth and width of a network determine the set of input/output transormations that it can perform, often referred to as its *expressivity*. The deeper and wider the network, the more *expressive* it is; that is, the larger the class of input/output transformations it can compute. In fact, it turns out that an infinitely wide *or* infinitely deep networks can in principle [compute (almost) *any* input/output transformation](https://en.wikipedia.org/wiki/Universal_approximation_theorem).

A classic mathematical demonstration of the power of depth is given by the so-called [XOR problem](https://medium.com/@jayeshbahire/the-xor-problem-in-neural-networks-50006411840b#:~:text=The%20XOr%2C%20or%20%E2%80%9Cexclusive%20or,value%20if%20they%20are%20equal.). This toy problem demonstrates how even a single hidden layer can drastically expand the set of input/output transformations a network can perform, relative to a shallow network with no hidden layers. The key intuition is that the hidden layer allows you to represent the input in a new format, which can then allow you to do almost anything you want with it. The *wider* this hidden layer, the more flexibility you have in this representation. In particular, if you have more hidden units than input units, then the hidden layer representation of the input is higher-dimensional than the raw data representation. This higher dimensionality effectively gives you more "room" to perform arbitrary computations in. It turns out that even with just this one hidden layer, if you make it wide enough you can actually approximate any input/output transformation you want. See [here](http://neuralnetworksanddeeplearning.com/chap4.html) for a neat visual demonstration of this.

In practice, however, it turns out that increasing depth seems to grant more expressivity with fewer units than increasing width does (for reasons that are not well understood). It is for this reason that truly *deep* networks are almost always used in machine learning, which is why this set of techniques is often referred to as *deep* learning.

That said, there is a cost to making networks deeper and wider. The bigger your network, the more parameters (i.e. weights and biases) it has, which need to be optimized! The extra expressivity afforded by higher width and/or depth thus carries with it (at least) two problems:
* optimizing more parameters usually requires more data
* a more highly parameterized network is more prone to overfit to the training data, so requires more sophisticated optimization algorithms to ensure generalization

## Gradient descent equations

Here we provide the equations for the three steps of the gradient descent algorithm, as applied to our decoding problem:

1. **Evaluate the loss** on the training data. For a mean squared error loss, this is given by
\begin{equation}
    L = \frac{1}{P}\sum_{n=1}^P (y^{(n)} - \tilde{y}^{(n)})^2
\end{equation}
where $y^{(n)}$ denotes the stimulus orientation decoded from the population response $\mathbf{r}^{(n)}$ to the $n$th stimulus in the training data, and $\tilde{y}^{(n)}$ is the true orientation of that stimulus. $P$ denotes the total number of data samples in the training set. In the syntax of our `train()` function above, $\mathbf{r}^{(n)}$ is given by `train_data[n, :]` and $\tilde{y}^{(n)}$ by `train_labels[n]`.

2. **Compute the gradient of the loss** with respect to each of the network weights. In our case, this entails computing the quantities
\begin{equation}
    \frac{\partial L}{\partial \mathbf{W}^{in}}, \frac{\partial L}{\partial \mathbf{b}^{in}}, \frac{\partial L}{\partial \mathbf{W}^{out}}, \frac{\partial L}{\partial \mathbf{b}^{out}}
\end{equation}
Usually, we would require lots of math in order to derive each of these gradients, and lots of code to compute them. But this is where PyTorch comes to the rescue! Using a cool technique called [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), PyTorch automatically calculates these gradients when the `.backward()` function is called.

  More specifically, when this function is called on a particular variable (e.g. `loss`, as above), PyTorch will compute the gradients with respect to each network parameter. These are computed and stored behind the scenes, and can be accessed through the `.grad` attribute of each of the network's parameters. As we saw above, however, we actually never need to look at or call these gradients when implementing gradient descent, as this can be taken care of by PyTorch's built-in optimizers, like `optim.SGD`.

3. **Update the network weights** by descending the gradient:
\begin{align}
    \mathbf{W}^{in} &\leftarrow \mathbf{W}^{in} - \alpha \frac{\partial L}{\partial \mathbf{W}^{in}} \\
    \mathbf{b}^{in} &\leftarrow \mathbf{b}^{in} - \alpha \frac{\partial L}{\partial \mathbf{b}^{in}} \\
    \mathbf{W}^{out} &\leftarrow \mathbf{W}^{out} - \alpha \frac{\partial L}{\partial \mathbf{W}^{out}} \\
    \mathbf{b}^{out} &\leftarrow \mathbf{b}^{out} - \alpha \frac{\partial L}{\partial \mathbf{b}^{out}}
\end{align}
where $\alpha$ is called the **learning rate**. This **hyperparameter** of the SGD algorithm controls how far we descend the gradient on each iteration. It should be as large as possible so that fewer iterations are needed, but not too large so as to avoid parameter updates from skipping over minima in the loss landscape.

While the equations written down here are specific to the network and loss function considered in this tutorial, the code provided above for implementing these three steps is completely general: no matter what loss function or network you are using, exactly the same commands can be used to implement these three steps.

## *Stochastic* gradient descent (SGD) vs. gradient descent (GD)

In this tutorial, we used the gradient descent algorithm, which differs in a subtle yet very important way from the more commonly used **stochastic gradient descent (SGD)** algorithm. The key difference is in the very first step of each iteration, where in the GD algorithm we evaluate the loss *at every data sample in the training set*. In SGD, on the other hand, we evaluate the loss only at a random subset of data samlpes from the full training set, called a **mini-batch**. At each iteration, we randomly sample a mini-batch to perform steps 1-3 on. All the above equations still hold, but now the $P$ data samples $\mathbf{r}^{(n)}, \tilde{y}^{(n)}$ denote a mini-batch of $P$ random samples from the training set, rather than the whole training set.

There are several reasons why one might want to use SGD instead of GD. The first is that the training set might be too big, so that we actually can't actually evaluate the loss on every single data sample in it. In this case, GD is simply infeasible, so we have no choice but to turn to SGD, which bypasses the restrictive memory demands of GD by sub-sampling the training set into smaller mini-batches.

But, even when GD is feasible, SGD turns out to be generally better. The stochasticity induced by the extra random sampling step in SGD effectively adds some noise in the search for local minima of the loss function. This can be really useful for avoiding potential local minima, and enforce that whatever minimum is converged to is a good one. This is particularly important when networks are wider and/or deeper, in which case the large number of parameters can lead to overfitting.

Here, we used only GD because (1) it is simpler, and (2) it suffices for the problem being considered here. Because we have so many neurons in our data set, decoding is not too challenging and doesn't require a particularly deep or wide network. The small number of parameters in our deep networks therefore can be optimized without a problem using GD.


# The Specific Factors Model

## Background

The SF model is a workhorse model in trade, growth, political economy and development.  The model is relatively easy to analyze -- it can be described compactly in terms of diagrams and yet is very rich in predictions.  In its simplest formulation there are two production sectors, say Agriculture and Manufacturing, with a linear homogenous (CRS) production technology in each sector. Production in each sector requires labor -- a mobile factor that can freely move between sectorsr -- as well as a factor specific to each sector, for example land $T$ in agriculture and capital $K$ of a specific kind in manufacturing. The model can be summarized as:

$$
\begin{aligned}
Q_a &= F(\bar T, L_a) \\
Q_m &= G(\bar K, L_m) \\
\bar{L} &=  L_a + L_m
\end{aligned}
$$

The model can be used to demonstrate gains to trade as well as how the distribution of incomes may change in response to changes in relative product prices, or changes in factor endowments.  The Specific Factors Model predicts that, in the absence of political redistribution mechanisms, specific factors in declining sectors will organize strong opposition to policies that might otherwise raise growth.  Variants of the model show up as models of rural to urban migration, adaptations of the Lewis model and other dual sector models of sectoral (mis)allocation.  The latter includes models such as the Harris-Todaro model that explains rural-urban migration, urban-bias and the growth of the urban informal sector.

The initial distribution of factor ownership and the degree of asset specificity may make a big difference in terms of what types of political coalitions mobilize for and against different policies.  This is the basic driving force in many political-economy models including Moav-Galor's (2006)cleverly titled "Das Human Kapital" growth model on why some regions made public investments in human capital which sped the transition from agriculture to manufacturing and enhanced growth, whereas similar policies were delayed in places where political/economic resistance was stronger --  for instance where landlords had stronger voice in political decisions.

The Specific Factors (SF) or Ricardo-Viner model is a close relative of the Hecksher-Ohlin-Samuelson (HOS) neoclassical trade model. The 2x2 HOS model assumes production in each of two sectors takes place by firms using constant returns to scale technologies with capital and labor as inputs and that both capital and labor are mobile across sectors. In the SF model only labor is inter-sectorally mobile and the fixed amounts of capital become 'specific' to the sector they are trapped within.

In effect the SF model therefore consists of three factors of production: mobile labor and two types of capital, one specific to each sector.  Let's label the two sectors are Agriculture and Manufacturing. In agriculture competitive firms bid to hire land and labor. In manufacturing competitive firms bid to hire capital and labor. Each factor of production will be competitively priced in equilibrium but only labor is priced on a national labor market.

The SF model is often described as a short-run version of the HOS model.  For example suppose we start with a HOS model equilibrium where labor wage and rental rate of capital have equalized across sectors (which also implies marginal products are equalized across sectors -- no productivity differences across sectors).  Now suppose that the relative price of manufacturing products suddenly rises (due to a change of world price, or government trade protection or other policies that favor manufacturing).  The higher relative product price should lead firms in the booming manufacturing sector to demand both more capital and labor. Correspondingly, demand for land and labor decline in the agricultural sector.  In the short run however only labor can be move from agriculture to manufacturing.  Agricultural workers can become factory workers but agricultural capital (say 'land' or 'tractors') cannot be easily converted to manufacturing capital (say 'weaving machines'). So labor moves from manufacturing to the agriculture, lowering the capital labor ratio in manufacturing and raising the land to labor ratio in agriculture.  The model thus predicts a surge in the real return to capital in the expanding sector and a real decline in the real return to capital in agriculture (land).  Hence the measured average and marginal product of capital will now diverge across sectors.  What happens to the real wage is more ambiguous: it rises measured in terms of purchasing power over agricultural goods but falls in terms of purchasing power over manufacturing goods.   Whether workers are better off or worse off following this price/policy change therefore comes down to how important agricultural and manufacturing goods are in their consumption basket. This result is labeled the neo-classical ambiguity.

Over the longer-term weaving machines cannot be transformed into tractors but over time new capital accumulation will build tractors and old weaving machines can be sold overseas or as scrap metal. Hence over time more capital will arrive into the manufacturing sector and leave the agricultural sector, whcih in turn will lead to even more movement of labor to manufacturing.  If this process continues capital has, in effect become mobile over time, and we end up getting closer to the predictions of the HOS model.

### Technology and Endowments
There are two sectors Agriculture and Manufacturing.  Production in each sector takes place with a linear homogenous (constant returns to scale) production functions. Agricultural production requires land which is non-mobile or specific to the sector and mobile labor.

$$Q_a = F(\bar T, L_a)$$

Manufacturing production requires specific capital $K$ and mobile labor.

$$Q_m = G(\bar K, L_m)$$

The quantity of land in the agricultural sector and the quantity of capital in the manufacturing sector are in fixed supply  during the period of analysis.  That means that firms within the agricultural (manufacturing) sector may compete with one another for the limited supply of the factor, but no new land (capital) can be supplied in the short-run period of analysis.  This of course means that the price of the factor will rise (or fall) quickly in response to swings in factor demand compared to the wage of labor whose supply is more elastic.

The market for mobile labor is competitive and the market clears at a wage where the sum of labor demands from each sector equals total labor supply. While the total labor supply in the economy is inelastic, the supply of labor to each sector will be elastic, since a rise in the wage in one sector will attract workers from the other sector.

$$L_a + L_m = \bar{L}$$

Notice that we can invert the two production function to get minimum labor requirement functions $L_a(Q_a)$ and $L_m(Q_m)$ which tell us the minimum amount of labor $L_i$ required in sector $i$ to produce quantity $Q_i$.  If we take these expressions and substitute them into the labor resource constraint we get an expression for the **production possibility frontier (PPF)** which summarizes the tradeoffs between sectors.

### Assumptions and parameters for visualizations

Let's get concrete and assume each sector employs a CRS Cobb-Douglas production function:

$$F(\bar T, L_a)=\bar T^{1-\alpha} \cdot  L_a^\alpha$$

$$G(\bar K, L_m)=\bar K^{1-\beta} \cdot  L_m^\beta$$

If $\alpha = \beta = \frac{1}{2}$ and $\bar T= \bar K = 100$.  Then

$$Q_a =  \sqrt{\bar T} \sqrt{L_a}  $$

$$Q_m = \sqrt{\bar K} \sqrt{L_m}$$

Substituting these into the labor resource constraint yields:

$$\frac{Q_a^2}{\bar T}+\frac{Q_m^2}{\bar K} = \bar L$$

or

$$Q_m = \sqrt{\bar K \bar L - \frac{\bar K Q_a^2}{\bar T} }   $$

If we make the further assumption that $\bar K = \bar L  = 100$ and $\bar L = 400$ then the PPF would look like this:

**NOTE:** If you are running this as a live jupyter notebook please first go to the [code section](#codesection) below and execute all the code cells there. Then return and run the code cells that follow sequentially.

```python
from sfm import *
```

```python
ppf(Tbar=100, Kbar=100, Lbar=400)
```

```python
interact(ppf, Tbar=(50,200,10), Kbar=(50,200,10), Lbar=fixed(400));
```

## Labor market equilibrium

Profit maximizing firms in each sector will hire labor up to the point where the marginal value product of labor (MVPL) equals the market wage. Since labor is mobile across sectors in equilibrium workers must be paid the same nominal wage $w$ in either sector:

$$ P_a \cdot MPL_a(\bar T, L_a) = w = P_m \cdot MPL_m(\bar K, L_m) $$

It will be useful to express the wage in real terms.  Divide each expression above by $P_m$ to get

$$ p \cdot MPL_a = \frac{w}{p_m} = MPL_m $$

$$\text{    where   } p = \frac{P_a}{P_m}$$

In the plots below we will place the real wage measured in terms of manufactured goods on the vertical axis of the labor demand and supply diagrams.

### Labor allocation across sectors

In a competitive market firms take the market prices of goods $p$ and wages $w$ as given. Since in equilibrium the labor market clears we can write $L_m = \bar L - L_a$ and substitute into the equilibrium condition above to get:

$$p \cdot MPL_a(L_A) = w = MPL_m(\bar L-L_A) $$

The left hand side $p \cdot MPL_a(L_A)=w$ can be solved to give us demand for labor in the agricultural sector as a function of the wage $L_a(w/p)$ and is plotted below.  The right hand side $w = MPL_m(\bar L-L_a)$ gives us demand for labor in the manufacturing sector $L_m$ as a function of the real wage but this in turn also gives us the supply of labor to the agricultural sector $L_a^s = \bar L - L_m(w/p)$ since firms in the agricultural sector can only attract workers to their sector by paying those workers just a bit above what they would be paid for the jobs they have to leave in the manufacturing sector.

The diagram below can therefore be interpreted as showing labor demand and supply in the agricultural sector and their intersection gives us the equilibrium real wage (measured on the vertical in terms of purchasing power over manufactured goods).

```python
sfmplot(p=1)
```

```python
interact(sfmplot, p=(0.5,2,0.1), Lbar=fixed(400), Tbar=(50,200,10), Kbar=(50,200,10), show=fixed('True'));
```

    interactive(children=(FloatSlider(value=1.2000000000000002, description='p', max=2.0, min=0.5), IntSlider(valu…

We can solve for the equilibrium labor allocation $L_a$ as a function of the relative price $p$, and this in turn also gives us the equilibrium real wage. When $\alpha=\beta = \frac{1}{2}$ the equilibrium condition $p \cdot MPL_a(L_a) = MPL_m(\bar L-L_a)$ becomes:

$$p \cdot \frac{\sqrt{\bar T}}{\sqrt{L_a}} = \frac{\sqrt{\bar K}}{\sqrt{\bar L - L_a}}$$

which we solve to find:

$$L_a^e = \frac{p^2 \bar T \bar L}{\bar K - p^2 \bar T}$$

For cases where $\alpha \ne \beta$ we find the optimal $L_a$ that solves this equilibrium condition numerically.

(target_header)=
## Autarky prices

Thus far we have a pretty complete model of how production allocations and real wages would be determined in a small open economy where producers face world price ratio $p$.  we explore comparative statics in this economy in more detail below.

If however the economy is inititally in autarky or closed to the world then we must also consider the role of domestic consumer preferences in the determination of domestic equilibrium product prices.

With Cobb-Douglas preferences $u(x,y) = \gamma \ln(x) + (1-\gamma) \ln(y)$ consumers demands can be written as a function of relative price $p=\frac{P_a}{P_m}$. Setting $P_m=1$ to make manufacturing the numeraire good, this can be written:

$$
\begin{align}
C_a(p) =& \gamma \cdot \frac{I(p)}{p} \\
C_m(p) =& (1-\gamma) \cdot I(p)
\end{align}
$$

Income $I$ in the expressions is given by the value of national production or GDP at these prices. Measured in manufactured goods:

$$I(p) = p \cdot F(\bar T, L_a(p)) + G(\bar K, \bar L - L_a(p)$$

By Walras' law we only need to find the relative price at which output equals demand in one of the two product markets so in the the code below we solve for equilibrium domestic prices from the condition $Q_a(p)=C_a(p)$.

For parameters $\alpha=\beta=\gamma = \frac{1}{2}$, $\bar T = \bar K =100$ and $\bar L=400$ the domestic equilibrium prices are unitary:

```python
p_autarky(Lbar, Tbar, Kbar)
```

    1.0

And this in turn leads to an equilibrium autarky allocation with $L_a = L_m = 200$ and a real wage $\frac{w}{p}=0.35$.

```python
eqn(p_autarky())
```

    (200.0, 0.35355339059327373)

The plot below shows the autarky production and consumption point (marked by an 'X') as well as the new production point (marked by a circle) and consumption point (marked by a square) if the country opened to trade with a world relative price $p=\frac{7}{4}$.

Opening to trade leads the country to expand agricultural production by re-allocating labor from manufacturing to agriculture. We confirm the expected **neoclassical ambiguity** result which is that an increase in the relative price of agricultural goods will lead to an *increase* in the real wage measured in terms of manufactured goods  $\frac{w}{p_m}$ ($=w$ since we have $P_m=1$) but a *decrease* in real wage measured in terms of agricultural goods or $\frac{w}{P_a} = \frac{w}{p}$ in our notation (since $\frac{w}{P_a} = \frac{w}{p_m} \cdot \frac{P_m}{P_a}$).

Diagrammatically

$$p \uparrow \rightarrow \frac{w}{p_m} \uparrow , \frac{w}{p_a} \downarrow$$

```python
pw = 7/4
Lao, wo = eqn(p=pw)
Lao, wo, wo/pw
```

    (301.5384615384615, 0.5038911092686594, 0.28793777672494825)

```python
foo = sfmtrade3(p=7/4)
```

```python
sfmtrade(p=7/4)
```

```python
interact(sfmtrade, p=(0.5,2,0.05));
```

## Effects of a price change on the income distribution

More explanations to be placed here...

```python
foo = sfmplot2(p=2)
```

```python
interact(sfmplot2, p=(0.1, 2,0.1));
```

    interactive(children=(FloatSlider(value=1.0, description='p', max=2.0, min=0.1), Output()), _dom_classes=('wid…

## Before and after plots

The trick here is to define a new plotting function that first plots a static plot and then allows an interaction.

```python
interact(sfmplot2, p=(0,2,0.2), Lbar=fixed(400), show=fixed(True));
```

    interactive(children=(FloatSlider(value=1.0, description='p', max=2.0, step=0.2), Output()), _dom_classes=('wi…


# Section 0 问题描述与完成项目流程

## 1. 问题描述

</img>

在该项目中，你将使用强化学习算法，实现一个自动走迷宫机器人。

1. 如上图所示，智能机器人显示在右上角。在我们的迷宫中，有陷阱（红色炸弹）及终点（蓝色的目标点）两种情景。机器人要尽量避开陷阱、尽快到达目的地。
2. 小车可执行的动作包括：向上走 `u`、向右走 `r`、向下走 `d`、向左走 `l`。
3. 执行不同的动作后，根据不同的情况会获得不同的奖励，具体而言，有以下几种情况。
    - 撞到墙壁：-10
    - 走到终点：50
    - 走到陷阱：-30
    - 其余情况：-0.1
4. 我们需要通过修改 `robot.py` 中的代码，来实现一个 Q Learning 机器人，实现上述的目标。

## 2. 完成项目流程

1. 配置环境，使用 `envirnment.yml` 文件配置名为 `robot-env` 的 conda 环境，具体而言，你只需转到当前的目录，在命令行/终端中运行如下代码，稍作等待即可。
```
conda env create -f envirnment.yml
```
安装完毕后，在命令行/终端中运行 `source activate robot-env`（Mac/Linux 系统）或 `activate robot-env`（Windows 系统）激活该环境。

2. 阅读 `main.ipynb` 中的指导完成项目，并根据指导修改对应的代码，生成、观察结果。
3. 导出代码与报告，上传文件，提交审阅并优化。
---

---

# Section 1 算法理解

## 1. 1 强化学习总览

强化学习作为机器学习算法的一种，其模式也是让智能体在“训练”中学到“经验”，以实现给定的任务。但不同于监督学习与非监督学习，在强化学习的框架中，我们更侧重通过智能体与环境的**交互**来学习。通常在监督学习和非监督学习任务中，智能体往往需要通过给定的训练集，辅之以既定的训练目标（如最小化损失函数），通过给定的学习算法来实现这一目标。然而在强化学习中，智能体则是通过其与环境交互得到的奖励进行学习。这个环境可以是虚拟的（如虚拟的迷宫），也可以是真实的（自动驾驶汽车在真实道路上收集数据）。

在强化学习中有五个核心组成部分，它们分别是：**环境（Environment）**、**智能体（Agent）**、**状态（State）**、**动作（Action）**和**奖励（Reward）**。在某一时间节点 $t$：

- 智能体在从环境中感知其所处的状态 $s_t$
- 智能体根据某些准则选择动作 $a_t$
- 环境根据智能体选择的动作，向智能体反馈奖励 $r_{t+1}$

通过合理的学习算法，智能体将在这样的问题设置下，成功学到一个在状态 $s_t$ 选择动作 $a_t$ 的策略 $\pi (s_t) = a_t$。

---

**问题 1**：请参照如上的定义，描述出 “机器人走迷宫这个问题” 中强化学习五个组成部分对应的实际对象：

- **环境** :
- **状态** :

- **动作** :
- **奖励** :

$$T(s^{'}, a, s) = P(s^{'}|a,s)$$

---

## 1.2 计算 Q 值

在我们的项目中，我们要实现基于 Q-Learning 的强化学习算法。Q-Learning 是一个值迭代（Value Iteration）算法。与策略迭代（Policy Iteration）算法不同，值迭代算法会计算每个”状态“或是”状态-动作“的值（Value）或是效用（Utility），然后在执行动作的时候，会设法最大化这个值。因此，对每个状态值的准确估计，是我们值迭代算法的核心。通常我们会考虑**最大化动作的长期奖励**，即不仅考虑当前动作带来的奖励，还会考虑动作长远的奖励。

在 Q-Learning 算法中，我们把这个长期奖励记为 Q 值，我们会考虑每个 ”状态-动作“ 的 Q 值，具体而言，它的计算公式为：

$$
q(s_{t},a) = R_{t+1} + \gamma \times\max_a q(a,s_{t+1})
$$

也就是对于当前的“状态-动作” $(s_{t},a)$，我们考虑执行动作 $a$ 后环境给我们的奖励 $R_{t+1}$，以及执行动作 $a$ 到达 $s_{t+1}$后，执行任意动作能够获得的最大的Q值 $\max_a q(a,s_{t+1})$，$\gamma$ 为折扣因子。

不过一般地，我们使用更为保守地更新 Q 表的方法，即引入松弛变量 $alpha$，按如下的公式进行更新，使得 Q 表的迭代变化更为平缓。

$$
q(s_{t},a) = (1-\alpha) \times q(s_{t},a) + \alpha \times(R_{t+1} + \gamma \times\max_a q(a,s_{t+1}))
$$

---
</img>

**问题 2**：根据已知条件求 $q(s_{t},a)$，在如下模板代码中的空格填入对应的数字即可。

已知：如上图，机器人位于 $s_1$，行动为 `u`，行动获得的奖励与题目的默认设置相同。在 $s_2$ 中执行各动作的 Q 值为：`u`: -24，`r`: -13，`d`: -0.29、`l`: +40，$\gamma$ 取0.9。

$$
\begin{align}
q(s_{t},a) & = R_{t+1} + \gamma \times\max_a q(a,s_{t+1}) \\
 & =() + ()*() \\
 & =()
\end{align}
$$

---

## 1.3 如何选择动作

在强化学习中，「探索-利用」问题是非常重要的问题。具体来说，根据上面的定义，我们会尽可能地让机器人在每次选择最优的决策，来最大化长期奖励。但是这样做有如下的弊端：
1. 在初步的学习中，我们的 Q 值会不准确，如果在这个时候都按照 Q 值来选择，那么会造成错误。
2. 学习一段时间后，机器人的路线会相对固定，则机器人无法对环境进行有效的探索。

因此我们需要一种办法，来解决如上的问题，增加机器人的探索。由此我们考虑使用 epsilon-greedy 算法，即在小车选择动作的时候，以一部分的概率随机选择动作，以一部分的概率按照最优的 Q 值选择动作。同时，这个选择随机动作的概率应当随着训练的过程逐步减小。

---

**问题 3**：在如下的代码块中，实现 epsilon-greedy 算法的逻辑，并运行测试代码。

```python
import random

actions = ['u','r','d','l']
qline = {'u':1.2, 'r':-2.1, 'd':-24.5, 'l':27}
epsilon = 0.3 # 以0.3的概率进行随机选择

def choose_action(epsilon):

    action = None
    if True: # 以某一概率
        pass # 实现对动作的随机选择
    else:
        pass # 否则选择具有最大 Q 值的动作
    return action
```

---

---

# Section 2 代码实现

## 2.1. `Maze` 类理解

我们首先引入了迷宫类 `Maze`，这是一个非常强大的函数，它能够根据你的要求随机创建一个迷宫，或者根据指定的文件，读入一个迷宫地图信息。

1. 使用 `Maze("file_name")` 根据指定文件创建迷宫，或者使用 `Maze(maze_size=(height,width))` 来随机生成一个迷宫。
2. 使用 `trap_number` 参数，在创建迷宫的时候，设定迷宫中陷阱的数量。
3. 直接键入迷宫变量的名字按回车，展示迷宫图像（如 `g=Maze("xx.txt")`，那么直接输入 `g` 即可。
4. 建议生成的迷宫尺寸，长在 6~12 之间，宽在 10～12 之间。

---

**问题 4**：在如下的代码块中，创建你的迷宫并展示。

```python
from Maze import Maze
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

## todo: 创建迷宫并展示
```

---

你可能已经注意到，在迷宫中我们已经默认放置了一个机器人。实际上，我们为迷宫配置了相应的 API，来帮助机器人的移动与感知。其中你随后会使用的两个 API 为 `maze.sense_robot()` 及 `maze.move_robot()`。

1. `maze.sense_robot()` 为一个无参数的函数，输出机器人在迷宫中目前的位置。
2. `maze.move_robot(direction)` 对输入的移动方向，移动机器人，并返回对应动作的奖励值。

---

**问题 5**：随机移动机器人，并记录下获得的奖励，展示出机器人最后的位置。

```python
rewards = []

## 循环、随机移动机器人10次，记录下奖励
for i in range(10):
    pass

## 输出机器人最后的位置
print()

## 打印迷宫，观察机器人位置
```

## 2.2. `Robot` 类实现

`Robot` 类是我们需要重点实现的部分。在这个类中，我们需要实现诸多功能，以使得我们成功实现一个强化学习智能体。总体来说，之前我们是人为地在环境中移动了机器人，但是现在通过实现 `Robot` 这个类，机器人将会自己移动。通过实现学习函数，`Robot` 类将会学习到如何选择最优的动作，并且更新强化学习中对应的参数。

首先 `Robot` 有多个输入，其中 `alpha=0.5, gamma=0.9, epsilon0=0.5` 表征强化学习相关的各个参数的默认值，这些在之前你已经了解到，`Maze` 应为机器人所在迷宫对象。

随后观察 `Robot.update` 函数，它指明了在每次执行动作时，`Robot` 需要执行的程序。按照这些程序，各个函数的功能也就明了了。

最后你需要实现 `Robot.py` 代码中的8段代码，他们都在代码中以 `#TODO` 进行标注，你能轻松地找到他们。

---

**问题 6**：实现 `Robot.py` 中的8段代码，并运行如下代码检查效果（记得将 `maze` 变量修改为你创建迷宫的变量名）。

```python
from Robot import Robot
robot = Robot(maze) # 记得将 maze 变量修改为你创建迷宫的变量名
robot.set_status(learning=True,testing=False)
print(robot.update())

maze
```

---

## 2.3 用 `Runner` 类训练 Robot

在实现了上述内容之后，我们就可以开始对我们 `Robot` 进行训练并调参了。我们为你准备了又一个非常棒的类 `Runner`，来实现整个训练过程及可视化。使用如下的代码，你可以成功对机器人进行训练。并且你会在当前文件夹中生成一个名为 `filename` 的视频，记录了整个训练的过程。通过观察该视频，你能够发现训练过程中的问题，并且优化你的代码及参数。

---

**问题 7**：尝试利用下列代码训练机器人，并进行调参。可选的参数包括：

- 训练参数
    - 训练次数 `epoch`
- 机器人参数：
    - `epsilon0` (epsilon 初值)
    - `epsilon`衰减（可以是线性、指数衰减，可以调整衰减的速度），你需要在 Robot.py 中调整
    - `alpha`
    - `gamma`
- 迷宫参数:
    - 迷宫大小
    - 迷宫中陷阱的数量

```python
## 可选的参数：
epoch = 20

epsilon0 = 0.5
alpha = 0.5
gamma = 0.9

maze_size = (6,6)
trap_number = 1
```

```python
from Runner import Runner

g = Maze(maze_size=maze_size,trap_number=trap_number)
r = Robot(g,alpha=alpha, epsilon0=epsilon0, gamma=gamma)
r.set_status(learning=True)

runner = Runner(r, g)
runner.run_training(epoch, display_direction=True)
runner.generate_movie(filename = "final1.mp4") # 你可以注释该行代码，加快运行速度，不过你就无法观察到视频了。
```

---

使用 `runner.plot_results()` 函数，能够打印机器人在训练过程中的一些参数信息。

- Success Times 代表机器人在训练过程中成功的累计次数，这应当是一个累积递增的图像。
- Accumulated Rewards 代表机器人在每次训练 epoch 中，获得的累积奖励的值，这应当是一个逐步递增的图像。
- Running Times per Epoch 代表在每次训练 epoch 中，小车训练的次数（到达终点就会停止该 epoch 转入下次训练），这应当是一个逐步递减的图像。

---

**问题 8**：使用 `runner.plot_results()` 输出训练结果，根据该结果对你的机器人进行分析。

- 指出你选用的参数如何，选用参数的原因。
- 建议你比较不同参数下机器人的训练的情况。
- 训练的结果是否满意，有何改进的计划。

```python
runner.plot_results()
```

(回答区)

---

**问题 9：** 请将如下的文件打包，提交文件给审阅者。

- `robot.py`
- `robot_maze.ipynb`
- 由 `robot_maze.ipynb` 导出的 `robot_maze.html`


# Surfinpy

#### Tutorial 3 - Pressure

Chemical potential can be converted to pressure values using

\begin{align}
P & = \frac{\mu_O}{k_B T} ,
\end{align}

where P is the pressure, $\mu$ is the chemical potential of oxygen, $k_B$ is the Boltzmnann constant and T is the temperature.

```python
from surfinpy import mu_vs_mu
```

```python
Oxygen_exp = mu_vs_mu.temperature_correction("O2.txt", 298)
Water_exp = mu_vs_mu.temperature_correction("H2O.txt", 298)
Oxygen_corrected = (-9.08 + -0.86 + Oxygen_exp) / 2
Water_corrected = -14.84 + 0.55 + Water_exp
```

```python
bulk = {'Cation' : 1, 'Anion' : 2, 'Energy' : -781.954938, 'F-Units' : 32}

pure =     {'Cation': 24, 'X': 48, 'Y': 0, 'Area': 60.0, 'Energy': -575.0,   'Label': 'Stoich',  'nSpecies': 1}
H2O =      {'Cation': 24, 'X': 48, 'Y': 2, 'Area': 60.0, 'Energy': -610.0,   'Label': '1 Water', 'nSpecies': 1}
H2O_2 =    {'Cation': 24, 'X': 48, 'Y': 4, 'Area': 60.0, 'Energy': -640.0,   'Label': '2 Water', 'nSpecies': 1}
H2O_3 =    {'Cation': 24, 'X': 48, 'Y': 8, 'Area': 60.0, 'Energy': -700.0,   'Label': '3 Water', 'nSpecies': 1}

Vo =       {'Cation': 24, 'X': 46, 'Y': 0, 'Area': 60.0, 'Energy': -560.0,   'Label': 'Vo', 'nSpecies': 1}
H2O_Vo =   {'Cation': 24, 'X': 46, 'Y': 2, 'Area': 60.0, 'Energy': -590.0,  'Label': 'Vo + 1 Water', 'nSpecies': 1}
H2O_Vo_2 = {'Cation': 24, 'X': 46, 'Y': 4, 'Area': 60.0, 'Energy': -620.0,  'Label': 'Vo + 2 Water', 'nSpecies': 1}
H2O_Vo_3 = {'Cation': 24, 'X': 46, 'Y': 6, 'Area': 60.0, 'Energy': -660.0, 'Label': 'Vo + 3 Water', 'nSpecies': 1}
H2O_Vo_4 = {'Cation': 24, 'X': 46, 'Y': 8, 'Area': 60.0, 'Energy': -680.0, 'Label': 'Vo + 4 Water', 'nSpecies': 1}

data = [pure, H2O_2, H2O_Vo, H2O,  H2O_Vo_2, H2O_3, H2O_Vo_3,  H2O_Vo_4, Vo]
```

```python
deltaX = {'Range': [ -10, -4],  'Label': 'O'}
deltaY = {'Range': [ -19, -12], 'Label': 'H_2O'}
```

```python
system =  mu_vs_mu.calculate(data, bulk, deltaX, deltaY, x_energy=Oxygen_corrected, y_energy=Water_corrected)
```

As before we can generate a basic plot of oxygen chemical potential vs water chemical potential at 298 K

```python
system.plot_phase(temperature=298)
```

We can also generate the same plot but with the $\mu$ values converted to pressure.

```python
system.plot_pressure(temperature=298)
```

Finally, we can also combine these two plots into one

```python
system.plot_mu_p(temperature=298)
```