# Week 6: Optimization techniques
**By Arash Ash, adapted from Neuromatch Academy [contents](https://github.com/NeuromatchAcademy/course-content-dl)**

---
# Tutorial Objectives

Objectives:
*   Necessity and importance of optimization
*   Introduction to commonly used optimization techniques
*   Optimization in non-convex loss landscapes
*   'Adaptive' hyperparameter tuning

---
# Setup

```python
# Imports
import time
import copy
import torch
import torchvision

import numpy as np
import ipywidgets as widgets
import matplotlib.pyplot as plt

import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision.datasets as datasets

from tqdm.auto import tqdm
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
plt.rc('axes', unicode_minus=False)
```

```python
# @title Helper functions
def print_params(model):
  for name, param in model.named_parameters():
    if param.requires_grad:
      print(name, param.data)
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    GPU is enabled in this notebook.

---
# Section 1. Introduction

```python
# @title Video 1: Introduction
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1VB4y1K7Vr", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"zm9oekdkJbQ", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')
```

## Discuss: Unexpected consequences

Can you think of examples from your own experience/life where poorly chosen incentives or objectives have lead to unexpected consequences?

---
# Section 2: Case study: successfully training an MLP for image classification

Many of the core ideas (and tricks) in modern optimization for deep learning can be illustrated in the simple setting of training an MLP to solve an image classification task. In this tutorial we will guide you through the key challenges that arise when optimizing high-dimensional, non-convex$^\dagger$ problems. We will use these challenges to motivate and explain some commonly used solutions.

**Disclaimer:** Some of the functions you will code in this tutorial are already implemented in Pytorch and many other libraries. For pedagogical reasons, we decided to bring these simple coding tasks into the spotlight and place a relatively higher emphasis in your understanding of the algorithms, rather than the use of a specific library.

In 'day-to-day' research projects you will likely to rely on the community-vetted, optimized libraries rather than the 'manual implementations' you will write today. In Section 8 you will have a chance to 'put it all together' and use the full power of Pytorch to tune the parameters of an MLP to classify handwritten digits.

$^\dagger$: A **convex** function has one, global minimum - a nice property, as an optimization algorithm won't get stuck in a local minimum that isn't a global one (e.g., $f(x)=x^2 + 2x + 1$). A **non-convex** function is wavy - has some 'valleys' (local minima) that aren't as deep as the overall deepest 'valley' (global minimum). Thus, the optimization algorithms can get stuck in the local minimum, and it can be hard to tell when this happens (e.g., $f(x) = x^4 + x^3 - 2x^2 - 2x$). See also **Section 5** for more details.

```python
# @title Video 2: Case Study - MLP Classification
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1GB4y1K7Ha", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"pJc2ENhYbqA", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

```

## Section 2.1: Data

We will use the MNIST dataset of handwritten digits. We load the data via the Pytorch `datasets` module, as you learned in W1D1.

**Note:** Although we can download the MNIST dataset directly from `datasets` using the optional argument `download=True`, we are going to download them from NMA directory on OSF to ensure network reliability.

```python
# @title Download MNIST dataset
import tarfile, requests, os

fname = 'MNIST.tar.gz'
name = 'MNIST'
url = 'https://osf.io/y2fj6/download'

if not os.path.exists(name):
  print('\nDownloading MNIST dataset...')
  r = requests.get(url, allow_redirects=True)
  with open(fname, 'wb') as fh:
    fh.write(r.content)
  print('\nDownloading MNIST completed.')

if not os.path.exists(name):
  with tarfile.open(fname) as tar:
    tar.extractall()
    os.remove(fname)
else:
  print('MNIST dataset has been dowloaded.')
```

    Downloading MNIST dataset...

    Downloading MNIST completed.

```python
def load_mnist_data(change_tensors=False, download=False):
  """Load training and test examples for the MNIST digits dataset

  Returns:
    train_data (tensor): training input tensor of size (train_size x 784)
    train_target (tensor): training 0-9 integer label tensor of size (train_size)
    test_data (tensor): test input tensor of size (70k-train_size x 784)
    test_target (tensor): training 0-9 integer label tensor of size (70k-train_size)

  """
  # Load train and test sets
  train_set = datasets.MNIST(root='.', train=True, download=download,
                             transform=torchvision.transforms.ToTensor())
  test_set = datasets.MNIST(root='.', train=False, download=download,
                            transform=torchvision.transforms.ToTensor())

  # Original data is in range [0, 255]. We normalize the data wrt its mean and std_dev.
  ## Note that we only used *training set* information to compute mean and std
  mean = train_set.data.float().mean()
  std = train_set.data.float().std()

  if change_tensors:
    # Apply normalization directly to the tensors containing the dataset
    train_set.data = (train_set.data.float() - mean) / std
    test_set.data = (test_set.data.float() - mean) / std
  else:
    tform = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                                            torchvision.transforms.Normalize(mean=[mean / 255.], std=[std / 255.])
                                            ])
    train_set = datasets.MNIST(root='.', train=True, download=download,
                               transform=tform)
    test_set = datasets.MNIST(root='.', train=False, download=download,
                              transform=tform)

  return train_set, test_set

train_set, test_set = load_mnist_data(change_tensors=True)
```

As we are just getting started, we will concentrate on a small subset of only 500 examples out of the 60.000 data points contained in the whole training set.

```python
# Sample a random subset of 500 indices
subset_index = np.random.choice(len(train_set.data), 500)

# We will use these symbols to represent the training data and labels, to stay
# as close to the mathematical expressions as possible.
X, y = train_set.data[subset_index, :], train_set.targets[subset_index]
```

Run the following cell to visualize the content of three examples in our training set. Note how the pre-processing we applied to the data changes the range of pixel values after normalization.

```python
# @title Run me!
num_figures = 3
fig, axs = plt.subplots(1, num_figures, figsize=(5 * num_figures, 5))

for sample_id, ax in enumerate(axs):
  # Plot the pixel values for each image
  ax.matshow(X[sample_id, :], cmap='gray_r')
  # 'Write' the pixel value in the corresponding location
  for (i, j), z in np.ndenumerate(X[sample_id, :]):
    text = '{:.1f}'.format(z)
    ax.text(j, i, text, ha='center',
            va='center', fontsize=6, c='steelblue')

  ax.set_title('Label: ' + str(y[sample_id].item()))
  ax.axis('off')

plt.show()
```

## Section 2.2: Model

As you will see next week, there are specific model architectures that are better suited to image-like data, such as Convolutional Neural Networks (CNNs). For simplicity, in this tutorial we will focus exclusively on Multi-Layer Perceptron (MLP) models as they allow us to highlight many important optimization challenges shared with more advanced neural network designs.

```python
class MLP(nn.Module):
  """ This class implements MLPs in Pytorch of an arbitrary number of hidden
  layers of potentially different sizes. Since we concentrate on classification
  tasks in this tutorial, we have a log_softmax layer at prediction time.
  """

  def __init__(self, in_dim=784, out_dim=10, hidden_dims=[], use_bias=True):
    """Constructs a MultiLayerPerceptron

    Args:
        in_dim (int): dimensionality of input data
        out_dim (int): number of classes
        hidden_dims (list): contains the dimensions of the hidden layers, an empty
            list corresponds to a linear model (in_dim, out_dim)
    """

    super(MLP, self).__init__()

    self.in_dim = in_dim
    self.out_dim = out_dim

    # If we have no hidden layer, just initialize a linear model (e.g. in logistic regression)
    if len(hidden_dims) == 0:
      layers = [nn.Linear(in_dim, out_dim, bias=use_bias)]
    else:
      # 'Actual' MLP with dimensions in_dim - num_hidden_layers*[hidden_dim] - out_dim
      layers = [nn.Linear(in_dim, hidden_dims[0], bias=use_bias), nn.ReLU()]

      # Loop until before the last layer
      for i, hidden_dim in enumerate(hidden_dims[:-1]):
        layers += [nn.Linear(hidden_dim, hidden_dims[i + 1], bias=use_bias),
                   nn.ReLU()]

      # Add final layer to the number of classes
      layers += [nn.Linear(hidden_dims[-1], out_dim, bias=use_bias)]

    self.main = nn.Sequential(*layers)

  def forward(self, x):
    # Flatten the images into 'vectors'
    transformed_x = x.view(-1, self.in_dim)
    hidden_output = self.main(transformed_x)
    output = F.log_softmax(hidden_output, dim=1)
    return output
```

Linear models constitute a very special kind of MLPs: they are equivalent to an MLP with *zero* hidden layers. This is simply an affine transformation, in other words a 'linear' map $W x$ with an 'offset' $b$; followed by a softmax function.

$$f(x) = \text{softmax}(W x + b)$$

Here $x \in \mathbb{R}^{784}$, $W \in \mathbb{R}^{10 \times 784}$ and $b \in \mathbb{R}^{10}$. Notice that the dimensions of the weight matrix are $10 \times 784$ as the input tensors are flattened images, i.e., $28 \times 28 = 784$-dimensional tensors and the output layer consists of $10$ nodes.

```python
# Empty hidden_dims means we take a model with zero hidden layers.
model = MLP(in_dim=784, out_dim=10, hidden_dims=[])

# We print the model structure with 784 inputs and 10 outputs
print(model)
```

    MLP(
      (main): Sequential(
        (0): Linear(in_features=784, out_features=10, bias=True)
      )
    )

## Section 2.3: Loss

While we care about the accuracy of the model, the 'discrete' nature of the 0-1 loss makes it challenging to optimize. In order to learn good parameters for this model, we will use the cross entropy loss (negative log-likelihood), which you saw in last lecture, as a surrogate objective to be minimized.

This particular choice of model and optimization objective leads to a *convex* optimization problem with respect to the parameters $W$ and $b$.

```python
loss_fn = F.nll_loss
```

## Section 2.4: Interpretability

In last lecture, you saw that inspecting the weights of a model can provide insights on what 'concepts' the model has learned. Here we show the weights of a partially trained model. The weights corresponding to each class 'learn' to _fire_ when an input of the class is detected.

```python
#@markdown Run _this cell_ to train the model. If you are curious about how the training
#@markdown takes place, double-click this cell to find out. At the end of this tutorial
#@markdown you will have the opportunity to train a more complex model on your own.

cell_verbose = False
partial_trained_model = MLP(in_dim=784, out_dim=10, hidden_dims=[])

if cell_verbose:
  print('Init loss', loss_fn(partial_trained_model(X), y).item()) # This matches around np.log(10 = # of classes)

optimizer = optim.Adam(partial_trained_model.parameters(), lr=7e-4)
for _ in range(200):
  loss = loss_fn(partial_trained_model(X), y)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

if cell_verbose:
  print('End loss', loss_fn(partial_trained_model(X), y).item()) # This should be less than 1e-2
```

```python
# Show class filters of a trained model
W = partial_trained_model.main[0].weight.data.numpy()

fig, axs = plt.subplots(1, 10, figsize=(15, 4))
for class_id in range(10):
  axs[class_id].imshow(W[class_id, :].reshape(28, 28), cmap='gray_r')
  axs[class_id].axis('off')
  axs[class_id].set_title('Class ' + str(class_id) )

plt.show()
```

---
# Section 3: High dimensional search

We now have a model with its corresponding trainable parameters as well as an objective to optimize. Where do we go to next? How do we find a 'good' configuration of parameters?

One idea is to choose a random direction and move only if the objective is reduced. However, this is inefficient in high dimensions and you will see how gradient descent (with a suitable step-size) can guarantee consistent improvement in terms of the objective function.

```python
# @title Video 3: Optimization of an Objective Function
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1aL411H7Ce", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"aSJTRdjRvvw", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')
```

## Coding Exercise 3: Implement gradient descent

In this exercise you will use PyTorch automatic differentiation capabilities to compute the gradient of the loss with respect to the parameters of the model. You will then use these gradients to implement the update performed by the gradient descent method.

```python
def zero_grad(params):
  """Clear up gradients as Pytorch automatically accumulates gradients from
  successive backward calls
  """
  for par in params:
    if not(par.grad is None):
      par.grad.data.zero_()

def random_update(model, noise_scale=0.1, normalized=False):
  """ Performs a random update on the parameters of the model
  """
  for par in model.parameters():
    noise = torch.randn_like(par)
    if normalized:
      noise /= torch.norm(noise)
    par.data +=  noise_scale * noise
```

```python
def gradient_update(loss, params, lr=1e-3):
  """Perform a gradient descent update on a given loss over a collection of parameters

  Args:
    loss (tensor): A scalar tensor containing the loss whose gradient will be computed
    params (iterable): Collection of parameters with respect to which we compute gradients
    lr (float): Scalar specifying the learning rate or step-size for the update
  """
  # Clear up gradients as Pytorch automatically accumulates gradients from
  # successive backward calls
  zero_grad(params)

  # Compute gradients on given objective
  loss.backward()

  with torch.no_grad():
    for par in params:
      #################################################
      ## TODO for students: update the value of the parameter ##
      raise NotImplementedError("Student exercise: implement gradient update")
      #################################################
      # Here we work with the 'data' attribute of the parameter rather than the
      # parameter itself.
      par.data -= ...

set_seed(seed=SEED)
model1 = MLP(in_dim=784, out_dim=10, hidden_dims=[])
print('\n The model1 parameters before the update are: \n')
print_params(model1)
loss = loss_fn(model1(X), y)

## Uncomment below to test your function
# gradient_update(loss, list(model1.parameters()), lr=1e-1)
# print('\n The model1 parameters after the update are: \n')
# print_params(model1)
```

    Random seed 2021 has been set.

     The model1 parameters before the update are:

    main.0.weight tensor([[-0.0264,  0.0010,  0.0173,  ...,  0.0297,  0.0278, -0.0221],
            [-0.0040, -0.0295, -0.0086,  ..., -0.0070,  0.0254, -0.0233],
            [ 0.0240, -0.0231,  0.0342,  ...,  0.0124,  0.0270, -0.0180],
            ...,
            [-0.0005,  0.0157,  0.0111,  ...,  0.0144, -0.0301, -0.0144],
            [ 0.0181,  0.0303,  0.0255,  ..., -0.0110, -0.0175,  0.0205],
            [ 0.0208, -0.0353, -0.0183,  ..., -0.0271,  0.0099,  0.0003]])
    main.0.bias tensor([-0.0290, -0.0033,  0.0100, -0.0320,  0.0022,  0.0221,  0.0307,  0.0243,
             0.0159, -0.0064])

```
 The model1 parameters after the update are:

main.0.weight tensor([[-0.0263,  0.0010,  0.0174,  ...,  0.0298,  0.0278, -0.0220],
        [-0.0047, -0.0302, -0.0093,  ..., -0.0077,  0.0248, -0.0240],
        [ 0.0234, -0.0237,  0.0335,  ...,  0.0117,  0.0263, -0.0187],
        ...,
        [-0.0006,  0.0156,  0.0110,  ...,  0.0143, -0.0302, -0.0145],
        [ 0.0164,  0.0286,  0.0238,  ..., -0.0127, -0.0191,  0.0188],
        [ 0.0206, -0.0354, -0.0184,  ..., -0.0272,  0.0098,  0.0002]])
main.0.bias tensor([-0.0292, -0.0018,  0.0115, -0.0370,  0.0054,  0.0155,  0.0317,  0.0246,
         0.0198, -0.0061])
```

## Comparing updates

These plots compare the effectiveness of updating random directions for the problem of optimizing the parameters of a high-dimensional linear model. We contrast the behavior at initialization and during an intermediate stage of training by showing the histograms of change in loss over 100 different random directions vs the changed in loss induced by the gradient descent update

**Remember:** since we are trying to minimize, here negative is better!

```python
# @markdown _Run this cell_ to visualize the results
fig, axs = plt.subplots(1, 2, figsize=(10, 4))

for id, (model_name, my_model) in enumerate([('Initialization', model),
                                              ('Partially trained', partial_trained_model)]):
  # Compue the loss we will be comparing to
  base_loss = loss_fn(my_model(X), y)

  # Compute the improvement via gradient descent
  dummy_model = copy.deepcopy(my_model)
  loss1 = loss_fn(dummy_model(X), y)
  gradient_update(loss1, list(dummy_model.parameters()), lr=1e-2)
  gd_delta = loss_fn(dummy_model(X), y) - base_loss

  deltas = []
  for trial_id in range(100):
    # Compute the improvement obtained with a random direction
    dummy_model = copy.deepcopy(my_model)
    random_update(dummy_model, noise_scale=1e-2)
    deltas.append((loss_fn(dummy_model(X), y) - base_loss).item())

  # Plot histogram for random direction and vertical line for gradient descent
  axs[id].hist(deltas, label='Random Directions', bins=20)
  axs[id].set_title(model_name)
  axs[id].set_xlabel('Change in loss')
  axs[id].set_ylabel('% samples')
  axs[id].axvline(0, c='green', alpha=0.5)
  axs[id].axvline(gd_delta.item(), linestyle='--', c='red', alpha=1,
                  label='Gradient Descent')

handles, labels = axs[id].get_legend_handles_labels()
fig.legend(handles, labels, loc='upper center',
           bbox_to_anchor=(0.5, 1.05),
           fancybox=False, shadow=False, ncol=2)

plt.show()
```

## Think! 3: Gradient descent vs. random search

Compare the behavior of gradient descent and random search based on the histograms above. Is any of the two methods more reliable? How can you explain the changes between behavior of the methods at initialization vs during training?

---
# Section 4: Poor conditioning

Already in this 'simple' logistic regression problem, the issue of bad conditioning is haunting us. Not all parameters are created equal and the sensitivity of the network to changes on the parameters will have a big impact in the dynamics of the optimization.

```python
# @title Video 4: Momentum
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1NL411H71t", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"3ES5O58Y_2M", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

```

We illustrate this issue in a 2-dimensional setting. We freeze all but two parameters of the network: one of them is an element of the weight matrix (filter) for class 0, while the other is the bias for class 7. This results in an optimization with two decision variables.

How much difference is there in the behavior of these two parameters under gradient descent? What is the effect of momentum in bridging that gap?

```python
# @markdown _Run this cell_ to setup some helper functions.

def loss_2d(model, u, v, mask_idx=(0, 378), bias_id=7):
  """Defines a 2-dim function by freezing all but two parameters of a linear
  model.

  Args:
    model (torch module): a pytorch 0-hidden layer (linear) model
    u (scalar): first free parameter
    v (scalar): second free parameter
    mask_idx (tuple): selects parameter in weight matrix replaced by u
    bias_idx (int): selects parameter in bias vector replaced by v

  Returns:
    scalar: loss of the 'new' model over inputs X, y (defined externally)
  """

  # We zero out the element of the weight tensor that will be
  # replaced by u
  mask = torch.ones_like(model.main[0].weight)
  mask[mask_idx[0], mask_idx[1]] = 0.
  masked_weights = model.main[0].weight * mask

  # u is replacing an element of the weight matrix
  masked_weights[mask_idx[0], mask_idx[1]] = u

  res = X.reshape(-1, 784) @ masked_weights.T + model.main[0].bias

  # v is replacing a bias for class 7
  res[:, 7] += v - model.main[0].bias[7]
  res =  F.log_softmax(res, dim=1)

  return loss_fn(res, y)

def plot_surface(U, V, Z, fig):
  """ Plot a 3D loss surface given meshed inputs U, V and values Z
  """
  ax = fig.add_subplot(1, 2, 2, projection='3d')
  ax.view_init(45, -130)

  surf = ax.plot_surface(U, V, Z, cmap=plt.cm.coolwarm,
                      linewidth=0, antialiased=True, alpha=0.5)

  # Select certain level contours to plot
  # levels = Z.min() * np.array([1.005, 1.1, 1.3, 1.5, 2.])
  # plt.contour(U, V, Z)# levels=levels, alpha=0.5)

  ax.set_xlabel('Weight')
  ax.set_ylabel('Bias')
  ax.set_zlabel('Loss', rotation=90)

  return ax

def plot_param_distance(best_u, best_v, trajs, fig, styles, labels,
                        use_log=False, y_min_v=-12.0, y_max_v=1.5):
  """ Plot the distance to each of the two parameters for a collection of 'trajectories'
  """
  ax = fig.add_subplot(1, 1, 1)

  for traj, style, label in zip(trajs, styles, labels):
    d0 = np.array([np.abs(_[0] - best_u) for _ in traj])
    d1 = np.array([np.abs(_[1] - best_v) for _ in traj])
    if use_log:
      d0 = np.log(1e-16 + d0)
      d1 = np.log(1e-16 + d1)
    ax.plot(range(len(traj)), d0, style, label='weight - ' + label)
    ax.plot(range(len(traj)), d1, style, label='bias - ' + label)
  ax.set_xlabel('Iteration')
  if use_log:
    ax.set_ylabel('Log distance to optimum (per dimension)')
    ax.set_ylim(y_min_v, y_max_v)
  else:
    ax.set_ylabel('Abs distance to optimum (per dimension)')
  ax.legend(loc='right', bbox_to_anchor=(1.5, 0.5),
            fancybox=False, shadow=False, ncol=1)

  return ax

def run_optimizer(inits, eval_fn, update_fn, max_steps=500,
                  optim_kwargs={'lr':1e-2}, log_traj=True):
  """Runs an optimizer on a given objective and logs parameter trajectory

  Args:
      inits list(scalar): initialization of parameters
      eval_fn (callable): function computing the objective to be minimized
      update_fn (callable): function executing parameter update
      max_steps (int): number of iterations to run
      optim_kwargs (dict): customize optimizer hyperparameters

  Returns:
      list[list]: trajectory information [*params, loss] for each optimization step
  """

  # Initialize parameters and optimizer
  params = [nn.Parameter(torch.tensor(_)) for _ in inits]
  # Methods like momentum and rmsprop keep and auxiliary vector of parameters
  aux_tensors = [torch.zeros_like(_) for _ in params]
  if log_traj:
    traj = np.zeros((max_steps, len(params)+1))
  for _ in range(max_steps):
    # Evaluate loss
    loss = eval_fn(*params)
    # Store 'trajectory' information
    if log_traj:
      traj[_, :] = [_.item() for _ in params] + [loss.item()]
    # Perform update
    if update_fn == gradient_update:
      gradient_update(loss, params, **optim_kwargs)
    else:
      update_fn(loss, params, aux_tensors, **optim_kwargs)
  if log_traj:
    return traj

L = 4.
xs = np.linspace(-L, L, 30)
ys = np.linspace(-L, L, 30)
U, V = np.meshgrid(xs, ys)
```

## Coding Exercise 4: Implement momentum

In this exercise you will implement the momentum update given by:

\begin{equation}
w_{t+1} = w_t - \eta \nabla J(w_t) + \beta (w_t - w_{t-1})
\end{equation}

It is convenient to re-express this update rule in terms of a recursion. For that, we define 'velocity' as the quantity:
\begin{equation}
v_{t-1} := w_{t} - w_{t-1}
\end{equation}

which leads to the two-step update rule:

\begin{equation}
v_t = - \eta \nabla J(w_t) + \beta (\underbrace{w_t - w_{t-1}}_{v_{t-1}})
\end{equation}

\begin{equation}
w_{t+1} \leftarrow w_t + v_{t}
\end{equation}

Pay attention to the positive sign of the update in the last equation, given the definition of $v_t$, above.

```python
def momentum_update(loss, params, grad_vel, lr=1e-3, beta=0.8):
  """Perform a momentum update over a collection of parameters given a loss and 'velocities'

  Args:
    loss (tensor): A scalar tensor containing the loss whose gradient will be computed
    params (iterable): Collection of parameters with respect to which we compute gradients
    grad_vel (iterable): Collection containing the 'velocity' v_t for each parameter
    lr (float): Scalar specifying the learning rate or step-size for the update
    beta (float): Scalar 'momentum' parameter
  """
  # Clear up gradients as Pytorch automatically accumulates gradients from
  # successive backward calls
  zero_grad(params)
  # Compute gradients on given objective
  loss.backward()

  with torch.no_grad():
    for (par, vel) in zip(params, grad_vel):
      #################################################
      ## TODO for students: update the value of the parameter ##
      raise NotImplementedError("Student exercise: implement momentum update")
      #################################################
      # Update 'velocity'
      vel.data = ...
      # Update parameters
      par.data += ...

set_seed(seed=SEED)
model2 = MLP(in_dim=784, out_dim=10, hidden_dims=[])
print('\n The model2 parameters before the update are: \n')
print_params(model2)
loss = loss_fn(model2(X), y)
initial_vel = [torch.randn_like(p) for p in model2.parameters()]

## Uncomment below to test your function
# momentum_update(loss, list(model2.parameters()), grad_vel=initial_vel, lr=1e-1, beta=0.9)
# print('\n The model2 parameters after the update are: \n')
# print_params(model2)
```

    Random seed 2021 has been set.

     The model2 parameters before the update are:

    main.0.weight tensor([[-0.0264,  0.0010,  0.0173,  ...,  0.0297,  0.0278, -0.0221],
            [-0.0040, -0.0295, -0.0086,  ..., -0.0070,  0.0254, -0.0233],
            [ 0.0240, -0.0231,  0.0342,  ...,  0.0124,  0.0270, -0.0180],
            ...,
            [-0.0005,  0.0157,  0.0111,  ...,  0.0144, -0.0301, -0.0144],
            [ 0.0181,  0.0303,  0.0255,  ..., -0.0110, -0.0175,  0.0205],
            [ 0.0208, -0.0353, -0.0183,  ..., -0.0271,  0.0099,  0.0003]])
    main.0.bias tensor([-0.0290, -0.0033,  0.0100, -0.0320,  0.0022,  0.0221,  0.0307,  0.0243,
             0.0159, -0.0064])

```
 The model2 parameters after the update are:

main.0.weight tensor([[ 1.5898,  0.0116, -2.0239,  ..., -1.0871,  0.4030, -0.9577],
        [ 0.4653,  0.6022, -0.7363,  ...,  0.5485, -0.2747, -0.6539],
        [-1.4117, -1.1045,  0.6492,  ..., -1.0201,  0.6503,  0.1310],
        ...,
        [-0.5098,  0.5075, -0.0718,  ...,  1.1192,  0.2900, -0.9657],
        [-0.4405, -0.1174,  0.7542,  ...,  0.0792, -0.1857,  0.3537],
        [-1.0824,  1.0080, -0.4254,  ..., -0.3760, -1.7491,  0.6025]])
main.0.bias tensor([ 0.4147, -1.0440,  0.8720, -1.6201, -0.9632,  0.9430, -0.5180,  1.3417,
         0.6574,  0.3677])
```

## Interactive Demo 4: Momentum vs. GD

The plots below show the distance to the optimum for both variables across the two methods, as well as the parameter trajectory over the loss surface.

```python
# @markdown Run this cell to enable the widget!
from matplotlib.lines import Line2D

# Find the optimum of this 2D problem using Newton's method
def run_newton(func, init_list=[0., 0.], max_iter=200):

  par_tensor = torch.tensor(init_list, requires_grad=True)
  t_g = lambda par_tensor: func(par_tensor[0], par_tensor[1])

  for _ in tqdm(range(max_iter)):
    eval_loss = t_g(par_tensor)
    eval_grad = torch.autograd.grad(eval_loss, [par_tensor])[0]
    eval_hess = torch.autograd.functional.hessian(t_g, par_tensor)
    # Newton's update is:  - inverse(Hessian) x gradient
    par_tensor.data -= torch.inverse(eval_hess) @ eval_grad

  return par_tensor.data.numpy()

set_seed(2021)
model = MLP(in_dim=784, out_dim=10, hidden_dims=[])
# Define 2d loss objectives and surface values
g = lambda u, v: loss_2d(copy.deepcopy(model), u, v)
Z = np.fromiter(map(g, U.ravel(), V.ravel()), U.dtype).reshape(V.shape)

best_u, best_v  = run_newton(func=g)

# Initialization of the variables
INITS = [2.5, 3.7]

# Used for plotting
LABELS = ['GD', 'Momentum']
COLORS = ['black', 'red']
LSTYLES = ['-', '--']

@widgets.interact_manual
def momentum_experiment(max_steps=widgets.IntSlider(300, 50, 500, 5),
                        lr=widgets.FloatLogSlider(value=1e-1, min=-3, max=0.7, step=0.1),
                        beta=widgets.FloatSlider(value=9e-1, min=0, max=1., step=0.01)
                        ):

  # Execute both optimizers
  sgd_traj = run_optimizer(INITS, eval_fn=g, update_fn=gradient_update,
                           max_steps=max_steps, optim_kwargs={'lr': lr})
  mom_traj = run_optimizer(INITS, eval_fn=g, update_fn=momentum_update,
                           max_steps=max_steps, optim_kwargs={'lr': lr, 'beta':beta})

  TRAJS = [sgd_traj, mom_traj]

  # Plot distances
  fig = plt.figure(figsize=(9,4))
  plot_param_distance(best_u, best_v, TRAJS, fig,
                      LSTYLES, LABELS, use_log=True, y_min_v=-12.0, y_max_v=1.5)

  # # Plot trajectories
  fig = plt.figure(figsize=(12, 5))
  ax = plot_surface(U, V, Z, fig)
  for traj, c, label in zip(TRAJS, COLORS, LABELS):
    ax.plot3D(*traj.T, c, linewidth=0.3, label=label)
    ax.scatter3D(*traj.T, '.-', s=1, c=c)

  # Plot optimum point
  ax.scatter(best_u, best_v, Z.min(), marker='*', s=80, c='lime', label='Opt.');
  lines = [Line2D([0], [0],
                  color=c,
                  linewidth=3,
                  linestyle='--') for c in COLORS]
  lines.append(Line2D([0], [0], color='lime', linewidth=0, marker='*'))
  ax.legend(lines, LABELS + ['Optimum'], loc='right',
            bbox_to_anchor=(.8, -0.1), ncol=len(LABELS) + 1)
```

    Random seed 2021 has been set.

      0%|          | 0/200 [00:00<?, ?it/s]

    interactive(children=(IntSlider(value=300, description='max_steps', max=500, min=50, step=5), FloatLogSlider(v…

## Think! 4: Momentum and oscillations

- Discuss how this specific example illustrates the issue of poor conditioning in optimization? How does momentum help resolve these difficulties?

- Do you see oscillations for any of these methods? Why does this happen?

- Finally, tune the learning rate and momentum parameters to achieve a loss below $10^{-6}$ (for both dimensions) within 100 iterations.

---
# Section 5: Non-convexity

The introduction of even just 1 hidden layer in the neural network transforms the previous convex optimization problem into a non-convex one. And with great non-convexity, comes great responsibility... (Sorry, we couldn't help it!)

**Note:** From this section onwards we will be dealing with non-convex optimization problems for the remaining of the tutorial.

```python
# @title Video 5: Overparametrization
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV16h41167Jr", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"7vUpUEKKl5o", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')
```

Take a couple of minutes to play with a more complex 3D visualization of the loss landscape of a neural network on a non-convex problem. Visit https://losslandscape.com/explorer.

1. Explore the features on the bottom left corner. You can see an explanation for each icon by clicking on the [i] button located on the top right corner.
2. Use the 'gradient descent' feature to perform a thought experiment:
    -   Choose an initialization
    -   Choose the learning rate
    -   Mentally formulate your hypothesis about what kind of trajectory you expect to observe
3. Run the experiment and contrast your intuition with the observed behavior.
4. Repeat this experiment a handful of times for several initialization/learning rate configurations

## Interactive Demo 5: Overparametrization to the rescue!

As you may have seen, the non-convex nature of the surface can lead the optimization process to get stuck in undesirable local-optima. There is ample empirical evidence supporting the claim that 'overparameterized' models are easier to train.

We will explore this assertion in the context of our MLP training. For this, we initialize a fixed model and construct several models by small random perturbations to the original initialized weights. Now, we train each of these perturbed models and see how the loss evolves. If we were in the convex setting, we should reach very similar objective values upon convergence since all these models were very close at the beginning of training, and in convex problems, every local optimum is also a global optimum.

Use the interactive plot below to visualize the loss progression for these perturbed models:

1. Select different settings from the `hidden_dims` drop-down menu.
2. Explore the effect of the number of steps and learning rate.

```python
# @markdown Execute this cell to enable the widget!

@widgets.interact_manual
def overparam(max_steps=widgets.IntSlider(150, 50, 500, 5),
              hidden_dims=widgets.Dropdown(options=["10", "20, 20", "100, 100"],
                                           value="10"),
              lr=widgets.FloatLogSlider(value=5e-2, min=-3, max=0, step=0.1),
              num_inits=widgets.IntSlider(7, 5, 10, 1)):

    X, y = train_set.data[subset_index, :], train_set.targets[subset_index]

    hdims = [int(s) for s in hidden_dims.split(',')]
    base_model = MLP(in_dim=784, out_dim=10, hidden_dims=hdims)

    fig, axs = plt.subplots(1, 1, figsize=(5, 4))

    for _ in tqdm(range(num_inits)):
      model = copy.deepcopy(base_model)
      random_update(model, noise_scale=2e-1)

      loss_hist = np.zeros((max_steps, 2))
      for step in range(max_steps):
        loss = loss_fn(model(X), y)
        gradient_update(loss, list(model.parameters()), lr=lr)
        loss_hist[step] = np.array([step, loss.item()])

      plt.plot(loss_hist[:, 0], loss_hist[:, 1])

    plt.xlabel('Iteration')
    plt.ylabel('Loss')
    plt.ylim(0, 3)
    plt.show()

    num_params = sum([np.prod(_.shape) for _ in model.parameters()])
    print('Number of paramaters in model:  ' + str(num_params))
```

    interactive(children=(IntSlider(value=150, description='max_steps', max=500, min=50, step=5), Dropdown(descrip…

### Think! 5.1: Width and depth of the network

- We see that as we increase the width/depth of the network, training becomes faster and more consistent across different initializations. What might be the reasons for this behavior?

- What are some potential downsides of this approach to dealing with non-convexity?

---
# Section 6: Full gradients are expensive

So far we have used only a small (fixed) subset of 500 training examples to perform the updates on the model parameters in our quest to minimize the loss. But what if we decided to use the training set? Do our current approach scale to datasets with tens of thousands, or millions of datapoints?

In this section we explore an efficient alternative to avoid having to perform computations on all the training examples before performing a parameter update.

```python
# @title Video 6: Mini-batches
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1ty4y1T7Uh", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"hbqUxpNBUGk", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Interactive Demo 6.1: Cost of computation

Evaluating a neural network is a relatively fast process. However, when repeated millions of times, the computational cost of performing forward and backward passes through the network starts to become significant.

In the visualization below, we show the time (averaged over 5 runs) of computing a forward and backward pass with a changing number of input examples. Choose from the different options in the drop-down box and note how the vertical scale changes depending on the size of the network.

**Remarks:** Note that the computational cost of a forward pass shows a clear linear relationship with the number of input examples, and the cost of the corresponding backward pass exhibits a similar computational complexity.

```python
# @markdown Execute this cell to enable the widget!

def measure_update_time(model, num_points):
  X, y = train_set.data[:num_points], train_set.targets[:num_points]
  start_time = time.time()
  loss = loss_fn(model(X), y)
  loss_time = time.time()
  gradient_update(loss, list(model.parameters()), lr=0)
  gradient_time = time.time()
  return loss_time - start_time, gradient_time - loss_time

@widgets.interact
def computation_time(hidden_dims=widgets.Dropdown(options=["1", "100", "50, 50"],
                                                  value="100")):

  hdims = [int(s) for s in hidden_dims.split(',')]
  model = MLP(in_dim=784, out_dim=10, hidden_dims=hdims)

  NUM_POINTS = [1, 5, 10, 100, 200, 500, 1000, 5000, 10000, 20000, 30000, 50000]
  times_list = []
  for _ in range(5):
    times_list.append(np.array([measure_update_time(model, _) for _ in NUM_POINTS]))

  times = np.array(times_list).mean(axis=0)

  fig, axs = plt.subplots(1, 1, figsize=(5,4))
  plt.plot(NUM_POINTS, times[:, 0], label='Forward')
  plt.plot(NUM_POINTS, times[:, 1], label='Backward')
  plt.xlabel('Number of data points')
  plt.ylabel('Seconds')
  plt.legend()
```

    interactive(children=(Dropdown(description='hidden_dims', index=1, options=('1', '100', '50, 50'), value='100'…

## Coding Exercise 6: Implement minibatch sampling

Complete the code in `sample_minibatch` so as to produce IID subsets of the training set of the desired size. (This is _not_ a trick question.)

```python
def sample_minibatch(input_data, target_data, num_points=100):
  """Sample a minibatch of size num_point from the provided input-target data

  Args:
    input_data (tensor): Multi-dimensional tensor containing the input data
    target_data (tensor): 1D tensor containing the class labels
    num_points (int): Number of elements to be included in minibatch

  Returns:
    batch_inputs (tensor): Minibatch inputs
    batch_targets (tensor): Minibatch targets
  """
  #################################################
  ## TODO for students: sample minibatch of data ##
  raise NotImplementedError("Student exercise: implement gradient update")
  #################################################
  # Sample a collection of IID indices from the existing data
  batch_indices = ...
  # Use batch_indices to extract entries from the input and target data tensors
  batch_inputs = input_data[...]
  batch_targets = target_data[...]

  return batch_inputs, batch_targets

## Uncomment to test your function
# x_batch, y_batch = sample_minibatch(X, y, num_points=100)
# print(f"The input shape is {x_batch.shape} and the target shape is: {y_batch.shape}")
```

```python
# to_remove solution
def sample_minibatch(input_data, target_data, num_points=100):
  """Sample a minibatch of size num_point from the provided input-target data

  Args:
    input_data (tensor): Multi-dimensional tensor containing the input data
    target_data (tensor): 1D tensor containing the class labels
    num_points (int): Number of elements to be included in minibatch

  Returns:
    batch_inputs (tensor): Minibatch inputs
    batch_targets (tensor): Minibatch targets
  """
  # Sample a collection of IID indices from the existing data
  batch_indices = np.random.choice(len(input_data), num_points)
  # Use batch_indices to extract entries from the input and target data tensors
  batch_inputs = input_data[batch_indices, :]
  batch_targets = target_data[batch_indices]

  return batch_inputs, batch_targets

## Uncomment to test your function
x_batch, y_batch = sample_minibatch(X, y, num_points=100)
print(f"The input shape is {x_batch.shape} and the target shape is: {y_batch.shape}")
```

    The input shape is torch.Size([100, 28, 28]) and the target shape is: torch.Size([100])

```
The input shape is torch.Size([100, 28, 28]) and the target shape is: torch.Size([100])
```

## Interactive Demo 6.2: *Compare* different minibatch sizes

What are the trade-offs induced by the choice of minibatch size? The interactive plot below shows the training evolution of a 2-hidden layer MLP with 100 hidden units in each hidden layer. Different plots correspond to a different choice of minibatch size. We have a fixed time budget for all the cases, reflected in the horizontal axes of these plots.

```python
# @markdown Execute this cell to enable the widget!

@widgets.interact_manual
def minibatch_experiment(batch_sizes='20, 250, 1000',
                         lrs='5e-3, 5e-3, 5e-3',
                         time_budget=widgets.Dropdown(options=["2.5", "5", "10"],
                                                      value="2.5")):

  batch_sizes = [int(s) for s in batch_sizes.split(',')]
  lrs = [float(s) for s in lrs.split(',')]

  LOSS_HIST = {_:[] for _ in batch_sizes}

  X, y = train_set.data, train_set.targets
  base_model = MLP(in_dim=784, out_dim=10, hidden_dims=[100, 100])

  for id, batch_size in enumerate(tqdm(batch_sizes)):
    start_time = time.time()
    # Create a new copy of the model for each batch size
    model = copy.deepcopy(base_model)
    params = list(model.parameters())
    lr = lrs[id]
    # Fixed budget per choice of batch size
    while (time.time() - start_time) < float(time_budget):
      data, labels = sample_minibatch(X, y, batch_size)
      loss = loss_fn(model(data), labels)
      gradient_update(loss, params, lr=lr)
      LOSS_HIST[batch_size].append([time.time() - start_time,
                                    loss.item()])

  fig, axs = plt.subplots(1, len(batch_sizes), figsize=(10, 3))
  for ax, batch_size in zip(axs, batch_sizes):
    plot_data = np.array(LOSS_HIST[batch_size])
    ax.plot(plot_data[:, 0], plot_data[:, 1], label=batch_size,
            alpha=0.8)
    ax.set_title('Batch size: ' + str(batch_size))
    ax.set_xlabel('Seconds')
    ax.set_ylabel('Loss')
  plt.show()
```

    interactive(children=(Text(value='20, 250, 1000', description='batch_sizes'), Text(value='5e-3, 5e-3, 5e-3', d…

**Remarks:** SGD works! We have an algorithm that can be applied (with the due precautions) to learn datasets of arbitrary size.

However, **note the diference in the vertical scale** across the plots above. When using a larger minibatch, we can perform fewer parameter updates as the forward and backward passes are more expensive.

This highlights the interplay between the minibatch size and the learning rate: when our minibatch is larger, we have a more confident estimator of the direction to move, and thus can afford a larger learning rate. On the other hand, extremely small minibatches are very fast computationally but are not representative of the data distribution and yield estimations of the gradient with high variance.

We encourage you to tune the value of the learning rate for each of the minibatch sizes in the previous demo, to achieve a training loss steadily below 0.5 within 5 seconds.

---
# Section 7: Adaptive methods

As of now, you should be aware that there are many knobs to turn when working on a machine learning problem. Some of these relate to the optimization algorithm, to the choice of model, or to the objective to minimize. Here are some prototypical examples:

- Problem: loss function, regularization coefficients (Day 5)
- Model: architecture, activations function
- Optimizer: learning rate, batch size, momentum coefficient

We concentrate on the choices that are directly related with optimization. In particular, we will explore some _automatic_ methods for setting the learning rate, in a way that fixes the poor-conditioning problem and is robust across different problems.

```python
# @title Video 7: Adaptive Methods
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1eq4y1W7JG", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"Zr6r2kfmQUM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')
```

## Coding Exercise 7: Implement RMSprop

In this exercise you will implement the update of the RMSprop optimizer:

\begin{align}
v_{t} &= \alpha v_{t-1} + (1 - \alpha) \nabla J(w_t)^2 \\ \\
w_{t+1} &= w_t - \eta \frac{\nabla J(w_t)}{\sqrt{v_t + \epsilon}}
\end{align}

where the non-standard operations (division of two vectors, squaring a vector, etc) are to be interpreted as element-wise operations, i.e., the operation is applied to each (pair of) entry[ies] of the vector(s) considered as real number(s).

Here, the $\epsilon$ hyperparameter provides numerical estability to the algorithm, by preventing the learning rate to become too big when $v_t$ is small. Typically, we set $\epsilon$ to a default small value, like $10^{-8}$.

```python
def rmsprop_update(loss, params, grad_sq, lr=1e-3, alpha=0.8, epsilon=1e-8):
  """Perform an RMSprop update on a collection of parameters

  Args:
    loss (tensor): A scalar tensor containing the loss whose gradient will be computed
    params (iterable): Collection of parameters with respect to which we compute gradients
    grad_sq (iterable): Moving average of squared gradients
    lr (float): Scalar specifying the learning rate or step-size for the update
    alpha (float): Moving average parameter
    epsilon (float): for numerical estability
  """
  # Clear up gradients as Pytorch automatically accumulates gradients from
  # successive backward calls
  zero_grad(params)
  # Compute gradients on given objective
  loss.backward()

  with torch.no_grad():
    for (par, gsq) in zip(params, grad_sq):
      #################################################
      ## TODO for students: update the value of the parameter ##
      # Use gsq.data and par.grad
      raise NotImplementedError("Student exercise: implement gradient update")
      #################################################
      # Update estimate of gradient variance
      gsq.data = ...
      # Update parameters
      par.data -=  ...

set_seed(seed=SEED)
model3 = MLP(in_dim=784, out_dim=10, hidden_dims=[])
print('\n The model3 parameters before the update are: \n')
print_params(model3)
loss = loss_fn(model3(X), y)
# Intialize the moving average of squared gradients
grad_sq = [1e-6*i for i in list(model3.parameters())]

## Uncomment below to test your function
# rmsprop_update(loss, list(model3.parameters()), grad_sq=grad_sq, lr=1e-3)
# print('\n The model3 parameters after the update are: \n')
# print_params(model3)
```

    Random seed 2021 has been set.

     The model3 parameters before the update are:

    main.0.weight tensor([[-0.0264,  0.0010,  0.0173,  ...,  0.0297,  0.0278, -0.0221],
            [-0.0040, -0.0295, -0.0086,  ..., -0.0070,  0.0254, -0.0233],
            [ 0.0240, -0.0231,  0.0342,  ...,  0.0124,  0.0270, -0.0180],
            ...,
            [-0.0005,  0.0157,  0.0111,  ...,  0.0144, -0.0301, -0.0144],
            [ 0.0181,  0.0303,  0.0255,  ..., -0.0110, -0.0175,  0.0205],
            [ 0.0208, -0.0353, -0.0183,  ..., -0.0271,  0.0099,  0.0003]])
    main.0.bias tensor([-0.0290, -0.0033,  0.0100, -0.0320,  0.0022,  0.0221,  0.0307,  0.0243,
             0.0159, -0.0064])

```
 The model3 parameters after the update are:

main.0.weight tensor([[-0.0240,  0.0031,  0.0193,  ...,  0.0316,  0.0297, -0.0198],
        [-0.0063, -0.0318, -0.0109,  ..., -0.0093,  0.0232, -0.0255],
        [ 0.0218, -0.0253,  0.0320,  ...,  0.0102,  0.0248, -0.0203],
        ...,
        [-0.0027,  0.0136,  0.0089,  ...,  0.0123, -0.0324, -0.0166],
        [ 0.0159,  0.0281,  0.0233,  ..., -0.0133, -0.0197,  0.0182],
        [ 0.0186, -0.0376, -0.0205,  ..., -0.0293,  0.0077, -0.0019]])
main.0.bias tensor([-0.0313, -0.0011,  0.0122, -0.0342,  0.0045,  0.0199,  0.0329,  0.0265,
         0.0182, -0.0041])
```

## Interactive Demo 7: Compare optimizers

Below, we compare your implementations of SGD, momentum and RMSprop. If you have successfully coded all the exercises so far: congrats! You are now *in the know* of some of the most commonly used and powerful tools of optimization for deep learning.

```python
# @markdown Execute this cell to enable the widget!
X, y = train_set.data, train_set.targets

@widgets.interact_manual
def compare_optimizers(
    batch_size=(25, 250, 5),
    lr=widgets.FloatLogSlider(value=2e-3, min=-5, max=0),
    max_steps=(50, 500, 5)):

  SGD_DICT = [gradient_update, 'SGD', 'black', '-', {'lr': lr}]
  MOM_DICT = [momentum_update, 'Momentum', 'red', '--', {'lr': lr, 'beta': 0.9}]
  RMS_DICT = [rmsprop_update, 'RMSprop', 'fuchsia', '-', {'lr': lr, 'alpha': 0.8}]

  ALL_DICTS = [SGD_DICT, MOM_DICT, RMS_DICT]

  base_model = MLP(in_dim=784, out_dim=10, hidden_dims=[100, 100])

  LOSS_HIST = {}

  for opt_dict in tqdm(ALL_DICTS):
    update_fn, opt_name, color, lstyle, kwargs = opt_dict
    LOSS_HIST[opt_name] = []

    model = copy.deepcopy(base_model)
    params = list(model.parameters())

    if opt_name != 'SGD':
      aux_tensors = [torch.zeros_like(_) for _ in params]

    for step in range(max_steps):
      data, labels = sample_minibatch(X, y, batch_size)
      loss = loss_fn(model(data), labels)
      if opt_name == 'SGD':
        update_fn(loss, params, **kwargs)
      else:
        update_fn(loss, params, aux_tensors, **kwargs)
      LOSS_HIST[opt_name].append(loss.item())

  fig, axs = plt.subplots(1, len(ALL_DICTS), figsize=(9, 3))
  for ax, optim_dict in zip(axs, ALL_DICTS):
    opt_name = optim_dict[1]
    ax.plot(range(max_steps), LOSS_HIST[opt_name], alpha=0.8)
    ax.set_title(opt_name)
    ax.set_xlabel('Iteration')
    ax.set_ylabel('Loss')
    ax.set_ylim(0, 2.5)
  plt.show()
```

    interactive(children=(IntSlider(value=135, description='batch_size', max=250, min=25, step=5), FloatLogSlider(…

### **Discussion**

Tune the 3 methods above in order to make each individually excel and discuss your findings. How do the methods compare in terms of robustness to small changes of the hyperparameters? How easy was it to find a good hyperparameter configuration?

**Remarks:** Note that RMSprop is allowing us to use a 'per-dimension' learning rate _without having to tune one learning rate for each dimension **ourselves**_. The method uses information collected about the variance of the gradients throughout training to **adapt** the step size for each of the parameters automatically. The savings in tuning efforts of RMSprop over SGD or 'plain' momentum are undisputed on this task.

Moreover, adaptive optimization methods are currently a highly active research domain, with many related algorithms like Adam, AMSgrad, Adagrad being used in practical application and theoretically investigated.

### Locality of Gradients
As we've seen throught this tutorial, poor conditioning can be a significant burden on convergence to an optimum while using gradient based optimization. Of the methods we've seen to deal with this issue, notice how both momentum and adaptive learning rates incorperate past gradient values into their update schemes. Why do we use past values of our loss function's gradient while updating our current MLP weights?

Recall from W1D2 that the gradient of a function, $\nabla f(w_t)$, is a **local** property and computes the direction of maximum change of $f(w_t)$ at the point $w_t$. However, when we train our MLP model we are hoping to find the __global__ optimum for our training loss. By incorperating past values of our function's gradient into our optimization schemes, we use more information about the overall shape of our function than just a single gradient alone can provide.

## Think! 7: Loss function and optimization

Can you think of other ways we can incorperate more information about our loss function into our optimization schemes?

---
# Summary

* Optimization is necessary to create Deep Learning models that are guaranteed to converge
* Stochastic Gradient Descent and Momentum are two commonly used optimization techniques
* RMSProp is a way of adaptive hyper parameter tuning which utilises a per-dimension learning rate
* Poor choice of optimization objectives can lead to unforeseen, undesirable consequences

If you have time left, you can read the Bonus material, were we put all together and we compare our model with a benchmark model.

---
# Bonus: Putting it all together

*Time estimate: ~40 mins*

We have progressively built a sophisticated optimization algorithm which is able to deal with a non-convex, poor-conditioned problem concerning tens of thousands of training examples. Now we present _you_ with a small challenge: beat us! :P

Your mission is to train an MLP model that can compete with a benchmark model which we have pre-trained for you. In this section you will be able to use the full Pytorch power: loading the data, defining the model, sampling minibatches as well as Pytorch's **optimizer implementations**.

There is a big engineering component behind the design of optimizers and their implementation can sometimes become tricky. So unless you are directly doing research in optimization, it's recommended to use an implementation provided by a widely reviewed open-source library.

```python
# @title Video 9: Putting it all together
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1MK4y1u7u2", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"DP9c13vLiOM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')
```

```python
# @title Download parameters of the benchmark model
import requests

fname = 'benchmark_model.pt'
url = "https://osf.io/sj4e8/download"
r = requests.get(url, allow_redirects=True)
with open(fname, 'wb') as fh:
  fh.write(r.content)

# Load the benchmark model's parameters
DEVICE = set_device()
if DEVICE == "cuda":
  benchmark_state_dict = torch.load(fname)
else:
  benchmark_state_dict = torch.load(fname, map_location=torch.device('cpu'))
```

    GPU is enabled in this notebook.

```python
# Create MLP object and update weights with those of saved model
benchmark_model = MLP(in_dim=784, out_dim=10,
                      hidden_dims=[200, 100, 50]).to(DEVICE)
benchmark_model.load_state_dict(benchmark_state_dict)

# Define helper function to evaluate models
def eval_model(model, data_loader, num_batches=np.inf, device='cpu'):

  loss_log, acc_log = [], []
  model.to(device=device)

  # We are just evaluating the model, no need to compute gradients
  with torch.no_grad():
    for batch_id, batch in enumerate(data_loader):
      # If we only evaluate a number of batches, stop after we reach that number
      if batch_id > num_batches:
        break
      # Extract minibatch data
      data, labels = batch[0].to(device), batch[1].to(device)
      # Evaluate model and loss on minibatch
      preds = model(data)
      loss_log.append(loss_fn(preds, labels).item())
      acc_log.append(torch.mean(1. * (preds.argmax(dim=1) == labels)).item())

  return np.mean(loss_log), np.mean(acc_log)
```

We define an optimizer in the following steps:

1. Load  the corresponding class that implements the parameter updates and other internal management activities, including:
    - create auxiliary variables,
    - update moving averages,
    - adjust learning rate.
2. Pass the parameters of the Pytorch model that the optimizer has control over. Note that different parameter groups can potentially be controlled by different optimizers.
3. Specify hyperparameters, including learning rate, momentum, moving average factors, etc.

## Exercise Bonus: Train your own model

Now, train the model with your preferred optimizer and find a good combination of hyperparameter settings.

```python
#################################################
## TODO for students: adjust training settings ##

# The three parameters below are in your full control
MAX_EPOCHS = 2  # select number of epochs to train
LR = 1e-5  # choose the step size
BATCH_SIZE = 64  # number of examples per minibatch

# Define the model and associated optimizer -- you may change its architecture!
model = MLP(in_dim=784, out_dim=10, hidden_dims=[200, 100, 50]).to(DEVICE)

# You can take your pick from many different optimizers
# Check the optimizer documentation and hyperparameter meaning before using!
# More details on Pytorch optimizers: https://pytorch.org/docs/stable/optim.html
# optimizer = torch.optim.SGD(model.parameters(), lr=LR, momentum=0.9)
# optimizer = torch.optim.RMSprop(model.parameters(), lr=LR, alpha=0.99)
# optimizer = torch.optim.Adagrad(model.parameters(), lr=LR)
optimizer = torch.optim.Adam(model.parameters(), lr=LR)
#################################################
```

```python
set_seed(seed=SEED)
# Print trainig stats every LOG_FREQ minibatches
LOG_FREQ = 200
# Frequency for evaluating the validation metrics
VAL_FREQ = 200
# Load data using a Pytorch Dataset
train_set_orig, test_set_orig = load_mnist_data(change_tensors=False)

# We separate 10,000 training samples to create a validation set
train_set_orig, val_set_orig = torch.utils.data.random_split(train_set_orig, [50000, 10000])

# Create the corresponding DataLoaders for training and test
g_seed = torch.Generator()
g_seed.manual_seed(SEED)

train_loader = torch.utils.data.DataLoader(train_set_orig,
                                           shuffle=True,
                                           batch_size=BATCH_SIZE,
                                           num_workers=2,
                                           worker_init_fn=seed_worker,
                                           generator=g_seed)
val_loader = torch.utils.data.DataLoader(val_set_orig,
                                         shuffle=True,
                                         batch_size=256,
                                         num_workers=2,
                                         worker_init_fn=seed_worker,
                                         generator=g_seed)
test_loader = torch.utils.data.DataLoader(test_set_orig,
                                          batch_size=256,
                                          num_workers=2,
                                          worker_init_fn=seed_worker,
                                          generator=g_seed)

# Run training
metrics = {'train_loss':[],
           'train_acc':[],
           'val_loss':[],
           'val_acc':[],
           'val_idx':[]}

step_idx = 0
for epoch in tqdm(range(MAX_EPOCHS)):

  running_loss, running_acc = 0., 0.

  for batch_id, batch in enumerate(train_loader):
    step_idx += 1
    # Extract minibatch data and labels
    data, labels = batch[0].to(DEVICE), batch[1].to(DEVICE)
    # Just like before, refresh gradient accumulators.
    # Note that this is now a method of the optimizer.
    optimizer.zero_grad()
    # Evaluate model and loss on minibatch
    preds = model(data)
    loss = loss_fn(preds, labels)
    acc = torch.mean(1.0 * (preds.argmax(dim=1) == labels))
    # Compute gradients
    loss.backward()
    # Update parameters
    # Note how all the magic in the update of the parameters is encapsulated by
    # the optimizer class.
    optimizer.step()
    # Log metrics for plotting
    metrics['train_loss'].append(loss.cpu().item())
    metrics['train_acc'].append(acc.cpu().item())

    if batch_id % VAL_FREQ == (VAL_FREQ - 1):
      # Get an estimate of the validation accuracy with 100 batches
      val_loss, val_acc = eval_model(model, val_loader,
                                     num_batches=100,
                                     device=DEVICE)
      metrics['val_idx'].append(step_idx)
      metrics['val_loss'].append(val_loss)
      metrics['val_acc'].append(val_acc)

      print(f"[VALID] Epoch {epoch + 1} - Batch {batch_id + 1} - "
            f"Loss: {val_loss:.3f} - Acc: {100*val_acc:.3f}%")

    # print statistics
    running_loss += loss.cpu().item()
    running_acc += acc.cpu().item()
    # Print every LOG_FREQ minibatches
    if batch_id % LOG_FREQ == (LOG_FREQ-1):
      print(f"[TRAIN] Epoch {epoch + 1} - Batch {batch_id + 1} - "
            f"Loss: {running_loss / LOG_FREQ:.3f} - "
            f"Acc: {100 * running_acc / LOG_FREQ:.3f}%")

      running_loss, running_acc = 0., 0.
```

    Random seed 2021 has been set.

      0%|          | 0/2 [00:00<?, ?it/s]

    [VALID] Epoch 1 - Batch 200 - Loss: 2.235 - Acc: 38.174%
    [TRAIN] Epoch 1 - Batch 200 - Loss: 2.274 - Acc: 30.945%
    [VALID] Epoch 1 - Batch 400 - Loss: 2.067 - Acc: 52.773%
    [TRAIN] Epoch 1 - Batch 400 - Loss: 2.166 - Acc: 44.289%
    [VALID] Epoch 1 - Batch 600 - Loss: 1.789 - Acc: 56.826%
    [TRAIN] Epoch 1 - Batch 600 - Loss: 1.935 - Acc: 55.352%
    [VALID] Epoch 2 - Batch 200 - Loss: 1.265 - Acc: 69.561%
    [TRAIN] Epoch 2 - Batch 200 - Loss: 1.373 - Acc: 66.117%
    [VALID] Epoch 2 - Batch 400 - Loss: 1.072 - Acc: 77.607%
    [TRAIN] Epoch 2 - Batch 400 - Loss: 1.168 - Acc: 74.141%
    [VALID] Epoch 2 - Batch 600 - Loss: 0.928 - Acc: 80.967%
    [TRAIN] Epoch 2 - Batch 600 - Loss: 0.997 - Acc: 79.094%

```python
fig, ax = plt.subplots(1, 2, figsize=(10, 4))

ax[0].plot(range(len(metrics['train_loss'])), metrics['train_loss'],
           alpha=0.8, label='Train')
ax[0].plot(metrics['val_idx'], metrics['val_loss'], label='Valid')
ax[0].set_xlabel('Iteration')
ax[0].set_ylabel('Loss')
ax[0].legend()

ax[1].plot(range(len(metrics['train_acc'])), metrics['train_acc'],
           alpha=0.8, label='Train')
ax[1].plot(metrics['val_idx'], metrics['val_acc'], label='Valid')
ax[1].set_xlabel('Iteration')
ax[1].set_ylabel('Accuracy')
ax[1].legend()
plt.tight_layout()
plt.show()
```

## Think! Bonus: Metrics

Which metric did you optimize when searching for the right configuration? The training set loss? Accuracy? Validation/test set metrics? Why? Discuss!

### Evaluation

We _finally_ can evaluate and compare the performance of the models on previously unseen examples.

Which model would you keep? (\*drum roll*)

```python
print('Your model...')
train_loss, train_accuracy = eval_model(my_model, train_loader, device=DEVICE)
test_loss, test_accuracy = eval_model(my_model, test_loader, device=DEVICE)
print(f'Train Loss {train_loss:.3f} / Test Loss {test_loss:.3f}')
print(f'Train Accuracy {100*train_accuracy:.3f}% / Test Accuracy {100*test_accuracy:.3f}%')

print('\nBenchmark model')
train_loss, train_accuracy = eval_model(benchmark_model, train_loader, device=DEVICE)
test_loss, test_accuracy = eval_model(benchmark_model, test_loader, device=DEVICE)
print(f'Train Loss {train_loss:.3f} / Test Loss {test_loss:.3f}')
print(f'Train Accuracy {100*train_accuracy:.3f}% / Test Accuracy {100*test_accuracy:.3f}%')
```

    Your model...
    Train Loss 2.470 / Test Loss 2.475
    Train Accuracy 10.340% / Test Accuracy 10.996%

    Benchmark model
    Train Loss 0.011 / Test Loss 0.025
    Train Accuracy 99.784% / Test Accuracy 99.316%


# Offshore Wind Farming

## Objective and Prerequisites

In this example, we will solve the problem of how to minimize the cost of laying underwater cables to collect electricity produced by an offshore wind farm. We will construct a mixed-integer programming (MIP)  model of this problem, implement this model in the Gurobi Python interface, and compute an optimal solution.

This modeling example is at the beginner level, where we assume that you know Python and that you have some knowledge about building mathematical optimization models.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

**Gurobi License**

In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EGU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_0ffshore-wind-farming_COM_EVAL_GITHUB_&utm_term=wind-power-generation&utm_content=C_JPM) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EGU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_0ffshore-wind-farming_ACADEMIC_EVAL_GITHUB_&utm_term=wind-power-generation&utm_content=C_JPM) as an *academic user*.

## Motivation

Global climate change has already had observable effects on the environment. Glaciers have shrunk, ice on rivers and lakes is breaking up earlier than expected, plant and animal species have  been affected and trees are flowering sooner than expected. The potential future effects of global climate change include more frequent wildfires, longer periods of drought in some regions and an increase in the number, duration and intensity of tropical storms. [1]

Climate change mitigation consists of actions to limit the magnitude or rate of global warming and its related effects.
The first challenge for climate change mitigation is eliminating the burning of coal, oil and, eventually, natural gas. This is perhaps the most daunting challenge as denizens of richer nations literally eat, wear, work, play and even sleep on the products made from fossil fuels. Also, citizens of developing nations want and arguably deserve the same comforts. There are no perfect solutions for reducing dependence on fossil fuels (for example, carbon neutral biofuels can drive up the price of food and lead to forest destruction, and while nuclear power does not emit greenhouse gases, it does produce radioactive waste). Other alternatives include plant-derived plastics, biodiesel, and wind power. [2]

Offshore wind power is the use of wind farms constructed in bodies of water, usually in the ocean, to harvest wind energy to generate electricity. Higher wind speeds are available offshore compared to on land, so offshore wind power’s electricity generation is higher per amount of capacity installed.

The advantage of locating wind turbines offshore is that the wind is much stronger off the coasts, and unlike wind over the continent, offshore breezes can be strong in the afternoon, matching the time when people are using the most electricity. Offshore turbines can also be located close to the load centers along the coasts, such as large cities, eliminating the need for new long-distance transmission lines.

## Problem Description

An offshore wind farm is a collection of wind turbines placed at sea to take advantage of the strong offshore winds. These strong winds produce more electricity, but offshore wind farms are more expensive to install and operate than those on land.

We will use a MIP model to reduce part of the cost of building an offshore wind farm. We will compute a plan for how to lay the underwater cables that connect the turbines. These cables are necessary to transfer the power produced by the turbines to land. The plan we compute will minimize the cost to install the underwater cables, while ensuring that each turbine is connected to the shore and each cable has sufficient capacity to handle the electrical current generated.

In our example, a wind farm is being built off the west coast of Denmark. There is a power station on the coast where all the electricity must be transferred to be distributed to the electric grid. There are also transfer stations in the wind farm where the power from several turbines can be collected and transferred along a single cable to the shore.

There are two factors we must consider when installing the cables. First, there is a fixed cost to lay a cable on the sea floor. This cost is proportional to the distance between the two stations the cable connects. Second, we must consider how much current will flow through the cables. Connections that carry large currents need thick cables. Thick cables are more expensive than thin cables.

The goal of this optimization problem is to decide which cables should be laid to connect the wind farm power network at a minimum cost.

The model of offshore wind farming optimization problem is an instance of a more general optimization model known as fixed charge network flow problem. Fixed charge network flow problems can be applied to a large number of business problems -for example, in the planning of communication and transport networks.

## Solution Approach

Mathematical programming is a declarative approach where the modeler formulates a mathematical optimization model that captures the key aspects of a complex decision problem. The Gurobi Optimizer solves such models using state-of-the-art mathematics and computer science.

A mathematical optimization model has five components, namely:

* Sets and indices.
* Parameters.
* Decision variables.
* Objective function(s).
* Constraints.

We now present a MIP formulation for the offshore wind farming problem.

## Model Formulation

### Sets and Indices

$G(V,E)$: Graph that represents the wind farm network, where $V$ is the set of vertices and $E$ is the set of edges. The turbines, transfer stations, and power stations are vertices in the set of vertices $V$ of the graph. The set of potential cables are the edges of the graph.

### Parameters

$s_{i} \in \mathbb{R}$: Power supplied at vertex $i \in V$. Since turbines supply power,  they are source vertices with $s_{i} > 0$. Transfer stations do not supply or remove power from the network so they have $s_{i} = 0$. The power station on the coast is a sink that remove all power from the wind farm so it has $s_{i} < 0$.

$u_{i,j} \in \mathbb{R}^+ $: Maximum current capacity a cable can handle from vertex $i \in V$ to vertex $j \in V$.

$c_{i,j} \in \mathbb{R}^+$: Cost per unit of current flow going from vertex $i \in V$ to vertex $j \in V$, i.e. the price we must pay to increase the thickness of the cable to handle an increase in current.

$f_{i,j} \in \mathbb{R}^+$: Fixed cost of laying a cable from vertex $i \in V$ to vertex $j \in V$, and is the result of multiplying the distance between vertices by the cost per mile.

### Decision Variables

$install_{i,j} \in \{0, 1 \}$: This variable is equal to 1 if we lay a cable from vertex $i \in V$ to vertex $j \in V$; and 0 otherwise.

$flow_{i,j} \geq 0$: This non-negative continuous variable represents the amount of current flowing from vertex $i \in V$ to vertex $j \in V$.

The goal of this optimization model is to decide which of these potential edges in the graph should be used at a minimum cost.

### Objective Function

- **Total costs**. We want to minimize the total cost to install the cables. The term on the left is the variable costs (i.e. those that vary according to the current in the cable). The term on right is the fixed cost to install the cable.

\begin{equation}
\text{Max} \quad Z = \sum_{(i,j) \in E} c_{i,j} \cdot flow_{i,j} + \sum_{(i,j) \in E} f_{i,j} \cdot install_{i,j}
\tag{0}
\end{equation}

### Constraints

- **Balance**. For each vertex $i \in V$, we want to ensure the conservation of current in the network.

\begin{equation}
\sum_{j:(i,j) \in E} flow_{i,j} - \sum_{j:(j,i) \in E} flow_{j,i} = s_{i} \quad \forall i \in V
\tag{1}
\end{equation}

- **Capacity**. For each edge $(i,j) \in E$, we want to enforce the limits on the maximum current capacity of each cable.

\begin{equation}
0 \leq flow_{i,j} \leq u_{i,j} \cdot install_{i,j}  \quad \forall (i,j) \in E
\tag{2}
\end{equation}

## Python Implementation

This example considers three turbines, one transfer station, and two power stations. The current flowing out at each vertex of the wind farm network is presented in the following table. Recall that since turbines supply power their capacity is positive. Transfer stations do not supply or remove power from the network so they have a capacity of zero.  The power stations on the coast are sinks that remove all power from the wind farm network so they have demand of power, in this case we use a negative number.

|  | Capacity in MW |
| --- | --- |
| vertex 1 | 4 |
| vertex 2 | 3 |
| vertex 3 | 2 |
| vertex 4 | 0 |
| vertex 5 | -6 |
| vertex 6 | -3 |

The capacity, flow cost, and fixed cost of each edge in the wind farm network are provided in the following table.

|  | Capacity in MW  | Flow cost in millions of Euros | Fixed cost in millions of Euros|
| --- | --- | --- | --- |
| Edge: (0,4) | 4 | 1 | 1 |
| Edge: (0,3) | 2 | 1 | 1 |
| Edge: (1,3) | 3 | 1 | 1 |
| Edge: (2,5) | 2 | 1 | 1 |
| Edge: (3,4) | 2 | 1 | 1 |
| Edge: (3,5) | 1 | 1 | 1 |

We now import the Gurobi Python Module. Then, we initialize the data structures with the given data.

```python
import gurobipy as gp
from gurobipy import GRB

# Parameters

vertices = {0: 4, 1: 3, 2: 2, 3: 0, 4: -6, 5: -3}
edges, cap, flow_cost, fixed_cost = gp.multidict({
    (0,4): [4,1,1],
    (0,3): [2,1,1],
    (1,3): [3,1,1],
    (2,5): [2,1,1],
    (3,4): [2,1,1],
    (3,5): [1,1,1]
})
```

### Model Deployment

We now determine the MIP model for the offshore wind farming problem, by defining the decision variables, constraints, and objective function. Next, we start the optimization process and Gurobi finds the plan to lay cables at the offshore wind farming network that minimizes total costs.

```python
# MIP  model formulation

m = gp.Model("offshore_wind_farming")

# Add variables
install = m.addVars(edges, vtype=GRB.BINARY, name="Install")
flow = m.addVars(edges, vtype=GRB.CONTINUOUS, name="Flow")

# Add constraints
m.addConstrs((flow.sum(v,'*') - flow.sum('*',v) == supply for v, supply in vertices.items()), name="Flow_conservation")
m.addConstrs((flow[e] <= cap[e]*install[e] for e in edges), name="Install2flow")

# Set objective
m.setObjective(flow.prod(flow_cost) + install.prod(fixed_cost), GRB.MINIMIZE)

m.optimize()
```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-
    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 12 rows, 12 columns and 24 nonzeros
    Model fingerprint: 0x03b3989f
    Variable types: 6 continuous, 6 integer (6 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 4e+00]
      Objective range  [1e+00, 1e+00]
      Bounds range     [1e+00, 1e+00]
      RHS range        [2e+00, 6e+00]
    Presolve removed 12 rows and 12 columns
    Presolve time: 0.00s
    Presolve: All rows and columns removed

    Explored 0 nodes (0 simplex iterations) in 0.01 seconds
    Thread count was 1 (of 8 available processors)

    Solution count 1: 17

    Optimal solution found (tolerance 1.00e-04)
    Best objective 1.700000000000e+01, best bound 1.700000000000e+01, gap 0.0000%

## Analysis

The result of the optimization model shows that the minimum total cost value is 17 million Euros. Let's see the solution that achieves that optimal result.

### Cable Installation Plan

This plan determines the layout of cables in the offshore wind farming network.

```python
# display which edges in the offshore wind farming network we plan to install.

for origin, end in install.keys():
    if (abs(install[origin, end].x) > 0.5):
        print(f"Install cable from location {origin + 1} to location {end + 1} in the offshore wind farming network ")

```

    Install cable from location 1 to location 5 in the offshore wind farming network
    Install cable from location 2 to location 4 in the offshore wind farming network
    Install cable from location 3 to location 6 in the offshore wind farming network
    Install cable from location 4 to location 5 in the offshore wind farming network
    Install cable from location 4 to location 6 in the offshore wind farming network

### Cable Capacity Plan

This plan determines the current flow capacity in MW of each cable installed.

```python
# Current flow capacity of each cable installed

for origin, end in flow.keys():
    if (abs(flow[origin, end].x) > 1e-6):
         print(f"The capacity of cable installed from location {origin + 1} to location {end + 1} is {flow[origin, end].x} MW ")
```

    The capacity of cable installed from location 1 to location 5 is 4.0 MW
    The capacity of cable installed from location 2 to location 4 is 3.0 MW
    The capacity of cable installed from location 3 to location 6 is 2.0 MW
    The capacity of cable installed from location 4 to location 5 is 2.0 MW
    The capacity of cable installed from location 4 to location 6 is 1.0 MW

##  Conclusion

In this example, we addressed an offshore wind farming problem where we want to minimize the cost of laying underwater cables to collect electricity produced by an offshore wind farm network. We learned how to formulate the problem as a MIP model. Also, we learned how to implement the MIP model formulation and solve it using the Gurobi Python API.

## References

[1] https://climate.nasa.gov/effects/

[2] https://www.scientificamerican.com/article/10-solutions-for-climate-change/

Copyright © 2020 Gurobi Optimization, LLC


Table of Contents

1  PageRank

1.1  Taxation

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', 'notebook_format'))

from formats import load_style
load_style(plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import numpy as np
import networkx as nx
import matplotlib.pyplot as plt

%watermark -a 'Ethen' -d -t -v -p numpy,networkx,matplotlib
```

    Ethen 2019-03-02 10:24:46

    CPython 3.6.4
    IPython 6.4.0

    numpy 1.14.2
    networkx 2.2
    matplotlib 2.2.3

# PageRank

PageRank is a function that assigns a number weighting each page in the Web, the intent is that the higher the PageRank of a page, the more important the page is. We can think of the Web as a directed graph, where the pages are the nodes and if there exists a link that connects page1 to page2 then there would be an edge connecting the two nodes.

Imagine an toy example where there are only 4 pages/nodes, ${A, B, C, D}$:

- $A$ has links connecting itself to each of ther other three pages.
- $B$ has links to $A$ and $D$.
- $D$ has links to $B$ and $C$.
- $C$ has links only to $A$.

```python
nodes = ['A', 'B', 'C', 'D']
edges = [
    ('A', 'B'),
    ('A', 'C'),
    ('A', 'D'),
    ('B', 'A'),
    ('B', 'D'),
    ('D', 'B'),
    ('D', 'C'),
    ('C', 'A')
]

graph = nx.DiGraph()
graph.add_nodes_from(nodes)
graph.add_edges_from(edges)
graph
```

    <networkx.classes.digraph.DiGraph at 0x10e7b0208>

```python
# change default style figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

# quick and dirty visualization of the graph we've defined above
nx.draw(graph, with_labels=True, node_color='skyblue', alpha=0.7)
```

Given this graph, we can build a transition matrix to depict what is the probability of landing on a given page after 1 step. If we look at an example below, the matrix has:

- $n$ rows and columns if there are $n$ pages
- Each element in the matrix, $m_{ij}$ takes on the value of $1 / k$ if page $i$ has $k$ edges and one of them is $j$. Otherwise $m_{ij}$ is 0.

```python
trans_matrix = nx.to_numpy_array(graph)
trans_matrix /= trans_matrix.sum(axis=1, keepdims=True)
trans_matrix
```

    array([[0.        , 0.33333333, 0.33333333, 0.33333333],
           [0.5       , 0.        , 0.        , 0.5       ],
           [1.        , 0.        , 0.        , 0.        ],
           [0.        , 0.5       , 0.5       , 0.        ]])

Now suppose we start at any of the $n$ pages of the Web with equal probability. Then the initial vector $v_0$ will have $1/n$ for each page. If $M$ is the transition matrix of the Web, then after one step, the distribution
of us landing on each of the page can be computed by a matrix vector multiplication. $v_0 M$

```python
n_nodes = trans_matrix.shape[0]
init_vector = np.repeat(1 / n_nodes, n_nodes)
init_vector @ trans_matrix
```

    array([0.375     , 0.20833333, 0.20833333, 0.20833333])

As we can see after 1 step, the probability of landing on the first page, page $A$, is higher than the probability of landing on other pages. We can repeat this matrix vector multiplication for multiple times and our results will eventually converge. Giving us an estimated probability of landing on each page, which in term is PageRank's estimate of how important a given page is when compared to all the other page in the Web.

```python
# we can tweak the number of iterations parameter
# and see that the resulting probability remains
# the same even if we increased the number to 50
n_iters = 30
result = init_vector
for _ in range(n_iters):
    result = result @ trans_matrix

result
```

    array([0.33333333, 0.22222222, 0.22222222, 0.22222222])

This sort of convergence behavior is an example of the Markov Chain processes. It is known that the distribution of $v = Mv$ converges, provided two conditions are met:

- The graph is strongly connected; that is, it is possible to get from any
node to any other node.
- There are no dead ends: nodes that have no edges out.

If we stare at the formula $v = Mv$ long enough, we can observe that our final result vector $v$ is an eigenvector of the matrix $M$ (recall an eigenvector of a matrix $M$ is a vector $v$ that satisfies $v = \lambda Mv$ for some constant eigenvalue $\lambda$).

## Taxation

The vanilla PageRank that we've introduced above needs some tweaks to handle data that can appear in real world scenarios. The two problems that we need to avoid is what's called **spider traps** and **dead end**.

**spider trap** is a set of nodes with edges, but these edges all links within the page itself. This causes the PageRank calculation to place all the PageRank score within the spider traps.

```python
# we replaced C's out-link, ('C', 'A'),
# from the list of edges with a link within
# the page itself ('C', 'C'), note that
# we can also avoid this problem by note
# including self-loops in the edges
nodes = ['A', 'B', 'C', 'D']
edges = [
    ('A', 'B'),
    ('A', 'C'),
    ('A', 'D'),
    ('B', 'A'),
    ('B', 'D'),
    ('D', 'B'),
    ('D', 'C'),
    ('C', 'C')
]

graph = nx.DiGraph()
graph.add_nodes_from(nodes)
graph.add_edges_from(edges)

# not showing the self-loop edge for node C
nx.draw(graph, with_labels=True, node_color='skyblue', alpha=0.7)
```

```python
# notice in the transition probability matrix, the third row, node C
# contains 1 for a single entry
trans_matrix = nx.to_numpy_array(graph)
trans_matrix /= trans_matrix.sum(axis=1, keepdims=True)
trans_matrix
```

    array([[0.        , 0.33333333, 0.33333333, 0.33333333],
           [0.5       , 0.        , 0.        , 0.5       ],
           [0.        , 0.        , 1.        , 0.        ],
           [0.        , 0.5       , 0.5       , 0.        ]])

```python
n_iters = 40
result = init_vector
for _ in range(n_iters):
    result = result @ trans_matrix

result
```

    array([6.04874159e-07, 8.81559995e-07, 9.99997632e-01, 8.81559995e-07])

As predicted, all the PageRank is at node $C$, since once we land there, there's no way for us to leave.

The other problem **dead end** describes pages that have no out-links, as a result pages that reaches these dead ends will not have any PageRank.

```python
# we remove C's out-link, ('C', 'A'),
# from the list of edges
nodes = ['A', 'B', 'C', 'D']
edges = [
    ('A', 'B'),
    ('A', 'C'),
    ('A', 'D'),
    ('B', 'A'),
    ('B', 'D'),
    ('D', 'B'),
    ('D', 'C')
]

graph = nx.DiGraph()
graph.add_nodes_from(nodes)
graph.add_edges_from(edges)
nx.draw(graph, with_labels=True, node_color='skyblue', alpha=0.7)
```

```python
# trick for numpy for dealing with zero division
# https://stackoverflow.com/questions/26248654/numpy-return-0-with-divide-by-zero
trans_matrix = nx.to_numpy_array(graph)
summed = trans_matrix.sum(axis=1, keepdims=True)
trans_matrix = np.divide(trans_matrix, summed,
                         out=np.zeros_like(trans_matrix), where=summed!=0)

# notice in the transition probability matrix, the third row, node C
# consists of all 0
trans_matrix
```

    array([[0.        , 0.33333333, 0.33333333, 0.33333333],
           [0.5       , 0.        , 0.        , 0.5       ],
           [0.        , 0.        , 0.        , 0.        ],
           [0.        , 0.5       , 0.5       , 0.        ]])

```python
n_iters = 40
result = init_vector
for _ in range(n_iters):
    result = result @ trans_matrix

result
```

    array([6.04874159e-07, 8.81559995e-07, 8.81559995e-07, 8.81559995e-07])

As we see, the result tells us the probability of us being anywhere goes to 0, as the number of steps increase.

To avoid the two problems mentioned above, we will modify the calculation of PageRank. At each step, we will give it a small probability of "teleporting" to a random page, rather than following an out-link from their current page. The notation form for the description above would be:

\begin{align}
v^\prime = \beta M v + (1 - \beta) e / n
\end{align}

where:

- $\beta$ is a chosen constant, usually in the range of 0.8 to 0.9.
- $e$ is a vector of all 1s with the appropriate number of elements so that the matrix addition adds up.
- $n$ is the number of pages/nodes in the Web graph.
- The term, $\beta M v$ denotes that at this step, there is a probability $\beta$ that we will follow an out-link from their present page.
- Notice the term $(1 - \beta) e / n$ does not depend on $v$, thus if there are some dead ends in the graph, there will always be some fraction of opportunity to jump out of that rabbit hole.
- This idea of adding $\beta$ is referred to as taxation (`networkx` package calls this damping factor).

```python
def build_trans_matrix(graph: nx.DiGraph, beta: float=0.9) -> np.ndarray:
    n_nodes = len(graph)
    trans_matrix = nx.to_numpy_array(graph)

    # assign uniform probability to dangling nodes (nodes without out links)
    const_vector = np.repeat(1.0 / n_nodes, n_nodes)
    row_sum = trans_matrix.sum(axis=1)
    dangling_nodes = np.where(row_sum == 0)[0]
    if len(dangling_nodes):
        for node in dangling_nodes:
            trans_matrix[node] = const_vector
            row_sum[node] = 1

    trans_matrix /= row_sum.reshape(-1, 1)
    return beta * trans_matrix + (1 - beta) * const_vector

trans_matrix = build_trans_matrix(graph)
trans_matrix
```

    array([[0.025, 0.325, 0.325, 0.325],
           [0.475, 0.025, 0.025, 0.475],
           [0.25 , 0.25 , 0.25 , 0.25 ],
           [0.025, 0.475, 0.475, 0.025]])

```python
n_iters = 20
result = init_vector
for _ in range(n_iters):
    result = result @ trans_matrix

result
```

    array([0.20408163, 0.26530612, 0.26530612, 0.26530612])

The result looks much more reasonable after introducing the taxation. We can also compare the it with the `pagerank` function from `networkx`.

```python
pagerank_score = nx.pagerank(graph, alpha=0.9)
pagerank_score
```

    {'A': 0.204081934263649,
     'B': 0.265306021912117,
     'C': 0.265306021912117,
     'D': 0.265306021912117}

Having seen how to calculate the PageRank vector, we should examine how this information is being used. Of course each search engine has its own secret formula that decides the ordering of pages in response to users search query. But we can guess that first, in order to be considered for the ranking at all, a page has to have at least one of the search terms in the query. Then amongst the qualified pages, a score is computed for each, and an important component of this score is the PageRank of the page (we can think of component as a feature to a machine learning ranker). Other components might include the presence or absence of search terms in prominent places, such as headers or the links to
the page itself.

# Reference

- [Networkx Documentation: PageRank](https://networkx.github.io/documentation/networkx-1.10/reference/generated/networkx.algorithms.link_analysis.pagerank_alg.pagerank.html)
- [Online Book: Mining Massive Dataset - Chapter 5 Link Analysis](http://infolab.stanford.edu/~ullman/mmds/book.pdf)