# Tutorial 1: Introduction to Reinforcement Learning

**Week 3, Day 2: Basic Reinforcement Learning (RL)**

**By Neuromatch Academy**

__Content creators:__  Matthew Sargent, Anoop Kulkarni, Sowmya Parthiban, Feryal Behbahani, Jane Wang

__Content reviewers:__ Ezekiel Williams, Mehul Rastogi, Lily Cheng, Roberto Guidotti, Arush Tagade, Kelson Shilling-Scrivo

__Content editors:__ Spiros Chavlis

__Production editors:__ Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

By the end of the tutorial, you should be able to:

1. Within the RL framework, be able to identify the different components: environment, agent, states, and actions.
2. Understand the Bellman equation and components involved.
3. Implement tabular value-based model-free learning (Q-learning and SARSA).
4. Discuss real-world applications and ethical issues of RL.

By completing the Bonus sections, you should be able to:
1. Run a DQN agent and experiment with different hyperparameters.
2. Have a high-level understanding of other (nonvalue-based) RL methods.

```
# @title Tutorial slides

# @markdown These are the slides for the videos in this tutorial

# @markdown If you want to locally download the slides, click [here](https://osf.io/m3kqy/download)
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/m3kqy/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

Run the following *Setup* cells in order to set up needed functions. Don't worry about the code for now!

**Note:** There is an issue with some images not showing up if you're using a Safari browser. Please switch to Chrome if this is the case.

```
# @title Install requirements
from IPython.display import clear_output
# @markdown we install the acme library, see [here](https://github.com/deepmind/acme) for more info

# @markdown WARNING: There may be errors and warnings reported during the installation.
# @markdown However, they should be ignored.
!apt-get install -y xvfb ffmpeg --quiet
!pip install --upgrade pip --quiet
!pip install imageio --quiet
!pip install imageio-ffmpeg
!pip install gym --quiet
!pip install enum34 --quiet
!pip install dm-env --quiet
!pip install pandas --quiet
!pip install keras-nightly==2.5.0.dev2021020510 --quiet
!pip install grpcio==1.34.0 --quiet
!pip install tensorflow --quiet
!pip install typing --quiet
!pip install einops --quiet
!pip install dm-acme --quiet
!pip install dm-acme[reverb] --quiet
!pip install dm-acme[tf] --quiet
!pip install dm-acme[envs] --quiet
!pip install dm-env --quiet

!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet
from evaltools.airtable import AirtableForm

# generate airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W3D2_T1','https://portal.neuromatchacademy.org/api/redirect/to/3e77471d-4de0-4e43-a026-9cfb603b5197')
clear_output()
```

```
# Import modules
import gym
import enum
import copy
import time
import acme
import torch
import base64
import dm_env
import IPython
import imageio
import warnings
import itertools
import collections

import numpy as np
import pandas as pd
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

import matplotlib as mpl
import matplotlib.pyplot as plt

import tensorflow.compat.v2 as tf

from acme import specs
from acme import wrappers
from acme.utils import tree_utils
from acme.utils import loggers
from torch.autograd import Variable
from torch.distributions import Categorical
from typing import Callable, Sequence

tf.enable_v2_behavior()
warnings.filterwarnings('ignore')
np.set_printoptions(precision=3, suppress=1)
```

```
# @title Figure settings
import ipywidgets as widgets       # interactive display
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
mpl.rc('image', cmap='Blues')
```

```
# @title Helper Functions
# @markdown Implement helpers for value visualisation

map_from_action_to_subplot = lambda a: (2, 6, 8, 4)[a]
map_from_action_to_name = lambda a: ("up", "right", "down", "left")[a]

def plot_values(values, colormap='pink', vmin=-1, vmax=10):
  plt.imshow(values, interpolation="nearest",
             cmap=colormap, vmin=vmin, vmax=vmax)
  plt.yticks([])
  plt.xticks([])
  plt.colorbar(ticks=[vmin, vmax])

def plot_state_value(action_values, epsilon=0.1):
  q = action_values
  fig = plt.figure(figsize=(4, 4))
  vmin = np.min(action_values)
  vmax = np.max(action_values)
  v = (1 - epsilon) * np.max(q, axis=-1) + epsilon * np.mean(q, axis=-1)
  plot_values(v, colormap='summer', vmin=vmin, vmax=vmax)
  plt.title("$v(s)$")

def plot_action_values(action_values, epsilon=0.1):
  q = action_values
  fig = plt.figure(figsize=(8, 8))
  fig.subplots_adjust(wspace=0.3, hspace=0.3)
  vmin = np.min(action_values)
  vmax = np.max(action_values)
  dif = vmax - vmin
  for a in [0, 1, 2, 3]:
    plt.subplot(3, 3, map_from_action_to_subplot(a))

    plot_values(q[..., a], vmin=vmin - 0.05*dif, vmax=vmax + 0.05*dif)
    action_name = map_from_action_to_name(a)
    plt.title(r"$q(s, \mathrm{" + action_name + r"})$")

  plt.subplot(3, 3, 5)
  v = (1 - epsilon) * np.max(q, axis=-1) + epsilon * np.mean(q, axis=-1)
  plot_values(v, colormap='summer', vmin=vmin, vmax=vmax)
  plt.title("$v(s)$")

def plot_stats(stats, window=10):
  plt.figure(figsize=(16,4))
  plt.subplot(121)
  xline = range(0, len(stats.episode_lengths), window)
  plt.plot(xline, smooth(stats.episode_lengths, window=window))
  plt.ylabel('Episode Length')
  plt.xlabel('Episode Count')
  plt.subplot(122)
  plt.plot(xline, smooth(stats.episode_rewards, window=window))
  plt.ylabel('Episode Return')
  plt.xlabel('Episode Count')
```

```
# @title Helper functions
def smooth(x, window=10):
  return x[:window*(len(x)//window)].reshape(len(x)//window, window).mean(axis=1)
```

```
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
```

```
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    WARNING: For this notebook to perform best, if possible, in the menu under `Runtime` -> `Change runtime type.`  select `GPU`

---
# Section 1: Introduction to Reinforcement Learning

*Time estimate: ~15mins*

```
# @title Video 1: Introduction to RL
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV18V411p7iK", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"BWz3scQN50M", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 1: Introduction to RL')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Acme: a research framework for reinforcement learning

**Acme** is a library of reinforcement learning (RL) agents and agent building blocks by Google DeepMind. Acme strives to expose simple, efficient, and readable agents, that serve both as reference implementations of popular algorithms and as strong baselines, while still providing enough flexibility to do novel research. The design of Acme also attempts to provide multiple points of entry to the RL problem at differing levels of complexity.

For more information see the github's repository [deepmind/acme](https://github.com/deepmind/acme).

---
# Section 2: General Formulation of RL Problems and Gridworlds

*Time estimate: ~30mins*

```
# @title Video 2: General Formulation and MDPs
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1k54y1E7Zn", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"h6TxAALY5Fc", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 2: General Formulation and MDPs')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

The agent interacts with the environment in a loop corresponding to the following diagram. The environment defines a set of **actions**  that an agent can take.  The agent takes an action informed by the **observations** it receives, and will get a **reward** from the environment after each action. The goal in RL is to find an agent whose actions maximize the total accumulation of rewards obtained from the environment.

## Section 2.1: The Environment

For this practical session we will focus on a **simple grid world** environment,which consists of a 9 x 10 grid of either wall or empty cells, depicted in black and white, respectively. The smiling agent starts from an initial location and needs to navigate to reach the goal square.

Below you will find an implementation of this Gridworld as a `dm_env.Environment`.

There is no coding in this section, but if you want, you can look over the provided code so that you can familiarize yourself with an example of how to set up a **grid world** environment.

```
# @title Implement GridWorld { form-width: "30%" }
# @markdown *Double-click* to inspect the contents of this cell.

class ObservationType(enum.IntEnum):
  STATE_INDEX = enum.auto()
  AGENT_ONEHOT = enum.auto()
  GRID = enum.auto()
  AGENT_GOAL_POS = enum.auto()

class GridWorld(dm_env.Environment):

  def __init__(self,
               layout,
               start_state,
               goal_state=None,
               observation_type=ObservationType.STATE_INDEX,
               discount=0.9,
               penalty_for_walls=-5,
               reward_goal=10,
               max_episode_length=None,
               randomize_goals=False):
    """Build a grid environment.

    Simple gridworld defined by a map layout, a start and a goal state.

    Layout should be a NxN grid, containing:
      * 0: empty
      * -1: wall
      * Any other positive value: value indicates reward; episode will terminate

    Args:
      layout: NxN array of numbers, indicating the layout of the environment.
      start_state: Tuple (y, x) of starting location.
      goal_state: Optional tuple (y, x) of goal location. Will be randomly
        sampled once if None.
      observation_type: Enum observation type to use. One of:
        * ObservationType.STATE_INDEX: int32 index of agent occupied tile.
        * ObservationType.AGENT_ONEHOT: NxN float32 grid, with a 1 where the
          agent is and 0 elsewhere.
        * ObservationType.GRID: NxNx3 float32 grid of feature channels.
          First channel contains walls (1 if wall, 0 otherwise), second the
          agent position (1 if agent, 0 otherwise) and third goal position
          (1 if goal, 0 otherwise)
        * ObservationType.AGENT_GOAL_POS: float32 tuple with
          (agent_y, agent_x, goal_y, goal_x)
      discount: Discounting factor included in all Timesteps.
      penalty_for_walls: Reward added when hitting a wall (should be negative).
      reward_goal: Reward added when finding the goal (should be positive).
      max_episode_length: If set, will terminate an episode after this many
        steps.
      randomize_goals: If true, randomize goal at every episode.
    """
    if observation_type not in ObservationType:
      raise ValueError('observation_type should be a ObservationType instace.')
    self._layout = np.array(layout)
    self._start_state = start_state
    self._state = self._start_state
    self._number_of_states = np.prod(np.shape(self._layout))
    self._discount = discount
    self._penalty_for_walls = penalty_for_walls
    self._reward_goal = reward_goal
    self._observation_type = observation_type
    self._layout_dims = self._layout.shape
    self._max_episode_length = max_episode_length
    self._num_episode_steps = 0
    self._randomize_goals = randomize_goals
    if goal_state is None:
      # Randomly sample goal_state if not provided
      goal_state = self._sample_goal()
    self.goal_state = goal_state

  def _sample_goal(self):
    """Randomly sample reachable non-starting state."""
    # Sample a new goal
    n = 0
    max_tries = 1e5
    while n < max_tries:
      goal_state = tuple(np.random.randint(d) for d in self._layout_dims)
      if goal_state != self._state and self._layout[goal_state] == 0:
        # Reachable state found!
        return goal_state
      n += 1
    raise ValueError('Failed to sample a goal state.')

  @property
  def layout(self):
    return self._layout

  @property
  def number_of_states(self):
    return self._number_of_states

  @property
  def goal_state(self):
    return self._goal_state

  @property
  def start_state(self):
    return self._start_state

  @property
  def state(self):
    return self._state

  def set_state(self, x, y):
    self._state = (y, x)

  @goal_state.setter
  def goal_state(self, new_goal):
    if new_goal == self._state or self._layout[new_goal] < 0:
      raise ValueError('This is not a valid goal!')
    # Zero out any other goal
    self._layout[self._layout > 0] = 0
    # Setup new goal location
    self._layout[new_goal] = self._reward_goal
    self._goal_state = new_goal

  def observation_spec(self):
    if self._observation_type is ObservationType.AGENT_ONEHOT:
      return specs.Array(
          shape=self._layout_dims,
          dtype=np.float32,
          name='observation_agent_onehot')
    elif self._observation_type is ObservationType.GRID:
      return specs.Array(
          shape=self._layout_dims + (3,),
          dtype=np.float32,
          name='observation_grid')
    elif self._observation_type is ObservationType.AGENT_GOAL_POS:
      return specs.Array(
          shape=(4,), dtype=np.float32, name='observation_agent_goal_pos')
    elif self._observation_type is ObservationType.STATE_INDEX:
      return specs.DiscreteArray(
          self._number_of_states, dtype=int, name='observation_state_index')

  def action_spec(self):
    return specs.DiscreteArray(4, dtype=int, name='action')

  def get_obs(self):
    if self._observation_type is ObservationType.AGENT_ONEHOT:
      obs = np.zeros(self._layout.shape, dtype=np.float32)
      # Place agent
      obs[self._state] = 1
      return obs
    elif self._observation_type is ObservationType.GRID:
      obs = np.zeros(self._layout.shape + (3,), dtype=np.float32)
      obs[..., 0] = self._layout < 0
      obs[self._state[0], self._state[1], 1] = 1
      obs[self._goal_state[0], self._goal_state[1], 2] = 1
      return obs
    elif self._observation_type is ObservationType.AGENT_GOAL_POS:
      return np.array(self._state + self._goal_state, dtype=np.float32)
    elif self._observation_type is ObservationType.STATE_INDEX:
      y, x = self._state
      return y * self._layout.shape[1] + x

  def reset(self):
    self._state = self._start_state
    self._num_episode_steps = 0
    if self._randomize_goals:
      self.goal_state = self._sample_goal()
    return dm_env.TimeStep(
        step_type=dm_env.StepType.FIRST,
        reward=None,
        discount=None,
        observation=self.get_obs())

  def step(self, action):
    y, x = self._state

    if action == 0:  # up
      new_state = (y - 1, x)
    elif action == 1:  # right
      new_state = (y, x + 1)
    elif action == 2:  # down
      new_state = (y + 1, x)
    elif action == 3:  # left
      new_state = (y, x - 1)
    else:
      raise ValueError(
          'Invalid action: {} is not 0, 1, 2, or 3.'.format(action))

    new_y, new_x = new_state
    step_type = dm_env.StepType.MID
    if self._layout[new_y, new_x] == -1:  # wall
      reward = self._penalty_for_walls
      discount = self._discount
      new_state = (y, x)
    elif self._layout[new_y, new_x] == 0:  # empty cell
      reward = 0.
      discount = self._discount
    else:  # a goal
      reward = self._layout[new_y, new_x]
      discount = 0.
      new_state = self._start_state
      step_type = dm_env.StepType.LAST

    self._state = new_state
    self._num_episode_steps += 1
    if (self._max_episode_length is not None and
        self._num_episode_steps >= self._max_episode_length):
      step_type = dm_env.StepType.LAST
    return dm_env.TimeStep(
        step_type=step_type,
        reward=np.float32(reward),
        discount=discount,
        observation=self.get_obs())

  def plot_grid(self, add_start=True):
    plt.figure(figsize=(4, 4))
    plt.imshow(self._layout <= -1, interpolation='nearest')
    ax = plt.gca()
    ax.grid(0)
    plt.xticks([])
    plt.yticks([])
    # Add start/goal
    if add_start:
      plt.text(
          self._start_state[1],
          self._start_state[0],
          r'$\mathbf{S}$',
          fontsize=16,
          ha='center',
          va='center')
    plt.text(
        self._goal_state[1],
        self._goal_state[0],
        r'$\mathbf{G}$',
        fontsize=16,
        ha='center',
        va='center')
    h, w = self._layout.shape
    for y in range(h - 1):
      plt.plot([-0.5, w - 0.5], [y + 0.5, y + 0.5], '-w', lw=2)
    for x in range(w - 1):
      plt.plot([x + 0.5, x + 0.5], [-0.5, h - 0.5], '-w', lw=2)

  def plot_state(self, return_rgb=False):
    self.plot_grid(add_start=False)
    # Add the agent location
    plt.text(
        self._state[1],
        self._state[0],
        u'😃',
        # fontname='symbola',
        fontsize=18,
        ha='center',
        va='center',
    )
    if return_rgb:
      fig = plt.gcf()
      plt.axis('tight')
      plt.subplots_adjust(0, 0, 1, 1, 0, 0)
      fig.canvas.draw()
      data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
      w, h = fig.canvas.get_width_height()
      data = data.reshape((h, w, 3))
      plt.close(fig)
      return data

  def plot_policy(self, policy):
    action_names = [
        r'$\uparrow$', r'$\rightarrow$', r'$\downarrow$', r'$\leftarrow$'
    ]
    self.plot_grid()
    plt.title('Policy Visualization')
    h, w = self._layout.shape
    for y in range(h):
      for x in range(w):
        # if ((y, x) != self._start_state) and ((y, x) != self._goal_state):
        if (y, x) != self._goal_state:
          action_name = action_names[policy[y, x]]
          plt.text(x, y, action_name, ha='center', va='center')

  def plot_greedy_policy(self, q):
    greedy_actions = np.argmax(q, axis=2)
    self.plot_policy(greedy_actions)

def build_gridworld_task(task,
                         discount=0.9,
                         penalty_for_walls=-5,
                         observation_type=ObservationType.STATE_INDEX,
                         max_episode_length=200):
  """Construct a particular Gridworld layout with start/goal states.

  Args:
      task: string name of the task to use. One of {'simple', 'obstacle',
        'random_goal'}.
      discount: Discounting factor included in all Timesteps.
      penalty_for_walls: Reward added when hitting a wall (should be negative).
      observation_type: Enum observation type to use. One of:
        * ObservationType.STATE_INDEX: int32 index of agent occupied tile.
        * ObservationType.AGENT_ONEHOT: NxN float32 grid, with a 1 where the
          agent is and 0 elsewhere.
        * ObservationType.GRID: NxNx3 float32 grid of feature channels.
          First channel contains walls (1 if wall, 0 otherwise), second the
          agent position (1 if agent, 0 otherwise) and third goal position
          (1 if goal, 0 otherwise)
        * ObservationType.AGENT_GOAL_POS: float32 tuple with
          (agent_y, agent_x, goal_y, goal_x).
      max_episode_length: If set, will terminate an episode after this many
        steps.
  """
  tasks_specifications = {
      'simple': {
          'layout': [
              [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
          ],
          'start_state': (2, 2),
          'goal_state': (7, 2)
      },
      'obstacle': {
          'layout': [
              [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
              [-1, 0, 0, 0, 0, 0, -1, 0, 0, -1],
              [-1, 0, 0, 0, -1, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
          ],
          'start_state': (2, 2),
          'goal_state': (2, 8)
      },
      'random_goal': {
          'layout': [
              [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, -1, -1, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, 0, 0, 0, 0, 0, 0, 0, 0, -1],
              [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
          ],
          'start_state': (2, 2),
          # 'randomize_goals': True
      },
  }
  return GridWorld(
      discount=discount,
      penalty_for_walls=penalty_for_walls,
      observation_type=observation_type,
      max_episode_length=max_episode_length,
      **tasks_specifications[task])

def setup_environment(environment):
  """Returns the environment and its spec."""

  # Make sure the environment outputs single-precision floats.
  environment = wrappers.SinglePrecisionWrapper(environment)

  # Grab the spec of the environment.
  environment_spec = specs.make_environment_spec(environment)

  return environment, environment_spec
```

We will use two distinct tabular GridWorlds:
* `simple` where the goal is at the bottom left of the grid, little navigation required.
* `obstacle` where the goal is behind an obstacle the agent must avoid.

You can visualize the grid worlds by running the cell below.

Note that **S** indicates the start state and **G** indicates the goal.

```
# Visualise GridWorlds

# Instantiate two tabular environments, a simple task, and one that involves
# the avoidance of an obstacle.
simple_grid = build_gridworld_task(
    task='simple', observation_type=ObservationType.GRID)
obstacle_grid = build_gridworld_task(
    task='obstacle', observation_type=ObservationType.GRID)

# Plot them.
simple_grid.plot_grid()
plt.title('Simple')

obstacle_grid.plot_grid()
plt.title('Obstacle')
```

In this environment, the agent has four possible  **actions**: `up`, `right`, `down`, and `left`.  The **reward** is `-5` for bumping into a wall, `+10` for reaching the goal, and `0` otherwise. The episode ends when the agent reaches the goal, and otherwise continues. The **discount** on continuing steps, is $\gamma = 0.9$.

Before we start building an agent to interact with this environment, let's first look at the types of objects the environment either returns (e.g., **observations**) or consumes (e.g., **actions**). The `environment_spec` will show you the form of the **observations**, **rewards** and **discounts** that the environment exposes and the form of the **actions** that can be taken.

```
# @title Look at environment_spec { form-width: "30%" }

# Note: setup_environment is implemented in the same cell as GridWorld.
environment, environment_spec = setup_environment(simple_grid)

print('actions:\n', environment_spec.actions, '\n')
print('observations:\n', environment_spec.observations, '\n')
print('rewards:\n', environment_spec.rewards, '\n')
print('discounts:\n', environment_spec.discounts, '\n')
```

    actions:
     DiscreteArray(shape=(), dtype=int32, name=action, minimum=0, maximum=3, num_values=4)

    observations:
     Array(shape=(9, 10, 3), dtype=dtype('float32'), name='observation_grid')

    rewards:
     Array(shape=(), dtype=dtype('float32'), name='reward')

    discounts:
     BoundedArray(shape=(), dtype=dtype('float32'), name='discount', minimum=0.0, maximum=1.0)

We first set the environment to its initial state by calling the `reset()` method which returns the first observation and resets the agent to the starting location.

```
environment.reset()
environment.plot_state()
```

Now we want to take an action to interact with the environment. We do this by passing a valid action to the `dm_env.Environment.step()` method which returns a `dm_env.TimeStep` namedtuple with fields `(step_type, reward, discount, observation)`.

Let's take an action and visualise the resulting state of the grid-world. (You'll need to rerun the cell if you pick a new action.)

**Note for kaggle users:** As kaggle does not render the forms automatically the students should be careful to notice the various instructions and manually play around with the values for the variables

```
# @title Pick an action and see the state changing
action = "right" #@param ["up", "right", "down", "left"] {type:"string"}

action_int = {'up': 0,
              'right': 1,
              'down': 2,
              'left':3 }
action = int(action_int[action])
timestep = environment.step(action)  # pytype: dm_env.TimeStep
environment.plot_state()
```

```
# @title Run loop  { form-width: "30%" }
# @markdown This function runs an agent in the environment for a number of
# @markdown episodes, allowing it to learn.

# @markdown *Double-click* to inspect the `run_loop` function.

def run_loop(environment,
             agent,
             num_episodes=None,
             num_steps=None,
             logger_time_delta=1.,
             label='training_loop',
             log_loss=False,
             ):
  """Perform the run loop.

  We are following the Acme run loop.

  Run the environment loop for `num_episodes` episodes. Each episode is itself
  a loop which interacts first with the environment to get an observation and
  then give that observation to the agent in order to retrieve an action. Upon
  termination of an episode a new episode will be started. If the number of
  episodes is not given then this will interact with the environment
  infinitely.

  Args:
    environment: dm_env used to generate trajectories.
    agent: acme.Actor for selecting actions in the run loop.
    num_steps: number of steps to run the loop for. If `None` (default), runs
      without limit.
    num_episodes: number of episodes to run the loop for. If `None` (default),
      runs without limit.
    logger_time_delta: time interval (in seconds) between consecutive logging
      steps.
    label: optional label used at logging steps.
  """
  logger = loggers.TerminalLogger(label=label, time_delta=logger_time_delta)
  iterator = range(num_episodes) if num_episodes else itertools.count()
  all_returns = []

  num_total_steps = 0
  for episode in iterator:
    # Reset any counts and start the environment.
    start_time = time.time()
    episode_steps = 0
    episode_return = 0
    episode_loss = 0

    timestep = environment.reset()

    # Make the first observation.
    agent.observe_first(timestep)

    # Run an episode.
    while not timestep.last():
      # Generate an action from the agent's policy and step the environment.
      action = agent.select_action(timestep.observation)
      timestep = environment.step(action)

      # Have the agent observe the timestep and let the agent update itself.
      agent.observe(action, next_timestep=timestep)
      agent.update()

      # Book-keeping.
      episode_steps += 1
      num_total_steps += 1
      episode_return += timestep.reward

      if log_loss:
        episode_loss += agent.last_loss

      if num_steps is not None and num_total_steps >= num_steps:
        break

    # Collect the results and combine with counts.
    steps_per_second = episode_steps / (time.time() - start_time)
    result = {
        'episode': episode,
        'episode_length': episode_steps,
        'episode_return': episode_return,
    }
    if log_loss:
      result['loss_avg'] = episode_loss/episode_steps

    all_returns.append(episode_return)

    # Log the given results.
    logger.write(result)

    if num_steps is not None and num_total_steps >= num_steps:
      break
  return all_returns
```

```
# @title Implement the evaluation loop { form-width: "30%" }
# @markdown This function runs the agent in the environment for a number of
# @markdown episodes, without allowing it to learn, in order to evaluate it.

# @markdown *Double-click* to inspect the `evaluate` function.

def evaluate(environment: dm_env.Environment,
             agent: acme.Actor,
             evaluation_episodes: int):
  frames = []

  for episode in range(evaluation_episodes):
    timestep = environment.reset()
    episode_return = 0
    steps = 0
    while not timestep.last():
      frames.append(environment.plot_state(return_rgb=True))

      action = agent.select_action(timestep.observation)
      timestep = environment.step(action)
      steps += 1
      episode_return += timestep.reward
    print(
        f'Episode {episode} ended with reward {episode_return} in {steps} steps'
    )
  return frames

def display_video(frames: Sequence[np.ndarray],
                  filename: str = 'temp.mp4',
                  frame_rate: int = 12):
  """Save and display video."""
  # Write the frames to a video.
  with imageio.get_writer(filename, fps=frame_rate) as video:
    for frame in frames:
      video.append_data(frame)

  # Read video and display the video.
  video = open(filename, 'rb').read()
  b64_video = base64.b64encode(video)
  video_tag = ('<video  width="320" height="240" controls alt="test" '
               'src="data:video/mp4;base64,{0}">').format(b64_video.decode())
  return IPython.display.HTML(video_tag)
```

## Section 2.2: The Agent

We will be implementing Tabular & Function Approximation agents. Tabular agents are purely in Python.

All agents will share the same interface from the Acme `Actor`. Here we borrow a figure from Acme to show how this interaction occurs:

### Agent interface

Each agent implements the following functions:

```python
class Agent(acme.Actor):
  def __init__(self, number_of_actions, number_of_states, ...):
    """Provides the agent the number of actions and number of states."""

  def select_action(self, observation):
    """Generates actions from observations."""

  def observe_first(self, timestep):
    """Records the initial timestep in a trajectory."""

  def observe(self, action, next_timestep):
    """Records the transition which occurred from taking an action."""

  def update(self):
    """Updates the agent's internals to potentially change its behavior."""
```

Remarks on the `observe()` function:

1. In the last method, the `next_timestep` provides the `reward`, `discount`, and `observation` that resulted from selecting `action`.

2. The `next_timestep.step_type` will be either `MID` or `LAST` and should be used to determine whether this is the last observation in the episode.

3. The `next_timestep.step_type` cannot be `FIRST`; such a timestep should only ever be given to `observe_first()`.

### Coding Exercise 2.1: Random Agent

Below is a partially complete implemention of an agent that follows a random (non-learning) policy. Fill in the ```select_action``` method.

The ```select_action``` method should return a random **integer** between 0 and ```self._num_actions``` (not a tensor or an array!)

```
class RandomAgent(acme.Actor):

  def __init__(self, environment_spec):
    """Gets the number of available actions from the environment spec."""
    self._num_actions = environment_spec.actions.num_values

  def select_action(self, observation):
    """Selects an action uniformly at random."""
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    # raise NotImplementedError("Student exercise: complete the select action method")
    #################################################
    # TODO return a random integer beween 0 and self._num_actions.
    # HINT: see the reference for how to sample a random integer in numpy:
    #   https://numpy.org/doc/1.16/reference/routines.random.html
    action = np.random.randint(self._num_actions)
    return action

  def observe_first(self, timestep):
    """Does not record as the RandomAgent has no use for data."""
    pass

  def observe(self, action, next_timestep):
    """Does not record as the RandomAgent has no use for data."""
    pass

  def update(self):
    """Does not update as the RandomAgent does not learn from data."""
    pass

# add event to airtable
atform.add_event('Coding Exercise 2.1: Random Agent')
```

```
# to_remove solution
class RandomAgent(acme.Actor):

  def __init__(self, environment_spec):
    """Gets the number of available actions from the environment spec."""
    self._num_actions = environment_spec.actions.num_values

  def select_action(self, observation):
    """Selects an action uniformly at random."""
    # TODO return a random integer beween 0 and self._num_actions.
    # HINT: see the reference for how to sample a random integer in numpy:
    #   https://numpy.org/doc/1.16/reference/routines.random.html
    action = np.random.randint(self._num_actions)
    return action

  def observe_first(self, timestep):
    """Does not record as the RandomAgent has no use for data."""
    pass

  def observe(self, action, next_timestep):
    """Does not record as the RandomAgent has no use for data."""
    pass

  def update(self):
    """Does not update as the RandomAgent does not learn from data."""
    pass

# add event to airtable
atform.add_event('Coding Exercise 2.1: Random Agent')
```

```
# @title Visualisation of a random agent in GridWorld { form-width: "30%" }

# Create the agent by giving it the action space specification.
agent = RandomAgent(environment_spec)

# Run the agent in the evaluation loop, which returns the frames.
frames = evaluate(environment, agent, evaluation_episodes=1)

# Visualize the random agent's episode.
display_video(frames)
```

    Episode 0 ended with reward -115.0 in 102 steps

<video  width="320" height="240" controls alt="test" src="data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAPsltZGF0AAACrgYF//+q3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1MiByMjg1NCBlOWE1OTAzIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNyAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD0tMiB0aHJlYWRzPTMgbG9va2FoZWFkX3RocmVhZHM9MSBzbGljZWRfdGhyZWFkcz0wIG5yPTAgZGVjaW1hdGU9MSBpbnRlcmxhY2VkPTAgYmx1cmF5X2NvbXBhdD0wIGNvbnN0cmFpbmVkX2ludHJhPTAgYmZyYW1lcz0zIGJfcHlyYW1pZD0yIGJfYWRhcHQ9MSBiX2JpYXM9MCBkaXJlY3Q9MSB3ZWlnaHRiPTEgb3Blbl9nb3A9MCB3ZWlnaHRwPTIga2V5aW50PTI1MCBrZXlpbnRfbWluPTEyIHNjZW5lY3V0PTQwIGludHJhX3JlZnJlc2g9MCByY19sb29rYWhlYWQ9NDAgcmM9Y3JmIG1idHJlZT0xIGNyZj0yNS4wIHFjb21wPTAuNjAgcXBtaW49MCBxcG1heD02OSBxcHN0ZXA9NCBpcF9yYXRpbz0xLjQwIGFxPTE6MS4wMACAAAAPS2WIhAAQ//73Tr8Cm0WXLk3F/QLI4li2gKqE+jfNNV2HlR9kFydYoBHOs/3rZBVNmErF2TFFLPNg7vDp6Iel6R1LZcP7SsFxp23ZmkS0sIx9jZRpwTDDyt1aphtX/963nqbaKSrWnQbuAglC3MOZ5nIdtpQk2Xc1u5yXaONjSSydg4xYLb7fhFN2juPdIepXCVU+nbppYqZ1ZXWIwo/hPTbUqyDjwjdPx97y4VKt+Sg/n0DGbdkuFJG5iQn1XlLRR0Lf4vRJhj/yEhm7GeFfTrL72jT2V9vBlXXdShCjiOsqhbe3gXzUXsT1zflo2T+uP2qKzGX67EZiYoKRBYutY6iL5/0SZ/UtjQ3YbfltlXMbNZfNnxhhDd3sqEwWMTU8sk9mX02YwATri7/5ZTAp5r4TtFdn9m1nUNT7qSLFNMuLtG9gUKLFGM3PYMGsOA/u+CXQWW4NTUiV2PlJDbp957uD837zjpOe4fzmkfF/rwHOqH2Z+ut5c9opmwBDjvU8K/o3O2Mu0xexmlD7APJ7t39hg+wnTNKibWH/ODS779jxBaPJvrFx488yL/daLbTwHooEWVUslBmJXO8dp73mKg7lc2vPVB7m2U3Ji0U01ZpX+3P9yvkh3hgJ3yRla1Da4xaWmm6pmDmNfdmsdaA0H+HpNszp8Yfflrceii7TouatvDvhGu/Fdu3i4BEyDUDr75w1ENajywzK7+ulWde+2pUowgRrc6JONe4DPEnjJEd+vEx5UjTM8H78ljwxgNk1psdA5dXuAnlh3/LjTBfqT+Sy/10K2al5FKdc/TPau+RJqwdQL4Uv9dwUMMtnwaEspE5ZVNfqX8mMS9sezwJCOpaBsUgnmm6R3ln/hwmRtQAgZYZ7N0rESb8gxXrbBAIEKqyzSs5E90PAaJS8n/kJt4bV9WR8W3yfiXUYWl7686zQYdns1MYb1vq1GVpTxpS8Pi7k4LxEoSVTIuI+HuPYcjUKbkR/xvuT9dkaIPVXBev98HYkX7BjtWr2F0r60O7nvXh/naeQ77Xxncy+jPIfOOgwU2QDdGf/2x/eupXIL2+mZMpx4evbn1Vvk8N7Zu/IaBxqHfl3F3swhrNM2ALx+IDpgjG1Dbnji5B8h0mMhTrIykHoI0sg+Q42Qw/kzCduZ6STJfxz/mKAXrK/ORpKQb3oiEcQ0REF3nQ/b8ziPAAqhzpDnCgPXFmebobrhSMlR8tY30KICSw2UBklFqfAXSGdSkPIKFO73F84CbBKMg21VrVEzo2dyijxBK/F8etKu3rHR6vk9RQQJt06A7BfZ48/LRHjEHu4QpJ9r5KyvYGVkQVAO42ZuJ1/FrEaNKQE+jv0lH2XRNjfC+yUdbqQFXk1trD+hqJQcwNOe1Kjj9PM2NbAOmwnwGamfkRnPiT56hliXRrfay1DtxhmOddbE1D2tDSw0lNFHNjEOE1uFsUZOBXmG/Er3rTlrYgEq1YqoK6Qnrh/hVaoePbAHMC7RM65HSfP7cOtnQ55e3wf1+Pi2uHIUNIbJTm1y9td/9U8siLxLZ3zpD9LDIVBYsCsQyKraDAfOOR2fHeBI9sBROQeZZCK080qQ5p0bexnzqX/DSTe6kMZlft5rC346xv4uW0R1b1Z6bVIdMlajs7ECdDh6Y5HuHrabFZjgO4IaHVPxAeIlxrtKFuJg8SYkOqVrvtlKxwYIlpRDnYA0TmMZRz70WmZtoD4ZbnUoprcTpSTgSmDjOq/RjMSsu/nWIkWgPl6X38hy0uWyTL0hFpIhUCn6XJ1p41cJDavBGtSwyTYWQQZb/u1B6B79fB0oaehLgZru3/MZaUHKPUMZEFxiq6PHf23fVM/R1g5VMThh+7NcOjKgy/nN4pNN7ZUvrGne6Lqkc9fR067Bd8hLnV3FgsU4b5yBkX8i3SL/2mjG/QC4phO4BO6dxGvcym08GtGhCcoakITd1JAlT3nFXhCb53Q77y+2iSfzaX9mL20DgkWiV9Af+0oZ+Eeu5qDNBOjhz/mkvlKNRJJe5SrgdwvrSvrZFqjifP3YwY2If42MUZgbi4Ds/LD7kw0FKHMmKxdRL0Mj+TkeNs+Tigc8wYTACf6/pj62abvhE0qJXFaPr7I4EFkphTmbN4nlxC1FqtSuE2Vjxp0lYx7Pmbpx6C9coDItmcWj2Er0w6/Oz09fRyLivH6lhYUkkTg+KnkQmRPEQvMkC6X335z8DxpCd/h17lHg2oXlFu8DZnuOqLKu4yWo/jMrsxDHBRmqYsoXHQVYTEBs2ik98nKa/UDas9OSQS8bTO4urXbEFYlnAhGLWHHgzWGFzGaf1StCRMzYqgLg6nejl1X/q5uXbWrhSh3+H4absRb/MTsklLU0/hNWI1XdJJM+jY36PXNx88O/5HVCFf7U7egad014L7s30o8CBEH13g4ab4465+hJyuX85IYiIQds+6oislqxJZsYPtJuPLqn/eGH/v39mnrXL+J3DNlA6SHHlLZz09rlH+mwr2omn02b1ASavjaSKV/wtKx+HEY9B7en/sGZ1wfLezKYP6EKwJqtCAvB3XhM/GJDeQtJriqqSmf8aZumO59wrIiM6Fu9nw6wVIgT4bFzXpsx39RvFf+buKGfNQSHcy2pOgxj1/V87+s8kqUKRcYd4fDQxkAQnsi0WMSTHZ90JPmbQuCXzl8z7dS2YrIzp30nN6XnYZHEaZR8Lz2C5wsdls7tTx7HwzoJbTH/8A9BMEB6GpY5pQJxk9AzeiKOagL1Lc8Uuh3tJ/+8+WJ2ewwAurx4MVjM8EAxyXf3xnm6r3EVz4wVUYkdVn5ULtiImKW4Bl8aXpmuGxlLCjUqp6KucbaWX/nhZ5qKyW3bZU/CCGu68bM9rnqzoOOBE7lz0hy1ynpeE5MMcIJlpeZSJloQRX4loOq1IYb8NTBg1Vjudqds/ZJ5WYRBdTDHBHCIxyKWcvUH0cllZ81z4OqNidFHlHFb/Mz5oONU9LJ1Lw2eUhUIJx7lnNWgI9AgQOmCUbusJdAwbdnM0hvr2rUToVtOYRuqA2by+pvTdFNhhZ+h3dFASa24205x9CrYEOV0DGgRDkHxZx92NarbrKTmI3RkCDX+4nAiHmwoV97/rSAeBuWQt4Q8hWpd01bDCX+P6ng1U6M8Y/kCq07RqweOxJi35VD7iUdxM7TbZ17E6mssamikB2q0845FhSxG5YvLOgJEXKotpluHZQ6oKShKYrtSA2dmtSsxIn1wnUAuIRRsAD0Y9yp/3MkaJRLxLVfaGK7Fq9av11mUq7Sw2OA1/dTmd3bkP+bzgGSUT+dTpe35xr3FBkvimBOBBORE+I8QEYFxvZBX+rVi/9urSYimkk8sAB6f+JXE48VPL++rU+SQQzKRU5bYEjOJV0ND+DrBL4gKuXHbR0ehhB5+7lKuPf6xE34A8f9aezMIc0GKVF2n+bUyibLQTgc1doVCJoe0Rc8w/V12A47dOg1Q2HOpGs2hPYoipP764lzqeJxKIRPMVc6WrCqo9tnb1S1a0AmnTU0qsmlpirehViaY6BXZzjOHVgj5CLoZdE2jYTt4rCXP42Lsk/+oqbvr6EGhNTiNp4SAGToXVE3MQ+vDlvTVqA3WJkKaiyecb7eShlxou2d3P8b0GMySswhnRAa9ULzXYIypkR7jjglcs2dY/wAk8wDyVTkedHtk8u1vUEHyV1r4ZM9qGKvQDsFJVn16wzcsqEymFyy6RLaVjf1QJtbE7IOGGj8C5MwZR4fnWmAw0VF33xiioG9lKrFMoeQy9OWWBVAqhfquaRVmWJUMnlHU6YbWoOzzAhGRvb2ZuY+p+lEChX/YtWkYQvwpa8bgZmPL6RrJf+qYh4GCNGv3aET8hdF5cELED6aQIfIScbLFCcuHCgjHwT1F4gNhX21gv2uJSmDOQSLtFgJ4WNgUGP7bSMZLLtLi4EM7B0zHaaZigZZV7QaqmfGyC6nBMR68rjgeNoHw/zGt0kC2xIPeMqxuuCEuXyjEKlQ2SSa/QxSicYn7BhGUVNKg7l0HBCoMuCzdHFNFkXsb/XGNm5Gampxh2Sz36kwbN14lgAUpJHPHqFZoeCs0rl4ARHQRJvjo3lcUqjn72WtJeFl0LxFzjcG94NqyfWfcVz+42uaRajpTr3Sqqh0UNkE2BvQnN5PMXbbmrtCoRNx+K2UtCxgaAQ5ZuVBw5ro4HSvOtijy+w5jgQ/O0FiTfGw7JL1ArLvs97cO3VkK3dqXh28JGWipH8TdBQXq5PlYPfSSpS2qAVsGqPD4Z4P/VvOaoGeUQdyWfUIbOrQmbGXWkhoh0t6ruMwP/cXlVcxj2GwQ6OgtvED5O7/NU1dp/KPthYEJSAEAiErFzqD+m83pK1owW25EZzxeiMyYmFUnK3piNCPub4CittfU5X86rAK71p4WEIbpL1vMTfjQItLQ9P3FsxgHtwln9sYZJpoHK96iWm41jtGtZl5P6yvWC/jqJ7v1kKQBFeuueOUWoj9eT/s0CLXcTnMPJoqw6OUh8OzRFMZdUneC3qsZ11/oZs+bhPbzW2ljZOhgWGW6jLwNMRIYkw+GW7VZCWo1h370GxgqaOsKSrKTN1U0mm5r2kIAJn63kjQnOra1VZZg8OWhN3P73m/lrpWkSSSZPkkADnLQVgjoCIiF0tfGbRHsMMaqCBfRhQQqVCvWX7Gjuo4l51Pk7bpAT0yQ7YQ7ev198QkOe8NYZudFnfNaDBvXudrftPO3xYlasRg+hzkXi91AORni/8zQH/2/0CRT0cQQVzIgZYg+UFIFJZpsurun9c4v4wkJ6+NszeCEJeFuqw/LuCgBTM7FL4dfB1VS4JIIu1n5cRR/udBB3uOWRifzgwblnY9mVLekiaIN+x2yaIpXbMd3AouuzIiA7OHpLe1y/HIbvtEJJw1BHu+rps1uITjs1fwq7+Iv/iTomQDZBJU1McdlxRt7WNOg485WQ0voHZL67dXC6c1I4MrfqGMoYYonFeT9YjAdMvrA1pVn9ZJtoHCRVWcnoAA/VU7EbH7xVV8sZFGY0vR9JZesfUpohmQxu+rpsdv8TePg5r1GovKAOR5ZzWxfyO9rGnQgDparmWsmHUlyXTPEosQscAFau2jLUxZvYd5oNC8VPwSzqTFUfKeRnTWv0hUMca/xcRN3AFRyGE3A+vDwHAj81X69j0CgumkLKsFUTsA39Au26aq3LH/OS1mLgeq7LPXmPoSYzVxKuiM2BvwAIIPYQAAAadBmiJsQQ/6KxOWVND7Aiym00ME2rqnaxhlnvs5wqOA21VNlmMbMrf9+qed3XtGyd8CJIyxzf515y7+vFEK3oOqto4EE60g86J/IqhtOcCYWigwWpZiBM5+PHHQrDWvsB9WSLNTP7pLg5ezuavM/bk9dLNsLFuB1unrrZJn8GopVpoDtuvt82EQykTI2KiLgZs3W4dCP/zHZwdPznlTiR3nv0zTO/nli4z/4dKB91i4MYcPlVsUDkjD9MNLh5M4CkctNE+lX9ruHm7b3hh4CyZJrbOCrRN3QZwyUo7NJjNBac0WLNcc0AgAv37f/lWByPv8M3+iryhpKOy+IL/0uTaF3StkL7etAQ3RfA659/wUSLCLsfC3bmAirKrBXO8d9/sHIKORP/5eNrQX7J8c2gPh3ZlUJkOPrWV5mZaUTHHOxlApoxTX00nkyVF7HPb7C0eoi00AmX8/POvT4l1FmktC17kGpxJwl2yhLVbmxXTZVH5rba3jMJXV6U/HxdBlTCuKvuFlkeYjBLWBcJSVS/q6zktAunVT3HWuYUL/kd3EViD0IvGhuEAAAABAAZ5BeQz/BBEKFUdBhem+hcGC5mTKoib1SfLZc2sZBKdpa0SHiw8ATRmoAfagvYGFr+cZ3RSZQJfg0jLR4Jk8wQAAAQ1BmkY8IZMphBD//XXzgKuDFYbWiVJ2EjY+o0sHX4pmjIuBAK24WzJhMEtke/FJZ7Mc7TrPA0u0QKGswzzEaWtxjswqiw300MXde5hlmuwfWvKPh2u9bohENRUSVMSsW17dEkixG2EGw0LCiOtN0dVAp1Obd/l+dmnMBfVInb5S9SXvjb00z+9aNjvLpifNj72v73o+51pq7vb5k5TtCnsvVO1xdzdhj40xVlCBt99rXa++iCnhRiSuaLKxdqXNZThsES+TRUXt1iO8EkZjbcvRhipICCaU9wiunHS7DkxJm/jzmzPkhapZ3c9TlatPL9CeYASAolxjkjykMcqQotQzjhit/+sYPaq6ZpIZMAAAALFBnmRqU8N/AS6bQGLLo5ThavhrUklADjeo8ZEgZBYzImq3uN5/uPr+eiVC5iU+IYxPpw8R7Clyid4uXl3F/5Fe0/FT/D1wm2ZVODJ+C9FWEu26eP4yp1TMciNpXzOzZTWObElHdFkdS0gncyCj4zetb8ZIi84q/fM/1RXHhfC+eztSGTcXG68SeuVaBCM3ySYcGFbA2lLZRUydEs+Xh4NTj4hQ6k15zsGPOREI95M3i4kAAAAlAZ6DdEM/AZz/VFA2WAHO1dEORZWC+MX9aC+u3IeAf/f/CxxswQAAAB4BnoVqQz8BSGichGPm6cMZHf4XXzn1m6PHeb759MEAAABTQZqKSahBaJlMCCH//XXzgKvPVYbWXLJ2HGY+kgMGXQpmiWgKlEanXYixIBfFYcrn4+0Hruzl8LpKK3sJS56HbHngKFLUcs2lOMmPYnvLA5akoOkAAAAlQZ6oRREsN/8BJvN4+Uf0Sj3x5Ox1hDugsmr/zHJA2dMUZ1GVgAAAABMBnsd0Qz8Airo9WIkRl287YoPMAAAAFQGeyWpDPwFIaJyPy+41j7n1FHfLaQAAADZBmsxJqEFsmUwUTBD//XXzgKuDFYbWiVJ2EjY+o0sHX4pmiCwAJoZYbBfB2BJE9/MsugrLToAAAACTAZ7rakM/AZOwpl3IRmiYOCiIANi/u4v2DH0dC0VqiLOCQoTd06j5Mi4wWM1v/dRMbpQ2fFvYmwT9RIAX6VKCj2+yi68aHezQdKPoN0ImE14c889eRbMqVLENG7TDzkUl+NkBwRWaeOWPHcaz1egxuyPtvPq1qMR5CP7KhP1SJIRDESBqEKODIbx+GkraOZTuDSOAAAAAwkGa8EnhClJlMCCH//1184Crz1WG1lyydhxmPpIDBl0KZolk3iUVo4phHB6lIJlPnyArS0Tgv9wHZE7YNZmkkPaYdw2H2tmsM/lPZFOmq/8lRr8QE/SWBq67yhtk6jyCb++kMYxEAayzSIsSs/rER+wCOOUewrrXFJPkLFB9CRrMjDX5oLvpwGq4eOv0XjHO/6+E5vB/Lp5fwIC9hH31YYbnxGfcGYt+B7KV4zrXwiIZTktAyKTY4zFOlcKenTAoX9WBAAAAIUGfDkU0TDf/ASbi0dx6hw76mJPDguTCDBgnBTOMxvRdwQAAAJABny10Qz8BSIufCovbNAghM4oANqNLaPPy8mIidmvrFuHtMhegdhzlWXxjrRLMGMyDF3G8Xwu+6PM7YMi7RgVSH/MivEZFyMZcVFXF4qAr3ZzS/Zf2vrPq3jjRCjtrgoaZpu6Z60WaV2oVpw36rx/OvVckb10/udYZv5UGhsC53tB/b4YmAocb/A/NUUswxx0AAAAPAZ8vakM/ALn3Mb/WjzkmAAAAvkGbM0moQWiZTAgh//1184CrgxWG1olSdhI2PqNLB1+KZoj3MP3mj0zxYvKApAXt7D4W0FlBGzDphALBronBf7gOyJ2wazNJIgL0PkMx2e0gZW9OBv/n8gFIdMssty++9nCsWnwDTIES8GV3vlhI7I3pEGhLeBnDlGfApfGW3g1B+yKwiGzP8x0j3c7ms57LT0kwM20IGXxo7L//t7O3W78dlSgFOrQjhIA5emTfu7yroNADyrllTS+ZYQ9zenQAAACnQZ9RRREsN/8BJvNmIDHWe/g9k/6AD91nbJ7Uk+zEJdOyaduovwwtfy7CCBBUNi27Q8T4T2Ss03d10iN2NbIdmhIPZQ7TUbBA7f6SVryTgEf/bdem6hdW7WHiQqO8uuBdQ5sN9jSe5YX0o9zWEW1NTPnpfI+KKxAhBSvuomy9r7dcELueDYPzIr+Oga2q17ZgPHDz5QoZd6eesIMQed3fnRZpXkYz400AAAArAZ9yakM/ALo0+UZtuMM075UB+lwo8WdZdxWCyzMR4n/qu3bmH7WWh6R4IAAAALVBm3dJqEFsmUwIIf/9dfOAq89VhtZcsnYcZj6SAwZdCmaILAAtbHMqWbWxXyGbB053uXELygoBpAHZpKqQ8lI5l41hk5CVabbPjzL47oQfkyeFrXgs3KLExT6TXkKqlLpex5Z83ShseUoXRsIorE2wCEz5idFVvhWM7jNHYPiI7lmKmnoeXNdSqi7alhDRLuA9wR3TB161hckg4CtldXbxKwrzYFKC4a3cH/81PkHaQQPTGdZgAAAAIEGflUUVLDf/ASbzTSPYMbYeB9/QqoxW6ukSd1j9cN+vAAAAmAGftHRDPwAc/fV2QXqtdBuOIrQOieACEZ2FCuX28Q7Muj6RiNACoLYwelPrtjt5GdPqz+IWoyMw4PkEsDWD+SJJmOvg+YGux90+PTzHnj1Q8c5BlwUHPQVtFO67T2TDkCSUqOUUXoPNO9wNl2Xe/zx4i5xruq+qwu2HRgqkG+JRl/YwYjSNXvGpV5CIHqgiFDKEyy3uVJFAAAAAjwGftmpDPwALPX2Yhi/pfLzrUm/EQAbGASUhYsLXs/9vjeipz/UEXuaRT6f8h9CflgtVP45hpFPe5W9KQfgGAWCMsL2T4C7Ra0mCmEm6HWaw12waFFdHaujB7vaNcet+RehWIWFjIaBXJl5Q50W7h8BJVlHfQWwrytn2OmOwnd+VcTel94yrN3OlJMD/+AANAAAAhUGbu0moQWyZTAgh//1184CrgxWG1olSdhI2PqNLB1+KZogsACZQcvlIuze+uhCOoDhUdFjVAbFPQgOWRBkeCRNtyTaKvtjzktAUTTVOnpv4sHZu1RUZfk4wIG/Ow2TrpDJKMAVxBJZS4NP3L7xRDjZ90y1oSs74h9TEDoLwHP+ZYq46i40AAAC3QZ/ZRRUsN/8BJvNNRqe2St9t350q2xgnmuFfBRSxeisfkGShM0ALcNCCKVWlDgXdwsvvifAtrqhPTf2+C/Xd0A3LNJO1RV+oF5ICa8Z0SK31UbSQ3yEkhX7UYFDzwqGfGLUlZye3THW3ZNEs2IinVaza9k5UZuI0WbzesHbU8jxvTkxM5WEg0wFq+9xeHg9qNSpAX1/OS0Oja/SM+JAmbd+xUxULgDjhJAUsEH8BgV4Cc6/a8xQwAAAAFwGf+HRDPwALWlsVy4iWvWyDkiIDkd6pAAAAFgGf+mpDPwAKzXx0yWW3k//IgNe8McAAAABCQZv/SahBbJlMCCH//XXzgKvPVYbWXLJ2HGY+kgMGXQpmiCwALWxzKlm1M1WiZBT62hMqdY8823e/HFbpAuNAqVNBAAAAIkGeHUUVLDf/ASbzTUantkrfbd+dKtsZQ6ZsHQh0IXnFzQMAAAATAZ48dEM/AArOQyWeEbIgkUZDIAAAAJwBnj5qQz8ABxWYCCsQP5sqpprzxsa15wY2VdfRpw4gajop49stnoHapwIOCBOxJt8EYuWvkmOm3nI1FvabxW0bOKeximR74kOXKiNLrZBMJLh7KKM8URj2nEXQSLMVFBYaWgub0CgwOnwac78HNSdUe5J/H2o8ngw7XWGD2ZlknSmv8Cs7hVnLGBgXbECd1D3admPd9cZdb7iv5HYAAADiQZoiSahBbJlMCCH//XXzgKuDFYbWiVJ2EjY+o0sHX4pmiCwAJlBy+Ui7G7pfEEZoRClx1dmpM5tzhsXXPmIt2cvpx32wVuvz4Gmp3x2+Cf8X5NSoa0RbkalkUTNZV5kfpcKgCFDjuLAubZdzHa9iDFw6+Hb3HdMZd4W/y9AqQFMvgQcWK/G6WYJoTcC7/xe+SPke1SLyxaMgoZqJhNAdoTw+L59ra67uj7YbOofsIZjdzPz5jyl+n5y5rwyRzlCl4e9sgW0pnORBsYoSMoMLtloQc+e9gFVnuhitnKPb6LRdYQAAACNBnkBFFSw3/wEm801Gp7ZK323fnSrbGUF5XFz2vt59GqR3CAAAAJgBnmFqQz8AC12BKRXxE8DvIAP3UcW3ngttz3AdnaN/C9JMzls21VZq1RylVNDQrhdS6lUxeTIiLHcheD+C9ezNloB30jc24lRcn6k5GtjofvntWG5eMw4UcHnMji9il7Vx+qOcBUZWOhhPY896mPy0XMsroKmtS4cWIZzGAQoWhp4OAqVgetfyi0jeW9+MgZR053WWrTFtvQAAADtBmmRJqEFsmUwUTBD//XXzgKvPVYbWXLJ2HGY+kgMGXQpmiCwALWxzKlm1CG+SAsT0MH0hHYuHraM2vgAAAJ0BnoNqQz8Bk7BeJc45Ab7njDUS6H7zMrz3KLoOrMIAQLO7i/YIXZywXjGmIeXnus8bL3w/azDtamPbhQt0HvcAjiji03xeyEo6jRRAIbKLHbJSOFYm2AKnPiwNTFrqGnnUWkH34i5B7LZLGJThe3LtPa/BqcLIJ9tif971rBv16Q/jN3Xbk9CqDc6jRjXUpGbnbMFn5o69tCWm9a3BAAAA+0GahknhClJlMFLBH/n7NGFXfkZtap3mmR4bB2lBy/r6fdNk0d148QBLL39nbVOmcIwNUuIp/+REf7BlCY0r+ASTW0kU163AJDpPcAEveJoEkGP/CVWO/ylmVSWPUQ7a/6WTJkySxOeIfCRYXnmDWPX+iNEZwEABmpPWekdr+/PlNjHMtF8XzcSNkAd3U/31u3K5GkQZZPZyR2Zf5+PRHGX6Jybz+MUY8lfnF1vXR00iocSYGGW0XZXuM/ZAOPvwALdMMZ/WoEkHwbq1nS09JsKDuUo0ggOVoSnxfy9Gf3WkLN8mZxBKzCnQCs6TxzRDumxW1BzoybR4IwbFAAAAKAGepWpDPwGOytPMCJIOKID3vyvbE0WAz0VQ8vPvZOUHFOFXFNkwh6UAAADbQZqqSeEOiZTAgj/5+zRhVuhGcDSdspkQOwBNNePYGc5LeWdP/qR4I71fZ21TpnCAHoZhsHMlnDZGxgd0p2rJQTjbIvT9PQEt6En5eHiqwR+/hqwjCsR8VW2RyMwEOv5ZBXbnvC7iZY1+ExezKeV8m8tHXiIJRZ6g8fK+KX4hewAVsZHUs+WVKeOPNQvLg+KJrPy0zRaNLeCIQdaqe46tTYuc654o48moGwownrYahkLSD9XzO5NhHiDp6z2Pjl1eK4EEiZxuR//wPTUtPitVWO5Bch9euMkOhS+BAAAAJ0GeyEUVPDf/ASODPLGdWzSqValIKR+1TFVLsoSYj1oZyt630xB34AAAAJoBnud0Qz8AC1pjCiPxE8DvIAP3UcW3ngttz3AdnaN/C9JMzls21VZq1RylVNDQrhR63XxVXSIix3IXg/gvXszZaAd9I3NuJUXJ+pORrY6H757VhuXjMOFHB5zI4vYpe1cfqjnAVGVjoYT2PPepj8tFzLK6CprUuHFiGcxgEKFoaeDgKlYHrX8otI3lvfjIGUdOehOlIeAT8QX9AAAAjgGe6WpDPwALP5ruUXQdBeABqtgaL9ghdnLBeMaYh5ee6zxsvfD9rMO1qY9uFC3Qe9wCOKOLTfF7ISjqNFEAhsosfS1cUVibYAnRRz4WQhXUNPOotIPvxFyD2WyWMSnC9uXae1+DU4WQT7bE/73rWDfr0h/Gbuu3J6FUGfhmevWC5V40fGLv7jYPjIxcKEEAAAEPQZruSahBaJlMCCP/+fs0YVd+Rm1qneaZHhsHaUHL+vp902TR3XjxAEsvf2dtU6ZwgB6GHuB0BsMgFbV/EsPy0BO1pn94m2CgJWwFPTea5+InyPESKd52CenlFSq2DLj6Y701spjuxTsIYDTiqnbJ5u/Du2jTMyIAY9JYRvL+xhd1SCvEq+ugLCbghZK2X5q4PLbp2uZE8PPvUa0Uv4a2zhoA9I0sHxGtExbO/uzZ+iXhXSxr/Gcp9QAeJO/F1qCeKUmn03/Dnjp2j2dr/TBGhfRAUHrrTrwZuTw3fBP6SYW3AHeNSCz9VLEI8fCTINZmJby7rpXoGo93S7E2GsJ4rPzwxmbX8Mg4H/CIp9mDUAAAAKJBnwxFESw3/wEm801Gp7ZK0C4wp8iDYFfkGS7CYAQnRPsPabdNiVs8/VRfU+1/oxso7hA435UxSOxNHz+9Zraf+KntGIs2wqKLOw50w1lRQYluP35G3gOKe+tsjseOlno2CfiUhLXUepPlP+7/lOaxBLqOLbE72ilH9zfzBdoWj+K6BEFHklyvgX6ADAfFHG0YsMf0f3xu+F3ajg81HTLiUOAAAAAVAZ8rdEM/AArOQyLaas6R+ATAp+G7AAAAFQGfLWpDPwAGwH/GGOy/8fJly1J5/wAAAEBBmzJJqEFsmUwII//5+zRgmCWsdQWIjCkYsuAJprx7AznJbyzp/9SPBHer7O2qdM4QA9DMK20N/2Sxpbz4/oZPAAAAGEGfUEUVLDf/ASbzTUantkrQLjCgM/NRIAAAAKEBn290Qz8AA4nBPa74K5QuAD4MIpvUmWwzLb+1XcAj/gwMvK2on3u5TBTQ4F2k7iw0/S9IImt50oYT8q1Z44miW6Zc8gvDK0HgKO2QHjGHBKFjIa8GRpWEXVyVRYBsgj8As43KFU9kITAkYkD6WzYrMDTgemN2WP3IkYsyyUW2GWskTbHN2QISLXusL3ET2MfUWU82LMMuHwvmQwp5iOy3YAAAAAwBn3FqQz8AA4rKlEkAAAAzQZt2SahBbJlMCCH//XXzgKuDFYbWiVJ2EjY+o0sHX4pmiCwAJlBy+Ui6/834QSZ8oNZgAAAAGEGflEUVLDf/ASbzTUantkrQLjCgNb2NwAAAAAwBn7N0Qz8AA4nAlEkAAAAMAZ+1akM/AAOKypRIAAAAM0GbukmoQWyZTAgh//1184Crz1WG1lyydhxmPpIDBl0KZogsAC1scypZtP/N9MscfrLOYQAAALdBn9hFFSw3/wEm801Gp7ZK0C4wo3/6+QZLltABdBoQRSq0ocC7uFl98T4FtdUJ6b+3wX67ugG5ZpJwvAL9QLyQE14znxsVRtJDJ8kkK/ajAoeeFQz4xakrOT26Y627JolmxEU6rWbXsnKjNxGizeb1g7ankeN6cmJnKwkGmAtX3uLw8HtRqVIC+v5yWh0bX6RnxIEzbvlvMVC304cI4Clgg/gMCvATnX7XruvOeuB3z/19ghiFJcEAAAAWAZ/3dEM/AAbCOAKw2+S4Ww+8AhG/wAAAABoBn/lqQz8ABrxdP+thWHGZaWkqzqj1WQWCwQAAADNBm/5JqEFsmUwIIf/9dfOAq4MVhtaJUnYSNj6jSwdfimaILAAmUHL5SLr/zfhBJnyg1mAAAAAYQZ4cRRUsN/8BJvNNRqe2StAuMKAz81EhAAAAoQGeO3RDPwADicE9rvgrlC4APgwim9SZbDMtv7VdwCP+DAy8raife7lMFNDgXaTuLDT9L0gia3nShhPyrVnjiaJbplzyC8MrQeAo7ZAeMYcEoWMhrwZGlYRdXJVFgGyCPwCzjcoVT2QhMCRiQPpbNiswNOB6Y3ZY/ciRizLJRbYZayRNsc3ZAhIte6wvcRPYx9RZTzYswy4fC+ZDCnmI7LdhAAAADAGePWpDPwADisqUSAAAAO9BmiJJqEFsmUwIIf/9dfOAq89VhtZcsnYcZj6SAwZdCmaILAAtbHMqWbUbkgSB8UIhS4+4qBGsU6HsqcyDWQwPts+d8Fz7tpTjw+sAaSfVdOXpH45NLtT8GJ7n3opHCFuGSIo8rM248VlKvzg/0+8ezQfA3fQZ3dLLLMNKHV2aTp1i9rtzmklU2YfhMSJ/qWzm/eWENVndyKpNipfB6WmHE6Lq/GRmfGZDQGNGhYFseAVuz5Z1tDEJ04Oo3bV5gJCs+xZ4A6aCvspMa0hPU8ThN3xnnKHBrZYWayu9lmcBZ6oQyzonupDLgpRfRscqgAAAACBBnkBFFSw3/wEm801Gp7ZK0C4wo+ZFmcTudmQqnhtfgQAAAJ4Bnn90Qz8ABxYblFfl4NuFeCufFAAmd5FN6ky2GZbf2q7gEf8GBl5W1E+93KYKaHAu0ncQoGqXpBdSTnf5hPyrVnjiaJbplzyC8MrQeAo7ZAeMYcEoWMhrsGfr73ANkpLaDU9yhVPZCEwJGJA+ls2KzA04HpjdliM2JGLMylFthlrJE2xzdkCEi17rC9xE9jH1FlPNizDLh8L5kSsvRQAAAJ0BnmFqQz8ABxa8WgMwk7C0qLrgroQcAJbeRTepMthmW39qu4BH/BgZeVtRPvdymCmhwLtJ3IeBql6QZFk53+YT8q1Z44miW6Zc8gvDK0HgKO2QHjGHBKFjIa905GIDZAy4BZxuUKp7IQmBIxIH0tmxWYGnA9MbstZ2awyD104urx2bG22ObsgQkWvdYXuInsY+osp5sWYZcPhfMhWXAAAAO0GaZEmoQWyZTBRMEP/9dfOAq4MVhtaJUnYSNj6jSwdfimaILAAmUHL5SLsIb5ICxPQwfSEeDgqfqgtfAAAAqAGeg2pDPwGTsF4lzjkBoGoQRBMKaZ8oQCvNlVNbgy+InlHMpFwlbLZ6B2qcCDggXz/2o6MbGg8MON+TzgSAtCfmlqa3ngjYwVPFT2jEWgIsosZsOV61EMUGJY/CB7MOyN2CqMc6Kb3qA5O1xh9BkuPUnyppzxBGuJI3COLbE72ik7WTW3BdoEigFdAh2CxYcr4Q6b7spdfbnY702lG54rvhd2op+LDMXQAAADRBmoZJ4QpSZTBSwR/9ds0YJhutuWD3LPFanijP1FXJcVpvEPwSbwR1RZ1Tn+zRjiH0Zh9BAAAAqAGepWpDPwGOysVfIypDo0ByUX/jJuIngukgBNM7RaHZGtue4Ds7Rv4XpJmctm2qrNWqOUqpoaFboBm678nOT2VU1xlDryZERZAuOjNgwF5P2zY3033zjxqCkgcLIVwISsXiANDoPtieJ1tUC4bQlwuw2c0leQxvGCy/zBri0pVa8jP8EPHK1dCs/wNhZLxg2HB7fRcAtBqNaA0fq59hJuBcYZM2cQTLkwAAAOlBmqpJ4Q6JlMCCH/1184CrgxWG1olSdhI2PqNLB1+KZogsACZQcvlIuxvtSJFjYP7uO+gecEjJs48y4VeWRG+ZtUfyWga+af4k3JAX24Gm4Q0DTP+eyObcIXC3AymYAf51Ybr6eLAMyMJfOxEKTpM+5bunx/u/zJ72u2meMgZsZ+xg4WNckESk/GouiOEKiiwMW3GJ9QhMm2bKguaunSRw9afJU7x0TWN6GrbnUbpXgvsUwz7xHDzT1/5grbuAakxAvAKFUdQZaU9V6icsUVzCYgLS8WW8VLKwFEUhCHIX2def3K8rVZPpMQAAACBBnshFFTw3/wEjgzyxnVs0qlWpSCZYlhk8RiiCf6UM3AAAAJ4Bnud0Qz8ABxN9XZBgQ9lIXBXPigATO8im9SZbDMtv7VdwCP+DAy8raife7lMFNDgXaTuMpG9L0gupJzv8wn5VqzxxNEt0y55BeGVoPAUdsgPGMOCULGQ13Qg41LANkpLaDU9yhVPZCEwJGJA+ls2KzA04HpjdliKSJGLMylFthlrJE2xzdkCEi17rC9xE9jH1FlPNizDLh8L5kTV2agAAAAwBnulqQz8AA4rKlEkAAAC8QZruSahBaJlMCCH//XXzgKvPVYbWXLJ2HGY+kgMGXQpmiCwALWxzKlm0/830zhM/+I0a6T3jAdmgMoG2Kr3+/RarzuySolxdVj9OgyBJKgep4ai1s+lo8h21hTz+95bNWh4Li2v4JUpq/P3rRzOvtRJSm1iMutAAoUe+Y3ni1y+N7obtQJGMW2//rdfirkJGW6fK1fim5myUwO970QT70H2eEsgLvAkHJLB3LUbW0vRVKz+27f/1R4v30UIAAAAhQZ8MRREsN/8BJvNNRqe2StAuMKA1wQyAxvSRNnf1Hp2AAAAAnAGfK3RDPwADicE9rvgrlC4APgwim9SZbDMtv7VdwCP+DAy8raife7lMFNDgXaTuLDT9L0gia3nShhPyrVnjiaJbplzyC8MrQeAo7ZAeMYcEoWMhrv3Zm8wpgGyCPwCzjcoVT2QhMCRiQPpbNiswNOB6Y3ZZys1hkHrnxdXjs2Ntsc3ZAhIte6wvcRPYx9RZTzYswy4fC+ZExa5lgQAAABkBny1qQz8AA4rLXQL6w8dNnIPfphAxlR9hAAAAvUGbMkmoQWyZTAgh//1184CrgxWG1olSdhI2PqNLB1+KZogsACZQcvlIuyN6T/+NajwgURc//Ukjv8cMPz0xNMLLT8bdMIa8he9lmpK8rlggJQLVJzzx6l5iEsJES/n91tH/QgGgRRdukY0spkjlqAlYm2w4agGXYLr20uOFqnwCk/PMTgohzfy/FWfg+P9RmVJwXpWXxLa1aTUc07P5OpxPknkrX5m6MGnQI9TyDSU4GyF0TKDH0XooDmnK3QAAAKdBn1BFFSw3/wEm801Gp7ZK0C4wpS4yVFfEAJpouBtwHufkzhpqzUsE6PE7WsUmXAPJmBj4ns5BW7+NsMuLxQN3p29udfPbAIXzxADhH/GSJT6FDWDpOe5ey7XNtBMVT6udNfbrrjjnbeoLU1kaOIoLllUAOKn5DzXVAXC8ma04iePAqtMl/Fsm+rTnzsqrSvmf/1oSDpiUvjVgD6M58q0uMMPE4aTUgAAAAJwBn290Qz8AB5obd5rxE8EYgAJ1naLQ7I1tz3AdnaN/C9JMzls21VZq1RylVNDQrhZBzdQDqrpERY7kMAfwXr2ZstAO+kbm3EqLk/UnI1sdD989qw3LxmHCjg85kcXsUvauP1RzgKjKx0MJ7HnvUx+Wi5lldBU1qXDixDOYwCFC0NPBwFSsD1r3AIOdbb34yBlHTn2GGvCS7bGgQLgAAACnAZ9xakM/AAguu6veAavnR2HWNCZ9/ORbyBuGmZ1tP/CUmKtgKbd6Lt+z0sQx8utLtpsjU9fP6hHX/iEYF+8izLGKkQZtQ4oZjtvAdRUayJK1WQ4jzlcUbzOWg4Xa4Ofm5GVhSRuYHvmLbYPQyiOQoAWEIcLIPiM+RjdCCr4t2DCmJIST+DXWKQOxAQcFw/9GpIGfBjiJHtxM3jxOgyCUFqhcH9N/uYEAAADIQZt2SahBbJlMCCH//XXzgKvPVYbWXLJ2HGY+kgMGXQpmiCwALWVtl157P+E9e/8B8Phd0geS/Bc/xx+3A+GnldMlK1UF0SnyL/3Qm+t0xoPU/qz7PaFV8h+Gyyr6ZZmy2a8WxEBgod7XY3Gmi4yOh7Y37HWsBD2aZmL4dZZC14CxagtxVzJeJdxnk9L8blsrOS0vt57xoqNKXk+ToukoB9/hJkqgQjN8ckZrTlUuVRc4v3XUBbIWvJxY45yC9O24mpgc4L2SdUAAAAAjQZ+URRUsN/8BJvNNRqe2StAuMKnlkfTC8/NJ4cTKQVbGzbgAAACjAZ+zdEM/AAzkcAVbfHSQDV86Ow6xoTQCByLeQRNUzOtp/4SkxVsBTbvRdv2eliGPl1pdtNkcLSmD4nwjAEVyLMsYqRBm1DihmusvA6io1kPVqshxHnK4o3mctBwu1wc/NyMrCkjcwPfMW2wehlEchQAsIQ4WQfEZ8jG6EFXxbsGFMSQkn8GusUgdiAg4Lh/6NSQM+DHESPbiZvHidBjuCJ3qQQAAAJUBn7VqQz8ADOWFPPmlvNAghC9AAIg7rE0q6MmIidmvrFuHtMhegdhzlWXxjrRLMGMyDF3G8Xwu+6PM7YMi7RgVSH/MivEZFyMZcVFXF4qAr3ZzQ3Zf2vrP3UI50Qo7a4KGsP+MztL+eevlij06rd0fgXZTJzdPCOGUdLyhzMFQ4K94HGUiACkKp9a8ffz2BTl02qXUgAAAACtBm7hJqEFsmUwUTBD//XXzgKuDFYbWiVJ2EjY+o0sHX4pmiCwAJkltQjoXAAAApQGf12pDPwGTsF4lzjkBoGoQW7F24B4ChvJRRwWn7yXBP5xtd7rE0q6MmIidmvrFuHtMhegderifpnuUplSUnJzK0ZyEb+x3R5nbBkXaEC1z/+ZFcY6LkYxshaaF4HAr3WOyGy/aaEvvFOBxEIG9j3kx9Ohtz8yhoieK1K+aGzENEp09MLc3aO4Ot44fs9Gb8RI5z18wCsP4+SKILdXJ7D8WPUx1gQAAALdBm9xJ4QpSZTAgh//9dfOAq89VhtZcsnYcZj6SAwZdCmaID+8LYjHDK/z2QDxrnZaQ8mkeFmEErYlLqGH8QW9AJ67CN0Ciqu1IM18BJugh4uaXINuLalGemWzgd3mgpJbLVTT6p2XajHdld8OiCf8eZWY4CDyojDsDW2jfc5iz6QUb+ofszOVGpSREVn1UEZKDNLgf2TSIPME5wRZdozvJoOI512JJeBvnYNRX90Uf+xD94sDQtIAAAAAnQZ/6RTRMN/8BJuKz1csHn0o9tF6yh9kgFyeMcSlbvPYzY2B5mgOBAAAAjgGeGXRDPwAgvgyLaZhLkIzRMHBpYAEN6gaL9gx9HQtFaoizgkKE3dOo+S7hrivFbjw3GP5Ow2fFvVtRSx2HZKVQMFETvFHvLWQVuxPpmE2QLAmubl28zdm62WZhm7tzjsY2v5ta57GAWSmg+LwaXfVz4ShP1JFHLrkbxq4yTerJ9CFOvsRB6Kvp6LunFSAAAACeAZ4bakM/ACLVcgAlq058Waxa9Xp8+rwrZOBsyHzhbZJWzuALxyXaHAoKNdXvZ0ZqdqnUXt2MaK9Hw1eXquxFClAUwhiZSPSmPb4SmgE/IcTjixcdKgvEXcNvl994QNf4avk2oOZS+BaDNVQ4/+nPJttG83Zx4W3RIilAojfizqtH2+5yKYJRRUxIQRe8Tg9vr7GHdnYwTxIfrrHd1CUAAACuQZoASahBaJlMCH///qeEACx7+0LCGgu8EKACX1pXFRWVdsc2zL0zlXZRcwq4KXZ3+10qtXfWIf0nJzFCNjUbURaU2eSBu5sYifwA5w/4swWdUz70DX/DnhNrg6FhXLWwuDI7uTJCiPfiJeDBdSOqS/59FwQf4U/rP5NeNglXbdyF8dXaD/D4glrrCN5TuDX0JoKAAgGvP7WNiYkUJFsw2LWiaujk2ioEFzwKpN3BAAAAKUGePkURLDf/ASbzTUantkrQLsI3BZFNzzjXjccYiPT6cAfeABdCGAXEAAAAoAGeXXRDPwAjvbPL/jmoEkALeO2ARNYter0+fXAmv1MNaHHEC2HdrvOcHeF1GzHBl9pcb1EXaqJWptmMSRhQjcvxlHu+1zZyxxIkPXbov2uM6WZ2wPu3JdEIaJ/ktyuUrXUBbzrC/58vxG5WyGsDn4gDS7fqmgkDNspNRTZ8CS3eeqczuFzpnzJkfioIkkDKK+UoEEIW113x6VLFG3VHv1wAAAAcAZ5fakM/ABdLCvv4plUAEuU1kECzUTC9zmUtTwAAAKpBmkRJqEFsmUwIb//+jLAArgaFDPF+sraa36guQ85v+DAyzABO3qUDbgPc/JnDTVmqttzA+TeTanPVZHiotg8RPMc+N+ml0o5UZ/+y2lvS3VQQS+l1A5xvo6Pol9jyargw+3nnDV+xWwz93aXZmWRHKkcdbDQdidMSECYVaSl9gNsm5DAqLRuFUrnjeclVku08KiXLOjcmtJ1h69HvZHhaKKwiCg3gqMwm9AAAAC9BnmJFFSw3/wEm801Gp7ZK0C7CSswP+CJPn/0aZjmvhU/di/XcEuOkg8QwT/hFsQAAAJwBnoF0Qz8AI65scrftYWHM8pgPPC91yi1YANixuRiFtdr6xbks6my0W37x7izMsABucc3HeCnyQM4cIs571/vrVBlSeVzbWGQ72hYMhDdd8+LZMTaY20dT90A5XMbjXd5AMwNLZF+DNJaBcZyf00B+srpZv1HLDyxS1EFuoR+0OZflLDAMpZ3iADzN+bXACrLFmcAZ7XHmFK7QW0AAAACWAZ6DakM/AAzjoZUea3hMJQBEx1otDyVF9ulwDHyxb5Fu3CVgzfz4ET8otm2lYMypvonEox152Z+B2akV68xJkofIt8Lp51qHku/OTU2w/NDLOt1Qd/RtnfmRB0Ac24u0Q4p/7luEyljBOBUtlpGUQ/qnJaPW8TfGGbh9vLExcvV3TxR/lcujy88csOYiUT3wBvhgT1rhAAAAqkGahUmoQWyZTAhn//44QAGRnnf8U56f50U2FTNUrrnZCTjbrAB+K5JwlA2o6iyfWGQpGtJDzqVzhUBwnonabjRM3ylcp27eC7MnMn3lkv88fXJzOr9O0Qkb6WhRoPEtHNxxMa3YOxPNnd1RLFFTWQVHOMub28NPK7ppF2KdPzyiKMIC8m4VD3yVXLXzfyt090cFhmLqKiZgfZhNHazXgr+YxnAOuQX0exxlAAAHyW1vb3YAAABsbXZoZAAAAAAAAAAAAAAAAAAAA+gAACE0AAEAAAEAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAbzdHJhawAAAFx0a2hkAAAAAwAAAAAAAAAAAAAAAQAAAAAAACE0AAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAEgAAABIAAAAAAAJGVkdHMAAAAcZWxzdAAAAAAAAAABAAAhNAAACAAAAQAAAAAGa21kaWEAAAAgbWRoZAAAAAAAAAAAAAAAAAAAMAAAAZgAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5kbGVyAAAABhZtaW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAAXWc3RibAAAAJZzdHNkAAAAAAAAAAEAAACGYXZjMQAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAEgASAASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABj//wAAADBhdmNDAWQADP/hABhnZAAMrNlBIJaEAAADAAQAAAMAYDxQplgBAAVo6+csiwAAABhzdHRzAAAAAAAAAAEAAABmAAAEAAAAABRzdHNzAAAAAAAAAAEAAAABAAADMGN0dHMAAAAAAAAAZAAAAAEAAAgAAAAAAQAADAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAMAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAQAAAAAAIAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAUAAAAAAEAAAgAAAAAAQAAAAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAAAQAAEAAAAAACAAAEAAAAAAEAAAwAAAAAAQAABAAAAAABAAAMAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAUAAAAAAEAAAgAAAAAAQAAAAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAUAAAAAAEAAAgAAAAAAQAAAAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAMAAAAAAEAAAQAAAAAAQAADAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAUAAAAAAEAAAgAAAAAAQAAAAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAAAQAADAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAUAAAAAAEAAAgAAAAAAQAAAAAAAAABAAAEAAAAAAEAAAgAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAABmAAAAAQAAAaxzdHN6AAAAAAAAAAAAAABmAAASAQAAAasAAABEAAABEQAAALUAAAApAAAAIgAAAFcAAAApAAAAFwAAABkAAAA6AAAAlwAAAMYAAAAlAAAAlAAAABMAAADCAAAAqwAAAC8AAAC5AAAAJAAAAJwAAACTAAAAiQAAALsAAAAbAAAAGgAAAEYAAAAmAAAAFwAAAKAAAADmAAAAJwAAAJwAAAA/AAAAoQAAAP8AAAAsAAAA3wAAACsAAACeAAAAkgAAARMAAACmAAAAGQAAABkAAABEAAAAHAAAAKUAAAAQAAAANwAAABwAAAAQAAAAEAAAADcAAAC7AAAAGgAAAB4AAAA3AAAAHAAAAKUAAAAQAAAA8wAAACQAAACiAAAAoQAAAD8AAACsAAAAOAAAAKwAAADtAAAAJAAAAKIAAAAQAAAAwAAAACUAAACgAAAAHQAAAMEAAACrAAAAoAAAAKsAAADMAAAAJwAAAKcAAACZAAAALwAAAKkAAAC7AAAAKwAAAJIAAACiAAAAsgAAAC0AAACkAAAAIAAAAK4AAAAzAAAAoAAAAJoAAACuAAAAFHN0Y28AAAAAAAAAAQAAADAAAABidWR0YQAAAFptZXRhAAAAAAAAACFoZGxyAAAAAAAAAABtZGlyYXBwbAAAAAAAAAAAAAAAAC1pbHN0AAAAJal0b28AAAAdZGF0YQAAAAEAAAAATGF2ZjU3LjgzLjEwMA==">

---
# Section 3: The Bellman Equation

*Time estimate: ~15mins*

```
# @title Video 3: The Bellman Equation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Lv411E7CB", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"cLCoNBmYUns", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 3: The Bellman Equation')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

In this tutorial we focus mainly on **value based methods**, where agents maintain a value for all state-action pairs and use those estimates to choose actions that maximize that **value** (instead of maintaining a policy directly, like in **policy gradient methods**).

We represent the **action-value function** (otherwise known as $\color{green}Q$-function associated with following/employing a policy $\pi$ in a given MDP as:

\begin{equation}
\color{green}Q^{\color{blue}{\pi}}(\color{red}{s},\color{blue}{a}) = \mathbb{E}_{\tau \sim P^{\color{blue}{\pi}}} \left[ \sum_t \gamma^t \color{green}{r_t}| s_0=\color{red}s,a_0=\color{blue}{a} \right]
\end{equation}

where $\tau = \{\color{red}{s_0}, \color{blue}{a_0}, \color{green}{r_0}, \color{red}{s_1}, \color{blue}{a_1}, \color{green}{r_1}, \cdots \}$

Recall that efficient value estimations are based on the famous **_Bellman Expectation Equation_**:

\begin{equation}
\color{green}Q^\color{blue}{\pi}(\color{red}{s},\color{blue}{a}) =    \sum_{\color{red}{s'}\in \color{red}{\mathcal{S}}}
\color{purple}P(\color{red}{s'} |\color{red}{s},\color{blue}{a})
\left(
  \color{green}{R}(\color{red}{s},\color{blue}{a}, \color{red}{s'})
  + \gamma \color{green}V^\color{blue}{\pi}(\color{red}{s'})
  \right)
\end{equation}

where $\color{green}V^\color{blue}{\pi}$ is the expected $\color{green}Q^\color{blue}{\pi}$ value for a particular state, i.e. $\color{green}V^\color{blue}{\pi}(\color{red}{s}) = \sum_{\color{blue}{a} \in \color{blue}{\mathcal{A}}} \color{blue}{\pi}(\color{blue}{a} |\color{red}{s}) \color{green}Q^\color{blue}{\pi}(\color{red}{s},\color{blue}{a})$.

---
# Section 4: Policy Evaluation

*Time estimate: ~30mins*

```
# @title Video 4: Policy Evaluation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV15f4y157zA", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"HAxR4SuaZs4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 4: Policy Evaluation')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

### Lecture footnotes:

**Episodic vs non-episodic environments:** Up until now, we've mainly been talking about episodic environments, or environments that terminate and reset (resampled) after a finite number of steps. However, there are also *non-episodic* environments, in which an agent cannot count on the environment resetting. Thus, they are forced to learn in a *continual* fashion.

**Policy iteration vs value iteration:** Compare the two equations below, noting that the only difference is that in value iteration, the second sum is replaced by a max.

*Policy iteration (using Bellman expectation equation)*
\begin{equation}
\color{green}Q_\color{green}{k}(\color{red}{s},\color{blue}{a}) \leftarrow  \color{green}{R}(\color{red}{s},\color{blue}{a}) +\gamma \sum_{\color{red}{s'}\in \color{red}{\mathcal{S}}}
\color{purple}P(\color{red}{s'} |\color{red}{s},\color{blue}{a})
\sum_{\color{blue}{a'} \in \color{blue}{\mathcal{A}}} \color{blue}{\pi_{k-1}}(\color{blue}{a'} |\color{red}{s'}) \color{green}{Q_{k-1}}(\color{red}{s'},\color{blue}{a'})
\end{equation}

*Value iteration (using Bellman optimality equation)*
\begin{equation}
\color{green}Q_\color{green}{k}(\color{red}{s},\color{blue}{a}) \leftarrow  \color{green}{R}(\color{red}{s},\color{blue}{a}) +\gamma \sum_{\color{red}{s'}\in \color{red}{\mathcal{S}}}
\color{purple}P(\color{red}{s'} |\color{red}{s},\color{blue}{a})
\max_{\color{blue}{a'}} \color{green}{Q_{k-1}}(\color{red}{s'},\color{blue}{a'})
\end{equation}

### Coding Exercise 4.1 Policy Evaluation Agent

Tabular agents implement a function `q_values()` returning a matrix of Q values
of shape: (`number_of_states`, `number_of_actions`)

In this section, we will implement a `PolicyEvalAgent` as an ACME actor: given an `evaluation_policy` $\pi_e$ and a `behaviour_policy` $\pi_b$, it will use the `behaviour_policy` to choose actions, and it will use the corresponding trajectory data to evaluate the `evaluation_policy` (i.e. compute the Q-values as if you were following the `evaluation_policy`).

Algorithm:

**Initialize** $Q(\color{red}{s}, \color{blue}{a})$ for all $\color{red}{s}$ ∈ $\mathcal{\color{red}S}$ and $\color{blue}a$ ∈ $\mathcal{\color{blue}A}(\color{red}s)$

**Loop forever**:

1. $\color{red}{s} \gets{}$current (nonterminal) state

2. $\color{blue}{a} \gets{} \text{behaviour_policy }\pi_b(\color{red}s)$

3. Take action $\color{blue}{a}$; observe resulting reward $\color{green}{r}$, discount $\gamma$, and state, $\color{red}{s'}$

4. Compute TD-error: $\delta = \color{green}R + \gamma Q(\color{red}{s'}, \underbrace{\pi_e(\color{red}{s'}}_{\color{blue}{a'}})) − Q(\color{red}s, \color{blue}a)$

4. Update Q-value with a small $\alpha$ step: $Q(\color{red}s, \color{blue}a) \gets Q(\color{red}s, \color{blue}a) + \alpha \delta$

We will use a uniform `random policy` as our `evaluation policy` here, but you could replace this with any policy you want, such as a greedy one.

```
# Uniform random policy
def random_policy(q):
  return np.random.randint(4)
```

```
class PolicyEvalAgent(acme.Actor):

  def __init__(self, environment_spec, evaluated_policy,
               behaviour_policy=random_policy, step_size=0.1):

    self._state = None
    # Get number of states and actions from the environment spec.
    self._number_of_states = environment_spec.observations.num_values
    self._number_of_actions = environment_spec.actions.num_values
    self._step_size = step_size
    self._behaviour_policy = behaviour_policy
    self._evaluated_policy = evaluated_policy
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    #raise NotImplementedError("Initialize your Q-values!")
    #################################################
    # TODO Initialize the Q-values to be all zeros.
    # (Note: can also be random, but we use zeros here for reproducibility)
    # HINT: This is a table of state and action pairs, so needs to be a 2-D
    #   array. See the reference for how to create this in numpy:
    #   https://numpy.org/doc/stable/reference/generated/numpy.zeros.html
    self._q = np.zeros((self._number_of_states, self._number_of_actions))
    self._action = None
    self._next_state = None

  @property
  def q_values(self):
    # return the Q values
    return self._q

  def select_action(self, observation):
    # Select an action
    return self._behaviour_policy(self._q[observation])

  def observe_first(self, timestep):
    self._state = timestep.observation

  def observe(self, action, next_timestep):
    s = self._state
    a = action
    r = next_timestep.reward
    g = next_timestep.discount
    next_s = next_timestep.observation

    # Compute TD-Error.
    self._action = a
    self._next_state = next_s
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    # raise NotImplementedError("Need to select the next action")
    #################################################
    # TODO Select the next action from the evaluation policy
    # HINT: Refer to step 4 of the algorithm above.
    next_a = self._evaluated_policy(self._q[next_s])
    self._td_error = r + g * self._q[next_s, next_a] - self._q[s, a]

  def update(self):
    # Updates
    s = self._state
    a = self._action
    # Q-value table update.
    self._q[s, a] += self._step_size * self._td_error
    # Update the state
    self._state = self._next_state

# add event to airtable
atform.add_event('Coding Exercise 4.1 Policy Evaluation Agent')
```

```
# to_remove solution
class PolicyEvalAgent(acme.Actor):

  def __init__(self, environment_spec, evaluated_policy,
               behaviour_policy=random_policy, step_size=0.1):

    self._state = None
    # Get number of states and actions from the environment spec.
    self._number_of_states = environment_spec.observations.num_values
    self._number_of_actions = environment_spec.actions.num_values
    self._step_size = step_size
    self._behaviour_policy = behaviour_policy
    self._evaluated_policy = evaluated_policy
    # TODO Initialize the Q-values to be all zeros.
    # (Note: can also be random, but we use zeros here for reproducibility)
    # HINT: This is a table of state and action pairs, so needs to be a 2-D
    #   array. See the reference for how to create this in numpy:
    #   https://numpy.org/doc/stable/reference/generated/numpy.zeros.html
    self._q = np.zeros((self._number_of_states, self._number_of_actions))
    self._action = None
    self._next_state = None

  @property
  def q_values(self):
    # return the Q values
    return self._q

  def select_action(self, observation):
    # Select an action
    return self._behaviour_policy(self._q[observation])

  def observe_first(self, timestep):
    self._state = timestep.observation

  def observe(self, action, next_timestep):
    s = self._state
    a = action
    r = next_timestep.reward
    g = next_timestep.discount
    next_s = next_timestep.observation

    # Compute TD-Error.
    self._action = a
    self._next_state = next_s
    # TODO Select the next action from the evaluation policy
    # HINT: Refer to step 4 of the algorithm above.
    next_a = self._evaluated_policy(self._q[next_s])
    self._td_error = r + g * self._q[next_s, next_a] - self._q[s, a]

  def update(self):
    # Updates
    s = self._state
    a = self._action
    # Q-value table update.
    self._q[s, a] += self._step_size * self._td_error
    # Update the state
    self._state = self._next_state

# add event to airtable
atform.add_event('Coding Exercise 4.1 Policy Evaluation Agent')
```

```
# @title Perform policy evaluation { form-width: "30%" }
# @markdown Here you can visualize the state value and action-value functions for the "simple" task.
num_steps = 1e3

# Create the environment
grid = build_gridworld_task(task='simple')
environment, environment_spec = setup_environment(grid)

# Create the policy evaluation agent to evaluate a random policy.
agent = PolicyEvalAgent(environment_spec, evaluated_policy=random_policy)

# run experiment and get the value functions from agent
returns = run_loop(environment=environment, agent=agent, num_steps=int(num_steps))

# get the q-values
q = agent.q_values.reshape(grid._layout.shape + (4, ))

# visualize value functions
print('AFTER {} STEPS ...'.format(num_steps))
plot_action_values(q, epsilon=1.)
```

---
# Section 5: Tabular Value-Based Model-Free Learning

*Time estimate: ~50mins*

```
# @title Video 5: Model-Free Learning
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1iU4y1n7M6", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"Y4TweUYnexU", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 5: Model-Free Learning')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

### Lecture footnotes:

**On-policy (SARSA) vs off-policy (Q-learning) TD control:** Compare the two equations below and see that the only difference is that for Q-learning, the update is performed assuming that a greedy policy is followed, which is not the one used to collect the data, hence the name *off-policy*.

*SARSA*
\begin{equation}
\color{green}Q(\color{red}{s},\color{blue}{a}) \leftarrow  \color{green}Q(\color{red}{s},\color{blue}{a}) +\alpha(\color{green}{r} + \gamma\color{green}{Q}(\color{red}{s'},\color{blue}{a'}) - \color{green}{Q}(\color{red}{s},\color{blue}{a}))
\end{equation}

*Q-learning*
\begin{equation}
\color{green}Q(\color{red}{s},\color{blue}{a}) \leftarrow  \color{green}Q(\color{red}{s},\color{blue}{a}) +\alpha(\color{green}{r} + \gamma\max_{\color{blue}{a'}} \color{green}{Q}(\color{red}{s'},\color{blue}{a'}) - \color{green}{Q}(\color{red}{s},\color{blue}{a}))
\end{equation}

## Section 5.1: On-policy control: SARSA Agent
In this section, we are focusing on control RL algorithms, which perform the **evaluation** and **improvement** of the policy synchronously. That is, the policy that is being evaluated improves as the agent is using it to interact with the environent.

The first algorithm we are going to be looking at is SARSA. This is an **on-policy algorithm** -- i.e: the data collection is done by leveraging the policy we're trying to optimize.

As discussed during lectures, a greedy policy with respect to a given $\color{Green}Q$ fails to explore the environment as needed; we will use instead an $\epsilon$-greedy policy with respect to $\color{Green}Q$.

### SARSA Algorithm

**Input:**
- $\epsilon \in (0, 1)$ the probability of taking a random action , and
- $\alpha > 0$ the step size, also known as learning rate.

**Initialize:** $\color{green}Q(\color{red}{s}, \color{blue}{a})$ for all $\color{red}{s}$ ∈ $\mathcal{\color{red}S}$ and $\color{blue}a$ ∈ $\mathcal{\color{blue}A}$

**Loop forever:**

1. Get $\color{red}s \gets{}$current (non-terminal) state

2. Select $\color{blue}a \gets{} \text{epsilon_greedy}(\color{green}Q(\color{red}s, \cdot))$

3. Step in the environment by passing the selected action $\color{blue}a$

4. Observe resulting reward $\color{green}r$, discount $\gamma$, and state $\color{red}{s'}$

5. Compute TD error: $\Delta \color{green}Q \gets
\color{green}r + \gamma \color{green}Q(\color{red}{s'}, \color{blue}{a'}) − \color{green}Q(\color{red}s, \color{blue}a)$,
 where $\color{blue}{a'} \gets \text{epsilon_greedy}(\color{green}Q(\color{red}{s'}, \cdot))$

5. Update $\color{green}Q(\color{red}s, \color{blue}a) \gets \color{green}Q(\color{red}s, \color{blue}a) + \alpha \Delta \color{green}Q$

### Coding Exercise 5.1: Implement $\epsilon$-greedy
Below you will find incomplete code for sampling from an $\epsilon$-greedy policy, to be used later when we implement an agent that learns values according to the SARSA algorithm.

```
def epsilon_greedy(
    q_values_at_s: np.ndarray,  # Q-values in state s: Q(s, a).
    epsilon: float = 0.1  # Probability of taking a random action.
    ):
  """Return an epsilon-greedy action sample."""
  #################################################
  # Fill in missing code below (...),
  # then remove or comment the line below to test your implementation
  # raise NotImplementedError("Student exercise: complete epsilon greedy policy function")
  #################################################
  # TODO generate a uniform random number and compare it to epsilon to decide if
  # the action should be greedy or not
  # HINT: Use np.random.random() to generate a random float from 0 to 1.
  if np.random.random() > epsilon:
    #TODO Greedy: Pick action with the largest Q-value.
    action = np.argmax(q_values_at_s)
  else:
    # Get the number of actions from the size of the given vector of Q-values.
    num_actions = np.array(q_values_at_s).shape[-1]
    # TODO else return a random action
    # HINT: Use np.random.randint() to generate a random integer.
    action = np.random.randint(num_actions)

  return action

# add event to airtable
atform.add_event('Coding Exercise 5.1: Implement  epsilon-greedy')
```

```
# to_remove solution
def epsilon_greedy(
    q_values_at_s: np.ndarray,  # Q-values in state s: Q(s, a).
    epsilon: float = 0.1  # Probability of taking a random action.
    ):
  """Return an epsilon-greedy action sample."""
  # TODO generate a uniform random number and compare it to epsilon to decide if
  # the action should be greedy or not
  # HINT: Use np.random.random() to generate a random float from 0 to 1.
  if epsilon < np.random.random():
    #TODO Greedy: Pick action with the largest Q-value.
    action = np.argmax(q_values_at_s)
  else:
    # Get the number of actions from the size of the given vector of Q-values.
    num_actions = np.array(q_values_at_s).shape[-1]
    # TODO else return a random action
    # HINT: Use np.random.randint() to generate a random integer.
    action = np.random.randint(num_actions)

  return action

# add event to airtable
atform.add_event('Coding Exercise 5.1: Implement  epsilon-greedy')
```

```
# @title Sample action from $\epsilon$-greedy { form-width: "30%" }
# @markdown With $\epsilon=0.5$, you should see that about half the time, you will get back the optimal
# @markdown action 3, but half the time, it will be random.

# Create fake q-values
q_values = np.array([0, 0, 0, 1])

# Set epsilon = 0.5
epsilon = 0.5
action = epsilon_greedy(q_values, epsilon=epsilon)
print(action)
```

    3

### Coding Exercise 5.2: Run your SARSA agent on the `obstacle` environment

This environment is similar to the Cliff-walking example from [Sutton & Barto](http://incompleteideas.net/book/RLbook2018.pdf) and allows us to see the different policies learned by on-policy vs off-policy methods. Try varying the number of steps.

```
class SarsaAgent(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               epsilon: float,
               step_size: float = 0.1
               ):

    # Get number of states and actions from the environment spec.
    self._num_states = environment_spec.observations.num_values
    self._num_actions = environment_spec.actions.num_values

    # Create the table of Q-values, all initialized at zero.
    self._q = np.zeros((self._num_states, self._num_actions))

    # Store algorithm hyper-parameters.
    self._step_size = step_size
    self._epsilon = epsilon

    # Containers you may find useful.
    self._state = None
    self._action = None
    self._next_state = None

  @property
  def q_values(self):
    return self._q

  def select_action(self, observation):
    return epsilon_greedy(self._q[observation], self._epsilon)

  def observe_first(self, timestep):
    # Set current state.
    self._state = timestep.observation

  def observe(self, action, next_timestep):
    # Unpacking the timestep to lighten notation.
    s = self._state
    a = action
    r = next_timestep.reward
    g = next_timestep.discount
    next_s = next_timestep.observation
    # Compute the action that would be taken from the next state.
    next_a = self.select_action(next_s)
    # Compute the on-policy Q-value update.
    self._action = a
    self._next_state = next_s
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    # raise NotImplementedError("Student exercise: complete the on-policy Q-value update")
    #################################################
    # TODO complete the line below to compute the temporal difference error
    # HINT: see step 5 in the pseudocode above.
    self._td_error = r + g * self._q[next_s, next_a] - self._q[s, a]

  def update(self):
    # Optional unpacking to lighten notation.
    s = self._state
    a = self._action
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    # raise NotImplementedError("Student exercise: complete value update")
    #################################################
    # Update the Q-value table value at (s, a).
    # TODO: Update the Q-value table value at (s, a).
    # HINT: see step 6 in the pseudocode above, remember that alpha = step_size!
    self._q[s, a] += self._step_size * self._td_error
    # Update the current state.
    self._state = self._next_state

# add event to airtable
atform.add_event('Coding Exercise 5.2: Run your SARSA agent on the obstacle environment')
```

```
# to_remove solution
class SarsaAgent(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               epsilon: float,
               step_size: float = 0.1
               ):

    # Get number of states and actions from the environment spec.
    self._num_states = environment_spec.observations.num_values
    self._num_actions = environment_spec.actions.num_values

    # Create the table of Q-values, all initialized at zero.
    self._q = np.zeros((self._num_states, self._num_actions))

    # Store algorithm hyper-parameters.
    self._step_size = step_size
    self._epsilon = epsilon

    # Containers you may find useful.
    self._state = None
    self._action = None
    self._next_state = None

  @property
  def q_values(self):
    return self._q

  def select_action(self, observation):
    return epsilon_greedy(self._q[observation], self._epsilon)

  def observe_first(self, timestep):
    # Set current state.
    self._state = timestep.observation

  def observe(self, action, next_timestep):
    # Unpacking the timestep to lighten notation.
    s = self._state
    a = action
    r = next_timestep.reward
    g = next_timestep.discount
    next_s = next_timestep.observation
    # Compute the action that would be taken from the next state.
    next_a = self.select_action(next_s)
    # Compute the on-policy Q-value update.
    self._action = a
    self._next_state = next_s
    # Compute the on-policy Q-value update.
    self._action = a
    self._next_state = next_s
    # TODO complete the line below to compute the temporal difference error
    # HINT: see step 5 in the pseudocode above.
    self._td_error = r + g * self._q[next_s, next_a] - self._q[s, a]

  def update(self):
    # Optional unpacking to lighten notation.
    s = self._state
    a = self._action
    # Update the Q-value table value at (s, a).
    # TODO: Update the Q-value table value at (s, a).
    # HINT: see step 6 in the pseudocode above, remember that alpha = step_size!
    self._q[s, a] += self._step_size * self._td_error
    # Update the current state.
    self._state = self._next_state

# add event to airtable
atform.add_event('Coding Exercise 5.2: Run your SARSA agent on the obstacle environment')
```

```
# @title Run SARSA agent and visualize value function

num_steps = 1e5 # @param {type:"number"}
num_steps = int(num_steps)

# Create the environment.
grid = build_gridworld_task(task='obstacle')
environment, environment_spec = setup_environment(grid)

# Create the agent.
agent = SarsaAgent(environment_spec, epsilon=0.1, step_size=0.1)

# Run the experiment and get the value functions from agent
returns = run_loop(environment=environment, agent=agent, num_steps=num_steps)
print('AFTER {0:,} STEPS ...'.format(num_steps))

# Get the Q-values and reshape them to recover grid-like structure of states.
q_values = agent.q_values
grid_shape = grid.layout.shape
q_values = q_values.reshape([*grid_shape, -1])

# Visualize the value and Q-value tables.
plot_action_values(q_values, epsilon=1.)

# Visualize the greedy policy.
environment.plot_greedy_policy(q_values)
```

##Section 5.2 Off-policy control: Q-learning Agent
Reminder: $\color{green}Q$-learning is a very powerful and general algorithm, that enables control (figuring out the optimal policy/value function) both on and off-policy.

**Initialize** $\color{green}Q(\color{red}{s}, \color{blue}{a})$ for all $\color{red}{s} \in \color{red}{\mathcal{S}}$ and $\color{blue}{a} \in \color{blue}{\mathcal{A}}$

**Loop forever**:

1. Get $\color{red}{s} \gets{}$current (non-terminal) state

2. Select $\color{blue}{a} \gets{} \text{behaviour_policy}(\color{red}{s})$

3. Step in the environment by passing the selected action $\color{blue}{a}$

4. Observe resulting reward $\color{green}{r}$, discount $\gamma$, and state, $\color{red}{s'}$

5. Compute the TD error: $\Delta \color{green}Q \gets \color{green}{r} + \gamma \color{green}Q(\color{red}{s'}, \color{blue}{a'}) − \color{green}Q(\color{red}{s}, \color{blue}{a})$,

where $\color{blue}{a'} \gets \arg\max_{\color{blue}{\mathcal A}} \color{green}Q(\color{red}{s'}, \cdot)$

6. Update $\color{green}Q(\color{red}{s}, \color{blue}{a}) \gets \color{green}Q(\color{red}{s}, \color{blue}{a}) + \alpha \Delta \color{green}Q$

Notice that the actions $\color{blue}{a}$ and $\color{blue}{a'}$ are not selected using the same policy, hence this algorithm being **off-policy**.

### Coding Exercise 5.3: Implement Q-Learning

```
QValues = np.ndarray
Action = int
# A value-based policy takes the Q-values at a state and returns an action.
ValueBasedPolicy = Callable[[QValues], Action]

class QLearningAgent(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               behaviour_policy: ValueBasedPolicy,
               step_size: float = 0.1):

    # Get number of states and actions from the environment spec.
    self._num_states = environment_spec.observations.num_values
    self._num_actions = environment_spec.actions.num_values

    # Create the table of Q-values, all initialized at zero.
    self._q = np.zeros((self._num_states, self._num_actions))

    # Store algorithm hyper-parameters.
    self._step_size = step_size

    # Store behavior policy.
    self._behaviour_policy = behaviour_policy

    # Containers you may find useful.
    self._state = None
    self._action = None
    self._next_state = None

  @property
  def q_values(self):
    return self._q

  def select_action(self, observation):
    return self._behaviour_policy(self._q[observation])

  def observe_first(self, timestep):
    # Set current state.
    self._state = timestep.observation

  def observe(self, action, next_timestep):
    # Unpacking the timestep to lighten notation.
    s = self._state
    a = action
    r = next_timestep.reward
    g = next_timestep.discount
    next_s = next_timestep.observation

    # Compute the TD error.
    self._action = a
    self._next_state = next_s
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    # raise NotImplementedError("Student exercise: complete the off-policy Q-value update")
    #################################################
    # TODO complete the line below to compute the temporal difference error
    # HINT: This is very similar to what we did for SARSA, except keep in mind
    # that we're now taking a max over the q-values (see lecture footnotes above).
    # You will find the function np.max() useful.
    self._td_error = r + g * self._q[next_s, np.argmax(self._q[next_s,:])] - self._q[s, a]

  def update(self):
    # Optional unpacking to lighten notation.
    s = self._state
    a = self._action
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    #raise NotImplementedError("Student exercise: complete value update")
    #################################################
    # Update the Q-value table value at (s, a).
    # TODO: Update the Q-value table value at (s, a).
    # HINT: see step 6 in the pseudocode above, remember that alpha = step_size!
    self._q[s, a] += self._step_size * self._td_error
    # Update the current state.
    self._state = self._next_state

# add event to airtable
atform.add_event('Coding Exercise 5.3: Implement Q-Learning')
```

```
# to_remove solution
QValues = np.ndarray
Action = int
# A value-based policy takes the Q-values at a state and returns an action.
ValueBasedPolicy = Callable[[QValues], Action]

class QLearningAgent(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               behaviour_policy: ValueBasedPolicy,
               step_size: float = 0.1):

    # Get number of states and actions from the environment spec.
    self._num_states = environment_spec.observations.num_values
    self._num_actions = environment_spec.actions.num_values

    # Create the table of Q-values, all initialized at zero.
    self._q = np.zeros((self._num_states, self._num_actions))

    # Store algorithm hyper-parameters.
    self._step_size = step_size

    # Store behavior policy.
    self._behaviour_policy = behaviour_policy

    # Containers you may find useful.
    self._state = None
    self._action = None
    self._next_state = None

  @property
  def q_values(self):
    return self._q

  def select_action(self, observation):
    return self._behaviour_policy(self._q[observation])

  def observe_first(self, timestep):
    # Set current state.
    self._state = timestep.observation

  def observe(self, action, next_timestep):
    # Unpacking the timestep to lighten notation.
    s = self._state
    a = action
    r = next_timestep.reward
    g = next_timestep.discount
    next_s = next_timestep.observation

    # Compute the TD error.
    self._action = a
    self._next_state = next_s
    # TODO complete the line below to compute the temporal difference error
    # HINT: This is very similar to what we did for SARSA, except keep in mind
    # that we're now taking a max over the q-values (see lecture footnotes above).
    # You will find the function np.max() useful.
    self._td_error = r + g * np.max(self._q[next_s]) - self._q[s, a]

  def update(self):
    # Optional unpacking to lighten notation.
    s = self._state
    a = self._action
    # Update the Q-value table value at (s, a).
    # TODO: Update the Q-value table value at (s, a).
    # HINT: see step 6 in the pseudocode above, remember that alpha = step_size!
    self._q[s, a] += self._step_size * self._td_error
    # Update the current state.
    self._state = self._next_state

# add event to airtable
atform.add_event('Coding Exercise 5.3: Implement Q-Learning')
```

### Run your Q-learning agent on the `obstacle` environment

```
# @title Run your Q-learning
epsilon =   1. # @param {type:"number"}
num_steps = 1e5  # @param {type:"number"}
num_steps = int(num_steps)

# environment
grid = build_gridworld_task(task='obstacle')
environment, environment_spec = setup_environment(grid)

# behavior policy
behavior_policy = lambda qval: epsilon_greedy(qval, epsilon=epsilon)

# agent
agent = QLearningAgent(environment_spec, behavior_policy, step_size=0.1)

# run experiment and get the value functions from agent
returns = run_loop(environment=environment, agent=agent, num_steps=num_steps)

# get the q-values
q = agent.q_values.reshape(grid.layout.shape + (4,))

# visualize value functions
print('AFTER {:,} STEPS ...'.format(num_steps))
plot_action_values(q, epsilon=0)

# visualise the greedy policy
grid.plot_greedy_policy(q)
plt.show()
```

### Experiment with different levels of greediness

* The default was $\epsilon=1.$, what does this correspond to?
* Try also $\epsilon =0.1, 0.5$. What do you observe? Does the behaviour policy affect the training in any way?

```
# @title Run the cell
epsilon = 0.9  # @param {type:"number"}
num_steps = 1e5  # @param {type:"number"}
num_steps = int(num_steps)

# environment
grid = build_gridworld_task(task='obstacle')
environment, environment_spec = setup_environment(grid)

# behavior policy
behavior_policy = lambda qval: epsilon_greedy(qval, epsilon=epsilon)

# agent
agent = QLearningAgent(environment_spec, behavior_policy, step_size=0.1)

# run experiment and get the value functions from agent
returns = run_loop(environment=environment, agent=agent, num_steps=num_steps)

# get the q-values
q = agent.q_values.reshape(grid.layout.shape + (4,))

# visualize value functions
print('AFTER {:,} STEPS ...'.format(num_steps))
plot_action_values(q, epsilon=epsilon)

# visualise the greedy policy
grid.plot_greedy_policy(q)
plt.show()
```

---
# Section 6: Function Approximation

*Time estimate: ~25mins*

```
# @title Video 6: Function approximation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1sg411M7cn", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"7_MYePyYhrM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 6: Function approximation')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

So far we only considered look-up tables for value-functions. In all previous cases every state and action pair $(\color{red}{s}, \color{blue}{a})$, had an entry in our $\color{green}Q$-table. Again, this is possible in this environment as the number of states is equal to the number of cells in the grid. But this is not scalable to situations where, say, the goal location changes or the obstacles are in different locations at every episode (consider how big the table could be in this situation?).

An example (not covered in this tutorial) is ATARI from pixels, where the number of possible frames an agent can see is exponential in the number of pixels on the screen.

But what we **really** want is just to be able to *compute* the Q-value, when fed with a particular $(\color{red}{s}, \color{blue}{a})$ pair. So if we had a way to get a function to do this work instead of keeping a big table, we'd get around this problem.

To address this, we can use **function approximation** as a way to generalize Q-values over some representation of the very large state space, and **train** them to output the values they should. In this section, we will explore $\color{green}Q$-learning with function approximation, which (although it has been theoretically proven to diverge for some degenerate MDPs) can yield impressive results in very large environments. In particular, we will look at [Neural Fitted Q (NFQ) Iteration](http://ml.informatik.uni-freiburg.de/former/_media/publications/rieecml05.pdf) and [Deep Q-Networks (DQN)](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf).

## Section 6.1 Replay Buffers
An important property of off-policy methods like $\color{green}Q$-learning is that they involve two policies: one for exploration and one that is being optimized (via the $\color{green}Q$-function updates). This means that we can generate data from the **behavior** policy and insert that data into some form of data storage---usually referred to as **replay**.

In order to optimize the $\color{green}Q$-function we can then sample data from the replay **dataset** and use that data to perform an update. An illustration of this learning loop is shown below.

In the next cell we will show how to implement a simple replay buffer. This can be as simple as a python list containing transition data. In more complicated scenarios we might want to have a more performance-tuned variant, we might have to be more concerned about how large replay is and what to do when its full, and we might want to sample from replay in different ways. But a simple python list can go a surprisingly long way.

```
# Simple replay buffer

# Create a convenient container for the SARS tuples required by deep RL agents.
Transitions = collections.namedtuple(
    'Transitions', ['state', 'action', 'reward', 'discount', 'next_state'])

class ReplayBuffer(object):
  """A simple Python replay buffer."""

  def __init__(self, capacity: int = None):
    self.buffer = collections.deque(maxlen=capacity)
    self._prev_state = None

  def add_first(self, initial_timestep: dm_env.TimeStep):
    self._prev_state = initial_timestep.observation

  def add(self, action: int, timestep: dm_env.TimeStep):
    transition = Transitions(
        state=self._prev_state,
        action=action,
        reward=timestep.reward,
        discount=timestep.discount,
        next_state=timestep.observation,
    )
    self.buffer.append(transition)
    self._prev_state = timestep.observation

  def sample(self, batch_size: int) -> Transitions:
    # Sample a random batch of Transitions as a list.
    batch_as_list = random.sample(self.buffer, batch_size)

    # Convert the list of `batch_size` Transitions into a single Transitions
    # object where each field has `batch_size` stacked fields.
    return tree_utils.stack_sequence_fields(batch_as_list)

  def flush(self) -> Transitions:
    entire_buffer = tree_utils.stack_sequence_fields(self.buffer)
    self.buffer.clear()
    return entire_buffer

  def is_ready(self, batch_size: int) -> bool:
    return batch_size <= len(self.buffer)
```

## Section 6.2: NFQ Agent
[Neural Fitted Q Iteration](http://ml.informatik.uni-freiburg.de/former/_media/publications/rieecml05.pdf) was one of the first papers to demonstrate how to leverage recent advances in Deep Learning to approximate the Q-value by a neural network.$^1$
In other words, the value $\color{green}Q(\color{red}{s}, \color{blue}{a})$ are approximated by the output of a neural network $\color{green}{Q_w}(\color{red}{s}, \color{blue}{a})$ for each possible action $\color{blue}{a} \in \color{blue}{\mathcal{A}}$.$^2$

When introducing function approximations, and neural networks in particular, we need to have a loss to optimize. But looking back at the tabular setting above, you can see that we already have some notion of error: the **TD error**.

By training our neural network to output values such that the *TD error is minimized*, we will also satisfy the Bellman Optimality Equation, which is a good sufficient condition to enforce, to obtain an optimal policy.
Thanks to automatic differentiation, we can just write the TD error as a loss, e.g., with an $\ell^2$ loss, but others would work too:

\begin{equation}
L(\color{green}w) = \mathbb{E}\left[ \left( \color{green}{r} + \gamma \max_\color{blue}{a'} \color{green}{Q_w}(\color{red}{s'}, \color{blue}{a'}) − \color{green}{Q_w}(\color{red}{s}, \color{blue}{a})  \right)^2\right].
\end{equation}

Then we can compute the gradient with respect to the parameters of the neural network and improve our Q-value approximation incrementally.

NFQ builds on $\color{green}Q$-learning, but if one were to update the Q-values online directly, the training can be unstable and very slow.
Instead, NFQ uses a replay buffer, similar to what we see implemented above (Section 6.1), to update the Q-value in a batched setting.

When it was introduced, it also was entirely off-policy using a uniformly random policy to collect data, which was prone to instability when applied to more complex environments (e.g. when the input are pixels or the tasks are longer and more complicated).
But it is a good stepping stone to the more complex agents used today. Here, we will look at a slightly different and modernised implementation of NFQ.

Below you will find an incomplete NFQ agent that takes in observations from a gridworld. Instead of receiving a tabular state, it receives an observation in the form of its (x,y) coordinates in the gridworld, and the (x,y) coordinates of the goal.

The goal of this coding exercise is to complete this agent by implementing the loss, using mean squared error.

---

$^1$ If you read the NFQ paper, they use a "control" notation, where there is a "cost to minimize", instead of "rewards to maximize", so don't be surprised if signs/max/min do not correspond.

$^2$ We could feed it $\color{blue}{a}$ as well and ask $Q_w$ for a single scalar value, but given we have a fixed number of actions and we usually need to take an $argmax$ over them, it's easiest to just output them all in one pass.

### Coding Exercise 6.1: Implement NFQ

```
# Create a convenient container for the SARS tuples required by NFQ.
Transitions = collections.namedtuple(
    'Transitions', ['state', 'action', 'reward', 'discount', 'next_state'])

class NeuralFittedQAgent(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               q_network: nn.Module,
               replay_capacity: int = 100_000,
               epsilon: float = 0.1,
               batch_size: int = 1,
               learning_rate: float = 3e-4):

    # Store agent hyperparameters and network.
    self._num_actions = environment_spec.actions.num_values
    self._epsilon = epsilon
    self._batch_size = batch_size
    self._q_network = q_network

    # Container for the computed loss (see run_loop implementation above).
    self.last_loss = 0.0

    # Create the replay buffer.
    self._replay_buffer = ReplayBuffer(replay_capacity)

    # Setup optimizer that will train the network to minimize the loss.
    self._optimizer = torch.optim.Adam(self._q_network.parameters(),lr = learning_rate)
    self._loss_fn = nn.MSELoss()

  def select_action(self, observation):
    # Compute Q-values.
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))  # Adds batch dimension.
    q_values = q_values.squeeze(0)   # Removes batch dimension

    # Select epsilon-greedy action.
    if self._epsilon < torch.rand(1):
      action = q_values.argmax(axis=-1)
    else:
      action = torch.randint(low=0, high=self._num_actions , size=(1,), dtype=torch.int64)
    return action

  def q_values(self, observation):
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))
    return q_values.squeeze(0).detach()

  def update(self):

    if not self._replay_buffer.is_ready(self._batch_size):
      # If the replay buffer is not ready to sample from, do nothing.
      return

    # Sample a minibatch of transitions from experience replay.
    transitions = self._replay_buffer.sample(self._batch_size)

    # Note: each of these tensors will be of shape [batch_size, ...].
    s = torch.tensor(transitions.state)
    a = torch.tensor(transitions.action,dtype=torch.int64)
    r = torch.tensor(transitions.reward)
    d = torch.tensor(transitions.discount)
    next_s = torch.tensor(transitions.next_state)

    # Compute the Q-values at next states in the transitions.
    with torch.no_grad():
      q_next_s = self._q_network(next_s)  # Shape [batch_size, num_actions].
      max_q_next_s = q_next_s.max(axis=-1)[0]
      # Compute the TD error and then the losses.
      target_q_value = r + d * max_q_next_s

    # Compute the Q-values at original state.
    q_s = self._q_network(s)

    # Gather the Q-value corresponding to each action in the batch.
    q_s_a = q_s.gather(1, a.view(-1,1)).squeeze(0)
    #################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your implementation
    #raise NotImplementedError("Student exercise: complete the NFQ Agent")
    #################################################
    # TODO Average the squared TD errors over the entire batch using
    # self._loss_fn, which is defined above as nn.MSELoss()
    # HINT: Take a look at the reference for nn.MSELoss here:
    #  https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html
    #  What should you put for the input and the target?
    loss = self._loss_fn(target_q_value, q_s_a)

    # Compute the gradients of the loss with respect to the q_network variables.
    self._optimizer.zero_grad()

    loss.backward()
    # Apply the gradient update.
    self._optimizer.step()

    # Store the loss for logging purposes (see run_loop implementation above).
    self.last_loss = loss.detach().numpy()

  def observe_first(self, timestep: dm_env.TimeStep):
    self._replay_buffer.add_first(timestep)

  def observe(self, action: int, next_timestep: dm_env.TimeStep):
    self._replay_buffer.add(action, next_timestep)

# add event to airtable
atform.add_event('Coding Exercise 6.1: Implement NFQ')
```

```
# to_remove solution
# Create a convenient container for the SARS tuples required by NFQ.
Transitions = collections.namedtuple(
    'Transitions', ['state', 'action', 'reward', 'discount', 'next_state'])

class NeuralFittedQAgent(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               q_network: nn.Module,
               replay_capacity: int = 100_000,
               epsilon: float = 0.1,
               batch_size: int = 1,
               learning_rate: float = 3e-4):

    # Store agent hyperparameters and network.
    self._num_actions = environment_spec.actions.num_values
    self._epsilon = epsilon
    self._batch_size = batch_size
    self._q_network = q_network

    # Container for the computed loss (see run_loop implementation above).
    self.last_loss = 0.0

    # Create the replay buffer.
    self._replay_buffer = ReplayBuffer(replay_capacity)

    # Setup optimizer that will train the network to minimize the loss.
    self._optimizer = torch.optim.Adam(self._q_network.parameters(),lr = learning_rate)
    self._loss_fn = nn.MSELoss()

  def select_action(self, observation):
    # Compute Q-values.
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))  # Adds batch dimension.
    q_values = q_values.squeeze(0)   # Removes batch dimension

    # Select epsilon-greedy action.
    if self._epsilon < torch.rand(1):
      action = q_values.argmax(axis=-1)
    else:
      action = torch.randint(low=0, high=self._num_actions , size=(1,), dtype=torch.int64)
    return action

  def q_values(self, observation):
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))
    return q_values.squeeze(0).detach()

  def update(self):

    if not self._replay_buffer.is_ready(self._batch_size):
      # If the replay buffer is not ready to sample from, do nothing.
      return

    # Sample a minibatch of transitions from experience replay.
    transitions = self._replay_buffer.sample(self._batch_size)

    # Note: each of these tensors will be of shape [batch_size, ...].
    s = torch.tensor(transitions.state)
    a = torch.tensor(transitions.action,dtype=torch.int64)
    r = torch.tensor(transitions.reward)
    d = torch.tensor(transitions.discount)
    next_s = torch.tensor(transitions.next_state)

    # Compute the Q-values at next states in the transitions.
    with torch.no_grad():
      q_next_s = self._q_network(next_s)  # Shape [batch_size, num_actions].
      max_q_next_s = q_next_s.max(axis=-1)[0]
      # Compute the TD error and then the losses.
      target_q_value = r + d * max_q_next_s

    # Compute the Q-values at original state.
    q_s = self._q_network(s)

    # Gather the Q-value corresponding to each action in the batch.
    q_s_a = q_s.gather(1, a.view(-1,1)).squeeze(0)
    # TODO Average the squared TD errors over the entire batch using
    # self._loss_fn, which is defined above as nn.MSELoss()
    # HINT: Take a look at the reference for nn.MSELoss here:
    #  https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html
    #  What should you put for the input and the target?
    loss = self._loss_fn(target_q_value, q_s_a)

    # Compute the gradients of the loss with respect to the q_network variables.
    self._optimizer.zero_grad()

    loss.backward()
    # Apply the gradient update.
    self._optimizer.step()

    # Store the loss for logging purposes (see run_loop implementation above).
    self.last_loss = loss.detach().numpy()

  def observe_first(self, timestep: dm_env.TimeStep):
    self._replay_buffer.add_first(timestep)

  def observe(self, action: int, next_timestep: dm_env.TimeStep):
    self._replay_buffer.add(action, next_timestep)

# add event to airtable
atform.add_event('Coding Exercise 6.1: Implement NFQ')
```

### Train and Evaluate the NFQ Agent

```
# @title Training the NFQ Agent
epsilon = 0.4 # @param {type:"number"}

max_episode_length = 200

# Create the environment.
grid = build_gridworld_task(
    task='simple',
    observation_type=ObservationType.AGENT_GOAL_POS,
    max_episode_length=max_episode_length)
environment, environment_spec = setup_environment(grid)

# Define the neural function approximator (aka Q network).
q_network = nn.Sequential(nn.Linear(4, 50),
                          nn.ReLU(),
                          nn.Linear(50, 50),
                          nn.ReLU(),
                          nn.Linear(50, environment_spec.actions.num_values))
# Build the trainable Q-learning agent
agent = NeuralFittedQAgent(
    environment_spec,
    q_network,
    epsilon=epsilon,
    replay_capacity=100_000,
    batch_size=10,
    learning_rate=1e-3)

returns = run_loop(
    environment=environment,
    agent=agent,
    num_episodes=500,
    logger_time_delta=1.,
    log_loss=True)
```

```
# @title Evaluating the agent (set $\epsilon=0$)
# Temporarily change epsilon to be more greedy; remember to change it back.
agent._epsilon = 0.0

# Record a few episodes.
frames = evaluate(environment, agent, evaluation_episodes=5)

# Change epsilon back.
agent._epsilon = epsilon

# Display the video of the episodes.
display_video(frames, frame_rate=6)
```

    Episode 0 ended with reward 10.0 in 5 steps
    Episode 1 ended with reward 10.0 in 5 steps
    Episode 2 ended with reward 10.0 in 5 steps
    Episode 3 ended with reward 10.0 in 5 steps
    Episode 4 ended with reward 10.0 in 5 steps

<video  width="320" height="240" controls alt="test" src="data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAHuttZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1MiByMjg1NCBlOWE1OTAzIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNyAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD0tMiB0aHJlYWRzPTMgbG9va2FoZWFkX3RocmVhZHM9MSBzbGljZWRfdGhyZWFkcz0wIG5yPTAgZGVjaW1hdGU9MSBpbnRlcmxhY2VkPTAgYmx1cmF5X2NvbXBhdD0wIGNvbnN0cmFpbmVkX2ludHJhPTAgYmZyYW1lcz0zIGJfcHlyYW1pZD0yIGJfYWRhcHQ9MSBiX2JpYXM9MCBkaXJlY3Q9MSB3ZWlnaHRiPTEgb3Blbl9nb3A9MCB3ZWlnaHRwPTIga2V5aW50PTI1MCBrZXlpbnRfbWluPTYgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTI1LjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAABAQZYiEABL//veBvzLLXyKvIfTfC7VC13ox/NGsEaZxW6rbpGyw6I5OJ9KVTBjElPEHLoIgCbyTgLe5ebRIiVMae3cS1a4uTm6huL8mEnRsYCS1qN/y3DzcTaGgMxdqLHx2iGsqJHhgpJNn1F0nVigs+DVcRt4hmjY0j7cMTg/jF94I18orHERpPbrUh6fmbsJy3B73lDVIfG6STsBKeV2W7TK9cjB4GfGzqSJ7SL531HcjLBuUCZOGPKN47saRRdyjlzCBtUYOQAD69dXST2SdmJrr5xuLZug2L4eMWVozOfRHCHgAPe0qtbhOZLX/BbwbLYWY3W1AM7+A0ZeOQRtbqUZS6w8qqvx47srfHPdvbNVE2LkDISDcm3eJxEkcIVYOj4pKYFPGK2MRHdcnRyRWvmJSxT/kQRy5rxPxXY9f+fgQWrASfUm1MwSOXRv3s4iVlaltDJOUAbrMD7F5gbch2HxmqlXsLEY+6ZzeqHsjdYbkkmvgwmQyVD4hOMryeAieJtHbkeFK/ubnz78JPUyr8khmyUNvMsAahPh+KIraZ/STZQfSLRvg1Fsble0tQ9MWVHXwlp6gjiUEI8ORY+5gCjWHpataamVyyzee4cCzwoVwQXtsCkLSomS12O5U3gUY/eX0loWUqzIGLpg0Hv6MAmkO26lf4FmkbnQelDIiSvc6fpiZsjGWnvrd7spEMMgRENoupIgveStOKdAw1LTVHWPWWnan92ryoUQ2O9YkpQhI48Y10jXOsjURzMxSK+fqas5V4y4FVYkKTMeiCW7K/vIVQUJq8gTwHIZsoHSQjBDYXgppDtupX+BN5DmWGMTanVG62ZQ47ChUtCfQEaabyGDFx0sSN3VfeTyoYEC6r4SOEdsQd8Upt8WXeI+xZs0QXamA/7v3f7s3VKnpVZlBjDqMG04wKszuf/ZBkvHLXPTAWUEGFgcjfabwiGl6X8JXSMOWDhHoicusfxYtMVQUgtEWHy+Uoz7C+l5SkJ7nafTNymKH+W/UNa5g9N/IR57SHtIdayO92a2OD+1GJXaeuh4DFPRgDE1YMmlKEe447wGGyV6AkGm8qADD/9s/0FYJ0A8fWENivjJHT494Im74VsNICWf45qZUXxLkCwN/f3hb4YDE31faaEig0bB77gUAbrL4q8a+NThOXaZ268NhLjekvdU0LCjcZnQY3uLa/GnwnHSsM0HU8CyG8Nh+qUmaeeODBtx9ddYYZsqEOg16KPDckpgy/i/D6Uf/YPDZS/KtlViODGpBKf8qOlZ/k1oPKWYVFNyC90WMfLIaNca1WHVfqzrakT1icaTHt05cr36gsjNZtaGbYZua1tkh8LPtIzCJtHnn0dpGVCtfKLujnDbQy7sVnsnJre820XlCZVIkA0M2GhREZ4Ch34JK8TjGKPmhH/VsO88osT6mK6QC7MfehWvCmqBF0ENJRC4EKFjChOD/ehzHEpyFUXq2KVwXZs9JBAElHSTrDfhnttff+XFI/vAo7di2I8uEDlaBzv2Kf2Eb8k0EzSeRKgsAvrrO79un6O9+meMRz5vfttfqCMl6xAMsEQ5DJRGO/oGPzUA6e7rdvfkrLl41nvmzO8eSwlwyU06dfw09ssbD94LBL5GXAhsl5jvwv8C+ZvH/wzKSnL2C4K0DGGsdtXcMxRn/feuysmc5sncc2bRV0hq3syyb310FQWus9qpBuMtWZ+x5zxMmRxaA+Ur1MAxRfTgxdxmmBz/bHUIbJRP4jKnyMVHn359isahMeE/ZEH/4abmEzh/rEItRRJHnO/gvpGsD9Zn4RP72GLOdkb09M26265cGO6To4IixjpocgxTMnoBgD47aUszrzoOZR/9W3dfow5PgjMl2JEu3nP99Z0Tf9Hf1wgTEkXYXcgfSjeNmo8WtiVdN9OLPe6snJhilbvFMWRqRIKWjJ18JS1cf6jsDKNtqfgQ0FpQNouqdclvNygYejGdUn5PkH9Pbx/6COTK4lclce2i7AYJLzVLlKwwe5vDSLQtfWy1dMC8/JQrav1Y8AzfBul0yL4mGMiBVffAHUQEQO521BZQyYTUsXwj4koXpwGrHaCQ4olMDbQPYl8RmhDOwJ9KJRK144LnmdEAdNn/pYGWhr5zNkBDg3YHavv6Rd/Vvq5ZDEBwL9ofo28boVYEwQCLGq+NQaV8tp5LYE7QUZkh3HWUJWXrYT3HV8G7yuO9ubItJ2MFUPvl7Zmg/CulaFpM+bqwvh7aOJzzDJJkoDozu3jg3piiIqJkWmf+bs+/H4BYGDgFHv1OLiFSxEQShU9zuDwBT/4eEiJJtG3FugDEcp2t8BFZQqDpHOUp0p6GMbP109jUNWcQUWmqln6Tp4Zk8pMxRWsntPIaKP3rfgKvRLEqnqN+I2gZciwDBSp4Cja3raZsZYhjfysdART/u/x22q/iOkcgwUbSi19kEEf2WV/CrOUf/78W0se+Y9pVRXEvYb6E37UCmNwXTdBCbavBftTUC41HqGPlchpTpGWQuKahzdr1Eew9lRFpU2iV4elNePf+IwtN7S7n7b/Yw8OP+WXqEl4g3g5Oo8Bsdi1izPCLSOyihK+u5L9tznhTlAAwbusg82rpLAJP2QdwuP7P3X3qEGGo/3J1vDf7pXVS94Yww/9tO7G1FsnNFyNgx7AAtNxc64nMUJsl2zasSit3oR/tSAT4mqdu803t8Ybc/hfeEjmyTvYZQiaXKxyMpaInIWRi7tRVeHwKEzzF/+DzdstDOkTfTpEZ+crdmQvtaur/IDEn+We672Dge/A4kxgFgcghvn96Up1a8XAfom5asmOf3G9HwE/+5QumLuLv9fpKxFZ/ZUb+NLBIEAx0CxYU6/zC566ECGhq2qrVNqBS7Q8ahWMkdkGrb6dUBEJPoTvwM8ANlXq0UhDVSGV7q0moHKdvMCscXLq0IvWCmSWqbYNVNWAyrqPU83vb1iOOdd1vNCTg5dm5oBg25nFfG2KnfhubIinSvol6SEw17sfJzkO8NnAJolG5N7Y6kbqgBFldnF2cauc2JNhvT2qJRyhnNOtIT4+Xz/WSKRaaPM4sF52XIpXOPsNliYsGWiSmwp3lBC/whAar1ICmzRq0caNR6CyyjmlxdIbWS9uOh/D2FrYWc+58VtyKCGf9MWm7wTOCRxU87lwIoeyAK2lWi5BOPaLljt4CimUIKxduI10K9t9rttnSTTrgnSAdCgrzIHHEBsTA8VziycP/VazuoO4XyHqw4sDEM1PzomIpsHcAywpb0S5Bw/XEEl3QdRVPC2i58vOWP7cJKNwj5tcuF6wNqBlkQpg1Wt3N//6Q9JBhCg+in2TD7uv9pxdnKd6TGyu++NYHtDKvU1NxG0pUFL/1AnJ/xMHbRqiz5TCbVHhoKXLM0P1REMjw9SLt3eLTH3hrWv6j7rlovhzwVKRNpedt2VHYAlDZtcqNNG51e8RvRgLG3nic3AAUvoSXD0EBMityBQdGp9UU0NEtag5LIBgE0D9l78n4Oa8vLrFoZRwR5v6imyQw6kqZvIaQHZ/eguhzhzKS0yvdDvi4B0hXc4cbB1dgNIR9cckc3AgvoQMkutBa0TDbZ0iPFCki+chc9zqEMGBPxV+uy+kakYiB64n+S+9dWuJI1ibbo+35UsiMrOwnwADYz3xYVXsIXKRfVybMyATIAp01fmI6sbtmvvhKtwAW2dMRqo/ZtTePzS3KGKfKfw2VAh0InrlY49rDns1A2t/eGDa8xY5dBdgNiJhcOLxvg3yo3RbjtwK1m46CDmJT1rR5dCyc1iXRnwZJC5DrPPx4sCzX5xl1DLKPS2fWkl8FSfiLZhH/HT8YJsS5MMKeI+EuUc4OGJfXIekINdrjNfM+b7x8QVWFQSnuMfWMQIYfGdpRq5Cv2UciYcnU4U6jUASfqn66JeHzZg9fZmImLymKpRcu/M+Wl87wgNMRAuN9pqum6TpIATvPRc8QgxCinNsJeAeg4Wtx2/H1HKwSkslGQZwcCeF7IpdBrnjrDvP0nm6ogpP+Gz4S2TvTZfgrPEC+8m9HZA834JHAmsEEAUbVU8GWyTIUAtQYvhQtX9w7hWEU/mhBgUkA+r0Rb8A15qBCJiaVxzJThig+JxHo8BCOCF4EeC6xNWkqvJGYBmEEZ8xZHKU10PNUITUykoCGxnvTCAb7ynQzd2Wl3W+Azjh3QS7Yh4rWZo00JZAO2RZeBVOqV0O0cyMsn3pkmUMSGvhqCfKOZeQJswYi4xBNv98c/a7xZM4/KNP7lCXxsrxnw1TBOV/hcHow7f6GgoXevOt4WgOU4zPRM40spoR77kaH8B30DJBRlhg5xKnM0koTfbBuv/0BBtByb+WkHztZqazAeqJpcJ81fpS2kFuthsQnnkKYaZpGSsitsmKvp5CT947d8V3aSn0D5by6k94kFVd2He8h7mD1P5ivscN8sJDyeCAdebOmOkDUXkrogjoqnkAc+/tIplcxchc39gpz+0MhQP4hilkL0Qku/uDnVanoFIZmk99tEJDVLNCmduNh83IFmcamlSX970cY0/7Z+AQzH014zxfN1z4DlrAvW59FtVpYLe8B1cR4YZYfrqxvp/GTVBMf8igFstszxf/B/BCh9ZwbT1cnORqAG05g8T8l6SR9YW/bLcTOYBm2LXgY4BLh60X2RclLjF+kyGpU6d39kJWbGWnz7XVYciZkVL7EkjIQPDgFrcvFWW16FPeOAAX6NvbkjVQTtYi8UfJMvbQ7Wltf9CfWQder3oyYEktxpqei6sAS2IfLniFYEJGRpXpryZ8NkRCXjBBFhJ2UIQKRaOyznzS1Qi0x6dN4Vg6cG4VhH6WGbuue4q40jpECh97IRsUDctralkB2tMli8oQhM5zZ6VrFdgtef0bpFMApSDZrLyOJK/0YccQe7YGOPf+QKgRf0HKLd3LPUSI7iX/Bpx9u0/QSK9AbvIST5DEWCpCvTcUWkuWCFxPoe7XTf0xLUolrQwJTzMjwEG1duij7lmvRT6F1GMWv/jYWN0/HJ67uZ7CfvRyxZKs/kfGq2ejkbQ2paX0OLQf/O0fbx8LksBc6R4mDGPjT7TMp3QlxiOQIt/dSR5pqe8RdNmrZUXLlKKZWFFN3PBpsATQQrHFji8B7Q2DWlFqTchtP7ycddjJ/VJDAbewrYA/dK55pmyCmeaB3d1z6YNxZNxR6k1yucgQG5rbZTk3J624V3mPYzKNw5L62sbi2OIgRBZR7ATkqiDj2H2lVbsIXDMGWnzj6ZFtn/IKdfA14dYUcWw5n687A46kboKE1O6juoQRH00xY5MBt7CtgD9xrds0jigonGKc7VG9oSTvz1xMS1tspybkSHSzQ7KNEudVvOKI49V7dbMA29iK7sbkeP/i24LgZFhX+iHz0VQ6TwFeo8vDNHvQi6NcvhkRET6H+JQQGSS6fM6IKVpipr4Lt/F7PMWvk4/0ilFX/2ivTVaAUqFN9QX0Mx2RbNSRVK0ZgzlAAAAwCWnEMAAAFfQZoibEE//rUqgNtudgCMrx+obs2udlQPJ+2QyztP2kejKr20Tp6MAf54TpHdtRcxKPCZFSWk+RVvqk7NGr3WDfQSEFub8Qos2sUIIgfjHWkx2Ay0OnpLLXOSABtYyDU9mINwzrxEqPu3SXLvBKNMDF6wlU2R0v/yTSosfROhK0mXpYTjGfjy0m+l/FlBGv/MQ7EI1iHAjnTeAXAq3stOUVC97pXQ5LtJZ0eNJapcuKGHshSe9N19ELTKINWd4IMHbCkjx88hzAZzBSMmkCSIXBVTbFfKfNepzHJ03J/eMcqvCs1/dg97lcp298k1+SqJq0SCZ8aKV1zcPHYC3L+H6R/PMibd0VNMZ3TscAtIRwoC0EF2NF7hgPT5gDsfawC08NnaoMw2+CnZUFMa2FjAjs4ngRTstinrxd7MpxsuvABdqP/s48SFAEFkTOYCVZ3G9e5u4NqHvNjAveDr8HQgAAAAMwGeQXkP/wE1GSnbR0z3ME0vrCbHgBYAHgpsyhrfs8eR3Zun7sc1xIoVM0VFzHEuPGkm4QAAAEtBmkY8IZMphBP//rUqgBGHqWsFCGN59cBb+EtRlIaqy8xNQ6buJRtHVBy7iKzrUrlakB9wU46w9Xgl/2A3HlAoeys4WfUc6iZi3nwAAAC0QZ5kalPBDwBEHKMJYjRdHcGbB8XcF8eAfpO/gDACWroBfUOG7Oq/0XvrVBmYdb/OZ3OxvKpExd/UzszvbZFGN6yovIzKClx/9JEruqmY/ig0QQfUkeOvxPTHgB5+781A2GUhw4hw3Qy2/eifN2tbzHKzHSRp01YmdMEOPPspEelSHsMRuAErzbPX3MCdUNAy6nsNZi6RWZgTWu2ag2RGKQAtGrpWJK6k1QpLIhV38gmQv+khAAAArgGeg3RD/wBhusRUoqAIAtMT43w265U5keIACckPwqJe3YzL0gLkgmYdy1hqpRbaKMfbMJxB/JHmuFWk/A/+0gEkoKlgMPol4srbzT+QRGxJWWUdmfMoatnZLLf0D7xbfhNDWhvhJESZAuJ0t+mj7muqMUCFlqE3IGPVz5mxJLiUe82uteH77rMk2JcKRXeybeUnd82KnMIo8fwOF79swnGm/lD0oxxlHkPlTkSF4QAAAK0BnoVqQ/8AYf7kAOPchFgRxFzq+AH19Pam2Mn3hpzKe97/w49nC1EoXZaul7pfMYaf6aJDFMsf5ubAXCI2KM3x65RjmuQdKvwOmTR02fmHm+PKAKX7/p66Q3QDxBOeCKnQuKzBUBG1lpnx0AlsEco4c+vNjx0vvx9La/bhzpMxWCO+ixQW7vW7529O+/IDBPvhiwx5llmz8HU8DYZpPiBXPWSOLYOby4yp0zbPeQAAAE5BmopJqEFomUwIJf/+qlUAQWcigHsDL4nCGVCOUKCoPu684y/qQlNT2O5L1mUHRvdLvRnX0mYiemq3A6png1gyA3iQEF9FVjtg5QKlQEkAAAAoQZ6oRREsEP8AbLaxJnmWaEUq/3yeHuq5knDgcb4ZGZ0E7aeKwpklpgAAAKsBnsd0Q/8Amz7MoWTUnPbQ/e6QAmr3fhUS9uxmXos1+BoT65Eg60hxEbGPtmE78ADQ0wfwMSFJ7QFSeTrq4BaMhIlVGg11QmfYT3OKPiAkp3efBHjQNHglxs86dwH3ufY1Z7n0VtxDLkdroypYV+eE+rudcLwcjAws9ilnfJ8TUwraDLR7Kln+4YHaLl2uf/OfiCPGG//nJ8gPpwhrwNZy2+q8+nSk47ZrbB8AAACtAZ7JakP/AGHLpOZACwAyC3nloXJvNelVEIzlXdH7N1nAXvrU95muRc5mei+5xsJWv1S4msB0guUmj1vPBmUFLj/6Uc1uP/38ODL1tAUKn4FXOY8APP3fmlPnSkOHIyG6GW37x9FIHq1J4RBy4QUhWJnTBC23CFIj0qQ9hoAMASR4xcXmEJ1Q0DLn0+VoFj3hSa8a7ZpOdnTugFk2jKxJXUmn8t5lyu/xd8mQBkEAAAA5QZrMSahBbJlMFEwT//61KoARg85187FJLTXxtI5cfa9aeh4kLRUHmR2HPoADcSHZQjt+kC8mMLvAAAAAIQGe62pD/wBhwBTXACwAg/fkbBSNt0I8zI3JhNHoGlPHYAAAADRBmvBJ4QpSZTAgl//+qlUAPlEY/JmqGQq82R0alQvgQfLIpzDYNQgxGbHB6ujS4J4YVeDjAAAArkGfDkU0TBD/ABswR/27Xwqg1b23vsQgALqugF9Q4bs6r/Re+tT0+eci5zN6NwpVImLv6mdmd7bIoxvWVF5GZQUuP/pPDzcf/v4IC2yD6kjx1+J6Y8APP3fmoGwykOHTCG6GW37y+jka1vMcrMdJGnTViZ0wQ48+ykR6VIewxG4ASvNs9fcwJ1Q0DLqew1mLpFZmBNa7ZqDZEYpAC0aulYkrqTVCksiFXf4k671dJwAAAKYBny10Q/8ADy6bTA7Pti0VU5j/wAE5IfhUS9uxmXpAXJBMw7lrDVSi20UY+2YTiD+SPNcKtJ+B/9pAJM4QUyI1M/oFW3mn8giNiSsso7M+ZQ1bOyWW/oH3i2/CaGtDfCSIkyBcTpb9NH3NdUYoELLUJuQMernzNiSXEo95tda8P33WZJsS4Uiu9k28pO75sVOYRR4/gcL2xWItarcajHHM9NCXbnPRAAAArAGfL2pD/wBh/uQA49yEWBHEXOr4AfXv7+TYyfeGnMp73v/Dj2cLUShdlq6Xul8xhp/pokMTn1sCJbH4G7CjN8euUY5rkHSr8Dpk0dNn5h5vjygCmYZX6UKN0A8QTngip0LiswVARtZaZ8dAJbBHKOHPrzY8dL78fS2v24c6TMVgjvosUFu71u+dvTvvyAwT74YsMeZZZs/B1PA2GaT4gVz1kji2DomD1Qur6+AAAABUQZs0SahBaJlMCCP//qmWAIgfNUAF1XYCzOQ251wDNYyyz6bY2axfUPZ9Uh7Dj20F3ni7LV0y0eZNkT/TRIe0zoNQ60ZqtcQv3ygji6t0j+xeX4qfAAAAKEGfUkURLBD/AEQj+vNtjF05KGkzJ3WFthk8Hnxq9A3sCGW9a5yLp98AAACoAZ9xdEP/ACavrNgdn4Y6MAH9IP4VEvbsZl6LNfgaE+uRIOtIcRGxj7ZhO/AA0NMNZa03sD5oUnk66uAWjISJVRoNdUJn2E9zij4gJKd3nwR40DR4JcbPOncB97n2NWe59FbcQy5Ha6MqWFfnhPq7nXC8HIwMLLnrd8nxNTCtoMtHsqWfznbaIuXMy/85+II8Yb/+cnyA+nCGvA1nLb6rz6bPAka5qJutAAAArQGfc2pD/wBhy6TmQAr8voyHaUIM98rNhcl3VIG3Zus4C99anufH/28s9yL7nGwla/VLiawHSC5SaPW88GZQUuP/pWT8OXYfwwMvW0BQqfgVc5jwA8/d+aU+dKQ4dTIboZbR4wikD1ak8Ig5cIKQrEzpghbbhCkR6VIew0AGAJI8YuLzCE6oaBl5naF8ZmPeFJrxrtmkx2dO6AWR6MrEldSafnbGXK7wTjcP/zT4AAAAI0GbdkmoQWyZTBRMEf/+qZYAiCzJ+HoY45yxnATXfzrHdrCZAAAAJgGflWpD/wBhwBTXACv17xaKDK48fWPThDr+yL3jzvks6ZFpEwqAAAAAs0GbmEnhClJlMFLD//6nhAEELDiBZmGZVJR0MKzNQj8VUQjOVd0fs3WcBe+tUEt3v9vLPYi+5xsJWv1S4msB0guUmj1vPBmUFLj/6X9aDl2H8CQL1tAUKn4FXOY8APP3fmlPnSkOUrEN0Mto8YRSB6tSeEQcuEFIViZ0wQttwhSI9KkPYaADAEkeMXF5hCdUNAy8ztC+MzHvCk1412zSY7OndALI9GViSupNPztjLld4JxnxAAAAqQGft2pD/wBdDILgd1Yx0ddq5U5kM2AEzIfhUS9uxmXpAXJBMw7lrDVSi20UY+2YTiD+SPNcKtJ+B/9pAJNCgUyI1M/oZW3mn8giNiSsso7M+ZQ1bOyWW/oH3i2/CaGtDfCSIkyBcTpb9NH3NdUYoELLUJuQMernzNiSXEo95tda8P33WZJsS4Uiu9k28pO75sVOYQ06GKWF7L2TxDfe3eSQft2EMCdznoEAAAQ9bW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAEEcAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAA2d0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAEEcAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAASAAAAEgAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAABBHAAAQAAABAAAAAALfbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAwAAAAyABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAACim1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAAAkpzdGJsAAAAlnN0c2QAAAAAAAAAAQAAAIZhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAASABIABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMGF2Y0MBZAAM/+EAGGdkAAys2UEgloQAAAMABAAAAwAwPFCmWAEABWjr5yyLAAAAGHN0dHMAAAAAAAAAAQAAABkAAAgAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAADYY3R0cwAAAAAAAAAZAAAAAQAAEAAAAAABAAAYAAAAAAEAAAgAAAAAAQAAKAAAAAABAAAQAAAAAAEAAAAAAAAAAQAACAAAAAABAAAoAAAAAAEAABAAAAAAAQAAAAAAAAABAAAIAAAAAAEAABgAAAAAAQAACAAAAAABAAAoAAAAAAEAABAAAAAAAQAAAAAAAAABAAAIAAAAAAEAACgAAAAAAQAAEAAAAAABAAAAAAAAAAEAAAgAAAAAAQAAGAAAAAABAAAIAAAAAAEAABgAAAAAAQAACAAAAAAcc3RzYwAAAAAAAAABAAAAAQAAABkAAAABAAAAeHN0c3oAAAAAAAAAAAAAABkAABLFAAABYwAAADcAAABPAAAAuAAAALIAAACxAAAAUgAAACwAAACvAAAAsQAAAD0AAAAlAAAAOAAAALIAAACqAAAAsAAAAFgAAAAsAAAArAAAALEAAAAnAAAAKgAAALcAAACtAAAAFHN0Y28AAAAAAAAAAQAAADAAAABidWR0YQAAAFptZXRhAAAAAAAAACFoZGxyAAAAAAAAAABtZGlyYXBwbAAAAAAAAAAAAAAAAC1pbHN0AAAAJal0b28AAAAdZGF0YQAAAAEAAAAATGF2ZjU3LjgzLjEwMA==">

```
# @title Visualise the learned $Q$ values

# Evaluate the policy for every state, similar to tabular agents above.

environment.reset()
pi = np.zeros(grid._layout_dims, dtype=np.int32)
q = np.zeros(grid._layout_dims + (4, ))
for y in range(grid._layout_dims[0]):
  for x in range(grid._layout_dims[1]):
    # Hack observation to see what the Q-network would output at that point.
    environment.set_state(x, y)
    obs = environment.get_obs()
    q[y, x] = np.asarray(agent.q_values(obs))
    pi[y, x] = np.asarray(agent.select_action(obs))

plot_action_values(q)
```

Compare the Q-values approximated with the neural network with the tabular case in **Section 5.3**. Notice how the neural network is generalizing from the visited states to the unvisited similar states, while in the tabular case we updated the value of each state only when we visited that state.

### Compare the greedy and behaviour ($\epsilon$-greedy) policies

```
# @title Compare the greedy policy with the agent's policy

# @markdown Notice that the agent's behavior policy has a lot more randomness,
# @markdown due to the high $\epsilon$. However, the greedy policy that's learned
# @markdown is optimal.

environment.plot_greedy_policy(q)
plt.figtext(-.08, .95, 'Greedy policy using the learnt Q-values')
plt.title('')
plt.show()

environment.plot_policy(pi)
plt.figtext(-.08, .95, "Policy using the agent's behavior policy")
plt.title('')
plt.show()
```

---
# Section 7: RL in the real world

*Time estimate: ~10mins*

```
# @title Video 7: Real-world applications and ethics
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Nq4y1X7AF", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"5kBtiW88QVw", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 7: Real-world applications and ethics')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Exercise 7: Group discussion

Form a group of 2-3 and have discussions (roughly 3 minutes each) of the following questions:

1. **Safety**: what are some safety issues that arise in RL that don’t arise with e.g. supervised learning?

2. **Generalization**: What happens if your RL agent is presented with data it hasn’t trained on? (“goes out of distribution”)

3. How important do you think **interpretability** is in the ethical and safe deployment of RL agents in the real world?

```
# to_remove explanation
"""
For TAs:

Encourage students to consider different examples of RL in the real
world that have safety issues that don't necessarily come up with supervised
learning, due to the fact that in RL, agents must *interact* with the world,
and explore.

Self-driving cars is a really salient example, but also consider using RL in
the health-care setting, where exploration would be unethical. What can be done
in this case? (Consider offline RL, using simulations, safeguards, etc.)

Interpretability can help with the safe deployment of deep RL algorithms
because we can have higher expectation that our agents will not perform
unpredictable actions, or if they do, we can at least understand why they did,
and work to prevent such occurrences in the future.

This should be a very open-ended discussion. Try to have everyone say at least
one thing. They can either take these 3 questions in turn, with 3-4 minutes
allotted to each, or address them all at once, and allow for a more natural
conversation.
""";
```

---
# Summary

*Time estimate: ~3mins*

In this tutorial we learn the most important aspects of RL. Within the RL framework, we are able to identify the different components: environment, agent, states, and actions. In addition, we learned and understand the Bellman equation and components involved.We implemented tabular value-based model-free learning (Q-learning and SARSA). Finally, we discussed real-world applications and ethical issues of RL.

If you have time left, in Bonus sections you can run a DQN agent and experiment with different hyperparameters (Bounus 1), and you can have a high-level understanding of other (nonvalue-based) RL methods (Bonus 2).

See also our *Appendix and further reading* for more information.

```
# @title Video 8: How to learn more
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1WM4y1T7G5", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"dKaOpgor5Ek", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 8: How to learn more')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

```
# @title Airtable Submission Link
from IPython import display as IPydisplay
IPydisplay.HTML(
   f"""
 <div>
   <a href= "{atform.url()}" target="_blank">
   </a>
   </div>""" )
```

  <a href= "https://portal.neuromatchacademy.org/api/redirect/to/3e77471d-4de0-4e43-a026-9cfb603b5197?data=eyJmb3JtX2lkIjogImFwcG43VmRQUnNlU29NWEVHIiwgInRhYmxlX25hbWUiOiAiVzNEMl9UMSIsICJhbnN3ZXJzIjoge30sICJldmVudHMiOiBbeyJldmVudCI6ICJpbml0IiwgInRzIjogMTYyOTI3MTc2OC4yNDk0MzQ3fSwgeyJldmVudCI6ICJWaWRlbyAxOiBJbnRyb2R1Y3Rpb24gdG8gUkwiLCAidHMiOiAxNjI5MjcxNzc1LjEwMjU2MTd9LCB7ImV2ZW50IjogIlZpZGVvIDI6IEdlbmVyYWwgRm9ybXVsYXRpb24gYW5kIE1EUHMiLCAidHMiOiAxNjI5MjcyNTIwLjU0NjM0M30sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDIuMTogUmFuZG9tIEFnZW50IiwgInRzIjogMTYyOTI3NDI5OS41MTEyOTY1fSwgeyJldmVudCI6ICJWaWRlbyAzOiBUaGUgQmVsbG1hbiBFcXVhdGlvbiIsICJ0cyI6IDE2MjkyNzQzNTUuMTg1MTMzfSwgeyJldmVudCI6ICJWaWRlbyA0OiBQb2xpY3kgRXZhbHVhdGlvbiIsICJ0cyI6IDE2MjkyNzUwNzUuMTgwODU4fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgNC4xIFBvbGljeSBFdmFsdWF0aW9uIEFnZW50IiwgInRzIjogMTYyOTI3ODM0MS4yODI4NzR9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSA0LjEgUG9saWN5IEV2YWx1YXRpb24gQWdlbnQiLCAidHMiOiAxNjI5Mjc4MzU0LjExMjI2fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgNC4xIFBvbGljeSBFdmFsdWF0aW9uIEFnZW50IiwgInRzIjogMTYyOTI3ODM3Ny4wNjEyMzExfSwgeyJldmVudCI6ICJWaWRlbyA1OiBNb2RlbC1GcmVlIExlYXJuaW5nIiwgInRzIjogMTYyOTI3ODUwNi45OTIyMjYxfSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgNS4xOiBJbXBsZW1lbnQgIGVwc2lsb24tZ3JlZWR5IiwgInRzIjogMTYyOTI4MDY4Ni4wMTI0OH0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDUuMjogUnVuIHlvdXIgU0FSU0EgYWdlbnQgb24gdGhlIG9ic3RhY2xlIGVudmlyb25tZW50IiwgInRzIjogMTYyOTI4MTAyMy43MjI0NDAyfSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgNS4zOiBJbXBsZW1lbnQgUS1MZWFybmluZyIsICJ0cyI6IDE2MjkyODEzOTkuMDg5NDE3N30sIHsiZXZlbnQiOiAiVmlkZW8gNjogRnVuY3Rpb24gYXBwcm94aW1hdGlvbiIsICJ0cyI6IDE2MjkyODE3NjMuNjIwMTIzOX0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDYuMTogSW1wbGVtZW50IE5GUSIsICJ0cyI6IDE2MjkyODM2MDkuOTQyMjE4M30sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDYuMTogSW1wbGVtZW50IE5GUSIsICJ0cyI6IDE2MjkyODM2MzguMzU0MDk3OH0sIHsiZXZlbnQiOiAiVmlkZW8gNzogUmVhbC13b3JsZCBhcHBsaWNhdGlvbnMgYW5kIGV0aGljcyIsICJ0cyI6IDE2MjkyODM4NjAuNTk1MTg3Mn0sIHsiZXZlbnQiOiAiVmlkZW8gODogSG93IHRvIGxlYXJuIG1vcmUiLCAidHMiOiAxNjI5Mjg0MzQ4LjcxMzQzMTh9LCB7ImV2ZW50IjogInVybCBnZW5lcmF0ZWQiLCAidHMiOiAxNjI5Mjg0MzQ5LjQ2MDY0MzN9XX0%3D" target="_blank">

---
# Bonus 1: DQN

*Time estimate: ~30mins*

```
# @title Video 9: Deep Q-Networks (DQN)
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Mo4y1Q7yD", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"HEDoNtV1y-w", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 9: Deep Q-Networks (DQN)')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Adopted from Mnih et al., 2015

In this section, we will look at an advanced deep RL Agent based on the following publication, [Playing Atari with Deep Reinforcement Learning](https://deepmind.com/research/publications/playing-atari-deep-reinforcement-learning), which introduced the first deep learning model to successfully learn control policies directly from high-dimensional pixel inputs using RL.

Here the agent will act directly on a pixel representation of the gridworld. You can find an incomplete implementation below.

### Bonus Coding Exercise 1: Run a DQN Agent

```
class DQN(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               network: nn.Module,
               replay_capacity: int = 100_000,
               epsilon: float = 0.1,
               batch_size: int = 1,
               learning_rate: float = 5e-4,
               target_update_frequency: int = 10):

    # Store agent hyperparameters and network.
    self._num_actions = environment_spec.actions.num_values
    self._epsilon = epsilon
    self._batch_size = batch_size
    self._q_network = q_network

    # create a second q net with the same structure and initial values, which
    # we'll be updating separately from the learned q-network.
    self._target_network = copy.deepcopy(self._q_network)

    # Container for the computed loss (see run_loop implementation above).
    self.last_loss = 0.0

    # Create the replay buffer.
    self._replay_buffer = ReplayBuffer(replay_capacity)
    # Keep an internal tracker of steps
    self._current_step = 0

    # How often to update the target network
    self._target_update_frequency = target_update_frequency
    # Setup optimizer that will train the network to minimize the loss.
    self._optimizer = torch.optim.Adam(self._q_network.parameters(), lr=learning_rate)
    self._loss_fn = nn.MSELoss()

  def select_action(self, observation):
    # Compute Q-values.
    # Sonnet requires a batch dimension, which we squeeze out right after.
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))  # Adds batch dimension.
    q_values = q_values.squeeze(0)   # Removes batch dimension

    # Select epsilon-greedy action.
    if self._epsilon < torch.rand(1):
      action = q_values.argmax(axis=-1)
    else:
      action = torch.randint(low=0, high=self._num_actions , size=(1,), dtype=torch.int64)
    return action

  def q_values(self, observation):
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))
    return q_values.squeeze(0).detach()

  def update(self):
    self._current_step += 1

    if not self._replay_buffer.is_ready(self._batch_size):
      # If the replay buffer is not ready to sample from, do nothing.
      return

    # Sample a minibatch of transitions from experience replay.
    transitions = self._replay_buffer.sample(self._batch_size)

    # Optionally unpack the transitions to lighten notation.
    # Note: each of these tensors will be of shape [batch_size, ...].
    s = torch.tensor(transitions.state)
    a = torch.tensor(transitions.action,dtype=torch.int64)
    r = torch.tensor(transitions.reward)
    d = torch.tensor(transitions.discount)
    next_s = torch.tensor(transitions.next_state)

    # Compute the Q-values at next states in the transitions.
    with torch.no_grad():
      #################################################
      # Fill in missing code below (...),
      # then remove or comment the line below to test your implementation
      # raise NotImplementedError("Student exercise: complete the DQN Agent")
      #################################################
      #TODO get the value of the next states evaluated by the target network
      #HINT: use self._target_network, defined above.
      q_next_s = self._target_network(next_s)  # Shape [batch_size, num_actions].
      max_q_next_s = q_next_s.max(axis=-1)[0]
      # Compute the TD error and then the losses.
      target_q_value = r + d * max_q_next_s

    # Compute the Q-values at original state.
    q_s = self._q_network(s)

    # Gather the Q-value corresponding to each action in the batch.
    q_s_a = q_s.gather(1, a.view(-1,1)).squeeze(0)

    # Average the squared TD errors over the entire batch
    loss = self._loss_fn(target_q_value, q_s_a)

    # Compute the gradients of the loss with respect to the q_network variables.
    self._optimizer.zero_grad()

    loss.backward()
    # Apply the gradient update.
    self._optimizer.step()

    if self._current_step % self._target_update_frequency == 0:
      self._target_network.load_state_dict(self._q_network.state_dict())
    # Store the loss for logging purposes (see run_loop implementation above).
    self.last_loss = loss.detach().numpy()

  def observe_first(self, timestep: dm_env.TimeStep):
    self._replay_buffer.add_first(timestep)

  def observe(self, action: int, next_timestep: dm_env.TimeStep):
    self._replay_buffer.add(action, next_timestep)

# Create a convenient container for the SARS tuples required by NFQ.
Transitions = collections.namedtuple(
    'Transitions', ['state', 'action', 'reward', 'discount', 'next_state'])
```

```
# to_remove solution
class DQN(acme.Actor):

  def __init__(self,
               environment_spec: specs.EnvironmentSpec,
               network: nn.Module,
               replay_capacity: int = 100_000,
               epsilon: float = 0.1,
               batch_size: int = 1,
               learning_rate: float = 5e-4,
               target_update_frequency: int = 10):

    # Store agent hyperparameters and network.
    self._num_actions = environment_spec.actions.num_values
    self._epsilon = epsilon
    self._batch_size = batch_size
    self._q_network = q_network

    # create a second q net with the same structure and initial values, which
    # we'll be updating separately from the learned q-network.
    self._target_network = copy.deepcopy(self._q_network)

    # Container for the computed loss (see run_loop implementation above).
    self.last_loss = 0.0

    # Create the replay buffer.
    self._replay_buffer = ReplayBuffer(replay_capacity)
    # Keep an internal tracker of steps
    self._current_step = 0

    # How often to update the target network
    self._target_update_frequency = target_update_frequency
    # Setup optimizer that will train the network to minimize the loss.
    self._optimizer = torch.optim.Adam(self._q_network.parameters(), lr=learning_rate)
    self._loss_fn = nn.MSELoss()

  def select_action(self, observation):
    # Compute Q-values.
    # Sonnet requires a batch dimension, which we squeeze out right after.
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))  # Adds batch dimension.
    q_values = q_values.squeeze(0)   # Removes batch dimension

    # Select epsilon-greedy action.
    if self._epsilon < torch.rand(1):
      action = q_values.argmax(axis=-1)
    else:
      action = torch.randint(low=0, high=self._num_actions , size=(1,), dtype=torch.int64)
    return action

  def q_values(self, observation):
    q_values = self._q_network(torch.tensor(observation).unsqueeze(0))
    return q_values.squeeze(0).detach()

  def update(self):
    self._current_step += 1

    if not self._replay_buffer.is_ready(self._batch_size):
      # If the replay buffer is not ready to sample from, do nothing.
      return

    # Sample a minibatch of transitions from experience replay.
    transitions = self._replay_buffer.sample(self._batch_size)

    # Optionally unpack the transitions to lighten notation.
    # Note: each of these tensors will be of shape [batch_size, ...].
    s = torch.tensor(transitions.state)
    a = torch.tensor(transitions.action,dtype=torch.int64)
    r = torch.tensor(transitions.reward)
    d = torch.tensor(transitions.discount)
    next_s = torch.tensor(transitions.next_state)

    # Compute the Q-values at next states in the transitions.
    with torch.no_grad():
      #TODO get the value of the next states evaluated by the target network
      #HINT: use self._target_network, defined above.
      q_next_s = self._target_network(next_s)  # Shape [batch_size, num_actions].
      max_q_next_s = q_next_s.max(axis=-1)[0]
      # Compute the TD error and then the losses.
      target_q_value = r + d * max_q_next_s

    # Compute the Q-values at original state.
    q_s = self._q_network(s)

    # Gather the Q-value corresponding to each action in the batch.
    q_s_a = q_s.gather(1, a.view(-1,1)).squeeze(0)

    # Average the squared TD errors over the entire batch
    loss = self._loss_fn(target_q_value, q_s_a)

    # Compute the gradients of the loss with respect to the q_network variables.
    self._optimizer.zero_grad()

    loss.backward()
    # Apply the gradient update.
    self._optimizer.step()

    if self._current_step % self._target_update_frequency == 0:
      self._target_network.load_state_dict(self._q_network.state_dict())
    # Store the loss for logging purposes (see run_loop implementation above).
    self.last_loss = loss.detach().numpy()

  def observe_first(self, timestep: dm_env.TimeStep):
    self._replay_buffer.add_first(timestep)

  def observe(self, action: int, next_timestep: dm_env.TimeStep):
    self._replay_buffer.add(action, next_timestep)

# add event to airtable
atform.add_event('Coding Exercise 7.1: Run a DQN Agent')

# Create a convenient container for the SARS tuples required by NFQ.
Transitions = collections.namedtuple(
    'Transitions', ['state', 'action', 'reward', 'discount', 'next_state'])
```

```
# @title Train and evaluate the DQN agent

epsilon = 0.25  # @param {type: "number"}
num_episodes = 500  # @param {type: "integer"}
max_episode_length = 200  # @param {type: "integer"}

grid = build_gridworld_task(
    task='simple',
    observation_type=ObservationType.GRID,
    max_episode_length=max_episode_length)
environment, environment_spec = setup_environment(grid)

# Build the agent's network.
class Permute(nn.Module):
  def __init__(self, order: list):
    super(Permute,self).__init__()
    self.order = order

  def forward(self, x):
    return x.permute(self.order)

q_network = nn.Sequential(Permute([0, 3, 1, 2]),
                          nn.Conv2d(3, 32, kernel_size=4,
                                    stride=2,padding=1),
                          nn.ReLU(),
                          nn.Conv2d(32, 64, kernel_size=3,
                                    stride=1, padding=1),
                          nn.ReLU(),
                          nn.MaxPool2d(3, 1),
                          nn.Flatten(),
                          nn.Linear(384, 50),
                          nn.ReLU(),
                          nn.Linear(50, environment_spec.actions.num_values)
                          )

agent = DQN(
    environment_spec=environment_spec,
    network=q_network,
    batch_size=10,
    epsilon=epsilon,
    target_update_frequency=25)

returns = run_loop(
    environment=environment,
    agent=agent,
    num_episodes=num_episodes,
    num_steps=100000)
```

```
# @title Visualise the learned $Q$ values
# Evaluate the policy for every state, similar to tabular agents above.
pi = np.zeros(grid._layout_dims, dtype=np.int32)
q = np.zeros(grid._layout_dims + (4,))
for y in range(grid._layout_dims[0]):
  for x in range(grid._layout_dims[1]):
    # Hack observation to see what the Q-network would output at that point.
    environment.set_state(x, y)
    obs = environment.get_obs()
    q[y, x] = np.asarray(agent.q_values(obs))
    pi[y, x] = np.asarray(agent.select_action(obs))

plot_action_values(q)
```

```
# @title Compare the greedy policy with the agent's policy

environment.plot_greedy_policy(q)
plt.figtext(-.08, .95, "Greedy policy using the learnt Q-values")
plt.title('')
plt.show()

environment.plot_policy(pi)
plt.figtext(-.08, .95, "Policy using the agent's epsilon-greedy policy")
plt.title('')
plt.show()
```

**Note:** You’ll get a better estimate of the value functions if you increase `num_episodes` and `max_episode_length`, but this will take longer to train. Feel free to play around after the day!

---
# Bonus 2: Beyond Value Based Model-Free Methods

*Time estimate: ~25mins*

```
# @title Video 10: Other RL Methods
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV14w411977Y", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"1N4Jm9loJx4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 10: Other RL Methods')

display(out)
```

## Cartpole task

Here we switch to training on a different kind of task, which has a continuous action space: Cartpole in [Gym](https://gym.openai.com/). As you recall from the video, policy-based methods are particularly well-suited for these kinds of tasks. We will be exploring two of those methods below.

```
# @title Make a CartPole environment, `gym.make('CartPole-v1')`
env = gym.make('CartPole-v1')

# Set seeds
env.seed(SEED)
set_seed(SEED)
```

## Bonus 2.1: Policy gradient

Now we will turn to policy gradient methods. Rather than defining the policy in terms of a value function, i.e. $\color{blue}\pi(\color{red}s) = \arg\max_{\color{blue}a}\color{green}Q(\color{red}s, \color{blue}a)$, we will directly parameterize the policy and write it as the distribution

\begin{equation}
\color{blue}a_t \sim \color{blue}\pi_{\theta}(\color{blue}a_t|\color{red}s_t).
\end{equation}

Here $\theta$ represent the parameters of the policy. We will update the policy parameters using gradient ascent to **maximize** expected future reward.

One convenient way to represent the conditional distribution above is as a function that takes a state $\color{red}s$ and returns a distribution over actions $\color{blue}a$.

Defined below is an agent which implements the REINFORCE algorithm.
REINFORCE (Williams 1992) is the simplest model-free general reinforcement learning technique.

The **basic idea** is to use probabilistic action choice. If the reward at the end turns out to be high, we make **all** actions in this sequence **more likely** (otherwise, we do the opposite).

This strategy could reinforce "bad" actions as well, however they will turn out to be part of trajectories with low reward and will likely not get accentuated.

From the lectures, we know that we need to compute

\begin{equation}
\nabla J(\theta)
= \mathbb{E}
\left[
  \sum_{t=0}^T \color{green} G_t
  \nabla\log\color{blue}\pi_\theta(\color{red}{s_t})
\right]
\end{equation}

where $\color{green} G_t$ is the sum over future rewards from time $t$, defined as

\begin{equation}
\color{green} G_t
= \sum_{n=t}^T \gamma^{n-t}
\color{green} R(\color{red}{s_t}, \color{blue}{a_t}, \color{red}{s_{t+1}}).
\end{equation}

The algorithm below will collect the state, action, and reward data in its buffer until it reaches a full trajectory. It will then update its policy given the above gradient (and the Adam optimizer).

A policy gradient trains an agent without explicitly mapping the value for every state-action pair in an environment by taking small steps and updating the policy based on the reward associated with that step. In this section, we will build a small network that trains using policy gradient using PyTorch.

The agent can receive a reward immediately for an action or it can receive the award at a later time such as the end of the episode.

The policy function our agent will try to learn is $\pi_\theta(a,s)$, where $\theta$ is the parameter vector, $s$ is a particular state, and $a$ is an action.

Monte-Carlo Policy Gradient approach will be used, which means the agent will run through an entire episode and then update policy based on the rewards obtained.

```
# @title Set the hyperparameters for Policy Gradient

num_steps = 300

learning_rate = 0.01  # @param {type:"number"}
gamma = 0.99  # @param {type:"number"}
dropout = 0.6 # @param {type:"number"}

# @markdown Only used in Policy Gradient Method:
hidden_neurons = 128  # @param {type:"integer"}
```

### Bonus Coding Exercise 2.1: Creating a simple neural network

Below you will find some incomplete code. Fill in the missing code to construct the specified neural network.

Let us define a simple feed forward neural network with one hidden layer of 128 neurons and a dropout of 0.6. Let's use Adam as our optimizer and a learning rate of 0.01. Use the hyperparameters already defined rather than using explicit values.

Using dropout will significantly improve the performance of the policy. Do compare your results with and without dropout and experiment with other hyper-parameter values as well.

```
class PolicyGradientNet(nn.Module):
  def __init__(self):
    super(PolicyGradientNet, self).__init__()
    self.state_space = env.observation_space.shape[0]
    self.action_space = env.action_space.n
    #################################################
    ## TODO for students: Define two linear layers
    ## from the first expression
    raise NotImplementedError("Student exercise: Create FF neural network.")
    #################################################
    # HINT: you can construct linear layers using nn.Linear(); what are the
    # sizes of the inputs and outputs of each of the layers? Also remember
    # that you need to use hidden_neurons (see hyperparameters section above).
    #   https://pytorch.org/docs/stable/generated/torch.nn.Linear.html
    self.l1 = ...
    self.l2 = ...
    self.gamma = ...
    # Episode policy and past rewards
    self.past_policy = Variable(torch.Tensor())
    self.reward_episode = []
    # Overall reward and past loss
    self.past_reward = []
    self.past_loss = []

  def forward(self, x):
    model = torch.nn.Sequential(
        self.l1,
        nn.Dropout(p=dropout),
        nn.ReLU(),
        self.l2,
        nn.Softmax(dim=-1)
    )
    return model(x)
```

```
# to_remove solution
class PolicyGradientNet(nn.Module):
  def __init__(self):
    super(PolicyGradientNet, self).__init__()
    self.state_space = env.observation_space.shape[0]
    self.action_space = env.action_space.n
    # HINT: you can construct linear layers using nn.Linear(); what are the
    # sizes of the inputs and outputs of each of the layers? Also remember
    # that you need to use hidden_neurons (see hyperparameters section above).
    #   https://pytorch.org/docs/stable/generated/torch.nn.Linear.html
    self.l1 = nn.Linear(self.state_space, hidden_neurons, bias=False)
    self.l2 = nn.Linear(hidden_neurons, self.action_space, bias=False)
    self.gamma = gamma
    # Episode policy and past rewards
    self.past_policy = Variable(torch.Tensor())
    self.reward_episode = []
    # Overall reward and past loss
    self.past_reward = []
    self.past_loss = []

  def forward(self, x):
    model = torch.nn.Sequential(
        self.l1,
        nn.Dropout(p=dropout),
        nn.ReLU(),
        self.l2,
        nn.Softmax(dim=-1)
    )
    return model(x)
```

Now let's create an instance of the network we have defined and use Adam as the optimizer using the learning_rate as hyperparameter already defined above.

```
policy = PolicyGradientNet()
pg_optimizer = optim.Adam(policy.parameters(), lr=learning_rate)
```

### Select Action

The `select_action()` function chooses an action based on our policy probability distribution using the PyTorch distributions package.  Our policy returns a probability for each possible action in our action space (move left or move right) as an array of length two such as [0.7, 0.3].  We then choose an action based on these probabilities, record our history, and return our action.

```
def select_action(state):
  #Select an action (0 or 1) by running policy model and choosing based on the probabilities in state
  state = torch.from_numpy(state).type(torch.FloatTensor)
  state = policy(Variable(state))
  c = Categorical(state)
  action = c.sample()

  # Add log probability of chosen action
  if policy.past_policy.dim() != 0:
    policy.past_policy = torch.cat([policy.past_policy, c.log_prob(action).reshape(1)])
  else:
    policy.past_policy = (c.log_prob(action).reshape(1))
  return action
```

### Update policy

This function updates the policy.

#### Reward $G_t$
We update our policy by taking a sample of the action value function $Q^{\pi_\theta} (s_t,a_t)$ by playing through episodes of the game.  $Q^{\pi_\theta} (s_t,a_t)$ is defined as the expected return by taking action $a$ in state $s$ following policy $\pi$.

We know that for every step the simulation continues we receive a reward of 1.  We can use this to calculate the policy gradient at each time step, where $r$ is the reward for a particular state-action pair.  Rather than using the instantaneous reward, $r$, we instead use a long term reward $ v_{t} $ where $v_t$ is the discounted sum of all future rewards for the length of the episode.   $v_{t}$ is then,

\begin{equation}
\color{green} G_t
= \sum_{n=t}^T \gamma^{n-t}
\color{green} R(\color{red}{s_t}, \color{blue}{a_t}, \color{red}{s_{t+1}}).
\end{equation}

where $\gamma$ is the discount factor (0.99).  For example, if an episode lasts 5 steps, the reward for each step will be [4.90, 3.94, 2.97, 1.99, 1].
Next we scale our reward vector by substracting the mean from each element and scaling to unit variance by dividing by the standard deviation.  This practice is common for machine learning applications and the same operation as Scikit Learn's __[StandardScaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)__.  It also has the effect of compensating for future uncertainty.

#### Update Policy: equation
After each episode we apply Monte-Carlo Policy Gradient to improve our policy according to the equation:

\begin{equation}
\Delta\theta_t = \alpha\nabla_\theta \, \log \pi_\theta (s_t,a_t)G_t
\end{equation}

We will then feed our policy history multiplied by our rewards to our optimizer and update the weights of our neural network using stochastic gradient **ascent**.  This should increase the likelihood of actions that got our agent a larger reward.

The following function ```update_policy``` updates the network weights and therefore the policy.

```
def update_policy():
  R = 0
  rewards = []

  # Discount future rewards back to the present using gamma
  for r in policy.reward_episode[::-1]:
    R = r + policy.gamma * R
    rewards.insert(0, R)

  # Scale rewards
  rewards = torch.FloatTensor(rewards)
  rewards = (rewards - rewards.mean()) / (rewards.std() +
                                          np.finfo(np.float32).eps)

  # Calculate loss
  pg_loss = (torch.sum(torch.mul(policy.past_policy,
                              Variable(rewards)).mul(-1), -1))
  # Update network weights
  # Use zero_grad(), backward() and step() methods of the optimizer instance.
  pg_optimizer.zero_grad()
  pg_loss.backward()
  # Update the weights
  for param in policy.parameters():
      param.grad.data.clamp_(-1, 1)
  pg_optimizer.step()

  # Save and intialize episode past counters
  policy.past_loss.append(pg_loss.item())
  policy.past_reward.append(np.sum(policy.reward_episode))
  policy.past_policy = Variable(torch.Tensor())
  policy.reward_episode= []
```

### Training
This is our main policy training loop.  For each step in a training episode, we choose an action, take a step through the environment, and record the resulting new state and reward.  We call update_policy() at the end of each episode to feed the episode history to our neural network and improve our policy.

```
def policy_gradient_train(episodes):
  running_reward = 10
  for episode in range(episodes):
    state = env.reset()
    done = False

    for time in range(1000):
      action = select_action(state)
      # Step through environment using chosen action
      state, reward, done, _ = env.step(action.item())

      # Save reward
      policy.reward_episode.append(reward)
      if done:
        break

    # Used to determine when the environment is solved.
    running_reward = (running_reward * gamma) + (time * (1 - gamma))

    update_policy()

    if episode % 50 == 0:
      print(f"Episode {episode}\tLast length: {time:5.0f}"
            f"\tAverage length: {running_reward:.2f}")

    if running_reward > env.spec.reward_threshold:
      print(f"Solved! Running reward is now {running_reward} "
            f"and the last episode runs to {time} time steps!")
      break
```

### Run the model

```
episodes = 500   #@param {type:"integer"}
policy_gradient_train(episodes)
```

### Plot the results

```
# @title Plot the training performance for policy gradient

def plot_policy_gradient_training():
  window = int(episodes / 20)

  fig, ((ax1), (ax2)) = plt.subplots(1, 2, sharey=True, figsize=[15, 4]);
  rolling_mean = pd.Series(policy.past_reward).rolling(window).mean()
  std = pd.Series(policy.past_reward).rolling(window).std()
  ax1.plot(rolling_mean)
  ax1.fill_between(range(len(policy.past_reward)),
                   rolling_mean-std, rolling_mean+std,
                   color='orange', alpha=0.2)
  ax1.set_title(f"Episode Length Moving Average ({window}-episode window)")
  ax1.set_xlabel('Episode'); ax1.set_ylabel('Episode Length')

  ax2.plot(policy.past_reward)
  ax2.set_title('Episode Length')
  ax2.set_xlabel('Episode')
  ax2.set_ylabel('Episode Length')

  fig.tight_layout(pad=2)
  plt.show()

plot_policy_gradient_training()
```

### Bonus Exercise 2.1: Explore different hyperparameters.

Try running the model again, by modifying the hyperparameters and observe the outputs. Be sure to rerun the function definition cells in order to pick up on the updated values.

What do you see when you

1. increase learning rate
2. decrease learning rate
3. decrease gamma ($\gamma$)
4. increase number of hidden neurons in the network

## Bonus 2.2: Actor-critic

Recall the policy gradient

\begin{equation}
\nabla J(\theta)
= \mathbb{E}
\left[
  \sum_{t=0}^T \color{green} G_t
  \nabla\log\color{blue}\pi_\theta(\color{red}{s_t})
\right]
\end{equation}

The policy parameters are updated using Monte Carlo technique and uses random samples. This introduces high variability in log probabilities and cumulative reward values. This leads to noisy gradients and can cause unstable learning.

One way to reduce variance and increase stability is subtracting the cumulative reward by a baseline:

\begin{equation}
\nabla J(\theta)
= \mathbb{E}
\left[
   \sum_{t=0}^T \color{green} (G_t  - b)
  \nabla\log\color{blue}\pi_\theta(\color{red}{s_t})
\right]
\end{equation}

Intuitively, reducing cumulative reward will make smaller gradients and thus smaller and more stable (hopefully) updates.

From the lecture slides, we know that in Actor Critic Method:
1. The “Critic” estimates the value function. This could be the action-value (the Q value) or state-value (the V value).
2. The “Actor” updates the policy distribution in the direction suggested by the Critic (such as with policy gradients).

Both the Critic and Actor functions are parameterized with neural networks. The "Critic" network parameterizes the Q-value.

```
# @title Set the hyperparameters for Actor Critic

learning_rate = 0.01  # @param {type:"number"}
gamma = 0.99  # @param {type:"number"}
dropout = 0.6

# Only used in Actor-Critic Method
hidden_size = 256  # @param {type:"integer"}

num_steps = 300
```

### Actor Critic Network

```
class ActorCriticNet(nn.Module):
  def __init__(self, num_inputs, num_actions, hidden_size, learning_rate=3e-4):
    super(ActorCriticNet, self).__init__()

    self.num_actions = num_actions
    self.critic_linear1 = nn.Linear(num_inputs, hidden_size)
    self.critic_linear2 = nn.Linear(hidden_size, 1)

    self.actor_linear1 = nn.Linear(num_inputs, hidden_size)
    self.actor_linear2 = nn.Linear(hidden_size, num_actions)

    self.all_rewards = []
    self.all_lengths = []
    self.average_lengths = []

  def forward(self, state):
    state = Variable(torch.from_numpy(state).float().unsqueeze(0))
    value = F.relu(self.critic_linear1(state))
    value = self.critic_linear2(value)

    policy_dist = F.relu(self.actor_linear1(state))
    policy_dist = F.softmax(self.actor_linear2(policy_dist), dim=1)

    return value, policy_dist
```

### Training

```
def actor_critic_train(episodes):
  all_lengths = []
  average_lengths = []
  all_rewards = []
  entropy_term = 0

  for episode in range(episodes):
    log_probs = []
    values = []
    rewards = []

    state = env.reset()
    for steps in range(num_steps):
      value, policy_dist = actor_critic.forward(state)
      value = value.detach().numpy()[0, 0]
      dist = policy_dist.detach().numpy()

      action = np.random.choice(num_outputs, p=np.squeeze(dist))
      log_prob = torch.log(policy_dist.squeeze(0)[action])
      entropy = -np.sum(np.mean(dist) * np.log(dist))
      new_state, reward, done, _ = env.step(action)

      rewards.append(reward)
      values.append(value)
      log_probs.append(log_prob)
      entropy_term += entropy
      state = new_state

      if done or steps == num_steps - 1:
        qval, _ = actor_critic.forward(new_state)
        qval = qval.detach().numpy()[0, 0]
        all_rewards.append(np.sum(rewards))
        all_lengths.append(steps)
        average_lengths.append(np.mean(all_lengths[-10:]))
        if episode % 50 == 0:
          print(f"episode: {episode},\treward: {np.sum(rewards)},"
                f"\ttotal length: {steps},"
                f"\taverage length: {average_lengths[-1]}")
        break

    # compute Q values
    qvals = np.zeros_like(values)
    for t in reversed(range(len(rewards))):
      qval = rewards[t] + gamma * qval
      qvals[t] = qval

    #update actor critic
    values = torch.FloatTensor(values)
    qvals = torch.FloatTensor(qvals)
    log_probs = torch.stack(log_probs)

    advantage = qvals - values
    actor_loss = (-log_probs * advantage).mean()
    critic_loss = 0.5 * advantage.pow(2).mean()
    ac_loss = actor_loss + critic_loss + 0.001 * entropy_term

    ac_optimizer.zero_grad()
    ac_loss.backward()
    ac_optimizer.step()

  # Store results
  actor_critic.average_lengths = average_lengths
  actor_critic.all_rewards = all_rewards
  actor_critic.all_lengths = all_lengths
```

### Run the model

```
episodes = 500   # @param {type:"integer"}

env.reset()

num_inputs = env.observation_space.shape[0]
num_outputs = env.action_space.n

actor_critic = ActorCriticNet(num_inputs, num_outputs, hidden_size)
ac_optimizer = optim.Adam(actor_critic.parameters())

actor_critic_train(episodes)
```

### Plot the results

```
# @title Plot the training performance for Actor Critic
def plot_actor_critic_training(actor_critic, episodes):
  window = int(episodes / 20)

  plt.figure(figsize=(15, 4))
  plt.subplot(1, 2, 1)

  smoothed_rewards = pd.Series(actor_critic.all_rewards).rolling(window).mean()
  std = pd.Series(actor_critic.all_rewards).rolling(window).std()

  plt.plot(smoothed_rewards, label='Smoothed rewards')
  plt.fill_between(range(len(smoothed_rewards)),
                   smoothed_rewards - std, smoothed_rewards + std,
                   color='orange', alpha=0.2)

  plt.xlabel('Episode')
  plt.ylabel('Reward')

  plt.subplot(1, 2, 2)
  plt.plot(actor_critic.all_lengths, label='All lengths')
  plt.plot(actor_critic.average_lengths, label='Average lengths')
  plt.xlabel('Episode')
  plt.ylabel('Episode length')
  plt.legend()

  plt.tight_layout()
  plt.show()

plot_actor_critic_training(actor_critic, episodes)
```

### Bonus Exercise 2.2.1: Effect of episodes on performance

Change the episodes from 500 to 3000 and observe the performance impact.

### Bonus Exercise 2.2.2: Effect of learning rate on performance

Modify the hyperparameters related to learning_rate and gamma and observe the impact on the performance.

Be sure to rerun the function definition cells in order to pick up on the updated values.

---
# Appendix and further reading

Books and lecture notes
*   [Reinforcement Learning: an Introduction by Sutton & Barto](http://incompleteideas.net/book/RLbook2018.pdf)
* [Algorithms for Reinforcement Learning by Csaba Szepesvari](https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf)

Lectures and course
*   [RL Course by David Silver](https://www.youtube.com/playlist?list=PLzuuYNsE1EZAXYR4FJ75jcJseBmo4KQ9-)
*   [Reinforcement Learning Course | UCL & DeepMind](https://www.youtube.com/playlist?list=PLqYmG7hTraZBKeNJ-JE_eyJHZ7XgBoAyb)
*   [Emma Brunskill Stanford RL Course](https://www.youtube.com/playlist?list=PLoROMvodv4rOSOPzutgyCTapiGlY2Nd8u)
*   [RL Course on Coursera by Martha White & Adam White](https://www.coursera.org/specializations/reinforcement-learning)

More practical:
* [Spinning Up in Deep RL by Josh Achiam](https://spinningup.openai.com/en/latest/)
*   [Acme white paper](https://arxiv.org/abs/2006.00979) & [Colab tutorial](https://github.com/deepmind/acme/blob/master/examples/tutorial.ipynb)

[Link to the tweet thread with resources recommended by the community](https://twitter.com/FeryalMP/status/1407272291579355136?s=20).

This Colab is based on the [EEML 2020 RL practical](https://colab.research.google.com/github/eemlcommunity/PracticalSessions2020/blob/master/rl/EEML2020_RL_Tutorial.ipynb) by Feryal Behbahani & Gheorghe Comanici. If you are interested in JAX you should try the colab. If you are interested in Tensorflow, there is also a version of the colab for the [MLSS 2020 RL Tutorial](https://github.com/Feryal/rl_mlss_2020) that you can try :)