# Deep Reinforcement Learning

In this session, we will use Open AI Gym to implement Q-learning, a classic algorthim in RL, and Deep Q-Networks (DQN), its deep learning counterpart. Hopefully this will give us some intuition for what we gain by moving from tabular methods to using neural networks for function approximation.

## Setup

First, let's install the necessary dependencies. This does not need to be run again if you restart the runtime, but it does if you factory reset the runtime.

```
!sudo apt-get install -y xvfb ffmpeg
!apt-get install x11-utils
!pip install 'gym==0.17.1'
!pip install 'pyglet==1.4.0'
!pip install pyvirtualdisplay
!pip install --upgrade tensorflow-probability
!pip install imageio-ffmpeg
```

    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    ffmpeg is already the newest version (7:3.4.6-0ubuntu0.18.04.1).
    The following NEW packages will be installed:
      xvfb
    0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
    Need to get 784 kB of archives.
    After this operation, 2,266 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.4 [784 kB]
    Fetched 784 kB in 2s (355 kB/s)
    debconf: unable to initialize frontend: Dialog
    debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)
    debconf: falling back to frontend: Readline
    debconf: unable to initialize frontend: Readline
    debconf: (This frontend requires a controlling tty.)
    debconf: falling back to frontend: Teletype
    dpkg-preconfigure: unable to re-open stdin:
    Selecting previously unselected package xvfb.
    (Reading database ... 144568 files and directories currently installed.)
    Preparing to unpack .../xvfb_2%3a1.19.6-1ubuntu4.4_amd64.deb ...
    Unpacking xvfb (2:1.19.6-1ubuntu4.4) ...
    Setting up xvfb (2:1.19.6-1ubuntu4.4) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following additional packages will be installed:
      libxxf86dga1
    Suggested packages:
      mesa-utils
    The following NEW packages will be installed:
      libxxf86dga1 x11-utils
    0 upgraded, 2 newly installed, 0 to remove and 25 not upgraded.
    Need to get 209 kB of archives.
    After this operation, 711 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga1 amd64 2:1.1.4-1 [13.7 kB]
    Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11-utils amd64 7.7+3build1 [196 kB]
    Fetched 209 kB in 2s (112 kB/s)
    Selecting previously unselected package libxxf86dga1:amd64.
    (Reading database ... 144575 files and directories currently installed.)
    Preparing to unpack .../libxxf86dga1_2%3a1.1.4-1_amd64.deb ...
    Unpacking libxxf86dga1:amd64 (2:1.1.4-1) ...
    Selecting previously unselected package x11-utils.
    Preparing to unpack .../x11-utils_7.7+3build1_amd64.deb ...
    Unpacking x11-utils (7.7+3build1) ...
    Setting up libxxf86dga1:amd64 (2:1.1.4-1) ...
    Setting up x11-utils (7.7+3build1) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Processing triggers for libc-bin (2.27-3ubuntu1) ...
    /sbin/ldconfig.real: /usr/local/lib/python3.6/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

    Requirement already satisfied: gym==0.17.1 in /usr/local/lib/python3.6/dist-packages (0.17.1)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.4.1)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.12.0)
    Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.4.0)
    Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.18.3)
    Requirement already satisfied: cloudpickle<1.4.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.3.0)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym==0.17.1) (0.16.0)
    Requirement already satisfied: pyglet==1.4.0 in /usr/local/lib/python3.6/dist-packages (1.4.0)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet==1.4.0) (0.16.0)
    Requirement already satisfied: pyvirtualdisplay in /usr/local/lib/python3.6/dist-packages (0.2.5)
    Requirement already satisfied: EasyProcess in /usr/local/lib/python3.6/dist-packages (from pyvirtualdisplay) (0.2.10)
    Requirement already up-to-date: tensorflow-probability in /usr/local/lib/python3.6/dist-packages (0.10.0rc0)
    Requirement already satisfied, skipping upgrade: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (1.12.0)
    Requirement already satisfied, skipping upgrade: gast>=0.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (0.3.3)
    Requirement already satisfied, skipping upgrade: cloudpickle>=1.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (1.3.0)
    Requirement already satisfied, skipping upgrade: decorator in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (4.4.2)
    Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (1.18.3)
    Requirement already satisfied: imageio-ffmpeg in /usr/local/lib/python3.6/dist-packages (0.4.1)

Now let's import the packages we will be using.

```
from __future__ import absolute_import, division, print_function

import base64
import imageio
import IPython
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import PIL.Image
import pyvirtualdisplay
import random
import tensorflow as tf
import gym

from gym.spaces import Discrete
from gym.spaces import Box
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam
from collections import defaultdict
from collections import deque
from pyvirtualdisplay import Display
```

```
print(tf.version.VERSION)
print(gym.version.VERSION)
```

    2.2.0-rc3
    0.17.1

## Introduction to Open AI Gym

Gym provides you with a set of [environments](https://gym.openai.com/envs/#classic_control). If we think of the classic RL framework schematic, Gym takes care of the environment, and you take care of the agent.

From this diagram, we can expect that we will interact with a Gym environment by giving it an action as input, and receiving a next state and reward as output.

It is then our job to implement some algorithm to learn the policy $\pi(a|s)$ for our agent to use to act within the environment.

### Setting up visualization

First, let's get plotting for Gym working in Colab. This will help give us an intuitive feel for how to work with the Gym environments.

```
def embed_mp4(filename):
    """Embeds an mp4 file in the notebook."""
    video = open(filename,'rb').read()
    b64 = base64.b64encode(video)
    tag = '''
    '''.format(b64.decode())

    return IPython.display.HTML(tag)
```

Before we get to implementing algorithms for our agents to learn a good policy, let's visualize an agent acting according to a random policy. At this point the visualizatation is just to give us an idea of what what an environment looks like, and later on we'll come back to see how we generate this video.

```
def create_random_policy_video(env, filename, num_episodes=5, fps=30):
    """Generates a visualization of an agent acting according to a random
    policy in the given environment."""
    display = Display(visible=0, size=(400, 300))
    display.start()
    filename = filename + ".mp4"
    with imageio.get_writer(filename, fps=fps) as video:
        for _ in range(num_episodes):
            done = False
            observation = env.reset()
            video.append_data(env.render(mode='rgb_array'))
            while not done:
                action = env.action_space.sample()
                observation, reward, done, info = env.step(action)
                video.append_data(env.render(mode='rgb_array'))
    display.stop()
    return embed_mp4(filename)
```

```
env = gym.make("MsPacman-v0")
create_random_policy_video(env, "video", num_episodes=1)
```

    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (210, 160) to (224, 160) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

This is pretty cool! In this example Gym gives us a great visualization of our agent playing PacMan (though not very well).

### Gym basics

#### Environments

As we would expect, an environment is defined by its state and action spaces.

First, of all there are two types of Gym [spaces](https://gym.openai.com/docs/#spaces):
- **`observation_space`**: defines the state space of the environment
- **`action_space`**: defines the action space of the environment

Spaces can be:
- `gym.spaces.Discrete`: fixed range of n values
- `gym.spaces.Box`: n-dimensional box

You can inspect the valid range for a `Box` by calling `Box.low()` and `Box.high()`.

```
env = gym.make("MsPacman-v0")
print(f'Action space: {env.action_space}')
print(f'Observation space: {env.observation_space}')
```

    Action space: Discrete(9)
    Observation space: Box(210, 160, 3)

We can see here that for `MsPacman-v0`, the `action_space` consits 9 possible actions, and the `observation_space` is a 210 x 160 x 3 box (rgb-image). We can also extract these dimensions using `Discrete.n` and `Box.shape`.

```
print(env.action_space.n)
print(env.observation_space.shape)
```

    9
    (210, 160, 3)

If we're curious, we can find the action meanings by calling,

```
print(env.unwrapped.get_action_meanings())
```

    ['NOOP', 'UP', 'RIGHT', 'LEFT', 'DOWN', 'UPRIGHT', 'UPLEFT', 'DOWNRIGHT', 'DOWNLEFT']

I would've guessed that the `action_space` would be just up, down, left, right, but apparently this implementation includes combination actions as well. Theoretically you don't "need" to know these details about the environment you're using because your algorithm should learn a good policy given whatever the available action space is, but I think it's still nice to get a sense of.

#### Key functions for interacting with the environment

We will mainly use three functions for interacting with the environment.

**`observation = env.reset()`**

- This function returns the starting state of an environment. We will call this function any time we want to start a new episode.

**`observation, reward, done, info = env.step(action)`**

- This function is how your agent takes actions in the environment; it defines the transition and reward function. It takes in an `action` as an argument, and returns the next `observation`(next_state), the `reward` (**float**), if the episode is `Done` (**bool**), and `info`, which we won't be using here but can contain helpful information for debugging.

**`action = env.action_space.sample()`**
- This is a helpful function for sampling a random action from the `action_space`. We will be using the $\epsilon$-greedy exploration strategy, so we will use this function when we want to select a random action.

If we look back at the code for `create_random_policy_video()`, we can see how we used these three functions to get the data for the video. Stripping away all code for plotting, the main loop is:

```
num_episodes = 1
env = gym.make("MsPacman-v0")

for _ in range(num_episodes):
    observation = env.reset()
    done = False
    while not done:
        action = env.action_space.sample()
        observation, reward, done, info = env.step(action)
```

In this notebook, we will generally replace the term `observation` with `state` becuase this is the wording we're more familiar with.

## Implementing RL algorithms

Now that we have all of the setup done, let's get to the fun part of actually implementing the algorithms!

### CartPole environment

For our implementations we are going to use the `CartPole-v1` environment. This is a simple environment where both of our algorithms (Q-learning and DQN) will be able to learn a good policy within a reasonably short amount of time.

```
env = gym.make("CartPole-v1")
```

The goal in this environment is to move the cart left or right in order to balance the pole so that it remains upright.

```
create_random_policy_video(env, "video", num_episodes= 10)
```

    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

```
print(f'Action space: {env.action_space}')
print(f'State space: {env.observation_space}')
```

    Action space: Discrete(2)
    State space: Box(4,)

This is a dramatically simpler environment than the MsPacman environment. The `observation_space` is a 4 dimensional array, and there are 2 possible actions. The CartPole [documentation](https://github.com/openai/gym/wiki/CartPole-v0) tells us the meanings of the observation and action spaces.

    Observation:
        Type: Box(4)
        Num	Observation               Min             Max
        0	Cart Position             -4.8            4.8
        1	Cart Velocity             -Inf            Inf
        2	Pole Angle                -24 deg         24 deg
        3	Pole Velocity At Tip      -Inf            Inf

    Actions:
        Type: Discrete(2)
        Num	Action
        0	Push cart to the left
        1	Push cart to the right

An episode terminates when the pole falls more than 12 degrees from vertical, the cart position moves off-screen, or the number of steps within the episode exceeds 500.

### Required functions for learning a policy

```
class Agent:

    def create_model(self):
        """This model will be used by the act() method to select actions,
        and will be updated during training """
        pass

    def act(self, test=False):
        """This function implements your policy and choose actions based on the
        current model. If test=True, actions chosen without exploration."""
        pass

    def update_model(self):
        """This function specifies how to update the model based on experience
        in the environment"""
        pass

    def train(self):
        """The main loop for training the model by selecting actions,
        interacting with the environment, and updating the model"""
        pass
```

Once we have a trained model, we can evaluate it's performance using a similar loop to the one above used to visualize the random policy. The only difference is that we replace the `env.action_space.sample()` function call with `agent.act()`.

    agent = Agent()
    agent = Agent.train()

    # run trained agent
    for _ in range(num_episodes):
        state = agent.env.reset()
        done = False
        while not done:
            action = agent.env.act(state, test=True)
            next_state, reward, done, info = env.step(action)

### Evaluation functions

Now we can use this loop to generate a video  visualizing the learned policy.

```
def create_learned_policy_video(agent, filename, num_episodes=5, fps=30):
    """Generates a video of the given agent acting accoring to its learned
    policy for the specified number of episodes."""
    display = Display(visible=0, size=(400, 300))
    display.start()
    filename = filename + ".mp4"
    with imageio.get_writer(filename, fps=fps) as video:
        for _ in range(num_episodes):
            done = False
            state = agent.env.reset()
            video.append_data(agent.env.render(mode='rgb_array'))
            while not done:
                action = agent.act(state, test=True)
                state, reward, done, info = agent.env.step(action)
                video.append_data(agent.env.render(mode='rgb_array'))
    display.stop()
    return embed_mp4(filename)
```

We will also want to evaluate the performance of the learned model. For evaluation trials we will not use $\epsilon$-greedy exploration, but instead always choose the best action according to our learned policy.

```
def evaluate_policy(agent, num_episodes=10):
    """Runs the agent through the specified number of episodes and prints the
    average return. """
    reward_history = []
    for _ in range(num_episodes):
        state = agent.env.reset()
        total_reward = 0
        done = False
        while not done:
            action = agent.act(state, test=True)
            next_state, reward, done, _ = agent.env.step(action)
            total_reward += reward
            state = next_state
        reward_history.append(total_reward)

    print("Exploit reward average: {}".format(np.mean(reward_history).round(2)))
```

## Q-Learning

Tabular Q-learning stores and updates a Q-value estimate for each state-action pair, $Q(s,a)$. Each of these Q-values is stored in a look-up table.

**Discretize environment observations**

Since the cartpole environment has an `observation_space` with continuous values, the number of Q-values we need to store and update would quickly explode, and also not be very useful. To avoid this we are going to use a wrapper for the environment that transforms the `obervation_space` from a continuous-valued `Box` to a discrete-valued `Discrete`. I got this wrapper from Lillian Weng's Q-learning [implementation](https://github.com/lilianweng/deep-reinforcement-learning-gym/blob/master/playground/utils/wrappers.py) (understanding the details of how she implements this isn't important for what we're focusing on).

```
class DiscretizedObservationWrapper(gym.ObservationWrapper):
    """This wrapper converts a Box observation into a single integer.
    """
    def __init__(self, env, n_bins=10, low=None, high=None):
        super().__init__(env)
        assert isinstance(env.observation_space, Box)

        low = self.observation_space.low if low is None else low
        high = self.observation_space.high if high is None else high

        self.n_bins = n_bins
        self.val_bins = [np.linspace(l, h, n_bins + 1) for l, h in
                         zip(low.flatten(), high.flatten())]
        self.observation_space = Discrete(n_bins ** low.flatten().shape[0])

    def _convert_to_one_number(self, digits):
        return sum([d * ((self.n_bins + 1) ** i) for i, d in enumerate(digits)])

    def observation(self, observation):
        digits = [np.digitize([x], bins)[0]
                  for x, bins in zip(observation.flatten(), self.val_bins)]
        return self._convert_to_one_number(digits)
```

### Algorithm

The next step is to implement the q-learning algorithm. The method names give you the skeleton of the implementation but the content is left for you to fill in. We've inserted detailed comments to guide your implementation. We've also left some code in that is not essential to the algorithm (e.g decaying the epsilon parameter each step, keeping track of reward history).

You need to fill in the content for three methods:

- `create_model()` - filled in already
- `act()`
- `update_model()`
- `train()`

#### create_model()

We've left the code for `create_model()` filled in because it is only creating a dictionary for storing the Q values. We are using `defaultdict (float)` rather than `dict` for a more efficient implementation. This automatically initializes any key entry to 0.0, rather than returning `KeyError`.

```
# define environement
env = gym.make("CartPole-v1")
env_discrete = DiscretizedObservationWrapper(
    env,
    n_bins=8,
    low=np.array([-2.4, -2.0, -0.42, -3.5]),
    high=np.array([2.4, 2.0, 0.42, 3.5])
)

# get example state-action pair
state = env_discrete.reset()
action = env_discrete.action_space.sample()

# define defaultdict and query the state-action pair
example = defaultdict(float)
example[state, action]  # *no KeyError*
```

    0.0

#### act()

For our implementation, we will be using the $\epsilon$-greedy exploration policy.

\begin{equation}
  a =
    \begin{cases}
      \text{random} & \text{with probability $\epsilon$}\\
      \arg\max_a Q(s,a)   & \text{otherwise}\\
    \end{cases}
\end{equation}

#### update_model()

This function should update the Q-value estimate using the Q-learning update rule based on the $(s,a,r,s'\text{done})$.

$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left[r_t + \gamma \max_{a'} Q(s',a') - Q(s,a) \right]$$

If the state is terminal (`done=True`) the update will be
$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left[r_t  - Q(s,a) \right]$$

#### train()

This function will run the main training loop. Here is the pseudocode for the Q-learning algorithm.

    create model (initialize q_values)
    for n_episodes
        initialize state
        while not done
            select action according to policy
            execute action; observe reward and next_state
            update model

This function will be used to train the agent as follows:

    agent = Agent(env)
    agent = Agent.train()

Remember, these are the environment api calls you will need to use in your implementation.
- `observation = env.reset()`
- `observation, reward, done, info = env.step(action)`
- `action = env.action_space.sample()`

#### Implementation

```
class QLearning:

    def __init__(self, env, gamma=0.9, alpha=0.5, epsilon=0.99,
                 epsilon_decay=0.9999, epsilon_min=0.1):
        self.env = env
        self.gamma = gamma
        self.alpha = alpha
        self.epsilon = epsilon
        self.epsilon_decay = epsilon_decay
        self.epsilon_min = epsilon_min
        self.actions = range(self.env.action_space.n)

    def create_model(self):
        """"For Q-learning the model is simply a dictionary for storing the
        tabular Q values."""
        self.Q = defaultdict(float)

    def act(self, state, test=False):
        """Choose action based on your current model using epsilon-greedy
        exploration"""
        # update epsilon
        self.epsilon *= self.epsilon_decay
        self.epsilon = max(self.epsilon_min, self.epsilon)
        epsilon = 0 if test else self.epsilon
        # take a random action with probability epsilon
        if np.random.random() < epsilon:
            return self.env.action_space.sample()
        # Pick the action with highest q value.
        qvals = {action: self.Q[state, action] for action in self.actions}
        max_q = max(qvals.values())
        # In case multiple actions have the same maximum q value.
        actions_with_max_q = [action for action, q in qvals.items() if q == max_q]
        action = np.random.choice(actions_with_max_q)
        return action

    def update_model(self, state, action, reward, next_state, done):
        # get max q value in next state
        max_q_next = max([self.Q[next_state, action] for a in self.actions])
        # Do not include the next state's value if currently at the terminal state.
        max_q_next = max_q_next * (1.0 - done)
        # Update q value of current state-action pair
        self.Q[state, action] += self.alpha * \
            (reward + self.gamma * max_q_next - self.Q[state, action])

    def train(self, num_episodes=20):
        """This is the main training loop for interacting with the environment
        and updating your model. We've left in code for storing training history."""
        self.reward_history = []
        # create model
        self.create_model()
        for episode in range(num_episodes):
            total_reward = 0.0
            done = False
            # initialize state
            state = self.env.reset()
            while not done:
                # select action according to policy
                action = self.act(state)
                # execute action
                next_state, reward, done, _ = self.env.step(action)
                # update model
                self.update_model(state, action, reward, next_state, done)
                # keep track of total reward in episode
                total_reward += reward
                # update state
                state = next_state

            # save total reward from epsisode and print training progress
            self.reward_history.append(total_reward)
            if episode % 500 == 0:
                print("episode {}: {} average reward".format(
                    episode,
                    np.mean(self.reward_history[max(0,episode-500):episode+1]).round(2)))

```

### Training an agent

Once you have your algorithm implemented, let's train an agent!

```
env = gym.make('CartPole-v1')
env_discrete = DiscretizedObservationWrapper(
    env,
    n_bins=8,
    low=np.array([-2.4, -2.0, -0.42, -3.5]),
    high=np.array([2.4, 2.0, 0.42, 3.5])
)

seed = 0
env.seed(seed)
env.action_space.seed(seed)
np.random.seed(seed)
random.seed(seed)

qlearning_agent = QLearning(env_discrete)
qlearning_agent.train(num_episodes=5000)
```

    episode 0: 52.0 average reward
    episode 500: 42.61 average reward
    episode 1000: 79.64 average reward
    episode 1500: 94.46 average reward
    episode 2000: 87.87 average reward
    episode 2500: 96.78 average reward
    episode 3000: 101.91 average reward
    episode 3500: 118.41 average reward
    episode 4000: 117.37 average reward
    episode 4500: 113.55 average reward

```
# visualize total reward per episode across training
plt.figure(figsize=(7,4))
plt.plot(qlearning_agent.reward_history, alpha=.3, color='teal', label='raw')
plt.plot(np.convolve(qlearning_agent.reward_history, np.ones((50,))/50, mode='valid'), color='purple', label='smoothed')
plt.xlabel('episode #', fontsize=15)
plt.ylabel('total reward per episode', fontsize=15)
plt.legend()
plt.show()
```

###Evaluating the agent

First, let's see what the average reward is across 100 trials when the agent is exploiting its learned policy (not using $\epsilon$-greedy exploration).

Now, let's visualize the agent acting according to its learned policy.

```
evaluate_policy(qlearning_agent, num_episodes=100)
```

    Exploit reward average: 121.67

```
create_learned_policy_video(qlearning_agent, "video", num_episodes=1)
```

    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

Woo, it learned something! This is definitely an improvement from random, but it's certainly not optimal (a reward of 500 is optimal). This agent could get better with more training but it would probably take a long time for it to reach optimal performance.

### What is the model learning?

We can tell that the model has learned something, but it would be nice to get some idea of what it's learned. In order to get a sense of this, we can visualize the learned Q-values across a set of states. For this example, we are going to plot Q-values as a function of pole velocity, while the cart position, cart velocity, and pole angle are all 0 (the pole is in the center, not moving, and upright).

Intuitively, the agent should have learned to push the cart right if the pole velocity is to the right (>0), and to push the cart left if the pole velocity is to the left (<0).

```
n_obs = 40
obs_array = np.zeros((n_obs, env.observation_space.shape[0]))
obs_array[:,0] = 0
obs_array[:,1] = 0
obs_array[:,2] = 0
obs_array[:,3] = np.linspace(-5, 5, n_obs)
# run model
q_values = np.zeros((n_obs, env.action_space.n))
for i, obs in enumerate(obs_array):
    obs_disctete = env_discrete.observation(obs)
    q_values[i] = [qlearning_agent.Q[obs_disctete, action] for action in qlearning_agent.actions]
# visualize results
plt.figure(figsize=(8,5))
plt.plot(obs_array[:,3], q_values[:,0], color='purple', label='push cart left', linewidth=2)
plt.plot(obs_array[:,3], q_values[:,1], color='green', label='push cart right', linewidth=2)
plt.vlines(0, q_values.min(), q_values.max(), linestyle='--', color='dimgray')
plt.xlabel('Pole Velocity', fontsize=15)
plt.ylabel('Q value', fontsize=15)
plt.title('Cart Position=0, Cart Velocity=0, Pole Angle=0', fontsize=15)
plt.legend(fontsize=13)
plt.show()
```

It does what we expect! The Q-values for a=right are larger than the Q-values for a=left when the pole velocity is greater than 0, and vice versa for when the pole velocity is less than 0.

## Deep Q-Networks (DQN)

Now that we've implemented Q-learning, let's move on to implementing DQNs!

### Algorithm

Similar to Q-Learning, the method names for the DQN class give you the skeleton of the implementation, while the content is left for you to fill in. For DQNs you will need to write a few more functions than you needed for Q-Learning.

You need to fill in the content for six functions:

- `create_model` - filled in already
- `act()`
- `remember()`
- `update_model()`
- `update_target()` - fillded in already
- `train()`

#### create_model()

For this implementation, we're going to use a two-layer densely connected network. This network will take in a state as input and output a Q-value estimate for each action within this state. Consequently, the input dim is the same as the `observation_space` shape (4 for the CartPole environment), and the output dim is the same as the `action_space` shape (2 for the CartPole environment).

We've left the code for `create_model()` filled in because it is largely determined by learning TensorFlow syntax, which isn't our focus here. We will use ReLu activation function and the Adam optimizer.

We will use mean squared error loss, as specified by the DQN loss function.

    def create_model(self):
        model = Sequential()
        model.add(Dense(24, input_dim=self.state_shape[0], activation="relu"))
        model.add(Dense(16, activation="relu"))
        model.add(Dense(self.env.action_space.n))
        model.compile(loss="mean_squared_error", optimizer=Adam(lr=self.learning_rate))
        return model

#### act()

We will again use the $\epsilon$-greedy exploration policy.

\begin{equation}
  a =
    \begin{cases}
      \text{random} & \text{with probability $\epsilon$}\\
      \arg\max_a Q(s,a;\theta)   & \text{otherwise}\\
    \end{cases}
\end{equation}

To get the Q-values from your Q-network, you can run

    q_values = self.model.predict(state)

#### remember()

After each step, we need to store the $(s, a,r,s', \text{done})$ experience in the replay memory. In this implementation we will store memories in a `deque` with a specified maximum legnth (memory capacity).

```
replay_memory = deque(maxlen=5)
print(replay_memory)

for i in range(7):
    replay_memory.append(i)
    print(replay_memory)

# in your implementation you will append the experience
# [state, action, reward, next_state, done] instead of i
```

    deque([], maxlen=5)
    deque([0], maxlen=5)
    deque([0, 1], maxlen=5)
    deque([0, 1, 2], maxlen=5)
    deque([0, 1, 2, 3], maxlen=5)
    deque([0, 1, 2, 3, 4], maxlen=5)
    deque([1, 2, 3, 4, 5], maxlen=5)
    deque([2, 3, 4, 5, 6], maxlen=5)

#### update_model()

To update the model, you'll need to:

1. sample a batch of experiences $(s_j, a_j, r_j, s_j', \text{done}_j)$ from memory
2. calculate the target output (`done` indicates if $s$ is terminal)

\begin{equation}
  y_j =
    \begin{cases}
      r_j + \gamma \max_{a'} Q(s_j',a'; \theta^-) & \text{if $s$ is not terminal}\\
      r_j   & \text{if $s$ is terminal}\\
    \end{cases}
\end{equation}

3. Perform gradient descent step according to

$$ L(\theta) = \left\langle \big( y_j - Q(s_j,a_j; \theta) \big)^2\right\rangle_{(s,a,r,s') \sim Uniform(Memory)}$$

For the third step, the TensorFlow code you will need is

    model.fit(batch_states, batch_target, epochs=1, verbose=0)

**NOTE**

The `batch_target` must be the same dimensions as the model output. This means you must have a target for every action for each input state in your batch of experiences $(s_j,a_j,r_j,s_j')$. For each action $a$ that is not in the experience batch, use the current output of the target model as the target value, $Q(s,a;\theta^-)$. Therefore, for each state $s_j$,

\begin{equation}
  \text{target} =
    \begin{cases}
      y_j             & \text{if $a$ is in experience batch $(s_j, a_j, r_j, s_j')$}\\
      Q(s,a;\theta^-)   & \text{if $a$ is NOT in experience batch $(s_j, a_j, r_j, s_j')$}\\
    \end{cases}
\end{equation}

**NOTE 2**

Here is a helpful line of code for reformating samples from memory, each of which will be a list `[state, action, reward, next_state, done]`, to a set of `np.array`s with dim (n_batch x __).

    batch_states, batch_actions, batch_rewards, batch_next_states, batch_done = map(np.asarray, zip(*memory_samples))

#### update_tartget()

This function is used to set the target network weights equal to the model network weights. This is only done periodically thoughout training, which reduces variance in the gradient across steps and stabilizes training.

We've left the code for `update_target()` filled in because, again, it is largely just Tensorflow syntax. You'll have to use this function appropriately within the main training loop.

    def update_target(self):
        weights = self.model.get_weights()
        self.target_model.set_weights(weights)

#### train()

This function will run the main training loop. Here is the pseudocode for the DQN algorithm.

    initialize Q-network (create model)
    initialize target network (create model and set weights equal to Q-network)
    for n_episodes
        initialize state
        while not done
            select action according to policy
            execute action; observe reward and next_state
            add experience (state, action, reward, next_state, done) to memory
            sample batch of (state, action, reward, next_state, done) experiences from memory and update model
            every C steps, update target model

Same as for Q-learning, the Gym api calls you will need are:
- `observation = env.reset()`
- `observation, reward, done, info = env.step(action)`
- `action = env.action_space.sample()`

The Tensorflow api calls you will need are:
- `model_output = model.predict(model_input)`
- `model.fit(model_input, model_target, epochs=1, verbose=0)`

#### Implementation

```
class DQN:
    def __init__(self, env, memory_cap=1000, gamma=0.9, epsilon=0.99,
        epsilon_decay=0.995, epsilon_min=0.01, learning_rate=0.005,
        batch_size=32, C=20):

        self.env = env
        self.memory = deque(maxlen=memory_cap)
        self.state_shape = env.observation_space.shape

        self.gamma = gamma
        self.epsilon = epsilon
        self.epsilon_min = epsilon_min
        self.epsilon_decay = epsilon_decay
        self.learning_rate = learning_rate
        self.batch_size = batch_size
        self.C = C

    def create_model(self):
        """We will use a two-layer perceptron. The input dim must equal the
        state space dim and the output dim must equal the action space dim,
        but you can play around with the size of the hidden layers. For DQNs,
        we need mean squared error loss."""
        model = Sequential()
        model.add(Dense(24, input_dim=self.state_shape[0], activation="relu"))
        model.add(Dense(16, activation="relu"))
        model.add(Dense(self.env.action_space.n))
        model.compile(loss="mean_squared_error", optimizer=Adam(lr=self.learning_rate))
        return model

    def act(self, state, test=False):
        """Choose action based on your current model using epsilon-greedy
        exploration"""
        # update epsilon
        self.epsilon *= self.epsilon_decay
        self.epsilon = max(self.epsilon_min, self.epsilon)
        epsilon = 0.01 if test else self.epsilon
        # take random action with probability epsilon
        if np.random.random() < epsilon:
            return self.env.action_space.sample()
        # reshape state to feed into model (tensorflow thing, shape must be
        # (1, input_dim), not (input_dim,).)
        state = state.reshape((1, self.state_shape[0]))
        # get q_values from model
        q_values = self.model.predict(state)[0]
        # get action (argmax of Q-values)
        action = np.argmax(q_values)
        return action

    def remember(self, state, action, reward, new_state, done):
        """Append experience to memory"""
        self.memory.append([state, action, reward, new_state, done])

    def update_model(self):
        """This function updates the q-network model. You must 1) sample a
        batch of experiences from the replay memory 2) calculate the target for
        each expereince, 3) update the model by calling model.fit()"""
        # only update model once have sufficient number of experiences in memory
        if len(self.memory) < self.batch_size:
            return
        # sample a batch of experiences from memory
        memory_samples = random.sample(self.memory, self.batch_size)
        # reformat samples
        batch_states, batch_actions, batch_rewards, batch_next_states, batch_done = map(np.asarray, zip(*memory_samples))
        # get target model predictions for current state
        batch_target = self.target_model.predict(batch_states)
        # get target model predictions for future state
        q_future = self.target_model.predict(batch_next_states).max(axis=1)
        # for actions within experience batch, replace target model predictions
        # with the target value (r + gamma * max Q(s,a) if s not terminal, r is s terminal)
        batch_target[range(self.batch_size), batch_actions] = batch_rewards + (1 - batch_done) * q_future * self.gamma
        self.model.fit(batch_states, batch_target, epochs=1, verbose=0)

    def update_target(self):
        """"Sets target weights equal to model weights."""
        weights = self.model.get_weights()
        self.target_model.set_weights(weights)

    def train(self, num_episodes=50):
        """This function implements the main training loop."""
        # keep track of total reward per episode
        self.reward_history = []
        # initialize model and target model
        self.model = self.create_model()
        self.target_model = self.create_model()
        self.target_model.set_weights(self.model.get_weights())
        # we need to keep track of steps now so we can update the model every
        # C steps
        step = 0
        # interact with environment and update model
        for episode in range(num_episodes):
            total_reward = 0
            done = False
            state = self.env.reset()
            while not done:
                # select action according to policy
                action = self.act(state)
                # execute action
                next_state, reward, done, _ = self.env.step(action)
                # add experience to memory
                self.remember(state, action, reward, next_state, done)
                # update model (sample batch of experiences for update)
                self.update_model()

                total_reward += reward
                step += 1
                state = next_state
                if step % self.C:
                    self.update_target()

            # save total reward from epsisode and print training progress
            self.reward_history.append(total_reward)
            print("episode {}: {} reward".format(episode, total_reward))
```

### Training an agent

```
env = gym.make('CartPole-v1')

seed = 2

env.seed(seed)
env.action_space.seed(seed)
np.random.seed(seed)
random.seed(seed)
tf.random.set_seed(seed)

dqn_agent = DQN(env, batch_size=32)
dqn_agent.train(num_episodes=35)
```

    episode 0: 12.0 reward
    episode 1: 14.0 reward
    episode 2: 19.0 reward
    episode 3: 19.0 reward
    episode 4: 10.0 reward
    episode 5: 10.0 reward
    episode 6: 21.0 reward
    episode 7: 16.0 reward
    episode 8: 30.0 reward
    episode 9: 11.0 reward
    episode 10: 11.0 reward
    episode 11: 50.0 reward
    episode 12: 26.0 reward
    episode 13: 23.0 reward
    episode 14: 40.0 reward
    episode 15: 113.0 reward
    episode 16: 24.0 reward
    episode 17: 43.0 reward
    episode 18: 38.0 reward
    episode 19: 8.0 reward
    episode 20: 42.0 reward
    episode 21: 64.0 reward
    episode 22: 54.0 reward
    episode 23: 70.0 reward
    episode 24: 91.0 reward
    episode 25: 30.0 reward
    episode 26: 121.0 reward
    episode 27: 213.0 reward
    episode 28: 500.0 reward
    episode 29: 500.0 reward
    episode 30: 364.0 reward
    episode 31: 401.0 reward
    episode 32: 342.0 reward
    episode 33: 309.0 reward
    episode 34: 290.0 reward

```
# visualize total reward per episode across training
plt.figure(figsize=(7,4))
plt.plot(dqn_agent.reward_history, alpha=1, color='purple', label='raw')
plt.xlabel('episode #', fontsize=15)
plt.ylabel('total reward per episode', fontsize=15)
plt.title('DQN Agent')
plt.show()
```

### Evaluating the agent

```
evaluate_policy(dqn_agent, num_episodes=1)
```

    Exploit reward average: 227.0

```
create_learned_policy_video(dqn_agent, "video", num_episodes=1)
```

    xdpyinfo was not found, X start can not be checked! Please install xdpyinfo!
    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

Yay, it learned the task! This is also not quite optimal, but its only been 35 sessions! This is a huge change from our Q-learning agent, which took 5000 episode to reach less than the perfomance of our DQN agents. This tells us how useful it is to keep the continous-valued inputs and use a neural network to appromimate the Q-value function.

### What is the model learning?

```
n_obs = 40
obs_array = np.zeros((n_obs, env.observation_space.shape[0]))
obs_array[:,0] = 0
obs_array[:,1] = 0
obs_array[:,2] = 0
obs_array[:,3] = np.linspace(-5, 5, n_obs)
# run model
q_values = dqn_agent.model.predict(obs_array)
# visualize results
plt.figure(figsize=(8,5))
plt.plot(obs_array[:,3], q_values[:,0], color='purple', label='push cart left', linewidth=2)
plt.plot(obs_array[:,3], q_values[:,1], color='green', label='push cart right', linewidth=2)
plt.vlines(0, q_values.min(), q_values.max(), linestyle='--', color='dimgray')
plt.xlabel('Pole Velocity', fontsize=15)
plt.ylabel('Q value', fontsize=15)
plt.title('Cart Position=0, Cart Velocity=0, Pole Angle=0', fontsize=15)
plt.legend(fontsize=13)
plt.show()
```

We can see here that our agent learned the switch between pushing the car right and left right at 0. It was able to learn this behavior from few examples because even if it didn't encounter every state along this continuum, it can generalize from nearby experiences. Pretty cool!

## Summary

To summarize, in this notebook we implemented the Q-Learning and DQN algorithms and trained an agent on each. Moving from Q-learning to DQNs gave us similar performance (slightly better for the DQN) after 5000 episodes for Q-learning and only 35 episodes for DQNs. This is a pretty amazing improvement, and gives us a hands-on feel for how much you gain by using neural networks for function approximation.

We focused on DQNs here, but Policy Gradient algorithms are a very effective class of algorithms for RL tasks. Our "What is the model learning?" plot may give us a hint for why this is. DQNs need to approximate the Q-value for each action, and must get the highest Q-value right in order for the policy to be correct. If the Q-values are very simialar this might by tricky and lead to noise in selecting the best action. Perhaps it is more simple to directly learn the action space, as Policy Gradients do.

There is a lot to learn in DeepRL, and this at least gives us a start!

## Citations

A lot of the code in the Colab notebook was inpired by some really helpful posts from other people:

- Lillian Weng's [deep-reinforcement-learning-gym](https://github.com/lilianweng/deep-reinforcement-learning-gym) repo (adapted her Q-learning implementation)
- Anita Hu's [TF2-RL](https://github.com/anita-hu/TF2-RL) repo (adapted her DQN implementation)
- TF [tutorial](https://www.tensorflow.org/agents/tutorials/1_dqn_tutorial) for training DQN with TF-Agents (adapted TF-Agents visualization functions for Gym)


# Tutorial 3: Learning to Act: Q-Learning
**Week 3, Day 4: Reinforcement Learning**

**By Neuromatch Academy**

__Content creators:__ Marcelo Mattar and Eric DeWitt with help from Byron Galbraith

__Content reviewers:__ Ella Batty, Matt Krause and Michael Waskom

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---

# Tutorial Objectives

*Estimated timing of tutorial: 40 min*

In this tutorial you will learn how to act in the more realistic setting of sequential decisions, formalized by Markov Decision Processes (MDPs). In a sequential decision problem, the actions executed in one state not only may lead to immediate rewards (as in a bandit problem), but may also affect the states experienced next (unlike a bandit problem). Each individual action may therefore affect affect all future rewards. Thus, making decisions in this setting requires considering each action in terms of their expected **cumulative** future reward.

We will consider here the example of spatial navigation, where actions (movements) in one state (location) affect the states experienced next, and an agent might need to execute a whole sequence of actions before a reward is obtained.

By the end of this tutorial, you will learn
* what grid worlds are and how they help in evaluating simple reinforcement learning agents
* the basics of the Q-learning algorithm for estimating action values
* how the concept of exploration and exploitation, reviewed in the bandit case, also applies to the sequential decision setting

```python
# @title Tutorial slides

# @markdown These are the slides for all videos in this tutorial.
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/2jzdu/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
# Imports

import numpy as np
import matplotlib.pyplot as plt
from scipy.signal import convolve as conv
```

```python
#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Plotting Functions

def plot_state_action_values(env, value, ax=None):
  """
  Generate plot showing value of each action at each state.
  """
  if ax is None:
    fig, ax = plt.subplots()

  for a in range(env.n_actions):
    ax.plot(range(env.n_states), value[:, a], marker='o', linestyle='--')
  ax.set(xlabel='States', ylabel='Values')
  ax.legend(['R','U','L','D'], loc='lower right')

def plot_quiver_max_action(env, value, ax=None):
  """
  Generate plot showing action of maximum value or maximum probability at
    each state (not for n-armed bandit or cheese_world).
  """
  if ax is None:
    fig, ax = plt.subplots()

  X = np.tile(np.arange(env.dim_x), [env.dim_y,1]) + 0.5
  Y = np.tile(np.arange(env.dim_y)[::-1][:,np.newaxis], [1,env.dim_x]) + 0.5
  which_max = np.reshape(value.argmax(axis=1), (env.dim_y,env.dim_x))
  which_max = which_max[::-1,:]
  U = np.zeros(X.shape)
  V = np.zeros(X.shape)
  U[which_max == 0] = 1
  V[which_max == 1] = 1
  U[which_max == 2] = -1
  V[which_max == 3] = -1

  ax.quiver(X, Y, U, V)
  ax.set(
      title='Maximum value/probability actions',
      xlim=[-0.5, env.dim_x+0.5],
      ylim=[-0.5, env.dim_y+0.5],
  )
  ax.set_xticks(np.linspace(0.5, env.dim_x-0.5, num=env.dim_x))
  ax.set_xticklabels(["%d" % x for x in np.arange(env.dim_x)])
  ax.set_xticks(np.arange(env.dim_x+1), minor=True)
  ax.set_yticks(np.linspace(0.5, env.dim_y-0.5, num=env.dim_y))
  ax.set_yticklabels(["%d" % y for y in np.arange(0, env.dim_y*env.dim_x,
                                                  env.dim_x)])
  ax.set_yticks(np.arange(env.dim_y+1), minor=True)
  ax.grid(which='minor',linestyle='-')

def plot_heatmap_max_val(env, value, ax=None):
  """
  Generate heatmap showing maximum value at each state
  """
  if ax is None:
    fig, ax = plt.subplots()

  if value.ndim == 1:
      value_max = np.reshape(value, (env.dim_y,env.dim_x))
  else:
      value_max = np.reshape(value.max(axis=1), (env.dim_y,env.dim_x))
  value_max = value_max[::-1,:]

  im = ax.imshow(value_max, aspect='auto', interpolation='none', cmap='afmhot')
  ax.set(title='Maximum value per state')
  ax.set_xticks(np.linspace(0, env.dim_x-1, num=env.dim_x))
  ax.set_xticklabels(["%d" % x for x in np.arange(env.dim_x)])
  ax.set_yticks(np.linspace(0, env.dim_y-1, num=env.dim_y))
  if env.name != 'windy_cliff_grid':
      ax.set_yticklabels(
          ["%d" % y for y in np.arange(
              0, env.dim_y*env.dim_x, env.dim_x)][::-1])
  return im

def plot_rewards(n_episodes, rewards, average_range=10, ax=None):
  """
  Generate plot showing total reward accumulated in each episode.
  """
  if ax is None:
    fig, ax = plt.subplots()

  smoothed_rewards = (conv(rewards, np.ones(average_range), mode='same')
                      / average_range)

  ax.plot(range(0, n_episodes, average_range),
          smoothed_rewards[0:n_episodes:average_range],
          marker='o', linestyle='--')
  ax.set(xlabel='Episodes', ylabel='Total reward')

def plot_performance(env, value, reward_sums):
  fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(16, 12))
  plot_state_action_values(env, value, ax=axes[0,0])
  plot_quiver_max_action(env, value, ax=axes[0,1])
  plot_rewards(n_episodes, reward_sums, ax=axes[1,0])
  im = plot_heatmap_max_val(env, value, ax=axes[1,1])
  fig.colorbar(im)
```

---
# Section 1: Markov Decision Processes

```python
# @title Video 1: MDPs and Q-learning
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1ft4y1Q7bX", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="8yvwMrUQJOU", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

**Grid Worlds**

As pointed out, bandits only have a single state and immediate rewards for our actions. Many problems we are interested in have multiple states and delayed rewards, i.e. we won't know if the choices we made will pay off over time, or which actions we took contributed to the outcomes we observed.

In order to explore these ideas, we turn the a common problem setting: the grid world. Grid worlds are simple environments where each state corresponds to a tile on a 2D grid, and the only actions the agent can take are to move up, down, left, or right across the grid tiles. The agent's job is almost always to find a way to a goal tile in the most direct way possible while overcoming some maze or other obstacles, either static or dynamic.

For our discussion we will be looking at the classic Cliff World, or Cliff Walker, environment. This is a 4x10 grid with a starting position in the lower-left and the goal position in the lower-right. Every tile between these two is the "cliff", and should the agent enter the cliff, they will receive a -100 reward and be sent back to the starting position. Every tile other than the cliff produces a -1 reward when entered. The goal tile ends the episode after taking any action from it.

Given these conditions, the maximum achievable reward is -11 (1 up, 9 right, 1 down). Using negative rewards is a common technique to encourage the agent to move and seek out the goal state as fast as possible.

---
# Section 2: Q-Learning

*Estimated timing to here from start of tutorial: 20 min*

Now that we have our environment, how can we solve it?

One of the most famous algorithms for estimating action values (aka Q-values) is the Temporal Differences (TD) **control** algorithm known as *Q-learning* (Watkins, 1989).

\begin{align}
Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \big(r_t + \gamma\max_\limits{a} Q(s_{t+1},a_{t+1}) - Q(s_t,a_t)\big)
\end{align}

where $Q(s,a)$ is the value function for action $a$ at state $s$, $\alpha$ is the learning rate, $r$ is the reward, and $\gamma$ is the temporal discount rate.

The expression $r_t + \gamma\max_\limits{a} Q(s_{t+1},a_{t+1})$ is referred to as the TD target while the full expression
\begin{align}
r_t + \gamma\max_\limits{a} Q(s_{t+1},a_{t+1}) - Q(s_t,a_t),
\end{align}
i.e. the difference between the TD target and the current Q-value, is referred to as the TD error, or reward prediction error.

Because of the max operator used to select the optimal Q-value in the TD target, Q-learning directly estimates the optimal action value, i.e. the cumulative future reward that would be obtained if the agent behaved optimally, regardless of the policy currently followed by the agent. For this reason, Q-learning is referred to as an **off-policy** method.

## Coding Exercise 2: Implement the Q-learning algorithm

In this exercise you will implement the Q-learning update rule described above. It takes in as arguments the previous state $s_t$, the action $a_t$ taken, the reward received $r_t$, the current state $s_{t+1}$, the Q-value table, and a dictionary of parameters that contain the learning rate $\alpha$ and discount factor $\gamma$. The method returns the updated Q-value table. For the parameter dictionary, $\alpha$: `params['alpha']` and $\gamma$: `params['gamma']`.

Once we have our Q-learning algorithm, we will see how it handles learning to solve the Cliff World environment.

You will recall from the previous tutorial that a major part of reinforcement learning algorithms are their ability to balance exploitation and exploration. For our Q-learning agent, we again turn to the epsilon-greedy strategy. At each step, the agent will decide with probability $1 - \epsilon$ to use the best action for the state it is currently in by looking at the value function, otherwise just make a random choice.

The process by which our the agent will interact with and learn about the environment is handled for you in the helper function `learn_environment`. This implements the entire learning episode lifecycle of stepping through the state observation, action selection (epsilon-greedy) and execution, reward, and state transition. Feel free to review that code later to see how it all fits together, but for now let's test out our agent.

```python
# @markdown Execute to get helper functions `epsilon_greedy`, `CliffWorld`, and `learn_environment`

def epsilon_greedy(q, epsilon):
  """Epsilon-greedy policy: selects the maximum value action with probabilty
  (1-epsilon) and selects randomly with epsilon probability.

  Args:
    q (ndarray): an array of action values
    epsilon (float): probability of selecting an action randomly

  Returns:
    int: the chosen action
  """
  if np.random.random() > epsilon:
    action = np.argmax(q)
  else:
    action = np.random.choice(len(q))

  return action

class CliffWorld:
  """
  World: Cliff world.
  40 states (4-by-10 grid world).
  The mapping from state to the grids are as follows:
  30 31 32 ... 39
  20 21 22 ... 29
  10 11 12 ... 19
  0  1  2  ...  9
  0 is the starting state (S) and 9 is the goal state (G).
  Actions 0, 1, 2, 3 correspond to right, up, left, down.
  Moving anywhere from state 9 (goal state) will end the session.
  Taking action down at state 11-18 will go back to state 0 and incur a
      reward of -100.
  Landing in any states other than the goal state will incur a reward of -1.
  Going towards the border when already at the border will stay in the same
      place.
  """
  def __init__(self):
    self.name = "cliff_world"
    self.n_states = 40
    self.n_actions = 4
    self.dim_x = 10
    self.dim_y = 4
    self.init_state = 0

  def get_outcome(self, state, action):
    if state == 9:  # goal state
      reward = 0
      next_state = None
      return next_state, reward
    reward = -1  # default reward value
    if action == 0:  # move right
      next_state = state + 1
      if state % 10 == 9:  # right border
        next_state = state
      elif state == 0:  # start state (next state is cliff)
        next_state = None
        reward = -100
    elif action == 1:  # move up
      next_state = state + 10
      if state >= 30:  # top border
        next_state = state
    elif action == 2:  # move left
      next_state = state - 1
      if state % 10 == 0:  # left border
        next_state = state
    elif action == 3:  # move down
      next_state = state - 10
      if state >= 11 and state <= 18:  # next is cliff
        next_state = None
        reward = -100
      elif state <= 9:  # bottom border
        next_state = state
    else:
      print("Action must be between 0 and 3.")
      next_state = None
      reward = None
    return int(next_state) if next_state is not None else None, reward

  def get_all_outcomes(self):
    outcomes = {}
    for state in range(self.n_states):
      for action in range(self.n_actions):
        next_state, reward = self.get_outcome(state, action)
        outcomes[state, action] = [(1, next_state, reward)]
    return outcomes

def learn_environment(env, learning_rule, params, max_steps, n_episodes):
  # Start with a uniform value function
  value = np.ones((env.n_states, env.n_actions))

  # Run learning
  reward_sums = np.zeros(n_episodes)

  # Loop over episodes
  for episode in range(n_episodes):
    state = env.init_state  # initialize state
    reward_sum = 0

    for t in range(max_steps):
      # choose next action
      action = epsilon_greedy(value[state], params['epsilon'])

      # observe outcome of action on environment
      next_state, reward = env.get_outcome(state, action)

      # update value function
      value = learning_rule(state, action, reward, next_state, value, params)

      # sum rewards obtained
      reward_sum += reward

      if next_state is None:
          break  # episode ends
      state = next_state

    reward_sums[episode] = reward_sum

  return value, reward_sums
```

```python
def q_learning(state, action, reward, next_state, value, params):
  """Q-learning: updates the value function and returns it.

  Args:
    state (int): the current state identifier
    action (int): the action taken
    reward (float): the reward received
    next_state (int): the transitioned to state identifier
    value (ndarray): current value function of shape (n_states, n_actions)
    params (dict): a dictionary containing the default parameters

  Returns:
    ndarray: the updated value function of shape (n_states, n_actions)
  """
  # Q-value of current state-action pair
  q = value[state, action]

  ##########################################################
  ## TODO for students: implement the Q-learning update rule
  # Fill out function and remove
  #raise NotImplementedError("Student excercise: implement the Q-learning update rule")
  ##########################################################

  # write an expression for finding the maximum Q-value at the current state
  if next_state is None:
    max_next_q = 0
  else:
    max_next_q = np.max(value[next_state])

  # write the expression to compute the TD error
  td_error = reward + params['gamma'] * max_next_q - q
  # write the expression that updates the Q-value for the state-action pair
  value[state, action] = q + params['alpha'] *td_error

  return value

# set for reproducibility, comment out / change seed value for different results
np.random.seed(1)

# parameters needed by our policy and learning rule
params = {
  'epsilon': 0.1,  # epsilon-greedy policy
  'alpha': 0.1,  # learning rate
  'gamma': 1.0,  # discount factor
}

# episodes/trials
n_episodes = 500
max_steps = 1000

# environment initialization
env = CliffWorld()

# solve Cliff World using Q-learning
results = learn_environment(env, q_learning, params, max_steps, n_episodes)
value_qlearning, reward_sums_qlearning = results

# Plot results
plot_performance(env, value_qlearning, reward_sums_qlearning)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W3D4_ReinforcementLearning/solutions/W3D4_Tutorial3_Solution_7a5ca920.py)

*Example output:*

If all went well, we should see four plots that show different aspects on our agent's learning and progress.

* The top left is a representation of the Q-table itself, showing the values for different actions in different states. Notably, going right from the starting state or down when above the cliff is clearly very bad.
* The top right figure shows the greedy policy based on the Q-table, i.e. what action would the agent take if it only took its best guess in that state.
* The bottom right is the same as the top, only instead of showing the action, it's showing a representation of the maximum Q-value at a particular state.
* The bottom left is the actual proof of learning, as we see the total reward steadily increasing after each episode until asymptoting at the maximum possible reward of -11.

Feel free to try changing the parameters or random seed and see how the agent's behavior changes.

---
# Summary

*Estimated timing of tutorial: 40 min*

In this tutorial you implemented a reinforcement learning agent based on Q-learning to solve the Cliff World environment. Q-learning combined the epsilon-greedy approach to exploration-expoitation with a table-based value function to learn the expected future rewards for each state.

---
# Bonus

---
## Bonus Section 1: SARSA

An alternative to Q-learning, the SARSA algorithm also estimates action values. However, rather than estimating the optimal (off-policy) values, SARSA estimates the **on-policy** action value, i.e. the cumulative future reward that would be obtained if the agent behaved according to its current beliefs.

\begin{align}
Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \big(r_t + \gamma Q(s_{t+1},a_{t+1}) - Q(s_t,a_t)\big)
\end{align}

where, once again, $Q(s,a)$ is the value function for action $a$ at state $s$, $\alpha$ is the learning rate, $r$ is the reward, and $\gamma$ is the temporal discount rate.

In fact, you will notices that the *only* difference between Q-learning and SARSA is the TD target calculation uses the policy to select the next action (in our case epsilon-greedy) rather than using the action that maximizes the Q-value.

### Bonus Coding Exercise 1: Implement the SARSA algorithm

In this exercise you will implement the SARSA update rule described above. Just like Q-learning, it takes in as arguments the previous state $s_t$, the action $a_t$ taken, the reward received $r_t$, the current state $s_{t+1}$, the Q-value table, and a dictionary of parameters that contain the learning rate $\alpha$ and discount factor $\gamma$. The method returns the updated Q-value table. You may use the `epsilon_greedy` function to acquire the next action. For the parameter dictionary, $\alpha$: `params['alpha']`, $\gamma$: `params['gamma']`, and $\epsilon$: `params['epsilon']`.

Once we have an implementation for SARSA, we will see how it tackles Cliff World. We will again use the same setup we tried with Q-learning.

```python
def sarsa(state, action, reward, next_state, value, params):
  """SARSA: updates the value function and returns it.

  Args:
    state (int): the current state identifier
    action (int): the action taken
    reward (float): the reward received
    next_state (int): the transitioned to state identifier
    value (ndarray): current value function of shape (n_states, n_actions)
    params (dict): a dictionary containing the default parameters

  Returns:
    ndarray: the updated value function of shape (n_states, n_actions)
  """
  # value of previous state-action pair
  q = value[state, action]

  ##########################################################
  ## TODO for students: implement the SARSA update rule
  # Fill out function and remove
  raise NotImplementedError("Student excercise: implement the SARSA update rule")
  ##########################################################

  # select the expected value at current state based on our policy by sampling
  # from it
  if next_state is None:
    policy_next_q = 0
  else:
    # write an expression for selecting an action using epsilon-greedy
    policy_action = ...
    # write an expression for obtaining the value of the policy action at the
    # current state
    policy_next_q = ...

  # write the expression to compute the TD error
  td_error = ...
  # write the expression that updates the Q-value for the state-action pair
  value[state, action] = ...

  return value

# set for reproducibility, comment out / change seed value for different results
np.random.seed(1)

# parameters needed by our policy and learning rule
params = {
  'epsilon': 0.1,  # epsilon-greedy policy
  'alpha': 0.1,  # learning rate
  'gamma': 1.0,  # discount factor
}

# episodes/trials
n_episodes = 500
max_steps = 1000

# environment initialization
env = CliffWorld()

# learn Cliff World using Sarsa
results = learn_environment(env, sarsa, params, max_steps, n_episodes)
value_sarsa, reward_sums_sarsa = results

# Plot results
plot_performance(env, value_sarsa, reward_sums_sarsa)
```

```python
def sarsa(state, action, reward, next_state, value, params):
  """SARSA: updates the value function and returns it.

  Args:
    state (int): the current state identifier
    action (int): the action taken
    reward (float): the reward received
    next_state (int): the transitioned to state identifier
    value (ndarray): current value function of shape (n_states, n_actions)
    params (dict): a dictionary containing the default parameters

  Returns:
    ndarray: the updated value function of shape (n_states, n_actions)
  """
  # value of previous state-action pair
  q = value[state, action]

  # select the expected value at current state based on our policy by sampling
  # from it
  if next_state is None:
    policy_next_q = 0
  else:
    # write an expression for selecting an action using epsilon-greedy
    policy_action = epsilon_greedy(value[next_state], params['epsilon'])
    # write an expression for obtaining the value of the policy action at the
    # current state
    policy_next_q = value[next_state, policy_action]

  # write the expression to compute the TD error
  td_error = reward + params['gamma'] * policy_next_q - q
  # write the expression that updates the Q-value for the state-action pair
  value[state, action] = q + params['alpha'] * td_error

  return value

# set for reproducibility, comment out / change seed value for different results
np.random.seed(1)

# parameters needed by our policy and learning rule
params = {
  'epsilon': 0.1,  # epsilon-greedy policy
  'alpha': 0.1,  # learning rate
  'gamma': 1.0,  # discount factor
}

# episodes/trials
n_episodes = 500
max_steps = 1000

# environment initialization
env = CliffWorld()

# learn Cliff World using Sarsa
results = learn_environment(env, sarsa, params, max_steps, n_episodes)
value_sarsa, reward_sums_sarsa = results

# Plot results
with plt.xkcd():
  plot_performance(env, value_sarsa, reward_sums_sarsa)
```

We should see that SARSA also solves the task with similar looking outcomes to Q-learning. One notable difference is that SARSA seems to be skittsh around the cliff edge and often goes further away before coming back down to the goal.

Again, feel free to try changing the parameters or random seed and see how the agent's behavior changes.

---
## Bonus Section 2: On-Policy vs Off-Policy

We have now seen an example of both on- and off-policy learning algorithms. Let's compare both Q-learning and SARSA reward results again, side-by-side, to see how they stack up.

```python
# @markdown Execute to see visualization

# parameters needed by our policy and learning rule
params = {
  'epsilon': 0.1,  # epsilon-greedy policy
  'alpha': 0.1,  # learning rate
  'gamma': 1.0,  # discount factor
}

# episodes/trials
n_episodes = 500
max_steps = 1000

# environment initialization
env = CliffWorld()

# learn Cliff World using Sarsa
np.random.seed(1)
results = learn_environment(env, q_learning, params, max_steps, n_episodes)
value_qlearning, reward_sums_qlearning = results
np.random.seed(1)
results = learn_environment(env, sarsa, params, max_steps, n_episodes)
value_sarsa, reward_sums_sarsa = results

fig, ax = plt.subplots()
ax.plot(reward_sums_qlearning, label='Q-learning')
ax.plot(reward_sums_sarsa, label='SARSA')
ax.set(xlabel='Episodes', ylabel='Total reward')
plt.legend(loc='lower right');
```

On this simple Cliff World task, Q-learning and SARSA are almost indistinguisable from a performance standpoint, but we can see that Q-learning has a slight-edge within the 500 episode time horizon. Let's look at the illustrated "greedy policy" plots again.

```python
# @markdown Execute to see visualization

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(16, 6))
plot_quiver_max_action(env, value_qlearning, ax=ax1)
ax1.set(title='Q-learning maximum value/probability actions')
plot_quiver_max_action(env, value_sarsa, ax=ax2)
ax2.set(title='SARSA maximum value/probability actions');
```

What should immediately jump out is that Q-learning learned to go up, then immediately go to the right, skirting the cliff edge, until it hits the wall and goes down to the goal. The policy further away from the cliff is less certain.

SARSA, on the other hand, appears to avoid the cliff edge, going up one more tile before starting over to the goal side. This also clearly solves the challenge of getting to the goal, but does so at an additional -2 cost over the truly optimal route.

Why do you think these behaviors emerged the way they did?