# Ray RLlib - Introduction to Reinforcement Learning

© 2019-2022, Anyscale. All Rights Reserved

_Reinforcement Learning_ is the category of machine learning that focuses on training one or more _agents_ to achieve maximal _rewards_ while operating in an environment. This lesson discusses the core concepts of RL, while subsequent lessons explore RLlib in depth. We'll use two examples with exercises to give you a taste of RL. If you already understand RL concepts, you can either skim this lesson or skip to the [next lesson](02-Introduction-to-RLlib.ipynb).

## What Is Reinforcement Learning?

Let's explore the basic concepts of RL, specifically the _Markov Decision Process_ abstraction, and to show its use in Python.

Consider the following image:

In RL, one or more **agents** interact with an **environment** to maximize a **reward**. The agents make **observations** about the **state** of the environment and take **actions** that are believed will maximize the long-term reward. However, at any particular moment, the agents can only observe the immediate reward. So, the training process usually involves lots and lot of replay of the game, the robot simulator traversing a virtual space, etc., so the agents can learn from repeated trials what decisions/actions work best to maximize the long-term, cumulative reward.

The trial and error search and delayed reward are the distinguishing characterists of RL vs. other ML methods ([Sutton 2018](06-RL-References.ipynb#Books)).

The way to formalize trial and error is the **exploitation vs. exploration tradeoff**. When an agent finds what appears to be a "rewarding" sequence of actions, the agent may naturally want to continue to **exploit** these actions. However, even better actions may exist. An agent won't know whether alternatives are better or not unless some percentage of actions taken **explore** the alternatives. So, all RL algorithms include a strategy for exploitation and exploration.

## RL Applications

RL has many potential applications. RL became "famous" due to these successes, including achieving expert game play, training robots, autonomous vehicles, and other simulated agents:

The famous Alpha Go Game

Playing Atari with Deep Reinforcement Learning

Learning to stack legos using RL

The famous walking man scenario

The autonous vehicles increasingly becoming prevalent

The cartonish two-legged robot, Cassie

Credits and References:
* [AlphaGo](https://www.youtube.com/watch?v=l7ngy56GY6k)
* [Breakout](https://towardsdatascience.com/tutorial-double-deep-q-learning-with-dueling-network-architectures-4c1b3fb7f756) ([paper](https://arxiv.org/abs/1312.5602))
* [Stacking Legos with Sawyer](https://robohub.org/soft-actor-critic-deep-reinforcement-learning-with-real-world-robots/)
* [Walking Man](https://openai.com/blog/openai-baselines-ppo/)
* [Autonomous Vehicle](https://www.daimler.com/innovation/case/autonomous/intelligent-drive-2.html)
* ["Cassie": Two-legged Robot](https://mime.oregonstate.edu/research/drl/robots/cassie/) (Uses Ray!)

Recently, other industry applications have emerged, include the following:

* **Process optimization:** industrial processes (factories, pipelines) and other business processes, routing problems, cluster optimization.
* **Ad serving and recommendations:** Some of the traditional methods, including _collaborative filtering_, are hard to scale for very large data sets. RL systems are being developed to do an effective job more efficiently than traditional methods.
* **Finance:** Markets are time-oriented _environments_ where automated trading systems are the _agents_.

## Markov Decision Processes

At its core, Reinforcement learning builds on the concepts of [Markov Decision Process (MDP)](https://en.wikipedia.org/wiki/Markov_decision_process), where the current state, the possible actions that can be taken, and overall goal are the building blocks.

An MDP model's sequential interactions with an external environment consists of the following:

- a **state space** where the current state of the system is sometimes called the **context**.
- a set of **actions** that can be taken at a particular state $s$ (or sometimes the same set for all states).
- a **transition function** that describes the probability of being in a state $s'$ at time $t+1$ given that the MDP was in state $s$ at time $t$ and an action $a$ was taken. The next state is selected stochastically based on these probabilities.
- a **reward function**, which determines the reward received at time $t$ following action $a$, based on the decision of **policy** $\pi$.

The goal of MDP is to develop a **policy** $\pi$ that specifies what action $a$ should be chosen for a given state $s$ so that the cumulative reward is maximized. When it is possible for the policy "trainer" to fully observe all the possible states, actions, and rewards, it can define a deterministic policy, fixing a single action choice for each state. In this scenario, the transition probabilities reduce to the probability of transitioning to state $s'$ given the current state is $s$, independent of actions, because the state now leads to a deterministic action choice. Various algorithms can be used to compute this policy.

Put another way, if the policy isn't deterministic, then the transition probability to state $s'$ at a time $t+1$ when action $a$ is taken for state $s$ at time $t$, is given by:

\begin{equation}
P_a(s',s) = P(s_{t+1} = s'|s_t=s,a)
\end{equation}

When the policy is deterministic, this transition probability reduces to the following, independent of $a$:

\begin{equation}
P(s',s) = P(s_{t+1} = s'|s_t=s)
\end{equation}

To be clear, a deterministic policy means that one and only one action will always be selected for a given state $s$, but the next state $s'$ will still be selected stochastically.

In the general case of RL, it isn't possible to fully know all this information, some of which might be hidden and evolving, so it isn't possible to specify a fully-deterministic policy.

Often this cumulative reward is computed using the **discounted sum** over all rewards observed:

\begin{equation}
\arg\max_{\pi} \sum_{t=1}^T \gamma^t R_t(\pi),
\end{equation}

where $T$ is the number of steps taken in the MDP (this is a random variable and may depend on $\pi$), $R_t$ is the reward received at time $t$ (also a random variable which depends on $\pi$), and $\gamma$ is the **discount factor**. The value of $\gamma$ is between 0 and 1, meaning it has the effect of "discounting" earlier rewards vs. more recent rewards.

The [Wikipedia page on MDP](https://en.wikipedia.org/wiki/Markov_decision_process) provides more details. Note what we said in the third bullet, that the new state only depends on the previous state and the action taken. The assumption is that we can simplify our effort by ignoring all the previous states except the last one and still achieve good results. This is known as the [Markov property](https://en.wikipedia.org/wiki/Markov_property). This assumption often works well and it greatly reduces the resources required.

## The Elements of RL

Here are the elements of RL that expand on MDP concepts (see [Sutton 2018](https://mitpress.mit.edu/books/reinforcement-learning-second-edition) for more details):

#### Policies

Unlike MDP, the **transition function** probabilities are often not known in advance, but must be learned. Learning is done through repeated "play," where the agent interacts with the environment.

This makes the **policy** $\pi$ harder to determine. Because the fully state space usually can't be fully known, the choice of an action $a$ for given state $s$ almostly always remains a stochastic choice, never deterministic, unlike MDP.

#### Reward Signal

The idea of a **reward signal** encapsulates the desired goal for the system and provides feedback for updating the policy based on how well particular events or actions contribute rewards towards the goal.

#### Value Function

The **value function** encapsulates the maximum cumulative reward likely to be achieved starting from a given state for an **episode**. This is harder to determine than the simple reward returned after taking an action. In fact, much of the research in RL over the decades has focused on finding better and more efficient implementations of value functions. To illustrate the challenge, repeatedly taking one sequence of actions may yield low rewards for a while, but eventually provide large rewards. Conversely, always choosing a different sequence of actions may yield a good reward at each step, but be suboptimal for the cumulative reward.

#### Episode

A sequence of steps by the agent starting in an initial state. At each step, the agent observes the current state, chooses the next action, and receives the new reward. Episodes are used for both training policies and replaying with an existing policy (called _rollout_).

#### Model

An optional feature, some RL algorithms develop or use a **model** of the environment to anticipate the resulting states and rewards for future actions. Hence, they are useful for _planning_ scenarios. Methods for solving RL problems that use models are called _model-based methods_, while methods that learn by trial and error are called _model-free methods_.

## Reinforcement Learning Example

Let's finish this introduction and let's learn about the popular "hello world" [1] example environment for RL, balancing a pole vertically on a moving cart, called `CartPole`. Then we'll see how to use RLlib to train a policy using a popular RL algorithm, _Proximal Policy Optimization_, again using `CartPole`.

[1] In books and tutorials on programming languages, it is a tradition that the very first program shown prints the message "Hello World!"

### CartPole and OpenAI

The popular [OpenAI "gym" environment](https://gym.openai.com/) provides MDP interfaces to a variety of simulated environments. Perhaps the most popular for learning RL is `CartPole`, a simple environment that simulates the physics of balancing a pole on a moving cart. The `CartPole` problem is described at https://gym.openai.com/envs/CartPole-v1. Here is an image from that website, where the pole is currently falling to the right, which means the cart will need to move to the right to restore balance:

This example fits into the MDP framework as follows:
- The **state** consists of the position and velocity of the cart (moving in one dimension from left to right) as well as the angle and angular velocity of the pole that is balancing on the cart.
- The **actions** are to decrease or increase the cart's velocity by one unit. A negative velocity means it is moving to the left.
- The **transition function** is deterministic and is determined by simulating physical laws. Specifically, for a given **state**, what should we choose as the next velocity value? In the RL context, the correct velocity value to choose has to be learned. Hence, we learn a _policy_ that approximates the optimal transition function that could be calculated from the laws of physics.
- The **reward function** is a constant 1 as long as the pole is upright, and 0 once the pole has fallen over. Therefore, maximizing the reward means balancing the pole for as long as possible.
- The **discount factor** in this case can be taken to be 1, meaning we treat the rewards at all time steps equally and don't discount any of them.

More information about the `gym` Python module is available at https://gym.openai.com/. The list of all the available Gym environments is in [this wiki page](https://github.com/openai/gym/wiki/Table-of-environments). We'll use a few more of them and even create our own in subsequent lessons.

```python
import gym
import numpy as np
import pandas as pd
import json
```

The code below illustrates how to create and manipulate MDPs in Python. An MDP can be created by calling `gym.make`. Gym environments are identified by names like `CartPole-v1`. A **catalog of built-in environments** can be found at https://gym.openai.com/envs.

```python
env = gym.make("CartPole-v1")
print("Created env:", env)
```

    Created env: <TimeLimit<CartPoleEnv<CartPole-v1>>>

Reset the state of the MDP by calling `env.reset()`. This call returns the initial state of the MDP.

```python
state = env.reset()
print("The starting state is:", state)
```

    The starting state is: [ 0.04093885 -0.03678606  0.02658814  0.01931461]

Recall that the state is the position of the cart, its velocity, the angle of the pole, and the angular velocity of the pole.

The `env.step` method takes an action. In the case of the `CartPole` environment, the appropriate actions are 0 or 1, for pushing the cart to the left or right, respectively. `env.step()` returns a tuple of four things:
1. the new state of the environment
2. a reward
3. a boolean indicating whether the simulation has finished
4. a dictionary of miscellaneous extra information

Let's show what happens if we take one step with an action of 0.

```python
action = 0
state, reward, done, info = env.step(action)
print(state, reward, done, info)
```

    [ 0.04020313 -0.23227903  0.02697443  0.3202664 ] 1.0 False {}

A **rollout** is a simulation of a policy in an environment. It is used both during training and when running simulations with a trained policy.

The code below performs a rollout in a given environment. It takes **random actions** until the simulation has finished and returns the cumulative reward.

```python
def random_rollout(env):
    state = env.reset()

    done = False
    cumulative_reward = 0

    # Keep looping as long as the simulation has not finished.
    while not done:
        # Choose a random action (either 0 or 1).
        action = np.random.choice([0, 1])

        # Take the action in the environment.
        state, reward, done, _ = env.step(action)

        # Update the cumulative reward.
        cumulative_reward += reward

    # Return the cumulative reward.
    return cumulative_reward
```

Try rerunning the following cell a few times. How much do the answers change? Note that the maximum possible reward for `CartPole-v1` is 500. You'll probably get numbers well under 500.

```python
reward = random_rollout(env)
print(reward)

reward = random_rollout(env)
print(reward)
```

    17.0
    48.0

### Exercise 1

Choosing actions at random in `random_rollout` is not a very effective policy, as the previous results showed. Finish implementing the `rollout_policy` function below, which takes an environment *and* a policy. Recall that the *policy* is a function that takes in a *state* and returns an *action*. The main difference is that instead of choosing a **random action**, like we just did (with poor results), the action should be chosen **with the policy** (as a function of the state).

> **Note:** Exercise solutions for this tutorial can be found [here](solutions/Ray-RLlib-Solutions.ipynb).

```python
def rollout_policy(env, policy):
    state = env.reset()

    done = False
    cumulative_reward = 0

    # EXERCISE: Fill out this function by copying the appropriate part of 'random_rollout'
    # and modifying it to choose the action using the policy.
    raise NotImplementedError

    # Return the cumulative reward.
    return cumulative_reward

def sample_policy1(state):
    return 0 if state[0] < 0 else 1

def sample_policy2(state):
    return 1 if state[0] < 0 else 0

reward1 = np.mean([rollout_policy(env, sample_policy1) for _ in range(100)])
reward2 = np.mean([rollout_policy(env, sample_policy2) for _ in range(100)])

print('The first sample policy got an average reward of {}.'.format(reward1))
print('The second sample policy got an average reward of {}.'.format(reward2))

assert 5 < reward1 < 15, ('Make sure that rollout_policy computes the action '
                          'by applying the policy to the state.')
assert 25 < reward2 < 35, ('Make sure that rollout_policy computes the action '
                           'by applying the policy to the state.')
```

We'll return to `CartPole` in lesson [01: Application Cart Pole](explore-rllib/01-Application-Cart-Pole.ipynb) in the `explore-rllib` section.

### RLlib Reinforcement Learning Example: Cart Pole with Proximal Policy Optimization

This section demonstrates how to use the _proximal policy optimization_ (PPO) algorithm implemented by [RLlib](http://rllib.io). PPO is a popular way to develop a policy. RLlib also uses [Ray Tune](http://tune.io), the Ray Hyperparameter Tuning framework, which is covered in the [Ray Tune Tutorial](../ray-tune/00-Ray-Tune-Overview.ipynb).

We'll provide relatively little explanation of **RLlib** concepts for now, but explore them in greater depth in subsequent lessons. For more on RLlib, see the documentation at http://rllib.io.

PPO is described in detail in [this paper](https://arxiv.org/abs/1707.06347). It is a variant of _Trust Region Policy Optimization_ (TRPO) described in [this earlier paper](https://arxiv.org/abs/1502.05477). [This OpenAI post](https://openai.com/blog/openai-baselines-ppo/) provides a more accessible introduction to PPO.

PPO works in two phases. In the first phase, a large number of rollouts are performed in parallel. The rollouts are then aggregated on the driver and a surrogate optimization objective is defined based on those rollouts. In the second phase, we use SGD (_stochastic gradient descent_) to find the policy that maximizes that objective with a penalty term for diverging too much from the current policy.

> **NOTE:** The SGD optimization step is best performed in a data-parallel manner over multiple GPUs. This is exposed through the `num_gpus` field of the `config` dictionary. Hence, for normal usage, one or more GPUs is recommended.

(The original version of this example can be found [here](https://raw.githubusercontent.com/ucbrise/risecamp/risecamp2018/ray/tutorial/rllib_exercises/)).

```python
import ray
from ray.rllib.agents.ppo import PPOTrainer, DEFAULT_CONFIG
from ray.tune.logger import pretty_print
```

Initialize Ray. If you are running these tutorials on your laptop, then a single-node Ray cluster will be started by the next cell. If you are running in the Anyscale platform, it will connect to the running Ray cluster.

```python
info = ray.init(ignore_reinit_error=True, log_to_driver=False)
print(info)
```

    2022-03-17 16:09:28,771	INFO services.py:1412 -- View the Ray dashboard at http://127.0.0.1:8265

    {'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-03-17_16-09-26_550801_99855/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-03-17_16-09-26_550801_99855/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-03-17_16-09-26_550801_99855', 'metrics_export_port': 62444, 'gcs_address': '127.0.0.1:53561', 'address': '127.0.0.1:53561', 'node_id': 'b948df77a0e95112f1432667f1fe3d29c78870984c4409b6b96232ab'}

> **Tip:** Having trouble starting Ray? See the [Troubleshooting](../reference/Troubleshooting-Tips-Tricks.ipynb) tips.

The next cell prints the URL for the Ray Dashboard. **This is only correct if you are running this tutorial on a laptop.** Click the link to open the dashboard.

If you are running on the Anyscale platform, use the URL provided by your instructor to open the Dashboard.

```python
print("Dashboard URL: http://{}".format(info["webui_url"]))
```

    Dashboard URL: http://127.0.0.1:8265

Instantiate a PPOTrainer object. We pass in a config object that specifies how the network and training procedure should be configured. Some of the parameters are the following.

- `num_workers` is the number of actors that the agent will create. This determines the degree of parallelism that will be used. In a cluster, these actors will be spread over the available nodes.
- `num_sgd_iter` is the number of epochs of SGD (stochastic gradient descent, i.e., passes through the data) that will be used to optimize the PPO surrogate objective at each iteration of PPO, for each _minibatch_ ("chunk") of training data. Using minibatches is more efficient than training with one record at a time.
- `sgd_minibatch_size` is the SGD minibatch size (batches of data) that will be used to optimize the PPO surrogate objective.
- `model` contains a dictionary of parameters describing the neural net used to parameterize the policy. The `fcnet_hiddens` parameter is a list of the sizes of the hidden layers. Here, we have two hidden layers of size 100, each.
- `num_cpus_per_worker` when set to 0 prevents Ray from pinning a CPU core to each worker, which means we could run out of workers in a constrained environment like a laptop or a cloud VM.

```python
config = DEFAULT_CONFIG.copy()
config['num_workers'] = 1
config['num_sgd_iter'] = 30
config['sgd_minibatch_size'] = 128
config['model']['fcnet_hiddens'] = [100, 100]
config['num_cpus_per_worker'] = 0
```

```python
agent = PPOTrainer(config, 'CartPole-v1')
```

    2022-03-17 16:09:49,745	WARNING util.py:55 -- Install gputil for GPU system monitoring.

Now let's train the policy on the `CartPole-v1` environment for `N` steps. The JSON object returned by each call to `agent.train()` contains a lot of information we'll inspect below. For now, we'll extract information we'll graph, such as `episode_reward_mean`. The _mean_ values are more useful for determining successful training.

```python
N = 10
results = []
episode_data = []
episode_json = []

for n in range(N):
    result = agent.train()
    results.append(result)

    episode = {'n': n,
               'episode_reward_min':  result['episode_reward_min'],
               'episode_reward_mean': result['episode_reward_mean'],
               'episode_reward_max':  result['episode_reward_max'],
               'episode_len_mean':    result['episode_len_mean']}

    episode_data.append(episode)
    episode_json.append(json.dumps(episode))

    print(f'{n:3d}: Min/Mean/Max reward: {result["episode_reward_min"]:8.4f}/{result["episode_reward_mean"]:8.4f}/{result["episode_reward_max"]:8.4f}')
```

      0: Min/Mean/Max reward:   8.0000/ 22.4157/ 92.0000
      1: Min/Mean/Max reward:   9.0000/ 37.3868/123.0000
      2: Min/Mean/Max reward:   9.0000/ 54.8000/138.0000
      3: Min/Mean/Max reward:  10.0000/ 84.6300/379.0000
      4: Min/Mean/Max reward:  10.0000/117.7400/397.0000
      5: Min/Mean/Max reward:  10.0000/149.7000/500.0000
      6: Min/Mean/Max reward:  10.0000/178.4400/500.0000
      7: Min/Mean/Max reward:  12.0000/209.8400/500.0000
      8: Min/Mean/Max reward:  12.0000/244.6100/500.0000
      9: Min/Mean/Max reward:  12.0000/270.0500/500.0000

Now let's convert the episode data to a Pandas `DataFrame` for easy manipulation. The results indicate how much reward the policy is receiving (`episode_reward_*`) and how many time steps of the environment the policy ran (`episode_len_mean`). The maximum possible reward for this problem is `500`. The reward mean and trajectory length are very close because the agent receives a reward of one for every time step that it survives. However, this is specific to this environment and not true in general.

```python
df = pd.DataFrame(data=episode_data)
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>n</th>
      <th>episode_reward_min</th>
      <th>episode_reward_mean</th>
      <th>episode_reward_max</th>
      <th>episode_len_mean</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>8.0</td>
      <td>22.415730</td>
      <td>92.0</td>
      <td>22.415730</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>9.0</td>
      <td>37.386792</td>
      <td>123.0</td>
      <td>37.386792</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>9.0</td>
      <td>54.800000</td>
      <td>138.0</td>
      <td>54.800000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3</td>
      <td>10.0</td>
      <td>84.630000</td>
      <td>379.0</td>
      <td>84.630000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4</td>
      <td>10.0</td>
      <td>117.740000</td>
      <td>397.0</td>
      <td>117.740000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>5</td>
      <td>10.0</td>
      <td>149.700000</td>
      <td>500.0</td>
      <td>149.700000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>6</td>
      <td>10.0</td>
      <td>178.440000</td>
      <td>500.0</td>
      <td>178.440000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>7</td>
      <td>12.0</td>
      <td>209.840000</td>
      <td>500.0</td>
      <td>209.840000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>8</td>
      <td>12.0</td>
      <td>244.610000</td>
      <td>500.0</td>
      <td>244.610000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>9</td>
      <td>12.0</td>
      <td>270.050000</td>
      <td>500.0</td>
      <td>270.050000</td>
    </tr>
  </tbody>
</table>
</div>

```python
df.columns.tolist()
```

    ['n',
     'episode_reward_min',
     'episode_reward_mean',
     'episode_reward_max',
     'episode_len_mean']

Let's plot the data. Since the length and reward means are equal, we'll only plot one line:

```python
df.plot(x="n", y=["episode_reward_mean", "episode_reward_min", "episode_reward_max"], secondary_y=True)
```

The model is quickly able to hit the maximum value of 500, but the mean is what's most valuable. After 10 steps, we're more than half way there.

FYI, here are two views of the whole value for one result. First, a "pretty print" output.

> **Tip:** The output will be long. When this happens for a cell, right click and select _Enable scrolling for outputs_.

```python
print(pretty_print(results[-1]))
```

    agent_timesteps_total: 40000
    custom_metrics: {}
    date: 2022-03-17_16-11-03
    done: false
    episode_len_mean: 270.05
    episode_media: {}
    episode_reward_max: 500.0
    episode_reward_mean: 270.05
    episode_reward_min: 12.0
    episodes_this_iter: 12
    episodes_total: 441
    experiment_id: 73459c6b55f748bdae8d42fe303b6f9a
    hostname: Juless-MacBook-Pro-16-inch-2019
    info:
      learner:
        default_policy:
          custom_metrics: {}
          learner_stats:
            cur_kl_coeff: 0.03750000149011612
            cur_lr: 4.999999873689376e-05
            entropy: 0.5147084593772888
            entropy_coeff: 0.0
            kl: 0.004637665580958128
            model: {}
            policy_loss: -0.00826053787022829
            total_loss: 832.4736328125
            vf_explained_var: 0.16725607216358185
            vf_loss: 832.4817504882812
      num_agent_steps_sampled: 40000
      num_agent_steps_trained: 40000
      num_steps_sampled: 40000
      num_steps_trained: 40000
      num_steps_trained_this_iter: 4000
    iterations_since_restore: 10
    node_ip: 127.0.0.1
    num_healthy_workers: 1
    off_policy_estimator: {}
    perf:
      cpu_util_percent: 0.0
      ram_util_percent: 51.42
    pid: 99855
    policy_reward_max: {}
    policy_reward_mean: {}
    policy_reward_min: {}
    sampler_perf:
      mean_action_processing_ms: 0.04614653261891873
      mean_env_render_ms: 0.0
      mean_env_wait_ms: 0.04571587329741563
      mean_inference_ms: 0.42432151722661793
      mean_raw_obs_processing_ms: 0.06788292425999021
    time_since_restore: 36.21382212638855
    time_this_iter_s: 3.654078960418701
    time_total_s: 36.21382212638855
    timers:
      learn_throughput: 3203.836
      learn_time_ms: 1248.504
      load_throughput: 16727034.895
      load_time_ms: 0.239
      sample_throughput: 553.77
      sample_time_ms: 7223.22
      update_time_ms: 1.596
    timestamp: 1647558663
    timesteps_since_restore: 40000
    timesteps_this_iter: 4000
    timesteps_total: 40000
    training_iteration: 10
    trial_id: default

We'll learn about more of these values as continue the tutorial.

The whole, long JSON blob, which includes the historical stats about episode rewards and lengths:

```python
results[-1]
```

    {'episode_reward_max': 500.0,
     'episode_reward_min': 12.0,
     'episode_reward_mean': 270.05,
     'episode_len_mean': 270.05,
     'episode_media': {},
     'episodes_this_iter': 12,
     'policy_reward_min': {},
     'policy_reward_max': {},
     'policy_reward_mean': {},
     'custom_metrics': {},
     'hist_stats': {'episode_reward': [34.0,
       241.0,
       191.0,
       171.0,
       83.0,
       237.0,
       151.0,
       124.0,
       18.0,
       206.0,
       66.0,
       140.0,
       32.0,
       132.0,
       186.0,
       144.0,
       73.0,
       148.0,
       106.0,
       182.0,
       159.0,
       75.0,
       191.0,
       186.0,
       275.0,
       184.0,
       236.0,
       252.0,
       317.0,
       152.0,
       174.0,
       397.0,
       103.0,
       262.0,
       156.0,
       328.0,
       261.0,
       189.0,
       243.0,
       141.0,
       12.0,
       176.0,
       330.0,
       250.0,
       323.0,
       214.0,
       270.0,
       272.0,
       289.0,
       286.0,
       243.0,
       295.0,
       261.0,
       383.0,
       500.0,
       163.0,
       215.0,
       140.0,
       342.0,
       165.0,
       295.0,
       439.0,
       300.0,
       398.0,
       475.0,
       329.0,
       330.0,
       88.0,
       367.0,
       242.0,
       500.0,
       476.0,
       381.0,
       401.0,
       388.0,
       500.0,
       182.0,
       370.0,
       500.0,
       500.0,
       424.0,
       456.0,
       500.0,
       456.0,
       406.0,
       500.0,
       500.0,
       355.0,
       500.0,
       500.0,
       90.0,
       255.0,
       341.0,
       261.0,
       375.0,
       337.0,
       500.0,
       190.0,
       173.0,
       350.0],
      'episode_lengths': [34,
       241,
       191,
       171,
       83,
       237,
       151,
       124,
       18,
       206,
       66,
       140,
       32,
       132,
       186,
       144,
       73,
       148,
       106,
       182,
       159,
       75,
       191,
       186,
       275,
       184,
       236,
       252,
       317,
       152,
       174,
       397,
       103,
       262,
       156,
       328,
       261,
       189,
       243,
       141,
       12,
       176,
       330,
       250,
       323,
       214,
       270,
       272,
       289,
       286,
       243,
       295,
       261,
       383,
       500,
       163,
       215,
       140,
       342,
       165,
       295,
       439,
       300,
       398,
       475,
       329,
       330,
       88,
       367,
       242,
       500,
       476,
       381,
       401,
       388,
       500,
       182,
       370,
       500,
       500,
       424,
       456,
       500,
       456,
       406,
       500,
       500,
       355,
       500,
       500,
       90,
       255,
       341,
       261,
       375,
       337,
       500,
       190,
       173,
       350]},
     'sampler_perf': {'mean_raw_obs_processing_ms': 0.06788292425999021,
      'mean_inference_ms': 0.42432151722661793,
      'mean_action_processing_ms': 0.04614653261891873,
      'mean_env_wait_ms': 0.04571587329741563,
      'mean_env_render_ms': 0.0},
     'off_policy_estimator': {},
     'num_healthy_workers': 1,
     'timesteps_total': 40000,
     'timesteps_this_iter': 4000,
     'agent_timesteps_total': 40000,
     'timers': {'sample_time_ms': 7223.22,
      'sample_throughput': 553.77,
      'load_time_ms': 0.239,
      'load_throughput': 16727034.895,
      'learn_time_ms': 1248.504,
      'learn_throughput': 3203.836,
      'update_time_ms': 1.596},
     'info': {'learner': {'default_policy': {'learner_stats': {'cur_kl_coeff': 0.03750000149011612,
         'cur_lr': 4.999999873689376e-05,
         'total_loss': 832.47363,
         'policy_loss': -0.008260538,
         'vf_loss': 832.48175,
         'vf_explained_var': 0.16725607,
         'kl': 0.0046376656,
         'entropy': 0.51470846,
         'entropy_coeff': 0.0,
         'model': {}},
        'custom_metrics': {}}},
      'num_steps_sampled': 40000,
      'num_agent_steps_sampled': 40000,
      'num_steps_trained': 40000,
      'num_steps_trained_this_iter': 4000,
      'num_agent_steps_trained': 40000},
     'done': False,
     'episodes_total': 441,
     'training_iteration': 10,
     'trial_id': 'default',
     'experiment_id': '73459c6b55f748bdae8d42fe303b6f9a',
     'date': '2022-03-17_16-11-03',
     'timestamp': 1647558663,
     'time_this_iter_s': 3.654078960418701,
     'time_total_s': 36.21382212638855,
     'pid': 99855,
     'hostname': 'Juless-MacBook-Pro-16-inch-2019',
     'node_ip': '127.0.0.1',
     'config': {'num_workers': 1,
      'num_envs_per_worker': 1,
      'create_env_on_driver': False,
      'rollout_fragment_length': 200,
      'batch_mode': 'truncate_episodes',
      'gamma': 0.99,
      'lr': 5e-05,
      'train_batch_size': 4000,
      'model': {'_use_default_native_models': False,
       '_disable_preprocessor_api': False,
       '_disable_action_flattening': False,
       'fcnet_hiddens': [100, 100],
       'fcnet_activation': 'tanh',
       'conv_filters': None,
       'conv_activation': 'relu',
       'post_fcnet_hiddens': [],
       'post_fcnet_activation': 'relu',
       'free_log_std': False,
       'no_final_linear': False,
       'vf_share_layers': False,
       'use_lstm': False,
       'max_seq_len': 20,
       'lstm_cell_size': 256,
       'lstm_use_prev_action': False,
       'lstm_use_prev_reward': False,
       '_time_major': False,
       'use_attention': False,
       'attention_num_transformer_units': 1,
       'attention_dim': 64,
       'attention_num_heads': 1,
       'attention_head_dim': 32,
       'attention_memory_inference': 50,
       'attention_memory_training': 50,
       'attention_position_wise_mlp_dim': 32,
       'attention_init_gru_gate_bias': 2.0,
       'attention_use_n_prev_actions': 0,
       'attention_use_n_prev_rewards': 0,
       'framestack': True,
       'dim': 84,
       'grayscale': False,
       'zero_mean': True,
       'custom_model': None,
       'custom_model_config': {},
       'custom_action_dist': None,
       'custom_preprocessor': None,
       'lstm_use_prev_action_reward': -1},
      'optimizer': {},
      'horizon': None,
      'soft_horizon': False,
      'no_done_at_end': False,
      'env': 'CartPole-v1',
      'observation_space': None,
      'action_space': None,
      'env_config': {},
      'remote_worker_envs': False,
      'remote_env_batch_wait_ms': 0,
      'env_task_fn': None,
      'render_env': False,
      'record_env': False,
      'clip_rewards': None,
      'normalize_actions': True,
      'clip_actions': False,
      'preprocessor_pref': 'deepmind',
      'log_level': 'WARN',
      'callbacks': ray.rllib.agents.callbacks.DefaultCallbacks,
      'ignore_worker_failures': False,
      'log_sys_usage': True,
      'fake_sampler': False,
      'framework': 'tf',
      'eager_tracing': False,
      'eager_max_retraces': 20,
      'explore': True,
      'exploration_config': {'type': 'StochasticSampling'},
      'evaluation_interval': None,
      'evaluation_duration': 10,
      'evaluation_duration_unit': 'episodes',
      'evaluation_parallel_to_training': False,
      'in_evaluation': False,
      'evaluation_config': {'num_workers': 1,
       'num_envs_per_worker': 1,
       'create_env_on_driver': False,
       'rollout_fragment_length': 200,
       'batch_mode': 'truncate_episodes',
       'gamma': 0.99,
       'lr': 5e-05,
       'train_batch_size': 4000,
       'model': {'_use_default_native_models': False,
        '_disable_preprocessor_api': False,
        '_disable_action_flattening': False,
        'fcnet_hiddens': [100, 100],
        'fcnet_activation': 'tanh',
        'conv_filters': None,
        'conv_activation': 'relu',
        'post_fcnet_hiddens': [],
        'post_fcnet_activation': 'relu',
        'free_log_std': False,
        'no_final_linear': False,
        'vf_share_layers': False,
        'use_lstm': False,
        'max_seq_len': 20,
        'lstm_cell_size': 256,
        'lstm_use_prev_action': False,
        'lstm_use_prev_reward': False,
        '_time_major': False,
        'use_attention': False,
        'attention_num_transformer_units': 1,
        'attention_dim': 64,
        'attention_num_heads': 1,
        'attention_head_dim': 32,
        'attention_memory_inference': 50,
        'attention_memory_training': 50,
        'attention_position_wise_mlp_dim': 32,
        'attention_init_gru_gate_bias': 2.0,
        'attention_use_n_prev_actions': 0,
        'attention_use_n_prev_rewards': 0,
        'framestack': True,
        'dim': 84,
        'grayscale': False,
        'zero_mean': True,
        'custom_model': None,
        'custom_model_config': {},
        'custom_action_dist': None,
        'custom_preprocessor': None,
        'lstm_use_prev_action_reward': -1},
       'optimizer': {},
       'horizon': None,
       'soft_horizon': False,
       'no_done_at_end': False,
       'env': 'CartPole-v1',
       'observation_space': None,
       'action_space': None,
       'env_config': {},
       'remote_worker_envs': False,
       'remote_env_batch_wait_ms': 0,
       'env_task_fn': None,
       'render_env': False,
       'record_env': False,
       'clip_rewards': None,
       'normalize_actions': True,
       'clip_actions': False,
       'preprocessor_pref': 'deepmind',
       'log_level': 'WARN',
       'callbacks': ray.rllib.agents.callbacks.DefaultCallbacks,
       'ignore_worker_failures': False,
       'log_sys_usage': True,
       'fake_sampler': False,
       'framework': 'tf',
       'eager_tracing': False,
       'eager_max_retraces': 20,
       'explore': True,
       'exploration_config': {'type': 'StochasticSampling'},
       'evaluation_interval': None,
       'evaluation_duration': 10,
       'evaluation_duration_unit': 'episodes',
       'evaluation_parallel_to_training': False,
       'in_evaluation': False,
       'evaluation_config': {},
       'evaluation_num_workers': 0,
       'custom_eval_function': None,
       'always_attach_evaluation_results': False,
       'sample_async': False,
       'sample_collector': ray.rllib.evaluation.collectors.simple_list_collector.SimpleListCollector,
       'observation_filter': 'NoFilter',
       'synchronize_filters': True,
       'tf_session_args': {'intra_op_parallelism_threads': 2,
        'inter_op_parallelism_threads': 2,
        'gpu_options': {'allow_growth': True},
        'log_device_placement': False,
        'device_count': {'CPU': 1},
        'allow_soft_placement': True},
       'local_tf_session_args': {'intra_op_parallelism_threads': 8,
        'inter_op_parallelism_threads': 8},
       'compress_observations': False,
       'metrics_episode_collection_timeout_s': 180,
       'metrics_num_episodes_for_smoothing': 100,
       'min_time_s_per_reporting': None,
       'min_train_timesteps_per_reporting': None,
       'min_sample_timesteps_per_reporting': 0,
       'seed': None,
       'extra_python_environs_for_driver': {},
       'extra_python_environs_for_worker': {},
       'num_gpus': 0,
       '_fake_gpus': False,
       'num_cpus_per_worker': 0,
       'num_gpus_per_worker': 0,
       'custom_resources_per_worker': {},
       'num_cpus_for_driver': 1,
       'placement_strategy': 'PACK',
       'input': 'sampler',
       'input_config': {},
       'actions_in_input_normalized': False,
       'input_evaluation': ['is', 'wis'],
       'postprocess_inputs': False,
       'shuffle_buffer_size': 0,
       'output': None,
       'output_compress_columns': ['obs', 'new_obs'],
       'output_max_file_size': 67108864,
       'multiagent': {'policies': {'default_policy': PolicySpec(policy_class=<class 'ray.rllib.policy.tf_policy_template.PPOTFPolicy'>, observation_space=Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32), action_space=Discrete(2), config={})},
        'policy_map_capacity': 100,
        'policy_map_cache': None,
        'policy_mapping_fn': None,
        'policies_to_train': None,
        'observation_fn': None,
        'replay_mode': 'independent',
        'count_steps_by': 'env_steps'},
       'logger_config': None,
       '_tf_policy_handles_more_than_one_loss': False,
       '_disable_preprocessor_api': False,
       '_disable_action_flattening': False,
       '_disable_execution_plan_api': False,
       'simple_optimizer': False,
       'monitor': -1,
       'evaluation_num_episodes': -1,
       'metrics_smoothing_episodes': -1,
       'timesteps_per_iteration': 0,
       'min_iter_time_s': -1,
       'collect_metrics_timeout': -1,
       'use_critic': True,
       'use_gae': True,
       'lambda': 1.0,
       'kl_coeff': 0.2,
       'sgd_minibatch_size': 128,
       'shuffle_sequences': True,
       'num_sgd_iter': 30,
       'lr_schedule': None,
       'vf_loss_coeff': 1.0,
       'entropy_coeff': 0.0,
       'entropy_coeff_schedule': None,
       'clip_param': 0.3,
       'vf_clip_param': 10.0,
       'grad_clip': None,
       'kl_target': 0.01,
       'vf_share_layers': -1},
      'evaluation_num_workers': 0,
      'custom_eval_function': None,
      'always_attach_evaluation_results': False,
      'sample_async': False,
      'sample_collector': ray.rllib.evaluation.collectors.simple_list_collector.SimpleListCollector,
      'observation_filter': 'NoFilter',
      'synchronize_filters': True,
      'tf_session_args': {'intra_op_parallelism_threads': 2,
       'inter_op_parallelism_threads': 2,
       'gpu_options': {'allow_growth': True},
       'log_device_placement': False,
       'device_count': {'CPU': 1},
       'allow_soft_placement': True},
      'local_tf_session_args': {'intra_op_parallelism_threads': 8,
       'inter_op_parallelism_threads': 8},
      'compress_observations': False,
      'metrics_episode_collection_timeout_s': 180,
      'metrics_num_episodes_for_smoothing': 100,
      'min_time_s_per_reporting': None,
      'min_train_timesteps_per_reporting': None,
      'min_sample_timesteps_per_reporting': 0,
      'seed': None,
      'extra_python_environs_for_driver': {},
      'extra_python_environs_for_worker': {},
      'num_gpus': 0,
      '_fake_gpus': False,
      'num_cpus_per_worker': 0,
      'num_gpus_per_worker': 0,
      'custom_resources_per_worker': {},
      'num_cpus_for_driver': 1,
      'placement_strategy': 'PACK',
      'input': 'sampler',
      'input_config': {},
      'actions_in_input_normalized': False,
      'input_evaluation': ['is', 'wis'],
      'postprocess_inputs': False,
      'shuffle_buffer_size': 0,
      'output': None,
      'output_compress_columns': ['obs', 'new_obs'],
      'output_max_file_size': 67108864,
      'multiagent': {'policies': {'default_policy': PolicySpec(policy_class=<class 'ray.rllib.policy.tf_policy_template.PPOTFPolicy'>, observation_space=Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32), action_space=Discrete(2), config={})},
       'policy_map_capacity': 100,
       'policy_map_cache': None,
       'policy_mapping_fn': None,
       'policies_to_train': None,
       'observation_fn': None,
       'replay_mode': 'independent',
       'count_steps_by': 'env_steps'},
      'logger_config': None,
      '_tf_policy_handles_more_than_one_loss': False,
      '_disable_preprocessor_api': False,
      '_disable_action_flattening': False,
      '_disable_execution_plan_api': False,
      'simple_optimizer': False,
      'monitor': -1,
      'evaluation_num_episodes': -1,
      'metrics_smoothing_episodes': -1,
      'timesteps_per_iteration': 0,
      'min_iter_time_s': -1,
      'collect_metrics_timeout': -1,
      'use_critic': True,
      'use_gae': True,
      'lambda': 1.0,
      'kl_coeff': 0.2,
      'sgd_minibatch_size': 128,
      'shuffle_sequences': True,
      'num_sgd_iter': 30,
      'lr_schedule': None,
      'vf_loss_coeff': 1.0,
      'entropy_coeff': 0.0,
      'entropy_coeff_schedule': None,
      'clip_param': 0.3,
      'vf_clip_param': 10.0,
      'grad_clip': None,
      'kl_target': 0.01,
      'vf_share_layers': -1},
     'time_since_restore': 36.21382212638855,
     'timesteps_since_restore': 40000,
     'iterations_since_restore': 10,
     'perf': {'cpu_util_percent': 0.0, 'ram_util_percent': 51.42}}

Let's plot the `episode_reward` values:

```python
episode_rewards = results[-1]['hist_stats']['episode_reward']
df_episode_rewards = pd.DataFrame(data={'episode':range(len(episode_rewards)), 'reward':episode_rewards})

df_episode_rewards.plot(x="episode", y="reward")
```

For a well-trained model, most runs do very well while occasional runs do poorly. Try plotting other results episodes by changing the array index in `results[-1]` to another number between `0` and `9`. (The length of `results` is `10`.)

### Exercise 2

The current network and training configuration are too large and heavy-duty for a simple problem like `CartPole`. Modify the configuration to use a smaller network (the `config['model']['fcnet_hiddens']` setting) and to speed up the optimization of the surrogate objective. (Fewer SGD iterations and a larger batch size should help.)

```python
# Make edits here:
config = DEFAULT_CONFIG.copy()
config['num_workers'] = 3
config['num_sgd_iter'] = 30
config['sgd_minibatch_size'] = 128
config['model']['fcnet_hiddens'] = [100, 100]
config['num_cpus_per_worker'] = 0

agent = PPOTrainer(config, 'CartPole-v1')
```

    2022-03-17 16:12:27,542	WARNING ppo.py:223 -- `train_batch_size` (4000) cannot be achieved with your other settings (num_workers=3 num_envs_per_worker=1 rollout_fragment_length=200)! Auto-adjusting `rollout_fragment_length` to 1333.

Train the agent and try to get a reward of 500. If it's training too slowly you may need to modify the config above to use fewer hidden units, a larger `sgd_minibatch_size`, a smaller `num_sgd_iter`, or a larger `num_workers`.

This should take around `N` = 20 or 30 training iterations.

```python
N = 5
results = []
episode_data = []
episode_json = []

for n in range(N):
    result = agent.train()
    results.append(result)

    episode = {'n': n,
               'episode_reward_mean': result['episode_reward_mean'],
               'episode_reward_max': result['episode_reward_max'],
               'episode_len_mean': result['episode_len_mean']}

    episode_data.append(episode)
    episode_json.append(json.dumps(episode))

    print(f'Max reward: {episode["episode_reward_max"]}')
```

    Max reward: 86.0
    Max reward: 114.0
    Max reward: 255.0
    Max reward: 373.0
    Max reward: 500.0

# Using Checkpoints

You checkpoint the current state of a trainer to save what it has learned. Checkpoints are used for subsequent _rollouts_ and also to continue training later from a known-good state.  Calling `agent.save()` creates the checkpoint and returns the path to the checkpoint file, which can be used later to restore the current state to a new trainer. Here we'll load the trained policy into the same process, but often it would be loaded in a new process, for example on a production cluster for serving that is separate from the training cluster.

```python
checkpoint_path = agent.save()
print(checkpoint_path)
```

    /Users/jules/ray_results/PPOTrainer_CartPole-v1_2022-03-17_16-12-27gdrnqw6v/checkpoint_000005/checkpoint-5

Now load the checkpoint in a new trainer:

```python
trained_config = config.copy()
test_agent = PPOTrainer(trained_config, "CartPole-v1")
test_agent.restore(checkpoint_path)
```

    2022-03-17 16:13:24,362	WARNING ppo.py:223 -- `train_batch_size` (4000) cannot be achieved with your other settings (num_workers=3 num_envs_per_worker=1 rollout_fragment_length=200)! Auto-adjusting `rollout_fragment_length` to 1333.
    2022-03-17 16:13:30,386	INFO trainable.py:495 -- Restored on 127.0.0.1 from checkpoint: /Users/jules/ray_results/PPOTrainer_CartPole-v1_2022-03-17_16-12-27gdrnqw6v/checkpoint_000005/checkpoint-5
    2022-03-17 16:13:30,387	INFO trainable.py:503 -- Current state after restoring: {'_iteration': 5, '_timesteps_total': 39990, '_time_total': 20.854461193084717, '_episodes_total': 686}

Use the previously-trained policy to act in an environment. The key line is the call to `test_agent.compute_action(state)` which uses the trained policy to choose an action. This is an example of _rollout_, which we'll study in a subsequent lesson.

Verify that the cumulative reward received roughly matches up with the reward printed above. It will be at or near 500.

```python
env = gym.make("CartPole-v1")
state = env.reset()
done = False
cumulative_reward = 0

while not done:
    action = test_agent.compute_single_action(state)  # key line; get the next action
    state, reward, done, _ = env.step(action)
    cumulative_reward += reward

print(cumulative_reward)
```

    375.0

```python
ray.shutdown()
```

The next lesson, [02: Introduction to RLlib](02-Introduction-to-RLlib.ipynb) steps back to introduce to RLlib, its goals and the capabilities it provides.


```python
from IPython.display import Markdown, display

display(Markdown("models.md"))
```

## Introduction

After an initial outbreak in early 2020 in Wuhan, China, Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has spread globally. SARS-CoV-2 is capable of sustained human-to-human trans-
mission, and may cause severe disease, especially in older individuals. The COVID-19 pandemic has, in general, shown a remarkedly low incidence among children and young adults. Children have a lower susceptibility to infection and a lower propensity to show clinical symptoms. Furthermore, pre-symptomatic transmission is a major contributor to SARS-CoV-2 spread.
Unfortunately, pharmaceutical interventions such as vaccination and antiviral drugs are not yet available. On March 13th, 2020, the Belgian governments imposed strict social restrictions after tracing methods had failed to prevent large-scale spread of SARS-CoV-2. One and a half months later, the curve was succesfully flattened and social restrictions were gradually relaxed during the months of May, June and the beginning of July. In spite of these relaxations, hospitalizations kept declining. It is expected that during the coming year(s), preventing COVID-19 outbreaks will depend mostly on the successful implementation of non-pharmaceutical interventions such as social distancing, testing, contact tracing and quarantine. Hence the need for well-informed models that can assist policymakers in choosing the best non-pharmaceutical interventions in case of another large-scale SARS-CoV-2 outbreak. Currently, two other models exist to make predictions for Belgium, the agent-based model of Willem et al. and the discrete-time, stochastic metapopulation model of Coletti et al.

We built a deterministic, continuous-time, age-stratified, extended SEIRD model and calibrated it to national Belgian hospitalization data. The model accounts for pre-symptomatic and asymptomatic transmission. Furthermore, the susceptibility to SARS-CoV-2, the severity of the disease and the susceptibility to a subclinical infection depend on the age of the individual. We used social contact rates from a 2012 study by Willem to model age-specific social mixing. Because detailed hospitalization data are not made publicly available by the Belgian Scientific Institute of Public Health (Sciensano), we derived age-stratified hospitalization parameters from data of 370 patients treated in two Ghent (Belgium) hospitals. To overcome this unconstructive data policy, we used the computed hospitalization parameters as temporary proxies to fit the model to the total number of patients in Belgian hospitals and ICU units. Using the model, we computed the basic reproduction number ($R_0$) during the March 2020 epidemic. Because the model systematically overestimates the hospitalizations during lockdown relaxation, we re-estimated the basic reproduction using data from May 2020 until July 2020.

## The model

### Model dynamics

The SEIR(D) model was first proposed in 1929 by two Scottish scientists. It is a compartmental model that subdivides the human population into four groups: 1) susceptible individuals, 2) exposed individuals in the latent phase 1, 3) infectious individuals capable of transmitting the disease and 4) individuals removed from the population either through immunization or death. Despite being a simple and idealized reality, the SEIR(D) dynamics are used extensively to predict the outbreak of infectious diseases and this was no different during the SARS-CoV-2 outbreak earlier this year.

In this work, we extend the SEIRD model to incorporate more expert knowledge on SARS-CoV-2. For that purpose, the infectious compartment is split into four parts. The first is a period of presymptomatic infectiousness because several studies have shown that pre-symptomatic transmission is a dominant transmission mechanism of SARS-CoV-2. After the period of pre-symptomatic transmission, three possible infectious outcomes are modeled: (1) Asymptomatic outcome, for individuals who show no symptoms at all, (2) Mild outcome, for individuals with mild symptoms who recover at home, and (3) Hospitalization, when mild symptoms worsen. Children and young adults have a high propensity to experience an asymptomatic or mild outcome, while older individual have a high propensity to be hospitalized.

In general, Belgian hospitals have two wards for COVID-19 patients: 1) Cohort, where patients are not monitored permanently and 2) Intensive care, for patients with the most severe symptoms. Intensive care includes permanent monitoring, the use of ventilators or the use of extracorporeal membrane oxygenation (ECMO). Patients generally spend limited time in the emergency room and/or in a buffer ward before going to Cohort. After spending limited time in Cohort, some patients are transferred to ICU. Patients can perish in both wards, but mortalities are generally lower in Cohort. After a stay in an ICU, patients return to Cohort for recovery in the hospital. During the recovery stay, mortality is limited. The above is a short summary of hospital dynamics based on interviewing Ghent University hospital staff and examining the hospital data.

 Extended SEIRD dynamics of the BIOMATH COVID-19 model. Nodes represent model states, edges denote transfers. An overview of the model parameters can be found on the bottom of this page.

### Model framework and equations

#### Age-stratification
We introduced heterogeneity in the deterministic implementation by means of age-stratification. Every population compartment is split into a number of age classes, the age-groups have different contact rates with other age-groups and the disease progresses differently for each age-group, making the model behaviour more realistic. Our age-stratified model consists of 9 age classes, i.e., [0-10[, [10-20[, [20-30[, [30-40[, [40-50[, [50-60[, [60-70[, [70-80[, [80- $\infty$[. The age-stratified implementation provides a good balance between added complexity and computational resources.

#### Deterministic framework

Our extended SEIRD model is implemented using two frameworks: a deterministic and a stochastic framework. The deterministic equations are obtained by writing down the following equation,

`rate of change = in - out`,

for every of the 11 population compartments. This results in the following system of coupled ordinary differential equations,

$$
\begin{eqnarray}
\dot{S_i} &=& - \beta s_i S_i \sum_{j=1}^{N} N_{c,ij}  \Big( \frac{I_j+A_j}{T_j} \Big) + \zeta R_i, \\
\dot{E_i} &=& \beta s_i  S_i \sum_{j=1}^{N} N_{c,ij} \Big( \frac{I_j+A_j}{T_j} \Big) - (1/\sigma) \cdot E_i,  \\
\dot{I_i} &=& (1/\sigma) E_i - (1/\omega) I_i, \\
\dot{A_i} &=& (\text{a}_i/\omega) I_i - (1/d_{\text{a}}) A_i, \\
\dot{M_i} &=&  ((1-\text{a}_i) / \omega ) I_i - ( (1-h_i)/d_m + h_i/d_{\text{hospital}} ) M_i, \\
\dot{ER_i} &=& (h_i/d_{\text{hospital}}) M_i - (1/d_{\text{ER}}) ER_i, \\
\dot{C_i} &=& c_i (1/d_{\text{ER}}) ER_i  - (m_{C, i}/d_{c,D}) C_i - ((1 - m_{C, i})/d_{c,R}) C_i, \\
\dot{ICU_i} &=& (1-c_i) (1/d_{\text{ER}}) ER_i - (m_{ICU,i}/d_{\text{ICU},D}) ICU_i  \\
&& - ((1-m_{ICU,i})/d_{\text{ICU},R}) ICU_i,\\
\dot{C}_{\text{ICU,rec,i}} &=& ((1-m_{ICU,i})/d_{\text{ICU},R}) ICU_i - (1/d_{\text{ICU,rec}}) C_{\text{ICU,rec,i}}, \\
\dot{D_i} &=&  (m_{ICU,i}/d_{\text{ICU},D}) ICU_i +  (m_{C,i}/d_{\text{c},D}) C_i , \\
\dot{R_i} &=&  (1/d_a) A_i + ((1-h_i)/d_m) M_i + ((1-m_{C,i})/d_{c,R}) C_i \\
&& + (1/d_{\text{ICU,rec}}) C_{\text{ICU,rec,i}} - \zeta R_i,
\end{eqnarray}
$$

for $i = 1,2,...,9$. Here, $T_i$ stands for total population, $S_i$ stands for susceptible, $E_i$ for exposed, $I_i$ for pre-symptomatic and infectious, $A_i$ for asymptomatic and infectious, $M_i$ for mildly symptomatic and infectious, $ER_i$ for emergency room and/or buffer ward, $C_i$ for cohort, $C_{\text{ICU,rec,i}}$ for a recovery stay in Cohort coming from Intensive Care, $ICU_i$ for Intensive Care Unit, $D_i$ for dead and $R_i$ for recovered. Using the above notation, all model states are 9x1 vectors,

$$
\begin{equation}
     \mathbf{S} = [S_1(t)\ S_2(t)\ ...\ S_i(t)]^T,
\end{equation}
$$

where $S_i(t)$ denotes the number of susceptibles in age-class i at time t after the introduction of the virus in the population.

These equations are implemented in the function `COVID19_SEIRD` located in `src/covid19model/models.py`. The integration is performed in `_sim_single` located in `src/covid19model/base.py` by using Scipy's `solve_ivp`. The integrator timestep depends on the rate of change of the system and the solver method is thus referred to as a 'continuous-time' solver. The implementation uses non-integer individuals.

#### Stochastic framework

By defining the probabilities of transitioning (propensities) from one state to another, a system of coupled stochastic difference equations (SDEs) can be obtained. The probability to transition from one state to another is assumed to be exponentially distributed. As an example, consider the average time a patient spends in an ICU when recovering, which is $d_{\text{ICU,R}} = 9.9$ days. The chances of survival in ICU are $(1-m_{\text{ICU,i}})$, where $m_{\text{ICU,i}}$ is the mortality in ICU for an individual in age group $i$. The probability of transitioning from state ICU to state $C_{\text{ICU,rec}}$ on any given day and for an individual in age group $i$ is,

$$
\begin{equation}
P(ICU_i \rightarrow C_{\text{ICU,rec,i}}) = 1 - \text{exp} \Bigg[ - \frac{1-m_{\text{ICU},i}}{d_{\text{ICU,R}}}\Bigg].
\end{equation}
$$

If a transitioning between states is defined as "succes", we can regard the number of individuals transitioning from ICU to a Cohort recovery ward as a binomial experiment. On a given day, the number of individuals transitioning is,

$$
\begin{equation}
(\text{ICU}_i \rightarrow C_{\text{ICU,rec,i}})(k) \sim \text{Binomial}\Bigg(\text{ICU}_i(k),\ 1 - \text{exp}\Bigg[- \frac{1-m_{\text{ICU,i}}}{d_{\text{ICU,R}}}\Bigg]\Bigg).
\end{equation}
$$

For a discrete stepsize $l$, there are 15 possible transitions,

$$
\begin{eqnarray}
(S_i \rightarrow E_i) (k) &\sim& \text{Binomial}\Bigg(S_i(k), 1 - \text{exp}\Bigg[- l \beta s_i \sum_{j=1}^{N} N_{c,ij}  \Big( \frac{I_j+A_j}{T_j} \Big) \Bigg]\Bigg)\\
(E_i \rightarrow I_i) (k) &\sim& \text{Binomial}\Bigg(E_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1}{\sigma}\Bigg]\Bigg)\\
(I_i \rightarrow A_i) (k) &\sim& \text{Binomial}\Bigg(I_i(k), 1 - \text{exp}\Bigg[- l\ \frac{a_i}{\omega}\Bigg]\Bigg)\\
(I_i \rightarrow M_i) (k) &\sim& \text{Binomial}\Bigg(I_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-a_i}{\omega}\Bigg]\Bigg)\\
(A_i \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(A_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1}{d_a}\Bigg]\Bigg)\\
(M_i \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(M_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-h_i}{d_m}\Bigg]\Bigg)\\
(M_i \rightarrow ER_i) (k) &\sim& \text{Binomial}\Bigg(M_i(k), 1 - \text{exp}\Bigg[- l\ \frac{h_i}{d_{\text{hospital}}}\Bigg]\Bigg)\\
(ER_i \rightarrow C_i) (k) &\sim& \text{Binomial}\Bigg(ER_i(k), 1 - \text{exp}\Bigg[- l\ \frac{c_i}{d_{\text{ER}}}\Bigg]\Bigg)\\
(ER_i \rightarrow ICU_i) (k) &\sim& \text{Binomial}\Bigg(ER_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-c_i}{d_{\text{ER}}}\Bigg]\Bigg)\\
(C_i \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(C_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-m_{C,i}}{d_{C,R}}\Bigg]\Bigg)\\
(ICU_i \rightarrow C_{\text{ICU,rec,i}}) (k) &\sim& \text{Binomial}\Bigg(ICU_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-m_{\text{ICU,i}}}{d_{ICU,R}}\Bigg]\Bigg)\\
(C_{\text{ICU,rec,i}} \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(C_{\text{ICU,rec,i}}(k), 1 - \text{exp}\Bigg[- l\ \frac{1}{d_{\text{ICU,rec}}}\Bigg]\Bigg)\\
(C_i \rightarrow D_i) (k) &\sim& \text{Binomial}\Bigg(C_i(k), 1 - \text{exp}\Bigg[- l\ \frac{m_{C,i}}{d_{C,D}}\Bigg]\Bigg)\\
(ICU_i \rightarrow D_i) (k) &\sim& \text{Binomial}\Bigg(ICU_i(k), 1 - \text{exp}\Bigg[- l\ \frac{m_{\text{ICU,i}}}{d_{\text{ICU,D}}}\Bigg]\Bigg)\\
(R_i \rightarrow S_i) (k) &\sim& \text{Binomial}\Bigg(R_i(k), 1 - \text{exp}\Bigg[- l\ \zeta \Bigg]\Bigg)\\
\end{eqnarray}
$$

And the system of equations becomes,

$$
\begin{eqnarray}
S_i(k+1) &=& S_i(k) + (R_i \rightarrow S_i) (k) - (S_i \rightarrow E_i) (k) \\
E_i(k+1) &=& E_i(k) + (S_i \rightarrow E_i) (k) - (E_i \rightarrow I_i) (k) \\
I_i(k+1) &=& I_i(k) + (E_i \rightarrow I_i) (k) - (I_i \rightarrow A_i) - (I_i \rightarrow M_i) (k) \\
A_i(k+1) &=& A_i(k) + (I_i \rightarrow A_i) (k) - (A_i \rightarrow R_i) (k) \\
M_i(k+1) &=& M_i(k) + (I_i \rightarrow M_i) (k) - (M_i \rightarrow R_i) (k) - (M_i \rightarrow ER_i) (k) \\
ER_i(k+1) &=& ER_i(k) + (M_i \rightarrow ER_i) (k) - (ER_i \rightarrow C_i) (k) - (ER_i \rightarrow ICU_i) (k) \\
C_i(k+1) &=& C_i(k) + (ER_i \rightarrow C_i) (k) - (C_i \rightarrow R_i) (k) - (C_i \rightarrow D_i) (k) \\
C_{\text{ICU,rec,i}}(k+1) &=& C_{\text{ICU,rec,i}}(k)  + (ICU_i \rightarrow C_{\text{ICU,rec,i}}) (k) - (C_{\text{ICU,rec,i}} \rightarrow R_i) (k) \\
R_i(k+1) &=& R_i(k) + (A_i \rightarrow R_i) (k)  + (M_i \rightarrow R_i) (k) + (C_i \rightarrow R_i) (k)\\
&& + (C_{\text{ICU,rec,i}} \rightarrow R_i) (k)  - (R_i \rightarrow S_i) (k) \\
D_i(k+1) &=& D_i(k) + (ICU_i \rightarrow D_i) (k) + (C_i \rightarrow D_i) (k) \\
\end{eqnarray}
$$

These equations are implemented in the function `COVID19_SEIRD_sto` located in `src/covid19model/models.py`. The computation itself is performed in the function `solve_discrete` located in `src/covid19model/base.py`. Please note that the deterministic model uses **differentials** in the model defenition and must be integrated, while the stochastic model uses **differences** and must be iterated. The discrete timestep is fixed at one day. The stochastic implementation only uses integer individuals, which is considered an advantage over the deterministic implementation.

### Spatial extension

Rather than considering the entire nation as a homogeneous collection of compartments, it is more realistic to consider spatially distinct regions that are mutually connected. The relevance of this approach becomes clear by looking at the spatial spread in the data (embedded video below): the occurence of confirmed cases is not entirely homogeneous.

The main motivation for investigating spatial relations is to quantify the effect of certain types of mobility on the severity and speed of the spread of COVID19, and the effect of spatial isolation of key regions. We consider e.g. the 43 Belgian arrondissements or the 10 Belgian provinces (plus Brussels-Capital). These regions are considered internally homogeneous and are connected through known or estimated mobility matrices.

Below, the spatially explicit model is worked out in a deterministic framework and a stochastic framework. These are largely based on the approach taken in [this 2020 paper](https://www.medrxiv.org/content/10.1101/2020.03.21.20040022v1) by Arenas et al.

#### Deterministic spatial framework

The deterministic approach is largely the same as in the non-spatial framework, only this time every 'patch' (either arrondissement or province) exhibits the dynamics explained before. We make the following assumptions from the mobility:
1. The effective population of a patch is not the same as the registered population, because (during the daytime) some people will leave their home patch and go to another, e.g. for their profession
2. People can get become exposed in their home patch or in the patch they visit
3. The overall mobility is scaled with a single parameter $p$, ranging between 0 and 1, where 0 means no mobility is possible, and 1 indicates regular mobility. This mobility parameter is typically below 1 in a pandemic situation, due to travelling restrictions.
4. Social contact rates are (slightly) different in every patch. Following Arenas et al., this translates to a rescaling of the overall contact rate with respect to density.

Mathematically, the **effective population** in a particular patch $g$ is

$$
\begin{equation}
T^g_{i,\text{eff}} = \sum_{h=1}^G [(1-p_i)\delta^{gh} + p_i P_i^{hg}]T_i^h,
\end{equation}
$$

where $G$ is the total number of patches (e.g. 43 arrondissements), $p_i$ the mobility parameter (stratified per age $i$), $\delta^{gh}$ the Kronecker delta, $P_i^{hg}$ the recurrent mobility *from* patch $h$ *to* patch $g$ (stratified per age $i$), and $T_i^h$ the registered population in age class $i$ and patch $h$. The same formula works goes for $I^g_{i,\text{eff}}$ and $E^g_{i,\text{eff}}$. Note that $T^g_{i,eff} = T^g_i$ if $p_i = 0$. Note: $P_i^{hg}$ is implemented in the code as `place`.

Because people can get exposed in their own *or* another patch, we need an expression for the **number of susceptibles** from patch $g$ that reside in patch $h$, in age class $i$:

$$
\begin{equation}
    S_i^{gh} = p_i P_i^{gh} S_i^g + (1-p_i) \delta^{gh} S_i^g
\end{equation}
$$

We now have a **social contact matrix** for every patch, $N^g_{c,ij}$. We follow the suggestion in Arenas et al. that the contact rate is proportional to a population density function $f$, or

$$
\begin{equation}
N^g_{c,ij} \propto f\left( \frac{T^g_\text{eff}}{a^g} \right).
\end{equation}
$$

Here $T^g_\text{eff}$ is again the effective population of patch $g$, and $a^g$ its area. The function $f$ is defined as

$$
\begin{equation}
    f(x) = 1 + (1-e^{-\xi x}),
\end{equation}
$$

and increases monotonically from 1 to 2 (see figure below)

This allows us to rewrite $N^g_{c,ij}$:

$$
\begin{equation}
    N^g_{c,ij} = z_i f\left( \frac{T^g_\text{eff}}{a^g} \right) N_{c,ij},
\end{equation}
$$

where Arenas et al. call $z_i$ the 'normalisation factor':

$$
\begin{equation}
    z_i = \frac{ T_i }{ \sum\limits_{h=1}^G f\left(\dfrac{T^h_\text{eff}}{a^h}\right) T^h_{i,\text{eff}}},
    \label{eq:z_i-normalisation}
\end{equation}
$$

with $T_i$ the total population in age class $i$.

Having that background, we define a quantity

$$
\begin{equation}
    B^g_i \equiv \sum_{j=1}^N \beta s_i z_i f\left(\frac{T_\text{eff}^g}{a^g}\right) N_{c,ij} \frac{I_{j,\text{eff}}^g + A_{j,\text{eff}}^g}{T_{j,\text{eff}}^g}
\end{equation}
$$

The **coupled ordinary differential equations** can be rewritten to the spatial situation
$$
\begin{eqnarray}
\dot{S}_i^g &=& - \sum_{h=1}^G S_i^{gh} B^h_i + \zeta R_i^g, \\
\dot{E}_i^g &=& \sum_{h=1}^G S_i^{gh} B^h_i - (1/\sigma) \cdot E_i^g,  \\
\dot{I}_i^g &=& (1/\sigma) E_i - (1/\omega) I_i^g, \\
\dot{A}_i^g &=& (\text{a}_i/\omega) I_i^g - (1/d_{\text{a}}) A_i^g, \\
\dot{M}_i^g &=&  ((1-\text{a}_i) / \omega ) I_i^g - ( (1-h_i)/d_m + h_i/d_{\text{hospital}} ) M_i^g, \\
\dot{ER}_i^g &=& (h_i/d_{\text{hospital}}) M_i^g - (1/d_{\text{ER}}) ER_i^g, \\
\dot{C}_i^g &=& c_i (1/d_{\text{ER}}) ER_i^g  - (m_{C, i}/d_{c,D}) C_i^g - ((1 - m_{C, i})/d_{c,R}) C_i^g, \\
\dot{ICU}+i^g &=& (1-c_i) (1/d_{\text{ER}}) ER_i^g - (m_{ICU,i}/d_{\text{ICU},D}) ICU_i^g  \\
&& - ((1-m_{ICU,i})/d_{\text{ICU},R}) ICU_i^g,\\
\dot{C}_{\text{ICU,rec,i}}^g &=& ((1-m_{ICU,i})/d_{\text{ICU},R}) ICU_i^g - (1/d_{\text{ICU,rec}}) C_{\text{ICU,rec,i}}^g, \\
\dot{D}_i^g &=&  (m_{ICU,i}/d_{\text{ICU},D}) ICU_i^g +  (m_{C,i}/d_{\text{c},D}) C_i^g , \\
\dot{R}_i^g &=&  (1/d_a) A_i + ((1-h_i)/d_m) M_i^g + ((1-m_{C,i})/d_{c,R}) C_i^g \\
&& + (1/d_{\text{ICU,rec}}) C_{\text{ICU,rec,i}}^g - \zeta R_i^g,
\end{eqnarray}
$$

All state matrices have dimensions $G \times N$, so integrating these ODEs takes (more than) $G$ times the computation time compared to the non-spatial case. Note that, if relevant and available, all parameters may be further stratified per patch $g$.

These equations are implemented in the function `COVID19_SEIRD_spatial` located in `src/covid19model/models.py`. When creating the simulation, these equations are integrated numerically.

#### Stochastic spatial framework

Extending the deterministic spatial framework to a (discrete) stochastic one is straightforward. The probability of being infected in patch $g$ in the next timestep (stepsize $\ell$) for a susceptible individual is

$$
\begin{equation}
    P^g_i = 1 - \exp\left( - \ell \sum\limits_{j=1}^N \beta s_i z_i f\left( \frac{T_\text{eff}^g}{a^g} \right) N_{c,ij} \frac{ I^{g}_{j, \text{eff}} + A^{g}_{j, \text{eff}}}{T_{j,\text{eff}}^g} \right)
\end{equation}
$$

The same delocalised infection happens here, again depending on the overall mobility $p_i$ and the recurrent mobility matrix $P_i^{gh}$:
\begin{equation}
    \Pi_i^g = P_i^g + p_i\sum\limits_{\substack{h=1 \\ h\neq g}}^G P_i^{gh}P_i^h
\end{equation}
The **overall probability of exposure** for an agent in age class $i$ and patch $g$, $\Pi_i^g$, is a sum of
1. The probability of being infected at home, *plus*
2. The probability of being infected outside the home patch, weighed by the mobility to that patch

Note that a higher mobility parameter does *not* necessarily mean that the exposure probability $\Pi_i^g$ is larger, because the probability of becoming infected in the home patch, $P_i^g$, is also dependant on the mobility parameter $p_i$.

This translates to the same **set of stochastic equations** as in the non-spatial case, but now with spatial stratification and a slightly adjusted probability in the binomial experiment:
$$
\begin{eqnarray}
(S_i^g \rightarrow E_i^g) (k) &\sim& \text{Binomial}\Bigg(S_i^g(k), \Pi_i^g(k) \Bigg)\\
(E_i^g \rightarrow I_i^g) (k) &\sim& \text{Binomial}\Bigg(E_i^g(k), \Pi_i^g(k) \Bigg)\\
\end{eqnarray}
$$

These in turn determine the discrete development of the stochastic model:

$$
\begin{eqnarray}
S_i^g(k+1) &=& S_i^g(k) + (R_i^g \rightarrow S_i^g) (k) - (S_i^g \rightarrow E_i^g) (k) \\
E_i^g(k+1) &=& E_i^g(k) + (S_i^g \rightarrow E_i^g) (k) - (E_i^g \rightarrow I_i^g) (k) \\
\end{eqnarray}
$$

... etcetera (see non-spatial case).

An **example** of the stochastic spatial framework is shown in the video below.

Simulation with the stochastic model and an initial condition where at time = 0, 5 exposed individuals in age class 30-40 years are 'released' in Arlon, all the way on the south-easter side of the country. At the onset, it is clear how the 'wave' of new exposures travels through the country, hitting the densely populated patches first. At day 40, measures are taken that bring down the effective reproduction constant below zero.

These equations are implemented in the function `COVID19_SEIRD_sto_spatial` located in `src/covid19model/models.py`. The computation itself is performed in the function `solve_discrete` located in `src/covid19model/base.py`. Please note that the deterministic model uses **differentials** in the model definition and must be integrated, while the stochastic model uses **differences** and must be iterated. The discrete timestep is fixed at one day. The stochastic implementation only uses integer individuals, which is considered an advantage over the deterministic implementation.

#### Spatial: remaining issues and tasks

The theory at the basis of both the deterministic and the stochastic spatial model is still subject to change. Some of these 'to-dos' are listed below.

1. In the original implementation of the spatial method, the type of contact in the exposure probability was related to whether or not the contact takes place in the home patch or not (see equation below, in the original notation). The reason is that e.g. home contact ($N_{\text{home},ij}$ is not relevant when visiting other places; if the individual does not work in his home patch, the work vs home, school, leisure, other human-to-human contacts must be seperate. Whilst this is certainly true, it is currently unclear whether this additional complication is worth pursuing.
$$
\begin{eqnarray}
    P(S_{i,g} \rightarrow E_{i,g}) (k) &=& 1 - \text{exp} \Bigg[ \underbrace{\text{P}_{g,g} \Bigg\{ - l \beta s_i \sum_{j=1}^{N} N_{\text{c, tot, ij}} \Bigg( \frac{I_{j,g} + A_{j,g}}{T_{j,g}} \Bigg) \Bigg\}}_{\text{individual working in residence patch}}  \\
    &+& \sum_{l=1\\ l \neq g}^{G} \text{P}_{g,l} \Bigg\{ \underbrace{- l \beta s_i \sum_{j=1}^{N} N_{\text{c, work, ij}} \Bigg( \frac{I_{j,l} + A_{j,l}}{T_{j,l}} \Bigg)}_{\text{work interactions in work patch (subscript l)}} \\
     &-& \underbrace{l \beta s_i \sum_{j=1}^{N} (N_{\text{c, home, ij}} + N_{\text{c, school, ij}} + N_{\text{c, leisure, ij}} + N_{\text{c, others, ij}}) \Bigg( \frac{I_{j,g} + A_{j,g}}{T_{j,g}} \Bigg)}_{\text{all other interactions in home patch (subscript g)}} \Bigg\} \Bigg]
\end{eqnarray}
$$
2. Many parameters, most notably the mobility parameter $p_i$, can (and often should) be stratified further.
3. The calibration methods are still under construction and are more difficult, mainly because of the higher number of possible choices:
    * Should every patch have its own $\beta^g$ value?
    * How do we go about choosing an initial condition for the calibration?
4. A good spatial model is largely dependant on good mobility data. The currently available data is a mobility matrix from 2011, which is quite certainly not representative for the 2020 pandemic situation. Ideally we have *current* mobility data, stratified per *age class* and per *activity* (work/school/leisure/...)
5. This model currently does *not* account for importation of exposed individuals, which in a highly populated region like Western Europe is undoubtedly an important factor.
6. The contact rate's dependence on the density may be omitted -- it has little effect -- but is currently implemented to parallel Arenas et al.'s approach.
7. Arguably, stratification at the level of arrondissements is too highly detailed in a 'big city' like Belgium. Data analysis of the current situation may point out that it is justified to aggregate some arrondissements with a similar time series morphology together.

### Transmission rates and social contact data

In our model, the transmission rate of the disease depends on the product of four contributions. The first contribution, $(I+A)/T$, is the fraction of contagious individuals in the population. The second contribution, $\mathbf{N}_c$, is the average number of human-to-human interactions per day. The third contribution, $s_i$, is the relative susceptiblity to SARS-CoV-2 infection in age group $i$, and the fourth contribution, $\beta$, is the probability of contracting COVID-19 when encountering a contagious individual under the assumption of 100 \% susceptibility to SARS-CoV-2 infection. We assume that the per contact transmission probability $\beta$ is independent of age and we will infer its distribution by calibrating the model to national Belgian hospitalization data. The number of human-human interactions, $\mathbf{N}_c$, are both place and age-dependent. These matrices assume the form of a 9x9 *interaction matrix* where an entry X, Y denotes the number of social contacts age group X has with age group Y per day. These matrices are available for homes, schools, workplaces, in public transport, and leisure activities, from a survey study by Lander Willem (2012). The total number of social interactions is given by the sum of the contributions in different places,

$$
\begin{equation}\label{eq:interaction_matrices}
\mathbf{N_{\text{c}}} = \mathbf{N_{\text{c, home}}} + \mathbf{N_{\text{c, schools}}} + \mathbf{N_{\text{c, work}}} + \mathbf{N_{\text{c, transport}}} + \mathbf{N_{\text{c, leisure}}} + \mathbf{N_{\text{c, others}}}.
\end{equation}
$$

Coefficients can be added to the contributing contact matrices to model a goverment policy. For instance, to model the Belgian lockdown, the mobility reductions deduced from the Google community mobility reports were used as coefficients for the different interaction matrices. We assumed workplace interactions were down to only 40 % of their prepandemic values before the lockdown.

### Modeling social intertia

The model takes into account the effect of *social inertia* when measures are taken. In reality, social restrictions or relaxations represent a change in behaviour which is gradual and cannot be modeled using a step-wise change of the social interaction matrix $\mathbf{N_c}$. This can be seen when closely inspecting the *Google community mobility report* above. Multiple functions can be used to model the effects of social compliance, e.g. a delayed or non-delayed ramp, or a logistic function. In our model, we use a delayed ramp to model compliance,

$$
\begin{equation}
\mathbf{N_{c}}^{k} = \mathbf{N_{\text{c, old}}} + f^{k} (\mathbf{N_{\text{c, new}}} - \mathbf{N_{\text{c, old}}})
\end{equation}
$$

where,

$$
\begin{equation}
    f^k=
\begin{cases}
	0.0,& \text{if } k\leq \tau\\
    \frac{k}{l} - \frac{\tau}{l},& \text{if } \tau < k\leq \tau + l\\
    1.0,              & \text{otherwise}
\end{cases}
\end{equation}
$$

where $\tau$ is the number of days before measures start having an effect and $l$ is the number of additional days after the time delay until full compliance is reached. Both parameters were calibrated to the daily number of hospitalizations in Belgium (notebooks `notebooks/0.1-twallema-calibration-deterministic.ipynb` and `notebooks/0.1-twallema-calibration-stochastic.ipynb`). $k$ denotes the number of days since a change in social policy.

### Basic reproduction number

The basic reproduction number $R_0$, defined as the expected number of secondary cases directly generated by one case in a population where all individuals are susceptible to infection, is computed using the next generation matrix (NGM) approach introducted by Diekmann. For our model, the basic reproduction number of age group $i$ is,

$$
\begin{equation}\label{eq:reproduction_number}
R_{0,i} = (a_i d_a + \omega) \beta s_i \sum_{j=1}^{N} N_{c,ij}
\end{equation}
$$

and the population basic reproduction number is calculated as the weighted average over all age groups using the demographics of Belgium. The detailed algebra underlying the computation equation of the basic reproduction number is presented in the supplementary materials of our manuscript (see section *Previous work*).

### Model parameters

An in-depth motivation of the model parameters is provided in our manuscript (see section *Previous work*). Hospital parameters were derived from a dataset obtained from two hospitals in Ghent (Belgium).

 Overview of BIOMATH COVID-19 model parameters.

docs/Markdown reading tool for Notebook.ipynb