# CIS-522 Week 11 Part 1
# Introduction to Reinforcement Learning

__Instructor:__ Dinesh Jayaraman

__Content creators:__ Chuning Zhu

---

```
#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)
my_pennkey = 'fparodi' #@param {type:"string"}
my_pod = 'superfluous-lyrebird' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','astute-jellyfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion']

```

## Recap the experience from last week

What did you learn last week. What questions do you have? [10 min discussion]

```
learning_from_previous_week = "learned attention, transformers. very much enjoyed that. i struggled understanding nlp and grus though. not sure if it's bc we're at the end of the semester or the pandemic or what. but i \u003C3 transformesr!" #@param {type:"string"}
```

---
# Setup

```
# imports

import math
import numpy as np
import IPython
from numbers import Number
from matplotlib import pyplot as plt
import matplotlib.patches as patches
from tqdm.auto import tqdm
```

```
# @title Plotting functions
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/"
              "course-content/master/nma.mplstyle")

# Plotting functions
def plot_episode_rewards(episode_rewards):
    fig = plt.figure()
    plt.plot(episode_rewards)
    plt.xlabel("Episode")
    plt.ylabel("Reward")
    fig.show()
```

```
# @title Gridworld Environment

'''
    A custom Gridworld environment with deterministic transitions. Adapted from
    CS 188 Gridworld env. There are four actions: up, left, down, right. The
    state is the (x, y) coordinates in the Grid.
'''
class Gridworld():

    def __init__(self, grid, living_reward=-1.0):
        self.h = len(grid)
        self.w = len(grid[0])
        self.living_reward = living_reward

        self.scale = math.ceil(max(self.h, self.w) / min(self.h, self.w))
        self.action_space = ['up', 'left', 'down', 'right']
        self.n_actions = 4

        self.init_grid(grid)

    def init_grid(self, grid):
        # Create reward grid. The reward grid is a numpy array storing the
        # reward given for entering each state.
        self.rew_grid = np.array([[self.living_reward if isinstance(e, str) else e
                                   for e in row] for row in grid], dtype=np.float)

        # Create grid. The grid is a numpy of chars.
        # S (start), T (terminal), C (cliff), # (block), or ' ' (regular).
        convert_fn = lambda e: 'T' if e >= self.living_reward else 'C'
        self.grid = np.array([[convert_fn(e) if isinstance(e, Number) else e
                               for e in row] for row in grid])

        # Find initial state
        start_indices = np.argwhere(self.grid == 'S')
        if len(start_indices) == 0:
            raise Exception('Grid has no start state')
        self.init_state = (start_indices[0][1], start_indices[0][0])

    def get_transition(self, state, action):
        '''
        Execute one action in the environment.
        Args:
            state (tuple): the (x, y) coordinates of the current state.
            action (int): the current action chosen from {0, 1, 2, 3}.

        Returns:
            next_state (tuple): the (x, y) coordinates of the next state.
            reward (float): the reward for the current time step.
        '''
        # Handle terminal states
        x, y = state
        if self.grid[y, x] == 'T':
            return state, 0

        # Handle invalid actions
        if action not in range(len(self.action_space)):
            raise Exception('Illegal action')

        # Default transitions
        named_action = self.action_space[action]
        nx, ny = x, y
        if named_action == 'up':
            ny -= 1
        elif named_action == 'left':
            nx -= 1
        elif named_action == 'down':
            ny += 1
        elif named_action == 'right':
            nx += 1

        # Handle special cases
        if nx < 0 or nx >= self.w or ny < 0 or ny >= self.h or self.grid[ny, nx] == '#':
            # Give living reward if next state is blocked or out of bounds
            reward = self.living_reward
            next_state = (x, y)
        else:
            reward = self.rew_grid[ny, nx]
            if self.grid[ny, nx] == 'C':
                next_state = self.init_state # falls off cliff
            else:
                next_state = (nx, ny) # transition to next state

        return next_state, reward

    def __render(self):
        # Render grid with matplotlib patches.
        fig, ax = plt.subplots(figsize=(self.h*self.scale, self.w*self.scale))
        ax.set_aspect('equal')
        ax.set_xlim(0, self.w)
        ax.set_ylim(0, self.h)
        ax.set_xticklabels([])
        ax.set_yticklabels([])
        ax.tick_params(length=0)
        plt.axis('off')
        for y in range(self.h):
            for x in range(self.w):
                cell_type = self.grid[y, x]
                if cell_type == 'S':
                    c = '#DAE8FC' # blue
                elif cell_type == '#':
                    c = '#CCCCCC' # gray
                elif cell_type == 'T':
                    c = '#D5E8D4' # green
                elif cell_type == 'C':
                    c = '#F8CECC' # red
                else:
                    c = '#FFFFFF' # white
                rect = patches.Rectangle((x, self.h-y-1), 1, 1, fc=c, ec='gray', lw=1)
                ax.add_patch(rect)
        return fig, ax

    def render_grid(self):
        fig, ax = self.__render()
        for y in range(self.h):
            for x in range(self.w):
                if self.grid[y, x] != '#':
                    # alternate: x+0.1, self.h-y-0.2
                    ax.text(x+0.5, self.h-y-0.5, str(self.rew_grid[y, x]), size='medium', ha='center', va='center')
        plt.title("Rewards")
        fig.show()

    def render_values(self, V):
        fig, ax = self.__render()
        for y in range(self.h):
            for x in range(self.w):
                ax.text(x+0.5, self.h-y-0.5, '{:.2f}'.format(V[y, x]), size='medium', ha='center', va='center')
        plt.title("Values")
        fig.show()

    def render_q_values(self, Q):
        fig, ax = self.__render()
        for y in range(self.h):
            for x in range(self.w):
                named_action = self.action_space[np.argmax(Q[y, x])]
                xl, xc, xr = x, x+0.5, x+1
                yt, yc, yb = self.h-y, self.h-y-0.5, self.h-y-1
                ce, tl, bl, tr, br = [xc, yc], [xl, yt], [xl, yb], [xr, yt], [xr, yb]
                if named_action == 'up':
                    xy = np.array([ce, tl, tr])
                elif named_action == 'left':
                    xy = np.array([ce, tl, bl])
                elif named_action == 'down':
                    xy = np.array([ce, bl, br])
                elif named_action == 'right':
                    xy = np.array([ce, br, tr])
                ax.plot([x, x+1], [self.h-y, self.h-y-1], 'gray', lw=1)
                ax.plot([x, x+1], [self.h-y-1, self.h-y], 'gray', lw=1)
                poly = patches.Polygon(xy, True, fc='#FFFF00', ec='gray')
                ax.add_patch(poly)
                ax.text(x+0.5, self.h-y-0.2, '{:.2f}'.format(Q[y, x, 0]), size='small', ha='center', va='center')
                ax.text(x+0.2, self.h-y-0.5, '{:.2f}'.format(Q[y, x, 1]), size='small', ha='center', va='center')
                ax.text(x+0.5, self.h-y-0.8, '{:.2f}'.format(Q[y, x, 2]), size='small', ha='center', va='center')
                ax.text(x+0.8, self.h-y-0.5, '{:.2f}'.format(Q[y, x, 3]), size='small', ha='center', va='center')
        fig.show()
        plt.title("Q-values")
        pass

    def render_policy(self, policy):
        fig, ax = self.__render()
        for y in range(self.h):
            for x in range(self.w):
                if policy[y, x] not in range(len(self.action_space)):
                    raise Exception('Illegal action')
                if self.grid[y, x] == 'T':
                    continue
                arrow_len = 0.3
                dx, dy = 0, 0
                named_action = self.action_space[policy[y, x]]
                if named_action == 'up':
                    dy = arrow_len
                elif named_action == 'left':
                    dx = -arrow_len
                elif named_action == 'down':
                    dy = -arrow_len
                elif named_action == 'right':
                    dx = arrow_len
                arrow = patches.FancyArrow(x+0.5, self.h-y-0.5, dx, dy, 0.03, True, color='#6C8EBF')
                ax.add_patch(arrow)
        plt.title("Policy")
        fig.show()

'''
GridworldEnv is a wrapper around Gridworld implementing an RL interface.
'''
class GridworldEnv(Gridworld):
    def __init__(self, grid, living_reward=-1.0):
        super().__init__(grid, living_reward)
        self.reset()

    def reset(self):
        '''
        Reset the agent to its initial state
        '''
        self.state = self.init_state
        return self.state

    def step(self, action):
        '''
        Execute one action in the environment.
        Args:
            action (int): the current action chosen from {0, 1, 2, 3}.

        Returns:
            next_state (tuple): (x, y) coordinates of the next state.
            reward (float): reward for the current time step.
            done (bool): True if a terminal state has been reached, False otherwise.
        '''
        next_state, reward = self.get_transition(self.state, action)
        self.state = next_state
        done = self.grid[self.state[1], self.state[0]] == 'T'
        return next_state, reward, done

# Pre-defined grids
def get_book_grid():
    grid = [['T',  ' ',  ' ',  ' '],
            [' ',  ' ',  ' ',  ' '],
            [' ',  ' ',  ' ',  ' '],
            ['S',  ' ',  ' ',  'T']]
    return GridworldEnv(grid)

def get_cliff_small():
    grid = [[' ',   ' ',  ' ',  ' ',  ' '],
            ['S',   ' ',  ' ',  ' ',   'T'],
            [-100, -100, -100, -100, -100]]
    return GridworldEnv(grid)

def get_cliff_walk():
    grid = [[' ' for _ in range(12)] for _ in range(3)]
    grid.append([-100 for _ in range(12)])
    grid[3][ 0] = 'S'
    grid[3][-1] = 'T'
    return GridworldEnv(grid)

def get_bridge_grid():
    grid = [[ '#',-100, -100, -100, -100, -100, '#'],
            [   1, 'S',  ' ',  ' ',  ' ',  ' ',  10],
            [ '#',-100, -100, -100, -100, -100, '#']]
    return GridworldEnv(grid)
```

---
# Section 1: Introduction

```
#@title Video : Intro to Reinforcement Learning

import time
try: t0;
except NameError: t0=time.time()

from IPython.display import YouTubeVideo
video = YouTubeVideo(id="cVTud58UfpQ", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=cVTud58UfpQ

Up to this point, we have mainly been concerned with supervised learning. In a supervised learning problem, we are provided with a dataset where each sample comes with a ground truth label (e.g. class label), and the goal is to learn to predict the label by minimizing some loss function. Reinforcement learning, on the other hand, is a framework for solving sequential decision-making problems. Consider an agent operating in some environment. The agent's goal is to carry out the best sequence of actions that maximizes the cumulative reward. This is difficult because the action at the current time step influences future states of the environment, which then feed back to the agent's observations. The following figure illustrates this setting.

What is the role of reinforcement learning in intelligence? According to Yann LeCun, if intelligence is a cake, then unsupervised learning is the bulk of the cake, supervised learning the icing, and reinforcement learning the cherry on top. The reason RL takes up such a small proportion is that very little learning in real world comes with explicit reward signal. This analogy is still debatable, as some RL folks argue that intelligence is more like a cake with lots of cherries on top, especially after the invention of [hindight experience replay](https://arxiv.org/abs/1707.01495). In addition, there are ways to solve sequential decision making problems without relying on shaped rewards, such as inverse reinforcement learning, which infers a reward function from experience, and learning from goals / demonstrations / examples. Another way to put RL in perspective is by comparing it with vision and natural language processing. If we decompose intelligence into perception, cognition (reasoning), and action (decision making), then vision coarsely corresponds to perception, NLP cognition, and RL action. Just like how vision can be combined with NLP for tasks like image captioning, RL can be organically combined with vision and NLP as well.

In this first tutorial, we will briefly step away from deep learning and study a few classic approaches in reinforcement learning. A good reference is Sutton and Barto's book, Reinforcement Learning: An Introduction. The [full text](http://incompleteideas.net/book/the-book.html) is avaliable online.

---
# Section 2: MDP and Bellman Equations

## Section 2.1: Markov Decision Process

```
#@title Video : Markov Decision Processes
try: t1;
except NameError: t1=time.time()

video = YouTubeVideo(id="GJEL-QkT2yk", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=GJEL-QkT2yk

We begin our study of reinforcement learning with a definition of  Markov decision process. A Markov decision process (MDP) is a tuple $(S, A, P, R, \gamma)$, where

- $S$ is the set of **states**.
- $A$ is the set of **actions**.
- $P$ defines the **transition probablities**. $P(s'|s, a)$ gives the probability of transitioning to state $s'$ by taking action $a$ at state $s$.
- $R$ is the **reward function**. $R(s, a)$ gives the reward of taking action $a$ at state $s$. $R$ can also be a function of state only.
- $\gamma$ is the **discount factor**. It controls how much future rewards matter to us. We will talk more about discount factor in the next video.

As an aside, we introduce partially observable MDP (POMDP). A POMDP additionally has a set of obervations $O$ and emission probabilities $\varepsilon$. $\varepsilon(o|s)$ gives the probability of observing $o$ at state $s$. This formulation is useful when we don't have access to explicit state information, but are provided with observations that may not fully reveal the underlying states. An example is reinforcement learning from images.

Come up with a sequential decision making problem and formalize it as an MDP. What are $S$, $A$, $P$, and $R$? Share your example with your pod.

```
MDP_example = 'the classic game theoretic example of having hot dog stand on the beach. the state would be the physical location of the stand; action would be moving the stand; P would be how likely you are to move given previous action/reward; reward is profits, relative to previous profits' #@param {type:"string"}
```

## Section 2.2 Solving MDPs

```
#@title Video : Solving MDPs

video = YouTubeVideo(id="meywaLPitZ4", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=meywaLPitZ4

A policy $\pi$ is a function mapping states to distributions over actions. At state $s$, we sample an action from the distribution $\pi(a|s)$ to execute in the environment. If all probability mass is assigned to one action, then the policy is deterministic. The goal of reinforcement learning is to find an optimal policy that maximizes the expected sum of discounted rewards: $$E\left[\sum_{t=0}^{\infty}\gamma^tR(s_t, a_t)\right]$$

Note that this objective assumes a continuous task, i.e. that $t$ extends to infinity. We can generalize it to episodic tasks with finite horizons by replacing $\infty$ with task horizon $T$. We may also discard the discount factor $\gamma$ in an episodic task.

Before we move on to some heavy math, consider this interesting (and somewhat philosophical) question: does life have a discount factor? Why or why not?

```
life_discount = 'of course it does. in everything we do we consider & discount future rewards.' #@param {type:"string"}
```

## Section 2.3: Bellman Equations

```
#@title Video : V, Q, and the Bellman Equation
try: t2;
except NameError: t2=time.time()

video = YouTubeVideo(id="tm39P5jT320", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=tm39P5jT320

We define the value of a state $s$ under policy $\pi$ as the expected future reward for following $\pi$ starting from $s$: $$V^{\pi}(s) = E_{\pi} \left[\sum_{t'=t}^{\infty} \gamma^{t'-t}R(s_{t'}, a_{t'}) \mid s_t = s\right] $$

We further define the value of a state-action pair $(s, a)$ under policy $\pi$ as the expected future reward for taking action $a$ at state $s$, *and then* following $\pi$. This is also known as the Q-value. $$Q^{\pi}(s, a) = E_{\pi} \left[\sum_{t'=t}^{\infty} \gamma^{t'-t}R(s_{t'}, a_{t'}) \mid s_t = s, a_t = a\right]$$

Observe that $V$ and $Q$ can be related by a simple equation:
$$V^{\pi}(s) = E_{a\sim \pi(a|s)}\left[Q^{\pi}(s, a)\right]$$

By definition, $V$ and $Q$ satisfy the following Bellman equations.

\begin{align*}
V^{\pi}(s) &= E_{a \sim \pi(a|s)}\left[R(s, a)+ \gamma E_{s' \sim P(s'|s, a)} \left[V^{\pi}(s')\right]\right] \\
Q^{\pi}(s, a) &= R(s, a) + \gamma E_{s' \sim P(s'|s, a)}\left[ E_{a' \sim \pi(a'|s')}\left[Q(s', a')\right]\right]
\end{align*}

The optimal value function capture the expected future reward if we start from state $s$ and act optimally in the future. Similarly, the optimal Q-function captures the expected future reward if we start from state $s$, take action $a$, and then act optimally in the future. They satisfy the Bellman optimality equations:

\begin{align*}
V^*(s) &= \max_{a\in A}\left(R(s, a) + \gamma E_{s' \sim P(s'|s, a)} \left[V^*(s')\right]\right)\\
Q^*(s, a) &= R(s, a) + \gamma E_{s' \sim P(s'|s, a)} \left[ \max_{a' \in A} Q^*(s', a')\right]
\end{align*}

If we have learned the optimal value function $V^*$ or Q-function $Q^*$, we can infer an optimal (deterministic) policy known as the greedy policy or argmax policy: $$\pi(s) = \arg\max_{a\in A}Q^*(s, a)$$

---
# Section 3: Gridworld

Before we dive into RL algorithms, let's get familiar with the running example we will use throughout this tutorial -- the Gridworld environment. As its name suggests, the Gridworld environment is an $m \times n$ grid. The states are x-y coordinates in the grid, with origin at the top-left corner. The agent starts from the initial state and aims to reach the terminal state. There are four actions: up, left, down, and right. Each action leads to a **deterministic** transition to the adjacent cell in the correspond direction. By default, a reward of -1 is issued for entering any non-terminal state, although our implementation allows you to define an arbitrary reward for each state.

To be more specific, our implementation admits four special cells: 'S' (start), 'T' (terminal), 'C' (cliff), and '#' (block). They are colored blue, green, red, and gray respectively. The agent spawns at the start cell, and aims to reach the terminal cell. If the agent "falls off the cliff," it will get a high penalty (-100) and be sent back to the start cell. If the agent tries to enter a block cell or go out of the grid, it will instead stay at the same place and get a reward of -1.

Familiarize yourself with the environment's interface by interacting with the following code cells.

```
# Get a pre-defined grid
gw = get_cliff_small()

# Render rewards
gw.render_grid()

# Render random values
values = np.random.rand(4, 12)
gw.render_values(values)

# Render random Q-values and argmax policy
q_values = np.random.randn(4, 12, 4)
gw.render_q_values(q_values)

# Render random policy
policy = np.random.choice(4, (4, 12)).astype(int)
gw.render_policy(policy)
```

In our Gridworld environment, states are represented by a tuple (x, y), and actions are encoded as 0, 1, 2, 3, corresponding to up, left, down, right. `reset()` resets the agent to its initial state and returns the initial state. `step(action)` executes an action in the environment. It returns the agent's next state, the reward, and a boolean value indicating whether or not the terminal state is reached. In the following cell, control the agent to reach the terminal state.

```
action_space = ['up', 'left', 'down', 'right']
def gw_step(gw, a):
    next_state, reward, done = gw.step(a)
    print(f'You moved {action_space[a]} to {next_state}, reward: {reward}, terminal state reached: {done}')

print(f"Initial state: {gw.reset()}") # reset to initial state
gw_step(gw, 0) # move up
gw_step(gw, 2) # move down
gw_step(gw, 3)
gw_step(gw, 3)
gw_step(gw, 3)
gw_step(gw, 3)

# Use gw_step() to reach the terminal state.

```

    Initial state: (0, 1)
    You moved up to (0, 0), reward: -1.0, terminal state reached: False
    You moved down to (0, 1), reward: -1.0, terminal state reached: False
    You moved right to (1, 1), reward: -1.0, terminal state reached: False
    You moved right to (2, 1), reward: -1.0, terminal state reached: False
    You moved right to (3, 1), reward: -1.0, terminal state reached: False
    You moved right to (4, 1), reward: -1.0, terminal state reached: True

A useful method of the `Gridworld` class is `get_transition(state, action)`. It takes in a state and an action and returns the next state and the reward. We will use this function for exercises 1-3 where we assume full knowledge of the environment's transitions. In a reinforcement learning setting, we only have access to `step(action)`.

```
# Show next state and reward for each action at state (0, 1)
print(gw.get_transition((0, 1), 0))
print(gw.get_transition((0, 1), 1))
print(gw.get_transition((0, 1), 2))
print(gw.get_transition((0, 1), 3))
```

    ((0, 0), -1.0)
    ((0, 1), -1.0)
    ((0, 1), -100.0)
    ((1, 1), -1.0)

---
# Section 4 Dynamic Programming

```
#@title Video : Policy and Value Iteration
try: t3;
except NameError: t3=time.time()

video = YouTubeVideo(id="l87rgLg90HI", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=l87rgLg90HI

## Section 4.1: Policy Iteration

If we have full knowledge of the environment, in particular its transitions, we can use dynamic programming to find the optimal policy. The first algorithm we will study is policy iteration. We start with policy evaluation, which computes the value function of the policy using the Bellman equation. We iteratively perform Bellman backup for the value of each state until convergence:
$$V(s) \leftarrow \sum_{a} \pi(a|s) \left(R(s, a) + \gamma\sum_{s'}P(s'|s, a)V(s')\right) $$
Since we have deterministic transitions, this simplifies to
$$V(s) \leftarrow \sum_{a} \pi(a|s) \left(R(s, a) + \gamma V(s')\right)$$
where $s'$ is the state we transition to by taking action $a$ at state $s$.

In the following excercise, you will evaluate a random policy which assigns equal probablities to all actions at each state. Complete one step of Bellman backup. You can get the next state and reward using `grid.get_transition((x, y), action)`.

### Exercise 1

```
# Random Policy evaluation
def random_policy_evaluation(grid, gamma=1.0):
    values = np.zeros_like(grid.rew_grid)
    iter = 0
    while True:
        eps = 0
        for y in range(grid.h):
            for x in range(grid.w):
                v = values[y, x]
                new_v = 0
                for action in range(grid.n_actions):
                     ###########################################################
                     # Fill in missing code below (...),
                     # then remove or comment the line below to test your function
                    #  raise NotImplementedError("Random policy evaluation")
                     ###########################################################
                     (new_x, new_y), reward = grid.get_transition((x, y), action)
                     new_v += 0.25 * (reward + gamma * values[new_y, new_x])

                values[y, x] = new_v
                eps = max(eps, abs(new_v - v))
        iter += 1
        if eps < 0.0001:
            print("Converged after {} iterations".format(iter))
            break
    return values

# # Uncomment to test
grid = get_book_grid()
values = random_policy_evaluation(grid)
grid.render_values(values)
```

[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W11_DeepRL/solutions/W11_Tutorial1_Solution_Ex01.py)

*Example output:*

```python
Converged after 114 iterations
```

Now we move on to the policy iteration algorithm. Policy iteration consists of two steps: policy evaluation and policy improvement. We first evaluate the policy, and then use the new values to derive a better policy by selecting the greedy action at each state. These steps are repeated until convergence. For an analysis of the theoretical guarantees of policy iteration, see [this page](http://incompleteideas.net/book/first/ebook/node42.html).

In the following exercise, you will implement the policy iteration algorithm. For policy evaluation, note that we have a deterministic greedy policy, so there's no need to iterate over actions. The backup thus becomes $V(s) \leftarrow R(s, \pi(s)) + \gamma V(s')$. For policy improvement, we do the same evaluation for all actions and store them in the action_values array, from which we derive the greedy policy. **Be careful when indexing into the value matrix**: values[y, x] stores the value of state (x, y).

### Exercise 2

```
# Policy Iteration
def policy_evaluation(grid, values, policy, gamma):
    while True:
        eps = 0
        for y in range(grid.h):
            for x in range(grid.w):
                v = values[y, x]
                ################################################################
                # Fill in missing code below (...),
                # then remove or comment the line below to test your function
                # raise NotImplementedError("Policy evaluation")
                ################################################################
                (new_x, new_y), reward = grid.get_transition((x, y), policy[y, x])
                new_v = reward + gamma * values[new_y, new_x]

                values[y, x] = new_v
                eps = max(eps, abs(new_v - v))
        if eps < 0.0001:
            break

def policy_improvement(grid, values, policy, gamma):
    converged = True
    for y in range(grid.h):
        for x in range(grid.w):
            old_action = policy[y, x]
            action_values = np.zeros(grid.n_actions, dtype=np.float)
            ####################################################################
            # Fill in missing code below (...),
            # then remove or comment the line below to test your function
            # raise NotImplementedError("Policy improvement")
            ####################################################################
            for action in range(grid.n_actions):
                (new_x, new_y), reward = grid.get_transition((x, y), action)
                action_values[action] = reward + gamma * values[new_y, new_x]
            policy[y, x] = np.argmax(action_values)

            if old_action != policy[y, x]:
                converged = False
    return converged

def policy_iteration(grid, gamma=1.0):
    policy = np.random.choice(grid.n_actions, (grid.h, grid.w)).astype(int)
    values = np.zeros_like(grid.rew_grid)
    converged = False
    while not converged:
        print("running policy evaluation")
        policy_evaluation(grid, values, policy, gamma)
        print("running policy improvement")
        converged = policy_improvement(grid, values, policy, gamma)
    return values, policy

# # Uncomment to test
grid = get_book_grid()
values, policy = policy_iteration(grid)
grid.render_values(values)
grid.render_policy(policy)
```

[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W11_DeepRL/solutions/W11_Tutorial1_Solution_Ex02.py)

*Example output:*

```python
running policy evaluation
running policy improvement
...
```

Construct the path from the policy visualization and see that following the policy from the initial state indeed leads to terminal state. Now change $\gamma$ to 1.0 and rerun the code. Does policy iteration still converge? Why are we stuck on policy evaluation? (This is a brain-teaser, so don't spend too much time on it, and don't let the code run for too long.)

```
convergence = "policy iteration doesn't converge as we get stuck at the first policy eval step" #@param {type:"string"}
```

[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W11_DeepRL/solutions/convergence.md)

## Section 4.2: Value Iteration

Value iteration can be thought of as a simplification of policy iteration, where we effectively combine the two steps in policy iteration into one. We still iterate over all states, but in each iteration the value update becomes $$V(s) \leftarrow \max_a R(s, a) + \gamma\sum_{s'}P(s'|s, a)V(s') $$ So instead of computing the state value and then selecting the greedy action, we directly store the maximum state-action value. This obviates the need to maintain an explicit policy. After the value matrix has converged, we can back out the optimal policy by taking the argmax, same as what we did in policy improvement.

Now it's your turn to implement the value iteration algorithm. You need to fill in the new update rule, and copy your code from policy improvment to reconstruct the optimal policy.

### Exercise 3

```
# Value Iteration

def value_iteration(grid, gamma=0.9):
    V = np.zeros_like(grid.rew_grid)
    while True:
        eps = 0
        for y in range(grid.h):
            for x in range(grid.w):
                v = values[y, x]
                action_values = np.zeros(grid.n_actions)
                ################################################################
                # Fill in missing code below (...),
                # then remove or comment the line below to test your function
                raise NotImplementedError("Value iteration")
                ################################################################
                for action in range(...):
                    action_values[action] = ...
                new_v = ...

                values[y, x] = new_v
                eps = max(eps, abs(new_v - v))
        if eps < 0.0001:
            break

    # Create greedy policy from values
    policy = np.zeros_like(grid.rew_grid).astype(int)
    for y in range(grid.h):
        for x in range(grid.w):
            action_values = np.zeros(grid.n_actions)
            ####################################################################
            # Copy your solution for policy improvement here
            raise NotImplementedError("Value iteration policy")
            ####################################################################
            for action in range(...):
                action_values[action] = ...
            policy[y, x] = ...

    return values, policy

# # Uncomment to test
# grid = get_book_grid()
# values, policy = value_iteration(grid)
# grid.render_values(values)
# grid.render_policy(policy)
```

[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W11_DeepRL/solutions/W11_Tutorial1_Solution_Ex03.py)

*Example output:*

---
# Section 5: Temporal Difference (TD) Learning

```
#@title Video : TD and Q Learning
try: t4;
except NameError: t4=time.time()

video = YouTubeVideo(id="rCk_hvwZ6iA", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=rCk_hvwZ6iA

## Section 5.1 Q-learning

Up until now we have assumed full access to the transitions of an environment. But in a typical reinforcement learning problem the dynamics is unknown. So how do we solve it? One way is to learn to approximate the dynamics using a function approximator (e.g. a neural net) and then apply dynamic programming or trajectory optimization. This is called model-based reinforcement learning, which we will cover next week. In this tutorial, we will study algorithms in the model-free regime. Specifically, we will investigate **Temporal Difference (TD) learning**.

The idea behind TD learning is to use $V(s_{t+1})$ as an imperfect proxy for the true value (Monte Carlo bootstrapping), and obtain a generalized equation to calculate the TD error:
$$\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$$

The expression $r_{t+1} + \gamma V(s_{t+1})$ is also called the TD target. We can then update the value using a learning rate $\alpha$.
$$ V(s_t) \leftarrow V(s_t) + \alpha \delta_t$$

**Q-learning** is an instantiation of TD learning, where the TD error is $$\delta_t = R(s_t, a_t) + \gamma \max_{a} Q(s_{t+1}, a_{t+1}) - Q(s_t, a_t)$$ and the full update rule is $$Q(s_t,a_t) \leftarrow Q(s_t, a_t) + \alpha \left(R(s_t, a_t) + \gamma \max_{a} Q(s_{t+1}, a_{t+1}) - Q(s_t, a_t)\right)$$

Because of the max operator used to select the optimal Q-value in the TD target, Q-learning directly estimates the optimal action value, i.e. the cumulative future reward that would be obtained if the agent behaved optimally, regardless of the policy currently followed by the agent. For this reason, Q-learning is referred to as an **off-policy** method. A sketch of the Q-learning algorithm is as follows:

```
for n episodes:
    for T steps:
        Select an action a_t using some policy derived from the current Q-values
        Execute a_t in the environment to get reward r and next state s_{t+1}
        Update Q(s_t, a_t) using (s_t, a_t, r, s_{t+1})
```

A remaining question is, how do we select an action base on the current Q-values? If the approximated Q-values are very bad, then greedily following the argmax policy may cause the agent to get stuck in some bad states. Thus, we instead adopt an **epsilon-greedy policy**, where we choose the argmax action with probability $(1-\epsilon)$ and take a random action otherwise. This relates to an important concept in reinforcement learning, namely exploration vs. exploitation.

```
# Epsilon-greedy policy
def epsilon_greedy(q_values, epsilon):
    if np.random.random() > epsilon:
        action = np.argmax(q_values)
    else:
        action = np.random.choice(len(q_values))
    return action

# General TD learning algorithm
def learn_gridworld(env, backup_rule, params, max_steps, n_episodes):
    values = np.zeros((env.h, env.w, env.n_actions))
    episode_actions = []
    episode_rewards = np.zeros(n_episodes)

    for episode in tqdm(range(n_episodes)):
        env.reset()
        total_reward = 0
        action_list = []

        for t in range(max_steps):
            state = env.state
            # Select action from epsilon-greedy policy
            action = epsilon_greedy(values[state[1], state[0]], params['epsilon'])
            action_list.append(action)
            # Execute action
            next_state, reward, done = env.step(action)
            # Update values
            values = backup_rule(state, action, reward, next_state, values, params)
            total_reward += reward
            if done:
                break

        episode_actions.append(action_list)
        episode_rewards[episode] = total_reward

    return values, episode_rewards
```

### Exercise 4

In this exercise, you will implement the update rule for Q-learning and test it on the Cliff World environment, where the agent needs to navigate to the other side of the cliff without falling off. You need to fill in the code for computing the TD error and updating the values matrix.

```
# Q-Learning
def q_learning_backup(state, action, reward, next_state, values, params):
    '''
    Compute a new set of q-values using the q-learning update rule.
    Args:
        state (tuple): s_t, a tuple of xy coordinates.
        action (int): a_t, an integer from {0, 1, 2, 3}.
        reward (float): the reward of executing a_t at s_t.
        next_state (tuple): s_{t+1}, a tuple of xy coordinates.
        values (ndarray): an (h, w, 4) numpy array of q-values. values[y, x, a]
                          stores the value of executing action a at state (x, y).
        params (dict): a dictionary of parameters.

    Returns:
        ndarray: the updated q-values.
    '''
    x, y = state
    nx, ny = next_state
    gamma = params['gamma']
    alpha = params['alpha']

    ####################################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your function
    raise NotImplementedError("Q-learning")
    ####################################################################
    q = ...
    max_next_q = ...

    # Compute TD error using q and max_next_q
    td_error = ...
    values[y, x, action] = ...

    return values

# # Uncomment to test
# env = get_cliff_walk()
# params = {'gamma': 1.0, 'alpha': 0.1 , 'epsilon': 0.1}
# max_steps = 1000
# n_episodes = 500
# q_values, episode_rewards = learn_gridworld(env, q_learning_backup, params, max_steps, n_episodes)

# plot_episode_rewards(episode_rewards)
# env.render_policy(np.argmax(q_values, axis=2))
# env.render_q_values(q_values)
```

[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W11_DeepRL/solutions/W11_Tutorial1_Solution_Ex04.py)

*Example output:*

## Section 5.2: SARSA

An alternative to Q-learning, the SARSA algorithm also estimates action values. However, rather than estimating the optimal (off-policy) values, SARSA estimates the **on-policy** action values, i.e. the cumulative future reward that would be obtained if the agent behaved according to its current beliefs.

\begin{align}
Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \big(R(s_t, a_t) + \gamma Q(s_{t+1}, \pi(s_{t+1})) - Q(s_t,a_t)\big)
\end{align}

In fact, you will notices that the *only* difference between Q-learning and SARSA is the TD target calculation uses the policy to select the next action (in our case epsilon-greedy) rather than using the action that maximizes the Q-value. You do not need to implement the SARSA algorithm. Run the following code cell and compare with Q-learning.

```
# SARSA
def sarsa_backup(state, action, reward, next_state, values, params):
    '''
    Compute a new set of q-values using the SARSA update rule.
    Args:
        state (tuple): s_t, a tuple of xy coordinates.
        action (int): a_t, an integer from {0, 1, 2, 3}.
        reward (float): the reward of executing a_t at s_t.
        next_state (tuple): s_{t+1}, a tuple of xy coordinates.
        values (ndarray): an (h, w, 4) numpy array of q-values. values[y, x, a]
                          stores the value of executing action a at state (x, y).
        params (dict): a dictionary of parameters.

    Returns:
        ndarray: the updated q-values.
    '''
    x, y = state
    nx, ny = next_state
    gamma = params['gamma']
    alpha = params['alpha']

    q = values[y, x, action]
    # Obtain on-policy action
    policy_action = epsilon_greedy(values[ny, nx], params['epsilon'])
    next_q = values[ny, nx, policy_action]

    # Compute TD error using q and max_next_q
    td_error = reward + (gamma * next_q - q)
    values[y, x, action] = q + alpha * td_error

    return values

env = get_cliff_walk()
params = {'gamma': 1.0, 'alpha': 0.1 , 'epsilon': 0.1}
max_steps = 1000
n_episodes = 500
q_values, episode_rewards = learn_gridworld(env, sarsa_backup, params, max_steps, n_episodes)

plot_episode_rewards(episode_rewards)
env.render_policy(np.argmax(q_values, axis=2))
env.render_q_values(q_values)
```

Compare the reward plots and policies of Q-learning and SARSA. Do they take the same path to reach the terminal state? Why does one look more conservative than the other?

```
q_vs_sarsa = 'they do not take the same path, the SARSA plot looks more conservative than Q-learning likely due to its TD target calculation' #@param {type:"string"}
```

[*Click for solution*](https://github.com/CIS-522/course-content/blob/main/tutorials/W11_DeepRL/solutions/q_vs_sarsa.md)

## Section 5.3 (Optional): Try your own grid

If time allows, feel free to try Q-learning or SARSA on one of the other pre-defined grids or a Gridworld of your own creation. Discuss your findings with your pod.

---
# Wrap-up and foreshadowing

```
#@title Video : Wrap-up
try: t5;
except NameError: t5=time.time()

video = YouTubeVideo(id="oJo0jb_h2sM", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=oJo0jb_h2sM

```
import time
import numpy as np
import urllib.parse
from IPython.display import IFrame

#@markdown #Run Cell to Show Airtable Form
#@markdown ##**Confirm your answers and then click "Submit"**

def prefill_form(src, fields: dict):
  '''
  src: the original src url to embed the form
  fields: a dictionary of field:value pairs,
  e.g. {"pennkey": my_pennkey, "location": my_location}
  '''
  prefill_fields = {}
  for key in fields:
      new_key = 'prefill_' + key
      prefill_fields[new_key] = fields[key]
  prefills = urllib.parse.urlencode(prefill_fields)
  src = src + prefills
  return src

#autofill time if it is not present
try: t0;
except NameError: t0 = time.time()
try: t1;
except NameError: t1 = time.time()
try: t2;
except NameError: t2 = time.time()
try: t3;
except NameError: t3 = time.time()
try: t4;
except NameError: t4 = time.time()
try: t5;
except NameError: t5 = time.time()
try: t6;
except NameError: t6 = time.time()

#autofill fields if they are not present
#a missing pennkey and pod will result in an Airtable warning
#which is easily fixed user-side.
try: my_pennkey;
except NameError: my_pennkey = ""
try: my_pod;
except NameError: my_pod = "Select"
try: learning_from_previous_week;
except NameError: learning_from_previous_week = ""
try: MDP_example;
except NameError: MDP_example = ""
try: life_discount;
except NameError: life_discount = ""
try: convergence;
except NameError: convergence = ""
try: q_vs_sarsa;
except NameError: q_vs_sarsa = ""

times = np.array([t1,t2,t3,t4,t5,t6])-t0

fields = {"pennkey": my_pennkey,
          "pod": my_pod,
          "learning_from_previous_week": learning_from_previous_week,
          "MDP_example": MDP_example,
          "life_discount": life_discount,
          "convergence": convergence,
          "q_vs_sarsa": q_vs_sarsa,
          "cumulative_times": times}

src = "https://airtable.com/embed/shrS0Ltpj30NO4Fr8?"

# now instead of the original source url, we do: src = prefill_form(src, fields)
display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))
```

## Feedback
How could this session have been better? How happy are you in your group? How do you feel right now?

Feel free to use the embeded form below or use this link:
https://airtable.com/shrNSJ5ECXhNhsYss

```
display(IFrame(src="https://airtable.com/embed/shrNSJ5ECXhNhsYss?backgroundColor=red", width = 800, height = 400))
```


# Meta

Title: Optimise Blending Weights with Bonus

Author: Yirun Zhang

[Source code](https://www.kaggle.com/gogo827jz/optimise-blending-weights-with-bonus-0)

# Model Blending Weights Optimisation

This demo shows how to use [scipy.optimize][1] to optimise your model blending weights using your models' OOFs.

**UPDATE:** Getting rid of the penalty term by using 'SLSQP' solver with a relatively small tolerance and Jacobian matrix.

**UPDATE:** Calculate the gradients with paper and pencil to accelerate the optimisation...

**UPDATE:** Add numba gradient function

[1]: https://docs.scipy.org/doc/scipy/reference/optimize.html

```python
# !pip install autograd --quiet
```

```python
import datetime
import pandas as pd
from time import time
# from autograd import grad
# import autograd.numpy as np
import numpy as np
from numba import njit
from scipy.optimize import minimize, fsolve
```

# Objective Function and Gradients

$$
F = -\frac{1}{NM}\sum_{m=1}^{M}\sum_{i=1}^{N}\left[ y_{i,m}{\rm log}\left( \sum_{k=1}^{K}w_{k}\hat{y}_{i,m,k} \right) + \left( 1 - y_{i,m} \right) {\rm log}\left( 1 - \sum_{k=1}^{K}w_{k}\hat{y}_{i,m,k} \right) \right],
$$

$$
\frac{\partial F}{\partial w_{k}} = -\frac{1}{NM}\sum_{m=1}^{M}\sum_{i=1}^{N}\left[ \frac{-y_{i,m}\hat{y}_{i,m,k}+\hat{y}_{i,m,k}^{2}w_{k}+\hat{y}_{i,m,k}\sum_{j=1, j\neq k}^{K}\left( w_{j}\hat{y}_{i,m,j}\right)}{\hat{y}_{i,m,k}^{2}w_{k}^{2}+2\hat{y}_{i,m,k}\sum_{j=1, j\neq k}^{K}\left( w_{j}\hat{y}_{i,m,j}\right)w_{k}-\hat{y}_{i,m,k}w_{k}+\left(\sum_{j=1, j\neq k}^{K}\left( w_{j}\hat{y}_{i,m,j}\right)\right)^{2}-\sum_{j=1, j\neq k}^{K}\left( w_{j}\hat{y}_{i,m,j}\right)} \right], \quad k = 1, ..., K.
$$

```python
# CPMP's logloss from https://www.kaggle.com/c/lish-moa/discussion/183010
def log_loss_numpy(y_pred):
    y_true_ravel = np.asarray(y_true).ravel()
    y_pred = np.asarray(y_pred).ravel()
    y_pred = np.clip(y_pred, 1e-15, 1 - 1e-15)
    loss = np.where(y_true_ravel == 1, - np.log(y_pred), - np.log(1 - y_pred))
    return loss.mean()

def func_numpy_metric(weights):
    oof_blend = np.tensordot(weights, oof, axes = ((0), (0)))
    return log_loss_numpy(oof_blend)

def grad_func(weights):
    oof_clip = np.clip(oof, 1e-15, 1 - 1e-15)
    gradients = np.zeros(oof.shape[0])
    for i in range(oof.shape[0]):
        a, b, c = y_true, oof_clip[i], np.zeros((oof.shape[1], oof.shape[2]))
        for j in range(oof.shape[0]):
            if j != i:
                c += weights[j] * oof_clip[j]
        gradients[i] = -np.mean((-a*b+(b**2)*weights[i]+b*c)/((b**2)*(weights[i]**2)+2*b*c*weights[i]-b*weights[i]+(c**2)-c))
    return gradients

@njit
def grad_func_jit(weights):
    oof_clip = np.minimum(1 - 1e-15, np.maximum(oof, 1e-15))
    gradients = np.zeros(oof.shape[0])
    for i in range(oof.shape[0]):
        a, b, c = y_true, oof_clip[i], np.zeros((oof.shape[1], oof.shape[2]))
        for j in range(oof.shape[0]):
            if j != i:
                c += weights[j] * oof_clip[j]
        gradients[i] = -np.mean((-a*b+(b**2)*weights[i]+b*c)/((b**2)*(weights[i]**2)+2*b*c*weights[i]-b*weights[i]+(c**2)-c))
    return gradients
```

# Model OOF Scores

Here are my oof scores. You may use your own oof scores.

```python
y_true = pd.read_csv('../input/lish-moa/train_targets_scored.csv', index_col = 'sig_id').values

oof_dict = {'Model 1': '../input/moa-oof-demo/oof1.npy',
            'Model 2': '../input/moa-oof-demo/oof2.npy',
            'Model 3': '../input/moa-oof-demo/oof3.npy'
           }

oof = np.zeros((len(oof_dict), y_true.shape[0], y_true.shape[1]))
for i in range(oof.shape[0]):
    oof[i] = np.load(list(oof_dict.values())[i])
```

```python
%%time

log_loss_scores = {}
for n, key in enumerate(oof_dict.keys()):
    score_oof = log_loss_numpy(oof[n])
    log_loss_scores[key] = score_oof
    print(f'{key} CV:\t', score_oof)
print('-' * 50)
```

    Model 1 CV:	 0.015347833625106137
    Model 2 CV:	 0.015271737693837504
    Model 3 CV:	 0.015401898519997256
    --------------------------------------------------
    CPU times: user 654 ms, sys: 181 ms, total: 834 ms
    Wall time: 834 ms

# Test Numba Gradient Function

```python
test_weights = np.array([1 / oof.shape[0]] * oof.shape[0])
```

```python
%timeit -r 10 grad_func(test_weights)
```

    791 ms ± 19.2 ms per loop (mean ± std. dev. of 10 runs, 1 loop each)

```python
%timeit -r 10 grad_func_jit(test_weights)
```

    539 ms ± 9.21 ms per loop (mean ± std. dev. of 10 runs, 1 loop each)

# Blending Weights Optimisation

Providing jac is optional because scipy uses its own 2-point finite difference estimation for the Jacobian matrix.

```python
tol = 1e-10
init_guess = [1 / oof.shape[0]] * oof.shape[0]
bnds = [(0, 1) for _ in range(oof.shape[0])]
cons = {'type': 'eq',
        'fun': lambda x: np.sum(x) - 1,
        'jac': lambda x: [1] * len(x)}

print('Inital Blend OOF:', func_numpy_metric(init_guess))
start_time = time()
res_scipy = minimize(fun = func_numpy_metric,
                     x0 = init_guess,
                     method = 'SLSQP',
                     jac = grad_func_jit, # grad_func
                     bounds = bnds,
                     constraints = cons,
                     tol = tol)
print(f'[{str(datetime.timedelta(seconds = time() - start_time))[2:7]}] Optimised Blend OOF:', res_scipy.fun)
print('Optimised Weights:', res_scipy.x)
```

    Inital Blend OOF: 0.015035763097305546
    [00:15] Optimised Blend OOF: 0.015019016150854788
    Optimised Weights: [0.38920215 0.54298912 0.06780872]

```python
print('Check the sum of all weights:', np.sum(res_scipy.x))
if np.sum(res_scipy.x) - 1 <= tol:
    print('Great! The sum of all weights equals to 1!')
else:
    print('Manual adjustion is needed to modify the weights.')
```

    Check the sum of all weights: 1.0
    Great! The sum of all weights equals to 1!

# Bonus (Lagrange Multiplier)

Congratulations! You have found this bonus. In this section, I optimise the blending weights in a more mathematical way using Lagrange Multiplier method. The following equation is the minimisation problem that we want to solve:

$$
\begin{align}
\min_{w_{1}, w_{2},..., w_{K}} \quad &-\frac{1}{NM}\sum_{m=1}^{M}\sum_{i=1}^{N}\left[ y_{i,m}{\rm log}\left( \sum_{k=1}^{K}w_{k}\hat{y}_{i,m,k} \right) + \left( 1 - y_{i,m} \right) {\rm log}\left( 1 - \sum_{k=1}^{K}w_{k}\hat{y}_{i,m,k} \right) \right], \qquad {\rm (1)} \\
s.t. \quad &\sum_{k=1}^{k}w_{k} = 1, \qquad {\rm (1a)} \\
& 0 \leqslant w_{k} \leqslant 1, \quad k = 1, ..., K, \qquad {\rm (1b)}
\end{align}
$$

where $N$ is the number of 'sigid' observations in the test data $(i = 1, ...,N)$;

$M$ is the number of scored MoA targets $(m = 1, ...,M)$;

$w_{k}$ is the blending weight for the $k$th model's prediction results $(k = 1, ...,K)$;

$\hat{y}_{i,m,k}$ is the $k$th model's predicted probability of the $m$th positive MoA response for the $n$th 'sigid';

$y_{i,m}$ is the groundtruth of the $m$th positive MoA response for the $n$th 'sigid', 1 for a positive response, 0 otherwise;

${\rm log}(.)$ is the natural (base e) logarithm.

According to the [Extreme Value Thereom][1], Constraint (1b) indicates Eq. (1) has absolute maximum and minimum values. We apply the [Lagrange Multiplier][2] method to this optimsiation problem. The new optimisation problem is expressed as follows:

$$
\begin{align}
\min_{w_{1}, w_{2},..., w_{K}} \quad &L = -\frac{1}{NM}\sum_{m=1}^{M}\sum_{i=1}^{N}\left[ y_{i,m}{\rm log}\left( \sum_{k=1}^{K}w_{k}\hat{y}_{i,m,k} \right) + \left( 1 - y_{i,m} \right) {\rm log}\left( 1 - \sum_{k=1}^{K}w_{k}\hat{y}_{i,m,k} \right) \right] - \lambda\left(\sum_{k=1}^{K}w_{k} - 1\right), \qquad {\rm (2)} \\
s.t. \quad &0 \leqslant w_{k} \leqslant 1, \quad k = 1, ..., K, \qquad {\rm (2b)}
\end{align}
$$

where $\lambda$ is the Lagrange multiplier.

The [Karush–Kuhn–Tucker condition][3] for the optimal solution is:
$$
\left\{\begin{matrix}
\frac{\partial L}{\partial w_{k}} = 0, & k = 1, ..., K, \\
\frac{\partial L}{\partial \lambda} = 0, &
\end{matrix}\right., \qquad {\rm (3)}
$$

From Eq. (3), we end up with $K+1$ equations that equal zero, we can simply use [autograd][4] to calculate the partial derivatives and [scipy.optimize.fsolve][5] to get the optimal solution.

[1]: https://en.wikipedia.org/wiki/Extreme_value_theorem
[2]: https://en.wikipedia.org/wiki/Lagrange_multiplier
[3]: https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions
[4]: https://github.com/HIPS/autograd
[5]: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fsolve.html

```python
# def Lagrange_func(params):
#     w1, w2, w3, _lambda = params
#     oof_blend = w1 * oof[0] + w2 * oof[1] + w3 * oof[2]
#     return log_loss_numpy(oof_blend) - _lambda * (w1 + w2 + w3 - 1)
```

```python
# grad_L = grad(Lagrange_func)
```

```python
# def Lagrange_obj(params):
#     w1, w2, w3, _lambda = params
#     dLdw1, dLdw2, dLdw3, dLdlam = grad_L(params)
#     return [dLdw1, dLdw2, dLdw3, w1 + w2 + w3 - 1]
```

```python
# start_time = time()
# w1, w2, w3, _lambda = fsolve(Lagrange_obj, [0.3, 0.3, 0.4, 1.0])
# print(f'[{str(datetime.timedelta(seconds = time() - start_time))[2:7]}] Optimised Weights:', [w1, w2, w3])
# oof_b = w1 * oof[0] + w2 * oof[1] + w3 * oof[2]
# print('Optimised Blend OOF:', log_loss_numpy(oof_b))
```

```python
# print('Check Condition (1a):', w1 + w2 + w3)
# if w1 + w2 + w3 - 1 <= tol:
#     print('Great! The sum of all weights equals to 1!')
# else:
#     print('Manual adjustion is needed to modify the weights.')
```


# IDMaxwellNRPy: An Einstein Toolkit Initial Data Thorn for Maxwell's equations

## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

### NRPy+ Source Code for this module: [Maxwell/MaxwellCartesian_ID.py](../edit/Maxwell/MaxwellCartesian_ID.py) [\[tutorial\]](Tutorial-MaxwellCartesian.ipynb) Constructs these Maxwell initial data as SymPy expressions.

## Introduction:
In this part of the tutorial, we will construct an Einstein Toolkit (ETK) thorn (module) that will set up *initial data* for Maxwell's equations. In the [Tutorial-MaxwellCartesian](Tutorial-MaxwellCartesian.ipynb), tutorial notebook we used NRPy+ to contruct the SymPy expressions for plane-wave initial data.

We will construct this thorn in two steps.

1. Call on NRPy+ to convert the SymPy expressions for the initial data into one C-code kernel.
2. Write the C code and linkages to the Einstein Toolkit infrastructure (i.e., the .ccl files) to complete this Einstein Toolkit module.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Call on NRPy+ to convert the SymPy expression for the Maxwell's equations initial data into a C-code kernel
1. [Step 2](#einstein): Interfacing with the Einstein Toolkit
    1. [Step 2.a](#einstein_c): Constructing the Einstein Toolkit C-code calling functions that include the C code kernels
    1. [Step 2.b](#einstein_ccl): CCL files - Define how this module interacts and interfaces with the larger Einstein Toolkit infrastructure
    1. [Step 2.c](#einstein_list): Add the C code to the Einstein Toolkit compilation list
1. [Step 3](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF

# Step 1: Call on NRPy+ to convert the SymPy expression for the Maxwell's equations initial data into a C-code kernel \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

After importing the core modules, since we are writing an ETK thorn, we'll need to set `"grid::GridFuncMemAccess"` to `"ETK"`. SymPy expressions for plane wave initial data are written inside [Maxwell/MaxwellCartesian_ID.py](../edit/Maxwell/MaxwellCartesian_ID.py), and we simply import them for use here.

```python
# Step 1: Call on NRPy+ to convert the SymPy expression
#         for the Maxwell's equations initial data into a C-code kernel

# Step 1a: Import needed NRPy+ core modules:
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import grid as gri               # NRPy+: Functions having to do with numerical grids
import loop as lp                # NRPy+: Generate C code loops
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
from outputC import lhrh         # NRPy+: Core C code output module
import finite_difference as fin  # NRPy+: Finite difference C code generation module

# Step 1b: This is an Einstein Toolkit (ETK) thorn. Here we
#          tell NRPy+ that gridfunction memory access will
#          therefore be in the "ETK" style.
par.set_parval_from_str("grid::GridFuncMemAccess","ETK")
#Set the spatial dimension parameter to 3.
par.set_parval_from_str("grid::DIM", 3)
DIM = par.parval_from_str("grid::DIM")

import Maxwell.MaxwellCartesian_Evol as mwrhs # This declare the parameter System_to_use needed for MaxwellCartesian_ID
# Step 1c: Call the MaxwellCartesian_ID() function from within the
#          Maxwell/MaxwellCartesian_ID.py.py module.
import Maxwell.MaxwellCartesian_ID as mwid
# Step 3: Set up the plane wave initial data. This sets uu_ID and vv_ID.
mwid.MaxwellCartesian_ID()

# Step 4: Register gridfunctions so they can be written to by NRPy.
# System I:
AID = ixp.register_gridfunctions_for_single_rank1("EVOL","AID")
EID = ixp.register_gridfunctions_for_single_rank1("EVOL","EID")
psiI = gri.register_gridfunctions("EVOL","psiI")

# Step 5: FIXME: ADD DESCRIPTION
for i in range(DIM):
    AID[i] = mwid.AidD[i]
    EID[i] = mwid.EidD[i]
psiI = mwid.psi_ID

# Step 4: Register gridfunctions so they can be written to by NRPy.
# System II:
AIID = ixp.register_gridfunctions_for_single_rank1("EVOL","AIID")
EIID = ixp.register_gridfunctions_for_single_rank1("EVOL","EIID")
psiII = gri.register_gridfunctions("EVOL","psiII")
Gamma = gri.register_gridfunctions("EVOL","Gamma")

# Step 5: FIXME: ADD DESCRIPTION
for i in range(DIM):
    AIID[i] = mwid.AidD[i]
    EIID[i] = mwid.EidD[i]
psiII = mwid.psi_ID
Gamma = mwid.Gamma_ID

# Step 6: Create the C code output kernel.
Maxwell_ID_to_print = [\
                        lhrh(lhs=gri.gfaccess("out_gfs","AID0"),rhs=AID[0]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","AID1"),rhs=AID[1]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","AID2"),rhs=AID[2]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","EID0"),rhs=EID[0]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","EID1"),rhs=EID[1]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","EID2"),rhs=EID[2]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","psiI"),rhs=psiI),\
                        lhrh(lhs=gri.gfaccess("out_gfs","AIID0"),rhs=AIID[0]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","AIID1"),rhs=AIID[1]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","AIID2"),rhs=AIID[2]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","EIID0"),rhs=EIID[0]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","EIID1"),rhs=EIID[1]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","EIID2"),rhs=EIID[2]),\
                        lhrh(lhs=gri.gfaccess("out_gfs","psiII"),rhs=psiII),\
                        lhrh(lhs=gri.gfaccess("out_gfs","Gamma"),rhs=Gamma),]
Maxwell_ID_CcodeKernel = fin.FD_outputC("returnstring",Maxwell_ID_to_print)

Maxwell_ID_looped = lp.loop(["i2","i1","i0"],["0","0","0"],["cctk_lsh[2]","cctk_lsh[1]","cctk_lsh[0]"],\
                            ["1","1","1"],["#pragma omp parallel for","",""],"",\
                            Maxwell_ID_CcodeKernel.replace("time","cctk_time"))

# Step 7: Create directories for the thorn if they don't exist.
!mkdir MaxwellID     2>/dev/null # 2>/dev/null: Don't throw an error if the directory already exists.
!mkdir MaxwellID/src 2>/dev/null # 2>/dev/null: Don't throw an error if the directory already exists.

# Step 8: Write the C code kernel to file.
with open("MaxwellID/src/Maxwell_ID.h", "w") as file:
    file.write(str(Maxwell_ID_looped))

```

# Step 2: Interfacing with the Einstein Toolkit \[Back to [top](#toc)\]
$$\label{einstein}$$

## Step 2.a: Constructing the Einstein Toolkit C-code calling functions that include the C code kernels \[Back to [top](#toc)\]
$$\label{einstein_c}$$

We will write another C file with the functions we need here.

```python
%%writefile MaxwellID/src/InitialData.c
#include <math.h>
#include <stdio.h>

#include "cctk.h"
#include "cctk_Parameters.h"
#include "cctk_Arguments.h"

void Maxwell_InitialData(CCTK_ARGUMENTS)
{
  DECLARE_CCTK_ARGUMENTS
  DECLARE_CCTK_PARAMETERS

  const CCTK_REAL *xGF = x;
  const CCTK_REAL *yGF = y;
  const CCTK_REAL *zGF = z;
  const CCTK_REAL *gammaDD00GF = gxx;
  const CCTK_REAL *gammaDD01GF = gxy;
  const CCTK_REAL *gammaDD02GF = gxz;
  const CCTK_REAL *gammaDD11GF = gyy;
  const CCTK_REAL *gammaDD12GF = gyz;
  const CCTK_REAL *gammaDD22GF = gzz;
#include "Maxwell_ID.h"
}
```

    Overwriting MaxwellID/src/InitialData.c

## Step 2.b: CCL files - Define how this module interacts and interfaces with the larger Einstein Toolkit infrastructure \[Back to [top](#toc)\]
$$\label{einstein_ccl}$$

Writing a module ("thorn") within the Einstein Toolkit requires that three "ccl" files be constructed, all in the root directory of the thorn:

1. `interface.ccl`: defines the gridfunction groups needed, and provides keywords denoting what this thorn provides and what it should inherit from other thorns. Specifically, this file governs the interaction between this thorn and others; more information can be found in the [official Einstein Toolkit documentation](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-178000D2.2).
With "implements", we give our thorn its unique name. By "inheriting" other thorns, we tell the Toolkit that we will rely on variables that exist and are declared "public" within those functions.

```python
%%writefile MaxwellID/interface.ccl
implements: MaxwellID
inherits: admbase MaxwellEvol grid

```

    Overwriting MaxwellID/interface.ccl

2. `param.ccl`: specifies free parameters within the thorn, enabling them to be set at runtime. It is required to provide allowed ranges and default values for each parameter. More information on this file's syntax can be found in the [official Einstein Toolkit documentation](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-183000D2.3).

```python
%%writefile MaxwellID/param.ccl
shares: grid

USES KEYWORD type

restricted:
CCTK_KEYWORD initial_data "Type of initial data"
{
  "toroid"      :: "Toroidal Dipole field"
} "toroid"

restricted:
CCTK_REAL amp "The amplitude of the initial wave packet"
{
 0.0:* :: "Should be positive"
} 1.0

restricted:
CCTK_REAL lam "A parameter describing the size of the wave package"
{
 0.0:* :: "Should be positive"
} 1.0

```

    Overwriting MaxwellID/param.ccl

3. `schedule.ccl`: allocates storage for gridfunctions, defines how the thorn's functions should be scheduled in a broader simulation, and specifies the regions of memory written to or read from gridfunctions. `schedule.ccl`'s official documentation may be found [here](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-186000D2.4).

We specify here the standardized ETK "scheduling bins" in which we want each of our thorn's functions to run.

```python
%%writefile MaxwellID/schedule.ccl

schedule Maxwell_InitialData at CCTK_INITIAL as Maxwell_InitialData
{
  STORAGE:       MaxwellEvol::system_I[3]
  STORAGE:       MaxwellEvol::system_II[3]
  LANG:          C
  READS: admbase::gxx(Everywhere)
  READS: admbase::gxy(Everywhere)
  READS: admbase::gxz(Everywhere)
  READS: admbase::gyy(Everywhere)
  READS: admbase::gyz(Everywhere)
  READS: admbase::gzz(Everywhere)
  READS: grid::x(Everywhere)
  READS: grid::y(Everywhere)
  READS: grid::y(Everywhere)
  WRITES: AI0GF(Everywhere)
  WRITES: AI1GF(Everywhere)
  WRITES: AI2GF(Everywhere)
  WRITES: EI0GF(Everywhere)
  WRITES: EI1GF(Everywhere)
  WRITES: EI2GF(Everywhere)
  WRITES: psiIGF(Everywhere)
  WRITES: AII0GF(Everywhere)
  WRITES: AII1GF(Everywhere)
  WRITES: AII2GF(Everywhere)
  WRITES: EII0GF(Everywhere)
  WRITES: EII1GF(Everywhere)
  WRITES: EII2GF(Everywhere)
  WRITES: psiIIGF(Everywhere)
  WRITES: GammaGF(Everywhere)
} "Initial data for Maxwell's equations"

```

    Overwriting MaxwellID/schedule.ccl

## Step 2.c: Add the C code to the Einstein Toolkit compilation list \[Back to [top](#toc)\]
$$\label{einstein_list}$$

We will also need `make.code.defn`, which indicates the list of files that need to be compiled. This thorn only has the one C file to compile.

```python
%%writefile MaxwellID/src/make.code.defn
SRCS = InitialData.c
```

    Overwriting MaxwellID/src/make.code.defn

# Step 3: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-ETK_thorn-IDMaxwellNRPy.pdf](Tutorial-ETK_thorn-IDMaxwellNRPy.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-ETK_thorn-IDMaxwellNRPy")
```

    Created Tutorial-ETK_thorn-IDMaxwellNRPy.tex, and compiled LaTeX file to
        PDF file Tutorial-ETK_thorn-IDMaxwellNRPy.pdf