# Deep Reinforcement Learning

Reinforcement learning is one of the hottest (if not the hottest) area of deep learning. It is possibly the closest that we have come to achieving general AI at this point in time. The reason it is touted as getting close to general AI is that we can use the same framework repeatedly in different environments to maximise future rewards. Apart from the ATARI games in which [Deep Reinforcement Learning got famous](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf) for and beating the current world GO chamption, it is now being used in finance and [teaching robots to walk](https://www.youtube.com/watch?v=JFJkpVWTQVM).

The general concept of RL is to simulate game a large number of times and to learn which actions constitutes a good move. One of the biggest challenges that RL faces is to infer what constitutes a good move. A good move at the current state might be a bad move for future states. This is definitely the case in playing chess for example where you constantly sacrifice pieces for larger future rewards.

## Belman Equation

The goal of reinforcement learning is to maximise its return. In this tutorial we get 'envirnonments' from OpenAI which provides us a game, the physics/ rules by which the game is controlled by and outputs the **rewards** achieved for a given **action**, at the current **state** of the game.

The bellman equation plays a crucial role in reinforcement learning.
\begin{align}
Q(S_t)(a_t) = r_t + \gamma\max_{a_{t+1}}Q(S_{t+1})(a_{t+1})
\end{align}
where $Q$ is a quality function, which depends on state $S_t$ and outputs the maximum possible future reward for action $a_t$. $\gamma$ is a discount factor. Most texts would prefer to depict the Quality function as $Q(S_t, a_t)$, however I wish to stress that $Q(S_t)$ outputs the maximum discounted future reward for all actions. $a_t$ is in this tutorial, a one-hot encoded variable which helps to choose a reward for all possible actions. **Notice how could recursively expand this function.**

## Pseudocode for DQN
Reference: https://github.com/yenchenlin/DeepLearningFlappyBird
```
Initialize replay memory D to size N
Initialize action-value function Q with random weights
for episode = 1, M do
    Initialize state s_1
    for t = 1, T do
        With probability ϵ select random action a_t
        otherwise select a_t=max_a  Q(s_t,a; θ_i)
        Execute action a_t in emulator and observe r_t and s_(t+1)
        Store transition (s_t,a_t,r_t,s_(t+1)) in D
        Sample a minibatch of transitions (s_j,a_j,r_j,s_(j+1)) from D
        Set y_j:=
            r_j for terminal s_(j+1)
            r_j+γ*max_(a^' )  Q(s_(j+1),a'; θ_i) for non-terminal s_(j+1)
        Perform a gradient step on (y_j-Q(s_j,a_j; θ_i))^2 with respect to θ
    end for
end for
```

## Reference:

- https://keon.io/deep-q-learning/
- http://mckinziebrandon.me/TensorflowNotebooks/2016/12/21/openai.html
- https://github.com/yenchenlin/DeepLearningFlappyBird
- [Playing Atari with Deep Reinforcement Learning](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)
- [OpenAI environments](https://gym.openai.com/envs/)
- https://www.youtube.com/watch?v=JFJkpVWTQVM

```python
!pip install gym
!conda install -y JSAnimation
```

```python
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import re

from keras.models import Sequential, Model
from keras.layers import Dense, Input, Dot
from keras.models import load_model, model_from_json
from keras.optimizers import Adam

import gym

from collections import deque

import time

%matplotlib inline

# Imports specifically so we can render outputs in Jupyter.
from JSAnimation.IPython_display import display_animation
from matplotlib import animation
from IPython.display import display
```

    Using TensorFlow backend.

```python
def create_model(n_states, n_actions):

    state = Input(shape=(n_states,))
    # TODO: Fill in the following with 2 Dense Layers
    # see https://keras.io/getting-started/functional-api-guide/
    x1 =
    x2 =
    out = Dense(n_actions)(x2)
    actions = Input(shape=(n_actions,))
    out2 = Dot(axes=-1)([out, actions])

    model = Model(inputs=[state, actions], outputs=out2)
    model.compile(loss='mse', optimizer='rmsprop')

    model2 = Model(inputs=state, outputs=out)

    return model, model2
```

```python
def train_data(minibatch, model):
    s_j_batch = np.array([d[0] for d in minibatch])
    a_batch = np.array([d[1] for d in minibatch])
    r_batch = np.array([d[2] for d in minibatch])
    s_j1_batch = np.array([d[3] for d in minibatch])
    terminal_batch = np.array([d[4] for d in minibatch])

    ##############################################################
    # The following 3 lines are the most important part of Deep RL
    ##############################################################
    readout_j1_batch = model.predict(s_j1_batch, batch_size=BATCH)
    y_batch = r_batch + GAMMA * np.max(readout_j1_batch, axis=1)
    y_batch[terminal_batch] = r_batch[terminal_batch]

    return s_j_batch, a_batch, y_batch
```

## OpenAI functions

- env = gym.make(environment_name) <- sets up the environment
- env.reset() <- resets the environment to starting point
- env.step(action) <- takes action and goes to state $S_{t+1}$
- env.render() <- render the output

```python
env = gym.make('CartPole-v0')
STATES, ACTIONS = env.observation_space.shape[0], env.action_space.n
model, out = create_model(STATES, ACTIONS)
INITIAL_EPSILON = 1e-1
FINAL_EPSILON = 1e-4
DECAY = 0.9
GAMMA = 0.9 # decay rate of past observations
OBSERVE = 5000. # timesteps to observe before training
REPLAY_MEMORY = 5000 # number of previous transitions to remember
TIME_LIMIT = 100000
BATCH = 128
# open up a game state to communicate with emulator
state = env.reset()

# store the previous observations in replay memory
D = deque(maxlen=REPLAY_MEMORY)
loss = []

a_t = np.zeros(ACTIONS)
a_t[np.random.choice(ACTIONS)] = 1
s_t, r_0, terminal, _ = env.step(np.argmax(a_t))

# start training
epsilon = INITIAL_EPSILON

up_time = [0]
start = time.time()
for t in range(TIME_LIMIT):
    # choose an action
    readout_t = out.predict(s_t[None, :])
    a_t = np.zeros([ACTIONS])
    if np.random.random() <= epsilon:
        a_t[np.random.choice(ACTIONS)] = 1
    else:
        a_t[np.argmax(readout_t)] = 1

    # scale down epsilon
    if epsilon > FINAL_EPSILON and t > OBSERVE:
        # TODO: decay the epsilon value
        epsilon =

    # run the selected action and observe next state and reward
    # store the transition in D
    # TODO: Take a step for the current environment (see lines above this cell)
    s_t1, r_t, terminal, _ =
    D.append((s_t, a_t, r_t, s_t1, terminal))

    if terminal:
        up_time.append(0)
        env.reset()
    else:
        up_time[-1] += r_t

    # only train if done observing
    if t > OBSERVE:
        # sample a minibatch to train on
        idx = np.random.choice(REPLAY_MEMORY, BATCH, replace=False)
        minibatch = [D[i] for i in idx]
        # get the batch variables
        s_t_batch, a_batch, y_batch = train_data(minibatch, out)
        # perform gradient step
        loss.append(model.train_on_batch([s_t_batch, a_batch], y_batch))

    # update the old state
    s_t = s_t1

    if t%(TIME_LIMIT/20)==0:
        print('Episode :', t, ', time taken: ', time.time() - start, 's, average up time: ',np.mean(up_time[-100:]))
        start = time.time()
```

    [2017-10-18 18:24:26,754] Making new env: CartPole-v0

    Episode : 0 , time taken:  0.04702281951904297 s, average up time:  1.0
    Episode : 5000 , time taken:  1.6890149116516113 s, average up time:  8.63
    Episode : 10000 , time taken:  11.951380014419556 s, average up time:  34.5
    Episode : 15000 , time taken:  13.010540962219238 s, average up time:  56.53
    Episode : 20000 , time taken:  11.975058794021606 s, average up time:  25.17
    Episode : 25000 , time taken:  11.041939973831177 s, average up time:  44.6
    Episode : 30000 , time taken:  11.701272964477539 s, average up time:  35.91
    Episode : 35000 , time taken:  11.184820890426636 s, average up time:  30.13
    Episode : 40000 , time taken:  11.266594886779785 s, average up time:  26.49
    Episode : 45000 , time taken:  11.33695912361145 s, average up time:  28.31
    Episode : 50000 , time taken:  11.028918027877808 s, average up time:  60.2
    Episode : 55000 , time taken:  11.333147048950195 s, average up time:  85.6
    Episode : 60000 , time taken:  11.429896831512451 s, average up time:  105.24
    Episode : 65000 , time taken:  10.855633974075317 s, average up time:  129.07
    Episode : 70000 , time taken:  10.938822984695435 s, average up time:  153.11
    Episode : 75000 , time taken:  10.609338998794556 s, average up time:  156.1
    Episode : 80000 , time taken:  10.61500597000122 s, average up time:  149.9
    Episode : 85000 , time taken:  10.69183897972107 s, average up time:  156.1
    Episode : 90000 , time taken:  10.760807037353516 s, average up time:  158.02
    Episode : 95000 , time taken:  11.041198015213013 s, average up time:  144.79

```python
plt.figure(figsize=(12, 5))
plt.plot(up_time)
plt.show()
```

```python
plt.figure(figsize=(12, 5))
plt.plot(loss)
plt.show()
```

**Only run the following if you are running it locally and not in Docker**

```python
def display_frames_as_gif(frames):
    """
    Displays a list of frames as a gif, with controls
    """
    #plt.figure(figsize=(frames[0].shape[1] / 72.0, frames[0].shape[0] / 72.0), dpi = 72)
    patch = plt.imshow(frames[0])
    plt.axis('off')

    def animate(i):
        patch.set_data(frames[i])

    anim = animation.FuncAnimation(plt.gcf(), animate, frames = len(frames), interval=50)
    display(display_animation(anim, default_mode='once'))
```

```python
env = gym.make('CartPole-v0')

# Run a demo of the environment
observation = env.reset()
cum_reward = 0
frames = []
for t in range(5000):
    # Render into buffer.
    frames.append(env.render(mode = 'rgb_array'))
    action = env.action_space.sample()
    observation, reward, done, info = env.step(action)
    if done:
        break
env.render(close=True)
display_frames_as_gif(frames)
print(t)
```

    [2017-10-18 18:28:03,126] Making new env: CartPole-v0

    <input id="_anim_sliderHEJQZZPHOHXHIASP" type="range" style="width:350px" name="points" min="0" max="1" step="1" value="0" onchange="animHEJQZZPHOHXHIASP.set_frame(parseInt(this.value));"></input>

    <button onclick="animHEJQZZPHOHXHIASP.slower()">–</button>
    <button onclick="animHEJQZZPHOHXHIASP.first_frame()"></button>
    <button onclick="animHEJQZZPHOHXHIASP.previous_frame()"></button>
    <button onclick="animHEJQZZPHOHXHIASP.reverse_animation()"></button>
    <button onclick="animHEJQZZPHOHXHIASP.pause_animation()"></button>
    <button onclick="animHEJQZZPHOHXHIASP.play_animation()"></button>
    <button onclick="animHEJQZZPHOHXHIASP.next_frame()"></button>
    <button onclick="animHEJQZZPHOHXHIASP.last_frame()"></button>
    <button onclick="animHEJQZZPHOHXHIASP.faster()">+</button>
  <form action="#n" name="_anim_loop_selectHEJQZZPHOHXHIASP" class="anim_control">
    <input type="radio" name="state" value="once" checked> Once </input>
    <input type="radio" name="state" value="loop" > Loop </input>
    <input type="radio" name="state" value="reflect" > Reflect </input>
  </form>

    12

```python
# Run a demo of the environment
observation = env.reset()
cum_reward = 0
frames = []
a_t = np.zeros([ACTIONS])
a_t[env.action_space.sample()] = 1
for t in range(5000):
    # Render into buffer.
    s_t, reward, done, info = env.step(np.argmax(a_t))
    frames.append(env.render(mode = 'rgb_array'))
    readout_t = out.predict(s_t[None, :])
    a_t = np.zeros([ACTIONS])
    a_t[np.argmax(readout_t)] = 1

    if done:
        break
env.render(close=True)
display_frames_as_gif(frames)
print(t)
```

    <input id="_anim_sliderUAAZUDIZGVQRHNPX" type="range" style="width:350px" name="points" min="0" max="1" step="1" value="0" onchange="animUAAZUDIZGVQRHNPX.set_frame(parseInt(this.value));"></input>

    <button onclick="animUAAZUDIZGVQRHNPX.slower()">–</button>
    <button onclick="animUAAZUDIZGVQRHNPX.first_frame()"></button>
    <button onclick="animUAAZUDIZGVQRHNPX.previous_frame()"></button>
    <button onclick="animUAAZUDIZGVQRHNPX.reverse_animation()"></button>
    <button onclick="animUAAZUDIZGVQRHNPX.pause_animation()"></button>
    <button onclick="animUAAZUDIZGVQRHNPX.play_animation()"></button>
    <button onclick="animUAAZUDIZGVQRHNPX.next_frame()"></button>
    <button onclick="animUAAZUDIZGVQRHNPX.last_frame()"></button>
    <button onclick="animUAAZUDIZGVQRHNPX.faster()">+</button>
  <form action="#n" name="_anim_loop_selectUAAZUDIZGVQRHNPX" class="anim_control">
    <input type="radio" name="state" value="once" checked> Once </input>
    <input type="radio" name="state" value="loop" > Loop </input>
    <input type="radio" name="state" value="reflect" > Reflect </input>
  </form>

    117

```python

```

ReinforcementLearning/Lesson 20 - Deep Q Learning.ipynb


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline

pd.plotting.register_matplotlib_converters()
```

# Discussion: Week 5

## Exercise: More Business Cycle Data

The file `business_cycle_data_actual_trend.csv`, available at https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/business_cycle_data_actual_trend.csv, contains actual and trend data for real GDP per capita, real consumption per capita, real investment per capita, real physical capital per capita, TFP,  hours per capita, real M2 per capita, the rate of PCE (personal consumption expenditures) inflation, the three month T-bill rate, and the unemployment rate at quarterly frequency. The GDP, consumption, investment, capital, and M2 data are in terms of 2012 dollars. Hours is measured as an index with the value in October 2012 set to 100. PCE inflation is measured at the percent change from one year prior. PCE inflation, unemployment, and the T-bill rate are expressed in units of percent.

In this Notebook, you will replicate the work from class 9 but for the following quantities:
* Real GDP per capita
* 3-month T-Bill rate
* PCE inflation rate
* Unemployment rate

Refer to the Notebook from Class 9 for reference: https://nbviewer.jupyter.org/github/letsgoexploring/econ126/blob/master/Lecture%20Notebooks/Econ126_Class_09.ipynb?flush_cache=true

## Part (a): Preliminaries

```python
# Read business_cycle_data_actual_trend.csv into a Pandas DataFrame with the first column set as the index and parse_dates=True
df = pd.read_csv('https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/business_cycle_data_actual_trend.csv',index_col=0,parse_dates=True)

# Print the last five rows of the data
df.tail()
```

      gdp
      gdp_trend
      consumption
      consumption_trend
      investment
      investment_trend
      hours
      hours_trend
      capital
      capital_trend
      tfp
      tfp_trend
      real_m2
      real_m2_trend
      t_bill_3mo
      t_bill_3mo_trend
      pce_inflation
      pce_inflation_trend
      unemployment
      unemployment_trend

      2018-07-01
      72.589059
      72.376813
      49.377099
      49.228728
      12.887706
      12.752714
      105.399877
      105.120948
      98.162281
      98.225755
      25.061921
      25.026057
      0.000050
      0.000050
      0.020400
      0.016665
      0.057662
      0.045360
      0.037667
      0.038054

      2018-10-01
      72.606447
      72.672948
      49.377294
      49.441990
      12.942796
      12.825832
      105.435736
      105.356245
      98.584308
      98.615485
      25.024779
      25.057209
      0.000050
      0.000050
      0.023167
      0.018115
      0.046050
      0.045643
      0.038000
      0.036738

      2019-01-01
      73.251081
      72.970978
      49.529684
      49.656456
      13.135114
      12.899000
      105.677466
      105.590970
      99.373072
      99.008495
      25.139192
      25.088650
      0.000050
      0.000051
      0.023867
      0.019566
      0.039181
      0.045903
      0.038667
      0.035432

      2019-04-01
      73.482233
      73.270527
      49.967211
      49.872074
      12.910780
      12.972253
      105.340443
      105.825548
      99.885397
      99.403981
      25.225494
      25.120254
      0.000050
      0.000051
      0.023000
      0.021015
      0.040987
      0.046152
      0.036333
      0.034133

      2019-07-01
      73.703609
      73.571388
      50.218595
      50.088709
      12.811444
      13.045774
      105.768114
      106.060465
      100.291597
      99.801352
      25.199125
      25.151929
      0.000051
      0.000051
      0.019800
      0.022462
      0.039933
      0.046396
      0.036333
      0.032835

```python
# CELL PROVIDED
# Construct a 2x2 grid of plots of GDP, consumption, t_bill_3mo, unemployment, and pce_inflation. Multiply
# t_bill_3mo, unemployment, and pce_inflation by 100 inside the plot command to plot in units of percent.
base_year = '2012'
fig = plt.figure(figsize=(12,8))

ax1 = fig.add_subplot(2,2,1)
ax1.plot(df.gdp,'-',lw=3,alpha = 0.7)
ax1.grid()
ax1.set_title('GDP per capita')
ax1.set_ylabel('Thousands of '+base_year+' dollars')

ax2 = fig.add_subplot(2,2,2)
ax2.plot(df.t_bill_3mo*100,'-',lw=3,alpha = 0.7)
ax2.grid()
ax2.set_title('3-month T-Bill mate')
ax2.set_ylabel('Thousands of '+base_year+' dollars')

ax3 = fig.add_subplot(2,2,3)
ax3.plot(df.unemployment*100,'-',lw=3,alpha = 0.7)
ax3.grid()
ax3.set_title('Unemployment rate')
ax3.set_ylabel('Thousands of '+base_year+' dollars')

ax4 = fig.add_subplot(2,2,4)
ax4.plot(df.pce_inflation*100,'-',lw=3,alpha = 0.7)
ax4.grid()
ax4.set_title('PCE inflation')
ax4.set_ylabel('Percent')

fig.tight_layout()
```

## Part (b): Cycles and Trends

Recalle: A time series $X_t$ can be decomposed into a trend component $X_t^{trend}$ and a cyclical component $X_t^{cycle}$ such that:

\begin{align}
X_t & = X_t^{trend} + X_t^{cycle}. \tag{1}
\end{align}

In equation (1), $X_t^{trend}$ is the long-run value about which $X_t$ fluctuates. $X^{cycle}_t$ is the amount by which $X_t$ excedes its trend.

Often times, it's useful to express the cyclical component of a time series as the difference between the (natural) log of the series and the log of the trend:

\begin{align}
\hat{x}_t & = \log\left(X_t\right) - \log\left(X_t^{trend}\right) \approx\frac{X_t-X_t^{trend}}{X_t^{trend}}
\end{align}

The log-deviation from trend is approximately equal to the percent deviation of the series from trend (divided by 100).

### GDP

```python
# CELL PROVIDED
# Construct a plot of real GDP with its trend with:
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'
#    3. A legend in an appropriate place
plt.plot(df.gdp,'-',lw=1,alpha = 0.7,label='actual')
plt.plot(df.gdp_trend,'r-',lw=3,alpha = 0.7,label='trend')
plt.grid()
plt.ylabel('Thousands of '+base_year+' dollars per person')
plt.title('GDP per capita')
plt.legend(loc='lower right',ncol=2)
```

```python
# CELL PROVIDED
# Create a new column called gdp_cycle equal to the log difference between actual GDP and trend GDP:
df['gdp_cycle'] = np.log(df['gdp']) - np.log(df['gdp_trend'])

# Plot the log deviation of GDP from its trend (times 100)
plt.plot(df.gdp_cycle*100,'b-',lw=3,alpha = 0.7)
plt.ylabel('Percent')
plt.title('GDP: Percent deviation from trend')
plt.grid()
```

### 3-month T-bill rate

```python
# Construct a plot of the 3-month T-bill rate (times 100) with its trend with:
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'
#    3. A legend in an appropriate place
plt.plot(df.t_bill_3mo*100,'-',lw=1,alpha = 0.7,label='actual')
plt.plot(df.t_bill_3mo_trend*100,'r-',lw=3,alpha = 0.7,label='trend')
plt.ylabel('Percent')
plt.title('3-month T-bill rate')
plt.legend(loc='upper right',ncol=2)
plt.grid()
```

```python
# Create a new column called t_bill_3mo_cycle equal to the DIFFERENCE (not log difference!) between actual
# 3-month T-bill rate and it's trend:
df['t_bill_3mo_cycle'] = df['t_bill_3mo'] - df['t_bill_3mo_trend']

# Plot the deviation of the 3-month T-bill rate from its trend (times 100)
plt.plot(df.t_bill_3mo_cycle*100,'b-',lw=3,alpha = 0.7,label='t-bill')
plt.plot(df.gdp_cycle*100,'r-',lw=1,alpha = 0.5,label='gdp')
plt.ylabel('Percent')
plt.title('3-month T-bill rate: Deviation from trend')
plt.legend(loc='lower right',ncol=2)
plt.grid()
```

### Unemployment rate

```python
# Construct a plot of the unemployment rate (times 100) with its trend with:
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'
#    3. A legend in an appropriate place
plt.plot(df.unemployment*100,'-',lw=1,alpha = 0.7,label='actual')
plt.plot(df.unemployment_trend*100,'r-',lw=3,alpha = 0.7,label='trend')
plt.ylabel('Percent')
plt.title('Unemployment rate')
plt.legend(loc='upper right',ncol=2)
plt.grid()
```

```python
# Create a new column called unemployment_cycle equal to the DIFFERENCE (not log difference!) between actual
# unemployment and it's trend:
df['unemployment_cycle'] = df['unemployment'] - df['unemployment_trend']

# Plot the deviation of the unemployment rate from its trend (times 100)
plt.plot(df.unemployment_cycle*100,'b-',lw=3,alpha = 0.7,label='unemployment')
plt.plot(df.gdp_cycle*100,'r-',lw=1,alpha = 0.5,label='gdp')
plt.ylabel('Percent')
plt.title('Unemployment rate: Deviation from trend')
plt.legend(loc='lower right',ncol=2)
plt.grid()
```

### PCE Inflation

```python
# Construct a plot of the pce_inflation rate (times 100) with its trend with:
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'
#    3. A legend in an appropriate place
plt.plot(df.pce_inflation*100,'-',lw=1,alpha = 0.7,label='actual')
plt.plot(df.pce_inflation_trend*100,'r-',lw=3,alpha = 0.7,label='trend')
plt.ylabel('Percent')
plt.title('PCE inflation rate')
plt.legend(loc='upper right',ncol=2)
plt.grid()
```

```python
# Create a new column called pce_inflation_cycle equal to the DIFFERENCE (not log difference!) between actual
# pce_inflation and it's trend:
df['pce_inflation_cycle'] = df['pce_inflation'] - df['pce_inflation_trend']

# Plot the deviation of the unemployment rate from its trend (times 100)
plt.plot(df.pce_inflation_cycle*100,'b-',lw=3,alpha = 0.7,label='inflation')
plt.plot(df.gdp_cycle*100,'r-',lw=1,alpha = 0.5,label='gdp')
plt.ylabel('Percent')
plt.title('PCE inflation rate: Deviation from trend')
plt.legend(loc='lower left',ncol=2)
plt.grid()
```

## Part (c): Statistics

```python
# Create a new variable called df_cycle that is a DataFrame with columns columns gdp_cycle, t_bill_3mo_cycle,
# pce_inflation_cycle, unemployment_cycle from df.
df_cycle = df[['gdp_cycle', 't_bill_3mo_cycle', 'pce_inflation_cycle', 'unemployment_cycle']]

# Print the first five rows of df_cycle
print(df_cycle.head())
```

                gdp_cycle  t_bill_3mo_cycle  pce_inflation_cycle  \
    1959-01-01   0.010128         -0.001898             0.010258
    1959-04-01   0.022773          0.000568             0.021152
    1959-07-01   0.013910          0.006169             0.021213
    1959-10-01   0.007594          0.013271             0.015073
    1960-01-01   0.016899          0.009906             0.000901

                unemployment_cycle
    1959-01-01            0.000541
    1959-04-01           -0.006929
    1959-07-01           -0.005399
    1959-10-01           -0.002199
    1960-01-01           -0.006988

```python
# Use the DataFrame method .mean() to find the average values of the gdp_cycle, t_bill_3mo_cycle,
# pce_inflation_cycle, unemployment_cycle columns
df_cycle.mean()
```

    gdp_cycle             -9.024880e-14
    t_bill_3mo_cycle      -2.635295e-15
    pce_inflation_cycle   -2.376328e-15
    unemployment_cycle    -1.469732e-15
    dtype: float64

```python
# Use the DataFrame method .std() to find the standard deviations of the gdp_cycle, t_bill_3mo_cycle,
# pce_inflation_cycle, unemployment_cycle columns
df_cycle.std()
```

    gdp_cycle              0.014307
    t_bill_3mo_cycle       0.011522
    pce_inflation_cycle    0.013217
    unemployment_cycle     0.007281
    dtype: float64

```python
# Use the DataFrame method .corr() to find the coeficients of correlation among the gdp_cycle, t_bill_3mo_cycle,
# pce_inflation_cycle, unemployment_cycle columns
df_cycle.corr()
```

      gdp_cycle
      t_bill_3mo_cycle
      pce_inflation_cycle
      unemployment_cycle

      gdp_cycle
      1.000000
      0.407666
      0.553497
      -0.848191

      t_bill_3mo_cycle
      0.407666
      1.000000
      0.241685
      -0.560634

      pce_inflation_cycle
      0.553497
      0.241685
      1.000000
      -0.376727

      unemployment_cycle
      -0.848191
      -0.560634
      -0.376727
      1.000000

**Questions**

1. Of the three quantities besides GDP in the columns of `df_cycle`, which varies the most over the business cycle?
2. Of the three quantities besides GDP in the columns of `df_cycle`, which varies the least over the business cycle?
3. Of the three quantities besides GDP in the columns of `df_cycle`, which is most correlated (in absolute value) with GDP over the business cycle?

**Answers**

1. The PCE inflation rate fluctuates the most over the business cycle.
2. The unemployment rate fluctuates the least over the business cycle.
3. The unemployment rate is the quantity that is most correlated with GDP over the cycle. Since the capital stock changes slowly over time, large fluctuations due primarily to large fluctuations in employment.

## Part (d): Bonus

Do the next cell for fun. You won't get extra credit, but do it if you want to and have time.

```python
# Plot the cyclical components of GDP, three month T-bill rate, PCE inflation rate, and the unemployment rate
# (all times 100) on the same set of axes
fig = plt.figure(figsize=(12,4))
ax = fig.add_subplot(1,1,1)
(df_cycle*100).plot(ax = ax,legend=False,lw=3,alpha=0.7)
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_ylabel('Percent')
ax.set_title('Select US Business cycle data')
ax.grid()
```


# ShiftIfYouCan Example Notebook

In this notebook we present a walkthrough example for the ShiftIfYouCan visualisation code.

```python
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
```

<style>.container { width:100% !important; }</style>

### Setup

First we have to import all needed modules.

```python
import numpy as np

# Internal functions
from modules.ext_libraries import f_measure, variations
from modules.operating import operation_count, process_operations, get_variation, get_summary
from modules.plotting import plot_operations
```

By using the magic `%matplotlib notebook`, interaction (zoom in, out, save) with the figure is enabled.

*Note: zoom-in is higly resource-intensive and may be quite slow*

```python
# Uncomment/comment this line if you do/don't need interaction
%matplotlib notebook

import matplotlib.pyplot as plt
FIGSIZE = (18, 2.5)
plt.rcParams['figure.figsize'] = FIGSIZE
```

## Start the Analysis

First we load the beat detections and ground-truth annotations. The examples provided correspond to a highly expressive piece of music.
Then we obtain a list of the corrected detections (`corrected`), that can be saved for further use within a tool such as [Sonic Visualiser](https://www.sonicvisualiser.org/).

We print a summary of the operations in a friendly manner, that provides the proposed **annotation efficiency**, defined as follows:
\begin{equation}
ae = t^{+} / (t^{+} + s + f^{+} + f^{-}),
\end{equation}
where $t^{+}$ is the number of true positives, $s$ is the number of shifts, and $f^{-}$ and $f^{+}$ correspond to false negatives and false positives, respectively.
that displays the number of each operation required to transform the detection, thus maximising the (transformed) F-Measure.

```python
# Load the beat detections (dets) and the ground-truth annotation (anns)
dets = np.loadtxt('dets.txt')
anns = np.loadtxt('hains006.beats')

# use only the first column (i.e. the time stamp) if these are 2D
if dets.ndim > 1:
    dets = dets[:, 0]
if anns.ndim > 1:
    anns = anns[:, 0]
```

```python
# Get matrix of operations and annotation efficiency
ops, ann_eff = operation_count(dets, anns)

# Get list of transformed detections
transformed = process_operations(ops)

# Save list of transformed detections
np.savetxt('dets_transformed.txt', transformed, fmt='%.2f')

# Get combined F-measure (tuple with initial F-measure and transformed F-measure)
comb_f_measure = f_measure(dets, anns), f_measure(transformed, anns)

# Display results
print(get_summary('dets', ann_eff, comb_f_measure))
```

    - - - - - - - - - - - - - - - - -
    dets
    - - - - - - - - - - - - - - - - -
    annotation_efficiency: 0.877
      # "good" detections: 57
      # insertions: 3
      # deletions: 1
      # shifts: 4

    (initial) f-measure: 0.905
    (transformed) f_measure: 1.000

# Visualisation

We provide an open-source Python implementation which graphically displays the minimum set and type of operations required to transform a sequence of initial beat  detections in such a way as to maximize the F-measure when comparing the transformed detections against a ground truth annotation sequence.

Here we present some examples to demonstrate the usefulness of this rich visualisation of beat tracking evaluation.
All the figures are plotted in interactive mode to allow the user to go into greater detail by zooming in specific parts of the plot.

Note: Besides the configuration provided by the parametererisation of the function plot_operations(), a group of module-level control variables is defined at the top of the module *plotting.py*, where all the plotting functionality is implemented.

```python
# Display visualisation
fig,ax = plot_operations(ops, anns, 'Original')
plt.tight_layout()
plt.show()

```

    <IPython.core.display.Javascript object>

We can manually zoom in by using the tools in the navigation bar of the figure. For demonstration purposes, we'll zoom programatically, e.g. the area between 68 secs and 73 secs.
In this area of the figure, we can observe the following:
- 1 **correct detection** (at 68.6 secs), clearly within the *inner* tolerance window, depicted as a grey background surrounding the annotation;
- 2 **shifts**, also within the *outer* tolerance window, depicted as a pink background surrounding the annotation; the original ("unshifted") locations of the detections are show as dotted lines (at 70.2 and 71.9 secs) and the final locations of the shifts are depicted as solid lines (at 69.7 and 72.6 secs).
- 1 **insertion** (at 70.8 secs). In this case, as there was not any detection (to consider as correct detection) available within the inner tolerance window, neither any detection (to consider as shift) within the outer tolerance window (only drawn when a shift is considered), a **insertion** was accounted.

```python
# Zoom the image
ax[0].set_xlim(68, 73) # only when in "interactive" mode
```

    (68.0, 73.0)

Finally, we can change the sizes of the inner and outer tolerance windows and update all the calculations and visualisation. We just have to pass the new values for both tolerance windows as (optional) parameters of the corresponding functions.

```python
# Get matrix of operations and annotation efficiency
ops, ann_eff = operation_count(dets, anns, inn_tol_win=0.05, out_tol_win=3.0)

# Get list of transformed detections
transformed = process_operations(ops)

# Save list of transformed detections
np.savetxt('dets_transformed.txt', transformed, fmt='%.2f')

# Get combined F-measure (tuple with initial F-measure and transformed F-measure)
comb_f_measure = f_measure(dets, anns, inn_tol_win=0.05), f_measure(transformed, anns, inn_tol_win=0.05)

# Display results
print(get_summary('dets', ann_eff, comb_f_measure))

# Display visualisation
fig,ax = plot_operations(ops, anns, inn_tol_win=0.05, out_tol_win=3.0)
plt.tight_layout()
plt.show()
```

    - - - - - - - - - - - - - - - - -
    dets
    - - - - - - - - - - - - - - - - -
    annotation_efficiency: 0.859
      # (correct) detections: 55
      # insertions: 2
      # deletions: 0
      # shifts: 7

    (initial) f-measure: 0.873
    (transformed) f_measure: 1.000

    <IPython.core.display.Javascript object>

### Metrical Ambiguity

To allow for metrical ambiguity in beat tracking evaluation, it is common to create a set of variations of the ground truth by interpolation and sub-sampling operations. In our implementation, we flip this behaviour, and instead create the following variants of the beat detections:
- *Offbeat*: 180 degrees out of phase from the original beat locations;
- *Double*: Beats at 2x the original tempo;
- *Half-odd*: Half of the original tempo, only the odd beats;
- *Half-even*: Half of the original tempo, only the even beats;
- *Triple*: Beats at 3x the original tempo;
- *Third-1*: A Third of the original tempo, 1st beat (1,4,3,2,..)
- *Third-2*: A Third of the original tempo, 2nd beat (2,1,4,3,..)
- *Third-3*: A Third of the original tempo, 3rd beat (3,2,1,4,..)

```python
# Get all variations of beat detections
dets_variations, types_variations = variations(dets)
```

Now let's analyse one of the provided variations: e.g. the **Offbeat** variation, which is a version 180 degrees out of phase from the original beat locations.
As we would expect, the values for the annotation efficiency are worse than for the original beat detection: 0.015 vs 0.877.

```python
# Select specific variation of the beat detections
type_variation = 'Offbeat'
dets_variation = get_variation(type_variation, dets_variations, types_variations)

# Get matrix of operations and annotation efficiency
ops, ann_eff = operation_count(dets_variation, anns)

# Get list of transformed detections
transformed = process_operations(ops)

# Get combined f-measure (tuple with initial f-measure and transformed f-measure)
comb_f_measure = f_measure(dets_variation, anns), f_measure(transformed, anns)

# Display summary results
print(get_summary(type_variation, ann_eff, comb_f_measure))

# Get the figure
fig = plot_operations(ops, anns, type_variation)

# Show the plot
plt.tight_layout()
plt.show()
```

    - - - - - - - - - - - - - - - - -
    Offbeat
    - - - - - - - - - - - - - - - - -
    annotation_efficiency: 0.015
      # (correct) detections: 1
      # insertions: 6
      # deletions: 3
      # shifts: 57

    (initial) f-measure: 0.016
    (transformed) f_measure: 1.000

    <IPython.core.display.Javascript object>

Now let's do the same analysis for the **Double** variation, where the beats occur at 2x the original tempo.

```python
# Select specific variation of the beat detections
type_variation = 'Double'
dets_variation = get_variation(type_variation, dets_variations, types_variations)

# Get matrix of operations and annotation efficiency
ops, ann_eff = operation_count(dets_variation, anns)

# Get list of transformed detections
transformed = process_operations(ops)

# Get combined f-measure (tuple with initial f-measure and transformed f-measure)
comb_f_measure = f_measure(dets_variation, anns), f_measure(transformed, anns)

# Display summary results
print(get_summary(type_variation, ann_eff, comb_f_measure))

# Get the figure
fig = plot_operations(ops, anns, type_variation)

# Show the plot
plt.tight_layout()
plt.show()
```

    - - - - - - - - - - - - - - - - -
    Double
    - - - - - - - - - - - - - - - - -
    annotation_efficiency: 0.468
      # (correct) detections: 58
      # insertions: 1
      # deletions: 60
      # shifts: 5

    (initial) f-measure: 0.620
    (transformed) f_measure: 1.000

    <IPython.core.display.Javascript object>


```python
import matplotlib.pyplot as plt
import matplotlib as mpl
import numpy as np
from sympy import *
init_printing()
import sys
sys.path.append('../')
import uPVT.PVT_correlations as PVT
```

## Газосодержание

### Газосодержание, корреляция Стендинга

Для расчета газосодержания используется корреляция, обратная корреляции Стендинга для давления насыщения нефти газом.

$$ R_s =  \gamma_g  \left( \frac{1.92 p}{\ 10^{y_g}}\right)^{1.204} $$

где:

$R_s$ - газосодержание, $м^3/м^3 $

$P$ - давление, $МПа$

$\gamma_g$ - относительная плотность газа, безразмерная величина

$y_g$ - мольная доля газа, $ y_g = 1.225 +0.00164 T - \frac{ 1.769}{\gamma_o}$

$\gamma_o$ - относительная плотность нефти, безразмерная величина

$ T $ - температура, $ ^{\circ}\mathrm{K}$

Газосодержание является одним из ключевых свойств нефти при расчётах производительности скважин и работы скважинного оборудования. Динамика изменения газосодержания во многом определяет количество свободного газа в потоке и должна учитываться при проведении расчётов.

Если известно газосодержание при давлении насыщения, то газосодержание при давлениях ниже давления насыщения может быть получено из пропорции:

$$ R_s =  \ R_{sb}\left( \frac{p}{\ P_b}\right)^{1.204} $$

где:

$R_s$ - газосодержание, $м^3/м^3 $

$P$ - давление, $МПа$

$P_b$ - давление насыщения, $МПа$

$R_{sb}$ - газосодержание при давлении насыщения, $м^3/м^3 $

Корреляции Standing базируются на 105 экспериментально определенных давлениях насыщения нефтяных систем Калифорнии. Диапазоны значений основных свойств, использованных для разработки данной корреляции, приведены в таблице ниже.

|
 Параметр  | Диапазон |
| :--- | :--- |
|
 давление насыщения,$P_b$ , $ МПа $  | 0.896…48.263  |
|
 температура, $^{\circ}\mathrm{K} $ | 310…400 |
|
 газосодержание при давлении насыщения, $R_{sb}$ , $м^3/м^3 $ | 3.6…254 |
|
 относительная плотность нефти по воде, $\gamma_o$ | 0.725…0.956  |
|
 относительная плотность газа, $\gamma_g$  | 0.59…0.95  |

ref "A Pressure-Volume-Temperature Correlation for Mixtures of California Oil and Gases", M.B. Standing, Drill. & Prod. Prac., API, 1947.

```python
# Корреляция Стендинга для газосодержания реализована
# в виде функции unf_Rs_Standing_m3m3 в модуле PVT_correlations.
# Подробные данные по функции включая исходный код приведены ниже

PVT.unf_rs_Standing_m3m3??
```

```python
# параметры определяющие диапазоны значений для построения графиков
p_set=np.arange(1,300,10)
t_set=np.arange(273,380,30)
t_set_def=np.array([313])
gg_set=np.arange(0.6,1,0.1)
gg_set_def=np.array([0.8])
go_set=np.arange(0.8,1,0.05)
go_set_def=np.array([0.86])
# функция для автоматизации построения графиков по давлению насыщения
def prep_plot(func,tset,goset,ggset,plot_title,plot_xlab,plot_ylab):
    for t in tset:
        for gg in ggset:
            for go in goset:
                Rs_set=[]
                for p in p_set:
                    Rs_set.append(func(p,t_K = t,gamma_gas = gg,gamma_oil = go))
                plt.plot(p_set, Rs_set, label='t = %1.0f $ ^{\circ}\mathrm{K}$'%t +
                                                 ' $\gamma_g$ = %1.2f'%gg +
                                                 ' $\gamma_o$ = %1.2f'%go)
    plt.title(plot_title)
    plt.ylabel(plot_ylab, color = 'black')
    plt.xlabel(plot_xlab, color = 'black')
    plt.legend()
```

```python
# код для построения графиков
plt.figure(figsize=(15,8))
f = PVT.unf_rs_Standing_m3m3
# рисуем первый график
plt.subplot(221)
prep_plot(f,t_set,go_set_def,gg_set_def,
              'Газосодержание от давления',
               '$P, MPa$',
               '$R_s, м^3/м^3$')
plt.grid()
# рисуем второй график
plt.subplot(222)
prep_plot(f,t_set_def,go_set,gg_set_def,
              'Газосодержание от давления',
               '$P, MPa$',
               '$R_s, м^3/м^3$')
plt.grid()
# рисуем третий график
plt.subplot(223)
prep_plot(f,t_set_def,go_set_def,gg_set,
              'Газосодержание от давления',
               '$P, MPa$',
               '$R_s, м^3/м^3$')

plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.25, wspace=0.35)
# рисуем все
plt.grid()
plt.show()
```

### Газосодержание, корреляция Веларде-Маккейна

Корреляция Веларде-Маккейна (1999) для газосодержания базируется на 718 лабораторных исследованиях разгазирования различных нефтей со всего мира.

$$ R_s = R_{sb}R_{sr}   $$

где:

$R_s$ - газосодержание, $м^3/м^3$

$R_{sb}$ - газосодержание при давлении насыщения, $м^3/м^3$

$R_{sr}=a_1p_r^{a_2} + (1-a_1)p_r^{a_3}$ - приведенное газосодержание

$p_r=\frac{(p-14,7)}{(p_b-14,7)}$ - степень превышения давления, $psig/psig$

$a_1=9.73 \cdot 10^{-7}\gamma_{gSP}^{1.672608}API^{0.929870}T^{0.247235}(p_b-14.7)^{1.056052}$

$a_2=0.022339 \gamma_{gSP}^{-1.004750}API^{0.337711}T^{0.132795}(p_b-14.7)^{0.302065}$

$a_3=0.725167 \gamma_{gSP}^{-1.485480}API^{-0.164741}T^{-0.091330}(p_b-14.7)^{0.047094}$

В приведенной таблице представлены диапазоны значений использованных для создания корреляции:

|
 Параметр  | Мин|Среднее|Макс|
| :--- | :---: |:---:|:---:|
|
 давление насыщения,$P_b$ , $ МПа $  | 2.861  |15.706|53.434|
|
 температура, $^{\circ}\mathrm{С} $ | 21 |86|160|
|
 относительная плотность газа на сепараторе, $\gamma_g$  | 0.555  |0.793|1.472|
|
 объемный коэффициент нефти при давлении насыщения, $B_{ob}$ , $ м^3/м^3 $ | 1.012  |1.358|2.042|

```python
# Корреляция Веларде для газосодержания реализована
# в виде функции unf_Rs_Velarde_m3m3 в модуле PVT_correlations.
# Подробные данные по функции включая исходный код приведены ниже

PVT.unf_rs_Velarde_m3m3??
```

```python
# параметры определяющие диапазоны значений для построения графиков
p_set=np.arange(1,11,0.25)
t_set=np.arange(294,400,30)
t_set_def=np.array([313])
gg_set=np.arange(0.6,1,0.1)
gg_set_def=np.array([0.8])
go_set=np.arange(0.8,1,0.05)
go_set_def=np.array([0.86])
# функция для автоматизации построения графиков по давлению насыщения
def prep_plot(func,tset,goset,ggset,plot_title,plot_xlab,plot_ylab):
    for t in tset:
        for gg in ggset:
            for go in goset:
                Rs_set=[]
                for p in p_set:
                    Rs_set.append(func(p,t_K = t,gamma_gas = gg,gamma_oil = go))
                plt.plot(p_set, Rs_set, label='t = %1.0f $ ^{\circ}\mathrm{K}$'%t +
                                                 ' $\gamma_g$ = %1.2f'%gg +
                                                 ' $\gamma_o$ = %1.2f'%go)
    plt.title(plot_title)
    plt.ylabel(plot_ylab, color = 'black')
    plt.xlabel(plot_xlab, color = 'black')
    plt.legend()
```

```python
# код для построения графиков
plt.figure(figsize=(15,8))
f = PVT.unf_rs_Velarde_m3m3
# рисуем первый график
plt.subplot(221)
plt.grid()
prep_plot(f,t_set,go_set_def,gg_set_def,
              'Газосодержание от давления',
               '$P, MPa$',
               '$R_s, м^3/м^3$')
# рисуем второй график
plt.subplot(222)
plt.grid()
prep_plot(f,t_set_def,go_set,gg_set_def,
              'Газосодержание от давления',
               '$P, MPa$',
               '$R_s, м^3/м^3$')
# рисуем третий график
plt.subplot(223)
prep_plot(f,t_set_def,go_set_def,gg_set,
              'Газосодержание от давления',
               '$P, MPa$',
               '$R_s, м^3/м^3$')

plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.25, wspace=0.35)
# рисуем все
plt.grid()
plt.show()
```

###  Газосодержание при давлении насыщения, Корреляция Маккейна если известны данные в сепараторе и резервуаре

Во многих корреляциях используется в качестве исходного параметра - газосодержание при давлении насыщения. Значение этого параметра может быть получено с помощью полевых данных как сумма отделяемого газового фактора в сепараторе и резервуаре для товарной нефти.

$$ R_{sb} = R_{sp} + R_{st}   $$

где:

$R_{sb}$ - газосодержание при давлении насыщения, $м^3/м^3$

$R_{sp}$ - газовый фактор, отделяемый в сепараторе, $м^3/м^3$

$R_{st}$ -  газовый фактор в резервуаре для товарной нефти, $м^3/м^3$

Данное уравнение справедливо только если пластовое давление выше давления насыщения

---

Расход газа и дебит нефти в сепараторе обычно измеряются, когда как в резервуаре газ обычно выпускается и не замеряется. Поэтому для более точной оценки газосодержания пластовой нефти при давлении насыщения необходимо оценить газовый фактор в резервуаре.
Таким образом, был разработана формула на основе GRACE-алгоритма на базе 898 исследований пластового флюида. В таблице приведены набор данных для исследования.

$$ ln R_{st} = 3.955 + 0.83z - 0.024z^2 + 0.075z^3 $$

$$ z =\sum_{n=1}^3 z_n $$

$$ z_n = C0_n + C1_nVAR_n + C2_nVAR_n^2 $$

|
 $$n$$  | $$VAR$$|$$CO$$|$$C1$$|$$C2$$|
| :--- | :---: |:---:|:---:|:---:|
|
 $$1$$  | $$ln p_{sp} $$ |$$-8.005$$|$$2.7$$|$$-0.161$$|
|
 $$2$$  | $$ln T_{sp}$$  |$$1.224$$|$$-0.5$$|$$0$$|
|
 $$3$$ | $$API$$  |$$-1.587$$|$$0.0441$$|$$-2.29 \cdot  10 ^{-5}$$|

$$T_{sp} - °F, p_{sp} - psia $$

Выражение для вычисления газового фактора в резервуаре требует знать температуру и давление в сепараторе, которые обычно не всегда бывают известны. Поэтому в этом случае можно использовать следующее уравнение.

$$ R_{sb} = 1.1618 R_{sp}   $$

```python
# Корреляция Маккейна для газосодержания при давлении насыщения реализована
# в виде функции unf_Rsb_Mccain_m3m3 в модуле PVT_correlations.
# Подробные данные по функции включая исходный код приведены ниже

PVT.unf_rsb_Mccain_m3m3??
```

```python
# параметры определяющие диапазоны значений для построения графиков
p_set=np.arange(1,11,0.25)
t_set=np.arange(294,400,30)
t_set_def=np.array([313])
go_set=np.arange(0.8,1,0.05)
go_set_def=np.array([0.86])
r_sp = 50
# функция для автоматизации построения графиков по давлению насыщения
def prep_plot(func,tset,goset,plot_title,plot_xlab,plot_ylab):
    for t in tset:
            for go in goset:
                Rs_set=[]
                for p in p_set:
                    Rs_set.append(func(r_sp,go,p,t))
                plt.plot(p_set, Rs_set, label='t = %1.0f $ ^{\circ}\mathrm{K}$'%t +
                                                  ' $\gamma_o$ = %1.2f'%go)
    plt.title(plot_title)
    plt.ylabel(plot_ylab, color = 'black')
    plt.xlabel(plot_xlab, color = 'black')
    plt.legend()
```

```python
# код для построения графиков
plt.figure(figsize=(15,8))
f = PVT.unf_rsb_Mccain_m3m3
# рисуем первый график
plt.subplot(221)
plt.grid()
prep_plot(f,t_set,go_set_def,
              'Газосодержание от давления в сепараторе',
               '$P, MPa$',
               '$R_s, м^3/м^3$')
# рисуем второй график
plt.subplot(222)
plt.grid()
prep_plot(f,t_set_def,go_set,
              'Газосодержание от давления в сепараторе',
               '$P, MPa$',
               '$R_s, м^3/м^3$')

plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.25, wspace=0.35)
# рисуем все
plt.show()
```


## Evade Signal Classification with FGSM

This notebook shows the performance of a digital FGSM attack on an Automatic Modulation Classification neural network model.

### Assumptions
- The dataset wrangling has already been completed (and is provided here)
- The implementation of model training on this dataset has already been completed
- The plotting code has already been completed

### Components Recreated in Tutorial
- FGSM attack constrained by a power ratio

### See Also
The code in this tutorial is a stripped down version of the code in ``rfml.attack.fgsm`` that simplifies discussion.  Further detail can be provided by directly browsing the source files.

### Install the library code and dependencies

```python
# Install the library code
#!pip install git+https://github.com/brysef/rfml.git@1.0
```

```python
# Ensure that the least loaded GPU is used
import setGPU

# Plotting Includes
import matplotlib.pyplot as plt

# External Includes
import numpy as np

import torch
from torch.nn.functional import cross_entropy
from torch.utils.data import DataLoader, TensorDataset

# Internal Includes
from rfml.data import Dataset, Encoder
from rfml.data import build_dataset

from rfml.nbutils import plot_acc_vs_spr, plot_confusion

from rfml.nn.eval import compute_accuracy
from rfml.nn.eval.confusion import _confusion_matrix
from rfml.nn.F import energy
from rfml.nn.model import build_model
from rfml.nn.train import build_trainer, PrintingTrainingListener
```

    setGPU: Setting GPU to: 1

### Configuration

```python
gpu = True       # Set to True to use a GPU for training
fig_dir = None   # Set to a file path if you'd like to save the plots generated
data_path = None # Set to a file path if you've downloaded RML2016.10A locally
```

## Train and Evaluate a Model on a Static Dataset

See ``module_2`` for a more extensive description of this procedure.
Here we just use the library for a quick implementation.
In practice, the model training would be separated from the attack, but, is juxtaposed with the attack here for simplicity as it is self-contained for demonstration.

```python
train, test, val, le = build_dataset("RML2016.10a", path=data_path)
# as_numpy returns x,y and x is shape BxCxIQxN
input_samples = val.as_numpy(le=le)[0].shape[3]
model = build_model(model_name="CNN", input_samples=input_samples, n_classes=len(le))
trainer = build_trainer(strategy="standard", max_epochs=3, gpu=gpu)
trainer.register_listener(PrintingTrainingListener())
trainer(model=model, training=train, validation=val, le=le)
acc = compute_accuracy(model=model, data=test, le=le)

print("Overall Testing Accuracy: {:.4f}".format(acc))
```

    Epoch 0 completed!
    		-Mean Training Loss: 1.383
    		-Mean Validation Loss: 1.201
    Epoch 1 completed!
    		-Mean Training Loss: 1.189
    		-Mean Validation Loss: 1.115
    Epoch 2 completed!
    		-Mean Training Loss: 1.133
    		-Mean Validation Loss: 1.062
    Training has Completed:

    =======================
    	Best Validation Loss: 1.062
    	Best Epoch: 2
    	Total Epochs: 2
    =======================
    Overall Testing Accuracy: 0.5576

### Strip Down the Testing Data
Strip the data down to only the highest SNR (18 dB in RML2016.10a).  This ensures that the classification accuracy would have been close to the highest and gives the best evaluation of the attack because it separates the decreased accuracy from being causes by low SNR vs being caused by the attack.

```python
mask = test.df["SNR"] >= 18
dl = DataLoader(test.as_torch(le=le, mask=mask), shuffle=True, batch_size=512)
```

### Evading Signal Classification with Direct Access to the Classifier

Here, we're simply going to sweep the **intensity** of the attack, denoted in this tutorial as $E_s/E_p$ or signal-to-perturbation ratio represented in dB.

Recall that the equation for FGSM [Goodfellow et. al] is

\begin{equation}
    x^* = x + \text{sign}(\nabla_X \mathcal{L}(f(\theta, X), y_s))
\end{equation}

For the purposes of a clean implementation in software, we can split this equation into multiple steps.
First, we must compute the *signed gradient*.

\begin{equation}
    \text{"signed gradient"} = \text{sign}(\nabla_X \mathcal{L}(f(\theta, X), y_s))
\end{equation}

This is done by utilizing the model and PyTorch to backpropogate the gradient to the input.
Once the gradient is known, we can apply a simple sign operation.

Then, the signed gradient must be scaled to achieve an allowed intensity.
Traditionally, CV literature uses a norm to constrain the perturbation (e.g. $\left\lVert {x - x^*} \right\rVert_p < \epsilon$).
In the context of wireless communications, it makes more sense to constrain the power ratio of the perturbation to the underlying signal as the absolute values of either the perturbation or signal generally do not matter.
Recall that we've assumed $E_s$ is $1$ and therefore can scale the perturbation using the following equation [Flowers et. al, Sadeghi/Larson].

\begin{equation}
    p = \sqrt{\frac{10^{\frac{-E_s/E_p}{10}}}{2 \times \text{sps}}} \times \text{"signed gradient"}
\end{equation}

Because the RML2016.10A dataset is not properly normalized for all examples, especially AM-SSB, we have to normalize the input data before running the attack to have a known power -- this is the only way to ensure that the attack intensity is correct.

To complete the FGSM algorithm we simply need to add the perturbation to the original example.

\begin{equation}
    \text{"adversarial example"} = x + p
\end{equation}

#### Citations

##### Goodfellow et al

Goodfellow, I., Shlens, J., and Szegedy, C. (2015).  Explaining and harnessing adversarial examples. In Int. Conf. on Learning Representations.

##### Flowers et al

Flowers, B., Buehrer,  R.  M.,  and Headley,  W. C. (2019). Evaluating adversarial evasion attacks in the context of wireless communications. IEEE Transactions on Information Forensics and Security, pages 1–1.

##### Sadeghi/Larson

Sadeghi, M. and Larsson, E. G. (2018). Adversarial attacks on deep-learning based radio signal classification.IEEEWireless Commun. Letters, pages 1–1.

#### Helper methods for the attack

```python
def fgsm(x, y, input_size, net, spr, sps):
    p = compute_signed_gradient(x=x, y=y, input_size=input_size, net=net)
    p = scale_perturbation(sg=p, spr=spr, sps=sps)

    return x + p

def compute_signed_gradient(x, y, input_size, net):
    # Ensure that the gradient is tracked at the input, add some noise to avoid any
    # actual zeros in the signal (dithering), and then ensure its the proper shape
    x.requires_grad = True
    _x = _dither(x)

    # Put the inputs/outputs onto the most probable device that the model is currently
    # on -- this could fail if the model gets split amongst multiple devies, but, that
    # doesn't happen in this code.
    _x = _x.to(net.device)
    y = y.to(net.device)

    # Perform forward/backward pass to get the gradient at the input
    _y = net(_x)
    loss = cross_entropy(_y, y)
    loss.backward()

    # Take the sign of the gradient that can be scaled later
    ret = torch.sign(x.grad.data)

    return ret

def scale_perturbation(sg, spr, sps):
    if spr == np.inf:
        return sg * 0
    multiplier = pow(10, -spr / 10.0)
    multiplier = multiplier / (2 * sps)
    multiplier = pow(multiplier, 0.5)

    return sg * multiplier

def _dither(x):
    snr = 100
    voltage = pow(pow(10.0, -snr / 10.0), 0.5)

    noise = x.data.new(x.size()).normal_(0.0, voltage)
    return x + noise

def _normalize(x):
    power = energy(x)
    # Make the dimensions match because broadcasting is too magical to
    # understand in its entirety... essentially want to ensure that we
    # divide each channel of each example by the sqrt of the power of
    # that channel/example pair
    power = power.view([power.size()[0], power.size()[1], 1, 1])

    return x / torch.sqrt(power)

def _sanity_check(desired_spr, adv_x, x):
    signal_power = energy(x)
    perturbation_power = energy(adv_x - x)
    _spr = 10*torch.log10(signal_power / perturbation_power)
    _spr = _spr.detach().numpy().mean()
    if np.abs(_spr - desired_spr) > 0.5:
        raise RuntimeError("Calculated SPR and desired SPR does not match: "
                           "Desired SPR={:0.2f}dB, Calculated SPR={:0.2f}dB, "
                           "Signal Power={:0.2f}dB, Perturbation Power={:0.2f}dB".format(
                               desired_spr,
                               _spr,
                               10.0*np.log10(signal_power.detach().numpy().mean()),
                               10.0*np.log10(perturbation_power.detach().numpy().mean()))
                          )
```

#### Top Level control loop for the attack

```python
# Ensure that the model is in "evaluation" mode
# -- Therefore batch normalization is not computed and dropout is not performed
# -- Note: This is the cause of a lot of bugs
model.eval()

acc_vs_spr = list()
sprs = list()

for spr in np.linspace(50.0, 0.0, num=26):
    right = 0
    total = 0

    for x, y in dl:
        # Before crafting an attack on X, ensure that it is normalized to have a
        # specified power -- RML2016.10a fluctuates a bit, especially on AM-SSB,
        # which will cause the attack intensity to be incorrect.
        x = _normalize(x)
        adv_x = fgsm(x, y, spr=spr, input_size=input_samples, sps=1, net=model)

        # Ensure that we've accurately represented the attack power
        _sanity_check(desired_spr=spr, adv_x=adv_x, x=x)

        predictions = model.predict(adv_x)
        right += (predictions == y).sum().item()
        total += len(y)

    acc = float(right) / total
    acc_vs_spr.append(acc)
    sprs.append(spr)

fig = plot_acc_vs_spr(acc_vs_spr=acc_vs_spr,
                      spr=sprs,
                      title="Performance of a Digital FGSM Attack"
                     )
if fig_dir is not None:
    file_path = "{fig_dir}/direct_access_fgsm.pdf".format(fig_dir=fig_dir)
    print("Saving Figure -> {file_path}".format(file_path=file_path))
    fig.savefig(file_path, format="pdf", transparent=True)
plt.show()
```

#### Constructing a Confusion Matrix

The model breaks down in a way that generally intuitively makes sense to the expert eye with knowledge of the underlying signal formats.  The analog signals are confused for other analog signals, the PSK/QAM signals are confused for other PSK/QAMs with differing modulation orders, etc.  This behavior was examined more closely for targeted adversarial attacks in [Bair et al.] where it was found that the adversarial distance [Papernot et al.], or put more simply, the adversarial perturbation power required to "transform" 50% of the input examples from a source to target class, could be used in a spectral clustering algorithm to recover the relationships between the signal formats.

##### Bair et al.

Bair, S., Delvecchio, M., Flowers, B., Michaels, A. J., andHeadley, W. C. (2019). On the limitations of targeted adversarial evasion attacks against deep learning enabled modulation recognition. In ACM Workshop on Wireless Security and Machine Learning (WiseML 2019).

#### Papernot et al.

Papernot,  N.,  McDaniel,  P.,  Jha,  S.,  Fredrikson,  M.,Celik,  Z.  B.,  and  Swami,  A.  (2016).   The  limitations  of  deep  learningin  adversarial  settings.   InIEEE  European  Symposium  on  Security  andPrivacy (EuroS&P), pages 372–387. IEEE.

```python
spr = 10  # dB

predictions = list()
labels = list()
for x, y in dl:
    # Before crafting an attack on X, ensure that it is normalized to have a
    # specified power -- RML2016.10a fluctuates a bit, especially on AM-SSB,
    # which will cause the attack intensity to be incorrect.
    x = _normalize(x)
    adv_x = fgsm(x, y, spr=spr, input_size=input_samples, sps=1, net=model)

    # Ensure that we've accurately represented the attack power
    _sanity_check(desired_spr=spr, adv_x=adv_x, x=x)

    _predictions = model.predict(adv_x)
    predictions.extend(_predictions.detach().numpy())
    labels.extend(y)

cmn = _confusion_matrix(predictions=predictions, labels=labels, le=le)

title = "Confusion Matrix with a {spr} dB FGSM Attack".format(spr=spr)
fig = plot_confusion(cm=cmn, labels=le.labels, title=title)
if fig_dir is not None:
    file_path = "{fig_dir}/fgsm_confusion_matrix.pdf"
    print("Saving Figure -> {file_path}".format(file_path=file_path))
    fig.savefig(file_path, format="pdf", transparent=True)
plt.show()
```


# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [ 1.77635684e-17  6.10622664e-17 -1.86656246e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.6674627590926071e-09
    dgamma error:  7.378954934067588e-13
    dbeta error:  2.6853898000928757e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 3.75e-10
    b1 relative error: 2.22e-03
    b2 relative error: 2.66e-07
    b3 relative error: 9.06e-11
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.17e-09
    gamma1 relative error: 6.44e-09
    gamma2 relative error: 2.41e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 1.11e-08
    b2 relative error: 2.50e-08
    b3 relative error: 2.23e-10
    beta1 relative error: 6.32e-09
    beta2 relative error: 5.69e-09
    gamma1 relative error: 5.94e-09
    gamma2 relative error: 4.14e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.315000; val_acc: 0.266000
    (Iteration 21 / 200) loss: 2.039345
    (Epoch 2 / 10) train acc: 0.397000; val_acc: 0.285000
    (Iteration 41 / 200) loss: 2.046870
    (Epoch 3 / 10) train acc: 0.486000; val_acc: 0.319000
    (Iteration 61 / 200) loss: 1.749425
    (Epoch 4 / 10) train acc: 0.523000; val_acc: 0.325000
    (Iteration 81 / 200) loss: 1.270384
    (Epoch 5 / 10) train acc: 0.581000; val_acc: 0.315000
    (Iteration 101 / 200) loss: 1.378948
    (Epoch 6 / 10) train acc: 0.639000; val_acc: 0.317000
    (Iteration 121 / 200) loss: 1.123740
    (Epoch 7 / 10) train acc: 0.658000; val_acc: 0.320000
    (Iteration 141 / 200) loss: 1.139865
    (Epoch 8 / 10) train acc: 0.700000; val_acc: 0.292000
    (Iteration 161 / 200) loss: 0.854841
    (Epoch 9 / 10) train acc: 0.801000; val_acc: 0.319000
    (Iteration 181 / 200) loss: 0.900679
    (Epoch 10 / 10) train acc: 0.768000; val_acc: 0.306000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696059
    (Epoch 6 / 10) train acc: 0.535000; val_acc: 0.345000
    (Iteration 121 / 200) loss: 1.557986
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.304000
    (Iteration 141 / 200) loss: 1.432189
    (Epoch 8 / 10) train acc: 0.628000; val_acc: 0.339000
    (Iteration 161 / 200) loss: 1.033931
    (Epoch 9 / 10) train acc: 0.661000; val_acc: 0.340000
    (Iteration 181 / 200) loss: 0.901033
    (Epoch 10 / 10) train acc: 0.726000; val_acc: 0.318000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers_ws[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
weight scale does not much affect the batchnorm layers, as when we normalise the layers, we prevent the explosion of gradients.

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
More is the batch size, better is the effect of batch normalisation on training.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
2. Layer Normalisation
3. Batch Normalisation

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_backward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 7.40148683e-16 -7.40148683e-17  0.00000000e+00  2.96059473e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.3050040640816791e-09
    dgamma error:  4.759518365681667e-12
    dbeta error:  2.6853898000928757e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
2 - If the number of features are small, the value of mean and standard deviation will have large variance when compared between different data points.

```python

```