## Weather Prediction with Deep Learning (30 minutes)
Denis Akhiyarov
    * Senior Consultant at Wood
    * Python.NET (Python for .NET) core developer
    * Faculty at North American University

https://notebooks.azure.com/denfromufa/libraries/pmlc

## Economics of weather forecasting?

1. Insurance and reinsurance

2. Transportation (air, taxi, boats)

3. Agriculture and fishing

4. Input for energy industry

5. Climate change and extreme events

     https://www.theweathernetwork.com

## Yandex Weather

https://yandex.com/company/blog/winter-is-coming-yandex-weather-nowcasting-helps-users-plan-for-precipitation

```
from IPython.display import YouTubeVideo
YouTubeVideo('aR5p7_g8qXQ', width=800, height=600)
```

## Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale
National Energy Research Scientific Computing Center

## Package imports

```
import pandas as pd
import numpy as np
from numpy import exp
from IPython.display import HTML, Latex, Math, display
import matplotlib.pyplot as plt
import matplotlib
import seaborn as sns
from sympy.printing import latex
from sympy.core.sympify import sympify
%matplotlib inline
pd.options.display.max_columns=50
pd.options.display.max_colwidth=100
matplotlib.rcParams['savefig.dpi'] = 300
matplotlib.rcParams['figure.dpi'] = 300
matplotlib.rcParams['figure.figsize'] = (3, 2)
fontsize=3
matplotlib.rcParams['figure.titlesize'] = fontsize
matplotlib.rcParams['axes.labelsize'] = fontsize
matplotlib.rcParams['font.size'] = fontsize
matplotlib.rcParams['axes.titlesize'] = fontsize
matplotlib.rcParams['legend.fontsize'] = fontsize
matplotlib.rcParams['xtick.labelsize'] = fontsize
matplotlib.rcParams['ytick.labelsize'] = fontsize
matplotlib.rcParams['lines.markersize'] = 2
```

## How to get weather data?!

Even Google Datasets does not explain well, multiple commercial offerings online

https://www.kaggle.com/selfishgene/historical-hourly-weather-data

Historic data of weather in Houston requested and retrieved from Department of Commerce > NOAA > NESDIS > NCEI

https://www.ncdc.noaa.gov/

## Data Exploration

Let's read the head of the table and look at the data using pandas

```
dfhtown=pd.read_csv("https://raw.githubusercontent.com/denfromufa/deepweather/master/9693687323818dat.txt", delim_whitespace=True)
dfhtown.head()
```

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW.1
      MW.2
      MW.3
      AW
      AW.1
      AW.2
      AW.3
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      0
      720617
      208
      201604010015
      ***
      0
      ***
      722
      CLR
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      79
      66
      ******
      29.67
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      1
      720617
      208
      201604010035
      330
      6
      ***
      722
      CLR
      *
      *
      *
      9.1
      **
      **
      **
      **
      **
      **
      **
      **
      *
      75
      66
      ******
      29.66
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      2
      720617
      208
      201604010055
      330
      3
      ***
      722
      CLR
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      73
      66
      ******
      29.67
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      3
      720617
      208
      201604010115
      330
      6
      ***
      722
      CLR
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      73
      66
      ******
      29.68
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      4
      720617
      208
      201604010135
      340
      7
      ***
      722
      CLR
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      72
      64
      ******
      29.70
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

The data is messy, the names of columns not clear, so let's look at html version. Again using pandas!

```
import sys
!pip install lxml html5lib
```

    Collecting lxml
      Downloading https://files.pythonhosted.org/packages/03/a4/9eea8035fc7c7670e5eab97f34ff2ef0ddd78a491bf96df5accedb0e63f5/lxml-4.2.5-cp36-cp36m-manylinux1_x86_64.whl (5.8MB)
        100% |████████████████████████████████| 5.8MB 5.4MB/s
    Collecting html5lib
      Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
        100% |████████████████████████████████| 122kB 26.1MB/s
    Requirement already satisfied: webencodings in /usr/local/lib/python3.6/dist-packages (from html5lib) (0.5.1)
    Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.6/dist-packages (from html5lib) (1.11.0)
    Installing collected packages: lxml, html5lib
    Successfully installed html5lib-1.0.1 lxml-4.2.5

```
dfheader=pd.read_html("https://raw.githubusercontent.com/denfromufa/deepweather/master/72061700208-2016-04_9693687323818dat.html")
```

```
def pd2html(df, header=False, index=False):
    return HTML(df.to_html(index=False, header=False))
pd2html(dfheader[0])
```

      AWS ID 720617
      WBAN ID 00208
      Name HOUSTON EXECUTIVE AIRPORT
      Country UNITED STATES
      State TEXAS
      Latitude +29.800
      Longitude -095.900
      Elevation +0051.2 (meters)

```
pd2html(dfheader[1][:2])
```

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW
      MW
      MW
      AW
      AW
      AW
      AW
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      NaN
      NaN
      GMT
      NaN
      MPH
      MPH
      NaN
      NaN
      NaN
      NaN
      NaN
      Miles
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      F
      F
      Mb
      inches
      Mb
      F
      F
      inches
      inches
      inches
      inches
      inches

## Data cleaning

Let's set the index of dataframe (table) to dates

```
dfhtown.set_index(dfhtown.columns[2], inplace=True)
dfhtown.head(3)
```

      USAF
      WBAN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW.1
      MW.2
      MW.3
      AW
      AW.1
      AW.2
      AW.3
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      YR--MODAHRMN

      201604010015
      720617
      208
      ***
      0
      ***
      722
      CLR
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      79
      66
      ******
      29.67
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      201604010035
      720617
      208
      330
      6
      ***
      722
      CLR
      *
      *
      *
      9.1
      **
      **
      **
      **
      **
      **
      **
      **
      *
      75
      66
      ******
      29.66
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      201604010055
      720617
      208
      330
      3
      ***
      722
      CLR
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      73
      66
      ******
      29.67
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

```
dfhtown.index=pd.to_datetime(dfhtown.index, format="%Y%m%d%H%M")
dfhtown.index.name='datetime'
dfhtown.replace(to_replace='\*+', value=np.nan, regex=True, inplace=True)
dfhtown.dropna(axis=1, inplace=True, how='all')
dfhtown.head()
```

      USAF
      WBAN
      DIR
      SPD
      GUS
      CLG
      SKC
      VSB
      AW
      TEMP
      DEWP
      ALT

      datetime

      2016-04-01 00:15:00
      720617
      208
      NaN
      0
      NaN
      722
      CLR
      10.0
      NaN
      79
      66
      29.67

      2016-04-01 00:35:00
      720617
      208
      330
      6
      NaN
      722
      CLR
      9.1
      NaN
      75
      66
      29.66

      2016-04-01 00:55:00
      720617
      208
      330
      3
      NaN
      722
      CLR
      10.0
      NaN
      73
      66
      29.67

      2016-04-01 01:15:00
      720617
      208
      330
      6
      NaN
      722
      CLR
      10.0
      NaN
      73
      66
      29.68

      2016-04-01 01:35:00
      720617
      208
      340
      7
      NaN
      722
      CLR
      10.0
      NaN
      72
      64
      29.70

Let's look at subset of columns with physical units from HTML version of the table

```
dfunits=dfheader[1][:2]
dfunits
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32

      0
      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW
      MW
      MW
      AW
      AW
      AW
      AW
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      1
      NaN
      NaN
      GMT
      NaN
      MPH
      MPH
      NaN
      NaN
      NaN
      NaN
      NaN
      Miles
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      F
      F
      Mb
      inches
      Mb
      F
      F
      inches
      inches
      inches
      inches
      inches

```
dfunits.rename(columns=dfunits.iloc[0]).iloc[1].loc[dfhtown.columns].dropna().to_frame()
```

      1

      SPD
      MPH

      GUS
      MPH

      VSB
      Miles

      TEMP
      F

      DEWP
      F

      ALT
      inches

* So SPD and GUS are some velocity measurements for wind
* VSB??
* TEMP - temperature
* DEWP - temperature dewpoint (used for humidity)
* ALT - perhaps the pressure in inches of liquid, such as mercury

The types of columns are very generic (object), so let's improve this!

```
dfhtown.dtypes
```

    USAF     int64
    WBAN     int64
    DIR     object
    SPD     object
    GUS     object
    CLG     object
    SKC     object
    VSB     object
    AW      object
    TEMP    object
    DEWP    object
    ALT     object
    dtype: object

```
dfhtown = dfhtown[['DIR','SPD','VSB','TEMP','DEWP','ALT']].astype('float64',raise_on_error=False)
dfhtown.head()
```

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: FutureWarning: the raise_on_error=False keyword is deprecated, use errors='ignore' instead
      """Entry point for launching an IPython kernel.

      DIR
      SPD
      VSB
      TEMP
      DEWP
      ALT

      datetime

      2016-04-01 00:15:00
      NaN
      0.0
      10.0
      79.0
      66.0
      29.67

      2016-04-01 00:35:00
      330.0
      6.0
      9.1
      75.0
      66.0
      29.66

      2016-04-01 00:55:00
      330.0
      3.0
      10.0
      73.0
      66.0
      29.67

      2016-04-01 01:15:00
      330.0
      6.0
      10.0
      73.0
      66.0
      29.68

      2016-04-01 01:35:00
      340.0
      7.0
      10.0
      72.0
      64.0
      29.70

```
dfhtown.dtypes
```

    DIR     float64
    SPD     float64
    VSB     float64
    TEMP    float64
    DEWP    float64
    ALT     float64
    dtype: object

The relative humidity (RH) is not given in table, so let's calculate using measured temperature and dewpoint temperature

```
rel_hum_eqn = "100*(exp((17.625*TD)/(243.04+TD))/exp((17.625*T)/(243.04+T)))"

```

```
dfhtown['T']=(dfhtown['TEMP']-32)/1.8
dfhtown['TD']=(dfhtown['DEWP']-32)/1.8
dfhtown['RH']=dfhtown.eval(rel_hum_eqn)
dfhtown[['RH']].rolling(window=24*7).mean().plot(style='.');
plt.ylabel('Relative Humidity [%]');
```

```
dfhtown.describe()
```

      DIR
      SPD
      VSB
      TEMP
      DEWP
      ALT
      T
      TD
      RH

      count
      20087.000000
      25833.000000
      23997.000000
      25844.000000
      25844.000000
      25853.000000
      25844.000000
      25844.000000
      25844.000000

      mean
      168.682232
      6.866527
      8.109147
      71.196835
      63.041751
      30.041237
      21.776019
      17.245417
      78.277455

      std
      108.760045
      5.283873
      2.605837
      12.736647
      13.240728
      0.162329
      7.075915
      7.355960
      19.340430

      min
      10.000000
      0.000000
      0.000000
      19.000000
      10.000000
      29.450000
      -7.222222
      -12.222222
      17.552120

      25%
      110.000000
      3.000000
      7.000000
      64.000000
      57.000000
      29.950000
      17.777778
      13.888889
      64.138382

      50%
      170.000000
      7.000000
      9.100000
      73.000000
      66.000000
      30.020000
      22.777778
      18.888889
      83.472995

      75%
      200.000000
      10.000000
      10.000000
      81.000000
      73.000000
      30.130000
      27.222222
      22.777778
      93.564177

      max
      990.000000
      36.000000
      10.000000
      99.000000
      81.000000
      30.790000
      37.222222
      27.222222
      100.000000

```
dfhtown.rolling(window=72).mean()[['TEMP']].plot();
```

The vapor pressure answers will be in units of millibars (mb) or hectoPascals (hPa).
To convert the saturated vapor pressure and/or the actual vapor pressure, the
temperature values must be converted to degrees Celsius (°C) .

```
vp='6.11*(10**((7.5*{T})/(237.3+{T})))'
dfhtown['e']=0.1*dfhtown.eval(vp.format(T='TD')) #converted from hPa to kPa
dfhtown['es']=0.1*dfhtown.eval(vp.format(T='T')) #converted from hPa to kPa
```

Let's verify calculation

```
ax1=dfhtown[['RH']].rolling(window=24*7).mean().plot(style='.')
(dfhtown.e/dfhtown.es*100.).rolling(window=24*7*2).mean().plot(style='.',ax=ax1)
plt.ylabel('Relative Humidity')
plt.title('Comparison of calculated humidity with various methods');
```

Back to apparent temperature (AT)!

```
AT = "-2.7 + 1.04*T + 2.0*e -0.65*v"
```

```
dfhtown['v']=dfhtown.SPD*0.44704 #MPH to m/s
```

```
dfhtown['AT']=dfhtown.eval(AT)
```

```
plt.figure()
dfhtown.eval('AT-T').rolling(window=7*24).mean().plot()
plt.title('Different between apparent temperature and measured temperature')
plt.ylabel('Delta T [deg. C]')
```

Comfortable apparent temperature **22-27** deg. C

https://en.wikipedia.org/wiki/Thermal_comfort

```
ixat = (dfhtown.AT>22) & (dfhtown.AT<27)
```

Comfortable hours to spend time outdoors are between 7am and 8pm

```
dfhtown['hour']=dfhtown.index.map(lambda x: x.hour)
ixhr = (dfhtown.AT>7) & (dfhtown.hour<20)
```

When humidity is above 95% - very high chances of rain! So not good for outdoors...

```
ixrh = dfhtown.RH<95
```

```
dfhtown[ixat & ixhr & ixrh].resample("H").sum().resample('M').AT.count()[:-1]
#.plot()
#plt.ylabel('Total hours comfortable outdoors in a month');
```

    datetime
    2016-04-30    720
    2016-05-31    744
    2016-06-30    720
    2016-07-31    744
    2016-08-31    744
    2016-09-30    720
    2016-10-31    744
    2016-11-30    720
    2016-12-31    744
    2017-01-31    744
    2017-02-28    672
    2017-03-31    744
    Freq: M, Name: AT, dtype: int64

## Deep learning approach

This part is based on the deep learning book "Deep Learning with Python" by Keras Lead François Chollet

https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/6.3-advanced-usage-of-recurrent-neural-networks.ipynb

```
import tensorflow as tf
print("tensorflow version = {}".format(tf.__version__))

import keras
print("keras version = {}".format(keras.__version__))

```

    tensorflow version = 1.12.0
    keras version = 2.2.4

## Transforming the data

The exact formulation of our problem will be the following: given data going as far back as `lookback` timesteps (a timestep is 10 minutes)
and sampled every `steps` timesteps, can we predict the temperature in `delay` timesteps?

We will use the following parameter values:

* `lookback = 360`, i.e. our observations will go back 5 days.
* `steps = 3`, i.e. our observations will be sampled at one data point per hour.
* `delay = 72`, i.e. our targets will be 24 hours in the future.

To get started, we need to do two things:

* Preprocess the data to a format a neural network can ingest. This is easy: the data is already numerical, so we don't need to do any
vectorization. However each timeseries in the data is on a different scale (e.g. temperature is typically between -20 and +30, but
pressure, measured in mbar, is around 1000). So we will normalize each timeseries independently so that they all take small values on a
similar scale.
* Write a Python generator that takes our current array of float data and yields batches of data from the recent past, alongside with a
target temperature in the future. Since the samples in our dataset are highly redundant (e.g. sample `N` and sample `N + 1` will have most
of their timesteps in common), it would be very wasteful to explicitly allocate every sample. Instead, we will generate the samples on the
fly using the original data.

We preprocess the data by subtracting the mean of each timeseries and dividing by the standard deviation. We plan on using the first
200,000 timesteps as training data, so we compute the mean and standard deviation only on this fraction of the data:

```
usedcols = 'DIR	TEMP	SPD	VSB	DEWP	ALT'.split('\t')
float_data = dfhtown[usedcols].dropna()
display(float_data.head(2))
float_data = float_data.values
float_data.shape
```

      DIR
      TEMP
      SPD
      VSB
      DEWP
      ALT

      datetime

      2016-04-01 00:35:00
      330.0
      75.0
      6.0
      9.1
      66.0
      29.66

      2016-04-01 00:55:00
      330.0
      73.0
      3.0
      10.0
      66.0
      29.67

    (18771, 6)

```
mean = float_data.mean(axis=0)
float_data -= mean
std = float_data.std(axis=0)
float_data /= std
```

```
pd.DataFrame(float_data, columns = usedcols).hist()
plt.xlim([-2,2])
plt.legend()
```

```
import seaborn as sns
#plt.matshow(pd.DataFrame(float_data, columns = usedcols).corr())

sns.heatmap(pd.DataFrame(float_data, columns = usedcols).corr(),
            xticklabels=usedcols,
            yticklabels=usedcols)

```

Now here is the data generator that we will use. It yields a tuple `(samples, targets)` where `samples` is one batch of input data and
`targets` is the corresponding array of target temperatures. It takes the following arguments:

* `data`: The original array of floating point data, which we just normalized in the code snippet above.
* `lookback`: How many timesteps back should our input data go.
* `delay`: How many timesteps in the future should our target be.
* `min_index` and `max_index`: Indices in the `data` array that delimit which timesteps to draw from. This is useful for keeping a segment
of the data for validation and another one for testing.
* `shuffle`: Whether to shuffle our samples or draw them in chronological order.
* `batch_size`: The number of samples per batch.
* `step`: The period, in timesteps, at which we sample data. We will set it 6 in order to draw one data point every hour.

```
def generator(data, lookback, delay, min_index, max_index,
              shuffle=False, batch_size=128, step=6):
    if max_index is None:
        max_index = len(data) - delay - 1
    i = min_index + lookback
    while True:
        if shuffle:
            rows = np.random.randint(
                min_index + lookback, max_index, size=batch_size)
        else:
            if i + batch_size >= max_index:
                i = min_index + lookback
            rows = np.arange(i, min(i + batch_size, max_index))
            i += len(rows)

        samples = np.zeros((len(rows),
                           lookback // step,
                           data.shape[-1]))
        targets = np.zeros((len(rows),))
        for j, row in enumerate(rows):
            indices = range(rows[j] - lookback, rows[j], step)
            samples[j] = data[indices]
            targets[j] = data[rows[j] + delay][1]
        yield samples, targets
```

Now let's use our abstract generator function to instantiate three generators, one for training, one for validation and one for testing. Each will look at different temporal segments of the original data: the training generator looks at the first 200,000 timesteps, the validation generator looks at the following 100,000, and the test generator looks at the remainder.

```
float_data.shape
```

    (18771, 6)

```
lookback = 360
step = 3
delay = 72
batch_size = 64
min_index_train = 0
max_index_train = 12000

min_index_valid = 12001
max_index_valid = 18000

def gen_train_val_test():
  train_gen = generator(float_data,
                        lookback=lookback,
                        delay=delay,
                        min_index=min_index_train,
                        max_index=max_index_train,
                        shuffle=True,
                        step=step,
                        batch_size=batch_size)
  val_gen = generator(float_data,
                      lookback=lookback,
                      delay=delay,
                      min_index=min_index_valid,
                      max_index=max_index_valid,
                      step=step,
                      batch_size=batch_size)
  test_gen = generator(float_data,
                       lookback=lookback,
                       delay=delay,
                       min_index=max_index_valid,
                       max_index=None,
                       step=step,
                       batch_size=batch_size)
  return train_gen, val_gen, test_gen

train_gen, val_gen, test_gen = gen_train_val_test()

# This is how many steps to draw from `val_gen`
# in order to see the whole validation set:
val_steps = (max_index_valid - min_index_valid - lookback) // batch_size

# This is how many steps to draw from `test_gen`
# in order to see the whole test set:
test_steps = (len(float_data) - max_index_valid - lookback) // batch_size
```

## A common sense, non-machine learning baseline

In our case, the temperature timeseries can safely be assumed to be continuous (the temperatures tomorrow are likely to be close to the temperatures today) as well as periodical with a daily period. Thus a common sense approach would be to always predict that the temperature 24 hours from now will be equal to the temperature right now. Let's evaluate this approach, using the Mean Absolute Error metric (MAE). Mean Absolute Error is simply equal to:

```
batch_maes = []
predsi = []
targetsi = []
for step in range(val_steps):
    samples, targets = next(val_gen)
    preds = samples[:, -1, 1]
    predslast = samples[:, -2, 1]
    preds = preds + preds - predslast
    mae = np.mean(np.abs(preds - targets))
    batch_maes.append(mae)
    predsi.extend(preds)
    targetsi.extend(targets)
print(np.mean(batch_maes))
plt.plot(predsi[:1000], 'b', linewidth=1, label='prediction');
plt.plot(targetsi[:1000], 'g', linewidth=1, label='target');
plt.legend();
```

## A basic machine learning approach

In the same way that it is useful to establish a common sense baseline before trying machine learning approaches, it is useful to try simple and cheap machine learning models (such as small densely-connected networks) before looking into complicated and computationally expensive models such as RNNs. This is the best way to make sure that any further complexity we throw at the problem later on is legitimate and delivers real benefits.

Here is a simply fully-connected model in which we start by flattening the data, then run it through two Dense layers. Note the lack of activation function on the last Dense layer, which is typical for a regression problem. We use MAE as the loss. Since we are evaluating on the exact same data and with the exact same metric as with our common sense approach, the results will be directly comparable.

```
lookback // step, float_data.shape[-1]
```

    (4, 6)

```
# from keras.models import Sequential
# from keras import layers
# from keras.optimizers import RMSprop

# model = Sequential()
# #model.add(layers.Flatten(input_shape=(lookback // step, float_data.shape[-1])))
# model.add(layers.Flatten(input_shape=(None, float_data.shape[-1])))
# model.add(layers.Dense(32, activation='relu'))
# model.add(layers.Dense(1))

# model.compile(optimizer=RMSprop(), loss='mae')
# history = model.fit_generator(train_gen,
#                               steps_per_epoch=500,
#                               epochs=20,
#                               #validation_data=val_gen,
#                               #validation_steps=val_steps
#                              )
```

```
# import matplotlib.pyplot as plt

# loss = history.history['loss']
# val_loss = history.history['val_loss']

# epochs = range(len(loss))

# plt.figure()

# plt.plot(epochs, loss, 'bo', label='Training loss')
# plt.plot(epochs, val_loss, 'b', label='Validation loss')
# plt.title('Training and validation loss')
# plt.legend()

# plt.show()
```

## Using Recurrent Neural Networks
### Gated Recurrent Unit - GRU cells
### Long Short-Term Memory - LSTM cells

```
from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

train_gen, val_gen, test_gen = gen_train_val_test()

model = Sequential()
#model.add(layers.GRU(16, implementation=2, input_shape=(None, float_data.shape[-1])))
model.add(layers.CuDNNGRU(16, input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=80,
                              epochs=20,
                              validation_data=val_gen,
                              validation_steps=val_steps)
```

    Epoch 1/20
    80/80 [==============================] - 4s 53ms/step - loss: 0.5599 - val_loss: 0.6423
    Epoch 2/20
    80/80 [==============================] - 3s 34ms/step - loss: 0.5089 - val_loss: 0.6319
    Epoch 3/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4931 - val_loss: 0.6019
    Epoch 4/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4921 - val_loss: 0.6151
    Epoch 5/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4785 - val_loss: 0.5905
    Epoch 6/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4696 - val_loss: 0.5925
    Epoch 7/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4587 - val_loss: 0.5977
    Epoch 8/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4596 - val_loss: 0.6064
    Epoch 9/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4414 - val_loss: 0.6027
    Epoch 10/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4455 - val_loss: 0.5918
    Epoch 11/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4400 - val_loss: 0.6055
    Epoch 12/20
    80/80 [==============================] - 3s 34ms/step - loss: 0.4353 - val_loss: 0.6077
    Epoch 13/20
    80/80 [==============================] - 3s 34ms/step - loss: 0.4438 - val_loss: 0.5976
    Epoch 14/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4410 - val_loss: 0.6175
    Epoch 15/20
    80/80 [==============================] - 3s 34ms/step - loss: 0.4300 - val_loss: 0.6130
    Epoch 16/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4284 - val_loss: 0.6249
    Epoch 17/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4225 - val_loss: 0.6348
    Epoch 18/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4215 - val_loss: 0.6311
    Epoch 19/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4149 - val_loss: 0.6473
    Epoch 20/20
    80/80 [==============================] - 3s 33ms/step - loss: 0.4138 - val_loss: 0.6512

```
import matplotlib.pyplot as plt

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
```

```
from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

train_gen, val_gen, test_gen = gen_train_val_test()

model = Sequential()
model.add(layers.GRU(64,
                     #dropout=0.1,
                     recurrent_dropout=0.2,
                     input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=20,
                              epochs=40,
                              validation_data=val_gen,
                              validation_steps=val_steps
                             )
```

    Epoch 1/40
    20/20 [==============================] - 14s 681ms/step - loss: 0.5384 - val_loss: 0.6619
    Epoch 2/40
    20/20 [==============================] - 12s 580ms/step - loss: 0.5094 - val_loss: 0.6319
    Epoch 3/40
    20/20 [==============================] - 12s 579ms/step - loss: 0.4975 - val_loss: 0.6321
    Epoch 4/40
    20/20 [==============================] - 12s 583ms/step - loss: 0.5255 - val_loss: 0.6550
    Epoch 5/40
    20/20 [==============================] - 11s 565ms/step - loss: 0.4929 - val_loss: 0.6330
    Epoch 6/40
    20/20 [==============================] - 11s 568ms/step - loss: 0.4793 - val_loss: 0.6046
    Epoch 7/40
    20/20 [==============================] - 11s 568ms/step - loss: 0.5135 - val_loss: 0.5886
    Epoch 8/40
    20/20 [==============================] - 11s 568ms/step - loss: 0.4978 - val_loss: 0.5772
    Epoch 9/40
    20/20 [==============================] - 11s 569ms/step - loss: 0.4998 - val_loss: 0.5809
    Epoch 10/40
    20/20 [==============================] - 12s 578ms/step - loss: 0.4775 - val_loss: 0.5757
    Epoch 11/40
    20/20 [==============================] - 12s 589ms/step - loss: 0.4688 - val_loss: 0.5736
    Epoch 12/40
    20/20 [==============================] - 12s 575ms/step - loss: 0.4839 - val_loss: 0.5967
    Epoch 13/40
    20/20 [==============================] - 11s 575ms/step - loss: 0.4776 - val_loss: 0.5867
    Epoch 14/40
    20/20 [==============================] - 12s 576ms/step - loss: 0.4865 - val_loss: 0.5834
    Epoch 15/40
    20/20 [==============================] - 12s 579ms/step - loss: 0.4770 - val_loss: 0.5748
    Epoch 16/40
    20/20 [==============================] - 12s 584ms/step - loss: 0.4607 - val_loss: 0.5719
    Epoch 17/40
    20/20 [==============================] - 12s 576ms/step - loss: 0.4675 - val_loss: 0.5597
    Epoch 18/40
    20/20 [==============================] - 12s 581ms/step - loss: 0.4635 - val_loss: 0.5591
    Epoch 19/40
    20/20 [==============================] - 12s 579ms/step - loss: 0.4852 - val_loss: 0.5771
    Epoch 20/40
    20/20 [==============================] - 12s 581ms/step - loss: 0.4883 - val_loss: 0.5506
    Epoch 21/40
    20/20 [==============================] - 12s 576ms/step - loss: 0.4676 - val_loss: 0.5601
    Epoch 22/40
    20/20 [==============================] - 12s 576ms/step - loss: 0.4618 - val_loss: 0.5478
    Epoch 23/40
    20/20 [==============================] - 12s 577ms/step - loss: 0.4768 - val_loss: 0.5490
    Epoch 24/40
    20/20 [==============================] - 12s 575ms/step - loss: 0.4565 - val_loss: 0.5515
    Epoch 25/40
    20/20 [==============================] - 12s 575ms/step - loss: 0.4495 - val_loss: 0.5472
    Epoch 26/40
    20/20 [==============================] - 12s 579ms/step - loss: 0.4497 - val_loss: 0.5440
    Epoch 27/40
    20/20 [==============================] - 12s 579ms/step - loss: 0.4610 - val_loss: 0.5578
    Epoch 28/40
    20/20 [==============================] - 12s 576ms/step - loss: 0.4596 - val_loss: 0.5565
    Epoch 29/40
    20/20 [==============================] - 12s 579ms/step - loss: 0.4537 - val_loss: 0.5612
    Epoch 30/40
    20/20 [==============================] - 12s 584ms/step - loss: 0.4424 - val_loss: 0.5560
    Epoch 31/40
    20/20 [==============================] - 11s 574ms/step - loss: 0.4567 - val_loss: 0.5604
    Epoch 32/40
    20/20 [==============================] - 12s 575ms/step - loss: 0.4421 - val_loss: 0.5516
    Epoch 33/40
    20/20 [==============================] - 12s 576ms/step - loss: 0.4481 - val_loss: 0.5545
    Epoch 34/40
    20/20 [==============================] - 12s 577ms/step - loss: 0.4515 - val_loss: 0.5609
    Epoch 35/40
    20/20 [==============================] - 11s 573ms/step - loss: 0.4476 - val_loss: 0.5535
    Epoch 36/40
    20/20 [==============================] - 11s 575ms/step - loss: 0.4398 - val_loss: 0.5546
    Epoch 37/40
    20/20 [==============================] - 12s 585ms/step - loss: 0.4577 - val_loss: 0.5578
    Epoch 38/40
    20/20 [==============================] - 12s 576ms/step - loss: 0.4358 - val_loss: 0.5625
    Epoch 39/40
    20/20 [==============================] - 11s 567ms/step - loss: 0.4330 - val_loss: 0.5700
    Epoch 40/40
    20/20 [==============================] - 11s 568ms/step - loss: 0.4429 - val_loss: 0.5629

```
import matplotlib.pyplot as plt

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
```

```
model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    gru_10 (GRU)                 (None, 64)                13632
    _________________________________________________________________
    dense_17 (Dense)             (None, 1)                 65
    =================================================================
    Total params: 13,697
    Trainable params: 13,697
    Non-trainable params: 0
    _________________________________________________________________

```
train_gen, val_gen, test_gen = gen_train_val_test()

batch_maes = []
predsi = []
targetsi = []
for step in range(val_steps):
    samples, targets = next(val_gen)
    preds = model.predict(samples)
    mae = np.mean(np.abs(preds - targets))
    #print(mae)
    batch_maes.append(mae)
    predsi.extend(preds)
    targetsi.extend(targets)
print(np.mean(batch_maes))
plt.plot(predsi[:1000], 'b', linewidth=1, label='prediction');
plt.plot(targetsi[:1000], 'g', linewidth=1, label='target');
plt.legend();
```

```

```


# Neuromatch Academy: Week 1, Day 5, Tutorial 2
# Dimensionality Reduction: Principal component analysis

__Content creators:__ Alex Cayco Gajic, John Murray

__Content reviewers:__ Roozbeh Farhoudi, Matt Krause, Spiros Chavlis, Richard Gao, Michael Waskom

---
# Tutorial Objectives

In this notebook we'll learn how to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

Overview:
- Calculate the eigenvectors of the sample covariance matrix.
- Perform PCA by projecting data onto the eigenvectors of the covariance matrix.
- Plot and explore the eigenvalues.

To quickly refresh your knowledge of eigenvalues and eigenvectors, you can watch this [short video](https://www.youtube.com/watch?v=kwA3qM0rm7c) (4 minutes) for a geometrical explanation. For a deeper understanding, this [in-depth video](https://www.youtube.com/watch?v=PFDu9oVAE-g&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab&index=14) (17 minutes) provides an excellent basis and is beautifully illustrated.

```python
# @title Video 1: PCA
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="-f6T9--oM0E", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=-f6T9--oM0E

---
# Setup
Run these cells to get the tutorial started.

```python
# Imports
import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import ipywidgets as widgets  # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper functions

def plot_eigenvalues(evals):
  """
  Plots eigenvalues.

  Args:
      (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """
  plt.figure(figsize=(4, 4))
  plt.plot(np.arange(1, len(evals) + 1), evals, 'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
  plt.xticks(np.arange(1, len(evals) + 1))
  plt.ylim([0, 2.5])

def sort_evals_descending(evals, evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats)    : Vector of eigenvalues
    evectors (numpy array of floats) : Corresponding matrix of eigenvectors
                                        each column corresponds to a different
                                        eigenvalue

  Returns:
    (numpy array of floats)          : Vector of eigenvalues after sorting
    (numpy array of floats)          : Matrix of eigenvectors after sorting
  """

  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:, index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:, 0],
                           1 / np.sqrt(2) * np.array([1, 1]))) > np.pi / 2:
      evectors[:, 0] = -evectors[:, 0]
    if np.arccos(np.matmul(evectors[:, 1],
                           1 / np.sqrt(2) * np.array([-1, 1]))) > np.pi / 2:
      evectors[:, 1] = -evectors[:, 1]
  return evals, evectors

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  scatter plot of their joint activity. The title indicates the sample
  correlation calculated from the data.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(X[:, 0], color='k')
  plt.ylabel('Neuron 1')
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(X[:, 1], color='k')
  plt.xlabel('Sample Number (sorted)')
  plt.ylabel('Neuron 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:, 0], X[:, 1], '.', markerfacecolor=[.5, .5, .5],
           markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:, 0], X[:, 1])[0, 1]))
  plt.show()

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

  Note that samples are sorted in ascending order for the first random
  variable.

  Args:
    var_1 (scalar)                     : variance of the first random variable
    var_2 (scalar)                     : variance of the second random variable
    cov_matrix (numpy array of floats) : desired covariance matrix

  Returns:
    (numpy array of floats)            : samples from the bivariate Gaussian,
                                          with each column corresponding to a
                                          different random variable
  """

  mean = np.array([0, 0])
  X = np.random.multivariate_normal(mean, cov_matrix, size=1000)
  indices_for_sorting = np.argsort(X[:, 0])
  X = X[indices_for_sorting, :]
  return X

def calculate_cov_matrix(var_1, var_2, corr_coef):
  """
  Calculates the covariance matrix based on the variances and
  correlation coefficient.

  Args:
    var_1 (scalar)         :  variance of the first random variable
    var_2 (scalar)         :  variance of the second random variable
    corr_coef (scalar)     :  correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  cov = corr_coef * np.sqrt(var_1 * var_2)
  cov_matrix = np.array([[var_1, cov], [cov, var_2]])
  return cov_matrix

def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats) : arbitrary 2D vector used for new basis

  Returns:
    (numpy array of floats)   : new orthonormal basis columns correspond to
                                basis vectors
  """

  u = u / np.sqrt(u[0] ** 2 + u[1] ** 2)
  w = np.array([-u[1], u[0]])
  W = np.column_stack((u, w))
  return W

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases. Similar to plot_data
  but with colors corresponding to projections onto basis 1 (red) and
  basis 2 (blue).
  The title indicates the sample correlation calculated from the data.

  Note that samples are re-sorted in ascending order for the first random
  variable.

  Args:
    Y (numpy array of floats) : Data matrix in new basis each column
                                corresponds to a different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(Y[:, 0], 'r')
  plt.ylabel('Projection \n basis vector 1')
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(Y[:, 1], 'b')
  plt.xlabel('Projection \n basis vector 1')
  plt.ylabel('Projection \n basis vector 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:, 0], Y[:, 1], '.', color=[.5, .5, .5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:, 0], Y[:, 1])[0, 1]))
  plt.show()

def change_of_basis(X, W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)   : Data matrix expressed in new basis
  """

  Y = np.matmul(X, W)
  return Y

def plot_basis_vectors(X, W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable
    W (numpy array of floats) : Square matrix representing new orthonormal
                                basis each column represents a basis vector

  Returns:
    Nothing.
  """

  plt.figure(figsize=[4, 4])
  plt.plot(X[:, 0], X[:, 1], '.', color=[.5, .5, .5], label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
           label='Basis vector 1')
  plt.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
           label='Basis vector 2')
  plt.legend()
  plt.show()

```

---
# Section 1: Calculate the eigenvectors of the the sample covariance matrix

As we saw in the lecture, PCA represents data in a new orthonormal basis defined by the eigenvectors of the covariance matrix. Remember that in the previous tutorial, we generated bivariate normal data with a specified covariance matrix $\bf \Sigma$, whose $(i,j)$th element is:
\begin{equation}
\Sigma_{ij} = E[ x_i x_j ] - E[ x_i] E[ x_j ] .
\end{equation}
However, in real life we don't have access to this ground-truth covariance matrix. To get around this, we can use the sample covariance matrix, $\bf\hat\Sigma$, which is calculated directly from the data. The $(i,j)$th element of the sample covariance matrix is:
\begin{equation}
 \hat \Sigma_{ij} =  \frac{1}{N_\text{samples}}{\bf x}_i^T {\bf x}_j - \bar {\bf x}_i \bar{\bf x}_j ,
\end{equation}
where ${\bf x}_i = [ x_i(1), x_i(2), \dots,x_i(N_\text{samples})]^T$ is a column vector representing all measurements of neuron $i$, and  $\bar {\bf x}_i$ is the mean of neuron $i$ across samples:
\begin{equation}
\bar {\bf x}_i = \frac{1}{N_\text{samples}} \sum_{k=1}^{N_\text{samples}} x_i(k).
\end{equation}
If we assume that the data has already been mean-subtracted, then we can write the sample covariance matrix in a much simpler matrix form:
\begin{align}
{\bf \hat \Sigma}
&= \frac{1}{N_\text{samples}} {\bf X}^T {\bf X}.
\end{align}

## Exercise 1: Calculation of the covariance matrix

Before calculating the eigenvectors, you must first calculate the sample covariance matrix.

**Steps**
* Complete the function `get_sample_cov_matrix` by first subtracting the sample mean of the data, then calculate $\bf \hat \Sigma$ using the equation above.
* Use `get_data` to generate bivariate normal data, and calculate the sample covariance matrix with your finished `get_sample_cov_matrix`. Compare this estimate to the true covariate matrix using `calculate_cov_matrix`.

```python
help(get_data)
help(calculate_cov_matrix)
```

    Help on function get_data in module __main__:

    get_data(cov_matrix)
        Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

        Note that samples are sorted in ascending order for the first random
        variable.

        Args:
          var_1 (scalar)                     : variance of the first random variable
          var_2 (scalar)                     : variance of the second random variable
          cov_matrix (numpy array of floats) : desired covariance matrix

        Returns:
          (numpy array of floats)            : samples from the bivariate Gaussian,
                                                with each column corresponding to a
                                                different random variable

    Help on function calculate_cov_matrix in module __main__:

    calculate_cov_matrix(var_1, var_2, corr_coef)
        Calculates the covariance matrix based on the variances and
        correlation coefficient.

        Args:
          var_1 (scalar)         :  variance of the first random variable
          var_2 (scalar)         :  variance of the second random variable
          corr_coef (scalar)     :  correlation coefficient

        Returns:
          (numpy array of floats) : covariance matrix

```python
def get_sample_cov_matrix(X):
  """
  Returns the sample covariance matrix of data X

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Covariance matrix
  """

  #################################################
  ## TODO for students: calculate the covariance matrix
  # Fill out function and remove
  raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  #################################################

  # Subtract the mean of X
  X = ...
  # Calculate the covariance matrix (hint: use np.matmul)
  cov_matrix = ...

  return cov_matrix

##########################################################
## TODO for students: generate bivariate Gaussian data
# with variances of 1 and a correlation coefficient of 0.8
# compare the true and sample covariance matrices
##########################################################
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

np.random.seed(2020)  # set random seed
# Uncomment below code to test your function
# cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
# print(cov_matrix)

# X = get_data(cov_matrix)
# sample_cov_matrix = get_sample_cov_matrix(X)
# print(sample_cov_matrix)
```

SAMPLE OUTPUT
```
[[1.  0.8]
 [0.8 1. ]]
[[0.99315313 0.82347589]
 [0.82347589 1.01281397]]
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial2_Solution_a98b14b7.py)

## Exercise 2: Eigenvectors of the Covariance matrix

Next you will calculate the eigenvectors of the covariance matrix. Plot them along with the data to check that they align with the geometry of the data.

**Steps:**
* Calculate the eigenvalues and eigenvectors of the sample covariance matrix. (**Hint:** use `np.linalg.eigh`, which finds the eigenvalues of a symmetric matrix).
* Use the provided code to sort the eigenvalues in descending order.
* Plot the eigenvectors on a scatter plot of the data, using the function `plot_basis_vectors`.

```python
help(sort_evals_descending)
help(plot_basis_vectors)
```

    Help on function sort_evals_descending in module __main__:

    sort_evals_descending(evals, evectors)
        Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
        eigenvectors to be in first two quadrants (if 2D).

        Args:
          evals (numpy array of floats)    : Vector of eigenvalues
          evectors (numpy array of floats) : Corresponding matrix of eigenvectors
                                              each column corresponds to a different
                                              eigenvalue

        Returns:
          (numpy array of floats)          : Vector of eigenvalues after sorting
          (numpy array of floats)          : Matrix of eigenvectors after sorting

    Help on function plot_basis_vectors in module __main__:

    plot_basis_vectors(X, W)
        Plots bivariate data as well as new basis vectors.

        Args:
          X (numpy array of floats) : Data matrix each column corresponds to a
                                      different random variable
          W (numpy array of floats) : Square matrix representing new orthonormal
                                      basis each column represents a basis vector

        Returns:
          Nothing.

```python
#################################################
## TO DO for students: Calculate and sort the eigenvalues in descending order
#################################################
# Calculate the eigenvalues and eigenvectors
# evals, evectors = ...
# Sort the eigenvalues in descending order
# evals, evectors = ...

# plot_basis_vectors(X, evectors)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial2_Solution_ca961dca.py)

*Example output:*

---
# Section 2: Perform PCA by projecting data onto the eigenvectors

To perform PCA, we will project the data onto the eigenvectors of the covariance matrix, i.e.:
\begin{equation}
\bf S = X W
\end{equation}
where $\bf S$ is an $N_\text{samples} \times N$ matrix representing the projected data (also called *scores*), and $W$ is an $N\times N$ orthogonal matrix, each of whose columns represents the eigenvectors of the covariance matrix (also called *weights* or *loadings*).

## Exercise 3: PCA implementation

You will now perform PCA on the data using the intuition you have developed so far. Fill in the function below to carry out the steps to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

**Steps:**
* First subtract the mean.
* Then calculate the sample covariance matrix.
* Then find the eigenvalues and eigenvectors and sort them in descending order.
* Finally project the mean-centered data onto the eigenvectors.

```python
help(change_of_basis)
help(plot_data_new_basis)
```

    Help on function change_of_basis in module __main__:

    change_of_basis(X, W)
        Projects data onto a new basis.

        Args:
          X (numpy array of floats) : Data matrix each column corresponding to a
                                      different random variable
          W (numpy array of floats) : new orthonormal basis columns correspond to
                                      basis vectors

        Returns:
          (numpy array of floats)   : Data matrix expressed in new basis

    Help on function plot_data_new_basis in module __main__:

    plot_data_new_basis(Y)
        Plots bivariate data after transformation to new bases. Similar to plot_data
        but with colors corresponding to projections onto basis 1 (red) and
        basis 2 (blue).
        The title indicates the sample correlation calculated from the data.

        Note that samples are re-sorted in ascending order for the first random
        variable.

        Args:
          Y (numpy array of floats) : Data matrix in new basis each column
                                      corresponds to a different random variable

        Returns:
          Nothing.

```python
def pca(X):
  """
  Sorts eigenvalues and eigenvectors in decreasing order.

  Args:
    X (numpy array of floats): Data matrix each column corresponds to a
                               different random variable

  Returns:
    (numpy array of floats)  : Data projected onto the new basis
    (numpy array of floats)  : Vector of eigenvalues
    (numpy array of floats)  : Corresponding matrix of eigenvectors

  """

  #################################################
  ## TODO for students: calculate the covariance matrix
  # Fill out function and remove
  raise NotImplementedError("Student excercise: sort eigenvalues/eigenvectors!")
  #################################################

  # Subtract the mean of X
  X = ...
  # Calculate the sample covariance matrix
  cov_matrix = ...
  # Calculate the eigenvalues and eigenvectors
  evals, evectors = ...
  # Sort the eigenvalues in descending order
  evals, evectors = ...
  # Project the data onto the new eigenvector basis
  score = ...

  return score, evectors, evals

#################################################
## TODO for students: Call the function to calculate the eigenvectors/eigenvalues
#################################################

# Perform PCA on the data matrix X
# score, evectors, evals = ...
# Plot the data projected into the new basis
# plot_data_new_basis(score)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial2_Solution_d65c4e04.py)

*Example output:*

## Plot and explore the eigenvalues

Finally, we will examine the eigenvalues of the covariance matrix. Remember that each eigenvalue describes the variance of the data projected onto its corresponding eigenvector. This is an important concept because it allows us to rank the PCA basis vectors based on how much variance each one can capture. First run the code below to plot the eigenvalues (sometimes called the "scree plot"). Which eigenvalue is larger?

```python
plot_eigenvalues(evals)
```

## Interactive Demo: Exploration of the correlation coefficient

Run the following cell and use the slider to change the correlation coefficient in the data. You should see the scree plot and the plot of basis vectors update.

**Questions:**
* What happens to the eigenvalues as you change the correlation coefficient?
* Can you find a value for which both eigenvalues are equal?
* Can you find a value for which only one eigenvalue is nonzero?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def refresh(corr_coef=.8):
  cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
  X = get_data(cov_matrix)
  score, evectors, evals = pca(X)
  plot_eigenvalues(evals)
  plot_basis_vectors(X, evectors)

_ = widgets.interact(refresh, corr_coef=(-1, 1, .1))
```

---
# Summary
- In this tutorial, we learned that goal of PCA is to find an orthonormal basis capturing the directions of maximum variance of the data. More precisely, the $i$th basis vector is the direction that maximizes the projected variance, while being orthogonal to all previous basis vectors. Mathematically, these basis vectors are the eigenvectors of the covariance matrix (also called *loadings*).
- PCA also has the useful property that the projected data (*scores*) are uncorrelated.
- The projected variance along each basis vector is given by its corresponding eigenvalue. This is important because it allows us rank the "importance" of each basis vector in terms of how much of the data variability it explains. An eigenvalue of zero means there is no variation along that direction so it can be dropped without losing any information about the original data.
- In the next tutorial, we will use this property to reduce the dimensionality of high-dimensional data.

---
# Bonus: Mathematical basis of PCA properties

```python
# @title Video 2: Properties of PCA
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="p56UrMRt6-U", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=p56UrMRt6-U


# MLSS2019: Bayesian Deep Learning

In this tutorial we will learn what basic building blocks are needed
to endow (deep) neural networks with uncertainty estimates.

The plan of the tutorial
1. [Setup and imports](#Setup-and-imports)
2. [Easy uncertainty in networks](#Easy-uncertainty-in-networks)
   1. [Bayesification via dropout and weight decay](#Bayesification-via-dropout-and-weight-decay)
   2. [Implementing function sampling with the DropoutLinear Layer](#Implementing-function-sampling-with-the-DropoutLinear-Layer)
   3. [Implementing-DropoutLinear](#Implementing-DropoutLinear)
   4. [Comparing sample functions to point-estimates](#Comparing-sample-functions-to-point-estimates)
3. [(optional) Dropout $2$-d Convolutional layer](#(optional)-Dropout-$2$-d-Convolutional-layer)
4. [(optional) A brief reminder on Bayesian and Variational Inference](#(optional)-A-brief-reminder-on-Bayesian-and-Variational-Inference)

**(note)**
* to view documentation on something  type in `something?` (with one question mark)
* to view code of something type in `something??` (with two question marks).

## Setup and imports

In this section we import necessary modules and functions and
define the computational device.

First, we install some boilerplate service code for this tutorial.

```python
!pip install -q --upgrade git+https://github.com/mlss-skoltech/tutorials.git#subdirectory=bayesian_deep_learning
```

Next, numpy for computing, matplotlib for plotting and tqdm for progress bars.

```python
import tqdm
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt
```

For deep learning stuff will be using [pytorch](https://pytorch.org/).

If you are unfamiliar with it, it is basically like `numpy` with autograd,
stricter data type enforcement, native GPU support, and tools for building
training and serializing models.

There are good introductory tutorials on `pytorch`, like this
[one](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html).

```python
import torch
import torch.nn.functional as F

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
```

We will need some functionality from scikit

```python
from sklearn.metrics import confusion_matrix
```

Next we import the boilerplate code.

* a procedure that implements a minibatch SGD **fit** loop
* a function, that **evaluates** the model on the provided dataset

```python
from mlss2019bdl import fit
```

```python
# pseudocode
def fit(model, dataset, criterion, ...):
    for epoch in epochs:
        for batch in dataset:
            loss = criterion(model, batch)  # forward pass

            grad = loss.backward()          # gradient via back propagation

            adam_step(grad)
```

```python
from mlss2019bdl import predict
```

```python
# pseudocode
def predict(model, dataset, ...):
    for input_batch in dataset:
        output.append(model(input_batch))  # forward pass

    return concatenate(output)
```

## Easy uncertainty in networks

Generate the initial small dataset $S_0 = (x_i, y_i)_{i=1}^{m_0}$
with $y_i = g(x_i)$, $x_i$ on a regular-spaced grid, and $
g
    \colon \mathbb{R} \to \mathbb{R}
    \colon x \mapsto \tfrac{x^2}4 + \sin \frac\pi2 x
$.

```python
from mlss2019bdl import dataset_from_numpy

X_train = np.linspace(-6.0, +6.0, num=20)[:, np.newaxis]
y_train = np.sin(X_train * np.pi / 2) + 0.25 * X_train**2

train = dataset_from_numpy(X_train, y_train, device=device)
```

```python
X_domain = np.linspace(-10., +10., num=251)[:, np.newaxis]

domain = dataset_from_numpy(X_domain, device=device)
```

Suppose we have the following model: a 3-layer fully connected
network with LeakyReLU activations.

```python
from torch.nn import Linear, Sequential
from torch.nn import LeakyReLU

model = Sequential(
    Linear(1, 512, bias=True),
    LeakyReLU(),

    Linear(512, 512, bias=True),
    LeakyReLU(),

    Linear(512, 1, bias=True),
)

model.to(device)
```

We fit our model on `train` using MSE loss and $\ell_2$ penalty on
weights (`weight_decay`):
$$
    \tfrac1{2 m} \|f_\omega(x) - y\|_2^2 + \lambda \|\omega\|_2^2
    \,, $$
where $\omega$ are all the learnable parameters of the network $f_\omega$.

Fit, ...

```python
fit(model, train, criterion="mse", n_epochs=2000, verbose=True, weight_decay=1e-3)
```

..., compute the predictions, ...

```python
y_pred = predict(model, domain)
```

..., and plot them.

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 5))

ax.scatter(X_train, y_train, c="black", s=40, label="train")

ax.plot(X_domain, y_pred.numpy(), c="C0", lw=2, label="prediction")

plt.legend();
```

This model seems to fit the train set adequately well. However, there is no
way to assess how confident this model is with respect to its predictions.
Indeed, the prediction $\hat{y}_x = f_\omega(x)$ is is a deterministic function
of the input $x$ and the learnt parameters $\omega$.

### `Bayesification` via dropout and weight decay

One inexpensive way to make any network into a stochastic function of its
input is to add dropout before any parameterized layer like `linear`
or `convolutional`, [Hinton et al. 2012](https://arxiv.org/abs/1207.0580).
Essentially, dropout applies a Bernoulli mask to the features of the input.

In [Gal, Y. (2016)](http://www.cs.ox.ac.uk/people/yarin.gal/website/thesis/thesis.pdf)
it has been shown that a simple, somewhat ad-hoc approach of
adding uncertainty quantification to networks through dropout,
coupled with $\ell_2$ weight penalty, is a special case of Variational Inference.

For input
$
    x\in \mathbb{R}^{[\mathrm{in}]}
$ the dropout layer acts like this:

$$
    y_j = x_j \, m_j
    \,, $$

where $m\in \mathbb{R}^{[\mathrm{in}]}$ with $
m_j \sim \pi_p(m_j)
    = \mathcal{Ber}\bigl(\bigl\{0, \tfrac1{1-p}\bigr\}, 1-p\bigr)
$,
i.e. equals $\tfrac1{1-p}$ with probability $1-p$ and $0$ otherwise.

#### (task) Always Active Dropout

Useful methods:
* `troch.rand(d1, ..., dn)` -- draw $d_1\times \ldots \times d_n$ tensor of uniform rv-s
* `troch.rand_like(other)` -- draw a tensor of uniform rv-s with the shape, data type and device as `other`

* `troch.bernoulli(pi)` -- draw tensor $t$ with independent $
t_\alpha \sim \mathcal{Ber}\bigl(\{0, 1\}, \pi_\alpha\bigr)
$ for each index $\alpha$
* `torch.full((d1, ..., dn), v)` -- a $d_1\times \ldots \times d_n$ tensor with the same value $v$

* `Tensor.to(other)` -- assume move `Tensor` to the device of the `other` and cast to its data type.

```python
from torch.nn import Module

class ActiveDropout(Module):
    # all building blocks of networks are inherited from Module!

    def __init__(self, p=0.5):
        super().__init__()  # init the base class

        self.p = p

    def forward(self, input):
        ## Exercise: implement feature dropout on input
        #  self.p - contains the specified dropout rate

        pass
```

#### (task) Rebuilding the model

Let's recreate the model above with this freshly minted dropout layer.
Then fit and plot it's prediction uncertainty due to forward pass stochasticity.

```python
def build_model(p=0.5):
    """Build a model with dropout layers' rate set to `p`."""

    return Sequential(
        ## Exercise: Use `ActiveDropout` before linear layers of our
        #  first network. Note that dropping out inputs is not a good idea

        # pass
    )
```

```python
model = build_model(p=0.5)

model.to(device)

fit(model, train, criterion="mse", n_epochs=2000, verbose=True,
    weight_decay=1e-3)
```

#### Sampling the random output

Let's take the test sample $\tilde{S} = (\tilde{x}_i)_{i=1}^m \in \mathcal{X}$
and repeat the stochastic forward pass $B$ times at each $x\in \tilde{S}$:

* for $b = 1 .. B$ do:

  1. draw $y_{bi} \sim f_\omega(\tilde{x}_i)$ for $i = 1 .. m$.

```python
def point_estimate(model, dataset, n_samples=1, verbose=False):
    """Draw pointwise samples with stochastic forward pass."""

    outputs = []
    for sample in tqdm.tqdm(range(n_samples), disable=not verbose):

        outputs.append(predict(model, dataset))

    return torch.stack(outputs, dim=0)

samples = point_estimate(model, domain, n_samples=101, verbose=True)
```

The approximate $95\%$ confidence band of predictions is...

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 5))
ax.scatter(X_train, y_train, c="black", s=40, label="train")

mean, std = samples.mean(dim=0).numpy(), samples.std(dim=0).numpy()
ax.plot(X_domain, mean + 1.96 * std, c="k")
ax.plot(X_domain, mean - 1.96 * std, c="k");
```

### Implementing function sampling with the DropoutLinear Layer

Let's inspect the draws $y_{bi}$ as $B$ functional samples:
$(x_i, y_{bi})_{i=1}^m$ - the $b$-th sample path. Below we
plot $5$ random paths.

```python
samples = point_estimate(model, domain, n_samples=101, verbose=True)

fig, ax = plt.subplots(1, 1, figsize=(12, 5))

ax.scatter(X_train, y_train, c="black", s=40, label="train")
ax.plot(X_domain[:, 0], samples[:5, :, 0].numpy().T, c="C0", lw=1, alpha=0.25);
```

It is clear that they are very erratic!

Computing stochastic forward passes with a new mask each time is equivalent
to drawing new **independent** prediction from for each point $x\in \tilde{S}$,
without considering that, in fact, at adjacent points the predictions should
be correlated. If we were interested in uncertainty at some particular point,
this would be okay: **fast and simple**.

However, if we are interested in the uncertainty of an integral **path-dependent**
measure of the whole estimated function, or are doing **optimization** of
the unknown true function taking estimation uncertainty into account, then
this clearly erratic behaviour of paths is undesirable. Ex. see
[blog: Gal, Y. 2016](http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_2248.html)

We need to implement some extra functionality on top of `pytorch`,
in order to draw realizations from the induced distribution over
functions, defined by a network, i.e. $
\bigl\{
    f_\omega\colon \mathcal{X}\to\mathcal{Y}
\bigr\}_{\omega \sim q(\omega)}
$
where $q(\omega)$ is a distribution over the parameters.

One of the design approaches is to allow layers
to cache random draws of their parameters for reuse
in all subsequent forward passes, until this is no
longer needed.

#### Freeze/unfreeze interface

This is a base **trait-class** `FreezableWeight` that adds interface
for freezing and unfreezing layer's random **weight** parameter.

```python
class FreezableWeight(Module):
    def __init__(self):
        super().__init__()
        self.unfreeze()

    def unfreeze(self):
        self.register_buffer("frozen_weight", None)

    def is_frozen(self):
        """Check if a frozen weight is available."""
        return isinstance(self.frozen_weight, torch.Tensor)

    def freeze(self):
        """Sample from the parameter distribution and freeze."""
        raise NotImplementedError()
```

Next, we declare a pair of functions:
* `freeze()` instructs each compatible layer of the model to **sample and freeze** its randomness
* `unfreeze()` requests the layers to **undo** this

```python
def freeze(model):
    for layer in model.modules():
        if isinstance(layer, FreezableWeight):
            layer.freeze()

    return model
```

```python
def unfreeze(model):
    for layer in model.modules():
        if isinstance(layer, FreezableWeight):
            layer.unfreeze()

    return model
```

#### (task) Sampling realizations

The algorithm to sample a random function is:
* for $b = 1... B$ do:

  1. draw an independent realization $f_b\colon \mathcal{X} \to \mathcal{Y}$
  with from the process $\{f_\omega\}_{\omega \sim q(\omega)}$
  2. get $\hat{y}_{bi} = f_b(\tilde{x}_i)$ for $i=1 .. m$

```python
def sample_function(model, dataset, n_samples=1, verbose=False):
    """Draw a realization of a random function."""

    ## Exercise: code a function similar to `point_estimate()`,
    ##  that collects the predictions from `frozen` models. Don't
    ##  forget to unfreeze before returning.

    pass
```

**(note)** although the internal loop in both functions looks
similar they, conceptually the functions differ:

```python
def point_estimate(f, S):
    for x in S:
        for w from f.q:  # different w for different x
            yield f(x, w)

def sample_function(f, S):
    for w from f.q:
        for x in S:      # same w for different x (thanks to freeze)
            yield f(x, w)
```

### Implementing `DropoutLinear`

Now we will merge `ActiveDropout` and `Linear` layers into one, which

1. (on forward pass) **drops out** the inputs, if necessary, and **applies** the linear (affine) transform
2. (on freeze) **randomly zeros** columns in a copy of the the weight matrix $W$

Preferably, we will try to preserve interface, so that the resulting
object is backwards compatible with `Linear`.

This way we would be able to draw realizations from the induced
distribution over functions defined by the network $
\bigl\{
    f_\omega\colon \mathcal{X}\to\mathcal{Y}
\bigr\}_{\omega \sim q(\omega)}
$
where $q(\omega)$ a distribution over the network parameters.

#### (task) Fused dropout-linear operation

On the inputs into a linear layer dropout acts like this: for input
$
    x\in \mathbb{R}^{[\mathrm{in}]}
$ and layer weights $
    W\in \mathbb{R}^{[\mathrm{out}] \times [\mathrm{in}]}
$
and bias $
    b\in \mathbb{R}^{[\mathrm{out}]}
$ the resulting effect is

$$
    \tilde{x} = x \odot m
    \,, \\
    y = \tilde{x} W^\top + b
%     = b + \sum_i x_i m_i W_i
    \,, $$

where $\odot$ is the elementwise product and $m\in \mathbb{R}^{[\mathrm{in}]}$
with $m_j \sim \pi_p(m_j) = \mathcal{Ber}\bigl(\bigl\{0, \tfrac1{1-p}\bigr\}, 1-p\bigr)$,
i.e. equals $\tfrac1{1-p}$ with probability $1-p$ and $0$ otherwise.

Let
$
    x\in \mathbb{R}^{[\mathrm{in}]}
$, $
    W\in \mathbb{R}^{[\mathrm{out}] \times [\mathrm{in}]}
$
and $
    b\in \mathbb{R}^{[\mathrm{out}]}
$. Let's use the following `torch`'s functions:

* `F.dropout(x, p, on/off)` -- independent Bernoulli dropout $x\mapsto x\odot m$
  for $m\sim \mathcal{Ber}\bigl(\bigl\{0, \tfrac1{1-p}\bigr\}, 1-p\bigr)$

* `F.linear(x, W, b)` -- affine transformation $x \mapsto x W^\top + b$

**(note)** the `.weight` of a linear layer in `pytorch` is an $
{
    [\mathrm{out}]
    \times [\mathrm{in}]
}
$ matrix.

```python
def DropoutLinear_forward(self, input):
    ## Exercise: If not frozen, then apply always active dropout,
    #  then linear transformation. If frozen, apply the transform
    #  using the frozen weight

    pass
```

#### Parameter freezer for our custom layer

For input
$
    x\in \mathbb{R}^{[\mathrm{in}]}
$ and a layer parameters $
    W\in \mathbb{R}^{[\mathrm{out}] \times [\mathrm{in}]}
$
and $
    b\in \mathbb{R}^{[\mathrm{out}]}
$ the effect in `DropoutLinear` is

$$
    y_j
        = \bigl[(x \odot m) W^\top + b\bigr]_j
        = b_j + \sum_i x_i m_i W_{ji}
        = b_j + \sum_i x_i \breve{W}_{ji}
    \,, $$

where the each column of $\breve{W}_i$ is, independently, either
$\mathbf{0} \in \mathbb{R}^{[\mathrm{out}]}$ with probability $p$ or
some (learnable) vector in $\mathbb{R}^{[\mathrm{out}]}$

$$
    \breve{W}_i \sim
\begin{cases}
    \mathbf{0}
        & \text{ w. prob } p \,, \\
    \tfrac1{1-p} M_i
        & \text{ w. prob } 1-p \,.
\end{cases}
$$

Thus the multiplicative effect of the random mask $m$ on $x$ can be
equivalently seen as a random **on/off** switch effect on the
**columns** of the matrix $W$.

```python
def DropoutLinear_freeze(self):
    """Apply dropout with rate `p` to columns of `weight` and freeze it."""
    # we leverage torch's broadcasting semantics and draw a one-row
    #  mask binary mask, that we later multiply the weight by.

    # let's draw the new weight
    with torch.no_grad():
        prob = torch.full_like(self.weight[:1, :], 1 - self.p)
        feature_mask = torch.bernoulli(prob) / prob

        frozen_weight = self.weight * feature_mask

    # and store it
    self.register_buffer("frozen_weight", frozen_weight)
```

Assemble the blocks into a layer

```python
class DropoutLinear(Linear, FreezableWeight):
    """Linear layer with dropout on inputs."""
    def __init__(self, in_features, out_features, bias=True, p=0.5):
        super().__init__(in_features, out_features, bias=bias)

        self.p = p

    forward = DropoutLinear_forward

    freeze = DropoutLinear_freeze
```

### Comparing sample functions to point-estimates

Let's rewrite the model builder function:

```python
def build_model(p=0.5):
    """Build a model with the custom layer and dropout rate set to `p`."""

    return Sequential(
        ## Exercise: Plug-in `DropoutLinear` layer into our second network.

        # pass
    )
```

Let's create a new instance and retrain the model.

```python
model = build_model(p=0.5)
model.to(device)

fit(model, train, criterion="mse", n_epochs=2000, verbose=True, weight_decay=1e-3)
```

... and obtain two estimates: pointwise and functional.

```python
samples_pe = point_estimate(model, domain, n_samples=51, verbose=True)
samples_sf = sample_function(model, domain, n_samples=51, verbose=True)

samples_pe.shape, samples_sf.shape
```

```python
(torch.Size([51, 251, 1]), torch.Size([51, 251, 1]))
```

Let's compare **point estimates**
with **function sampling**.

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 5))

ax.plot(X_domain[:, 0], samples_pe[:10, :, 0].numpy().T,
        c="C1", lw=1, alpha=0.5)

ax.plot(X_domain[:, 0], samples_sf[:10, :, 0].numpy().T,
        c="C0", lw=2, alpha=0.5)

ax.scatter(X_train, y_train, c="black", s=40,
           label="train", zorder=+10);
```

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 5))

ax.scatter(X_train, y_train, c="black", s=40, label="train")

mean, std = samples_sf.mean(dim=0).numpy(), samples_sf.std(dim=0).numpy()
ax.plot(X_domain, mean + 1.96 * std, c="C0")
ax.plot(X_domain, mean - 1.96 * std, c="C0");

mean, std = samples_pe.mean(dim=0).numpy(), samples_pe.std(dim=0).numpy()
ax.plot(X_domain, mean + 1.96 * std, c="C1")
ax.plot(X_domain, mean - 1.96 * std, c="C1");
```

Pros of `point-estimate`:
* uses stochastic forward passes -- no need to for extra code and classes

Cons of `point-estimate`:
* samples from the predictive distribution at adjacent inputs are independent

**(note)**
The parameter distribution of the layer we've built is

$$
    q(\omega\mid \theta)
        = \prod_i q(\omega_i\mid \theta_i)
        = \prod_i \bigl\{
            p \delta_{\mathbf{0}} (\omega_i)
            + (1 - p) \delta_{\tfrac1{1-p} \theta_i}(\omega_i)
        \bigr\}
    \,, $$

where $\omega_i$ is the $i$-th column of $\omega$, $\delta_a$ is a
**point-mass** distribution at $a$, and $\theta$ is the learnt
approximate posterior mean of $\omega$.

Under benign assumptions and certain relaxations
[Gal, Y. 2016 (eq. (6.3) p.109, Prop. 4 p.149)](http://www.cs.ox.ac.uk/people/yarin.gal/website/thesis/thesis.pdf)
has shown that a deep network with dropout rate $p$
and $\ell_2$ weight penalty (`weight_decay`) performs (doubly)
**stochastic variational inference** with the following stochastic
approximate **evidence lower bound**: for the dataset of size $N$
and random batches $B$ of size $\lvert B \rvert = m$

$$
    \frac1{N} \underbrace{
        \mathbb{E}_{\omega\sim q(\omega\mid \theta)} \log p(D \mid \omega)
        - KL\bigl(q(\omega\mid \theta) \big\| \pi(\omega) \bigr)
    }_{\tfrac1{N}  ELBO(\theta)}
    \approx \frac1{\lvert B \rvert}
        \sum_{i\in B} \log p(y_i \mid x_i, \omega^{(1)}_i, \ldots, \omega^{(L)}_i)
        - \sum_{l=1}^L
            \frac{1-p^{(l)}}{2 s^2 N} \|\theta^{(l)}\|_2^2
%             - [\mathrm{in}_{(l)}] \, \mathbb{H}(\mathcal{Ber}(p^{(l)}))
%         + \mathrm{const}
\,, $$
where $\omega_i^{(l)}$ are independently drawn from $q(\omega \mid \theta)$
(one random draw per element in $B$) and $s^2$ is the prior variance.

Thus `weight_decay` should be decreasing with $p$ and $N$:
$$ \lambda = \frac{1-p}{2 s^2 N} \,. $$

#### Question(s) (to ponder in your spare time)

* what happens to the confidence bands, when you increase the number
  of path-wise and pointwise samples?

* what will happen if you change the dropout rate $p$ and keep `n_epochs` at 2000?

* what happens if for $p=\tfrac12$ we use much less `n_epochs`?

* how does different settings of `weight_decay` affect the bands?

Try to rebuild the model with different $p \in (0, 1)$ using `build_model(p)`, use
`fit(..., n_epochs=...)`, and then plot the predictive bands.

```python
from mlss2019bdl.plotting import plot1d_bands

# model = fit(build_model(p=...), train, n_epochs=..., weight_decay=..., criterion="mse")
# plot1d_bands(X_domain, sample_function(model, domain, n_samples=101), c="C0")
```

```python
pass
```

### (optional) Dropout $2$-d Convolutional layer

Typically, in convolutional neural networks the dropout acts upon the feature
(channel) information and not on the spatial dimensions. Thus entire channels
are dropped out and for $
    x \in \mathbb{R}^{
        [\mathrm{in}]
        \times h
        \times w}
$ and $
    y \in \mathbb{R}^{
        [\mathrm{out}]
        \times h'
        \times w'}
$ the full effect of the `Dropout+Conv2d` layer is

$$
    y_{lij} = ((x \odot m) \ast W_l)_{ij} + b_l
        = b_l + \sum_k \sum_{pq} x_{k i_p j_q} m_k W_{lkpq}
    \,, \tag{conv-2d} $$

where i.i.d $m_k \sim \mathcal{Ber}\bigl(\bigl\{0, \tfrac1{1-p}\bigr\}, 1-p\bigr)$,
and indices $i_p$ and $j_q$ represent the spatial location in $x$ that correspond
to the $p$ and $q$ elements in the kernel $
    W\in \mathbb{R}^{
        [\mathrm{out}]
        \times [\mathrm{in}]
        \times h
        \times w}
$ relative to $(i, j)$ coordinates in $y$.
The exact values of $i_p$ and $j_q$ depend on the configuration of the
convolutional layer, e.g. stride, kernel size and dilation.

**(note)** Informative illustrations on the effects of convolution
parameters can be found in [Convolution arithmetic](https://github.com/vdumoulin/conv_arithmetic)
repo.

## (optional) A brief reminder on Bayesian and Variational Inference

Bayesian Inference is a principled framework of reasoning about uncertainty.

In Bayesian Inference (**BI**) we *assume* that the observation
data $D$ follows a *model* $m$ with data generating distribution
$p(D\mid m, \omega)$ *governed by unknown parameters* $\omega$.
The goal of **BI** is to reason about the model and/or its parameters,
and new data given the observed data $D$ and our assumptions, i.e
to seek the **posterior** parameter and predictive distributions:

$$\begin{align}
    p(d \mid D, m)
        % &= \mathbb{E}_{
        %     \omega \sim p(\omega \mid D, m)
        % } p(d \mid D, \omega, m)
        &= \int p(d \mid D, \omega, m) p(\omega \mid D, m) d\omega
    \,, \\
    p(\omega \mid D, m)
        &= \frac{p(D\mid \omega, m) \, \pi(\omega \mid m)}{p(D\mid m)}
    \,.
\end{align}
$$

* the **prior** distribution $\pi(\omega \mid m)$ reflects our belief
  before having made the observations

* the data distribution $p(D \mid \omega, m)$ reflects our assumptions
  about the data generating process, and determines the parameter
  **likelihood** (Gaussian, Categorical, Poisson)

Unless the distributions and likelihoods are conjugate, posterior in
Bayesian inference is typically intractable and it is common to resort
to **Variational Inference** or **Monte Carlo** approximations.

This key idea of this approach is to seek an approximation $q(\omega)$
to the intractable posterior $p(\omega \mid D, m)$, via a variational
optimization problem over some tractable family of distributions $\mathcal{Q}$:

$$
    q^*(\omega)
        \in \arg \min_{q\in \mathcal{Q}} \mathrm{KL}(q(\omega) \| p(\omega \mid D, m))
    \,, $$

where the Kullback-Leibler divergence between $P$ and $Q$ ($P\ll Q$)
with densities $p$ and $q$, respectively, is given by

$$
    \mathrm{KL}(q(\omega) \| p(\omega))
%         = \mathbb{E}_{\omega \sim Q} \log \tfrac{dQ}{dP}(\omega)
        = \mathbb{E}_{\omega \sim q(\omega)}
            \log \tfrac{q(\omega)}{p(\omega)}
    \,. \tag{kl-div} $$

Note that the family of variational approximations $\mathcal{Q}$ can be
structured **arbitrarily**: point-mass, products, mixture, dependent on
input, having mixed hierarchical structure, -- any valid distribution.

Although computing the divergence w.r.t. the unknown posterior
is still hard and intractable, it is possible to do away with it
through the following identity, which is based on the Bayes rule.

For **any** $q(\omega) \ll p(\omega \mid D; \phi)$ and any model $m$

$$
\begin{align}
    \overbrace{
        \log p(D \mid m)
    }^{\text{evidence}}
        &= \overbrace{
            \mathbb{E}_{\omega \sim q} \log p(D\mid \omega, m)
        }^{\text{expected conditional likelihood}}
        - \overbrace{
            \mathrm{KL}(q(\omega)\| \pi(\omega \mid m))
        }^{\text{proximity to prior belief}}
        \\
        &+ \underbrace{
            \mathrm{KL}(q(\omega)\| p(\omega \mid D, m))
        }_{\text{posterior approximation}}
\end{align}
    \,. \tag{master-identity}
$$

Instead of minimizing the divergence of the approximation from the posterior,
we maximization the **Evidence Lower Bound** with respect to $q(\omega)$:

$$
    q^* \in
    \arg\max_{q\in Q}
        \mathcal{L}(q) =
            \mathbb{E}_{\omega \sim q} \log p(D\mid \omega, m)
            - \mathrm{KL}(q(\omega)\| \pi(\omega \mid m))
    \,. \tag{max-ELBO} $$

* the expected $\log$-likelihood favours $q$ that place their mass on
parameters $\omega$ that explain $D$ under the specified model $m$.

* the negative KL-divergence discourages the approximation $q$
from straying too far away from to the prior belief $\pi$ under $m$.

We typically assume that the following setup (conditioning on model $m$ is omitted):
* the likelihood factorizes $
p(D \mid \omega)
    = \prod_i p(y_i, x_i \mid \omega)
    \propto \prod_i p(y_i \mid x_i, \omega)
$
for $D = (x_i, y_i)_{i=1}^N$

* the approximation is parameterized by $\theta$: $q(\omega\mid \theta)$

* the prior on $\omega$ itself depends on hyper-parameters $\lambda$, that
  can be fixed, or variable ($\pi(\omega \mid \lambda)$).

In this case the variational objective (evidence lower bound)

$$
    \log p(D\mid \lambda )
        \geq \mathcal{L}(\theta, \lambda)
            = \mathbb{E}_{\omega \sim q(\omega \mid \theta)}
                \sum_i \log p_\phi(y_i \mid x_i, \omega)
            - KL(q(\omega \mid \theta) \| \pi(\omega \mid \lambda))
    $$

is maximized with respect to $\theta$ (to approximate the posterior).

Priors can be
* *subjective*, i.e. reflective prior beliefs (but not arbitrary),
* *objective*, i.e. reflecting our lack of knowledge,
* *empirical*, i.e. learnt from data (we also optimize over hyper-parameters $\lambda$)

The stochastic variant of ELBO is formed by randomly batching
the dataset $D$:

$$
    \mathcal{L}(\theta, \lambda)
        \approx \mathcal{L}_\mathrm{SGVB}(\theta, \lambda)
        = \lvert D \rvert \tfrac1{\lvert B \rvert}
            \sum_{b \in B} \mathbb{E}_{\omega \sim q(\omega \mid \theta)}
                \log p(y_b \mid x_b, \omega)
        - KL(q(\omega \mid \theta) \| \pi(\omega \mid \lambda))
    \,. $$

* Stochastic optimization follows noisy unbiased gradient estimates, which are
usually cheap, allow escaping from local optima, and optimize the objective in
expectation.

In order to get a gradient of $
    F_\theta = \mathbb{E}_{\omega \sim q(\omega \mid \theta)} f(\omega)
$ w.r.t $\omega$ we use either:

###### (REINFORCE)
$
\nabla_\theta F_\theta
    = \mathbb{E}_{\omega \sim q(\omega \mid \theta)}
        (f(\omega) - b_\theta) \nabla_\theta \log q(\omega \mid \theta)
$
* for some $b_\theta$ that is used to control variance

###### (reparameterization)
$
\nabla_\theta F_\theta
    = \nabla_\theta \mathbb{E}_{\varepsilon \sim q(\varepsilon)}
        f(g(\theta; \varepsilon))
    = \mathbb{E}_{\varepsilon \sim q(\varepsilon)}
        \nabla_\theta g(\theta; \varepsilon)
            \nabla_\omega f(\omega) \big\vert_{\omega = g(\theta; \varepsilon)}
$
* when there are $q$ and differentiable $g$ such that sampling from
$q(\omega \mid \theta)$ is equivalent to $\omega = g(\theta; \varepsilon)$
with $\varepsilon \sim q(\varepsilon)$.

The variational approximation might yield high dimensional integrals,
which are slow/prohibitive to compute. To make the computations faster
without foregoing much of the precision, we may use Monte Carlo methods:

$$
    \mathbb{E}_{\omega \sim q(\omega\mid \theta)} \, f(\omega)
        \overset{\text{MC}}{\approx}
            \frac1{\lvert \mathcal{W}\rvert}
                \sum_{\omega \in \mathcal{W}} f(\omega)
    \,,
$$

where $\mathcal{W} = (\omega_b)_{b=1}^B$ is a sample of independent draws
from $q(\omega\mid \theta)$.

If we also approximate the expectation in the gradient of ELBO
via Monte Carlo we get **doubly stochastic variational objective**:

$$
    \nabla_\theta \mathcal{L}_\mathrm{DSVB}(\theta, \lambda)
        \approx
            \lvert D \rvert \tfrac1{\lvert B \rvert}
                \sum_{b \in B}
                    \mathop{gradient}(x_b, y_b, \omega_b)
            - \nabla_\theta KL(q(\omega \mid \theta) \| \pi(\omega \mid \lambda))
    \,. $$

Usually one independent draw of $\omega$ per element of $B$ is used.

We can use the same sampling approach to computing the gradient of the divergence term.

A good overview of Bayesian Inference can be found at [bdl101.ml](http://bdl101.ml/),
in [this lecture](http://mlg.eng.cam.ac.uk/zoubin/talks/lect1bayes.pdf),
[this paper](https://arxiv.org/abs/1206.7051.pdf), or
[this review](https://arxiv.org/abs/1601.00670.pdf),
among other great resources. It is also possible to consult
the references at [wiki](https://en.wikipedia.org/wiki/Bayesian_inference).

We can estimate the divergence term in the ELBO
with Monte Carlo, or, for example, for the predictive distribution
we have

$$
\begin{align}
    \mathbb{E}_{y\sim p(y\mid x, D, m)} \, g(y)
        &\overset{\text{BI}}{=}
            \mathbb{E}_{\omega\sim p(\omega \mid D, m)}
                \mathbb{E}_{y\sim p(y\mid x, \omega, D, m)} \, g(y)
        \\
        &\overset{\text{VI}}{\approx}
            \mathbb{E}_{\omega\sim q(\omega)}
                \mathbb{E}_{y\sim p(y\mid x, \omega, D, m)} \, g(y)
        \\
        &\overset{\text{MC}}{\approx}
%             \hat{\mathbb{E}}_{\omega \sim \mathcal{W}}
%                 \mathbb{E}_{y\sim p(y\mid x, \omega, D, m)} \, g(y)
            \frac1{\lvert \mathcal{W}\rvert} \sum_{\omega \in \mathcal{W}}
                \mathbb{E}_{y\sim p(y\mid x, \omega, D, m)} \, g(y)
    \,,
\end{align}
$$

where $\mathcal{W} = (\omega_b)_{b=1}^B \sim q(\omega)$
-- iid samples from the variational approximation.

Machine Learning Summer School 2019 (Moscow, Russia)/tutorials/bayesian_deep_learning/Bayesian Deep Learning part1.ipynb