## Weather Prediction with Deep Learning (30 minutes)
Denis Akhiyarov
    * Senior Consultant at Wood
    * Python.NET (Python for .NET) core developer
    * Faculty at North American University

https://notebooks.azure.com/denfromufa/libraries/pmlc

## Economics of weather forecasting?

1. Insurance and reinsurance

2. Transportation (air, taxi, boats)

3. Agriculture and fishing

4. Input for energy industry

5. Climate change and extreme events

     https://www.theweathernetwork.com

## Models of weather predictions?
Unified model - UK Met Office

## Yandex Weather

https://yandex.com/company/blog/winter-is-coming-yandex-weather-nowcasting-helps-users-plan-for-precipitation

```
from IPython.display import YouTubeVideo
YouTubeVideo('aR5p7_g8qXQ', width=800, height=600)
```

## Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale
National Energy Research Scientific Computing Center

## Package imports

```
import pandas as pd
import numpy as np
from numpy import exp
from IPython.display import HTML, Latex, Math, display
import matplotlib.pyplot as plt
import matplotlib
import seaborn as sns
from sympy.printing import latex
from sympy.core.sympify import sympify
%matplotlib inline
pd.options.display.max_columns=50
pd.options.display.max_colwidth=100
matplotlib.rcParams['savefig.dpi'] = 300
matplotlib.rcParams['figure.dpi'] = 300
matplotlib.rcParams['figure.figsize'] = (3, 2)
fontsize=3
matplotlib.rcParams['figure.titlesize'] = fontsize
matplotlib.rcParams['axes.labelsize'] = fontsize
matplotlib.rcParams['font.size'] = fontsize
matplotlib.rcParams['axes.titlesize'] = fontsize
matplotlib.rcParams['legend.fontsize'] = fontsize
matplotlib.rcParams['xtick.labelsize'] = fontsize
matplotlib.rcParams['ytick.labelsize'] = fontsize
matplotlib.rcParams['lines.markersize'] = 2
```

## How to get weather data?!

Even Google Datasets does not explain well, multiple commercial offerings online

https://www.kaggle.com/selfishgene/historical-hourly-weather-data

Historic data of weather in Houston requested and retrieved from Department of Commerce > NOAA > NESDIS > NCEI

https://www.ncdc.noaa.gov/

https://www7.ncdc.noaa.gov/CDO/cdopoemain.cmd?datasetabbv=DS3505

## Data Exploration and Cleaning

Let's read the head of the table and look at the data using pandas

```
dftxtown=pd.read_csv("https://raw.githubusercontent.com/denfromufa/deepweather/master/data/8004287785567dat.txt",
                    delim_whitespace=True)
dftxtown.head()
```

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW.1
      MW.2
      MW.3
      AW
      AW.1
      AW.2
      AW.3
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      0
      720648
      230
      201511010015
      ***
      0
      ***
      722
      SCT
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      68
      64
      ******
      29.86
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      1
      720648
      230
      201511010035
      010
      6
      ***
      50
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      68
      66
      ******
      29.87
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      2
      720648
      230
      201511010055
      010
      7
      ***
      50
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      68
      64
      ******
      29.88
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      3
      720648
      230
      201511010115
      350
      10
      ***
      28
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      70
      63
      ******
      29.89
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      4
      720648
      230
      201511010135
      340
      11
      ***
      60
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      68
      63
      ******
      29.91
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

```
dftxtown.shape
```

    (549546, 33)

```
dftxtown.USAF.value_counts()
```

    999999    316540
    720648     77469
    722530     42688
    722590     41489
    722510     38613
    722580     32747
    Name: USAF, dtype: int64

```
dftxtown.WBAN.value_counts()
```

    23907    316540
    230       77469
    12921     42688
    3927      41489
    12924     38613
    13960     32747
    Name: WBAN, dtype: int64

```
dfhtown=pd.read_csv("https://raw.githubusercontent.com/denfromufa/deepweather/master/data/8115087791588dat.txt",
                    delim_whitespace=True)
dfhtown.head()
```

    /usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:2718: DtypeWarning: Columns (24) have mixed types. Specify dtype option on import or set low_memory=False.
      interactivity=interactivity, compiler=compiler, result=result)

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW.1
      MW.2
      MW.3
      AW
      AW.1
      AW.2
      AW.3
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      0
      720637
      223
      201511010015
      330
      6
      ***
      14
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      61
      **
      **
      **
      *
      70
      68
      ******
      29.8
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      1
      720637
      223
      201511010035
      010
      5
      ***
      14
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      61
      **
      **
      **
      *
      70
      68
      ******
      29.82
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **

      2
      720637
      223
      201511010055
      350
      8
      ***
      19
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      61
      **
      **
      **
      *
      69
      68
      ******
      29.8
      ******
      ***
      ***
      0.01
      *****
      *****
      *****
      **

      3
      720637
      223
      201511010115
      340
      6
      ***
      34
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      69
      68
      ******
      29.82
      ******
      ***
      ***
      0.01
      *****
      *****
      *****
      **

      4
      720637
      223
      201511010135
      340
      5
      ***
      19
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      69
      68
      ******
      29.83
      ******
      ***
      ***
      0.01
      *****
      *****
      *****
      **

```
dfhtown.shape
```

    (78216, 33)

```
dftxtown = pd.concat([dfhtown,dftxtown])

dftxtown.shape
```

    (627762, 33)

```
dftxtown['USAFWBAN'] = dftxtown['USAF'].astype(str) + dftxtown['WBAN'].astype(str)
del dftxtown['USAF']
del dftxtown['WBAN']
dftxtown.head()
```

      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW.1
      MW.2
      MW.3
      AW
      AW.1
      AW.2
      AW.3
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD
      USAFWBAN

      0
      201511010015
      330
      6
      ***
      14
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      61
      **
      **
      **
      *
      70
      68
      ******
      29.8
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **
      720637223

      1
      201511010035
      010
      5
      ***
      14
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      61
      **
      **
      **
      *
      70
      68
      ******
      29.82
      ******
      ***
      ***
      *****
      *****
      *****
      *****
      **
      720637223

      2
      201511010055
      350
      8
      ***
      19
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      61
      **
      **
      **
      *
      69
      68
      ******
      29.8
      ******
      ***
      ***
      0.01
      *****
      *****
      *****
      **
      720637223

      3
      201511010115
      340
      6
      ***
      34
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      69
      68
      ******
      29.82
      ******
      ***
      ***
      0.01
      *****
      *****
      *****
      **
      720637223

      4
      201511010135
      340
      5
      ***
      19
      ***
      *
      *
      *
      10.0
      **
      **
      **
      **
      **
      **
      **
      **
      *
      69
      68
      ******
      29.83
      ******
      ***
      ***
      0.01
      *****
      *****
      *****
      **
      720637223

```
dftxtown.USAFWBAN.value_counts()
```

    99999923907    316540
    720637223       78216
    720648230       77469
    72253012921     42688
    7225903927      41489
    72251012924     38613
    72258013960     32747
    Name: USAFWBAN, dtype: int64

```
dftxtown = dftxtown[dftxtown['USAFWBAN'].astype(int).isin(
    [99999923907, 72251012924, 72253012921, 7225903927, 720637223])]
```

```
dftxtown = dftxtown.pivot(index='YR--MODAHRMN', columns='USAFWBAN')

dftxtown.head()
```

      DIR
      SPD
      GUS
      CLG
      SKC
      ...
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      USAFWBAN
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      ...
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907

      YR--MODAHRMN

      201511010000
      None
      330
      070
      320
      ***
      None
      15
      5
      10
      6
      None
      ***
      ***
      ***
      ***
      None
      ***
      ***
      ***
      ***
      None
      OVC
      OVC
      OVC
      ***
      ...
      None
      *****
      *****
      *****
      0.00
      None
      *****
      *****
      0.00T*****
      *****
      None
      *****
      *****
      *****
      *****
      None
      *****
      *****
      **
      *****
      None
      **
      **
      NaN
      **

      201511010005
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      ...
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      **

      201511010010
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      ...
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      **

      201511010015
      330
      None
      None
      None
      ***
      6
      None
      None
      None
      ***
      ***
      None
      None
      None
      ***
      14
      None
      None
      None
      ***
      ***
      None
      None
      None
      ***
      ...
      *****
      None
      None
      None
      *****
      *****
      None
      None
      None
      *****
      *****
      None
      None
      None
      *****
      *****
      None
      None
      None
      *****
      **
      None
      None
      None
      **

      201511010020
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      None
      None
      None
      None
      ***
      ...
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      *****
      None
      None
      None
      None
      **

5 rows × 150 columns

```
dftxtown.index = pd.to_datetime(dftxtown.index, format='%Y%m%d%H%M')
```

The data is messy, the names of columns not clear, so let's look at html version. Again using pandas!

```
import sys
!pip install lxml html5lib
```

    Collecting lxml
      Downloading https://files.pythonhosted.org/packages/03/a4/9eea8035fc7c7670e5eab97f34ff2ef0ddd78a491bf96df5accedb0e63f5/lxml-4.2.5-cp36-cp36m-manylinux1_x86_64.whl (5.8MB)
        100% |████████████████████████████████| 5.8MB 5.6MB/s
    Collecting html5lib
      Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
        100% |████████████████████████████████| 122kB 29.6MB/s
    Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.6/dist-packages (from html5lib) (1.11.0)
    Requirement already satisfied: webencodings in /usr/local/lib/python3.6/dist-packages (from html5lib) (0.5.1)
    Installing collected packages: lxml, html5lib
    Successfully installed html5lib-1.0.1 lxml-4.2.5

```
def pd2html(df, header=False, index=False):
    return HTML(df.to_html(index=False, header=False))
```

```
dfheaders = []
for _city in [99999923907, 72251012924, 72253012921, 72259003927]:
  dfheaders.append(pd.read_html(
      "https://raw.githubusercontent.com/denfromufa/deepweather/master/data/{}-2018-11_8004287785567dat.html".format(_city)))
  display(pd2html(dfheaders[-1][0]))
```

      AWS ID 999999
      WBAN ID 23907
      Name AUSTIN 33 NW
      Country UNITED STATES
      State TEXAS
      Latitude +30.622
      Longitude -098.085
      Elevation +0414.8 (meters)

      AWS ID 722510
      WBAN ID 12924
      Name CORPUS CHRISTI INTERNATIONAL AIRPORT
      Country UNITED STATES
      State TEXAS
      Latitude +27.774
      Longitude -097.512
      Elevation +0013.4 (meters)

      AWS ID 722530
      WBAN ID 12921
      Name SAN ANTONIO INTERNATIONAL AIRPORT
      Country UNITED STATES
      State TEXAS
      Latitude +29.544
      Longitude -098.484
      Elevation +0240.5 (meters)

      AWS ID 722590
      WBAN ID 03927
      Name DALLAS/FT WORTH INTERNATIONAL AP
      Country UNITED STATES
      State TEXAS
      Latitude +32.898
      Longitude -097.019
      Elevation +0170.7 (meters)

```
for dfheader in dfheaders:
  display(pd2html(dfheader[1][:2]))
```

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW
      MW
      MW
      AW
      AW
      AW
      AW
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      NaN
      NaN
      GMT
      NaN
      MPH
      MPH
      NaN
      NaN
      NaN
      NaN
      NaN
      Miles
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      F
      F
      Mb
      inches
      Mb
      F
      F
      inches
      inches
      inches
      inches
      inches

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW
      MW
      MW
      AW
      AW
      AW
      AW
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      NaN
      NaN
      GMT
      NaN
      MPH
      MPH
      NaN
      NaN
      NaN
      NaN
      NaN
      Miles
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      F
      F
      Mb
      inches
      Mb
      F
      F
      inches
      inches
      inches
      inches
      inches

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW
      MW
      MW
      AW
      AW
      AW
      AW
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      NaN
      NaN
      GMT
      NaN
      MPH
      MPH
      NaN
      NaN
      NaN
      NaN
      NaN
      Miles
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      F
      F
      Mb
      inches
      Mb
      F
      F
      inches
      inches
      inches
      inches
      inches

      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW
      MW
      MW
      AW
      AW
      AW
      AW
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      NaN
      NaN
      GMT
      NaN
      MPH
      MPH
      NaN
      NaN
      NaN
      NaN
      NaN
      Miles
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      F
      F
      Mb
      inches
      Mb
      F
      F
      inches
      inches
      inches
      inches
      inches

Let's set the index of dataframe (table) to dates

```
# dfhtown.set_index(dfhtown.columns[2], inplace=True)
# dfhtown.head(3)
```

```
#dfhtown.index=pd.to_datetime(dfhtown.index, format="%Y%m%d%H%M")
#dfhtown.index.name='datetime'
#dfhtown.replace(to_replace='\*+', value=np.nan, regex=True, inplace=True)
#dfhtown.dropna(axis=1, inplace=True, how='all')
#dfhtown.head()
```

Let's look at subset of columns with physical units from HTML version of the table

```
dfunits=dfheader[1][:2]
dfunits
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32

      0
      USAF
      WBAN
      YR--MODAHRMN
      DIR
      SPD
      GUS
      CLG
      SKC
      L
      M
      H
      VSB
      MW
      MW
      MW
      MW
      AW
      AW
      AW
      AW
      W
      TEMP
      DEWP
      SLP
      ALT
      STP
      MAX
      MIN
      PCP01
      PCP06
      PCP24
      PCPXX
      SD

      1
      NaN
      NaN
      GMT
      NaN
      MPH
      MPH
      NaN
      NaN
      NaN
      NaN
      NaN
      Miles
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      F
      F
      Mb
      inches
      Mb
      F
      F
      inches
      inches
      inches
      inches
      inches

```
dfunits.rename(columns=dfunits.iloc[0]).iloc[1].loc[dfhtown.columns].dropna().to_frame()
```

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: FutureWarning:
    Passing list-likes to .loc or [] with any missing label will raise
    KeyError in the future, you can use .reindex() as an alternative.

    See the documentation here:
    http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
      """Entry point for launching an IPython kernel.

      1

      YR--MODAHRMN
      GMT

      SPD
      MPH

      GUS
      MPH

      VSB
      Miles

      TEMP
      F

      DEWP
      F

      SLP
      Mb

      ALT
      inches

      STP
      Mb

      MAX
      F

      MIN
      F

      PCP01
      inches

      PCP06
      inches

      PCP24
      inches

      PCPXX
      inches

      SD
      inches

* So SPD and GUS are some velocity measurements for wind
* VSB??
* TEMP - temperature
* DEWP - temperature dewpoint (used for humidity)
* ALT - perhaps the pressure in inches of liquid, such as mercury

The types of columns are very generic (object), so let's improve this!

```
dfhtown.dtypes
```

    USAF             int64
    WBAN             int64
    YR--MODAHRMN     int64
    DIR             object
    SPD             object
    GUS             object
    CLG             object
    SKC             object
    L               object
    M               object
    H               object
    VSB             object
    MW              object
    MW.1            object
    MW.2            object
    MW.3            object
    AW              object
    AW.1            object
    AW.2            object
    AW.3            object
    W               object
    TEMP            object
    DEWP            object
    SLP             object
    ALT             object
    STP             object
    MAX             object
    MIN             object
    PCP01           object
    PCP06           object
    PCP24           object
    PCPXX           object
    SD              object
    dtype: object

```
dfhtown = dfhtown[['DIR','SPD','VSB','TEMP','DEWP','ALT']].astype('float64',raise_on_error=False)
dfhtown.head()
```

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: FutureWarning: the raise_on_error=False keyword is deprecated, use errors='ignore' instead
      """Entry point for launching an IPython kernel.

      DIR
      SPD
      VSB
      TEMP
      DEWP
      ALT

      0
      330
      6
      10.0
      70
      68
      29.8

      1
      010
      5
      10.0
      70
      68
      29.82

      2
      350
      8
      10.0
      69
      68
      29.8

      3
      340
      6
      10.0
      69
      68
      29.82

      4
      340
      5
      10.0
      69
      68
      29.83

```
dfhtown.dtypes
```

    DIR     object
    SPD     object
    VSB     object
    TEMP    object
    DEWP    object
    ALT     object
    dtype: object

```
dftxtown = dftxtown.filter(regex="|".join(
    ['DIR','SPD','VSB','TEMP','DEWP','ALT'])).replace(
    '^\*+$', np.nan, regex=True).astype(
    float, errors='ignore').dropna(
    how='all', axis='columns').dropna(
    axis='rows', how='all').resample('60min').median().dropna()

dftxtown.head()
```

      DIR
      SPD
      VSB
      TEMP
      DEWP
      ALT

      USAFWBAN
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927

      YR--MODAHRMN

      2015-11-01 00:00:00
      330.0
      335.0
      205.0
      330.0
      6.0
      15.5
      10.0
      12.0
      6.0
      10.0
      10.0
      10.0
      10.0
      70.0
      73.5
      75.0
      61.5
      65.0
      68.0
      65.0
      61.5
      55.5
      29.80
      29.82
      29.880
      29.870

      2015-11-01 01:00:00
      340.0
      340.0
      335.0
      330.0
      6.0
      14.0
      9.5
      10.0
      5.0
      10.0
      10.0
      4.5
      10.0
      69.0
      72.0
      69.0
      61.0
      63.0
      68.0
      68.0
      63.0
      55.0
      29.83
      29.85
      29.905
      29.900

      2015-11-01 02:00:00
      340.0
      10.0
      60.0
      325.0
      6.0
      9.0
      7.0
      10.5
      2.0
      10.0
      10.0
      10.0
      10.0
      69.0
      69.0
      67.0
      60.5
      63.0
      67.0
      68.0
      62.0
      54.0
      29.85
      29.85
      29.900
      29.920

      2015-11-01 03:00:00
      330.0
      330.0
      185.0
      330.0
      3.0
      10.0
      6.5
      10.0
      2.0
      10.0
      10.0
      10.0
      10.0
      69.0
      69.0
      67.0
      60.0
      63.0
      67.0
      68.0
      62.0
      54.0
      29.88
      29.89
      29.930
      29.935

      2015-11-01 04:00:00
      320.0
      350.0
      320.0
      340.0
      6.0
      8.0
      8.0
      6.0
      6.0
      10.0
      10.0
      10.0
      10.0
      69.0
      68.0
      66.0
      60.0
      62.0
      66.0
      67.0
      61.0
      53.0
      29.89
      29.92
      29.940
      29.950

The relative humidity (RH) is not given in table, so let's calculate using measured temperature and dewpoint temperature

```
dftxtown.describe()
```

      DIR
      SPD
      VSB
      TEMP
      DEWP
      ALT

      USAFWBAN
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927

      count
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000
      17417.000000

      mean
      176.411839
      159.598668
      188.619165
      197.955733
      6.364443
      13.489436
      9.861917
      11.759689
      4.200149
      9.236444
      9.423962
      9.432451
      9.522444
      71.564535
      74.741833
      71.756588
      69.538640
      68.487053
      62.041569
      64.474823
      56.713441
      52.911782
      30.023518
      29.984744
      30.016027
      30.004219

      std
      128.549854
      130.006679
      180.261087
      150.182745
      3.843356
      5.929081
      4.549815
      5.508809
      2.950346
      1.876862
      1.699020
      1.753733
      1.651627
      12.728135
      12.349128
      14.300348
      16.677465
      15.204274
      13.568524
      13.105968
      14.386607
      16.269214
      0.166065
      0.172456
      0.174348
      0.196544

      min
      10.000000
      10.000000
      10.000000
      10.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      22.000000
      26.000000
      20.000000
      15.000000
      15.000000
      12.000000
      9.000000
      5.000000
      1.500000
      29.450000
      29.400000
      29.420000
      29.370000

      25%
      105.000000
      100.000000
      100.000000
      130.000000
      3.000000
      9.000000
      6.500000
      8.000000
      2.000000
      10.000000
      10.000000
      10.000000
      10.000000
      64.000000
      68.000000
      62.500000
      58.000000
      59.000000
      55.000000
      58.000000
      47.500000
      41.000000
      29.920000
      29.880000
      29.900000
      29.880000

      50%
      150.000000
      140.000000
      150.000000
      170.000000
      6.000000
      13.000000
      9.000000
      11.000000
      4.000000
      10.000000
      10.000000
      10.000000
      10.000000
      73.000000
      77.000000
      73.000000
      71.000000
      70.000000
      67.000000
      69.000000
      62.000000
      57.000000
      30.010000
      29.970000
      30.000000
      29.980000

      75%
      220.000000
      170.000000
      200.000000
      220.000000
      9.000000
      17.000000
      13.000000
      15.000000
      6.000000
      10.000000
      10.000000
      10.000000
      10.000000
      81.000000
      84.000000
      82.000000
      83.000000
      79.500000
      73.000000
      74.000000
      68.000000
      67.000000
      30.110000
      30.080000
      30.120000
      30.120000

      max
      990.000000
      990.000000
      990.000000
      990.000000
      26.000000
      41.000000
      31.000000
      36.000000
      17.000000
      10.000000
      10.000000
      10.000000
      10.000000
      98.000000
      102.000000
      104.000000
      108.000000
      108.000000
      80.000000
      83.000000
      76.000000
      79.000000
      30.770000
      30.790000
      30.760000
      30.810000

```
dftxtown.rolling(window=72).mean()[['TEMP']].plot();
```

The vapor pressure answers will be in units of millibars (mb) or hectoPascals (hPa).
To convert the saturated vapor pressure and/or the actual vapor pressure, the
temperature values must be converted to degrees Celsius (°C) .

## Deep learning approach

This part is based on the deep learning book "Deep Learning with Python" by Keras Lead François Chollet

https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/6.3-advanced-usage-of-recurrent-neural-networks.ipynb

```
import tensorflow as tf
print("tensorflow version = {}".format(tf.__version__))

import keras
print("keras version = {}".format(keras.__version__))

```

    tensorflow version = 1.12.0
    keras version = 2.2.4

## Transforming the data

The exact formulation of our problem will be the following: given data going as far back as `lookback` timesteps (a timestep is 10 minutes)
and sampled every `steps` timesteps, can we predict the temperature in `delay` timesteps?

We will use the following parameter values:

* `lookback = 360`, i.e. our observations will go back 5 days.
* `steps = 3`, i.e. our observations will be sampled at one data point per hour.
* `delay = 72`, i.e. our targets will be 24 hours in the future.

To get started, we need to do two things:

* Preprocess the data to a format a neural network can ingest. This is easy: the data is already numerical, so we don't need to do any
vectorization. However each timeseries in the data is on a different scale (e.g. temperature is typically between -20 and +30, but
pressure, measured in mbar, is around 1000). So we will normalize each timeseries independently so that they all take small values on a
similar scale.
* Write a Python generator that takes our current array of float data and yields batches of data from the recent past, alongside with a
target temperature in the future. Since the samples in our dataset are highly redundant (e.g. sample `N` and sample `N + 1` will have most
of their timesteps in common), it would be very wasteful to explicitly allocate every sample. Instead, we will generate the samples on the
fly using the original data.

We preprocess the data by subtracting the mean of each timeseries and dividing by the standard deviation. We plan on using the first
200,000 timesteps as training data, so we compute the mean and standard deviation only on this fraction of the data:

```
usedcols = 'DIR	TEMP	SPD	VSB	DEWP	ALT'.split('\t')
float_data = dftxtown[usedcols].dropna()
display(float_data.head(2))
float_data = float_data.values
float_data.shape

# 720637223 TEMP at index 4
```

      DIR
      TEMP
      SPD
      VSB
      DEWP
      ALT

      USAFWBAN
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      99999923907
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927
      720637223
      72251012924
      72253012921
      7225903927

      YR--MODAHRMN

      2015-11-01 00:00:00
      330.0
      335.0
      205.0
      330.0
      70.0
      73.5
      75.0
      61.5
      65.0
      6.0
      15.5
      10.0
      12.0
      6.0
      10.0
      10.0
      10.0
      10.0
      68.0
      65.0
      61.5
      55.5
      29.80
      29.82
      29.880
      29.87

      2015-11-01 01:00:00
      340.0
      340.0
      335.0
      330.0
      69.0
      72.0
      69.0
      61.0
      63.0
      6.0
      14.0
      9.5
      10.0
      5.0
      10.0
      10.0
      4.5
      10.0
      68.0
      68.0
      63.0
      55.0
      29.83
      29.85
      29.905
      29.90

    (17417, 26)

```
mean = float_data.mean(axis=0)
float_data -= mean
std = float_data.std(axis=0)
float_data /= std
```

```
import seaborn as sns
#plt.matshow(pd.DataFrame(float_data, columns = usedcols).corr())

sns.heatmap(pd.DataFrame(float_data #, columns = usedcols
                        ).corr(),
            #xticklabels=usedcols,
            #yticklabels=usedcols
           )

```

Now here is the data generator that we will use. It yields a tuple `(samples, targets)` where `samples` is one batch of input data and
`targets` is the corresponding array of target temperatures. It takes the following arguments:

* `data`: The original array of floating point data, which we just normalized in the code snippet above.
* `lookback`: How many timesteps back should our input data go.
* `delay`: How many timesteps in the future should our target be.
* `min_index` and `max_index`: Indices in the `data` array that delimit which timesteps to draw from. This is useful for keeping a segment
of the data for validation and another one for testing.
* `shuffle`: Whether to shuffle our samples or draw them in chronological order.
* `batch_size`: The number of samples per batch.
* `step`: The period, in timesteps, at which we sample data. We will set it 6 in order to draw one data point every hour.

```
def generator(data, lookback, delay, min_index, max_index,
              shuffle=False, batch_size=128, step=6, target_col = 4):
    if max_index is None:
        max_index = len(data) - delay - 1
    i = min_index + lookback
    while True:
        if shuffle:
            rows = np.random.randint(
                min_index + lookback, max_index, size=batch_size)
        else:
            if i + batch_size >= max_index:
                i = min_index + lookback
            rows = np.arange(i, min(i + batch_size, max_index))
            i += len(rows)

        samples = np.zeros((len(rows),
                           lookback // step,
                           data.shape[-1]))
        targets = np.zeros((len(rows),))
        for j, row in enumerate(rows):
            indices = range(rows[j] - lookback, rows[j], step)
            samples[j] = data[indices]
            targets[j] = data[rows[j] + delay][target_col]
        yield samples, targets
```

Now let's use our abstract generator function to instantiate three generators, one for training, one for validation and one for testing. Each will look at different temporal segments of the original data: the training generator looks at the first 200,000 timesteps, the validation generator looks at the following 100,000, and the test generator looks at the remainder.

```
float_data.shape
```

    (17417, 26)

```
lookback = 120
step = 2
delay = 24
batch_size = 64
min_index_train = 0
max_index_train = 10000

min_index_valid = 10001
max_index_valid = 15000

def gen_train_val_test():
  train_gen = generator(float_data,
                        lookback=lookback,
                        delay=delay,
                        min_index=min_index_train,
                        max_index=max_index_train,
                        shuffle=True,
                        step=step,
                        batch_size=batch_size,
                        )
  val_gen = generator(float_data,
                      lookback=lookback,
                      delay=delay,
                      min_index=min_index_valid,
                      max_index=max_index_valid,
                      step=step,
                      batch_size=batch_size)
  test_gen = generator(float_data,
                       lookback=lookback,
                       delay=delay,
                       min_index=max_index_valid,
                       max_index=None,
                       step=step,
                       batch_size=batch_size)
  return train_gen, val_gen, test_gen

train_gen, val_gen, test_gen = gen_train_val_test()

# This is how many steps to draw from `val_gen`
# in order to see the whole validation set:
val_steps = (max_index_valid - min_index_valid - lookback) // batch_size

# This is how many steps to draw from `test_gen`
# in order to see the whole test set:
test_steps = (len(float_data) - max_index_valid - lookback) // batch_size
```

## Using Recurrent Neural Networks

```
from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

train_gen, val_gen, test_gen = gen_train_val_test()

model = Sequential()
#model.add(layers.GRU(16, implementation=2,
#input_shape=(None, float_data.shape[-1])))
model.add(layers.CuDNNGRU(16, input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=80,
                              epochs=20,
                              validation_data=val_gen,
                              validation_steps=val_steps)
```

    Epoch 1/20
    80/80 [==============================] - 2s 26ms/step - loss: 0.5922 - val_loss: 0.6375
    Epoch 2/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.5104 - val_loss: 0.6275
    Epoch 3/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.5025 - val_loss: 0.6144
    Epoch 4/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4821 - val_loss: 0.6178
    Epoch 5/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4826 - val_loss: 0.6109
    Epoch 6/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4596 - val_loss: 0.6060
    Epoch 7/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4594 - val_loss: 0.6049
    Epoch 8/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4569 - val_loss: 0.6054
    Epoch 9/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4423 - val_loss: 0.6053
    Epoch 10/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4455 - val_loss: 0.6020
    Epoch 11/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4540 - val_loss: 0.6206
    Epoch 12/20
    80/80 [==============================] - 2s 19ms/step - loss: 0.4419 - val_loss: 0.6046
    Epoch 13/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4392 - val_loss: 0.6060
    Epoch 14/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4265 - val_loss: 0.6053
    Epoch 15/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4277 - val_loss: 0.6062
    Epoch 16/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4192 - val_loss: 0.6109
    Epoch 17/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4100 - val_loss: 0.6205
    Epoch 18/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4174 - val_loss: 0.5982
    Epoch 19/20
    80/80 [==============================] - 2s 19ms/step - loss: 0.4117 - val_loss: 0.6049
    Epoch 20/20
    80/80 [==============================] - 2s 20ms/step - loss: 0.4042 - val_loss: 0.6012

```
import matplotlib.pyplot as plt

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
```

```
from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

train_gen, val_gen, test_gen = gen_train_val_test()

model = Sequential()
model.add(layers.GRU(64,
                     #dropout=0.1,
                     recurrent_dropout=0.2,
                     input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=20,
                              epochs=40,
                              validation_data=val_gen,
                              validation_steps=val_steps
                             )
```

    Epoch 1/40
    20/20 [==============================] - 6s 284ms/step - loss: 0.6371 - val_loss: 0.6260
    Epoch 2/40
    20/20 [==============================] - 5s 250ms/step - loss: 0.5394 - val_loss: 0.6149
    Epoch 3/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.5198 - val_loss: 0.6113
    Epoch 4/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4908 - val_loss: 0.6027
    Epoch 5/40
    20/20 [==============================] - 5s 252ms/step - loss: 0.4836 - val_loss: 0.5992
    Epoch 6/40
    20/20 [==============================] - 5s 253ms/step - loss: 0.4994 - val_loss: 0.5950
    Epoch 7/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4732 - val_loss: 0.5982
    Epoch 8/40
    20/20 [==============================] - 5s 250ms/step - loss: 0.4837 - val_loss: 0.5894
    Epoch 9/40
    20/20 [==============================] - 5s 252ms/step - loss: 0.4661 - val_loss: 0.5924
    Epoch 10/40
    20/20 [==============================] - 5s 256ms/step - loss: 0.4549 - val_loss: 0.5918
    Epoch 11/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4544 - val_loss: 0.5959
    Epoch 12/40
    20/20 [==============================] - 5s 249ms/step - loss: 0.4512 - val_loss: 0.5914
    Epoch 13/40
    20/20 [==============================] - 5s 247ms/step - loss: 0.4606 - val_loss: 0.5981
    Epoch 14/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4742 - val_loss: 0.6195
    Epoch 15/40
    20/20 [==============================] - 5s 253ms/step - loss: 0.4678 - val_loss: 0.5974
    Epoch 16/40
    20/20 [==============================] - 5s 250ms/step - loss: 0.4386 - val_loss: 0.5893
    Epoch 17/40
    20/20 [==============================] - 5s 252ms/step - loss: 0.4637 - val_loss: 0.5913
    Epoch 18/40
    20/20 [==============================] - 5s 250ms/step - loss: 0.4450 - val_loss: 0.5899
    Epoch 19/40
    20/20 [==============================] - 5s 252ms/step - loss: 0.4529 - val_loss: 0.6058
    Epoch 20/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4499 - val_loss: 0.5855
    Epoch 21/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4301 - val_loss: 0.5837
    Epoch 22/40
    20/20 [==============================] - 5s 253ms/step - loss: 0.4418 - val_loss: 0.6022
    Epoch 23/40
    20/20 [==============================] - 5s 255ms/step - loss: 0.4544 - val_loss: 0.5834
    Epoch 24/40
    20/20 [==============================] - 5s 257ms/step - loss: 0.4294 - val_loss: 0.5842
    Epoch 25/40
    20/20 [==============================] - 5s 253ms/step - loss: 0.4338 - val_loss: 0.5848
    Epoch 26/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4459 - val_loss: 0.6002
    Epoch 27/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4251 - val_loss: 0.6003
    Epoch 28/40
    20/20 [==============================] - 5s 250ms/step - loss: 0.4304 - val_loss: 0.5995
    Epoch 29/40
    20/20 [==============================] - 5s 249ms/step - loss: 0.4172 - val_loss: 0.5846
    Epoch 30/40
    20/20 [==============================] - 5s 252ms/step - loss: 0.4310 - val_loss: 0.5959
    Epoch 31/40
    20/20 [==============================] - 5s 252ms/step - loss: 0.4293 - val_loss: 0.5930
    Epoch 32/40
    20/20 [==============================] - 5s 250ms/step - loss: 0.4159 - val_loss: 0.6055
    Epoch 33/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4166 - val_loss: 0.5795
    Epoch 34/40
    20/20 [==============================] - 5s 254ms/step - loss: 0.4202 - val_loss: 0.5873
    Epoch 35/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4210 - val_loss: 0.5930
    Epoch 36/40
    20/20 [==============================] - 5s 250ms/step - loss: 0.4052 - val_loss: 0.6047
    Epoch 37/40
    20/20 [==============================] - 5s 255ms/step - loss: 0.4094 - val_loss: 0.5857
    Epoch 38/40
    20/20 [==============================] - 5s 252ms/step - loss: 0.4222 - val_loss: 0.6491
    Epoch 39/40
    20/20 [==============================] - 5s 249ms/step - loss: 0.4132 - val_loss: 0.5803
    Epoch 40/40
    20/20 [==============================] - 5s 251ms/step - loss: 0.4085 - val_loss: 0.6060

```
import matplotlib.pyplot as plt

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
```

```
model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    gru_3 (GRU)                  (None, 64)                17472
    _________________________________________________________________
    dense_5 (Dense)              (None, 1)                 65
    =================================================================
    Total params: 17,537
    Trainable params: 17,537
    Non-trainable params: 0
    _________________________________________________________________

```
train_gen, val_gen, test_gen = gen_train_val_test()

batch_maes = []
predsi = []
targetsi = []
for step in range(val_steps):
    samples, targets = next(val_gen)
    preds = model.predict(samples)
    mae = np.mean(np.abs(preds - targets))
    #print(mae)
    batch_maes.append(mae)
    predsi.extend(preds)
    targetsi.extend(targets)
print(np.mean(batch_maes))
plt.plot(predsi[:1000], 'b', linewidth=1, label='prediction');
plt.plot(targetsi[:1000], 'g', linewidth=1, label='target');
plt.legend();
```

```

```


# Masked Language Modeling

In this lab, we will overview the **masked language modeling** objective, and how the **Transformer** architecture is used for large-scale masked language modeling.

```python
%pylab inline
import os, sys, glob, json, math
import pandas as pd
from tqdm import tqdm
from pprint import pprint
from collections import defaultdict
import torch
import torch.nn as nn

%load_ext autoreload
%autoreload 2
pd.set_option('display.max_colwidth', -1)
```

    Populating the interactive namespace from numpy and matplotlib

Let set the random seeds for reproducability.

```python
SEED = 1234

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True
```

## Background

Recently, Devlin et al. published [BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding](https://arxiv.org/pdf/1810.04805.pdf).

**B**idirectional

**E**ncoder

**R**epresentations from

**T**ransformers

#### Goal:
1. **pre-train** a model that produces language representations.
2. **fine-tune** the model on a task.

## Masked Language Model Objective

Randomly mask some of the tokens from the input, predict original vocabulary id of each masked token.

- Given sequence $x_1,\ldots,x_N$.

- Form **mask** $m_1,\ldots,m_N$ where $m_i\in \{0,1\}$.
    - E.g. $m_i=1$ with probability 0.15

- Form **masked sequence** $\tilde{x}_1,\ldots,\tilde{x}_N$.
    - $\tilde{x}_i=\begin{cases} x_i & m_i=0\\ \texttt{[MASK]} & m_i=1\end{cases}$

#### $$\mathcal{L}_{\text{MLM}}=-\sum_{\underbrace{i | m_i=1}_{\text{MASKED POSITIONS}}}\log p_{\theta}(\underbrace{x_i}_{\text{TRUE TOKEN}}|\underbrace{\tilde{x}_1,\ldots,\tilde{x}_N}_{\text{MASKED SEQUENCE}})$$

<!-- Below, we will discuss the exact form of $\tilde{x}_i$ that the BERT authors used. -->

<!-- #### Diagram of BERT Implementation -->
<!--  -->

## Transformers

So far we have modeled a sequence by factorizing the joint distribution into conditionals, and **parameterizing each conditional with a recurrent network**:

#### $$p_{\theta}(x_1,\ldots,x_T)=\prod_{t=1}^T p_{\theta}(x_t | x_{<t})$$
\begin{align}
h_t &= RNN(x_{t-1}, h_t)\\
p_{\theta}(x_t | x_{<t}) &=\text{softmax}\left(Wh_t+b\right),
\end{align}

where $\theta$ are the model parameters (RNN parameters, $W, b$, embedding matrix).

#### Alternative

An alternative proposed in [[Vaswani et al 2017](https://arxiv.org/pdf/1706.03762.pdf)] is to parameterize each conditional with a **particular feed-forward architecture** called the **Transformer**. With this model, it is possible to compute all conditionals with a **single feed-forward pass**:
\begin{align}
(h_1,\ldots,h_T) &= Transformer(x)\\
p_{\theta}(x_t | x_{<t}) &= \text{softmax}\left(Wh_t + b\right)
\end{align}

We will discuss briefly the key ideas, the overall **Transformer architecture (encoder only)**, and how they are used in Pytorch.

### High-Level View

We can view the Transformer encoder as mapping a sequence to a sequence of vectors.

Let's step through the key ideas of how this mapping is designed, and discuss some of its resulting properties.

### Key Idea 1: Position Embeddings

**Unlike RNNs which can learn positional information via the hidden state over time, the Transformer has no notion of time**.

Thus we encode inputs with **position** as well as **token** embeddings:

```python
input_sequence = ['<s>', 'my', 'pet', '[M]', '<s>']

max_len = 10

vocab = {'<s>': 0, 'my': 1, 'pet': 2, 'dog': 3, 'cat': 4, 'lion': 5, '[M]': 6}

dim = 6

token_embed = nn.Embedding(len(vocab), embedding_dim=dim)  # an embedding for each token
position_embed = nn.Embedding(max_len, embedding_dim=dim)  # an empbedding to count for the position of the token
```

```python
input_vector = torch.tensor([vocab[x] for x in input_sequence]).unsqueeze(1)   # get the numerical representation of the token

input_embeddings = token_embed(input_vector) + position_embed(torch.arange(len(input_vector))).unsqueeze(1)  # add the input embedding to the position embedding
input_embeddings.size()
```

    torch.Size([5, 1, 6])

**Warning!!** The pytorch Transformer classes accept input as `Length x Batch x Dim`

#### Key Idea 2: Modularity
The Transformer (encoder) is composed of a stack of **N identical layers**.

```python
import torch.nn as nn
nn.TransformerEncoder?
```

    Init signature: nn.TransformerEncoder(encoder_layer, num_layers, norm=None)
    Docstring:
    TransformerEncoder is a stack of N encoder layers

    Args:
        encoder_layer: an instance of the TransformerEncoderLayer() class (required).
        num_layers: the number of sub-encoder-layers in the encoder (required).
        norm: the layer normalization component (optional).

    Examples::
        >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
        >>> transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
        >>> src = torch.rand(10, 32, 512)
        >>> out = transformer_encoder(src)
    Init docstring: Initializes internal Module state, shared by both nn.Module and ScriptModule.
    File:           ~/anaconda3/envs/aims/lib/python3.7/site-packages/torch/nn/modules/transformer.py
    Type:           type
    Subclasses:

#### The `forward` passes the input through the N layers, then normalizes it:

**Warning!!** The forward function accepts input as `Length x Batch x Dim`

```python
# nn.TransformerEncoder.forward??
```

```python
encoder_layer = nn.TransformerEncoderLayer(d_model=dim, nhead=2, dim_feedforward=64, dropout=0.1) # attention is all you need config nn.TransformerEncoderLayer(d_model=512, nhead=8)

encoder = nn.TransformerEncoder(encoder_layer, num_layers=4)
```

```python
outputs = encoder(input_embeddings)

print("input size: \t%s" % str(tuple(input_embeddings.shape)))
print("output size:\t%s" % str(tuple(outputs.shape)))             # contextualize embedding of each token
outputs
```

    input size: 	(5, 1, 6)
    output size:	(5, 1, 6)

    tensor([[[ 0.5391, -1.9445,  1.0577, -0.4564,  0.0294,  0.7747]],

            [[-0.7556,  0.5501, -1.3479,  1.5129,  0.7344, -0.6939]],

            [[ 0.7792, -1.9722,  0.9250,  0.5844,  0.1997, -0.5160]],

            [[-0.9059, -1.2809,  0.1254, -0.3488,  1.7034,  0.7068]],

            [[ 1.1515, -1.2860,  0.2149,  1.3268, -0.3691, -1.0381]]],
           grad_fn=<NativeLayerNormBackward>)

```python
print(token_embed(input_vector)) # the original token embedding
```

    tensor([[[-0.1117, -0.4966,  0.1631, -0.8817,  0.0539,  0.6684]],

            [[-0.0597, -0.4675, -0.2153,  0.8840, -0.7584, -0.3689]],

            [[-0.3424, -1.4020,  0.3206, -1.0219,  0.7988, -0.0923]],

            [[-0.7690, -1.5606, -0.5309,  0.2178,  1.3232,  1.8169]],

            [[-0.1117, -0.4966,  0.1631, -0.8817,  0.0539,  0.6684]]],
           grad_fn=<EmbeddingBackward>)

#### Each layer has two parts, **self-attention** and a feed-forward transformation:

```python
# nn.TransformerEncoderLayer??
```

```python
# nn.TransformerEncoderLayer.forward??
```

### Key Idea 3: Self-Attention

In the RNN, the hidden state contains information about previous tokens.
The Transformer instead performs **attention** over all inputs at a given layer. 'Attention' computes an output vector by taking a weighted sum of input vectors. The weights are 'attention weights'. The Transformer uses **scaled dot-product attention**:
#### $$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

and 'Multi-head Attention' refers to applying several of these operations in parallel.

#### *Key Property*: Each output vector of a layer $n$ can using information from **all** inputs to the layer $n$.

Thus each **final output vector** can incorporate information from **all input words**.

(If we want to prevent information flow such as in left-to-right language modeling, we can use masking).

```python
nn.MultiheadAttention?
```

    Init signature:
    nn.MultiheadAttention(
        embed_dim,
        num_heads,
        dropout=0.0,
        bias=True,
        add_bias_kv=False,
        add_zero_attn=False,
        kdim=None,
        vdim=None,
    )
    Docstring:
    Allows the model to jointly attend to information
    from different representation subspaces.
    See reference: Attention Is All You Need

    .. math::
        \text{MultiHead}(Q, K, V) = \text{Concat}(head_1,\dots,head_h)W^O
        \text{where} head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)

    Args:
        embed_dim: total dimension of the model.
        num_heads: parallel attention heads.
        dropout: a Dropout layer on attn_output_weights. Default: 0.0.
        bias: add bias as module parameter. Default: True.
        add_bias_kv: add bias to the key and value sequences at dim=0.
        add_zero_attn: add a new batch of zeros to the key and
                       value sequences at dim=1.
        kdim: total number of features in key. Default: None.
        vdim: total number of features in key. Default: None.

        Note: if kdim and vdim are None, they will be set to embed_dim such that
        query, key, and value have the same number of features.

    Examples::

        >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
        >>> attn_output, attn_output_weights = multihead_attn(query, key, value)
    Init docstring: Initializes internal Module state, shared by both nn.Module and ScriptModule.
    File:           ~/anaconda3/envs/aims/lib/python3.7/site-packages/torch/nn/modules/activation.py
    Type:           type
    Subclasses:

```python
attn = nn.MultiheadAttention(dim, 2, dropout=0.0)

attn_outputs, attn_weights = attn.forward(query=outputs, key=outputs, value=outputs)

print("input shape: %s" % (str(tuple(outputs.size()))))
print("output shape: %s" % (str(tuple(attn_outputs.size()))))
print(outputs)

print("\nattn weights shape: %s" % (str(tuple(attn_weights.size()))))
print(attn_weights)
```

    input shape: (5, 1, 6)
    output shape: (5, 1, 6)
    tensor([[[ 0.5391, -1.9445,  1.0577, -0.4564,  0.0294,  0.7747]],

            [[-0.7556,  0.5501, -1.3479,  1.5129,  0.7344, -0.6939]],

            [[ 0.7792, -1.9722,  0.9250,  0.5844,  0.1997, -0.5160]],

            [[-0.9059, -1.2809,  0.1254, -0.3488,  1.7034,  0.7068]],

            [[ 1.1515, -1.2860,  0.2149,  1.3268, -0.3691, -1.0381]]],
           grad_fn=<NativeLayerNormBackward>)

    attn weights shape: (1, 5, 5)
    tensor([[[0.1703, 0.2043, 0.1716, 0.2744, 0.1794],
             [0.2384, 0.1427, 0.2343, 0.1328, 0.2517],
             [0.1686, 0.1739, 0.1843, 0.2152, 0.2581],
             [0.1906, 0.1497, 0.2084, 0.2162, 0.2351],
             [0.1768, 0.1457, 0.1723, 0.2346, 0.2706]]], grad_fn=<DivBackward0>)

#### Summary

```python
class Transformer(nn.Module):
    def __init__(self, vocab_size, max_len, dim=8, num_layers=4, nhead=2):
        super().__init__()
        self.token_embed = nn.Embedding(vocab_size, dim)
        self.position_embed = nn.Embedding(max_len, dim)
        encoder_layer = nn.TransformerEncoderLayer(d_model=dim, nhead=nhead, dim_feedforward=64, dropout=0.0)
        self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
        self.projection = nn.Linear(dim, vocab_size)

    def features(self, token_indices):
        pos = torch.arange(len(token_indices), device=token_indices.device).unsqueeze(1)
        x = self.token_embed(token_indices) + self.position_embed(pos)
        x = self.encoder(x)
        return x

    def forward(self, token_indices):
        x = self.features(token_indices)
        x = self.projection(x)
        return x
```

```python
input_vector.size()
```

    torch.Size([5, 1])

```python
model = Transformer(len(vocab), max_len=100)

model.features(input_vector)
```

    tensor([[[-0.1841,  0.3811,  1.4767,  0.3744, -2.2709, -0.2429, -0.0631,
               0.5290]],

            [[ 0.6175,  0.9034,  1.0025, -0.8787,  1.1249, -1.4216, -0.0295,
              -1.3185]],

            [[-0.3699,  0.7491,  0.4773, -0.6109,  1.7609,  0.3687, -0.6204,
              -1.7548]],

            [[-1.1181,  0.9639, -0.8665,  0.5216, -1.5619,  0.5622,  0.0716,
               1.4273]],

            [[-0.1518, -0.3191,  1.9927,  0.0086, -1.1681, -1.3354,  0.1103,
               0.8627]]], grad_fn=<NativeLayerNormBackward>)

```python
model.features(input_vector).shape
```

    torch.Size([5, 1, 8])

```python
model
```

    Transformer(
      (token_embed): Embedding(7, 8)
      (position_embed): Embedding(100, 8)
      (encoder): TransformerEncoder(
        (layers): ModuleList(
          (0): TransformerEncoderLayer(
            (self_attn): MultiheadAttention(
              (out_proj): Linear(in_features=8, out_features=8, bias=True)
            )
            (linear1): Linear(in_features=8, out_features=64, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
            (linear2): Linear(in_features=64, out_features=8, bias=True)
            (norm1): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (norm2): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (dropout1): Dropout(p=0.0, inplace=False)
            (dropout2): Dropout(p=0.0, inplace=False)
          )
          (1): TransformerEncoderLayer(
            (self_attn): MultiheadAttention(
              (out_proj): Linear(in_features=8, out_features=8, bias=True)
            )
            (linear1): Linear(in_features=8, out_features=64, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
            (linear2): Linear(in_features=64, out_features=8, bias=True)
            (norm1): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (norm2): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (dropout1): Dropout(p=0.0, inplace=False)
            (dropout2): Dropout(p=0.0, inplace=False)
          )
          (2): TransformerEncoderLayer(
            (self_attn): MultiheadAttention(
              (out_proj): Linear(in_features=8, out_features=8, bias=True)
            )
            (linear1): Linear(in_features=8, out_features=64, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
            (linear2): Linear(in_features=64, out_features=8, bias=True)
            (norm1): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (norm2): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (dropout1): Dropout(p=0.0, inplace=False)
            (dropout2): Dropout(p=0.0, inplace=False)
          )
          (3): TransformerEncoderLayer(
            (self_attn): MultiheadAttention(
              (out_proj): Linear(in_features=8, out_features=8, bias=True)
            )
            (linear1): Linear(in_features=8, out_features=64, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
            (linear2): Linear(in_features=64, out_features=8, bias=True)
            (norm1): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (norm2): LayerNorm((8,), eps=1e-05, elementwise_affine=True)
            (dropout1): Dropout(p=0.0, inplace=False)
            (dropout2): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (projection): Linear(in_features=8, out_features=7, bias=True)
    )

```python
model.token_embed(input_vector)
```

    tensor([[[-1.1013, -1.4392, -0.7411, -1.5282,  0.5737, -1.4641, -0.3159,
               0.9548]],

            [[-0.0104,  0.0784,  0.5036, -0.7424,  1.9766, -0.9712,  0.1476,
              -1.6098]],

            [[-0.7888, -0.8349,  1.3856, -2.7129,  1.0593,  1.2605,  0.0641,
              -0.4017]],

            [[ 0.0079, -1.4143, -2.0439, -0.7530,  0.1951,  1.4279, -0.3208,
               1.5795]],

            [[-1.1013, -1.4392, -0.7411, -1.5282,  0.5737, -1.4641, -0.3159,
               0.9548]]], grad_fn=<EmbeddingBackward>)

## Back to Masked Language Modeling

Recall the **key property** of Transformers: due to self-attention, each output vector can incorporate information from *all* input tokens.

This is useful for masked language modeling, where we want to use information from the entire context when predicting the masked token(s).

#### MLM on Persona-Chat

```python
import utils
raw_datasets, datasets, vocab = utils.load_personachat()
```

    100%|██████████| 133176/133176 [00:31<00:00, 4251.03it/s]
    100%|██████████| 133176/133176 [00:00<00:00, 168611.57it/s]
    100%|██████████| 16181/16181 [00:00<00:00, 169260.05it/s]

    Vocab size: 19157

```python
from torch.utils.data.dataloader import DataLoader

trainloader = DataLoader(datasets['train'], batch_size=4, collate_fn=lambda x: utils.pad_collate_fn(vocab.get_id('<pad>'), x))
validloader = DataLoader(datasets['valid'], batch_size=4, collate_fn=lambda x: utils.pad_collate_fn(vocab.get_id('<pad>'), x))
```

```python
batch = next(trainloader.__iter__())
batch
```

    tensor([[ 0,  0,  0,  0],
            [ 4,  4,  4, 22],
            [ 5,  5, 18, 23],
            [ 6, 13, 17, 24],
            [ 7, 14, 19, 15],
            [ 8, 15, 13, 25],
            [ 9, 16, 20, 26],
            [10, 17, 21, 27],
            [11, 12, 12, 28],
            [12,  0,  0, 29],
            [ 0,  2,  2, 30],
            [ 2,  2,  2, 24],
            [ 2,  2,  2,  4],
            [ 2,  2,  2, 31],
            [ 2,  2,  2, 32],
            [ 2,  2,  2, 27],
            [ 2,  2,  2, 33],
            [ 2,  2,  2, 34],
            [ 2,  2,  2, 35],
            [ 2,  2,  2, 36],
            [ 2,  2,  2, 24],
            [ 2,  2,  2,  0]])

```python
def mask_tokens(inputs, mask_prob, pad_token_id, mask_token_id, vsize):
    """ Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original."""
    inputs = inputs.clone()
    labels = inputs.clone()
    # Sample tokens in each sequence for masked-LM training
    masked_indices = torch.bernoulli(torch.full(labels.shape, mask_prob)).bool()
    masked_indices = masked_indices & (inputs != pad_token_id)
    labels[~masked_indices] = -1  # We only compute loss on masked tokens

    # 80% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK])
    indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
    inputs[indices_replaced] = mask_token_id

    # 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(vsize, labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random]

    # The rest of the time (10% of the time) we keep the masked input tokens unchanged
    return inputs, labels
```

```python
inputs, labels = mask_tokens(batch, mask_prob=0.15, mask_token_id=vocab.get_id('[M]'), pad_token_id=vocab.get_id('<pad>'), vsize=len(vocab))
print("Mask token id: %d" % vocab.get_id('[M]'))
inputs
```

    Mask token id: 1

    tensor([[ 0,  0,  0,  0],
            [ 1,  4,  4, 22],
            [ 5,  5, 18, 23],
            [ 6, 13, 17,  1],
            [ 7,  1, 19, 15],
            [ 8, 15, 13,  1],
            [ 9, 16, 20, 26],
            [10, 17, 21, 27],
            [ 1, 12, 12, 28],
            [12,  0,  0, 29],
            [ 0,  2,  2, 30],
            [ 2,  2,  2, 24],
            [ 2,  2,  2,  4],
            [ 2,  2,  2, 31],
            [ 2,  2,  2, 32],
            [ 2,  2,  2, 27],
            [ 2,  2,  2,  1],
            [ 2,  2,  2, 34],
            [ 2,  2,  2, 35],
            [ 2,  2,  2, 36],
            [ 2,  2,  2, 24],
            [ 2,  2,  2,  0]])

```python
labels
```

    tensor([[-1, -1, -1, -1],
            [ 4, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, 24],
            [-1, 14, -1, -1],
            [-1, -1, -1, 25],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [11, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, 33],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1],
            [-1, -1, -1, -1]])

```python
model = Transformer(len(vocab), max_len=200)
```

```python
logits = model(inputs)
logits.size()
```

    torch.Size([22, 4, 19157])

```python
labels.size()
```

    torch.Size([22, 4])

```python
criterion = nn.CrossEntropyLoss(ignore_index=-1)
```

```python
logits_ = logits.view(-1, logits.size(2))
labels_ = labels.view(-1)

criterion(logits_, labels_)
```

    tensor(10.2056, grad_fn=<NllLossBackward>)

```python
if False:
    import torch.optim as optim
    from tqdm import tqdm, trange
    from collections import defaultdict
    from torch.utils.data.dataloader import DataLoader

    trainloader = DataLoader(datasets['train'], batch_size=64, collate_fn=lambda x: utils.pad_collate_fn(vocab.get_id('<pad>'), x))
    validloader = DataLoader(datasets['valid'], batch_size=64, collate_fn=lambda x: utils.pad_collate_fn(vocab.get_id('<pad>'), x))

    device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')

    model = Transformer(len(vocab), max_len=65, dim=256, nhead=8).to(device)

    model_parameters = [p for p in model.parameters() if p.requires_grad]
    optimizer = optim.Adam(model_parameters, lr=0.001)

    criterion = nn.CrossEntropyLoss(ignore_index=-1).to(device)

    stats = defaultdict(list)

    for epoch in range(50):
        for step, batch in enumerate(trainloader):
            model.train()
            # Mask the batch
            inputs, labels = mask_tokens(batch, mask_prob=0.15,
                                         pad_token_id=vocab.get_id('<pad>'),
                                         mask_token_id=vocab.get_id('[M]'),
                                         vsize=len(vocab))
            inputs = inputs.to(device)
            labels = labels.to(device)

            logits = model(inputs)
            logits_ = logits.view(-1, logits.size(2))
            labels_ = labels.view(-1)

            optimizer.zero_grad()
            loss = criterion(logits_, labels_)

            loss.backward()
            optimizer.step()

            stats['train_loss'].append(loss.item())
            stats['train_loss_log'].append(loss.item())
            if (step % 500) == 0:
                avg_loss = sum(stats['train_loss_log']) / len(stats['train_loss_log'])
                print("Epoch %d Step %d\tTrain Loss %.3f" % (epoch, step, avg_loss))
                stats['train_loss_log'] = []

        for batch in validloader:
            model.eval()
            with torch.no_grad():
                # Mask the batch
                inputs, labels = mask_tokens(batch, mask_prob=0.15,
                                             pad_token_id=vocab.get_id('<pad>'),
                                             mask_token_id=vocab.get_id('[M]'),
                                             vsize=len(vocab))
                inputs = inputs.to(device)
                labels = labels.to(device)

                logits = model(inputs)
                logits_ = logits.view(-1, logits.size(2))
                labels_ = labels.view(-1)

                loss = criterion(logits_, labels_)
                stats['valid_loss'].append(loss.item())
        print("=== Epoch %d\tValid Loss %.3f" % (epoch, stats['valid_loss'][-1]))
```

### Example Conditionals

#### Load model

```python
device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')

checkpoint = utils.load('model', 'model', best=True)
options = checkpoint['options']
stats = checkpoint['stats']

model = utils.Transformer(len(vocab), options['max_len'],
                          dim=options['dim'],
                          nhead=options['nhead'])
model.load_state_dict(checkpoint['model_dict'])
```

    <All keys matched successfully>

```python
model.eval()
model = model.to(device)
```

```python
sentences = [['<s>', 'i', 'have', 'a', 'pet', '[M]', '.', '<s>'],
             ['<s>', 'i', 'have', 'two', 'pet', '[M]', '.', '<s>'],
             ['<s>', 'my', '[M]', 'is', 'a', 'lawyer', '.', '<s>'],
             ['<s>', 'my', '[M]', 'is', 'a', '[M]', '.', '<s>'],
             ['<s>', 'i', '[M]', '[M]', '[M]', 'sometimes', '.' , '<s>']]

def get_top_masked_tokens(tokens, vocab, device, top=10):
    ids = torch.tensor([vocab.get_id(x) for x in tokens], device=device).unsqueeze(1)
    masked = ids == vocab.get_id('[M]')

    logits = model(ids)[masked]
    probs = torch.softmax(logits, -1)

    print(' '.join(tokens))
    for ps in probs:
        probs, idxs = ps.sort(descending=True)

        for i in range(top):
            print("\t%s (%.4f)" % (vocab.get_token(idxs[i].item()),
                                   probs[i].item()))
        print()
```

```python
for s in sentences:
    get_top_masked_tokens(s, vocab, device)
```

    <s> i have a pet [M] . <s>
    	cat (0.0707)
    	dog (0.0533)
    	sibling (0.0342)
    	puppy (0.0340)
    	sister (0.0302)
    	retriever (0.0265)
    	daughter (0.0264)
    	shepard (0.0232)
    	named (0.0213)
    	brother (0.0208)

    <s> i have two pet [M] . <s>
    	cats (0.1525)
    	dogs (0.0874)
    	girls (0.0748)
    	boys (0.0501)
    	brothers (0.0499)
    	wives (0.0420)
    	children (0.0386)
    	kids (0.0377)
    	sisters (0.0333)
    	, (0.0219)

    <s> my [M] is a lawyer . <s>
    	mother (0.2872)
    	dad (0.2481)
    	mom (0.1561)
    	husband (0.0864)
    	father (0.0363)
    	brother (0.0230)
    	job (0.0144)
    	sister (0.0143)
    	parents (0.0131)
    	wife (0.0104)

    <s> my [M] is a [M] . <s>
    	mother (0.2330)
    	dad (0.2212)
    	mom (0.1373)
    	husband (0.1110)
    	brother (0.0364)
    	father (0.0357)
    	sister (0.0285)
    	job (0.0145)
    	wife (0.0141)
    	parents (0.0127)

    	teacher (0.0899)
    	lawyer (0.0456)
    	nurse (0.0426)
    	cop (0.0414)
    	mechanic (0.0386)
    	doctor (0.0259)
    	pilot (0.0195)
    	journalist (0.0163)
    	dancer (0.0148)
    	hairdresser (0.0123)

    <s> i [M] [M] [M] sometimes . <s>
    	am (0.1669)
    	love (0.1151)
    	like (0.0823)
    	do (0.0610)
    	have (0.0457)
    	can (0.0237)
    	hate (0.0237)
    	just (0.0199)
    	get (0.0172)
    	enjoy (0.0159)

    	a (0.1115)
    	in (0.0584)
    	not (0.0565)
    	to (0.0477)
    	my (0.0177)
    	the (0.0163)
    	i (0.0146)
    	into (0.0119)
    	for (0.0111)
    	like (0.0101)

    	, (0.0760)
    	. (0.0338)
    	right (0.0203)
    	but (0.0201)
    	up (0.0146)
    	and (0.0135)
    	not (0.0133)
    	music (0.0083)
    	food (0.0074)
    	coffee (0.0065)

## Back to *BERT*

**B**idirectional

**E**ncoder

**R**epresentations from

**T**ransformers

#### - Masked Language Modeling at scale

#### - Learned representations are useful downstream

#### Great implementation in [transformers](https://github.com/huggingface/transformers):

```python
# !pip install transformers
```

```python
import torch
from transformers import (
    BertForMaskedLM,
    BertTokenizer
)
```

### Details -- Model Variants

- $\text{BERT}_{\text{BASE}}$: 12 layers, hidden dimension 768, 12 attention heads (**110 million parameters**)
- $\text{BERT}_{\text{LARGE}}$: 24 layers, hidden dimension 1024, 16 attention heads (**340 million parameters**)

```python
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)
model = BertForMaskedLM.from_pretrained('bert-base-cased', output_attentions=True)

if torch.cuda.is_available():
    model.cuda()
```

    HBox(children=(FloatProgress(value=0.0, description='Downloading', max=213450.0, style=ProgressStyle(descripti…

    HBox(children=(FloatProgress(value=0.0, description='Downloading', max=361.0, style=ProgressStyle(description_…

    HBox(children=(FloatProgress(value=0.0, description='Downloading', max=435779157.0, style=ProgressStyle(descri…

### Details -- Input Implementation

- `[CLS]` token: starts each sequence. Used as aggregate sequence representation.
- `[SEP]` token: separates two segments (e.g. two sentences).
- **Segment embedding**: learned embedding for every token indicating whether it belongs
to sentence A or sentence B.
- **Position embedding**: learned.

**Exercise:** Which downstream tasks would two sequences be useful for?

### Tokenization

#### BERT represents text using **subword** tokens with a 30k token vocabulary.

(more info [here](https://github.com/google/sentencepiece) and in the papers mentioned there)

<!-- - **Token embedding**: WordPiece embeddings with 30k token vocabulary. -->

```python
tokenizer.tokenize("Pretraining is cool.")
```

    ['Pre', '##tra', '##ining', 'is', 'cool', '.']

```python
tokenizer.tokenize("BERT represents text using subwords.")
```

    ['B', '##ER', '##T', 'represents', 'text', 'using', 'sub', '##words', '.']

### Examining Learned Conditionals (& Representations)

**Probing tasks** can be used to examine aspects of what the model has learned.

Following [Petroni et al 2019](https://arxiv.org/pdf/1909.01066.pdf) we probe for '**knowledge**' that the model has learned by querying for masked out objects, e.g.:

The task also illustrates some aspects of the **conditional distributions** and **contextualized representations** that the model has learned.

(image from [Petroni et al 2019])

**Exercise:** The authors only consider *single-token* prediction. Why?

#### Probing Task

We use a dataset from [Petroni et al 2019](https://github.com/facebookresearch/LAMA).

```python
import utils
data = utils.load_lama_squad(download=True)
data[0]
```

    {'masked_sentences': ['To emphasize the 50th anniversary of the Super Bowl the [MASK] color was used.'],
     'obj_label': 'gold',
     'id': '56be4db0acb8001400a502f0_0',
     'sub_label': 'Squad'}

```python
results = []

model.eval()
for example in tqdm(data, total=len(data)):
    sentence, label = example['masked_sentences'][0], example['obj_label']
    inp = torch.tensor([
        [tokenizer.cls_token_id] +
        tokenizer.encode(sentence) +
        [tokenizer.sep_token_id]
    ], device=device)

    mask = (inp == tokenizer.vocab[tokenizer.mask_token])
    out, attn = model(inp)

    probs, token_ids = out[mask].softmax(1).topk(10)
    probs = probs[0].tolist()
    token_ids = token_ids[0].tolist()

    tokens = [tokenizer.ids_to_tokens[i] for i in token_ids]

    results.append({
        'sentence': sentence,
        'label': label,
        'top_tokens': tokens,
        'top_probs': probs,
        'correct@1': tokens[0] == label,
        'attn': attn
    })

print("correct@1: %.3f" % (
    len([r for r in results if r['correct@1']]) / len(results)
))
```

    100%|██████████| 305/305 [00:07<00:00, 38.36it/s]

    correct@1: 0.121

```python
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

correct = [r for r in results if r['correct@1']]
wrong = [r for r in results if not r['correct@1']]

def show(idx=0, attn_layer=0, is_correct=True):
    result = correct[idx] if is_correct else wrong[idx]

    # --- format the result into a string
    top_str = '\n\t'.join([
        ('\t%s\t(%.4f)' % (tokens, probs))
        for tokens, probs in zip(result['top_tokens'], result['top_probs'])
    ])
    print("%s\n\tlabel:\t%s\n\n\ttop:%s" % (
        result['sentence'],
        result['label'],
        top_str
    ))

    # --- visualize attention
    print("Attention weights (12 heads) from layer %d:" % attn_layer)
    fig, axs = plt.subplots(3, 4, figsize=(18, 12))

    toks = ['[CLS]'] + tokenizer.tokenize(result['sentence']) + ['[SEP]']
    for i, ax in enumerate(axs.reshape(-1)):
        ax.matshow(result['attn'][attn_layer][0][i].data.cpu().numpy(), cmap='gray')

        ax.set_xticks(range(len(toks)))
        ax.set_xticklabels(toks, rotation=90, fontsize=15)
        ax.set_yticks(range(len(toks)))
        ax.set_yticklabels(toks, fontsize=15)
    plt.tight_layout()

interactive(
    show,
    idx=(0, min(len(correct), len(wrong))-1),
    attn_layer=range(12),
    is_correct=True
)
```

    interactive(children=(IntSlider(value=0, description='idx', max=36), Dropdown(description='attn_layer', option…