# SysArmy Surveys: Analysis of argentine techies' salaries evolution (2014.02 - 2017.02)

## Datasets

The used datasets have been published on the SysArmy blog since they started taking the surveys.  Each survey contains differencies on their columns, mostly additions across time, but some names and order have changed to.

In order to normalize the different structures, the next section includes a
 couple of functions to facilitate this process.  Below, is the list of datasets used at the moment.

### Salaries

| Period  | Path             | URL                                                                                |
|---------|------------------|-------------------------------------------------------------------------------------------------------------|
| 2014.02 | data/2014.02.csv | https://drive.google.com/open?id=0B7UapTwn9AahWkdEWnltRjUwVjQ                                               |
| 2015.01 | data/2015.01.csv | https://drive.google.com/open?id=0B7UapTwn9AahSHl1aGNfTlNrYWc                                               |
| 2016.01 | data/2016.01.csv | https://drive.google.com/open?id=0B7UapTwn9AahLTJmSE5zOUQ0aGc                                               |
| 2016.02 | data/2016.02/    | https://drive.google.com/open?id=0B7UapTwn9AahUGtySFBjdHBCcWs                                               |
| 2017.01 | data/2017.01/    | https://docs.google.com/spreadsheets/d/1diZG7ruHJ4OgO2-tbnO-ZnhjxjOnYjMVSxl7I-4LT54/edit#gid=1424918437     |
| 2017.02 | data/2017.02/    | https://docs.google.com/spreadsheets/d/1f1CAmzbVtoL66IwXJWEf0Mn-QbMNGZIoCQ_b1HB-91E                         |

The datasets detailed above were downloaded and included "as is" in the `data/` folder on this repository.  The idea of this analysis is to not preprocess anything, and just normalize the data on the fly, of course this is possible given that the datasets are not so big.

### Argentine Peso AR\$ to US\$

In one of the analysis (GDP) we will compare the salaries through time using the dollar as currency of reference. We do this in order to understand the numbers in the global economy, putting aside the effects of inflation.
In relation to the inflation, the official numbers are not so good for an appropiate analysis: between 2014 and 2015 the methodoly changed, and it's been an entire year (April 2016 - April 2017) without measurement.

#### Links of interest

* https://es.tradingeconomics.com/argentina/inflation-cpi
* http://data.worldbank.org/indicator/NY.GDP.MKTP.CD?end=2016&locations=AR&start=1962&view=chart

| Period  | Path                 | URL                                                                                |
|---------|----------------------|---------------------------------------------------------------------------------------------------------|
| 5Y      | data/usdars-cur.json | https://www.bloomberg.com/markets/api/bulk-time-series/price/USDARS%3ACUR?timeFrame=5_YEAR              |

## Data preparation

```python
import pandas as pd
import numpy as np
import re
from matplotlib import cm
from functools import reduce
from IPython.display import display, Markdown as md

def getPeriodDF(period, publish_date, csv_path, columns_src, columns_dst, transforms = []):
    # given a csv_path, extract src columns into dst columns by applying a transform function for a certain period name
    df_src = pd.read_csv(csv_path)
    df_dst = pd.DataFrame(columns=['period', 'publish_date']+columns_dst)
    df_dst[columns_dst] = df_src[columns_src]
    df_dst['period'] = period
    df_dst['publish_date'] = pd.to_datetime(publish_date)
    # apply custom transformations
    for transform in transforms:
        df_src, df_dst = transform(df_src, df_dst)
    return df_dst

def transformNormalize(df_src, df_dst):
    # cast to float the salary column, some datasets gets wrongly parsed due to crappy user input
    df_dst['salary'] = df_dst['salary'].astype(float)
    df_dst['age'] = df_dst['age'].astype(str)
    # normalize stack column
    re_separator = ' *, *| *; *| *\. *| *\| *| *\/ *| *- *'
    re_garbage = '"""|\*|nan|='
    df_dst['stack'] = df_dst['stack'].apply(lambda s: re.split(re_separator, re.sub(re_garbage, '', str(s)).strip().lower()))
    # munge the dataset, removing entries that we consider not realistic for our analysis
    munging_mask = (df_dst.salary > 5000) & (df_dst.salary < 300000)
    df_src = df_src.loc[munging_mask]
    df_dst = df_dst.loc[munging_mask]
    return df_src, df_dst

def transformNet2Brute(df_src, df_dst):
    # filter wether net or brute income
    mask = np.array(df_src['Bruto o neto?'].str.contains('Neto'), dtype = bool)
    net_factor = np.array([1.0] * len(df_dst));
    # scales up net salary into brute, given Argentina's social security contributions
    net_factor[ mask ] = 1/0.83
    df_dst['salary'] = net_factor * df_dst['salary']
    return df_src, df_dst

def transformNet2Brute201702(df_src, df_dst):
    # filter wether net or brute income
    mask = np.array(df_src['¿Porcentaje, bruto o neto?'].str.contains('Neto'), dtype = bool)
    net_factor = np.array([1.0] * len(df_dst));
    # scales up net salary into brute, given Argentina's social security contributions
    net_factor[ mask ] = 1/0.83
    df_dst['salary'] = net_factor * df_dst['salary']
    return df_src, df_dst

src_common_cols = [
    'Tengo',
    'Años de experiencia',
    'Años en el puesto actual',
    'Trabajo de',
    'Tecnologías que utilizás']

dst_cols = [
    'salary',
    'location',
    'age',
    'yoe',
    'yip',
    'role',
    'stack']

df1 = getPeriodDF('2014.02', '2015-01-01', '../data/2014.02.csv',
                  ['Salario bruto mensual (AR$)', 'Trabajo en'] + src_common_cols, dst_cols,
                  [transformNormalize])

df2 = getPeriodDF('2015.01', '2015-09-01', '../data/2015.01.csv',
                  ['Salario bruto mensual (AR$)', 'Trabajo en'] + src_common_cols, dst_cols,
                  [transformNormalize])

df3 = getPeriodDF('2016.01', '2016-02-01', '../data/2016.01.csv',
                  ['Salario mensual (AR$)', 'Trabajo en'] + src_common_cols, dst_cols,
                  [transformNormalize, transformNet2Brute])

df4 = getPeriodDF('2016.02', '2016-08-01', '../data/2016.02/argentina.csv',
                  ['Salario mensual (en tu moneda local)', 'Argentina'] + src_common_cols, dst_cols,
                  [transformNormalize, transformNet2Brute])

df5 = getPeriodDF('2017.01', '2017-02-01', '../data/2017.01/argentina.csv',
                  ['Salario mensual (en tu moneda local)', 'Argentina'] + src_common_cols, dst_cols,
                  [transformNormalize, transformNet2Brute])

df6 = getPeriodDF('2017.02', '2017-09-01', '../data/2017.02/argentina.csv',
                  ['Salario mensual (en tu moneda local)', 'Argentina'] + src_common_cols, dst_cols,
                  [transformNormalize, transformNet2Brute201702])

# compute the union of all the datasets as a signe pandas dataframe
df = pd.concat([df1, df2, df3, df4, df5, df6])

# extract the list of periods
periods = df.period.unique()

# extract the list of roles
roles = df.role.unique()

# generate a list of colors by period for easy visualization on next figures
colors = cm.rainbow(np.linspace(0, 1, len(periods)))

# extract the list of stacks
all_stacks = reduce(lambda res, it: np.concatenate((res, it)), df['stack'], np.array([]))
data = np.transpose(np.unique(all_stacks, return_counts=True))
dfs = pd.DataFrame(data, columns=['stack', 'count'])
dfs['count'] = dfs['count'].astype(int)

# extract the list of roles
data = np.transpose(np.unique(df['role'].apply(lambda s: s.strip()), return_counts=True))
dfr = pd.DataFrame(data, columns=['role', 'count'])
dfr['count'] = dfr['count'].astype(int)

# create a dataframe with the most frequent stack names and mask of rows in main df matching it
dfs = dfs.query('count > 100 & stack != ""').sort_values(by=['count'], ascending=[0])
dfs['df_mask'] = dfs['stack'].apply(lambda stack: np.array(df['stack'].apply(lambda s: np.isin(stack, s))))

# create a dataframe with the most frequent role names and mask of rows in main df matching it
dfr = dfr.query('count > 100 & role != ""').sort_values(by=['count'], ascending=[0])
dfr['df_mask'] = dfr['role'].apply(lambda role: np.array(df['role'] == role))

# show results
display(md('# Normalized data'))
display(md('## Survey entries'))
display(df.head())
# we take only the top stacks/roles because the fields were free-text, and this resulted
# in many outliers, ie: random text without continuity for a proper time-series analysis
display(md('## Top stacks: names, frquency and entries mask'))
display(dfs.head())
display(md('## Top roles: names, frquency and entries mask'))
display(dfr.head())
```

# Normalized data

## Survey entries

      period
      publish_date
      salary
      location
      age
      yoe
      yip
      role
      stack

      0
      2014.02
      2015-01-01
      21000.00
      Ciudad Autónoma de Buenos Aires
      27 - 30
      5 - 7
      Menos de un año
      SysAdmin
      [linux, cloud]

      1
      2014.02
      2015-01-01
      10000.00
      Ciudad Autónoma de Buenos Aires
      35 - 40
      10+
      2 - 4
      SysAdmin
      [linux]

      2
      2014.02
      2015-01-01
      16000.00
      Córdoba
      27 - 30
      3 - 5
      1 - 2
      DevOps
      [linux, cloud]

      3
      2014.02
      2015-01-01
      23771.13
      Ciudad Autónoma de Buenos Aires
      27 - 30
      5 - 7
      1 - 2
      SysAdmin
      [linux]

      4
      2014.02
      2015-01-01
      16000.00
      Ciudad Autónoma de Buenos Aires
      30 - 33
      10+
      2 - 4
      DevOps
      [linux]

## Top stacks: names, frquency and entries mask

      stack
      count
      df_mask

      712
      windows
      4958
      [False, False, False, False, False, False, Fal...

      320
      linux
      3988
      [True, True, True, True, True, False, False, F...

      692
      vmware
      2193
      [False, False, False, False, False, False, Fal...

      124
      containers
      817
      [False, False, False, False, False, False, Fal...

      154
      docker
      817
      [False, False, False, False, False, False, Fal...

## Top roles: names, frquency and entries mask

      role
      count
      df_mask

      480
      Developer
      6195
      [False, False, False, False, False, False, Fal...

      1409
      SysAdmin / DevOps
      2905
      [False, False, False, False, False, False, Fal...

      679
      HelpDesk
      2162
      [False, False, False, False, False, False, Fal...

      976
      Networking
      1142
      [False, False, False, False, False, False, Tru...

      1057
      PM
      767
      [False, False, False, False, False, True, Fals...

```python
import scipy.stats as sps
import matplotlib.pyplot as plt

def getPlotSalarySeries(currencySeries):
    s = currencySeries.apply(lambda x: float('{0:.2f}'.format(round(x/1000, 1))))
    mu, sigma = np.mean(s), np.std(s)
    median = np.median(s)
    t = np.linspace(mu - 3*sigma, mu + 3*sigma, 1000)
    d = sigma**2/mu
    return mu, sigma, t, d, median, s

dfrt = pd.DataFrame(columns=['role', 'count', 'mu', 'sigma', 'period', 'publish_date'])

i = 0
for period in periods:
    for j, row in dfr.iterrows():
        dfrs = df.loc[ row['df_mask'] & (df['period'] == period) ]
        if len(dfrs) == 0: continue
        mu, sigma, t, d, m, s = getPlotSalarySeries(dfrs['salary'])
        dfrt.loc[i] = [row['role'], len(dfrs), mu, sigma, period, np.max(dfrs['publish_date'])]
        i = i+1

# the ranking of technologies most frequent in the datasets order by mean of salary
dfrt = dfrt.query('count > 1').sort_values(by=['role', 'period'], ascending=[0, 0])
display(md('## Top roles: numbers by period'))
dfrt.head()
```

## Top roles: numbers by period

      role
      count
      mu
      sigma
      period
      publish_date

      59
      SysAdmin / DevOps
      639.0
      34.385915
      15.772861
      2017.02
      2017-09-01

      47
      SysAdmin / DevOps
      640.0
      28.695000
      14.261597
      2017.01
      2017-02-01

      35
      SysAdmin / DevOps
      636.0
      24.607075
      11.653473
      2016.02
      2016-08-01

      23
      SysAdmin / DevOps
      990.0
      20.572828
      13.491941
      2016.01
      2016-02-01

      17
      SysAdmin
      248.0
      18.471371
      8.054694
      2015.01
      2015-09-01

```python
import matplotlib.patches as mpatches

%matplotlib inline
plt.style.use(['classic', 'ggplot'])
plt.rcParams.update({'figure.figsize': (10.0, 6.0), 'font.size': 8})

trending_roles = np.unique(dfrt['role'])
roles_colors = cm.rainbow(np.linspace(0, 1, len(trending_roles)))
publish_dates = np.unique(dfrt['publish_date'])

y = []
for i, role in enumerate(trending_roles):
    role_y = []
    for j, date in enumerate(publish_dates):
        dfrs = dfrt.query('role == @role & publish_date == @date')
        count = 0 if len(dfrs['count']) == 0 else dfrs['count'].values[0]
        role_y.append(count)
    y.append(role_y)

# define data points for stackplot
x = publish_dates
y = np.row_stack(y)
percent = np.divide(y, y.sum(axis=0).astype(float) * 100)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.stackplot(x, percent, colors=roles_colors)
ax.set_title('Level of participation in surveys by role across time')
ax.set_ylabel('Percent (%)')

# creating the legend manually
plt.legend([mpatches.Patch(color=roles_colors[i]) for i, role in enumerate(trending_roles)],
           trending_roles,
           bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

display(md('## Level of participation in surveys by role'))
display(md("""
This initial graph aims to help understanding who participated in the surveys on each period.

It is very important to understand the universe of samples, which may be a segment of the entire tech community in Argentina. There're reasons to think on this.
A valid one is that SysArmy may have started as group of sysadmins/security guys -mainly, perhaps due to the topics they post about-, then spreaded more to the devs community.

In order to prove this hypothesis, we graph the percentual level of participation by role (only top roles).
"""))
plt.show()
display(md("""
The results threw that Developers were from the beginning the majority group.  This is against the initial hypothesis, but not so falsy, neither the last conclusion:
1. It is clearly visible that the Developers group has grew through time starting on September 2015.
2. Helpdesk however, have decreased since February 2016.
3. "Storage / Backup" have falled and then dissapeared.
4. "Otro" (others) has been raising from the begining -then dissapeared-, but marked the tendency of the universe of samples which tends to diverge in respect to the role.
5. The rest of roles maintain some consistency over time.
"""))
```

## Histogram of Incomes by period (Argentina)

```python
%matplotlib inline
plt.rcParams.update({'figure.figsize': (10.0, 18.0), 'font.size': 8})
fig, ax = plt.subplots(len(periods), 1)
for i, period in enumerate(periods):
    dft = df[df['period'] == period]
    mu, sigma, t, d, m, s = getPlotSalarySeries(dft.salary)
    n, bins, patches = ax[i].hist(s, 160, normed=1, alpha=0.75, color=colors[i])
    ax[i].set_ylabel('density')
    ax[i].set_title(period + ' Histogram of Income')
    ax[i].text(40, .025, '$\mu=%.2f,\ \sigma=%.2f,\ n=%d$' % (mu, sigma, len(s)))
    ax[i].axis([0, 160, 0, 0.1])
    ax[i].grid(True)
ax[len(periods)-1].set_xlabel('income [1000 AR$/mo]')
fig.canvas.draw()

display(md('## Density by range of salary'))
display(md("""
The following figures shows the density ranges of income for the Universe (Argentina samples) across time.
The histograms helps understand the probability density functions behind the data, which present some postive skew,
and a tendency to be shifted to the right through time (inflation adjustments).

It is also noticeable that the levels of dispersion increases since 2015, something that we will analyize next.
"""))
display(plt.show())
```

## Mean and Standard deviation evolution

The following graph represents the income mean $\mu$ across time, along with the standard deviation $\sigma$.
This is an interesting time-series information, because describes better the raise in currency numbers (not purchasing power) of salaries.  The tendency is the growthness, which is for sure good, reflecting a correction over time of the numbers due to the inflation.

The next question would be, Are these corrections enough to at least maintain some stability on the purchasing levels?
To answer this, we may need information about levels of inflation.  As described in the Datasets section, there's no good data available of inflation in Argentina.  A workaround to help answer this question, from a different perspective, but a valid one, is to base the analysis on the dollar currency and see the evolution of income (cotinue reading).

```python
%matplotlib inline
plt.rcParams.update({'figure.figsize': (10.0, 6.0), 'font.size': 8})
mus, sigmas, dates = [], [], []
for period in periods:
    dft = df[df['period'].str.contains(period)]
    mu, sigma, t, d, m, s = getPlotSalarySeries(dft.salary)
    mus.append(mu)
    sigmas.append(sigma)
    dates.append(np.max(dft['publish_date']))

plt.plot(dates, mus, '+', label='$\mu$', ls='-', c='b')
plt.plot(dates, sigmas, '+', label='$\sigma$', ls=':', c='g')
plt.xlabel('date'); plt.ylabel('income [1000 AR$/mo]')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
```

### Currency evolution (Argentine Peso)

Continuing with the previous analysis, we obtained the data of AR\$ to US\$ over time, of the last 5 years.  The next step would be to sample from these series, only few points of reference for the periods that our surveys datasets correspond.

So we plot below, the peso evolution in green, and overlapped in blue, the sampled values of these series for the periods of our interest.

```python
import json
import datetime
with open('../data/usdars-cur.json') as data_file:
    data = json.load(data_file)
    df_currency = pd.DataFrame(data[0]['price'])
    df_currency.date = pd.to_datetime(df_currency.date)

%matplotlib inline
plt.rcParams.update({'figure.figsize': (10.0, 6.0), 'font.size': 8})
delta = datetime.timedelta(days=15)
currencies = []
for date in dates:
    currency_series = df_currency[(df_currency.date >= date-delta) & (df_currency.date <= date+delta)]
    currencies.append(float(currency_series.value[:1]))

plt.xlabel('date'); plt.ylabel('1US\$ [AR\$]')
plt.plot(dates, currencies, '+', label='sampled currency for the chosen periods', ls='-', c='b')
plt.plot(df_currency.date, df_currency.value, label='by date currency from Bloomberg.com', ls='-', c='g')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
```

## Monthly nominal GDP

Once obtained the peso value in dollar currency, we calculate the salaries mean $\mu$ over time, in dollars:
Monthly Gross Domestic Product, in US dollars, tell us the purchasing capacity of Argentines salaries in nominal values.  To compute the values, we just divide each $\mu$ in pesos by the dollar rate at its given moment.

$$
\sum_i{nom(GDP_i)} = \sum_i{\mu_i US\$} = \sum_i{\frac {\mu_i AR\$}{exchrate_i{\frac {AR\$}{US\$}}}}
$$

The results can be compared with World Bank data for Argentina provided here:

http://data.worldbank.org/indicator/NY.GDP.PCAP.PP.CD?end=2016&locations=AR&start=2010

```python
%matplotlib inline
plt.rcParams.update({'figure.figsize': (13.0, 5.0), 'font.size': 8})

fig, ax = plt.subplots(1, 2)
ax[0].set_xlabel('date'); ax[0].set_ylabel('income [1000/mo]')
ax[0].plot(dates, mus, '+', label='$\mu$ AR\$', ls='-', c='b')
ax[0].legend()

ax[1].set_xlabel('date'); ax[1].set_ylabel('income [1000/mo]')
ax[1].plot(dates, np.divide(mus, currencies), '+', label='$\mu$ US\$', ls='-', c='r')
ax[1].legend()
fig.canvas.draw()

display(md("""
### Evolution of mean income in AR$ (left), in US\$ (right)

As the figure show (left), the salary in pesos continuosly grows.
There's however, an interesting plateau in between the end of 2015 and beginning of 2016.
The two periods occurred before and after the Argentina presidential elections of October/December 2015.
It mustn't be coincidence that the numbers got practically freezed, perhaps due to the uncertainty caused by such national event.

There's another clear insight that we can identify on the figure at the right side.
Sadly the income in dollars falled down dramatically right after the elections.  This was due to the removal of the "cepo cambiario",
which came along with a drastic devaluation of the local currency: http://www.bbc.com/mundo/noticias/2015/12/151217_argentina_fin_cepo_devaluacion_irm
"""))
display(plt.show())
```

## Distribution of Income by period (Argentina)

### Assuming Normal distribution

Despite the fact that histograms throw distributions more similar to a $\gamma$ (Gamma) function, and the marked possitive skewness, for simplicity we will use the $N$ (Gaussian) distribution.

* Probability density function

$$
\large
{\displaystyle f(x\;|\;\mu ,\sigma ^{2})={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}\;e^{-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}}}
$$

* Gaussian density paramters

$$
\large
\begin{align}
\mu &= \frac{1}{N} \sum_i x_i \\
\sigma^2 &= \frac{1}{N} \sum_i x_i^2
\end{align}
$$

The following figure, shows for the full dataset by period, the mean ($\mu$), and the standard deviation ($\sigma$).  It is noticieable the shifting to the right of the curves across time, indicating that salaries somehow followed the inflation tendency.

A new insight can be distinguished from this graph: the standard deviation of the salaries (the amplitude of the gaussian shape) starts increasing since 2016.  Once again, after the presidential elections, the course of the economy changed, and then the rules and behavior of salaries.
Here, an interpretation would be, that the tendency of incomes is now more sparse, meaning that for the same type of work and experience -in general- there's people with totally different salaries (wider ranges).

```python
%matplotlib inline
plt.rcParams.update({'figure.figsize': (10.0, 6.0), 'font.size': 8})
plt.xlim(0,100)
for i, period in enumerate(periods):
    dft = df[df.period.str.contains(period)]
    mu, sigma, t, d, m, s = getPlotSalarySeries(dft.salary)
    plt.plot(t, sps.norm.pdf(t, mu, sigma), label=period + '\n$\mu=%.2f,\ \sigma=%.2f,\ n=%d$\n' % (mu, sigma, len(s)), ls='-', c=colors[i])
    plt.axvline(x=mu, ls=':', c=colors[i])
plt.xlabel('income [1000 AR$/mo]'); plt.ylabel('density, $N$')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
```

## Interactive dynamic dashboard

The next figure is dynamic. It has some controls that lets you to filter data and so, refresh the visualization comparing diferent dimensionality reductions, with the average by dataset, using the probability denstiy functions as curve.

_NOTE: in order to make this work, you may need to run this notebook locally, using Jupyter Notebooks_

```python
from IPython.display import clear_output
import ipywidgets as widgets

filters = ['role', 'yoe', 'yip', 'location', 'age']
i_filters = [{'name': r, 'i': i} for i, r in enumerate(filters)]
filters_opts = list(map(lambda r: list(df[r].unique()), filters))
filters_checkboxes = list(map(lambda r: widgets.Checkbox(value=False, description='Use '+r, name='use_'+r, disabled=False), filters))
filters_dropdowns = list(map(lambda r: widgets.Dropdown(options=filters_opts[r['i']], value=filters_opts[r['i']][0], name=r['name'], description=r['name']+':', disabled=not filters_checkboxes[r['i']].value), i_filters))

%matplotlib notebook
plt.rcParams.update({'figure.figsize': (13.0, 6.0), 'font.size': 8})
fig, ax = plt.subplots()
ax.set_xlim(-20,100)
ax.set_ylim(0,0.08)
ax.set_xlabel('income [1000 AR$/mo]'); ax.set_ylabel('density, $N$')

def on_filters_change(change):
    for i in range(len(filters)):
        checkbox = filters_checkboxes[i]
        dropdown = filters_dropdowns[i]
        if change['owner'].name == checkbox.name:
            dropdown.disabled = not change['new']
            break
    plot_filtered_monthly_income()

def plot_filtered_monthly_income(change = None):
    plt.rcParams.update({'figure.figsize': (10.0, 8.0), 'font.size': 8})
    if not ax.lines:
        for i, period in enumerate(periods):
            dft = df[df.period.str.contains(period)]
            mu, sigma, t, d, m, s = getPlotSalarySeries(dft.salary)
            ax.plot(t, sps.norm.pdf(t, mu, sigma), label=period, ls=':', c=colors[i])
            ax.plot(t, sps.norm.pdf(t, mu, sigma), label=period + ' $\mu=%.2f,\ \sigma=%.2f,\ n=%d$' % (mu, sigma, len(s)), ls='-', c=colors[i])
    else:
        for i, period in enumerate(periods):
            dft = df[df.period.str.contains(period)]
            for dropdown in filters_dropdowns:
                dft = dft if dropdown.disabled else dft[dft[dropdown.name].str.contains(dropdown.value)]
            mu, sigma, t, d, m, s = getPlotSalarySeries(dft.salary)
            line = ax.lines[2*i+1]
            if len(dft) > 2:
                line.set_xdata(t)
                line.set_ydata(sps.norm.pdf(t, mu, sigma))
                line.set_label(period + ' $\mu=%.2f,\ \sigma=%.2f,\ n=%d$' % (mu, sigma, len(s)))
            else:
                line.set_xdata([]), line.set_ydata([])
    title = 'Segment:' + ''.join([
        '' if dropdown.disabled else ' %s: %s' % (dropdown.name, dropdown.value)
            for dropdown in filters_dropdowns])
    ax.set_title(title)
    ax.legend(bbox_to_anchor=(1.05, 1), loc=1, borderaxespad=0.)
    ax.grid(True)
    fig.canvas.draw()
    plt.ion()
    plt.show()

boxes = []
for i in range(len(filters_dropdowns)):
    filters_checkboxes[i].observe(on_filters_change, names='value')
    filters_dropdowns[i].observe(plot_filtered_monthly_income, names='value')
    boxes.append(widgets.Box([filters_checkboxes[i], filters_dropdowns[i]]))

display(widgets.Box(boxes))
plot_filtered_monthly_income()
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

## Comparative analysis by technology

Below, we will rank technologies by most frequent.  We will exclude technologies with less than 100 samples in total -because they are in most cases, isolated, and doesn't follow clear trends.

From the resulting table we will plot the results in two ways:

1. First as a time-series comparative of the technologies by period, indicatin the $\mu$ mean salary.
2. Secondly, as a scatter, having the salary mean $\mu$ in the X axis, the standard deviation $\sigma$ in the Y axis and the amount of samples as the size of the dot.

```python
dfst = pd.DataFrame(columns=['stack', 'count', 'mu', 'sigma', 'period', 'publish_date'])

i = 0
for period in periods:
    for j, row in dfs.iterrows():
        dfss = df.loc[ row['df_mask'] & (df['period'] == period) ]
        if len(dfss) == 0: continue
        mu, sigma, t, d, m, s = getPlotSalarySeries(dfss['salary'])
        dfst.loc[i] = [row['stack'], len(dfss), mu, sigma, period, np.max(dfss['publish_date'])]
        i = i+1

# the ranking of technologies most frequent in the datasets order by mean of salary
dfst = dfst.query('count >= 1').sort_values(by=['stack', 'period'], ascending=[0, 0])
display(md('## Top stacks: numbers by period'))
dfst.head()
```

## Top stacks: numbers by period

      stack
      count
      mu
      sigma
      period
      publish_date

      61
      windows
      459.0
      32.508061
      15.077103
      2017.02
      2017-09-01

      48
      windows
      2844.0
      25.242827
      13.018835
      2017.01
      2017-02-01

      34
      windows
      512.0
      23.590430
      9.715138
      2016.02
      2016-08-01

      19
      windows
      831.0
      19.829122
      13.524452
      2016.01
      2016-02-01

      9
      windows
      177.0
      18.333898
      11.019761
      2015.01
      2015-09-01

```python
%matplotlib inline
plt.rcParams.update({'figure.figsize': (10.0, 6.0), 'font.size': 8})

trending_stacks = np.unique(dfst['stack'])
stacks_colors = cm.rainbow(np.linspace(0, 1, len(trending_stacks)))
for i, stack in enumerate(trending_stacks):
    dfp = dfst.query('stack == @stack')
    plt.plot(dfp['publish_date'], dfp['mu'], '+', label='$\mu$ AR\$ %s' % stack, ls='-', c=stacks_colors[i])
    plt.xlabel('date'); plt.ylabel('$\mu$')
    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
display(md('## Mean salary of the most frequent technologies (stacks) through time'))
```

```python
%matplotlib inline
plt.rcParams.update({'figure.figsize': (13.0, 30.0), 'font.size': 10})

fig, ax = plt.subplots(len(periods), 1)
for i, period in enumerate(periods):
    ## plot only the top ranked technologies in terms of salary for the period sample
    dftss = dfst.query("@period == period").sort_values(by=['mu', 'sigma'], ascending=[0, 0])[:30]
    ax[i].scatter(
        dftss['sigma'], dftss['mu'], marker='o', c=dftss.index, s=100*dftss['count']**(1/2), alpha=0.3,
        cmap=plt.get_cmap('Spectral'))

    for label, x, y in zip(dftss['stack'], dftss['sigma'], dftss['mu']):
        ax[i].annotate(
            label,
            xy=(x, y), xytext=(30, -30),
            textcoords='offset points', ha='right', va='bottom',
            bbox=dict(boxstyle='round,pad=0.5', fc='white', alpha=0.5),
            arrowprops=dict(arrowstyle = '->', connectionstyle='arc3,rad=0'))

    ax[i].set_xlabel('$\mu$'); ax[i].set_ylabel('$\sigma$')
    ax[i].set_title('%s - Technologies by salary, standard deviation and num of samples.' % period)
    fig.canvas.draw()
display(md('## Scatter of technologies by mean, dispersion, frequency and time'))
display(md('_NOTE: It\'s pending for these results to drive some conclusion (WIP)._'))
display(md('_NOTE: The circles radius represents the frequency (number of entries) by stack in an inversely cuadratic representation._'))
```


```python
try:
    import openmdao.api as om
    import dymos as dm
except ImportError:
    !python -m pip install openmdao[notebooks]
    !python -m pip install dymos[docs]
    import openmdao.api as om
    import dymos as dm
```

# Organizing Phases into Trajectories

The majority of real-world use cases of optimal control involve complex trajectories that cannot be modeled with a single phase.
For instance, different phases of a trajectory may have different equations of motion, different control parameterizations, or different path constraints.
Phases are also necessary if the user wishes to impose intermediate constraints upon some variable, by imposing them as boundary constraints at a phase junction.

The *Trajectory* class in Dymos is intended to simplify the development of multi-phase problems.
It serves as a Group which contains the various phases belonging to the trajectory, and it provides linkage constraints that dictate how phases are linked together.
This enables trajectories that are not only a sequence of phases in time, but may include branching behavior, allowing us to do things like track/constrain the path of a jettisoned rocket stage.

It supports a `get_values` method similar to that of Phases that allows the user to retrieve the value of a variable within the trajectory.
When verifying an answer with explicit simulation, the `simulate` method of Trajectory can simulate all of its member phases in parallel, providing a significant performance improvement for some cases.

## Instantiating a Trajectory

Instantiating a Trajectory is simple.  Simply invoke `Trajectory()`.  The trajectory object
itself is an OpenMDAO `Group` which serves as a container for its constituent Phases.

- phases
    An OpenMDAO `Group` or `ParallelGroup` holding the member phases
- linkages
    A Dymos `PhaseLinkageComp` that manages all of the linkage constraints that dictate how the phases are connected.

## Adding Phases

Phases are added to a Trajectory using the `add_phase` method.

```{eval-rst}
    .. automethod:: dymos.Trajectory.add_phase
        :noindex:
```

##  Defining Phase Linkages

Having added phases to the Trajectory, they now exist as independent Groups within the OpenMDAO model.
In order to enforce continuity among certain variables across phases, the user must declare which variables are to be continuous in value at each phase boundary.
There are two methods in dymos which provide this functionality.
The `add_linkage_constraint` method provides a very general way of coupling two phases together.
It does so by generating a constraint of the following form:

\begin{align}
    c = \mathrm{sign}_a \mathrm{var}_a + \mathrm{sign}_b \mathrm{var}_b
\end{align}

Method `add_linkage_constraint` lets the user specify the variables and phases to be compared for this constraint, as well as the location of the variable in each phase (either 'initial' or 'final')
By default this method is setup to provide continuity in a variable between two phases:
- the sign of variable `a` is +1 while the sign of variable `b` is -1.
- the location of variable `a` is 'final' while the location of variable `b` is 'initial'.
- the default value of the constrained quantity is 0.0.

In this way, the default behavior constrains the final value of some variable in phase `a` to be the same as the initial value of some variable in phase `b`.
Other values for these options can provide other functionality.
For instance, to simulate a mass jettison, we could require that the initial value of `mass` in phase `b` be 1000 kg less than the value of mass at the end of phase `a`.
Providing arguments `equals = 1000, units='kg` would achieve this.

Similarly, specifying other values for the locations of the variables in each phase can be used to ensure that two phases start or end at the same condition - such as the case in a branching trajectory or a rendezvous.

While `add_linkage_constraint` gives the user a powerful capability, providing simple state and time continuity across multiple phases would be a very verbose undertaking using this method.
The `link_phases` method is intended to simplify this process.
In the finite-burn orbit raising example, there are three phases:  `burn1`, `coast`, `burn2`.
This case is somewhat unusual in that the thrust acceleration is modeled as a state variable.
The acceleration needs to be zero in the coast phase, but continuous between `burn1` and `burn2`, assuming no mass was jettisoned during the coast and that the thrust magnitude doesn't change.

### add_linkage_constraint

```{eval-rst}
    .. automethod:: dymos.Trajectory.add_linkage_constraint
        :noindex:
```

### link_phases

```{eval-rst}
    .. automethod:: dymos.Trajectory.link_phases
        :noindex:
```

## Examples of using the `link_phases` method

**Typical Phase Linkage Sequence**

A typical phase linkage sequence, where all phases use the same ODE (and therefore have
the same states), are simply linked sequentially in time.

```python
t.link_phases(['phase1', 'phase2', 'phase3'])
```

**Adding an Additional Linkage**

If the user wants some control variable, `u`, to be continuous in value between `phase2` and
`phase3` only, they could indicate that with the following code:

```python
t.link_phases(['phase2', 'phase3'], vars=['u'])
```

**Branching Trajectories**

For a more complex example, consider the case where there are two phases which branch off
from the same point, such as the case of a jettisoned stage.  The nominal trajectory
consists of the phase sequence `['a', 'b', 'c']`.  Let phase `['d']` be the phase that tracks
the jettisoned component to its impact with the ground.  The linkages in this case
would be defined as:

```python
t.link_phases(['a', 'b', 'c'])
t.link_phases(['b', 'd'])
```

**Specifying Linkage Locations**

Phase linkages assume that, for each pair, the state/control values at the end (`'final'`)
of the first phase are linked to the state/control values at the start of the second phase
(`'initial'`).

The user can override this behavior, but they must specify a pair of location strings for
each pair given in `phases`.  For instance, in the following example phases `a` and `b`
have the same initial time and state, but phase `c` follows phase `b`.  Note since there
are three phases provided, there are two linkages and thus two pairs of location
specifiers given.

```python
t.link_phases(['a', 'b', 'c'], locs=[('initial', 'initial'), ('final', 'initial')])
```

##  Trajectory-Level Parameters

Often times, there are parameters which apply to the entirety of a trajectory that potentially need to be optimized.
If we implemented these as parameters within each phase individually, we would need some constraints to ensure that they held the same value within each phase.
To avoid this complexity, Dymos Trajectory objects support their own Parameters.

Like their Phase-based counterparts, Trajectory parameters produce may be design variables for the problem or used as inputs to the trajectory from external sources.

When using Trajectory parameters, their values are connected to each phase as an Input Parameter within the Phase.
Because ODEs in different phases may have different names for parameters (e.g. 'mass', 'm', 'm_total', etc) Dymos allows the user to specify the targeted ODE parameters on a phase-by-phase basis using the `targets` and `target_params` option.
It can take on the following values.

*  If `targets` is `None` the trajectory parameter will be connected to the phase input parameter of the same name in each phase, if it exists (otherwise it is not connected to that phase).

*  Otherwise targets should be specified as a dictionary. And the behavior depends on the value associated with each phase name:

    * If the phase name is not in the given dictionary, attempt to connect to an existing parameter of the same name in that phase.

    * If the associated value is None, explicitly omit a connection to that phase.

    * If the associated value is a string, connect to an existing input parameter whose name is given by the string in that phase.

    * If the associated value is a Sequence, create an input parameter in that phase connected to the ODE targets given by the Sequence.

## Explicit Simulation of Trajectories

The `simulate` method on Trajectory is similar to that of the `simulate` method of Phases.  When
invoked, it will perform a simulation of each Phase in the trajectory.


# Practical 3A: Reinforcement Learning

© Deep Learning Indaba. Apache License 2.0.

Adapted for Khipu.

## Introduction

In this practical, we will cover the basics of reinforcement learning, which has successfully been used to [control robotic hands](https://openai.com/blog/learning-dexterity/); play [Chess](https://deepmind.com/blog/article/alphazero-shedding-new-light-grand-games-chess-shogi-and-go), [Go](https://en.wikipedia.org/wiki/AlphaGo), and [StarCraft](https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii). It can even be used for supervised learning problems. For example, training a neural net to optimize a non-differentiable objective (such as accuracy, or the [BLEU score in machine translation](https://arxiv.org/abs/1609.08144)) or [finding a good neural net architecture](https://arxiv.org/abs/1611.01578) for a particular supervised learning problem.

## Learning Objectives

* Understand the problem reinforcement learning tries to solve.
* Understand the terminology used in reinforcement learning: **environment**, **action**, **observation**, **reward**, **agent**,  **policy**, **episode**.
* Learn how to use the [OpenAI Gym](https://gym.openai.com/) reinforcement learning environments.
* Learn how to solve the classic reinforcement learning problem of balancing a pole on a moving cart using Random Search and Policy Gradients. Understand some of the limitations of these methods.

**IMPORTANT: Please fill out the exit ticket form before you leave the practical: https://forms.gle/FLQ78eAEQprsWXp38**

## Imports

```python
#@title Imports (RUN ME!)

# Note: enviroments like CartPole-v0 require a display to render. We need to install pyvirtualdisplay etc
# in order to render from these environments

!pip install pyglet~=1.3.2 > /dev/null 2>&1
!pip install 'gym[atari]' > /dev/null 2>&1
!apt-get install python-opengl -y > /dev/null 2>&1
!apt install xvfb -y > /dev/null 2>&1
!pip install pyvirtualdisplay > /dev/null 2>&1
!pip install tensorflow==2.0.0-beta0 > /dev/null 2>&1

from IPython.display import HTML

# Start virtual display
from pprint import pprint
import logging
from pyvirtualdisplay import Display
logging.getLogger("pyvirtualdisplay").setLevel(logging.ERROR)

display = Display(visible=0, size=(1024, 768))
display.start()
import os
os.environ["DISPLAY"] = ":" + str(display.display) + "." + str(display.screen)
```

## The Reinforcement Learning Problem

So far we have encountered **supervised learning**, where we have an input and a target value or class that we want to predict. We have also encountered **unsupervised learning**, where we are only given an input and look for patterns in that input. In this practical, we look into **reinforcement learning**, which can loosely be defined as training an **agent** to maximise the total **reward** it obtains through many interactions with an **environment**.

At timestep $t$, the agent can make an **observation** of the environment $o_t$. For example, if the environment is a computer game, the observation could be the pixel values of the current screen.

The environment defines a set of **actions** that an agent can take.  The agent performs an action $a_t$ informed by the observations it has made, and will receive a **reward** $r_t$ from the environment after every action. The *reinforcement learning problem* is to find an agent whose actions maximize the total rewards obtained from the environment over many actions.

The following diagram illustrates the interaction between the agent and environment. We will explore each of the terms in more detail throughout this practical.

**Optional Recommended Reading**:
- [*OpenAI Spinning Up: Key Concepts in RL*](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html) is a great summary of the terminology used in reinforcement learning.

### The Environment

We will focus on the cartpole environment for this practical. This environment consists of a pole attached to a cart via a hinge, with the pole initially balanced close to upright. The agent needs to move the cart to the left or to the right in order to prevent the pole from falling over.

We will use the OpenAI Gym implementation of the cartpole environment. [OpenAI Gym](https://gym.openai.com/) is probably the most popular set of reinforcement learning environments (the available environments in Gym can be seen [here](https://gym.openai.com/envs)). Every Gym environment has the same interface, allowing code written for one environment to work for all of them. Popular reinforcement learning frameworks, such as [Ray](https://ray.readthedocs.io/en/latest/index.html), often use the Gym interface as [their default interface](https://ray.readthedocs.io/en/latest/rllib-env.html#openai-gym) for reinforcement learning environments.

Let us import Gym and open a cartpole environment:

```python
import numpy as np
import gym
env_cartpole = gym.make('CartPole-v1')
```

The first step to using a Gym environment is to initialize the environment to some initial configuration using the `reset` method. The `reset` method also returns the first observation of the environment:

```python
env_cartpole.reset()
```

These four numbers represent the position and velocity of the cart and pole, `(cart position, cart velocity, pole angle, velocity of the top of the pole)`. We will want our agent to use this observation when deciding to move left or right.

We also want to see an image of the system that we can interpret. Gym provides the `render` method to do this:

```python
import matplotlib.pyplot as plt
plt.imshow(env_cartpole.render(mode='rgb_array'));
```

Note that reinitializing this environment with `reset` will randomly change the starting angle of the pole. By running this multiple times, see if you can notice this:

```python
env_cartpole.reset()
plt.imshow(env_cartpole.render(mode='rgb_array'));
```

Now we want to take an action using the `step` method. But which actions are allowed, and how are they represented? We can find this out using the `action_space` property of Gym environments:

```python
env_cartpole.action_space
```

In Gym, `Discrete(n)` means actions are represented by the integers `0,1,...,n-1`. Action spaces always provide a way to take a random action sampled from the space:

```python
[env_cartpole.action_space.sample() for _ in range(10)]
```

Now let us take an action using the `step` method, which takes an action and returns a tuple `(observation, reward, done, info)`:

```python
action = 1
env_cartpole.step(action)
```

**Question**: Does the action `0` move the cart to the left or to the right? Use the `step` and `render` methods to figure this out.

What is `done`? Many environments are **episodic**, which means they have a natural end-point, after which more actions cannot be taken. For example in chess, the game ends after a checkmate. In the case of cart-pole, this end-point is when the pole falls down too far. Then no matter the actions taken, the pole will keep falling. Gym will set `done` as `True` in this case, and will print a warning if `step` is called again. The `reset` method needs to be called to reinitialize the environment when `done` is `True`.  We won't use `info` (see the [official Gym docs](https://gym.openai.com/docs/#observations) for more information).

In addition, many Gym environments have a maximum number of steps they can take, after which `done` is set to `True`. This can be seen using the `_max_episode_steps` property:

```python
env_cartpole._max_episode_steps
```

**Question**: What is an appropriate reward for a game of chess? Suppose the environment is one where your agent plays against an existing chess-playing computer. Remember that there are three outcomes in chess, a win, loss and draw.

#### Question

Write a function `actions_till_done(env, act)` that takes an environment `env` and action `act`, reinitializes `env` with `reset`, and returns the number of actions taken before `done` is `True`. If you always take the `0` action, how many steps can you take on average before `done` is `True`?

Reveal the cell below by double-clicking and running it, to check your answer when you're done or you get stuck!

```python
#@title Answer { display-mode: "form" }
def actions_till_done(env, action):
    env.reset()
    done = False
    count = 0
    while done == False:
        _, _, done, _, = env.step(action)
        count += 1
    return count

# Get an estimate for how many 0 actions you can take on average before the
# pole falls over
np.mean([actions_till_done(env_cartpole, 0) for _ in range(100)])
```

## Agents and Policies

A **policy** is a function used by an agent to decide what actions to take. This function takes as input an observation and returns either an action (a **deterministic policy**) or a probability distribution over possible actions (a **stochastic policy**).

**Note**: the terms **policy** and **agent** are often used interchangeably:
> Because the policy is essentially the agent’s brain, it’s not uncommon to substitute the word “policy” for “agent”, e.g. saying “The policy is trying to maximize reward.” ~ [OpenAI Spinning Up](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#policies)

We will consider an **agent** to be a function whose input is the observation, and it returns a dictionary which always contains the `"Action"` key. It might also contain other keys, such as the action probabilities, the value function, etc.

The simplest possible policy for a cartpole agent is one that always takes the `0` action, no matter what the observation is:

```python
def agent_left(observation):
    return {"Action":0}
```

```python
agent_left(None)
```

## Rewards and Returns

Recall that in Reinforcement Learning, the **reward** is a scalar signal that the agent observes upon acting on the environment and which it will try to maximize. In the cartpole example, a reward of **+1** is provided for every timestep that the pole remains upright (less than 15 degrees from the vertical).

Is our agent `agent_left` any good? To answer that, we need some measure of 'goodness', which is usually related to the total return obtained during an episode. This is called the **return**, $R$, and there are different ways of defining it. The most obvious definition for return is to simply add up all the rewards the agent received during an episode. If the episode took $T$ steps, and got a reward $r_t$ at each step, then this return is $R = \sum_{t=1}^{T} r_{t}$, where $r_t$ is the reward received at timestep $t$.

One issue is that some environments will have different initial states, agents may take actions stochastically and the rewards themselves could be stochastic. The solution is to average the returns over multiple starting points. Here is a function that does this:

```python
def average_episodic_return(env, agent, episodes=10, max_steps_per_episode=500):
    episode_rewards = []
    for episode in range(episodes):
        total_rewards = 0
        obs = env.reset()
        for t in range(max_steps_per_episode):
            out = agent(obs) # we don't care about the probabilities here
            assert ("Action" in out), "The key 'Action' was missing from the agents output."
            obs, rew, done, _ = env.step(out["Action"])
            total_rewards += rew
            # check if we are done, if so, exit loop
            if done:
                break

        episode_rewards.append(total_rewards)

    return {
        "AverageEpisodicReturn": np.mean(episode_rewards),
        "StandardDeviation":np.sqrt(np.var(episode_rewards))
    }
```

Now we can see how good our agent `agent_left` is:

```python
average_episodic_return(env_cartpole, agent_left)
```

## Animating Agents

This code cell defines a function `animate_agent(environment, agent)` that produces a human understandable animation of the `agent` controlling the `environment`. **Note:** It is not necessary to understand these functions to follow this practical.

```python
from collections import deque
import matplotlib.animation
import matplotlib.pyplot as plt
%matplotlib inline

def animate_agent(env, agent, max_steps = 400):
    obs = env.reset()
    frames = deque()
    frames.append(env.render(mode='rgb_array'))
    for _ in range(max_steps):
        out = agent(obs)
        assert ("Action" in out), "The key 'Action' was missing from the agents output."
        action = out["Action"]
        obs, _ , done, _ = env.step(action)
        frames.append(env.render(mode='rgb_array'))
        if done:
            break

    return animate_frames(frames)

def animate_frames(frames):
    new_height = 2.2
    original_height = frames[0].shape[0]
    original_width = frames[0].shape[1]
    new_width = (new_height / original_height) * original_width
    fig = plt.figure(figsize=(new_width, new_height), dpi = 120)

    ax = plt.Axes(fig, [0., 0., 1., 1.])
    ax.set_axis_off()
    fig.add_axes(ax)
    patch = ax.imshow(frames[0], aspect='auto', animated=True, interpolation='bilinear')
    animate = lambda i: patch.set_data(frames[i])

    ani = matplotlib.animation.FuncAnimation(fig, animate, frames=len(frames), interval = 50)

    plt.close()
    return HTML(ani.to_jshtml())
```

Now we can easily get an animation of the cartpole environment controlled by `policy_left`:

```python
def agent_left(observation):
    return {"Action":0}

animate_agent(env_cartpole, agent_left)
```

## Agent 1: Random Action Agent

Rather than having an agent whose policy is to always take the same action, we could also randomly choose between possible actions

**Question**: would you expect the agent taking random actions to get a higher or lower return than the `agent_left` agent for cartpole?

```python
def agent_random(observation):
    return {"Action":np.random.choice([0, 1])}
```

**Note**: that we can easily write a general random agent for any gym environment using `env.action_space.sample()`.

Computing the average return of `agent_random`:

```python
average_episodic_return(env_cartpole, agent_random)
```

And animating it:

```python
animate_agent(env_cartpole, agent_random)
```

### Random Agent Tasks

1. [**ALL**] Create a random agent for "MsPacman-v0" environment using `env.action_space.sample()` and animate the agent using `animate_agent` for 300 steps.

## Agent 2: Neural Net Policy Trained with Random Search

To improve on the random agent `agent_random`, we need a policy that takes the observation of the environment into account. In this section, we will show how to create an agent that solves the cartpole problem without any fancy reinforcement learning methods.

First, define an agent with a single hidden-layer multi-layer perceptron (MLP) policy implemented using NumPy. Its input is a vector of length 4 and its output is either `0` or `1`:

```python
class AgentMLPNumpy:
    def __init__(self, num_hidden):
        self.W1 = np.random.randn(num_hidden, 4)
        self.W2 = np.random.randn(2, num_hidden)

    def __call__(self, x):
        y = self.W1.dot(x)
        y = np.tanh(y)
        y = self.W2.dot(y)
        return {"Action":np.argmax(y)}
```

The reate an instance of `AgentMLPNumpy` (with random weight initializations), and compute its return. If the return is better than previous agents, then keep this agent. Otherwise, throw it away and try a new randomly initialized agent `AgentMLPNumpy`. Repeat this process `n` times.

### Random Search Tasks

1. [**ALL**] Write a function `find_good_mlp_agent(env, num_tries, mlp_size)` that tries `num_tries` instances of `AgentMLPNumpy`, each with different weights. It should return both the agent that got the best return, along with that return.
2. [**All**] Try out different values of `num_tries` and `mlp_size` until you find values that produces an agent getting a return of 500. **Hint**: `num_tries` shouldn't need to be larger than 300.
3. [**ALL**] Visualize your best agent controlling cartpole using `animate_agent`.

### Extra Reading on Random Search

Variations of random search have proven to be very succesful for training agents to solve the reinforcement learning problem. Here are some examples:
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://openai.com/blog/evolution-strategies/), OpenAI, 2017
- [Simple random search provides a competitive approach to reinforcement learning](https://arxiv.org/abs/1803.07055), H. Mania *et al*, 2018
- [Welcoming the Era of Deep Neuroevolution](https://eng.uber.com/deep-neuroevolution/), Uber AI Labs, 2017

Note that random search used to solve the *reinforcement learning problem* is not generally considered to be a *reinforcement learning method* (as implied by the titles of some of the readings above, eg. "Evolution Strategies as a Scalable Alternative to Reinforcement Learning"). The reason for this:

> Reinforcement learning, like many topics whose names end with “ing,” such as machine learning and mountaineering, is simultaneously a problem, a class of solution methods that work well on the problem, and the field that studies this problem and its solution methods. It is convenient to use a single name for all three things, but at the same time essential to keep the three conceptually separate. In particular, the distinction between problems and solution methods is very important in reinforcement learning; failing to make this distinction is the source of many confusions. ~ *Reinforcement Learning*, Sutton and Barto, **2nd Edition**

## Agent 3: Neural Net Agent Trained with Policy Gradients

Unlike the random policy, the MLP policy trained with random search *does* take environment observations into account when choosing to take an action. However, the training procedure is clearly not optimal: each policy in the search makes no use of observations and rewards seen by previous policies. For environments where it takes significant time and money to take actions and make observations (for example, a real robot picking strawberries), this is incredibly wasteful. In this section, we present a method that does take previous observations into account when learning.

**Warning**: This section requires more mathematics than previous sections, and hence more precise notation. It will thus be significantly more abstract.

### Policy Gradient Theory

As we saw earlier, the reinforcement learning problem is to find an agent/policy that maximizes the return (rewards). We will now use a neural net to represent a stochastic policy  $\pi_\mathbf{\theta}(a_t|o_t)$, where $\mathbf{\theta}$ is the set of learnable parameters in the neural net, and $\pi_\mathbf{\theta}(a_t|o_t)$ is a probability distribution over the possible actions conditioned on the observation.

A **trajectory**  $\tau$ is a sequence of observations of the environment and the corresponding actions an agent takes. For an episodic task that ends after $T$ steps:
$$\tau = (o_0, a_0, o_1, a_1, \ldots , o_T, a_T)$$
The return (or total rewards) is a function of the trajectory, $R(\tau)$.

We want to find the values of $\theta$ that maximize the **expected return** over trajectories:
$$J(\pi_\theta)= \mathop{\mathbb{E}} _{\tau \sim \pi_{\theta}}[R(\tau)]$$

The usual way of efficiently finding good values of $\theta$ is to differentiate the objective with respect to $\theta$ and then use gradient descent. Luckily, we can obtain an *estimate* of this gradient. The key result is the *Policy Gradient Theorem*, which says that you can evaluate the gradient using the gradients of the policy itself:
$$\nabla_\theta J(\pi_\theta) =
\mathop{\mathbb{E}} _{\tau \sim \pi_{\theta}}\left[ \sum_{t=0}^{T}\Phi_t\nabla_\theta\log \pi_\theta(a_t|o_t) \right]
$$
There are many possibilities for $\Phi_t$ (see page 2 [here](https://arxiv.org/pdf/1506.02438.pdf) for a list of these possibilities). We  will use $\Phi_t = \hat{R}_t \equiv \sum_{t'=t}^{T} r_{t'}$, sometimes known as the [reward-to-go](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you). Note that $\mathop{\mathbb{E}} _{\tau \sim \pi_{\theta}}$ is an expectation over trajectories $\tau$ which can be estimated by using a sample mean. Let us call this estimate of the gradient $\hat{g}$:

$$\hat{g} = \frac{1}{|D|}\sum_{\tau \in \mathcal{D}}
\sum_{t=0}^{T}\hat{R}_t\nabla_\theta\log \pi_\theta(a_t|o_t)
$$
where $\mathcal{D}$ is a set of trajectories of size $|\mathcal{D}|$. The more trajectories we sample from, the lower the variance of the estimator $\hat{g}$. But to simplify things for this practical, we will only be sampling from only one trajectory at a time to get our gradient estimate. Then:
$$\hat{g} =
\sum_{t=0}^{T}\hat{R}_t\nabla_\theta\log \pi_\theta(a_t|o_t)
$$
Translating this into a loss to be *maximized*:
$$\mathcal{L}=\sum_{t=0}^{T}\hat{R}_t\log \pi_\theta(a_t|o_t) $$

This looks very similar to the usual supervised learning cross-entropy loss, $\sum_{i}\log p(y_i|x_i) $, where $y_i$ is the label and $x_i$ is the feature. Indeed, learning with policy gradients is almost the same doing supervised learning with the following small changes:

- we use the action $a_t$ taken when we saw $o_t$ as the label ($y_i$)
- the loss of each example gets multiplied (or *weighted*) by $\hat{R}_t$. This increases the log probability for good actions and decrease it by bad actions.
- we are doing gradient *ascent* rather than the usual gradient *descent*
- the observations $o_t$ are correlated with each other, and hence are not independent and identically distributed (i.i.d) as the features $x_i$ are assumed to be.

In summary:

| Reinforcement Learning         	|  Supervised Learning 	|
|-------------------------------------	|-----------------------------------------------------		|
|  $\sum_{t}\log \pi_\theta(a_t|o_t) \hat{R}_t$     	| $\sum_{i}\log p(y_i|x_i) $    |
|  action $a_t$    | label $y_i$    |
|  observation $o_t$    | feature $x_i$    |
| $\pi_\theta(a_t|o_t)$   |   $p(y_i|x_i)$  |
|  gradient ascent     	|  gradient descent    |
| non-i.i.d observations $o_t$ | i.i.d features $x_i$

For those interested in the derivation, there are two excellent resources:
- [Part 3: Intro to Policy Optimization](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html) of OpenAI Spinning Up. Note that they often use *state* where we used *observation*.
- Chapter 13 of Sutton and Barto, *Reinforcement Learning: and Introduction*, 2nd Edition. A free copy can be found [here](http://incompleteideas.net/book/RLbook2018.pdf).

**Historical Note**: The name "policy gradient" comes from the fact that we're directly taking the gradient of the policy. The particular flavour of policy gradient which uses the loss function above, along with the Monte-carlo approximation of the objective is known as the **REINFORCE** algorithm ([Williams 1992](https://link.springer.com/article/10.1007/BF00992696)).

#### Optional extra reading: More on Policy Gradients

This section is for the mathematically inclined, and gives more insight into the Policy Gradient Theorem.

Outside of the reinforcement learning setting, the Policy Gradient Theorem is known as a [score function estimator](http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/):
$$
\frac{\partial }{\partial \theta}\mathop{\mathbb{E}} _{x \sim p_{\theta}}[f(x)]=\mathop{\mathbb{E}} _{x \sim p_{\theta}}[f(x)\frac{\partial }{\partial \theta}\log p(x;\theta)]
$$

The power of this theorem comes from the ability to estimate the RHS, even if $f(x)$ is a *black-box function* (i.e. one for which we don't have the analytical form for). We can do this by replacing the mean by the sample mean:
$$\mathop{\mathbb{E}} _{x \sim p_{\theta}}[f(x)\frac{\partial }{\partial \theta}\log p(x;\theta)]
\approx \frac{1}{N}\sum_{n=1}^{N}  f(\hat{x}^{(n)})\frac{\partial }{\partial \theta}\log p(\hat{x}^{(n)};\theta)$$

where $\hat{x}^{(n)}$ is sampled from $p(x;\theta)$. In the reinforcement learning setting, $x$ is the action sampled from the parameterized policy $p(x;\theta)$, and $f(x)$ is the reward function.

The derivation of the score function estimator:
$$
\begin{align}
\frac{\partial }{\partial \theta}\mathop{\mathbb{E}} _{x \sim p_{\theta}}[f(x)]
&= \frac{\partial }{\partial \theta} \int p(x; \theta)f(x) \, dx \\
&= \int  \frac{\partial }{\partial \theta} p(x; \theta)f(x) \, dx \\
&= \int  p(x; \theta) \frac{\partial }{\partial \theta} \log p(x; \theta)f(x) \, dx \ \ \ \text{ (Log-Derivative trick)}\\
&= \mathop{\mathbb{E}} _{x \sim p_{\theta}}[f(x)\frac{\partial }{\partial \theta}\log p(x;\theta)]
\end{align}
$$

The Log-Derivative trick used in the third line is the property, for any function $f(x)$:
$$\frac{d f(x)}{dx}=\frac{f(x)}{f(x)}\frac{d f(x)}{dx}=f(x)\frac{d\log(f(x))}{dx}
$$
The right-hand side was obtained using the chain rule and the fact that $d \log(x)/dx=1/x$.

The score function estimator is valid under **very** general conditions: $p(x;\theta)$ needs to be a continuous function of $\theta$, but does not need to be a continuous function of $x$. Furthermore, $f(x)$ can be discontinuous and $x$ can be either a discrete or continuous random variable (a reference for these conditions is [here](https://arxiv.org/abs/1506.05254)). This should reassure us when using this theorem for reinforcement learning, as it means we don't need to worry about showing that our environments produce a smooth or even continuous return function (which might not be true, or might be very hard to show for most environments)!

**Example: A Toy Model**

To get some intuition for how the score function estimator works, lets consider a toy model: Let $f(x)=-(x-k)^2$ (where $k$ is some real number) and let $p(x;\mu,\sigma)$ be normally distributed $\mathcal{N}(\mu, \sigma)$ with mean $\mu$ and variance $\sigma$:
$$
p(x;\mu,\sigma) = \frac{e^{-\frac{1}{2\sigma} (x-\mu)^2}}{\sqrt{2 \pi } \sigma}
$$

In reinforcement learning, the task is to find the values of $\mu$ and $\sigma$ that maximize $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$. Here is a summary as to how this toy problem fits into the reinforcement learning framework:

|     Reinforcement Learning Concept    	|  Toy Example 	|
|-------------------------------------	|-----------------------------------------------------		|
|  Action     	| choice of $x$    |
|  Policy    | $p(x;\mu,\sigma)$    |
|  Observation $o_t$     | None    |
| Reward $r_t$   |   $-(x-k)^2$  |

Unlike the general reinforcement learning case where $f(x)$ is a black-box,  we can analytically evaluate $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$ with our example:
$$
\begin{align}
\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]
&= \int^{\infty}_{-\infty} f(x) p(x;\mu,\sigma) dx \\
&=\int^{\infty}_{-\infty} \left[-(x-k)^2\right] \left[\frac{e^{-\frac{1}{2\sigma} (x-\mu)^2}}{\sqrt{2 \pi } \sigma}\right]dx \\
&=-(k-\mu)^2 - \sigma^2
\end{align}
$$

The maximimum of $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$ is when the distribution $p(x;\mu,\sigma)$ has mean $\mu=k$ (which corresponds to the position of the maximum of $f(x)=-(x-k)^2$) and variance $\sigma=0$ (so it is infinitely 'spiked', i.e. a [Dirac Delta function](https://en.wikipedia.org/wiki/Dirac_delta_function)). This makes sense: to maximize $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$, we want all of the probability mass of $p(x;\mu,\sigma)$ to be concentrated at the maximum of $f(x)=-(x-k)^2$, i.e. $x=k$.

We can now also easily verify that using the score function estimator is equivalent to differentiating $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$ directly: as we have computed $
\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$, we immediately have that
$$\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)] = \frac{\partial }{\partial \mu}\left[-((\mu-k)^2 + \sigma^2)\right]=2(k-\mu)
$$
But using the score function estimator instead:
$$\begin{align}
\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]
&= \mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)\frac{\partial }{\partial \mu}\log p(x;\mu,\sigma)] \\
&= \mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[-(x-k)^2 \frac{\partial }{\partial \mu}\log \left(\frac{e^{-\frac{1}{2\sigma} (x-\mu)^2}}{\sqrt{2 \pi } \sigma}\right)] \\
&= \mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[\frac{(x-k)^2(\mu-x)}{\sigma^2}] \\
&=2(k-\mu)
\end{align}
$$

Note: If you have trouble following that last step see [moments of the Gaussian](https://en.wikipedia.org/wiki/Normal_distribution#Moments)

It is left as an exercise to verify that the same holds for $\frac{\partial }{\partial \sigma}\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)] $.

**Sampling using the Toy Model**

So far, the score function estimator has not given us anything extra: we can
directly compute the derivatives of $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$, and we don't even need derivatives to find its minimum. But lets pretend that $f(x)$ is a black box: we can only evaluate $f(x)$ at a finite number of values of $x$. We can no longer compute $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$ directly. How do we find the values of $\mu$ and $\sigma$ that maximize $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$?

**Further simplification**: set $\sigma=1$ and $k=0$, so $f(x)=-x^2$ and the task is to find the optimal value of $\mu$.

**Approach 1: Direct Search.** We can replace the mean $\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}$ with the sample mean:

$$
\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)]\approx \frac{1}{N} \sum_{n=1}^{N}f(\hat{x}^{(n)})
$$
where $\hat{x}^{(n)} \sim p(x;\mu,1)$ is the $n^{th}$ sample from the distribution $p(x;\mu,1)$. The sample mean is equivalent to the expectation in the limit $N\to\infty$. Then we simply try out lots of different values of $\mu$ and $\sigma$ and take the ones that maximize the sample mean. The random search method explored in the previous section is an example of this approach.

**Approach 2: Gradient Ascent**. If we can get an estimate of the derivative of $\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)]$ w.r.t $\mu$, we can modify our initial values of $\mu$ by gradient ascent rather than just trying out random values. Using the score function estimator and then taking the sample mean:

$$
\begin{align}
\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]
&= \mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)\frac{\partial }{\partial \mu}\log p(x;\mu,1)] \\
&= \mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x) (x-\mu)] \\
&\approx \frac{1}{N} \sum_{n=1}^{N}f(\hat{x}^{(n)}) (\hat{x}^{(n)}-\mu) \\
&\equiv \eta_N
\end{align}$$
Our estimator $\eta_N \equiv \sum_{n=1}^{N}f(\hat{x}^{(n)}) (\hat{x}^{(n)}-\mu)$ allows us to estimate the gradient $\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu,\sigma}}[f(x)]$ using a finite number of samples of $p(x;\mu,1)$. In this case, we were able to explicitly compute $\frac{\partial }{\partial \mu}\log p(x;\mu,1)$ as we have an analytical expression for $p(x;\mu,1)$, which is usually not the case for reinforcement learning (where $p(x;\mu,1)$ is often a neural net).
Let us implement this estimator $\eta_N$:

```python
def f_estimator(x, mean):
    return (-x**2) * (x - mean)

def sf_estimator(mean, sample_size, f):
    x = np.random.normal(mean, 1, sample_size)
    return np.mean(f(x, mean))
```

If our starting value is $\mu=3$, then the exact derivate is $2\times (-3)=-6$, and our estimated derivative is:

```python
sample_size=200
mean=3
sf_estimator(mean, sample_size, f_estimator)
```

**Baselines and Variance Reduction**

We can also get an idea of the **variance** of our derivative estimates, which tells us how noisy these estimates are:

```python
def sf_estimator_variance(mean, mean_sample_size, grad_f, variance_samples = 200):
    grads = [grad_f(mean, mean_sample_size, f_estimator) for _ in range(variance_samples)];
    return np.var(grads)
```

With a sample size of 200, the variance of our derivative estimator is:

```python
var1 = sf_estimator_variance(mean, sample_size, sf_estimator)
var1
```

One obvious way of reducing the variance is to increase the sample size:

```python
sample_size2 = 500
sf_estimator_variance(mean, sample_size2, sf_estimator)
```

In reinforcement learning, taking extra samples is often expensive. It turns out that we can reduce the variance *without* taking any more samples! This idea is very important in reinforcement learning, where it is known as the [baseline method](http://jmlr.csail.mit.edu/papers/volume5/greensmith04a/greensmith04a.pdf). The general procedure is also known as the [control variate method](https://arxiv.org/abs/1906.10652).

To see how this works, let us derive an expression for the variance of our estimator:

$$
\begin{align}
\text{Var}[\eta_N] &= \text{Var}\left[\frac{1}{N}\sum_{n=1}^{N}f(x) (x-\mu) \right] \ \ \  \ (x \sim \mathcal{N}(\mu, 1))  \\
&=\frac{1}{N^2}\text{Var}\left[\sum_{n=1}^{N}f(x) (x-\mu) \right] \ \ \ \ (\text{Var}[aX]=a^2\text{Var}[X]) \\
&= \frac{1}{N^2}\sum_{n=1}^{N}\text{Var}\left[f(x) (x-\mu) \right] \ \ \ \ (\text{Var}[X+Y]=\text{Var}[X]+\text{Var}[Y] \ \text{when} \ X,Y \ \text{independent}) \\
&= \frac{1}{N^2}\cdot N\cdot\text{Var}\left[f(x) (x-\mu) \right] \\
&= \frac{1}{N}\text{Var}\left[f(x) (x-\mu) \right]
\end{align}
$$
As we already saw, the obvious way of reducing the variance $\text{Var}[\eta_N]$ is to increase $N$, but this comes at the cost of needing to take more samples. We have learnt though that the variance decreases as $\mathcal{O}(1/N)$ with sample size.

Another way is with the following trick: replace the $f(x)$ with a new $\tilde{f}(x)$ that leaves the gradient $\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu}}[(f(x))]$ unchanged, but reduces the variance of our gradient estimator $\eta_N$, i.e reduces $\text{Var}[\eta_N]$. A simple choice is $\tilde{f}(x)=f(x)-b(x)$ for any function $b(x)$ that does not depend on $\mu$. It is easy to check that this leaves $\eta_N$ unchanged:
$$\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu}}[(\tilde{f}(x))]
=\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu}}[(f(x)-b(x))]
=\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)]+\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu}}[b(x)]
=\frac{\partial }{\partial \mu}\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)]$$
Now [note that](https://en.wikipedia.org/wiki/Variance#Sum_of_variables)
$$
\text{Var}[f-b]= \text{Var}[f]-2\text{Cov}[f,b]+\text{Var}[b]
$$
So if $f$ and $b$ are strongly correlated, then the covariance $2\text{Cov}[f,b]$ can be greater than the extra variance term $\text{Var}[b]$, reducing the total variance $\text{Var}[f-b]$.

A common choice for $b$ is $b=\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)]$. This choice of $b$ is clearly correlated with $f(x)$, as increases in $f(x)$ will lead to increases in the mean $\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)]$ (and decreases will lead to decreases in the mean).

Let us check for our toy example that this does actually reduce the variance. First, note that $\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x)] = -(1+\mu^2)$ and $\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x) (x-\mu) ] = -2\mu$ (leaving out the details of this calculation). Then (skipping the details of the computation: this is what computer algebra systems are there for!):
$$
\begin{align}
\text{Var}[\eta_N]
&=  \frac{1}{N}\text{Var}\left[f(x) (x-\mu)\right] \\
&=  \frac{1}{N}\int^{\infty}_{-\infty}\left(f(x)(x-\mu) -\mathop{\mathbb{E}} _{x \sim p_{\mu}}[f(x) (x-\mu) ] \right)^2 p(x;\mu)dx \\
&= \frac{1}{N}\left( \mu^4 +14\mu^2+15\right) \\
\end{align}
$$
And similarly for $\tilde{f}(x)$:
$$
\begin{align}
\text{Var}[\tilde{\eta}_N] &= \frac{1}{N}\text{Var}\left[\tilde{f}(x) (x-\mu)\right] \\
&=\frac{1}{N}\left( 8\mu^2+10\right)
\end{align}
$$
The variance is indeed reduced for all values of $\mu$, as the difference in variances $\text{Var}[\eta_N] -\text{Var}[\tilde{\eta}_N]$ is a positive quantity:
$$
\text{Var}[\eta_N] -\text{Var}[\tilde{\eta}_N]=\frac{1}{N}(\mu^4+6 \mu^2+5)
$$

To get more insight into how the addition of the baseline changes the distribution of $f(x) (x-\mu)$ (where $x$ is normally distributed $\mathcal{N}(\mu, 1)$), we could analytically compute what this distribution is and plot it, and hopefully see how the baseline changes its shape. This is tricky as $f(x) (x-\mu)$ is a cubic polynomial, which is non-bijective in a non-trivial way, complicating the use of the standard probability density [change-of-variables formula](https://en.wikipedia.org/wiki/Probability_density_function#Function_of_random_variables_and_change_of_variables_in_the_probability_density_function). Rather than do this analytically, it is much easier to sample values of $x$ and plot the histogram of values of $f(x) (x-\mu)$. The Seaborn plotting package makes this very easy:

```python
def f_estimator_baseline(x, mean):
  return (-x**2 - (-1 - mean**2)) * (x-mean)

mean2 = 7
sample_size = 10000
x_rand=np.random.normal(mean2, 1, sample_size)
rands_f = f_estimator(x_rand, mean2)
rands_f_baseline = f_estimator_baseline(x_rand, mean2)

import seaborn as sns
sns.set_style('darkgrid')
sns.kdeplot(rands_f, shade=True, label="Without Baseline", legend=True)
sns.kdeplot(rands_f_baseline, shade=True, label="With Baseline", legend=True)
```

This shows how the addition of the baseline changes the distribution to be a lot *sharper*, which means that it has less variance. The sampling we did above allows us to check our analytic calculations as well: with $\mu=7$, $\text{Var}\left[f(x) (x-\mu)\right] = 3102$ and $\text{Var}\left[\tilde{f}(x) (x-\mu)\right] = 402$. This is similar to what we see with our sampled data:

```python
print("Variance without Baseline: ", np.var(rands_f))
print("Variance with Baseline: ", np.var(rands_f_baseline))
```

The sample means should be similar (but only the same in the limit $N\to\infty$):

```python
print("Mean without Baseline: ", np.mean(rands_f))
print("Mean with Baseline: ", np.mean(rands_f_baseline))
```

In practice, we don't know the exact mean of $f(x)$ to use as a baseline, but we can just take the sample mean.

**Question**: verify that using `f_estimator_baseline` instead of `f_estimator` in `sf_estimator_variance` reduces the variance

**Extra Reading**
- A great review of score function estimators:
    * [Monte Carlo Gradient Estimation in Machine Learning](https://arxiv.org/abs/1906.10652), Mohamed *et al*, 2019

### Episode Trajectory Collector

Let us implement a function `get_episode_trajectory(env, agent)` that controls the environment `env` with `agent` and returns lists of rewards, observations and actions taken at each timestep:

```python
def get_episode_trajectory(env, agent, max_steps=1000):
    observation_list = []
    reward_list = []
    action_list = []
    value_list = []

    done = False
    obs = env.reset()
    for _ in range(max_steps):
        observation_list.append(obs)
        out = agent(obs)
        assert ("Action" in out), "The key 'Action' was missing from the agents output."
        action = out["Action"]
        obs, rew, done, _, = env.step(action)
        reward_list.append(rew)
        action_list.append(action)
        if "Value" in out:
            value_list.append(out["Value"])

        if done:
            break

    ret = {
        "Observations": observation_list,
        "Actions": action_list,
        "Rewards": np.array(reward_list, dtype=np.float32)
    }
    if len(value_list) > 0:
        ret["Values"] = value_list

    return ret
```

Let us try out `get_episode_trajectory` on cartpole using the random agent we implemented earlier:

```python
traj = get_episode_trajectory(env_cartpole, agent_random, max_steps=3)
pprint(traj)
```

### Reward-to-go $\hat{R}_t$

We need to compute the reward-to-go $\hat{R}_t= \sum_{t'=t}^{T} r_{t'}$. Here is a simple NumPy implementation:

```python
def reward_to_go(rewards):
    return np.flip(np.cumsum(np.flip(rewards)))
```

Checking that this works as expected:

```python
reward_to_go([1, 1.2, 1])
```

#### Optional extra reading: estimator issues

The estimator $\Phi_t \equiv \hat{R}_t= \sum_{t'=t}^{T} r_{t'}$ we used has a number of problems. For example, if we don't get to the end of an episode (as it takes too long and we stop sampling), or our task is not episodic, we will have incorrect estimates of the advantages of our states. One popular alternative that works around this difficulty is the GAE-Lamba estimator:
 - [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438), Schulman *et al*, 2015
 - A good blog post on this: https://danieltakeshi.github.io/2017/04/02/notes-on-the-generalized-advantage-estimation-paper/

### TensorFlow Policy Net

Let us use a neural net policy with parameters $\theta$, so we have $\pi_\theta(a|s) = NN(s; \theta)$, where $NN(s; \theta)$ is some potentially complex function represented by a neural network with parameters $\theta$:

```python
import tensorflow as tf
from tensorflow.keras.layers import Dense, Flatten, Conv2D
from tensorflow.keras import Model
```

```python
class AgentMLPTF(Model):
  def __init__(self):
    super(AgentMLPTF, self).__init__()
    self.d1 = Dense(15, activation='tanh')
    self.d2 = Dense(2)

  def call(self, x):
    # 1. Define Policy
    batch = True
    if x.ndim == 1:
        batch = False
        x = np.expand_dims(x, axis=0)
    x = self.d1(x)
    action_logits = self.d2(x)

    # 2. Sample policy to get action
    action = tf.random.categorical(action_logits, 1)
    action = action.numpy().flatten()
    if not batch:
        action = np.asscalar(action)

    return {"Action":action, "LogProbability":action_logits}
```

```python
agent_mlp_tf = AgentMLPTF()
```

```python
obs = np.array([0.1,0.2,0.3,0.4])
agent_mlp_tf(obs)
```

We can also verify that this works on a batch of observations:

```python
obs_batch = np.array([[0.1,0.2,0.3,0.4], [0.5,0.3,0.2,0.1]])
agent_mlp_tf(obs_batch)
```

### Loss Function

**Exercise [ALL]**: implement the loss function $\mathcal{L}=\sum_{t=0}^{T}\hat{R}_t\log \pi_\theta(a_t|s_t) $ using TensorFlow operations. Call this function `loss_pg(actions, log_probs, returns)`.

```python
#@title Answer { display-mode: "form" }

def loss_pg(actions, log_probs, returns):
    action_masks = tf.one_hot(actions, 2, dtype=np.float64)
    log_probs = tf.reduce_sum(action_masks * tf.nn.log_softmax(log_probs), axis=1)
    return -tf.reduce_sum(returns * log_probs)

# Verify that this works on some example data:

actions = [1,0,0]
logits = np.array([[0.2,0.8],[0.2,0.8],[0.6,0.4]])
weights = [2.3, 4.3, 2.1]
loss_pg(actions, logits, weights)

# Note: this is equivalent to:
def loss_pg2(actions, log_probs, returns):
    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)
    return tf.reduce_mean(returns * loss(actions, log_probs))
```

### Train the Policy Net

Now we can put all the pieces together in the training loop. It simply collects a single trajectory and uses it to do a single gradient update of the agent:

```python
def train_policy_grad(env, agent, num_epochs=300):
    optimizer = tf.keras.optimizers.Adam(learning_rate=1e-2)
    log_reward = 0
    log_reward_list = []
    logging_period = 20

    for epoch in range(num_epochs):
        # get the training data
        traj = get_episode_trajectory(env, agent)
        obs = np.stack(traj["Observations"])
        rew = traj["Rewards"]
        actions = traj["Actions"]

        # compute 'reward-to-go'
        rew_2_go = reward_to_go(rew)

        # compute gradients + update weights
        with tf.GradientTape() as tape:
            logits = agent(obs)["LogProbability"]
            loss = loss_pg(actions, logits, rew_2_go)

        gradients = tape.gradient(loss, agent.trainable_variables)
        optimizer.apply_gradients(zip(gradients, agent.trainable_variables))

        # log the reward
        log_reward += np.sum(rew)
        if (epoch % logging_period) == 0:
            template = 'Training Epoch {}, Averaged Return: {}'
            print(template.format(epoch, log_reward / logging_period))
            log_reward_list.append(log_reward / logging_period)
            log_reward = 0

    return (range(0, num_epochs, logging_period), log_reward_list)
```

Note that `train_policy_grad(env, agent)` takes a TensorFlow agent as input and mutates it during training. Create and train a `AgentMLPTF` agent:

```python
agent_mlp_tf = AgentMLPTF()
(episodes, rewards) = train_policy_grad(env_cartpole, agent_mlp_tf)
```

We can plot the average rewards obtained versus the number of episodes trained for:

```python
plt.plot(episodes, rewards, 'bo')
plt.xlabel('Episode Number')
plt.ylabel('Average Episode Reward')
plt.show()
```

Let us animate this agent controlling the cartpole environment:

```python
animate_agent(env_cartpole, agent_mlp_tf)
```

### Policy Gradient Tasks

1. [**ALL**] Try using TensorFlows default initial learning rate of the ADAM optimizer in `train_policy_grad`. What happens? Try come up with an explanation for what is going on.

**Variance Reduction**: We are trying to estimate the policy gradient using $\hat{g} = \frac{1}{|D|}\sum_{\tau \in \mathcal{D}}
\sum_{t=0}^{T}\hat{R}_t\nabla_\theta\log \pi_\theta(a_t|o_t)
$. This gradient is only exact if we have an infinite number of samples. Otherwise it is *noisy*: we will get a different estimate of $\hat{g}$ for every sample we take.

2. [**ALL**] The simplest way of reducing the variance of $\hat{g}$ is to use a *baseline*: rather than using $\hat{R}_t$, use $\hat{R}_t-b$ where $b$ is the mean of $\hat{R}$, i.e. $b=\frac{1}{T}\sum_{t=1}^{T}\hat{R}$. Modify `train_policy_grad` to use this. Does it speed up training? An in-depth discussion of why this works is in the section *Optional extra reading: More on Policy Gradients*.

3. [**Advanced**] The policy used to solve the Cartpole environment only required the most recent observation to inform its actions. Observations taken before the most recent one give no extra information that the policy could use to improve its decision making. This is called the [Markov Property](http://www.cs.cmu.edu/~10601b/slides/MDP_RL.pdf). Most real-world environments are not like this. Consider the problem of using a reinforcement learning agent to control a self-driving car, where an observation is a frame from its onboard video cameras. List various ways in which the Markov Property is violated in this environment (this is also known as a *partially-observed environment*). How might you modify the architecture of your policy net to operate in a non-Markovian environment like this?

**IMPORTANT: Please fill out the exit ticket form before you leave the practical: https://forms.gle/FLQ78eAEQprsWXp38**

### Optional extra reading: Policy Gradients

- The simple REINFORCE algorithm shown here is very unstable. There are a number of improvements to this method, the most popular of which is probably PPO:
    * [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347), Schulman *et al*, 2017
    * OpenAI SpinningUp: https://spinningup.openai.com/en/latest/algorithms/ppo.html
    * OpenAI Blog: https://openai.com/blog/openai-baselines-ppo/
- PPO borrows many ideas from TRPO. [Depth First Learning](https://www.depthfirstlearning.com/2018/TRPO) has a course with a lot of good resources on TRPO.
- The idea of being able to differentiate through stochastic nodes in computation graphs is a very powerful idea, and it is used in a number of other areas in deep learning, such as *variational autoencoders*.
    * For the general setting, see [Gradient Estimation Using Stochastic Computation Graphs](https://arxiv.org/abs/1506.05254), Schulman *et al*, 2015.
- For a skeptical take on policy gradients, see Ben Recht's post [The Policy of Truth](https://www.argmin.net/2018/02/20/reinforce/).
- Karpathy has [good post](http://karpathy.github.io/2016/05/31/rl/) on reinforcement learning using policy gradients, including an implementation of policy gradients that solves Pong using only NumPy in [131 lines of Python](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5)!


# Exploring Jupyter Notebook - Magic Functions
Author: Rahul Yedida

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
plt.plot(range(100))
```

```python
%time x = range(10000)
```

    Wall time: 0 ns

```python
%%timeit x = range(10000)
max(x)
```

    1000 loops, best of 3: 857 µs per loop

```python
%%HTML
<i> Image in jupyter notebook</i>

```

 Image in jupyter notebook

```latex
%%latex
\begin{align}
Gradient: \nabla J = -2H^T (Y - HW)
\end{align}
```

\begin{align}
Gradient: \nabla J = -2H^T (Y - HW)
\end{align}

```python
!pip install ipython-sql
```

    Collecting ipython-sql
      Downloading ipython_sql-0.3.8-py3-none-any.whl
    Requirement already satisfied: sqlalchemy>=0.6.7 in c:\users\yrahu\anaconda3-3.4\lib\site-packages (from ipython-sql)
    Collecting prettytable (from ipython-sql)
      Downloading prettytable-0.7.2.zip
    Collecting sqlparse (from ipython-sql)
      Downloading sqlparse-0.2.4-py2.py3-none-any.whl
    Collecting ipython-genutils>=0.1.0 (from ipython-sql)
      Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl
    Requirement already satisfied: ipython>=1.0 in c:\users\yrahu\anaconda3-3.4\lib\site-packages (from ipython-sql)
    Requirement already satisfied: six in c:\users\yrahu\anaconda3-3.4\lib\site-packages (from ipython-sql)
    Building wheels for collected packages: prettytable
      Running setup.py bdist_wheel for prettytable: started
      Running setup.py bdist_wheel for prettytable: finished with status 'done'
      Stored in directory: C:\Users\yrahu\AppData\Local\pip\Cache\wheels\b6\90\7b\1c22b89217d0eba6d5f406e562365ebee804f0d4595b2bdbcd
    Successfully built prettytable
    Installing collected packages: prettytable, sqlparse, ipython-genutils, ipython-sql
    Successfully installed ipython-genutils-0.2.0 ipython-sql-0.3.8 prettytable-0.7.2 sqlparse-0.2.4

```python
%load_ext sql
```

```python
!which -a python
```

    'which' is not recognized as an internal or external command,
    operable program or batch file.

```python
sys.executable
```

```python
import sys
```

```python
sys.executable
```

    'C:\\Users\\yrahu\\Anaconda3\\python.exe'

```python
!C:\Users\yrahu\Anaconda3\python -m pip install ipython-sql
```

    Collecting ipython-sql
      Using cached ipython_sql-0.3.8-py3-none-any.whl
    Requirement already satisfied (use --upgrade to upgrade): ipython-genutils>=0.1.0 in c:\users\yrahu\anaconda3\lib\site-packages (from ipython-sql)
    Collecting prettytable (from ipython-sql)
      Using cached prettytable-0.7.2.zip
    Requirement already satisfied (use --upgrade to upgrade): sqlalchemy>=0.6.7 in c:\users\yrahu\anaconda3\lib\site-packages (from ipython-sql)
    Requirement already satisfied (use --upgrade to upgrade): six in c:\users\yrahu\anaconda3\lib\site-packages (from ipython-sql)
    Collecting sqlparse (from ipython-sql)
      Using cached sqlparse-0.2.4-py2.py3-none-any.whl
    Requirement already satisfied (use --upgrade to upgrade): ipython>=1.0 in c:\users\yrahu\anaconda3\lib\site-packages (from ipython-sql)
    Building wheels for collected packages: prettytable
      Running setup.py bdist_wheel for prettytable: started
      Running setup.py bdist_wheel for prettytable: finished with status 'done'
      Stored in directory: C:\Users\yrahu\AppData\Local\pip\Cache\wheels\b6\90\7b\1c22b89217d0eba6d5f406e562365ebee804f0d4595b2bdbcd
    Successfully built prettytable
    Installing collected packages: prettytable, sqlparse, ipython-sql
    Successfully installed ipython-sql-0.3.8 prettytable-0.7.2 sqlparse-0.2.4

    You are using pip version 8.1.0, however version 9.0.1 is available.
    You should consider upgrading via the 'python -m pip install --upgrade pip' command.

```python
%sql sqlite://
```

    'Connected: None@None'

```sql
%%sql
CREATE TABLE classroom(name, age, totalmarks);
INSERT INTO classroom VALUES("Raj", 20, 89);
INSERT INTO classroom VALUES("Poonam", 24, 84);
```

    Done.
    1 rows affected.
    1 rows affected.

    []

```python
%sql SELECT * FROM classroom;
```

    Done.

        name
        age
        totalmarks

        Raj
        20
        89

        Poonam
        24
        84

```python

```

notebooks/Exploring Jupyter Notebooks - Magic Functions.ipynb