## CCNSS 2018 Module 3: Reinforcement learning and planning
# Tutorial 1: Value estimation during classical conditioning using TD-learning

# **Introduction **

In this tutorial, we will learn how to estimate state-value functions in a classical conditioning paradigm using Temporal Difference (TD) learning and examine TD-errors at the presentation of the conditioned and unconditioned stimulus (CS and US) under different CS-US contingencies.

__Environment:__

- The classical conditioning environment is composed of a sequence of states that the agent deterministically transitions through.
- The agent experiences the environment in episodes or trials.
- Episodes terminate by transitioning to the inter-trial-interval (ITI) state and they are initiated from the ITI state as well. We clamp the value of the terminal/ITI states to zero.
- Within each episode, the agent is presented a CS and US (reward).
- For each exercise, we use a different CS-US contingency.
- The goal of the agent is to learn to predict expected rewards from each state in the trial.

__Definitions:__

1. Returns:
\begin{align}
G_{t} = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + ... = \sum \limits_{k = 1}^{\infty} \gamma^{k-1} r_{t+k}
\end{align}

2. Value:
\begin{align}
V(s_{t}) = \mathbb{E} [ G_{t} | s_{t}] = \mathbb{E} [r_{t+1} + \gamma V_{t+1} | s_{t}]
\end{align}

3. TD-error:
\begin{align}
\delta_{t} = r_{t+1} + \gamma V(s_{t+1}) - V(s_{t})
\end{align}

4. Value updates:
\begin{align}
V(s_{t}) \leftarrow V(s_{t}) + \alpha \delta_{t}
\end{align}

**Run the following code for your implementation:**

```python
% matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from pylab import *
```

```python
class classical_conditioning():

    def __init__(self, n_steps):

        # Task variables
        self.n_steps = n_steps
        self.n_actions = 0

        # Reward variables
        self.reward_state = [0,0]
        self.reward_magnitude = reward_magnitude
        self.reward_probability = reward_probability
        self.reward_time = reward_time

        # Time step at which the conditioned stimulus is presented
        self.cs_time = int(n_steps/4) - 1

        # Create a state dictionary
        self.create_state_dictionary()

    def define_reward(self, reward_magnitude, reward_time):

        """
        Determine reward state and magnitude of reward
        """
        if reward_time >= self.n_steps - self.cs_time:
            self.reward_magnitude = 0

        else:
            self.reward_magnitude = reward_magnitude
            self.reward_state = [1, reward_time]

    def get_outcome(self, current_state, action = 0):

        """
        Determine next state and reward
        """
        # Update state
        if current_state < self.n_steps - 1:
            next_state = current_state + 1
        else:
            next_state = 0

        # Check for reward
        if self.reward_state == self.state_dict[current_state]:
            reward = self.reward_magnitude
        else:
            reward = 0

        return next_state, reward

    def create_state_dictionary(self):

        """
        This dictionary maps number of time steps/ state identities
        in each episode to some useful state attributes:

        state      - 0 1 2 3 4 5 (cs) 6 7 8 9 10 11 12 ...
        is_delay   - 0 0 0 0 0 0 (cs) 1 1 1 1  1  1  1 ...
        t_in_delay - 0 0 0 0 0 0 (cs) 1 2 3 4  5  6  7 ...
        """
        d = 0

        self.state_dict = {}
        for s in range(self.n_steps):
            if s <= self.cs_time:
                self.state_dict[s] = [0,0]
            else:
                d += 1 # Time in delay
                self.state_dict[s] = [1,d]

```

### Unless specified otherwise, use the following parameter values:

```python
reward_time = 10
reward_magnitude = 10
reward_probability = 1

n_trials = 20000
n_steps = 40

gamma = 0.98  # temporal discount factor
alpha = 0.001 # learning rate
```

---

# **Exercise 1:**

Implement TD-learning to estimate the state-value function in the classical-conditioning world with guaranteed rewards, with a fixed magnitude, at a fixed delays after the CS. Save the history of estimated value function and TD-errors over learning.

1. Plot TD-errors on an example trial before learning.
2. Plot the estimated value function for each state after learning has converged.
3. Plot the value function and TD-errors over learning.

```python

```

# **Exercise 2:**

Implement TD-learning in the context of rewards of varying magnitudes. Use a set of discrete reward magnitudes. On each trial, randomly select a reward from the specified set.
1. Plot TD-errors on a set of trials before learning, as many as the number of reward magnitudes. On each trial present a different reward magnitude. Overlay the TD-errors on these set of trials on the same plot.
2. Plot the estimated state-value function for each state after learning has converged.
3. Plot TD-errors on a set of trials, on for each reward magnitude (as in (1)), after learning has converged.

```python
reward_magnitudes = [10, 6, 4, 1]
```

```python

```

# **Exercise 3: **

Implement TD-learning in the context of probabilistic rewards. For a fixed probability of reward delivery ($P(r) < 1$), on each trial randomly determine whether or not to present a reward on that trial. Keep reward magnitude constant.
1. Plot the estimated value function after learning.
2. Plot TD-errors on a trial with reward delivered and one with reward omitted, after learning. Overlay the two on the same plot.
3. Repeat (1) and (2) for different probabilities of reward delivery. How does the value function and TD-errors change for different probabilities of reward delivery?

```python
reward_probabilities = [0.1, 0.4, 0.6, 0.9]
```

```python

```

# **Exercise 4:**

Implement TD-learning in the context of rewards presented at variable delays since CS. Use a set of discrete times since CS. On each trial, randomly select the delay between CS ans US.

1. Plot TD-errors on a set of trials before learning, as many as the number of CS-US delays. On each trial present a different delay between the CS and reward (US). Overlay the TD-errors on these set of trials on the same plot.
2. Plot the estimated state value function after learning.
3. Plot TD-errors on a set of trials, one for each delay duration, after learning.
4. Implement (1) to (3) in the context of probabilitic rewards ($P(r) < 1$). On each trial, randomly select the delay between CS-US and whether or not to present a reward on that trial. How do TD-errors change when the reward probability is changed?

```python
# Start by using: reward_times = [10 20]
# Once you think you understand why the estimated value function looks the way it does,
# then try reward_times = np.arange(10,21,3) and reward_times = np.arange(10,21,1)
```

```python
reward_probabilities = [0.1, 0.4, 0.6, 0.9, 1]
```

```python

```

# **Exercise 5: **

In this exercise we will implement a commonly used heuristic used in modeling activity of dopamine neurons, TD-reset.
Implement TD-learning as in previous exercises, but set TD-error to zero on all steps after reward (US).

1. Plot value function and TD-errors.
2. Can you explain how the reset is changing the TD-errors and value function?
3. How do they change if rewards are probabilistic ($P(r) < 1$)?

```python
reward_times = np.arange(10,21,1)
reward_probabilities = [0.1, 0.4, 0.6, 0.9, 1]
```

```python

```

# **Exercise 6:**

Implement Exercise 5, after making the following small change: instead of resetting the TD-error, set the next_state after reward delivery as the ITI state, irrespective of the time of reward.

1. Compare value functions and TD-errors in Exercise 5 with those using the modified environment.
2. Can you explain the differences in these two cases?
3. How does this change in the state representation compare with the reset we introduced in Exercise 4 in terms of our interpretation regarding how an agent understand the classical conditioning paradigm?

```python
reward_times = np.arange(10,21,1)
reward_probabilities = [0.1, 0.4, 0.6, 0.9, 1]
```

```python

```

## Are you done?

Awesome! In that case, ask me and I will point you to an interesting paper to read. Given these results, you can evaluate the main claims in that paper with a more critical eye!

```python

```


## Surfinpy

#### Tutorial 1 - Bulk Phase diagrams

In this tutorial we learn how to generate a basic bulk phase diagram from DFT energies.  This enables the comparison of the thermodynamic stability of various different bulk phases under different chemical potentials giving valuable insight in to the syntheis of solid phases.  This example will consider a series of bulk phases which can be defined through a reaction scheme across all phases, thus for this example including an example bulk phase, H2O and CO2 as reactions and A as a generic product.

\begin{align}
x\text{Bulk} + y\text{H}_2\text{O} + z\text{CO}_2 \rightarrow \text{A}
\end{align}

##### Methodology

The system is in equilibrium when the chemical potentials of the reactants and product are equal; i.e. the change in Gibbs free energy is $\delta G_{T,p} = 0$.

\begin{align}
	\delta G_{T,p} = \mu_A - x\mu_{\text{Bulk}} - y\mu_{\text{H}_2\text{O}} - z\mu_{\text{CO}_2} = 0
\end{align}

Assuming that H2O and CO2 are gaseous species, $\mu_{CO_2}$ and $\mu_{H_2O}$ can be written as

\begin{align}
	\mu_{\text{H}_2\text{O}} = \mu^0_{\text{H}_2\text{O}} + \Delta\mu_{\text{H}_2\text{O}}
\end{align}
and
\begin{align}
	\mu_{\text{CO}_2} = \mu^0_{\text{CO}_2} + \Delta\mu_{\text{CO}_2}
\end{align}

The chemical potential $\mu^0_x$ is the partial molar free energy of any reactants or products (x) in their standard states, in this example we assume all solid components can be expressed as

\begin{align}
\mu_{\text{component}} = \mu^0_{\text{component}}
\end{align}

Hence, we can now rearrange the equations to produce;

\begin{align}
	\mu^0_A - x\mu^0_{\text{Bulk}} - y\mu^0_{\text{H}_2\text{O}} - z\mu^0_{\text{CO}_2} = y\Delta\mu_{\text{H}_2\text{O}} + z\Delta\mu_{\text{CO}_2}
\end{align}

As $\mu^0_A$  corresponds to the partial molar free energy of product A, we can replace the left side with the Gibbs free energy ($\Delta G_{\text{f}}^0$).

\begin{align}
	\delta G_{T,p} = \Delta G_{\text{f}}^0 - y\Delta\mu_{\text{H}_2\text{O}} - z\Delta\mu_{\text{CO}_2}
\end{align}

At equilibrium $\delta G_{T,p} = 0$, and hence

\begin{align}
	\Delta G_{\text{f}}^0 = y\Delta\mu_{\text{H}_2\text{O}} + z\Delta\mu_{\text{CO}_2}
\end{align}

Thus, we can find the values of $\Delta\mu_{\text{H}_2\text{O}}$ and $\Delta\mu_{\text{CO}_2}$ (or $(p_{\text{H}_2\text{O}})^y$ and $(p_{\text{CO}_2})^z$) when solid phases are in thermodynamic equilibrium; i.e. they are more or less stable than bulk.  This procedure can then be applied to all phases to identify which is the most stable, provided that the free energy $\Delta G_f^0$ is known for each phase.

The free energy can be calculated using

\begin{align}
\Delta G^{0}_{f} = \sum\Delta G_{f}^{0,\text{products}} - \sum\Delta G_{f}^{0,\text{reactants}}
\end{align}

Where for this tutorial the free energy (G) is equal to the calculated DFT energy (U0).

```python
import matplotlib.pyplot as plt
from surfinpy import bulk_mu_vs_mu as bmvm
from surfinpy import utils as ut
from surfinpy import data
```

The first thing to do is input the data that we have generated from our DFT simulations. The input data needs to be contained within a `surfinpy.data` object. First we have created a `surfinpy.data.ReferenceDataSet` object for the bulk data (reference data), where `cation` is the number of cations, `anion` is the number of anions, `energy` is the DFT energy and `funits` is the number of formula units.

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 1, energy = -92.0, funits = 10)
```

Next we create the bulk `surfinpy.data.DataSet` objects - one for each surface or "phase". `cation` is the number of cations, `x` is in this case the number of oxygen species (corresponding to the X axis of the phase diagram), `y` is the number of in this case water molecules (corresponding to the Y axis of our phase diagram), `energy` is the DFT energy, `label` is the label for the phase (appears on the phase diagram) and finally `nSpecies` is the number of adsorbin species.

```python
Bulk = data.DataSet(cation = 10, x = 0, y = 0, energy = -92.0, label = "Bulk")
A = data.DataSet(cation = 10, x = 5, y = 20, energy = -468.0, label = "A")
B = data.DataSet(cation = 10, x = 0, y = 10, energy = -228.0, label = "B")
C = data.DataSet(cation = 10, x = 10, y = 30, energy = -706.0, label = "C")
D = data.DataSet(cation = 10, x = 10, y = 0, energy = -310.0, label = "D")
E = data.DataSet(cation = 10, x = 10, y = 50, energy = -972.0, label = "E")
F = data.DataSet(cation = 10, x = 8, y = 10, energy = -398.0, label = "F")

```

Next we need to create a list of our data. Don't worry about the order, surfinpy will sort that out for you.

```python
data = [Bulk, A, B, C,  D, E, F]
```

We now need to generate our X and Y axis, or more appropriately, our chemical potential values. These exist in a dictionary. 'Range' corresponds to the range of chemcial potential values to be considered and 'Label' is the axis label.  Additionally, the x and y energy need to be specified.

```
    deltaX = {'Range': Range of Chemical Potential,
              'Label': Species Label}

```

```python
deltaX = {'Range': [ -3, 2],  'Label': 'CO_2'}
deltaY = {'Range': [ -3, 2], 'Label': 'H_2O'}
x_energy=-20.53412969
y_energy=-12.83725889
```

And finally we can generate our plot using these 6 variables of data.

```python
system = bmvm.calculate(data, bulk, deltaX, deltaY, x_energy, y_energy)

ax = system.plot_phase(figsize=(6, 4.5))
plt.show()
```

```python

```


SNIJPUNT VAN TWEE RECHTEN

Naast de snijpunten met de assen, is het ook belangrijk dat je snijpunten met andere rechten kunt berekenen. Dit kan bijvoorbeeld helpen bij het bepalen van de oplossing van een vraagstuk of bij het grafisch oplossen van een probleem.

Ook hiervoor bestaan er reeds functies in Python in de module SymPy.

### Nodige modules importeren

```python
import numpy as np
import matplotlib.pyplot as plt

from sympy import Symbol
from sympy.solvers import solve
```

1. Twee rechten met gegeven vergelijking plotten

In de notebook 'Rechten' leerde je reeds een grafiek van een rechte plotten. Nu moeten er twee rechten getoond worden. Dit vergt slechte een kleine aanpassing in de code.

### Opdracht 1.1
Je zal a.d.h.v. de volgende code-cel de grafiek van *twee* rechten met gegeven vergelijking laten zien.

De twee rechten worden gegeven door de vergelijkingen: $ y = -3x + 5$ en $ y = 2x + 16 $.

De code staat echter *niet* in de juiste volgorde.

Zet alles in de goede volgorde en voer het script uit.

```python
# TWEE RECHTEN MET GEGEVEN VERGELIJKING PLOTTEN

plt.figure()                                                           # creër tekenvenster
plt.show()                                                             # toon grafiek

# vergelijking van de eerste rechte: y = -3x + 5
# vergelijking van de tweede rechte: y = 2x + 16
y1 = -3 * x + 5                                                        # overeenkomstige y-coördinaten voor eerste rechte
y2 = 2 * x + 16                                                        # overeenkomstige y-coördinaten voor tweede rechte

x = np.arange(-9, 9, 1)                                                # kies x-coördinaten van punten die geplot worden

plt.axis(xmin=-10, xmax=10, ymin=-30, ymax=40)                         # vensterinstellingen
plt.xticks(np.arange(-10, 11, step=1))                                 # verdeling x-as
plt.yticks(np.arange(-30, 45, step=5))                                 # verdeling y-as
plt.hlines(0, -10, 10)                                                 # x-as
plt.vlines(0, -30, 40)                                                 # y-as
plt.grid(True)                                                         # rooster

plt.plot(x, y1, color="blue", linewidth=1.0, linestyle="solid")        # plot eerste rechte
plt.plot(x, y2, color="red", linewidth=4.0, linestyle="dashed")        # plot tweede rechte
```

2. Twee rechten met gegeven vergelijking met elkaar snijden

De module SymPy is geschikt om vergelijkingen op te lossen, coördinaten te berekenen, nulwaarden te bepalen en stelsels op te lossen. Om het snijpunt van twee rechten te bepalen, moet een stelsel vergelijkingen worden opgelost. Dus ook om met Python het snijpunt van twee rechten te berekenen, gebruik je de module Sympy. De nodige zaken zijn reeds geïmporteerd.

Naast de zaken die hierboven werden opgesomd, is SymPy ook handig omdat het rationale getallen niet gaat benaderen maar laat staan als breuken. Worteltekens gaat SymPy ook steeds behouden, eventueel vereenvoudigd.

### Voorbeeld
A.d.h.v. de volgende code-cel bereken je het snijpunt van de twee snijdende rechten uit puntje 1. met Python.

Om het snijpunt van de twee rechten te bepalen, moet je een stelsel van twee vergelijkingen oplossen met x en y als onbekenden:
$$
\left\{
    \begin{array}{ll}
        -3 x + 5 - y = 0 \\
        2 x + 16 - y = 0
    \end{array}
\right.
.$$
**Je zal in Python ook die rol van x en y als onbekenden moeten aangeven.**

**Merk op dat de vergelijkingen herleid zijn naar nul.**

Bekijk grondig de code in de cel en voer ze uit.

```python
# TWEE RECHTEN MET GEGEVEN VERGELIJKING SNIJDEN

# x en y beschouwen als onbekenden
x = Symbol("x")
y = Symbol("y")

# vergelijking van de eerste rechte: y = -3x + 5
# vergelijking van de tweede rechte: y = 2x + 16

# snijpunt berekenen door het stelsel met vergelijkingen −3 𝑥 + 5 − 𝑦 = 0  en 2 𝑥 + 16 − 𝑦 = 0 op te lossen naar x en y
snijpunt = solve((-3 * x + 5 - y, 2 * x + 16 - y), x, y)

print(snijpunt)
```

De functie `solve()` geeft een *dictionary* terug. Let op de accolades, die aangeven dat de volgorde in een dictionary niet van belang is. Voor het probleem hier is die volgorde natuurlijk wel van belang: de volgorde wordt benadrukt door de *keys* van de dictionary, nl. x en y.

Meer uitleg over dictionaries vind je in de notebooks uit Programmeerelementen over dataypes.

```python
# betere weergave van berekende snijpunt
print("Het snijpunt van de twee rechten is: ", (snijpunt[x], snijpunt[y]))
```

```python
# SNIJPUNT PLOTTEN

plt.figure()

plt.plot(snijpunt[x], snijpunt[y], color="plum", marker="o")

plt.show()
```

3. Het snijpunt van twee rechten berekenen en laten zien

### Opdracht 3.1
Voeg de voorbeeldcode uit puntje 2. samen met de (aangepaste) code uit opdracht 1.1 en test uit.

Bij uitvoering van de code:
-  wordt het snijpunt van de twee rechten bepaald;
-  worden de twee rechten getoond;
-  wordt ook het snijpunt getoond.

Tip: zorg ervoor dat je niet twee keer dezelfde variabelenaam gebruikt.

```python

```

    Bij het aanpakken van dit probleem komen concepten van computationeel denken aan bod:

    - Decompositie, want het probleem bestaat uit twee deelproblemen: het berekenen van de coördinaat van het snijpunt en het maken van een grafiek. Bij de opbouw van een grafiek moet je goed nadenken over de verschillende elementen die in de grafiek moeten worden opgenomen, o.a. de twee rechten en het snijpunt.

    - Voor beide deelproblemen stel je een algoritme op; die algoritmes worden samengevoegd tot ze het script vormen dat zal worden uitgevoerd.

    - Patroonherkenning: de opbouw van beide algoritmes verloopt volgens een vast patroon.

    De volgorde van de twee algoritmes is niet vrij. Het snijpunt kan pas weergegeven worden op de grafiek, als zijn coördinaat gekend is. Hier moet je dus bewust stilstaan bij de sequentie, een van de principes van programmeren.

Alles wordt samengvat in dit schema:

    .

4. Oefeningen

### Oefening 4.1
Doe hetzelfde voor twee andere schuine rechten, nl. de rechte met vergelijking $y = 4 x - 6$ en de schuine rechte met vergelijking $y = 7x + 31$.

```python

```

### Oefening 4.2
Doe hetzelfde voor twee andere rechten, nl. de horizontale rechte met vergelijking $y = 4$ en de schuine rechte met vergelijking
$y = - 8 x + 12$.

```python

```

### Oefening 4.3
Doe hetzelfde voor twee andere rechten, nl. de horizontale rechte met vergelijking $y = -2$ en de verticale rechte met vergelijking $x = 6$.

```python

```

Notebook Python in wiskunde, zie AI Op School, van F. wyffels, C. Boitsios & N. Gesquière is in licentie gegeven volgens een Creative Commons Naamsvermelding-NietCommercieel-GelijkDelen 4.0 Internationaal-licentie.


## Physics 152L Radioactivity Lab ##

### 1. Read in the data

Import the usual libraries, and then read in the data from the saved CSV file into a Pandas dataframe.

We can use df.info() to get a basic summary of the data.

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

df = pd.read_csv("./RadioactivityLab.csv")

df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 686 entries, 0 to 685
    Data columns (total 20 columns):
     #   Column                                      Non-Null Count  Dtype
    ---  ------                                      --------------  -----
     0   Time (s) Background                         75 non-null     float64
     1   Geiger Counts (counts/sample) Background    75 non-null     float64
     2   RootN (cts/5s) Background                   75 non-null     float64
     3   Background Subtracted (cts/5s) Background   75 non-null     float64
     4   Time (s) Background2                        686 non-null    float64
     5   Geiger Counts (counts/sample) Background2   686 non-null    int64
     6   RootN (cts/5s) Background2                  686 non-null    float64
     7   Background Subtracted (cts/5s) Background2  686 non-null    float64
     8   Time (s) Barium                             144 non-null    float64
     9   Geiger Counts (counts/sample) Barium        144 non-null    float64
     10  RootN (cts/5s) Barium                       144 non-null    float64
     11  Background Subtracted (cts/5s) Barium       144 non-null    float64
     12  Time (s) Radium                             84 non-null     float64
     13  Geiger Counts (counts/sample) Radium        84 non-null     float64
     14  RootN (cts/5s) Radium                       84 non-null     float64
     15  Background Subtracted (cts/5s) Radium       84 non-null     float64
     16  Time (s) RadSamples                         240 non-null    float64
     17  Geiger Counts (counts/sample) RadSamples    240 non-null    float64
     18  RootN (cts/5s) RadSamples                   240 non-null    float64
     19  Background Subtracted (cts/5s) RadSamples   240 non-null    float64
    dtypes: float64(19), int64(1)
    memory usage: 107.3 KB

### 2. Splitting Up The Data ###

Inspection of the dataframe shows us that there are five data sets from the different
parts of the experiment that are combined into our single CSV file, and hence into our
initial dataframe.

We can use df.filter() to parse this main dataframe and create separate dataframes for
each part of the experiment.

```python
background = df.filter(['Time (s) Background',
                        'Geiger Counts (counts/sample) Background',
                        'RootN (cts/5s) Background',
                        'Background Subtracted (cts/5s) Background'], axis=1).dropna()
background2 = df.filter(['Time (s) Background2',
                        'Geiger Counts (counts/sample) Background2',
                        'RootN (cts/5s) Background2',
                        'Background Subtracted (cts/5s) Background2'], axis=1).dropna()
barium = df.filter(['Time (s) Barium',
                        'Geiger Counts (counts/sample) Barium',
                        'RootN (cts/5s) Barium',
                        'Background Subtracted (cts/5s) Barium'], axis=1).dropna()
radium = df.filter(['Time (s) Radium',
                        'Geiger Counts (counts/sample) Radium',
                        'RootN (cts/5s) Radium',
                        'Background Subtracted (cts/5s) Radium'], axis=1).dropna()
radsamples = df.filter(['Time (s) RadSamples',
                        'Geiger Counts (counts/sample) RadSamples',
                        'RootN (cts/5s) RadSamples',
                        'Background Subtracted (cts/5s) RadSamples'], axis=1).dropna()
```

### 3.  Part I:  Background Measurments ###

We begin by plotting the data from part I.

In order to calculate the average background rate, with uncertainty, we do the following:
* Get the total integrated counts
* Get the total time
* Calculate the average rate
* Calculate the uncertainty, using the fact that $\Delta N = \sqrt{N}$.

There are two background measurements in the CSV file, and so we will do the same thing
for each of these two data sets.

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(background['Time (s) Background'],
         background['Geiger Counts (counts/sample) Background'],'r.', label = 'Raw Data')
plt.plot(background['Time (s) Background'],
         background['RootN (cts/5s) Background'],'b--', label = 'Cumulative Error')
plt.plot(background['Time (s) Background'],
         background['Background Subtracted (cts/5s) Background'],'g-',
         label = 'Background Subtracted')
plt.legend()

entries = len(background)
count_sum = background['Geiger Counts (counts/sample) Background'].sum()
end_time = background['Time (s) Background'][entries-1]
rate = count_sum/end_time*60
erate = np.sqrt(count_sum)/end_time*60.0

print ('Counts = ',count_sum,' Time = ',end_time,' Rate = ',rate,' +/- ',erate)
```

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(background2['Time (s) Background2'],
         background2['Geiger Counts (counts/sample) Background2'],'r.',label = 'Raw Data' )
plt.plot(background2['Time (s) Background2'],
         background2['RootN (cts/5s) Background2'],'b--', label = 'Cumulative Error' )
plt.plot(background2['Time (s) Background2'],
         background2['Background Subtracted (cts/5s) Background2'],
         'g-', label = 'Background Subtracted')
plt.legend()

entries = len(background2)
count_sum = background2['Geiger Counts (counts/sample) Background2'].sum()
end_time = background2['Time (s) Background2'][entries-1]
rate = count_sum/end_time*60
erate = np.sqrt(count_sum)/end_time*60.0

print ('Counts = ',count_sum,' Time = ',end_time,' Rate = ',rate,' +/- ',erate)
```

### 3.  Barium Measurements

Plot the barium data.  The expected distribution is:

\begin{equation}
R(t) = R_0 e^{-Bt} \nonumber
\end{equation}

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(barium['Time (s) Barium'],
         barium['Background Subtracted (cts/5s) Barium'],'g.',
         label='Background Subtracted')
```

So, the distribution looks roughly exponential, which is good.  But, in order to extract the parameters of this distribution, i.e. $R_0$ and $B$, we take a slightly different approach.

Taking the natural logarithm of both sides, we have that:

\begin{equation}
ln(R) = ln(R_0 e^{-Bt}) = ln(R_0) - Bt \nonumber
\end{equation}

Thus, if we plot $ln(R)$ vs. $t$, then we expect a $linear$ graph, with a positive y-intercept (=$ln(R_0)$) and negative slope (=$-B$).

We can use the linear regression fitting model from sklearn to fit this data, and extract
the numerical values of the slope and intercept.

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(barium['Time (s) Barium'],
         np.log(barium['Background Subtracted (cts/5s) Barium']),'g.',
         label='Background Subtracted')

from sklearn import linear_model

reg = linear_model.LinearRegression()

x_data = barium['Time (s) Barium'].values.reshape(-1, 1)
y_data = np.log(barium['Background Subtracted (cts/5s) Barium'].values.reshape(-1, 1))

reg.fit(x_data,y_data)

print ('Fit: y_intercept = ',reg.intercept_)
print ('Fit: slope = ',reg.coef_)

y_fit = x_data*reg.coef_ + reg.intercept_
plt.plot(x_data,y_fit,'r-',label='Linear Fit')
plt.legend()
```

Note that the linear fit is NOT a particularly good one, because by the end of the data set, it
appears that the radioactive material has mostly decayed away, and we are not seeing the
same exponential fall-off.

So, what to do?  Well, we can limit the region of the fit to just the first 450 second or so, where the decay does appear to follow the expected log-linear dependence.

```python
barium2 = barium[barium['Time (s) Barium'] < 450]

fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(barium['Time (s) Barium'],
         np.log(barium['Background Subtracted (cts/5s) Barium']),'g.',
         label='Background Subtracted')

from sklearn import linear_model

reg = linear_model.LinearRegression()

x_data = barium2['Time (s) Barium'].values.reshape(-1, 1)
y_data = np.log(barium2['Background Subtracted (cts/5s) Barium'].values.reshape(-1, 1))

reg.fit(x_data,y_data)

print ('Fit: y_intercept = ',reg.intercept_)
print ('Fit: slope = ',reg.coef_)

print ('Half-Life = ',float(-np.log(2)/reg.coef_),' seconds.')

y_fit = x_data*reg.coef_ + reg.intercept_
plt.plot(x_data,y_fit,'r-',label='Linear Fit')
plt.legend()
```

###  4. Measurements with Different Absorbers ###

Let's begin by looking at the raw data.

We that there are five distinct "sections" in the data set that correspond to five
different experimental conditions:

* Gamma + Beta Source with no absorber
* Gamma + Beta Source with absorber to block beta particles
* Gamma + Beta Source with Cardboard absorber
* Gamma + Beta Source with Aluminum absorber
* Gamma + Beta Source with Lead absorber

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radium['Time (s) Radium'],radium['Background Subtracted (cts/5s) Radium'],'g-')
```

For each of the five measurements, we will set a time range (t1min and t1max).  Then, we filter the data set to include only measurements during these time windows.  Once we have the filtered data set, we can calculate an average rate over that time window.  In each case, we plot a horizontal line at the appropriate rate for that time window on top of the raw data.

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radium['Time (s) Radium'],radium['Background Subtracted (cts/5s) Radium'],'g-')

t1min = 0.0
t1max = 90.0
gammabeta = radium[radium['Time (s) Radium']>t1min]
gammabeta = gammabeta[gammabeta['Time (s) Radium']<t1max]

gammabeta_rate = gammabeta['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Gamma+Beta Rate = ',gammabeta_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([gammabeta_rate,gammabeta_rate])
plt.plot(x,y,'b-')

t1min = 100.0
t1max = 170.0
gamma = radium[radium['Time (s) Radium']>t1min]
gamma = gamma[gamma['Time (s) Radium']<t1max]

gamma_rate = gamma['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Gamma Rate = ',gamma_rate,' cts/5 sec')
print ('Beta Rate = ',gammabeta_rate-gamma_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([gamma_rate,gamma_rate])
plt.plot(x,y,'b-')

print ("")

t1min = 180.0
t1max = 240.0
cardboard = radium[radium['Time (s) Radium']>t1min]
cardboard = cardboard[cardboard['Time (s) Radium']<t1max]

cardboard_rate = cardboard['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Cardboard Rate = ',cardboard_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([cardboard_rate,cardboard_rate])
plt.plot(x,y,'b-')

t1min = 260.0
t1max = 305.0
aluminum = radium[radium['Time (s) Radium']>t1min]
aluminum = aluminum[aluminum['Time (s) Radium']<t1max]

aluminum_rate = aluminum['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Aluminum Rate = ',aluminum_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([aluminum_rate,aluminum_rate])
plt.plot(x,y,'b-')

t1min = 340.0
t1max = 410.0
lead = radium[radium['Time (s) Radium']>t1min]
lead = lead[lead['Time (s) Radium']<t1max]

lead_rate = lead['Background Subtracted (cts/5s) Radium'].mean()*1.0
print ('Lead Rate = ',lead_rate,' cts/5 sec')

x = np.array([t1min,t1max])
y = np.array([lead_rate,lead_rate])
plt.plot(x,y,'b-')
```

###  4. Measurements with Different Sources (With and Without Lead Absorber)

In this part of the experiment, there are ten separate measurements, corresponding to five different sources, with data taken both with and without a lead absorber.

We begin by plotting the raw data, and we can clearly see the ten different situations.

As in the last section, we will calculate the average rate for each of these ten time windows, and plot this average rate on top of the data.  Of course, we need to define the ten time windows (as four arrays - two for no absorber and two for absorber).

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radsamples['Time (s) RadSamples'],
         radsamples['Background Subtracted (cts/5s) RadSamples'],'g-')
```

```python
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')
plt.plot(radsamples['Time (s) RadSamples'],
         radsamples['Background Subtracted (cts/5s) RadSamples'],'g-')

tsmin = np.array([0.0,250.0,500.0,720.0,990.0])
tsmax = np.array([105.0,340.0,580.0,830.0,1070.0])
tlmin = np.array([120.0,375.0,650.0,850.0,1100.0])
tlmax = np.array([220.0,460.0,720.0,950.0,1190.0])

for i in range(5):
    t1min = tsmin[i]
    t1max = tsmax[i]
    rate = radsamples[radsamples['Time (s) RadSamples']>t1min]
    rate = rate[rate['Time (s) RadSamples']<t1max]

    no_lead_rate = rate['Background Subtracted (cts/5s) RadSamples'].mean()*1.0
    print ('NoLeadRate[',i+1,'] = ',no_lead_rate,' cts/5 sec')

    x = np.array([t1min,t1max])
    y = np.array([no_lead_rate,no_lead_rate])
    plt.plot(x,y,'b-')

    t1min = tlmin[i]
    t1max = tlmax[i]
    rate = radsamples[radsamples['Time (s) RadSamples']>t1min]
    rate = rate[rate['Time (s) RadSamples']<t1max]

    lead_rate = rate['Background Subtracted (cts/5s) RadSamples'].mean()*1.0
    print ('LeadRate[',i+1,'] = ',lead_rate,' cts/5 sec')

    x = np.array([t1min,t1max])
    y = np.array([lead_rate,lead_rate])
    plt.plot(x,y,'b-')
    print()
```

```python

```

```python

```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 9: Introduction to Business Cycle Data

The *business cycle* is the fluctuation of many macroeconomic quantities that last for about 1.5 to 8 years. Colloquially, the term refers to the alternating periods of expansion and contraction in the macroeconomy. Business cycle fluctuations are costly because they are associated with misallocations of capital and labor. Recessions are particularly painful for workers that become unemployed and for the families of workers who become unemployed. The costs of the business cycle have driven research into understanding the cause of the cycle.

The collective set of theories to explain the cycle is called *business cycle theory* and eventually we'll study and critique two competing theoretical perspectives. However, before we approach the theory, we should first uncover some empirical facts about the business cycle that models should be able to explain.

In this lecture, we will:

1. Visualize the difference between the trend and cyclical components of GDP, consumption, investent, and hours.
2. Compute the log-deviations from trend of GDP, consumption, investent, and hours.
3. Compute summary statistics about the business cycle that models of the cycle should be able to explain.

## Data

The file `rbc_data_actual_trend.csv`, available at https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/rbc_data_actual_trend.csv, contains actual and trend data for real GDP per capita, real consumption per capita, real investment per capita, real physical capital per capita, TFP, and hours per capita at quarterly frequency. The GDP, consumption, investment, and capital data are in terms of 2012 dollars. Hours is measured as an index with the value in October 2012 set to 100. All of the data are *real* quantities. That is, there are no *nominal* quantities like money or inflation or a nominal interest rate. The reason is that the first theory that we will encounter is called *real business cycle* or RBC theory and, in that theory, there is no place for nominal quantities. RBC theory seeks to explain fluctuations in real quantities as being primarily due to TFP shocks; i.e., shocks to the production function.

```python
# Read business_cycle_data_actual_trend.csv into a Pandas DataFrame with the first column set as the index and parse_dates=True

# Print the last five rows of the data

```

```python
# Construct a 2x2 grid of plots of GDP, consumption, investment, and hours. CELL PROVIDED
base_year = '2012'
fig = plt.figure(figsize=(12,8))

ax1 = fig.add_subplot(2,2,1)
ax1.plot(df.gdp,'-',lw=3,alpha = 0.7)
ax1.grid()
ax1.set_title('GDP per capita')
ax1.set_ylabel('Thousands of '+base_year+' dollars')

ax2 = fig.add_subplot(2,2,2)
ax2.plot(df.consumption,'-',lw=3,alpha = 0.7)
ax2.grid()
ax2.set_title('Consumption per capita')
ax2.set_ylabel('Thousands of '+base_year+' dollars')

ax3 = fig.add_subplot(2,2,3)
ax3.plot(df.investment,'-',lw=3,alpha = 0.7)
ax3.grid()
ax3.set_title('Investment per capita')
ax3.set_ylabel('Thousands of '+base_year+' dollars')

ax4 = fig.add_subplot(2,2,4)
ax4.plot(df.hours,'-',lw=3,alpha = 0.7)
ax4.grid()
ax4.set_title('Hours per capita')
ax4.set_ylabel('Index ('+base_year+'=100)')

fig.tight_layout()
```

## Cycles and Trends

Business cycle theory is tested against data. However, data on the business cycle data is not readily available and must be constructed. A time series $X_t$ can be decomposed into a trend component $X_t^{trend}$ and a cyclical component $X_t^{cycle}$ such that:

\begin{align}
X_t & = X_t^{trend} + X_t^{cycle}. \tag{1}
\end{align}

In equation (1), $X_t^{trend}$ is the long-run value about which $X_t$ fluctuates. $X^{cycle}_t$ is the amount by which $X_t$ excedes its trend. The process for decomposing a series into trend and cyclical components is called *filtering* and is more technical than we want to get into. We'll take for granted that such procedures exist.

Often times, it's useful to express the cyclical component of a time series as the difference between the (natural) log of the series and the log of the trend:

\begin{align}
\hat{x}_t & = \log\left(X_t\right) - \log\left(X_t^{trend}\right) \approx\frac{X_t-X_t^{trend}}{X_t^{trend}}
\end{align}

The log-deviation from trend is approximately equal to the percent deviation of the series from trend (divided by 100).

### GDP

```python
# Construct a plot of real GDP with its trend with:
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'

```

```python
# Create a new column called gdp_cycle equal to the difference between acual and trend GDP

# Create a new column called gdp_cycle_dev equal to the log difference between actual GDP and trend GDP:

# Plot the log deviation of GDP from its trend (times 100)

```

### Consumption, Investment, and Hours

```python
# Create three new columns called cons_cycle, invest_cycle, and hours_cycle equal to the cyclical components of the
# respective series

# Create a new column called cons_cycle_dev, invest_cycle_dev, and hours_cycle_dev equal to the log difference between
# the actual and trend values of the respective series:

```

```python
# Construct a plot of consumption with its trend
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'

```

```python
# Plot the log deviation of consumption from its trend (times 100)

```

```python
# Construct a plot of investment with its trend
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'

```

```python
# Plot the log deviation of investment from its trend (times 100)

```

```python
# Construct a plot of hours with its trend
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'

```

```python
# Plot the log deviation of hours from its trend (times 100)

```

## US Business Cycle Statistics

```python
# Create a new variable called df_cycle that is a DataFrame with columns columns gdp_cycle_dev, cons_cycle_dev,
# invest_cycle_dev, and hours_cycle_dev from df.

# Print the first five rows of df_cycle

```

```python
# Use the DataFrame method .mean() to find the average values of the gdp_cycle_dev, cons_cycle_dev, invest_cycle_dev,
# and hours_cycle_dev columns

```

```python
# Use the DataFrame method .std() to find the standard deviations of the gdp_cycle_dev, cons_cycle_dev, invest_cycle_dev,
# and hours_cycle_dev columns

```

```python
# Use the DataFrame method .corr() to find the coefficients of correlation among the gdp_cycle_dev, cons_cycle_dev,
# invest_cycle_dev, and hours_cycle_dev columns

```

**Questions**

1. Which quantity varies the most over the business cycle?
2. Which quantity varies the least over the business cycle?
3. Which quantity is most correlated with GDP over the business cycle?

**Answers**

1.

2.

3.

```python
# Plot the cyclical components of GDP, consumption, investment, and hours (all times 100) on the same set of axes

```


Table of Contents

1  Word2vec (Skipgram)

1.1  Model Details

1.2  The Hidden Layer

1.3  The Output Layer

2  Improving Word2vec

2.1  Negative Sampling

2.1.1  Mathematical Notation

2.1.2  Selecting Negative Samples

2.2  Subsampling Frequenct Words

2.3  Detecting Phrases

3  Implementation

4  Final Thoughts

4.1  Hyperparameters

4.2  Applications

4.3  Resources

5  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import numpy as np
import pandas as pd
from time import time
from joblib import cpu_count
from collections import Counter
from gensim.models import Word2Vec
from sklearn.datasets import fetch_20newsgroups

%watermark -a 'Ethen' -d -t -v -p numpy,scipy,pandas,sklearn,gensim
```

    Ethen 2019-06-14 09:43:53

    CPython 3.6.4
    IPython 7.5.0

    numpy 1.16.1
    scipy 1.2.1
    pandas 0.24.2
    sklearn 0.20.3
    gensim 3.6.0

# Word2vec (Skipgram)

At a high level `Word2Vec` is a unsupervised learning algorithm that uses a shallow neural network (with one hidden layer) to learn the vectorial representations of all the unique words/phrases for a given corpus. The advantage that word2vec offers is it tries to preserve the semantic meaning behind those terms. For example, a document may employ the words "dog" and "canine" to mean the same thing, but never use them together in a sentence. Ideally, the word2vec algorithm would be able to learn the context and place them together in similar vector semantic space.

We'll start off by using the Gensim's implementation of the algorithm to provide a high-level intuition.

```python
# the .data attribute will access the raw data, where
# each element is a document
newsgroups_train = fetch_20newsgroups(subset='train')

# gensim’s Word2vec expects a sequence of sentences as its input,
# where each sentence a list of words. We'll be lazy for now
# and not perform any sort of text preprocessing
sentences = [doc.strip().split() for doc in newsgroups_train.data]

# example output of the data
print('raw data:\n\n', newsgroups_train.data[0])
print('example input:\n', sentences[0])
```

    raw data:

     From: lerxst@wam.umd.edu (where's my thing)
    Subject: WHAT car is this!?
    Nntp-Posting-Host: rac3.wam.umd.edu
    Organization: University of Maryland, College Park
    Lines: 15

     I was wondering if anyone out there could enlighten me on this car I saw
    the other day. It was a 2-door sports car, looked to be from the late 60s/
    early 70s. It was called a Bricklin. The doors were really small. In addition,
    the front bumper was separate from the rest of the body. This is
    all I know. If anyone can tellme a model name, engine specs, years
    of production, where this car is made, history, or whatever info you
    have on this funky looking car, please e-mail.

    Thanks,
    - IL
       ---- brought to you by your neighborhood Lerxst ----

    example input:
     ['From:', 'lerxst@wam.umd.edu', "(where's", 'my', 'thing)', 'Subject:', 'WHAT', 'car', 'is', 'this!?', 'Nntp-Posting-Host:', 'rac3.wam.umd.edu', 'Organization:', 'University', 'of', 'Maryland,', 'College', 'Park', 'Lines:', '15', 'I', 'was', 'wondering', 'if', 'anyone', 'out', 'there', 'could', 'enlighten', 'me', 'on', 'this', 'car', 'I', 'saw', 'the', 'other', 'day.', 'It', 'was', 'a', '2-door', 'sports', 'car,', 'looked', 'to', 'be', 'from', 'the', 'late', '60s/', 'early', '70s.', 'It', 'was', 'called', 'a', 'Bricklin.', 'The', 'doors', 'were', 'really', 'small.', 'In', 'addition,', 'the', 'front', 'bumper', 'was', 'separate', 'from', 'the', 'rest', 'of', 'the', 'body.', 'This', 'is', 'all', 'I', 'know.', 'If', 'anyone', 'can', 'tellme', 'a', 'model', 'name,', 'engine', 'specs,', 'years', 'of', 'production,', 'where', 'this', 'car', 'is', 'made,', 'history,', 'or', 'whatever', 'info', 'you', 'have', 'on', 'this', 'funky', 'looking', 'car,', 'please', 'e-mail.', 'Thanks,', '-', 'IL', '----', 'brought', 'to', 'you', 'by', 'your', 'neighborhood', 'Lerxst', '----']

```python
# apart from the input sentence, the only additional paramter
# we'll set is to specify use all possible cpu to train the model
workers = cpu_count()

start = time()
word2vec = Word2Vec(sentences, workers=workers)
elapse = time() - start
print('elapse time:', elapse)

# obtain the learned word vectors (.wv.vectors)
# and the vocabulary/word that corresponds to each word vector
word_vectors = pd.DataFrame(word2vec.wv.vectors, index=word2vec.wv.index2word)
print('word vector dimension: ', word_vectors.shape)
word_vectors.head()
```

    elapse time: 11.304150104522705
    word vector dimension:  (44593, 100)

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      90
      91
      92
      93
      94
      95
      96
      97
      98
      99

      the
      0.800643
      -1.986527
      0.977325
      -0.605457
      -0.930851
      0.529532
      0.863752
      -0.062699
      -0.299589
      -0.153011
      ...
      0.242826
      -1.691200
      -2.657021
      1.045379
      -1.026594
      0.227805
      1.456893
      -0.545802
      -0.542880
      -0.391885

      to
      0.219040
      1.816056
      -2.354737
      -0.099213
      -4.378406
      -0.780509
      -1.826088
      -1.877190
      -3.262380
      -0.298119
      ...
      0.217874
      -4.101723
      -0.673022
      -1.751550
      -1.243698
      2.592210
      1.250851
      1.222139
      1.038643
      -1.465834

      of
      -0.350918
      -1.358160
      0.871220
      -0.373295
      -0.763746
      1.521049
      -2.074889
      0.788673
      0.642699
      0.719654
      ...
      1.431210
      -1.377120
      -0.577969
      -0.647040
      -0.602083
      0.210434
      2.037229
      -0.518271
      1.195434
      0.720554

      a
      -2.368490
      -1.427992
      -0.131853
      1.031120
      -2.994447
      0.075319
      -0.848089
      0.646724
      -0.175409
      0.728912
      ...
      0.025669
      -0.203032
      -3.961305
      2.411133
      -2.031027
      0.556540
      2.303332
      -1.757780
      -0.780398
      -2.324198

      and
      -2.012166
      0.505729
      1.470638
      0.043076
      -0.796829
      -1.076530
      -0.330089
      -0.839426
      0.635126
      0.202480
      ...
      -0.248939
      -1.219598
      -1.832169
      -1.141254
      -0.892952
      -0.717763
      1.994657
      0.249245
      0.011719
      -1.332175

5 rows × 100 columns

After the model has learned the word vector for each valid word in our corpus, we can use them to look up related words and phrases (words that have similar semantic meaning) for a given term of interest by comparing distances between the vectors using distance metric such as cosine distance.

```python
word2vec.wv.most_similar(positive=['computer'], topn=5)
```

    [('keyboard', 0.8483761548995972),
     ('machine', 0.845702052116394),
     ('application', 0.8282846212387085),
     ('network', 0.8205068111419678),
     ('board', 0.8138819932937622)]

Apart from finding similar words using distance metric such as cosine distance, word vectors have the remarkable property that analogies between words seem to be encoded in the difference between word vectors. For example there seems to be a constant male-female difference vector:

$$
\begin{align}
W(woman) - W(man)
&\tilde{=} W(aunt) - W(uncle) \\
&\tilde{=} W(queen) - W(king)
\end{align}
$$

This property means we can also perform vector manipulation (e.g. addition and subtraction) with our word vectors. For example, you might have heard or saw the famous example of: King - male + female = queen. In the next section, we'll be taking a look at this model's inner workings. The explanations "borrows" heavily from the blogs listed below.

- [Blog: Word2Vec Tutorial - The Skip-Gram Model](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/)
- [Blog: Word2Vec Tutorial Part 2 - Negative Sampling](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/).

## Model Details

The way the model works underneath the hood is that trains a neural network by do the following: Given a specific word in the middle of a sentence (the input word), look at the words nearby and pick one at random.

There is a **window size** hyperparameter to the algorithm that quantifies the word "nearby". A typical window size might be 5, meaning 5 words behind and 5 words ahead (10 in total). There are some implementations that does something even fancier: Instead of using a fix $k$ window around each word, the window is uniformly distributed from $1, 2, ..., K$, where $K$ is the max window size we specify.

The diagram below shows some of the training samples (word pairs) we would take from the sentence "The quick brown fox jumps over the lazy dog." We'll used a small window size of 2 just for the example. The word highlighted in blue is the input word.

After feeding a bunch of word pairs to the network, it is going to tell us the probability for every word in our vocabulary being "nearby" word that we chose. The output probabilities are going to relate to how likely it is find each vocabulary word nearby our input word. For example, if we gave the trained network the input word "Soviet", the output probabilities should be much higher for words like "Union" and "Russia" than for unrelated words like "watermelon" and "kangaroo".

So how is this all represented? First of all, we know we can't feed a word just as a text string to a neural network (or probably any machine learning model), i.e. we need a way to represent the words to the network. To do this, we first build a vocabulary of words from our training documents. We'll assume that our corpus has a vocabulary size of 10,000.

We're going to represent an input word like "ants" as a one-hot vector. This vector will have 10,000 components (one for every unique word in our vocabulary) and we'll place a "1" in the position corresponding to the word "ants", and 0s in all of the other positions. The output of the network is a single vector (also with 10,000 components) containing, for every word in our vocabulary, the probability that a randomly selected nearby word is that vocabulary word. Here's the architecture of our single-layer neural network.

There is no activation function on the hidden layer neurons, but the output neurons use softmax. We’ll come back to this later.

When training this network on word pairs, the input is a one-hot vector representing the input word and the training output is also a one-hot vector representing the output word. But when we evaluate the trained network on an input word, the output vector will actually be a probability distribution (i.e., a bunch of floating point values, not a one-hot vector).

An alternative diagram that depicts that Skip-gram model architecture well:

Where we're using the centre word $w_{(t)}$ to predict the surrounding words and the training objective is to learn word vector representations, a.k.a projections that are good at predicting the nearby words.

## The Hidden Layer

Let's say that we wish to learn word vectors with 300 features. The number of features is a hyperparameter that we would have to tune to our application to see which one yields the best result. So the hidden layer is going to be represented by a weight matrix with 10,000 rows (one for every word in our vocabulary) and 300 columns (one for every hidden neuron).

Now if we look at what would happen when we multiply the 1 x 10,000 one-hot vector representation of the word with a 10,000 x 300 matrix that represents the hidden layer's weight, it will effectively just select the matrix row corresponding to the "1". The following figure is a small example that does a matrix multiplication of a 1 x 5 one hot vector with a 5 x 3 hidden layer's weight to give you a visual.

This means that the hidden layer of this model is really just operating as a lookup table and the output of the hidden layer is essentially the "word vector" for the input word.

## The Output Layer

The 1 x 300 word vector for "ants" then gets fed to the output layer. The output layer is a Softmax regression classifier. There's another documentation on Softmax Regression [here](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/deep_learning/softmax.ipynb), but the gist of it is that each output neuron, one per word in our vocabulary will produce an output probability between 0 and 1 and the sum of all these output values will add up to 1.

Specifically, each output neuron has a weight vector which it multiplies against the word vector from the hidden layer, then it applies the function `exp(x)` to the result. Finally, in order to get the outputs to sum up to 1, we divide this result by the sum of the results from all 10,000 output nodes. Here's an illustration of calculating the output probability for the word "car".

Note that neural network does not know anything about the offset of the output word relative to the input word. In other words, it does not learn a different set of probabilities for the word before the input versus the word after.

Recall that in the beginning of the documentation, we mentioned that the goal for word2vec is to represent each word in the corpus as the vector representation while trying to preserve semantic meaning. This means that if two different words have very similar "contexts" (that is, what words are likely to appear around them), then our model needs to output very similar results for these two words. And one way for the network to output similar context predictions for these two words is if the word vectors are similar. So to hit the notion home, if two words have similar contexts, then word2vec is motivated to learn similar word vectors for these two words!

# Improving Word2vec

You may have noticed that the skip-gram neural network contains a huge number of weights ... For our example with 300 features and a vocab of 10,000 words, that's 3M weights in the hidden layer and output layer each! Training this on a large dataset would be slow and prone to overfitting, so the word2vec authors introduced a number of tweaks to make training feasible.

- Modifying the optimization objective with a technique they called "Negative Sampling", which causes each training sample to update only a small percentage of the model's weights.
- Subsampling frequent words to decrease the number of training examples.
- Treating common word pairs or phrases as single word in the model.

It's worth noting that subsampling frequent words and applying Negative Sampling not only reduced the compute burden of the training process, but also improved the quality of their resulting word vectors as well.

## Negative Sampling

Training a neural network means taking a training example and adjusting all of the neuron weights slightly so that it predicts that training sample more accurately. In other words, each training sample will tweak all of the weights in the neural network. As we discussed above, the size of our word vocabulary means that our skip-gram neural network has a tremendous number of weights, all of which would be updated slightly by every one of our billions of training samples! Negative sampling addresses this by having each training sample only modify a small percentage of the weights, rather than all of them. Here's how it works:

When training the network on the word pair ("fox", "quick"), we want the "correct output" of the network, that is the output neuron corresponding to "quick" to output a 1, and for all of the other thousands of output neurons to output a 0.

With negative sampling, we are instead going to randomly select just a small number of "negative" words (let's say 5) to update the weights for. (In this context, a "negative" word is one for which we want the network to output a 0 for). We will also still update the weights for our "positive" word (which is the word "quick" in our current example).

> The paper says that selecting 5-20 words works well for smaller datasets, and we can get away with only 2-5 words for large datasets.

Recall that the output layer of our model has a weight matrix that's 300 x 10,000. So we will just be updating the weights for our positive word ("quick"), plus the weights for 5 other words that we want to output 0. That's a total of 6 output neurons, and 1,800 weight values total. That's only 0.06% of the 3M weights in the output layer!

In the hidden layer, only the weights for the input word are updated (this is true whether you're using Negative Sampling or not).

### Mathematical Notation

This section goes back and re-visit the mathematical notation for Word2vec skipgram's objective function. Hopefully, the math won't look so daunting after having an understanding of the model from a non-mathematical standpoint.

In this model, we are given a corpus of words $w$ and their contexts $c$ (the word pair of the targeted word that we've sampled). Our goal is to set the parameters $\theta$ of $p(c | w; \theta)$ to maximize our objective function $J_\theta$, i.e. the corpus probability, with regards to our model parameters $\theta$:

$$
\begin{align}
J_{\theta} &=
arg\underset{\theta}{max} \prod_{w \in Text} \bigg[ \prod_{c \in C(w)} p(c | w; \theta) \bigg]
\end{align}
$$

Here $C(w)$ is word $w$'s set of contexts.

One approach for parameterizing the $p(c | w; \theta)$ part of the skip-gram model is the classic softmax objective function:

$$
\begin{align}
p(c | w; \theta) &= \dfrac{exp(v_c \cdot v_w)}{ \sum_{c' \in C} exp(v_c' \cdot v_w)}
\end{align}
$$

Where:

- $v_c$ and $v_w$ are the vector representations for word $c$ and $w$ respectively
- $C$ is the set of all available contexts

While the objective function above can be computed, it is computationally expensive to do so due to the summation $\sum_{c' \in C} exp(v_c' \cdot v_w)$ since there can be thousands if not million of them. And this is where negative sampling comes in.

Instead of trying to estimate the probability of the word pair directly, we train a model to differentiate the target word from noise (negative sample). We can thus reduce the problem of predicting the correct word to a binary classification task, where the model tries to distinguish positive/genuine data from noise/negative samples.

We know for the word pair we generated from the data, we want to maximize its probability, i.e.

$$
\begin{align}
arg\underset{\theta}{max} \prod_{(w, c) \in D} p(D = 1 \big| c, w; \theta)
\end{align}
$$

Here:

- $D$ is the set of all word and context pairs we extract from the text
- $p(D = 1 \big| w, c)$ the probability that the word pair $(w, c)$ came from the corpus data

We then generate the set $D'$ of random (w, c) pairs, assuming they are all incorrect. The name "negative sampling" stems from the set $D'$ of randomly sampled negative examples. Note that when we pick one random word from the vocabulary, there is some tiny chance that the picked word is actually a valid context. If we consider the large number of vocabulary we have, we can argue that the probability is really really tiny, but a lot of the packages still do take care of removing these "accidents".

Since this is a binary classification loss, we can use logistic regression to minimize the negative log-likelihood, leading to the objective function:

\begin{align}
J_{\theta}
&= arg\underset{\theta}{max}
\prod_{(w, c) \in D} p(D = 1 \big| c, w; \theta) \prod_{(w, c) \in D'} p(D = 0 \big| c, w; \theta) \\
&= arg\underset{\theta}{max}
\prod_{(w, c) \in D} p(D = 1 \big| c, w; \theta) \prod_{(w, c) \in D'} \big(1 - p(D = 1 \big| c, w; \theta)\big) \\
&= arg\underset{\theta}{max}
\sum_{(w, c) \in D} log \big(p(D = 1 \big| c, w; \theta)\big) \sum_{(w, c) \in D'} log \big(1 - p(D = 1 \big| c, w; \theta)\big) \\
&= arg\underset{\theta}{max}
\sum_{(w, c) \in D} log \big( \dfrac{1}{1 + \text{exp}(-v_c \cdot v_w)} \big) \sum_{(w, c) \in D'} log \big (1 - \dfrac{1}{1 + \text{exp}(-v_c \cdot v_w)} \big) \\
&= arg\underset{\theta}{max}
\sum_{(w, c) \in D} log \big( \dfrac{1}{1 + \text{exp}(-v_c \cdot v_w)} \big) \sum_{(w, c) \in D'} log \big (\dfrac{1}{1 + \text{exp}(v_c \cdot v_w)} \big) \\
\end{align}

Note that there're various other approaches to approximate the expensive softmax objective function. Negative sampling is soley discussed here because it is fast and works really really well. If you would like to go deeper with this topic, the following link might be a good place to start. [Blog: On word embeddings - Part 2: Approximating the Softmax](http://ruder.io/word-embeddings-softmax/index.html)

### Selecting Negative Samples

One little detail that's missing from the description above is how do we select the negative samples.

The negative samples are chosen using the unigram distribution. Essentially, the probability of selecting a word as a negative sample is related to its frequency, with more frequent words being more likely to be selected as negative samples. Instead of using the raw frequency for $w_i$, $\text{freq}(w_i)$, in the original word2vec paper, each word is given a weight that's equal to it's frequency (word count) raised to the 3/4 power. The probability for selecting a word is just it's weight divided by the sum of weights for all words.

$$
\begin{align}
P(w_i) = \frac{ {\text{freq}(w_i)}^{3/4} }{\sum_{j=0}^{n} \left({\text{freq}(w_j)}^{3/4} \right) }
\end{align}
$$

This decision to raise the frequency to the 3/4 power appears to be empirical; as the author claims it outperformed other functions (e.g. just using unigram distribution).

Side note: The way this selection is implemented in the original word2vec C code is interesting. They have a large array with 100M elements (which they refer to as the unigram table). They fill this table with the index of each word in the vocabulary multiple times, and the number of times a word’s index appears in the table is given by $P(w_i) \times \text{table_size}$. Then, to actually select a negative sample, we just generate a random integer between 0 and 100M, and use the word at that index in the table. Since the higher probability words occur more times in the table, we're more likely to pick those.

## Subsampling Frequenct Words

Word2vec has two additional parameters for discarding some of the input words: words appearing less
than `min-count` times are not considered as either words or contexts, and in addition frequent words are down-sampled as defined by the `sample` parameter.

There are two potential issues with frequently appeared words like "the":

- When looking at word pairs that includes "the", e.g. ("fox", "the"), "the" doesn't tell us much about the meaning of "fox", since it appears in the context of pretty much every word.
- We will have more than enough samples of ("the", "the other word for the word pair") than we need to learn a good vector for "the".

Word2Vec implements a "subsampling" scheme to address this. For each word we encounter in our training text, there is a chance that we will discard it from the text. The probability that we cut the word is related to the word's frequency.

$$
\begin{align}
\text{probability of keeping the word } w_i
&= (\sqrt{\frac{z(w_i)}{0.001}} + 1) \cdot \frac{0.001}{z(w_i)}
\end{align}
$$

Where:

- $z(w_i)$ is the fraction of the total words in the corpus that are that word. For example, if the word "peanut" occurs 1,000 times in a 1 billion word corpus, then z("peanut") = 1E-6.
- There is also a parameter called `sample` which controls how much subsampling occurs, and the default value is 0.001. Smaller values of `sample` mean words are less likely to be kept

Here are some interesting observations of this subsampling function (again this is using the default sample value of 0.001).

- $\text{probability of keeping the word } w_i = 1$ (100% chance of being kept) when $z(w_i) <= 0.0026$. This means that only words which represent more than 0.26% of the total words will be subsampled
- $\text{probability of keeping the word } w_i = 0.5$ (50% chance of being kept) when $z(w_i) <= 0.00746$
- $\text{probability of keeping the word } w_i = 0.033$ (3.3% chance of being kept) when $z(w_i) = 1.0$. That is, if the corpus consisted entirely of word $w_i$, which of course is ridiculous

## Detecting Phrases

Word pair like "Boston Globe" (a newspaper) has a much different meaning than the individual words "Boston" and "Globe". So it makes sense to treat "Boston Globe", wherever it occurs in the text, as a single word with its own word vector representation.

The formula our phrase models will use to determine whether two tokens $A$ and $B$ constitute a phrase is:

$$
\begin{align}
\frac{count(A B) - count_{min}}{count(A) \cdot count(B)} \cdot N > threshold
\end{align}
$$

Where:

- $count(A)$ is the number of times token $A$ appears in the corpus
- $count(B)$ is the number of times token $B$ appears in the corpus
- $count(A B)$ is the number of times the tokens $A B$ appear in the corpus in this specific order
- $N$ is the total size of the corpus vocabulary
- $count_{min}$ is a user-defined parameter to ensure that accepted phrases occur a minimum number of times,
- $threshold$ is a user-defined parameter to control how strong of a relationship between two tokens the model requires before accepting them as a phrase

As we can infer the formula is designed to make phrases out of words which occur together often relative to the number of individual occurrences. And a higher threshold value will favors phrases made of infrequent words in order to avoid making phrases out of common words like "and the" or "this is".

# Implementation

The following code chunks 1) basic preprocess the raw text. 2) trains a Phrase model to glue words that commonly appear next to each other into bigrams. 3) trains the Word2vec model.

There also a pure python script [here](https://github.com/ethen8181/machine-learning/tree/master/deep_learning/word2vec/word2vec_workflow.py) if you're interested.

```python
import os
import re
from string import punctuation
from nltk.corpus import stopwords
from gensim.models import Phrases
from gensim.models import Word2Vec
from gensim.models.phrases import Phraser
from gensim.models.word2vec import LineSentence
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS
```

```python
def export_unigrams(unigram_path, texts, stop_words):
    """
    Preprocessed the raw text and export it to a .txt file,
    where each line is one document, for what sort of preprocessing
    is done, please refer to the `normalize_text` function

    Parameters
    ----------
    unigram_path : str
        output file path of the preprocessed unigram text.

    texts : iterable
        iterable can be simply a list, but for larger corpora,
        consider an iterable that streams the sentences directly from
        disk/network using Gensim's Linsentence or something along
        those line.

    stop_words : set
        stopword set that will be excluded from the corpus.
    """
    with open(unigram_path, 'w', encoding='utf_8') as f:
        for text in texts:
            cleaned_text = normalize_text(text, stop_words)
            f.write(cleaned_text + '\n')

def normalize_text(text, stop_words):
    # remove special characters\whitespaces
    text = re.sub(r'[^a-zA-Z\s]', '', text, re.I | re.A)

    # lower case & tokenize text
    tokens = re.split(r'\s+', text.lower().strip())

    # filter stopwords out of text &
    # re-create text from filtered tokens
    cleaned_text = ' '.join(token for token in tokens if token not in stop_words)
    return cleaned_text
```

```python
# a set of stopwords built-in to various packages
# we can always expand this set for the
# problem that we are working on, here we also included
# python built-in string punctuation mark
STOPWORDS = set(stopwords.words('english')) | set(punctuation) | set(ENGLISH_STOP_WORDS)

# create a directory called 'model' to
# store all outputs in later section
MODEL_DIR = 'model'
if not os.path.isdir(MODEL_DIR):
    os.mkdir(MODEL_DIR)

UNIGRAM_PATH = os.path.join(MODEL_DIR, 'unigram.txt')
if not os.path.exists(UNIGRAM_PATH):
    start = time()
    export_unigrams(UNIGRAM_PATH, texts=newsgroups_train.data, stop_words=STOPWORDS)
    elapse = time() - start
    print('text preprocessing, elapse', elapse)
```

    text preprocessing, elapse 1.8021948337554932

```python
PHRASE_MODEL_CHECKPOINT = os.path.join(MODEL_DIR, 'phrase_model')
if os.path.exists(PHRASE_MODEL_CHECKPOINT):
    phrase_model = Phrases.load(PHRASE_MODEL_CHECKPOINT)
else:
    # use LineSentence to stream text as oppose to
    # loading it all into memory
    unigram_sentences = LineSentence(UNIGRAM_PATH)
    start = time()
    phrase_model = Phrases(unigram_sentences)
    elapse = time() - start
    print('training phrase model, elapse', elapse)
    phrase_model.save(PHRASE_MODEL_CHECKPOINT)
```

    training phrase model, elapse 3.066694736480713

```python
def export_bigrams(unigram_path, bigram_path, phrase_model):
    """
    Use the learned phrase model to create (potential) bigrams,
    and output the text that contains bigrams to disk

    Parameters
    ----------
    unigram_path : str
        input file path of the preprocessed unigram text

    bigram_path : str
        output file path of the transformed bigram text

    phrase_model : gensim's Phrase model object

    References
    ----------
    Gensim Phrase Detection
    - https://radimrehurek.com/gensim/models/phrases.html
    """

    # after training the Phrase model, create a performant
    # Phraser object to transform any sentence (list of
    # token strings) and glue unigrams together into bigrams
    phraser = Phraser(phrase_model)
    with open(bigram_path, 'w') as fout, open(unigram_path) as fin:
        for text in fin:
            unigram = text.split()
            bigram = phraser[unigram]
            bigram_sentence = ' '.join(bigram)
            fout.write(bigram_sentence + '\n')
```

```python
BIGRAM_PATH = os.path.join(MODEL_DIR, 'bigram.txt')
if not os.path.exists(BIGRAM_PATH):
    start = time()
    export_bigrams(UNIGRAM_PATH, BIGRAM_PATH, phrase_model)
    elapse = time() - start
    print('converting words to phrases, elapse', elapse)
```

    converting words to phrases, elapse 14.514541864395142

The next two code chunks is a slight digression from the overall process, this is simply used assess the amount of memory that is potentially saved by doing these preprocessing, i.e. loading all the raw text
and preprocessed text into memory and comparing the two.

```python
import sys

def get_size(obj, seen=None):
    """
    Recursively finds size of objects, shamelessly "borrowed"
    from link listed in reference

    References
    ----------
    https://goshippo.com/blog/measure-real-size-any-python-object/
    """
    size = sys.getsizeof(obj)
    if seen is None:
        seen = set()

    obj_id = id(obj)
    if obj_id in seen:
        return 0

    # Important mark as seen *before* entering recursion to gracefully handle
    # self-referential objects
    seen.add(obj_id)
    if isinstance(obj, dict):
        size += sum([get_size(v, seen) for v in obj.values()])
        size += sum([get_size(k, seen) for k in obj.keys()])
    elif hasattr(obj, '__dict__'):
        size += get_size(obj.__dict__, seen)
    elif hasattr(obj, '__iter__') and not isinstance(obj, (str, bytes, bytearray)):
        size += sum([get_size(i, seen) for i in obj])

    return size
```

```python
X = []
with open(BIGRAM_PATH) as f:
    for line in f:
        X.append(line)

original_size = get_size(newsgroups_train.data)
preprocessed_size = get_size(X)
ratio = original_size / preprocessed_size
print('original size uses {} times the amount of memory'.format(ratio))
del X
```

    original size uses 1.579115410629266 times the amount of memory

The Word2vec also accepts several key parameters that affect both training speed and quality. Hopefully these will now make more sense after we covered some theory of the algorithm.

- `iter`: Number of iterations to train the model.
- `min_count`: For pruning the internal dictionary. Words that appear only once or twice in a billion-word corpus are probably uninteresting typos and garbage. In addition, there's not enough data to make any meaningful training on those words, so it's best to ignore them. A reasonable value for min_count is between 0-100, depending on the size of the dataset.
- `size`: Refers to the hidden layers size, i.e. dimension of our word vectors. Bigger size values require more training data, but can lead to better (more accurate) models. Reasonable values are in the tens to hundreds
- `workers`: Number of cores/threads used for training.
- `window`: Only terms that occur within a window-neighbourhood of a term in a sentence are associated with it during training. The default value is 4. Unless your text contains big sentences, it might be safe to leave it at that.
- `sg`: This defines the algorithm. If equal to 1, the skip-gram technique is used. Else, the CBoW method is employed. The word2vec framework we introduced here is skip-gram, where we're using the target word to predict the context word, where CBow is simply flipping it the other way around, i.e. using the context word to predict the target word. Based on the original author's [paper](https://arxiv.org/abs/1310.4546), Skip-gram is reported to be a better framework.

You can refer to the full list of parameters [here](http://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec).

```python
WORD2VEC_CHECKPOINT = os.path.join(MODEL_DIR, 'word2vec')
if os.path.exists(WORD2VEC_CHECKPOINT):
    word2vec = Word2Vec.load(WORD2VEC_CHECKPOINT)
else:
    # the old way of streaming data into gensim's Word2Vec was to
    # leverage the LineSentence generator, i.e.
    # sentences = LineSentence(BIGRAM_PATH)
    # word2vec = Word2Vec(sentences, workers=cpu_count())
    # the latest and faster way is to directly pass the path to
    # the corpus_file argument or what they call file based training mode
    start = time()
    word2vec = Word2Vec(corpus_file=BIGRAM_PATH, workers=cpu_count())
    elapse = time() - start
    print('training word2vec, elapse', elapse)
    word2vec.save(WORD2VEC_CHECKPOINT)
```

    training word2vec, elapse 5.049929141998291

After that we can print out similar words to a targeted word that we've specified and use our human judgement of whether the embedding that was learned makes sense. The next code chunk is simply satisfying a personal interest in how the `most_similar` function is implemented in gensim.

```python
# once we’re finished training the model (i.e. no more updates, only querying)
# we can store the word vectors and delete the model to trim unneeded model memory
vocab = word2vec.wv.vocab
word_vectors = word2vec.wv.vectors
index2word = word2vec.wv.index2word
del word2vec
```

```python
from sklearn.preprocessing import normalize

def most_similar(word_vectors, vocab, index2word,
                 positive=None, negative=None, topn=5):
    """
    No frill version of the function `word2vec.wv.most_similar`;
    Find the top-N most similar words. Positive words contribute
    positively towards the similarity, negative words negatively.

    Parameters
    ----------
    word_vectors : 2d ndarray
        the learned word vectors.

    vocab : word2vec.wv.vocab
        dictionary like object where the key is the word
        and the value has a .index attribute that allows
        us to look up the index for a given word.

    index2word : word2vec.wv.index2word
        list like object that serves as the looking up the word
        of a given index.

    positive/negative : list, default None
        list of positive or negative words to look for similar words,
        positive words will get assign a +1 weight and negative words
        will get assign a -1 weight when doing the vector addition.

    topn : int
        top-n similar words to find.
    """
    # normalize word vectors up front makes the cosine distance
    # calculation later easier
    normed_vectors = normalize(word_vectors)

    # assign weight to positive and negative words
    if positive is None:
        positive = []
    else:
        positive = [(word, 1.0) if isinstance(word, str) else word
                    for word in positive]
    if negative is None:
        negative = []
    else:
        negative = [(word, -1.0) if isinstance(word, str) else word
                    for word in negative]

    # compute the weighted average of all the query words
    queries = []
    all_word_index = set()
    for word, weight in positive + negative:
        # gensim's Word2vec word2vec.wv.vocab.index
        # stores the index of a given word
        word_index = vocab[word].index
        word_vector = normed_vectors[word_index]
        queries.append(weight * word_vector)
        all_word_index.add(word_index)

    if not queries:
        raise ValueError('cannot compute similarity with no input')

    # find topn most similar words measured by cosine distance
    query_vector = np.mean(queries, axis=0)
    query_vector /= np.sqrt(np.sum(query_vector ** 2))

    # cosine similarity between the query vector and all the existing word vectors
    scores = np.dot(normed_vectors, query_vector)

    actual_len = topn + len(all_word_index)
    sorted_index = np.argpartition(scores, -actual_len)[-actual_len:]
    best = sorted_index[np.argsort(scores[sorted_index])[::-1]]

    result = [(index2word[index], scores[index])
              for index in best if index not in all_word_index]
    return result[:topn]
```

```python
word = 'computer'
most_similar(word_vectors, vocab, index2word, positive = [word])
```

    [('software', 0.9712179),
     ('pc', 0.9687412),
     ('box', 0.96866494),
     ('modem', 0.9682968),
     ('network', 0.9651587)]

# Final Thoughts

## Hyperparameters

The most crucial decisions that affect the performance are the choice of the model
architecture, the size of the vectors, the subsampling rate, and the size of the training window. Because Word2vec training is an unsupervised task, there's no good way to objectively evaluate the result, it all depends on the end application.

## Applications

Word2Vec and the concept of word embeddings originated in the domain of NLP, however the idea of words in the context of a sentence or a surrounding word window can be generalized to other problem domain dealing with sequences or sets of related data points. For example:

- A direct application of Word2Vec to a classical engineering task was recently presented by Spotify. They abstracted the ideas behind Word2Vec to apply them not simply to words in sentences but to any object in any sequence, in this case to songs in a playlist. Songs are treated as words and other songs in a playlist as their surrounding context, depending on whether the playlists in question were genre specific the vocabulary encompassed a number of songs from that collection. Now in order to recommend songs to a user one merely has to examine a neighborhood of the 'song embeddings' of songs the user already likes
- Similarly, we could recommend a user who to connect to in a social network setting by examining the graph of relationships, where the nodes represent words and a path through the graph represents a sentence, then nodes that occur in the context of similar other nodes, would be close together in the vector space.

These examples show that the general applicability of Word2Vec based algorithms is very rich and that it behooves practitioners to examine their problem domain with respect to sequences of objects that occur in some meaningful context.

## Resources

A lot of the non-mathematical notes were taken from Chris McCormick's blog post. If you wish to learn more about the topic, the following link also contains different resources that Chris has curated (including a commented version of word2vec's original C code, derivation of word2vec's gradient update, etc.). [Blog: Word2Vec Resources](http://mccormickml.com/2016/04/27/word2vec-resources/)

If you would like to practice using a deep learning framework such as Tensorflow to implement the word2vec model. The following resources are good places to start, the reason that its not included in this documentation is because it was an order of magnitude slower than Gensim's Word2vec and the result weren't as good as well.

- [Note: CS 20SI Lecture note 4: How to structure your model in TensorFlow](http://web.stanford.edu/class/cs20si/lectures/notes_04.pdf)
- [Blog: Word2Vec word embedding tutorial in Python and TensorFlow](http://adventuresinmachinelearning.com/word2vec-tutorial-tensorflow/)

# Reference

- [Blog: Word2vec Tutorial](http://rare-technologies.com/word2vec-tutorial/)
- [Blog: Word2Vec Tutorial - The Skip-Gram Model](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/)
- [Blog: Word2Vec Tutorial Part 2 - Negative Sampling](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/)
- [Blog: Natural Language Processing Made Easy – using SpaCy (in Python)](https://www.analyticsvidhya.com/blog/2017/04/natural-language-processing-made-easy-using-spacy-%E2%80%8Bin-python/)
- [Paper: Yoav Goldberg, Omer Levy (2014) word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://arxiv.org/abs/1402.3722)
- [Paper: Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean (2013) Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546)
- [Youtube: PyData DC 2016: Modern NLP in Python](https://www.youtube.com/watch?v=6zm9NC9uRkk)
- [Jupyter Notebook: PyData DC 2016: Modern NLP in Python](http://nbviewer.jupyter.org/github/skipgram/modern-nlp-in-python/blob/master/executable/Modern_NLP_in_Python.ipynb)
- [Jupyter Notebook: Gensim Docs - 2Vec File-based Training: API Tutorial](https://nbviewer.jupyter.org/github/RaRe-Technologies/gensim/blob/develop/docs/notebooks/Any2Vec_Filebased.ipynb)


# Quantum and classical neural networks for Max-Cut

- toc: true
- badges: false
- comments: false
- categories: [gnn, qnn]
- author: Sergei Tsimbalist
- use_math: true

```python
#hide

import tensorflow as tf
import tensorflow_quantum as tfq
import cirq
import sympy
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit

import warnings
warnings.filterwarnings("ignore")

import networkx as nx

import pandas as pd

from plotnine import *
from plotnine.data import *

from qutip import Bloch
```

# Introduction

Both deep learning and quantum computing have been making the headlines in the recent past - these two topics are seen as "the big things" in computer science. But while deep learning has already delivered a lot, and the question here is: "how far can it go?", the field of quantum computing is still waiting for better hardware - there is a lot more theory in the field describing algorithms that could do cool things that are otherwise difficult for classical computers, but the quantum hardware isn't there yet.

At the intersection of machine learning and quantum computing, there exists the field of quantum machine learning, which encompasses two main goals: analyzing quantum data (physics experiments, etc...) and implementing machine learning on quantum computers. Recently, there has been as increasing research into the quantum sibling of neural networks - parameterized quantum cirquits, which are also called quantum neural networks. Unfortunately, at the present moment it does not seem that quantum neural networks will be of much use for classical deep learning problems - their main application is for data that could have some quantum properties. One of the very few exceptions to this that I have found is the maximum cut graph problem.

## MaxCut problem

The problem of maximum cut can be stated as follows: we want to find set of nodes $S$ such that the number or the weight of all edges coming from $S$ to $\overline{S}$, which specifies the cut $S:\overline{S}$, is maximal.

Below is a random cut of a 10-node graph: yellow and purple nodes belong to different cut partitions, red edges define the cut.

```python
#collapse-hide

def draw_cut(graph, cut):
    ncolors = []
    for n in graph.nodes:
        if n in cut:
            ncolors.append(0)
        else:
            ncolors.append(1)
    ecolors = []
    for e in graph.edges:
        if (e[0] in cut and e[1] not in cut) or (e[0] not in cut and e[1] in cut):
            ecolors.append('red')
        else:
            ecolors.append('black')
    fig, ax = plt.subplots(1, 1, figsize=(8, 6));
    nx.draw_networkx(graph, ax=ax, node_color=ncolors, edge_color=ecolors, with_labels=False)

ex_graph = nx.erdos_renyi_graph(n=10,p=0.5)
draw_cut(ex_graph, [0,3,7,9])
```

This problem is interesting for this study as it is NP-hard - there is exact no polynomial time algorithm for it. The practical applications where this problem is used inlude network design and theoretical physics. As it turns out, this problem can be also stated in the terms of physics - the so called Ising model - which then can be neatly mapped to a quantum circuit.

## Graph Neural Networks

Even at the first glance, the naive apprach from supervised deep learning - using graph matrix as input and maximum cuts as output, should not work: firstly, there can be many maximum-cuts, finding them for large graphs is a problem on itself, and a simple neural network will not exploit the relational properties between the nodes, which is important for a graph.

In order to exploit the relational structure of a graph, different methods are known: one can employ techniques from convolutional neural networks or recurrent neural networks. Additionally, for each node (or edge) we want to create some informative embedding.

The image below demonstrates an idea behind graph (convolutional) networks - we look at each node and their neighbours using shared parameters, similarly to how convolution filters works in CNNs.

If you want to know more, you can watch this video for the introduction to the topic.
> youtube: https://youtu.be/zCEYiCxrL_0

## Quantum Computing

Below is an image of a Block sphere. If we take the upwards direction as qubit zero (or ket zero $|0\rangle$, as quantum scientists call it) and downwards as ket one $|1\rangle$, then a qubit can be anything on the surphace of this unit sphere, for example the green arrow represents a qubit that is in some superposition of $|0\rangle$ and $|1\rangle$. However, we cannot access the state of a qubit directly, instead we would need to perform a measurement operation, that would give us either zero or one in a classical bit.

```python
#collapse-hide

bloch_sphere = Bloch()
bloch_sphere.add_vectors([-1/np.sqrt(3),1/np.sqrt(3),1/np.sqrt(3)])
bloch_sphere.show()
bloch_sphere.clear()
```

A quantum circuit is normally represented as this: there are $n$ qubits, in this case 3, each having their own line, you can apply gates on one or more qubits. Usually at the end of the circuit you perform a measurement and get 3 classical bits of information. Running the circuit multiple times and averaring you can get an estimate of the output value, which is as a rule probabilistic.

The example below is a random 3 qubit circuit composed of:
* $Rx$ - rotation about the x axis (check Bloch sphere above)
* $Ry$ - rotation about the y axis
* $H$ - Hadamard - a superposition gate
* $cX$ gate - controlled NOT gate

```python
#collapse-hide

q0, q1, q2 = cirq.GridQubit.rect(1, 3)

circuit = cirq.Circuit(
    (
        cirq.rx(0.5).on(q0))**0.5,
    cirq.ry(0.7).on(q1),
    cirq.CNOT(control=q0, target=q1),
    cirq.H(q2),cirq.CNOT(control=q1, target=q2))

SVGCircuit(circuit)
```

### Quantum Neural Networks

As mentioned previously, a quantum neural network is nothing more than a parameterized quantum circuit.

The image below is of the same quantum circuit, but now it is parameterized by two variables $a, b$ - in essence this is a quantum neural network, you could learn these parameters with respect to some desired output.

```python
#collapse-hide

q0, q1, q2 = cirq.GridQubit.rect(1, 3)
a, b = sympy.symbols('a b')

circuit = cirq.Circuit(
    (
        cirq.rx(a).on(q0)),
    cirq.ry(b).on(q1),
    cirq.CNOT(control=q0, target=q1),
    cirq.H(q2),cirq.CNOT(control=q1, target=q2))

SVGCircuit(circuit)
```

To get a popular introduction to the topic of Quantum Machine Learning, you can watch this video from Tensorflow Dev Summit:

> youtube: https://youtu.be/-o9AhIz1uvo

In general, quantum and classical neural networks can be combined in many different ways, depending on our goals. This scheme from Google demonstrates a hybrid quantum-classical neural network and its basic functionality.

Quantum neural networks used here will be explored below, under the Methods section.

# Methods

## Graph NN for MaxCut

For the graph neural network I will use implementation provided by this paper Experimental performance of graph neural networks on random instances of max-cut, as this was actually the only working implementation that I have found. You can consult the paper about the details of how graphs are encoded for the input and what the loss funcion for this problem looks like.

## Quantum NN for MaxCut

For the quantum model I will the use implementation provided in Tensorflow Quantum, check chapter IV.C for details. In short, the graph for which we want to find the max-cut is mapped to a parameterized quantum circuit, then the cost function is also represented as a quantum circuit. When appended together this parameterized circuit will sort of describe the energy level of the system, given the parameters, and we will want to minimize the energy.

```python
#hide_input

import matplotlib.image as mpimg
image = mpimg.imread("qcirq.png")
print('Parameterized quantum circuit example for MaxCut on 5 node graph')
plt.figure(figsize = (15,2))
plt.imshow(image)
plt.axis('off')
plt.show()
```

## Meta learning QNN

Main drawback of the previous model is that we find parameters only for one specific graph. For this reason the same paper descibes in chapter V.A a meta-learning framework for the QNN. The idea is to use a recurrent neural network to learn the parameter updates based on previous parameter values and the loss. This model should be able to generalize to different graphs and even different graph sizes.

In general, this model is descibed as:

$\theta_{i+1}, h_{i+1} = RNN(\theta_{i}, h_{i}, y_{i}) $

$y_{i} = QNN(\theta_{i}) $

where $\theta_{i}$ is a set of parameters for the QNN (vector with just 2 paramters in this case), $h_{i}$ is the hidden state of the RNN, and $y_i$ is the energy level, or 'loss' produced by the QNN. Number of RNN timesteps will be 5.

## Experiments

Relatively small graphs will be used (15 nodes), because as it turned out my hardware cannot handle large quantum tensorflow models. On the positive side, during test phase its possible to search over all possible cuts of which there are $2^n$ ($n$ number of nodes) to find the optimum and report the achieved ratio of that optimum. The accuracy of the model will then be averaged ratio of optimum over all test samples.

### Generating random graphs

Graphs for training and testing will be generated using an Erdos-Renyi random graph model with different parameters, to see if the graph structure has any effect on the performance. The paramter $p$ of Erodos-Renyi model descibes the probability of an edge being included in the graph.

```python
#hide-input

graph = nx.erdos_renyi_graph(n=10,p=0.7)
print('Erdos-renyi random graph with p=0.7')
fig, ax = plt.subplots(1, 1, figsize=(8, 6));
nx.draw_networkx(graph, ax=ax)
```

```python
#hide-input

graph = nx.erdos_renyi_graph(n=10,p=0.3)
print('Erdos-renyi random graph with p=0.3')
fig, ax = plt.subplots(1, 1, figsize=(8, 6));
nx.draw_networkx(graph, ax=ax)
```

# Results

## Accuracy of Max-Cut prediction

```python
#hide_input

df = pd.read_csv('exp1.csv')

p = ggplot(df, aes( y='acc', x='graph', fill='model'))+ geom_boxplot() + ylim([0,1]) + facet_wrap('model', nrow=1) + scale_x_discrete(labels=['p=0.3','p=0.5','p=0.7'])
p
```

Interestingly, when I first run the experiments I was a little bit surprised that all models performed better for graphs with more edges. However, evaluating the expectation for a random cut has shown where this bias comes from.

Thing to notice here are:
* The QNN has only 2 parameters and the RNN has 7735, whereas GNN has 80000
* GNN outperforms both alternatives
* There appears to be little difference between accruacy of standalone QNN and when used in the meta learning framework with RNN
* Results are better than random model

## Dependence of accuracy on the training size

On of the things I noticed during the experiments is that the RNN-QNN does not need many training samples to perform well, so I also decided to test the training set size vs the accuracy of prediction.

```python
#hide_input

df = pd.read_csv('exp2.csv')

p = ggplot(df, aes( y='acc', x='factor(tr_size)', fill='model'))+ geom_boxplot() + ylim([0,1]) + xlab('train size') + facet_grid('model ~ graph')
p
```

Meta learning model appears to benefit little from additional training samples, compared to the graph neural network. I think its possible to speculate, that this is because the QNN itself encodes a lot of knowledge about the problem, but the question is then why does it underperform compared to the GNN.

# Conclusion

These results should be taken in the context that practical quantum computers are still far away and all experiments of this sort are done in simulators, so the real cost-benefit analysis between the methods is not possible at the moment. While these results show that the quantum model does not appear better, perhaps further work in the quantum machine learning will yield different results.

Publication of this post has benefitted from the scholarship provided by IT academy http://studyitin.ee

# References

https://arxiv.org/abs/1908.05767

https://arxiv.org/pdf/2003.02989.pdf

https://github.com/tensorflow/quantum/blob/research/metalearning_qaoa/metalearning_qaoa.ipynb

https://github.com/tensorflow/quantum/blob/research/qaoa/qaoa.ipynb

https://github.com/ElainaYao/maxCut