### Project 3: Tennis

```python
from IPython.display import HTML
style = "<style>.pseudo-code ul { list-style-type: none;}</style>"
HTML(style)
```

## 1: Learning algorithm

### Description of algorithm

Deep learning and reinforcement learning have recently been combined in different ways, including variants of the "Deep Q-Network" (DQN) as used in the Navigation project. While DQN works well with high-dimensional _observation spaces_, it can only handle discrete and low dimensional _action spaces_.

In physical control tasks such as this one, we have continuous and high dimensional action spaces and algothithms like _deep deterministic policy gradient_ (DDPG) are better suited. In brief, DDPG is a "model-free, off-policy actor-critic algorithm using deep function approximators" ([Lillicrap et al., 2015](https://arxiv.org/abs/1509.02971)).

For this project, the DDPG implementation from Project 2: Continuous Control has been reused without modification - apart from minor configuration changes, including hyper parameters values, number of episodes, and time steps per episode. For reference, the DDPG algorithm is summarized below:

* Randomly initialize critic network $Q(s,a|\theta^Q)$ and actor $\mu(s|\theta^\mu)$ with weights $\theta^Q$ and $\theta^\mu.$
* Initialize target network $Q'$ and $\mu'$ with weights $\theta^{Q'} \leftarrow \theta^Q, \theta^{\mu'} \leftarrow \theta^\mu$
* Initialize replay buffer $R$

   **for** episode = 1, M **do**

   * Initialize a random process $\mathcal{N}$ for action exploration

   * Receive initial observation state $s_1$

   * **for** t = 1, T **do**
     * Select action $a_t = \mu(s_t|\theta^\mu) + \mathcal{N}_t$ according to the current policy and exploration noise

     * Execute action $a_t$ and observe reward $r_t$ and observe new state $s_{t+1}$

     * Store transition $(s_t, a_t, r_t, s_{t+1})$ in $R$

     * Sample a random minibatch of $\mathcal{N}$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$

     * Set $y_i = r_i + \gamma Q'(s_{i+1}, \mu'(s_{i+1}|\theta^{\mu'})|\theta^{Q'})$

     * Update cretic by minimizing the loss: $L = \frac{1}{N} \sum_i(y_i - Q(s_i, a_i|\theta^Q))^2$

     * Update the actor policy using the sampled policy gradient:

     * \begin{equation} \nabla_{\theta^\mu} J \approx \frac{1}{N} \sum_i \nabla_a Q(s,a|\theta^Q)|_{s=s_i, a=\mu(s_i)} \nabla_{\theta^\mu}\mu(s|\theta^\mu)|_{s_i}  \end{equation}

     * Update the target networks:

   \begin{align}
   \theta^{Q'} &\leftarrow \tau\theta^Q + (t-\tau) \theta^{Q'}\\
   \theta^{\mu'} &\leftarrow \tau\theta^\mu + (t-\tau)\theta^{\mu'}
   \end{align}

   * **end for**

  **end for**

### Chosen Hyperparameters

```
BUFFER_SIZE = int(1e5)  # replay buffer size
BATCH_SIZE = 128        # minibatch size
GAMMA = 0.99            # discount factor
TAU = 1e-3              # for soft update of target parameters
LR_ACTOR = 2e-4         # learning rate of the actor
LR_CRITIC = 3e-4        # learning rate of the critic
WEIGHT_DECAY = 0        # L2 weight decay
NUM_AGENTS = 2          # number of agents
UPDATE_RATE = 20        # number of time steps between updates
NUM_UPDATES = 10        # how many times train the agens on each update
EPSILON = 1.0           # initial noise magnitude
EPSILON_DECAY = 0.01    # noise decay per episode
```

### Neural Networks

Two similarly structured networks are used for the _actor_ and the _critic_. For this problem ``state_size = 33``
 and ``action_size = 4``.

#### Actor
A basic three-layered feed-forward network with fully connected layers is used for the actor:

* BatchNorm 1
* Layer 1: (state_size, 128)
* ReLU 1
* BatchNorm 2
* Layer 2: (128, 128)
* ReLU 2
* BatchNorm 3
* Layer 3: (128, action_size)
* Tanh

#### Critic
A slightly wider three-layered feed-forward network with fully connected layers is used for the critic:

* Layer 1: (state_size, 256)
* ReLU 1
* BatchNorm
* Layer 2: (cat(128, action_size), 128)
* ReLU 2
* Layer 3: (128, action_size)

## 2: Plot of Rewards

This setup was able to solve the problem in 1507 episodes. A plot of (average) score per episode is illustraded below:

## 3: Ideas for Future Work

While the problem is solved reasonably quickly with DDPG, the work could possibly be improved in several ways including:

* Search systematically for optimal hyperparameters and neural network size/shape.
* Try other algorithms, including Truncated Natural Policy Gradient (TNPG) and Trust Region Policy Optimization (TRPO) which have been shown to achieve better performance in some tasks while being more stable.


# Section 7: Capstone Project

---

# Table of Contents

* [System Requirements (Part 1)](#System-Requirements)
  * [Model Introduction](#Model-Introduction)
  * [Requirements Analysis](#Requirements-Analysis)
  * [Visual System Mapping: Causal Loop Diagram](#Visual-System-Mapping:-Causal-Loop-Diagram)
  * [Visual System Mapping: Stock & Flow Diagram](#Visual-System-Mapping:-Stock-&-Flow-Diagram)
  * [Mathematical Specification](#Mathematical-Specification)
* [System Design (Part 2)](#System-Design)
  * [Differential Specification](#Differential-Specification)
  * [cadCAD Standard Notebook Layout](#cadCAD-Standard-Notebook-Layout)
    0. [Dependencies](#0.-Dependencies)
    1. [State Variables](#1.-State-Variables)
    2. [System Parameters](#2.-System-Parameters)
    3. [Policy Functions](#3.-Policy-Functions)
    4. [State Update Functions](#4.-State-Update-Functions)
    5. [Partial State Update Blocks](#5.-Partial-State-Update-Blocks)
    6. [Configuration](#6.-Configuration)
    7. [Execution](#7.-Execution)
    8. [Simulation Output Preparation](#8.-Simulation-Output-Preparation)
* [System Validation (Part 3)](#System-Validation)
  * [What-if Matrix](#What-if-Matrix)
  * [System Analysis](#System-Analysis)

---

# System Requirements

## Model Introduction

Project Anthropocene is a model that enables the insightful analysis of the impact of carbon dioxide (CO2) on the Earth's temperature.

## Requirements Analysis

[Link to System Analysis](#System-Analysis)

### Questions

**Planned analysis:** How does the Earth's temperature evolve over the next 100 years under various assumptions regarding CO2 emissions?

1. How will the __Earth's average temperature__ and the __rate of annual temperature change__ develop over the next 100 years, if we keep CO2 emissions __unchanged__ at today’s annual emission levels vs. a __doubling__ of today’s emission levels.
2. How will the __Earth's average temperature__ and the __rate of annual temperature change__ develop over the next 100 years if we are able to reduce annual CO2 emissions to __zero__ after a given number of years?

## Visual System Mapping: Causal Loop Diagram

The overall __relationships__ in the model are the following:
* The __Earth's temperature is determined by what's called radiation balance__, i.e. how much radiation comes in via the Sun, minus how much is dissipating into space. If this balance is positive, heat accumulates, and the Earth warms up; if it is negative, the Earth cools down.
* The __radiation balance__ is driven by the Sun's radiation, which tends to make the Earth hotter, and the Earth's radiation, which makes heat dissipate and the planet colder.
* The __radiation balance is influenced by the well-known greenhouse effect__, i.e. the stronger the greenhouse effect, the more radiation from Earth gets trapped in the atmosphere unable to dissipate into space and the higher the radiation balance. Quick primer on the greenhouse effect: https://en.wikipedia.org/wiki/Greenhouse_effect
* __CO2__ contributes strongly to the greenhouse effect.

## Visual System Mapping: Stock & Flow Diagram

## Mathematical Specification

The Anthropocene system is an IVP (initial value problem) which is described by the following equations:

\begin{align}
\tag{1}
dCO_2(t) = \begin{cases}
    \mathcal{N}(\mu,  \sigma) & \forall t \in [0, t_w] \\
     \mathcal{N}(0,  \sigma) & \forall t \in [t_w, \infty]
            \end{cases}
\end{align}

\begin{align}
\tag{2}
\alpha(t) = 1 - e^{-\beta * CO_2(t)}
\end{align}

\begin{align}
\tag{3}
Y(t) = \alpha(t) Z(t)
\end{align}

\begin{align}
\tag{4}
Z(t) = K T(t)^4
\end{align}

\begin{align}
\tag{5}
dT(t) = \gamma(a + Y(t) - Z(t))
\end{align}

Where $\mathcal{N}$ represents the normal distribution with mean $\mu$ and standard deviation $\sigma$. For each timestep $t$ we have the $CO_2(t)$, $\alpha(t)$, $Y(t)$, $T(t)$ and $Z(t)$ being the atmospheric $CO_2$ concentration, the atmosphere reflectance, the reflected radiation, the surface temperature and the outgoing radiation respectively. Also, we have the $\beta$, $\gamma$, $a$, $K$, $t_w$ constants as being a $CO_2$ to reflectance conversion factor, a radiation to temperature conversion factor, the Sun's yearly radiance, a constant for the blackbody effect and the year where emissions are stopped.

The system is tightly coupled and is both non-linear and stochastic, which can make mathematical treatment difficult, and as such, the characterization of it will be made easier through an experimental approach and computational simulations, which is exactly what **cadCAD** enables us to do.

# System Design

## Differential Specification

## cadCAD Standard Notebook Layout

### 0. Dependencies

```python
import pandas as pd
import numpy as np
from random import normalvariate
import plotly.express as px

from cadCAD.configuration.utils import config_sim
from cadCAD.configuration import Experiment
from cadCAD.engine import ExecutionContext, Executor
```

### 1. State Variables

The states we are interested in, their state variables and their initial values are:

* The __atmosphere's CO2 concentration__ in parts per million (ppm): `co2`, initial value 400
* The __earth's surface temperature__ in Kelvin (K): `temperature`, initial value 290

```python
initial_state = {
    'co2': 400,
    'temperature': 290
}
```

### 2. System Parameters

**The system parameters we need to define are:**

* The sun radiation: `sun_radiation` with value `1361`
* A constant representing the relationship between temperature and radiation: `temperature_constant` with value `1e-4`
* A constant representing CO2 impact on the radiation balance via the greenhouse effect: `co2_reflectance_factor` with value `1e-3`
* A unit conversion constant that relates how much gigatons of CO2 we need to have an additional part per million unit in the atmosphere's concentration: `co2_gigatons_to_ppm` with value `1.2e-1`
* The standard deviation for the stochastic process generating the yearly CO2 concentration: `co2_stdev` with value `40` ppm
* A constant representing how much heat dissipitates into space: `heat_dissipation_constant` with value `2075`

**There are two parameters which we want to sweep, which are:**

* A parameter which represents the annual CO2 emissions in units of billion tons, which is the `co2_annual_emissions`. Let's sweep three values for it: `0`, `40` and `80`. The first value simulates a scenario where we stop all emissions at once, while using `40` means no additional emissions beyond what we already emmit every year, and `80` means that we are going to double our emissions.
* The `year_of_the_wakening`, which is the number of years that must pass before we set the `co2_annual_emissions` to zero. Let's sweep four values for it: `0`, `10`, `50` and `100`.

```python
system_params = {
    'sun_radiation': [1361],
    'temperature_constant': [1e-4],
    'co2_reflectance_factor': [1e-3],
    'co2_gigatons_to_ppm': [1.2e-1],
    'co2_stdev': [40],
    'heat_dissipation_constant': [2075],
    'co2_annual_emissions': [40, 80, 40, 80, 40, 80, 40, 80],
    'year_of_the_wakening': [0, 0, 10, 10, 50, 50, 100, 100]
}
```

```python
assert 1e10 == 1*10**10
```

### 3. Policy Functions

```python
def p_co2_emissions(params,
                    subbstep,
                    state_history,
                    previous_state):
    # Parameters & variables
    mean = params['co2_annual_emissions']
    std = params['co2_stdev']
    conversion_factor = params['co2_gigatons_to_ppm']
    t_w = params['year_of_the_wakening']
    t = previous_state['timestep']

    # Logic
    if t > t_w:
        mean = 0
    else:
        mean = mean
    value = normalvariate(mean, std) * conversion_factor

    # Output
    return {'add_co2': value}
```

```python
def p_sun_radiation(params,
                    substep,
                    state_history,
                    previous_state):
    # Parameters & variables
    g = params['temperature_constant']
    a = params['sun_radiation']

    # Logic
    temp_change = g * a

    # Output
    return {'add_temperature': temp_change}
```

```python
def p_earth_cooling(params,
                    substep,
                    state_history,
                    previous_state):
    # Parameters & variables
    g = params['temperature_constant']
    K = params['heat_dissipation_constant']
    T = previous_state['temperature']

    # Logic
    temp_change = -(g * K * (T / 300) ** 4)

    # Output
    return {'add_temperature': temp_change}
```

```python
def p_greenhouse_effect(params,
                        substep,
                        state_history,
                        previous_state):
    # Parameters & variables
    g = params['temperature_constant']
    K = params['heat_dissipation_constant']
    beta = params['co2_reflectance_factor']
    T = previous_state['temperature']
    CO2 = previous_state['co2']

    # Logic
    alpha = (1 - np.exp(-beta * CO2))
    temp_change = g * alpha * K * (T / 300) ** 4

    # Output
    return {'add_temperature': temp_change}
```

### 4. State Update Functions

```python
def s_co2(params,
          substep,
          state_history,
          previous_state,
          policy_input):
    # Parameters & variables
    current_co2 = previous_state['co2']
    co2_change = policy_input['add_co2']

    # Logic
    new_co2 = max(current_co2 + co2_change, 0)

    # Output
    return ('co2', new_co2)
```

```python
def s_temperature(params,
                  substep,
                  state_history,
                  previous_state,
                  policy_input):
    # Parameters & variables
    current_temp = previous_state['temperature']
    temp_change = policy_input['add_temperature']

    # Logic
    new_temp = max(current_temp + temp_change, 0)

    # Output
    return ('temperature', new_temp)
```

### 5. Partial State Update Blocks

```python
partial_state_update_blocks = [
    {
        'label': 'Temperature dynamics', # Useful metadata to describe our partial state update blocks
        'policies': {
            'sun_radiation': p_sun_radiation,
            'earth_cooling': p_earth_cooling,
            'greenhouse_effect': p_greenhouse_effect
        },
        'variables': {
            'temperature': s_temperature

        }
    },
    {
        'label': 'CO2 dynamics', # Useful metadata to describe our partial state update blocks
        'policies': {
            'co2_emissions': p_co2_emissions
        },
        'variables': {
            'co2': s_co2
        }

    }
]

```

### 6. Configuration

```python
MONTE_CARLO_RUNS = 50
SIMULATION_TIMESTEPS = 100

sim_config = config_sim(
    {
        'N': MONTE_CARLO_RUNS,
        'T': range(SIMULATION_TIMESTEPS),
        'M': system_params,
    }
)

from cadCAD import configs
del configs[:] # Clear any prior configs

experiment = Experiment()
experiment.append_configs(
    sim_configs=sim_config,
    initial_state=initial_state,
    partial_state_update_blocks=partial_state_update_blocks
)
```

### 7. Execution

```python
exec_context = ExecutionContext()
run = Executor(exec_context=exec_context, configs=configs)

(system_events, tensor_field, sessions) = run.execute()
```

### 8. Simulation Output Preparation

```python
# Get system events and attribute index
df = (pd.DataFrame(system_events)
        .assign(years=lambda df: df.timestep)
        .assign(temperature_celsius=lambda df: df.temperature - 273)
        .query('timestep > 0')
     )

# Clean substeps
first_ind = (df.substep == 0) & (df.timestep == 0)
last_ind = df.substep == max(df.substep)
inds_to_drop = (first_ind | last_ind)
df = df.loc[inds_to_drop].drop(columns=['substep'])

# Attribute parameters to each row
df = df.assign(**configs[0].sim_config['M'])
for i, (_, n_df) in enumerate(df.groupby(['simulation', 'subset', 'run'])):
    df.loc[n_df.index] = n_df.assign(**configs[i].sim_config['M'])
```

# System Validation

[Link to Requirements Analysis](#Requirements-Analysis)

## What-if Matrix

What-if-question | Type of experiment | Variables / parameters | Values / Ranges to be tested
- | - | - | -
How will the __Earth's average temperature__ develop over the next 100 years, if we keep CO2 emissions __unchanged__ at today’s annual emission levels vs. a __doubling__ of today’s emission levels? | Parameter Sweep + Monte Carlo runs | co2_annual_emissions | 40 and 80 Gigatons
How will the __rate of annual temperature change__ develop over the next 100 years if we keep CO2 emissions __unchanged__ at today’s annual emission levels vs. a __doubling__ of today’s emission levels? | Parameter Sweep + Monte Carlo runs | co2_annual_emissions | 40 and 80 Gigatons
How will the __rate of annual temperature change__ develop over the next 100 years if we are able to reduce annual CO2 emissions to __zero__ after a given number of years? | Parameter Sweep + Monte Carlo runs | year_of_the_wakening | 0, 10, 50 and 100 years
How will the __Earth's average temperature__ develop over the next 100 years if we are able to reduce annual CO2 emissions to __zero__ after a given number of years? | Parameter Sweep + Monte Carlo runs | year_of_the_wakening | 0, 10, 50 and 100 years

## System Analysis

### Analysis 1: How will the Earth's average temperature develop over the next 100 years, if we keep CO2 emissions unchanged at today’s annual emission levels vs. a doubling of today’s emission levels?

```python
fig_df = df.query('year_of_the_wakening == 100')

fig = px.scatter(
    fig_df,
    x=fig_df.years,
    y=fig_df.temperature_celsius,
    color=fig_df.co2_annual_emissions.astype(str),
    opacity=0.1,
    trendline="lowess",
    labels={'color': 'Yearly CO2 emissions (Gt)'}
)

fig.show()
```

```python
fig_df = df.query('year_of_the_wakening == 100')

fig = px.box(
    fig_df,
    x=fig_df.years,
    y=fig_df.temperature_celsius,
    color=fig_df.co2_annual_emissions.astype(str),
    points=False,
    labels={'color': 'Yearly CO2 emissions (Gt)'}
)

fig.show()
```

### Analysis 2: How will the rate of annual temperature change develop over the next 100 years if we keep CO2 emissions unchanged at today’s annual emission levels vs. a doubling of today’s emission levels?

```python
fig_df = (df.query('year_of_the_wakening == 100')
            .assign(annual_temperature_increase=lambda df: df.temperature.diff())
            .query('years > 1'))

fig = px.scatter(
    fig_df,
    x=fig_df.years,
    y=fig_df.annual_temperature_increase,
    opacity=0.1,
    trendline="lowess",
    color=fig_df.co2_annual_emissions.astype(str),
    labels={'color': 'Yearly CO2 emissions (Gt)'}
)

fig.show()
```

### Analysis 3: How will the rate of annual temperature change develop over the next 100 years if we are able to reduce annual CO2 emissions to zero after a given number of years?

```python
fig_df = (df.query('co2_annual_emissions == 40')
            .assign(annual_temperature_increase=lambda df: df.temperature.diff())
            .query('years > 1'))

fig = px.scatter(
    fig_df,
    x=fig_df.years,
    y=fig_df.annual_temperature_increase,
    opacity=0.1,
    trendline="lowess",
    color=fig_df.year_of_the_wakening.astype(str),
    labels={'color': 'Year of the wakening (years)'}
)

fig.show()
```

### Analysis 4: How will the Earth's average temperature develop over the next 100 years if we are able to reduce annual CO2 emissions to zero after a given number of years?

```python
fig_df = (df.query('co2_annual_emissions == 40')
            .assign(temperature_increase=lambda df: df.temperature - df.temperature.iloc[0]))

fig = px.scatter(
    fig_df,
    x=fig_df.years,
    y=fig_df.temperature_increase,
    opacity=0.1,
    trendline="lowess",
    color=fig_df.year_of_the_wakening.astype(str),
    labels={'color': 'Year of the wakening (years)'}
)

fig.show()
```

# Congratulations!


# Deciphering crop circle July 28th, 2012 message with Braille's writing system

```
# import necessary libraries to the notebook
from sympy import Matrix
import matplotlib.pyplot as plt
from numpy import array, fliplr, flipud, sort, zeros, matrix
from IPython.core.display import HTML
from pybrl import brailles, ordered_unicodes, matrixcodes, decodings

%matplotlib inline
```

Original picture

    Google  for crop circle appearence at 28th of July, 2012:

## Picture encoded to binary matrices

```
# 12*12 bits message
message = [
[1,0,1,1,1,0,1,1,1,0,1,0],
[0,0,0,1,0,1,1,1,1,0,1,0],
[0,1,1,0,0,0,1,0,1,0,0,0],
[1,0,0,1,0,0,1,1,1,0,1,1],
[0,0,0,1,0,0,0,1,1,1,1,0],
[0,0,0,0,0,1,1,0,0,0,0,1],
[1,0,1,0,0,0,1,1,0,0,1,1],
[0,0,1,1,0,1,0,1,1,0,1,0],
[0,1,0,0,0,0,1,0,1,0,0,1],
[1,1,0,1,0,1,0,1,0,0,1,0],
[0,0,1,1,1,1,0,1,0,0,1,1],
[0,0,0,0,1,0,0,0,1,0,1,0]
]

# inverse
message_inverse = [
[0,1,0,0,0,1,0,0,0,1,0,1],
[1,1,1,0,1,0,0,0,0,1,0,1],
[1,0,0,1,1,1,0,1,0,1,1,1],
[0,1,1,0,1,1,0,0,0,1,0,0],
[1,1,1,0,1,1,1,0,0,0,0,1],
[1,1,1,1,1,0,0,1,1,1,1,0],
[0,1,0,1,1,1,0,0,1,1,0,0],
[1,1,0,0,1,0,1,0,0,1,0,1],
[1,0,1,1,1,1,0,1,0,1,1,0],
[0,0,1,0,1,0,1,0,1,1,0,1],
[1,1,0,0,0,0,1,0,1,1,0,0],
[1,1,1,1,0,1,1,1,0,1,0,1]
]

# black and white plot
def show_matrix(Z):
    N = len(Z)
    G = zeros((N,N,3))
    G[Z<0.5] = [0,0,0]
    G[Z>0.5] = [1,1,1]
    plt.imshow(G, interpolation="nearest")
    plt.show()
```

## Black and white matrix + inversed version

```
#show_matrix(matrix(message))
#show_matrix(matrix(message_inverse))
HTML(
"""
<p>

</p>
""")
```

Braille symbols (1-64)

https://en.wikipedia.org/wiki/Braille

https://en.wikipedia.org/wiki/Braille_Patterns

```
HTML('<p style="font-size: 170%; margin-top: 20px">'+''.join(ordered_unicodes)+'</p>')
```

⠀⠁⠂⠃⠄⠅⠆⠇⠈⠉⠊⠋⠌⠍⠎⠏⠐⠑⠒⠓⠔⠕⠖⠗⠘⠙⠚⠛⠜⠝⠞⠟⠠⠡⠢⠣⠤⠥⠦⠧⠨⠩⠪⠫⠬⠭⠮⠯⠰⠱⠲⠳⠴⠵⠶⠷⠸⠹⠺⠻⠼⠽⠾⠿

Braille symbol meanings

https://en.wikipedia.org/wiki/Braille_ASCII

http://www.brailleauthority.org/literary/ebae2002.pdf

http://www.brailleauthority.org/ueb/symbols_list.pdf

http://www.htctu.fhda.edu/trainings/manuals/alt/grade_two_day_2.pdf

Using numpy and pybrl libraries to flip and decode message matrix:

```
def decode(arr, row_skip = 3, column_skip = 2):
    y = []
    for i in xrange(0, len(arr), row_skip):
        for j in xrange(0, len(arr[0]), column_skip):
            m = Matrix(arr)[i:i+row_skip, j:j+column_skip]
            if row_skip < column_skip:
                l = [m[:][3], m[:][0], m[:][4], m[:][1], m[:][5], m[:][2]]
            else:
                l = m[:]
            x = str(l).replace("\n", '').replace(" ", '')
            for mc, d, b in zip(matrixcodes, decodings, brailles):
                if [item for sublist in mc for item in sublist] == l:
                    x = b+' '+d
                    break
            y.append(x)
    return y
```

```
print ' | '.join(decode(message)), "\r\n"
print ' | '.join(decode(fliplr(message))), "\r\n"
print ' | '.join(decode(flipud(message))), "\r\n"
print ' | '.join(decode(fliplr(flipud(message)))), "\r\n"
```

    ⠡ ch/CHILD/5-character | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠟ q/QUITE/5-question | ⠇ l/LIKE/5-lord/5-6-ful/RELEASE | ⠃ b/BUT/2 | ⠁ a/1 | ⠘ 4-5-  | ⠠ 6- | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠓ h/HAVE/5-here/4-5-6-had/8 | ⠫ ed | ⠡ ch/CHILD/5-character | ⠓ h/HAVE/5-here/4-5-6-had/8 | ⠐ 5- | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠆ be/bb/; | ⠫ ed | ⠉ c/CAN/4-5-6-cannot/3 | ⠚ j/JUST/0 | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠘ 4-5-  | ⠄ 3- | ⠗ r/RATHER/5-right

    ⠘ 4-5-  | ⠸ 4-5-6- | ⠻ er | ⠊ i/9 | ⠫ ed | ⠌ st/STILL | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠚ j/JUST/0 | ⠫ ed | ⠄ 3- | ⠃ b/BUT/2 | ⠈ ´/@ | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠰ 5-6- | ⠫ ed | ⠂ ea/, | ⠚ j/JUST/0 | ⠌ st/STILL | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠠ 6- | ⠃ b/BUT/2 | ⠳ ou/OUT/5-ought | ⠓ h/HAVE/5-here/4-5-6-had/8 | ⠉ c/CAN/4-5-6-cannot/3

    ⠤ com/- | ⠲ dis/dd/. | ⠳ ou/OUT/5-ought | ⠰ 5-6- | ⠁ a/1 | ⠗ r/RATHER/5-right | ⠌ st/STILL | ⠖ to/ff/! | ⠐ 5- | ⠵ z/AS | ⠃ b/BUT/2 | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠄ 3- | ⠰ 5-6- | ⠈ ´/@ | ⠵ z/AS | ⠖ to/ff/! | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠌ st/STILL | ⠵ z/AS | ⠔ in | ⠷ OF | ⠇ l/LIKE/5-lord/5-6-ful/RELEASE | ⠆ be/bb/;

    ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠈ ´/@ | ⠆ be/bb/; | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠖ to/ff/! | ⠤ com/- | ⠵ z/AS | ⠘ 4-5-  | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠂ ea/, | ⠲ dis/dd/. | ⠡ ch/CHILD/5-character | ⠵ z/AS | ⠲ dis/dd/. | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠁ a/1 | ⠆ be/bb/; | ⠠ 6- | ⠰ 5-6- | ⠸ 4-5-6- | ⠾ WITH | ⠢ en/ENOUGH | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠡ ch/CHILD/5-character

```
print ' | '.join(decode(message, row_skip = 2, column_skip = 3)), "\r\n"
print ' | '.join(decode(fliplr(message), row_skip = 2, column_skip = 3)), "\r\n"
print ' | '.join(decode(flipud(message), row_skip = 2, column_skip = 3)), "\r\n"
print ' | '.join(decode(fliplr(flipud(message)), row_skip = 2, column_skip = 3)), "\r\n"
```

    ⠨ 4-6- | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠿ FOR/full | ⠒ con/cc/: | ⠱ wh/WHICH/5-where/4-5-whose | ⠁ a/1 | ⠯ AND | ⠆ be/bb/; | ⠀ SPACE/empty | ⠌ st/STILL | ⠱ wh/WHICH/5-where/4-5-whose | ⠜ ar | ⠬ ing | ⠅ k/KNOWLEDGE/5-know | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠲ dis/dd/. | ⠓ h/HAVE/5-here/4-5-6-had/8 | ⠅ k/KNOWLEDGE/5-know | ⠪ ow | ⠢ en/ENOUGH | ⠠ 6- | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠔ in | ⠲ dis/dd/.

    ⠒ con/cc/: | ⠿ FOR/full | ⠵ z/AS | ⠨ 4-6- | ⠃ b/BUT/2 | ⠯ AND | ⠄ 3- | ⠜ ar | ⠱ wh/WHICH/5-where/4-5-whose | ⠜ ar | ⠡ ch/CHILD/5-character | ⠀ SPACE/empty | ⠚ j/JUST/0 | ⠳ ou/OUT/5-ought | ⠅ k/KNOWLEDGE/5-know | ⠩ sh/SHALL | ⠊ i/9 | ⠪ ow | ⠅ k/KNOWLEDGE/5-know | ⠖ to/ff/! | ⠚ j/JUST/0 | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠈ ´/@

    ⠄ 3- | ⠗ r/RATHER/5-right | ⠢ en/ENOUGH | ⠖ to/ff/! | ⠚ j/JUST/0 | ⠨ 4-6- | ⠕ o/5-one | ⠔ in | ⠥ u/US/5-under/4-5-upon | ⠨ 4-6- | ⠳ ou/OUT/5-ought | ⠖ to/ff/! | ⠀ SPACE/empty | ⠡ ch/CHILD/5-character | ⠎ s/SO/5-some/4-5-6-spirit/4-6-less/5-6-ness | ⠣ gh/RELEASE CAPS/< | ⠎ s/SO/5-some/4-5-6-spirit/4-6-less/5-6-ness | ⠈ ´/@ | ⠽ y/YOU/5-young/5-6-ity/6-ally/ | ⠰ 5-6- | ⠅ k/KNOWLEDGE/5-know | ⠫ ed | ⠿ FOR/full | ⠒ con/cc/:

    ⠓ h/HAVE/5-here/4-5-6-had/8 | ⠊ i/9 | ⠗ r/RATHER/5-right | ⠁ a/1 | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠕ o/5-one | ⠨ 4-6- | ⠲ dis/dd/. | ⠓ h/HAVE/5-here/4-5-6-had/8 | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠨ 4-6- | ⠍ m/MORE/5-mother/4-5-6-many | ⠎ s/SO/5-some/4-5-6-spirit/4-6-less/5-6-ness | ⠣ gh/RELEASE CAPS/< | ⠌ st/STILL | ⠀ SPACE/empty | ⠘ 4-5-  | ⠽ y/YOU/5-young/5-6-ity/6-ally/ | ⠠ 6- | ⠣ gh/RELEASE CAPS/< | ⠒ con/cc/: | ⠿ FOR/full | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠅ k/KNOWLEDGE/5-know

```
print ' | '.join(decode(message_inverse)), "\r\n"
print ' | '.join(decode(fliplr(message_inverse))), "\r\n"
print ' | '.join(decode(flipud(message_inverse))), "\r\n"
print ' | '.join(decode(fliplr(flipud(message_inverse)))), "\r\n"
```

    ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠢ en/ENOUGH | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠠ 6- | ⠸ 4-5-6- | ⠼ ble/# | ⠾ WITH | ⠧ v/VERY | ⠟ q/QUITE/5-question | ⠢ en/ENOUGH | ⠬ ing | ⠔ in | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠬ ing | ⠯ AND | ⠢ en/ENOUGH | ⠹ th/THIS/5-through/4-5-those | ⠔ in | ⠶ were/gg/() | ⠥ u/US/5-under/4-5-upon | ⠡ ch/CHILD/5-character | ⠧ v/VERY | ⠻ er | ⠨ 4-6-

    ⠧ v/VERY | ⠇ l/LIKE/5-lord/5-6-ful/RELEASE | ⠄ 3- | ⠵ z/AS | ⠔ in | ⠳ ou/OUT/5-ought | ⠢ en/ENOUGH | ⠥ u/US/5-under/4-5-upon | ⠔ in | ⠻ er | ⠼ ble/# | ⠷ OF | ⠢ en/ENOUGH | ⠏ p/PEOPLE/5-part | ⠔ in | ⠽ y/YOU/5-young/5-6-ity/6-ally/ | ⠥ u/US/5-under/4-5-upon | ⠳ ou/OUT/5-ought | ⠅ k/KNOWLEDGE/5-know | ⠟ q/QUITE/5-question | ⠼ ble/# | ⠌ st/STILL | ⠬ ing | ⠶ were/gg/()

    ⠛ g/GO/5-6-ong/7 | ⠍ m/MORE/5-mother/4-5-6-many | ⠌ st/STILL | ⠏ p/PEOPLE/5-part | ⠾ WITH | ⠨ 4-6- | ⠳ ou/OUT/5-ought | ⠩ sh/SHALL | ⠯ AND | ⠊ i/9 | ⠼ ble/# | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠻ er | ⠏ p/PEOPLE/5-part | ⠷ OF | ⠊ i/9 | ⠩ sh/SHALL | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠳ ou/OUT/5-ought | ⠊ i/9 | ⠫ ed | ⠈ ´/@ | ⠸ 4-5-6- | ⠹ th/THIS/5-through/4-5-those

    ⠅ k/KNOWLEDGE/5-know | ⠷ OF | ⠹ th/THIS/5-through/4-5-those | ⠡ ch/CHILD/5-character | ⠩ sh/SHALL | ⠛ g/GO/5-6-ong/7 | ⠊ i/9 | ⠧ v/VERY | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠽ y/YOU/5-young/5-6-ity/6-ally/ | ⠍ m/MORE/5-mother/4-5-6-many | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠊ i/9 | ⠍ m/MORE/5-mother/4-5-6-many | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠾ WITH | ⠹ th/THIS/5-through/4-5-those | ⠟ q/QUITE/5-question | ⠏ p/PEOPLE/5-part | ⠇ l/LIKE/5-lord/5-6-ful/RELEASE | ⠁ a/1 | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠞ t/THAT/5-time/4-6-ount/5-6-ment

```
print ' | '.join(toBraille(message_inverse, row_skip = 2, column_skip = 3)), "\r\n"
print ' | '.join(toBraille(fliplr(message_inverse), row_skip = 2, column_skip = 3)), "\r\n"
print ' | '.join(toBraille(flipud(message_inverse), row_skip = 2, column_skip = 3)), "\r\n"
print ' | '.join(toBraille(fliplr(flipud(message_inverse)), row_skip = 2, column_skip = 3)), "\r\n"
```

    ⠗ r/RATHER/5-right | ⠢ en/ENOUGH | ⠀ SPACE/empty | ⠭ x/IT | ⠎ s/SO/5-some/4-5-6-spirit/4-6-less/5-6-ness | ⠾ WITH | ⠐ 5- | ⠹ th/THIS/5-through/4-5-those | ⠿ FOR/full | ⠳ ou/OUT/5-ought | ⠎ s/SO/5-some/4-5-6-spirit/4-6-less/5-6-ness | ⠣ gh/RELEASE CAPS/< | ⠓ h/HAVE/5-here/4-5-6-had/8 | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠡ ch/CHILD/5-character | ⠍ m/MORE/5-mother/4-5-6-many | ⠬ ing | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠕ o/5-one | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠟ q/QUITE/5-question | ⠅ k/KNOWLEDGE/5-know | ⠫ ed | ⠍ m/MORE/5-mother/4-5-6-many

    ⠭ x/IT | ⠀ SPACE/empty | ⠊ i/9 | ⠗ r/RATHER/5-right | ⠼ ble/# | ⠐ 5- | ⠻ er | ⠣ gh/RELEASE CAPS/< | ⠎ s/SO/5-some/4-5-6-spirit/4-6-less/5-6-ness | ⠣ gh/RELEASE CAPS/< | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠿ FOR/full | ⠥ u/US/5-under/4-5-upon | ⠌ st/STILL | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠖ to/ff/! | ⠵ z/AS | ⠕ o/5-one | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠩ sh/SHALL | ⠥ u/US/5-under/4-5-upon | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠅ k/KNOWLEDGE/5-know | ⠷ OF

    ⠻ er | ⠨ 4-6- | ⠝ n/NOT/5-name/4-6-sion/5-6-tion/6-ation | ⠩ sh/SHALL | ⠥ u/US/5-under/4-5-upon | ⠗ r/RATHER/5-right | ⠪ ow | ⠫ ed | ⠚ j/JUST/0 | ⠗ r/RATHER/5-right | ⠌ st/STILL | ⠩ sh/SHALL | ⠿ FOR/full | ⠞ t/THAT/5-time/4-6-ount/5-6-ment | ⠱ wh/WHICH/5-where/4-5-whose | ⠜ ar | ⠱ wh/WHICH/5-where/4-5-whose | ⠷ OF | ⠂ ea/, | ⠏ p/PEOPLE/5-part | ⠺ w/WILL/5-work/4-5-word/4-5-6-world | ⠔ in | ⠀ SPACE/empty | ⠭ x/IT

    ⠬ ing | ⠵ z/AS | ⠨ 4-6- | ⠾ WITH | ⠮ THE/5-there/4-5-these/4-5-6-their | ⠪ ow | ⠗ r/RATHER/5-right | ⠍ m/MORE/5-mother/4-5-6-many | ⠬ ing | ⠡ ch/CHILD/5-character | ⠗ r/RATHER/5-right | ⠲ dis/dd/. | ⠱ wh/WHICH/5-where/4-5-whose | ⠜ ar | ⠳ ou/OUT/5-ought | ⠿ FOR/full | ⠧ v/VERY | ⠂ ea/, | ⠟ q/QUITE/5-question | ⠜ ar | ⠭ x/IT | ⠀ SPACE/empty | ⠑ e/EVERY/5-ever/4-6-ance/5-6-ence/5 | ⠺ w/WILL/5-work/4-5-word/4-5-6-world

We can see from above decoding samples, that one of them gives reasonable continuous and unbreaking form of the message, while still leaving it up to a reader, that it meant with few words, namely CHILD and Q PLANET:

```
HTML("""

<p style="font-size: 200%; margin: 20px">
⠅ KNOWLEDGE ⠷ OF ⠹ THIS ⠡ CHILD ⠩ SHALL
</p>
<p style="font-size: 200%; margin: 20px">
⠛ g ⠊ i ⠧ v ⠑ e ⠽ YOU ⠍ MORE
</p>
<p style="font-size: 200%; margin: 20px">
⠞ t ⠊ i ⠍ m ⠑ e ⠾ WITH ⠹ THIS
</p>
<p style="font-size: 200%; margin: 20px">
⠟ q ⠏ p ⠇ l ⠁ a ⠝ n ⠑ e ⠞ t
</p>

""")
```

⠅ KNOWLEDGE ⠷ OF ⠹ THIS ⠡ CHILD ⠩ SHALL

⠛ g ⠊ i ⠧ v ⠑ e ⠽ YOU ⠍ MORE

⠞ t ⠊ i ⠍ m ⠑ e ⠾ WITH ⠹ THIS

⠟ q ⠏ p ⠇ l ⠁ a ⠝ n ⠑ e ⠞ t

## The [MIT](http://choosealicense.com/licenses/mit/) License

Copyright (c) 2015 Marko Manninen

Crop circle July 28th, 2012 Braille message.ipynb


# What is Deep Learning

To understand what deep learning is, we first need to understand the relationship deep learning has with machine learning, neural networks, and artificial intelligence.

The best way to think of this relationship is to visualize them as concentric circles:

At the outer most ring you have artificial intelligence (using computers to reason). One layer inside of that is machine learning. With artificial neural networks and deep learning at the center.

Broadly speaking, deep learning is a more approachable name for an artificial neural network. The “deep” in deep learning refers to the depth of the network. An artificial neural network can be very shallow.

Neural networks are inspired by the structure of the cerebral cortex. At the basic level is the perceptron, the mathematical representation of a biological neuron. Like in the cerebral cortex, there can be several layers of interconnected perceptrons.

The first layer is the input layer. Each node in this layer takes an input, and then passes its output as the input to each node in the next layer. There are generally no connections between nodes in the same layer and the last layer produces the outputs.

We call the middle part the hidden layer. These neurons have no connection to the outside (e.g. input or output) and are only activated by nodes in the previous layer.

Think of deep learning as the technique for learning in neural networks that utilizes multiple layers of abstraction to solve pattern recognition problems. In the 1980s, most neural networks were a single layer due to the cost of computation and availability of data.

Machine learning is considered a branch or approach of Artificial intelligence, whereas deep learning is a specialized type of machine learning.

Machine learning involves computer intelligence that doesn’t know the answers up front. Instead, the program will run against training data, verify the success of its attempts, and modify its approach accordingly. Machine learning typical requires a sophisticated education, spanning software engineering and computer science to statistical methods and linear algebra.

There are two broad classes of machine learning methods:

   * Supervised learning
   * Unsupervised learning

In supervised learning, a machine learning algorithm uses a labeled dataset to infer the desired outcome. This takes a lot of data and time, since the data needs to be labeled by hand. Supervised learning is great for classification and regression problems.

For example, let’s say that we were running a company and want to determine the effect of bonuses on employee retention. If we had historical data – i.e. employee bonus amount and tenure – we could use supervised machine learning.

With unsupervised learning, there aren’t any predefined or corresponding answers. The goal is to figure out the hidden patterns in the data. It’s usually used for clustering and associative tasks, like grouping customers by behavior. Amazon’s “customers who also bought…” recommendations are a type of associative task.

While supervised learning can be useful, we often have to resort to unsupervised learning. Deep learning has proven to be an effective unsupervised learning technique.

## Why is Deep Learning Important?

Computers have long had techniques for recognizing features inside of images. The results weren’t always great. Computer vision has been a main beneficiary of deep learning. Computer vision using deep learning now rivals humans on many image recognition tasks.

Facebook has had great success with identifying faces in photographs by using deep learning. It’s not just a marginal improvement, but a game changer: “Asked whether two unfamiliar photos of faces show the same person, a human being will get it right 97.53 percent of the time. New software developed by researchers at Facebook can score 97.25 percent on the same challenge, regardless of variations in lighting or whether the person in the picture is directly facing the camera.”

Speech recognition is a another area that’s felt deep learning’s impact. Spoken languages are so vast and ambiguous. Baidu – one of the leading search engines of China – has developed a voice recognition system that is faster and more accurate than humans at producing text on a mobile phone. In both English and Mandarin.

What is particularly fascinating, is that generalizing the two languages didn’t require much additional design effort: “Historically, people viewed Chinese and English as two vastly different languages, and so there was a need to design very different features,” Andrew Ng says, chief scientist at Baidu. “The learning algorithms are now so general that you can just learn.”

Google is now using deep learning to manage the energy at the company’s data centers. They’ve cut their energy needs for cooling by 40%. That translates to about a 15% improvement in power usage efficiency for the company and hundreds of millions of dollars in savings.

## Deep Learning Microservices

Here’s a quick overview of some deep learning use cases and microservices.

Illustration Tagger. An implementation of Illustration2Vec, this microservice can tag an image with the safe, questionable, or explicit rating, the copyright, and general category tag to understand what’s in the image. DeepFilter is a style transfer service for applying artistic filters to images.

The age classifier uses face detection to determine the age of a person in a photo. The Places 365 Classifier uses a pre-trained CNN and based on Places: An Image Database for Deep Scene Understanding B. Zhou, et al., 2016 to identify particular locations in images, such as a courtyard, drugstore, hotel room, glacier, mountain, etc. Lastly, there is InceptionNet, a direct implementation of Google’s InceptionNet using TensorFlow. It takes an image (such as a car), and returns the top 5 classes the model predicts are relevant to the image.

## Open Source Deep Learning Frameworks

Deep learnings is made accessible by a number of open source projects. Some of the most popular technologies include, but are not limited to, Deeplearning4j (DL4j), Theano, Torch, TensorFlow, and Caffe. The deciding factors on which one to use are the tech stack they target, and if they are low-level, academic, or application focused. Here’s an overview of each:

DL4J:

   * JVM-based
   * Distrubted
   * Integrates with Hadoop and Spark

Theano:

   * Very popular in Academia
   * Fairly low level
   * Interfaced with via Python and Numpy

Torch:

   * Lua based
   * In house versions used by Facebook and Twitter
   * Contains pretrained models

TensorFlow:

   * Google written successor to Theano
   * Interfaced with via Python and Numpy
   * Highly parallel
   * Can be somewhat slow for certain problem sets

Caffe:

   * Not general purpose. Focuses on machine-vision problems
   * Implemented in C++ and is very fast
   * Not easily extensible
   * Has a Python interface

## McCulloch and Pitts Neuron

In 1943, McCulloch and Pitts introduced a mathematical model of a neuron. It consisted of three components:

1. A set of **weights** $w_i$ corresponding to synapses (inputs)
2. An **adder** for summing input signals; analogous to cell membrane that collects charge
3. An **activation function** for determining when the neuron fires, based on accumulated input

The neuron model is shown schematically below. On the left are input nodes $\{x_i\}$, usually expressed as a vector. The strength with which the inputs are able to deliver the signal along the synapse is determined by their corresponding weights $\{w_i\}$. The adder then sums the inputs from all the synapses:

$$h = \sum_i w_i x_i$$

The parameter $\theta$ determines whether or not the neuron fires given a weighted input of $h$. If it fires, it returns a value $y=1$, otherwise $y=0$. For example, a simple **activation function** is using $\theta$ as a simple fixed threshold:

$$y = g(h) = \left\{ \begin{array}{l}
1, \text{if } h \gt \theta \\
0, \text{if } h \le \theta
\end{array} \right.$$

this activation function may take any of several forms, such as a logistic function.

A single neuron is not interesting, nor useful, from a learning perspective. It cannot learn; it simply receives inputs and either fires or not. Only when neurons are joined as a **network** can they perform useful work.

Learning takes place by changing the weights of the connections in a neural network, and by changing the parameters of the activation functions of neurons.

## Perceptron

A collection of McCullough and Pitts neurons, along with a set of input nodes connected to the inputs via weighted edges, is a perceptron, the simplest neural network.

Each neuron is independent of the others in the perceptron, in the sense that its behavior and performance depends only on its own weights and threshold values, and not of those for the other neurons. Though they share inputs, they operate independently.

The number of inputs and outputs are determined by the data. Weights are stored as a `N x K` matrix, with N observations and K neurons, with $w_{ij}$ specifying the weight on the *i*th observation on the *j*th neuron.

In order to use the perceptron for statistical learning, we compare the outputs $y_j$ from each neuron to the obervation target $t_j$, and adjust the input weights when they do not correspond (*e.g.* if a neuron fires when it should not have).

$$t_j - y_j$$

We use this difference to update the weight $w_{ij}$, based on the input and a desired **learning rate**. This results in an update rule:

$$w_{ij} \leftarrow w_{ij} + \eta (t_j - y_j) x_i$$

After an incremental improvement, the perceptron is shown the training data again, resulting in another update. This is repeated until the performance no longer improves. Having a learning rate less than one results in a more stable learning rate, though this stability is traded off against having to expose the network to the data multiple times. Typical learning rates are in the 0.1-0.4 range.

An additional input node is typically added to the perceptron model, which is a constant value (usually -1, 0, or 1) that acts analogously to an intercept in a regression model. This establishes a baseline input for the case when all inputs are zero.

## Learning with Perceptrons

1. Initialize weights $w_{ij}$ to small, random numbers.
2. For each t in T iterations
    * compute activation for each neuron *j* connected to each input vector *i*
    $$y_j = g\left( h=\sum_i w_{ij} x_i \right) = \left\{ \begin{array}{l}
1, \text{if } h \gt 0 \\
0, \text{if } h \le 0
\end{array} \right.$$
    * update weights
    $$w_{ij} \leftarrow w_{ij} + \eta (t_j - y_j) x_i$$

This algorithm is $\mathcal{O}(Tmn)$

### Example: Logical functions

Let's see how the perceptron learns by training it on a couple of of logical functions, AND and OR. For two variables `x1` and `x2`, the AND function returns 1 if both are true, or zero otherwise; the OR function returns 1 if either variable is true, or both. These functions can be expressed as simple lookup tables.

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
from scipy import optimize
from ipywidgets import *
from IPython.display import SVG
from sklearn import datasets

```

```python
AND = pd.DataFrame({'x1': (0,0,1,1), 'x2': (0,1,0,1), 'y': (0,0,0,1)})
AND
```

      x1
      x2
      y

      0
      0
      0
      0

      1
      0
      1
      0

      2
      1
      0
      0

      3
      1
      1
      1

First, we need to initialize weights to small, random values (can be positive and negative).

```python
w = np.random.randn(3)*1e-4
```

Then, a simple activation function for calculating $g(h)$:

```python
g = lambda inputs, weights: np.where(np.dot(inputs, weights)>0, 1, 0)
```

Finally, a training function that iterates the learning algorithm, returning the adapted weights.

```python
def train(inputs, targets, weights, eta, n_iterations):

    # Add the inputs that match the bias node
    inputs = np.c_[inputs, -np.ones((len(inputs), 1))]

    for n in range(n_iterations):

        activations = g(inputs, weights);
        weights -= eta*np.dot(np.transpose(inputs), activations - targets)

    return(weights)
```

Let's test it first on the AND function.

```python
inputs = AND[['x1','x2']]
target = AND['y']

w = train(inputs, target, w, 0.25, 10)
```

Checking the performance:

```python
g(np.c_[inputs, -np.ones((len(inputs), 1))], w)
```

    array([0, 0, 0, 1])

Thus, it has learned the function perfectly. Now for OR:

```python
OR = pd.DataFrame({'x1': (0,0,1,1), 'x2': (0,1,0,1), 'y': (0,1,1,1)})
OR
```

      x1
      x2
      y

      0
      0
      0
      0

      1
      0
      1
      1

      2
      1
      0
      1

      3
      1
      1
      1

```python
w = np.random.randn(3)*1e-4
```

```python
inputs = OR[['x1','x2']]
target = OR['y']

w = train(inputs, target, w, 0.25, 20)
```

```python
g(np.c_[inputs, -np.ones((len(inputs), 1))], w)
```

    array([0, 1, 1, 1])

Also 100% correct.

### Exercise: XOR

Now try running the model on the XOR function, where a one is returned for either `x1` or `x2` being true, but *not* both. What happens here?

```python
# Write your answer here
```

Let's explore the problem graphically:

```python
AND.plot(kind='scatter', x='x1', y='x2', c='y', s=50, colormap='winter')
plt.plot(np.linspace(0,1.4), 1.5 - 1*np.linspace(0,1.4), 'k--');
```

```python
OR.plot(kind='scatter', x='x1', y='x2', c='y', s=50, colormap='winter')
plt.plot(np.linspace(-.4,1), .5 - 1*np.linspace(-.4,1), 'k--');
```

```python
XOR = pd.DataFrame({'x1': (0,0,1,1), 'x2': (0,1,0,1), 'y': (0,1,1,0)})

XOR.plot(kind='scatter', x='x1', y='x2', c='y', s=50, colormap='winter');
```

The perceptron tries to find a separating hyperplane for the two response classes. Namely, a set of weights that satisfies:

$$\mathbf{x_1}\mathbf{w}^T=0$$

and:

$$\mathbf{x_2}\mathbf{w}^T=0$$

Hence,

$$\begin{aligned}
\mathbf{x}_1\mathbf{w}^T &= \mathbf{x}_2\mathbf{w}^T \\
\Rightarrow (\mathbf{x}_1 - \mathbf{x}_2) \mathbf{w}^T &= 0
\end{aligned}$$

This means that either the norms of $\mathbf{x}_1 - \mathbf{x}_2$ or $\mathbf{w}$ are zero, or the cosine of the angle between them is equal to zero, due to the identity:

$$\mathbf{a}\mathbf{b} = \|a\| \|b\| \cos \theta$$

Since there is no reason for the norms to be zero in general, we need the two vectors to be at right angles to one another. So, we need a weight vector that is perpendicular to the decision boundary.

Clearly, for the XOR function, the output classes are not linearly separable. So, the algorithm does not converge on an answer, but simply cycles through two incorrect solutions.

## Multi-layer Perceptron

The solution to fitting more complex (*i.e.* non-linear) models with neural networks is to use a more complex network that consists of more than just a single perceptron. The take-home message from the perceptron is that all of the learning happens by adapting the synapse weights until prediction is satisfactory. Hence, a reasonable guess at how to make a perceptron more complex is to simply **add more weights**.

There are two ways to add complexity:

1. Add backward connections, so that output neurons feed back to input nodes, resulting in a **recurrent network**
2. Add neurons between the input nodes and the outputs, creating an additional ("hidden") layer to the network, resulting in a **multi-layer perceptron**

The latter approach is more common in applications of neural networks.

How to train a multilayer network is not intuitive. Propagating the inputs forward over two layers is straightforward, since the outputs from the hidden layer can be used as inputs for the output layer. However, the process for updating the weights based on the prediction error is less clear, since it is difficult to know whether to change the weights on the input layer or on the hidden layer in order to improve the prediction.

Updating a multi-layer perceptron (MLP) is a matter of:

1. moving forward through the network, calculating outputs given inputs and current weight estimates
2. moving backward updating weights according to the resulting error from forward propagation.

In this sense, it is similar to a single-layer perceptron, except it has to be done twice, once for each layer.

# Backpropagation

Backpropagation is a method for efficiently computing the gradient of the cost function of a neural network with respect to its parameters.  These partial derivatives can then be used to update the network's parameters using, e.g., gradient descent.  This may be the most common method for training neural networks.  Deriving backpropagation involves numerous clever applications of the chain rule for functions of vectors.

## Review: The chain rule

The chain rule is a way to compute the derivative of a function whose variables are themselves functions of other variables.  If $C$ is a scalar-valued function of a scalar $z$ and $z$ is itself a scalar-valued function of another scalar variable $w$, then the chain rule states that
$$
\frac{\partial C}{\partial w} = \frac{\partial C}{\partial z}\frac{\partial z}{\partial w}
$$
For scalar-valued functions of more than one variable, the chain rule essentially becomes additive.  In other words, if $C$ is a scalar-valued function of $N$ variables $z_1, \ldots, z_N$, each of which is a function of some variable $w$, the chain rule states that
$$
\frac{\partial C}{\partial w} = \sum_{i = 1}^N \frac{\partial C}{\partial z_i}\frac{\partial z_i}{\partial w}
$$

## Notation

In the following derivation, we'll use the following notation:

$L$ - Number of layers in the network.

$N^n$ - Dimensionality of layer $n \in \{0, \ldots, L\}$.  $N^0$ is the dimensionality of the input; $N^L$ is the dimensionality of the output.

$W^m \in \mathbb{R}^{N^m \times N^{m - 1}}$ - Weight matrix for layer $m \in \{1, \ldots, L\}$.  $W^m_{ij}$ is the weight between the $i^{th}$ unit in layer $m$ and the $j^{th}$ unit in layer $m - 1$.

$b^m \in \mathbb{R}^{N^m}$ - Bias vector for layer $m$.

$\sigma^m$ - Nonlinear activation function of the units in layer $m$, applied elementwise.

$z^m \in \mathbb{R}^{N^m}$ - Linear mix of the inputs to layer $m$, computed by $z^m = W^m a^{m - 1} + b^m$.

$a^m \in \mathbb{R}^{N^m}$ - Activation of units in layer $m$, computed by $a^m = \sigma^m(h^m) = \sigma^m(W^m a^{m - 1} + b^m)$.  $a^L$ is the output of the network.  We define the special case $a^0$ as the input of the network.

$y \in \mathbb{R}^{N^L}$ - Target output of the network.

$C$ - Cost/error function of the network, which is a function of $a^L$ (the network output) and $y$ (treated as a constant).

## Backpropagation in general

In order to train the network using a gradient descent algorithm, we need to know the gradient of each of the parameters with respect to the cost/error function $C$; that is, we need to know $\frac{\partial C}{\partial W^m}$ and $\frac{\partial C}{\partial b^m}$.  It will be sufficient to derive an expression for these gradients in terms of the following terms, which we can compute based on the neural network's architecture:

- $\frac{\partial C}{\partial a^L}$: The derivative of the cost function with respect to its argument, the output of the network
- $\frac{\partial a^m}{\partial z^m}$: The derivative of the nonlinearity used in layer $m$ with respect to its argument

To compute the gradient of our cost/error function $C$ to $W^m_{ij}$ (a single entry in the weight matrix of the layer $m$), we can first note that $C$ is a function of $a^L$, which is itself a function of the linear mix variables $z^m_k$, which are themselves functions of the weight matrices $W^m$ and biases $b^m$.  With this in mind, we can use the chain rule as follows:

$$\frac{\partial C}{\partial W^m_{ij}} = \sum_{k = 1}^{N^m} \frac{\partial C}{\partial z^m_k} \frac{\partial z^m_k}{\partial W^m_{ij}}$$

Note that by definition
$$
z^m_k = \sum_{l = 1}^{N^m} W^m_{kl} a_l^{m - 1} + b^m_k
$$
It follows that $\frac{\partial z^m_k}{\partial W^m_{ij}}$ will evaluate to zero when $i \ne k$ because $z^m_k$ does not interact with any elements in $W^m$ except for those in the $k$th row, and we are only considering the entry $W^m_{ij}$.  When $i = k$, we have

\begin{align*}
\frac{\partial z^m_i}{\partial W^m_{ij}} &= \frac{\partial}{\partial W^m_{ij}}\left(\sum_{l = 1}^{N^m} W^m_{il} a_l^{m - 1} + b^m_i\right)\\
&= a^{m - 1}_j\\
\rightarrow \frac{\partial z^m_k}{\partial W^m_{ij}} &= \begin{cases}
0 & k \ne i\\
a^{m - 1}_j & k = i
\end{cases}
\end{align*}

The fact that $\frac{\partial C}{\partial a^m_k}$ is $0$ unless $k = i$ causes the summation above to collapse, giving

$$\frac{\partial C}{\partial W^m_{ij}} = \frac{\partial C}{\partial z^m_i} a^{m - 1}_j$$

or in vector form

$$\frac{\partial C}{\partial W^m} = \frac{\partial C}{\partial z^m} a^{m - 1 \top}$$

Similarly for the bias variables $b^m$, we have

$$\frac{\partial C}{\partial b^m_i} = \sum_{k = 1}^{N^m} \frac{\partial C}{\partial z^m_k} \frac{\partial z^m_k}{\partial b^m_i}$$

As above, it follows that $\frac{\partial z^m_k}{\partial b^m_i}$ will evaluate to zero when $i \ne k$ because $z^m_k$ does not interact with any element in $b^m$ except $b^m_k$.  When $i = k$, we have

\begin{align*}
\frac{\partial z^m_i}{\partial b^m_i} &= \frac{\partial}{\partial b^m_i}\left(\sum_{l = 1}^{N^m} W^m_{il} a_l^{m - 1} + b^m_i\right)\\
&= 1\\
\rightarrow \frac{\partial z^m_i}{\partial b^m_i} &= \begin{cases}
0 & k \ne i\\
1 & k = i
\end{cases}
\end{align*}

The summation also collapses to give

$$\frac{\partial C}{\partial b^m_i} = \frac{\partial C}{\partial z^m_i}$$

or in vector form

$$\frac{\partial C}{\partial b^m} = \frac{\partial C}{\partial z^m}$$

Now, we must compute $\frac{\partial C}{\partial z^m_k}$.  For the final layer ($m = L$), this term is straightforward to compute using the chain rule:

$$
\frac{\partial C}{\partial z^L_k} = \frac{\partial C}{\partial a^L_k} \frac{\partial a^L_k}{\partial z^L_k}
$$

or, in vector form

$$
\frac{\partial C}{\partial z^L} = \frac{\partial C}{\partial a^L} \frac{\partial a^L}{\partial z^L}
$$

The first term $\frac{\partial C}{\partial a^L}$ is just the derivative of the cost function with respect to its argument, whose form depends on the cost function chosen.  Similarly, $\frac{\partial a^m}{\partial z^m}$ (for any layer $m$ includling $L$) is the derivative of the layer's nonlinearity with respect to its argument and will depend on the choice of nonlinearity.  For other layers, we again invoke the chain rule:

\begin{align*}
\frac{\partial C}{\partial z^m_k} &= \frac{\partial C}{\partial a^m_k} \frac{\partial a^m_k}{\partial z^m_k}\\
&= \left(\sum_{l = 1}^{N^{m + 1}}\frac{\partial C}{\partial z^{m + 1}_l}\frac{\partial z^{m + 1}_l}{\partial a^m_k}\right)\frac{\partial a^m_k}{\partial z^m_k}\\
&= \left(\sum_{l = 1}^{N^{m + 1}}\frac{\partial C}{\partial z^{m + 1}_l}\frac{\partial}{\partial a^m_k} \left(\sum_{h = 1}^{N^m} W^{m + 1}_{lh} a_h^m + b_l^{m + 1}\right)\right) \frac{\partial a^m_k}{\partial z^m_k}\\
&= \left(\sum_{l = 1}^{N^{m + 1}}\frac{\partial C}{\partial z^{m + 1}_l} W^{m + 1}_{lk}\right) \frac{\partial a^m_k}{\partial z^m_k}\\
&= \left(\sum_{l = 1}^{N^{m + 1}}W^{m + 1\top}_{kl} \frac{\partial C}{\partial z^{m + 1}_l}\right) \frac{\partial a^m_k}{\partial z^m_k}\\
\end{align*}

where the last simplification was made because by convention $\frac{\partial C}{\partial z^{m + 1}_l}$ is a column vector, allowing us to write the following vector form:

$$\frac{\partial C}{\partial z^m} = \left(W^{m + 1\top} \frac{\partial C}{\partial z^{m + 1}}\right) \circ \frac{\partial a^m}{\partial z^m}$$

Note that we now have the ingredients to efficiently compute the gradient of the cost function with respect to the network's parameters:  First, we compute $\frac{\partial C}{\partial z^L_k}$ based on the choice of cost function and nonlinearity.  Then, we recursively can compute $\frac{\partial C}{\partial z^m}$ layer-by-layer based on the term $\frac{\partial C}{\partial z^{m + 1}}$ computed from the previous layer and the nonlinearity of the layer (this is called the "backward pass").

## Backpropagation in practice

As discussed above, the exact form of the updates depends on both the chosen cost function and each layer's chosen nonlinearity.  The following two table lists the some common choices for nonlinearities and the required partial derivative for deriving the gradient for each layer:

| Nonlinearity | $a^m = \sigma^m(z^m)$ | $\frac{\partial a^m}{\partial z^m}$ | Notes |
|--------------|---|---|---|
| Sigmoid      | $\frac{1}{1 + e^{z^m}}$ | $\sigma^m(z^m)(1 - \sigma^m(z^m)) = a^m(1 - a^m)$ | "Squashes" any input to the range $[0, 1]$ |
| Tanh         | $\frac{e^{z^m} - e^{-z^m}}{e^{z^m} + e^{-z^m}}$ | $1 - (\sigma^m(z^m))^2 = 1 - (a^m)^2$ | Equivalent, up to scaling, to the sigmoid function |
| ReLU         | $\max(0, z^m)$ | $0, z^m < 0;\; 1, z^m \ge 0$ | Commonly used in neural networks with many layers|

Similarly, the following table collects some common cost functions and the partial derivative needed to compute the gradient for the final layer:

| Cost Function | $C$                                  | $\frac{\partial C}{\partial a^L}$ | Notes |
|---------------|--------------------------------------|-----------------------------------|---|
| Squared Error | $\frac{1}{2}(y - a^L)^\top(y - a^L)$ | $y - a^L$                         | Commonly used when the output is not constrained to a specific range |
| Cross-Entropy | $(y - 1)\log(1 - a^L) - y\log(a^L)$  | $\frac{a^L - y}{a^L(1 - a^L)}$    | Commonly used for binary classification tasks; can yield faster convergence |

In practice, backpropagation proceeds in the following manner for each training sample:

1. Forward pass: Given the network input $a^0$, compute $a^m$ recursively by
 $$a^1 = \sigma^1(W^1 a^0 + b^1), \ldots, a^L = \sigma^L(W^L a^{L - 1} + b^L)$$
1. Backward pass: Compute
$$\frac{\partial C}{\partial z^L} = \frac{\partial C}{\partial a^L} \frac{\partial a^L}{\partial z^L}$$
for the final layer based on the tables above, then recursively compute
$$\frac{\partial C}{\partial z^m} = \left(W^{m + 1\top} \frac{\partial C}{\partial z^{m + 1}}\right) \circ \frac{\partial a^m}{\partial z^m}$$
for all other layers.  Plug these values into
$$\frac{\partial C}{\partial W^m} = \frac{\partial C}{\partial z^m_i} a^{m - 1 \top}$$
and
$$\frac{\partial C}{\partial b^m} = \frac{\partial C}{\partial z^m}$$
to obtain the updates.

### Example: Sigmoid network with cross-entropy loss using gradient descent

A common network architecture is one with fully connected layers where each layer's nonlinearity is the sigmoid function $a^m = \frac{1}{1 + e^{z^m}}$ and the cost function is the cross-entropy loss $(y - 1)\log(1 - a^L) - y\log(a^L)$.  To compute the updates for gradient descent, we first compute (based on the tables above)
\begin{align*}
\frac{\partial C}{\partial z^L} &= \frac{\partial C}{\partial a^L} \frac{\partial a^L}{\partial z^L}\\
&= \left(\frac{a^L - y}{a^L(1 - a^L)}\right)a^L(1 - a^L)\\
&= a^L - y
\end{align*}
From here, we can compute
\begin{align*}
\frac{\partial C}{\partial z^{L - 1}} &= \left(W^{L\top} \frac{\partial C}{\partial z^L} \right) \circ \frac{\partial a^{L - 1}}{\partial z^{L - 1}}\\
&= W^{L\top} (a^L - y) \circ a^{L - 1}(1 - a^{L - 1})\\
\frac{\partial C}{\partial z^{L - 2}} &= \left(W^{L - 1\top} \frac{\partial C}{\partial z^{L - 1}} \right) \circ \frac{\partial a^{L - 2}}{\partial z^{L - 2}}\\
&= W^{L - 1\top} \left(W^{L\top} (a^L - y) \circ a^{L - 1}(1 - a^{L - 1})\right) \circ a^{L - 2}(1 - a^{L - 2})
\end{align*}
and so on, until we have computed $\frac{\partial C}{\partial z^m}$ for $m \in \{1, \ldots, L\}$.  This allows us to compute  $\frac{\partial C}{\partial W^m_{ij}}$ and $\frac{\partial C}{\partial b^m_i}$, e.g.
\begin{align*}
\frac{\partial C}{\partial W^L} &= \frac{\partial C}{\partial z^L} a^{L - 1 \top}\\
&= (a^L - y)a^{L - 1\top}\\
\frac{\partial C}{\partial W^{L - 1}} &= \frac{\partial C}{\partial z^{L - 1}} a^{L - 2 \top}\\
&= W^{L\top} (a^L - y) \circ a^{L - 1}(1 - a^{L - 1}) a^{L - 2\top}
\end{align*}
and so on.  Standard gradient descent then updates each parameter as follows:
$$W^m = W^m - \lambda \frac{\partial C}{\partial W^m}$$
$$b^m = b^m - \lambda \frac{\partial C}{\partial b^m}$$
where $\lambda$ is the learning rate.  This process is repeated until some stopping criteria is met.

## Toy Python example

Due to the recursive nature of the backpropagation algorithm, it lends itself well to software implementations.  The following code implements a multi-layer perceptron which is trained using backpropagation with user-supplied nonlinearities, layer sizes, and cost function.

```python
# Ensure python 3 forward compatibility
from __future__ import print_function
import numpy as np

def sigmoid(x):
    return 1/(1 + np.exp(-x))

class SigmoidLayer:
    def __init__(self, n_input, n_output):
        self.W = np.random.randn(n_output, n_input)
        self.b = np.random.randn(n_output, 1)
    def output(self, X):
        if X.ndim == 1:
            X = X.reshape(-1, 1)
        return sigmoid(self.W.dot(X) + self.b)

class SigmoidNetwork:

    def __init__(self, layer_sizes):
        '''
        :parameters:
            - layer_sizes : list of int
                List of layer sizes of length L+1 (including the input dimensionality)
        '''
        self.layers = []
        for n_input, n_output in zip(layer_sizes[:-1], layer_sizes[1:]):
            self.layers.append(SigmoidLayer(n_input, n_output))

    def train(self, X, y, learning_rate=0.2):
        X = np.array(X)
        y = np.array(y)
        if X.ndim == 1:
            X = X.reshape(-1, 1)
        if y.ndim == 1:
            y = y.reshape(1, -1)

        # Forward pass - compute a^n for n in {0, ... L}
        layer_outputs = [X]
        for layer in self.layers:
            layer_outputs.append(layer.output(layer_outputs[-1]))

        # Backward pass - compute \partial C/\partial z^m for m in {L, ..., 1}
        cost_partials = [layer_outputs[-1] - y]
        for layer, layer_output in zip(reversed(self.layers), reversed(layer_outputs[:-1])):
            cost_partials.append(layer.W.T.dot(cost_partials[-1])*layer_output*(1 - layer_output))
        cost_partials.reverse()

        # Compute weight gradient step
        W_updates = []
        for cost_partial, layer_output in zip(cost_partials[1:], layer_outputs[:-1]):
            W_updates.append(cost_partial.dot(layer_output.T)/X.shape[1])
        # and biases
        b_updates = [cost_partial.mean(axis=1).reshape(-1, 1) for cost_partial in cost_partials[1:]]

        for W_update, b_update, layer in zip(W_updates, b_updates, self.layers):
            layer.W -= W_update*learning_rate
            layer.b -= b_update*learning_rate

    def output(self, X):
        a = np.array(X)
        if a.ndim == 1:
            a = a.reshape(-1, 1)
        for layer in self.layers:
            a = layer.output(a)
        return a
```

```python
nn = SigmoidNetwork([2, 2, 1])
X = np.array([[0, 1, 0, 1],
              [0, 0, 1, 1]])
y = np.array([0, 1, 1, 0])
for n in range(int(1e3)):
    nn.train(X, y, learning_rate=1.)
print("Input\tOutput\tQuantized")
for i in [[0, 0], [1, 0], [0, 1], [1, 1]]:
    print("{}\t{:.4f}\t{}".format(i, nn.output(i)[0, 0], 1*(nn.output(i)[0] > .5)))
```

    Input	Output	Quantized
    [0, 0]	0.5125	[1]
    [1, 0]	0.4927	[0]
    [0, 1]	0.9788	[1]
    [1, 1]	0.0161	[0]

```python
logistic = lambda h, beta: 1./(1 + np.exp(-beta * h))

@interact(beta=(-1, 25))
def logistic_plot(beta=5):
    hvals = np.linspace(-2, 2)
    plt.plot(hvals, logistic(hvals, beta))
```

This has the advantage of having a simple derivative:

$$\frac{dg}{dh} = \beta g(h)(1 - g(h))$$

Alternatively, the hyperbolic tangent function is also sigmoid:

$$g(h) = \tanh(h) = \frac{\exp(h) - \exp(-h)}{\exp(h) + \exp(-h)}$$

```python
hyperbolic_tangent = lambda h: (np.exp(h) - np.exp(-h)) / (np.exp(h) + np.exp(-h))

@interact(theta=(-1, 25))
def tanh_plot(theta=5):
    hvals = np.linspace(-2, 2)
    h = hvals*theta
    plt.plot(hvals, hyperbolic_tangent(h))
```

Gradient Descent
---
The simplest algorithm for iterative minimization of differentiable functions is known as just **gradient descent**.
Recall that the gradient of a function is defined as the vector of partial derivatives:

$$\nabla f(x) =  [{\partial{f}{x_1}, \partial{f}{x_2}, \ldots, \partial{f}{x_n}}]$$

and that the gradient of a function always points towards the direction of maximal increase at that point.

Equivalently, it points *away* from the direction of maximum decrease - thus, if we start at any point, and keep moving in the direction of the negative gradient, we will eventually reach a local minimum.

This simple insight leads to the Gradient Descent algorithm. Outlined algorithmically, it looks like this:

1. Pick a point $x_0$ as your initial guess.
2. Compute the gradient at your current guess:
$v_i = \nabla f(x_i)$
3. Move by $\alpha$ (your step size) in the direction of that gradient:
$x_{i+1} = x_i + \alpha v_i$
4. Repeat steps 1-3 until your function is close enough to zero (until $f(x_i) < \varepsilon$ for some small tolerance $\varepsilon$)

Note that the step size, $\alpha$, is simply a parameter of the algorithm and has to be fixed in advance.

Notice that the hyperbolic tangent function asymptotes at -1 and 1, rather than 0 and 1, which is sometimes beneficial, and its derivative is simple:

$$\frac{d \tanh(x)}{dx} = 1 - \tanh^2(x)$$

Performing gradient descent will allow us to change the weights in the direction that optimially reduces the error. The next trick will be to employ the **chain rule** to decompose how the error changes as a function of the input weights into the change in error as a function of changes in the inputs to the weights, mutliplied by the changes in input values as a function of changes in the weights.

$$\frac{\partial E}{\partial w} = \frac{\partial E}{\partial h}\frac{\partial h}{\partial w}$$

This will allow us to write a function describing the activations of the output weights as a function of the activations of the hidden layer nodes and the output weights, which will allow us to propagate error backwards through the network.

The second term in the chain rule simplifies to:

$$\begin{align}
\frac{\partial h_k}{\partial w_{jk}} &= \frac{\partial \sum_l w_{lk} a_l}{\partial w_{jk}}  \\
&= \sum_l \frac{\partial w_{lk} a_l}{\partial w_{jk}} \\
& = a_j
\end{align}$$

where $a_j$ is the activation of the jth hidden layer neuron.

For the first term in the chain rule above, we decompose it as well:

$$\frac{\partial E}{\partial h_k} = \frac{\partial E}{\partial y_k}\frac{\partial y_k}{\partial h_k} = \frac{\partial E}{\partial g(h_k)}\frac{\partial g(h_k)}{\partial h_k}$$

The second term of this chain rule is just the derivative of the activation function, which we have chosen to have a conveneint form, while the first term simplifies to:

$$\frac{\partial E}{\partial g(h_k)} = \frac{\partial}{\partial g(h_k)}\left[\frac{1}{2} \sum_k (t_k - y_k)^2 \right] = t_k - y_k$$

Combining these, and assuming (for illustration) a logistic activiation function, we have the gradient:

$$\frac{\partial E}{\partial w} = (t_k - y_k) y_k (1-y_k) a_j$$

Which ends up getting plugged into the weight update formula that we saw in the single-layer perceptron:

$$w_{jk} \leftarrow w_{jk} - \eta (t_k - y_k) y_k (1-y_k) a_j$$

Note that here we are *subtracting* the second term, rather than adding, since we are doing gradient descent.

We can now outline the MLP learning algorithm:

1. Initialize all $w_{jk}$ to small random values
2. For each input vector, conduct forward propagation:
    * compute activation of each neuron $j$ in hidden layer (here, sigmoid):
    $$h_j = \sum_i x_i v_{ij}$$
    $$a_j = g(h_j) = \frac{1}{1 + \exp(-\beta h_j)}$$
    * when the output layer is reached, calculate outputs similarly:
    $$h_k = \sum_k a_j w_{jk}$$
    $$y_k = g(h_k) = \frac{1}{1 + \exp(-\beta h_k)}$$
3. Calculate loss for resulting predictions:
    * compute error at output:
    $$\delta_k = (t_k - y_k) y_k (1-y_k)$$
4. Conduct backpropagation to get partial derivatives of cost with respect to weights, and use these to update weights:
    * compute error of the hidden layers:
    $$\delta_{hj} = \left[\sum_k w_{jk} \delta_k \right] a_j(1-a_j)$$
    * update output layer weights:
    $$w_{jk} \leftarrow w_{jk} - \eta \delta_k a_j$$
    * update hidden layer weights:
    $$v_{ij} \leftarrow v_{ij} - \eta \delta_{hj} x_i$$

Return to (2) and iterate until learning completes. Best practice is to shuffle input vectors to avoid training in the same order.

Its important to be aware that because gradient descent is a hill-climbing (or descending) algorithm, it is liable to be caught in local minima with respect to starting values. Therefore, it is worthwhile training several networks using a range of starting values for the weights, so that you have a better chance of discovering a globally-competitive solution.

One useful performance enhancement for the MLP learning algorithm is the addition of **momentum** to the weight updates. This is just a coefficient on the previous weight update that increases the correlation between the current weight and the weight after the next update. This is particularly useful for complex models, where falling into local mimima is an issue; adding momentum will give some weight to the previous direction, making the resulting weights essentially a weighted average of the two directions. Adding momentum, along with a smaller learning rate, usually results in a more stable algorithm with quicker convergence. When we use momentum, we lose this guarantee, but this is generally seen as a small price to pay for the improvement momentum usually gives.

A weight update with momentum looks like this:

$$w_{jk} \leftarrow w_{jk} - \eta \delta_k a_j + \alpha \Delta w_{jk}^{t-1}$$

where $\alpha$ is the momentum (regularization) parameter and $\Delta w_{jk}^{t-1}$ the update from the previous iteration.

The multi-layer pereptron is implemented below in the `MLP` class. The implementation uses the scikit-learn interface, so it is uses in the same way as other supervised learning algorithms in that package.

Deep Learning/Reference Notebooks/01--Deep Learning Introduction Part 1.ipynb


# Cross-Resonance 门

*版权所有 (c) 2021 百度量子计算研究所，保留所有权利。*

## 内容概要

本教程介绍如何使用量脉生成 Cross-Resonance（CR）门的优化脉冲序列。与之前教程中的 iSWAP 和 CZ 门的实现不同，CR 门是使用全微波驱动实现的。本教程的概要如下：

- 背景介绍
- 准备工作
- 构造哈密顿量
- 通过量脉云服务生成与优化脉冲
- 总结

## 背景介绍

**基本原理**

与我们之前介绍的量子门不同的是，CR 门只使用微波来实现两个量子比特的相互作用，这样我们就可以避免由磁通而引起的噪声。CR 门的物理实现涉及两个耦合的频率固定的量子比特，并且使驱动脉冲的频率等于目标量子比特（Target Qubit）的频率，如下图所示：

我们先关注系统的有效哈密顿量（详见参考文献 \[1\]）。在双旋转坐标系中，我们用驱动强度 $A$、失谐量 $\Delta$、驱动相位 $\phi_0$ 和耦合强度 $g_{01}$ 表示 cross-resonance 效应的有效哈密顿量（为了简洁，我们令 $\hbar=1$）：

$$
\hat{H}_{\rm eff} = \frac{A}{4\Delta}g_{01}(\hat{\sigma}_0^z\hat{\sigma}_1^x\cos{\phi_0}+\hat{\sigma}_0^z\hat{\sigma}_1^y\sin{\phi_0}).
$$

当 $\phi_0=0$ 时，cross-resonance 效应产生了 $\hat{\sigma}^z_0\otimes\hat{\sigma}_1^x$ 有效耦合。因此，我们可以从上面的有效哈密顿量推导出时间演化算符：

$$
U_{\rm CR}(\theta)=e^{-i\frac{\theta}{2}\hat{\sigma}^z_0\otimes\hat{\sigma}^x_1},
$$

其中 $\theta=\Omega_0 g_{01}t/(2\Delta)$（$t$ 是门时间）。可见，cross-resonance 效应使得量子比特 $q_1$（目标量子比特）的旋转取决于量子比特 $q_0$（控制量子比特）的状态。

通过上面的推导, CR 门的矩阵形式是（有关更多详细信息，请参阅 \[2\]）：
$$
U_{\rm CR}(\theta) = \begin{bmatrix}
\cos{\frac{\theta}{2}} & -i\sin{\frac{\theta}{2}} & 0 & 0 \\
-i\sin{\frac{\theta}{2}} & \cos{\frac{\theta}{2}} & 0 & 0 \\
0 & 0 & \cos{\frac{\theta}{2}} & i\sin{\frac{\theta}{2}} \\
0 & 0 & i\sin{\frac{\theta}{2}} & \cos{\frac{\theta}{2}}
\end{bmatrix}.
$$

特别的，当 $\theta=-\frac{\pi}{2}$ 时，CR 门的矩阵表示为：

$$
U_{\rm CR}(-\pi/2) = \frac{\sqrt{2}}{2}
\begin{bmatrix}
1 & i & 0 & 0 \\
i & 1 & 0 & 0 \\
0 & 0 & 1 & -i \\
0 & 0 & -i & 1
\end{bmatrix}.
$$

**应用**

在分析了实现 CR 门的一些基本原理之后，我们现在关注 CR 门在量子计算中的应用：其中之一是通过增加两个额外的单量子比特门来实现 CNOT 门。

在本教程中，我们对由两个三能级的量子比特组成的系统进行建模，并以目标量子比特（量子比特 $q_1$）的频率向控制量子比特（量子比特 $q_0$）施加驱动脉冲。通过旋转波近似（RWA），哈密顿量可以表示为（更多细节请参阅 \[1\]）：

$$
\hat{H}_{\rm sys} = (\omega_{\rm q0}-\omega_{\rm d})\hat{a}_{0}^{\dagger }\hat{a}_0 + (\omega_{\rm q1}-\omega_{\rm d})\hat{a}_1^\dagger \hat{a}_1 + \frac{\alpha_0}{2} \hat{a}^{\dagger2}_0\hat{a}^2_0 + \frac{\alpha_1}{2} \hat{a}^{\dagger2}_1\hat{a}^2_1+\frac{g}{2}(\hat{a}_0\hat{a}_1^\dagger + \hat{a}_0^\dagger\hat{a}_1) + \Omega_0^x(t)\frac{\hat{a}^\dagger_0+\hat{a}_0}{2}.
$$

其中各种符号定义见下表：

|符号|定义|
|:--------:|:----------:|
|$\omega_{\rm qi}$| 量子比特 $q_i$ 的频率|
|$\omega_{\rm d}$|驱动频率|
|$\hat{a}_i^{\dagger}$|产生算符|
|$\hat{a}_i$|湮灭算符|
|$\alpha_i$| 量子比特 $q_i$ 的非谐性|
|$g$|耦合强度|
|$\Omega_0^x$(t)| X 通道的脉冲函数|

## 准备工作

成功安装量脉后，您可以按照本教程运行下面的量脉程序。要运行此教程，您需要从量脉（Quanlse）和其它常用的 Python 库导入以下包：

```python
# Import Hamiltonian-related module
from Quanlse.QHamiltonian import QHamiltonian as QHam
from Quanlse.QOperator import driveX, number, duff

# Import optimizer for the cross-resonance gate
from Quanlse.remoteOptimizer import remoteOptimizeCr

# Import tools to analyze the result
from Quanlse.Utils.Functions import project

# Import numpy and math
from numpy import round
from math import pi
```

## 构造哈密顿量

现在，我们需要使用量脉来构造哈密顿量。在量脉中，所有关于哈密顿量的信息都存储在一个字典中。我们首先定义一些构造哈密顿量的字典所需的基本参数：采样周期、系统中的量子比特数量和系统的能级。为了初始化该哈密顿量的字典，我们从模块 `QHamiltonian` 调用函数 `QHamiltonian()`。

```python
# Sampling period
dt = 1.0

# Number of qubits
qubits = 2

# System energy level
level = 3

# Initialize the Hamiltonian
ham = QHam(subSysNum=qubits, sysLevel=level, dt=dt)
```

现在我们可以开始构造哈密顿量了。在开始之前，我们需要定义几个常量作为函数的参数：

```python
# Parameters setting
qubitArgs = {
    "coupling": 0.0038 * (2 * pi),  # Coupling of Q0 and Q1
    "qubit_freq0": 5.114 * (2 * pi),  # Frequency of Q0
    "qubit_freq1": 4.914 * (2 * pi),  # Frequency of Q1
    "drive_freq0": 4.914 * (2 * pi),  # Drive frequency on Q0
    "drive_freq1": 4.914 * (2 * pi),  # Drive frequency on Q1
    "qubit_anharm0": -0.33 * (2 * pi),  # Anharmonicity of Q0
    "qubit_anharm1": -0.33 * (2 * pi)  # Anharmonicity of Q1
}
```

然后我们需要在之前初始化的哈密顿的字典中添加以下项：

$$
\begin{align}
\hat{H}_{\rm drift} &= (\omega_{\rm q0}-\omega_{\rm d}) \hat{a}_0^\dagger \hat{a}_0 + (\omega_{\rm q1}-\omega_{\rm d}) \hat{a}_1^\dagger \hat{a}_1 + \frac{\alpha_0}{2} \hat{a}_0^{\dagger}\hat{a}_0^{\dagger}\hat{a}_0
\hat{a}_0 + \frac{\alpha_1}{2} \hat{a}_1^{\dagger}\hat{a}_1^{\dagger}\hat{a}_1 \hat{a}_1 , \\
\hat{H}_{\rm coup} &= \frac{g_{01}}{2}(\hat{a}_0 \hat{a}_1^\dagger+\hat{a}^\dagger_0 \hat{a}_1). \\
\end{align}
$$

在量脉的 `Operator` 模块中，我们提供了一些允许用户快速构造常用算符的工具。失谐项 $(\omega_{\rm q}-\omega_{\rm d})\hat{a}^\dagger\hat{a}$ 和非谐项 $\frac{\alpha}{2}\hat{a}^\dagger\hat{a}^\dagger\hat{a}\hat{a}$ 可以分别使用 `Operator` 模块中的 `number(n)` 和 `duff(n)` 生成：这两个函数 `number(n)` 和 `duff(n)` 分别返回 $n\times n$ 的粒子数算符和 Duffing 算符。耦合项的形式为 $\frac{g}{2}(\hat{a}_i^\dagger\hat{a}_j+\hat{a}_i\hat{a}_j^\dagger)$，可以使用函数 `addCoupling()` 直接添加到哈密顿量中。

```python
# Add the detuning and the anharmonicity terms
for qu in range(2):
    # Add the detuning term(s).
    ham.addDrift(number, qu, (qubitArgs[f"qubit_freq{qu}"] - qubitArgs[f"drive_freq{qu}"]))
    # Add the anharmonicity term(s).
    ham.addDrift(duff, qu, qubitArgs[f"qubit_anharm{qu}"] / 2)

# Add the coupling term
ham.addCoupling([0, 1], qubitArgs["coupling"] / 2)
```

需要注意的是，量脉的优化函数会自动添加控制项：

$$
\hat{H}_{\rm ctrl}(t) = \Omega_0^x(t)\frac{\hat{a}^\dagger_0+\hat{a}_0}{2},
$$

所以我们不需要手动添加这项。

系统的哈密顿量构造完成后，我们可以进行量子系统的模拟。

## 通过量脉云服务生成与优化脉冲

在本地设备上处理优化过程通常需要很长时间，而我们提供的云服务可以显著加速此过程。要使用量脉云服务，用户需要从 http://quantum-hub.baidu.com 获取 token。

```python
# Import tools to get access to cloud service
from Quanlse import Define

# To use remoteOptimizerCr on cloud, paste your token (a string) here
Define.hubToken = ''
```

为了找到 CR 门的优化脉冲，我们使用函数 `remoteOptimizeCr()`。 此函数以我们先前定义的哈密顿量、振幅的界、门的持续时间、最大迭代次数和目标失真度。通过调用 `remoteOptimizeCr()`，用户可以向量脉云服务提交优化任务。如果用户想要进一步减少失真度，我们建议用户尝试增加门时间 `tg`（CR 门的持续时间约为 200 到 400 纳秒）。用户还可以通过设置更大的 `aBound` 和 `maxIter` 来增大搜索空间。

在本教程中，用来评估生成的量子门性能的失真度为 ${\rm infid} = 1 - \frac{1}{d}\left|{\rm Tr}[U^\dagger_{\rm goal}P(U)]\right|$，其中 $U_{\rm goal}$ 是目标酉变换 $U_{\rm CR}(-\pi /2)$，$d$ 是 $U_{\rm goal}$ 的维度，$U$ 是前面定义的三级体系的实际酉演化。注意，$P(U)$ 描述投影到计算子空间的演化。

```python
# Set amplitude bound
aBound = (-1.0, 3.0)

# Run the optimization
gateJob, infidelity = remoteOptimizeCr(ham, aBound=aBound, tg=200, maxIter=5, targetInfidelity=0.005)
```

我们可以使用 `plot()` 可视化生成的脉冲。有关 `plot()` 的详细信息，请参考 [single-qubit-gate](https://quanlse.baidu.com/#/doc/tutorial-single-qubit)。

```python
# Print waves and the infidelity
gateJob.plot()
print(f'infidelity: {infidelity}')
```

使用函数 `simulate()` 以及 `project()`，用户可以获得系统投影到计算子空间的演化后的矩阵 $P(U)$。

```python
# Print the system's evolution
result = ham.simulate(job=gateJob)
process2d = project(result[0]["unitary"], qubits, level, 2)
print("The projected evolution P(U):\n", round(process2d, 2))
```

此外，如果想要获取生成脉冲的每个时刻 `dt` 的数值，可以使用函数 `getPulseSequences()`。该函数采用哈密顿的字典和脉冲的通道名称作为输入参数。

```python
gateJob.generatePulseSequence(driveX(3), 0)
gateJob.plot()
```

## 总结

通过构造系统哈密顿量并在量脉云服务上产生优化的脉冲，我们成功地设计了一个可以实现高保真度的 cross-resonance 门的脉冲。用户可以通过点击这个链接 [tutorial-cr-gate.ipynb](https://github.com/baidu/Quanlse/blob/main/Tutorial/CN/tutorial-cr-cn.ipynb) 跳转到此 Jupyter Notebook 文档相应的 GitHub 页面并且运行这个程序。我们鼓励用户尝试不同于本教程的参数值以获得最佳结果。

## 参考文献

\[1\] [Rigetti, Chad, and Michel Devoret. "Fully microwave-tunable universal gates in superconducting qubits with linear couplings and fixed transition frequencies." *Physical Review B* 81.13 (2010): 134507.](https://journals.aps.org/prb/abstract/10.1103/PhysRevB.81.134507)

\[2\] [Nielsen, Michael A., and Isaac L. Chuang. Quantum Computation and Quantum Information: 10th Anniversary Edition. Cambridge University Press, 2010.](https://doi.org/10.1017/CBO9780511976667)