# Detrending, Stylized Facts and the Business Cycle

In an influential article, Harvey and Jaeger (1993) described the use of unobserved components models (also known as "structural time series models") to derive stylized facts of the business cycle.

Their paper begins:

    "Establishing the 'stylized facts' associated with a set of time series is widely considered a crucial step
    in macroeconomic research ... For such facts to be useful they should (1) be consistent with the stochastic
    properties of the data and (2) present meaningful information."

In particular, they make the argument that these goals are often better met using the unobserved components approach rather than the popular Hodrick-Prescott filter or Box-Jenkins ARIMA modeling techniques.

statsmodels has the ability to perform all three types of analysis, and below we follow the steps of their paper, using a slightly updated dataset.

```python
%matplotlib inline

import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt

from IPython.display import display, Latex
```

## Unobserved Components

The unobserved components model available in statsmodels can be written as:

$$
y_t = \underbrace{\mu_{t}}_{\text{trend}} + \underbrace{\gamma_{t}}_{\text{seasonal}} + \underbrace{c_{t}}_{\text{cycle}} + \sum_{j=1}^k \underbrace{\beta_j x_{jt}}_{\text{explanatory}} + \underbrace{\varepsilon_t}_{\text{irregular}}
$$

see Durbin and Koopman 2012, Chapter 3 for notation and additional details. Notice that different specifications for the different individual components can support a wide range of models. The specific models considered in the paper and below are specializations of this general equation.

### Trend

The trend component is a dynamic extension of a regression model that includes an intercept and linear time-trend.

$$
\begin{align}
\underbrace{\mu_{t+1}}_{\text{level}} & = \mu_t + \nu_t + \eta_{t+1} \qquad & \eta_{t+1} \sim N(0, \sigma_\eta^2) \\\\
\underbrace{\nu_{t+1}}_{\text{trend}} & = \nu_t + \zeta_{t+1} & \zeta_{t+1} \sim N(0, \sigma_\zeta^2) \\
\end{align}
$$

where the level is a generalization of the intercept term that can dynamically vary across time, and the trend is a generalization of the time-trend such that the slope can dynamically vary across time.

For both elements (level and trend), we can consider models in which:

- The element is included vs excluded (if the trend is included, there must also be a level included).
- The element is deterministic vs stochastic (i.e. whether or not the variance on the error term is confined to be zero or not)

The only additional parameters to be estimated via MLE are the variances of any included stochastic components.

This leads to the following specifications:

|                                                                      | Level | Trend | Stochastic Level | Stochastic Trend |
|----------------------------------------------------------------------|-------|-------|------------------|------------------|
| Constant                                                             | ✓     |       |                  |                  |
| Local Level
 (random walk)                                     | ✓     |       | ✓                |                  |
| Deterministic trend                                                  | ✓     | ✓     |                  |                  |
| Local level with deterministic trend
 (random walk with drift) | ✓     | ✓     | ✓                |                  |
| Local linear trend                                                   | ✓     | ✓     | ✓                | ✓                |
| Smooth trend
 (integrated random walk)                         | ✓     | ✓     |                  | ✓                |

### Seasonal

The seasonal component is written as:

$$
\gamma_t = - \sum_{j=1}^{s-1} \gamma_{t+1-j} + \omega_t \qquad \omega_t \sim N(0, \sigma_\omega^2)
$$

The periodicity (number of seasons) is `s`, and the defining character is that (without the error term), the seasonal components sum to zero across one complete cycle. The inclusion of an error term allows the seasonal effects to vary over time.

The variants of this model are:

- The periodicity `s`
- Whether or not to make the seasonal effects stochastic.

If the seasonal effect is stochastic, then there is one additional parameter to estimate via MLE (the variance of the error term).

### Cycle

The cyclical component is intended to capture cyclical effects at time frames much longer than captured by the seasonal component. For example, in economics the cyclical term is often intended to capture the business cycle, and is then expected to have a period between "1.5 and 12 years" (see Durbin and Koopman).

The cycle is written as:

$$
\begin{align}
c_{t+1} & = c_t \cos \lambda_c + c_t^* \sin \lambda_c + \tilde \omega_t \qquad & \tilde \omega_t \sim N(0, \sigma_{\tilde \omega}^2) \\\\
c_{t+1}^* & = -c_t \sin \lambda_c + c_t^* \cos \lambda_c + \tilde \omega_t^* & \tilde \omega_t^* \sim N(0, \sigma_{\tilde \omega}^2)
\end{align}
$$

The parameter $\lambda_c$ (the frequency of the cycle) is an additional parameter to be estimated by MLE. If the seasonal effect is stochastic, then there is one another parameter to estimate (the variance of the error term - note that both of the error terms here share the same variance, but are assumed to have independent draws).

### Irregular

The irregular component is assumed to be a white noise error term. Its variance is a parameter to be estimated by MLE; i.e.

$$
\varepsilon_t \sim N(0, \sigma_\varepsilon^2)
$$

In some cases, we may want to generalize the irregular component to allow for autoregressive effects:

$$
\varepsilon_t = \rho(L) \varepsilon_{t-1} + \epsilon_t, \qquad \epsilon_t \sim N(0, \sigma_\epsilon^2)
$$

In this case, the autoregressive parameters would also be estimated via MLE.

### Regression effects

We may want to allow for explanatory variables by including additional terms

$$
\sum_{j=1}^k \beta_j x_{jt}
$$

or for intervention effects by including

$$
\begin{align}
\delta w_t \qquad \text{where} \qquad w_t & = 0, \qquad t < \tau, \\\\
& = 1, \qquad t \ge \tau
\end{align}
$$

These additional parameters could be estimated via MLE or by including them as components of the state space formulation.

## Data

Following Harvey and Jaeger, we will consider the following time series:

- US real GNP, "output", ([GNPC96](https://research.stlouisfed.org/fred2/series/GNPC96))
- US GNP implicit price deflator, "prices", ([GNPDEF](https://research.stlouisfed.org/fred2/series/GNPDEF))
- US monetary base, "money", ([AMBSL](https://research.stlouisfed.org/fred2/series/AMBSL))

The time frame in the original paper varied across series, but was broadly 1954-1989. Below we use data from the period 1948-2008 for all series. Although the unobserved components approach allows isolating a seasonal component within the model, the series considered in the paper, and here, are already seasonally adjusted.

All data series considered here are taken from [Federal Reserve Economic Data (FRED)](https://research.stlouisfed.org/fred2/). Conveniently, the Python library [Pandas](https://pandas.pydata.org/) has the ability to download data from FRED directly.

```python
# Datasets
from pandas_datareader.data import DataReader

# Get the raw data
start = '1948-01'
end = '2008-01'
us_gnp = DataReader('GNPC96', 'fred', start=start, end=end)
us_gnp_deflator = DataReader('GNPDEF', 'fred', start=start, end=end)
us_monetary_base = DataReader('AMBSL', 'fred', start=start, end=end).resample('QS').mean()
recessions = DataReader('USRECQ', 'fred', start=start, end=end).resample('QS').last().values[:,0]

# Construct the dataframe
dta = pd.concat(map(np.log, (us_gnp, us_gnp_deflator, us_monetary_base)), axis=1)
dta.columns = ['US GNP','US Prices','US monetary base']
dta.index.freq = dta.index.inferred_freq
dates = dta.index._mpl_repr()
```

To get a sense of these three variables over the timeframe, we can plot them:

```python
# Plot the data
ax = dta.plot(figsize=(13,3))
ylim = ax.get_ylim()
ax.xaxis.grid()
ax.fill_between(dates, ylim[0]+1e-5, ylim[1]-1e-5, recessions, facecolor='k', alpha=0.1);
```

## Model

Since the data is already seasonally adjusted and there are no obvious explanatory variables, the generic model considered is:

$$
y_t = \underbrace{\mu_{t}}_{\text{trend}} + \underbrace{c_{t}}_{\text{cycle}} + \underbrace{\varepsilon_t}_{\text{irregular}}
$$

The irregular will be assumed to be white noise, and the cycle will be stochastic and damped. The final modeling choice is the specification to use for the trend component. Harvey and Jaeger consider two models:

1. Local linear trend (the "unrestricted" model)
2. Smooth trend (the "restricted" model, since we are forcing $\sigma_\eta = 0$)

Below, we construct `kwargs` dictionaries for each of these model types. Notice that rather that there are two ways to specify the models. One way is to specify components directly, as in the table above. The other way is to use string names which map to various specifications.

```python
# Model specifications

# Unrestricted model, using string specification
unrestricted_model = {
    'level': 'local linear trend', 'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
}

# Unrestricted model, setting components directly
# This is an equivalent, but less convenient, way to specify a
# local linear trend model with a stochastic damped cycle:
# unrestricted_model = {
#     'irregular': True, 'level': True, 'stochastic_level': True, 'trend': True, 'stochastic_trend': True,
#     'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
# }

# The restricted model forces a smooth trend
restricted_model = {
    'level': 'smooth trend', 'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
}

# Restricted model, setting components directly
# This is an equivalent, but less convenient, way to specify a
# smooth trend model with a stochastic damped cycle. Notice
# that the difference from the local linear trend model is that
# `stochastic_level=False` here.
# unrestricted_model = {
#     'irregular': True, 'level': True, 'stochastic_level': False, 'trend': True, 'stochastic_trend': True,
#     'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
# }
```

We now fit the following models:

1. Output, unrestricted model
2. Prices, unrestricted model
3. Prices, restricted model
4. Money, unrestricted model
5. Money, restricted model

```python
# Output
output_mod = sm.tsa.UnobservedComponents(dta['US GNP'], **unrestricted_model)
output_res = output_mod.fit(method='powell', disp=False)

# Prices
prices_mod = sm.tsa.UnobservedComponents(dta['US Prices'], **unrestricted_model)
prices_res = prices_mod.fit(method='powell', disp=False)

prices_restricted_mod = sm.tsa.UnobservedComponents(dta['US Prices'], **restricted_model)
prices_restricted_res = prices_restricted_mod.fit(method='powell', disp=False)

# Money
money_mod = sm.tsa.UnobservedComponents(dta['US monetary base'], **unrestricted_model)
money_res = money_mod.fit(method='powell', disp=False)

money_restricted_mod = sm.tsa.UnobservedComponents(dta['US monetary base'], **restricted_model)
money_restricted_res = money_restricted_mod.fit(method='powell', disp=False)
```

Once we have fit these models, there are a variety of ways to display the information. Looking at the model of US GNP, we can summarize the fit of the model using the `summary` method on the fit object.

```python
print(output_res.summary())
```

                                Unobserved Components Results
    =====================================================================================
    Dep. Variable:                        US GNP   No. Observations:                  241
    Model:                    local linear trend   Log Likelihood                 769.632
                       + damped stochastic cycle   AIC                          -1527.264
    Date:                       Tue, 02 Feb 2021   BIC                          -1506.456
    Time:                               07:02:49   HQIC                         -1518.877
    Sample:                           01-01-1948
                                    - 01-01-2008
    Covariance Type:                         opg
    ====================================================================================
                           coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------------
    sigma2.irregular  1.091e-06   7.29e-06      0.150      0.881   -1.32e-05    1.54e-05
    sigma2.level      4.091e-06   4.93e-05      0.083      0.934   -9.26e-05       0.000
    sigma2.trend      2.961e-06    1.4e-06      2.112      0.035    2.14e-07    5.71e-06
    sigma2.cycle       3.91e-05   2.56e-05      1.524      0.127   -1.12e-05    8.94e-05
    frequency.cycle      0.4454      0.047      9.478      0.000       0.353       0.537
    damping.cycle        0.8684      0.042     20.732      0.000       0.786       0.951
    ===================================================================================
    Ljung-Box (L1) (Q):                   0.00   Jarque-Bera (JB):                 9.29
    Prob(Q):                              0.96   Prob(JB):                         0.01
    Heteroskedasticity (H):               0.27   Skew:                            -0.05
    Prob(H) (two-sided):                  0.00   Kurtosis:                         3.96
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using the outer product of gradients (complex-step).

For unobserved components models, and in particular when exploring stylized facts in line with point (2) from the introduction, it is often more instructive to plot the estimated unobserved components (e.g. the level, trend, and cycle) themselves to see if they provide a meaningful description of the data.

The `plot_components` method of the fit object can be used to show plots and confidence intervals of each of the estimated states, as well as a plot of the observed data versus the one-step-ahead predictions of the model to assess fit.

```python
fig = output_res.plot_components(legend_loc='lower right', figsize=(15, 9));
```

Finally, Harvey and Jaeger summarize the models in another way to highlight the relative importances of the trend and cyclical components; below we replicate their Table I. The values we find are broadly consistent with, but different in the particulars from, the values from their table.

```python
# Create Table I
table_i = np.zeros((5,6))

start = dta.index[0]
end = dta.index[-1]
time_range = '%d:%d-%d:%d' % (start.year, start.quarter, end.year, end.quarter)
models = [
    ('US GNP', time_range, 'None'),
    ('US Prices', time_range, 'None'),
    ('US Prices', time_range, r'$\sigma_\eta^2 = 0$'),
    ('US monetary base', time_range, 'None'),
    ('US monetary base', time_range, r'$\sigma_\eta^2 = 0$'),
]
index = pd.MultiIndex.from_tuples(models, names=['Series', 'Time range', 'Restrictions'])
parameter_symbols = [
    r'$\sigma_\zeta^2$', r'$\sigma_\eta^2$', r'$\sigma_\kappa^2$', r'$\rho$',
    r'$2 \pi / \lambda_c$', r'$\sigma_\varepsilon^2$',
]

i = 0
for res in (output_res, prices_res, prices_restricted_res, money_res, money_restricted_res):
    if res.model.stochastic_level:
        (sigma_irregular, sigma_level, sigma_trend,
         sigma_cycle, frequency_cycle, damping_cycle) = res.params
    else:
        (sigma_irregular, sigma_level,
         sigma_cycle, frequency_cycle, damping_cycle) = res.params
        sigma_trend = np.nan
    period_cycle = 2 * np.pi / frequency_cycle

    table_i[i, :] = [
        sigma_level*1e7, sigma_trend*1e7,
        sigma_cycle*1e7, damping_cycle, period_cycle,
        sigma_irregular*1e7
    ]
    i += 1

pd.set_option('float_format', lambda x: '%.4g' % np.round(x, 2) if not np.isnan(x) else '-')
table_i = pd.DataFrame(table_i, index=index, columns=parameter_symbols)
table_i
```

      $\sigma_\zeta^2$
      $\sigma_\eta^2$
      $\sigma_\kappa^2$
      $\rho$
      $2 \pi / \lambda_c$
      $\sigma_\varepsilon^2$

      Series
      Time range
      Restrictions

      US GNP
      1948:1-2008:1
      None
      40.91
      29.61
      390.9
      0.87
      14.11
      10.91

      US Prices
      1948:1-2008:1
      None
      49.17
      31.36
      0.01
      0.92
      10
      5.89

      $\sigma_\eta^2 = 0$
      21.78
      NaN
      49.66
      0.89
      15.37
      4.93

      US monetary base
      1948:1-2008:1
      None
      69.19
      18.83
      192.6
      0.89
      23.21
      0

      $\sigma_\eta^2 = 0$
      18.91
      NaN
      247.5
      0.89
      23.83
      0


# 13 Root Finding (Students)

An important tool in the computational tool box is to find roots of equations for which no closed form solutions exist:

We want to find the roots $x_0$ of

$$
f(x_0) = 0
$$

## Problem: Projectile range
The equations of motion for the projectile with linear air resistance (see *12 ODE applications*) can be solved exactly.

As a reminder: the linear drag force is
$$
\mathbf{F}_1 = -b_1 \mathbf{v}\\
b := \frac{b_1}{m}
$$

Equations of motion with force due to gravity $\mathbf{g} = -g \hat{\mathbf{e}}_y$

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= - g \hat{\mathbf{e}}_y -b \mathbf{v}
\end{align}

### Analytical solution of the equations of motion
(Following Wang Ch 3.3.2)

Solve $x$ component of the velocity

$$
\frac{dv_x}{dt} = -b v_x
$$

by integration:

$$
v_x(t) = v_{0x} \exp(-bt)
$$

The drag force reduces the forward velocity to 0.

Integrate again to get the $x(t)$ component

$$
x(t) = x_0 + \frac{v_{0x}}{b}\left[1 - \exp(-bt)\right]
$$

Integrating the $y$ component of the velocity

$$
\frac{dv_y}{dt} = -g - b v_y
$$

gives

$$
v_y = \left(v_{0y} + \frac{g}{b}\right) \exp(-bt) - \frac{g}{b}
$$

and integrating again

$$
y(t) = y_0 + \frac{v_{0y} + \frac{g}{b}}{b} \left[1 - \exp(-bt)\right] - \frac{g}{b} t
$$

(Note: This shows immediately that the *terminal velocity* is

$$
\lim_{t\rightarrow\infty} v_y(t) = - \frac{g}{b},
$$

i.e., the force of gravity is balanced by the drag force.)

#### Analytical trajectory

To obtain the **trajectory $y(x)$** eliminate time (and for convenience, using the origin as the initial starting point, $x_0 = 0$ and $y_0 = 0$. Solve $x(t)$ for $t$

$$
t = -\frac{1}{b} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

and insert into $y(t)$:

$$
y(x) = \frac{x}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

#### Plot

Plot the analytical solution $y(x)$ for $\theta = 30^\circ$ and $v_0 = 100$ m/s.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
```

The function `y_lindrag()` should compute $y(x)$.

```python
def y_lindrag(x, v0, b1=0.2, g=9.81, m=0.5):
    b = b1/m
    v0x, v0y = v0
    # IMPLEMENT FUNCTION

def initial_v(v, theta):
    x = np.deg2rad(theta)
    return v * np.array([np.cos(x), np.sin(x)])
```

```python
X = np.concatenate([np.linspace(0, 42, 100), np.linspace(42, 45, 1000)])
Y = y_lindrag(X, initial_v(100, 30), b1=1)
```

```python
# PLOT
```

Compare to the numerical solution (from **12 ODE Applications**):

```python
import ode

def simulate(v0, h=0.01, b1=0.2, g=9.81, m=0.5):
    def f(t, y):
        # y = [x, y, vx, vy]
        return np.array([y[2], y[3], -b1/m * y[2], -g - b1/m * y[3]])

    vx, vy = v0
    t = 0
    positions = []
    y = np.array([0, 0, vx, vy], dtype=np.float64)

    while y[1] >= 0:
        positions.append([t, y[0], y[1]])  # record t, x and y
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)
```

```python
r = simulate(initial_v(100, 30), h=0.01, b1=1)
```

```python
plt.plot(X, Y, lw=2, label="analytical")
plt.plot(r[:, 1], r[:, 2], '--', label="RK4")
plt.legend(loc="best")
plt.xlabel("$x$ (m)"); plt.ylabel("$y$ (m)")
```

### Predict the range $R$
How far does the ball or projectile fly, i.e., that value $x=R$ where $y(R) = 0$:

$$
\frac{R}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bR}{v_{0x}}\right) = 0
$$

This *transcendental equation* can not be solved in terms of elementary functions.

Use a **root finding** algorithm.

## Root-finding with the Bisection algorithm
**Bisection** is the simplest (but very robust) root finding algorithm that uses trial-and-error:

* bracket the root
* refine the brackets
* see [13_Root-finding-algorithms (PDF)](13_Root-finding-algorithms.pdf)

More specifically:
1. determine a bracket that contains the root: $a < x_0 < b$ (i.e., an interval $[a, b]$ with $f(a) > 0$ and $f(b) < 0$ or $f(a) < 0$ and $f(b) > 0$)
2. cut bracket in half: $x' = \frac{1}{2}(a + b)$
3. determine in which half the root lies: either in $[a, x']$ or in $[x', b]$: If $f(a) f(x') > 0$ then the root lies in the right half $[x', b]$, otherwise the left half $[a, x']$.
4. Change the boundaries $a$ or $b$.
5. repeat until $|f(x')| < \epsilon$.

### Implementation of Bisection

- Test that the initial bracket contains a root; if not, return `None` (and possibly print a warning).
- If either of the bracket points is a root then return the bracket point.
- Allow `Nmax` iterations or until the convergence criterion `eps` is reached.
- Bonus: print a message if no root was found after `Nmax` iterations, but print the best guess and the error (but return `None`).

```python
def bisection(f, a, b, Nmax=100, eps=1e-14):
    # IMPLEMENT FUNCTION
```

### Finding the range with the bisection algorithm

Define the trial function:

```python
def f(x):
    v0 = initial_v(100, 30)
    b1 = 1.
    return y_lindrag(x, v0, b1=b1)
```

The initial bracket $[a_\text{initial}, b_\text{initial}]$ is a little bit difficult for this function: choose the right bracket near the point where the argument of the logarithm becomes 0 (which is actually the maximum $x$ value $\lim_{t\rightarrow +\infty} x(t) = \frac{v_{0x}}{b}$):

$$
b_\text{initial} = \frac{v_{0x}}{b} - \epsilon'
$$

where $\epsilon'$ is a small number.

```python
v = initial_v(100, 30)
b1 = 1.
m = 0.5
b = b1/m
# COMPLETE: bisection( , eps=1e-6)
```

### Find the range as a function of the initial angle

```python
b1 = 1.
m = 0.5
b = b1/m
v0 = 100
u = []

# IMPLEMENT

```

```python
# PLOT
```

Write a function `find_range()` to calculate the range for a given initial velocity $v_0$ and plot $R(\theta)$ for $10\,\text{m/s} ≤ v_0 ≤ 100\,\text{m/s}$.

```python
def find_range(v0, b1=1, m=0.5):
    b = b1/m
    u = []
    for theta in np.arange(1, 90):
        v = initial_v(v0, theta)
        # IMPLEMENT THE REST ...
    return np.array(u)
```

```python
for v0 in (10, 25, 50, 75, 100):
    u = find_range(v0)
    plt.plot(u[:, 0], u[:, 1], label="{} m/s".format(v0))
plt.legend(loc="best")
plt.xlabel(r"$\theta$ (degrees)")
plt.ylabel(r"range $R$ (m)")
```

## Newton-Raphson algorithm
(see derivation in class and in the PDF or [Newton's Method](http://mathworld.wolfram.com/NewtonsMethod.html) on MathWorld)

### Activity: Implement Newton-Raphson
1. Implement the Newton-Raphson algorithm
2. Test with $g(x)$.

   $$
   g(x) = 2 \cos x - x
   $$

3. Bonus: test performance of `newton_raphson()` against `bisection()`.

```python
def g(x):
    return 2*np.cos(x) - x
```

```python
xvals = np.linspace(0, 7, 30)
plt.plot(xvals, np.zeros_like(xvals), 'k--')
plt.plot(xvals, g(xvals))
```

```python
def newton_raphson(f, x, h=3e-1, Nmax=100, eps=1e-14):
    # IMPLEMENT ME
```

```python
newton_raphson(g, 0)
```

```python

```


# Backpropagation

- Arguably, Backpropagation is one of the most important algorithms today. Among other people, Geoffrey Hinton is responsible for making this algorithm popular.

**Backpropagation is an extension of the gradient calculation in the Gradient Descent algorithms.**

Like in Gradient Descent, the gradient of a loss function is used to calculate by how much to update the weigths in the layers.

The information about the gradient is then sent backwards through the network. For each layer, a separate gradient is calculated based on the gradient of the later layers.

### Let's make an Artificial Neural Network (ANN), from scratch, so that we can understand the concepts better, step-by-step.
- **We'll start with making some random data with the make_moons method in Scikit Learn**

## Implementing a Feed-Forward-Network

### Step 1 - Create some training data:

```python
import numpy as np
from sklearn.datasets import make_moons
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
X, y = make_moons(n_samples=50, noise=0.2, random_state=42)
```

```python
X.shape
```

    (50, 2)

```python
y.shape
```

    (50,)

### Step 2 - Create a Scatterplot and colour by y

```python
# selects everything from column 0!
# X[:,0]
# selects everything from column 1!
# X[:,1]
```

```python
plt.scatter(X[:,0], X[:,1], c=y)
#c is the colour parameter, and y is an array of 1s and 0s
```

### Step 3 - Use Numpy to add an extra column for the bias to your input:

```python
X = np.hstack([X, np.ones((X.shape[0], 1))]) # This hstack just adds an extra column!
```

```python
X[:5]
```

    array([[-0.15856989,  0.98480618,  1.        ],
           [ 1.10900374, -0.62160223,  1.        ],
           [ 0.39641523,  0.20740869,  1.        ],
           [ 1.79239122,  0.07511293,  1.        ],
           [-0.84739307,  0.71154296,  1.        ]])

```python
X.shape
```

    (50, 3)

- hstack does the same thing as pandas concat, horizontally attaching a column of ones to the input data, X
- we can now treat the bias as an additional input

#### np.vstack would add a row of 1s!

### Step 4 - Implement a sigmoid function that accepts NumPy arrays:
- Just a quick note to make: ANNs are actually used primarily as a *supervised learning* method (well, at least in this class)

```python
def sigmoid(x):
    '''sigmoid function that accepts numpy arrays '''

    sig = 1 / (1 + np.e**(-x))

    return sig
```

#### And test it:

```python
a = np.array([-10.0, -1.0, 0.0, 1.0, 10.0])
expected = np.array([0.0, 0.27, 0.5, 0.73, 1.0])
assert np.all(sigmoid(a).round(2) == expected)
```

- if the results of our sigmoid function are within 2 decimals rounding from the expected results,then we know it's calculating correctly

#### Let's remind ourselves of what the sigmoid function actually does:

```python
b = np.array([-99])
sigmoid(b)
```

    array([1.01122149e-43])

### Step 5 - Initialize the network with random weights (for both the hidden and output layers)

We need:

- inner weights: (3, 2)
- outer weights: (3, 1)
    - The first dimension represents the number of inputs!
    - The second dimension represents the number of neurons!

**Hint:** If we represent each layers’ weights with a numpy array, what should its shape be, given that – for example – the first layer has 3 inputs that feed 2 neurons?

```python
# creating random weights
inner_weights = np.random.normal(0.0, 1.0, size=(3, 2))
outer_weights = np.random.normal(0.0, 1.0, size=(3, 1))
```

```python
inner_weights
```

    array([[ 0.2491557 ,  0.69894957],
           [ 0.99885573, -0.34461507],
           [-0.33067308, -0.21645169]])

```python
outer_weights
```

    array([[-1.87113825],
           [ 0.51101215],
           [-0.53039483]])

```python
weights = [inner_weights, outer_weights]
```

```python
weights[0]
```

    array([[ 0.2491557 ,  0.69894957],
           [ 0.99885573, -0.34461507],
           [-0.33067308, -0.21645169]])

- it's really important that the weights are in this shape! i.e. the first element in the tuple (3) is the number of input weights (X1, X2, bias)
- and the second element in the tuple (2 or 1) represents the number of neurons in the first and second layers, respectively.

### Step 6 - Implement a feed-forward function that replicates the feed-forward network illustrated in the figure above

```python
def feed_forward(X, weights):

    # 1. calculate the dot product of X (50,3)
    # and the weights of the first layer (3, 2)
    #results in output shape of (50,2)

    dot_product = np.dot(X, weights[0])

    # 2. apply the sigmoid function on the result
    # applying sigmoid results of (50,2) matrix gives us (50,2)

    output1 = sigmoid(dot_product)

    # 3. append an extra columns of 1s for the bias to the result (see Step 3)
    # results in (50, 3) matrix

    hidden_X = np.hstack([output1, np.ones((output1.shape[0], 1))])

    # 4. calculate the dot product of the hidden output X (50,3)
    # and the weights of the second layer (3, 1) - results in (50,1)

    hidden_dot =  np.dot(hidden_X, weights[1])

    # 5. apply the sigmoid function on the result
    # don't forget to append bias column!

    output2 = sigmoid(hidden_dot)

    #6. return all intermediate results

    return output1, output2
```

#### And test the ANN F-F function:

```python
out1, out2 = feed_forward(X, weights)
assert out1.shape == (50, 2)
assert out2.shape == (50, 1)
```

```python
Xref = np.array([[1.0, 2.0, 1.0]])
whidden = np.array([[1.0, 2.0, 0.0],
                    [-1.0, -2.0, 0.0]]).T

wout = np.array([1.0, -1.0, 0.5]).T

w = [whidden, wout]

output1, output2 = feed_forward(Xref, w)
assert np.all(output1.round(2) == np.array([[0.99, 0.01]]))
# assert np.all(output2.round(2) == np.array([0.82])
```

### A nice way to manually calculate the accuracy of the initial guess:
- flatten both ytrue and ypred (so that they're both (50,) arrays)
- round the ypred (either up to 1 or down to 0)
- then compare them to eachother to get a boolean array (True where the values are the same, False where they are not)
- sum the boolean array and divide by the length (i.e. take the mean)

```python
def get_accuracy(ypred,ytrue):

    acc = sum(ytrue.flatten() == ypred.flatten().round().astype(np.int64))

    return acc/len(ytrue)
```

```python
get_accuracy(out2, y)
```

    0.5

#### 50% is our baseline - let's hope we can improve it!

## Backpropagation

So far, we have just put in random numbers and fed forward into our network. Now we will work backwards to work out the really large errors, and to establish what the weights should be!

Backpropagation is an extension of the gradient calculation in the Gradient Descent algorithms.

Like in Gradient Descent, the gradient of a loss function is used to calculate by how much to update the weigths in the layers.

The information about the gradient is then sent backwards through the network. For each layer, a separate gradient is calculated based on the gradient of the later layers.

### Step 1 -  Run your code for the Feed-Forward-Network, make sure it works

### Step 2 - Create small random weights for both layers of the network:

- create separate weights for both layers
- include the biase in the weights
- use the numpy.random functions
- to speed up training, assign weights in the range:

\begin{align}
w \approx \pm \sqrt{\frac{1}{n_{neurons}}}
\end{align}

### Step 3 - Implement a function that calculates a simple error from the output:

\begin{align}
error = \hat y - y_{true}
\end{align}

In our case, our ${\hat y}$ is our second output (out2) and $y_{true}$ is the original y values from the make moons dataset!

```python
y = y.reshape(50, 1)
```

```python
def diff_func(ypred, ytrue):

    diff = ypred - ytrue

    return diff
```

```python
diff = diff_func(out2, y)
```

```python
diff.shape
```

    (50, 1)

### Step 4 - Implement a Log Loss function
\begin{align}
loss = \sum -(y_{true} * log(y_{pred}) + (1 - y_{true}) * log(1 - y{pred}))
\end{align}

The function should take the true output and a predicted probability and returns the sum of log loss values over all data points

```python
def log_loss(ytrue, ypred):

    logloss = sum(-(ytrue * np.log(ypred) + (1-ytrue) * np.log(1-ypred)))

    return logloss
```

```python
log_loss(y, out2)
```

    array([40.08035897])

```python
log_loss(y, out2).shape
```

    (1,)

#### Test the function with an assertion:

```python
ytrue = np.array([0.00, 0.00, 1.00, 1.00])
ypred = np.array([0.01, 0.99, 0.01, 0.99])
assert round(log_loss(ytrue, ypred), 2) == 9.23
```

```python
error = diff_func(out2, y) * log_loss(y, out2)
```

```python
error.shape
```

    (50, 1)

```python
sum(error)
```

    array([-539.99162595])

```python
def error_function(ypred, ytrue):

    diff = ypred - ytrue

    logloss = sum(-(ytrue * np.log(ypred) + (1-ytrue) * np.log(1-ypred)))

    error = diff * logloss

    return error
```

```python
sum(error_function(out2, y))
```

    array([-539.99162595])

**Note:** With a ypred of exactly 1.0 or 0.0 the log loss function will return a **NaN value**.

### Step 5 - Implement the first part of the backpropagation algorithm according to the equations:

\begin{align}\begin{aligned}\bigtriangledown_{out} = out_{hidden} \bullet error\\w_{out} \leftarrow w_{out} + \bigtriangledown_{out} * LR\end{aligned}\end{align}

Write the code that modifies the weights of the output layer:

1. Initialize the weights
2. Calculate the output of both layers for each data point
3. Calculate the error for each data point
4. Add a bias to the hidden output
5. Calculate weight modifications for the output layer by the equations above
    - Modify each weight of the output layer by `-sigmoid'(output) * loss * hidden_output`

6. Apply the weight modification for each data point separately.

#### Execute the function once. This is Step B!

```python
# def backpropagationB(ypred):

#     gradient_y = ypred * (1 - ypred) * error

#     return gradient_y
```

```python
# grad_y = backpropagationB(out2)
```

```python
# grad_y.shape
```

```python
# out1_with_bias = np.hstack([out1, np.ones((out1.shape[0], 1))])
```

```python
# out1_with_bias.shape
```

```python
# LR= 0.05
```

```python
#this will error, need to transpose!
# delta_wo = np.dot(-grad_y, out1_with_bias) * LR
```

#### You're going to get an error, we need to transpose!

```python
# delta_wo = np.dot(-grad_y.T, out1_with_bias) * LR
```

```python
# delta_wo
```

```python
# def backpropagation1(ypred):

#     gradient_y = ypred * (1 - ypred) * error

#     out1_with_bias = np.hstack([out1, np.ones((out1.shape[0], 1))])

#     LR = 0.05

#     delta_wo = np.dot(-gradient_y.T, out1_with_bias) * LR

#     return delta_wo
```

```python
# backpropagation1(out2)
```

```python
# def backpropagation1(ypred, ytrue, hidden_out, LR=0.05):

#     diff = ypred - ytrue
#     logloss = sum(-(ytrue * np.log(ypred) + (1-ytrue) * np.log(1-ypred)))
#     error = diff * logloss

#     gradient_y = ypred * (1 - ypred) * error
#     out_with_bias = np.hstack([hidden_out, np.ones((hidden_out.shape[0], 1))])
#     delta_wo = np.dot(-gradient_y.T, out1_with_bias) * LR

#     new_weights = weights[1] + delta_wo.T

#     return new_weights
```

```python
# new_weights = [weights[0],backpropagation1(out2, y, out1)]
# new_weights
```

```python
# new_weights = [wH_new, wO_new]
```

```python
# new_weights[0].shape, new_weights[1].shape
# #shapes look right!!
```

```python
def backprop(weights,output1,output2,ytrue,X_input, LR_O = 0.05, LR_H = 50):
    #separate learning rates for outer and inner weights.

    wH =weights[0]
    wO =weights[1]

    #STEP A:
    error = (output2-ytrue.reshape(-1,1))*log_loss(ytrue.reshape(-1,1), output2) #need to reshape the y true to a (50,1)!

    #STEP B:
    sig_deriv = output2*(1-output2)
    y_grad = sig_deriv*error

    #STEP C:
    hidden_out_with_bias = np.hstack([output1,np.ones((output1.shape[0],1))])
    delta_wo = np.dot(-y_grad.T,hidden_out_with_bias)*LR_O
    wO_new = wO+delta_wo.T #old weights + delta weights -> new weights!

    #STEP D:
    sig_deriv_2 = output1 * (1-output1)
    H_grad = sig_deriv_2  * np.dot(y_grad , wO_new[:2].T) #remove the bias, since it is not backpropagated!

    #STEP E:
    delta_wH = np.dot(-H_grad.T,X_input)*LR_H
    wH_new = wH+delta_wH.T #old weights + delta weights -> new weights!

    return wH_new, wO_new
```

```python
wH_new, wO_new = backprop(weights,out1,out2,y,X)
```

```python
ACC_VEC = []
LOSS_VEC = []
for i in range(500):
    out1, out2 =  feed_forward(X, weights)
    ACC_VEC.append(get_accuracy(out2,y))
    LOSS_VEC.append(sum(log_loss(y, out2)))
    new_weights = backprop(weights,out1,out2,y,X)
    weights = new_weights
```

    /Users/maximcondon/anaconda3/envs/deep_learning/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: overflow encountered in power
      after removing the cwd from sys.path.

```python
max(ACC_VEC), min(LOSS_VEC)
#90% max accuracy!
```

    (0.66, 28.350432483965868)

```python
import matplotlib.pyplot as plt
plt.figure(figsize=(8,5))
plt.plot(LOSS_VEC)
plt.title('LOSS OVER TRAINING EPOCHS')
```

```python
plt.figure(figsize=(8,5))
plt.plot(ACC_VEC)
plt.title('ACCURACY OVER TRAINING EPOCHS')
```

### Still a bit clunky and needs some hyperparameter optimization, but not bad for doing it from scratch.

What could we do better (in theory)?
- Better initialization method (e.g. He, Glorot, Lecun)
- Batch Normalization
- Better fine-tuning of the learning rates and epochs
- Use Keras :)

### Step 6 - Implement an optimization loop:

- Wrap steps 2-5 in a loop so that they are repeatedly applied to all data points
- print the total loss in each iteration
- stop after a predefined number of iterations
- When the loss is becoming smaller, your implementations works.

#### You must make sure that the loss is really dropping. Do not even try to continue with the second part of the backpropagation if it doesnt.

(Implement the second part of the backpropagation algorithm – propagate the output gradient to the hidden layer:

Calculate hidden_loss as sig'(output) * output_weight for each hidden neuron and each data point
Modify each weight in the hidden layer by -sigmoid'(hidden_output) * hidden_loss * input
Apply the weight modification for each data point separately.
Execute the function once.)

```python
# new_out1, new_out2 = feed_forward(X, new_weights)
```

```python
# new_out1.shape, new_out2.shape
```

```python
# new_error = error_function(new_out2, y)
```

```python
# sum(new_error)
```

Now let's put it in a looooop

```python

```