# Spectral Analysis of Deterministic Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Short-Time Fourier Transform

The discrete Fourier transform (DFT) is not very well suited for the analysis of instationary signals when applied to the entire signal at once. Furthermore practical signals, for instance, an antenna signal, cannot be analyzed in an on-line manner by the DFT. This motivates to split a long signal into segments and compute the DFT on these segments. This transform is known as the [short-time Fourier transform](https://en.wikipedia.org/wiki/Short-time_Fourier_transform) (STFT).

The STFT $X[\mu, n]$ of a signal $x[k]$ is defined as

\begin{equation}
X[\mu, n] = \sum_{k = n}^{n+N-1} x[k] \, w[k-n] \;  \mathrm{e}^{\,-\mathrm{j}\,\frac{2 \pi}{N} k \mu}
\end{equation}

where $w[k]$ denotes a window function of length $N$ which is normalized by $\sum_{k=0}^{N-1} w[k] = 1$. Starting from $k=n$, the signal $x[k]$ is windowed by $w[k]$ to a segment of length $N$. This windowed segment is then transformed by a DFT of length $N$.

The STFT has many applications in digital signal processing, for instance, in the spectral analysis of signals or the processing of instationary signals. The resulting spectrum $X[\mu, n]$ depends on the frequency index $\mu$ and the time index $n$. The spectral domain is therefore also termed as [time-frequency domain](https://en.wikipedia.org/wiki/Time%E2%80%93frequency_representation) and techniques using the STFT as time-frequency processing.

The properties of the STFT depend on

* the length $N$ of the segments,
* the overlap between the segments, and
* the window function $w[k]$.

The size $N$ of the segments and the window function influence the spectral and temporal resolution of the STFT. The time index $n$ of the STFT can be increased by an arbitrary step size. The step size determines the overlap between two consecutive STFTs. For instance, the spectra $X[\mu, n]$ and $X[\mu, n+1]$ have $N-1$ overlapping samples. The overlap is sometimes given as percentage of the segment length $N$.

## The Spectrogram

The magnitude $|X[\mu, n]|$ of the STFT is known as the [spectrogram](https://en.wikipedia.org/wiki/Spectrogram) of a signal. It is frequently used to analyze signals in the time-frequency domain, for instance by a [spectrum analyzer](https://en.wikipedia.org/wiki/Spectrum_analyzer).

**Example**

The following example computes the spectrogram of a chirp/sweep signal.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 8192  # length of the signal
L = 256  # length of one segment
overlap = 128  # overlap between segments

# generate signal
k = np.arange(N)
x = sig.chirp(k, 0, N, .5)

# compute and plot spectrogram
plt.figure(figsize = (10, 5))
plt.specgram(x, NFFT=L, noverlap=overlap, sides='onesided')
plt.xlabel(r'$n$')
plt.ylabel(r'$\Omega$ / $\pi$')
cb = plt.colorbar()
cb.set_label(r'$|X[\Omega,n]|$ in dB')
plt.autoscale(tight=True)
```

**Exercise**

* Can you characterize the signal from its spectrogram? How would it sound like?
* Change the segment length `L` and the overlap `overlap` between segments. Rerun the example. What changes in the spectrogram?

The signal consists of single frequencies that increase with time. A higher segment length leads to a narrower main lobe and thus better selection of the current frequency. With a higher overlap, the spectrum has a higher temporal resolution.

**Example**

The following example computes the spectrogram of a speech signal containing a short sentence. The source signal is loaded using the [`soundfile`](http://pysoundfile.readthedocs.io/en/latest/) module for Python.

```python
import soundfile as sf

L = 1024  # length of one segment
overlap = 512  # overlap between segments

# read speech signal from file
x, fs = sf.read('../data/speech_8k.wav')
x = x/np.max(np.abs(x))

# compute and plot spectrogram
plt.figure(figsize = (10, 5))
plt.specgram(x, NFFT=L, Fs=fs, noverlap=overlap, sides='onesided')
plt.xlabel(r'$n$ in s')
plt.ylabel(r'$f$ in Hz')
cb = plt.colorbar()
cb.set_label(r'$|X[f,n]|$ in dB')
plt.autoscale(tight=True)
plt.ylim([0, 2000]);
```

**Exercise**

* A speech signal exhibits a specific spectral structure. What insights can you gain from the spectrum above?
* Change the segment length and overlap of the STFT. How does the spectrogram change?

In case of a vowel a speech signal is periodic due to the underlying speech production mechanism. A periodic signal has a line spectrum consisting of a fundamental frequency and various harmonics with multiple of the fundamental frequency. The [harmonic structure of the vowels](https://en.wikipedia.org/wiki/Formant) is visible in the spectrogram.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


**Exercise set 11**
==============

>In this exercise, we will process signals and you will learn how you
>can remove noise, obtain numerical derivatives of noisy signals, and
>correct near-infrared spectra by removing unwanted scattering effects.

**Exercise 11.1**

Savitzky–Golay filtering can be used to both smooth a signal and to obtain numerical
derivatives. We will here check that we indeed can obtain the numerical derivative
with a Savitzky–Golay filter. Note: A slightly more involved version of this
question can be found in the 2020 practice exam.

We consider a case where we set the window length to 3 and we have measured a signal, $y = f(x)$,
at uniformly spaced locations $x$. We set the spacing between points to be $h$. If we let
$x_0$ be the first point where we have measured the signal, then the locations of
the other points are given by $x_i = x_0 + i \times h$ for $i = 0, 1, 2, \ldots$.

**(a)**  Consider a specific window of length 3 where we have the set of points
$(x_0, f(x_0))$,
$(x_1, f(x_1))$, and,
$(x_2, f(x_2))$,
At a first step, we transform the original $x_i$ to scaled coordinates $z_i$
so that $z_0 = -1$, $z_1 = 0$, and $z_2 = 1$,
This can be done
with the following transformation:

\begin{equation}
z_ i = \frac{x_i - \alpha}{\beta} .
\label{eq:transform}
\tag{1}\end{equation}

Verify that $\alpha = x_0 + h$ and $\beta = h$ gives the desired transformation.

```python
# Your code here
```

**Your answer to question 11.1(a):** *Double click here*

**(b)**  In the window we are considering, we are going to fit a polynomial of order 2,
\begin{equation}
y = b_0 + b_1 z + b_2 z^2 .
\tag{2}\end{equation}

where we use $z$ as the variable. That is, we are now fitting to the set of points,
$(z_0, f(x_0))$,
$(z_1, f(x_1))$, and
$(z_2, f(x_2))$,
This fitting can be done with least squares and we can formulate the problem as,

\begin{equation}
\mathbf{y} = \mathbf{J} \mathbf{b},
\label{eq:ls}
\tag{3}
\end{equation}

where

\begin{equation}
\mathbf{y} =
\begin{pmatrix}
f(x_0) \\
f(x_1) \\
f(x_2) \\
\end{pmatrix}
, \quad
\mathbf{b} = \begin{pmatrix}
b_0 \\
b_1 \\
b_2 \\
\end{pmatrix},
\tag{4}
\end{equation}

and $\mathbf{J}$ is a matrix created from $z_0$, $z_1$, and $z_2$.
What does the matrix $\mathbf{J}$ contain in this case and how would you construct it? Below
you can find a partially filled out $\mathbf{J}$ (one column is missing), can you
complete it by filling in the missing values?

\begin{equation}
\mathbf{J} =
\begin{pmatrix}
1 &  ? & 1 \\
1 &  ? & 0 \\
1 &  ? & 1 \\
\end{pmatrix}
\tag{5}\end{equation}

```python
# Your code here
```

**Your answer to question 11.1(b):** *Double click here*

**(c)**  If you were to move the window and consider 3 new points,
say, $(x_{10}, f(x_{10}))$, $(x_{11}, f(x_{11}))$, and $(x_{12}, f(x_{12}))$,
would the
$\mathbf{J}$-matrix above change? Briefly justify your answer.

```python
# Your code here
```

**Your answer to question 11.1(c):** *Double click here*

**(d)**  The least-squares solution to Eq. (3) can be written as,

\begin{equation}
  \mathbf{b} =
\begin{pmatrix}
b_0 \\
b_1 \\
b_2 \\
\end{pmatrix} =
\begin{pmatrix}
0  &  1  &   0 \\
-1/2  &  0  & 1/2 \\
1/2  & -1  & 1/2 \\
\end{pmatrix}
\begin{pmatrix}
f(x_0) \\
f(x_1) \\
f(x_2)
\end{pmatrix}.
\tag{6}
\end{equation}

Explain how you can use this equation to obtain the first derivative
at the central location where $z=0$.

```python
# Your code here
```

**Your answer to question 11.1(d):** *Double click here*

**(e)**  An alternative numerical expression for the derivative at the central
point is the central-difference expression:

\begin{equation}
f^\prime (x) = \frac{f(x + h) - f(x - h)}{2 h}.
\tag{7}\end{equation}

Show that this expression is the same as you can obtain from the Savitzky–Golay coefficients.
Three hints:

*  Consider the expression for the derivative using the Savitzky–Golay coefficients at
   the central location $x_1$ ($z_1 = 0$).

*  Use that $x_i = x_0 + i\times h$, i.e. $x_0 = x_1 - h$ and $x_2 = x_1 + h$.

*  Use the chain rule for the polynomial,
   $\frac{\text{d} y}{\text{d} x} = \frac{\text{d} y}{\text{d} z}
   \frac{\text{d} z}{\text{d} x}$, and show that $\frac{\text{d} z}{\text{d} x} = \frac{1}{h}$ using
   Eq. (1).

```python
# Your code here
```

**Your answer to question 11.1(e):** *Double click here*

**Exercise 11.2**

In this exercise, we will test out the Savitzky–Golay filter for smoothing
and numerical differentiation. We will use a test signal which has been generated
from the following analytical function,

\begin{equation}
y(t) = \sin 8t - 1.8t^2 + 0.5t^3.
\label{eq:signal}
\tag{8}\end{equation}

The signal is available in the file [`Data/signal.txt`](Data/signal.txt).
In addition, we will investigate a test signal which is generated from the same
analytical function, but with noise added. This signal is available in the file
[`Data/signal_noise.txt`](Data/signal_noise.txt).

In `scipy`, a Savitzky–Golay filter can be created by using
the method `savgol_filter` from [`scipy.signal`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.savgol_filter.html).
An example is given below:

```python
import numpy as np
from scipy.signal import savgol_filter
from matplotlib import pyplot as plt
plt.style.use('seaborn-notebook')

signal = np.loadtxt('Data/signal_noise.txt')[:, 1]  # Load signal.
# Apply a Savitzky-Golay filter, length 21 and polynomial order of 3
smooth = savgol_filter(signal, 21, polyorder=3)
# Plot the original signal and the filtered one.
plt.plot(signal, label='Original signal')
plt.plot(smooth, label='Filtered signal')
plt.legend()
plt.show()
```

**(a)**  Consider first the signal without the noise and apply a
Savitzky–Golay smoothing filter to it. Here you can experiment
with the window size and polynomial order, for instance, a window size of $5$
or $7$ and a polynomial order of $3$ or $5$.
Does the filter you are considering change the original signal?

```python
# Your code here
```

**Your answer to question 11.2(a):** *Double click here*

**(b)**  Create a Savitzky–Golay filter for first-order differentiation with a
window size of $7$ and polynomial order of $3$. Apply this to the signal
without noise and compare the differentiated signal with the analytical
derivative of Eq. (8).

Note: In order to obtain the derivative,
you will have to supply the spacing between your points using the
`delta` parameter of the `savgol_filter` method.
Further, you will have to tell the method to compute the derivative by
setting the parameter `deriv`  to the order of the derivative
you want. An example of this can be found below:

```python
import numpy as np
from scipy.signal import savgol_filter
from matplotlib import pyplot as plt
plt.style.use('seaborn-notebook')

data = np.loadtxt('Data/signal_noise.txt')  # Load signal.
x = data[:, 0]
delta_x = x[1] - x[0]  # Spacing between x-points
signal = data[:, 1]
# Apply a Savitzky-Golay filter, length 51 and polynomial order of 3,
# and obtain the first derivative:
deriv = savgol_filter(signal, 51, polyorder=3, delta=delta_x, deriv=1)
# Plot the original signal and the filtered one.
plt.plot(x, signal, label='Original signal')
plt.plot(x, deriv, label='Derivative of signal')
plt.legend()
plt.show()
```

```python
# Your code here
```

**Your answer to question 11.2(b):** *Double click here*

**(c)**  Create a Savitzky–Golay filter for smoothing of the signal *with* noise.
Here, you have to experiment with the window size to use. Use a polynomial
order of $3$, and compare your smoothed signal with the original signal, and
the signal without noise.

```python
# Your code here
```

**Your answer to question 11.2(c):** *Double click here*

**(d)**  Create a Savitzky–Golay filter for first-order differentiation of the signal
*with* noise. Here, you have to experiment with the window size to use.
Use a polynomial order of $3$ and compare your differentiated signal
with the analytical derivative of Eq. (8).

```python
# Your code here
```

**Your answer to question 11.2(d):** *Double click here*

**Exercise 11.3**

In this exercise, we will try to smooth a signal by performing convolution.
We will attempt to smooth the signal given in Eq. (8)
with added noise (data file: [`Data/signal_noise.txt`](Data/signal_noise.txt)). A short example
of performing convolution with `scipy/numpy` can be found below:

```python
import numpy as np
from scipy.signal import windows
from matplotlib import pyplot as plt
plt.style.use('seaborn-notebook')

signal = np.loadtxt('Data/signal_noise.txt')[:, 1]  # Load signal.
window = windows.boxcar(21)  # Create a rectangular window of 21 points
window /= window.sum()  # Normalize the window.
conv = np.convolve(signal, window, mode='same')  # Calculate convolution.
# Plot the original signal and the filtered one.
plt.plot(signal, label='Original signal')
plt.plot(conv, label='Filtered signal (convolution, rectangular)')
plt.legend()
plt.show()
```

**(a)**  For smoothing the signal, we will use a windowing function for the convolution.
In the code above we use a simple rectangular window for the smoothing,
but in `scipy`, there are several windowing functions
available. A list of the available windowing
functions can be found [here](https://docs.scipy.org/doc/scipy/reference/signal.windows.html) and
they behave differently. Here,
you can experiment with different windowing functions and/or different
windowing lengths. Are you able to remove most of the noise from
the given signal?

```python
# Your code here
```

**Your answer to question 11.3(a):** *Double click here*

**Exercise 11.4**

You have measured the signal given in the file [`Data/spike.txt`](Data/spike.txt). Unfortunately,
the signal contains a prominent spike that you would like to remove. In the code below,
we attempt to remove the spike by doing a convolution with
a rectangular window. Run this code and observe the results.

```python
import numpy as np
from matplotlib import pyplot as plt
from scipy.signal import windows
plt.style.use('seaborn-notebook')

def median_filter(yvalues, width=11):
    """Apply a median filter to the given y-values."""
    medians = []
    half = width // 2
    length = len(yvalues) - 1
    for i, _ in enumerate(yvalues):
        left = max(0, i - half)
        right = min(length, i + half)
        med = np.median(yvalues[left:right + 1])
        medians.append(med)
    return np.array(medians)

signal = np.loadtxt('Data/spike.txt')
window = window = windows.boxcar(21)
window /= window.sum()
conv = np.convolve(signal, window, mode='same')  # Do convolution
filter_median = median_filter(signal, width=21)  # Apply median filter
# Plot the results:
fig, axes = plt.subplots(constrained_layout=True, ncols=2)
axes[0].plot(signal, label='Original signal')
axes[0].legend()
axes[1].plot(signal, alpha=0.5, label='Original signal')
axes[1].plot(conv, label='Signal after convolution')
axes[1].plot(filter_median, label='Signal after median filter')
# Zoom in around the spike:
axes[1].set_xlim(600, 950)
axes[1].set_ylim(0, 2)
axes[1].legend()
plt.show()
```

**(a)**  As you can see from the results of applying the convolution,
the spike has not been completely removed, rather,
it has been made broader. Can you explain what seems to be
happening around the spike when you apply the convolution?

```python
# Your code here
```

**Your answer to question 11.4(a):** *Double click here*

**(b)**  The code given above also includes a so-called *median* filter.
The median filter
returns the median value of the signal in a given window. Applying
this filter seems to do a better job of removing the spike. Can you explain
why this is the case?

```python
# Your code here
```

**Your answer to question 11.4(b):** *Double click here*

**Exercise 11.5**

Multiple Scatter Correction (MSC) is one approach to remove non-linear
effects in near-infrared (NIR) spectra. Such effects may arise as a result
of scattering effects in a sample.
This will generate additional variance which is not related to the
chemical interesting information contained in the spectrum, and we would,
therefore, like to remove it.

The file [`Data/nir_msc.txt`](Data/nir_msc.txt) contains $222$ spectra that have been
sampled at $121$ wavelengths. Each row contains a spectrum, and each
column represents a single wavelength. You will now apply MSC to
correct these spectra.

**(a)**  Find a representative spectrum by taking the mean of the $222$ spectra.
We will refer to this representative spectrum as $f(x)$ in the following.

```python
# Your code here
```

**Your answer to question 11.5(a):** *Double click here*

**(b)**  Correct each spectrum, $h_i(x)$, by first fitting it to a linear
equation,

\begin{equation}
h_i(x) = a_i f(x) + b_i ,
\end{equation}

and then remove the scattering effects by taking,

\begin{equation}
h_{i, \text{corrected}}(x) = \frac{h_i(x) -b_i}{a_i} .
\end{equation}

```python
# Your code here
```

**Your answer to question 11.5(b):** *Double click here*

**(c)**  Plot the corrected and uncorrected spectra.
Does this look like what you would expect?

```python
# Your code here
```

**Your answer to question 11.5(c):** *Double click here*

**(d)**  As stated above MSC will remove some of the (presumably) uninteresting
variance. We can quantify this by calculating the sum of squares, $SS_0$,

\begin{equation}
SS_0 = \sum_i \sum_j (x_{ij} - \overline{x}_j)^2,
\end{equation}

where $x_{ij}$ is the absorbance for spectrum $i$ at wavelength $j$, and
$\overline{x}_j$ is the mean of all spectra for wavelength $j$.
Calculate $SS_0$ for both the corrected and uncorrected spectra.
Does MSC reduce $SS_0$ as we expect?

```python
# Your code here
```

**Your answer to question 11.5(d):** *Double click here*


# Decentralization Planning

## Objective and Prerequisites

Ready for a mathematical optimization modeling challenge? Put your skills to the test with this example, where you’ll learn how to model and solve a decentralization planning problem. You’ll have to figure out – given a set of departments of a company, and potential cities where these departments can be located – the “best” location for each department in order to maximize gross margins.

This model is example 10 from the fifth edition of Model Building in Mathematical Programming by H. Paul Williams on pages 265 and 317-319.

This modeling example is at the advanced level, where we assume that you know Python and the Gurobi Python API and that you have advanced knowledge of building mathematical optimization models. Typically, the objective function and/or constraints of these examples are complex or require advanced features of the Gurobi Python API.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

## Problem Description

A large company wants to move some of its departments out of London. Doing so will result in reduced costs in some areas
(such as cheaper housing, government incentives, easier recruitment, etc.), and increased costs in other areas (such as communication between departments). The cost implications for all possible locations of each department have been calculated.
The goal is to determine where to locate each department in order to maximize the total difference between the reduced costs  from relocating and the increased communication costs between departments.

The company comprises five departments (A, B, C, D and E). The possible cities for relocation are Bristol and Brighton, or a department may be kept in London. None of these cities (including London) may be the location for more than three of the departments.

## Model Formulation

### Sets and Indices

$d,d2 \in \text{Departments}=\{A,B,C,D,E\}$

$c,c2 \in \text{Cities}=\{\text{Bristol}, \text{Brighton}, \text{London}\}$

### Parameters

$\text{benefit}_{d,c} \in \mathbb{R}^+$: Benefit -in thousands of dollars per year, derived from relocating department $d$  to city $c$.

$\text{communicationCost}_{d,c,d2,c2} \in \mathbb{R}^+$: Communication cost -in thousands of dollars per year, derived from relocating department $d$  to city $c$ and relocating department $d2$  to city $c2$.

We define the set $dcd2c2 = \{(d,c,d2,c2) \in \text{Departments} \times \text{Cities} \times \text{Departments} \times \text{Cities}: \text{communicationCost}_{d,c,d2,c2} > 0  \}$

### Decision Variables

$\text{locate}_{d,c} \in \{0,1 \}$: This binary variable is equal 1, if department $d$  is located at city $c$, and 0 otherwise.

$y_{d,c,d2,c2} = \text{locate}_{d,c}*\text{locate}_{d2,c2} \in \{0,1 \}$: This auxiliary binary variable is equal 1, if department $d$ is located at city $c$ and department $d2$ is located at city $c2$, and 0 otherwise.

### Constraints

**Department location**: Each department must be located in only one city.

\begin{equation}
\sum_{c \in \text{Cities}} \text{locate}_{d,c} = 1 \quad \forall d \in \text{Departments}
\end{equation}

**Departments limit**: No city may be the location for more than three departments.

\begin{equation}
\sum_{d \in \text{Departments}} \text{locate}_{d,c} \leq 3 \quad \forall c \in \text{Cities}
\end{equation}

**Logical Constraints**:

- If $y_{d,c,d2,c2} = 1$ then $\text{locate}_{d,c} = 1$ and $\text{locate}_{d2,c2} = 1$.

\begin{equation}
y_{d,c,d2,c2} \leq \text{locate}_{d,c} \quad \forall (d,c,d2,c2) \in dcd2c2
\end{equation}

\begin{equation}
y_{d,c,d2,c2} \leq \text{locate}_{d2,c2} \quad \forall (d,c,d2,c2) \in dcd2c2
\end{equation}

-  If $\text{locate}_{d,c} = 1$ and $\text{locate}_{d2,c2} = 1 $ then $y_{d,c,d2,c2} = 1$.

\begin{equation}
\text{locate}_{d,c} + \text{locate}_{d2,c2} - y_{d,c,d2,c2} \leq 1 \quad  \forall (d,c,d2,c2) \in dcd2c2
\end{equation}

### Objective Function

**Gross margin**: Maximize the gross margin of relocation.

\begin{equation}
\text{Maximize} \quad Z = \sum_{d \in \text{Departments}} \sum_{c \in \text{Cities}} \text{benefit}_{d,c}*\text{locate}_{d,c} -
\sum_{d,c,d2,c2 \in dcd2c2} \text{communicationCost}_{d,c,d2,c2}*y_{d,c,d2,c2}
\end{equation}

This linear integer programming formulation of the decentralization problem is in fact a linearization of a quadratic assignment formulation of this problem. With Gurobi 9.0, you can directly solve  the quadratic assignment formulation of the decentralization problem without the auxiliary variables and the logical constraints.

### Objective Function

**Gross margin**: Maximize the gross margin of relocation.

\begin{equation}
\text{Maximize} \quad Z = \sum_{d \in \text{Departments}} \sum_{c \in \text{Cities}} \text{benefit}_{d,c}*\text{locate}_{d,c} -
\sum_{d,c,d2,c2 \in dcd2c2} \text{communicationCost}_{d,c,d2,c2}*\text{locate}_{d,c}*\text{locate}_{d2,c2}
\end{equation}

### Constraints

**Department location**: Each department must be located in only one city.

\begin{equation}
\sum_{c \in \text{Cities}} \text{locate}_{d,c} = 1 \quad \forall d \in \text{Departments}
\end{equation}

**Departments limit**: No city may be the location for more than three departments.

\begin{equation}
\sum_{d \in \text{Departments}} \text{locate}_{d,c} \leq 3 \quad \forall c \in \text{Cities}
\end{equation}

## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
%pip install gurobipy
```

```python
import pandas as pd

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input data
We define all the input data for the model.

```python
# Lists of deparments and cities

Deparments = ['A','B','C','D','E']
Cities = ['Bristol', 'Brighton', 'London']

# Create a dictionary to capture benefits -in thousands of dollars from relocation.

d2c, benefit = gp.multidict({
    ('A', 'Bristol'): 10,
    ('A', 'Brighton'): 10,
    ('A', 'London'): 0,
    ('B', 'Bristol'): 15,
    ('B', 'Brighton'): 20,
    ('B', 'London'): 0,
    ('C', 'Bristol'): 10,
    ('C', 'Brighton'): 15,
    ('C', 'London'): 0,
    ('D', 'Bristol'): 20,
    ('D', 'Brighton'): 15,
    ('D', 'London'): 0,
    ('E', 'Bristol'): 5,
    ('E', 'Brighton'): 15,
    ('E', 'London'): 0
})

# Create a dictionary to capture the communication costs -in thousands of dollars from relocation.

dcd2c2, communicationCost = gp.multidict({
    ('A','London','C','Bristol'): 13,
    ('A','London','C','Brighton'): 9,
    ('A','London','C','London'): 10,
    ('A','London','D','Bristol'): 19.5,
    ('A','London','D','Brighton'): 13.5,
    ('A','London','D','London'): 15,
    ('B','London','C','Bristol'): 18.2,
    ('B','London','C','Brighton'): 12.6,
    ('B','London','C','London'): 14,
    ('B','London','D','Bristol'): 15.6,
    ('B','London','D','Brighton'): 10.8,
    ('B','London','D','London'): 12,
    ('C','London','E','Bristol'): 26,
    ('C','London','E','Brighton'): 18,
    ('C','London','E','London'): 20,
    ('D','London','E','Bristol'): 9.1,
    ('D','London','E','Brighton'): 6.3,
    ('D','London','E','London'): 7,
    ('A','Bristol','C','Bristol'): 5,
    ('A','Bristol','C','Brighton'): 14,
    ('A','Bristol','C','London'): 13,
    ('A','Bristol','D','Bristol'): 7.5,
    ('A','Bristol','D','Brighton'): 21,
    ('A','Bristol','D','London'): 19.5,
    ('B','Bristol','C','Bristol'): 7,
    ('B','Bristol','C','Brighton'): 19.6,
    ('B','Bristol','C','London'): 18.2,
    ('B','Bristol','D','Bristol'): 6,
    ('B','Bristol','D','Brighton'): 16.8,
    ('B','Bristol','D','London'): 15.6,
    ('C','Bristol','E','Bristol'): 10,
    ('C','Bristol','E','Brighton'): 28,
    ('C','Bristol','E','London'): 26,
    ('D','Bristol','E','Bristol'): 3.5,
    ('D','Bristol','E','Brighton'): 9.8,
    ('D','Bristol','E','London'): 9.1,
    ('A','Brighton','C','Bristol'): 14,
    ('A','Brighton','C','Brighton'): 5,
    ('A','Brighton','C','London'): 9,
    ('A','Brighton','D','Bristol'): 21,
    ('A','Brighton','D','Brighton'): 7.5,
    ('A','Brighton','D','London'): 13.5,
    ('B','Brighton','C','Bristol'): 19.6,
    ('B','Brighton','C','Brighton'): 7,
    ('B','Brighton','C','London'): 12.6,
    ('B','Brighton','D','Bristol'): 16.8,
    ('B','Brighton','D','Brighton'): 6,
    ('B','Brighton','D','London'): 10.8,
    ('C','Brighton','E','Bristol'): 28,
    ('C','Brighton','E','Brighton'): 10,
    ('C','Brighton','E','London'): 18,
    ('D','Brighton','E','Bristol'): 9.8,
    ('D','Brighton','E','Brighton'): 3.5,
    ('D','Brighton','E','London'): 6.3
})
```

## Model Deployment

We create a model and the variables. These binary decision variables define the city at which each department will be located.

Solving quadratic assignment problems  with Gurobi is as easy as configuring the global parameter `nonConvex`, and setting this parameter to the value of 2.

```python
model = gp.Model('decentralization')

# Set global parameters
model.params.nonConvex = 2

# locate deparment d at city c
locate = model.addVars(d2c, vtype=GRB.BINARY, name="locate")
```

    Using license file c:\gurobi\gurobi.lic
    Changed value of parameter nonConvex to 2
       Prev: -1  Min: -1  Max: 2  Default: -1

Each department must be located in exactly one city.

```python
# Department location constraint

department_location = model.addConstrs((gp.quicksum(locate[d,c] for c in Cities) == 1 for d in Deparments),
                                    name='department_location')
```

No city may be the location for more than three departments.

```python
# Limit on number of departments

departments_limit = model.addConstrs((gp.quicksum(locate[d,c] for d in Deparments) <= 3 for c in Cities),
                                    name='departments_limit')
```

We now set the optimization objective, which is to maximize gross margins.

```python
model.setObjective((gp.quicksum(benefit[d,c]*locate[d,c] for d,c in d2c)
                    - gp.quicksum(communicationCost[d,c,d2,c2]*locate[d,c]*locate[d2,c2] for d,c,d2,c2 in dcd2c2) ),
                   GRB.MAXIMIZE)
```

```python
# Verify model formulation

model.write('decentralizationQA.lp')

# Run optimization engine

model.optimize()
```

    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 8 rows, 15 columns and 30 nonzeros
    Model fingerprint: 0x2ad3c449
    Model has 54 quadratic objective terms
    Variable types: 0 continuous, 15 integer (15 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [5e+00, 2e+01]
      QObjective range [7e+00, 6e+01]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 3e+00]
    Found heuristic solution: objective -73.9000000
    Presolve time: 0.00s
    Presolved: 62 rows, 69 columns, 192 nonzeros
    Variable types: 0 continuous, 69 integer (69 binary)

    Root relaxation: objective -6.750000e+01, 14 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0   67.50000    0   10  -73.90000   67.50000   191%     -    0s
    H    0     0                     -33.9000000   67.50000   299%     -    0s
    H    0     0                     -16.3000000   67.50000   514%     -    0s
    H    0     0                      14.9000000   67.50000   353%     -    0s
         0     0   30.00000    0   22   14.90000   30.00000   101%     -    0s

    Cutting planes:
      Gomory: 3
      MIR: 16
      Zero half: 4
      Mod-K: 2
      RLT: 25

    Explored 1 nodes (43 simplex iterations) in 0.04 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 4: 14.9 -16.3 -33.9 -73.9

    Optimal solution found (tolerance 1.00e-04)
    Best objective 1.490000000000e+01, best bound 1.490000000000e+01, gap 0.0000%

## Analysis

The optimal relocation plan and associated financial report follows.

```python
relocation_plan = pd.DataFrame(columns=["Department", "City"])

count = 0

for c in Cities:
    for d in Deparments:
        if(locate[d,c].x > 0.5):
            count += 1
            relocation_plan = relocation_plan.append({"Department": d, "City": c }, ignore_index=True )
relocation_plan.index=['']*count
relocation_plan
```

      Department
      City

      A
      Bristol

      D
      Bristol

      B
      Brighton

      C
      Brighton

      E
      Brighton

```python
print("\n\n_________________________________________________________________________________")
print(f"Financial report")
print("_________________________________________________________________________________")
total_benefit = 0
for c in Cities:
    for d in Deparments:
        if(locate[d,c].x > 0.5):
            total_benefit += 1000*benefit[d,c]

dollars_benefit = '${:,.2f}'.format(total_benefit)
print(f"The yearly total benefit is {dollars_benefit} dollars")

total_communication_cost = 0
for d,c,d2,c2 in dcd2c2:
    if(locate[d,c].x*locate[d2,c2].x > 0.5):
        total_communication_cost += 1000*communicationCost[d,c,d2,c2]

dollars_communication_cost = '${:,.2f}'.format(total_communication_cost)
print(f"The yearly total communication cost is {dollars_communication_cost} dollars")

total_gross_margin = total_benefit - total_communication_cost
dollars_gross_margin = '${:,.2f}'.format(total_gross_margin)
print(f"The yearly total gross margin is {dollars_gross_margin} dollars")
```

    _________________________________________________________________________________
    Financial report
    _________________________________________________________________________________
    The yearly total benefit is $80,000.00 dollars
    The yearly total communication cost is $65,100.00 dollars
    The yearly total gross margin is $14,900.00 dollars

## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC


```python
import math
import numpy as np
import functools
import sympy as sym
```

## Valid permutations example: D=2, L=2, R=2
- $\textbf{[D,L,R,R,D,L]}$
- $\textbf{[L,L,D,R,R,D]}$
- $\textbf{[L,R,D,L,R,D]}$

## Rules:
- Permutations must not end in R
- Permutations must not include an R followed by an L

## Invalid permutations examples: D=2, L=2, R=2
- $\textbf{[D,D,L,L,R,R]}$

- $\textbf{[R,L,L,D,R,D]}$

## Total number of permutations ($T$):
$$T = \frac{(D+R+L)!}{D! \times R! \times L!}$$

```python
def get_all_permutations(D, R, L):
    return np.math.factorial(D + R + L) // (np.math.factorial(D) * np.math.factorial(R) * np.math.factorial(L))
```

```python

```

## Number of permutations that end in R ($E_R$):
$$
E_R =
\begin{cases}
    \frac{(D+R+L-1)!}{D! \times (R-1)! \times L!} & \text{if } R>0\\
    0 & \text{otherwise}
\end{cases}
$$

```python
def get_permutations_ending_in_R(D, R, L):
    if R > 0:
        return np.math.factorial(D + R + L - 1) // (np.math.factorial(D) * np.math.factorial(R-1) * np.math.factorial(L))
    else:
        return 0
```

### Examples:
- $[R,D,L,D,L,|R]$
- $[D,L,L,R,D,|R]$
- $[D,R,D,L,L,|R]$
- $[L,D,R,D,L,|R]$

```python

```

## Number of permutations that end in D and there exist at least one RL ($E_D$):

$$
E_D =
\begin{cases}
    \sum_{i=1}^{\min(R,L)} (-1)^{i+1} \frac{(D+R+L-i-1)!}{(D-1)! \times (R-i)! \times (L-i)! \times (i)!} & \text{if } \min(R,L) > 0\\
    0 & \text{otherwise}
\end{cases}
$$

```python
def get_permutations_ending_in_D_where_any_RL_exists(D, R, L, method=np.math):
    max_RL = min(R,L)
    if max_RL > 0:
        sign = -1
        perms = 0
        for num_RL in np.linspace(1, max_RL, max_RL):
            sign *= -1
            current_perm = method.factorial(R + D + L - num_RL - 1) // (method.factorial(D-1) * method.factorial(R-num_RL) * method.factorial(L-num_RL) * method.factorial(num_RL))
            perms += sign * current_perm

        return perms
    else:
        return 0
```

### Examples:
- $[R,D,L,(RL),|D]$
- $[(RL),L,R,D,|D]$
- $[D,R,(RL),L,|D]$
### Duplicate cases:
- $[D,R,L,(RL),|D]$
- $[D,(RL),R,L,|D]$
- $[D,(RL),(RL),|D]$

```python

```

## Number of permutations that end in L and there exist at least one RL ($E_L$):

$$
E_L =
\begin{cases}
    \sum_{i=1}^{\min(R,L)} (-1)^{i+1} \frac{(D+R+L-i-1)!}{(D)! \times (R-i)! \times (L-i-1)! \times (i)!} & \text{if } \min(R,L-1) > 0\\
    0 & \text{otherwise}
\end{cases}
$$

```python
def get_permutations_ending_in_L_where_any_RL_exists(D, R, L):
    max_RL = min(R,L-1)
    if max_RL > 0:
        sign = -1
        perms = 0
        for num_RL in np.linspace(1, max_RL, max_RL):
            sign *= -1
            perms += sign * np.math.factorial(R + D + L - num_RL - 1) // (np.math.factorial(D) * np.math.factorial(R-num_RL) * np.math.factorial(L-1-num_RL) * np.math.factorial(num_RL))
        return perms
    else:
        return 0
```

### Examples:
- $[R,D,D,(RL),|L]$
- $[(RL),D,R,D,|L]$
- $[D,(RL),D,R,|L]$
### Extra cases:
- $[D,R,D,L,R,|L]$
- $[D,L,R,D,R,|L]$
- $[D,D,L,R,R,|L]$

```python

```

## Number of permutations that end in RL and RL exists only at the end ($E_{RL}$):

$$
E_{RL} =
\begin{cases}
    \sum_{i=1}^{\min(R,L)} (-1)^{i+1} \frac{(D+R+L-i-1)!}{(D)! \times (R-i)! \times (L-i)! \times (i-1)!} & \text{if } \min(R,L) > 0\\
    0 & \text{otherwise}
\end{cases}
$$

```python
def get_permutations_ending_in_RL_where_RL_exists_only_at_the_end(D, R, L):
    max_RL = min(R,L)
    if max_RL > 0:
        sign = -1
        perms = 0
        for num_RL in np.linspace(1, max_RL, max_RL):
            sign *= -1
            perms += sign * np.math.factorial(R + D + L - num_RL - 1) // (np.math.factorial(D) * np.math.factorial(R-num_RL) * np.math.factorial(L-num_RL) * np.math.factorial(num_RL-1))
        return perms
    else:
        return 0
```

### Examples:
- $[R,D,L,D,|R,L]$
- $[D,L,D,R,|R,L]$
- $[D,R,D,L,|R,L]$
### Cases that should not be considered:
- $[D,R,L,D,|R,L] \rightarrow [D,(RL),D,R,|L]$
- $[R,L,D,D,|R,L] \rightarrow [(RL),D,D,R,|L]$

```python

```

## Number of permutations where there is no R at the end and there is no R followed by an L:

$$
C = T - E_R - E_D - E_L - E_{RL}
$$

```python
def get_coeff(D, R, L):

    total = D + R + L
    all_permutations = get_all_permutations(D, R, L)
    permutations_ending_in_R = get_permutations_ending_in_R(D, R, L)
    permutations_ending_in_D_where_any_RL_exists = get_permutations_ending_in_D_where_any_RL_exists(D, R, L)
    permutations_ending_in_L_where_any_RL_exists = get_permutations_ending_in_L_where_any_RL_exists(D, R, L)
    permutations_ending_in_RL_where_RL_exists_only_at_the_end = get_permutations_ending_in_RL_where_RL_exists_only_at_the_end(D, R, L)

    coeff = all_permutations
    coeff -= permutations_ending_in_R
    coeff -= permutations_ending_in_D_where_any_RL_exists
    coeff -= permutations_ending_in_L_where_any_RL_exists
    coeff -= permutations_ending_in_RL_where_RL_exists_only_at_the_end

    return coeff#, [all_permutations, permutations_ending_in_R, permutations_ending_in_D_where_any_RL_exists, permutations_ending_in_L_where_any_RL_exists, permutations_ending_in_RL_where_RL_exists_only_at_the_end]
```

```python
get_coeff(10,0,1)
```

    11

```python

```

# All known coefficients

```python
demo = [[1,1,1],
        [2,1,1],
        [1,2,1],
        [1,1,2],
        [3,1,1],
        [2,2,1],
        [2,1,2],
        [1,3,1],
        [1,2,2],
        [1,1,3],
        [4,1,1],
        [3,2,1],
        [3,1,2],
        [2,3,1],
        [2,2,2],
        [2,1,3],
        [1,4,1],
        [1,3,2],
        [1,2,3],
        [1,1,4],
        [5,1,1],
        [4,2,1],
        [4,1,2],
        [3,3,1],
        [3,2,2],
        [3,1,3],
        [2,4,1],
        [2,3,2],
        [2,2,3],
        [2,1,4],
        [1,5,1],
        [1,4,2],
        [1,3,3],
        [1,2,4],
        [1,1,5],
        [6,1,1],
        [5,2,1],
        [5,1,2],
        [4,3,1],
        [4,2,2],
        [4,1,3],
        [3,4,1],
        [3,3,2],
        [3,2,3],
        [3,1,4],
        [2,5,1],
        [2,4,2],
        [2,3,3],
        [2,2,4],
        [2,1,5],
        [1,6,1],
        [1,5,2],
        [1,4,3],
        [1,3,4],
        [1,2,5],
        [1,1,6]]
```

```python
for drl in demo:
    print(get_coeff(drl[0], drl[1], drl[2]))
```

    2
    6
    2
    3
    12
    9
    12
    2
    3
    4
    20
    24
    30
    12
    18
    20
    2
    3
    4
    5
    30
    50
    60
    40
    60
    60
    15
    24
    30
    30
    2
    3
    4
    5
    6
    42
    90
    105
    100
    150
    140
    60
    100
    120
    105
    18
    30
    40
    45
    42
    2
    3
    4
    5
    6
    7

```python

```

# Large inputs

```python
get_coeff(1, 100, 100)
```

    (101,
     [18200251445876759514246239592574316938775422524758080705105320,
      9054851465610328116540417707748416387450458967541333684132000,
      90548514656103281165404177077484163874504589675413336841319,
      9054851465610328116540417707748416387450458967541333684121900,
      10000])

```python

```

## Check values that the algorithm holds

```python
@functools.lru_cache(maxsize=None)
def a(n):
    if n >= 2:
        return 3*a(n-1) - a(n-2)
    else:
        return 1
```

```python
# 1, 2, 5, 13, 34, 89, 233, 64

def verify_permutation_algorithm(cap):
    tot = 0
    for p1 in range(1, cap + 1):
        for p2 in range(0, (cap - p1 + 1)):
            p3 = cap - p1 - p2
            tot += get_coeff(p1,p2,p3)[0]
            # print(p1, p2, p3, get_coeff(p1,p2,p3))
    tot += get_coeff(0,0,1)[0]

    return(a(cap + 1) == tot)
```

```python
for i in range(200,201):
    print(i, verify_permutation_algorithm(i))
```

    200 True


```python
from sympy import symbols, sin, cos, sqrt, trigsimp
from sympy.physics.mechanics import ReferenceFrame, Point, dynamicsymbols, init_vprinting
```

```python
init_vprinting()
```

```python
r, h, l, m, g = symbols('r, h, l, m, g', positive=True, real=True)
theta = dynamicsymbols('theta')
```

```python
N = ReferenceFrame('N')
A = N.orientnew('A', 'Axis', (theta, N.z))
```

```python
R = r * A.y + r * theta * A.x + h / 2 * A.y
R
```

```python
R.express(N)
```

```python
o = Point('o')
o.set_vel(N, 0)
p = o.locatenew('p', R)
p.set_vel(A, R.diff(dynamicsymbols._t, A))
p.v1pt_theory(o, N, A)
```

```python
p.vel(N).magnitude().simplify()
```

```python
R.diff(dynamicsymbols._t, A).simplify()
```

```python
(r * theta.diff() * A.x + (theta.diff() * A.z).cross(r * theta * A.x + (r + h / 2) * A.y)).simplify()
```

```python
sqrt((r * theta * theta.diff())**2 + ((r + h / 2) * theta.diff() - r * theta.diff())**2).simplify()
```

```python
v = R.diff(dynamicsymbols._t, N).magnitude().simplify()
v
```

```python
v.subs({theta.diff(): symbols('theta_d', real=True)}).simplify()
```

```python
(sqrt((R.magnitude().diff(dynamicsymbols._t)**2 + (R.magnitude() * theta.diff())**2)) - v).simplify()
```

```python
I = m / 12 * (l**2 + h**2)
I
```

```python
T = I / 2 * theta.diff()**2 + m / 2 * v**2
T
```

```python
U = -m * g * (r + h / 2 - R.dot(N.y))
U.simplify()
```

```python
U.diff(theta).simplify()
```

```python
(m * g * ((r + h / 2) * cos(theta) + r * theta * sin(theta))).diff(theta).simplify()
```

```python
L = T - U
```

```python
eom = L.diff(theta.diff()).diff(dynamicsymbols._t) - L.diff(theta)
eom.simplify().expand()
```

```python
eom.simplify() / m * 12
```

```python
Istar = eom.expand().coeff(theta.diff().diff())
Istar
```

```python
(eom - Istar * theta.diff().diff()).simplify()
```

```python
lin_eom = eom.subs({cos(theta): 1, sin(theta): theta, theta**2: 0, theta.diff()**2: 0}).simplify()
(lin_eom.expand() / m).simplify()
```

```python
m_ = lin_eom.expand().coeff(theta.diff().diff())
k_ = lin_eom.expand().coeff(theta)
m_, k_
```

```python
sqrt(k_ / m_).simplify()
```

Stability

```python
(k_ / m / g).simplify()
```

```python
ct = cos(theta)
st = sin(theta)
I_ = r * r * theta * theta + h * h / 3 + l * l / 12
eom_mont = -(r * r * theta.diff() * theta.diff() * theta + g * (r * theta * ct - h * st / 2)) / I_
```

```python
I_
```

```python
eom_mont.simplify()
```

```python
(-eom_mont * I_ + I_ * theta.diff().diff() - eom / m).simplify()
```

```python
(eom / m * 12).simplify()
```


```python
import sympy as sym
```

```python
print('Indefinite and Definite Calculator')

print('1). Indefinite Calculator')

print('a). Basic Integral')

f = lambda x: input('Input your function : ' )

x = sym.symbols('x')

print('The result of the integration is : ', sym.integrate(f(x)))

print('-------------------------------------------------------------')

print('b). Intergal Beetween 2 function with Basic Operation')

f = lambda x: input('Input your 1st function : ' )
g = lambda x: input('Input your 2nd function : ' )

x = sym.symbols('x')

print('The result of the integration is : ', sym.integrate(f(x) + g(x)))

print('-------------------------------------------------------------')

print('2). Definite Calculator')

print('a). Basic Integral')

f = lambda x: input('Input your function : ' )
a = float(input('Input your lower bound : '))
b = float(input('Input your upper bound : '))

x = sym.symbols('x')

print('The result of the integration is : ', sym.integrate(f(x), (x, a, b)))

print('-------------------------------------------------------------')

print('b). Intergal Beetween 2 function with Basic Operation')

f = lambda x: input('Input your 1st function : ' )
g = lambda x: input('Input your 2nd function : ' )
a = float(input('Input your lower bound : '))
b = float(input('Input your upper bound : '))

x = sym.symbols('x')

print('The result of the integration is : ', sym.integrate(f(x) + g(x)), (x, a, b))

print('-------------------------------------------------------------')

LaunchButton = input('Type "RESTART" if you want restart your iteration, or type "NO" to stop : ')

if LaunchButton == 'RESTART':
  main()
elif LaunchButton == 'NO':
  print('Thank Your for choosing this calculator')

print('-------------------------------------------------------------')
```

    Indefinite and Definite Calculator
    1). Indefinite Calculator
    a). Basic Integral
    Input your function : x
    The result of the integration is :  x**2/2
    -------------------------------------------------------------
    b). Intergal Beetween 2 function with Basic Operation
    Input your 1st function : x
    Input your 2nd function : x
    The result of the integration is :  xx**2/2
    -------------------------------------------------------------
    2). Definite Calculator
    a). Basic Integral
    Input your lower bound : 0
    Input your upper bound : 2
    Input your function : x
    The result of the integration is :  2.00000000000000
    -------------------------------------------------------------
    b). Intergal Beetween 2 function with Basic Operation
    Input your lower bound : 0
    Input your upper bound : 3
    Input your 1st function : x
    Input your 2nd function : x
    The result of the integration is :  xx**2/2 (x, 0.0, 3.0)
    -------------------------------------------------------------
    Type "RESTART" if you want restart your iteration, or type "NO" to stop : NO
    Thank Your for choosing this calculator
    -------------------------------------------------------------

Note : Almost finished, but i don't know how to make sum operation on the integrate formulas


```python
import numpy as np
from types import SimpleNamespace
%load_ext autoreload
%autoreload 2

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')

# Import additional libraries:
```

# The logit model

In the following, we will consider the logit model for a binary discrete choice. That is, an agent is either taking a specific action, or not taking it (Think buying a car, exit the labor market etc).

We imagine that the benefit of taking the action in question is described by a linear utility index $y_i^*$. This depends on two exogenous variables $x_1$ and $x_2$ and a random shock $\epsilon$:

$$
\begin{aligned}
	y_{i}^*  =&\beta_{0} + \beta_{1}x^1_{i} + \beta_{2}x^2_{i} + \epsilon_i \\
    =& x_{i}\beta + \epsilon_i \\
    & \epsilon \sim logistic(0,1)
\end{aligned}
$$

The econometrician does not observe the utility index; only the *actual choice* based on the index is observed. We therefore associate the indicator variable $y_i$ with the choice taken by individual $i$

$$
\begin{aligned}
	y_i & = 1 \Leftrightarrow y^* > 0 \Leftrightarrow \text{Choice is taken} \\
	y_i & = 0 \Leftrightarrow y^* \le 0 \Leftrightarrow \text{Choice is not taken}
\end{aligned}
$$

Because we assume that the utility shocks follow a logistic distribution, we can formulate the **probability** that an individual chooses to take the action by

$$
\begin{aligned}
	P(y_i = 1|x_i;\beta) &= \frac{\exp(x_i \beta)}{1 + \exp(x_i \beta)} \\
	P(y_i = 0|x_i;\beta) &= 1 - P(y_i = 1|x_i;\beta)
\end{aligned}
$$

We can now use the formulation of choice probabilities to estimate the parameters $\beta$ by maximum likelihood. That is, we write up the log-likelihood function

$$
\begin{equation}
	LL(\beta) = \sum \limits_{i=1}^{N}y_i\log(P(y_i = 1|x_i;\beta)) + (1-y_i)\log(1-P(y_i = 1|x_i;\beta))
\end{equation}
$$

Maximizing $LL(\beta)$ with respect to $\beta$ yields the estimated parameters $\hat{\beta}$

$$
\hat{\beta} = \arg\max \limits_{\beta}LL(\beta)
$$

The function `DGP()` will create the $N$ observations of $(y_i,x_i)$:

```python
def DGP(mp):
    ''' The data generating process behind binary choice model

    Args:
        mp (SimpleNamespace): object containing parameters for data generation

    Returns:
        y_obs (ndarray): indicator for binary choices made by individuals
        x_obs (ndarray): independent variables

    '''

    # a. Exogenous variables
    x0 = np.tile(1.0, mp.N)
    x1 = np.random.normal(**mp.x1_distr)
    x2 = np.random.normal(**mp.x2_distr)
    x_obs = np.vstack((x0, x1, x2)).T

    # b. Probabilities of action choice
    y_prb = np.exp(x_obs @ mp.beta) / (1 + np.exp(x_obs @ mp.beta))

    # c. Draw binary choices from the binomial distribution
    y_obs = np.random.binomial(1, y_prb)
    return y_obs, x_obs
```

Create your data using the following parameterization:

```python
# Parameters
mp = SimpleNamespace()
mp.beta = np.array([0.15, 0.1, 0.2])
mp.N = 100_000
mp.x1_distr = {'loc': 4, 'scale': 3, 'size': mp.N}
mp.x2_distr = {'loc': 1, 'scale': 0.5, 'size': mp.N}

# Create data
np.random.seed(2021)
y_obs, x_obs = DGP(mp)
```

**Question 1:** Create a function that calculates the log-likelihood of your data based on a $\beta$. That is, the function must take as arguments an array `beta`, `y_obs` and `x_obs`

```python
# Example
def log_likelihood(beta, y_obs, x_obs):
    pass
```

**Question 2:** Make a 3d-plot of the likelihood function where $\beta_1$ and $\beta_2$ are on the horizontal axes, and the log-likelihood is on the vertical axis. Visually confirm that it peaks at the data generating $\beta_1$ and $\beta_2$.

*Note:* You can let $\beta_0$=`mp.beta[0]`. Make sure that `mp.beta[1]` and `mp.beta[2]` are in the grids over $\beta_1$ and $\beta_2$.

**Question 3:** Estimate $\beta$ by maximum likelihood. You may use a gradient-free approach or gradients if you will.

**Question 4:** Based on your estimated parameters, simulate a choice `y_sim` pr individual in `x_obs`. Create an output table that shows the following 4 statistics:
The number of times where:
* `y_obs` = 1 and `y_sim` = 1
* `y_obs` = 1 and `y_sim` = 0
* `y_obs` = 0 and `y_sim` = 1
* `y_obs` = 0 and `y_sim` = 0

Comment on the distribution of occurances across cells in the table.

**Question 5:** Test if your initial guess of $\beta$ will have an impact on the final estimate. Why do you think there is/is not an impact?

# Consumption saving with borrowing

We are now considering the consumption-savings model with an extension: households may **borrow** money in the first period. Additionally, there are **2 kinds** of households: the first type will likely see a **low level** of period 2 income, whereas the second type will likely see a **high** second period income.

A household lives for 2 periods and makes decisions on consumption and saving in each period.

**Second period:**
Solving the consumer problem in the second period is similar to the baseline case we have seen before.

The household gets utility from **consuming** and **leaving a bequest** (warm glow),

$$
\begin{aligned}
v_{2}(m_{2})&= \max_{c_{2}}\frac{c_{2}^{1-\rho}}{1-\rho}+\nu\frac{(a_2+\kappa)^{1-\rho}}{1-\rho}\\
\text{s.t.} \\
a_2 &= m_2-c_2 \\
a_2 &\geq 0
\end{aligned}
$$

where
* $m_t$ is cash-on-hand
* $c_t$ is consumption
* $a_t$ is end-of-period assets
* $\rho > 1$ is the risk aversion coefficient
* $\nu > 0 $ is the strength of the bequest motive
* $\kappa > 0$ is the degree of luxuriousness in the bequest motive
* $a_2\geq0$ ensures the household *cannot* die in debt

**First period:**
The household gets utility from immediate consumption.  Household takes into account that next period income is stochastic.

$$
\begin{aligned}
v_1(m_1)&=\max_{c_1}\frac{c_{1}^{1-\rho}}{1-\rho}+\beta\mathbb{E}_{1}\left[v_2(m_2)\right]\\&\text{s.t.}&\\
a_1 & = m_1 - c_1 \\
m_2 & = (1+r)a_1 + y_2 \\
y_{2} & = \begin{cases}
1 - \Delta & \text{with prob. } P_{low}\\
1 + \Delta & \text{with prob. } P_{high}
\end{cases}\\
a_1 & > -\frac{1-\Delta}{1+r}
\end{aligned}
$$

where
* $\beta > 0$ is the discount factor
* $\mathbb{E}_1$ is the expectation operator conditional on information in period 1
* $y_2$ is income in period 2
* $\Delta \in (0,1)$ is the level of income risk
* $r$ is the interest rate
* $\frac{1-\Delta}{1+r} > c_1 - m_1$ ensures the household cannot borrow *more* than it will be able to repay in next period when $y_2$ is received.

The **2 types** of households are defined by their different $(P_{low},P_{high})$:
* Type 1:
    * $P_{low} = 0.9$
    * $P_{high} = 0.1$
* Type 2:
    * $P_{low} = 0.1$
    * $P_{high} = 0.9$

```python
# Parameters
rho = 3
kappa = 0.5
nu = 0.1
r = 0.04
beta = 0.95
Delta = 0.5
# Add income prb parameters

# Tip: for each household type, create a SimpleNamespace
# or dictionary for storing all the parameters
```

**Question 1** Solve the model for each type of household. Plot the value functions $v_1(m_1)$ and $v_2(m_2)$ in one graph for each household type. Comment on the differences.

**Question 2** From the model solution, obtain the optimal consumption functions $c_1^*(m_1)$ and $c_2^*(m_2)$. Plot these in one graph for each type of household. Comment on the observed differences between household types.

**Question 3** Simulate `simN` households of each type based on the distribution of $m_1$ below. You can use the same distribution for both household types. What is the fraction of households who *borrow* in period 1, $c_1 > m_1$, in each group?

```python
np.random.seed(2021)
simN = 1000
# No one gets negative m in first period
sim_m1 = np.fmax(np.random.normal(1, 1, size = simN), 0)
```

# Division by Newton's method

One can obtain the numerical ratio of 2 real numbers **using only multiplication** and harnessing Newton's method! This may be helpful when the numbers are very large because division methods of large numbers is costly.

Our objective is to find the numerical $x$
$$
x = \frac{n}{d}
$$
given the two numbers $n,d$.

**First note** that if we can find the numeric value $\tilde{d}$
$$
\tilde{d} = \frac{1}{d}
$$
then we can readily obtain $x$ by
$$
x = n \times \tilde{d}
$$

Therefore, our objective comes down to finding the value of $\tilde{d}$ and the rest is trivial.

**Second, note** that Newton's method can be used to find the root $x^*$ of a function $f(x)$ by the iteration steps

$$
x_{k+1} = x_{k} - \frac{f(x_{k})}{f^{\prime}(x_{k})} \equiv \mathcal{N}(x_k)
$$

This means that if we can define some function $f(x)$ such that
$$
f(x) = 0 \Leftrightarrow x = \frac{1}{d}
$$
then the root $x^*$ provides us with the numerical value that we want.

**Third, note** that the function $g(x)$
$$
g(x) = \frac{1}{x}-d
$$
has the property $g(\tilde{d}) = 0$, which means that $g(x)$ is a good candidate for $f(x)$.

**Question 1:** By applying the function $g(x)$ in Newton's method, we can avoid any use of division during the run of the algorithm.
Derive the expression $\frac{g(x)}{g'(x)}$. Do you see why there is no division involved?

**Question 2:** Implement the algorithm below in code and test it.

Division algorithm
1. Choose a tolerance level $\epsilon > 0$. Provide an initial guess $\tilde{d}_0 \ne 0$. Set $k = 0$.
2. Calculate a new candidate root $\tilde{d}_{k+1} = \mathcal{N}(\tilde{d}_k)$.
3. If $|\tilde{d}_{k+1} - \tilde{d}_{k}| < \epsilon$ then stop and return $x = n \times \tilde{d}_{k+1}$.
4. Else, set $k = k + 1$ and return to step 2.

**Important:** if the starting point $\tilde{d}_0$ is too far off target, then you might not get convergence.
You can test your implementation with the example:
$n$ = 37.581
$d$ = 5.9
$\tilde{d}_0$ = 0.2

```python
def newton_division(n, d, d0, max_iter=500, tol=1e-8):
    pass
```


# Nonlinear Program (NLP) Tutorial
For instructions on how to run these tutorial notebooks, please see the [README](https://github.com/RobotLocomotion/drake/blob/master/tutorials/README.md).

## Important Note
Please refer to the [MathematicalProgram Tutorial](./mathematical_program.ipynb) for constructing and solving a general optimization program in Drake.

## Nonlinear Program
A Nonlinear Programming (NLP) problem is a special type of optimization problem. The cost and/or constraints in an NLP are nonlinear functions of decision variables. The mathematical formulation of a general NLP is
\begin{align}
\min_x&\; f(x)\\
\text{subject to }& g_i(x)\leq 0
\end{align}
where $f(x)$ is the cost function, and $g_i(x)$ is the i'th constraint.

An NLP is typically solved through gradient-based optimization (like gradient descent, SQP, interior point methods, etc). These methods rely on the gradient of the cost/constraints $\partial f/\partial x, \partial g_i/\partial x$. pydrake can compute the gradient of many functions through automatic differentiation, so very often the user doesn't need to manually provide the gradient.

## Setting the objective
The user can call `AddCost` function to add a nonlinear cost into the program. Note that the user can call `AddCost` repeatedly, and the program will evaluate the *summation* of each individual cost as the total cost.

### Adding a cost through a python function
We can define a cost through a python function, and then add this python function to the objective through `AddCost` function. When adding a cost, we should provide the variable associated with that cost, the syntax is `AddCost(cost_evaluator, vars=associated_variables)`, which means the cost is evaluated on the `associated_variables`.In the code example below, We first demonstrate how to construct an optimization program with 3 decision variables, then we show how to add a cost through a python function.

```python
from pydrake.solvers.mathematicalprogram import MathematicalProgram, Solve
import numpy as np

# Create an empty MathematicalProgram named prog (with no decision variables,
# constraints or costs)
prog = MathematicalProgram()
# Add three decision variables x[0], x[1], x[2]
x = prog.NewContinuousVariables(3, "x")
```

```python
def cost_fun(z):
    cos_z = np.cos(z[0] + z[1])
    sin_z = np.sin(z[0] + z[1])
    return cos_z**2 + cos_z + sin_z
# Add the cost evaluated with x[0] and x[1].
cost1 = prog.AddCost(cost_fun, vars=[x[0], x[1]])
print(cost1)
```

Notice that by changing the argument `vars` in `AddCost` function, we can add the cost to a different set of variables. In the code example below, we use the same python function `cost_fun`, but impose this cost on the variable `x[0], x[2]`.

```python
cost2 = prog.AddCost(cost_fun, vars=[x[0], x[2]])
print(cost2)
```

### Adding cost through a lambda function
A more compact approach to add a cost is through a lambda function. For example, the code below adds a cost $x[1]^2 + x[0]$ to the optimization program.

```python
# Add a cost x[1]**2 + x[0] using a lambda function.
cost3 = prog.AddCost(lambda z: z[0]**2 + z[1], vars = [x[1], x[0]])
print(cost3)
```

If we change the associated variables, then it represents a different cost. For example, we can use the same lambda function, but add the cost $x[1]^2 + x[2]$ to the program by changing the argument to `vars`

```python
cost4 = prog.AddCost(lambda z: z[0]**2 + z[1], vars = x[1:])
print(cost4)
```

## Adding quadratic cost
In NLP, adding a quadratic cost $0.5x^TQx+ b'x+c$ is very common. pydrake provides multiple functions to add quadratic cost, including
- `AddQuadraticCost`
- `AddQuadraticErrorCost`
- `AddL2NormCost`

### AddQuadraticCost
We can add a simple quadratic expression as a cost.

```python
cost4 = prog.AddQuadraticCost(x[0]**2 + 3 * x[1]**2 + 2*x[0]*x[1] + 2*x[1] * x[0] + 1)
print(cost4)
```

If the user knows the matrix form of `Q` and `b`, then it is faster to pass in these matrices to `AddQuadraticCost`, instead of using the symbolic quadratic expression as above.

```python
# Add a cost x[0]**2 + 2*x[1]**2 + x[0]*x[1] + 3*x[1] + 1.
cost5 = prog.AddQuadraticCost(
    Q=np.array([[2., 1], [1., 4.]]),
    b=np.array([0., 3.]),
    c=1.,
    vars=x[:2])
print(cost5)
```

### AddQuadraticErrorCost
This function adds a cost of the form $(x - x_{des})^TQ(x-x_{des})$.

```python
cost6 = prog.AddQuadraticErrorCost(
    Q=np.array([[1, 0.5], [0.5, 1]]),
    x_desired=np.array([1., 2.]),
    vars=x[1:])
print(cost6)
```

### AddL2NormCost
This function adds a quadratic cost of the form $(Ax-b)^T(Ax-b)$

```python
# Add the L2 norm cost on (A*x[:2] - b).dot(A*x[:2]-b)
cost7 = prog.AddL2NormCost(
    A=np.array([[1., 2.], [2., 3], [3., 4]]),
    b=np.array([2, 3, 1.]),
    vars=x[:2])
print(cost7)
```

## Adding constraints

Drake supports adding constraints in the following form
\begin{align}
lower \leq g(x) \leq upper
\end{align}
where $g(x)$ returns a numpy vector.

The user can call `AddConstraint(g, lower, upper, vars=x)` to add the constraint. Here `g` must be a python function (or a lambda function).

```python
## Define a python function to add the constraint x[0]**2 + 2x[1]<=1, -0.5<=sin(x[1])<=0.5
def constraint_evaluator1(z):
    return np.array([z[0]**2+2*z[1], np.sin(z[1])])

constraint1 = prog.AddConstraint(
    constraint_evaluator1,
    lb=np.array([-np.inf, -0.5]),
    ub=np.array([1., 0.5]),
    vars=x[:2])
print(constraint1)

# Add another constraint using lambda function.
constraint2 = prog.AddConstraint(
    lambda z: np.array([z[0]*z[1]]),
    lb=[0.],
    ub=[1.],
    vars=[x[2]])
print(constraint2)
```

## Solving the nonlinear program

Once all the constraints and costs are added to the program, we can call the `Solve` function to solve the program and call `GetSolution` to obtain the results. Solving an NLP requires an initial guess on all the decision variables. If the user doesn't specify an initial guess, we will use a zero vector by default.

```python
## Solve a simple nonlinear
# min               -x0
# subject to x1 - exp(x0) >= 0
#            x2 - exp(x1) >= 0
#            0 <= x0 <= 100
#            0 <= x1 <= 100
#            0 <= x2 <= 10
prog = MathematicalProgram()
x = prog.NewContinuousVariables(3)
# The cost is a linear function, so we call AddLinearCost
prog.AddLinearCost(-x[0])
# Now add the constraint x1-exp(x0)>=0 and x2-exp(x1)>=0
prog.AddConstraint(
    lambda z: np.array([z[1]-np.exp(z[0]), z[2]-np.exp(z[1])]),
    lb=[0, 0],
    ub=[np.inf, np.inf],
    vars=x)
# Add the bounding box constraint 0<=x0<=100, 0<=x1<=100, 0<=x2<=10
prog.AddBoundingBoxConstraint(0, 100, x[:2])
prog.AddBoundingBoxConstraint(0, 10, x[2])

# Now solve the program with initial guess x=[1, 2, 3]
result = Solve(prog, np.array([1.,2.,3.]))
print(f"Is optimization successful? {result.is_success()}")
print(f"Solution to x: {result.GetSolution(x)}")
print(f"optimal cost: {result.get_optimal_cost()}")
```

### Setting the initial guess
Some NLPs might have many decision variables. In order to set the initial guess for these decision variables, we provide a function `SetInitialGuess` to set the initial guess of a subset of decision variables. For example, in the problem below, we want to find the two closest points $p_1$ and $p_2$, where $p_1$ is on the unit circle, and $p_2$ is on the curve $y=x^2$, we can set the initial guess for these two variables separately by calling `SetInitialGuess`.

```python
import matplotlib.pyplot as plt
prog = MathematicalProgram()
p1 = prog.NewContinuousVariables(2, "p1")
p2 = prog.NewContinuousVariables(2, "p2")

# Add the constraint that p1 is on the unit circle centered at (0, 2)
prog.AddConstraint(
    lambda z: [z[0]**2 + (z[1]-2)**2],
    lb=np.array([1.]),
    ub=np.array([1.]),
    vars=p1)

# Add the constraint that p2 is on the curve y=x*x
prog.AddConstraint(
    lambda z: [z[1] - z[0]**2],
    lb=[0.],
    ub=[0.],
    vars=p2)

# Add the cost on the distance between p1 and p2
prog.AddQuadraticCost((p1-p2).dot(p1-p2))

# Set the value of p1 in initial guess to be [0, 1]
prog.SetInitialGuess(p1, [0., 1.])
# Set the value of p2 in initial guess to be [1, 1]
prog.SetInitialGuess(p2, [1., 1.])

# Now solve the program
result = Solve(prog)
print(f"Is optimization successful? {result.is_success()}")
p1_sol = result.GetSolution(p1)
p2_sol = result.GetSolution(p2)
print(f"solution to p1 {p1_sol}")
print(f"solution to p2 {p2_sol}")
print(f"optimal cost {result.get_optimal_cost()}")

# Plot the solution.
plt.figure()
plt.plot(np.cos(np.linspace(0, 2*np.pi, 100)), 2+np.sin(np.linspace(0, 2*np.pi, 100)))
plt.plot(np.linspace(-2, 2, 100), np.power(np.linspace(-2, 2, 100), 2))
plt.plot(p1_sol[0], p1_sol[1], '*')
plt.plot(p2_sol[0], p2_sol[1], '*')
plt.axis('equal')
plt.show()
```


# Neural networks

Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.

The field of artificial neural networks has a long history of
development, and is closely connected with the advancement of computer
science and computers in general. A model of artificial neurons was
first developed by McCulloch and Pitts in 1943 to study signal
processing in the brain and has later been refined by others. The
general idea is to mimic neural networks in the human brain, which is
composed of billions of neurons that communicate with each other by
sending electrical signals.  Each neuron accumulates its incoming
signals, which must exceed an activation threshold to yield an
output. If the threshold is not overcome, the neuron remains inactive,
i.e. has zero output.

This behaviour has inspired a simple mathematical model for an artificial neuron.

$$
\begin{equation}
 y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u)
\label{artificialNeuron} \tag{1}
\end{equation}
$$

Here, the output $y$ of the neuron is the value of its activation function, which have as input
a weighted sum of signals $x_i, \dots ,x_n$ received by $n$ other neurons.

Conceptually, it is helpful to divide neural networks into four
categories:
1. general purpose neural networks for supervised learning,

2. neural networks designed specifically for image processing, the most prominent example of this class being Convolutional Neural Networks (CNNs),

3. neural networks for sequential data such as Recurrent Neural Networks (RNNs), and

4. neural networks for unsupervised learning such as Deep Boltzmann Machines.

In natural science, DNNs and CNNs have already found numerous
applications. In statistical physics, they have been applied to detect
phase transitions in 2D Ising and Potts models, lattice gauge
theories, and different phases of polymers, or solving the
Navier-Stokes equation in weather forecasting.  Deep learning has also
found interesting applications in quantum physics. Various quantum
phase transitions can be detected and studied using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive achievements to reveal the
potential of DNNs to facilitate the study of quantum systems.

In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural.

The applications are not limited to the natural sciences. There is a
plethora of applications in essentially all disciplines, from the
humanities to life science and medicine.

An artificial neural network (ANN), is a computational model that
consists of layers of connected neurons, or nodes or units.  We will
refer to these interchangeably as units or nodes, and sometimes as
neurons.

It is supposed to mimic a biological nervous system by letting each
neuron interact with other neurons by sending signals in the form of
mathematical functions between layers.  A wide variety of different
ANNs have been developed, but most of them consist of an input layer,
an output layer and eventual layers in-between, called *hidden
layers*. All layers can contain an arbitrary number of nodes, and each
connection between two nodes is associated with a weight variable.

Neural networks (also called neural nets) are neural-inspired
nonlinear models for supervised learning.  As we will see, neural nets
can be viewed as natural, more powerful extensions of supervised
learning methods such as linear and logistic regression and soft-max
methods we discussed earlier.

### Feed-forward neural networks

The feed-forward neural network (FFNN) was the first and simplest type
of ANNs that were devised. In this network, the information moves in
only one direction: forward through the layers.

Nodes are represented by circles, while the arrows display the
connections between the nodes, including the direction of information
flow. Additionally, each arrow corresponds to a weight variable
(figure to come).  We observe that each node in a layer is connected
to *all* nodes in the subsequent layer, making this a so-called
*fully-connected* FFNN.

### Convolutional Neural Network

A different variant of FFNNs are *convolutional neural networks*
(CNNs), which have a connectivity pattern inspired by the animal
visual cortex. Individual neurons in the visual cortex only respond to
stimuli from small sub-regions of the visual field, called a receptive
field. This makes the neurons well-suited to exploit the strong
spatially local correlation present in natural images. The response of
each neuron can be approximated mathematically as a convolution
operation.  (figure to come)

Convolutional neural networks emulate the behaviour of neurons in the
visual cortex by enforcing a *local* connectivity pattern between
nodes of adjacent layers: Each node in a convolutional layer is
connected only to a subset of the nodes in the previous layer, in
contrast to the fully-connected FFNN.  Often, CNNs consist of several
convolutional layers that learn local features of the input, with a
fully-connected layer at the end, which gathers all the local data and
produces the outputs. They have wide applications in image and video
recognition.

### Recurrent neural networks

So far we have only mentioned ANNs where information flows in one
direction: forward. *Recurrent neural networks* on the other hand,
have connections between nodes that form directed *cycles*. This
creates a form of internal memory which are able to capture
information on what has been calculated before; the output is
dependent on the previous computations. Recurrent NNs make use of
sequential information by performing the same task for every element
in a sequence, where each element depends on previous elements. An
example of such information is sentences, making recurrent NNs
especially well-suited for handwriting and speech recognition.

### Other types of networks

There are many other kinds of ANNs that have been developed. One type
that is specifically designed for interpolation in multidimensional
space is the radial basis function (RBF) network. RBFs are typically
made up of three layers: an input layer, a hidden layer with
non-linear radial symmetric activation functions and a linear output
layer (''linear'' here means that each node in the output layer has a
linear activation function). The layers are normally fully-connected
and there are no cycles, thus RBFs can be viewed as a type of
fully-connected FFNN. They are however usually treated as a separate
type of NN due the unusual activation functions.

## Multilayer perceptrons

One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.

Such networks are often called *multilayer perceptrons* (MLPs).

According to the *Universal approximation theorem*, a feed-forward
neural network with just a single hidden layer containing a finite
number of neurons can approximate a continuous multidimensional
function to arbitrary accuracy, assuming the activation function for
the hidden layer is a **non-constant, bounded and
monotonically-increasing continuous function**.

Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.

The output $y$ is produced via the activation function $f$

$$
y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(z),
$$

This function receives $x_i$ as inputs.
Here the activation $z=(\sum_{i=1}^n w_ix_i+b_i)$.
In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
of the outputs of *all* neurons in the previous layer.

First, for each node $i$ in the first hidden layer, we calculate a weighted sum $z_i^1$ of the input coordinates $x_j$,

$$
\begin{equation} z_i^1 = \sum_{j=1}^{M} w_{ij}^1 x_j + b_i^1
\label{_auto1} \tag{2}
\end{equation}
$$

Here $b_i$ is the so-called bias which is normally needed in
case of zero activation weights or inputs. How to fix the biases and
the weights will be discussed below.  The value of $z_i^1$ is the
argument to the activation function $f_i$ of each node $i$, The
variable $M$ stands for all possible inputs to a given node $i$ in the
first layer.  We define  the output $y_i^1$ of all neurons in layer 1 as

$$
\begin{equation}
 y_i^1 = f(z_i^1) = f\left(\sum_{j=1}^M w_{ij}^1 x_j  + b_i^1\right)
\label{outputLayer1} \tag{3}
\end{equation}
$$

where we assume that all nodes in the same layer have identical
activation functions, hence the notation $f$. In general, we could assume in the more general case that different layers have different activation functions.
In this case we would identify these functions with a superscript $l$ for the $l$-th layer,

$$
\begin{equation}
 y_i^l = f^l(u_i^l) = f^l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)
\label{generalLayer} \tag{4}
\end{equation}
$$

where $N_l$ is the number of nodes in layer $l$. When the output of
all the nodes in the first hidden layer are computed, the values of
the subsequent layer can be calculated and so forth until the output
is obtained.

The output of neuron $i$ in layer 2 is thus,

$$
\begin{equation}
 y_i^2 = f^2\left(\sum_{j=1}^N w_{ij}^2 y_j^1 + b_i^2\right)
\label{_auto2} \tag{5}
\end{equation}
$$

$$
\begin{equation}
 = f^2\left[\sum_{j=1}^N w_{ij}^2f^1\left(\sum_{k=1}^M w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]
\label{outputLayer2} \tag{6}
\end{equation}
$$

where we have substituted $y_k^1$ with the inputs $x_k$. Finally, the ANN output reads

$$
\begin{equation}
 y_i^3 = f^3\left(\sum_{j=1}^N w_{ij}^3 y_j^2 + b_i^3\right)
\label{_auto3} \tag{7}
\end{equation}
$$

$$
\begin{equation}
 = f_3\left[\sum_{j} w_{ij}^3 f^2\left(\sum_{k} w_{jk}^2 f^1\left(\sum_{m} w_{km}^1 x_m + b_k^1\right) + b_j^2\right)
  + b_1^3\right]
\label{_auto4} \tag{8}
\end{equation}
$$

We can generalize this expression to an MLP with $l$ hidden
layers. The complete functional form is,

$$
\begin{equation}
y^{l+1}_i = f^{l+1}\left[\!\sum_{j=1}^{N_l} w_{ij}^3 f^l\left(\sum_{k=1}^{N_{l-1}}w_{jk}^{l-1}\left(\dots f^1\left(\sum_{n=1}^{N_0} w_{mn}^1 x_n+ b_m^1\right)\dots\right)+b_k^2\right)+b_1^3\right]
\label{completeNN} \tag{9}
\end{equation}
$$

which illustrates a basic property of MLPs: The only independent
variables are the input values $x_n$.

This confirms that an MLP, despite its quite convoluted mathematical
form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors $\hat{x} \in \mathbb{R}^n \rightarrow
\hat{y} \in \mathbb{R}^m$.

Furthermore, the flexibility and universality of an MLP can be
illustrated by realizing that the expression is essentially a nested
sum of scaled activation functions of the form

$$
\begin{equation}
 f(x) = c_1 f(c_2 x + c_3) + c_4
\label{_auto5} \tag{10}
\end{equation}
$$

where the parameters $c_i$ are weights and biases. By adjusting these
parameters, the activation functions can be shifted up and down or
left and right, change slope or be rescaled which is the key to the
flexibility of a neural network.

We can introduce a more convenient notation for the activations in an A NN.

Additionally, we can represent the biases and activations
as layer-wise column vectors $\hat{b}_l$ and $\hat{y}_l$, so that the $i$-th element of each vector
is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively.

We have that $\mathrm{W}_l$ is an $N_{l-1} \times N_l$ matrix, while $\hat{b}_l$ and $\hat{y}_l$ are $N_l \times 1$ column vectors.
With this notation, the sum becomes a matrix-vector multiplication, and we can write
the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as

$$
\begin{equation}
 \hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) =
 f_2\left(\left[\begin{array}{ccc}
    w^2_{11} &w^2_{12} &w^2_{13} \\
    w^2_{21} &w^2_{22} &w^2_{23} \\
    w^2_{31} &w^2_{32} &w^2_{33} \\
    \end{array} \right] \cdot
    \left[\begin{array}{c}
           y^1_1 \\
           y^1_2 \\
           y^1_3 \\
          \end{array}\right] +
    \left[\begin{array}{c}
           b^2_1 \\
           b^2_2 \\
           b^2_3 \\
          \end{array}\right]\right).
\label{_auto6} \tag{11}
\end{equation}
$$

### Matrix-vector notation  and activation

The activation of node $i$ in layer 2 is

$$
\begin{equation}
 y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
 f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
\label{_auto7} \tag{12}
\end{equation}
$$

This is not just a convenient and compact notation, but also a useful
and intuitive way to think about MLPs: The output is calculated by a
series of matrix-vector multiplications and vector additions that are
used as input to the activation functions. For each operation
$\mathrm{W}_l \hat{y}_{l-1}$ we move forward one layer.

### Activation functions

A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s).  As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem

  * Non-constant

  * Bounded

  * Monotonically-increasing

  * Continuous

The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.

Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic *Sigmoid*

$$
f(x) = \frac{1}{1 + e^{-x}},
$$

and the *hyperbolic tangent* function

$$
f(x) = \tanh(x)
$$

The *sigmoid* function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called *one-sided*. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs.  has
become the most popular for *deep neural networks*

```python
%matplotlib inline

"""The sigmoid function (or the logistic curve) is a
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm."""

import numpy
import matplotlib.pyplot as plt
import math as mt

z = numpy.arange(-5, 5, .1)
sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
sigma = sigma_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, sigma)
ax.set_ylim([-0.1, 1.1])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('sigmoid function')

plt.show()

"""Step Function"""
z = numpy.arange(-5, 5, .02)
step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
step = step_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, step)
ax.set_ylim([-0.5, 1.5])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('step function')

plt.show()

"""Sine Function"""
z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
t = numpy.sin(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, t)
ax.set_ylim([-1.0, 1.0])
ax.set_xlim([-2*mt.pi,2*mt.pi])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('sine function')

plt.show()

"""Plots a graph of the squashing function used by a rectified linear
unit"""
z = numpy.arange(-2, 2, .1)
zero = numpy.zeros(len(z))
y = numpy.max([zero, z], axis=0)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, y)
ax.set_ylim([-2.0, 2.0])
ax.set_xlim([-2.0, 2.0])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('Rectified linear unit')

plt.show()
```

## The multilayer  perceptron (MLP)

The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
1. A neural network with one or more layers of nodes between the input and the output nodes.

2. The multilayer network structure, or architecture, or topology, consists of an input layer, one or more hidden layers, and one output layer.

3. The input nodes pass values to the first hidden layer, its nodes pass the information on to the second and so on till we reach the output layer.

As a convention it is normal to call  a  network with one layer of input units, one layer of hidden
units and one layer of output units as  a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.

For an MLP network there is no direct connection between the output nodes/neurons/units and  the input nodes/neurons/units.
Hereafter we will call the various entities of a layer for nodes.
There are also no connections within a single layer.

The number of input nodes does not need to equal the number of output
nodes. This applies also to the hidden layers. Each layer may have its
own number of nodes and activation functions.

The hidden layers have their name from the fact that they are not
linked to observables and as we will see below when we define the
so-called activation $\hat{z}$, we can think of this as a basis
expansion of the original inputs $\hat{x}$. The difference however
between neural networks and say linear regression is that now these
basis functions (which will correspond to the weights in the network)
are learned from data. This results in  an important difference between
neural networks and deep learning approaches on one side and methods
like logistic regression or linear regression and their modifications on the other side.

### From one to many layers, the universal approximation theorem

A neural network with only one layer, what we called the simple
perceptron, is best suited if we have a standard binary model with
clear (linear) boundaries between the outcomes. As such it could
equally well be replaced by standard linear regression or logistic
regression. Networks with one or more hidden layers approximate
systems with more complex boundaries.

As stated earlier,
an important theorem in studies of neural networks, restated without
proof here, is the [universal approximation
theorem](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf).

It states that a feed-forward network with a single hidden layer
containing a finite number of neurons can approximate continuous
functions on compact subsets of real functions. The theorem thus
states that simple neural networks can represent a wide variety of
interesting functions when given appropriate parameters. It is the
multilayer feedforward architecture itself which gives neural networks
the potential of being universal approximators.

## Deriving the back propagation code for a multilayer perceptron model

As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
The unknowwn quantities are our weights $w_{ij}$ and we need to find an algorithm for changing them so that our errors are as small as possible.
This leads us to the famous [back propagation algorithm](https://www.nature.com/articles/323533a0).

The questions we want to ask are how do changes in the biases and the
weights in our network change the cost function and how can we use the
final output to modify the weights?

To derive these equations let us start with a plain regression problem
and define our cost function as

$$
{\cal C}(\hat{W})  =  \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2,
$$

where the $t_i$s are our $n$ targets (the values we want to
reproduce), while the outputs of the network after having propagated
all inputs $\hat{x}$ are given by $y_i$.  Below we will demonstrate
how the basic equations arising from the back propagation algorithm
can be modified in order to study classification problems with $K$
classes.

With our definition of the targets $\hat{t}$, the outputs of the
network $\hat{y}$ and the inputs $\hat{x}$ we
define now the activation $z_j^l$ of node/neuron/unit $j$ of the
$l$-th layer as a function of the bias, the weights which add up from
the previous layer $l-1$ and the forward passes/outputs
$\hat{a}^{l-1}$ from the previous layer as

$$
z_j^l = \sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,
$$

where $b_k^l$ are the biases from layer $l$.  Here $M_{l-1}$
represents the total number of nodes/neurons/units of layer $l-1$. The
figure here illustrates this equation.  We can rewrite this in a more
compact form as the matrix-vector products we discussed earlier,

$$
\hat{z}^l = \left(\hat{W}^l\right)^T\hat{a}^{l-1}+\hat{b}^l.
$$

With the activation values $\hat{z}^l$ we can in turn define the
output of layer $l$ as $\hat{a}^l = f(\hat{z}^l)$ where $f$ is our
activation function. In the examples here we will use the sigmoid
function discussed in our logistic regression lectures. We will also use the same activation function $f$ for all layers
and their nodes.  It means we have

$$
a_j^l = f(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}.
$$

### Derivatives and the chain rule

From the definition of the activation $z_j^l$ we have

$$
\frac{\partial z_j^l}{\partial w_{ij}^l} = a_i^{l-1},
$$

and

$$
\frac{\partial z_j^l}{\partial a_i^{l-1}} = w_{ji}^l.
$$

With our definition of the activation function we have that (note that this function depends only on $z_j^l$)

$$
\frac{\partial a_j^l}{\partial z_j^{l}} = a_j^l(1-a_j^l)=f(z_j^l)(1-f(z_j^l)).
$$

With these definitions we can now compute the derivative of the cost function in terms of the weights.

Let us specialize to the output layer $l=L$. Our cost function is

$$
{\cal C}(\hat{W^L})  =  \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2=\frac{1}{2}\sum_{i=1}^n\left(a_i^L - t_i\right)^2,
$$

The derivative of this function with respect to the weights is

$$
\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \left(a_j^L - t_j\right)\frac{\partial a_j^L}{\partial w_{jk}^{L}},
$$

The last partial derivative can easily be computed and reads (by applying the chain rule)

$$
\frac{\partial a_j^L}{\partial w_{jk}^{L}} = \frac{\partial a_j^L}{\partial z_{j}^{L}}\frac{\partial z_j^L}{\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1},
$$

### Bringing it together, first back propagation equation

We have thus

$$
\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \left(a_j^L - t_j\right)a_j^L(1-a_j^L)a_k^{L-1},
$$

Defining

$$
\delta_j^L = a_j^L(1-a_j^L)\left(a_j^L - t_j\right) = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)},
$$

and using the Hadamard product of two vectors we can write this as

$$
\hat{\delta}^L = f'(\hat{z}^L)\circ\frac{\partial {\cal C}}{\partial (\hat{a}^L)}.
$$

This is an important expression. The second term on the right handside
measures how fast the cost function is changing as a function of the $j$th
output activation.  If, for example, the cost function doesn't depend
much on a particular output node $j$, then $\delta_j^L$ will be small,
which is what we would expect. The first term on the right, measures
how fast the activation function $f$ is changing at a given activation
value $z_j^L$.

Notice that everything in the above equations is easily computed.  In
particular, we compute $z_j^L$ while computing the behaviour of the
network, and it is only a small additional overhead to compute
$f'(z^L_j)$.  The exact form of the derivative with respect to the
output depends on the form of the cost function.
However, provided the cost function is known there should be little
trouble in calculating

$$
\frac{\partial {\cal C}}{\partial (a_j^L)}
$$

With the definition of $\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely

$$
\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \delta_j^La_k^{L-1}.
$$

It is also easy to see that our previous equation can be written as

$$
\delta_j^L =\frac{\partial {\cal C}}{\partial z_j^L}= \frac{\partial {\cal C}}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L},
$$

which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely

$$
\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}\frac{\partial b_j^L}{\partial z_j^L}=\frac{\partial {\cal C}}{\partial b_j^L},
$$

That is, the error $\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias.

We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are

**The starting equations.**

$$
\begin{equation}
\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \delta_j^La_k^{L-1},
\label{_auto8} \tag{13}
\end{equation}
$$

and

$$
\begin{equation}
\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)},
\label{_auto9} \tag{14}
\end{equation}
$$

and

$$
\begin{equation}
\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L},
\label{_auto10} \tag{15}
\end{equation}
$$

An interesting consequence of the above equations is that when the
activation $a_k^{L-1}$ is small, the gradient term, that is the
derivative of the cost function with respect to the weights, will also
tend to be small. We say then that the weight learns slowly, meaning
that it changes slowly when we minimize the weights via say gradient
descent. In this case we say the system learns slowly.

Another interesting feature is that is when the activation function,
represented by the sigmoid function here, is rather flat when we move towards
its end values $0$ and $1$ (see the above Python codes). In these
cases, the derivatives of the activation function will also be close
to zero, meaning again that the gradients will be small and the
network learns slowly again.

We need a fourth equation and we are set. We are going to propagate
backwards in order to the determine the weights and biases. In order
to do so we need to represent the error in the layer before the final
one $L-1$ in terms of the errors in the final output layer.

### Final back propagating equation

We have that (replacing $L$ with a general layer $l$)

$$
\delta_j^l =\frac{\partial {\cal C}}{\partial z_j^l}.
$$

We want to express this in terms of the equations for layer $l+1$. Using the chain rule and summing over all $k$ entries we have

$$
\delta_j^l =\sum_k \frac{\partial {\cal C}}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}=\sum_k \delta_k^{l+1}\frac{\partial z_k^{l+1}}{\partial z_j^{l}},
$$

and recalling that

$$
z_j^{l+1} = \sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},
$$

with $M_l$ being the number of nodes in layer $l$, we obtain

$$
\delta_j^l =\sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l),
$$

This is our final equation.

We are now ready to set up the algorithm for back propagation and learning the weights and biases.

### Setting up the Back propagation algorithm

The four equations  provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.

First, we set up the input data $\hat{x}$ and the activations
$\hat{z}_1$ of the input layer and compute the activation function and
the pertinent outputs $\hat{a}^1$.

Secondly, we perform then the feed forward till we reach the output
layer and compute all $\hat{z}_l$ of the input layer and compute the
activation function and the pertinent outputs $\hat{a}^l$ for
$l=2,3,\dots,L$.

Thereafter we compute the ouput error $\hat{\delta}^L$ by computing all

$$
\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}.
$$

Then we compute the back propagate error for each $l=L-1,L-2,\dots,2$ as

$$
\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).
$$

Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\dots,2$ and update the weights and biases according to the rules

$$
w_{jk}^l\leftarrow  = w_{jk}^l- \eta \delta_j^la_k^{l-1},
$$

$$
b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l,
$$

The parameter $\eta$ is the learning parameter discussed in connection with the gradient descent methods.
Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.