# Session 9: Introduction to Support Vector Machines

------------------------------------------------------
Introduction to Data Science & Machine Learning

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

[Support vector machines (SVMs)](http://scikit-learn.org/stable/modules/svm.html) are a particularly powerful and flexible class of supervised algorithms for both classification and regression.
In this section, we will develop the intuition behind support vector machines and their use in classification problems.

Recommended Bibliography:

- Chapter 7 of Pattern Recognition and Machine Learning, Christopher Bishop, 2006
- Chapter 17 of [Bayesian Reasoning and Machine Learning](http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf), David Barber, 2010
- Chapter 14 of Machine Learning: a probabilistic perspective, Kevin Murphy 2012
- This excellent [post](http://efavdb.com/svm-classification/) by Andrew Ng.

We begin with the standard imports:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# use seaborn plotting defaults
import seaborn as sns; sns.set()
```

Consider the simple case of a classification task, in which the two classes of points are well separated. Check out first what the [make blobs](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_blobs.html) function does.

Also, to learn about color maps in matplotlib check [here](https://matplotlib.org/users/colormaps.html)

```python
from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=100, centers=2,
                  random_state=10, cluster_std=3)    #With random_state we fix the random seed

#We separate away some data for test
X_test = X[50:-1,:]
y_test = y[50:-1]

X = X[:50,:]
y = y[:50]

print("The shape of X is ",X.shape)

print("y is a label vector. The first 10 labels are:", y[:10])

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu')

plt.rcParams["figure.figsize"] = [8,8]

```

A linear discriminative classifier would attempt to draw a straight line separating the two sets of data, and thereby create a model for classification. ** However, note there are many possible solutions!! **

```python
xfit = np.linspace(-5, 15)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu')

m,b = (1.5, -9.3)
plt.plot(xfit, m * xfit + b, '-k')

m,b = (1, -5.5)
plt.plot(xfit, m * xfit + b, '-m')

m,b = (-0.2,0.1)
plt.plot(xfit, m * xfit + b, '-g')

plt.rcParams["figure.figsize"] = [8,8]
```

** Which one do you think separates best the data? **

Lets plot some *test data* with the right category.

```python
xfit = np.linspace(-5, 15)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu',label='Train')

plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test, s=50, cmap='RdBu',marker='x',label='Test')

plt.legend()

m,b = (1.5, -9.3)
plt.plot(xfit, m * xfit + b, '-k')

m,b = (1, -5.5)
plt.plot(xfit, m * xfit + b, '-m')

m,b = (-0.2,0.1)
plt.plot(xfit, m * xfit + b, '-g')

plt.rcParams["figure.figsize"] = [8,8]
```

# Maximizing the *Margin*

Support vector machines offer one way to improve on this.
The intuition is this: rather than simply drawing a zero-width line between the classes, we can draw around each line a *margin* of some width, up to the nearest point. The largest the margin is, the more robust the model is and generalizes better.

Here is an example of how this might look:

```python
xfit = np.linspace(-5, 15)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu')

m,b,d = (1.5, -9.3,0.01)
yfit = m * xfit + b
plt.plot(xfit,yfit, '-k')
plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',color='#AAAAAA', alpha=0.4)

m,b,d = (1, -5.5,1)
yfit = m * xfit + b
plt.plot(xfit,yfit, '-m')
plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',color='#AAAAAA', alpha=0.4)

m,b,d = (-0.2,0.1,0.05)
yfit = m * xfit + b
plt.plot(xfit,yfit, '-g')
plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',color='#AAAAAA', alpha=0.4)

plt.rcParams["figure.figsize"] = [8,8]
```

In support vector machines, the line that maximizes this margin is the one we will choose as the optimal model.
Support vector machines are an example of such a *maximum margin* estimator.

# A little bit of geometry: hyperplanes

### Hyperplanes

First of all, for $\mathbf{x}\in\mathbb{R}^D$, consider the function
\begin{align}
y(\mathbf{x}) = \mathbf{w}^T\mathbf{x}+w_0
\end{align}

Then, the set of points $\mathbf{x}\in\mathbb{R}^D$ such that $y(\mathbf{x})=0$ is called an [**hyperplane**](http://mathworld.wolfram.com/Hyperplane.html), a subspace of dimension $D-1$. E.g., for $D=2$, then the hyperplan is a line. For $D=3$ is a plane.

#### Some useful results with hyperplanes

A) If $\mathbf{x}_A$ and $\mathbf{x}_B$ are two points in the hyperplane, then the vector $\mathbf{v}=\alpha(\mathbf{x}_A-\mathbf{x}_B)$ is *orthogonal* to $\mathbf{w}$ for $\alpha\in\mathbb{R}$. Thus, $\mathbf{w}$ is orthogonal to any vector **contained** in the hyperplane.

B) The normal distance between the origin $\mathbf{0}$ and the closest point in the hyperplane is given by $-\frac{w_0}{||\mathbf{w}||_2}$, where a negative distance denotes that the hyperplane lies below the origin, and $||\mathbf{w}||_2=\sqrt{\mathbf{w}^T\mathbf{w}}$.

C) The normal distance to any point $\mathbf{x}^*\in\mathbb{R}^D$ to the hyperplane is
\begin{align}
\frac{y(\mathbf{x}^*)}{||\mathbf{w}||_2}
\end{align}
This distance is positive for points **above** the hyperplane, and negative for points **below** the hyperplane.

#### Some useful results with hyperplanes (Proofs)

*Proof for A):* if $\mathbf{x}_A$ and $\mathbf{x}_B$ are two points in the hyperplane, then $\mathbf{w}^T\mathbf{x}_A+w_0=\mathbf{w}^T\mathbf{x}_B+w_0=0$. Thus, $\mathbf{w}^T\left(\mathbf{x}_A-\mathbf{x}_B\right)=0$.

*Proof for B):* Note that, for any $\mathbf{x}$ (not necesarily in the hyperplane), the projection of $\mathbf{x}$ into $\mathbf{w}$ is
\begin{align}
\frac{ \mathbf{w}^T\mathbf{x}}{||\mathbf{w}||_2} = \frac{||\mathbf{w}||_2||\mathbf{x}||_2\cos(\phi)}{||\mathbf{w}||_2}=||\mathbf{x}||_2\cos(\phi),
\end{align}
where $\phi$ is the angle between $\mathbf{x}$ and $||\mathbf{w}||_2$. Note that this projection is the normal distance between the $\mathbf{0}$ point and the hyperplane orthogonal to $ ||\mathbf{w}||_2$ that passes through $\mathbf{x}$.

Further, if $\mathbf{x}$ belongs to the hyperplane $y(\mathbf{x}) = \mathbf{w}^T\mathbf{x}+w_0$, then
\begin{align}
||\mathbf{x}||_2\cos(\phi) = -\frac{w_0}{||\mathbf{w}||_2}
\end{align}

*Proof for C) Given any point $\mathbf{x}$ in the hyperplane $y(\mathbf{x}) = \mathbf{w}^T\mathbf{x}+w_0=0$, the normal distance of $\mathbf{x}^*$ to the hyperplane is the proyection of $(\mathbf{x}^*-\mathbf{x})$ into $\mathbf{w}$ (since $\mathbf{w}$ is perpedincular to the plane). Hence
\begin{align}
\text{distance} = \frac{\mathbf{w}^T(\mathbf{x}^*-\mathbf{x})}{||\mathbf{w}||_2}=\frac{\mathbf{w}^T\mathbf{x}-\mathbf{w}^T\mathbf{x}^*}{||\mathbf{w}||_2}=\frac{\mathbf{w}^T\mathbf{x}^*-\mathbf{w}^T\mathbf{x}}{||\mathbf{w}||_2}=\frac{\mathbf{w}^T\mathbf{x}^*+w_0}{||\mathbf{w}||_2}=\frac{y(\mathbf{x}^*)}{||\mathbf{w}||_2}
\end{align}

# Support Vector Machines

Assume there exists an hyperplane that separates our data ** in a transformed space** $\mathcal{D}=(\phi(\mathbf{x})^{(i)},t^{(i)})$, $i=1,\ldots,N$, where $\mathbf{x}\in\mathbb{R}^D$ and $t^{(i)}\in\{-1,+1\}$. Then, there must exist an hyperplane $y(\mathbf{x}) = \mathbf{w}^T\phi(\mathbf{x})+w_0=0$ that verifies
\begin{align}
t^{(i)}y(\mathbf{x}^{(i)})\geq 0, ~~ \forall (\mathbf{x}^{(i)},t^{(i)})
\end{align}

## The SVM: maximizing the margin

The SVM optimization problem reads as follows

\begin{align}
\arg\max_{\mathbf{w},w_0} \left\{ \min_{i} \frac{t^{(i)}y(\mathbf{x}^{(i)})}{||\mathbf{w}||_2}\right\}
\end{align}

However, this can be written in a simpler way. Since the distance of any point $\mathbf{x}^{(i)}$ to the hyperplane is invariant to a scale of the form $\mathbf{w}\leftarrow \eta\mathbf{w}$, $w_0\leftarrow\eta w_0$, **then we can freely set the minimum distance to the hyperplane to $||\mathbf{w}||_2$**, rescaling the whole problem. Thus, any point in the training set must verify

\begin{align}
t^{(i)}y(\mathbf{x}^{(i)})\geq 1, ~~ \forall (\mathbf{x}^{(i)},t^{(i)})
\end{align}

And the equivalent problem can be written follows:

\begin{align}
&\min_{\mathbf{w},w_0} = \frac{1}{2} ||\mathbf{w}||^2_2\\
&\text{subject to}
&t^{(i)}y(\mathbf{x}^{(i)})\geq 1, ~ i=1,\ldots,N
\end{align}

where we have introduced $||\mathbf{w}||^2_2$ as objective function and the factor $\frac{1}{2}$ for later convenience. This optimization problem is an example of a [**Quadratic Programming (QP)**](https://sites.math.washington.edu/~burke/crs/408f/notes/qp/qp1.pdf) optimization problem. Very efficient solvers are known for these kind of problems. Complexity scales cubic in the input dimension, i.e., $\mathcal{O}(D^3)$.

Lets visualize the solution for our running example.

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear', C=1E10)   # We use a linear kernel (no transformation).
                                       # Also, we explain below the role of C
model.fit(X, y)
```

    SVC(C=10000000000.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

The following function plots the SVM decision boundaries for us:

```python
def plot_svc_decision_function(model, ax=None, plot_support=True):
    """Plot the decision function for a 2D SVC"""
    if ax is None:            #If no figure handle is provided, it opens the current figure
        ax = plt.gca()

    xlim = ax.get_xlim()
    ylim = ax.get_ylim()

    # create grid to evaluate model
    x = np.linspace(xlim[0], xlim[1], 30)    #30 points in the grid axis
    y = np.linspace(ylim[0], ylim[1], 30)
    Y, X = np.meshgrid(y, x)                 # We create a grid with the x,y coordinates defined above

    # From the grid to a list of (x,y) values.
    # Check Numpy help for ravel()

    xy = np.vstack([X.ravel(), Y.ravel()]).T
    P = model.decision_function(xy).reshape(X.shape)

    # plot decision boundary and margins
    # In levels we provide a list of floating point numbers indicating
    #the level curves to draw, in increasing order; e.g., to draw just the zero contour pass
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

    # plot support vectors
    if plot_support:
        ax.scatter(model.support_vectors_[:, 0],
                   model.support_vectors_[:, 1],
                   s=300, linewidth=1, marker='p')
    ax.set_xlim(xlim)
    ax.set_ylim(ylim)
```

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model,plot_support=False)

plt.rcParams["figure.figsize"] = [8,8]
```

# Understanding the SVM solution:  kernels and the dual problem

There's lot more that we can say about how SVM perform. To this end, we have to go deeper into the optimization problem itself:

\begin{align}
&\min_{\mathbf{w},w_0} = \frac{1}{2} ||\mathbf{w}||^2_2\\
&\text{subject to}\\
&t^{(i)}y(\mathbf{x}^{(i)})\geq 1, ~ i=1,\ldots,N
\end{align}

## Introducing Lagrange Multipliers

Given the constraints of the problem, the Lagrange function that has to be optimized is of the form

\begin{align}
\mathcal{L}(\mathbf{w},w_0,\mathbf{a}) = \frac{||\mathbf{w}||^2_2}{2} -\sum_{i=1}^N a_i (t^{(i)}y(\mathbf{x}^{(i)})-1),
\end{align}
where $a_i\geq 0$, $i=1,\ldots,N$ are the Lagrange multipliers. If we compute the gradient of $\mathcal{L}(\mathbf{w},w_0,\mathbf{a})$ w.r.t. $\mathbf{w}$ and $w_0$ and equalize to zero we get the following conditions

\begin{align}
\mathbf{w} &=\sum_{i=1}^N a_i t^{(i)} \phi(\mathbf{x}^{(i)}), ~~~~~~
0 = \sum_{i=1}^{N} a_i  t^{(i)}
\end{align}
## The Dual Problem

If we introduce the above expressions in the the Lagrange function, our optimization problem reads

\begin{align}
\mathcal{L}(\mathbf{a}) &= \sum_{i=1}^{N}a_i -\frac{1}{2}\sum_{i=1}^N \sum_{j=1}^N a_i a_j k(\mathbf{x}^{(i)},\mathbf{x}^{(j)})\\
&\text{subject to} \\
a_i&\geq 0, ~ i=1,\ldots,N\\
0 &= \sum_{i=1}^{N} a_i  t^{(i)}
\end{align}

where $k(\mathbf{x}^{(i)},\mathbf{x}^{(j)})=\phi(\mathbf{x}^{(i)})^T\phi(\mathbf{x}^{(j)})$ is the **kernel** between points $\mathbf{x}^{(i)}$ and $\mathbf{x}^{(j)}$. This problem is another instance of **Quadratic Programming**. The resolution of the problem in this dual space is $\mathcal{O}(N^3)$ complex.

Given the solution, we classify a new point according to the sign of
\begin{align}
y(\mathbf{x}^*) = \sum_{i=1}^{N} a_i t^{(i)} k(\mathbf{x}^{(i)},\mathbf{x}^*)+w_0
\end{align}

Lets visualize the fit of a kernel SVM with the **RBF** kernel

```python
from sklearn.datasets.samples_generator import make_circles
Xc, yc = make_circles(20, factor=0.25, noise=.1)

model2 = SVC(kernel='rbf',C=10^6).fit(Xc, yc)

f,ax = plt.subplots(1,2)
ax[0].scatter(Xc[:, 0], Xc[:, 1], c=yc, s=50, cmap='autumn')
ax[1].scatter(Xc[:, 0], Xc[:, 1], c=yc, s=50, cmap='autumn')
plot_svc_decision_function(model2, ax=ax[1],plot_support=False)

plt.rcParams["figure.figsize"] = [8,8]
```

# SVMs are sparse!

There is even more we can say about the SVM solution. In fact, we will see that the SVM solution is determine only by a **subset** of training points, which are known as **support vectors**.

Despite we do not prove it, given the problem
\begin{align}
\mathcal{L}(\mathbf{a}) &= \sum_{i=1}^{N}a_i -\frac{1}{2}\sum_{i=1}^N \sum_{j=1}^N a_i a_j k(\mathbf{x}^{(i)},\mathbf{x}^{(j)})\\
&\text{subject to} \\
a_i&\geq 0, ~ i=1,\ldots,N\\
0 &= \sum_{i=1}^{N} a_i  t^{(i)}
\end{align}

the [**Karush-Kuhn-Tucker (KKT)**](http://www.onmyphd.com/?p=kkt.karush.kuhn.tucker) conditions require that the solution of the problem must verify the following. For $i=1,\ldots,N$,

- $a_i\geq 0$
- $t^{(i)}(y(\mathbf{x}^{(i)})-1)\geq 0$
- $a_i\left(t^{(i)}(y(\mathbf{x}^{(i)})-1)\right)=0$

If you want to understand how to prove this results, check out Appendix E in Bishop's book.

## Support Vectors

The third condition above implies that, for any point in our training set, either $a_i=0$ or $y(\mathbf{x}^{(i)})-1$. This means that either the point **do not participate in the SVM solution** or that the point lies **exactly in the margin**. Points for which $a_i>0$ are called **support vectors** and are the only ones defining the separation hyperplane and the prediction for future values!:

\begin{align}
\mathbf{w} &=\sum_{i=1}^N a_i t^{(i)} \phi(\mathbf{x}^{(i)})\\
w_0 &= \frac{1}{N_S}\sum_{i: a_i>0} \left(t^{(i)}-\sum_{j: a_j>0} a_j t^{(j)}k(\mathbf{x}^{(i)},\mathbf{x}^{(j)})\right)\\
y(\mathbf{x}^*) &= \sum_{i=1}^{N} a_i t^{(i)} k(\mathbf{x}^{(i)},\mathbf{x}^*)+w_0
\end{align}

The fraction of support vectors w.r.t. to the total number of training points must be read as a measure of the complexity of the model and how much it is exposed to **overfitting**. The more we have, the poorest generalization we can expect.

Lets plot the support vectors in the two examples we have done before.

```python
f,ax = plt.subplots(1,2)

ax[0].scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model, ax=ax[0],plot_support=True)

ax[1].scatter(Xc[:, 0], Xc[:, 1], c=yc, s=50, cmap='autumn')
plot_svc_decision_function(model2, ax=ax[1],plot_support=True)

```

# Dealing with Non-separable datasets

So far, the whole SVM formulation builds up over the assumption that the data is separable by an hyperplane in a transformed space. I.e., that there exists an hyperplane $y(\mathbf{x}) = \mathbf{w}^T\phi(\mathbf{x})+w_0=0$ that verifies
\begin{align}
t^{(i)}y(\mathbf{x}^{(i)})\geq 1, ~~ \forall (\mathbf{x}^{(i)},t^{(i)}),
\end{align}
where points with equality are the support vectors.

In order to **relax** this assumption and prevent **overfitting**, we could allow certain **training** points to be missclassified. We introduce the so-called **slack** variables:

\begin{align}
t^{(i)}y(\mathbf{x}^{(i)})\geq 1-\xi_i, ~~ \forall (\mathbf{x}^{(i)},t^{(i)})
\end{align}
where $\xi_i\geq 0$:
- Training points for which $\xi_i\leq 1$ are correctly classified.
- Training points for which $\xi_i > 1$ are in the wrong side of the decision boundary $y(\mathbf{x})=0$.

## Optimization problem with slack variables

The optimization problem can now be written as follows:
\begin{align}
&\min_{\mathbf{w},w_0} = \frac{1}{2} ||\mathbf{w}||^2_2 + C \sum_{i=1}^{N}\xi_i\\
&\text{subject to}\\
&t^{(i)}y(\mathbf{x}^{(i)})\geq 1-\xi_i, ~ i=1,\ldots,N\\
&\xi_i\geq 0, ~ i=1,\ldots,N
\end{align}
where note that the regularization is controlled by the $C$ parameter. For $C\rightarrow\infty$ we recover the original problem, as the only solution tends to $\xi_i=0$ for $i=1,\ldots,N$, and the original SVM formulation is recovered.

The Lagrange function to be optimized is now

\begin{align}
\mathcal{L}(\mathbf{w},w_0,\mathbf{a},\mathbf{b}) = \frac{||\mathbf{w}||^2_2}{2} + C \sum_{i=1}^{N}\xi_i -\sum_{i=1}^N a_i (t^{(i)}y(\mathbf{x}^{(i)})-1+\xi_i)-\sum_{i=1}^{N}b_i\xi_i,
\end{align}

### KKT conditions

The KKT conditions associated to the new optimization problem are as follows. For $i=1,\ldots,N$

$$\xi_n\geq 0$$

$$a_i\geq 0$$

$$b_i\geq 0$$

$$t^{(i)} (\mathbf{w}^T \phi(\mathbf{x})+b)-1+\xi_n)\geq 0$$

$$b_i\xi_n=0$$

$$a_i\left[t^{(i)} (y(\mathbf{x})-1+\xi_n)\right]=0$$

### Support Vectors

As before, the last condition implies that, for any point in our training set, either $a_i=0$ or $y(\mathbf{x}^{(i)})-1$. This means that either the point **do not participate in the SVM solution** or that the point lies **exactly in the margin**. Points for which $a_i>0$ are called **support vectors** and are the only ones defining the separation hyperplane and the prediction for future values!

### Dual problem

If we compute the gradient Lagrange function w.r.t. $\mathbf{w}$, $w_0$ and $\xi_i$ and equalize to zero we get the following conditions

\begin{align}
\mathbf{w} &=\sum_{i=1}^N a_i t^{(i)} \phi(\mathbf{x}^{(i)}), ~~~~~~
0 = \sum_{i=1}^{N} a_i  t^{(i)}, ~~~~~~~ a_i = C-b_i
\end{align}

If we substitute them in the Lagrange function, we derive the dual optimization problem, with complexity $\mathcal{O}(N^3)$.

Lets play with parameter $C$ with another example

```python
X, y = make_circles(100, factor=0.5, noise=.2)

model_lin = SVC(kernel='linear',C=10^8 ).fit(X, y)

f,ax = plt.subplots(1,2)

ax[0].scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
ax[1].scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model_lin, ax=ax[1],plot_support=True) # Try to plot the suppor vectors by changing the flag

plt.rcParams["figure.figsize"] = [8,8]
```

Clearly, a linear Kernel does not do the job

```python
model_rbf = SVC(kernel='rbf',C=1e6).fit(X, y)
```

```python
f, axes = plt.subplots(1, 2)
axes[0].scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model_rbf ,ax=axes[0], plot_support=False)
axes[1].scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model_rbf ,ax=axes[1], plot_support=True)

plt.rcParams["figure.figsize"] = [20,15]
```

And now with $C=10$ ...

```python
model_rbf_2 = SVC(kernel='rbf',C=1e1).fit(X, y)
```

```python
f, axes = plt.subplots(1, 2)
axes[0].scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model_rbf_2,ax=axes[0], plot_support=False)
axes[1].scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model_rbf_2,ax=axes[1], plot_support=True)
```

```python

```

Notebooks/Session 9 SVMs/.ipynb_checkpoints/Introduction to SVMs-checkpoint.ipynb


```python
import sympy as sy
from sympy.solvers.solveset import linsolve
import pandas_datareader as pdr
import datetime as dt
import statsmodels.api as sm
import pandas as pd
```

#  Terminologies Of Identification

Before discussing simultaneous-equation models, we should clarify some terminologies that are commonly seen in macroeconomic literature.

The **identification** was originally a statistical term which describes a model with multiple sets of parameters, but generates the same distribution, therefore the parameters can't be identified by investigating the generated observations.

In econometric research, identification concerns more about identifying the causality, especially in multiple equation models. For easy demonstration, below is a time series model of three equations. The subscript $t$ represents the time period.

This is all called **structural model**, because they may portray the structure of economy or behaviors of an economic agent.

\begin{align}
Y_{1t} &= \quad\qquad+\beta_{12}Y_{2t}+\beta_{13}Y_{3t} + \gamma_{11} X_{1t}+ \gamma_{12} X_{2t} +\gamma_{13} X_{3t}+u_{1t}\\
Y_{2t} &= \beta_{11}Y_{1t}\quad\quad\qquad+\beta_{13}Y_{3t} + \gamma_{21} X_{1t}+ \gamma_{22} X_{2t} +\gamma_{23} X_{3t}+u_{2t}\\
Y_{2t} &= \beta_{11}Y_{1t}+\beta_{12}Y_{2t} +\quad\qquad+ \gamma_{31} X_{1t}+ \gamma_{32} X_{2t} +\gamma_{33} X_{3t}+u_{2t}\\
\end{align}

$Y_1$, $Y_2$ and $Y_3$ are **endogenous** variables, i.e. values are determined within the model; $X_1$, $X_2$ and $X_3$  are the **predetermined** variables, which are treated as non-stochastic; $u_1$, $u_2$ and $u_3$ are disturbance terms. Note that we have ceased using terms such as _dependent_ or _independent_ variables as in single equation model any more.

The _predetermined_ variables are divided into two categories: **exogenous** (such as $X_{1t}, X_{1, t-1}$) and **lagged endogenous** (such as $Y_{1, t-1}$), which are not determined by the model in the current time period. But in practice, the researchers themselves should classify variables based on theoretical grounds or experience.

A common macroeconomic practice is to convert the structural form equations into the **reduced form equations**, which express endogenous variables solely in terms of the predetermined variables and disturbance terms.

It would be considerably concise to define corresponding matrices as following
$$
\boldsymbol{Y}_t =
\begin{bmatrix}
Y_{1t} \\
Y_{2t}\\
Y_{3t}
\end{bmatrix}
$$

$$
\boldsymbol{X}_t =
\begin{bmatrix}
X_{1t} \\
X_{2t}\\
X_{3t}
\end{bmatrix}
$$

$$
\boldsymbol{\beta} =
\begin{bmatrix}
0 & \beta_{12}  & \beta_{13}\\
\beta_{11} & 0 &\beta_{13} \\
\beta_{11} & \beta_{12} &0
\end{bmatrix}
$$

$$
\boldsymbol{\Gamma} =
\begin{bmatrix}
\gamma_{11} & \gamma_{12}  & \gamma_{13}\\
\gamma_{21} & \gamma_{22} &\gamma_{23} \\
\gamma_{31} & \gamma_{32} &\gamma_{33}
\end{bmatrix}
$$

$$
\boldsymbol{u}_t =
\begin{bmatrix}
u_{1}\\
u_{2} \\
u_{3}
\end{bmatrix}
$$

Then structural model in matrix form is
$$
\boldsymbol{Y}_t = \boldsymbol{\beta}\boldsymbol{Y}_t + \boldsymbol{\Gamma}\boldsymbol{X}_t + \boldsymbol{u}_t
$$
Rewrite as reduced-form
$$
\boldsymbol{Y}_t = (\boldsymbol{I}-\boldsymbol{\beta})^{-1}\boldsymbol{\Gamma}\boldsymbol{X}_t+(\boldsymbol{I}-\boldsymbol{\beta})^{-1}\boldsymbol{u}_t
$$

#  Keynesian Cross Model

Here is structural model, it describes the consumption of the economy
$$
\begin{array}{ll}
\text { Consumption function: } & C_{t}=\beta_{0}+\beta_{1} Y_{t}+u_{t} \quad 0<\beta_{1}<1 \\
\text { Income identity: } & Y_{t}=C_{t}+I_{t}
\end{array}
$$

Join both equations and write as reduced form
$$
Y_t  = \frac{\beta_0}{1-\beta_1}+\frac{I_t}{1-\beta_1}+\frac{u_t}{1-\beta_1}
$$

Or write as
$$
Y_{t}=\Pi_{0}+\Pi_{1} I_{t}+w_{t}\\
\begin{aligned}
\Pi_{0} &=\frac{\beta_{0}}{1-\beta_{1}} \\
\Pi_{1} &=\frac{1}{1-\beta_{1}} \\
w_{t} &=\frac{u_{t}}{1-\beta_{1}}
\end{aligned}
$$
Substitute back to consumption function
$$
\begin{gathered}
C_{t}=\Pi_{2}+\Pi_{3} I_{t}+w_{t} \\
\Pi_{2}=\frac{\beta_{0}}{1-\beta_{1}}\\
\Pi_{3}=\frac{\beta_{1}}{1-\beta_{1}} \\
w_{t}=\frac{u_{t}}{1-\beta_{1}}
\end{gathered}
$$

Once we have the reduced-form, we can estimate reduced-form coefficients by OLS without suffering from inconsistency.

#  Underidentification

This is the basic demand and supply model
\begin{align}
\text{Demand function:}& \quad Q_{t}^{d}=\alpha_{0}+\alpha_{1} P_{t}+u_{1 t} \quad \alpha_{1}<0\\
\text{Supply function:}& \quad Q_{t}^{s}=\beta_{0}+\beta_{1} P_{t}+u_{2 t} \quad \beta_{1}>0\\
\text{Equilibrium condition:}& \quad \mathrm{Q}_{t}^{d}=\mathrm{Q}_{t}^{s}
\end{align}
If $u_{1t}$ changes, it means the demand will change due to factor other than $P_t$, which causes the demand curve shifting; so is $u_{2t}$, which causes supply curve shifting.

However, notice that curve shifting will cause both $P$ and $Q$ change too, this means that $u_{1t}$ and $P_t$  are not independently distributed, so are $u_{2t}$ and $P_t$. This means we can't use OLS to estimate each equations independently.

Equate the demand and supply
$$\alpha_{0}+\alpha_{1} P_{t}+u_{1 t}=\beta_{0}+\beta_{1} P_{t}+u_{2 t}$$
Solve for equilibrium $P_t$
$$
\begin{gathered}
P_{t}=\Pi_{0}+v_{t} \\
\Pi_{0}=\frac{\beta_{0}-\alpha_{0}}{\alpha_{1}-\beta_{1}} \\
v_{t}=\frac{u_{2 t}-u_{1 t}}{\alpha_{1}-\beta_{1}}
\end{gathered}
$$
Substitute back to demand function to solve for equilibrium quantity
$$
\begin{gathered}
Q_{t}=\Pi_{1}+w_{t} \\
\Pi_{1}=\frac{\alpha_{1} \beta_{0}-\alpha_{0} \beta_{1}}{\alpha_{1}-\beta_{1}} \\
w_{t}=\frac{\alpha_{1} u_{2 t}-\beta_{1} u_{1 t}}{\alpha_{1}-\beta_{1}}
\end{gathered}
$$

We obtain the reduced-form equations in terms of structural coefficients and disturbance term. We can estimate the reduced-form coefficients $\Pi_{0}$  and $\Pi_{1}$ by OLS.

Let's say $\Pi_{0}=3$  and $\Pi_{1}=4$, but how do we pin down structural coefficients $\alpha_0$, $\alpha_1$, $\beta_0$ and $\beta_1$.
$$
\Pi_{0}=\frac{\beta_{0}-\alpha_{0}}{\alpha_{1}-\beta_{1}}\\
\Pi_{1}=\frac{\alpha_{1} \beta_{0}-\alpha_{0} \beta_{1}}{\alpha_{1}-\beta_{1}}
$$
Two equations, but four unknowns, there are infinite amount of combination of them to satisfy the restriction, this is the exact question of identification - how to identify the structural coefficient even if we have reduced-form coefficients?

The answer to this case: _we can't_.

Because there is not enough information to solve the system, we call this **underidentification**.

Since the problem of underidentification is due to lack of information, how about we give it more information and test if it can be identified.
\begin{align}
\text{Demand function:}& \quad Q^d_{t}=\alpha_{0}+\alpha_{1} P_{t}+\alpha_{2} I_{t}+u_{1 t} \quad &\alpha_{1}<0, \alpha_{2}>0\\
\text{Supply function:}&\quad Q^s_{t}=\beta_{0}+\beta_{1} P_{t}+u_{2 t} \quad &\beta_{1}>0
\end{align}
where $I$ is the income of the family.

Equate both function
$$
\alpha_{0}+\alpha_{1} P_{t}+\alpha_{2} I_{t}+u_{1 t}=\beta_{0}+\beta_{1} P_{t}+u_{2 t}
$$
And solve for the equilibrium $P_t$
$$
P_{t}=\Pi_{0}+\Pi_{1} I_{t}+v_{t}\\
\begin{aligned}
\Pi_{0} &=\frac{\beta_{0}-\alpha_{0}}{\alpha_{1}-\beta_{1}} \\
\Pi_{1} &=-\frac{\alpha_{2}}{\alpha_{1}-\beta_{1}} \\
v_{t} &=\frac{u_{2 t}-u_{1 t}}{\alpha_{1}-\beta_{1}}
\end{aligned}
$$
Substitute back to demand function to solve for $Q_t$
$$
\begin{aligned}
&Q_{t}=\Pi_{2}+\Pi_{3} I_{t}+w_{t} \\
&\Pi_{2}=\frac{\alpha_{1} \beta_{0}-\alpha_{0} \beta_{1}}{\alpha_{1}-\beta_{1}} \\
&\Pi_{3}=-\frac{\alpha_{2} \beta_{1}}{\alpha_{1}-\beta_{1}} \\
&w_{t}=\frac{\alpha_{1} u_{2 t}-\beta_{1} u_{1 t}}{\alpha_{1}-\beta_{1}}
\end{aligned}
$$

Let's assume $\Pi_0 = 2$, $\Pi_1 = 3$,  $\Pi_2 = 4$ and  $\Pi_3 = 5$, we have a system of four equations and five unknowns
\begin{align}
2& =\frac{\beta_{0}-\alpha_{0}}{\alpha_{1}-\beta_{1}} \\
3& =-\frac{\alpha_{2}}{\alpha_{1}-\beta_{1}} \\
4&=\frac{\alpha_{1} \beta_{0}-\alpha_{0} \beta_{1}}{\alpha_{1}-\beta_{1}} \\
5&=-\frac{\alpha_{2} \beta_{1}}{\alpha_{1}-\beta_{1}}
\end{align}
One free variable, the system is still underidentified!

#  Exact Identification

We keep adding variables in the model, we will see if it can be identified

\begin{align}
\text{Demand function:}& \quad Q_{t}=\alpha_{0}+\alpha_{1} P_{t}+\alpha_{2} I_{t}+u_{1 t}\\
\text{Supply function:}& \quad Q_{t}=\beta_{0}+\beta_{1} P_{t}+\beta_{2} P_{t-1}+u_{2 t}
\end{align}

Equate both functions
$$
\alpha_{0}+\alpha_{1} P_{t}+\alpha_{2} I_{t}+u_{1 t}=\beta_{0}+\beta_{1} P_{t}+\beta_{2} P_{t-1}+u_{2 t}
$$
Solve for $P_t$

$$
\begin{gathered}
P_{t}=\Pi_{0}+\Pi_{1} I_{t}+\Pi_{2} P_{t-1}+v_{t} \\
\Pi_{0}=\frac{\beta_{0}-\alpha_{0}}{\alpha_{1}-\beta_{1}} \\
\Pi_{1}=-\frac{\alpha_{2}}{\alpha_{1}-\beta_{1}} \\
\Pi_{2}=\frac{\beta_{2}}{\alpha_{1}-\beta_{1}} \\
v_{t}=\frac{u_{2 t}-u_{1 t}}{\alpha_{1}-\beta_{1}}
\end{gathered}
$$

Substitute back into demand function to solve for $Q_t$
$$
Q_{t}=\Pi_{3}+\Pi_{4} I_{t}+\Pi_{5} P_{t-1}+w_{t}\\
\Pi_{3}=\frac{\alpha_{1} \beta_{0}-\alpha_{0} \beta_{1}}{\alpha_{1}-\beta_{1}} \\
\Pi_{4}=-\frac{\alpha_{2} \beta_{1}}{\alpha_{1}-\beta_{1}} \\
\Pi_{5}=\frac{\alpha_{1} \beta_{2}}{\alpha_{1}-\beta_{1}} \\
w_{t}=\frac{\alpha_{1} u_{2 t}-\beta_{1} u_{1 t}}{\alpha_{1}-\beta_{1}}
$$

Finally we obtain six equations and six unknowns. Assume $\Pi_0 = 2$, $\Pi_1 = -3$,  $\Pi_2 = 4$,  $\Pi_3 = 5$, $\Pi_4 = 6$ and $\Pi_5 = 7$, then solve the system, we obtain all the structural coefficients

```python
a0, a1, a2, b0, b1, b2 = sy.symbols('a0, a1, a2, b0, b1, b2')

eq0 = sy.Eq(2 * (a1 - b1) - (b0 - a0), 0)
eq1 = sy.Eq(-3 * (a1 - b1) + a2, 0)
eq2 = sy.Eq(4 * (a1 - b1) - b2, 0)
eq3 = sy.Eq(5 * (a1 - b1) - a1*b0 + a0*b1, 0)
eq4 = sy.Eq(6 * (a1 - b1) + a2*b1, 0)
eq5 = sy.Eq(7 * (a1 - b1) - a1*b2, 0)

sy.solve([eq0, eq1, eq2, eq3, eq4, eq5], (a0, a1, a2, b0, b1, b2))
```

    [(3/2, 7/4, 45/4, 9, -2, 15)]

#  Over Identification

What will happen if we keep adding more variables than necessary? The system will be overidentified, here is the example
\begin{align}
\text{Demand function:}& \quad Q_{t}=\alpha_{0}+\alpha_{1} P_{t}+\alpha_{2} I_{t}+\alpha_{3} R_{t}+u_{1 t}\\
\text{Supply function:}& \quad Q_{t}=\beta_{0}+\beta_{1} P_{t}+\beta_{2} P_{t-1}+u_{2 t}
\end{align}

Combine and solve for $P_t$ and $Q_t$
$$
\begin{aligned}
P_{t} &=\Pi_{0}+\Pi_{1} I_{t}+\Pi_{2} R_{t}+\Pi_{3} P_{t-1}+v_{t} \\
Q_{t} &=\Pi_{4}+\Pi_{5} I_{t}+\Pi_{6} R_{t}+\Pi_{7} P_{t-1}+w_{t}
\end{aligned}
$$
where $R_t$ represents wealth.
$$
\begin{align}
\Pi_{0}&=\frac{\beta_{0}-\alpha_{0}}{\alpha_{1}-\beta_{1}} \quad &\Pi_{1}=-\frac{\alpha_{2}}{\alpha_{1}-\beta_{1}} \\
\Pi_{2}&=-\frac{\alpha_{3}}{\alpha_{1}-\beta_{1}} \quad &\Pi_{3}=\frac{\beta_{2}}{\alpha_{1}-\beta_{1}} \\
\Pi_{4}&=\frac{\alpha_{1} \beta_{0}-\alpha_{0} \beta_{1}}{\alpha_{1}-\beta_{1}} \quad &\Pi_{5}=-\frac{\alpha_{2} \beta_{1}}{\alpha_{1}-\beta_{1}} \\
\Pi_{6}&=-\frac{\alpha_{3} \beta_{1}}{\alpha_{1}-\beta_{1}} \quad &\Pi_{7}=\frac{\alpha_{1} \beta_{2}}{\alpha_{1}-\beta_{1}} \\
w_{t}&=\frac{\alpha_{1} u_{2 t}-\beta_{1} u_{1 t}}{\alpha_{1}-\beta_{1}} \quad &v_{t}=\frac{u_{2 t}-u_{1 t}}{\alpha_{1}-\beta_{1}}
\end{align}
$$

Eight equations and seven unknowns, if you get lucky there can be a unique set of structural parameters, however mostly no unique solutions. Therefore, it doesn't mean more variables in the system is better.

As below shows, $\beta_0$ can't be identified.

```python
a0, a1, a2, a3, b0, b1, b2 = sy.symbols('a0, a1, a2, a3, b0, b1, b2')

eq0 = sy.Eq(2 * (a1 - b1) - (b0 - a0), 0)
eq1 = sy.Eq(-3 * (a1 - b1) + a2, 0)
eq2 = sy.Eq(4 * (a1 - b1) + a3, 0)
eq3 = sy.Eq(5 * (a1 - b1) - b2, 0)
eq4 = sy.Eq(6 * (a1 - b1) - a1*b0 + a0*b1, 0)
eq5 = sy.Eq(7 * (a1 - b1) + a2*b1, 0)
eq6 = sy.Eq(8 * (a1 - b1) + a3*b1, 0)
eq7 = sy.Eq(9 * (a1 - b1) - a1*b2, 0)

sy.solve([eq0, eq1, eq2, eq3, eq4, eq5, eq6, eq7], (a0, a1, a2, a3, b0, b1, b2))
```

    [(b0, -7/3, 0, 0, b0, -7/3, 0)]

#  Order Condition

We will use an example to show how to perform order condition and rank condition.

Consider an extended Keynesian model
$$
\begin{array}{ll}
\text { Consumption function: } & C_{t} & =\beta_{1}+\beta_{2} Y_{t}-\beta_{3} T_{t}+u_{1 t} \\
\text { Investment function: } & I_{t} & =\alpha_{0}+\alpha_{1} Y_{t-1}+u_{2 t} \\
\text { Taxation function: } & T_{t} & =\gamma_{0}+\gamma_{1} Y_{t}+u_{3 t} \\
\text { Income identity: } & Y_{t} & =C_{t}+I_{t}+G_{t}
\end{array}
$$
In the model the endogenous variables are $C_t$, $I_t$, $T_t$, and $Y_t$ and the predetermined variables are $G_t$ and $Y_{t-1}$

**Step 1:** Arrange all equations that variables stay on the left side, disturbance terms on the right side.
$$
\begin{array}{ll}
\text { Consumption function: } & C_{t} -\beta_{1}+\beta_{2} Y_{t}+\beta_{3} T_{t}=u_{1 t} \\
\text { Investment function: } & I_{t} -\alpha_{0}-\alpha_{1} Y_{t-1}=u_{2 t} \\
\text { Taxation function: } & T_{t} -\gamma_{0}-\gamma_{1} Y_{t}=u_{3 t} \\
\text { Income identity: } & Y_{t} -C_{t}-I_{t}-G_{t}=0
\end{array}
$$

**Step 2:** Extract the coefficient matrix like below
$$
\qquad\quad\text{ Coefficients of the Variables }\\
\begin{aligned}
&\begin{array}{cccccccccc}
 &1 & C_{t} & I_{t} & T_{t} & Y_{t} & Y_{t-1} & G_{t}\\
\text{Cons. Func.} & -\beta_1 & 1 & 0 & \beta_{3} & \beta_2 & 0 & 0  \\
\text{Inv. Func.}  & -\alpha_0 & 0 & 1 & 0 & 0 & -\alpha_1 & 0\\
\text{Tax. Func.}  & -\gamma_0 & 0 & 0 & 1 & -\gamma_1 & 0 & 0  \\
\text{Inco. Id.} & 0 & -1 & -1 & 0 & 1 & 0 & -1 \\
\end{array}
\end{aligned}
$$

**Step 3:** To identify each equation, we follow a rule called **Order Condition**.

In a model of $M$ simultaneous equations, in order for an equation to be identified, the number of predetermined variables excluded from the equation must not be less than the number of endogenous variables included in that equation less 1, that is,
$$
K-k \geq m-1
$$

$$
\begin{align}
M&= \text{number of endogenous variables in the model}\\
m&= \text{number of endogenous variables in a given equation}\\
K&= \text{number of predetermined variables in the model}\\
k&= \text{number of predetermined variables in a given equation}
\end{align}
$$

If $K-k=m-1$, the equation is exact identified, but if $K-k>m-1$, it is overidentified.

Write a table like below

$$
\begin{array}{cccc}
& \text { No. of Predetermined } & \text { No. of Endogenous } & \\
&\text { Variables Excluded, } & \text { Variables Included, } & \\
 & (K-k) & \text { Less One, }(m-1) & \text { Identified? } \\
\text{Cons. Func.} & 2-0=2 & 3-1=2 & \text { Exactly } \\
\text{Inv. Func.} & 2-1=1 & 1-1=0 & \text { Overidentified } \\
\text{Tax. Func.} & 2-0=2 & 2-1=1 & \text { Overidentified } \\
\text{Inco. Id.} & 2-1=1 & 3-1=2 & \text { Underidentified }
\end{array}
$$

#  Limited Information Methods

**Limited information methods** (LIM) are estimation methods that used on each equation in the system individually, totally ignoring restrictions on other equations. In contrast, **full information methods** (FIM) estimate all equation simultaneously, taking account of restrictions from each equation.

However, in practice FIM are seldom used for a variety of reason. Let alone computation burden, of most importance is sensitivity to specification error, if any equation out of hundreds of suffer from specification error, the whole model's estimation could be biased.

Here we only discuss LIM, specifically there three common methods:
1. Ordinary Least Squares
2. Indirect Least Squares
3. Two-Stage Least Squares

##  Ordinary Least Squares

In general, we don't use OLS for system of simultaneous equations due to interdependence of disturbance term and endogenous variables, the estimation results are biased and inconsistent.

However there is one scenario that OLS can be applied in the contexts of simultaneous equations. This is the case of **recursive** models, consider the following model.

$$
\begin{array}{ll}
Y_{1 t}&=\beta_{11}                                      &+\gamma_{11} X_{1 t}+\gamma_{12} X_{2 t}+u_{1 t} \\
Y_{2 t}&=\beta_{21}+\beta_{22} Y_{1 t}                   &+\gamma_{21} X_{1 t}+\gamma_{22} X_{2 t}+u_{2 t} \\
Y_{3 t}&=\beta_{31}+\beta_{32} Y_{1 t}+\beta_{33} Y_{2 t}&+\gamma_{31} X_{1 t}+\gamma_{32} X_{2 t}+u_{3 t}
\end{array}
$$
where $Y$ and $X$ are endogenous and exogenous variables respectively. Assumptions of disturbance terms
$$
\operatorname{Cov}\left(u_{1 t}, u_{2 t}\right)=\operatorname{Cov}\left(u_{1 t}, u_{3 t}\right)=\operatorname{Cov}\left(u_{2 t}, u_{3 t}\right)=0
$$
which can be called **zero contemporaneous correlation**, if you calculate covariance matrix of the disturbance term vector $\mathbb{u}$, it is expected to be a diagonal matrix.

Now, if you pay close attention, you can see the first equation can actually be estimated by OLS, because all exogenous variables are on right-hand side.

The equation two has $Y_{1t}$ on the right hand side, however the $u_{1t}$ from $Y_{1t}$ and $u_{2t}$ from $Y_{2t}$ has zero covariance, again it can be estimated by OLS.

Similarly, the third equations can also be estimated by OLS, because $Y_{1t}$ and $Y_{2t}$ are predetermined in the previous two equations.

## Indirect Least Squares

**Indirect Least Squares** is the method for obtaining structural coefficients in exact identified model.

\begin{align}
\text{Demand function:}& \quad Q_{t}=\alpha_{0}+\alpha_{1} P_{t}+\alpha_{2} X_{t}+u_{1 t}\\
\text{Supply function:}& \quad Q_{t}=\beta_{0}+\beta_{1} P_{t}+\beta_{2} P_{t-1}+u_{2 t}
\end{align}

$Q$ and $P$ has their conventional meaning, $X$ can be anything, for this case, we deem it as income. The reduced-form equations are
$$
\begin{aligned}
P_{t}&=\Pi_{0}+\Pi_{1} X_{t}+\Pi_{2} P_{t-1}+v_{t} \\
Q_{t}&=\Pi_{3}+\Pi_{4} X_{t}+\Pi_{5} P_{t-1}+w_{t}
\end{aligned}
$$

Now, let's try with real data, which is excerpt from _Economic Report of the President 2020_.

(Example will be updated...)

## Two Stage Least Squares

Will be updated...

```python

```

8. Identification and Simultaneous-Equation Models.ipynb


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Theorems

The theorems of the Laplace transformation relate basic time-domain operations to their equivalents in the Laplace domain. They are of use for the computation of Laplace transforms of signals composed from modified [standard signals](../continuous_signals/standard_signals.ipynb) and for the computation of the response of systems to an input signal. The theorems allow further to predict the consequences of modifying a signal or system by certain operations.

### Temporal Scaling Theorem

A signal $x(t)$ is given for which the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ exists. The Laplace transform of the [temporally scaled signal](../continuous_signals/operations.ipynb#Temporal-Scaling) $x(a t)$ with $a \in \mathbb{R} \setminus \{0\}$ reads

\begin{equation}
\mathcal{L} \{ x(a t) \} = \frac{1}{|a|} \cdot X \left( \frac{s}{a} \right)
\end{equation}

The Laplace transformation of a temporally scaled signal is given by weighting the inversely scaled Laplace transform of the unscaled signal with $\frac{1}{|a|}$. The scaling of the Laplace transform can be interpreted as a scaling of the complex $s$-plane. The region of convergence (ROC) of the temporally scaled signal $x(a t)$ is consequently the inversely scaled ROC of the unscaled signal $x(t)$

\begin{equation}
\text{ROC} \{ x(a t) \} = \left\{ s: \frac{s}{a} \in \text{ROC} \{ x(t) \} \right\}
\end{equation}

Above relation is known as scaling theorem of the Laplace transform. The scaling theorem can be proven by introducing the scaled signal $x(a t)$ into the definition of the Laplace transformation

\begin{equation}
\mathcal{L} \{ x(a t) \} = \int_{-\infty}^{\infty} x(a t) \, e^{- s t} \; dt = \frac{1}{|a|} \int_{-\infty}^{\infty} x(t') \, e^{-\frac{s}{a} t'} \; dt' = \frac{1}{|a|} \cdot X \left( \frac{s}{a} \right)
\end{equation}

where the substitution $t' = a t$ was used. Note that a negative value of $a$ would result in a reversal of the integration limits. In this case a second reversal of the integration limits together with the sign of the integration element $d t'= a \, dt$ was consolidated into the absolute value of $a$.

### Convolution Theorem

The convolution theorem states that the Laplace transform of the convolution of two signals $x(t)$ and $y(t)$ is equal to the scalar multiplication of their Laplace transforms $X(s)$ and $Y(s)$

\begin{equation}
\mathcal{L} \{ x(t) * y(t) \} = X(s) \cdot Y(s)
\end{equation}

under the assumption that both Laplace transforms $X(s) = \mathcal{L} \{ x(t) \}$ and $Y(s) = \mathcal{L} \{ y(t) \}$ exist, respectively. The ROC of the convolution $x(t) * y(t)$ includes at least the intersection of the ROCs of $x(t)$ and $y(t)$

\begin{equation}
\text{ROC} \{ x(t) * y(t) \} \supseteq \text{ROC} \{ x(t) \} \cap \text{ROC} \{ y(t) \}
\end{equation}

The theorem can be proven by introducing the [definition of the convolution](../systems_time_domain/convolution.ipynb) into the [definition of the Laplace transform](definition.ipynb) and changing the order of integration

\begin{align}
\mathcal{L} \{ x(t) * y(t) \} &= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} x(\tau) \cdot y(t-\tau) \; d \tau \right) e^{-s t} \; dt \\
&= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} y(t-\tau) \, e^{-s t} \; dt \right) x(\tau) \; d\tau \\
&= Y(s) \cdot \int_{-\infty}^{\infty} x(\tau) \, e^{-s \tau} \; d \tau \\
&= Y(s) \cdot X(s)
\end{align}

The convolution theorem is very useful in the context of linear time-invariant (LTI) systems. The output signal $y(t)$ of an LTI system is given as the convolution of the input signal $x(t)$ with the impulse response $h(t)$. The signals can be represented either in the time or Laplace domain. This leads to the following equivalent representations of an LTI system in the time and Laplace domain, respectively

Calculation of the system response by transforming the problem into the Laplace domain can be beneficial since this replaces the evaluation of the convolution integral by a scalar multiplication. In many cases this procedure simplifies the calculation of the system response significantly. A prominent example is the [analysis of a passive electrical network](network_analysis.ipynb). The convolution theorem can also be useful to derive an unknown Laplace transform. The key is here to express the signal as convolution of two other signals for which the Laplace transforms are known. This is illustrated by the following example.

**Example**

The Laplace transform of the convolution of a causal cosine signal $\epsilon(t) \cdot \cos(\omega_0 t)$ with a causal sine signal $\epsilon(t) \cdot \sin(\omega_0 t)$ is derived by the convolution theorem

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot ( \cos(\omega_0 t) * \sin(\omega_0 t) \}
= \frac{s}{s^2 + \omega_0^2} \cdot \frac{\omega_0}{s^2 + \omega_0^2}
= \frac{\omega_0 s}{(s^2 + \omega_0^2)^2}
\end{equation}

where the [Laplace transforms of the causal cosine and sine signals](properties.ipynb#Transformation-of-the-cosine-and-sine-signal) were used. The ROC of the causal cosine and sine signal is $\Re \{ s \} > 0$. The ROC for their convolution is also $\Re \{ s \} > 0$, since no poles and zeros cancel out. Above Laplace transform has one zero $s_{00} = 0$, and two poles of second degree $s_{\infty 0} = s_{\infty 1} = j \omega_0$ and $s_{\infty 2} = s_{\infty 3} = - j \omega_0$.

This example is evaluated numerically in the following. First the convolution of the causal cosine and sine signal is computed

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, tau = sym.symbols('t tau', real=True)
s = sym.symbols('s', complex=True)
w0 = sym.symbols('omega0', positive=True)

x = sym.integrate(sym.cos(w0*tau) * sym.sin(w0*(t-tau)), (tau, 0, t))
x = x.doit()
x
```

For the sake of illustration let's plot the signal for $\omega_0 = 1$

```python
sym.plot(x.subs(w0, 1), (t, 0, 50), xlabel=r'$t$', ylabel=r'$x(t)$');
```

The Laplace transform is computed

```python
X, a, cond = sym.laplace_transform(x, t, s)
X, a
```

which exists for $\Re \{ s \} > 0$. Its zeros are given as

```python
sym.roots(sym.numer(X), s)
```

and its poles as

```python
sym.roots(sym.denom(X), s)
```

### Temporal Shift Theorem

The [temporal shift of a signal](../continuous_signals/operations.ipynb#Temporal-Shift) $x(t - \tau)$ for $\tau \in \mathbb{R}$ can be expressed by the convolution of the signal $x(t)$ with a shifted Dirac impulse

\begin{equation}
x(t - \tau) = x(t) * \delta(t - \tau)
\end{equation}

This follows from the sifting property of the Dirac impulse. Applying a two-sided Laplace transform to the left- and right-hand side and exploiting the convolution theorem yields

\begin{equation}
\mathcal{L} \{ x(t - \tau) \} = X(s) \cdot e^{- s \tau}
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$ is assumed to exist. Note that $\mathcal{L} \{ \delta(t - \tau) \} = e^{- s \tau}$ can be derived from the definition of the two-sided Laplace transform together with the sifting property of the Dirac impulse. The Laplace transform of a shifted signal is given by multiplying the Laplace transform of the original signal with $e^{- s \tau}$. The ROC does not change

\begin{equation}
\text{ROC} \{ x(t-\tau) \} = \text{ROC} \{ x(t) \}
\end{equation}

This result is known as shift theorem of the Laplace transform. For a causal signal $x(t)$ and $\tau > 0$ the shift theorem of the one-sided Laplace transform is equal to the shift theorem of the two-sided transform.

#### Transformation of the rectangular signal

The Laplace transform of the [rectangular signal](../continuous_signals/standard_signals.ipynb#Rectangular-Signal) $x(t) = \text{rect}(t)$ is derived by expressing it by the Heaviside signal

\begin{equation}
\text{rect}(t) = \epsilon \left(t + \frac{1}{2} \right) - \epsilon \left(t - \frac{1}{2} \right)
\end{equation}

Applying the shift theorem to the [transform of the Heaviside signal](definition.ipynb#Transformation-of-the-Heaviside-Signal) and the linearity of the Laplace transform yields

\begin{equation}
\mathcal{L} \{ \text{rect}(t) \} = \frac{1}{s} e^{s \frac{1}{2}} - \frac{1}{s} e^{- s \frac{1}{2}} = \frac{\sinh \left( \frac{s}{2} \right) }{\frac{s}{2}}
\end{equation}

where $\sinh(\cdot)$ denotes the [hyperbolic sine function](https://en.wikipedia.org/wiki/Hyperbolic_function#Sinh). The ROC of the Heaviside signal is given as $\Re \{ s \} > 0$. Applying [l'Hopitals rule](https://en.wikipedia.org/wiki/L'H%C3%B4pital's_rule) the pole at $s=0$ can be disregarded leading to

\begin{equation}
\text{ROC} \{ \text{rect}(t) \} = \mathbb{C}
\end{equation}

For illustration, the magnitude of the Laplace transform $|X(s)|$ is plotted in the $s$-plane, as well as $X(\sigma)$ and $X(j \omega)$ for the real and imaginary part of the complex frequency $s = \sigma + j \omega$.

```python
sigma, omega = sym.symbols('sigma omega')
X = sym.sinh(s/2)*2/s

sym.plotting.plot3d(abs(X.subs(s, sigma+sym.I*omega)), (sigma, -5, 5), (omega, -20, 20),
                    xlabel=r'$\Re\{s\}$', ylabel=r'$\Im\{s\}$', title=r'$|X(s)|$')

sym.plot(X.subs(s, sigma) , (sigma, -5, 5), xlabel=r'$\Re\{s\}$', ylabel=r'$X(s)$', ylim=(0, 3))

sym.plot(X.subs(s, sym.I*omega) , (omega, -20, 20), xlabel=r'$\Im\{s\}$', ylabel=r'$X(s)$');
```

**Exercise**

* Derive the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ of the causal rectangular signal $x(t) = \text{rect} (a t - \frac{1}{2 a})$
* Derive the Laplace transform of the [triangular signal](../fourier_transform/theorems.ipynb#Transformation-of-the-triangular-signal) $x(t) = \Lambda(a t)$ with $a \in \mathbb{R} \setminus \{0\}$

### Differentiation Theorem

Derivatives of signals are the fundamental operations of differential equations. Ordinary differential equations (ODEs) with constant coefficients play an important role in the theory of time-invariant (LTI) systems. Consequently, the representation of the derivative of a signal in the Laplace domain is of special interest.

#### Two-sided transform

A differentiable signal $x(t)$ whose temporal derivative $\frac{d x(t)}{dt}$ exists is given. Using the [derivation property of the Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse), the derivative of the signal can be expressed by the convolution

\begin{equation}
\frac{d x(t)}{dt} = \frac{d \delta(t)}{dt} * x(t)
\end{equation}

Applying a two-sided Laplace transformation to the left- and right-hand side together with the [convolution theorem](#Convolution-Theorem) yields the Laplace transform of the derivative of $x(t)$

\begin{equation}
\mathcal{L} \left\{ \frac{d x(t)}{dt} \right\} = s \cdot X(s)
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$. The two-sided Laplace transform $\mathcal{L} \{ \frac{d \delta(t)}{dt} \} = s$ can be derived by applying the definition of the Laplace transform together with the derivation property of the Dirac impulse. The ROC is given as a superset of the ROC for $x(t)$

\begin{equation}
\text{ROC} \left\{ \frac{d x(t)}{dt} \right\} \supseteq \text{ROC} \{ x(t) \}
\end{equation}

due to the zero at $s=0$ which may cancel out a pole.

Above result is known as differentiation theorem of the two-sided Laplace transform. It states that the differentiation of a signal in the time domain is equivalent to a multiplication of its spectrum by $s$.

#### One-sided transform

Many practical signals and systems are causal, hence $x(t) = 0$ for $t < 0$. A causal signal is potentially discontinuous for $t=0$. The direct application of above result for the two-sided Laplace transform is not possible since it assumes that the signal is differentiable for every time $t$. The potential discontinuity at $t=0$ has to be considered explicitly for the derivation of the differentiation theorem for the one-sided Laplace transform [[Girod et al.](index.ipynb#Literature)]

\begin{equation}
\mathcal{L} \left\{ \frac{d x(t)}{dt} \right\} = s \cdot X(s) - x(0+)
\end{equation}

where $x(0+) := \lim_{\epsilon \to 0} x(0+\epsilon)$ denotes the right sided limit value of $x(t)$ for $t=0$. The ROC is given as a superset of the ROC of $x(t)$

\begin{equation}
\text{ROC} \left\{ \frac{d x(t)}{dt} \right\} \supseteq \text{ROC} \{ x(t) \}
\end{equation}

due to the zero at $s=0$ which may cancel out a pole. The one-sided Laplace transform of a causal signal is equal to its two-sided transform. Above result holds therefore also for the two-sided transform of a causal signal.

The main application of the differentiation theorem is the transformation and solution of differential equations under consideration of initial values. Another application area is the derivation of transforms of signals which can be expressed as derivatives of other signals.

### Integration Theorem

An integrable signal $x(t)$ for which the integral $\int_{-\infty}^{t} x(\tau) \; d\tau$ exists is given. The integration can be represented as convolution with the rectangular signal $\epsilon(t)$

\begin{equation}
\int_{-\infty}^{t} x(\tau) \; d\tau = \int_{-\infty}^{\infty} x(\tau) \cdot \epsilon(t - \tau) \; d\tau = \epsilon(t) * x(t)
\end{equation}

as illustrated below

Two-sided Laplace transformation of the left- and right-hand side of above equation, application of the convolution theorem and using the Laplace transform of the Heaviside signal $\epsilon(t)$ yields

\begin{equation}
\mathcal{L} \left\{ \int_{-\infty}^{t} x(\tau) \; d\tau \right\}
= \frac{1}{s} \cdot X(s)
\end{equation}

The ROC is given as a superset of the intersection of the ROC of $x(t)$ and the right $s$-half-plane

\begin{equation}
\text{ROC} \left\{ \int_{-\infty}^{t} x(\tau) \; d\tau \right\} \supseteq \text{ROC} \{ x(t) \} \cap \{s : \Re \{ s \} > 0\}
\end{equation}

due to the pole at $s=0$. This integration theorem holds also for the one-sided Laplace transform.

#### Transformation of the ramp signal

The Laplace transform of the causal [ramp signal](https://en.wikipedia.org/wiki/Ramp_function) $t \cdot \epsilon(t)$ is derived by the integration theorem. The ramp signal can be expressed as integration over the Heaviside signal

\begin{equation}
t \cdot \epsilon(t) = \int_{-\infty}^{t} \tau \cdot \epsilon(\tau) \; d \tau
\end{equation}

Laplace transformation of the left- and right-hand side and application of the integration theorem together with the Laplace transform of the Heaviside signal yields

\begin{equation}
\mathcal{L} \{ t \cdot \epsilon(t) \} = \frac{1}{s^2}
\end{equation}

with

\begin{equation}
\text{ROC} \{ t \cdot \epsilon(t) \} = \{s : \Re \{ s \} > 0\}
\end{equation}

**Exercise**

* Derive the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ of the signal $x(t) = t^n \cdot \epsilon(t)$ with $n \geq 0$ by repeated application of the integration theorem.
* Compare your result to the numerical result below. Note that $\Gamma(n+1) = n!$ for $n \in \mathbb{N}$.

```python
n = sym.symbols('n', integer=True)

X, a, cond = sym.laplace_transform(t**n, t, s)
X, a, cond
```

### Modulation Theorem

The complex modulation of a signal $x(t)$ is defined as $e^{s_0 t} \cdot x(t)$ with $s_0 \in \mathbb{C}$. The Laplace transform of a modulated signal is derived by introducing it into the definition of the two-sided Laplace transform

\begin{align}
\mathcal{L} \left\{ e^{s_0 t} \cdot x(t) \right\} &=
\int_{-\infty}^{\infty} e^{s_0 t} x(t) \, e^{-s t} \; dt =
\int_{-\infty}^{\infty} x(t) \, e^{- (s - s_0) t} \; dt \\
&= X(s-s_0)
\end{align}

where $X(s) = \mathcal{L} \{ x(t) \}$. Modulation of the signal $x(t)$ leads to a translation of the $s$-plane into the direction given by the complex value $s_0$. Consequently, the ROC is also shifted

\begin{equation}
\text{ROC} \{ e^{s_0 t} \cdot x(t) \} = \{s: s - \Re \{ s_0 \} \in \text{ROC} \{ x(t) \} \}
\end{equation}

This relation is known as modulation theorem.

**Example**

The Laplace transform of the signal $t^n \cdot \epsilon(t)$

\begin{equation}
\mathcal{L} \{ t^n \cdot \epsilon(t) \} = \frac{n!}{s^{n+1}}
\end{equation}

for $\Re \{ s \} > 0$ was derived in the previous example. This result can be extended to the class of signals $t^n e^{-s_0 t} \epsilon(t)$ with $s_0 \in \mathbb{C}$ using the modulation theorem

\begin{equation}
\mathcal{L} \{ t^n e^{-s_0 t} \epsilon(t) \} = \frac{n!}{(s + s_0)^{n+1}} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}.
\end{equation}

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


以下の論文のPSOアルゴリズムに従った。

http://ci.nii.ac.jp/els/110006977755.pdf?id=ART0008887051&type=pdf&lang=en&host=cinii&order_no=&ppv_type=0&lang_sw=&no=1452683083&cp=

```python
%matplotlib inline

import numpy as np
import pylab as pl
import math
from sympy import *
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from mpl_toolkits.mplot3d import Axes3D
```

```python
def TSP_map(N): #100×100の正方格子内にN個の点を配置する関数
    TSP_map = []
    X = [i for i in range(100)]
    Y = [i for i in range(100)]
    x = np.array([])
    y = np.array([])
    for i in range(N):
        x = np.append(x, np.random.choice(X))
        y = np.append(y, np.random.choice(Y))
    for i in range(N):
        TSP_map.append([x[i], y[i]])
    return TSP_map
```

```python
class PSO:
    def __init__(self, N, pN, omega, alpha, beta):
        self.N = N
        self.pN = pN
        self.omega = omega
        self.alpha = alpha
        self.beta = beta
        self.city = TSP_map(N)

    def initialize(self):
        ptcl = np.array([])
        for i in range(self.pN):
            a = np.random.choice([i for i in range(self.N - 1)])
            b = np.random.choice([i for i in range(a, self.N)])
            V = [a, b]
            ptcl = np.append(ptcl, particle(i, V, self.N, self.omega, self.alpha, self.beta))
        self.ptcl = ptcl
        return self.ptcl

    def one_simulate(self):
        for i in range(self.pN):
            self.ptcl[i].SS_id()
            self.ptcl[i].SS_gd(self.p_gd_X)
            self.ptcl[i].new_V()
            self.ptcl[i].new_X()
            self.ptcl[i].P_id(self.city)

    def simulate(self, sim_num):
        for i in range(self.pN):
            self.ptcl[i].initial(self.city)
        self.p_gd_X = self.P_gd()
        for i in range(sim_num):
            self.one_simulate()
            self.p_gd_X = self.P_gd()
        self.p_gd_X = self.P_gd()
        return self.p_gd_X

    def P_gd(self):
        P_gd = self.ptcl[0].p_id
        no = 0
        for i in range(self.pN):
            if P_gd < self.ptcl[i].p_id:
                P_gd = self.ptcl[i].p_id
                self.no = i
        return self.ptcl[self.no].p_id_X
```

```python
class particle:
    def __init__(self, No, V, num_city, omega, alpha, beta): #noは粒子の番号（ナンバー）である。
        self.No = No
        self.V = V
        self.num_city = num_city
        self.omega = omega
        self.alpha = alpha
        self.beta = beta
        self.X = self.init_X()

    def initial(self, city):
        self.ss_id = []
        self.ss_gd = []
        self.P_id(city)

    def init_X(self):
        c = np.array([i for i in range(self.num_city)])
        np.random.shuffle(c)
        return c

    def SO(self, V, P):
        SO = []
        for i in range(len(V)):
            if V[i] != P[i]:
                t = np.where(V == P[i])
                t = int(t[0])
                a = V[i]
                b = V[t]
                V[i] = b
                V[t] = a
                SO.append([i, t])
        return SO

    def SS_id(self):
        self.ss_id = self.SO(self.X, self.p_id_X)

    def SS_gd(self, p_gd_X):
        self.ss_gd = self.SO(self.X, p_gd_X)

    def select(self, V, v, p):
        select_v = np.array([])
        for i in range(len(V)):
            x = np.random.choice([1, 0], p=[p, 1-p])
            if x == 1:
                select_v = np.append(select_v,V[i])
        return select_v

    def new_V(self):
        V = np.array([])
        self.select(V, self.V, self.omega)
        self.select(V, self.SS_id, self.alpha)
        self.select(V, self.ss_gd, self.beta)
        self.V = V
        return self.V

    def new_X(self):
        for i in range(len(self.V)):
            j = self.V[i][0]
            k = self.V[i][1]
            a = self.X[j]
            b = self.X[k]
            self.X[j] = b
            self.X[k] = a
        return self.X

    def P_id(self, city): #二都市間の距離を足し上げてP_idを求める関数。
        P_id = 0
        for i in range(self.num_city):
            if i != self.num_city-1:
                x1 = city[self.X[i]][0]
                y1 = city[self.X[i]][1]
                x2 = city[self.X[i+1]][0]
                y2 = city[self.X[i+1]][1]
            else:
                x1 = city[self.X[i]][0]
                y1 = city[self.X[i]][1]
                x2 = city[self.X[0]][0]
                y2 = city[self.X[0]][1]
            a = np.array([x1, y1])
            b = np.array([x2, y2])
            u = b - a
            p = np.linalg.norm(u)
            P_id += p
        if P_id < self.P_id:
            self.p_id = P_id
            self.p_id_X = self.X
        return self.p_id
```

10都市で行う。パラメータは左から順に、（都市の数、粒子の数、前期の速度の影響率、localベストの影響率、globalベストの影響率)である。

```python
pso = PSO(10, 10, 0.3, 0.1, 0.3)
```

都市の座標とプロット。

```python
pso.city
```

    [[40.0, 26.0],
     [16.0, 74.0],
     [74.0, 49.0],
     [55.0, 20.0],
     [20.0, 14.0],
     [24.0, 75.0],
     [87.0, 84.0],
     [15.0, 87.0],
     [8.0, 9.0],
     [34.0, 70.0]]

```python
x = []
y = []
for i in range(len(pso.city)):
    x.append(pso.city[i][0])
    y.append(pso.city[i][1])
plt.scatter(x, y)
```

粒子の初期化。

```python
pso.initialize()
```

    array([<__main__.particle instance at 0x10b55ae18>,
           <__main__.particle instance at 0x10ac81e18>,
           <__main__.particle instance at 0x10ac81ab8>,
           <__main__.particle instance at 0x10b4bba70>,
           <__main__.particle instance at 0x10b4bbe60>,
           <__main__.particle instance at 0x10b4bbea8>,
           <__main__.particle instance at 0x10b4bbd40>,
           <__main__.particle instance at 0x10b4bbbd8>,
           <__main__.particle instance at 0x10b4bbab8>,
           <__main__.particle instance at 0x10b4bbb00>], dtype=object)

100回シミュレーションした。

```python
pso.simulate(100)
```

```python
x = []
y = []
for i in pso.p_gd_X:
    x.append(pso.city[i][0])
    y.append(pso.city[i][1])
plt.plot(x, y)
```

```python
pso.no
```

    1

```python

```

```python

```


# hw 6: estimators
learning objectives:
* solidify "what is an estimator"
* evaluate the effectiveness of an estimator computationally (through simulation)
* understand the notion of unbiasedness, consistency, and efficiency of an estimator and evaluate these qualities computationally (through simulation)

```julia
using StatsBase, Random, Statistics, PyPlot
```

### what is the most important problem in your field?

(0) It is important to effectively communicate with people from different fields. Introduce yourself to someone in the class from a different field of study (someone you haven't met). Ask them, "what is the most important problem in your field of study?". Then, argue/explain what you think is the most important problem in *your* field of study.

Write the name of the student you spoke with and their field of study.

I'd also be interested to read about what you think is the most important problem in your field (a few sentences), but this is not required.

("field of study" can be as general as "chemistry" or as specific as "self-assembly of nanoparticles".)

### counting
(1) A biologist takes a random sample of six fish from a lake. The lake has ten distinct species of fish. It is possible for the biologist to select more than one fish of any given species. e.g., one outcome is: one fish of species $X$, two fish of species $Y$, and four fish of species $Z$. How many different samples could the biologist draw from the lake?
* the sampling is done without replacement, but this is irrelevant since the lake contains more than six fish of each species
* the order in which the fish are sampled does not matter
* fish of a given species are indistinguishable

**solution:** this combinatorics problem can be solved with the stars and bars formulation. here, the stars are the six fish, while the bars separate groups of fish according to their species. we need 9 bars to represent the partitioning of fish among 10 bins representing the different species.

For example, the following stars and bars formulation represents the outcome: 1 fish of species 1, 2 fish of species 2, 2 fish of species 4, and 1 fish of species 9:

 $* | ** | | ** | | | | | * | $

the number of possible distributions of the fish species among the six samples is equal to the number of ways to permute the [indistinguishable] stars and bars, which is:

$\dfrac{(6+9)!}{6!9!} = 5005$

### capture, mark, release, recapture

In ecology, one wishes to estimate the size of a population (e.g. turtles). It is too costly and impractical to count *every* member of a population. One strategy is to:
1. capture a random sample from the population, without replacement
2. mark/tag each member of this random sample
3. release the marked sample back into the population
4. after sufficient time has passed, recapture another sample
5. count the number of marked members from the recaptured sample

We assume that:
* when the captured and marked sample is released back into the wild, they randomly (homogenously) mix with the rest of the (unmarked) population before we recapture
* marking a member of the population does not change its likelihood of being recaptured
* the time between capture/mark/release and recapture/count is short enough to neglect deaths, births, and migration out of the population

define the variables:
* $n$: the total number of turtles in the population (unknown)
* $k$: number of turtles captured, marked, and released in the first phase
* $k_r$: number of turtles recaptured
* $m$: the number of recaptured turtles that had marks on them

##### Lincoln-Petersen estimator
A very intuitive estimator is found by imposing that the proportion found marked in the recaptured sample is equal to the proportion of the population that was captured/marked/released in the first phase.

\begin{equation}
\frac{k}{n}=\frac{m}{k_r}
\end{equation}

giving the Lincoln-Petersen estimator for the population size $n$:
\begin{equation}
\hat{n} = \frac{k k_r}{m}
\end{equation}

##### Chapmen estimator
Chapmen derived a different estimator that we will compare to the Lincoln-Petersen estimator below.
\begin{equation}
\hat{n} = \frac{(k + 1) (k_r +1)}{m+1} - 1
\end{equation}

#### estimating the population of turtles on a small island
We wish to estimate the population of turtles on a small island by a capture, mark, release, and recapture strategy.

(2) create a mutable data structure `Turtle` that represents a turtle in the population. It should have a single attribute, `marked`, that indicates whether it has been marked or not.

```julia
mutable struct Turtle
    marked::Bool
end
```

(3) write a function `create_population(nb_turtles::Int)` that creates a population of `nb_turtles` unmarked `Turtle`s. return the turtles as an `Array{Turtle}`.

```julia
function create_population(nb_turtles::Int)
    return [Turtle(false) for i = 1:nb_turtles]
end

turtles = create_population(500)
```

    500-element Array{Turtle,1}:
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     ⋮
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)

(4) write a function `count_marked(turtles::Array{Turtle})` that takes in a population of turtles and returns the number of these turtles that are marked.

```julia
turtles = create_population(500)
count_marked(turtles) # should return zero
```

```julia
function count_marked(turtles::Array{Turtle})
    nb_marked = 0
    for t in turtles
        if t.marked
            nb_marked += 1
        end
    end
    return nb_marked
end

count_marked(turtles)
```

    0

(5) write a function `capture_mark_release!` that takes in two arguments:
* `turtles::Array{Turtle}` the population of turtles
* `nb_capture_mark_release::Int` the number of turtles to randomly capture (select without replacement) and mark

and modifies the `marked` attribute of `nb_capture_mark_release` randomly selected turtles in `turtles` to denote that they have been marked. think about why the function has an `!`.

```julia
turtles = create_population(500)
capture_mark_release!(turtles, 45)
count_marked(turtles) # should return 45
```

```julia
function capture_mark_release!(turtles::Array{Turtle}, nb_capture_mark_release::Int)
    for t in sample(1:length(turtles), nb_capture_mark_release, replace=false)
        turtles[t].marked = true
    end
end

turtles = create_population(500)
capture_mark_release!(turtles, 45)
count_marked(turtles) # should return 45
```

    45

(6) write a function `recapture(turtles::Array{Turtle}, nb_recapture::Int)` that returns a random sample (without replacement) of `nb_recapture` turtles from `turtles` in the form of an `Array{Turtle}`.

```julia
turtles = create_population(500)
capture_mark_release!(turtles, 45)
recaptured_turtles = recapture(turtles, 50) # should return Array{Turtle} with 50 elements
```

```julia
function recapture(turtles::Array{Turtle}, nb_recapture::Int)
    return sample(turtles, nb_recapture, replace=false)
end

turtles = create_population(500)
capture_mark_release!(turtles, 45)
recaptured_turtles = recapture(turtles, 50) # should return Array{Turtle} with 50 elements
```

    50-element Array{Turtle,1}:
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(true)
     ⋮
     Turtle(false)
     Turtle(false)
     Turtle(true)
     Turtle(true)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)
     Turtle(false)

(7) write two functions, one for each estimator:

* `chapman_estimator(nb_capture_mark_release::Int, nb_recapture::Int, nb_marked_in_recaptured::Int)`
* `lincoln_petersen_estimator(nb_capture_mark_release::Int, nb_recapture::Int, nb_marked_in_recaptured::Int)`

that each take in the entire population of turtles (marked and unmarked), `turtles`, and the array of recaptured turtles, `recaptured_turtles`, and returns the respective estimate $\hat{n}$ of the number of turtles. here, `nb_capture_mark_release` is $k$, `nb_recapture` is $k_r$, and `nb_marked_in_recaptured` is $m$.

```julia
function chapman_estimator(nb_capture_mark_release::Int, nb_recapture::Int,
        nb_marked_in_recaptured::Int)
    return (nb_capture_mark_release + 1) * (nb_recapture + 1) / (nb_marked_in_recaptured + 1) - 1

end

function lincoln_petersen_estimator(nb_capture_mark_release::Int, nb_recapture::Int,
        nb_marked_in_recaptured::Int)
    return nb_capture_mark_release * nb_recapture / nb_marked_in_recaptured
end
```

    lincoln_petersen_estimator (generic function with 1 method)

(8) write a function `sim_capture_mark_release_recapture` that takes in:

* `nb_turtles::Int` $=n$
* `nb_capture_mark_release::Int` $=k$
* `nb_recapture::Int` $=k_r$
* `estimator::Function` either `chapman_estimator` or `lincoln_peterson_estimator` that you wrote above

and returns $\hat{n}$, the estimate of the number of turtles in this simulation.

use all of the functions you wrote above.

```julia
nb_turtles = 200
nb_capture_mark_release = 50
nb_recapture = 42
estimator = lincoln_petersen_estimator
n̂ = sim_capture_mark_release_recapture(nb_turtles, nb_capture_mark_release,
                                            nb_recapture, estimator)
```

```julia
function sim_capture_mark_release_recapture(nb_turtles::Int, nb_capture_mark_release::Int,
                                            nb_recapture::Int, estimator::Function)
    # create turtle population
    turtles = create_population(nb_turtles)

    # randomly capture, mark, and release
    capture_mark_release!(turtles, nb_capture_mark_release)

    # recapture
    recaptured_turtles = recapture(turtles, nb_recapture)

    # count number of marked in recaptured sample
    nb_marked_in_recaptured = count_marked(recaptured_turtles)

    # estimate the number of turtles
    n̂ = estimator(nb_capture_mark_release, nb_recapture, nb_marked_in_recaptured)
    return n̂
end

nb_turtles = 200
nb_capture_mark_release = 50
nb_recapture = 42
estimator = lincoln_petersen_estimator
n̂ = sim_capture_mark_release_recapture(nb_turtles, nb_capture_mark_release,
                                            nb_recapture, estimator)
```

    175.0

(9) evaluate/compare the biasedness of the Chapman and Lincoln-Peterson estimators by plotting the distribution of $\hat{n}$ over many capture, mark, release, recapture simulations. Label which histogram corresponds to which estimator (in a legend if in the same histogram panel or in a title if in two subplots). plot as a vertical line the true number of turtles for comparison. print off the average and standard deviation of $\hat{n}$ over the simulations.

```julia
nb_turtles = 200
nb_capture_mark_release = 50
nb_recapture = 42
estimators = [chapman_estimator, lincoln_petersen_estimator]
estimator_names = ["Chapman", "Lincoln-Petersen"]

nb_sims = 5000 # number of simulations
```

    5000

```julia
figure()
xlabel("estimated # turtles")
ylabel("density")
axvline(x=nb_turtles, color="k")

n̂s = [NaN for s = 1:nb_sims]
for (e, estimator) in enumerate(estimators)
    for s = 1:nb_sims
        n̂s[s] = sim_capture_mark_release_recapture(nb_turtles, nb_capture_mark_release,
                                                   nb_recapture, estimator)
    end
    println("estimator: ", estimator_names[e])
    println("\t⟨n̂⟩ = ", mean(n̂s))
    println("\tstd(n̂) = ", std(n̂s))

    hist(n̂s, label=estimator_names[e], alpha=0.4, density=true)
end
legend()
```

(10) comment on which estimator, Chapman or Lincoln-Petersen, that appears to be most unbiased.

The Chapman estimator appears to be unbiased; the Lincoln-Peterson estimator appears to be biased since the average estimated number of turtles among the simulations is not equal to the true number of turtles.

(11) evaluate the consistency of the Chapman estimator by plotting the simulated distribution of $\hat{n}$ for several different values of `nb_recapture`.

```julia
color_norm = PyPlot.matplotlib.colors.Normalize(vmin=50, vmax=150)
kr_to_color = PyPlot.cm.ScalarMappable(norm=color_norm, cmap=get_cmap("viridis")).to_rgba

figure()
for nb_recapture in 50:25:150
    for s = 1:nb_sims
        n̂s[s] = sim_capture_mark_release_recapture(nb_turtles, nb_capture_mark_release,
                                                   nb_recapture, chapman_estimator)
    end
    println("k_r = ", nb_recapture)
    println("\t⟨n̂⟩ = ", mean(n̂s))
    println("\tstd(n̂) = ", std(n̂s))

    hist(n̂s, label="# recapture = $nb_recapture", alpha=0.4,
        density=true, color=kr_to_color(nb_recapture))
end
xlabel("estimated # turtles")
ylabel("density")
legend()
```

the Chapman estimator appears to be consistent because the density concentrates on the true number of turtles as we increase the number of recaptured turtles. to support this conclusion (in addition to the visual histogram), the standard deviation of the estimated number of turtles decreases as the number of recaptured turtles increases.


```python
from decodes.core import *
from decodes.io.jupyter_out import JupyterOut
import math

out = JupyterOut.unit_square( )
```

# Transformations in Code
todo

## Xform Objects in Decod.es

```python
"""
Xform Initialization
An Xform object is initialized with values configured to match those of an
identity matrix.
"""

class Xform(object):

    def __init__(self):
        self._m = [0.0]*16
        self.c11, self.c22, self.c33, self.c44, = 1.0, 1.0, 1.0, 1.0
```

```python
"""
Transformation Management
Here, a method is defined that attempts to apply a spatial transformation as
described by this Xform matrix to any geometric object given. An appropriate
mechanism for doing so is applied on a case-by-case basis, with constituent
Points and Vecs transformed separately, stripping away the translation portion
of the matrix where needed. If no appropriate means is defined, an error is
raised.
"""
def transform(self,other):

    # if other is a Point
    if isinstance(other, Point):
        # apply the transformation and return a new Point
        tup = self._xform_tuple(other.to_tuple())
        pt = Point(tup[0],tup[1],tup[2])
        return pt

    # if other is a Vec
    if isinstance(other, Vec):
        # apply the transformation and return a new Vec
        tup = self._xform_tuple(other.to_tuple())
        vec = Vec(tup[0],tup[1],tup[2])
        return vec

    # if other is a LinearEntity
    if isinstance(other, LinearEntity):
        # apply the transformation to the LinearEntity start point
        other._pt = other._pt*self
        xf = self.strip_translation()
        # apply the transformation to the LinearEntity vector
        other._vec = other._vec*xf
        # assemble and return a new LinearEntity
        return other

    # if other is a CS
    if isinstance(other, CS):
        cs = other
        # apply the transformation to the CS origin
        tup = self._xform_tuple(cs.origin.to_tuple())
        origin = Point(tup[0],tup[1],tup[2])
        xf = self.strip_translation()
        # apply the transformation to each of the CS axis Vecs
        tup = xf._xform_tuple(cs.x_axis.to_tuple())
        x_axis = Vec(tup[0],tup[1],tup[2])
        tup = xf._xform_tuple(cs.y_axis.to_tuple())
        y_axis = Vec(tup[0],tup[1],tup[2])
        # assemble and return a new CS
        ret = CS(origin, x_axis, y_axis)
        return ret

    # if other is a Circle
    if isinstance(other, Circle):
        # apply the transformation as if this Circle was a Plane
        pln = other.plane * self
        cir = Circle(pln,other.rad)
        # assemble and return a new Circle
        return cir

    # if other is a Plane
    if isinstance(other, Plane):
        pln = other
        # apply the transformation to the Plane origin
        tup = self._xform_tuple(pln.origin.to_tuple())
        origin = Point(tup[0],tup[1],tup[2])
        xf = self.strip_translation()
        # apply the transformation to the Plane normal Vec
        tup = xf._xform_tuple(pln.normal.to_tuple())
        normal = Vec(tup[0],tup[1],tup[2]).normalized()
        # assemble and return a new Plane
        pln = Plane(origin, normal)
        pln.copy_props(other)
        return pln

    # if no transformation is defined for this object, raise an error.
    raise NotImplementedError("I'm sorry. I can't transform that.")

```

```python
"""
A Rotation of a Segment
Following an earlier example that defined a matrix for rotating vectors, a
given Segment seg is rotated by 90 degrees about the origin.
"""
xf = Xform()
xf.c11 = 0
xf.c12 = -1
xf.c21 = 1
xf.c22 = 0

seg *= xf
```

```python
"""
Variable Rotation of Segments
Here, a function is defined that rotates a given termination Point of a Segment
a variable amount as determined by its distance to an attractor Point. For this
purpose, a single Xform object is continually altered and applied to each Point. Two given Intervals, ival_dist and ival_spin, control the amount of rotation.
"""
xf = Xform()
def spin(pt):
    # calculate an angle of rotation
    ang = Interval.remap(pt.dist(attr_pt),ival_dist,ival_spin)
    sint, cost = sin(ang), cos(ang)
    # set the components of the matrix to the desired rotation
    xf.c11, xf.c12, xf.c21, xf.c22 = cost, -sint, sint, cost
    return pt * xf

segs = [Segment(spin(seg.spt),spin(seg.ept)) for seg in segs]
```

```python
"""
Combining Transformations
Xform objects may be combined before being applied to a geometric object.
Here, four transformations are defined by calling on static methods of the
Xform class, and then combined before being recursively applied to a given
collection of Segments. Note that the order of combination plays a significant
role in the nature of the result.
"""
xf_rot = Xform.rotation(angle = radians(15))
xf_trn = Xform.translation(Vec(0.50, 0.33))
xf_scl = Xform.scale(1.02)
xf_mir = Xform.mirror("world_yz")

xf_sum = xf_trn * xf_rot * xf_scl

# figure is a given collection of Segments
figs = [figure]
for i in range(count):
    figs.append([xf_sum*seg for seg in figs[-1]])
```

## A Library of Spatial Transformations

### Change of Basis

\begin{align}
     {\large M_{\vec{s_{1}},\vec{s_{2}},\vec{s_{3}} \to \vec{e_{1}},\vec{e_{2}},\vec{e_{3}}} } =
    \begin{bmatrix}
        s_{1x} & s_{2x} & s_{3x} & 0 \\
        s_{1y} & s_{2y} & s_{3y} & 0 \\
        s_{1z} & s_{2z} & s_{3z} & 0 \\
        0 & 0 & 0 & 1 \\
    \end{bmatrix}
\end{align}

\begin{align}
     {\large M_{\vec{e_{1}},\vec{e_{2}},\vec{e_{3}} \to \vec{t_{1}},\vec{t_{2}},\vec{t_{3}} } } =
    \begin{bmatrix}
        t_{1x} & t_{1x} & t_{1x} & 0 \\
        t_{2y} & t_{2y} & t_{2y} & 0 \\
        t_{3z} & t_{3z} & t_{3z} & 0 \\
        0 & 0 & 0 & 1 \\
    \end{bmatrix}
\end{align}

\begin{align}
    \begin{bmatrix}
        1 & 0 & 0 & o_{tx} \\
        0 & 1 & 0 & o_{ty} \\
        0 & 0 & 1 & o_{tz} \\
        0 & 0 & 0 & 1 \\
    \end{bmatrix}
     \times
    {\large M_{\vec{e_{1}},\vec{e_{2}},\vec{e_{3}} \to \vec{t_{1}},\vec{t_{2}},\vec{t_{3}} } }
     \times
     {\large M_{\vec{s_{1}},\vec{s_{2}},\vec{s_{3}} \to \vec{e_{1}},\vec{e_{2}},\vec{e_{3}}} }
     \times
    \begin{bmatrix}
        1 & 0 & 0 & -o_{sx} \\
        0 & 1 & 0 & -o_{sy} \\
        0 & 0 & 1 & -o_{sz} \\
        0 & 0 & 0 & 1 \\
    \end{bmatrix}
\end{align}

```python

```

107 - Transformations and Intersections/256 - Transformations in Code.ipynb


# Smith Sphere

The [smith chart](http://en.wikipedia.org/wiki/Smith_chart) is a nomogram used frequently in RF/Microwave Engineering. Since its inception it has been recognised that projecting the chart onto the reimen sphere [1].

[1]H. . Wheeler, “Reflection Charts Relating to Impedance Matching,” IEEE Transactions on Microwave Theory and Techniques, vol. 32, no. 9, pp. 1008–1021, Sep. 1984.

```python
#from IPython.display import SVG
#SVG('pics/smith_sphere.svg')
from galgebra.printer import Format, Fmt
```

```python
from galgebra import ga
from galgebra.ga import Ga
from sympy import *
Format()

(o3d,er,ex,es) = Ga.build('e_r e_x e_s',g=[1,1,1])
(o2d,zr,zx) = Ga.build('z_r z_x',g=[1,1])

Bz = er^ex # impedance plance
Bs = es^ex # reflection coefficient plane
Bx = er^es
I = o3d.I()

def down(p, N):
    '''
    stereographically project a vector in G3 downto the bivector N
    '''
    n= -1*N.dual()
    return -(n^p)*(n-n*(n|p)).inv()

def up(p):
    '''
    stereographically project a vector in G2 upto the space G3
    '''
    if (p^Bz).obj == 0:
        N = Bz
    elif  (p^Bs).obj == 0:
        N = Bs

    n = -N.dual()

    return   n + 2*(p*p + 1).inv()*(p-n)

a,b,c,z,s,n = [o3d.mv(k,'vector') for k in ['a','b','c','z','s' ,'n']]
```

Starting with an impedance vector $z$, defined by a vector in the impedance plane  $B_z$, this vector has two scalar components ( $z^r$, $z^x$) known as resistance and reactance

```python
Bz.dual()
```

\begin{equation*} - \boldsymbol{e}_{s} \end{equation*}

```python
Bz.is_zero()
```

    False

```python
z = z.proj([er,ex])
z
```

\begin{equation*} z^{r} \boldsymbol{e}_{r} + z^{x} \boldsymbol{e}_{x} \end{equation*}

stereographically  up-projecting this onto the sphere to point $p$,

```python
p = up(z)
p
```

\begin{equation*} \frac{2 z^{r}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{r} + \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{x} + \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{s} \end{equation*}

```python
simplify(p.norm2())
```

$\displaystyle 1$

If we stereo-project this back onto the impedance plane

```python
down(p, Bz)
```

\begin{equation*} z^{r} \boldsymbol{e}_{r} + z^{x} \boldsymbol{e}_{x} \end{equation*}

```python
down(p,Bs).simplify()
```

\begin{equation*} \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{x} + \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{s} \end{equation*}

```python
(z-er)*(z+er).inv()
```

\begin{equation*} \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1}  - \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{r}\wedge \boldsymbol{e}_{x} \end{equation*}

```python
p
```

\begin{equation*} \frac{2 z^{r}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{r} + \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{x} + \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{s} \end{equation*}

```python
R=((-pi/4)*Bx).exp()
R
```

\begin{equation*} \frac{\sqrt{2}}{2}  - \frac{\sqrt{2}}{2} \boldsymbol{e}_{r}\wedge \boldsymbol{e}_{s} \end{equation*}

```python
R*p*R.rev()
```

\begin{equation*} \frac{- {\left ( z^{r} \right )}^{2} - {\left ( z^{x} \right )}^{2} + 1}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{r} + \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{x} + \frac{2 z^{r}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{s} \end{equation*}

```python
down(R*p*R.rev(),Bz)
```

\begin{equation*} \frac{- {\left ( z^{r} \right )}^{2} - {\left ( z^{x} \right )}^{2} + 1}{{\left ( z^{r} \right )}^{2} - 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{r} + \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} - 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e}_{x} \end{equation*}


## 3.1 Introduction

In Chapter 2, we discussed the network topology, the weights associated with edges, the combination matrix, the average consensus algorithm, and how to implement it in the real CPU clusters with BlueFog. In this section, we will discuss how to solve the general optimization problem in a decentralized manner.

### 3.1.1 The optimization problem

Consider $n$ computing nodes collaborate to solve the problem:

$$\min_{x \in \mathbb{R}^d} \quad \frac{1}{n}\sum_{i=1}^n f_i(x)$$

where $f_i(x)$ is a local and private function held by node $i$. Each node $i$ can access its own variable $x$ or gradient $\nabla f_i(x)$, but it has to communicate to access information from other nodes.

If each $f_i(x)$ is assumed to be smooth, the leading algorithm to solve the above problem is gradient descent:

\begin{align}
x^{(k+1)} = \frac{1}{n}\sum_{i=1}^n \Big(x^{(k)} - \alpha \nabla f_i(x^{(k)}) \Big)
\end{align}

The above recursion can be conducted in a distributed manner. First, each node $i$ conducts a local gradient descent. Second, all nodes will synchronize with each other to achieve the global average. As a result, the above algorithm is named as **distributed gradient descent**.

The global average operation in distributed gradient descent can be conducted either via Parameter-Server or All-Reduce, but it will incur either a significant bandwidth cost or high latency. This motivates us to solve the above problem in a **decentralized** manner in which each node only needs to communicate with its direct neighbors.

### 3.1.2 Organization

The contents of this chapter is organized as follows:

- Sec.3.2: Decentralized gradient descent: the adapt-then-combine (ATC) version

- Sec.3.3: Decentralized gradient descent: the adapt-with-combination (AWC) version

- Sec.3.4: The influence of network topologies on decentralized gradient descent

- Sec.3.5: Decentralized stochastic gradient descent

### 3.1.3 Initialize BlueFog and test it

All contents in this section are displayed in Jupyter notebook, and all experimental examples are written with BlueFog and iParallel. Readers not familiar with how to run BlueFog in ipython notebook environment is encouraged to read Sec. [HelloWorld section] first. In the following codes, we will initialize BlueFog and test whether it works normally.

In the following code, you should be able to see the id of your CPUs. We use 8 CPUs to conduct the following experiment.

```
import ipyparallel as ipp

rc = ipp.Client(profile="bluefog")
rc.ids
```

    [0, 1, 2, 3, 4, 5, 6, 7]

Let each agent import necessary modules and then initialize BlueFog. You should be able to see the printed information like:

> \[stdout:0\] Hello, I am 1 among 8 processes
>
> ...

```
%%px
import numpy as np
import bluefog.torch as bf
import torch
from bluefog.common import topology_util
import networkx as nx

bf.init()
print(f"Hello, I am {bf.rank()} among {bf.size()} processes")
```

    [stdout:0] Hello, I am 3 among 8 processes
    [stdout:1] Hello, I am 5 among 8 processes
    [stdout:2] Hello, I am 6 among 8 processes
    [stdout:3] Hello, I am 2 among 8 processes
    [stdout:4] Hello, I am 0 among 8 processes
    [stdout:5] Hello, I am 1 among 8 processes
    [stdout:6] Hello, I am 7 among 8 processes
    [stdout:7] Hello, I am 4 among 8 processes

Push seed to each agent so that the simulation can be reproduced.

```
dview = rc[:]  # A DirectView of all engines
dview.block = True

# Push the data into all workers
dview.push({"seed": 2021}, block=True)
```

    [None, None, None, None, None, None, None, None]

After running the following code, you should be able to see the printed information like

> \[stdout:0\] I received seed as value:  2021
>
> ...

```
%%px
print("I received seed as value: ", seed)
```

    [stdout:0] I received seed as value:  2021
    [stdout:1] I received seed as value:  2021
    [stdout:2] I received seed as value:  2021
    [stdout:3] I received seed as value:  2021
    [stdout:4] I received seed as value:  2021
    [stdout:5] I received seed as value:  2021
    [stdout:6] I received seed as value:  2021
    [stdout:7] I received seed as value:  2021

Congratulations! Your BlueFog is initialized and tested successfully.


```python
import matplotlib.pyplot as plt
import matplotlib as mpl
import numpy as np
import sympy as sy
sy.init_printing()
```

```python
def round_expr(expr, num_digits):
    return expr.xreplace({n : round(n, num_digits) for n in expr.atoms(sy.Number)})
```

#  Systems of First-Order Equations

In time series analysis, we study difference equations by writing them into a linear system. For instance,

$$
3y_{t+3} - 2y_{t+2} + 4y_{t+1} - y_t = 0
$$

We define

$$
\mathbf{x}_t =
\left[
\begin{matrix}
y_t\\
y_{t+1}\\
y_{t+2}
\end{matrix}
\right], \qquad
\mathbf{x}_{t+1} =
\left[
\begin{matrix}
y_{t+1}\\
y_{t+2}\\
y_{t+3}
\end{matrix}
\right]
$$

Rerrange the difference equation for better visual shape,

$$
y_{t+3} = \frac{2}{3}y_{t+2} - \frac{4}{3}y_{t+1} + \frac{1}{3}y_{t}
$$

The difference equation can be rewritten as

$$
\mathbf{x}_{t+1} = A \mathbf{x}_{t}
$$

That is,

$$
\left[
\begin{matrix}
y_{t+1}\\
y_{t+2}\\
y_{t+3}
\end{matrix}
\right] =
\left[
\begin{matrix}
0 & 1 & 0\\
0 & 0 & 1\\
\frac{1}{3} & -\frac{4}{3} & \frac{2}{3}
\end{matrix}
\right]
\left[
\begin{matrix}
y_t\\
y_{t+1}\\
y_{t+2}
\end{matrix}
\right]
$$

In general, we make sure the difference equation look like:

$$
y_{t+k} = a_1y_{t+k-1} + a_2y_{t+k-2} + ... + a_ky_{t}
$$

then rewrite as $\mathbf{x}_{t+1} = A \mathbf{x}_{t}$, where

$$
\mathbf{x}_t =
\left[
\begin{matrix}
y_{t}\\
y_{t+1}\\
\vdots\\
y_{t+k-1}
\end{matrix}
\right], \quad
\mathbf{x}_{t+1} =
\left[
\begin{matrix}
y_{t+1}\\
y_{t+2}\\
\vdots\\
y_{t+k}
\end{matrix}
\right]
$$

And also

$$A=\left[\begin{array}{ccccc}
0 & 1 & 0 & \cdots & 0 \\
0 & 0 & 1 & & 0 \\
\vdots & & & \ddots & \vdots \\
0 & 0 & 0 & & 1 \\
a_{k} & a_{k-1} & a_{k-2} & \cdots & a_{1}
\end{array}\right]$$

#  Markov Chains

Markov chain is a type of stochastic process, commonly modeled by difference equation, we will be slightly touching the surface of this topic by walking through an example.

Markov chain is also described by the first-order difference equation $\mathbf{x}_{t+1} = P\mathbf{x}_t$, where $\mathbf{x}_t$ is called state vector, $A$ is called stochastic matrix.

Suppose there are 3 cities $A$, $B$ and $C$, the proportion of population migration among cities are constructed in the stochastic matrix below

$$
M =
\left[
\begin{matrix}
.89 & .07 & .10\\
.07 & .90 & .11\\
.04 & .03 & .79
\end{matrix}
\right]
$$

For instance, the first column means that $89\%$ of population will stay in city $A$, $7\%$ will move to city $B$ and $4\%$ will migrate to city $C$. The first row means $7\%$ of city $B$'s population will immigrate into $A$, $10\%$ of city $C$'s population will immigrate into $A$.

Suppose the initial population of 3 cities are $(593000, 230000, 709000)$, convert the entries into percentage of total population.

```python
x = np.array([593000, 230000, 709000])
x = x/np.sum(x);x
```

    array([0.38707572, 0.15013055, 0.46279373])

Input the stochastic matrix

```python
M = np.array([[.89, .07, .1], [.07, .9, .11], [.04, .03, .79]])
```

After the first period, the population proportion among cities are

```python
x1 = M@x
x1
```

    array([0.4012859 , 0.2131201 , 0.38559399])

The second period

```python
x2 = M@x1
x2
```

    array([0.41062226, 0.26231345, 0.3270643 ])

The third period

```python
x3 = M@x2
x3
```

    array([0.41652218, 0.30080273, 0.28267509])

We can construct a loop till $\mathbf{x}_{100} = M\mathbf{x}_{99}$, then plot the dynamic path. Notice that the curve is flattening after 20 periods, and we call it convergence to steady-state.

```python
k = 100
X = np.zeros((k, 3))
X[0] = M@x
i = 0

while i+1 < 100:
    X[i+1] = M@X[i]
    i = i + 1
```

```python
fig, ax = plt.subplots(figsize = (12, 12))

la = ['City A', 'City B', 'City C']
s = '$%.3f$'
for i in [0, 1, 2]:
    ax.plot(X[:, i], lw = 3, label = la[i] )
    ax.text(x = 20, y = X[-1,i], s = s %X[-1,i], size = 16)

ax.axis([0, 20, 0, .6]) # No need to show more of x, it reaches steady-state around 20 periods
ax.legend(fontsize = 16)
ax.grid()
ax.set_title('Dynamics of Population Percentage')
plt.show()
```

#  Eigenvalue and -vector in Markov Chain

If the $M$ in last example is diagonalizable, there will be $n$ linearly independent eigenvectors and corresponding eigenvalues, $\lambda_1$,...,$\lambda_n$. And eigenvalues can always be arranged so that $\left|\lambda_{1}\right| \geq\left|\lambda_{2}\right| \geq \cdots \geq\left|\lambda_{n}\right|$.

Also, because any initial vector $\mathbb{x}_0 \in \mathbb{R}^n$, we can use the basis of eigenspace (eigenvectors) to represent all $\mathbf{x}$.

$$
\mathbf{x}_{0}=c_{1} \mathbf{v}_{1}+\cdots+c_{n} \mathbf{v}_{n}
$$

This is called eigenvector decomposition of $\mathbf{x}_0$. Multiply by $A$

$$
\begin{aligned}
\mathbf{x}_{1}=A \mathbf{x}_{0} &=c_{1} A \mathbf{v}_{1}+\cdots+c_{n} A \mathbf{v}_{n} \\
&=c_{1} \lambda_{1} \mathbf{v}_{1}+\cdots+c_{n} \lambda_{n} \mathbf{v}_{n}
\end{aligned}
$$

In general, we have a formula for $\mathbf{x}_k$

$$
\mathbf{x}_{k}=c_{1}\left(\lambda_{1}\right)^{k} \mathbf{v}_{1}+\cdots+c_{n}\left(\lambda_{n}\right)^{k} \mathbf{v}_{n}
$$

Now we test if $M$ has $n$ linearly independent eigvectors.

```python
M = sy.Matrix([[.89, .07, .1], [.07, .9, .11], [.04, .03, .79]]);M
```

```python
M.is_diagonalizable()
```

    True

$M$ is diagonalizable, which also means that $M$ has $n$ linearly independent eigvectors.

```python
P, D = M.diagonalize()
P = round_expr(P,4); P # user-defined round function at the top of the notebook
```

```python
D = round_expr(D,4); D
```

First we find the $\big[\mathbf{x}_0\big]_C$, i.e. $c_1, c_2, c_3$

```python
x0 = sy.Matrix([[.3870], [.1501], [0.4627]]);x0
```

```python
P_aug = P.row_join(x0)
P_aug_rref = round_expr(P_aug.rref()[0],4); P_aug_rref
```

```python
c = sy.zeros(3, 1)
for i in [0, 1, 2]:
    c[i] = P_aug_rref[i, 3]
c = round_expr(c,4);c
```

Now we can use the formula to compute $\mathbf{x}_{100}$, it is the same as we have plotted in the graph.

```python
x100 = c[0] * D[0, 0]**100 * P[:, 0]\
+ c[1] * D[1, 1]**100 * P[:, 1]\
+ c[2] * D[2, 2]**100 * P[:, 2]
x100 = round_expr(x100,4);x100
```

This is close enough to the steady-state.

#  Fractal Pictures

Here is an example of fractal geometry, illustrating how dynamic system and affine transformation can create fractal pictures.

The algorithem is perform 4 types of affine transformation. The corresponding probabilites are $p_1, p_2, p_3, p_4$.

$$
\begin{array}{l}
T_{1}\left(\left[\begin{array}{l}
x \\
y
\end{array}\right]\right)=\left[\begin{array}{rr}
0.86 & 0.03 \\
-0.03 & 0.86
\end{array}\right]\left[\begin{array}{l}
x \\
y
\end{array}\right]+\left[\begin{array}{l}
0 \\
1.5
\end{array}\right], p_{1}=0.83 \\
T_{2}\left(\left[\begin{array}{l}
x \\
y
\end{array}\right]\right)=\left[\begin{array}{lr}
0.2 & -0.25 \\
0.21 & 0.23
\end{array}\right]\left[\begin{array}{l}
x \\
y
\end{array}\right]+\left[\begin{array}{l}
0 \\
1.5
\end{array}\right], p_{2}=0.09 \\
T_{3}\left(\left[\begin{array}{l}
x \\
y
\end{array}\right]\right)=\left[\begin{array}{rr}
-0.15 & 0.27 \\
0.25 & 0.26
\end{array}\right]\left[\begin{array}{l}
x \\
y
\end{array}\right]+\left[\begin{array}{l}
0 \\
0.45
\end{array}\right], p_{3}=0.07 \\
T_{4}\left(\left[\begin{array}{l}
x \\
y
\end{array}\right]\right)=\left[\begin{array}{ll}
0 & 0 \\
0 & 0.17
\end{array}\right]\left[\begin{array}{l}
x \\
y
\end{array}\right]+\left[\begin{array}{l}
0 \\
0
\end{array}\right], p_{4}=0.01
\end{array}
$$

The codes below are self-explanatory.

```python
A = np.array([[[.86, .03],
              [-.03, .86]],
             [[.2, -.25],
              [.21, .23]],
             [[-.15, .27],
              [.25, .26]],
             [[0., 0.],
              [0., .17]]])
a = np.array([[[0,1.5]],
             [[0,1.5]],
             [[0,0.45]],
             [[0,0]]])

p1 = 1*np.ones(83)
p2 = 2*np.ones(9)
p3 = 3*np.ones(7)
p4 = 4*np.ones(1)
p = np.hstack((p1,p2,p3,p4))

k = 30000
fig, ax = plt.subplots(figsize = (5,8))
X = np.zeros((2,k))
for i in range(k-1):
    n = np.random.randint(0, 100)
    if p[n] == 1:
        X[:,i+1] = A[0,:,:]@X[:,i]+a[0,:,:]
    elif p[n] == 2:
        X[:,i+1]  = A[1,:,:]@X[:,i]+a[1,:,:]
    elif p[n] == 3:
        X[:,i+1]  = A[2,:,:]@X[:,i]+a[2,:,:]
    else:
        X[:,i+1]  = A[3,:,:]@X[:,i]+a[3,:,:]
ax.scatter(X[0,:],X[1,:], s = 1, color = 'g')
plt.show()
```

Chapter 14 - Applications to Dynamic System.ipynb