# A practical introduction to polynomial chaos with the chaospy package

**Vinzenz Gregor Eck**, Expert Analytics, Oslo

**Jacob Sturdy**, Department of Structural Engineering, NTNU

**Leif Rune Hellevik**, Department of Structural Engineering, NTNU
Date: **Sep 25, 2018**

```
# ipython magic
%matplotlib notebook
%load_ext autoreload
%autoreload 2
```

```
# plot configuration
import matplotlib
import matplotlib.pyplot as plt
plt.style.use("ggplot")
# import seaborn as sns # sets another style
matplotlib.rcParams['lines.linewidth'] = 3
fig_width, fig_height = (7.0,5.0)
matplotlib.rcParams['figure.figsize'] = (fig_width, fig_height)
```

```
import chaospy as cp
import numpy as np
from numpy import linalg as LA
```

# Introduction

To conduct UQSA with the
polynomial chaos (PC) method we use the python package `chaospy`.
This package
includes many features and methods to apply non-intrusive polynomial chaos to
any model with
few lines of code.
Since, `chaospy` allows for convenient
definition of random variables, calculation of joint distributions and
generation of samples,
we apply `chaospy` for the Monte Carlo method as well.
In this notebook we will briefly describe the theory of polynomial chaos and
give a practical step by step guide
for the application of `chaospy`.

# Polynomial chaos

For a more in depth
introduction to the polynomial chaos method see [Xiu 2010](https://press.princeton.edu/titles/9229.html) or [Smith 2013](http://bookstore.siam.org/cs12/)

## Concept

The main concept of polynomial chaos is to approximate the stochastic
model response of a model with stochastic input as polynomial expansion relating
the response to the inputs.
For simplicity, we consider a model function $f$
which takes random variables $\mathbf{Z}$ and non-random variables as
spatial
coordinates $x$ or time $t$, as input:

$$
\begin{equation}
Y = y{}(x, t, \mathbf{Z})
\label{_auto1} \tag{1}
\end{equation}
$$

Then we seek to generate a polynomial expansion such that one can approximate the
model response.

$$
\begin{equation}
Y = y{}(x, t, \mathbf{Z}) \approx y_{N} = \sum_{i=1}^{P} v_i(x,
t) \Phi_i(\mathbf{Z}),
\label{_auto2} \tag{2}
\end{equation}
$$

where $v_i$ are the expansion coefficients, which are only dependent on the non-
stochastic parameters, and $\Phi_i$ are orthogonal polynomials, which depend
solely on the random input parameter $\mathbf{Z}$.
Once the polynomial expansion
is calculated, we can analytically calculate the statistics of approximated
model output.

Since polynomial chaos generally requires fewer model evaluations
to estimates statistics than Monte Carlo methods, this method is preferable
applied to computational-demanding models.

### Applications
Polynomial chaos may be applied in a wide variety of situations. Requirements
for convergence are simply that the function be square integrable with respect
to the inner product associated with the orthogonal polynomials. In other words
if the function has a finite variance, polynomial chaos may be applied without
worrying too much. The caveat to this fact is that convergence may not
necessarily be fast: The smoother the function, the faster the convergence.
Polynomial chaos can handle discontinuities, however, it may be advisable to
reformulate the problem in these situations (e.g. see [Feinberg et al. 2015](https://www.sciencedirect.com/science/article/pii/S1877750315300119)).

## Orthogonal polynomials

As stated above, the polynomial chaos
expansion consists of a sum of basis polynomials in the input parameters
$\mathbf{Z}$. By using orthogonal polynomials as the basis polynomials, the
efficiency of the convergence may be improved and the use of the expansion in
uncertainty quantification and sensitivity analysis is simplified.
Orthogonality of functions is a general concept developed for functional analysis within an
inner product space. Typically the inner product of two functions is defined as
a weighted integral of the product of the functions over a domain of interest.

In particular, the inner product of
two polynomials is defined as the expected value of their product for the purposes of polynomial chaos. I.e. the
integral of their product weighted with respect to the distribution of random
input parameters.
Two polynomials are orthogonal if their inner product is 0,
and a set of orthogonal polynomials is orthogonal if every pair of distinct
polynomials are orthogonal.

The following equalities hold for the orthogonal
basis polynomials used in  polynomial chaos

$$
\begin{align*}
 \mathbb{E}(\Phi_i(\mathbf{Z})\Phi_j(\mathbf{Z})) &= \langle
\Phi_i(\mathbf{Z}),\Phi_j(\mathbf{Z}) \rangle\\
 &= \int_{\Omega}
\Phi_i(z)\Phi_j(z)w(z) dz \\
 &= \int_{\Omega} \Phi_i(z)\Phi_j(z)f(z) dz  \\
 &=
{\int_{\Omega} \Phi_i(z)\Phi_j(z) dF_Z(\mathbf{z})} = h_j \delta_{ij},
\end{align*}
$$

where $h_j$ is equal to the normalisation factor of the used polynomials, and
$\Phi_i(\mathbf{Z})$ indicates the substitution of the random variable
$\mathbf{Z}$ as the polynomial's variable.
Note that $\Phi_0$ is a polynomial of
degree zero, i.e. a constant thus $\mathbb{E}(\Phi_0\Phi_j) = \Phi_0
\mathbb{E}(\Phi_j) = 0$ which implies that $\mathbb{E}(\Phi_j) = 0$ for $j \neq
0$.

Once the random inputs $\mathbf{Z}$ are properly defined with
marginal distributions, orthogonal polynomials can be constructed.
For most univariate distributions, polynomial bases functions are defined, and listed in
the Asker Wilkeys scheme.
A set of orthogonal polynomials can be created from
those basis polynomials with orthogonalization methods as Cholesky
decomposition, three terms recursion, and modified Gram-Schmidt.

## Expansion coefficients

There are two non-intrusive ways of approximating a polynomial
chaos expansion coefficients:

### Regression

Supposing a polynomial expansion
approximation $y_{N} = \sum_i v_i \Phi_i$,
then *stochastic collocation*
specifies a set of nodes, $\Theta_{M} =
\left\{\mathbf{z}^{(s)}\right\}_{s=1}^{P}$, where the deterministic values of
$y_{N}=y$.
The task is thus to find suitable coefficients $v_i$ such that this
condition is satisfied.
Considering the existence of a set of collocation nodes
$\left\{\mathbf{z}^{(s)}\right\}_{s=1}^{P}$, then $y_{N} = \sum_i v_i \Phi_i$
can be
 formed into linear system of equations for the coefficients $v_i$ at
these nodes:

$$
\begin{equation}
\begin{bmatrix}
\Phi_0(\mathbf{z}^{(1)}) & \cdots &
\Phi_P(\mathbf{z}^{(1)}) \\
\vdots & & \vdots \\
\Phi_0(\mathbf{z}^{(N)}) &
\cdot & \Phi_P(\mathbf{z}^{(n)})
\end{bmatrix}
\begin{bmatrix}
v_{0}\\
\vdots \\
v_{P}
\end{bmatrix}
= \begin{bmatrix}
y(\mathbf{z}^{(1)}) \\
\vdots \\
y
(\mathbf{z}^{(N)})
\end{bmatrix}
\end{equation}
\label{eq:stochColl} \tag{3}
$$

Now we can use regression to achieve the relaxed condition that $y_{N}$ is
"sufficiently close" to $y$ at $\left\{\mathbf{z}^{(s)}\right\}_{s=1}^{P}$.
This
is done by choosing a larger number of samples so that ([3](#eq:stochColl)) is
over determined and
then minimizing the appropriate error ${\lvert\lvert {y_{N}
\rvert\rvert}-y}_{R}$ over $\left\{\mathbf{z}^{(s)}\right\}_{s=1}^{P}$.
Ordinary
least squares, Ridge-Regression, and Tikhonov-Regularization are all regression
methods that may be applied to this problem.

### Pseudo-spectral projection

Discrete projection refers to the approximation of coefficients for $y_{N} =
\sum_{i=1}^{P} v_i \Phi_i$, by directly approximating the orthogonal projection
coefficients

$$
\begin{equation}
v_i = \frac{1}{h_i} \mathbb{E}(y \Phi_i) = \frac{1}{h_i}
\int_{\Omega} y \Phi_i dF_z,
\label{_auto3} \tag{4}
\end{equation}
$$

using a quadrature scheme to calculate the integral $\int_{\Omega} y \Phi_i
dF_z$ as a sum $\sum_{i=1}^{P} w_i y(\mathbf{z}^{(i)})$, where $w_i$ are the
quadrature weights.

This results in an approximation $\tilde{v}_{i}$ of $v_{i}$
the error between the final approximation may be split

$$
\begin{equation}
\lVert{y_{N} - y}\rVert \leq \lVert \sum_{i=1}^{P} v_i \Phi_i -
y{} \rVert + \lVert {\sum_{i=1}^{P} \left( v_i - \tilde{v}_i \right)\Phi_i}
\rVert
\label{eq:ps_error} \tag{5}
\end{equation}
$$

where the first term is called the truncation error and the second term the
quadrature error.
Thus one may consider the maximal accuracy for a given
polynomial order $P$ and see this should be achieved as the quadrature error is
reduced to almost 0 by increased number of collocation nodes.

## Statistics

Once the expansion is generated, it can be used directly to calculated
statistics for the uncertainty and sensitivity analysis.
The two most common
measures for uncertainty quantification, **expected value** and **variance**,
can be calculated by
inserting the expansion into the definition of the
measures.

The expected value is equal to the first expansion coefficient:

$$
\begin{equation}
  \begin{aligned}
{\mathbb{E}}(Y) \approx \int_{\Omega} \sum_{i=1}^{P} v_i \Phi_i(\mathbf{z}{})
dF_Z(\mathbf{z}) = v_1.
  \end{aligned}
    \label{eq:expectedValue_gPCE}
\tag{6}
\end{equation}
$$

The variance is the sum of squared expansion coefficients multiplied by
normalisation constants of the polynomials:

$$
\begin{equation}
    \begin{aligned}
    \operatorname{Var}(Y) &\approx
{\mathbb{E}}{({v(x,t,\mathbf{Z})}(\mathbf{Z})-{\mathbb{E}}(Y))^2} =
\int_{\Omega}({v(x,t,\mathbf{Z})}(\mathbf{z}) - v_1)^2 dF_Z(\mathbf{z}) \\
&=  \int_{\Omega}\left(\sum_{i=1}^{P} v_i \Phi_i(\mathbf{z}) \right)^2
dF_Z(\mathbf{z}) - v_1^2 = \sum_{i=1}^{P} v_i^2
\int_{\Omega}{\Phi^2_i(\mathbf{z})}dF_Z(\mathbf{z}) - v_1^2 \\
    &=
\sum_{i=1}^{P} v_i^2 h_i - v_1^2 = \sum_{i=2}^{P} v_i^2 h_i
  \end{aligned}
\label{eq:variance_gPCE} \tag{7}
\end{equation}
$$

(Note the orthogonality of individual terms implies their covariance is zero,
thus the variance is simply the sum of the variances of the terms.)

The Sobol
indices may be calculated quite simply from the expansion terms due to the fact
that the _ANOVA_ decomposition is unique. Thus, the main effect of a parameter
$z_i$ is simply the variance of all terms only in $z_i$. Explicitly let
$\mathcal{A}_{i} = \{k | \Phi_{k}(\mathbf{z}) = \Phi_{k}(z_i)\} $ ,i.e.
$\mathcal{A}_{i}$ is the set of all indices of basis functions depending only on
$z_i$ then

$$
\begin{equation}
    \begin{aligned}
    f_i &= \sum_{k\in \mathcal{A}_{i}}
v_{k} \Phi_{k} \implies \\
    S_i &= \frac{1}{\operatorname{Var}(Y)} \sum_{k\in
\mathcal{A}_{i}} \operatorname{Var}(v_{k} \Phi_{k}) \\
    S_i &=
\frac{1}{\operatorname{Var}(Y)} \sum_{k\in \mathcal{A}_{i}} v_{k}^2 h_{k}
\end{aligned}
  \label{eq:sensitivity_gPCE} \tag{8}
\end{equation}
$$

and similarly one may define $\mathcal{A}_{ij}$ for pairwise combinations of
inputs and further to calculate all orders of sensitivities.

# Chaospy

The python package `chaospy` an introductory paper to
the package including a comparison to other software packages is
presented here [Feinberg 2015](https://www.sciencedirect.com/science/article/pii/S1877750315300119).
You can find an introduction, tutorials and
the source code at the [projects homepage](https://github.com/jonathf/chaospy)
The installation of the package can be done via `pip`:

```
pip install chaospy
```

In the following we will use the import naming convention of the package creator
to import the package in python:

```
import chaospy as cp
```

Therefore it will be convenient to see whenever a method of the package is
applied.

The package `chaospy` is doc-string annotated, which means that every
method provides a short help text with small examples.
To show the method
documentation simply type a `?` after the method name in a ipython console or
notebook.
As shown in the following two examples:

```
# show help for uniform distributions
cp.Uniform?
```

```
# show help for sample generation
cp.samplegen?
```

## Steps for polynomial chaos analysis with chaospy

To conduct UQSA analysis with polynomial chaos we
need to follow the following steps:

* Definition of the marginal and joint distributions

* Generation of the orthogonal polynomials

* Linear regression

  * Generation of samples

  * `Evaluation of the model for all samples`

  * Generation of the polynomial chaos expansion

* Pseudo-spectral projection

  * Generation of integration nodes and weights

  * `Evaluation of the model for all nodes`

  * Generation of the polynomial chaos expansion

* Calculations of all statistics

Note, that steps  **3 Linear regression**  and **4 Pseudo-spectral projection** are interchangeable. They are simply different methods of
calculating the expansion coefficients. In both cases
one generates a set of points in the parameter space where the model must be evaluated (steps 3.b and 4.b,
respectively).

## Step 1: Definition of marginal and joint distributions

The analysis of a each model starts with the
definition of the marginal distributions for each random model input, i.e.
describing it as
random variable.
Univariate random variables can be defined
with `chaospy` by calling the class-constructor of a distribution type, e.g
`cp.Normal()`, with arguments to describe the particular distribution, e.g. mean
value and standard deviation for `cp.Normal`.
The help function can be used to
find out more about the required arguments, e.g. `help(cp.Normal)`.

In the
following an example for 3 random variables with uniform, normal and log-normal
distribution:

```
# simple distributions
rv1 = cp.Uniform(0, 1)
rv2 = cp.Normal(0, 1)
rv3 = cp.Lognormal(0, 1, 0.2, 0.8)
print(rv1, rv2, rv3)
```

After all random input variables are defined with univariate random variables a
multi-variate random variable and its joint distribution
can be constructed with
the following command:

```
# joint distributions
joint_distribution = cp.J(rv1, rv2, rv3)
print(joint_distribution)
```

It is also possible to construct independent identical distributed random
variables from any univariate variable:

```
# creating iid variables
X = cp.Normal()
Y = cp.Iid(X, 4)
print(Y)
```

All 64 distributions available in the chaospy package can be found in the
following table:

Distributions   	      implemented     	        in chaospy

   Alpha
	    Birnbaum-Sanders
Laplace                    	    Power log-normal

   Anglit                 	    Fisher-Snedecor          	    Levy
	    Power normal

   Arcsinus               	    Fisk/log-
logistic        	    Log-gamma                  	    Raised cosine

Beta                   	    Folded Cauchy
	    Log-laplace
Rayleigh

   Brandford
	    Folded normal
Log-normal                 	    Reciprocal

   Burr                   	    Frechet                  	    Log-
uniform                	    Right-skewed Gumbel

   Cauchy                 	    Gamma                    	    Logistic
	    Student-T

   Chi                    	    Gen.
exponential         	    Lomax
   Triangle

   Chi-square             	    Gen.
extreme value       	    Maxwell
   Truncated exponential

   Double Gamma           	    Gen. gamma
	    Mielke's beta-kappa
Truncated normal

   Double Weibull
	    Gen. half-logistic
Nakagami                   	    Tukey-Lambda

   Epanechnikov           	    Gilbrat                  	    Non-
central chi-squared    	    Uniform

   Erlang                 	    Truncated Gumbel         	    Non-
central Student-T      	    Wald

   Exponential            	    Gumbel                   	    Non-
central F              	    Weibull

   Exponential power      	    Hypergeometric secant    	    Normal
	    Wigner

   Exponential Weibull    	    Kumaraswamy
	    Pareto (first kind)
Wrapped Cauchy

## Step 2: Orthogonal Polynomials

The orthogonal
polynomials can be generated with different methods, in `chaospy` there are 4
methods implemented. The
most stable method, and therefore most advised is the
*three terms recursion* method.

Orthogonalization Method

   Cholesky decomposition
	    cp.orth\_chol

Three terms recursion       	    cp.orth\_ttr

   Modified Gram-Schmidt       	    cp.orth\_gs

Regarding the *three terms recursion* method:
For the distributions Normal, Uniform, Gamma,
Log-normal, Triangle, Beta and stochastic independent variable combinations of
those,
the three terms recursion coefficients are known.
For all other
distributions the coefficients are estimated numerically.
The *three terms
recursion* method is then also called **discretized stieltjes method**.

The
most stable method and therefore most applied method is the **three terms
recursion** (**discretized stieltjes method**) method.

We will look at all in a
small example, try to increase the polynomial order and the instabilities of the
methods become visible.

```
# example orthogonalization schemes
# a normal random variable
n = cp.Normal(0, 1)

x = np.linspace(0,1, 50)
# the polynomial order of the orthogonal polynomials
polynomial_order = 3

poly = cp.orth_chol(polynomial_order, n, normed=True)
print('Cholesky decomposition {}'.format(poly))
ax = plt.subplot(131)
ax.set_title('Cholesky decomposition')
_=plt.plot(x, poly(x).T)
_=plt.xticks([])

poly = cp.orth_ttr(polynomial_order, n, normed=True)
print('Discretized Stieltjes / Three terms reccursion {}'.format(poly))
ax = plt.subplot(132)
ax.set_title('Discretized Stieltjes ')
_=plt.plot(x, poly(x).T)

poly = cp.orth_gs(polynomial_order, n, normed=True)
print('Modified Gram-Schmidt {}'.format(poly))
ax = plt.subplot(133)
ax.set_title('Modified Gram-Schmidt')
_=plt.plot(x, poly(x).T)
```

## Step 3.: Linear regression

The linear regression method requires to conduct
the three following steps:

1. Generation of samples

2. `Evaluation of the model for all samples`

3. Generation of the polynomial chaos expansion

In the following steps, we will not consider the model evaluation.

### Step 3.a: Sampling

Once a random variable is defined or a joint
random variable -referred as distribution here, the following
method can be
used to generate as set of samples:

```
# sampling in chaospy
u = cp.Uniform(0,1)
u.sample?
```

The method takes the arguments **size** which is the number of samples and
**rule** which is the applied sampling scheme.
The following example shows the
creation of 2 set of samples for the sampling schemes *(Pseudo-)Random* and
*Hammersley*.

```
# example sampling
u1 = cp.Uniform(0,1)
u2 = cp.Uniform(0,1)
joint_distribution = cp.J(u1, u2)
number_of_samples = 350
samples_random = joint_distribution.sample(size=number_of_samples, rule='R')
samples_hammersley = joint_distribution.sample(size=number_of_samples, rule='M')

fig1, ax1 = plt.subplots()
ax1.set_title('Random')
ax1.scatter(*samples_random)
ax1.set_xlabel("Uniform 1")
ax1.set_ylabel("Uniform 2")
ax1.axis('equal')

fig2, ax2 = plt.subplots()
ax2.set_title('Hammersley sampling')
ax2.scatter(*samples_hammersley)
ax2.set_xlabel("Uniform 1")
ax2.set_ylabel("Uniform 2")
ax2.axis('equal')
```

All sampling schemes implemented in chaospy are listed in the following table:

Key
Name      	 Nested

   C      	    Chebyshev nodes
   no

   NC
	    Nested Chebyshev    	    yes

   K      	    Korobov
	    no

   R
	    (Pseudo-)Random     	    no

   RG     	    Regular
grid        	    no

   NG     	    Nested grid
   yes

   L
	    Latin hypercube     	    no

   S      	    Sobol
	    yes

   H
	    Halton              	    yes

   M
Hammersley          	    yes

### Step 3.a.i: Importing and exporting samples

It may be useful to export the samples from `chaospy` for use in another program. The most useful
format for exporting the samples likely depends on the external program, but it
is quite straightforward to save the samples as a `CSV` file with a delimeter of your
choice:

```
# example save samples to file
# Creates a csv file where each row corresponds to the sample number and each column with teh variables in the joint distribution
csv_file = "csv_samples.csv"
sep = '\t'
header = ["u1", "u2"]
header = sep.join(header)
np.savetxt(csv_file, samples_random, delimiter=sep, header=header)
```

Each row of the csv file now contains a single sample from the joint
distribution with the columns corresponding to each component.

Now you may
evaluate these samples with an external program and save the resulting data into
a similarly formatted `CSV` file. Again each row should correspond to a single
sample value and each column to different components of the model output.

```
# example load samples from file
# loads a csv file where the samples/or model evaluations for each sample are saved
# with one sample per row. Multiple components ofoutput can be stored as separate columns
filepath = "external_evaluations.csv"
data = np.loadtxt(filepath)
```

### Step 3.c: Polynomial Chaos Expansion

After the model is evaluated for all
samples, the polynomial chaos expansion can be generated with the following
method:

```
# linear regression in chaospy
cp.fit_regression?
```

A complete example for polynomial chaos expansion using
the linear regression follows.
The model applied the very simple mathematical
expression:

$$
\begin{equation}
 y(z_1, z_2) = z_1 + z_1 z_2
\label{eq:dummy_model} \tag{9}
\end{equation}
$$

The random variables for $Z_1, Z_2$ are defined as simple uniform random
variables:

$$
\begin{equation}
Z_1 = \mbox{U}(0,1), \quad Z_2 = \mbox{U}(0,1)
\label{eq:dummy_rv} \tag{10}
\end{equation}
$$

The mean of this should be $\frac{3}{4}$, the variance should be
$\frac{31}{144}$ and the sensitivites to $Z_1$ and $Z_2$ are respectively
$\frac{3}{31}$ and $\frac{27}{31}$.

Here is the annotated example code with all
steps required to generate a polynomial chaos expansion with
linear regression:

```
# example linear regression
# 1. define marginal and joint distributions
u1 = cp.Uniform(0,1)
u2 = cp.Uniform(0,1)
joint_distribution = cp.J(u1, u2)

# 2. generate orthogonal polynomials
polynomial_order = 3
poly = cp.orth_ttr(polynomial_order, joint_distribution)

# 3.1 generate samples
number_of_samples = cp.bertran.terms(polynomial_order, len(joint_distribution))
samples = joint_distribution.sample(size=number_of_samples, rule='R')

# 3.2 evaluate the simple model for all samples
model_evaluations = samples[0]+samples[1]*samples[0]

# 3.3 use regression to generate the polynomial chaos expansion
gpce_regression = cp.fit_regression(poly, samples, model_evaluations)
print("Success")
```

## Step 4: Pseudo-spectral projection

### Step 4.a: Quadrature nodes and weights

Once a random variable is defined or
joint random variables, also referred as distribution here, the following
method
can be used to generate nodes and weights for different quadrature methods:

```
# quadrature in polychaos
cp.generate_quadrature?
```

We will look at the following arguments of the method: **order** is the order of
the quadrature, **domain** is
the distribution one is working on, **rule** is the *name* or *key* of the
quadrature rule to apply.

In the following example we look at some quadrature
nodes for the same uniform variables as for the sampling,
for Optimal Gaussian
quadrature and Clenshaw-Curtis quadrature.

```
# example quadrature
u1 = cp.Uniform(0,1)
u2 = cp.Uniform(0,1)
joint_distribution = cp.J(u1, u2)

order = 5

nodes_gaussian, weights_gaussian = cp.generate_quadrature(order=order, domain=joint_distribution, rule='G')
nodes_clenshaw, weights_clenshaw = cp.generate_quadrature(order=order, domain=joint_distribution, rule='C')

print('Number of nodes gaussian quadrature: {}'.format(len(nodes_gaussian[0])))
print('Number of nodes clenshaw-curtis quadrature: {}'.format(len(nodes_clenshaw[1])))

fig1, ax1 = plt.subplots()
ax1.scatter(*nodes_gaussian, marker='o', color='b')
ax1.scatter(*nodes_clenshaw, marker= 'x', color='r')
ax1.set_xlabel("Uniform 1")
ax1.set_ylabel("Uniform 2")
ax1.axis('equal')
```

In the following all quadrature rules implemented in chaospy are highlighted:

 Collection of quadrature
rules	    Name  	 Key

   Optimal Gaussian quadrature
	    Gaussian     	    G

   Gauss-Legendre quadrature
   Legendre     	    E

   Clenshaw-Curtis quadrature         	    Clenshaw     	    C

   Leja quadrature                    	    Leja         	    J

   Hermite Genz-Keizter 16 rule       	    Genz         	    Z

   Gauss-Patterson quadrature rule    	    Patterson    	    P

It is also possible to use sparse grid quadrature. For this
purpose Clenshaw-Curtis method is advised since it is nested.

In the following
example we show sparse vs. normal quadrature nodes:

```
# example sparse grid quadrature
u1 = cp.Uniform(0,1)
u2 = cp.Uniform(0,1)
joint_distribution = cp.J(u1, u2)

order = 2
# sparse grid has exponential growth, thus a smaller order results in more points
nodes_clenshaw, weights_clenshaw = cp.generate_quadrature(order=order, domain=joint_distribution, rule='C')
nodes_clenshaw_sparse, weights_clenshaw_sparse = cp.generate_quadrature(order=order, domain=joint_distribution, rule='C', sparse=True)

print('Number of nodes normal clenshaw-curtis quadrature: {}'.format(len(nodes_clenshaw[0])))
print('Number of nodes clenshaw-curtis quadrature with sparse grid : {}'.format(len(nodes_clenshaw_sparse[0])))

fig1, ax1 = plt.subplots()
ax1.scatter(*nodes_clenshaw, marker= 'x', color='r')
ax1.scatter(*nodes_clenshaw_sparse, marker= 'o', color='b')
ax1.set_xlabel("Uniform 1")
ax1.set_ylabel("Uniform 2")
ax1.axis('equal')
```

### Step 4.c: Polynomial Chaos Expansion

After the model is evaluated for all
integration nodes, the polynomial chaos expansion can be generated with the
following method:

```
# spectral projection in chaospy
cp.fit_quadrature?
```

In the following snippet we show again a complete example for polynomial chaos expansion
using the pseudo spectral approach to calculate the
expansion coefficients.
The
model applied the same simple mathematical expression as before:

$$
\begin{equation}
 y(z_1, z_2) = z_1 + z_1
z_2
\label{eq:dummy_model_repeat} \tag{11}
\end{equation}
$$

The random variables for $Z_1, Z_2$ are defined as simple uniform random
variables:

$$
\begin{equation}
Z_1 = \mbox{U}(0,1), \quad Z_2 = \mbox{U}(0,1)
\label{eq:dummy_rv_repeat} \tag{12}
\end{equation}
$$

```
# example spectral projection
# 1. define marginal and joint distributions
u1 = cp.Uniform(0,1)
u2 = cp.Uniform(0,1)
joint_distribution = cp.J(u1, u2)

# 2. generate orthogonal polynomials
polynomial_order = 3
poly = cp.orth_ttr(polynomial_order, joint_distribution)

# 4.1 generate quadrature nodes and weights
order = 5
nodes, weights = cp.generate_quadrature(order=order, domain=joint_distribution, rule='G')

# 4.2 evaluate the simple model for all nodes
model_evaluations = nodes[0]+nodes[1]*nodes[0]

# 4.3 use quadrature to generate the polynomial chaos expansion
gpce_quadrature = cp.fit_quadrature(poly, nodes, weights, model_evaluations)
print("Success")
```

## Step 5: Statistical Analysis

Once the polynomial chaos expansion has been generated either with **pseudo-spectral projection** or with **regression** method,
the calculation of the statistical properties is straightforward.
The following listing gives an overview of all available
methods take all the same input parameter the **polynomial-expansion** and
the
**joint-distribution** (see also example below).

Note, that one can also
calculate uncertainty statistics on distributions only as well.

### Uncertainty quantification

* Expected value: `cp.E`

* Variance: `cp.Var`

* Standard deviation: `cp.Std`

* Curtosis: `cp.Kurt`

* Skewness: `cp.Skew`

* Distribution of Y: `cp.QoI_Dist`

* Prediction intervals: `cp.Perc`, which is a method to calculate percentiles: an additional argument defining the percentiles
needs to be passed.

If multiple quantities of interest are available:

* Covariance matrix: `cp.Cov`

* Correlation matrix: `cp.Corr`

* Spearman correlation: `cp.Spearman`

* Auto-correlation function: `cp.Acf`

```
# example uq
exp_reg = cp.E(gpce_regression, joint_distribution)
exp_ps =  cp.E(gpce_quadrature, joint_distribution)

std_reg = cp.Std(gpce_regression, joint_distribution)
str_ps = cp.Std(gpce_quadrature, joint_distribution)

prediction_interval_reg = cp.Perc(gpce_regression, [5, 95], joint_distribution)
prediction_interval_ps = cp.Perc(gpce_quadrature, [5, 95], joint_distribution)

print("Expected values   Standard deviation            90 % Prediction intervals\n")
print(' E_reg |  E_ps     std_reg |  std_ps                pred_reg |  pred_ps')
print('  {} | {}       {:>6.3f} | {:>6.3f}       {} | {}'.format(exp_reg,
                                                                  exp_ps,
                                                                  std_reg,
                                                                  str_ps,
                                                                  ["{:.3f}".format(p) for p in prediction_interval_reg],
                                                                  ["{:.3f}".format(p) for p in prediction_interval_ps]))
```

### Sensitivity analysis

The variance bases sensitivity indices can be
calculated directly from the expansion.
The `chaospy` package provides the
following methods:

* first order indices: `cp.Sens_m`

* second order indices: `cp.Sens_m2`

* total indices: `cp.Sens_t`

Here is an example for the first and
total indices for both expansions:

```
# example sens
sensFirst_reg = cp.Sens_m(gpce_regression, joint_distribution)
sensFirst_ps = cp.Sens_m(gpce_quadrature, joint_distribution)

sensT_reg = cp.Sens_t(gpce_regression, joint_distribution)
sensT_ps = cp.Sens_t(gpce_quadrature, joint_distribution)

print("First Order Indices           Total Sensitivity Indices\n")
print('       S_reg |  S_ps                 ST_reg |  ST_ps  \n')
for k, (s_reg, s_ps, st_reg, st_ps) in enumerate(zip(sensFirst_reg, sensFirst_ps, sensT_reg, sensT_ps)):
    print('S_{} : {:>6.3f} | {:>6.3f}         ST_{} : {:>6.3f} | {:>6.3f}'.format(k, s_reg, s_ps, k, st_reg, st_ps))
```


# DSCI 572 Lab 3

## Installations

You'll first need to install TensorFlow, which should be possible with

```
pip install tensorflow
```

Note that we will be using TensorFlow 2. If you had TensorFlow 1 installed previously, you may need to remove it or upgrade.

```python
import tensorflow as tf
tf.__version__ # should print 2.0.0
```

    '2.0.0'

```python
# keras imports
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation
from tensorflow.keras import utils
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
import tensorflow.keras.datasets
```

```python
# other imports
import numpy as np
import pandas as pd

from sklearn import datasets
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler

from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression

from sklearn.neural_network import MLPClassifier, MLPRegressor

import matplotlib.pyplot as plt
%matplotlib inline

import time
```

```python
plt.rcParams['font.size'] = 16
```

## Instructions
rubric={mechanics:10}

Follow the [general lab instructions](https://ubc-mds.github.io/resources_pages/general_lab_instructions/).

## Exercise 0: neural networks "by hand"

#### 0(a)
rubric={reasoning:10}

Suppose that we use a neural network with one hidden layer and ReLU activations for a regression problem. After training, we obtain the following parameters:

$$\begin{align}W^{(0)} &= \begin{bmatrix}-2 & 2 & -1\\-1 & -2 & 0\end{bmatrix},  &b^{(0)}&=\begin{bmatrix}2 \\ 0\end{bmatrix} \\ W^{(1)} &= \begin{bmatrix}3 & 1\end{bmatrix},  &b^{(1)}&=-10\end{align}$$

For a training example with features $x = \begin{bmatrix}3 \\-2 \\ 2\end{bmatrix}$ what are the values in this network of $x^{(1)}$ and $\hat{y}$? Show your work.

#### 0(b)
rubric={reasoning:10}

Draw this neural network using a circle/arrow diagram. Label the diagram with the weight/bias values given above.

If you want to draw this diagram by hand, that is fine: you can take a photo of the drawing and put it in here. If you are doing so, make sure you upload the image to your repo!

#### 0(c)
rubric={accuracy:3,quality:3}

For optimization purposes, the parameters above would typically be combined into a single parameter vector, for example:

$$w_\text{all} = \begin{bmatrix}W^{(0)}_\text{flattened}\\ b^{(0)} \\ W^{(1)}_\text{flattened} \\ b^{(1)}\end{bmatrix} = \begin{bmatrix}-2\\2\\-1\\-1\\-2\\0\\2\\0\\3\\1\\-10\end{bmatrix}$$

The reason for this is that optimization functions, like `scipy.optimize.minimize` or the `gradient_descent` and `stochastic_gradient_descent` functions you've written, generally all tend to assume that the input is a 1D numpy array. They would not work if you fed in, say, a list of lists of arrays.

Your task: write a python function `combine_params` that takes in a list of weights and a list of biases, and returns the flattened vector.

```python
W0 = np.array([[-2,2,-1],[-1,-2,0]])
b0 = np.array([2,0])
W1 = np.array([3,1])
b1 = np.array([-10])

all_params = [W0, b0, W1, b1]
all_params
```

```python
my_answer = combine_params(all_params)
desired_answer = np.array([-2, 2, -1, -1, -2, 0, 2, 0, 3, 1, -10])

assert np.array_equal(my_answer, desired_answer)
```

## Exercise 1: digits warm-up

The code below loads scikit-learn's built-in handwritten digits dataset and fits the following classifiers:

- KNN
- random forest
- RBF SVM
- logistic regression
- 1-hidden-layer neural network using sklearn

```python
digits = datasets.load_digits()
X, y = digits['data'], digits['target']
X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=1)

print(X_train.shape[0], 'train samples')
print(X_valid.shape[0], 'valid samples')
```

```python
scaler = StandardScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_valid = scaler.transform(X_valid)
```

```python
classifiers = {
    'knn'           : KNeighborsClassifier(),
    'random forest' : RandomForestClassifier(n_estimators=50),
    'SVM'           : SVC(C=100, gamma="scale"),
    'logistic reg'  : LogisticRegression(multi_class="multinomial", solver="lbfgs"),
    'sklearn NN'    : MLPClassifier()
}

train_scores = dict()
valid_scores = dict()
training_times = dict()

for classifier_name, classifier_obj in classifiers.items():
    print("Fitting", classifier_name)
    t = time.time()
    classifier_obj.fit(X_train, y_train)

    training_times[classifier_name] = time.time() - t
    train_scores[classifier_name] = classifier_obj.score(X_train, y_train)
    valid_scores[classifier_name] = classifier_obj.score(X_valid, y_valid)
```

```python
pd.options.display.float_format = '{:,.2f}'.format # make things look prettier when printing
data = {"train acc": train_scores, "valid acc" : valid_scores, "training time (s)" : training_times}
df = pd.DataFrame(data, columns=data.keys())
df.index = list(classifiers.keys())
df
```

As you can see, this dataset isn't very exciting in the sense that most of the methods get a high test accuracy after very little time. We'll use this dataset for a few more moments, and then move onto another one soon.

### 1(a)
rubric={accuracy:5}

Using Keras, create a neural network with the same architecture as the sklearn NN above. You should one-hot encode the labels using `tensorflow.keras.utils.to_categorical` (although this step can be skipped by using [KerasClasifier](https://www.tensorflow.org/api_docs/python/tf/keras/wrappers/scikit_learn/KerasClassifier), we'll avoid using that here). Briefly discuss your results.

### 1(b)
rubric={viz:5}

For the same network above, make the following two plots:

1. A plot of accuracy vs. optimization epochs. You should have two curves on this plot, one of train and one for test.
2. A plot of the loss vs. optimization epochs. You should have two curves on this plot, one of train and one for test.

Some notes:

- To get access to this information, you can use the history object that is returned by `fit`.
- If you're wondering what the difference is, accuracy is the percentage of examples that are correctles classified (always between $0$ and $1$), whereas the loss is literally the function being optimized (like `loss_lr` in lab 1, not necessarily between $0$ and $1$). The loss can't just be the number of incorrectly classifier examples, since this isn't a continuous function and would be too hard to optimize.

### 1(c)
rubric={reasoning:5,quality:5}

The optimization problem of training a neural network is non-convex. To explore this, try training
your network several times. You will get different results due to different random initializations and different randomness in the optimization method itself. Explore how the training/validation error changes across
runs.

### (optional) 1(d)
rubric={reasoning:1}

Continuing with the above, try also exploring how the model parameters themselves (the weights) change across runs (you'll need to think of a reasonable way of comparing the weights of two networks).
Do you observe substantially different weight sets all with a similar prediction accuracy... or something different?

Note: to inspect the weights themselves, use the `get_weights()` function for the Keras model/layer object. Also, you can use your `combine_params` function from Exercise 0 if you push.

```python
xx[0].shape
```

## Exercise 2: big digits

Next, we'll move on to the famous the MNIST digits -- a classic dataset for deep learning. The MNIST data set is  bigger than the digits dataset built into sklearn: the images are larger ($28\times28$ instead of $8\times8$) and there are more of them ($70000$ insetad of $1797$). In total, we're dealing with $70000\times28\times28\approx 55$ million training pixels instead of $1797\times8\times8\approx80000$ training pixels (about $500$ times more data).

If you're interested, check out [this site](http://yann.lecun.com/exdb/mnist/) showing the progress on this dataset from 1998 to 2012.

The following code loads the MNIST dataset. The first time you run it, the data will be downloaded. In future times, it will use the local version.

```python
# the data, shuffled and split between train and test sets
(X_train_img, y_train), (X_test_img, y_test) = tensorflow.keras.datasets.mnist.load_data()
```

```python
X_train_img.shape
```

    (60000, 28, 28)

```python
# display a random training example
i = np.random.randint(0,len(X_train_img))
plt.imshow(X_train_img[i],cmap='gray')
plt.title('This is a %d' % y_train[i]);
```

```python
X_train = X_train_img.reshape(60000, 28*28)
X_test = X_test_img.reshape(10000, 28*28)
X_train = X_train / 255 # this is the same a scaling, since the pixel intensities range from 0 to 255
X_test = X_test / 255

print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')
```

    60000 train samples
    10000 test samples

### 2(a)
rubric={reasoning:15}

1. Which of the classifiers from Exercise 1 scale well to this larger data set? You are free to make arguments using theory (big-O running times) and/or experiment (timed runs) as you see fit. Keep in mind that we've increased both $n$ (number of examples) and $d$ (number of features). Don't subject yourself to experiments that take tens of minutes or hours. It is fine to declare defeat after a couple of minutes and say that a method doesn't scale. But, when that happens, try to say a little bit about why that might be the case.
2. For those methods where the running time is reasonable (say, a couple minutes of computation), how does the accuracy compare between the methods?

### 2(b)
rubric={reasoning:15}

The code below runs a bigger Keras model on the full MNIST data set.

```python
batch_size = 128
num_classes = 10
epochs = 5

# convert class vectors to binary class matrices
Y_train = utils.to_categorical(y_train, num_classes)
Y_test = utils.to_categorical(y_test, num_classes)

model = Sequential()
model.add(Dense(256, input_shape=(X_train.shape[1],), activation='relu'))
model.add(Dense(256, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

start_time = time.time()

history = model.fit(X_train, Y_train,
                    batch_size=batch_size, epochs=epochs,
                    verbose=1, validation_data=(X_test, Y_test))

score = model.evaluate(X_train, Y_train, verbose=0)
print('Train accuracy:', score[1])

score = model.evaluate(X_test, Y_test, verbose=0)
print('Test accuracy:', score[1])

elapsed_time = time.time()-start_time
print("---Running Time: %s seconds ---" % elapsed_time)
```

For the neural network implemented with Keras above, explore the effects of the different hyperparameters on accuracy. Try at least 3 variations on what you're given above. In each case, briefly discuss your results. Some things you can consider trying:
  - adding/removing layer(s)
  - changing the [activation functions](https://keras.io/activations/) from `relu` to `tanh`
  - adding [regularization](https://keras.io/regularizers/) such as dropout
  - changing the way the weights are [initialized](https://keras.io/initializations/)
  - changing the [optimizer](https://keras.io/optimizers/) from adam to something else, like SGD. Read the documentation and try changing the _hyperparameters of the optimizer_ in addition to just the type of optimizer (e.g., $\alpha$ in gradient descent is a hyperparameter of the optimizer). It should not be difficult to completely mess up your training procedure, for example by tampering with the learning rate.

**NOTE: if at any point things are just way too slow, you can use a subset of the data to speed things up. But try to still draw interesting conclusions to the extent possible.**

## Exercise 3: pondering neural networks
rubric={writing:5}

The above writing marks are allocated to the overall writing quality of your answers below to the sub-parts of Exercise 3. For each question, no need to write more than 3 sentences. You may also include some code / code output in your answers.

#### 3(a)
rubric={reasoning:3}

Explain why a 1-layer (zero-hidden-layer) neural network with a linear activation is equivalent to linear regression. If you just wanted to do linear regression, what are the disadvantages of using Keras instead of a package like R's `lm`?

#### 3(b)
rubric={reasoning:3}

Consider a regression problem with $d=500$ features. You use a 2-hidden-layer neural network with hidden layer sizes $100$ and $200$. How many parameters does the model have? Justify your answer.

#### 3(c)
rubric={reasoning:3}

Hyperparameter optimization would be easy if we could independently tune each hyperparameter. Then, with $m$ hyperparameters, we'd just have $m$ one-dimensional optimization problems. However, in reality the hyperparameters interact, which leaves you with the more daunting $m$-dimensional optimization problem. Consider for example a regularization hyperparameter (like dropout) and an architecture parameter (like the number of units or layers). Why do you think these hyperparameters would "interact" with each other? For example, if the regularization strength was very large, would you expect bigger or smaller networks to perform well?

#### 3(d)
rubric={reasoning:3}

If you look at the [documentation for scikit-learn's neural network classifier](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html), you'll see that the user only needs to input the sizes of the hidden layers; like other scikit-learn classifiers, there's no need for the user to explicitly enter the number of features when creating the model object. However, Keras requires us to set the `input_dim` argument in the first `Dense` layer, which specifies the number of features. Why do you think Keras requires this whereas scikit-learn doesn't?

## (optional) Exercise 4(a)
rubric={reasoning:1}

It should be the case that logistic regression is equivalent to a neural network with no hidden layers.

Using sklearn's `MLPClassifier`, fit a zero-hidden-layer neural network on the synthetic 2D data (generated below) and compare the learned weights to what you get with `LogisticRegression`. Do you get essentially the same model with the neural network? You'll need to fiddle with the hyperparameters (optimization, regularization) to get something equivalent.

If you want to be very careful, see if you can get the coefficients to be exactly the same up to several decimal places (I managed to do this).  However, this might not be a good use of your time.

```python
n = 100
d = 2
np.random.seed(123)
X = np.random.rand(n,2)
y = np.random.rand(n) > 0.7
X[y==0,0] += .7
X[y==0,1] += .3
```

### (optional) Exercise 4(b)
rubric={reasoning:1}

Can you get this working with Keras as well? I tried for about 20 minutes and wasn't successful (speaking of questionable uses of time...), so if you figure it out please send me a message and I'll add your solution to the official solutions. Some reasons why this is annoying:

- One has to be very careful with the (L2) regularization. The interpretation of the regularization strength may vary across packages. It might be easier to turn off regularization for both.
- It's not clear whether to use a `softmax` activation + `categorical_crossentropy` loss + one-hot enoded labels or `sigmoid` activation + `binary_crossentropy` loss + regular labels. I have slightly more faith in the former approach. For this reason, it might be easier to first try and get this working with linear regression, and then move on to logistic.
- The SGD-based optimizers available in Keras aren't well suited to this kind of toy dataset.

## (optional) Exercise 5: backprop
rubric={reasoning:1}

From scratch using only raw numpy, implement a one-hidden-layer neural network for regression using ReLUs. Show your computation of the gradients alongside the code. You can ignore the fact that the ReLU function is not differentiable when its input equals zero.

Note: there are probably a lot of resources out there where people give their "raw" neural network implementations. If you're going to do this and you consult any sources, make sure you cite them.

Note: this is a lot more work than most optional questions.


# Ramsey-Cass-Koopmans model

Ramsey model from Chapter 2 of D. Romer's *Advanced Economics* looks as follows...

\begin{align}
    \dot{k}(t) =& k(t)^{\alpha} - c(t) - (g + n + \delta)k(t),\\
    \dot{c}(t) =& \Bigg[\frac{1}{\theta} \bigg[\alpha k(t)^{\alpha - 1} - \delta - \rho - \theta g\bigg]\Bigg]c(t)
\end{align}
...with boundary conditions...
\begin{align}
    k(0) =& k_0 \\
    lim_{t\rightarrow \infty} c(t) =& c^*
\end{align}

...where...

* Elasticity of output with respect to capital: $0 < \alpha < 1$
* Coefficient of relative risk aversion: $0 < \theta$
* Discount rate: $0 < \rho$
* Breakeven investment: $0 < g + n + \delta$
* Convergent lifetime utility: $0 < \rho - n - (1 - \theta)g$

Note that we are assuming two things...

1. Cobb-Douglas production
2. Constant Relative Risk Aversion (CRRA) preferences

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sn
import sympy as sym

import pycollocation
```

```python
pycollocation.__version__
```

    '0.3.0a0'

```python
pycollocation.
```

```python
def extensive_output(A, K, L, **params):
    """Output is a function of technology, A, capital, K, and labor, L."""
    alpha = params['alpha']
    return K**alpha * (A * L)**(1 - alpha)

def flow_utility(C, **params):
    """Flow utility from consumption (per capita)."""
    theta = params['theta']
    return (C**(1 - theta) - 1) / (1 - theta)

def steady_state_capital(A0, delta, g, K0, L0, n, rho, alpha, theta):
    """
    Equilibrium value of capital (per unit effective labor) with
    Cobb-Douglas production and Constant Relative Risk Aversion (CRRA)
    preferences.

    """
    return (alpha / (delta + rho + theta * g))**(1 / (1 - alpha))

```

```python
class RamseyModelLike(object):

    @property
    def flow_utility(self):
        return self._flow_utility

    @flow_utility.setter
    def flow_utility(self, u):
        self._flow_utility = u

    @property
    def extensive_output(self):
        return self._extensive_output

    @extensive_output.setter
    def extensive_output(self, F):
        self._extensive_output = F

    @staticmethod
    def marginal_product_capital(Y, K):
        """Marginal product of capital."""
        return sym.diff(Y, K, 1)

    @staticmethod
    def marginal_product_labor(Y, L):
        """Marginal product of labor."""
        return sym.diff(Y, L, 1)

    @staticmethod
    def break_even_investment(k, delta, g, n, **params):
        """
        Return level of investment (per unit effective labor) required
        to maintain current level of capital (per unit effective labor).

        """
        return (g + n + delta) * k

    def actual_investment(self, k, c, **params):
        """Return investment (per unit effective labor)."""
        return self.intensive_output(k, **params) - c

    def equation_motion_capital(self, t, k, c, A0, delta, g, n, rho, **params):
        """Equation of motion for capital (per unit effective labor)."""
        k_dot = (self.actual_investment(k, c, **params) -
                 self.break_even_investment(k, delta, g, n, **params))
        return k_dot

    def intensive_output(self, k, **params):
        """
        Intensive output is a function of capital (per unit effective labor)
        and model parameters.

        """
        return self.extensive_output(1, k, 1, **params)

class SolowModelLike(RamseyModelLike):

    def consumption(t, k, s, **params):
        """Household consumes a fixed fraction of output (per unit effective labor)."""
        return (1 - s) * self.intensive_output(k, **params)

    def equation_motion_capital(self, t, k, A0, delta, g, n, rho, s, **params):
        """Equation of motion for capital (per unit effective labor)."""
        c = self.consumption(t, k, s, **params)
        k_dot = super(SolowModelLike, self).equation_motion_capital(t, k, c, A0, delta, g, n, rho, **params)
        return k_dot

```

```python
model = RamseyModelLike()
```

```python
# set the user required model primitives
model.extensive_output = extensive_output
model.flow_utility = flow_utility
```

## Using pyCollocation

```python
def steady_state_capital_locus(self, k, delta, g, n, **params):
        """
        Locus of points (k, c) such that :math:`\dot{k}=0`.

        Function returns the level of consumption (per unit
        effective labor) required in order to maintain a given
        level of capital (per unit effective labor).

        """
        locus = (self.intensive_output(k, **params) -
                 self.break_even_investment(k, delta, g, n, **params))
        return locus

def steady_state_consumption(self, A0, delta, g, K0, L0, n, rho, **params):
        """Equilibrium value for consumption (per unit effective labor)."""
        k_star = steady_state_capital(A0, delta, g, K0, L0, n, rho, **params)
        c_star = self.steady_state_capital_locus(k_star, delta, g, n, **params)
        return c_star

def find_steady_state(self, ):
        """Compute the equilibrium values of capital stock and consumption (per unit effective labor)."""
        result = optimize.root()
        return result
```

```python

```

```python
from pycollocation import SymbolicTwoPointBVPLike

class RamseyProblem(RamseyModelLike, SymbolicTwoPointBVPLike):

    def __init__(self, F, u, params):
        self.extensive_output = F
        self.flow_utility = u
        self.params = params

    @property
    def _change_of_variables(self):
        """Substitutions for C and r."""
        return {'C': self._per_capita_consumption, 'r': self.real_interest_rate}

    @property
    def _growth_rate_c(self):
        """Growth rate of consumption (per unit effective labor)."""
        return self._growth_rate_C - self._symbolic_params['g']

    @property
    def _growth_rate_C(self):
        """Growth rate of consumption (per unit labor)."""
        return self.consumption_euler_equation.subs(self._change_of_variables)

    @property
    def _initial_condition_capital(self):
        """Intitial condition for capital stock (per unit effective labor)."""
        return self.params['K0'] / (self.params['A0'] * self.params['L0'])

    @property
    def _per_capita_consumption(self):
        """Symbolic expression for per capita consumption."""
        t, c = self._symbolic_vars[0::2]
        A0, g = self._symbolic_params['A0'], self._symbolic_params['g']
        return A0 * sym.exp(g * t) * c

    @property
    def boundary_conditions(self):
        """Boundary conditions for the model."""
        k0 = self._initial_condition_capital
        c_star = self.steady_state_consumption(**self.params)
        bcs = {'lower': [self._symbolic_vars[1] - k0],
               'upper': [self._symbolic_vars[2] - c_star]}
        return bcs

    @property
    def c_dot(self):
        """
        Symbolic expression defining the equation of motion for
        consumption (per unit effective labor).

        """
        return self._growth_rate_c * self._symbolic_vars[2]

    @property
    def consumption_euler_equation(self):
        """
        Representative household's consumption Euler equation.

        The Euler equation implies that the growth rate of per
        capita consumption is proportional to the difference
        between the real interest rate and the discount rate of
        the representative household.

        """
        epsilon = self.elasticity_intertemporal_substitution
        r = self.real_interest_rate
        consumption_growth = epsilon * (r - self._symbolic_params['rho'])
        return consumption_growth

    @property
    def dependent_vars(self):
        """Return a list of model dependent variables."""
        return ['k', 'c']

    @property
    def elasticity_intertemporal_substitution(self):
        """
        Elasticity of intertemporal substitution is reciprical of the
        coefficient of relative risk aversion.

        """
        return 1 / self.relative_risk_aversion_coef

    @property
    def independent_var(self):
        """Return the model independent variable as a string."""
        return 't'

    @property
    def k_dot(self):
        """Symbolic expression defining the equation of motion for capital stock (per unit effective labor)."""
        return self.equation_motion_capital(*self._symbolic_vars, **self._symbolic_params)

    @property
    def real_interest_rate(self):
        """Return the real interest rate (net depreciation)."""
        Y = self.extensive_output(1, self._symbolic_vars[1], 1, **self._symbolic_params)
        mpk = self.marginal_product_capital(Y, self._symbolic_vars[1])
        return mpk - self._symbolic_params['delta']

    @property
    def relative_risk_aversion_coef(self):
        """Return the coefficient of relative risk aversion."""
        C = sym.symbols("C")
        u = self.flow_utility(C, **self._symbolic_params)
        crra = -((C * sym.diff(u, C, 2)) / sym.diff(u, C, 1))
        return crra

    @property
    def rhs(self):
        """Symbolic expressions for the RHS of the system of ODE."""
        return {'k': self.k_dot, 'c': self.c_dot}

```

```python

```

```python
params = {'A0': 1.0, 'K0': 1.0, 'L0': 1.0, 'alpha': 0.33, 'delta': 0.04, 'theta': 1.05, 'rho': 0.05,
          'g': 0.02, 'n': 0.02}
```

```python
problem = RamseyProblem(extensive_output, flow_utility, params)
```

```python
problem.equilibrium.find_equilibrium(np.array([1.0, 1.0]))
```

      status: 1
     success: True
         qtf: array([ -1.38042791e-11,   7.55008202e-12])
        nfev: 18
           r: array([-0.04309798,  1.12955614,  0.41948545])
         fun: array([ -2.90878432e-14,   6.40918761e-14])
           x: array([ 5.08458095,  1.30350166])
     message: 'The solution converged.'
        fjac: array([[-0.80048697,  0.59935015],
           [-0.59935015, -0.80048697]])

```python
problem.params['theta'] = 2.0
```

```python
problem.equilibrium.find_equilibrium(np.array([1.0, 1.0]))
```

      status: 1
     success: True
         qtf: array([ -7.46717468e-11,   3.60795540e-11])
        nfev: 17
           r: array([-0.05987336,  1.16740084,  0.22841909])
         fun: array([ -1.34781075e-13,   1.34857166e-13])
           x: array([ 4.01639759,  1.26090543])
     message: 'The solution converged.'
        fjac: array([[-0.94214291,  0.33521147],
           [-0.33521147, -0.94214291]])

```python
problem.equation_motion_capital(0.0, 1.0, 1.0, **problem.params)
```

    -0.08

```python
solver = pycollocation.OrthogonalPolynomialSolver(problem)
```

```python
solver._basis_function_factory?
```

```python
class RamseySolverLike(object):

    def initial_polynomial(ts, ks, **kwargs):
        """Eventually need some to link to underlying basis functions!"""
        return self._basis_function_factory(ts, ks, **kwargs)

    def initial_coefs(ts, ks, cs,**kwargs):
        """Return dictionary of coefs."""
        capital = initial_polynomial(ts, ks, deg, domain)
        consumption = initial_polynomial(ts, cs, deg, domain)
        coefs = {'k': capital.coef, 'c': consumption.coef}
        return coefs
```

```python
# define our domain of approximation
k0 = problem._initial_condition_capital
domain = [0.0, 100.0]

# initial guess for capital
N = 1000
ts = np.linspace(domain[0], domain[1], N)
ks = steady_state_capital(**params) - (steady_state_capital(**params) - k0) * np.exp(-ts)
cs = np.log(ks)

def initial_polynomial(ts, ks, deg, domain):
    """Eventually need some to link to underlying basis functions!"""
    return np.polynomial.Chebyshev.fit(ts, ks, deg, domain)

def initial_coefs(ts, ks, cs, deg, domain):
    """Return dictionary of coefs."""
    capital = initial_polynomial(ts, ks, deg, domain)
    consumption = initial_polynomial(ts, cs, deg, domain)
    coefs = {'k': capital.coef, 'c': consumption.coef}
    return coefs
```

```python
coefs_guess = initial_coefs(ts, ks, cs, 35, domain)
solver.solve(kind="Chebyshev",
             coefs_dict=coefs_guess,
             domain=domain)
```

```python
solver.result.x
```

    array([  4.37225405e+00,   1.29407760e+00,  -9.72862387e-01,
             6.08921292e-01,  -3.18201558e-01,   1.37285311e-01,
            -4.62850602e-02,   9.22882158e-03,   2.25265029e-03,
            -4.07400457e-03,   3.21211999e-03,  -2.07493833e-03,
             1.24105453e-03,  -7.22117265e-04,   4.18592259e-04,
            -2.44435831e-04,   1.44396565e-04,  -8.63551433e-05,
             5.22449113e-05,  -3.19390959e-05,   1.97071578e-05,
            -1.22605018e-05,   7.68428916e-06,  -4.84844392e-06,
             3.07792569e-06,  -1.96514557e-06,   1.26163055e-06,
            -8.14606070e-07,   5.29424467e-07,  -3.47071441e-07,
             2.30580740e-07,  -1.56776432e-07,   1.10978890e-07,
            -8.39515497e-08,   4.59631210e-08,  -9.06485050e-09,
             1.17873943e+00,   2.28498625e-01,  -1.76366595e-01,
             1.16303877e-01,  -6.68076608e-02,   3.42707361e-02,
            -1.62089828e-02,   7.36702585e-03,  -3.37782840e-03,
             1.63268450e-03,  -8.50106910e-04,   4.74261607e-04,
            -2.78167064e-04,   1.68519209e-04,  -1.04250866e-04,
             6.54391427e-05,  -4.15333271e-05,   2.65970433e-05,
            -1.71602060e-05,   1.11430936e-05,  -7.27662551e-06,
             4.77545510e-06,  -3.14799526e-06,   2.08356680e-06,
            -1.38419780e-06,   9.22827089e-07,  -6.17389026e-07,
             4.14583842e-07,  -2.79625272e-07,   1.89715826e-07,
            -1.29871121e-07,   9.02377686e-08,  -6.43792306e-08,
             4.78481011e-08,  -3.82330997e-08,   1.72751855e-08])

```python
class RamseyVisualizer(pycollocation.Visualizer):

    def steady_state_capital_locus(self, k, delta, g, n, **params):
        """
        Locus of points (k, c) such that :math:`\dot{k}=0`.

        Function returns the level of consumption (per unit
        effective labor) required in order to maintain a given
        level of capital (per unit effective labor).

        """
        locus = (self.solver.model.intensive_output(k, **params) -
                 self.solver.model.break_even_investment(k, delta, g, n, **params))
        return locus
```

```python
# now create a visualizer for plot the solution...
visualizer = pycollocation.Visualizer(solver)
```

```python
# plot the solution
visualizer.interpolation_knots = np.linspace(domain[0], domain[1], N)
visualizer.solution.plot()
plt.show()
```

```python
# plot the normalized residuals (they should be small everywhere!)
visualizer.normalized_residuals.plot(logy=True)
plt.show()
```

```python

```

```python

```


# Lambda School Data Science - A First Look at Data

## Lecture - let's explore Python DS libraries and examples!

The Python Data Science ecosystem is huge. You've seen some of the big pieces - pandas, scikit-learn, matplotlib. What parts do you want to see more of?

```
# TODO - we'll be doing this live, taking requests
# and reproducing what it is to look up and learn things
1 + 1
```

    2

## Assignment - now it's your turn

Pick at least one Python DS library, and using documentation/examples reproduce in this notebook something cool. It's OK if you don't fully understand it or get it 100% working, but do put in effort and look things up.

```
widget_info = {
    'fixed costs': 10000000,
    'variable costs': 20000,
    'price point': 50000
}
```

```
from sympy import Symbol, solve
widgets = Symbol('x')
y = Symbol('y')

revenue = widget_info['price point'] * widgets - y
expences = widget_info['variable costs'] * widgets + widget_info['fixed costs'] - y

```

```
def solve_for_widgets(func_1, func_2):
  solution = solve((func_1, func_2))
  return float(solution[widgets])

def solve_for_revenue(func_1, func_2):
  solution = solve((func_1, func_2))
  return float(solution[y])

number_of_widgets = solve_for_widgets(revenue, expences)
total_revenue = solve_for_revenue(revenue, expences)
```

```
from numpy import arange
import matplotlib.pyplot as plt

x = arange(0, (number_of_widgets * 100))

revenue_func = widget_info['price point'] * x
expence_func = widget_info['variable costs'] * x + widget_info['fixed costs']

plt.plot(x, revenue_func, color='black')
plt.plot(x, expence_func, color='r')
plt.xlim(0, (number_of_widgets * 1.8))
plt.ylim(0, (total_revenue * 1.8))
plt.plot(number_of_widgets, total_revenue, marker='o', markersize='10', color='g')
plt.title('Break Even Analysis')
plt.xlabel('Widgets')
plt.ylabel('Dollars')
print('Break Even Point:\n', number_of_widgets, 'widgets')
plt.show();
```

### Assignment questions

After you've worked on some code, answer the following questions in this text block:

1.  Describe in a paragraph of text what you did and why, as if you were writing an email to somebody interested but nontechnical.

2.  What was the most challenging part of what you did?

3.  What was the most interesting thing you learned?

4.  What area would you like to explore with more time?

I wanted to build a visual representation of a cost volume analysis that would allow a user to see how diffrent costs affect a widgets profitability.

The user would input the fixed costs, variable costs, and price point of a given widget into the dictionary data structure in the first code block.The program would then output the break even point expressed as a number of widgets and graphically.

The most challenging part of this assignment was actualy fairly trival in retrospect. In order to correctly generalize the code so that it would work for any number of cost combinations the x and y limits of the graph must scale with the data. Tuning this parameter was the most difficult part as matplotlibs autoscale function didn't work quite as I had hoped.

The most interesting thig I learned during this assignment is the utility of Sympy in creating graphical representations in Matplotlib.

If I had more time I would like to explore how economies of scale would affect the cost structure of the widget.

I have used the sympy library for these calculations along with numpy and matplotlib for the graphical representation.

[Sympy](https://www.sympy.org/en/index.html) is amazing for solving math problems in a symbolic fashion.

## Stretch goals and resources

Following are *optional* things for you to take a look at. Focus on the above assignment first, and make sure to commit and push your changes to GitHub (and since this is the first assignment of the sprint, open a PR as well).

- [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/)
- [scikit-learn documentation](http://scikit-learn.org/stable/documentation.html)
- [matplotlib documentation](https://matplotlib.org/contents.html)
- [Awesome Data Science](https://github.com/bulutyazilim/awesome-datascience) - a list of many types of DS resources

Stretch goals:

- Find and read blogs, walkthroughs, and other examples of people working through cool things with data science - and share with your classmates!
- Write a blog post (Medium is a popular place to publish) introducing yourself as somebody learning data science, and talking about what you've learned already and what you're excited to learn more about.


```python
%matplotlib inline
from sympy import *
init_printing(use_unicode=True)
```

```python
r, u, v, c, r_c, u_c, v_c, E, p, r_p, u_p, v_p, e, a, b, q, b_0, b_1, b_2, b_3, q_0, q_1, q_2, q_3, q_4, q_5, t, g, c_0, c_1, c_2, c_3, c_4, c_5 = symbols('r u v c r_c u_c v_c E p r_p u_p v_p e a b q b_0 b_1 b_2 b_3 q_0 q_1 q_2 q_3 q_4 q_5 t g c_0 c_1 c_2 c_3 c_4 c_5',positive=True)
```

```python
cprm, thetaprm, Pprm, betaprm, rtheta, gammaprm, rhoprm, Eprm, sigmaprm = symbols('cprm thetaprm Pprm betaprm r_theta gammaprm rhoprm Eprm sigmaprm',positive=True)
```

```python
gamma,r_p,u_p,e = symbols('gamma r_p u_p e',positive=True)
```

####$f_{2}(c,p) = \dfrac{1}{2}r_{c}c^{2}+\dfrac{1}{4}u_{c}c^{4}+\dfrac{1}{6}v_{c}c^{6}+\dfrac{1}{2}r_{p}p^{2}+\dfrac{1}{4}u_{p}p^{4}-\gamma cp-\dfrac{1}{2}ec^{2}p^{2}-Ep$

```python
f2 = ((1/2)*r_c*c**2+(1/4)*u_c*c**4+(1/6)*v_c*c**6+(1/2)*r_p*p**2+(1/4)*u_p*p**4-E*p-gamma*c*p-e*c**2*p**2/2)
nsimplify(f2)
```

###Rescaling

```python
P, C, w, rho, beta, tau, Epr = symbols('P C w rho beta tau E^{\prime}')
```

```python
fP = nsimplify(f2.subs(p,w*Pprm))
fP
```

```python
fP = nsimplify(fP.subs(w,sqrt(r_p/u_p)))
fP
```

```python
fP = expand(fP/(r_p**2/u_p),r_p)
fP
```

```python
fP = fP.subs([(E,Epr/sqrt(u_p)*r_p**(3/2)),(c,cprm*sqrt(r_p/e))])
fP
```

```python
fP = fP.subs([(cprm,thetaprm),(r_c,rtheta*e*r_p/u_p),(u_c,gammaprm*e**2/u_p),(v_c,rhoprm*e**3/(u_p*r_p)),(gamma,betaprm*r_p*sqrt(e/u_p))])
fP
```

```python
fhelp = nsimplify((1/2)*rtheta*thetaprm**2+(1/4)*gammaprm*thetaprm**4+(1/6)*rhoprm*thetaprm**6+(1/2)*Pprm**2+(1/4)*Pprm**4
        +(1/6)*sigmaprm*Pprm**6-betaprm*thetaprm*Pprm-(1/2)*Pprm**2*thetaprm**2-Eprm*Pprm)
fhelp
```

```python
Ep = solve(fhelp.diff(Pprm),Eprm)[0]
Ep
```

```python
fhelp.diff(thetaprm)
```

```python
P_min = solve(fhelp.diff(thetaprm),Pprm)[0]
P_min
```

```python
Ep = Ep.subs(Pprm,P_min)
Ep
```

```python
series(Ep,thetaprm,n=7)
```

```python
# Series expansion of $E$ out to $\mathcal{O(\theta^{\prime 7})}$:

# $\beta E = \theta^{\prime}\left(-\beta^{\prime 2} + r_{\theta}^{\prime}\right) + \theta^{\prime 3}\left(\gamma^{\prime} - r_{\theta}^{\prime} + \dfrac{r_{\theta}^{\prime 3}}{\beta^{\prime 2}} - \dfrac{r_{\theta}^{\prime 2}}{\beta^{\prime 2}}\right) + \theta^{\prime 5}\left(-\gamma^{\prime} + \rho^{\prime} + \dfrac{3\gamma^{\prime}r_{\theta}^{\prime 2}}{\beta^{\prime 2}} - \dfrac{2\gamma^{\prime}r_{\theta}}{\beta^{\prime 2}} + \dfrac{r_{\theta}^{\prime 2}}{\beta^{\prime 2}} - \dfrac{3r_{\theta}^{\prime 4}}{\beta^{\prime 4}} + \dfrac{2r_{\theta}^{\prime 3}}{\beta^{\prime 4}}\right)$

# The coefficients of $E$ i.t.o. $a$ where $r_{\theta}^{\prime} = \beta^{\prime 2} + a$:

# $B(a) = \dfrac{a^{3}}{\beta^{\prime 2}} + a^{2}\left(3 - \dfrac{1}{\beta^{\prime 2}}\right) + a(3\beta^{\prime 2} - 3) + \beta^{\prime 4} - 2\beta^{\prime 2} + \gamma^{\prime}$

# $C(a) = -\dfrac{3a^{4}}{\beta^{\prime 4}} + a^{3}\left(\dfrac{2}{\beta^{\prime 4}} - \dfrac{12}{\beta^{\prime 2}}\right) + a^{2}\left(\dfrac{3\gamma^{\prime}}{\beta^{\prime 2}} - 18 + \dfrac{7}{\beta^{\prime 2}}\right) + a\left(6\gamma^{\prime} - 12\beta^{\prime 2} - \dfrac{2\gamma^{\prime}}{\beta^{\prime 2}} + 8\right) - 3\beta^{\prime 4} + 3\beta^{\prime 2}\gamma^{\prime} + 3\beta^{\prime 2} - 3\gamma^{\prime} + \rho^{\prime}$

# $R(a) = B(a)^{2} - \dfrac{20aC(a)}{9}$ (not going to write this one out, it's long)
```

```python
rth = betaprm**2+a*betaprm

B = (gammaprm-rtheta+(rtheta**3-rtheta**2)/betaprm**2)/betaprm

C = (-gammaprm+rhoprm+3*gammaprm*(rtheta/betaprm)**2-2*gammaprm*rtheta/betaprm**2
     +(rtheta/betaprm)**2+rtheta**5*sigmaprm/betaprm**4-3*(rtheta/betaprm)**4+2*rtheta**3/betaprm**4)/betaprm
```

```python
B = collect(expand(B.subs(rtheta,rth)),a)
B
```

```python
C = collect(expand(C.subs(rtheta,rth)),a)
C
```

```python
b0 = betaprm**3-2*betaprm+gammaprm/betaprm
b1 = 3*betaprm**2-3
b2 = 3*betaprm-1/betaprm
b3 = 1

c0 = betaprm**5*sigmaprm-3*betaprm**3+3*betaprm*gammaprm+3*betaprm-3*gammaprm/betaprm+rhoprm/betaprm
c1 = 5*betaprm**4*sigmaprm+6*gammaprm-12*betaprm**2+8-2*gammaprm/betaprm**2
c2 = 10*betaprm**3*sigmaprm+(3*gammaprm+7)/betaprm-18*betaprm
c3 = 10*betaprm**2*sigmaprm-12+2/betaprm**2
c4 = 5*betaprm*sigmaprm-3/betaprm
c5 = sigmaprm
```

```python
gammap = expand(solve(b0-b_0,gammaprm)[0])
sigmap = expand(solve(c0-c_0,sigmaprm)[0])
```

```python
replacements = [(gammaprm,gammap),(sigmaprm,sigmap)]
```

```python
sigmap = simplify(sigmap.subs(gammaprm,gammap))
```

```python
sigmap
```

```python
gammap
```

```python
b0
```

```python
b1
```

```python
b2
```

```python
b3
```

```python
B_a = b3*a**3+b2*a**2+b1*a+b_0
B_a
```

```python
# c0 = c0.subs(sigmaprm,sigmap).subs(gammaprm,gammap)
c0
```

```python
c1 = c1.subs(sigmaprm,sigmap).subs(gammaprm,gammap)
expand(c1)
```

```python
c2 = c2.subs(sigmaprm,sigmap).subs(gammaprm,gammap)
expand(c2)
```

```python
c3 = c3.subs(sigmaprm,sigmap).subs(gammaprm,gammap)
expand(c3)
```

```python
c4 = c4.subs(sigmaprm,sigmap).subs(gammaprm,gammap)
expand(c4)
```

```python
c5 = c5.subs(sigmaprm,sigmap).subs(gammaprm,gammap)
c5
```

```python
C_a = c_5*a**5+c4*a**4+c3*a**3+c2*a**2+c1*a+c_0
collect(expand(C_a),a)
```

```python
series(B_a**2-20*a*C_a/9,a,n=7)
```

```python
series(B**2-20*a*C/9,a,n=7)
```

```python
Etrun = a*thetaprm+b*thetaprm**3+c*thetaprm**5
Etrun
```

```python
collect(Etrun.subs([(b,B),(c,C)]),thetaprm)
```

```python
theta_L = solve(Etrun.diff(thetaprm),thetaprm)[1]
theta_U = solve(Etrun.diff(thetaprm),thetaprm)[3]
theta_L,theta_U
```

```python
E_L = simplify(Etrun.subs(thetaprm,theta_U))
E_U = simplify(Etrun.subs(thetaprm,theta_L))
E_L,E_U
```

```python

```


# University of Applied Sciences Munich
## Kalman Filter Tutorial

---
(c) Lukas Köstler (lkskstlr@gmail.com)

```python
import ipywidgets as widgets
from ipywidgets import interact_manual
from IPython.display import display

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (6, 3)

import numpy as np
%matplotlib notebook
```

```python
def normal_pdf(x, mu=0.0, sigma=1.0):
    return 1.0 / np.sqrt(2*np.pi*sigma**2) * np.exp(-0.5/sigma**2 * (x-mu)**2)
```

#### Possible sources:
+ (One of the most prominent books on robotics) https://docs.ufpr.br/~danielsantos/ProbabilisticRobotics.pdf
+ (Many nice pictures) http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/
+ (Stanford lecture slides) https://stanford.edu/class/ee363/lectures/kf.pdf

## Kalman Filter in Multiple Dimensions
---
* The concept is exactly the same, just the math looks more difficult.
* Just use the equations given in a book/paper/wikipedia

### Transition Model
---
\begin{align}
\mathbf{x}_{n+1} = A \mathbf{x}_n + B \mathbf{u}_{n+1} + error
\end{align}

with $\mathbf{x} \in \mathcal{R}^{d_x}, \mathbf{u} \in \mathcal{R}^{d_u}, A \in \mathcal{R^{d_x,d_x}}, B \in \mathcal{R^{d_x,d_u}}$. And $error \sim N \left( 0, R \right)$ and $R \in \mathcal{R^{d_x,d_x}}$

### Measurement Model
---
\begin{align}
\mathbf{z}_{n} = C \mathbf{x}_n + error
\end{align}

with $\mathbf{z} \in \mathcal{R}^{d_z}, C \in \mathcal{R^{d_z,d_x}}$. And $error \sim N \left( 0, Q \right)$ and $Q \in \mathcal{R^{d_z,d_z}}$

### Algorithm (1/2)
---
\begin{align}
\text{mean:} && \mathbf{\mu}_n \\
\text{covariance:} && \Sigma_n \\
\text{pred. mean:} && \mathbf{\mu}_{n+1 \vert n} \\
\text{pred. cov:} && \Sigma_{n+1 \vert n}\\
\text{kalman gain:} && K_n
\end{align}

with $\mathbf{\mu} \in \mathcal{R}^{d_x}, \Sigma \in \mathcal{R}^{d_x, d_x}, K \in \mathcal{R}^{d_x, d_z}$

### Algorithm (2/2)
---
\begin{align}
\mathbf{\mu}_{n+1 \vert n} &= A \mathbf{\mu}_n + B \mathbf{u}_{n+1} \\
\Sigma_{n+1 \vert n} &= A \Sigma_n A^T + R \\
K_{n+1} &= \Sigma_{n+1 \vert n} C^T \left(C \Sigma_{n+1 \vert n} C^T + Q \right)^{-1} \\
\mathbf{\mu}_{n+1 \vert n+1} &= \mathbf{\mu}_{n+1 \vert n} + K_{n+1} \left( z_{n+1} - C \mathbf{\mu}_{n+1 \vert n}\right) \\
\Sigma_{n+1} &= \left(I - K_{n+1} C \right) \Sigma_{n+1 \vert n}
\end{align}

## Let's try this:
---
* Example: robot localization on a 2D map in mind
 \begin{equation}
 \mathbf{x}_n = \begin{bmatrix}
 x \\
 y \\
 \varphi
 \end{bmatrix}
 \end{equation}
 Note: $\mathbf{x}_n$ is the state and $x$ is the vehicles position in the plane. Sorry for the confusing nomenclature.
* The control at time $t_n$ is denoted $\mathbf{u}_n$. This could be
 \begin{equation}
 \mathbf{u}_n = \begin{bmatrix}
 v \\
 \omega \\
 \end{bmatrix}
 \end{equation}
 with $v$ the velocity of the car and $\omega$ the angular velocity of the car (or more precisely of the velocity vector).

### Transition Model
---
* Remember
$$ \mathbf{x}_{n+1} = A \mathbf{x}_n + B \mathbf{u}_{n+1} $$

* Simplest possible form of motion (by far not the best. Use the models in Thrun et. al: Probabilistic Robotics):
\begin{align}
\mathbf{x}_{n+1} = \begin{bmatrix}
 x_{n+1} \\
 y_{n+1} \\
 \varphi_{n+1}
 \end{bmatrix}
 =
 \mathbf{x}_n +
 \begin{bmatrix}
 v \Delta t \, cos(\varphi_n) \\
 v \Delta t \, sin(\varphi_n) \\
 \Delta t \, \omega
 \end{bmatrix}
\end{align}

* This is not linear :(

## To the rescue: Extended Kalman Filter
---
* What we want:
$$ \mathbf{x}_{n+1} = A \mathbf{x}_n + B \mathbf{u}_n $$

* What we have:
\begin{equation}
\mathbf{x}_{n+1} = \underbrace{f\left(\mathbf{x}_{n}, \mathbf{u}_{n}\right)}_{\text{non-linear function}}
\end{equation}

* Remedy: Taylor Expansion

## Taylor Expansion
---
* A function can be approximated by it's derivatives (actually much more theory but we do not care)
* So:
\begin{align}
f(x) &= f(x_0) + \frac{df}{dx}(x_0) (x-x_0) + O(x^2) \\[1.3ex]
     &= f(x_0) + f'(x_0) (x-x_0) + \dots
\end{align}

* Example: $f(x) = sin(x) \rightarrow f'(x) = cos(x)$, $x_0 = 0$:
\begin{align}
f(x) &= f(x_0) + f'(x_0) (x-x_0) + O(x^2) \\[1.3ex]
     &= 0 + 1 (x-0) + \dots \\[1.3ex]
     &= x
\end{align}

```python
%%capture
xx = np.linspace(-2,2,1000)
fig = plt.figure()
plt.plot(xx, np.sin(xx), label="$f(x) = sin(x)$")
plt.plot(xx, xx, '--', label="$f(x) = x$")
plt.legend()
plt.title("First Order Taylor Expansion of $sin(x)$")
```

```python
display(fig)
```

    <IPython.core.display.Javascript object>

## Taylor Expansion
---
* This also works in higher dimensions. The derivative is replaced with the jacobi matrix:
\begin{align}
f(\mathbf{x}) &= f(\mathbf{x}_0) + A (\mathbf{x}-\mathbf{x}_0) + O(\vert \vert x \vert \vert ^2) \\[1.3ex]
A &= \frac{d}{d \mathbf{x}} f (\mathbf{x}_0)\\[1.3ex]
\phantom{1}
\end{align}

* And this gives us the linear equation we need!

## Extended Kalman Filter
---
* In each step linearize around the current mean and use the linear Kalman equations explained before
* Let's do an example!

## Extended Kalman Filter
---
* In each step linearize around the current mean and use the linear Kalman equations explained before
* Let's do an example!

## EKF Localization
---

from Probabilistic Robotics chapter 7.5
* State:
\begin{equation}
 \mathbf{x}_n = \begin{bmatrix}
 x \\
 y \\
 \theta
 \end{bmatrix}
 \end{equation}
 $x, y$ positions on the map. $\theta$ angle between the forward direction and the global $x$ axis.

* Control:
\begin{equation}
 \mathbf{u}_n = \begin{bmatrix}
 v \\
 \omega
 \end{bmatrix}
 \end{equation}
 $v$ is the velocity in forward direction. $\omega$ is the rotational velocity of the forward direction.

* Perfect Motion Model (I will do a lengthy derivation on the black board):
\begin{equation}
 \mathbf{x'} = \begin{bmatrix}
 x' \\
 y' \\
 \theta'
 \end{bmatrix}
 =
 \begin{bmatrix}
 x \\
 y \\
 \theta
 \end{bmatrix}
 +
 \begin{bmatrix}
 r \left( - sin \theta + sin \left(\theta + \omega \Delta t\right)\right) \\
 r \left( + cos \theta - cos \left(\theta + \omega \Delta t\right)\right) \\
 \omega \Delta t
 \end{bmatrix}
 =:
 g \left(\mathbf{x}, \mathbf{u}\right)
 \end{equation}

* $r$ is the radius of the circle on which the robot moves with velocity $v$. Out of the triple $v, r, \omega$ two can be choosen and the last one obtained by calculation, e.g. $r = v / \omega$

* Let's linearize this for the EKF: $g \left(\mathbf{x}_n, \mathbf{u}_{n+1}\right) \approx g \left(\mathbf{\mu}_n, \mathbf{u}_{n+1}\right) + A \left( \mathbf{x}_n - \mathbf{\mu}_n \right)$

\begin{equation}
 A = \begin{bmatrix}
 1 & 0 & r \,cos \left(\theta_n\right) - r \,cos \left(\theta_n + \omega_{n+1} \Delta t\right) \\
 0 & 1 & r \,sin \left(\theta_n\right) - r \,sin \left(\theta_n + \omega_{n+1} \Delta t\right) \\
 0 & 0 & 1
 \end{bmatrix}
 \end{equation}

* For $g \left(\mathbf{\mu}_n, \mathbf{u}_{n+1}\right)$ just insert into the equations from above.

* Measurement (Again: derivations will be done on the blackboard):
* The resulting algorithm is given in the following (table taken from Probabilistic Robotics by Thrun et al)

### Where to go from here? (1/2)
* Also estimate the positions of the landmarks -> SLAM
* Fast SLAM: [link](http://robots.stanford.edu/papers/Montemerlo03a.html)
* Many good algorithms at openslam: [link](https://openslam.org)
* MRPT seems to offer a ros package which fits exactly (2D Landmarks with range and bearing): [link](http://wiki.ros.org/mrpt_slam)
* Might be difficult to incorporate own motion model. Might also not be necessary.
* Visual Methods:
 * ORB SLAM (ETH): [link](https://github.com/raulmur/ORB_SLAM2)
 * LSD SLAM (TUM): [link](https://github.com/tum-vision/lsd_slam)
 * Calibrate your camera: [link](https://github.com/ethz-asl/kalibr)
 * Might need some thinking in how to integrate the cones; Maybe use SLAM for localization and run own algorithm for cones on top (state is only the cone locations)
* Also global optimization (in time), sometimes called bundle adjustement, might be nice. [g2o](http://wiki.ros.org/g2o) is a nice package for this

### Where to go from here? (2/2)
* The Unscented Kalman Filter does not need derivatives and is thus ways easier to use
* Particle Filters do also not use derivatives but they are much more expensive
* Grid based filters can be very fast on GPUs (and CPUs) and offer a good solution to the localization problem. In my opinion cones on a plane don't really need a grid