# Python Lecture 3: On Lists, Iteration, and Modules

### Lecture Notes by Jakov Ivan S. Dumbrique (jdumbrique@ateneo.edu)

MATH 100.2: Topics in Financial Mathematics II \
First Semester, S.Y. 2021-2022 \
Ateneo de Manila University

Today, we will learn three topics in Python:

0. Lists
1. Iteration
2. Modules

## Section 0: Lists
A list is the simplest data structure. It is a finite, mutable sequence of elements, where each element is accessible by an index.

```python
# Lists store sequences
# a list is denoted by square brackets
empty_li = []
# You can start with a prefilled list
# commas separate the elements of a list
li = [1, 2, 4, 3]
other_li = [4, 5, 6]

# lists can contain elements of different types
one_li = [1, 1.0, True, 'one']
```

```python
li
```

    [1, 2, 4, 3]

```python
# Access an element of a list using its index.
# Unlike in R, Python uses zero-based indexing.
li[0]   # => 1
# Look at the last element
li[-1]  # => 3

# Looking out of bounds is an IndexError
# li[4]  # Raises an IndexError
```

    3

```python
li
```

    [1, 2, 4, 3]

```python
# Indexing and slicing
# The start index is included, the end index is not
# (It's a closed/open range for you mathy types.)
li[1:3]   # Return list from index 1 to 3 => [2, 4]
li[2:]    # Return list starting from index 2 => [4, 3]
li[:3]    # Return list from beginning until index 3  => [1, 2, 4]
li[::2]   # Return list selecting every second entry => [1, 4]
li[::-1]  # Return list in reverse order => [3, 4, 2, 1]
# Use any combination of these to make advanced slices
# li[start:end:step]
```

    [3, 4, 2, 1]

```python
# Add an element to the end of a list with append
# li.append(1), not append(li,1)
# all Python objects have data, as well as (built-in) methods
# this mutates the li object in place
li.append(2)    # li is now [1, 2, 4, 3, 2]

# Concatenate lists with "extend()"
li.extend(other_li)  # Now li is [1, 2, 4, 3, 2, 4, 5, 6]

# You can also use '+' to add lists
# Note: unlike in 'li.extend', values for li and for other_li are not modified.
li + one_li  # => [1, 2, 4, 3, 2, 4, 5, 6, 1, 1.0, True, 'one']

# Check for existence in a list with "in"
1 in li  # => True

# get the length of the list (i.e. number of elements in a list)
len(li)  # => 8
```

### Check out the other methods of list objects in [https://docs.python.org/3/tutorial/datastructures.html](https://docs.python.org/3/tutorial/datastructures.html).

Sources:
1. https://ballmemes.com/i/arrays-start-at-0-speak-nice-doge-d54d0129c5e04f8bb7a7cf558186d009
2. Other meme: https://www.reddit.com/r/ProgrammerHumor/comments/b0aw8b/array_based_memes_for_non_matlab_teens/

You may read the article [Why numbering should start at 0](https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html) by Edsger Dijkstra, the inventor of Dijkstra's algorithm, a shortest-path algorithm which you'll learn in your Operations Research class in 5th year.

## Section 1: Iteration

Aside from branching, iteration is another way of controlling the flow of your code.

```python
"""
For loops iterate over lists
prints:
    dog is a mammal
    cat is a mammal
    mouse is a mammal
"""
for animal in ["dog", "cat", "mouse"]:
    print(f"{animal} is a mammal")
```

    dog is a mammal
    cat is a mammal
    mouse is a mammal

```python
"""
"range(number)" returns an iterable of numbers
from zero to the given number
prints:
    0
    1
    2
    3
"""
for i in range(4):
    print(i)
```

    0
    1
    2
    3

```python
"""
"range(lower, upper)" returns an iterable of numbers
from the lower number to the upper number
prints:
    4
    5
    6
    7
"""
for i in range(4, 8):
    print(i)
```

    4
    5
    6
    7

```python
"""
"range(lower, upper, step)" returns an iterable of numbers
from the lower number to the upper number, while incrementing
by step. If step is not indicated, the default value is 1.
prints:
    4
    6
"""
for i in range(4, 8, 2):
    print(i)
```

    4
    6

```python
"""
To loop over a list, and retrieve both the index and the value of each item in the list
prints:
    0 dog
    1 cat
    2 mouse
"""
animals = ["dog", "cat", "mouse"]
for i, value in enumerate(animals):
    print(i, value)
```

    0 dog
    1 cat
    2 mouse

```python
"""
While loops go until a condition is no longer met.
prints:
    0
    1
    2
    3
"""
x = 0
while x < 4:
    print(x)
    x += 1  # Shorthand for x = x + 1
```

    0
    1
    2
    3

Exercise

Suppose you invest $\$500$ in an account that pays an interest of $5.75\%$ p.a. How much will it be worth in $1.5$ years if the interest rate is
1. compounded annually?
2. compounded semiannually?
3. compounded quarterly?
4. compounded monthly?
5. compounded weekly?
6. compounded daily?

Use lists and for loops to succinctly answer the above question.

```python
def get_future_value(P, r, T, type_interest, m=None):
    """
    Input: principal P, interest rate r p.a., type of interest (whether simple or compounded m times a year),
    and investment time period T in years

    Returns the future value of a principal P invested for T years at an
    interest rate r p.a. (simple or compounded m times a year)
    """
    if type_interest == 'simple':
        A = P * (1 + r*T)
    elif type_interest == 'comp':
        A = P * (1 + r/m)**(m*T)

    return A
```

```python
m_lst = [1, 2, 4, 12, 52, 360]
for m in m_lst:
    A = get_future_value(500, 0.0575, 1.5, 'comp', m)
    print(f"The future value if the interest is compounded {m} times a year is ${A}.")
```

    The future value if the interest is compounded 1 times a year is $543.739105494308.
    The future value if the interest is compounded 2 times a year is $544.3767255859376.
    The future value if the interest is compounded 4 times a year is $544.7048313758173.
    The future value if the interest is compounded 12 times a year is $544.9271497450427.
    The future value if the interest is compounded 52 times a year is $545.013435824494.
    The future value if the interest is compounded 360 times a year is $545.0356531431054.

## Section 2: Modules

```python
# You can import modules
import math
print(math.sqrt(16))  # => 4.0

# You can get specific functions from a module
from math import ceil, floor
print(ceil(3.7))   # => 4.0
print(floor(3.7))  # => 3.0

# You can import all functions from a module.
# Warning: this is not recommended
from math import *

# You can shorten module names
import math as m
math.sqrt(16) == m.sqrt(16)  # => True

# Python modules are just ordinary Python files. You
# can write your own, and import them. The name of the
# module is the same as the name of the file.

# You can find out which functions and attributes
# are defined in a module.
import math
dir(math)
```

    4.0
    4
    3

    ['__doc__',
     '__file__',
     '__loader__',
     '__name__',
     '__package__',
     '__spec__',
     'acos',
     'acosh',
     'asin',
     'asinh',
     'atan',
     'atan2',
     'atanh',
     'ceil',
     'comb',
     'copysign',
     'cos',
     'cosh',
     'degrees',
     'dist',
     'e',
     'erf',
     'erfc',
     'exp',
     'expm1',
     'fabs',
     'factorial',
     'floor',
     'fmod',
     'frexp',
     'fsum',
     'gamma',
     'gcd',
     'hypot',
     'inf',
     'isclose',
     'isfinite',
     'isinf',
     'isnan',
     'isqrt',
     'ldexp',
     'lgamma',
     'log',
     'log10',
     'log1p',
     'log2',
     'modf',
     'nan',
     'perm',
     'pi',
     'pow',
     'prod',
     'radians',
     'remainder',
     'sin',
     'sinh',
     'sqrt',
     'tan',
     'tanh',
     'tau',
     'trunc']

```python
math.exp(1) #e^1
```

    2.718281828459045

Definition: Continuously Compounded Interest

Suppose that an amount $P$ is invested for $T$ years at an interest rate of $r$ per annum (p.a.). If the interest rate is **compounded continuously**, then the terminal value of the investment is
\begin{align}
    A = Pe^{rT} \label{Continuously Compounding Interest}
\end{align}

Exercise

Suppose you invest $\$500$ in an account that pays a continuously-compounded interest of $5.75\%$ p.a. How much will it be worth in $1.5$ years?

Change the function below to included cont. comp. interest. Use the `math.exp()` function in `math` module.

```python
def get_future_value(P, r, T, type_interest, m=None):
    """
    Input: principal P, interest rate r p.a., type of interest (whether simple or compounded m times a year),
    and investment time period T in years

    Returns the future value of a principal P invested for T years at an
    interest rate r p.a. (simple or compounded m times a year)
    """
    if type_interest == 'simple':
        A = P * (1 + r*T)
    elif type_interest == 'comp':
        A = P * (1 + r/m)**(m*T)

    return A
```

```python

```


# Task 4: Time-dependent general Hartree-Fock solver

Having constructed and solved the general Hartree-Fock ground state problem, we now set out to shine a monochromatic dipole laser on the system.
This means appending a time-dependent operator to the Hamiltonian.
The operator describes a semi-classical electric field in the dipole approximation in the length gauge.
For a more thorough discussion of this operator check out chapters 2 - 2.4 in {cite}`joachain2012atoms`.
The operator will in the one-dimensional case be described by
\begin{align}
    \hat{h}_I(t) = -f(t) \hat{d},
\end{align}
where $\hat{d} \equiv q\hat{x}$ is the dipole moment operator with $q = -1$ the electron charge, and $f(t)$ the time-dependent laser field.
From {cite}`zanghellini_2004` we have $f(t) = \mathcal{E}_0 \sin(\omega t)$, (note that $\omega$ is not necessarily the same as the harmonic oscillator well frequency) which describes a monochromatic laser field that is always active.

## Time-dependent Hartree-Fock

In the time-dependent Hartree-Fock method we use an ansatz for the many-body wave function as
\begin{align}
    | \Psi(t) \rangle = | \Phi(t) \rangle = | \phi_1(t) \phi_2(t) \dots \phi_n(t) \rangle,
\end{align}
where the time-dependence is kept in the molecular orbitals.
```{note}
Add derivation of the time-dependent Hartree-Fock equations.
```
The time-evolution of the molecular orbitals is described by the time-dependent Hartree-Fock equation
\begin{align}
    i \frac{\text{d}}{\text{d} t} | \phi_p(t) \rangle
    = \hat{f}(t) | \phi_p(t) \rangle,
\end{align}
where $\hat{f}(t)$ is the time-dependent Fock operator and we have set $\hbar = 1$.
We have chosen to expand the time-independent molecular orbitals in a known basis of atomic orbitals (the harmonic oscillator eigenfunctions).
Here as well we choose to expand our time-dependent molecular orbitals in a time-independent basis of atomic orbitals, and let the time-evolution occur in the coefficients.
That is,
\begin{align}
    | \phi_p(t) \rangle
    = \sum_{\mu = 1}^{l} C_{\mu p}(t) | \psi_{\mu} \rangle,
\end{align}
where $\{\psi_{\mu}\}_{\mu = 1}^{l}$ is a time-independent atomic orbital basis which we assume to be orthonormal.
Inserting this expansion into the time-dependent Hartree-Fock equations and left-projecting with $\langle \psi_{\mu}|$ we get
\begin{gather}
    i \frac{\text{d}}{\text{d} t} \sum_{\nu = 1}^{l} C_{\nu p}(t) | \psi_{\nu} \rangle
    = \hat{f}(t) \sum_{\nu = 1}^{l} C_{\nu p}(t) | \psi_{\nu} \rangle
    \\
    \implies
    i \sum_{\nu = 1}^{l} \dot{C}_{\nu p}(t)
    \langle \psi_{\mu} | \psi_{\nu} \rangle
    = i \dot{C}_{\mu p}(t)
    = \langle \psi_{\mu} | \hat{f}(t) | \psi_{\nu} \rangle C_{\nu p}(t)
    = f_{\mu \nu}(t) C_{\nu p}(t),
\end{gather}
where we now need to find the matrix elements of the time-dependent Fock operator in the atomic orbital basis.
As the laser field interaction operator $\hat{h}_I(t)$ is a one-body operator, the time-dependent Fock operator will need to add this term.


# Realization of Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Direct Form Structures

The output signal $y[k] = \mathcal{H} \{ x[k] \}$ of a recursive linear-time invariant (LTI) system is given by

\begin{equation}
y[k] = \frac{1}{a_0} \left( \sum_{m=0}^{M} b_m \; x[k-m] - \sum_{n=1}^{N} a_n \; y[k-n] \right)
\end{equation}

where $a_n$ and $b_m$ denote constant coefficients and $N$ the order. Note that systems with $M > N$ are in general not stable. The computational realization of above equation requires additions, multiplications, the actual and past samples of the input signal $x[k]$, and the past samples of the output signal $y[k]$. Technically this can be realized by

* adders
* multipliers, and
* unit delays or storage elements.

These can be arranged in different topologies. A certain class of structures, which is introduced in the following, is known as *direct form structures*. Other known forms are for instance [cascaded sections](cascaded_structures.ipynb), parallel sections, lattice structures and state-space structures.

For the following it is assumed that $a_0 = 1$. This can be achieved for instance by dividing the remaining coefficients by $a_0$.

### Direct Form I

The [direct form I](https://en.wikipedia.org/wiki/Digital_filter#Direct_Form_I) is derived by rearranging above equation for $a_0 = 1$

\begin{equation}
y[k] = \sum_{m=0}^{M} b_m \; x[k-m] + \sum_{n=1}^{N} - a_n \; y[k-n]
\end{equation}

It is now evident that we can realize the recursive filter by a superposition of a non-recursive and a recursive part. With the elements given above, this results in the following block-diagram

This representation is not canonical since $N + M$ unit delays are required to realize a system of order $N$. A benefit of the direct form I is that there is essentially only one summation point which has to be taken care of when considering quantized variables and overflow. The output signal $y[k]$ for the direct form I is computed by realizing above equation.

The block diagram of the direct form I can be interpreted as the cascade of two systems. Denoting the signal in between both as $w[k]$ and discarding initial values we get

\begin{align}
w[k] &= \sum_{m=0}^{M} b_m \; x[k-m] = h_1[k] * x[k] \\
y[k] &= w[k] + \sum_{n=1}^{N} - a_n \; w[k-n] = h_2[k] * w[k] = h_2[k] * h_1[k] * x[k]
\end{align}

where $h_1[k] = [b_0, b_1, \dots, b_M]$ denotes the impulse response of the non-recursive part and $h_2[k] = [1, -a_1, \dots, -a_N]$ for the recursive part. From the last equality of the second equation and the commutativity of the convolution it becomes clear that the order of the cascade can be exchanged.

### Direct Form II

The [direct form II](https://en.wikipedia.org/wiki/Digital_filter#Direct_Form_II) is yielded by exchanging the two systems in above block diagram and noticing that there are two parallel columns of delays which can be combined, since they are redundant. For $N=M$ it is given as

Other cases with $N \neq M$ can be considered for by setting coefficients to zero. This form is a canonical structure since it only requires $N$ unit delays for a recursive filter of order $N$. The output signal $y[k]$ for the direct form II is computed by the following equations

\begin{align}
w[k] &= x[k] + \sum_{n=1}^{N} - a_n \; w[k-n] \\
y[k] &= \sum_{m=0}^{M} b_m \; w[k-m]
\end{align}

The samples $w[k-m]$ are termed *state* (variables) of a digital filter.

### Transposed Direct Form II

The block diagrams above can be interpreted as linear [signal flow graphs](https://en.wikipedia.org/wiki/Signal-flow_graph). The theory of these graphs provides useful transformations into different forms which preserve the overall transfer function. Of special interest is the *transposition* or *reversal* of a graph which can be achieved by

* exchanging in- and output,
* exchanging signal split and summation points, and
* reversing the directions of the signal flows.

Applying this procedure to the direct form II shown above for $N=M$ yields the transposed direct form II

The output signal of the transposed direct form II is given as

\begin{equation}
y[k] = b_0 x[k] + \sum_{m=1}^{M} b_m x[k-n] - \sum_{n=1}^{N} a_n y[k-n]
\end{equation}

Using the signal before the $n$-th delay unit as internal state $w_n[k]$ we can reformulate this into a set of difference equations for computation of the output signal

\begin{align}
w_n[k] &=
\begin{cases}
w_{n+1}[k-1] - a_n y[k] + b_n x[k] & \text{for } n=0,1,\dots,N-1 \\
-a_N y[k]  + b_N x[k] & \text{for } n=N
\end{cases}\\
y[k] &= w_1[k-1] + b_0 x[k]
\end{align}

#### Example - Computation of output signal

The following example illustrates the computation of the impulse response $h[k]$ of a 2nd-order recursive system using the transposed direct form II as realized by `scipy.signal.lfilter`.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

p = 0.90*np.exp(-1j*np.pi/4)
a = np.poly([p, np.conj(p)])  # denominator coefficients
b = [1, 0, 0]  # numerator coefficients
N = 40  # number of computed samples

# generate input signal (= Dirac impulse)
k = np.arange(N)
x = np.where(k == 0, 1.0, 0.0)

# filter signal using transposed direct form II
h = sig.lfilter(b, a, x)

# plot output signal
plt.figure(figsize=(8, 4))
plt.stem(h, use_line_collection=True)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
plt.axis([-1, N, -1.5, 1.5])
plt.grid()
```

**Exercise**

* Compute the step-response $h_\epsilon[k] = \mathcal{H} \{ \epsilon[k] \}$ of the filter by modifying above example.

Solution: The step response can be computed by changing the input signal to

```python
x = np.where(k>=0, 1.0, 0.0)
```

or by cumulative summation of the impulse response due to the relation $h_\epsilon[k] = \sum_{\kappa =0}^k h[k]$ of the step response to the impulse response

```python
he = np.cumsum(h)
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# Neuromatch Academy: Week 1, Day 5, Tutorial 2
# Dimensionality Reduction: Principal component analysis

---

In this notebook we'll learn how to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

Steps:
 1. Calculate the eigenvectors of the sample covariance matrix.
 2. Perform PCA by projecting data onto the eigenvectors of the covariance matrix.
 3. Plot the eigenvalues.

---

```python
#@title Video: PCA
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="Ze0uK1ipEn0", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=Ze0uK1ipEn0

# Setup
Run these cells to get the tutorial started.

```python
#library imports
import time                        # import time
import numpy as np                 # import numpy
import scipy as sp                 # import scipy
import math                        # import basic math functions
import random                      # import basic random number generator functions

import matplotlib.pyplot as plt    # import matplotlib
from IPython import display
```

```python
#@title Figure Settings
%matplotlib inline
fig_w, fig_h = (8, 8)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
plt.style.use('ggplot')
%config InlineBackend.figure_format = 'retina'
```

```python
#@title Helper functions
# new helper functions
def plot_eigenvalues(evals):
  """
  Plots eigenvalues.

  Args:
     (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """
  plt.plot(np.arange(1,len(evals)+1),evals,'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
  plt.xticks(np.arange(1,len(evals)+1))
  plt.ylim([0,2.5])

def sort_evals_descending(evals,evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats):   Vector of eigenvalues
    evectors (numpy array of floats):   Corresponding matrix of eigenvectors
                                each column corresponds to a different eigenvalue

  Returns:
    (numpy array of floats) : Vector of eigenvalues after sorting
    (numpy array of floats) : Matrix of eigenvectors after sorting
  """
  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:,index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:,0], 1./np.sqrt(2)*np.array([1,1]))) > np.pi/2.:
      evectors[:,0] = -evectors[:,0]
    if np.arccos(np.matmul(evectors[:,1], 1./np.sqrt(2)*np.array([-1,1]))) > np.pi/2.:
      evectors[:,1] = -evectors[:,1]
  return evals, evectors

  # Old helper functions from Tutorial 1

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  scatter plot of their joint activity. The title indicates the sample correlation
  calculated from the data.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(X[:,0],color='k')
  plt.ylabel('Neuron 1')
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(X[:,1],color='k')
  plt.xlabel('Sample Number (sorted)')
  plt.ylabel('Neuron 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:,0],X[:,1],'.',markerfacecolor=[.5,.5,.5], markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:,0],X[:,1])[0,1]))

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

  Note that samples are sorted in ascending order for the first random variable.

  Args:
    var_1 (scalar):    variance of the first random variable
    var_2 (scalar):    variance of the second random variable
    cov_matrix (numpy array of floats): desired covariance matrix

  Returns:
    (numpy array of floats) : samples from the bivariate Gaussian, with
                              each column corresponding to a different random variable
  """
  mean = np.array([0,0])
  X = np.random.multivariate_normal(mean,cov_matrix,size = 1000)
  indices_for_sorting = np.argsort(X[:,0])
  X = X[indices_for_sorting,:]
  return X

def calculate_cov_matrix(var_1,var_2,corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation coefficient.

  Args:
    var_1 (scalar):    variance of the first random variable
    var_2 (scalar):    variance of the second random variable
    corr_coef (scalar):      correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  cov = corr_coef * np.sqrt(var_1 * var_2)
  cov_matrix = np.array([[var_1,cov],[cov,var_2]])
  return cov_matrix

def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats):    arbitrary 2D vector used for new basis

  Returns:
    (numpy array of floats) : new orthonormal basis
                                columns correspond to basis vectors
  """
  u = u / np.sqrt(u[0]**2 + u[1]**2)
  w = np.array([-u[1],u[0]])
  W = np.column_stack((u,w))
  return W

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases. Similar to plot_data but
  with colors corresponding to projections onto basis 1 (red) and basis 2 (blue).
  The title indicates the sample correlation calculated from the data.

  Note that samples are re-sorted in ascending order for the first random variable.

  Args:
    Y (numpy array of floats):   Data matrix in new basis
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(Y[:,0],'r')
  plt.ylabel('Projection \n basis vector 1')
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(Y[:,1],'b')
  plt.xlabel('Projection \n basis vector 1')
  plt.ylabel('Projection \n basis vector 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:,0],Y[:,1],'.',color=[.5,.5,.5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:,0],Y[:,1])[0,1]))

def change_of_basis(X,W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix
                                each column corresponding to a different random variable
    W (numpy array of floats):  new orthonormal basis
                                columns correspond to basis vectors

  Returns:
    (numpy array of floats) : Data matrix expressed in new basis
  """
  Y = np.matmul(X,W)
  return Y

def plot_basis_vectors(X,W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable
    W (numpy array of floats):   Square matrix representing new orthonormal basis
                                  each column represents a basis vector

  Returns:
    Nothing.
  """
  plt.figure(figsize=[4,4])
  plt.plot(X[:,0],X[:,1],'.',color=[.5,.5,.5],label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0,W[0,0]],[0,W[1,0]],color='r',linewidth=3,label = 'Basis vector 1')
  plt.plot([0,W[0,1]],[0,W[1,1]],color='b',linewidth=3,label = 'Basis vector 2')
  plt.legend()

```

# Calculate the eigenvectors of the the sample covariance matrix.

PCA represents data in a new basis defined by the eigenvectors of the covariance matrix. In the previous tutorial, we generated bivariate normal data with a specified covariance matrix $\bf \Sigma$, whose $(i,j)$th element is:
\begin{equation}
\Sigma_{ij} = E[ x_i x_j ] - E[ x_i] E[ x_j ] .
\end{equation}
However, in real life we don't have access to the ground-truth covariance matrix. To get around this, we can use the sample covariance matrix, $\bf\hat\Sigma$, calculated directly from the data. The $(i,j)$th element of the sample covariance matrix is:
\begin{equation}
 \hat \Sigma_{ij} =  \frac{1}{N_\text{samples}}{\bf x}_i^T {\bf x}_j - \bar {\bf x}_i \bar{\bf x}_j ,
\end{equation}
where ${\bf x}_i = [ x_i(1), x_i(2), \dots,x_i(N_\text{samples})]^T$ is a column vector representing all measurements of neuron i, and  $\bar {\bf x}_i$ is a scalar representing its sample mean:
\begin{equation}
\bar {\bf x}_i = \frac{1}{N_\text{samples}} \sum_{k=1}^{N_\text{samples}} x_i(k).
\end{equation}
If we assume that the data has already been mean-subtracted, then we can write:
\begin{align}
{\bf \hat \Sigma}
&= \frac{1}{N_\text{samples}} {\bf X}^T {\bf X}.
\end{align}

---

#### Exercise

Before we move on to calculating the eigenvectors, we'll first calculate the sample covariance matrix from the data. Your job is to finish the function `get_sample_cov_matrix`. Then you can use the provided code to generate bivariate normal data with `get_data`, and calculate the sample covariance matrix with your finished `get_sample_cov_matrix`. Then you can compare this estimate to the true covariate matrix using `calculate_cov_matrix`,

```python
help(get_data)
help(calculate_cov_matrix)
```

    Help on function get_data in module __main__:

    get_data(cov_matrix)
        Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

        Note that samples are sorted in ascending order for the first random variable.

        Args:
          var_1 (scalar):    variance of the first random variable
          var_2 (scalar):    variance of the second random variable
          cov_matrix (numpy array of floats): desired covariance matrix

        Returns:
          (numpy array of floats) : samples from the bivariate Gaussian, with
                                    each column corresponding to a different random variable

    Help on function calculate_cov_matrix in module __main__:

    calculate_cov_matrix(var_1, var_2, corr_coef)
        Calculates the covariance matrix based on the variances and correlation coefficient.

        Args:
          var_1 (scalar):    variance of the first random variable
          var_2 (scalar):    variance of the second random variable
          corr_coef (scalar):      correlation coefficient

        Returns:
          (numpy array of floats) : covariance matrix

```python
def get_sample_cov_matrix(X):
  """
    Returns the sample covariance matrix of data X

    Args:
      X (numpy array of floats):   Data matrix
                                   each column corresponds to a different random variable

    Returns:
      (numpy array of floats) : Covariance matrix
  """
  ###################################################################
  ## Insert your code here to:
  ##                Subtract the mean of X
  ##                Calculate the covariance matrix (hint: use np.matmul)

  #X = ...
  #cov_matrix = ...

  #uncomment once you've filled in the function
  raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  ###################################################################
  return cov_matrix

## Uncomment below code to:
##                generate bivariate Gaussian data with variances of 1
##                and a correlation coefficient of 0.8
##                compare the true and sample covariance matrices
###################################################################
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

#cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
#print(cov_matrix)

#X = get_data(cov_matrix)
#sample_cov_matrix = get_sample_cov_matrix(X)
#print(sample_cov_matrix)

```

```python
# to_remove solution

def get_sample_cov_matrix(X):
  """
    Returns the sample covariance matrix of data X

    Args:
      X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

    Returns:
      (numpy array of floats) : Covariance matrix
  """
  X = X - np.mean(X,0)
  cov_matrix = 1./X.shape[0]*np.matmul(X.T,X)
  return cov_matrix

cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
print(cov_matrix)

X = get_data(cov_matrix)
sample_cov_matrix = get_sample_cov_matrix(X)
print(sample_cov_matrix)

```

    [[1.  0.8]
     [0.8 1. ]]
    [[0.96463129 0.74853726]
     [0.74853726 0.94341816]]

#### Exercise
Next we will compare the eigenvectors of the covariance matrix to the geometrical view of the data.

**Suggestions:**
* Calculate the eigenvalues and eigenvectors of the sample covariance matrix. (**Hint:** use `np.linalg.eig`).
* Use the provided code to sort the eigenvalues in descending order.
* Plot the eigenvectors on a scatter plot of the data, using the function `plot_basis_vectors`.

```python
help(sort_evals_descending)
help(plot_basis_vectors)
```

    Help on function sort_evals_descending in module __main__:

    sort_evals_descending(evals, evectors)
        Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
        eigenvectors to be in first two quadrants (if 2D).

        Args:
          evals (numpy array of floats):   Vector of eigenvalues
          evectors (numpy array of floats):   Corresponding matrix of eigenvectors
                                      each column corresponds to a different eigenvalue

        Returns:
          (numpy array of floats) : Vector of eigenvalues after sorting
          (numpy array of floats) : Matrix of eigenvectors after sorting

    Help on function plot_basis_vectors in module __main__:

    plot_basis_vectors(X, W)
        Plots bivariate data as well as new basis vectors.

        Args:
          X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable
          W (numpy array of floats):   Square matrix representing new orthonormal basis
                                        each column represents a basis vector

        Returns:
          Nothing.

```python
###################################################################
## Insert your code here to:
##                Calculate the eigenvalues and eigenvectors
##                Sort the eigenvalues in descending order
##                plot the eigenvectors against the data
###################################################################

# evals, evectors = ...
# evals, evectors = ..
# plot_basis_vectors(X,evectors)

```

```python
# to_remove solution

evals, evectors = np.linalg.eig(cov_matrix)
evals, evectors = sort_evals_descending(evals,evectors)

with plt.xkcd():
  plot_basis_vectors(X,evectors)

```

# Perform PCA by projecting data onto the eigenvectors.

To perform PCA, we will project the data onto the eigenvectors of the covariance matrix, i.e.:
\begin{equation}
\bf S = X W
\end{equation}
where $\bf S$ is an $N_\text{samples} \times N$ matrix representing the projected data (also called *scores*), and $W$ is an $N\times N$ orthogonal matrix, each of whose columns represents the eigenvectors of the covariance matrix (also called *weights* or *loadings*).

#### Exercise
We will now perform PCA on the data using the intuition we have built. Fill in the function below to carry out the steps to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

**Suggestions**
* First subtract the mean.
* Then calculate the covariance.
* Then find the eigenvalues and order them in descending order.
* Finally project the mean-centered data onto the eigenvectors.

```python
help(change_of_basis)
help(plot_data_new_basis)
```

    Help on function change_of_basis in module __main__:

    change_of_basis(X, W)
        Projects data onto a new basis.

        Args:
          X (numpy array of floats) : Data matrix
                                      each column corresponding to a different random variable
          W (numpy array of floats):  new orthonormal basis
                                      columns correspond to basis vectors

        Returns:
          (numpy array of floats) : Data matrix expressed in new basis

    Help on function plot_data_new_basis in module __main__:

    plot_data_new_basis(Y)
        Plots bivariate data after transformation to new bases. Similar to plot_data but
        with colors corresponding to projections onto basis 1 (red) and basis 2 (blue).
        The title indicates the sample correlation calculated from the data.

        Note that samples are re-sorted in ascending order for the first random variable.

        Args:
          Y (numpy array of floats):   Data matrix in new basis
                                        each column corresponds to a different random variable

        Returns:
          Nothing.

```python
def pca(X):
  """
  Sorts eigenvalues and eigenvectors in decreasing order.

  Args:
     X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    (numpy array of floats) : Data projected onto the new basis
    (numpy array of floats) : Vector of eigenvalues
    (numpy array of floats) : Corresponding matrix of eigenvectors

  """
  ###################################################################
  ## Insert your code here to ...
  ##                Subtract the mean of X
  ##                Calculate the sample covariance matrix
  ##                Calculate the eigenvalues and eigenvectors
  ##                Sort the eigenvalues in descending order
  ##                Project the data onto the new eigenvector basis
  ###################################################################
  return score, evectors, evals

###################################################################
## Insert your code here to:
##                perform PCA on the data matrix X
##                Plot the data projected into the new basis
###################################################################

# score, evectors, evals = ...
# plot_data_new_basis(score)
```

```python
# to_remove solution

def pca(X):
  """
  Performs PCA on multivariate data.

  Args:
     X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    (numpy array of floats) : Data projected onto the new basis
    (numpy array of floats) : Vector of eigenvalues
    (numpy array of floats) : Corresponding matrix of eigenvectors

  """
  X = X - np.mean(X,0)
  cov_matrix = get_sample_cov_matrix(X)
  evals, evectors = np.linalg.eig(cov_matrix)
  evals, evectors = sort_evals_descending(evals,evectors)
  score = change_of_basis(X,evectors)
  return score, evectors, evals

with plt.xkcd():
  score, evectors, evals = pca(X)
  plot_data_new_basis(score)
```

# Plot and explore the eigenvalues.

```python
#@title Video: Properties of PCA
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="uPcrB8B9S7A", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=uPcrB8B9S7A

Finally, we will examine the eigenvalues of the covariance matrix, which represent the variance of the projected data. Run the code below to plot the eigenvalues (sometimes called the "scree plot"). Which eigenvalue is larger?

```python
 plot_eigenvalues(evals)
```

Now, run the following cell and use the slider to change the correlation coefficient in the data. You should see the scree plot and the plot of basis vectors update.

**Questions:**
* What happens to the eigenvalues as you change the correlation coefficient?
* Can you find a value when both eigenvalues are equal?
* Can you find a value when only one eigenvalue is nonzero?

```python
###### MAKE SURE TO RUN THIS CELL VIA THE PLAY BUTTON TO ENABLE SLIDERS ########

import ipywidgets as widgets

def refresh(corr_coef = .8):
  cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
  X = get_data(cov_matrix)
  score, evectors, evals = pca(X)
  plot_eigenvalues(evals)
  plot_basis_vectors(X,evectors)

_ = widgets.interact(refresh,
  corr_coef = (-1, 1, .1))

```


# Matrix Formalism of the Newton-Euler equations

Renato Naville Watanabe

In this notebook will be shown two examples of how to use a matrix formalism to perform inverse dynamics analysis. It does not consist a comprehensive treatise about the subject. It is rather an introduction based on examples. Nevertheless, the reader of this notebook will have sufficient knowledge to read recent texts on biomechanics and other multibody dynamic analysis.

## Inverse dynamics

For the inverse dynamics analysis, we will obtain the joint torques and forces, from the joint kinematics and external forces.

Adapted from Erdemir et al. (2007)

  As an example, we will consider the problem of estimating the forces and torques in the ankle and knee joints during the gait, considering a 3D movement. At this point, we consider that the  accelerations, angular velocities, angular accelerations, masses, moments of inertia and rotation matrices necessary  to compute the forces and moments are known.

 The free-body diagram of the gait, considering a 3D movement is very similar [to the 2D case](GaitAnalysis2D.ipynb). The equations of forces and moments are described by the Newton-Euler equations (for a revision on Tridimensional Newton-Euler equations click [here](Tridimensional%20rigid%20body%20Kinetics.ipynb)):
\begin{align}
\overrightarrow{F_A} + \overrightarrow{GRF} + m_F\overrightarrow{g} &= m_F\overrightarrow{a_{cm_F}}\\
\overrightarrow{M_A} + \overrightarrow{M_{GRF}}+ \overrightarrow{M_{FA}}&=I_F\overrightarrow{\dot{\omega_F}} + \overrightarrow{\omega_F} \times (I_F\overrightarrow{\omega_F})\\
\overrightarrow{F_K} -\overrightarrow{F_A} + m_S\overrightarrow{g} &= m_S\overrightarrow{a_{cm_S}}\\
    \overrightarrow{M_K} - \overrightarrow{M_A} + \overrightarrow{M_{FA}} + \overrightarrow{M_{FK}} &= I_S\overrightarrow{\dot{\omega_S}} + \overrightarrow{\omega_S} \times (I_S\overrightarrow{\omega_S})
\end{align}
where

- $\overrightarrow{g} = -9.81\hat{j}$;
- $m_F$ and $m_S$ are the masses of the foot and the shank, respectively;
- $\overrightarrow{GRF}$ is the ground reaction force being applied to the foot;
- $\overrightarrow{a_{cm_F}}$ and $\overrightarrow{a_{cm_S}}$ are the accelerations of the center of mass of the foot and the shank, respectively;
- $\overrightarrow{\omega_F}$ and $\overrightarrow{\omega_S}$  are the angular accelerations of the foot and shank, respectively, described at a basis attached to the segment, and $\overrightarrow{\dot{\omega_F}}$ and $\overrightarrow{\dot{\omega_S}}$ are their time-derivatives;
- $I_S$ and $I_F$ are the matrices of inertia of the shank and the foot, respectively;
- $\overrightarrow{F_K}$, $\overrightarrow{F_A}$, $\overrightarrow{M_A}$ and $\overrightarrow{M_A}$ are the forces and moments at the ankle and knee joints, respectively

Note that each of these equations have components at each of the three directions. Additionally, note that the equations of the forces are described in the global basis, and the equations of the moments must be described in the basis attached to the segment relative to that equation. So, it is a good idea to make this clear with a more precise notation. We will denote as a superscript in the vectors the segment where the basis that we are describing the vector is fixed. So for example, $\overrightarrow{M_A^F}$ is the vector of the moment due to the muscle forces of the ankle, described in the basis fixed at the foot. So, the equations can be rewritten as:

\begin{align}
\overrightarrow{F_A^G} + \overrightarrow{GRF^G} + m_F\overrightarrow{g^G} &= m_F\overrightarrow{a_{cm_F}^G}\\
\overrightarrow{M_A^F} + \overrightarrow{M_{GRF}^F}+ \overrightarrow{M_{FA}^F}&=I_F\overrightarrow{\dot{\omega_F^F}} + \overrightarrow{\omega_F^F} \times (I_F\overrightarrow{\omega_F^F})\\
\overrightarrow{F_K^G} -\overrightarrow{F_A^G} + m_S\overrightarrow{g^G} &= m_S\overrightarrow{a_{cm_S}^G}\\
    \overrightarrow{M_K^S} - \overrightarrow{M_A^S} + \overrightarrow{M_{FA}^S} + \overrightarrow{M_{FK}^S} &= I_S\overrightarrow{\dot{\omega_S^S}} + \overrightarrow{\omega_S^S} \times (I_S\overrightarrow{\omega_S^S})
\end{align}
where the superscript $G$ denotes the global frame of reference, the superscript $S$ denotes the frame of reference in the shank and the superscript $F$ denotes the frame of reference at the foot.

The moments due to the ground reaction force, the force at the ankle and the force at the knee are computed by cross-multiplying them by their moment-arms. As the forces and the moment-arms are described in the global basis, we must multiply them by the rotation matrix of the basis corresponding to the segment. So, the equations can be rewritten as:

\begin{align}
\overrightarrow{F_A^G} + \overrightarrow{GRF^G} + m_F\overrightarrow{g^G} &= m_F\overrightarrow{a_{cm_F}^G}\\
\overrightarrow{M_A^F} + R_F(\overrightarrow{r_{cop/cm_F}^G}\times \overrightarrow{GRF^G})+ R_F(\overrightarrow{r_{A/cm_F}^G}\times \overrightarrow{F_A}^G)&=I_F\overrightarrow{\dot{\omega_F^F}} + \overrightarrow{\omega_F^F} \times (I_F\overrightarrow{\omega_F^F})\\
\overrightarrow{F_K^G} -\overrightarrow{F_A^G} + m_S\overrightarrow{g^G} &= m_S\overrightarrow{a_{cm_S}^G}\\
\overrightarrow{M_K^S} - \overrightarrow{M_A^S} - R_S(\overrightarrow{r_{A/cm_S}^G}\times \overrightarrow{F_A^G}) + R_S(\overrightarrow{r_{K/cm_S}^G}\times \overrightarrow{F_K^G}) &= I_S\overrightarrow{\dot{\omega_S^S}} + \overrightarrow{\omega_S^S} \times (I_S\overrightarrow{\omega_S^S})
\end{align}
where $R_S$ is the rotation matrix of the basis attached to the shank and $R_F$ is the rotation matrix of the basis attached to the foot.

Now, we can note that the vectors $\overrightarrow{M_K^S}$ and $\overrightarrow{M_K^F}$ are the same vectors described in different basis. So we could use only one of the descriptions and use rotation matrices to convert from one to another. To pass the vector from the foot coordinates to the shank coordinate, we must first multiply it by the inverted rotation matrix of the foot and then multiply it by the rotation matrix of the shank. So, $\overrightarrow{M_A^S} = R_SR_F^{-1}\overrightarrow{M_A^F}$ and the equations above can be rewritten as:

\begin{align}
\overrightarrow{F_A^G} + \overrightarrow{GRF^G} + m_F\overrightarrow{g^G} &= m_F\overrightarrow{a_{cm_F}^G}\\
\overrightarrow{M_A^F} + R_F(\overrightarrow{r_{cop/cm_F}^G}\times \overrightarrow{GRF^G})+ R_F(\overrightarrow{r_{A/cm_F}^G}\times \overrightarrow{F_A}^G)&=I_F\overrightarrow{\dot{\omega_F^F}} + \overrightarrow{\omega_F^F} \times (I_F\overrightarrow{\omega_F^F})\\
\overrightarrow{F_K^G} -\overrightarrow{F_A^G} + m_S\overrightarrow{g^G} &= m_S\overrightarrow{a_{cm_S}^G}\\
\overrightarrow{M_K^S} - R_SR_F^{-1}\overrightarrow{M_A^F} - R_S(\overrightarrow{r_{A/cm_S}^G}\times \overrightarrow{F_A^G}) + R_S(\overrightarrow{r_{K/cm_S}^G}\times \overrightarrow{F_K^G}) &= I_S\overrightarrow{\dot{\omega_S^S}} + \overrightarrow{\omega_S^S} \times (I_S\overrightarrow{\omega_S^S})
\end{align}

Now, we divide the equations above in the matrices defined previously:
\begin{equation}
\underbrace{\left[\begin{array}{cccc} m_FI_3& [0]& [0]& [0]\\ [0]& I_F  & [0] & [0] \\ [0] &[0] &  m_SI_3& [0] \\ [0] & [0] & [0] & I_S\end{array}\right]}_{M}\cdot\left[\begin{array}{c}\overrightarrow{a_{cm_F}^G}\\\overrightarrow{\dot{\omega_F^F}}\\\overrightarrow{a_{cm_S}^G}\\\overrightarrow{\dot{\omega_S^S}} \\ \end{array}\right] = \underbrace{\left[\begin{array}{c}[0]\\ - \overrightarrow{\omega_F^F} \times (I_F\overrightarrow{\omega_F^F})  \\ [0] \\ - \overrightarrow{\omega_S^S} \times (I_S\overrightarrow{\omega_S^S}) \end{array}\right]}_{C} + \underbrace{\left[\begin{array}{c}  m_F\overrightarrow{g^G}\\ [0]\\  m_S\overrightarrow{g^G} \\  [0] \end{array}\right]}_{G} + \underbrace{\left[\begin{array}{c}  \overrightarrow{F_A^G}\\ \overrightarrow{M_A^F}+R_F(\overrightarrow{r_{A/cm_F}^G}\times \overrightarrow{F_A}^G)\\  \overrightarrow{F_K^G} - \overrightarrow{F_A^G} \\ \overrightarrow{M_K^S} - R_SR_F^{-1}\overrightarrow{M_A^F} - R_S(\overrightarrow{r_{A/cm_S}^G}\times \overrightarrow{F_A^G}) + R_S(\overrightarrow{r_{K/cm_S}^G}\times \overrightarrow{F_K^G})  \end{array}\right]}_{Q} + \underbrace{\left[\begin{array}{c} \overrightarrow{GRF^G}\\  R_F(\overrightarrow{r_{cop/cm_F}^G}\times \overrightarrow{GRF^G})\\  [0] \\  [0] \end{array}\right]}_{E}
\end{equation}

where $I_3$ is the identity matrix 3x3.

To perform the inverse dynamics, we still cannot isolate the vector of forces and moments. As the vector $F$ has cross-products we must define the a new operator that performs the cross-product through a matrix multiplication.

We can note that the cross-product between the vectors $\vec{v}$ and $\vec{w}$ has the following result:

\begin{equation}
    \vec{v} \times \vec{w} = \left[\begin{array}{c}v_x\\v_y\\v_z \end{array}\right] \times \left[\begin{array}{c}w_x\\w_y\\w_z \end{array}\right] = \left[\begin{array}{c}v_yw_z - v_zw_y\\v_zw_x - v_xw_z\\v_xw_y - v_yw_x \end{array}\right] = \left[\begin{array}{ccc}0&-v_z&v_y\\v_z&0&-v_x\\-v_y&v_x&0 \end{array}\right]\cdot\left[\begin{array}{c}w_x\\w_y\\w_z \end{array}\right]
\end{equation}

So we can define a new operator known as skew-symmetric matrix:

\begin{equation}
    S(\vec{v}) \triangleq \left[\begin{array}{ccc}0&-v_z&v_y\\v_z&0&-v_x\\-v_y&v_x&0 \end{array}\right]
\end{equation}

Therefore:

\begin{equation}
    \vec{v} \times \vec{w} = S(\vec{v})\cdot\vec{w}
\end{equation}

Now, we will use this operator in  the equation we found previously:

\begin{equation}
\left[\begin{array}{cccc} m_FI_3& [0]& [0]& [0]\\ [0]& I_F  & [0] & [0] \\ [0] &[0] &  m_SI_3& [0] \\ [0] & [0] & [0] & I_S\end{array}\right]\cdot\left[\begin{array}{c}\overrightarrow{a_{cm_F}^G}\\\overrightarrow{\dot{\omega_F^F}}\\\overrightarrow{a_{cm_S}^G}\\\overrightarrow{\dot{\omega_S^S}} \\ \end{array}\right] = \left[\begin{array}{c}[0]\\ - \overrightarrow{\omega_F^F} \times (I_F\overrightarrow{\omega_F^F})  \\ [0] \\ - \overrightarrow{\omega_S^S} \times (I_S\overrightarrow{\omega_S^S}) \end{array}\right] + \left[\begin{array}{c}  m_F\overrightarrow{g^G}\\ [0]\\  m_S\overrightarrow{g^G} \\  [0] \end{array}\right] + \left[\begin{array}{c}  \overrightarrow{F_A^G}\\ \overrightarrow{M_A^F}+R_F(S(\overrightarrow{r_{A/cm_F}^G})\cdot\overrightarrow{F_A}^G)\\  \overrightarrow{F_K^G} - \overrightarrow{F_A^G} \\ \overrightarrow{M_K^S} - R_SR_F^{-1}\overrightarrow{M_A^F} - R_S(S(\overrightarrow{r_{A/cm_S}^G})\cdot\overrightarrow{F_A^G}) + R_S(S(\overrightarrow{r_{K/cm_S}^G})\cdot\overrightarrow{F_K^G})  \end{array}\right] + \left[\begin{array}{c} \overrightarrow{GRF^G}\\  R_F(\overrightarrow{r_{cop/cm_F}^G}\times \overrightarrow{GRF^G})\\  [0] \\  [0] \end{array}\right]
\end{equation}

Now it is possible to write the vector $F$ as multiplication of a matrix by a vector:

\begin{equation}
\left[\begin{array}{cccc} m_FI_3& [0]& [0]& [0]\\ [0]& I_F  & [0] & [0] \\ [0] &[0] &  m_SI_3& [0] \\ [0] & [0] & [0] & I_S\end{array}\right]\cdot\left[\begin{array}{c}\overrightarrow{a_{cm_F}^G}\\\overrightarrow{\dot{\omega_F^F}}\\\overrightarrow{a_{cm_S}^G}\\\overrightarrow{\dot{\omega_S^S}} \\ \end{array}\right] = \left[\begin{array}{c}[0]\\ - \overrightarrow{\omega_F^F} \times (I_F\overrightarrow{\omega_F^F})  \\ [0] \\ - \overrightarrow{\omega_S^S} \times (I_S\overrightarrow{\omega_S^S}) \end{array}\right] + \left[\begin{array}{c}  m_F\overrightarrow{g^G}\\ [0]\\  m_S\overrightarrow{g^G} \\  [0] \end{array}\right] + \left[\begin{array}{ccc}  I_3& [0]& [0]& [0]\\ R_FS\left(\overrightarrow{r_{A/cm_F}^G}\right)&I_3& [0]& [0]\\  -I_3& [0]& I_3 & [0] \\ -R_SS\left(\overrightarrow{r_{A/cm_S}^G}\right)& - R_SR_F^{-1} & R_SS\left(\overrightarrow{r_{K/cm_S}^G}\right) & I_3      \end{array}\right]\cdot\left[\begin{array}{c}  \overrightarrow{F_A^G}\\ \overrightarrow{M_A^F}\\  \overrightarrow{F_K^G}\\ \overrightarrow{M_K^S}\end{array}\right] + \left[\begin{array}{c} \overrightarrow{GRF^G}\\  R_F(\overrightarrow{r_{cop/cm_F}^G}\times \overrightarrow{GRF^G})\\  [0] \\  [0] \end{array}\right]
\end{equation}

So, the final equation to compute the forces and torques is obtained by multiplying everything by the inverse of the matrix multipliying the vector of forces:

\begin{equation}
\left[\begin{array}{c}  \overrightarrow{F_A^G}\\ \overrightarrow{M_A^F}\\  \overrightarrow{F_K^G}\\ \overrightarrow{M_K^S}\end{array}\right]  = \left[\begin{array}{ccc}  I_3& [0]& [0]& [0]\\ R_FS\left(\overrightarrow{r_{A/cm_F}^G}\right)&I_3& [0]& [0]\\  -I_3& [0]& I_3 & [0] \\ -R_SS\left(\overrightarrow{r_{A/cm_S}^G}\right)& - R_SR_F^{-1} & R_SS\left(\overrightarrow{r_{K/cm_S}^G}\right) & I_3      \end{array}\right]^{-1}\cdot\left(\left[\begin{array}{cccc} m_FI_3& [0]& [0]& [0]\\ [0]& I_F  & [0] & [0] \\ [0] &[0] &  m_SI_3& [0] \\ [0] & [0] & [0] & I_S\end{array}\right]\cdot\left[\begin{array}{c}\overrightarrow{a_{cm_F}^G}\\\overrightarrow{\dot{\omega_F^F}}\\\overrightarrow{a_{cm_S}^G}\\\overrightarrow{\dot{\omega_S^S}} \\ \end{array}\right] - \left[\begin{array}{c}[0]\\ - \overrightarrow{\omega_F^F} \times (I_F\overrightarrow{\omega_F^F})  \\ [0] \\ - \overrightarrow{\omega_S^S} \times (I_S\overrightarrow{\omega_S^S}) \end{array}\right] -\left[\begin{array}{c} \overrightarrow{GRF^G}\\  R_F(\overrightarrow{r_{cop/cm_F}^G}\times \overrightarrow{GRF^G})\\  [0] \\  [0] \end{array}\right] - \left[\begin{array}{c}  m_F\overrightarrow{g^G}\\ [0]\\  m_S\overrightarrow{g^G} \\  [0] \end{array}\right]\right)
\end{equation}

With the last equation, we can obtain all the forces and moments using only one line of code. Computationally, it is less prone to errors and more efficient.

So, generically, the steps to perform the analysis of inverse dynamics is:

- write the equations of Newton-Euler for each segment. Write explicitly the basis at which  each vector is described.

- use the rotation matrices of the basis to pass the description of a vector to another basis. Use it in a way that the same vector is described at just a single frame of reference.

- write the cross-products as a product between the skew-symmetric matrix $S$ of the first vector and the second vector.

- write the equations in the matrix format, repeated here:

\begin{equation}
    M(q)\ddot{q} = C(q,\dot{q}) + G(q) + Q +  E
\end{equation}

- write explicitly the vector containing the unknown forces and moments $Q$, as a multiplication of a matrix and vector containing only the unknown forces.

- isolate the vector containing only the unknown forces by multiplying the whole equation by the inverse of the matrix multiplying the vector with the forces.

## Problems

1) Solve the problems 18.3.20 and 18.3.24 of the Ruina and Rudra's book by using the Lagrangian formalism (it is much easier than use the Newton-Euler formalism) and then use the matrix formalism to obtain the expressions of the angular accelerations.

2) Write the matrices to find the forces and torques in a tridimensional double pendulum, consisted of two cylindrical bars. Consider that you know all the masses, moments of inertia, rotation matrices, accelerations, angular velocities and angular accelerations necessary to solve the problem.

## References

- YAMAGUCHI, G. T. Dynamic modeling of musculoskeletal motion: a vectorized approach for biomechanical analysis in three dimensions., 2001

- CRAIG, J. Introduction to robotics. , 1989

- JAIN, A. Robot and multibody dynamics. , 2011

- SPONG, M. W.; HUTCHINSON, S.; VIDYASAGAR, M. Robot modeling and control., 2006

- ERDEMIR, A. et al. Model-based estimation of muscle forces exerted during movements. Clinical Biomechanics, v. 22, n. 2, p. 131–154, 2007.

- STANEV, D.; MOUSTAKAS, K. Simulation of constrained musculoskeletal systems in task space. IEEE Transactions on Biomedical Engineering, v. 65, n. 2, p. 307–318, 2018.

- ZAJAC FE, GORDON ME , [Determining muscle's force and action in multi-articular movement](https://drive.google.com/open?id=0BxbW72zV7WmUcC1zSGpEOUxhWXM&authuser=0). Exercise and Sport Sciences Reviews, 17, 187-230. , 1989

- RUINA A, RUDRA P. [Introduction to Statics and Dynamics](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press. , 2015


## ILDEBRANDO MAGNANI

im975@nyu.edu

15/04/2016

## Scikit-Learn for Linear Regression, Cross-Validation and Ridge Regression:

Scikit-Learn:

*  Simple and efficient tools for data analysis and data mining
*  Accessible to everybody, and reusable in various contexts
*  Built on NumPy, SciPy, and matplotlib
*  Open source, commercially usable

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
from math import pi
```

We will begin by stating our "true model", defined  as $y = \cos(\pi x)$.

```python
def y(x):
    return np.cos(pi*x)
```

We now add some random "noise" to it in order to generate 25 data points, i.e., 25 $(x_i, y_i)$-tuples that will form our dataset. To do so, we have

\begin{equation}
y = \cos(\pi x) + \varepsilon,    \    where   \ x\sim U[-1,1] \ and \      \varepsilon\sim N(0,0.5)
\end{equation}

```python
x = np.linspace(-1, 1, 100)
X = np.random.uniform(-1, 1, 25)
X_data = X.reshape(25, 1)

y_obs_list = []

for i in range(len(X)):
    y_obs = y(X[i]) + np.random.normal(0, 0.5)
    y_obs_list.append(y_obs)

X_data = X.reshape(25, 1)
Y_data = np.asarray(y_obs_list).reshape(25, 1)
```

Here we want to plot the true model and the $(x_i, y_i)$-tuples we generated above.

```python
fig, ax = plt.subplots(figsize=(8,6))

ax.plot(x, y(x))
ax.scatter(X_data, Y_data)
```

Now, we use Scikit for Linear Regression. The module "LinearRegression" fits a linear model with coefficients $\beta = (\beta_1, \beta_2, \dots, \beta_p)$ to minimize the residual sum of squares between the observed responses in the dataset, and the responses predicted by the linear approximation. Mathematically it solves a problem of the form:

\begin{equation}
\min_{\beta} \lVert X\beta - y \rVert_{2}^2
\end{equation}

The vector of coefficients $\beta = (\beta_1, \beta_2, \dots, \beta_p)$ is designated as "coef\_" and $\beta_0$ as "intercept\_".

LinearRegression will take in its "fit" method arrays X, y and will store the coefficients $\beta$ of the linear model in its coef\_ member. By default, the "score" method returns the coefficient of determination $R^2$ of the prediction.

```python
from sklearn import linear_model
```

```python
regr = linear_model.LinearRegression()
regr.fit(X_data, Y_data)

print('intercept: \n', regr.intercept_)
print('coefficient: \n', regr.coef_)
print("Mean Squared Error: %.2f" % np.mean((regr.predict(X_data) - Y_data) ** 2))
print('R squared: %.2f' % regr.score(X_data, Y_data))
```

    intercept:
     [ 0.01233033]
    coefficient:
     [[ 0.13812041]]
    Mean Squared Error: 0.85
    R squared: 0.01

We now plot the results, which are obviously not very significant.

```python
fig, ax = plt.subplots(figsize=(8,8))

ax.plot(x, y(x), label = "True Model")
ax.scatter(X_data, Y_data, label = "Training Points")
ax.plot(X_data, regr.predict(X_data), label = "Regression Line")
plt.legend(loc='lower center')
plt.show
```

To make things more interesting, we are now going to use Scikit's linear models trained on nonlinear functions of the data. This approach maintains the generally fast performance of linear methods, while allowing them to fit a much wider range of data.

Here, we randomly generate a $(3\times2)$ matrix $X = [x_1, x_2]$ of observations, and we will transform it to $X = [1, x_1, x_2, x_{1}^2, x_1 x_2, x_{2}^2]$ using Scikit's "PolynomialFeatures" preprocessor. This preprocessor transforms an input data matrix into a new data matrix of a given degree. It can be used as follows:

```python
from sklearn.preprocessing import PolynomialFeatures
```

```python
X_data_example = np.random.random((3, 2))

print(X_data_example)

poly = PolynomialFeatures(degree=2)

poly_X_data_example = poly.fit_transform(X_data_example)
print(poly.fit_transform(X_data_example))
```

    [[ 0.82780005  0.94438455]
     [ 0.79859015  0.93094532]
     [ 0.20454604  0.37746037]]
    [[ 1.          0.82780005  0.94438455  0.68525291  0.78176157  0.89186217]
     [ 1.          0.79859015  0.93094532  0.63774623  0.74344377  0.8666592 ]
     [ 1.          0.20454604  0.37746037  0.04183908  0.07720802  0.14247633]]

This sort of preprocessing can be streamlined with the "Pipeline" tools. In fact, Pipeline can be used to chain multiple estimators into one. This is useful as there is often a fixed sequence of steps in processing the data, for example feature selection, normalization and classification. Pipeline serves two purposes here:

* Convenience: You only have to call fit and predict once on your data to fit a whole sequence of estimators.

* Joint parameter selection: You can grid search over parameters of all estimators in the pipeline at once.

Now, we are going to fit the same dataset with polynomials of different degrees (2, 5, 14) and plot results. We will compute the MSE of the predictor to see how well the polynomials do in their approximation.

```python
from sklearn.pipeline import make_pipeline

x_plot = np.linspace(-1, 1, 100).reshape(100,1)
regression = linear_model.LinearRegression()

for degree in [2, 5, 14]:
    # chain PolynomialFeatures and LinearRegression into one
    # estimator (make_pipeline is just a shorthand for Pipeline)
    model = make_pipeline(PolynomialFeatures(degree), regression)
    model.fit(X_data, Y_data)

    #predict using the linear model
    y_plot = model.predict(x_plot)

    #plot
    fig, ax = plt.subplots(figsize=(8,8))
    ax.plot(x, y(x), label = "True Model")
    ax.scatter(X_data, Y_data, label = "Training Points")
    ax.plot(x_plot, y_plot, label="Degree %d" % degree)
    plt.legend(loc='lower center')

    #compute the mean squared error (MSE)
    MSE = np.mean((model.predict(X_data) - Y_data) ** 2)
    print("Mean Squared Error for Degree %d" % degree, ":")
    print(MSE)
    plt.title("Degree {}\nMSE = {:.2e}".format(degree, MSE))

    plt.xlim(-1, 1)
    plt.ylim(-2, 2)

plt.show()
```

We can see that a linear function (polynomial with degree 1) is not sufficient to fit the training samples. This is called underfitting. A polynomial of degree 5 approximates the true function almost perfectly. For higher degrees the model will overfit the training data, i.e. it learns the noise of the training data.

Note: Learning the parameters of a prediction function and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data. This situation is called overfitting. To avoid it, it is common practice when performing a (supervised) machine learning experiment to hold out part of the available data as a test set "X_test", "Y_test".

In scikit-learn a random split into training and test sets can be quickly computed with the "train_test_split" helper function. We can quickly sample a training set while holding out 40% of the data for testing (evaluating) our classifier:

```python
from sklearn.cross_validation import train_test_split
```

```python
X_train, X_test, Y_train, Y_test = train_test_split(X_data, Y_data, test_size=0.4, random_state=0)
```

```python
X_train.shape, Y_train.shape
```

    ((15, 1), (15, 1))

```python
X_test.shape, Y_test.shape
```

    ((10, 1), (10, 1))

We can train a linear regression model on (X-train, Y-train) data, and then compute the Mean Squared Error on the (X-test, Y-test) set.

```python
regr = linear_model.LinearRegression()
regr.fit(X_train, Y_train)

# The coefficients
print('intercept: \n', regr.intercept_)
print('coefficient: \n', regr.coef_)
# The mean square error computed on validation sets
print("Mean Squared Error: %.2f" % np.mean((regr.predict(X_test) - Y_test) ** 2))
```

    intercept:
     [ 0.15169192]
    coefficient:
     [[ 0.2190235]]
    Mean Squared Error: 1.34

We can do the same with "polynomial fitting".

```python
regression = linear_model.LinearRegression()

for degree in [1, 2, 5, 14]:
    model = make_pipeline(PolynomialFeatures(degree), regression)
    model.fit(X_train, Y_train)
    print("MSE on VALIDATION SET for Degree %d" % degree, ":")
    print(np.mean((model.predict(X_test) - Y_test) ** 2))
```

    MSE on VALIDATION SET for Degree 1 :
    1.33851766521
    MSE on VALIDATION SET for Degree 2 :
    0.506630898514
    MSE on VALIDATION SET for Degree 5 :
    0.747112367299
    MSE on VALIDATION SET for Degree 14 :
    4826715154.43

What we observe is a giant increase in the MSE for the polynomial of 14th degree. This is an obvious consequence of overfitting, i.e., with a 14th degree polynomal we are minimizing empirical risk, and not the "true" risk.

By partitioning the available data into 2 sets, we drastically reduce the number of samples which can be used for learning the model, and the results can depend on a particular random choice for the pair of (train, validation) sets.

A solution to this problem is a procedure called cross-validation (CV for short). A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV. In the basic approach, called k-fold CV, the training set is split into k smaller sets (other approaches are described below, but generally follow the same principles). The following procedure is followed for each of the k “folds”:

*  A model is trained using k-1 of the folds as training data;
*  The resulting model is validated on the remaining part of the data (i.e., it is used as a test set to compute a  performance measure such as accuracy).

The performance measure reported by k-fold cross-validation is then the average of the values computed in the loop. This approach can be computationally expensive, but does not waste too much data (as it is the case when fixing an arbitrary test set), which is a major advantage in problem such as inverse inference where the number of samples is very small.

```python
from sklearn.cross_validation import cross_val_score
```

The simplest way to use cross-validation is to call the "cross_val_score" helper function on the estimator and the dataset. By default, the score computed at each CV iteration is the score method of the estimator. It is possible to change this by using the scoring parameter.

The following example demonstrates how to estimate the "cross-validated" MSE of LinearRegression() on (X-data,Y-data) by splitting the data, fitting a model on k-th training set and computing the MSE 5 consecutive times on the k-th test set (with different splits each time):

```python
scores = cross_val_score(linear_model.LinearRegression(), X_data, Y_data, cv = 5, scoring = 'mean_squared_error')
print("MSE array:", scores)
```

    MSE array: [-1.14612157 -1.47367855 -0.95452968 -0.72913903 -0.77076311]

When the cv argument is an integer, "cross_val_score uses" the KFold by default.

"The Mean Square Error returned by sklearn.cross_validation.cross_val_score is always a negative. While being a designed decision so that the output of this function can be used for maximization given some hyperparameters, it's extremely confusing when using cross_val_score directly".

https://github.com/scikit-learn/scikit-learn/issues/2439

From Scikit Documentation: "Whether score_func is a score function (default), meaning high is good,
or a loss function, meaning low is good. In the latter case, the scorer
object will sign-flip the outcome of the score_func".

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html

```python
from sklearn.metrics.scorer import SCORERS
print(SCORERS.keys())
```

    dict_keys(['recall_micro', 'mean_absolute_error', 'f1_samples', 'adjusted_rand_score', 'average_precision', 'recall', 'mean_squared_error', 'r2', 'precision_weighted', 'recall_weighted', 'accuracy', 'f1_macro', 'f1', 'precision', 'roc_auc', 'f1_micro', 'precision_samples', 'recall_samples', 'f1_weighted', 'log_loss', 'precision_micro', 'recall_macro', 'median_absolute_error', 'precision_macro'])

There are many built-in scoring metrics. However, we can always define our own metric:

```python
def my_own_scoring(regr, X_data, Y_data):
    return np.mean(regr.predict(X_data) == Y_data)

cross_val_score(regr, X_data, Y_data, cv = 5, scoring = my_own_scoring)
```

    array([ 0.,  0.,  0.,  0.,  0.])

Scikit-Learn provides many tools to generate indices that can be used to generate dataset splits according to different cross validation strategies.

For example, we have the "KFold", which divides all the samples in k groups of samples, called folds (if k = n, this is equivalent to the Leave One Out strategy), of equal sizes (if possible). The prediction function is learned using k - 1 folds, and the fold left out is used for test.

Example of 5-fold cross-validation on a dataset with 25 samples:

```python
from sklearn.cross_validation import KFold
```

```python
kf = KFold(n = len(X_data), n_folds = 5, shuffle = True)
MSE_kth_list = []
print("len(kf):", len(kf))

for train_index, test_index in kf:
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X_data[train_index], X_data[test_index]
    Y_train, Y_test = Y_data[train_index], Y_data[test_index]
    regr = linear_model.LinearRegression().fit(X_train, Y_train)
    MSE_kth = np.mean((regr.predict(X_test) - Y_test) ** 2)
    MSE_kth_list.append(MSE_kth)
```

    len(kf): 5
    TRAIN: [ 0  1  2  3  4  5  6  8  9 10 11 13 14 15 16 17 20 21 23 24] TEST: [ 7 12 18 19 22]
    TRAIN: [ 1  2  3  4  5  7  8  9 10 12 14 15 16 17 18 19 20 21 22 24] TEST: [ 0  6 11 13 23]
    TRAIN: [ 0  1  2  3  5  6  7  8 10 11 12 13 14 15 17 18 19 20 22 23] TEST: [ 4  9 16 21 24]
    TRAIN: [ 0  1  4  6  7  8  9 10 11 12 13 15 16 18 19 20 21 22 23 24] TEST: [ 2  3  5 14 17]
    TRAIN: [ 0  2  3  4  5  6  7  9 11 12 13 14 16 17 18 19 21 22 23 24] TEST: [ 1  8 10 15 20]

Each fold is constituted by two arrays: the first one is related to the training set, and the second one to the test set. Thus, one can create the training/test sets using numpy indexing.

We know print the MSE that results from the 5-folds average .

```python
print("Average k-fold MSE:", np.mean(MSE_kth_list))
```

    Average k-fold MSE: 0.996664633076

Another example of cross-validation strategy is the "LeaveOneOut" (or LOO, a simple cross-validation. Each learning set is created by taking all the samples except one, the test set being the sample left out. Thus, for n samples, we have n different training sets and n different tests set. This cross-validation procedure does not waste much data as only one sample is removed from the training set:

```python
from sklearn.cross_validation import LeaveOneOut
```

```python
loo = LeaveOneOut(len(X_data))

for train_index, test_index in loo:
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X_data[train_index], X_data[test_index]
    Y_train, Y_test = Y_data[train_index], Y_data[test_index]
```

    TRAIN: [ 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [0]
    TRAIN: [ 0  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [1]
    TRAIN: [ 0  1  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [2]
    TRAIN: [ 0  1  2  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [3]
    TRAIN: [ 0  1  2  3  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [4]
    TRAIN: [ 0  1  2  3  4  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [5]
    TRAIN: [ 0  1  2  3  4  5  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [6]
    TRAIN: [ 0  1  2  3  4  5  6  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [7]
    TRAIN: [ 0  1  2  3  4  5  6  7  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [8]
    TRAIN: [ 0  1  2  3  4  5  6  7  8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [9]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 11 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [10]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 12 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [11]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 13 14 15 16 17 18 19 20 21 22 23 24] TEST: [12]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 14 15 16 17 18 19 20 21 22 23 24] TEST: [13]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 15 16 17 18 19 20 21 22 23 24] TEST: [14]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 16 17 18 19 20 21 22 23 24] TEST: [15]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 17 18 19 20 21 22 23 24] TEST: [16]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 18 19 20 21 22 23 24] TEST: [17]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 19 20 21 22 23 24] TEST: [18]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 20 21 22 23 24] TEST: [19]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 21 22 23 24] TEST: [20]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 22 23 24] TEST: [21]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 23 24] TEST: [22]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 24] TEST: [23]
    TRAIN: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23] TEST: [24]

---------------------------------------------------------------------------------------------------------------------

We now turn to Ridge Regression.

Applying Tikhonov regularization argument to OLS, we can find a new (this time biased) estimator for $\beta$ coefficients that has always lower MSE. In particular, the estimator is designed to solve the problem

\begin{equation}
\min_{b\in\mathbb{R}} \left[\sum_{n=1}^{N} (y_n - \mathbf{x_n'b})^2 + \lambda \lVert\mathbf{b}\rVert_2^2\right]
\end{equation}

where $\lambda \geq 0$ is called "regularization parameter". In other words, we are minimizing empirical risk plus a regularization term that penalizes large values of $\lVert\mathbf{b}\rVert$. It can be shown that there always exists a $\lambda > 0$ such that the $MSE(\beta_{\lambda})$ < $MSE(\beta)$, even if $\beta_{\lambda}$ is always biased. "The reduction in mean squared error over the least squares estimator occurs because, for some intermediate
value of $\lambda$, the variance of $\beta_{\lambda}$ falls by more than enough to offset the extra bias" (Stachurski, Econometric Theory). The outcome of Ridge regression depends a lot on the non-trivial choice of the $\lambda$ value.

As with other linear models, Ridge will take in its fit method arrays $X$, $y$ and will store the coefficients $\beta$ of the linear model in its "coef\_" member:

```python
from sklearn.linear_model import Ridge
```

```python
Ridge_regr = Ridge(alpha=0.5)
Ridge_regr.fit(X_data, Y_data)
```

    Ridge(alpha=0.5, copy_X=True, fit_intercept=True, max_iter=None,
       normalize=False, random_state=None, solver='auto', tol=0.001)

```python
print('Intercept: \n', Ridge_regr.intercept_)
print('Coefficient: \n', Ridge_regr.coef_)
print("Mean Squared Error: %.2f" % np.mean((Ridge_regr.predict(X_data) - Y_data) ** 2))
print('R^2: %.2f' % Ridge_regr.score(X_data, Y_data))
```

    Intercept:
     [ 0.01240888]
    Coefficient:
     [[ 0.1301897]]
    Mean Squared Error: 0.85
    R^2: 0.01

RidgeCV implements ridge regression with built-in cross-validation of the alpha parameter. It defaults to Generalized Cross-Validation (GCV), an efficient form of leave-one-out (LOO) cross-validation:

```python
from sklearn.linear_model import RidgeCV
```

```python
# a list of possible lambda values
lambda_list = [0.1, 1, 5, 10]

RidgeCV_regr = RidgeCV(alphas = lambda_list, scoring = 'mean_squared_error')
RidgeCV_regr.fit(X_data, Y_data)
```

    RidgeCV(alphas=[0.1, 1, 5, 10], cv=None, fit_intercept=True, gcv_mode=None,
        normalize=False, scoring='mean_squared_error', store_cv_values=False)

```python
print("Cross-Validated alpha:", RidgeCV_regr.alpha_)
```

    Cross-Validated alpha: 10.0

---------------------------------------------------------------------------------------------------------------------

In this final example, we demonstrates the problems of underfitting and overfitting and how we can use Ridge regression with polynomial features to approximate nonlinear functions. The plot shows the function that we want to approximate, which is a part of the cosine function. In addition, the samples from the real function and the approximations of different models are displayed. The models have polynomial features of different degrees.

We are using Ridge regression throguh "RidgeCV", so we choose our regularization parameter $\lambda$ by means of cross-validation. We evaluate quantitatively overfitting / underfitting by using cross-validation. We calculate the mean squared error (MSE) on the validation set, the higher, the less likely the model generalizes correctly from the training data.

```python
from sklearn.pipeline import Pipeline
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import Ridge

x_plot = np.linspace(-1, 1, 100).reshape(100,1)

for degree in [1, 2, 5, 14]:
    model = make_pipeline(PolynomialFeatures(degree), RidgeCV_regr)
    model.fit(X_data, Y_data)
    print("Cross-Validated Lambda for degree: %d" % degree)
    print(RidgeCV_regr.alpha_)

    # Evaluate the models using crossvalidation
    scores = cross_val_score(model, X_data, Y_data, scoring="mean_squared_error", cv=10)

    y_plot = model.predict(x_plot)

    #plot
    fig, ax = plt.subplots(figsize=(8,8))
    ax.plot(x, y(x), label = "True Model")
    ax.scatter(X_data, Y_data, label = "Training Points")
    ax.plot(x_plot, y_plot, label="Degree %d" % degree)
    plt.legend(loc='lower center')
    print("Cross-Validated MSE:")
    print(-scores.mean())
    plt.title("Degree {}\nMSE = {:.2e}\n Chosen Lambda {}".format(degree, -scores.mean(), RidgeCV_regr.alpha_))
    plt.xlim(-1, 1)
    plt.ylim(-2, 2)

plt.show()
```

---------------------------------------------------------------------------------------------------------------------

# References:

* Econometric Theory, Stachurski
* Lectures on Scikit-Learn, Andreas Mueller
* Scikit-Learn Documentation


```python
%pylab inline
pylab.rcParams['figure.figsize'] = (16.0, 8.0)
```

    Populating the interactive namespace from numpy and matplotlib

# Summarizing the Monte Carlo output

The result of a Monte Carlo simulation is a set of samples from the probability distribution associated with the measurand
$$ \{\mathbf{y}^{(k)},\,k=1,\ldots,M\} $$

The aim is to derive characteristic information about the measurand from this set:

1) best estimate

2) uncertainty associated with the best estimate

3) intervals/regions with a prescribed coverage probability

## Univariate measurand

1) best estimate
\begin{align}
y =& mean \{ y^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M} \sum_{k=1}^M y^{(k)}
\end{align}
2) uncertainty associated with the best estimate
\begin{align}
u_y =& std\{ y^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M-1} \sum_{k=1}^M (y-y^{(k)})^2
\end{align}

### Exercise 4.1

Draw randomly from the normal distribution with mean $\mu=1.3$ and standard deviation $\sigma=0.4$ and calculate best estimate and its uncertainy using 100, 200, 300, ..., 100 000 draws. Compare estimate and uncertainty with mean and standard deviation of the original distribution for the different number of draws.

```python
from scipy.stats import norm

Y = norm.rvs(loc=1.3, scale=0.4, size=100000)

draws = arange(100,len(Y)+100,100)
means = array([Y[:M].mean() for M in draws])
uncs  = array([Y[:M].std() for M in draws])

figure()
subplot(121)
plot(draws, means)
axhline(1.3,color="k",linestyle="--")
subplot(122)
plot(draws, uncs)
axhline(0.4,color="k",linestyle="--");
```

Intervals with a prescribed coverage probability can be calculated from the Monte Carlo outcome as follows

1) Sort the Monte Carlo outcome in ascending order
``` python
    sort(Y)
```
2) For propability $P$ calculate the corresponding fraction of Monte Carlo draws $q=int(PM)$

3) Set lower bound of interval as $r=int(0.5(M-q))$ for a probabilistically symmetrical interval

4) Calculate upper bound of interval as $r+q$

### Exercise 4.2

Draw randomly from the distribution from Exercise 4.1 and calculate the 95% probabilistally symetrical coverage interval from 1000, 10000 and 100000 draws. Compare the result to the exact 95% interval.

```python
from scipy.stats import norm

P = 0.95 # sought probability of coverage interval

draws = 100000
Y = norm.rvs(loc=1.3, scale=0.4, size=draws)

# step 1 - sort random draws
Ys = sort(Y)
# step 2 - calculate portion of draws corresponding to P
q  = int(draws*P)
# step 3 - set lower bound of coverage interval
r  = int(0.5*(draws-q))
# interval bounds
ylow = Ys[r]
yhigh = Ys[r+q]
interval = [ylow, yhigh]
true_interval = [1.3 - 1.96*0.4, 1.3 + 1.96*0.4]

print("calculated interval")
print(interval)
print("true interval")
print(true_interval)

```

    calculated interval
    [0.51871191644717352, 2.082216089835121]
    true interval
    [0.516, 2.084]

## Multivariate measurand

1) best estimate
\begin{align}
\mathbf{y} =& mean \{ \mathbf{y}^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M} \sum_{k=1}^M \mathbf{y}^{(k)}
\end{align}
2) uncertainty associated with the best estimate
\begin{align}
U_\mathbf{y} =& cov\{ \mathbf{y}^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M-1} \sum_{k=1}^M (\mathbf{y}-\mathbf{y}^{(k)})(\mathbf{y}-\mathbf{y}^{(k)})^T
\end{align}

### Exercise 4.3

Draw randomly from the normal distribution with mean
$$\mathbf{\mu}=\left( \begin{array}{c}
0.4 \\ -1.5
\end{array}\right)
$$
and covariance
$$
\Sigma=\left(\begin{array}{cc}
0.09 & -0.2 \\ -0.2 & 1.44
\end{array}\right)
$$
and calculate best estimate and its uncertainy using 1000, 10000 and 100 000 draws. Compare estimate and uncertainty with mean and covariance of the original distribution for the different number of draws.

```python
from scipy.stats import multivariate_normal

Y = multivariate_normal.rvs(mean=[0.4, -1.5],
                            cov = array([[0.09, -0.2],[-0.2, 1.44]]),
                            size = 100000)
draws = [1000, 10000, 100000]
means = array([Y[:M].mean(axis=0) for M in draws])
uncs  = array([cov(Y[:M], rowvar=0) for M in draws])

print("calculated estimates")
print(means)
print("calculated covariances")
print(uncs)
```

    calculated estimates
    [[ 0.40676864 -1.48146002]
     [ 0.39609434 -1.48938041]
     [ 0.40065093 -1.49893305]]
    calculated covariances
    [[[ 0.0858938  -0.19229542]
      [-0.19229542  1.41033116]]

     [[ 0.08787505 -0.19790337]
      [-0.19790337  1.45852861]]

     [[ 0.08906988 -0.19908451]
      [-0.19908451  1.4405801 ]]]

Regions with a prescribed coverage probability can be calculated from the multivariate Monte Carlo outcome as follows

1) Calculate the Cholesky decomposition of the sample covariance matrix $U_{\mathbf{y}}=\mathbf{LL}^T$

2) Transform the Monte Carlo outcomes
$$ \mathbf{y}_{(k)} = \mathbf{L}^{-1}(\mathbf{y}^{(k)}-\mathbf{y})$$
and sort according to the distance measure
$$ d^2_{(k)} = \mathbf{y}_{(k)}^T\mathbf{y}_{(k)} $$

3) calculate $k_P$ such that a fraction $P$ of all Monte Carlo outcomes satisfies $d_{(k)}<k_P$

This defines the ellipsoidal region $(\mathbf{\eta}-\mathbf{y})^TU_{\mathbf{y}}^{-1}(\mathbf{\eta}-\mathbf{y})<k^2_P$

For a bivariate normal distribution, the factor for a 95% coverage ellipsoidal region is given as the 95% quantile of the $\chi^2$ distribution with 2 degrees of freedom.

### Exercise 4.4

Calculate 100 000 random draws from the distribution from Exercise 4.3 and calculate the 95% coverage region. Compare to the true 95% coverage region.

```python
def calculate_ellipse(mu, Sigma, kP):
    vals, vecs = linalg.eigh(Sigma)
    order = vals.argsort()[::-1]
    vals = vals[order]
    vecs = vecs[:,order]
    theta = degrees(np.arctan2(*vecs[:,0][::-1]))
    width, height = kP * sqrt(vals)
    return width, height, theta

```

```python
from scipy.stats import multivariate_normal, chi2
from matplotlib.patches import Ellipse

mu = array([0.4, -1.5])
Sigma = array([[0.09, -0.2],[-0.2, 1.44]])

dist = multivariate_normal(mean = mu, cov = Sigma)
P = 0.95
draws = 10000
q = int(P*draws)

Y = multivariate_normal.rvs(mean = mu, cov = Sigma, size = draws)

y = Y.mean(axis=0)
Uy= cov(Y, rowvar=0)

# step 1 - Cholesky decomposition of covariance matrix
L = cholesky(Uy)
# step 2 - Transform the Monte Carlo outcomes
Ys = linalg.solve(L, (Y - tile(y,(draws,1))).T).T
d = diag(dot(Ys,Ys.T))
# and sort distances
ds = sort(d)
# step 3 - calculate kP as the scale parameter of the ellipse
kP = ds[q]
width, height, theta = calculate_ellipse(y, Uy, kP)

```

```python
figure()
scatter(Y[:,0],Y[:,1],edgecolor="none",alpha=0.5)
ellip=Ellipse(xy=y, width=width, height=height, angle=theta,facecolor="none")
gca().add_artist(ellip)
w,h,t = calculate_ellipse(mu, Sigma, chi2.ppf(0.95,2))
ellip_true = Ellipse(xy=mu,width=w,height=h,angle=t,facecolor="none",edgecolor="r")
gca().add_artist(ellip_true);
```

05 Summarizing the Monte Carlo output.ipynb