# Lecture 6: Matrix decompositions

## Syllabus
**Week 1:** Python intro
**Week 2:** Matrices, vectors, norms, ranks
**Week 3:** Linear systems, eigenvectors, eigenvalues
**Week 4:** Matrix decompositions: LU, QR, SVD + test

##Recap of the previous lecture
- Eigenvectors and their applications (Schrodinger + PageRank)
- Gershgorin circles
- Computing eigenvectors using power method, $x := Ax$
- Schur theorem, $A = U T U^*$
- Normal matrices $AA^* = A^* A$.

## Today lecture
Today we will talk about the matrix factorizations
- LU decomposition and Gaussian elimination
- QR decomposition and Gram-Schmidt orthogonalization

##What is the LU factorization

From the title, it is clear, that LU factorization is the representation of a given matrix $A$ as a product
of a **lower triangular matrix** $L$ and an **upper triangular** matrix $U$.
The matrix $L$ has to be non-singular with ones on the diagonal, and the matrix $U$ has to be non-singular.

**Main goal** of the LU decomposition is to solve linear systems, because
$$
    A^{-1} f = (L U)^{-1} f = U^{-1} L^{-1} f,
$$
and this reduces to the solution of a linear system
$$
     L y = f,
$$
and
$$
   U x = y.
$$
Solving linear systems with triangular matrices is **easy**: you just solve the equation with one unknown, put into the second equation, and so on. The computational cost of such algorithm in $\mathcal{O}(n^2)$ arithmetic operations, once $LU$ factorization is known.

Note that the inverse of triangular matrix is a triangular matrix.

## LU decomposition: the algorithm
We can do some **symbolic computations** using the Python sympy package for our Hilbert matrix, to illustrate how LU decomposition looks like.

```
import numpy as np
import sympy
from sympy.matrices import Matrix
import IPython
sympy.init_printing(use_latex=True)
n = 5
w = Matrix(n, n, lambda i, j: 1/(i + j +  sympy.Integer(1)/2))
L, U, tmp = w.LUdecomposition()
#Generate the final expression
expr = (L + U)
fn = '%s \\times %s = %s' % (sympy.latex(L), sympy.latex(U), sympy.latex(w))
IPython.display.Math(fn)
```

$$\left[\begin{matrix}1 & 0 & 0 & 0 & 0\\\frac{5}{7} & 1 & 0 & 0 & 0\\\frac{5}{9} & \frac{14}{11} & 1 & 0 & 0\\\frac{5}{11} & \frac{189}{143} & \frac{9}{5} & 1 & 0\\\frac{5}{13} & \frac{84}{65} & \frac{198}{85} & \frac{44}{19} & 1\end{matrix}\right] \times \left[\begin{matrix}\frac{2}{5} & \frac{2}{7} & \frac{2}{9} & \frac{2}{11} & \frac{2}{13}\\0 & \frac{8}{441} & \frac{16}{693} & \frac{24}{1001} & \frac{32}{1365}\\0 & 0 & \frac{128}{127413} & \frac{128}{70785} & \frac{256}{109395}\\0 & 0 & 0 & \frac{512}{8690825} & \frac{2048}{15011425}\\0 & 0 & 0 & 0 & \frac{32768}{9256590525}\end{matrix}\right] = \left[\begin{matrix}\frac{2}{5} & \frac{2}{7} & \frac{2}{9} & \frac{2}{11} & \frac{2}{13}\\\frac{2}{7} & \frac{2}{9} & \frac{2}{11} & \frac{2}{13} & \frac{2}{15}\\\frac{2}{9} & \frac{2}{11} & \frac{2}{13} & \frac{2}{15} & \frac{2}{17}\\\frac{2}{11} & \frac{2}{13} & \frac{2}{15} & \frac{2}{17} & \frac{2}{19}\\\frac{2}{13} & \frac{2}{15} & \frac{2}{17} & \frac{2}{19} & \frac{2}{21}\end{matrix}\right]$$

A simple implementation of the LU factorization can be done in 3 cycles:

```
L = Matrix(n, n, lambda i, j: 0)
U = Matrix(n, n, lambda i, j: 0)
a = w.copy() #Our matrix
for k in xrange(n): #Eliminate one row

    L[k, k] = 1
    for i in xrange(k+1, n):
        L[i, k] = a[i, k] / a[k, k]
        for j in xrange(k+1, n):
            a[i, j] = a[i, j] - L[i, k] * a[k, j]
    for j in xrange(k, n):
        U[k, j] = a[k, j]
```

```
U
```

## Strictly regular matrices and LU decomposition
A matrix $A$ is strictly regular, if all of its **principal minors** (i.e, submatrices in the first $k$ rows and $k$ columns) are non-singular. In this case, there always exists a LU-decomposition.  The reverse is also true.

**Proof**. If there is a LU-decomposition, then the matrix is strictly regular -- this follows from the fact that to get a minor you multiply a corresponding submatrix of $L$ by a corresponding submatrix of $U$, and they are non-singular (since non-singularity of triangular matrices is equivalent to the fact that their diagonal elements are not equal to zero).

The other way can be proven by **induction**. Suppose that we know that for all $(n-1) \times (n-1)$ matrices will non-singular minors LU-decomposition exists.

Then, consider the block form
$$
    A = \begin{bmatrix}
          a & c^{\top} \\
          b & D
    \end{bmatrix},
$$
where $D$ is $(n-1) \times (n-1)$.

Then we do "block elimination":
$$
     \begin{bmatrix}
     1 & 0 \\
     -z & I
     \end{bmatrix}
     \begin{bmatrix}
     a & c^{\top} \\
     b & D
     \end{bmatrix}=
     \begin{bmatrix}
     a & c^{\top} \\
     0 & A_1
     \end{bmatrix},
$$
where $z = \frac{b}{a}, \quad A_1 = D - \frac{1}{a} b c^{\top}$.
We can show that $A_1$ is also strictly regular, thus it has (by induction) the LU-decomposition:
$$
   A_1 = L_1 U_1,
$$

And then
$$
   L = \begin{bmatrix}
    1 & 0 \\
    z & L_1
     \end{bmatrix}, \quad
     U = \begin{bmatrix}
     a & c^{\top} \\
     0 & U_1
     \end{bmatrix}
$$
we get $LA = U$ and $L$ and $U$ have the necessary triangular structure.

## When LU fails
What happens, if the matrix is not strictly regular (or the **pivots** in the Gaussian elimination are really small?). There is classical $2 \times 2$ example of a bad LU decomposition.  The matrix we look at is
$$
    A = \begin{pmatrix}
    \varepsilon & 1 \\
    1 & 1
    \end{pmatrix}
$$
If $\varepsilon$ is sufficiently small, we **might** fail.

```
import numpy as np
eps = 1.1 * 1e-16
a = [[eps, 1],[1,  1]]
a = np.array(a)
a0 = a.copy()
n = a.shape[0]
L = np.zeros((n, n))
U = np.zeros((n, n))
for k in xrange(n): #Eliminate one row
    L[k, k] = 1
    for i in xrange(k+1, n):
        L[i, k] = a[i, k] / a[k, k]
        for j in xrange(k+1, n):
            a[i, j] = a[i, j] - L[i, k] * a[k, j]
    for j in xrange(k, n):
        U[k, j] = a[k, j]

print 'L * U - A:'
print np.dot(L, U) - a0
print L
```

    L * U - A:
    [[ 0.  0.]
     [ 0.  1.]]
    [[  1.00000000e+00   0.00000000e+00]
     [  9.09090909e+15   1.00000000e+00]]

## The concept of pivoting
We can do pivoting, i.e. permute rows and columns to maximize $A_{kk}$ that we divide over.
The simplest but effective strategy is the **row pivoting**: at each step, select the index that is maximal in modulus, and put it onto the diagonal.

It gives us the decomposition

$$A = P L U,$$
where $P$ is a **permutation matrix**.

What makes pivoting good?
It is made good by the fact that

$$
   | L_{ij}|<1,
$$
but the elements of $U$ can grow, up to $2^n$! (in practice, this is very rarely encountered).

## Cholesky factorization
A very important class of matrices is the class of **symmetric positive definite matrices**.
By positive definiteness we mean that the associated **quadratic form**  is always positive
$$
   (Ax, x) > 0, \quad A > 0
$$

In this case, the LU-factorization always exists and moreover, it can be written in the form of **Cholesky factorization**:
$$
   A = R R^*
$$
And we always can select pivots on the diagonal!

## Summary on the LU for dense matrices
- Principal submatrices non-singular - there exists LU
- Pivoting is needed to avoid error growth
- Complexity is $\mathcal{O}(n^3)$ for the factorization step (can be speeded by using Strassen ideas) and $\mathcal{O}(n^2)$ for a solve

## LU-factorization for sparse matrices
The $\mathcal{O}(n^3)$ cost can be reduced, if the matrix $A$ is sparse.
For example, for a three-diagonal matrix, the factors $L$ and $U$ will be bidiagonal, and could be computed in $\mathcal{O}(n)$ cost.
Some demo...

```
import numpy as np
import scipy as sp
from scipy.sparse import csc_matrix, linalg as sla
%matplotlib inline
import matplotlib.pyplot as plt
plt.xkcd()
#A = csc_matrix([[1,2,0,4],[1,0,0,1],[1,0,2,1],[2,2,1,0.]])

n = 10
ex = np.ones(n);
lp1 = sp.sparse.spdiags(np.vstack((ex,  -2*ex, ex)), [-1, 0, 1], n, n, 'csc');
e = sp.sparse.eye(n)
A = sp.sparse.kron(lp1, e) + sp.sparse.kron(e, lp1)
A = csc_matrix(A)

fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].spy(A)
ax[0].set_title('Sparsity pattern of a matrix')
lu = sla.splu(A)
ax[1].spy(lu.L)
ax[1].set_title('Sparsity of an L factor')
```

```

```

    <5x5 sparse matrix of type '<type 'numpy.float64'>'
    	with 13 stored elements in Compressed Sparse Column format>

## QR-decomposition
The next decomposition: **QR** decomposition. Again from the name it is clear that a matrix is represented as a product
$$
    A = Q R,
$$
where $Q$ is an **orthogonal (unitary)** matrix and $R$ is **upper triangular**.
The matrix sizes: $Q$ is $n \times m$, $R$ is $m \times m$.

This algorithm plays a crucial role in many problems:
- Computing orthogonal bases ior a linear space
- Used in the preprocessing step for the SVD
- QR-algorithm for the computation of eigenvectors and eigenvalues (one of the 10 most important algorithms of the 20th century) is    based on the QR-decomposition

## Theorem
Every rectangular $n \times m$ matrix, $n \geq m$ matrix has a QR-decomposition.
There are several ways to prove it and compute it.

- (mathematical) Using the Gram matrices and Cholesky factorization
- (geometrical) Using the Gram-Schmidt orthogonalization
- (practical) Using Householder/Givens transformations

## Mathematical way
$$A = QR,$$
then $A^* A = R^* R$, the matrix $A^* A$ is called **Gram matrix**, and its elements are scalar products of the columns of $A$.

It is simple to show that $A^* A$ is positive definite:
$$
   (A^* A y, y) = (Ay, Ay) = \Vert Ay \Vert^2  \geq 0.
$$
Therefore, $A^* A = R^* R$ always exists.
Then the matrix $A R^{-1}$ is orthogonal:
$$
   (A R^{-1})^* (AR^{-1})= R^{-*} A^* A R^{-1} = R^{-*} R^* R R^{-1} = I.
$$

**Gram matrices are not good for numerical stability**!

Let us show that on random and Hilbert matrices :)
The Gram-matrix method may lead to the loss of orthogonality.

```
import numpy as np
n = 10
r = 5
#a = np.random.randn(n, r)
a = [[1.0/(i+j+0.5) for i in xrange(r)] for j in xrange(n)]
a = np.array(a)
q, Rmat = np.linalg.qr(a)
e = np.eye(r)
print 'Built-in QR orth', np.linalg.norm(np.dot(q.T, q) - e)
gram_matrix = a.T.dot(a)
Rmat1 = np.linalg.cholesky(gram_matrix)
q1 = np.dot(a, np.linalg.inv(Rmat1.T))
print 'Via Gram matrix:', np.linalg.norm(np.dot(q1.T, q1) - e)
```

    Built-in QR orth 8.10893700925e-16
    Via Gram matrix: 1.78151575967e-08

## Second way: Gram-Schmidt orthogonalization
QR-decomposition is a mathematical way of writing down the Gram-Schmidt orthogonalization process.
Given a sequence of vectors $a_1, \ldots, a_m$ we want to find orthogonal basis $q_1, \ldots, q_m$ such that every $a_i$ is a linear combination of such vectors.

**Gram-Schmidt:**
1. $q_1 := a_1/\Vert a_1 \Vert$
2. $q_2 := a_2 - (a_2, q_1) q_1, \quad q_2 := q_2/\Vert q_2 \Vert$
3. $q_3 := a_3 - (a_3, q_1) q_1 - (a_3, q_2) q_2), \quad q_2 := q_3/\Vert q_3 \Vert$
4. And go on
Note that the transformation from $Q$ to $A$ has triangular structure, since from the $k$-th vector we subtract only the previous ones.

## Gram-Schmidt is unstable
Gram-Schmidt can be very unstable (i.e., the produced vectors will be not orthogonal, especially if $q_k$ has small norm).
This is called **loss of orthogonality**.
There is a remedy, called **modified Gram-Schmidt** method.
Instead of doing
$$q_k := a_k - (a_k, q_1) q_1 - \ldots - (a_k, q_{k-1}) q_{k-1}$$
we do it step-by-step. First we set $q_k := a_k$ and orthogonalize sequentially:
$$
   q_k := q_k - (q_k, q_1)q_1, \quad q_k := q_{k} - (q_k,q_2)q_2, \ldots
$$
In exact arithmetic, it is the same. In floating point it is absolutely different!

Note that the complexity in $\mathcal{O}(nm^2)$ operations

## QR-decomposition: the (almost) practical way

If $A = QR$, then
$$
R = Q^* A,
$$
and we need to find a certain orthogonal matrix $Q$ that brings a matrix into orthogonal form.
For simplicity, we will look for an $n \times n$ matrix such that
$$
   Q^* A = \begin{bmatrix}
   * & * & *  \\
   0 & * & * \\
   0 & 0 & * \\
   &0_{(n-m) \times m}
   \end{bmatrix}
$$

We will do it column-by-column.
First, we find a Householder matrix $H_1 = (I - 2 uu^{\top})$ such that (we illustrate on a $4 \times 3$ matrix)

$$
   H_1 A = \begin{bmatrix}
    * & * & * \\
    0 & * & * \\
    0 & * & * \\
    0 & * & *
   \end{bmatrix}
$$

Then,
$$
   H_2 H_1 A = \begin{bmatrix}
    * & * & * \\
    0 & * & * \\
    0 & 0 & * \\
    0 & 0 & *
   \end{bmatrix},
$$
where
$$
  H_2 = \begin{bmatrix}
  1 & 0 \\
  0 & H'_2,
  \end{bmatrix}
$$
and $H'_2$ is a $3 \times 3$ Householder matrix.

Finally,
$$
   H_3 H_2 H_1 A = \begin{bmatrix}
    * & * & * \\
    0 & * & * \\
    0 & 0 & * \\
    0 & 0 & 0
   \end{bmatrix},
$$

A simple implementation of the Householder method in Python looks as follows.

```
#Initialization
import numpy as np
n = 4
r = 3
a = np.random.randn(n, r)
k = 0
```

```
def householder(x):
    u = x.copy()
    u = u / np.linalg.norm(u)
    u[0] = u[0] - 1 #Or plus
    u = u / np.linalg.norm(u)
    u = np.reshape(u, (-1, 1))
    return u

x = a[:, k:k+1]
u0 = householder(a[k:, k:k+1])
u = np.zeros((n, 1))
u[k:] = u0
H = np.eye(n) - 2 * u.dot(u.T)
a = H.T.dot(a)
print a
k = k + 1

```

    [[  2.55804239e+00  -3.02487125e-01   1.22104370e-01]
     [  2.18724250e-16   1.12507545e+00  -3.09474748e-01]
     [ -2.64480936e-17  -3.45664708e-16   4.80083115e-01]
     [  2.76406866e-17   1.07140777e-16  -5.55111512e-17]]

## The QR algorithm
The QR algorithm (Kublanovskaya and Francis independently proposed it in 1961). Do not **mix** QR algorithm and QR decomposition.
QR-decomposition is the representation of a matrix, whereas QR algorithm uses QR decomposition computes the eigenvalues!

Computation of the eigenvectors and eigenvalues of a matrix.

- Initialization: $$A_0 := A$$
- Iteration step: $$Q_k R_k = A_k, \quad A_{k+1} = R_k Q_k,$$
so we compute the QR-factorization of a matrix, and then multiply them in the opposite order.
Good news: the spectrum of $A_k$ is the same as of $A_0$, and $A_k$ converges to a triangular matrix!
It is too tempting not to implement it :)

```
import numpy as np
n = 4
a = [[1.0/(i + j + 0.5) for i in xrange(n)] for j in xrange(n)]
niters = 10
for k in xrange(niters):
    q, rmat = np.linalg.qr(a)
    a = rmat.dot(q)
    print 'current a:'
    print a

```

    current a:
    [[  2.40047183e+00   1.43485636e-01   4.99605047e-03  -5.56291523e-05]
     [  1.43485636e-01   3.59286592e-01   1.60534687e-02  -1.94225770e-04]
     [  4.99605047e-03   1.60534687e-02   1.60692682e-02  -2.80885670e-04]
     [ -5.56291523e-05  -1.94225770e-04  -2.80885670e-04   2.40684296e-04]]
    current a:
    [[  2.41031150e+00   2.09437993e-02   3.18722176e-05   5.45279213e-09]
     [  2.09437993e-02   3.50196113e-01   6.87806955e-04   1.28079871e-07]
     [  3.18722176e-05   6.87806955e-04   1.53250849e-02   4.17732654e-06]
     [  5.45279205e-09   1.28079871e-07   4.17732654e-06   2.35678648e-04]]
    current a:
    [[  2.41051991e+00   3.04114601e-03   2.02621961e-07  -5.33227597e-13]
     [  3.04114601e-03   3.49989111e-01   3.00995528e-05  -8.62074652e-11]
     [  2.02621961e-07   3.00995528e-05   1.53236760e-02  -6.42429491e-08]
     [ -5.33147641e-13  -8.62074827e-11  -6.42429491e-08   2.35677492e-04]]
    current a:
    [[  2.41052431e+00   4.41545673e-04   1.28806658e-09   1.32059800e-16]
     [  4.41545673e-04   3.49984720e-01   1.31786000e-06   5.80333420e-14]
     [  1.28806664e-09   1.31786000e-06   1.53236733e-02   9.88053919e-10]
     [  5.21260158e-17   5.80510077e-14   9.88053906e-10   2.35677492e-04]]
    current a:
    [[  2.41052440e+00   6.41081268e-05   8.18816607e-12  -7.99356474e-17]
     [  6.41081268e-05   3.49984627e-01   5.77009674e-08  -2.14109052e-17]
     [  8.18822358e-12   5.77009674e-08   1.53236733e-02  -1.51962427e-11]
     [ -5.09637195e-21  -3.90911829e-17  -1.51962302e-11   2.35677492e-04]]
    current a:
    [[  2.41052440e+00   9.30787458e-06   5.19949264e-14   7.99300814e-17]
     [  9.30787458e-06   3.49984626e-01   2.52637036e-09  -1.76560777e-17]
     [  5.20524342e-14   2.52637031e-09   1.53236733e-02   2.33729939e-13]
     [  4.98273388e-25   2.63237618e-20   2.33717423e-13   2.35677492e-04]]
    current a:
    [[  2.41052440e+00   1.35141258e-06   2.73389069e-16  -7.99300126e-17]
     [  1.35141258e-06   3.49984625e-01   1.10614263e-10   1.76826923e-17]
     [  3.30896669e-16   1.10614211e-10   1.53236733e-02  -3.60708065e-15]
     [ -4.87162969e-29  -1.77262591e-23  -3.59456477e-15   2.35677492e-04]]
    current a:
    [[  2.41052440e+00   1.96211922e-07  -5.54040645e-17   7.99300027e-17]
     [  1.96211922e-07   3.49984625e-01   4.84316731e-12  -1.76827548e-17]
     [  2.10350596e-18   4.84311567e-12   1.53236733e-02   6.78001457e-17]
     [  4.76300288e-33   1.19367537e-26   5.52842647e-17   2.35677492e-04]]
    current a:
    [[  2.41052440e+00   2.84880569e-08  -5.74941943e-17  -7.99300013e-17]
     [  2.84880568e-08   3.49984625e-01   2.12101875e-13   1.76827614e-17]
     [  1.33719609e-20   2.12050235e-13   1.53236733e-02  -1.33661508e-17]
     [ -4.65679822e-37  -8.03813647e-30  -8.50269816e-19   2.35677492e-04]]
    current a:
    [[  2.41052440e+00   4.13618795e-09  -5.75074807e-17   7.99300011e-17]
     [  4.13618792e-09   3.49984625e-01   9.33601458e-15  -1.76827623e-17]
     [  8.50053870e-23   9.28437503e-15   1.53236733e-02   1.25289581e-17]
     [  4.55296169e-41   5.41283161e-33   1.30771163e-20   2.35677492e-04]]

## Take home message
- LU decomposition, Cholesky, their complexity is $\mathcal{O}(n^3)$, used to solve dense linear systems
- QR decomposition, find orthogonal basis, the basic component for the computation of eigenvalues and eigenvectors

## Next time
- How do we compute LU factorizations for large-scale problems?
- And: the SVD (with some examples)

```
n = 100
a = np.random.randn(n, n)
%timeit np.linalg.qr(a)
%timeit scipy.linalg.lu(a)
```

    1000 loops, best of 3: 777 µs per loop
    1000 loops, best of 3: 218 µs per loop

```
%timeit l, u = np.linalg.cholesky?
```

```
import scipy

```

##### Questions?

```
from IPython.core.display import HTML
def css_styling():
    styles = open("./styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        /*width:80%;*/
        /*margin-left:auto !important;
        margin-right:auto;*/
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 1.2;
        font-size: 160%;
        /*width:70%;*/
        /*margin-left:auto;*/
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 110%;
        color:#CD2305;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #CD2305;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


$\newcommand{\myvector}[2]{\left\{ \begin{array}{c}#1\\#2\end{array} \right\}}$
$\newcommand{\mymatrix}[4]{\left[ \begin{array}{c}#1&#2\\#3&#4\end{array} \right]}$
$\newcommand{\sM}[1]{\boldsymbol{s}_{\mathrm{M#1}}}$
$\newcommand{\rvector}[1]{\boldsymbol{r}_{\mathrm{#1}}}$

# APPENDIX D:  Bayesian and least-squares results divergence

This Appendix describes a minimal inverse kinematics model for which Bayesian and least-sqaures results diverge.

### Model

The planar mechanism depicted below has two degrees of freedom (DOF): one translational and one rotational. The first (slider) segment moves to a position $y$ and the second segment rotates about point A to an angle $\theta$. Two markers (M1 and M2) are rigidly fixed to the rotating segment and have fixed local positions:  $\sM{1} = \myvector{35}{0}$ and $\sM{2} = \myvector{45}{0}$.

The three points' true global positions are:

\begin{align}
\rvector{A} &= \myvector{0}{y}\\ \\
\rvector{M1} &= \rvector{A} + \boldsymbol{R} \ \sM{1}\\ \\
\rvector{M2} &= \rvector{A} + \boldsymbol{R} \ \sM{2}
\end{align}

where $\boldsymbol{R}$ is the rotation matrix:

\begin{equation}
\boldsymbol{R} = \mymatrix{\cos\theta}{-\sin\theta}{\sin\theta}{\cos\theta}
\end{equation}

### Inverse kinematics (IK) problem

Imagine that we have measured the following marker positions:

\begin{align}
\rvector{M1}' &= \myvector{33.51}{12.11}\\ \\
\rvector{M2}' &= \myvector{42.63}{16.18}
\end{align}

The IK problem is to estimate $y$ and $\theta$ given these measurements.

For argument's sake let's say we also happen to know that the true values of the unknown variables are: $y$ = 0 and $\theta$ = 20 deg. In this case the true global marker positions are:

\begin{align}
\rvector{M1} &= \myvector{32.89}{11.97}\\ \\
\rvector{M2} &= \myvector{42.29}{15.39}
\end{align}

Let's assemble everything we know into Python for later use in our IK solutions.

```python
%matplotlib notebook

from math import sin,cos,radians,degrees
import numpy as np
from matplotlib import pyplot

#(0) Define known values:
### local positions:
sM1        = np.array([35.0, 0.0])
sM2        = np.array([45.0, 0.0])
### measurements:
rpM1       = np.array([33.51, 12.11])
rpM2       = np.array([42.63, 16.18])
### true values: (not used in IK solutions, but useful for visualization)
y_true     = 0.0
theta_true = radians(20)

#(1) Compute global marker positions given y and theta:
def rotation_matrix(theta):
    '''Construct rotation matrix'''
    c,s    = cos(theta), sin(theta)
    R      = np.matrix(  [[c, -s], [s, c]] )
    return R

def rotate(R, r):
    '''Rotate a position vector r using rotation matrix R'''
    return np.asarray(  (R * np.mat(r).T)  ).flatten()

def get_positions(y, theta):
    '''Compute global marker positions given y and theta'''
    rA     = np.array([0, y])       #global A position
    R      = rotation_matrix(theta)
    rM1    = rA + rotate(R, sM1)    #global M1 position
    rM2    = rA + rotate(R, sM2)    #global M2 position
    return rM1,rM2

rM1,rM2    = get_positions(y_true, theta_true)

#(2) Plot:
pyplot.figure()
ax = pyplot.axes()
ax.plot(rM1[0],  rM1[1],  'o', markeredgecolor='k', markerfacecolor='w', ms=40)
ax.plot(rM2[0],  rM2[1],  'o', markeredgecolor='k', markerfacecolor='w', ms=40)
ax.plot(rpM1[0], rpM1[1], 'o', markeredgecolor='r', markerfacecolor='r', ms=20)
ax.plot(rpM2[0], rpM2[1], 'o', markeredgecolor='r', markerfacecolor='r', ms=20)
ax.text(33, 11, 'True position',     color='k', size=14)
ax.text(34, 12, 'Measured position', color='r', size=14)
ax.set_xlabel('Global X position  (mm)', size=18)
ax.set_ylabel('Global Y position  (mm)', size=18)
pyplot.axis('equal')
pyplot.show()
```

    <IPython.core.display.Javascript object>

# Least-squares solution

This IK problem is relatively simple, so it could be solved using an analytical approach similar to the one presented in Appendix B. However, we'll be a bit lazy and instead solve the IK problem numerically, using the "get_positions" function that we've already written above.

Once we set values for $y$ and $\theta$ we know the global positions of $\rvector{M1}$ and $\rvector{M2}$, so we can compute the difference between those positions and our measurements as:

\begin{align}
\boldsymbol{\epsilon}_1 &= \rvector{M1}' - \rvector{M1}\\
\boldsymbol{\epsilon}_2 &= \rvector{M2}' - \rvector{M2}
\end{align}

We can then express total measurement error as follows:

\begin{equation}
f(y, \theta) =
\big| \ \boldsymbol{\epsilon}_1 \ \big| \ ^2 \
+ \
\big| \ \boldsymbol{\epsilon}_2 \ \big| \ ^2
\end{equation}

Although we have called $f(y, \theta)$ a "measurement error" function, the opposite perspective is equally apt: "guess error".  That is, if we know our measurement values ($\rvector{M1}'$, $\rvector{M2}'$) and then use $y$ and $\theta$ to guess the true marker positions ($\rvector{M1}$, $\rvector{M2}$), then $f(y, \theta)$ represents our guess error. We nevertheless use "measurement error" below because we presume there is no guess which can yield zero error.

For later purposes let's bundle our two unknown variables into a generalized "unknowns" vector $\boldsymbol{x}$ as follows:

\begin{align}
\boldsymbol{x} = \myvector{y}{\theta}
\end{align}

Now our error function is:

\begin{align}
f(\boldsymbol{x}) =
\big| \ \boldsymbol{\epsilon}_1 \ \big| \ ^2 \
+ \
\big| \ \boldsymbol{\epsilon}_2 \ \big| \ ^2
\end{align}

Let's implement this function in Python and then explore its output to make sure it's returning reasonable values.

```python
def measurement_error(x):
    y,theta = x
    rM1,rM2 = get_positions(y, theta)
    e1,e2   = rpM1 - rM1, rpM2 - rM2
    e1,e2   = np.linalg.norm(e1), np.linalg.norm(e2)
    f       = e1**2 + e2**2
    return f

x = [0, radians(20)]
print( measurement_error(x) )
```

    1.14563304519

Here the total measurement error is 1.146 mm$^2$. By moving to a different vertical position (e.g. $y$=2) or a different angular position ($\theta$=10 deg) we see that measurement error increases:

```python
print(   measurement_error( [2, radians(20)] )   )
print(   measurement_error( [0, radians(10)] )   )
```

    5.43207890941
    110.13801125

Since we can compute the error function $f(y,\theta)$ for arbitrary $y$ and $\theta$ values we can view it as a three-dimensional surface as follows:

```python
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D

### Compute error for a range of y and theta values:
Y       = np.linspace(-2, 5, 41)
THETA   = np.radians( np.linspace(14, 23, 41) )
Y,THETA = np.meshgrid(Y, THETA)
F       = [measurement_error(x)   for x in zip(Y.flatten(), THETA.flatten())]
F       = np.reshape(F, Y.shape)
F[F>4]  = 4  #cap all values at 4 to more clearly see the function's minimum

### Plot:
fig     = pyplot.figure()
ax      = fig.gca(projection='3d')
ax.plot_surface(Y, np.degrees(THETA), F, cmap=cm.jet, rstride=1, cstride=1, linewidth=0.2)
ax.set_xlabel(r'$y$  (mm)', size=18)
ax.set_ylabel(r'$\theta$  (deg)', size=18)
ax.set_zlabel(r'$f(y, \theta)$', size=18)
pyplot.show()
```

    <IPython.core.display.Javascript object>

In the figure above it appears that our error function's minimum value occurs for approximately $y$=1.5 and $\theta$=19, but let's find its minimum value algorithmically using **scipy.optimize.minimize** as follows:

```python
from scipy import optimize

x0       = [0, radians(20)]  #initial (y, theta) guess
results  = optimize.minimize(measurement_error, x0)

print('Least-squares estimates:')
print('   y     = %.3f' %results.x[0])
print('   theta = %.3f' %degrees(results.x[1]) )
```

    Least-squares estimates:
       y     = 1.321
       theta = 18.700

Due to measurement error our final estimates for $y$ and $\theta$ are slightly different from their true values ($y$=0, $\theta$=20).

# Bayesian approach

The figure below depicts a stochastic forward-kinematics model for this system.  Once the numerical values of $y$ and $\theta$ are known we know the values of our deterministic variables $\rvector{M1}$ and $\rvector{M2}$.  Similarly, once the numerical value of our measurement error $\epsilon$ is known we will also know the value of our generalized measurement vector $\boldsymbol{q}$, where $\boldsymbol{q}$ contains all four measured coordinates from $\rvector{M1}'$ and $\rvector{M2}'$. Let's implement this model in PyMC.

```python
import pymc

q_observed = np.asarray([rpM1, rpM2]).flatten() #measured positions
tau        = 20  #presumed measurement precision (we'll relax this later)
y          = pymc.Normal("y", 0, 1)  #prior for y
theta      = pymc.Uniform("theta", radians(-45), radians(45))  #prior for theta

@pymc.deterministic
def observations_model(y=y, theta=theta):
    rM1,rM2 = get_positions(y, theta)
    q       = np.asarray([rM1, rM2]).flatten()
    return q
q_model   = pymc.Normal("q", observations_model, tau, value=q_observed, observed=True)
```

Now that we have a model of our observations we can set values for $y$ and $\theta$ and then check what kinds of results our model produces.

```python
y.set_value(0)
theta.set_value( radians(20) )
print( q_model.random() )
print( q_model.random() )
print( q_model.random() )
```

    [ 32.78484058  11.89083546  42.45642173  16.02977362]
    [ 33.28954332  11.9619234   42.19980949  15.71036284]
    [ 32.77358118  12.14726452  42.56477381  15.23909662]

The first two columns represent $\rvector{M1}'$ and the last two columns represent $\rvector{M2}'$.  We can see that these values are similar to our actual measured values, so our model appears to have been specified correctly.

Variability exists in these values because we have not set the value of our last stochastic variable ($\epsilon$); this variability is built into the model above via the parameter "tau" which specifies our measurement precision.  If we raise tau to a very high value we will see that the model's results become much less variable.

```python
tau       = 1e9
q_model   = pymc.Normal("q", observations_model, tau, value=q_observed, observed=True)
print( q_model.random() )
print( q_model.random() )
print( q_model.random() )
print( q_model.random() )
print( q_model.random() )
```

    [ 32.88928498  11.97068501  42.28618804  15.39089926]
    [ 32.88924399  11.97076292  42.28616353  15.39089276]
    [ 32.88931438  11.97073356  42.2861079   15.39092759]
    [ 32.88925868  11.97070977  42.2862221   15.39091137]
    [ 32.88926231  11.97067152  42.2861532   15.390917  ]

Now let's relax our assumptions regarding tau's true value, then use PyMC to run Markov-Chain Monte-Carlo simulations with the goal of finding posterior distributions for $y$ and $\theta$ (and tau) which are most consistent with the observed marker positions.

```python
tau        = pymc.Normal("tau", 20, 1)  #prior for measurement precision
y          = pymc.Normal("y", 0, 1)     #prior for y
theta      = pymc.Uniform("theta", radians(-45), radians(45))  #prior for theta

@pymc.deterministic
def observations_model(y=y, theta=theta):
    rM1,rM2 = get_positions(y, theta)
    q       = np.asarray([rM1, rM2]).flatten()
    return q
q_model   = pymc.Normal("q", observations_model, tau, value=q_observed, observed=True)

mcmc      = pymc.MCMC([q_model, y, theta, tau])
mcmc.sample(40000, 20000)
Y         = mcmc.trace('y')[:]
THETA     = np.degrees( mcmc.trace('theta')[:] )
TAU       = np.degrees( mcmc.trace('tau')[:] )

print('\n\nBayesian estimates:')
print('   y     = %.3f' %Y.mean() )
print('   theta = %.3f' %THETA.mean() )
```

     [-----------------100%-----------------] 40000 of 40000 complete in 7.7 sec

    Bayesian estimates:
       y     = 1.145
       theta = 18.923

Our Bayesian estimates for $y$ and $\theta$ are somewhat closer to their true values than are our least-squares estimates. But this is just for one case.  Let's now compare the two approaches' results more systematically.

# Systematic approach comparison

First let's pick some new true values for $y$ and $\theta$ and then generate ten random datasets based on those true values.

```python
y_true      = -0.1
theta_true  = radians(5)
rM1,rM2     = get_positions(y_true, theta_true)
q_true      = np.array([rM1, rM2]).flatten()

nIterations = 10
noise_amp   = 0.5
np.random.seed(0)
Q_obs       = q_true + noise_amp * np.random.randn(nIterations, 4)

print(Q_obs)
```

    [[ 35.74884061   3.1505296   45.31813041   4.94245502]
     [ 35.80059343   2.46181206  45.30380562   3.74632982]
     [ 34.81520501   3.15575025  44.9007832    4.54914518]
     [ 35.2473333    3.0112885   45.05069303   3.98884559]
     [ 35.61385397   2.84787186  44.98529526   3.39496055]
     [ 33.59031953   3.27726029  45.26097951   3.45092591]
     [ 36.00169175   2.22326816  44.85164067   3.7284165 ]
     [ 35.63320404   3.68513038  44.90623513   4.01108968]
     [ 34.42292156   1.96005276  44.65480534   3.90018291]
     [ 35.48195977   3.55164092  44.63509801   3.67085705]]

These are the ten measurements we'll test. Let's first compute the least-squares solutions for each set of measurements.

```python
def measurement_error(x, q_obs):
    y,theta = x
    rpM1    = q_obs[:2]
    rpM2    = q_obs[2:]
    rM1,rM2 = get_positions(y, theta)
    e1,e2   = rpM1 - rM1, rpM2 - rM2
    e1,e2   = np.linalg.norm(e1), np.linalg.norm(e2)
    f       = e1**2 + e2**2
    return f

def solution_ls(q_obs):
    x0      = [y_true, theta_true]  #initial (y, theta) guess
    results = optimize.minimize(measurement_error, x0, args=(q_obs,))
    y,theta = results.x
    return y, degrees(theta)

np.set_printoptions(precision=3, suppress=True)
RESULTS_LS  = np.array([solution_ls(q_obs)  for q_obs in Q_obs])
print('Least-squares results:')
print('   y     = %s' %RESULTS_LS[:,0])
print('   theta = %s' %RESULTS_LS[:,1])
```

    Least-squares results:
       y     = [ 0.568  0.501 -0.742  0.672  1.625  1.289 -0.372  2.901 -3.663  3.141]
       theta = [ 4.988  3.732  6.596  4.055  2.144  2.974  4.801  1.356  9.488  0.674]

Next let's get Bayesian estimates for each dataset.

```python
def solution_bayesian(q_obs):
    x0      = [y_true, theta_true]  #initial (y, theta) guess
    tau     = pymc.Normal("tau", 1/(noise_amp**2), 1)
    y       = pymc.Normal("y", y_true, 1, value=y_true)
    theta   = pymc.Uniform("theta", radians(-45), radians(45))

    @pymc.deterministic
    def observations_model(y=y, theta=theta):
        rM1,rM2 = get_positions(y, theta)
        q       = np.asarray([rM1, rM2]).flatten()
        return q
    q_model   = pymc.Normal("q", observations_model, tau, value=q_obs, observed=True)

    mcmc      = pymc.MCMC([q_model, y, theta, tau])
    mcmc.sample(40000, 20000, progress_bar=False)
    Y         = mcmc.trace('y')[:]
    THETA     = np.degrees( mcmc.trace('theta')[:] )

    return Y.mean(), THETA.mean()

RESULTS_B = np.zeros((nIterations,2))
for i,q_obs in enumerate(Q_obs):
    print('Iteration %d of %d...' %(i+1, nIterations))
    y,theta = solution_bayesian(Q_obs[0])
    RESULTS_B[i] = [y, theta]
    print('   y = %.3f,  theta = %.3f' %(y,theta))

```

    Iteration 1 of 10...
       y = 0.112,  theta = 5.617
    Iteration 2 of 10...
       y = 0.061,  theta = 5.694
    Iteration 3 of 10...
       y = 0.044,  theta = 5.708
    Iteration 4 of 10...
       y = 0.176,  theta = 5.506
    Iteration 5 of 10...
       y = 0.095,  theta = 5.625
    Iteration 6 of 10...
       y = 0.083,  theta = 5.662
    Iteration 7 of 10...
       y = 0.151,  theta = 5.557
    Iteration 8 of 10...
       y = 0.068,  theta = 5.681
    Iteration 9 of 10...
       y = 0.052,  theta = 5.699
    Iteration 10 of 10...
       y = 0.007,  theta = 5.772

Here is a summary of IK error for the two approaches:

```python
x_true   = [y_true, degrees(theta_true)]
error_LS = RESULTS_LS - x_true
error_B  = RESULTS_B - x_true

print('Average absolute error (least-squares):')
print('   y: %.3f,  theta: %.3f' %tuple( np.abs(error_LS).mean(axis=0) ) )
print('Average absolute error (Bayesian):')
print('   y: %.3f,  theta: %.3f' %tuple( np.abs(error_B).mean(axis=0) ) )

```

    Average absolute error (least-squares):
       y: 1.587,  theta: 2.136
    Average absolute error (Bayesian):
       y: 0.185,  theta: 0.652

We can see that, on average, the Bayesian approach produces smaller errors for both $y$ and $\theta$.

However, you may have noticed that our calculations above have cheated a bit by using the true values of $y$ and $\theta$ as the algorithmic starting points.  Thus the results above suggest only that the Bayesian approach performs better than the least-squares approach when both start from the known true values of $y$ and $\theta$. In real IK applications the true values of $y$ and $\theta$ are of course unknown.  The main manuscript uses the true values of $y$ and $\theta$ as starting points only for LS estimation, to maximize its chances of converging to the true solution. The main manuscript then uses the LS solution as the starting point for the Bayesian estimate, to avoid potential biases associated with starting from the true solution. In other words, it gives LS estimation a huge advantage in finding the true maximum, but it gives Bayesian estimation no such advantage.

Finally, let's check how the least-squares and Bayesian approaches compare on a case-by-case basis.

```python
error_difference = np.abs(error_LS) - np.abs(error_B)
print(error_difference)
```

    [[ 0.457 -0.605]
     [ 0.44   0.574]
     [ 0.498  0.888]
     [ 0.496  0.439]
     [ 1.53   2.231]
     [ 1.206  1.364]
     [ 0.021 -0.358]
     [ 2.833  2.963]
     [ 3.412  3.789]
     [ 3.134  3.554]]

Positive values indicate that the least-squares solution had greater error and negative values indicate the opposite.  This shows that the Bayesian estimates for $y$ were better in all ten cases, and that the Bayesian estimates for $\theta$ were better in 8 of 10 cases.

# Summary

This Appendix shows that Bayesian and least-squares (LS) inverse kinematics (IK) estimates can diverge even for simple planar rotations. The results suggest that Bayesian IK performs better in general than LS-IK. Generally, *probabilistically* matching noisy data to a stochastic model via Bayesian techniques outperforms LS estimates derived from noisy data.


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of Passive Electrical Networks

The Laplace transform is a well-established tool for the analysis of differential equations including initial values. [Electrical networks](https://en.wikipedia.org/wiki/Electrical_network) composed of linear passive elements, like resistors, capacitors and inductors can be described mathematically by linear ordinary differential equations (ODEs) with constant coefficients. The Laplace transform provides an elegant way of analyzing such networks. This is illustrated in the following.

### Complex Impedances and Equivalent Networks

The concept of complex impedances is used to analyze passive electrical networks in the Laplace domain. Let's first take a look at the ODEs describing the relation between voltage $u(t)$ and current $i(t)$ for linear passive elements. They are summarized in the second column of the following table

| Element | $\quad \qquad \qquad \quad \quad$ Temporal Domain $\qquad \qquad \qquad \quad \quad$ | $\qquad \qquad$ Laplace Domain $\qquad \qquad$ | Impedance $Z(s)$ |
|:---:|:---:|:---:|:---:|
|  | $u(t) = R \cdot i(t)$ | $U(s) = R \cdot I(s)$ | $R$ |
|  | $\begin{matrix} u(t) = L \frac{d}{dt} i(t) \\ i(t) = \frac{1}{L} \int_{0}^{t} u(\tau) d\tau + i(0+) \epsilon(t) \end{matrix}$ | $\begin{matrix} U(s) = s L I(s) - L i(0+) \\ I(s) = \frac{1}{s L} U(s) + \frac{1}{s} i(0+) \end{matrix}$ | $s L$ |
|  | $\begin{matrix} u(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau + u(0+) \epsilon(t) \\ i(t) = C \frac{d}{dt} u(t) \end{matrix}$ | $\begin{matrix} U(s) = \frac{1}{s C} I(s) + \frac{1}{s} u(0+) \\ I(s) = s C U(s) - C u(0+) \end{matrix}$ | $\frac{1}{s C}$

It was assumed that the voltage $u(t)=0$ and current $i(t)=0$ for $t < 0$, hence that both are causal signals. The initial values $u(0+)$ and $i(0+)$ denote their right-sided limit values for $t=0$. For instance $u(0+) = \lim_{\epsilon \to 0} u(0 + \epsilon)$. They initial values represent the energy stored in the capacitors and inductors at time instant $t=0$, respectively. The analysis of a passive electrical network is performed by applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) resulting in an ODE, describing the relation between input and output voltage for instance. This ODE has to be solved explicitly. See for instance the [previous network analysis example](../systems_time_domain/network_analysis.ipynb).

The time-domain relations can be transformed into the Laplace domain by applying the [differentiation and integration theorem](table_theorems_transforms.ipynb#Properties-and-Theorems) of the Laplace transform. The results are summarized in the third column. The differentiation and integration are now represented by algebraic operations. Kirchhoff's circuit laws can be applied straightforwardly to the transformed quantities using the transformed relations. This is due to the fact that the Laplace transform is a linear operation. The result is an algebraic equation that can be solved straightforward with respect to the desired quantities.

When the initial values $i(0+)$ or $i(0+)$ are zero, the elements can be characterized in the Laplace domain by their [complex impedances](https://en.wikipedia.org/wiki/Electrical_impedance). The complex impedance $Z(s)$ is defined as follows

\begin{equation}
Z(s) = \frac{U(s)}{I(s)}
\end{equation}

Complex impedances can be used to represent a passive electrical network in the Laplace domain. The analysis of an electrical network in the Laplace domain is illustrated by the example given in the next section. Note that similar considerations also apply to mechanical systems and other problems that can be described by ODEs.

### Example: Second-Order Low-Pass Filter

The second-order low-pass filter from the [previous example](../systems_time_domain/network_analysis.ipynb) is analyzed using the Laplace transform. First the step response for zero initial values is computed followed by an analysis including initial values in a second stage.

#### Output signal for zero initial values

It is assumed that no energy is stored in the capacitor and inductor for $t<0$. Consequently, the initial values can be discarded. The equivalent network in the Laplace domain is derived by transforming the input $x(t)$ and output $y(t)$, and introducing the complex impedances from above table for its elements.

Applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) with the complex impedances of the network elements yields the output signal $Y(s)$ in relation to the input $X(s)$ in the Laplace domain as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot X(s)
\end{equation}

This relation is defined in `SymPy` for subsequent evaluation.

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

s = sym.symbols('s', complex=True)
t, R, L, C = sym.symbols('t R L C', positive=True)
X = sym.Function('X')(s)

Y = 1/(L*C*s**2 + R*C*s + 1) * X
Y
```

The response $y(t)$ of the network to a [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) at its input, is computed by setting the input to $x(t) = \epsilon(t)$. The Laplace transform $Y(s)$ of the output signal is hence given as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot \frac{1}{s}
\end{equation}

The output signal $y(t)$ is computed by inverse Laplace transform of $Y(s)$ for the normalized values $L = .5$, $R = 1$, $C = .4$.

```python
RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}

y = sym.inverse_laplace_transform(Y.subs(RLC).subs(X, 1/s), s, t)
y
```

The result is simplified for sake of readability

```python
y = y.simplify()
y
```

```python
sym.plot(y, (t, 0, 5), xlabel='$t$', ylabel='$y(t)$');
```

The computation of the output signal $y(t)$ did not require the solution of the underlying ODE as in the [previous example](../systems_time_domain/network_analysis.ipynb). Based on the equivalent network in the Laplace domain, only the computation of an inverse Laplace transform was required. Above result is equal to the [solution of the ODE for an Heaviside signal at the input](../systems_time_domain/network_analysis.ipynb#Step-Response).

#### Output signal including initial values

Now the analysis is performed for non-zero initial values. As initial values, the normalized voltage $u_\text{C}(0+) = -1$ at the capacitor and the normalized current $i_\text{L}(0+) = 0$ at the inductor is assumed. The network is analyzed again using Kirchhoff's circuit laws, but now the initial values are not discarded. Using the Laplace domain representation of the network elements from above table, this results in

\begin{align}
Y(s) &= \underbrace{\frac{1}{L C s^2 + R C s + 1} \cdot X(s)}_{Y_\text{ext}(s)} \\
&+ \underbrace{\frac{R C + L C s}{L C s^2 + R C s + 1} \cdot y(0+) + \frac{L}{L C s^2 + R C s + 1} \cdot i_\text{L}(0+)}_{Y_\text{int}(s)}
\end{align}

where the fact has been used that the initial voltage $u_\text{C}(0+)$ at the capacitor is equal to the initial value of the output $y(0+)$. The index for the current $i_\text{L}(t)$ at the inductor is discarded in the remainder for brevity. The terms have been sorted with respect to their dependence on the input $X(s)$, and the initial values $y(0+)$ and $i(0+)$. The part of the output signal which depends only on the input is termed as *external* part $Y_\text{ext}(s)$. The parts of the output signal which depend only on the initial values are termed as *internal* parts $Y_\text{int}(s)$. The output signal is given as superposition of both contributions

\begin{equation}
y(t) = y_\text{ext}(t) + y_\text{int}(t)
\end{equation}

where $y_\text{ext}(t) = \mathcal{L}^{-1} \{ Y_\text{ext}(s) \}$ and $y_\text{int}(t) = \mathcal{L}^{-1} \{ Y_\text{int}(s) \}$.

The external part of the output signal has already been computed in the previous section

```python
yext = y
yext.simplify()
```

The Laplace transform of the internal part $Y_\text{int}(s)$ is defined for evaluation

```python
i0, y0 = sym.symbols('i0 y0', real=True)

Yint = (R*C + L*C*s) / (L*C*s**2 + R*C*s + 1) * y0 + L / (L*C*s**2 + R*C*s + 1) * i0
Yint
```

Now the inverse Laplace transform is computed for the initial values $y(0+)$ and $i(0+)$, and the specific values of $R$, $L$ and $C$ given above

```python
yint = sym.inverse_laplace_transform(Yint.subs(RLC).subs(i0, 0).subs(y0, -1), s, t)
yint
```

The output signal $y(t)$ is given as superposition of the external and internal contribution

```python
y = yext + yint
y.simplify()
```

The internal $y_\text{int}(t)$ (green line) and external $y_\text{ext}(t)$ (blue line) part, as well as the output signal $y(t)$ (red line) is plotted for illustration

```python
p1 = sym.plot(yext, (t, 0, 5), line_color='b', xlabel='$t$', ylabel='$y(t)$', show=False)
p2 = sym.plot(yint, (t, 0, 5), line_color='g', show=False)
p3 = sym.plot(y, (t, 0, 5), line_color='r', show=False)
p1.extend(p2)
p1.extend(p3)
p1.show()
```

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# XGBoost

Zhanghan Wang, PB18111697

## Objective
\begin{align}
    \tilde{L}^{(t)}&=\sum\limits_{i=1}^n [g_if_t(\textbf{x}_i) + \frac{1}{2}h_if_t(\textbf{x}_i)]+\Omega(f_t)\\
    &=\sum\limits_{i=1}^n [g_if_t(\textbf{x}_i) + \frac{1}{2}h_if_t(\textbf{x}_i)]+\gamma T+\frac{1}{2}\lambda\sum\limits_{j=1}^T\omega_j^2\\
    &=\sum\limits_{j=1}^T\left[\left(\sum\limits_{i\in I_j}g_i\right)\omega_j+\frac{1}{2}\left(\sum\limits_{i\in I_j}h_i +\lambda \right)\omega^2_j\right]+\gamma T
\end{align}
where $$g_i=\partial_{\hat{y}^{(t-1)}}l(y_i,\hat{y}_i^{(t-1)})\quad \mbox{and} \quad h_i=\partial^2_{\hat{y}^{(t-1)}}l(y_i,\hat{y}_i^{(t-1)})$$
and $l(y_1, y_2)$ is the loss function

- note that in the code, I use $l(y_1, y_2)=\frac{1}{2}(y_1-y_2)^2$ as loss function, i.e.
$$g_i=\hat{y}^{(t-1)}-y_i\quad \mbox{and} \quad h_i=1$$

## Optimal Value

Thus, for a fixed tree structure, we have the optimal weights of leaves:
$$\omega^*_j=-\frac{\sum_{i\in I_j}g_i}{\sum_{i\in I_j}h_i+\lambda},$$
and the corresponding optimal value is
$$\tilde{\mathcal{L}}^{(t)}(q)=-\frac{1}{2}\sum\limits_{j=1}^T\frac{\left(\sum_{i\in I_j}g_i\right)^2}{\sum_{i\in I_j}h_i+\lambda}+\gamma T.$$

## Split Gain
With the $\tilde{\mathcal{L}}^{(t)}(q)$ above, we have
$$\mathcal{L}_{split}=\frac{1}{2}\left[\frac{\left(\sum_{i\in I_L}g_i\right)^2}{\sum_{i\in I_L}h_i+\lambda}+\frac{\left(\sum_{i\in I_R}g_i\right)^2}{\sum_{i\in I_R}h_i+\lambda}-\frac{\left(\sum_{i\in I}g_i\right)^2}{\sum_{i\in I}h_i+\lambda}\right]$$

```python
import numpy as np

x_train = np.load('./data/train_data.npy')
y_train = np.load('./data/train_target.npy')
x_test = np.load('./data/test_data.npy')
y_test = np.load('./data/test_target.npy')
```

```python
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)
print(x_train[:10])
print(y_train[:10])
```

    (615, 8) (615,)
    (153, 8) (153,)
    [[4.00e+00 1.54e+02 7.20e+01 2.90e+01 1.26e+02 3.13e+01 3.38e-01 3.70e+01]
     [2.00e+00 1.12e+02 8.60e+01 4.20e+01 1.60e+02 3.84e+01 2.46e-01 2.80e+01]
     [1.00e+00 1.35e+02 5.40e+01 0.00e+00 0.00e+00 2.67e+01 6.87e-01 6.20e+01]
     [3.00e+00 1.07e+02 6.20e+01 1.30e+01 4.80e+01 2.29e+01 6.78e-01 2.30e+01]
     [3.00e+00 1.02e+02 7.40e+01 0.00e+00 0.00e+00 2.95e+01 1.21e-01 3.20e+01]
     [1.00e+00 9.70e+01 6.60e+01 1.50e+01 1.40e+02 2.32e+01 4.87e-01 2.20e+01]
     [0.00e+00 1.04e+02 6.40e+01 2.30e+01 1.16e+02 2.78e+01 4.54e-01 2.30e+01]
     [7.00e+00 1.14e+02 6.60e+01 0.00e+00 0.00e+00 3.28e+01 2.58e-01 4.20e+01]
     [4.00e+00 8.30e+01 8.60e+01 1.90e+01 0.00e+00 2.93e+01 3.17e-01 3.40e+01]
     [0.00e+00 1.37e+02 8.40e+01 2.70e+01 0.00e+00 2.73e+01 2.31e-01 5.90e+01]]
    [0 0 0 1 0 0 0 1 0 0]

```python
import numpy as np
import networkx as nx
import os.path as osp
from collections import namedtuple
from networkx.drawing.nx_agraph import graphviz_layout
import matplotlib.pyplot as plt

Criterion = namedtuple('Criterion', ('idx', 'threshold'))

class GradientTreeNode:
    def __init__(self, x_train, y_train, y_last_pred, lamb=1,
                 split_threshold_by_GH=False, split_threshold=0.1):
        assert len(x_train.shape) == 2
        assert len(y_train.shape) == 1
        assert x_train.shape[0] == y_train.shape[0]
        self.x_train = x_train
        self.y_train = y_train
        self.lamb = 1
        self.split_threshold_by_GH = split_threshold_by_GH
        self.split_threshold = split_threshold
        self.y_last_pred = y_last_pred
        self.g = self.g_func(self.y_train, self.y_last_pred)
        self.h = self.h_func(self.y_train, self.y_last_pred)
        self.G = self.g.sum()
        self.H = self.h.sum()
        if split_threshold_by_GH:
            self.split_threshold = np.sqrt(self.G**2 / (self.H + self.lamb))
        self.weight = -self.G / (self.H + self.lamb)
        self.left = None
        self.right = None
        self.criterion = Criterion('?', '?')

    def __repr__(self):
        if self.left is None and self.right is None:
            return f'{self.weight:.2f}'
        else:
            return f'x{self.criterion.idx}<{self.criterion.threshold: .2f}?'

    @property
    def feature_dim(self):
        return self.x_train.shape[1]

    @property
    def m(self):
        return self.x_train.shape[0]

    # gradient statistics
    @staticmethod
    def g_func(y_train, y_last_pred):
        return y_last_pred - y_train

    @staticmethod
    def h_func(y_train, y_last_pred):
        return np.ones(y_train.shape)

    def split(self):
        # datas
        x_train = self.x_train
        g, h = self.g, self.h
        G, H = self.G, self.H
        lamb = self.lamb
        # test if all instances are in the same class
        unique_y = np.unique(self.y_train)
        if unique_y.shape[0] == 1:
            return [self]
        # otherwise, find the best feature to split the node
        split = [0, 0]
        best_score = -np.inf
        gains = np.zeros((x_train.shape[-1::-1]))
        for k in range(self.feature_dim):
            G_L = 0
            H_L = 0
            args = np.argsort(x_train[:, k], axis=0)
            for i in range(len(args)):
                G_L += g[args[i]]
                H_L += h[args[i]]
                G_R = G - G_L
                H_R = H - H_L
                gain = G_L ** 2 / (H_L + lamb) + G_R ** 2 / (H_R + lamb) - G ** 2 / (H + lamb)
                gains[k][i] = gain
                if gain > best_score:
                    best_score = gain
                    split = [k, i]
        # test if best_score is less than the thershold
        if best_score < self.split_threshold:
            return [self]
        # otherwise, we can split this node
        args = np.argsort(x_train[:, split[0]], axis=0)
        self.criterion = Criterion(split[0], x_train[args[split[1]], split[0]])
        left_idx = args[:split[1] + 1]
        right_idx = args[split[1] + 1:]
        assert len(left_idx) > 0 and len(right_idx) > 0
        self.left = GradientTreeNode(self.x_train[left_idx], self.y_train[left_idx], self.y_last_pred[left_idx],
                                     self.lamb,
                                     split_threshold_by_GH=self.split_threshold_by_GH,
                                     split_threshold=self.split_threshold)
        self.right = GradientTreeNode(self.x_train[right_idx], self.y_train[right_idx], self.y_last_pred[right_idx],
                                      self.lamb,
                                      split_threshold_by_GH=self.split_threshold_by_GH,
                                      split_threshold=self.split_threshold)
        return self.left.split() + self.right.split()

    def predict(self, x):
        assert len(x.shape) == 1 and x.shape[0] == self.feature_dim
        criterion = self.criterion
        if self.left is None or self.right is None:
            return self.weight
        if x[criterion.idx] < criterion.threshold:
            return self.left.predict(x)
        else:
            return self.right.predict(x)

class GradientTree:
    def __init__(self, x_train, y_train, y_last_pred, lamb=1,
                 split_threshold_by_GH=False, split_threshold=0.01, gamma=1):
        self.x_train = x_train
        self.y_train = y_train
        self.y_last_pred = y_last_pred
        self.lamb = lamb
        self.split_threshold = split_threshold
        self.root = GradientTreeNode(x_train, y_train, y_last_pred, lamb=lamb,
                                     split_threshold_by_GH=split_threshold_by_GH,
                                     split_threshold=split_threshold)
        self.leaves = []
        self.graph = None
        self.gamma = gamma

    @property
    def obj(self):
        Gs = np.array([leaf.G for leaf in self.leaves])
        Hs = np.array([leaf.H for leaf in self.leaves])
        obj = -np.sum((Gs ** 2 / (Hs + self.lamb))) / 2 + self.gamma * len(self.leaves)
        return obj

    def train(self):
        self.leaves = self.root.split()

    def predict(self, x_test, y_test):
        correct_count = 0
        y_test_pred = []
        y_test_pred_prob = []
        for x, y in zip(x_test, y_test):
            y_pred_prob = self.root.predict(x)
            y_pred = 1 if y_pred_prob >= 0.5 else 0
            y_test_pred.append(y_pred)
            y_test_pred_prob.append(y_pred_prob)
            if y_pred == y:
                correct_count += 1
        acc = correct_count / len(y_test)
        y_test_pred = np.array(y_test_pred)
        y_test_pred_prob = np.array(y_test_pred_prob)
        return y_test_pred, y_test_pred_prob, acc

    def generate_graph(self):
        graph = nx.DiGraph()
        queue = [self.root]
        graph.add_nodes_from([(id(self.root), {'criterion': str(self.root)})])
        while len(queue) > 0:
            top = queue[0]
            queue = queue[1:]
            if top.left is None or top.right is None:
                assert top.left is None and top.right is None
                continue
            graph.add_nodes_from([(id(top.left), {'criterion': str(top.left)}),
                                  (id(top.right), {'criterion': str(top.right)})])
            graph.add_edges_from([(id(top), id(top.left)), (id(top), id(top.right))])
            queue.append(top.left)
            queue.append(top.right)
        self.graph = graph
        return graph

    def draw_tree(self):
        graph = self.generate_graph()
        labels = {}
        for node in graph.nodes.data():
            labels[node[0]] = node[1]['criterion']
        pos = graphviz_layout(graph, prog='dot')
        nx.draw(graph, pos=pos, with_labels=True, labels=labels)
```

```python
import numpy as np
from typing import Union, List

class XGBoost:
    def __init__(self, x_train, y_train,
                 num_trees=10, lamb=1,
                 split_threshold_by_GH=False, split_threshold: Union[int, List, np.ndarray] = 1,
                 gamma=1):
        self.x_train = x_train
        self.y_train = y_train
        self.forest = []
        self.num_trees = num_trees
        self.lamb = lamb
        self.split_threshold_by_GH = split_threshold_by_GH
        self.split_threshold = split_threshold
        self.gamma = gamma

    def train(self):
        x_train = self.x_train
        y_train = self.y_train
        y_last_pred = np.zeros(y_train.shape)
        split_threshold = self.split_threshold
        if type(split_threshold) == int:
            np.repeat(split_threshold, self.num_trees)
        else:
            assert len(split_threshold) == self.num_trees

        for tree_idx in range(self.num_trees):
            gt = GradientTree(x_train, y_train, y_last_pred,
                              lamb=self.lamb,
                              split_threshold_by_GH=self.split_threshold_by_GH,
                              split_threshold=split_threshold[tree_idx], gamma=self.gamma)
            gt.train()
            self.forest.append(gt)
            y_last_pred += gt.predict(x_train, y_train)[1]

    def predict(self, x_test, y_test):
        y_test_pred_prob = np.zeros(y_test.shape)
        for gt in self.forest:
            y_test_pred_prob += gt.predict(x_test, y_test)[1]
        y_test_pred = np.array(list(map(lambda x: 1 if x > 0.5 else 0, y_test_pred_prob)))
        acc = np.sum(y_test_pred - y_test == 0) / y_test.shape[0]
        y_test_pred = np.array(y_test_pred)
        y_test_pred_prob = np.array(y_test_pred_prob)
        return y_test_pred, y_test_pred_prob, acc

```

```python
x_train = np.load('./data/train_data.npy')
y_train = np.load('./data/train_target.npy')
x_test = np.load('./data/test_data.npy')
y_test = np.load('./data/test_target.npy')

gt = GradientTree(x_train, y_train, np.zeros(y_train.shape), lamb=1, split_threshold=5, split_threshold_by_GH=True)
gt.train()
y_train_pred = gt.predict(x_train, y_train)
print(np.unique(y_train_pred[1]), y_train_pred[2])
y_test_pred = gt.predict(x_test, y_test)
print(np.unique(y_test_pred[1]), y_test_pred[2])
gt.draw_tree()
```

```python
import numpy as np
import matplotlib.pyplot as plt

x_train = np.load('./data/train_data.npy')
y_train = np.load('./data/train_target.npy')
x_test = np.load('./data/test_data.npy')
y_test = np.load('./data/test_target.npy')

# split_threshold = [5]
split_threshold = [5, 2, 1.7]
xgb = XGBoost(x_train, y_train, num_trees=len(split_threshold), lamb=1,
              split_threshold_by_GH=True, split_threshold=split_threshold,
              gamma=1)
xgb.train()
y_train_pred = xgb.predict(x_train, y_train)
# print(np.unique(y_train_pred[1]))
print(f'train acc: {y_train_pred[2]}')
y_test_pred = xgb.predict(x_test, y_test)
# print(np.unique(y_test_pred[1]))
print(f'test acc: {y_test_pred[2]}')

fig = plt.figure(figsize=(12, 9))
subplot_size = [int(np.ceil(len(xgb.forest) / (int(np.sqrt(len(xgb.forest))) + 1))),
                int(np.sqrt(len(xgb.forest))) + 1]
for tree_idx, tree in enumerate(xgb.forest):
    ax = fig.add_subplot(*subplot_size, tree_idx+1)
    plt.title(f'tree {tree_idx}')
    tree.draw_tree()
plt.tight_layout()
plt.show()

```

```python
from sklearn.ensemble import GradientBoostingClassifier

gbc = GradientBoostingClassifier()
gbc.fit(x_train, y_train)
print(f'train acc: {np.sum(gbc.predict(x_train) - y_train == 0) / y_train.shape[0]}')
print(f'test acc: {np.sum(gbc.predict(x_test) - y_test == 0) / y_test.shape[0]}')
```

    train acc: 0.9349593495934959
    test acc: 0.7254901960784313


# Statistical Data Modeling

Pandas, NumPy and SciPy provide the core functionality for building statistical models of our data. We use models to:

- Concisely **describe** the components of our data
- Provide **inference** about underlying parameters that may have generated the data
- Make **predictions** about unobserved data, or expected future observations.

This section of the tutorial illustrates how to use Python to build statistical models of low to moderate difficulty from scratch, and use them to extract estimates and associated measures of uncertainty.

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
```

Estimation
==========

An recurring statistical problem is finding estimates of the relevant parameters that correspond to the distribution that best represents our data.

In **parametric** inference, we specify *a priori* a suitable distribution, then choose the parameters that best fit the data.

* e.g. $\mu$ and $\sigma^2$ in the case of the normal distribution

```python
x = np.array([ 1.00201077,  1.58251956,  0.94515919,  6.48778002,  1.47764604,
        5.18847071,  4.21988095,  2.85971522,  3.40044437,  3.74907745,
        1.18065796,  3.74748775,  3.27328568,  3.19374927,  8.0726155 ,
        0.90326139,  2.34460034,  2.14199217,  3.27446744,  3.58872357,
        1.20611533,  2.16594393,  5.56610242,  4.66479977,  2.3573932 ])
_ = plt.hist(x, bins=8)
```

### Fitting data to probability distributions

We start with the problem of finding values for the parameters that provide the best fit between the model and the data, called point estimates. First, we need to define what we mean by ‘best fit’. There are two commonly used criteria:

* **Method of moments** chooses the parameters so that the sample moments (typically the sample mean and variance) match the theoretical moments of our chosen distribution.
* **Maximum likelihood** chooses the parameters to maximize the likelihood, which measures how likely it is to observe our given sample.

### Discrete Random Variables

$$X = \{0,1\}$$

$$Y = \{\ldots,-2,-1,0,1,2,\ldots\}$$

**Probability Mass Function**:

For discrete $X$,

$$Pr(X=x) = f(x|\theta)$$

***e.g. Poisson distribution***

The Poisson distribution models unbounded counts:

$$Pr(X=x)=\frac{e^{-\lambda}\lambda^x}{x!}$$

* $X=\{0,1,2,\ldots\}$
* $\lambda > 0$

$$E(X) = \text{Var}(X) = \lambda$$

### Continuous Random Variables

$$X \in [0,1]$$

$$Y \in (-\infty, \infty)$$

**Probability Density Function**:

For continuous $X$,

$$Pr(x \le X \le x + dx) = f(x|\theta)dx \, \text{ as } \, dx \rightarrow 0$$

***e.g. normal distribution***

$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right]$$

* $X \in \mathbf{R}$
* $\mu \in \mathbf{R}$
* $\sigma>0$

$$\begin{align}E(X) &= \mu \cr
\text{Var}(X) &= \sigma^2 \end{align}$$

### Example: Nashville Precipitation

The dataset `nashville_precip.txt` contains [NOAA precipitation data for Nashville measured since 1871](http://bit.ly/nasvhville_precip_data). The gamma distribution is often a good fit to aggregated rainfall data, and will be our candidate distribution in this case.

```python
precip = pd.read_table("../data/nashville_precip.txt", index_col=0, na_values='NA', delim_whitespace=True)
precip.head()
```

```python
_ = precip.hist(sharex=True, sharey=True, grid=False)
plt.tight_layout()
```

The first step is recognixing what sort of distribution to fit our data to. A couple of observations:

1. The data are skewed, with a longer tail to the right than to the left
2. The data are positive-valued, since they are measuring rainfall
3. The data are continuous

There are a few possible choices, but one suitable alternative is the **gamma distribution**:

$$x \sim \text{Gamma}(\alpha, \beta) = \frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)}$$

The ***method of moments*** simply assigns the empirical mean and variance to their theoretical counterparts, so that we can solve for the parameters.

So, for the gamma distribution, the mean and variance are:

$$ \hat{\mu} = \bar{X} = \alpha \beta $$
$$ \hat{\sigma}^2 = S^2 = \alpha \beta^2 $$

So, if we solve for these parameters, we can use a gamma distribution to describe our data:

$$ \alpha = \frac{\bar{X}^2}{S^2}, \, \beta = \frac{S^2}{\bar{X}} $$

Let's deal with the missing value in the October data. Given what we are trying to do, it is most sensible to fill in the missing value with the average of the available values. We will learn more sophisticated methods for handling missing data later in the course.

```python
precip.fillna(value={'Oct': precip.Oct.mean()}, inplace=True)
```

Now, let's calculate the sample moments of interest, the means and variances by month:

```python
precip_mean = precip.mean()
precip_mean
```

```python
precip_var = precip.var()
precip_var
```

We then use these moments to estimate $\alpha$ and $\beta$ for each month:

```python
alpha_mom = precip_mean ** 2 / precip_var
beta_mom = precip_var / precip_mean
```

```python
alpha_mom, beta_mom
```

We can use the `gamma.pdf` function in `scipy.stats.distributions` to plot the ditribtuions implied by the calculated alphas and betas. For example, here is January:

```python
from scipy.stats.distributions import gamma

precip.Jan.hist(normed=True, bins=20)
plt.plot(np.linspace(0, 10), gamma.pdf(np.linspace(0, 10), alpha_mom[0], beta_mom[0]))
```

Looping over all months, we can create a grid of plots for the distribution of rainfall, using the gamma distribution:

```python
axs = precip.hist(normed=True, figsize=(12, 8), sharex=True, sharey=True, bins=15, grid=False)

for ax in axs.ravel():

    # Get month
    m = ax.get_title()

    # Plot fitted distribution
    x = np.linspace(*ax.get_xlim())
    ax.plot(x, gamma.pdf(x, alpha_mom[m], beta_mom[m]))

    # Annotate with parameter estimates
    label = 'alpha = {0:.2f}\nbeta = {1:.2f}'.format(alpha_mom[m], beta_mom[m])
    ax.annotate(label, xy=(10, 0.2))

plt.tight_layout()
```

Maximum Likelihood
==================

**Maximum likelihood** (ML) fitting is usually more work than the method of moments, but it is preferred as the resulting estimator is known to have good theoretical properties.

There is a ton of theory regarding ML. We will restrict ourselves to the mechanics here.

Say we have some data $y = y_1,y_2,\ldots,y_n$ that is distributed according to some distribution:

$$Pr(Y_i=y_i | \theta)$$

Here, for example, is a **Poisson distribution** that describes the distribution of some discrete variables, typically *counts*:

```python
y = np.random.poisson(5, size=100)
plt.hist(y, bins=12, normed=True)
plt.xlabel('y'); plt.ylabel('Pr(y)')
```

The product $\prod_{i=1}^n Pr(y_i | \theta)$ gives us a measure of how **likely** it is to observe values $y_1,\ldots,y_n$ given the parameters $\theta$. Maximum likelihood fitting consists of choosing the appropriate function $l= Pr(Y|\theta)$ to maximize for a given set of observations. We call this function the *likelihood function*, because it is a measure of how likely the observations are if the model is true.

> Given these data, how likely is this model?

In the above model, the data were drawn from a Poisson distribution with parameter $\lambda =5$.

$$L(y|\lambda=5) = \frac{e^{-5} 5^y}{y!}$$

So, for any given value of $y$, we can calculate its likelihood:

```python
poisson_like = lambda x, lam: np.exp(-lam) * (lam**x) / (np.arange(x)+1).prod()

lam = 6
value = 10
poisson_like(value, lam)
```

```python
np.sum(poisson_like(yi, lam) for yi in y)
```

```python
lam = 8
np.sum(poisson_like(yi, lam) for yi in y)
```

We can plot the likelihood function for any value of the parameter(s):

```python
lambdas = np.linspace(0,15)
x = 5
plt.plot(lambdas, [poisson_like(x, l) for l in lambdas])
plt.xlabel('$\lambda$')
plt.ylabel('L($\lambda$|x={0})'.format(x))
```

How is the likelihood function different than the probability distribution function (PDF)? The likelihood is a function of the parameter(s) *given the data*, whereas the PDF returns the probability of data given a particular parameter value. Here is the PDF of the Poisson for $\lambda=5$.

```python
lam = 5
xvals = np.arange(15)
plt.bar(xvals, [poisson_like(x, lam) for x in xvals])
plt.xlabel('x')
plt.ylabel('Pr(X|$\lambda$=5)')
```

Why are we interested in the likelihood function?

A reasonable estimate of the true, unknown value for the parameter is one which **maximizes the likelihood function**. So, inference is reduced to an optimization problem.

Going back to the rainfall data, if we are using a gamma distribution we need to maximize:

$$\begin{align}l(\alpha,\beta) &= \sum_{i=1}^n \log[\beta^{\alpha} x^{\alpha-1} e^{-x/\beta}\Gamma(\alpha)^{-1}] \cr
&= n[(\alpha-1)\overline{\log(x)} - \bar{x}\beta + \alpha\log(\beta) - \log\Gamma(\alpha)]\end{align}$$

(*Its usually easier to work in the log scale*)

where $n = 2012 − 1871 = 141$ and the bar indicates an average over all *i*. We choose $\alpha$ and $\beta$ to maximize $l(\alpha,\beta)$.

Notice $l$ is infinite if any $x$ is zero. We do not have any zeros, but we do have an NA value for one of the October data, which we dealt with above.

### Finding the MLE

To find the maximum of any function, we typically take the *derivative* with respect to the variable to be maximized, set it to zero and solve for that variable.

$$\frac{\partial l(\alpha,\beta)}{\partial \beta} = n\left(\frac{\alpha}{\beta} - \bar{x}\right) = 0$$

Which can be solved as $\beta = \alpha/\bar{x}$. However, plugging this into the derivative with respect to $\alpha$ yields:

$$\frac{\partial l(\alpha,\beta)}{\partial \alpha} = \log(\alpha) + \overline{\log(x)} - \log(\bar{x}) - \frac{\Gamma(\alpha)'}{\Gamma(\alpha)} = 0$$

This has no closed form solution. We must use ***numerical optimization***!

Numerical optimization alogarithms take an initial "guess" at the solution, and iteratively improve the guess until it gets "close enough" to the answer.

Here, we will use Newton-Raphson algorithm:

$$x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}$$

Which is available to us via SciPy:

```python
from scipy.optimize import newton
```

Here is a graphical example of how Newtone-Raphson converges on a solution, using an arbitrary function:

```python
# some function
func = lambda x: 3./(1 + 400*np.exp(-2*x)) - 1
xvals = np.linspace(0, 6)
plt.plot(xvals, func(xvals))
plt.text(5.3, 2.1, '$f(x)$', fontsize=16)

# zero line
plt.plot([0,6], [0,0], 'k-')

# value at step n
plt.plot([4,4], [0,func(4)], 'k:')
plt.text(4, -.2, '$x_n$', fontsize=16)

# tangent line
tanline = lambda x: -0.858 + 0.626*x
plt.plot(xvals, tanline(xvals), 'r--')

# point at step n+1
xprime = 0.858/0.626
plt.plot([xprime, xprime], [tanline(xprime), func(xprime)], 'k:')
plt.text(xprime+.1, -.2, '$x_{n+1}$', fontsize=16)
```

To apply the Newton-Raphson algorithm, we need a function that returns a vector containing the **first and second derivatives** of the function with respect to the variable of interest. In our case, this is:

```python
from scipy.special import psi, polygamma

dlgamma = lambda m, log_mean, mean_log: np.log(m) - psi(m) - log_mean + mean_log
dl2gamma = lambda m, *args: 1./m - polygamma(1, m)
```

where `log_mean` and `mean_log` are $\log{\bar{x}}$ and $\overline{\log(x)}$, respectively. `psi` and `polygamma` are complex functions of the Gamma function that result when you take first and second derivatives of that function.

```python
# Calculate statistics
log_mean = precip.mean().apply(np.log)
mean_log = precip.apply(np.log).mean()
```

Time to optimize!

```python
# Alpha MLE for December
alpha_mle = newton(dlgamma, 2, dl2gamma, args=(log_mean[-1], mean_log[-1]))
alpha_mle
```

And now plug this back into the solution for beta:

$$ \beta  = \frac{\alpha}{\bar{X}} $$

```python
beta_mle = alpha_mle/precip.mean()[-1]
beta_mle
```

We can compare the fit of the estimates derived from MLE to those from the method of moments:

```python
dec = precip.Dec
dec.hist(normed=True, bins=10, grid=False)
x = np.linspace(0, dec.max())
plt.plot(x, gamma.pdf(x, alpha_mom[-1], beta_mom[-1]), 'm-', label='Moment estimator')
plt.plot(x, gamma.pdf(x, alpha_mle, beta_mle), 'r--', label='ML estimator')
plt.legend()
```

For some common distributions, SciPy includes methods for fitting via MLE:

```python
from scipy.stats import gamma

gamma.fit(precip.Dec)
```

This fit is not directly comparable to our estimates, however, because SciPy's `gamma.fit` method fits an odd 3-parameter version of the gamma distribution.

### Example: truncated distribution

Suppose that we observe $Y$ truncated below at $a$ (where $a$ is known). If $X$ is the distribution of our observation, then:

$$ P(X \le x) = P(Y \le x|Y \gt a) = \frac{P(a \lt Y \le x)}{P(Y \gt a)}$$

(so, $Y$ is the original variable and $X$ is the truncated variable)

Then X has the density:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

Suppose $Y \sim N(\mu, \sigma^2)$ and $x_1,\ldots,x_n$ are independent observations of $X$. We can use maximum likelihood to find $\mu$ and $\sigma$.

First, we can simulate a truncated distribution using a `while` statement to eliminate samples that are outside the support of the truncated distribution.

```python
x = np.random.normal(size=10000)

# Truncation point
a = -1

# Resample until all points meet criterion
x_small = x < a
while x_small.sum():
    x[x_small] = np.random.normal(size=x_small.sum())
    x_small = x < a

_ = plt.hist(x, bins=100)
```

We can construct a log likelihood for this function using the conditional form:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

The denominator normalizes the truncated distribution so that it integrates to one.

```python
from scipy.stats.distributions import norm

trunc_norm = lambda theta, a, x: -(np.log(norm.pdf(x, theta[0], theta[1])) -
                                      np.log(1 - norm.cdf(a, theta[0], theta[1]))).sum()
```

For this example, we will use another optimization algorithm, the **Nelder-Mead simplex algorithm**. It has a couple of advantages:

- it does not require derivatives
- it can optimize (minimize) a vector of parameters

SciPy implements this algorithm in its `fmin` function:

```python
from scipy.optimize import fmin

fmin(trunc_norm, np.array([1,2]), args=(-1, x))
```

In general, simulating data is a terrific way of testing your model before using it with real data.

### Kernel density estimates

In some instances, we may not be interested in the parameters of a particular distribution of data, but just a smoothed representation of the data at hand. In this case, we can estimate the disribution *non-parametrically* (i.e. making no assumptions about the form of the underlying distribution) using kernel density estimation.

```python
# Some random data
y = np.random.random(15) * 10
y
```

```python
x = np.linspace(0, 10, 100)
# Smoothing parameter
s = 0.4
# Calculate the kernels
kernels = np.transpose([norm.pdf(x, yi, s) for yi in y])
plt.plot(x, kernels, 'k:')
plt.plot(x, kernels.sum(1))
plt.plot(y, np.zeros(len(y)), 'ro', ms=10)
```

SciPy implements a Gaussian KDE that automatically chooses an appropriate bandwidth. Let's create a bi-modal distribution of data that is not easily summarized by a parametric distribution:

```python
# Create a bi-modal distribution with a mixture of Normals.
x1 = np.random.normal(0, 3, 50)
x2 = np.random.normal(4, 1, 50)

# Append by row
x = np.r_[x1, x2]
```

```python
plt.hist(x, bins=8, normed=True)
```

```python
from scipy.stats import kde

density = kde.gaussian_kde(x)
xgrid = np.linspace(x.min(), x.max(), 100)
plt.hist(x, bins=8, normed=True)
plt.plot(xgrid, density(xgrid), 'r-')
```

### Exercise: Cervical dystonia analysis

Recall the cervical dystonia database, which is a clinical trial of botulinum toxin type B (BotB) for patients with cervical dystonia from nine U.S. sites. The response variable is measurements on the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS), measuring severity, pain, and disability of cervical dystonia (high scores mean more impairment). One way to check the efficacy of the treatment is to compare the distribution of TWSTRS for control and treatment patients at the end of the study.

Use maximum likelihood to calculate the mean and variance of TWSTRS at week 16 for one of the treatments and the control group. Assume that the distribution of the `twstrs` variable is normal:

$$f(x \mid \mu, \sigma^2) = \sqrt{\frac{1}{2\pi\sigma^2}} \exp\left\{ -\frac{1}{2} \frac{(x-\mu)^2}{\sigma^2} \right\}$$

**HINT**: the normal distribution PDF is available in the `scipy.stats` module.

```python
cdystonia = pd.read_csv("../data/cdystonia.csv")
cdystonia[cdystonia.obs==6].hist(column='twstrs', by=cdystonia.treat, bins=8);
```

```python
# Write your answer here
```


```python
from sympy import init_printing, solve, symbols, Eq
init_printing(use_latex=True)
A,B,C,D,E,F,x,y = symbols('A,B,C,D,E,F,x,y')

eq1 = Eq(A*x + B*y, C)
eq2 = Eq(D*x + E*y, F)
solve([eq1,eq2],[x,y])
```

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy import pi,cos,sin,tan,sqrt,arcsin,arccos,arctan2
%matplotlib inline

def plotvector(p0,u,*args,**kwargs):
    """
    Gráfica un vector, dados su punto inicial
    y sus componentes rectangulares
    """
    plt.plot([p0[0],p0[0]+u[0]],[p0[1],p0[1]+u[1]],*args,**kwargs)

for t2 in np.linspace(0,pi,5):
    r1,r2,r3,r4 = 7,3,8,6
    t1 = 0
    z = sqrt(r1**2 + r2**2 - 2*r1*r2*cos(t2))
    alpha = arcsin((r2/z)*sin(t2))
    beta = arccos(-(r3**2 - r4**2 - z**2)/(2*r4*z))
    gamma = arcsin((z/r3)*sin(beta))
    t3 = pi - (alpha+beta+gamma)
    t4 = pi - (alpha+beta)

    R1 = np.array([r1*cos(t1),r1*sin(t1)])
    R2 = np.array([r2*cos(t2),r2*sin(t2)])
    R3 = np.array([r3*cos(t3),r3*sin(t3)])
    R4 = np.array([r4*cos(t4),r4*sin(t4)])

    clr = np.array([t2,t2,t2])/pi
    plotvector([0,0], R2, color=clr)
    plotvector(R2, R3, color=clr)
    plotvector(R1, R4, color=clr)
    plt.axis('equal')
    plt.grid(ls="--")
```

```python
z,t3*180/pi,t4*180/pi,alpha*180/pi,beta*180/pi
```

```python
plt.plot?
```

```python

```


# Matrix and compartive statistics review

The following notebook is a review of matrices and comparative statistics with examples in python.

The equations and examples are from the following book I highly recommend using to brush up on mathamtics commonly used in economics coursework:
- Dowling, E. T. (2012). Introduction to mathematical economics. McGraw-Hill.
    - [Amazon link](https://www.amazon.com/Schaums-Introduction-Mathematical-Economics-Outlines/dp/0071762515/ref=sr_1_7?dchild=1&keywords=mathematics+economics&qid=1593200726&sr=8-7)

# Table of contents
- [1. Matrix basics](#1.-Matrix-basics)

- [2. Special determinants](#2.-Special-determinants)

- [3. Comparative statistics](#3.-Comparative-statistics)

# 1. Matrix basics

```python
import numpy as np
np.random.seed(1)
```

## 1.1 Scalar multiplication

```python
A = np.random.randint(20, size=(2,2))
A
```

    array([[ 5, 11],
           [12,  8]])

```python
A*3
```

    array([[15, 33],
           [36, 24]])

## 1.2 Matrix addition

```python
A = np.random.randint(20, size=(2,2))
B = np.random.randint(20, size=(2,2))
A+B
```

    array([[ 9, 27],
           [ 6, 27]])

## 1.3 Matrix mulitiplication

```python
A = np.random.randint(20, size=(2,2))
B = np.random.randint(20, size=(2,2))
```

```python
A@B
```

    array([[178, 256],
           [228, 288]])

### 1.4 Identity matrix and empty

```python
np.eye(3)
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 1.]])

```python
np.zeros(shape=(2,2))
```

    array([[0., 0.],
           [0., 0.]])

## 1.5 Matrix inversion

```python
A = np.random.randint(20, size=(2,2))
A
```

    array([[14, 18],
           [ 4,  9]])

```python
#Determinate
round(np.linalg.det(A))
```

    54.0

```python
#Invert matrix
np.linalg.inv(A).round(3)
```

    array([[ 0.167, -0.333],
           [-0.074,  0.259]])

# 2. Special determinants

## 2.1 Jacobian

```python
import sympy as sy
x1, x2 = sy.symbols('x1 x2', integer=True)
y1 = 5*x1+3*x2
y2 = 25*x1**2+30*x1*x2+9*x2**2
```

```python
y1
```

$\displaystyle 5 x_{1} + 3 x_{2}$

```python
y2
```

$\displaystyle 25 x_{1}^{2} + 30 x_{1} x_{2} + 9 x_{2}^{2}$

```python
independent_variables = [x1, x2]
functions = [y1, y2]
Jacobian = sy.Matrix(np.zeros(shape=(len(functions), len(independent_variables))))
count = 0
for funcs in functions:
    for iv in independent_variables:
        Jacobian[count] = sy.diff(funcs, iv)
        count+=1
Jacobian
```

$\displaystyle \left[\begin{matrix}5 & 3\\50 x_{1} + 30 x_{2} & 30 x_{1} + 18 x_{2}\end{matrix}\right]$

```python
if Jacobian.det()==0:
    print("Functional dependence")
```

    Functional dependence

## 2.2 Hessian

### 2.2.1 Sympy example: Definition

```python
from sympy import Function, hessian
from sympy.abc import x, y
f = Function('f')(x, y)
hessian(f, (x, y))
```

$\displaystyle \left[\begin{matrix}\frac{\partial^{2}}{\partial x^{2}} f{\left(x,y \right)} & \frac{\partial^{2}}{\partial y\partial x} f{\left(x,y \right)}\\\frac{\partial^{2}}{\partial y\partial x} f{\left(x,y \right)} & \frac{\partial^{2}}{\partial y^{2}} f{\left(x,y \right)}\end{matrix}\right]$

### 2.2.1 Sympy example: From 2.1

```python
count = 0
Hessian = Jacobian.copy()
for _ in range(0,2):
    for iv in independent_variables: #Reverses list order for hessian
        Hessian[count] = sy.diff(Hessian[count],iv)
        count+=1
Hessian
```

$\displaystyle \left[\begin{matrix}0 & 0\\50 & 18\end{matrix}\right]$

```python
H1 = Hessian[0]
H2 = Hessian.det()
if H1>0 and H2>0:
    print('Positive definite')
    print('Minimum point')
if H1<0 and H2>0:
    print('Negative definite')
    print('Max point')
```

## 2.3 Discriminant
- Tests for positive or negative definiteness of quadratic equations

```python
x, y = sy.symbols('x y', integer=True)
z = 2*x**2 + 5*x*y+8*y**2
z
```

$\displaystyle 2 x^{2} + 5 x y + 8 y^{2}$

```python
Discrim = sy.Matrix([[2,(5/2)], [(5/2),8]])
Discrim
```

$\displaystyle \left[\begin{matrix}2 & 2.5\\2.5 & 8\end{matrix}\right]$

```python
d1 = Discrim[0]
d2 = Discrim.det() #deter
if d1 and d2>0:
    print("Positive definite")
elif d1<0 and d2>0:
    print('Negative definite')
```

    Positive definite

## 2.4 Higher order hessian

```python
x1, x2, x3 = sy.symbols('x1 x2 x3', integer=True)
z = -5*x1**2+10*x1+x1*x3-2*x2**2+4*x2+2*x2*x3-4*x3**2
z
```

$\displaystyle - 5 x_{1}^{2} + x_{1} x_{3} + 10 x_{1} - 2 x_{2}^{2} + 2 x_{2} x_{3} + 4 x_{2} - 4 x_{3}^{2}$

```python
focs = []
for idx,iv in enumerate([x1,x2,x3]):
    foc = sy.diff(z,iv)
    focs.append(foc)
for idx, foc in enumerate(focs):
    print("FOC %s:" %(idx+1), foc)
```

    FOC 1: -10*x1 + x3 + 10
    FOC 2: -4*x2 + 2*x3 + 4
    FOC 3: x1 + 2*x2 - 8*x3

```python
A,b = sy.linear_eq_to_matrix(focs, [x1, x2, x3])
```

```python
Hessian = A
Hessian
```

$\displaystyle \left[\begin{matrix}-10 & 0 & 1\\0 & -4 & 2\\1 & 2 & -8\end{matrix}\right]$

```python
H1 = Hessian[0]
H2 = Hessian[0:2,0:2].det()
H3 = Hessian.det()
if H1>0 and H2>0 and H3>0:
    print('Positive definite')
    print('Minimum point')
elif H1<0 and H2>0 and H3<0:
    print('Negative definite')
    print('Maximum point')
```

    Negative definite
    Maximum point

## 2.5 Bordered Hessian

```python
from sympy import Function, hessian, pprint
from sympy.abc import x, y
f = Function('f')(x, y)
constraint = Function('g')(x, y)
hessian(f, (x, y), [constraint])
```

$\displaystyle \left[\begin{matrix}0 & \frac{\partial}{\partial x} g{\left(x,y \right)} & \frac{\partial}{\partial y} g{\left(x,y \right)}\\\frac{\partial}{\partial x} g{\left(x,y \right)} & \frac{\partial^{2}}{\partial x^{2}} f{\left(x,y \right)} & \frac{\partial^{2}}{\partial y\partial x} f{\left(x,y \right)}\\\frac{\partial}{\partial y} g{\left(x,y \right)} & \frac{\partial^{2}}{\partial y\partial x} f{\left(x,y \right)} & \frac{\partial^{2}}{\partial y^{2}} f{\left(x,y \right)}\end{matrix}\right]$

## 2.6 Eigenvalues & Eigenvectors

```python
def eigen(matrix):
    trace = np.trace(A)
    det = round(np.linalg.det(A),0)
    eig_values = np.round((np.sort((trace+np.array([+1,-1])*np.sqrt(trace**2-(4*det)))/2)),1)
    solu1, solu2 = [eig_values[:][0], eig_values[:][1]]
    print("Original matrix: \n",matrix)
    print("Eigen-values:\n {}".format(eig_values))
    #Classification of matrix
    if solu1>0 and solu2>0:
        print('Pos definite')
    if solu1<0 and solu2<0:
        print('Neg definite')
    if (solu1==0 or solu2==0) and (solu1>0 and solu2>0):
        print('Pos semi-def')
    if (solu1==0 or solu2==0) and (solu1<0 and solu2<0):
        print('Pos semi-def')
    if (solu1<0 and solu2>0) or (solu1>0 and solu2<0):
        print('Indefinite')
```

```python
A = np.random.randint(20, size=(2,2))
eigen(A)
```

    Original matrix:
     [[17  0]
     [13  9]]
    Eigen-values:
     [ 9. 17.]
    Pos definite

# 3. Comparative statistics

## 3.1 One endogenous variable

$$
Q_d = m-nP+kY\\
Q_s = a+bP
$$

### 3.1.1 Explicit function
$$P* =\frac{m-a+kY}{b+n}$$

### 3.1.2 Implicit function

$$\frac{dP^*}{dY}= - \frac{F_Y}{F_P}$$

```python
from sympy.abc import x,n,p,k,y,a,b,m
f = m-n*p+k*y-a-b*p
f
```

$\displaystyle - a - b p + k y + m - n p$

```python
-sy.diff(f,y)/sy.diff(f,p)
```

$\displaystyle - \frac{k}{- b - n}$

## 3.2 N-endogenous variables
- `Comparative statistics:` requires a unique equilibrium condition for each endogenous variable
- Measuring the effect of an exogenous variable on the endgenous variables involves taking the total derivative of each equilibrium conditions
    - w.r.t to the particular exogenous variable and solving for each of the partial derivatives

$$
F^1(y_1, y_2; x_1, x_2) = 0 \\
F^2(y_1, y_2; x_1, x_2) = 0
$$

#### Note:
- #### Exogenous variables: $x_1$ and $x_2$
- #### Endogenous variables: $y_1$ and $y_2$

## 3.3 Comparative statistics for optimization problems
- Apply comparative statistics to the first order conditions to determine initial optimal values

```python
from sympy.abc import r, K, w, L, P, Q
Q = Function('Q')(K, L)
π = p*Q-r*K-w*L
π
```

$\displaystyle - K r - L w + p Q{\left(K,L \right)}$

### 3.3.1 F.O.C

```python
focs = []
for idx,iv in enumerate([K,L]):
    foc = sy.diff(π,iv)
    focs.append(foc)
focs[0]
```

$\displaystyle p \frac{\partial}{\partial K} Q{\left(K,L \right)} - r$

### 3.3.2 Jacobian

- For optimization of a system the Deteriminant of the Jacobian>0

```python
Jacobian = sy.Matrix([[π.diff(K,K), π.diff(K,L)],[π.diff(L,K), π.diff(L,L)]])
Jacobian
```

$\displaystyle \left[\begin{matrix}p \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} & p \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)}\\p \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & p \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)}\end{matrix}\right]$

```python
B = []
for foc in focs:
    B.append(foc.diff(r))
B
```

    [-1, 0]

### 3.3.3 Find derivatives

```python
J = Jacobian.det()
```

```python
J1 = Jacobian.copy()
J1[0] =1
J1[2] =0
J1
```

$\displaystyle \left[\begin{matrix}1 & p \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)}\\0 & p \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)}\end{matrix}\right]$

```python
J2 = Jacobian.copy()
J2[1] =1
J2[3] =0
J2
```

$\displaystyle \left[\begin{matrix}p \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} & 1\\p \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & 0\end{matrix}\right]$

#### 3.3.3.1 Find $\frac{\partial \bar{K}}{\partial r}$

```python
J1.det()/J
```

$\displaystyle \frac{p \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)}}{p^{2} \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} - p^{2} \left(\frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)}\right)^{2}}$

#### 3.3.3.2 Find $\frac{\partial \bar{L}}{\partial r}$

```python
J2.det()/J
```

$\displaystyle - \frac{p \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)}}{p^{2} \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} - p^{2} \left(\frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)}\right)^{2}}$

## 3.4 Comparative statistics in constrained optimization
- Optimize comparative statistics with constraints

```python
from sympy.abc import r, K, w, L, P, Q, B
lamda = sy.symbols('lamda')
Q = Function('Q')(K, L)
π = Q+lamda*(B-r*K-w*L)
π
```

$\displaystyle \lambda \left(B - K r - L w\right) + Q{\left(K,L \right)}$

### 3.4.1 F.O.C

```python
focs = []
for idx,iv in enumerate([K,L,lamda]):
    foc = sy.diff(π,iv)
    focs.append(foc)
```

```python
focs[0]
```

$\displaystyle - \lambda r + \frac{\partial}{\partial K} Q{\left(K,L \right)}$

```python
focs[1]
```

$\displaystyle - \lambda w + \frac{\partial}{\partial L} Q{\left(K,L \right)}$

```python
focs[2]
```

$\displaystyle B - K r - L w$

### 3.4.2 Jacobian

```python
independent_variables = [K,L,lamda]
functions = focs
Jacobian = sy.Matrix(np.zeros(shape=(len(functions), len(independent_variables))))
count = 0
for funcs in functions:
    for iv in independent_variables:
        Jacobian[count] = sy.diff(funcs, iv)
        count+=1
Jacobian
```

$\displaystyle \left[\begin{matrix}\frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} & \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & - r\\\frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} & - w\\- r & - w & 0\end{matrix}\right]$

```python
J_deter = Jacobian.det()
```

### 3.4.3 Find derivatives

```python
def deriv_convert(matrix, col=0):
    deriv_col = iter([0, 0, -1])
    derivs = Jacobian.copy()
    for i in range(1,len(Jacobian)+1):
        if i%3==col:
            derivs[i-1] = next(deriv_col)
    return derivs
```

#### 3.3.4.1 Find $\frac{\partial \bar{K}}{\partial B}$

```python
k_b = deriv_convert(derivs, col=1)
k_b
```

$\displaystyle \left[\begin{matrix}0 & \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & - r\\0 & \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} & - w\\-1 & - w & 0\end{matrix}\right]$

```python
k_b.det()/J_deter
```

$\displaystyle \frac{- r \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} + w \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)}}{- r^{2} \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} + 2 r w \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} - w^{2} \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)}}$

#### 3.3.4.2 Find $\frac{\partial \bar{L}}{\partial B}$

```python
L_b = deriv_convert(derivs, col=2)
L_b
```

$\displaystyle \left[\begin{matrix}\frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} & 0 & - r\\\frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & 0 & - w\\- r & -1 & 0\end{matrix}\right]$

```python
L_b.det()/J_deter
```

$\displaystyle \frac{r \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} - w \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)}}{- r^{2} \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} + 2 r w \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} - w^{2} \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)}}$

#### 3.3.4.3 Find $\frac{\partial \bar{\lambda}}{\partial B}$

```python
lamb_b = deriv_convert(derivs, col=0)
lamb_b
```

$\displaystyle \left[\begin{matrix}\frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} & \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & 0\\\frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} & \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} & 0\\- r & - w & -1\end{matrix}\right]$

```python
lamb_b.det()/J_deter
```

$\displaystyle \frac{- \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)} \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} + \left(\frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)}\right)^{2}}{- r^{2} \frac{\partial^{2}}{\partial L^{2}} Q{\left(K,L \right)} + 2 r w \frac{\partial^{2}}{\partial L\partial K} Q{\left(K,L \right)} - w^{2} \frac{\partial^{2}}{\partial K^{2}} Q{\left(K,L \right)}}$

## 3.5 Envelope theorem

- `Envelope theorem:` Measures the effect of a change in exogenous variables on the optimal value of the objective function
    - This can be achieved by simply taking the derivative of the Lagrangian function w.r.t the desired exogenous variable and evaluating the derivative at the values of the optimal solution

```python
B, x, px, y, py, lamb= sy.symbols('B x px y py lamda', integer=True)

u = Function('u')(x, y)
constraint = lamb*(B-px*x-py*y)
U = u+constraint
print('Budget constraint:')
U
```

    Budget constraint:

$\displaystyle \lambda \left(B - px x - py y\right) + u{\left(x,y \right)}$

```python
focs = []
for idx,iv in enumerate([px, py, B]):
    foc = sy.diff(U,iv)
    focs.append(foc)
```

- $\lambda$: Marginal uility of money
    - The extra utility derived from a change in income

- The first and second order conditions are negative:
    - $\uparrow \lambda \rightarrow$  negative impact on the quantity of good consumed

```python
focs[0]
```

$\displaystyle - \lambda x$

```python
focs[1]
```

$\displaystyle - \lambda y$

```python
focs[2]
```

$\displaystyle \lambda$

## 3.6 Concave programming
- Optimize comparative statistics with inequality constraints
- Assume that functions are concave

```python
B, x, px, y, py, lamb= sy.symbols('B x px y py lamda', integer=True)

u = Function('u')(x, y)
constraint = lamb*(B-px*x-py*y)
U = u+constraint
print('Budget constraint:')
U
```

    Budget constraint:

$\displaystyle \lambda \left(B - px x - py y\right) + u{\left(x,y \right)}$

```python
#1.A
sy.diff(U, x)
```

$\displaystyle - \lambda px + \frac{d}{d x} u{\left(x,y \right)}$

```python
# 1.B
sy.diff(U, y)
```

$\displaystyle - \lambda py + \frac{d}{d y} u{\left(x,y \right)}$

```python
#2.A
sy.diff(U,lamb)
```

$\displaystyle B - px x - py y$