# Linear Algebra

Sources: [Deep Learning](www.deeplearningbook.org)

```python
# Library imports
import numpy as np
from scipy import linalg
```

Definitions and notation:

- **Scalar**: a single number, such as $s \in \mathbb{R}$ or $n \in \mathbb{N}$.
- **Vector**: an array of numbers in order. If each element $x_i \in \mathbb{R}$ for vector $\mathbf{a}$, then vector $\mathbf{a}$ lies in set $\mathbb{R}^n$. Vectors in machine learning are typically column vectors (shape $n \times 1$). You can think of vectors as identifying points in space, with each element giving the coordinate along a diﬀerent axis.

\begin{align}
\mathbf{a} = \sum_{i=1}^n a_i b_i
\end{align}

- **Matrix**: 2D array of numbers, each element has two indices. A matrix $\mathbf{A}$ with $m$ rows and $n$ columns, then $\mathbf{A} \in \mathbb{R}^{m \times n}$. Elements of a matrix are identified as $A_{i, j}$ where the subscripts identify the $i$-th row and $j$-th column for the item.

\begin{align}
\mathbf{A} = \begin{bmatrix}
A_{1, 1} & A_{1, 2} \\
A_{2, 1} & A_{2, 2}
\end{bmatrix}
\end{align}

- **Tensor**: an array $\mathsf{A}$ with more than two axes. Elements are identified by $\mathsf{A}_{i, j, k}$.
- **Transpose**: the transpose of a matrix is the mirror image of the matrix across the main diagonal (running down and to right):

\begin{align}
\mathbf{A} = \begin{bmatrix}
A_{1, 1} & A_{1, 2} \\
A_{2, 1} & A_{2, 2} \\
A_{3, 1} & A_{3, 2}
\end{bmatrix} \Rightarrow
\mathbf{A}^{\operatorname{T}} = \begin{bmatrix}
A_{1, 1} & A_{2, 1} & A_{3, 1} \\
A_{1, 2} & A_{2, 2} & A_{3, 2}
\end{bmatrix}
\end{align}

- A **diagonal** matrix consists mostly of zeroes and has entries only along the main diagonal ($\mathbf{D}$ is diagonal if and only if $\mathbf{D}_{i, j} = 0$ for all $i \ne j$). A square diagonal matrix is written as a vector diag($\mathbf{v}$) which indicates the non-zero entries. They are easy to compute with since diag($\mathbf{v}$)$\mathbf{x}$ is just element-wise multiplication between the vectors (or the Hadamard product, defined below). Additionally, assuming all diagonal entries are non-zero, the inverse is diag($\mathbf{v}\text{)}^{-1}=$ diag($[1/v_1, \ldots , 1/v_n]^{\operatorname{T}}$).
- A **symmetric matrix** is one that's equal to its own transpose: $\mathbf{A} = \mathbf{A}^{\operatorname{T}}$.

Computations with vectors and matrices:

- The **dot product** of two vectors $\mathbf{a}$ and $\mathbf{b}$ (which have the same dimensionality) is defined as the sum of the element-wise products:

\begin{align}
\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^n a_i b_i
\end{align}

- **Matrix multiplication** of $A$ and $B$ only works if $A$ has the same number of columns as $B$ has rows. So if $A$ is $m \times n$ and $B$ is $n \times p$, the result $C$ is of shape $m \times p$

$$
C_{i, j} = \sum_k \mathbf{A}_{i, k} \mathbf{B}_{k, j}
$$

- There is an element-wise product defined for two matrices, which is the **Hadamard product** and is denoted ($\mathbf{A} \odot \mathbf{B}$)
- A system of equations can be written as $\mathbf{Ax} = \mathbf{b}$, where $\mathbf{A} \in \mathbb{R}^{m \times n}$ is a known matrix, $\mathbf{b} \in \mathbb{R}^{m}$ is a known vector, and $\mathbf{x} \in \mathbb{R}^{n}$ is a vector of unknown variables to solve for.

```python
# Vectors and dot products
a = np.array([1, 2, 3, 4]).reshape(4,)
print('Vector a:', a)

b = np.array([1, 0, 2, 1]).reshape(4,)
print('Vector b:', b)

print('Dot product of a and b:', np.dot(a, b))
```

    Vector a: [1 2 3 4]
    Vector b: [1 0 2 1]
    Dot product of a and b: 11

```python
# Matrix multiplication
A = np.array([5, 2, 10, 1, 0, 7]).reshape(3, 2)
print('Matrix A:')
print(A)

B = np.array([1, 3, 0, 1]).reshape(2, 2)
print('Matrix B:')
print(B)

print('AB =')
print(np.matmul(A, B))
```

    Matrix A:
    [[ 5  2]
     [10  1]
     [ 0  7]]
    Matrix B:
    [[1 3]
     [0 1]]
    AB =
    [[ 5 17]
     [10 31]
     [ 0  7]]

## Matrix Multiplication Properties

Matrix multiplication is both distributive $\mathbf{A}(\mathbf{B} + \mathbf{C}) = \mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C}$ as well as associative $\mathbf{A}(\mathbf{B} \mathbf{C}) = (\mathbf{A}\mathbf{B}) \mathbf{C}$.

However, matrix multiplication is NOT commutative $\mathbf{A} \mathbf{B} \ne \mathbf{B} \mathbf{A}$. That said, the dot product between two vectors is commutative: $\mathbf{x}^{\operatorname{T}} \mathbf{y} = \mathbf{y}^{\operatorname{T}} \mathbf{x}$.

The transpose of a matrix product can be written as $\mathbf{AB}^{\operatorname{T}} = \mathbf{B}^{\operatorname{T}} \mathbf{A}^{\operatorname{T}}$.

```python
# Transpose examples
print('Matrix A:')
print(A)
print('Transpose of A:')
print(A.T)
```

    Matrix A:
    [[ 5  2]
     [10  1]
     [ 0  7]]
    Transpose of A:
    [[ 5 10  0]
     [ 2  1  7]]

```python
# Dot product examples
print('Vector a:')
print('Vector b:')
print('Tranpose of a dot b:')
print(np.dot(a.T, b))
print('Tranpose of b dot a:')
print(np.dot(b.T, a))
```

    Vector a:
    Vector b:
    Tranpose of a dot b:
    11
    Tranpose of b dot a:
    11

```python
# Distributive property example
C = np.array([4, 4, 0, 1]).reshape(2, 2)
print('A(B + C):')
print(np.matmul(A, B+C))

print('AB + AC:')
print(np.matmul(A, B) + np.matmul(A, C))
```

    A(B + C):
    [[25 39]
     [50 72]
     [ 0 14]]
    AB + AC:
    [[25 39]
     [50 72]
     [ 0 14]]

```python
# Associative property example
print('A(BC):')
print(np.matmul(A, np.matmul(B, C)))

print('(AB)C:')
print(np.matmul(np.matmul(A, B), C))
```

    A(BC):
    [[20 37]
     [40 71]
     [ 0  7]]
    (AB)C:
    [[20 37]
     [40 71]
     [ 0  7]]

## Identity and Inverse Matrices

The **identity matrix** is a matrix that does not change any vector when you multiply the vector by that matrix. The identity matrix that preserves $n$-dimensional vectors is denoted $\mathbf{I}_n$:

\begin{align}
\mathbf{I}_n \in \mathbb{R}^{n \times n} \text{and } \forall \mathbf{x} \in \mathbb{R}^n, \, \mathbf{I}_n \mathbf{x} = \mathbf{x}
\end{align}

The structure of an identity matrix has $1$'s along the main diagonal and zeroes for all other entries. For example:

\begin{align}
\mathbf{I}_3 = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{bmatrix}
\end{align}

A **matrix inverse** of $\mathbf{A}$ is written as $\mathbf{A}^{-1}$ and is defined so $\mathbf{A}^{-1} \mathbf{A} = \mathbf{I}_n$. It's also possible to define an inverse that's multiplied on the right, such that $\mathbf{A} \mathbf{A}^{-1} = \mathbf{I}_n$. For square matrices ($m = n$), the left and right inverses are the same.

This is useful in theory to solve a system of linear equations $\mathbf{Ax} = \mathbf{b}$, where the solution is $\mathbf{x} = \mathbf{A}^{-1} \mathbf{b}$. This assumes that the inverse exists, for that to happen, the equation $\mathbf{Ax} = \mathbf{b}$ has exactly one solution (versus no solutions or an infinite number of solutions).

```python
# Identity and inverse examples
I4 = np.eye(4)
print('Identity matrix (4x4):')
print(I4)
print()

print('Vector a:')
print(a)
print()

print('I_4 * a:')
print(np.matmul(I4, a))
```

    Identity matrix (4x4):
    [[1. 0. 0. 0.]
     [0. 1. 0. 0.]
     [0. 0. 1. 0.]
     [0. 0. 0. 1.]]

    Vector a:
    [1 2 3 4]

    I_4 * a:
    [1. 2. 3. 4.]

## Linear Combinations and Span

- A **linear combination** of a set of vectors $\{\mathbf{v}^{(1)}, \ldots , \mathbf{v}^{(n)} \}$ is given be multiplying each vector $\mathbf{v}^{(i)}$ by a scalar and adding the results:

\begin{align}
\displaystyle \sum_i = c_i \mathbf{v}^{(i)}
\end{align}

- The **span** of a set of vectors is the set of all points obtainable by linear combination of the original vectors.

Finding whether $\mathbf{Ax} = \mathbf{b}$ has a solution boils down to the following:

- $\mathbf{b}$ is in the span of the columns of $\mathbf{A}$ (also known as the **column space**, or **range** of $\mathbf{A}$)
- If $\mathbf{b} \in \mathbb{R}^m$, then the column space of $\mathbf{A}$ is all of $\mathbb{R}^m$. (If not, there's a potential value of $\mathbf{b}$ with no solution)
    - This implies that $\mathbf{A}$ have at least $m$ columns - $n \ge m$ (the matrix is at least as wide as it is tall) - otherwise the dimensionality of the column space would be less than $m$. For example, if $\mathbf{A}$ has shape $3 \times 2$ (so $\mathbf{b} \in \mathbb{R}^3$), it would be a system of $3$ equations with only $2$ unknown variables. At best, this could trace out a plane in $\mathbb{R}^3$, so the system would only have a solution if $\mathbf{b}$ fell on that plane. Note: the condition $n \ge m$ is only necessary for every point to have a solution, but doesn't guarantee that columns are independent (not redundant)
    - The columns must be **linearly independent** (a set of vectors is linearly independent if no vector is a linear combination of other vectors in the set - if you added a vector to the set that were a linear combination of others, it would not add points to the set's span)
    - Therefore, for the column space to encompass all $\mathbb{R}^m$, it must have a set of $m$ linearly independent columns
- To guarantee that the matrix $\mathbf{A}$ has an inverse, there must be *at most* one solution for each value of $\mathbf{b}$. To do so, $\mathbf{A}$ must have exactly $m$ columns (otherwise, there can be more than one way of parameterizing the solution)

To summarize, in order to find the solution of the system of linear equations using the inverse, $\mathbf{A}$ must be a **square matrix** ($m = n$) and all columns are linearly independent. If $\mathbf{A}$ isn't square, or is square but has linearly dependent columns (known as a **singular**), it's still possible to solve, but just not using the matrix inversion technique.

```python
# Using scipy to solve a system of linear equations
solve_A = np.array([1., 2., 7., 1.]).reshape(2, 2)
solve_b = np.array([1., 20.])
print('A:')
print(solve_A)
print('b:')
print(solve_b)
print()

# Get inverse of A
solve_A_inv = linalg.inv(solve_A)

solution = np.matmul(solve_A_inv, solve_b)
print('Inverse solution:')
print(solution)
print()

# Use scipy's solver
alt_sol = linalg.solve(solve_A, solve_b)
print('Scipy sover solution:')
print(alt_sol)
print()

#Check
print('Check')
print(solve_A[0, 0] * solution[0] + solve_A[0, 1] * solution[1])
print(solve_A[1, 0] * solution[0] + solve_A[1, 1] * solution[1])
```

    A:
    [[1. 2.]
     [7. 1.]]
    b:
    [ 1. 20.]

    Inverse solution:
    [ 3. -1.]

    Scipy sover solution:
    [ 3. -1.]

    Check
    1.0
    19.999999999999996

## Norms

- A function for the size of a vector is called a **norm**, or more formally, norms are functions that map vectors to non-negative values. The $L^p$ norm for $p \in \mathbb{R}$, $p \ge 1$ is:

\begin{align}
\Vert \mathbf{x} \Vert_p = \left( \displaystyle \sum_i \vert x_i \vert^p \right)^{\frac{1}{p}}
\end{align}

- The $L^2$ norm ($p = 2$) is the **Euclidean norm**, and is so common it can be seen written as $\Vert \mathbf{x} \Vert$. Another common norm is using the square of the $L^2$ norm, or simply $\mathbf{x}^{\mathsf{T}} \mathbf{x}$
- A **unit vector** is one with unit norm: $\Vert \mathbf{x} \Vert_2 = 1$
- The $L^1$ norm (which is the sum of the absolute values of vector entries) is important in machine learning when the difference between zero and non-zero elements is important.

\begin{align}
\Vert \mathbf{x} \Vert_1 = \displaystyle \sum_i \vert x_i \vert
\end{align}

- Occasionally the $L^{\infty}$ is used, which is defined as:

\begin{align}
\Vert \mathbf{x} \Vert_{\infty} = \text{max}_i \vert x_i \vert
\end{align}

- In deep learning, the **Frobenius norm** is used to calculate the size of a matrix

\begin{align}
\Vert \mathbf{A} \Vert_F = \sqrt{\displaystyle \sum_{i, j} A_{i, j}^2}
\end{align}

- The dot product in terms of norms, where $\theta$ is the angle between the vectors, is:

\begin{align}
\mathbf{x}^{\operatorname{T}} \mathbf{y} = \Vert \mathbf{x} \Vert_2 \Vert \mathbf{y} \Vert_2 \cos \theta
\end{align}

- A vector $\mathbf{x}$ and a vector $\mathbf{y}$ are **orthogonal** to each other if $\mathbf{x}^{\operatorname{T}} \mathbf{y} = 0$ (assuming both vectors have non-zero norms), which means they are at a $90^{\circ}$ angle to each other. In $\mathbb{R}^n$, at most $n$ vectors with non-zero norms can be mutually orthogonal.
- **Orthonormal** vectors are both orthogonal and have unit norms.
- An **orthogonal matrix** is a square matrix where the rows are mutually orthonormal and the columns are mutually orthonormal:

\begin{align}
\mathbf{A}^{\operatorname{T}} \mathbf{A} = \mathbf{A} \mathbf{A}^{\operatorname{T}} = \mathbf{I} \\
\text{and } \mathbf{A}^{-1} = \mathbf{A}^{\operatorname{T}}
\end{align}

```python
# Norm examples
a_norm_l2 = linalg.norm(a)  # default is L2 for vector; Frobenius for matrix
a_norm_max = linalg.norm(a, ord=np.inf)
A_norm_frob = linalg.norm(A)

print('Vector a:')
print(a)
print()

print('L2 norm for a:')
print(a_norm_l2)
print()

print('L-inf norm (max of abs values) for a:')
print(a_norm_max)
print()

print('Matrix A:')
print(A)
print()

print('Frobenius norm for A:')
print(A_norm_frob)
```

    Vector a:
    [1 2 3 4]

    L2 norm for a:
    5.477225575051661

    L-inf norm (max of abs values) for a:
    4.0

    Matrix A:
    [[ 5  2]
     [10  1]
     [ 0  7]]

    Frobenius norm for A:
    13.379088160259652

## Eigenvalues, Eigenvectors, and Eigendecomposition

- An **eigenvector** of a square matrix $\mathbf{A}$ is a nonzero vector $\mathbf{v}$ such that multiplication by $\mathbf{A}$ alters only the scale of $\mathbf{v}$:

\begin{align}
\mathbf{Av} = \lambda \mathbf{v}
\end{align}

- The scalar $\lambda$ is known as the **eigenvalue** that corresponds to the above eigenvector
- Generally, you focus on unit eigenvectors. If $\mathbf{v}$ is an eigenvector of $\mathbf{A}$, then any rescaled vector $s\mathbf{v}$ for $s \in \mathbb{R}, \, s \ne 0$ is also an eigenvector with the same $\lambda$

Another way to look at eigenvectors and eigenvalues is through the lens that matrix multiplication applies a linear transformation to that original matrix $\mathbf{A}$. That transformation may rotate, flip, or stretch space in some way, but the eigenvectors of $\mathbf{A}$ are special, in that they'll remain on their original span, just scaled by $\lambda$ after the transformation. To find the eigenvectors, first re-write the right side of the above equation from scalar multiplication to matrix multiplication, and rearrange:

\begin{align}
\mathbf{Av} = (\lambda \mathbf{I}) \mathbf{v} \\
\mathbf{Av} - (\lambda \mathbf{I}) \mathbf{v} = \mathbf{0} \\
(\mathbf{A} - \lambda \mathbf{I}) \mathbf{v} = \mathbf{0}
\end{align}

The resulting equation shows $\mathbf{A}$ less the $\lambda$ value on the main diagonal multiplied by vector $\mathbf{v}$ gives the zero vector. When a nonzero matrix times a nonzero vector results in the zero vector, that means the determinant of the matrix must be zero. To find the eigenvectors and eigenvalues for $\mathbf{A}$, solve for $\text{det(}\mathbf{A} - \lambda \mathbf{I}\text{)} = 0$

- **Eigendecomposition** decomposes a matrix into a set of eigenvectors and eigenvalues. If you concatenate all eigenvectors $\mathbf{v}^{(i)}$ into a matrix $\mathbf{V} = [\mathbf{v}^{(1)}, \ldots , \mathbf{v}^{(n)}]$ and all eigenvalues into a vector $\mathbf{\lambda} = [\lambda^{(1)}, \ldots , \lambda^{(n)}]^{\operatorname{T}}$, then the decomposition of $\mathbf{A}$ is:

\begin{align}
\mathbf{A} = \mathbf{V} \text{diag(} \mathbf{\lambda}\text{)} \mathbf{V}^{-1}
\end{align}

Constructing a matrix with specific eigenvectors and eigenvalues allows you to stretch space in a specific way. In $\mathbb{R}^2$, picture a unit circle  (where the set of all unit vectors is $\mathbf{u}$), showing the existing eigenvectors for a matrix $\mathbf{A}$. After multiplying $\mathbf{Au}$, eigenvector $\mathbf{v}^{(1)}$ will be scaled by $\lambda_1$ and eigenvector $\mathbf{v}^{(2)}$ will be scaled by $\lambda_2$, with the unit circle distorting to encompass those new vector locations.

Not all matrices have real-valued eigendecompositions. However, every real symmetric matrix can be decomposed into an expression using only real-valued eigenvectors and eigenvalues:

\begin{align}
\mathbf{A} = \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^{\operatorname{T}}
\end{align}

- $\mathbf{Q}$ is an orthogonal matrix composed of eigenvectors of $\mathbf{A}$
- $\mathbf{\Lambda}$ (capital lambda) is a diagonal matrix where the eigenvalue $\Lambda_{i, i}$ is associated with the eigenvector in column $i$ of $\mathbf{Q}$, denoted as $\mathbf{Q}_{:, i}$
- Since $\mathbf{Q}$ is an orthogonal matrix, $\mathbf{A}$ scales space by $\lambda_i$ in the direction $\mathbf{v}^{(i)}$ (akin to the unit circle description above)
- While an eigendecomposition will exist for a real symmetric matrix, it isn't guaranteed to be unique

What an eigendecomposition of a matrix can tell you:

- The matrix is singular if and only if any of the eigenvalues are zero
- The decomposition (of real symmetric matrix) can be use to optimize quadratic expressions in the form $f(\mathbf{x}) = \mathbf{x}^{\operatorname{T}} \mathbf{Ax}$ (where $\Vert \mathbf{x} \Vert_2 = 1$)
- A matrix whose eigenvalues are all positive is called **positive definite**. This guarantees that $\mathbf{x}^{\operatorname{T}} \mathbf{Ax} = 0 \Rightarrow \mathbf{x} = 0$
- A matrix whose eigenvalues are all positive or zero is called **positive semidefinite**, which guarantees that $\forall \mathbf{x} \text{, } \mathbf{x}^{\operatorname{T}} \mathbf{Ax} \ge 0$

```python
# Eigenvector and eigenvalue example
tmp = np.array([3., 1., 0, 2.]).reshape(2, 2)
eig_vals = linalg.eig(tmp)

print('Matrix:')
print(tmp)
print()

print('Eigenvalues:')
print(eig_vals[0])
print()

print('Eigenvectors:')
print(eig_vals[1])
```

    Matrix:
    [[3. 1.]
     [0. 2.]]

    Eigenvalues:
    [3.+0.j 2.+0.j]

    Eigenvectors:
    [[ 1.         -0.70710678]
     [ 0.          0.70710678]]

## Singular Value Decomposition

- **Singular value decomposition (SVD)** is a way to factor a matrix into singular vectors and singular values. It sheds similar light on a matrix as eigendecomposition, but is more universally applicable - every real matrix has an SVD, even ones that aren't square (a requirement for eigendecomposition)

\begin{align}
\mathbf{A} = \mathbf{U}\mathbf{D}\mathbf{V}^{\operatorname{T}}
\end{align}

- $\mathbf{A}$ is shape $m \times n$
- $\mathbf{U}$ is shape $m \times m$ and is an orthogonal matrix. Its columns are the **left-singular vectors**
- $\mathbf{D}$ is shape $m \times n$ (not necessarily square) and is a diagonal matrix. The elements of $\mathbf{D}$ are the **singular values** of $\mathbf{A}$
- $\mathbf{V}$ is shape $n \times n$ and is an orthogonal matrix. Its columns are the **right-singular vectors**
- The left-singular vectors of $\mathbf{A}$ are the eigenvectors of $\mathbf{A} \mathbf{A}^{\operatorname{T}}$
- The right-singular vectors of $\mathbf{A}$ are the eigenvectors of $\mathbf{A}^{\operatorname{T}} \mathbf{A}$
- The nonzero singular values of $\mathbf{A}$ are the square roots of the eigenvalues of both $\mathbf{A}^{\mathsf{T}} \mathbf{A}$ and $\mathbf{A} \mathbf{A}^{\operatorname{T}}$

SVD is useful to partially generalize matrix inversion (which isn't defined for non-square matrices).

```python
# SVD example
U, s, Vh = linalg.svd(A)

print('Shape of matrix A:', A.shape)
print('Matrix A:')
print(A)
print()

print('Shape of U:', U.shape)
print('U:')
print(U)
print()

print('Shape of s:', s.shape)
print('s:')
print(s)
print()

print('Shape of Vh:', Vh.shape)
print('Vh:')
print(Vh)
```

    Shape of matrix A: (3, 2)
    Matrix A:
    [[ 5  2]
     [10  1]
     [ 0  7]]

    Shape of U: (3, 3)
    U:
    [[-0.46824193  0.0953727  -0.87843813]
     [-0.86978761 -0.2248469   0.43921906]
     [-0.15562458  0.96971538  0.18823674]]

    Shape of s: (2,)
    s:
    [11.41254422  6.98239461]

    Shape of Vh: (2, 2)
    Vh:
    [[-0.96727649 -0.25372463]
     [-0.25372463  0.96727649]]

```python
# Checking SVD output to eigenvalue claims (true, order is different)
A_AT = np.matmul(A, A.T)
eig_A_AT = linalg.eig(A_AT)

AT_A = np.matmul(A.T, A)
eig_AT_A = linalg.eig(AT_A)

print('Eigenvectors of A*A^T (should match cols of U):')
print(eig_A_AT[1])
print()
print('Left-singular vectors of A (U):')
print(U)
print()

print('Eigenvectors of A^T*A (should match cols of Vh):')
print(eig_AT_A[1])
print()
print('Right-singular vectors of A (Vh):')
print(Vh)
print()

print('Eigenvalues of A*A^T:')
print(eig_A_AT[0])
print('Eigenvalues of A^T*A:')
print(eig_AT_A[0])
print('Squares of singular values of A:')
print(s**2)
```

    Eigenvectors of A*A^T (should match cols of U):
    [[ 0.46824193  0.87843813  0.0953727 ]
     [ 0.86978761 -0.43921906 -0.2248469 ]
     [ 0.15562458 -0.18823674  0.96971538]]

    Left-singular vectors of A (U):
    [[-0.46824193  0.0953727  -0.87843813]
     [-0.86978761 -0.2248469   0.43921906]
     [-0.15562458  0.96971538  0.18823674]]

    Eigenvectors of A^T*A (should match cols of Vh):
    [[ 0.96727649 -0.25372463]
     [ 0.25372463  0.96727649]]

    Right-singular vectors of A (Vh):
    [[-0.96727649 -0.25372463]
     [-0.25372463  0.96727649]]

    Eigenvalues of A*A^T:
    [ 1.30246165e+02+0.j -1.86517468e-14+0.j  4.87538345e+01+0.j]
    Eigenvalues of A^T*A:
    [130.24616546+0.j  48.75383454+0.j]
    Squares of singular values of A:
    [130.24616546  48.75383454]

## The Trace Operator

- The **trace operator** gives the sum of all the diagonal entries of a matrix:

\begin{align}
\text{Tr(} \mathbf{A} \text{)} = \displaystyle \sum_i \mathbf{A}_{i, i}
\end{align}

It's useful to express certain relationships is simpler terms than resorting to summation notation. For example, another way to write the Frobenius norm is:

\begin{align}
\Vert \mathbf{A} \Vert_F = \sqrt{\text{Tr(} \mathbf{A} \mathbf{A}^{\operatorname{T}} \text{)}}
\end{align}

Some properties of the trace operator are:

- It's invariant to the transpose operator:

\begin{align}
\text{Tr(} \mathbf{A} \text{)} = \text{Tr(} \mathbf{A}^{\operatorname{T}} \text{)}
\end{align}

- For square matrices, it's invariant to cyclic permutations:

\begin{align}
\text{Tr(} \mathbf{ABC} \text{)} = \text{Tr(} \mathbf{CAB} \text{)} = \text{Tr(} \mathbf{BCA} \text{)}
\end{align}

- A scalar is its own trace $a = \text{Tr(}a \text{)}$

## The Determinant

- The **determinant** of a square matrix, $\text{det(} \mathbf{A} \text{)}$, is a function that maps matrices to a real scalar. Visually, it can be interpreted as the factor by which multiplying by the matrix (aka the linear transformation described by the matrix on the basis vectors) expands or contracts space. In $\mathbb{R}^2$ you can think of how the area of a given region changes, in $\mathbb{R}^3$ it's how the volume changes. For example, the matrix below moves $\hat{i} = [1, 0]^{\operatorname{T}}$ to $[3, 0]^{\operatorname{T}}$ and $\hat{j} = [0, 1]^{\operatorname{T}}$ to $[0, 2]^{\operatorname{T}}$, so the new area of the unit square after applying the matrix transformation is $6$, which is the determinant.

\begin{align}
\begin{bmatrix}
3 & 0 \\
0 & 2
\end{bmatrix}
\end{align}

- The determinant is also equal to the product of all the eigenvalues of the matrix.
- A negative determinant indicates that the transformation inverts space (akin to flipping a sheet of paper over in 2D).
- A determinant of $1$ indicates that multiplying by the matrix preserves the area or volume of space.
- When the determinant is zero, it implies the matrix transformation squishes space to a lower dimension. In 2D, that means space is reduced from a plane to a line or single point. In 3D, a cube is reduced to a plane, line, or point. It also indicates that the columns of the matrix are linearly dependent.

The formula for finding a determinant for a matrix in $\mathbb{R}^2$ is:

\begin{align}
\text{det}\left( \begin{bmatrix}
a & b \\
c & d
\end{bmatrix} \right) = ad - cb
\end{align}

The formula for finding a determinant for a matrix in $\mathbb{R}^3$ is:

\begin{align}
\text{det}\left( \begin{bmatrix}
a & b & c \\
d & e & f \\
g & h & i
\end{bmatrix} \right) =
a \, \text{det}\left( \begin{bmatrix}
e & f \\
h & i
\end{bmatrix} \right) -
b \, \text{det}\left( \begin{bmatrix}
d & f \\
g & i
\end{bmatrix} \right) +
c \, \text{det}\left( \begin{bmatrix}
d & e \\
g & h
\end{bmatrix} \right)
\end{align}


# Mobile robot - unicycle model

```python
import numpy as np
import sympy as sy
sy.init_printing()
```

```python
s = sy.symbols('s', real=False)
x, y, theta = sy.symbols('x, y, theta')
x0, y0, theta0 = sy.symbols('x0, y0, theta0')
omega, v = sy.symbols('omega, v')
omega0, v0 = sy.symbols('omega_0, v_0')
zx, zy, ztheta = sy.symbols('z_x, z_y, z_theta')
ztheta
```

```python
xi = sy.Matrix([theta, x, y])
dxidt = sy.Matrix([omega, v*sy.cos(theta), v*sy.sin(theta)])
```

## Linearization

```python
A1 = dxidt.diff(theta).subs([(theta, theta0), (v, v0)])
A1
```

```python
A = sy.zeros(3,3)
A[:, 0] = A1
A
```

```python
A*A
```

## Measurements

```python
phi = sy.atan2(y, x) - theta + 180
d = sy.sqrt(x*x + y*y)
d
```

```python
C1 = sy.Matrix([[phi.diff(theta).subs([(x, x0), (y, y0)]),
                 phi.diff(x).subs([(x, x0), (y, y0)]),
                 phi.diff(y).subs([(x, x0), (y, y0)])]])
C1
```

```python
C2 = sy.Matrix([[d.diff(theta).subs([(x, x0), (y, y0)]),
                 d.diff(x).subs([(x, x0), (y, y0)]),
                 d.diff(y).subs([(x, x0), (y, y0)])]])
C2
```

```python
C = sy.Matrix([C1, C2])
C
```

```python
Obs = sy.Matrix([C, C*A, C*A*A])
Obs
```

```python
Obs.rank()
```

```python

```


# 1D Maze with Q Learning

In this notebook we attempt to solve the 1D Maze reinforcement learning problem. However, instead of using _TD Learning_ we will be using _Q Learning_. The difference is in the value function and the delta function. When using Q learning, one considers the value of a __state action pair__. Being in a state and going left has a different value than being in a state and going right. The Q function and delta function look like so:

$\boldsymbol{Q}(\boldsymbol{s},\boldsymbol{a}) = \boldsymbol{r}(\boldsymbol{s} + 1) + \gamma \boldsymbol{r}(\boldsymbol{s} + 2) \ldots \gamma^{\boldsymbol{n}-1} \boldsymbol{r}(\boldsymbol{s}+\boldsymbol{n})$

$\delta(\boldsymbol{s},\boldsymbol{a}) = (\boldsymbol{r}(\boldsymbol{s}+1) + \gamma \boldsymbol{Q}(\boldsymbol{s}+1, \boldsymbol{a}+1)) - \boldsymbol{Q}(\boldsymbol{s}, \boldsymbol{a})$

The Q Table for a maze of length four would look like the following:

##### 6 State:
| State: | 0 | 1 | 2 | 3 | 4 | 5 |
| --- | --- | --- | --- | --- | --- | --- |
| Left | 0 | 1 | 1/2 | 1/4 | 1/8 | 1/4 |
| Right | 0 | 1/4 | 1/8 | 1/4 | 1/2 | 1 |

```python
import numpy as np
from sympy import *
init_printing(use_latex=True)
from fractions import Fraction

EPOCH = 100
GAMMA = .5
EPSILON = .5
class Maze:
    def __init__(self, goal, length):
        self.reward = np.zeros(length)
        self.value = np.zeros((2,length))
        self.goal = goal
        self.length = length

        self.reward[goal] = 1

    # Performs one episode of learning on the maze
    # S is the starting state
    def episode(self, s):
         #get action
        if (self.Q(s,0) >= self.Q(s,1)):
            a = 0
        else:
            a = 1
        while s != self.goal:
            self.value[a, s] += self.delta(s,a)
            left, right = self.orient(s)
            if (a == 0):
                s = left
            else:
                s = right
            # Epsilon soft
            if (np.random.random() > EPSILON):
                if (self.Q(s,0) >= self.Q(s,1)):
                    a = 0
                else:
                    a = 1
            else:
                a = np.random.choice([0,1])

    # delta(s,a) = r(s+1) + GAMMA * maxQ(s+1,a+1) - Q(s,a)
    def delta(self, s, a):
        return  self.Q(s, a) - self.value[a, s]

    # Q(s,a) = r(s+1) + GAMMA * Q(s+1, a+1)
    def Q(self, s, a):
        # Get next state and action
        left, right = self.orient(s)
        if (a == 0):
            nextS = left
        else:
            nextS = right

        # Get value
        if (s == self.goal):
            # no future moves (only reward)
            return self.reward[nextS]
        else:
            if (self.value[0, nextS] >= self.value[1, nextS]):
                nextValue = self.value[0, nextS]
            else:
                nextValue = self.value[1, nextS]
            return self.reward[nextS] + GAMMA * nextValue

    def orient(self, s):
        # Obtain which state is to the left and which is to the right
        if (s == 0):
            left = self.length-1
            right = s+1
        elif (s == self.length-1):
            left = s-1
            right = 0
        else:
            left = s-1
            right = s+1

        return left, right

    # Print the maze
    def display(self):
        print("L: ", end='')
        for i in self.value[0]:
            print( "| %3s" %Fraction(i), end='')
        print("|")
        print("R: ", end='')
        for i in self.value[1]:
            print( "| %3s" %Fraction(i), end='')
        print("|")
        print()

```

```python
maze = Maze(0, 6)
print("The new matrix is:")
maze.display()

for i in range(EPOCH):
    s = np.random.randint(0,maze.length)
    maze.episode(s)
print("The trained matrix is:")
maze.display()
```

    The new matrix is:
    L: |   0|   0|   0|   0|   0|   0|
    R: |   0|   0|   0|   0|   0|   0|

    The trained matrix is:
    L: |   0|   1| 1/2| 1/4| 1/8| 1/4|
    R: |   0| 1/4| 1/8| 1/4| 1/2|   1|

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import interact
```

$%\newcommand{\fou}[2]{\widehat{#1}^{(#2)}}$
$\newcommand{\fou}[2]{\widehat{#1}}$

# Item XI

$%\newcommand{\fou}[2]{\widehat{#1}^{(#2)}}$
$\newcommand{\fou}[2]{\widehat{#1}}$
Use the Fourier transfrom to solve the following PDE:

\begin{align}
u_t - k u_{xx} &= 0 \text{ and } t>0
\\u(x,0) &= f(x) \, , \,  x \in \mathbb{R}
\end{align}
Assume $f$ and $u \in L^{2}(\mathbb{R})$.

---

We apply the Fourier transform with respect to the variable $x$ on both equations.

In the first one:
\begin{align}
\fou{u_t}{x} - k \fou{u_{xx}}{x} &= 0
\\ \fou{u_t}{x} - k(2\pi i \xi)^2 \fou{u}{x} &= 0
\\ \fou{u_t}{x} + k(2\pi \xi)^2 \fou{u}{x} &= 0 \, .
\end{align}
We now use the following property (see this [link](https://www.math.ubc.ca/~feldman/m267/pdeft.pdf)):
$$
\fou{\frac{\partial u}{\partial t}}{x}(\xi,t) = \frac{\partial \fou{u}{x}(\xi,t)}{\partial t} \,.
$$ and
\begin{align}
\\ \left(\fou{u}{x}\right)_t + k(2\pi \xi)^2 \fou{u}{x} &= 0
\\ \left(\fou{u}{x}\right)_t &= -k(2\pi \xi)^2 \fou{u}{x} \, .
\end{align}
The solution to this ordinary PDE is:
\begin{align}
\fou{u}{x}(\xi,t) = c(\xi)e^{-k(2\pi \xi)^2 t}
\end{align}

We can see that at $t=0$:
\begin{align}
\fou{u}{x}(\xi,0) &= c(\xi)
\\ \fou{f}{x}(\xi) &= c(\xi)
\end{align}

Finally, the solution is:
\begin{align}
\fou{u}{x}(\xi,t) &= \fou{f}{x}(\xi)e^{-k(2\pi \xi)^2 t}
\\ u(x,t) &= \int_{-\infty}^{\infty} \fou{f}{x}(\xi)e^{-k(2\pi \xi)^2 t} e^{2\pi i \xi x} d\xi
\\ &= \int_{-\infty}^{\infty} \fou{f}{x}(\xi)e^{-k(2\pi \xi)^2 t} e^{2\pi i \xi x} d\xi
\end{align}

```python
def discretize(fn,xi,xf,N=101):
    xs = np.linspace(xi,xf,num=N)
    fs = fn(xs)
    return fs

# The discrete fourier transform
def discrete_fourier_transform(fs,derivate=0,inverse=False):
    N = fs.shape[0]
    ns = np.arange(N)
    if inverse:
        ts = [(1.0/N)*np.sum(fs*np.exp(1j*2*np.pi*k/N*ns)) for k in range(N)]
    else:
        ts = [np.sum(fs*np.exp(-1j*2*np.pi*k/N*ns)) for k in range(N)]
    return np.array(ts)
```

```python
def solve_pde(f,K,xi,xf,tf,N=300,M=50):
    fx = discretize(f,xi,xf,N=N) # u(x,0)
    ees = np.arange(N)/(xf-xi)
    fxt = discrete_fourier_transform(fx)
    ts = np.linspace(0,tf,num=M)
    funct = lambda t: fxt*np.exp(-K*(2*np.pi*ees)**2*t)
    uts = [discrete_fourier_transform(funct(t),inverse=True) for t in ts]
    return uts
```

```python
X_I = -4
X_F = 4
KK = 0.2
T_F = 10.0

# We test with a function, say:
F_MAX = 1
fn = lambda x: np.exp(-x**2.0)
us = solve_pde(fn,K=KK,xi=X_I,xf=X_F,tf=T_F)
xs = np.linspace(X_I,X_F,num=us[0].shape[0])

def plot_at_time(i=0):
    plt.plot(xs,us[i],'-')
    plt.ylim((0,F_MAX))
    plt.grid()
    plt.title("$u(x,t=%f)$"%(T_F*i/(len(us)-1.0)))
    plt.show()

interact(plot_at_time,i=(0,len(us)-1),continuous_update=False)
```

    interactive(children=(IntSlider(value=0, description='i', max=49), Output()), _dom_classes=('widget-interact',…

    <function __main__.plot_at_time(i=0)>

```python

```


# O método de Euler modificado
*Método Preditor-Corretor*

Vamos considerar novamente o problema de valor inicial:

\begin{equation}
\begin{aligned}
\dot{u}(t) &= f(t, u(t)), \quad t>t_{1} \\
u\left(t_{1}\right) &= a
\end{aligned}
\end{equation}

Primeiro, integramos de $t_1$ até $t_2$

\begin{equation}
\begin{aligned}
\int_{t_{1}}^{t_{2}} \dot{u}(t) dt &=\int_{t_{1}}^{t_{2}} f(t, u(t)) dt \\
u\left(t_{2}\right)-u\left(t_{1}\right) &=\int_{t_{1}}^{t_{2}} f(t, u(t)) dt \\
u\left(t_{2}\right) &=u\left(t_{1}\right)+\int_{t_{1}}^{t_{2}} f(t, u(t)) d t
\end{aligned}
\end{equation}

Agora, aplicamos a regra do trapézio na integral do lado direito da expressão:

\begin{equation}
\int_{t_{1}}^{t_{2}} f(t, u(t)) d t=\left[\frac{f\left(t_{1}, u\left(t_{1}\right)\right)+f\left(t_{2}, u\left(t_{2}\right)\right)}{2}\right] h+O\left(h^{3}\right)
\end{equation}

Não conhecemos o valor de $u(t_2)$, vamos aproximá-lo usando o método de Euler:
\begin{equation}
\tilde{u}\left(t_{2}\right)=u\left(t_{1}\right)+h f\left(t_{1}, u\left(t_{1}\right)\right)
\end{equation}

Portanto, temos:

\begin{equation}
\begin{aligned}
u\left(t_2\right) &=u\left(t_1\right)+\int_{t_1}^{t_2} f(t, u(t)) d t \\
& \approx u\left(t_1\right)+\left[\frac{f\left(t_1, u\left(t_1\right)\right)+f\left(t_2, u\left(t_2\right)\right)}{2}\right] h \\
& \approx u\left(t_1\right)+\left[\frac{f\left(t_1, u\left(t_1\right)\right)+f\left(t_2, \tilde{u}\left(t_2\right)\right)}{2}\right] h
\end{aligned}
\end{equation}

Obtemos então a fórmula de recorrência:
\begin{equation}
\begin{aligned}
\tilde{u}_{k+1} &=u_k+h f\left(t_k, u_k\right) \\
u_{k+1} &=u_k+\frac{h}{2}\left(f\left(t_k, u_k\right)+f\left(t_{k+1}, \tilde{u}_{k+1}\right)\right) \\
u_{1} &=a
\end{aligned}
\end{equation}

Podemos reescrever e deixar da seguinte forma:

\begin{equation}
\begin{aligned}
k_{1} &=f\left(t_n, u_n\right) \\
k_{2} &=f\left(t_{n+1}, u_{n+1}\right) \\
u_{n+1} &={u}_{n}+h \frac{k_{1}+k_{2}}{2} \\
u_1 &=a
\end{aligned}
\end{equation}

(aqui, trocamos os índices $k$ por $n$ para evitar a confusão com $k_1$ e $k_2$):

Aqui $k_1$ e $k_2$ são variáveis auxiliares que representam as **inclinações** e devem ser calculadas a cada passo. Esta notação é compatível com a notação usada nos métodos de Runge-Kutta, uma família de esquemas iterativos para aproximar problemas de valor inicial, da qual o método de Euler e o método de Euler melhorado são casos particulares. Veremos os métodos de Runge-Kutta a seguir.

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn-poster')
```

```python
def Euler(f,h,s0,tmax=10):
  nt = int(tmax/h)
  u = np.zeros(nt)
  tempo = np.linspace(0,tmax,nt)
  u[0] = s0

  for k in np.arange(1,nt):
      u[k] = u[k-1] + h*f(tempo[k-1],u[k-1])

  return tempo,u
```

```python
def faz_grafico_euler(tempo,y,f_analitica,h=0.01):
  plt.figure(figsize = (12, 8))
  plt.plot(tempo, y, 'bo--', label='Numérica com h={}'.format(h))
  plt.plot(tempo,f_analitica, 'g', label='Analítica')
  plt.title('Solução Aproximada \
  Solução Analítica')
  plt.xlabel('t')
  plt.ylabel('f(t)')
  plt.grid()
  plt.legend(loc='lower left')
  plt.show()
```

# Exemplo 1

Use o método de Euler modificado para resolver o problema de valor inicial:

\begin{equation}
\dot{u} = -0.5 u + 2+ t
\end{equation}

com $u(0)=8$.

```python
def Euler_mod(f,h=0.1,s0=1,tmax=10):
  nt = int(tmax/h)
  u = np.zeros(nt)
  tempo = np.linspace(0,tmax,nt)
  u[0] = s0
  for k in np.arange(1,nt):
      k1 = f(tempo[k-1],u[k-1])
      u_euler = u[k-1] + h*k1
      k2 = f(tempo[k],u_euler)
      u[k] = u[k-1]+h*(k1+k2)/2
  return tempo,u

```

```python
# Resolvendo o exemplo 1
f = lambda t,s: -0.5*s + 2 + t
```

```python
tempo, y = Euler_mod(f,0.1,8,10)
```

```python
f_exata = lambda t: 2*t+8*np.exp(-t/2)
```

```python
faz_grafico_euler(tempo,y,f_exata(tempo),h=0.1)

```

```python
tempo, y_euler = Euler(f,0.1,8,10)

```

```python
faz_grafico_euler(tempo,y_euler,f_exata(tempo),h=0.1)

```

```python

```


# Bond Pricing with Vasicek Model

Author:

Stanislav Khrapov

khrapovs@gmail.com

http://sites.google.com/site/khrapovs/

## Introduction

The following code is the example of adapting methodology of

Duffie, D., Pan, J., & Singleton, K. J. (2000). Transform Analysis and Asset Pricing for Affine Jump-Diffusions. Econometrica, 68(6), 1343–1376.

to the Vasicek model of interest rates.

Set up environment

```
import scipy.integrate as si
import numpy as np
import matplotlib.pylab as plt
import functools
```

## The Model

\begin{equation}
dr_{t}=\kappa\left(\mu-r_{t}\right)dt+\sigma dW_{t}.
\end{equation}

Set up parameters

```
h = 1
kappa, mu, sigma = 1.5, .5, .1
theta = [kappa, mu, sigma]
u = np.linspace(0, 1e3, 1e2)
```

## Characteristic function

### Analytic solution

The analytic solution for the characteristic function is exponentially affine

\begin{equation}
E_{t}\left[\exp\left(iur_{T}\right)\right]=\exp\left(\alpha_{t}\left(u\right)+\beta_{t}\left(u\right)r_{t}\right),
\end{equation}
with

\begin{eqnarray*}
\beta_{t}\left(u\right) & = & iue^{-\kappa h}\\
\alpha_{t}\left(u\right) & = & \mu iu\left(1-e^{-\kappa h}\right)-\frac{\sigma^{2}}{4\kappa}u^{2}\left(1-e^{-2\kappa h}\right)
\end{eqnarray*}

```
def alpha(u, h, theta):
    kappa, mu, sigma = theta
    e = np.exp( - kappa * h )
    return 1j * u * mu * (1 - e) - u ** 2 * sigma ** 2 * (1 - e ** 2) / kappa / 4

def beta(u, h, theta):
    kappa, mu, sigma = theta
    return 1j * u * np.exp( - kappa * h )
```

Evaluate these functions on the grid.

```
a = alpha(u, h, theta)
b = beta(u, h, theta)
sol_analytic = np.array([a, b])
```

### Numeric solution

We need to solve the following system of ODE:

\begin{eqnarray*}
\dot{\alpha}_{t} & = & -\kappa\mu\beta_{t}-\frac{1}{2}\sigma^{2}\beta_{t}^{2},\\
\dot{\beta}_{t} & = & \kappa\beta_{t},
\end{eqnarray*}

with terminal conditions $\beta_{T}=iu$ and $\alpha_{T}=0$.

The following function is the right hand side of the ODE system. Since the solution is actually in terms of distance $h$ to the terminal date and not in calendar time $t$, the left-hand side derivatives go with minus.

```
def df(t, x, theta):
    kappa, mu, sigma = theta
    da = - kappa * mu * x[1] - .5 * sigma ** 2 * x[1] ** 2
    db = kappa * x[1]
    return [-da, -db]
```

This function solves the system of complex valued ODE.

```
def f(df, u, h, theta):
    dt = h / 1e1
    df_args = functools.partial(df, theta = theta)
    r = si.complex_ode(df_args)
    sol = []
    for v in u:
        x0 = [0., v * 1j]
        r.set_initial_value(x0, 0)
        while r.successful() and r.t < h:
            r.integrate(r.t + dt)
        sol.append(r.y)
    return np.array(sol).T
```

Separate solution into functions $\alpha_t(u)$ and $\beta_t(u)$

```
alpha_num = lambda u, h, theta: f(df, u, h, theta)[0]
beta_num = lambda u, h, theta: f(df, u, h, theta)[1]
```

Evaluate numerical solutions at specific values.

```
a = alpha_num(u, h, theta)
b = beta_num(u, h, theta)
sol_integrate = np.array([a, b])
```

### Compare solutions

Plot them.

```
fig, axes = plt.subplots(2, 2, figsize = (8, 6), sharex = True)
for sol in [sol_analytic, sol_integrate]:
    axes[0,0].plot(u, sol[0].real)
    axes[1,0].plot(u, sol[0].imag)
    axes[0,1].plot(u, sol[1].real)
    axes[1,1].plot(u, sol[1].imag)
axes[0,0].set_title('Alpha real')
axes[1,0].set_title('Alpha imag')
axes[0,1].set_title('Beta real')
axes[1,1].set_title('Beta imag')
axes[1,0].set_xlabel('u')
axes[1,1].set_xlabel('u')
axes[0,1].legend(['Analytic','Numeric'])
plt.show()
```

### Compare characteristic functions

Define conditional characteristic functions.

```
rt = 1.5
psi_analytic = lambda u: np.exp( alpha(u, h, theta) + beta(u, h, theta) * rt )
psi_numeric = lambda u: np.exp( alpha_num(u, h, theta) + beta_num(u, h, theta) * rt )
```

Plot them.

```
x = np.linspace(0, 60, 1e2)
psi_a = psi_analytic(x)
psi_n = psi_numeric(x)

fig, axes = plt.subplots(2, 1, figsize = (8, 6), sharex = True)
for psi in [psi_a, psi_n]:
    axes[0].plot(x, psi.real)
    axes[1].plot(x, psi.imag)
axes[1].set_xlabel('u')
axes[0].set_title('Real')
axes[1].set_title('Imag')
axes[0].legend(['Analytic','Numeric'])
plt.show()
```

### Compare transition densities

Fast Fourier inverse of charcteristic function. Returns density.

```
def CFinverse(psi, A = -1e2, B = 1e2, N = 1e5):
    eta = (N - 1) / N * 2 * np.pi / (B - A)
    lmbd = (B - A) / (N - 1)
    k = np.arange(0, N)
    x = A + lmbd * k
    v = eta * k
    y = psi(v) * np.exp(- 1j * A * v) * eta / np.pi
    f = np.fft.fft(y)
    return x, f.real
```

Compute densities. Note how much time was spent on each operation.

```
N = 1e4
%time x, density_analytic = CFinverse(psi_analytic, N = N)
%time x, density_numeric = CFinverse(psi_numeric, N = N)
```

    CPU times: user 8.27 ms, sys: 7 µs, total: 8.27 ms
    Wall time: 7.86 ms
    CPU times: user 40.8 s, sys: 15.9 ms, total: 40.8 s
    Wall time: 40.8 s

Plot transition densities.

```
plt.plot(x, density_analytic)
plt.plot(x, density_numeric)
plt.xlim([0, 2])
plt.legend(['Analytic','Numeric'])
plt.xlabel('x')
plt.ylabel('f(x)')
plt.show()
```

# Bond Pricing

Thanks to Duffie, Pan, & Singleton (2000, Econometrica) we know how to compute the following transform for AJD models:

\begin{equation}
\mathbb{T}^{\chi}\left(u,Y_{t},t,T\right)=E_{t}\left[\exp\left(-\int_{t}^{T}r_{s}ds\right)\exp\left(ur_{T}\right)\right]
=\exp\left(\alpha_{t}(u)+\beta_{t}(u)r_{t}\right).
\end{equation}

But in order to compute the bond price we obviously need to set $u=0$ in this transform:

\begin{equation}
B_{t,T}=E_{t}\left[\exp\left(-\int_{t}^{T}r_{s}ds\right)\right]=\mathbb{T}^{\chi}\left(0,r_{t},t,T\right)=\exp\left(\alpha_{t}\left(0\right)+\beta_{t}\left(0\right)r_{t}\right),
\end{equation}

In case of Vasicek model we need to solve the following system of ODE

\begin{eqnarray*}
\dot{\beta}_{t} & = & 1+\kappa\beta_{t},\\
\dot{\alpha}_{t} & = & -\kappa\mu\beta_{t}-\frac{1}{2}\sigma^{2}\beta_{t}^{2},
\end{eqnarray*}
with terminal conditions $\beta_{T}=u$ and $\alpha_{T}=0$.

## Analytic solution

The analytic solution to this problem is known to be

\begin{eqnarray*}
\beta_{t}\left(0\right) & = & \frac{1}{\kappa}\left(e^{-\kappa h}-1\right),\\
\alpha_{t}\left(0\right) & = & -\left(\mu-\frac{\sigma^{2}}{2\kappa^{2}}\right)\left(\beta_{t}\left(0\right)+h\right)-\frac{\sigma^{2}}{4\kappa}\beta_{t}^{2}\left(0\right).
\end{eqnarray*}
It is coded below.

```
def beta(h, theta):
    kappa, mu, sigma = theta
    return ( np.exp( - kappa * h ) - 1 ) / kappa

def alpha(h, theta):
    kappa, mu, sigma = theta
    a = - (mu - .5 * sigma ** 2 / kappa ** 2) * (beta(h, theta) + h)
    a -= .25 * sigma ** 2 / kappa * beta(h, theta) ** 2
    return  a
```

Most of the time we are actually interested in yields, rather than in bond prices directly.

\begin{equation}
y_{t,T}= -\frac{1}{h}\log B_{t,T}=-\frac{1}{h}\alpha_{t}\left(0\right)-\frac{1}{h}\beta_{t}\left(0\right)r_{t}
=A\left(h\right) + B\left(h\right)r_t
\end{equation}

```
A_exact = lambda h, theta: - alpha(h, theta) / h
B_exact = lambda h, theta: - beta(h, theta) / h
```

We can compute these for the interval of $H$ days in case of analytic solution.

```
H = np.linspace(.1, 10, 1e2)
a = A_exact(H, theta)
b = B_exact(H, theta)
sol_analytic = np.array([a, b])
```

## Numeric solution

For numerical solution we need to set up the system of ODEs.

```
def df(t, x, theta):
    kappa, mu, sigma = theta
    da = - kappa * mu * x[1] - .5 * sigma ** 2 * x[1] ** 2
    db = 1 + kappa * x[1]
    return [-da, -db]
```

The solver is adjusted since for bond pricign we only need the value of functions $\alpha_t(u)$ and $\beta_t(u)$ at zero but for variety of $h$.

```
def f(df, H, theta):
    df_args = functools.partial(df, theta = theta)
    r = si.ode(df_args)
    sol = []
    for h in H:
        dt = h / 1e2
        x0 = [0, 0]
        r.set_initial_value(x0, 0)
        while r.successful() and r.t < h:
            r.integrate(r.t + dt)
        sol.append(r.y)
    return np.array(sol).T
```

The solutions are written as function below.

```
alpha_num = lambda H, theta: f(df, H, theta)[0]
beta_num = lambda H, theta: f(df, H, theta)[1]

A_numeric = lambda H, theta: - alpha_num(H, theta) / H
B_numeric = lambda H, theta: - beta_num(H, theta) / H
```

We can evaluate these at a specific time interval.

```
a = A_numeric(H, theta)
b = B_numeric(H, theta)
sol_integrate = np.array([a, b])
```

## Comparison

Now plot the solutions and the resulting yield.

```
fig, axes = plt.subplots(3, 1, figsize = (8, 6), sharex = True)
for sol in [sol_analytic, sol_integrate]:
    axes[0].plot(H, sol[0])
    axes[1].plot(H, sol[1])
    axes[2].plot(H, sol[0] + sol[1] * rt)
axes[0].set_title('A')
axes[1].set_title('B')
axes[2].set_title('Yields conditional on $r_0=$ ' + str(rt))
axes[2].legend(['Analytic','Numeric'])
axes[2].set_xlabel('H')
axes[2].set_ylabel('%')
plt.show()
```


```python
import numpy as np
import ipyvolume as ipv
import sympy as sy
from sympy.geometry import Plane as syPlane, Point3D as syPoint3D
import tqdm
```

```python
xyz = np.array((np.random.random(1000), np.random.random(1000), np.random.random(1000)))
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
ipv.show()
```

Let's see how we can define a thick plane (i.e. two co-planar planes separated by a certain distance) by selecting a point in space and then adding a normal vector and a thickness.

It would be nice if we could get the initial plane-point by clicking on a point in the ipyvolume plot. Let's look into that later.

If the normal vector $\boldsymbol{n}$ is $(a,b,c)$ then a plane in $(x,y,z)$ is parameterized by

$$
ax+by+cz+d=0
$$

You can either find three points $P_i$ in the plane and use the cross product of two differences of points to find a, b, c, because

$$
\boldsymbol{n} = (P_3-P_1) \times (P_2-P_1)
$$

or if you define $\boldsymbol{n}$ yourself, you can just directly find $d$ by plugging in any point in the plane in the previous equation.

Then you can use the edges of your box to define the extreme points of your plane to be plotted.

See e.g. https://stackoverflow.com/a/13473027/1199693

```python
def normalize_vector(n):
    n_size = n[0] * n[0] + n[1] * n[1] + n[2] * n[2]
    n = (n[0]/n_size, n[1]/n_size, n[2]/n_size)
    return n

def plot_plane(p, n):
    """
    Draw a plane.

    p: a point in the plane (x, y, z; any iterable)
    n: the normal vector to the plane (x, y, z; any iterable)
    """
    n = normalize_vector(n)

    # find d
    d = -(n[0] * p[0] + n[1] * p[1] + n[2] * p[2])

    # get box limits in two dimensions
    x_lim = (0, 1)
    z_lim = (0, 1)
    x, z = np.meshgrid(x_lim, z_lim)

    # find corresponding y coordinates
    y = -(n[0] * x + n[2] * z + d)/n[1]

    # plot
    ipv.plot_surface(x, y, z)

def thick_plane_points(p, n, thickness):
    """
    Given a point, a normal vector and a thickness, return two points
    along the normal that are `thickness` apart.
    """
    n = normalize_vector(n)

    p1 = (p[0] + 0.5 * thickness * n[0],
          p[1] + 0.5 * thickness * n[1],
          p[2] + 0.5 * thickness * n[2])
    p2 = (p[0] - 0.5 * thickness * n[0],
          p[1] - 0.5 * thickness * n[1],
          p[2] - 0.5 * thickness * n[2])
    return p1, p2

def plot_thick_plane(p, n, thickness=0):
    """
    Draw two co-planar planes, separated by a distance `thickness`.

    p: a point in the plane (x, y, z; any iterable)
    n: the normal vector to the plane (x, y, z; any iterable)
    thickness: the distance between the two co-planar planes
    """
    if thickness <= 0:
        plot_plane(point, normal)
    else:
        # find points in the two planes and plot them
        p1, p2 = thick_plane_points(p, n, thickness)

        plot_plane(p1, n)
        plot_plane(p2, n)
```

```python
point = (0.5, 0.5, 0.5)
normal = (0, 1, 0)  # make it normalized to one
thickness = 0.1

plot_thick_plane(point, normal, thickness=thickness)
```

Now, we also need to actually filter the points. Let's do that with Sympy.

```python
def filter_points_plane(points_xyz, plane_point, plane_normal, plane_thickness):
    point1, point2 = thick_plane_points(plane_point, plane_normal, plane_thickness)
    plane1 = syPlane(syPoint3D(point1), normal_vector=plane_normal)
    plane2 = syPlane(syPoint3D(point2), normal_vector=plane_normal)

    p_filtered = []
    for p_i in tqdm.tqdm(points_xyz.T):
        sy_point_i = syPoint3D(tuple(p_i))
        if plane1.distance(sy_point_i) <= thickness and plane2.distance(sy_point_i) <= thickness:
            p_filtered.append(p_i)
    return p_filtered
```

```python
p_filtered = filter_points_plane(xyz, point, normal, thickness)
```

```python
len(p_filtered)
```

```python
ipv.scatter(*np.array(p_filtered).T, marker='circle_2d', color='blue')
```

```python

```


# Integrating Orbits

This is a demonstration of Euler's method for integrating orbits.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

We consider low mass objects orbiting the Sun.  We work in units of AU, yr, and solar masses.  From Kepler's third law:

\begin{equation}
4 \pi^2 a^3 = G M P^2
\end{equation}

If $a$ is in AU, $P$ is in yr, and $M$ is in solar masses, then
\begin{equation}
a^3 = P^2
\end{equation}
and therefore
\begin{equation}
4 \pi^2 = G
\end{equation}

We work in coordinates with the Sun at the origin.

Equations:
\begin{align*}
\frac{dx}{dt} &= u \\
\frac{dy}{dt} &= v \\
\frac{du}{dt} &= - \frac{GMx}{r^3} \\
\frac{dv}{dt} &= - \frac{GMy}{r^3}
\end{align*}

```python
# assuming 1 solar mass
GM = 4*np.pi**2
```

```python
def rhs(x, y, u, v):
    """ RHS of the equations of motion.  X is the input coordinate
        vector and V is the input velocity vector """

    # current radius
    r = np.sqrt(x**2 + y**2)

    # position derivatives
    xdot = u
    ydot = v

    # velocity derivatives
    udot = -GM*x/r**3
    vdot = -GM*y/r**3

    return xdot, ydot, udot, vdot

```

A simple class for storing the solution history and plotting it

```python
class Orbit:
    def __init__(self):
        self.t = []
        self.x = []
        self.y = []
        self.u = []
        self.v = []

    def add_point(self, time, xnew, ynew, unew, vnew):
        self.x.append(xnew)
        self.y.append(ynew)
        self.u.append(unew)
        self.v.append(vnew)

    def plot(self):
        fig = plt.figure()
        ax = fig.add_subplot(111)

        _ = ax.plot(orbit.x, orbit.y)
        _ = ax.set_aspect("equal", "datalim")
        return fig
```

```python
def integrate(x0, y0, u0, v0, tmax, dt):
    """integrate the orbit with initial conditions X0, V0, using a
    timestep dt for a duration tmax"""

    t = 0
    x = x0
    y = y0
    u = u0
    v = v0

    o = Orbit()
    o.add_point(t, x, y, u, v)

    while (t < tmax):

        xdot, ydot, udot, vdot = rhs(x, y, u, v)
        x += dt * xdot
        y += dt * ydot
        u += dt * udot
        v += dt * vdot

        t += dt

        o.add_point(t, x, y, u, v)

        if t + dt > tmax:
            dt = tmax - t

    return o
```

Initial conditions are at perihelion with counter clockwise orbit:
\begin{align*}
x &= 0 \\
y &= a(1-e) \\
u &= -\sqrt{\frac{GM}{a} \frac{1+e}{1-e}} \\
v &= 0
\end{align*}

```python
def init_conditions(a, e):
    x0 = 0.0
    y0 = a*(1.0 - e)
    u0 = -np.sqrt((GM/a)* (1.0 + e) / (1.0 - e))
    v0 = 0.0

    return x0, y0, u0, v0
```

To integrate the orbit, we set the semi-major axis and eccentricity, specify one period (`tmax = 1`), and a timestep

```python
a = 1.0
e = 0.0
x0, y0, u0, v0 = init_conditions(a, e)

orbit = integrate(x0, y0, u0, v0, 1.0, 0.00001)

```

```python
fig = orbit.plot()
```

```python

```


# Problem Formulation

$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}$

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

%matplotlib inline
SEED = 0
```

## Mutual information estimation

**How to formulate the problem of mutual information estimation?**

The problem of estimating the mutual information is:

---

Given $n$ samples

$$(\R{X}_1,\R{Y}_1),\dots, (\R{X}_n,\R{Y}_n) \stackrel{iid}{\sim} P_{\R{X},\R{Y}}\in \mc{P}(\mc{X},\mc{Y})$$

i.i.d. drawn from an *unknown* probability measure $P_{\R{X},\R{Y}}$ from the space $\mc{X}\times \mc{Y}$, estimate the *mutual information (MI)*

$$
\begin{align}
I(\R{X}\wedge\R{Y}) &:= E\left[\log \frac{d P_{\R{X},\R{Y}}(\R{X},\R{Y})}{d (P_{\R{X}} \times P_{\R{Y}})(\R{X},\R{Y})} \right].
\end{align}
$$ (MI)

---

Run the following code, which uses `numpy` to
- generate i.i.d. samples from a multivariate gaussian distribution, and
- store the samples as numpy arrays assigned to `XY`.

```python
# Seeded random number generator for reproducibility
XY_rng = np.random.default_rng(SEED)

# Sampling from an unknown probability measure
rho = 1 - 0.19 * XY_rng.random()
mean, cov, n = [0, 0], [[1, rho], [rho, 1]], 1000
XY = XY_rng.multivariate_normal(mean, cov, n)
plt.scatter(XY[:, 0], XY[:, 1], s=2)
plt.show()
```

You may find more details from the official documentations of [`multivariate_normal`][mn] and [`scatter`][sc].

[mn]: https://numpy.org/doc/stable/reference/random/generated/numpy.random.multivariate_normal.html
[sc]: https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.scatter.html

You can also get help directly in [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/getting_started/overview.html):

- **Docstring**:
  - Move the cursor to the object and
    - click `Help->Show Contextual Help` or
    - click `Shift-Tab` if you have limited screen space.

- **Directory**:
  - Right click on a notebook and choose `New Console for Notebook`.
  - Run `dir(obj)` for a previously defined object `obj` to see the available methods/properties of `obj`.

---

**Exercise** What is unknown about the above sampling distribution?

The density is

$$
\frac{d P_{\R{X},\R{Y}}}{dxdy} = \mc{N}_{\M{0},\left[\begin{smallmatrix}1 & \rho \\ \rho & 1\end{smallmatrix}\right]}(x,y)
$$

but $\rho$ is unknown (uniformly random over $[0.8,0.99)$).

---

To show the data samples using `pandas`:

```python
XY_df = pd.DataFrame(XY, columns=["X", "Y"])
XY_df
```

      X
      Y

      0
      -0.029492
      0.285584

      1
      0.030093
      -0.233446

      2
      -0.671256
      -0.029712

      3
      -0.744867
      -1.091092

      4
      1.379858
      1.073218

      ...
      ...
      ...

      995
      -1.740407
      -0.884540

      996
      -0.370652
      0.244189

      997
      -0.590843
      -0.864294

      998
      2.180489
      1.733067

      999
      -0.461016
      -0.254751

1000 rows × 2 columns

To plot the data using `seaborn`:

```python
def plot_samples_with_kde(df, **kwargs):
    p = sns.PairGrid(df, **kwargs)
    p.map_lower(sns.scatterplot, s=2)  # scatter plot of samples
    p.map_upper(sns.kdeplot)  # kernel density estimate for pXY
    p.map_diag(sns.kdeplot)  # kde for pX and pY
    return p

plot_samples_with_kde(XY_df)
plt.show()
```

---

**Exercise** Complete the following code by replacing the blanks `___` so that `XY_ref` stores the i.i.d. samples of $(\R{X}',\R{Y}')$ where $\R{X}'$ and $\R{Y}'$ are zero-mean independent gaussian random variables with unit variance.

```Python
...
cov_ref, n_ = ___, n
XY_ref = XY_ref_rng_ref.___(mean, ___, n_)
...
```

```python
XY_ref_rng = np.random.default_rng(SEED)
### BEGIN SOLUTION
cov_ref, n_ = [[1, 0], [0, 1]], n
XY_ref = XY_ref_rng.multivariate_normal(mean, cov_ref, n_)
### END SOLUTION
XY_ref_df = pd.DataFrame(XY_ref, columns=["X'", "Y'"])
plot_samples_with_kde(XY_ref_df)
plt.show()
```

---

## Divergence estimation

**Can we generalize the problem further?**

Estimating MI may be viewed as a special case of the following problem:

---

Estimate the KL *divergence*

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) &:= E\left[\log \frac{d P_{\R{Z}}(\R{Z})}{d P_{\R{Z}'}(\R{Z})} \right]
\end{align}
$$ (D)

using
- a sequence $\R{Z}^n:=(\R{Z}_1,\dots, \R{Z}_n)\sim P_{\R{Z}}^n$ of i.i.d. samples from $P_{\R{Z}}$ if $P_{\R{Z}}$ is unknown, and
- another sequence ${\R{Z}'}^{n'}\sim P_{\R{Z}'}^{n'}$ of i.i.d. samples from $P_{\R{Z}'}$  if $P_{\R{Z}'}$, the *reference measure* of $P_{\R{Z}}$, is also unknown.

---

---
**Exercise**

Although $\R{X}^n$ and $\R{Y}^n$ for MI estimation should have the same length, $\R{Z}^n$ and ${\R{Z}'}^{n'}$ can have different lengths, i.e., $n \not\equiv n'$. Why?

**Solution** The dependency between $\R{Z}$ and $\R{Z}'$ does not affect the divergence.

---

Regarding the mutual information as a divergence from joint to product distributions, the problem can be further generalized to estimtate other divergences such as the $f$-divergence:

For a strictly convex function $f$ with $f(1)=0$,

$$
\begin{align}
D_f(P_{\R{Z}}\|P_{\R{Z}'}) &:= E\left[ f\left(\frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')}\right) \right].
\end{align}
$$ (f-D)

$f$-divergence in {eq}`f-D` reduces to KL divergence when $f=u \log u$:

$$
\begin{align}
E\left[ \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')} \log \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')}  \right] &= \int_{\mc{Z}} \color{gray}{d P_{\R{Z}'}(z)} \cdot \frac{d P_{\R{Z}}(z)}{\color{gray}{d P_{\R{Z}'}(z)}} \log \frac{d P_{\R{Z}}(z)}{d P_{\R{Z}'}(z)}.
\end{align}
$$

---

**Exercise**

Show that $D_f(P_{\R{Z}}\|P_{\R{Z}'})\geq 0$ with equality iff $P_{\R{Z}}=P_{\R{Z}'}$ using Jensen's inequality and the properties of $f$.

**Solution**

It is a valid divergence because, by Jensen's inequality,

$$
D_f(P_{\R{Z}}\|P_{\R{Z}'}) \geq  f\bigg( \underbrace{E\left[ \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')} \right]}_{=1}\bigg) = 0
$$

with equality iff $P_{\R{Z}}=P_{\R{Z}'}$.

---

Regarding the divergence as an expectation, it is approximated by the sample average:

$$
\begin{align}
D_f(P_{\R{Z}}\|P_{\R{Z}'}) &\approx
\frac1n \sum_{i\in [n]} f\left(\frac{d P_{\R{Z}}(\R{Z}'_i)}{d P_{\R{Z}'}(\R{Z}'_i)}\right).
\end{align}
$$ (avg-f-D)

However, this is not a valid estimate because it involves the unknown measures $P_{\R{Z}}$ and $P_{\R{Z}'}$.

One may further estimate the *density ratio*

$$
\begin{align}
z \mapsto \frac{d P_{\R{Z}}(z)}{d P_{\R{Z}'}(z)}
\end{align}
$$ (dP-ratio)

or estimate the density defined respective to some reference measure $\mu$:

$$
\begin{align}
p_{\R{Z}}&:=\frac{dP_{\R{Z}}}{d\mu} \in \mc{P}_{\mu}(\mc{Z}).
\end{align}
$$ (density)


# Announcements
- No Problem Set this week, Problem Set 4 will be posted on 9/28.
- Stay on at the end of lecture if you want to ask questions about Problem Set 3.

# Ordinary Differential Equations - higher order methods

Based on notes and notebooks by Niels Henrik Aase, Thorvald Ballestad, Vasilis Paschalidis and Jon Andreas Støvneng

## Algorithms for initial value problem ODEs

Assume we have a first-order differential equation which can be expressed in the form

$$ \frac{dy}{dt} = g(y,t) $$

We will solve this on constant-interval mesh of the independent variable $t$ defined by

$$ t_n = t_0 + n h $$

### Forward-Euler method

In Lecture 10 we derived Euler's method, which simply solves the first-order forward difference approximation to $dy/dt$

$$ \frac{y_{i+1}-y_i}{h} = g(y_i,t_i)$$

as

$$ y_{i+1} = y_i + h g(y_i,t_i) \label{Euler_fwd}\tag{3}$$

```python
# Importing the necessary libraries
import numpy as np # NumPy is used to generate arrays and to perform some mathematical operations
import matplotlib.pyplot as plt # Used for plotting results

```

```python
def forwardEuler_step(t, y, h, g, *P):
    """
    Implements a single step of the forward-Euler finite-difference scheme
    Parameters:
            t: time t
            y: Numerical approximation of y at time t
            h: Step size
            g: RHS of our ODE (RHS = Right hand side). Can be any function with signature g(t,y,*P).
            *P: tuple of parameters, arguments to g
        Returns:
            next_y: Numerical approximation of y at time t+h
    """
    next_y = y + h*g(t, y, *P)
    return next_y

```

We now need some sort of framework which will take this function and do the integration for us. Let's rewrite `full_Euler` from Lecture 10 to be more general:

```python
def odeSolve(t0, y0, tmax, h, g, method, *P):
    """ A full numerical aproximation of an ODE in a set time interval. Performs consecutive steps of `method`
    with step size h from start time until the end time. Also takes into account the initial values of the ODE

    Parameters:
            t0: start time
            y0 : Initial condition for y at t = t0
            tmax: The end of the interval where the `method` is integrated, t_N
            h: Step size
            g: RHS of our ODE (RHS = Right hand side). Can be any function with signature g(t,y,*P).
            *P: tuple of parameters, arguments to g
        Returns:
            t_list: Evenly spaced discrete list of time with spacing h.
                    Starting time = start_t, and end time = end_t
            y_list: Numerical approximation of y at times t_list
    """
    # make the t-mesh; guarantees we stop precisely at tmax
    t_list = np.arange(t0,tmax+h,h)
    # allocate space for the solution
    y_list = np.zeros_like(t_list)
    # set the initial condition
    y_list[0] = y0

    # find out the size of the t-mesh, and then integrate forward one meshpoint per iteration of the loop
    n, = t_list.shape
    for i in range(0,n-1):
        y_list[i+1] = method(t_list[i], y_list[i], h, g, *P)

    # return the solution
    return t_list,y_list
```

Armed with this machinery, let's set up another simple problem and try it out.

Last time, we looked at exponential growth, let's solve exponential decay this time:

$$ \frac{dy}{dt} = - c y, \quad y[0] = 1 $$

First, we provide a function to implement the RHS:

```python
def expRHS(t, y, c):
    """
    Implements the RHS (y'(x)) of the DE
    """
    return -c*y
```

Now we set up the problem to compute and plot the result, along with a plot of the magnitude of the fractional error

### Runge-Kutta Schemes

The idea of the Runge-Kutta schemes is to take advantage of derivative information at the times between $t_i$ and $t_{i+1}$ to increase the order of accuracy.

For example, in the midpoint method, the derivative at the initial time is used to approximate the derivative at the midpoint of the interval, $f(y_i+\frac{1}{2}hf(y_i,t_i), t_i+\frac{1}{2}h)$. The derivative at the midpoint is then used to advance the solution to the next step. The method can be written in two stages $k_i$,

$$ \begin{aligned} \begin{array}{l} k_1 = h f(y_i,t_i)\\ k_2 = h f(y_i+\frac{1}{2}k_1, t_n+\frac{1}{2}h)\\ y_{i+1} = y_i + k_2 \end{array} \end{aligned}\label{RK2}\tag{4} $$

The midpoint method is known as a __2nd-order Runge-Kutta__ formula.

In general, an explicit 2-stage Runge-Kutta method can be written as,
$$ \begin{array}{l} k_1 = h f(y_n,t_n)\\ k_2 = h f(y_n+b_{21}k_1, t_n+a_2h)\ \\ y_{n+1} = y_n + c_1k_1 +c_2k_2 \label{explicitrk2}\tag{5}\end{array} $$

The scheme is said to be *explicit* since a given stage does not depend *implicitly* on itself, as in the backward Euler method , or on a later stage.

Other explicit second-order schemes can be derived by comparing Eq.(\ref{explicitrk2}) to other second-order expansions and matching terms to determine the coefficients $a_2$, $b_{21}$, $c_1$ and $c_2$.

### Explicit Fourth-Order Runge-Kutta Method

Explicit Runge-Kutta methods are popular as each stage can be calculated with one function evaluation. In contrast, implicit Runge-Kutta methods usually involves solving a non-linear system of equations in order to evaluate the stages. As a result, explicit schemes are much less expensive to implement than implicit schemes.

The higher-order Runge-Kutta methods can be derived by in manner similar to the midpoint formula. An s-stage method is compared to a Taylor method and the terms are matched up to the desired order.

As it happens to be, The Fourth Order Runge-Kutta Method uses three such test-points and is the most widely used Runge-Kutta Method. You might ask why we don't use five, ten or even more test-points, and the answer is quite simple: It is not computationally free to calculate all these test-points, and the gain in accuracy rapidly decreases beyond the fourth order of the method. That is, if high precision is of such importance that you would require a tenth-order Runge-Kutta, then you're better off reducing the step size $h$, than increasing the order of the method.

Also, there exists other more sophisticated methods which can be both faster and more accurate for equivalent choices of $h$, but obviously, may be a lot more complicated to implement. See for instance Richardson Extrapolation, the Bulirsch-Stoer method, Multistep methods, Multivalue methods and Predictor-Corrector methods.

The classic fourth-order Runge-Kutta formula is:
$$ \begin{array}{l} k_1 = h f(y_n,t_n)\\ k_2 = h f(y_n+\frac{k_1}{2}, t_n+\frac{h}{2})\\ k_3 = h f(y_n+\frac{k_2}{2}, t_n+\frac{h}{2})\\ k_4 = h f(y_n+k_3, t_n+h)\\ y_{n+1} = y_n + \frac{k_1}{6}+ \frac{k_2}{3}+ \frac{k_3}{3} + \frac{k_4}{6} \label{RK4}\tag{6}\end{array} $$

```python
def RK2_step(t, y, h, g, *P):
    """
    Implements a single step of the second-order, explicit midpoint method
    """
    thalf = t + 0.5*h
    k1 = h * g(t, y, *P)
    k2 = h * g(thalf, y + 0.5*k1, *P)

    return y +k2
```

```python
def RK4_step(t, y, h, g, *P):
    """
    Implements a single step of a fourth-order, explicit Runge-Kutta scheme
    """
    thalf = t + 0.5*h
    k1 = h * g(t, y, *P)
    k2 = h * g(thalf, y + 0.5*k1, *P)
    k3 = h * g(thalf, y + 0.5*k2, *P)
    k4 = h * g(t + h, y + k3, *P)
    return y + (k1 + 2*k2 + 2*k3 + k4)/6
```

```python
# set up problem
c = 1.0
h = 0.5
t0 = 0.0
y0 = 1.0
tmax = 5.0

# call the solver for RK2
t, y = odeSolve(t0, y0, tmax, h, expRHS, RK2_step, c)

# plot the result
fig,ax = plt.subplots(1,2)
ans = np.exp(-c*t)
ax[0].plot(t,ans,'r')
ax[0].set_xlabel('t')
ax[0].set_ylabel('y')

ax[0].plot(t,y,'o','RK2')
err_RK2 = np.abs((ans-y)/ans)

# call the solver for Euler
t, y = odeSolve(t0, y0, tmax, h, expRHS, forwardEuler_step, c)
ax[0].plot(t,y,'o','Euler')
err = np.abs((ans-y)/ans)

# call the solver for RK2
t, y4 = odeSolve(t0, y0, tmax, h, expRHS, RK4_step, c)
ax[0].plot(t,y4,'o','RK4')
err_RK4 = np.abs((ans-y4)/ans)

# along with the errors
err_RK2 = np.abs((ans-y)/ans)
ax[1].plot(t, err_RK2, 'o',label = "RK2")
ax[1].plot(t, err_RK4, 'o',label = "RK4")
ax[1].plot(t, err, 'o',label = "Euler")
ax[1].set_xlabel('t')
ax[1].set_ylabel('fractional error')
ax[1].legend()
# now also overplot the error we calculated for forward-Euler

# this gives better spacing between axes
plt.tight_layout()
plt.show()
```

### Systems of First-Order ODEs

Next, we turn to systems of ODE's. We'll take as our example the Lotke-Volterra equations, a simple model of population dynamics in an ecosystem (with many other uses as well).

Imagine a population of rabbits and of foxes on a small island. The rabbits eat a plentiful supply of grass and
would breed like, well, rabbits, with their population increasing exponentially with time in the absence of preditors. The foxes eat the rabbits, and would die out exponentially in time with no food supply. The rate at which foxes eat rabbits depends upon the product of the fox and rabbit populations.

The equations for the population of the rabbits $R$ and foxes $F$ in this simple model is then

\begin{eqnarray*}
\frac{dR}{dt} &= \alpha R - \beta R F \\
\frac{dF}{dt} &= \delta R F - \gamma F
\end{eqnarray*}

Without the cross terms in $RF$, these are just two decay equations of the form we have used as an example above.

A random set of parameters (I am not a biologist!) might be that a rabbit lives four years, so $\alpha=1/4$ and
a fox lives 10 years, so $\gamma=1/10$. Let's pick the other parameters as $\beta = 1$ and $\delta = 1/4$.

We can express the unknown populations as a vector of length two: $y = (R, F)$. The rate of change of populations then can also be expressed as a vector $dy/dt = (dR/dt, DF/dt)$. With such a definition, we can write the RHS function of our system as

```python
def lvRHS(t, y, *P):
    # Lotke-Volterra system RHS

    # unpack the parameters from the array P
    alpha, beta, gamma, delta = P

    # make temporary variables with rabbit and fox populations
    R = y[0]
    F = y[1]

    # LV system
    dRdt = alpha * R - beta * R * F
    dFdt = delta * R * F - gamma * F

    # return an array of derivatives with same order as input vector
    return np.array([ dRdt, dFdt ])
```

We now have to generalize our odeSolve function to allow more than one equation

```python
def odeSolve(t0, y0, tmax, h, RHS, method, *P):
    """
    ODE driver with constant step-size, allowing systems of ODE's
    """
    # make array of times and find length of array
    t = np.arange(t0,tmax+h,h)
    ntimes,  = t.shape

    # find out if we are solving a scalar ODE or a system of ODEs, and allocate space accordingly

    if type(y0) in [int, float]:  # check if primitive type -- means only one eqn
        neqn = 1
        y = np.zeros( ntimes )
    else:                         # otherwise assume a numpy array -- a system of more than one eqn
        neqn, = y0.shape
        y = np.zeros( (ntimes, neqn) )

    # set first element of solution to initial conditions (possibly a vector)
    y[0] = y0

    # march on...
    for i in range(0,ntimes-1):
        y[i+1] = method(t[i], y[i], h, RHS, *P)

    return t,y

```

Now we can solve our system of two coupled ODEs. Note that the solution is now a vector of 2D vectors... the first index is the solution time, the second the variable:

```python
alpha = 1.0
beta = 0.025
gamma = 0.4
delta = 0.01

h = 0.2
t0 = 0.0
y0 = np.array([ 30, 10 ])
tmax = 50

# call the solver
t, y = odeSolve(t0, y0, tmax, h, lvRHS, RK4_step, alpha, beta, gamma, delta)

fig,ax = plt.subplots()
ax.plot(t,y[:,0],'b', label='prey')
ax.plot(t,y[:,1],'r', label='preditor')
ax.set_xlabel('time')
ax.set_ylabel('population')
ax.legend()

plt.tight_layout()
plt.show()
```

### Higher Order Derivatives and Sets of 1st order ODEs

The trick to solving ODEs with higher derivatives is turning them into systems of first-order ODEs.

As a simple example, consider the second-order differential equation describing the van der Pol oscillator

$$ \frac{d^2 x}{dt^2} - a (1-x^2) \frac{dx}{dt} + x = 0 $$

We turn this into a pair of first-order ODEs by defining an auxiliary function $v(t) = dx/dt$ and writing the system as

\begin{align}
\begin{split}
\frac{dv}{dt} &= a (1-x^2) v - x\\
\frac{dx}{dt} &= v
\end{split}
\end{align}

Note that there are only functions (and the independent variable) on the RHS; all "differentials" are on the LHS.

Now that we have a system of first-order equations ,we can proceed as above. A function describing
the RHS of this system is

```python
def vdpRHS(t, y, a):
    # we store our function as the array [x, x']
    return np.array([

        y[1],                          # dx/dt = v
        a*(1-y[0]**2)*y[1] - y[0]      # dv/dt = a*(1-x**2)*v - x

    ])
```

```python
a = 15 # parameter

h = 0.01
t0 = 0.0
y0 = np.array([ 0,  1])
tmax = 50

# call the solver
t, y = odeSolve(t0, y0, tmax, h, vdpRHS, RK4_step, a)

fig,ax = plt.subplots()
ax.plot(t,y[:,0],'b', label='x')
ax.plot(t,y[:,1],'r--', label='v')
ax.set_xlabel('time')
ax.legend()
ax.set_title(f"van der Pol Oscillator for a={a}")

plt.tight_layout()
plt.show()

```

A somewhat more complex example is the Lane-Emden equation, which is really just Poisson's equation in spherical symmetry for the graviational potential of a self-gravitating fluid whose pressure is related to its density as $P\propto\rho^\gamma$. Such a system is called a _polytrope_, and is often used in astrophysics as a simple model for the structure of system such as a a star in which outward pressure and inward gravity are in equilibrium.

Let $\xi$ be the dimensionless radius of the system, and let $\theta$ be related to the density as
$\rho = \rho_c \theta^n$, where $\rho_c$ is the density at the origin and $n = 1/(\gamma-1)$. We then have the dimensionless second-order differential equation

$$ \frac{1}{\xi^2}\frac{d}{d\xi}\left(\xi^2\frac{d\theta}{d\xi}\right) + \theta^n = 0 $$

Note that the first term is just the divergence $\nabla\cdot\theta$ in spherical symmetry.

If we expand out the first term, we have

$$ \frac{d^2\theta}{d\xi^2} + \frac{2}{\xi}\frac{d\theta}{d\xi} + \theta^n = 0 $$

Defining an auxiliary function $v(\xi) = d\theta/d\xi$, we can then convert this into a system of two first-order ODEs:

\begin{align}
\begin{split}
\frac{dv}{d\xi} &= -\frac{2}{\xi} v - \theta^n \\
\frac{d\theta}{d\xi} &= v
\end{split}
\end{align}

Again, we have "derivatives" only on the LHS and no derivatives on the RHS of our system.

Looking at this expression, one can see right away that at the origin $\xi=0$ we will have a numerical problem; we are dividing by zero.
Analytically, this is not a problem, since $v/\xi\rightarrow0$ as $\xi\rightarrow0$, but here we need to address this numerically.

The first approach is to take care of the problem in our RHS function:

```python
def leRHS(x, y, n):

    dthetadx = y[1]

    if x==0:
        dvdx = -y[0]**n
    else:
        dvdx = -2/x*y[1] - y[0]**n

    return np.array([ dthetadx, dvdx ])
```

This is somewhat clunky, however, and you would first have to convince yourself that in fact $v(\xi)\rightarrow0$ faster than $\xi$ (don't just take my word for it!).

Instead, we could use a more direct RHS function

```python
def leRHS(x, y, n):

    dthetadx = y[1]

    dvdx = -2/x*y[1] - y[0]**n

    return np.array([ dthetadx, dvdx ])
```

and expand the solution in a Taylor series about the origin to get a starting value for our numerical integration at a small distance away from the origin. To do this, write

$$\theta(\xi) = a_0 + a_1 \xi + a_2 \xi^2 + \dots$$

The first thing to notice is that, by symmetry, only even powers of $\xi$ will appear in the solution.
Thus we will have

$$ \theta(\xi) = a_0 + a_2 \xi^2 + a_4 \xi^4 + \dots$$

By the boundary condition $\theta(0) = 1$, we have immediately that $a_0 = 1$.

Next, substitute $\theta(\xi) = 1 + a_2 \xi^2 + a_4 \xi ^4 + O(\xi^6)$ into the Lane-Emden equation. $\theta$ and its first two derivatives are

\begin{align}
\begin{split}
\theta(\xi) &= 1 + a_2 \xi^2 + a_4 \xi^4 + O(\xi^6)\\
\theta'(\xi) &= 2 a_2 \xi + 4 a_4 \xi^3 + O(\xi^5) \\
\theta''(\xi) &= 2 a_2 + 12 a_4 \xi^2 + O(\xi^4)
\end{split}
\end{align}

Putting these into the Lane-Emden equation, we have
\begin{align}
\begin{split}
2 a_2 + 12 a_4 \xi^2 + O(\xi^4) + \frac{2}{\xi} (2 a_2 x + 4 a_4 \xi^3 + O(\xi^5))  &= -\theta^n \\
6 a_2 + 20 a_4 \xi^2 + O(\xi^4) &= -\theta^n
\end{split}
\end{align}

A boundary condition $\theta(0)=1$, and thus we have $a_2 = -1/6$. Away from zero, then, we have

\begin{align}
\begin{split}
-1 + 20 a_4 \xi^2 + O(\xi^4) &= -\left(1 - 1/6 \xi^2 + a_4 \xi^4 + O(\xi^6)\right)^n
\end{split}
\end{align}

The term on the RHS is $ 1 - n \xi^2/6 + O(\xi^4)$, and so we must have $a_4 = n/120$.

Thus, the series expansion of the solution around the origin is

$$ \theta(\xi) = 1 - \frac{1}{6}\xi^2 + \frac{n}{120} \xi^4 + \dots $$

We can now use this expansion to take a first step slightly away from the origin before beginning our
numerical integration, thus avoiding the divide by zero. Note that this series solution near the origin is $O(h^5)$ and so is a good match for RK4 if we take the same (or smaller) step-size.

```python
n = 3

xi0 = 0.01                            # starting value of xi for our numerical integration
theta0 = 1 - xi0**2/6 + n*xi0**4/120  # Taylor series solution to the DE near zero derived above
theta0p = -xi0/3 + n*xi0**3/30
y0 = np.array([ theta0,  theta0p])    # set IC's for numerical integration
print(f"IC at {xi0:10.5e}: {y0[0]:10.5e}, {y0[1]:10.5e}")

h = 0.1
tmax = 8

# call the solver
t, y = odeSolve(xi0, y0, tmax, h, leRHS, RK4_step, n)

fig,ax = plt.subplots()
ax.plot(t,y[:,0],'b', label=r'$\theta(\xi)$')
ax.plot(t,y[:,1],'r--', label=r'$\frac{d\theta}{d\xi}$')
ax.plot([0,tmax],[0,0],'k')
ax.set_xlabel(r'$\xi$')
ax.set_title(f"Lane Emden Equation for n={n}")
ax.legend()

plt.tight_layout()
plt.show()
```

For values of $n\le5$, the solutions of the Lane Emden equation (the so-called Lane-Emden functions of index $n$) decrease to zero at finite $\xi$. Since this is the radius at which the density goes to zero, we can interpret it as the surface of the self-gravitating body (for example, the radius of the star). Knowing this value $\xi_1$ is thus interesting... Let us see how to determine it numerically.

Cleary, we are looking for the solution to $\theta(\xi_1)=0$; this is just root-finding, which we already know how to do. Instead of using some closed-form function, however, the value of the function $\theta(\xi)$ must in this case be determined numerically. But we have just figured out how to do this!

Let's use the bisection method for our root-finding algorithm; here is a quick version (no error checking!)

```python
def bisection(func, low, high, eps, *P):
    flow = func(low, *P)
    fhigh = func(high, *P)
    mid = 0.5*(low+high)
    fmid = func(mid,*P)

    while (high-low)> eps:
        if fmid*flow < 0:
            high = mid
            fhigh = fmid
        else:
            low = mid
            flow = mid

        mid = 0.5*(low+high)
        fmid = func(mid,*P)

    return low
```

Now let us make a function which returns $\theta(\xi)$, the solution to the Lane-Emden equation at $\xi$

```python
def theta(xi, n):
    h = 1e-4
    xi0 = 1e-4
    theta0 = 1 - xi0**2/6 + n*xi0**4/120
    theta0p = -xi0/3 + n*xi0**3/30
    y0 = np.array([ theta0,  theta0p])
    t, y = odeSolve(xi0, y0, xi, h, leRHS, RK4_step, n)

    return y[-1,0]
```

Using these, we can compute the surface radius of the polytrope

```python
n = 3
xi1 = bisection(theta, 6, 8, 1e-5, n)
print(f"xi_1 = {xi1:7.5f}")
```

A more careful treatment gives a value $\xi_1 = 6.89685...$, so we are doing pretty well...

```python

```


Programming exercise 1: Manuel, Niclas, Veli <br>
This code approximates the solution to the two dimensional Poisson problem by discretizing:
\begin{align}
-\Delta u & = f \; in \; \Omega = (0,1)^2\\
u & = 0 \; on \; \partial \Omega
\end{align}
First, we write a function that gives back the sparse matrix of size $(n-1)^2 \times (n-1)^2$ which we will need to solve to get the approximate solution of the Poisson problem with dirichlet boundary conditions.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.sparse import csr_matrix, isspmatrix_csr
from scipy.sparse import csc_matrix
from scipy.sparse.linalg import spsolve
import numpy.linalg as LA
from mpl_toolkits.mplot3d import Axes3D
import scipy
import time

def A_matrix(n):
    if n<=1:#check if n is bigger 1, throw error if not
        return print("Error: Input has to be an integer n>1")
    else:
        #nb=[-1]*(n-1) alternative but slower way in this case
        nb = [-1 for i in range(n-1)] #list of length n-1 filled with -1, since we use it twice
        data = np.array([[2]*(n-1), nb, nb])#diagonal entries
        diags = np.array([0, -1, 1])#on which diagonals the entries should be put
        del nb #since we dont use nb anymore
        L=scipy.sparse.spdiags(data, diags, n-1, n-1)
        A=scipy.sparse.kronsum(L,L)
        A=n*n*A
        return A

#Example calculation for n=4:
print("This is the resulting matrix:\n", A_matrix(4).toarray())
#This is the resulting matrix:\n", .toarray()
```

    This is the resulting matrix:
     [[ 64 -16   0 -16   0   0   0   0   0]
     [-16  64 -16   0 -16   0   0   0   0]
     [  0 -16  64   0   0 -16   0   0   0]
     [-16   0   0  64 -16   0 -16   0   0]
     [  0 -16   0 -16  64 -16   0 -16   0]
     [  0   0 -16   0 -16  64   0   0 -16]
     [  0   0   0 -16   0   0  64 -16   0]
     [  0   0   0   0 -16   0 -16  64 -16]
     [  0   0   0   0   0 -16   0 -16  64]]

Now we implement a jacobi iteration to solve the system Au=b with:
\begin{align}
u^{(k+1)} = u^{(k)} + D^{-1} (b-Au^{(k)})
\end{align}
with stopping criterion $\lVert u^{(k+1)} - u^{(k)}\rVert < \epsilon$ for $\epsilon >0$.

```python
def jacobi_iteration(A,u1,b1,e,n):
    D=(1/(4*n*n)*scipy.sparse.eye((n-1)*(n-1)))#inverse diagonal matrix
    while True:
        q=A.dot(u1)
        #k=len(q)
        #q=np.reshape(q,(k,1))
        r=b1-q
        v= D.dot(r) + u1
        if LA.norm(v-u1)<e:#if difference between steps is small, end
            return v
        u1=v
```

Now we want to approximately solve the example $f(x,y)=5\pi^2sin(2\pi x)sin(\pi y)$.

```python
def f(x,y):#the function for which we want to solve the Poisson equation
    return 5*np.pi*np.pi*np.sin(2*np.pi*x)*np.sin(np.pi*y)

def u(n):#gives start vector [00...00] of right size back
    k=(n-1)*(n-1)
    u1=[0]*k
    u1=np.reshape(u1,(k,1))
    return u1

def grid_ij(n):#grid with ij indexing
    x=np.arange(1/n, 1, 1/n)
    y=np.arange(1/n, 1, 1/n)
    XX,YY=np.meshgrid(x, y, sparse=False, indexing='ij')
    return XX,YY

def grid_xy(n):#grid with coordinate like indexing
    x=np.arange(1/n, 1, 1/n)
    y=np.arange(1/n, 1, 1/n)
    XX,YY=np.meshgrid(x, y, sparse=False, indexing='xy')
    return XX,YY

def b(n):#calculates vector of values of f(x,y)
    z=grid_ij(n)
    print(z)
    l=len(z[0])
    a=np.reshape(z,l*l*2, order='F')
    print(a)
    a=a.reshape((l*l,2))
    print(a)
    result=f(a[:,:1],a[:,1:2])
    return result

def solve(n):#applies jacobi iteration
    A=A_matrix(n)
    b1=b(n)
    u1=u(n)
    e=1e-10#tolerance
    return jacobi_iteration(A,u1,b1,e,n)
b(6)
n_list=[8,16,32,64,128]                   #different grid sizes for calculations
approx_result = [solve(x) for x in n_list]#calculating different approximations
x=np.arange(1/128, 1, 1/128)
y=np.arange(1/128, 1, 1/128)
XX,YY=np.meshgrid(x, y, sparse=False, indexing='xy')
ZZ=b1.reshape(np.array(grid_ij(128)[0]).shape)#function values on grid points in right format
#Axes3D.plot_surface(XX,YY,ZZ)
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(XX, YY, ZZ)
#scipy.sparse.linalg.spsolve(A_matrix(128), b(128)) for comparing results, works as intended
```

This is the surface plot of our approximated solution.

```python
def exact_sol(x,y):#analytical solution to the poisson equation for the example
    return np.sin(2*np.pi*x)*np.sin(np.pi*y)

def apply_exact_sol(n):#calculates vector of values of exact_sol(x,y)
    z=grid_ij(n)
    l=len(z[0])
    a=np.reshape(z,l*l*2, order='F')
    a=a.reshape((l*l,2))
    result=exact_sol(a[:,:1],a[:,1:2])
    return result

numb=np.arange(0., 0.12,0.001)
h_list=[1/n_list[i] for i in range(5)]
exact_result = [apply_exact_sol(x) for x in n_list]
#print(exact_result[4])
max_diff=[np.amax(abs(approx_result[i]-exact_result[i])) for i in range(5)]
plt.xlabel('grid size h')
plt.ylabel('l_inf error')
plt.title('convergence plot')
plt.grid(True)
plt.plot(h_list,max_diff,label="differences")
plt.plot(numb,3*numb**2,label="3*x^2")
plt.legend()
plt.show()
```


```python
# basic calculus

from sympy import *
x, y, z, t = symbols('x y z t')
init_printing(use_unicode=True)
```

```python
diff(x**2)
```

```python
diff(x**x)
```

```python
diff(cos(x)*sin(x)+exp(x)*sin(y) + cos(z)*exp(t))
```

```python
diff(cos(x)*sin(x)+exp(x)*sin(y) + cos(z)*exp(t), x)
```

```python
f = cos(x)*sin(x)+exp(x)*sin(y) + cos(z)*exp(t)
```

```python
f
```

```python
diff(f, x)
```

```python
dfx = diff(f, x)
simplify(dfx)
```

```python
integrate(f,x)
```

```python
intf = Integral(f,x)
```

```python
intf.doit()
```

```python
g = x**5
```

```python
diff(g)
```

```python
diff(g, x, 1)
```

```python
diff(g, x, 2)
```

```python
diff(g, x, x)
```

```python
diff(g, x, x, y)
```

```python
df = Derivative(g, x, x, y, y, t)
```

```python
df
```

```python
df.doit()
```

```python
# integration
```

```python
# indefinite integral
integrate(sin(x),x)
```

```python
# definite integral
integrate(exp(-x), (x, 0, oo))
```

```python
integrate(exp(-x**2-y**2), (x,-oo,oo), (y,0,oo) )
```

```python
intf = Integral(exp(-x**2-y**2), x)
```

```python
pprint(intf)
```

    ⌠
    ⎮     2    2
    ⎮  - x  - y
    ⎮ ℯ          dx
    ⌡

```python
# limits
limit(sin(x)/x, x, 0)
```

```python
# Taylor Series expansion
f = exp(x)
```

```python
f.series(x, 0, 4)
```

```python
f.series(x, 2, 4).removeO()
```

```python

```