# Linear equations: introduction

This notebook is based on Chapter 2 of

> Süli, Endre and Mayers, David F. _An introduction to numerical analysis_. Cambridge University Press, Cambridge, 2003.
<https://doi.org/10.1017/CBO9780511801181> (ebook in [Helka](https://helka.helsinki.fi/permalink/358UOH_INST/1h3k2rg/alma9926836783506253))

Although numerical analysis has a long history, it can be argued, as is done [here](http://history.siam.org/), that "modern" numerical analysis, characterized by the synergy of the programmable electronic computer, mathematical analysis, and the opportunity and need to solve large and complex problems in applications, begins with the paper

> von Neumann, John and Goldstine, Herman H. _Numerical inverting of matrices of high order_. Bulletin of the AMS 53, 1021-1099, 1947. <https://doi.org/10.1090/S0002-9904-1947-08909-6>

Numerical inverting of matrices is the topic of this notebook.

# Determinants and computational cost

We will consider linear equations
$Ax = b$
where $A$ is an invertible $n \times n$ matrix and $b$ is a vector in $\mathbb R^n$,

$$
A =
\begin{pmatrix}
a_{11} & a_{12} & \dots & a_{1n}
\\
a_{21} & a_{22} & \dots & a_{2n}
\\
\vdots & \vdots & \ddots & \vdots
\\
a_{n1} & a_{n2} & \dots & a_{nn}
\end{pmatrix},
\qquad
b =
\begin{pmatrix}
b_1
\\
b_2
\\
\vdots
\\
b_n
\end{pmatrix}.
$$

Recall that $\det(A)$ can be computed by expanding along row $i$

$$
a_{i1} A_{i1} + a_{i2} A_{i2} + \dots + a_{in} A_{in} = \det(A),
$$

where $A_{ij} = (-1)^{i+j} M_{ij}$ and $M_{ij}$ is the determinant of the matrix obtained by deleting the row $i$ and column $j$ of $A$. Note that for $k \ne i$

$$
a_{i1} A_{k1} + a_{i2} A_{k2} + \dots + a_{in} A_{kn} = 0,
$$

since this is the determinant of the matrix where row $k$ is replaced by row $i$ in $A$.

To summarize,

$$
a_{i1} A_{k1} + a_{i2} A_{k2} + \dots + a_{in} A_{kn}
=
\begin{cases}
\det(A) & i = k,
\\
0 & i \ne k.
\end{cases}
$$

We assumed that $A$ is invertible, equivalently $\det(A) \ne 0$. Hence

$$
\det(A)^{-1} (a_{i1} A_{k1} + a_{i2} A_{k2} + \dots + a_{in} A_{kn})
= \delta_{ik}
= \begin{cases}
1 & i = k,
\\
0 & i \ne k.
\end{cases}
$$

Equivalently, $A A^{-1} = I$ with

$$
A^{-1}
= \frac{1}{\det(A)}
\begin{pmatrix}
A_{11} & A_{21} & \dots & A_{n1}
\\
A_{12} & _{22} & \dots & A_{n2}
\\
\vdots & \vdots & \ddots & \vdots
\\
A_{1n} & A_{2n} & \dots & A_{nn}
\end{pmatrix}.
$$

This formula for the inverse $A^{-1}$ is called [Cramer's rule](https://en.wikipedia.org/wiki/Cramer's_rule). However, the formula does not give an efficient computational method. Let us consider the number $d_n$ of multiplication operations needed to compute $\det(A)$ of a matrix of size $n \times n$ using

$$
a_{i1} A_{i1} + a_{i2} A_{i2} + \dots + a_{in} A_{in} = \det(A).
$$

Then $d_n \ge n d_{n-1}$ and $d_n \ge n!$. The number $d_n$ is one measure of [computational complexity](https://en.wikipedia.org/wiki/Computational_complexity).

```python
import numpy as np
n = 100
print(f'{np.math.factorial(n) = :.1e}')
```

Let's find out if the world's fastest computer can handle $10^{156}$ floating-point operations (FLOPs)

<https://www.wolframalpha.com/input/?i=fastest+computer>

[Peta](https://en.wikipedia.org/wiki/Peta-) is a decimal unit prefix denoting multiplication by $10^{15}$. So it would take at least $10^{156 - 15 - 3} = 10^{138}$ seconds to compute $\det(A)$ for a matrix of size $100 \times 100$. This is _much_ longer time than the age of the universe $4.3 \cdot 10^{17}$ seconds, see

<https://www.wolframalpha.com/input/?i=age+of+universe>

To summarize, computations based on determinant expansions can be used only for very small matrices.

# Gaussian elimination

Gaussian elimination is one of the algorithms that are familiar to us. It consists of steps of the form

1. Add a multiple of a row to another row
2. Swap the position of two rows

Let's solve $Ax = b$
where
$$
A = \begin{pmatrix}
 1  &	 2  &	 3  	\\
 2  &	 4  &	 5  	 \\
 3  &	 5  &	 6
\end{pmatrix}, \quad
b = \begin{pmatrix}
1 \\ 0 \\ 0
\end{pmatrix}.
$$

Writing $a_1, a_2, a_3$ for the rows of $A$, we have

\begin{align}
&\begin{pmatrix}
 1  &	 2  &	 3  &|&	 1 \\
 2  &	 4  &	 5  &|&	 0 \\
 3  &	 5  &	 6  &|&	 0
\end{pmatrix}
\xrightarrow{a_2 += -2 a_1}
\begin{pmatrix}
 1  &	 2  &	 3  &|&	 1 \\
 0  &	 0  &	 -1  &|&	 -2 \\
 3  &	 5  &	 6  &|&	 0
\end{pmatrix}
\\
&\xrightarrow{a_3 += -3 a_1}
\begin{pmatrix}
 1  &	 2  &	 3  &|&	 1 \\
 0  &	 0  &	 -1  &|&	 -2 \\
 0  &	 -1  &	 -3  	&|& -3
\end{pmatrix}
\xrightarrow{a_2 \leftrightarrow a_3}
\begin{pmatrix}
 1  &	 2  &	 3  &|&	 1 \\
 0  &	 -1  &	 -3  &|&	 -3 \\
 0  &	 0  	& -1  &|&	 -2
\end{pmatrix}
\end{align}

Now $x$ can be solved by back substitution

$$
x_3 = 2, \quad x_2 = 3 - 3x_3 = -3, \quad x_1 = 1 - 2x_2 - 3x_3 = 1.
$$

## Example: Matrices and vectors in NumPy

Matrices and vectors are created using `array`.

```python
a = np.array([
[1,2,3],
[2,4,5],
[3,5,6],
], dtype=float)
b = np.array([1, 0, 0], dtype=float)

```

You can go ahead and implement the Gaussian elimination in Python, and the below sketch can be used as a starting point.

```python
import logging
from logging import debug, error

def forward_solve(a, b):
    '''Forward steps in Gaussian elimination. A sketch!'''
    n = a.shape[0]
    for j in range(n-1):
        d = a[j,j]
        if d == 0: # This is dangerous! Why?
            error('Need to swap rows. This is not implemented!')
            raise NotImplementedError()
        for k in range(j+1,n):
            mu = - a[k,j]/d
            a[k] = a[k] + mu*a[j]
            b[k] = b[k] + mu*b[j]
        debug(f'After forward step for col {j+1} obtained:\n'
              f'{np.block([a,b[:,np.newaxis]])}')
```

```python
def backward_solve(a, b, x):
    '''Backward steps in Gaussian elimination'''
    n = a.shape[0]
    for j in range(n-1,-1,-1):
        y = b[j]
        for k in range(j+1, n):
            y -= a[j,k]*x[k]
        x[j] = y/a[j,j]
        debug(f'Solved x{j+1} = {x[j]}')

def solve_demo(a, b):
    '''Solve ax = b'''
    # Take copies as we don't want to change the original matrices
    a, b = a.copy(), b.copy()
    forward_solve(a, b)
    n = a.shape[0]
    x = np.zeros(n)
    backward_solve(a, b, x)
    return x
```

```python
def test_solve(a, b):
    x = solve_demo(a,b)
    xtrue = np.linalg.solve(a, b)
    print(f'{    x = }')
    print(f'{xtrue = }')

a = np.array([
[1,2,3],
[2,3,4],
[3,4,6],
], dtype=float)
b = np.array([1, 1, 0], dtype=float)
test_solve(a, b)
```

```python
# Let's try with the matrix from the example that we computed by hand
a = np.array([
[1,2,3],
[2,4,5],
[3,5,6],
], dtype=float)
b = np.array([1, 0, 0], dtype=float)
try:
    test_solve(a, b)
except NotImplementedError as e:
    pass
```

```python
a = np.array([
[1, 0, 0, 0.1, 0],
[0, 1, 0, 0.1, 0],
[0, 0, 1, 0.1, 0],
[1, 1, 1, 0.3, 1],
[0, 0, 0, 1.0, 0]
])
b = np.array([0, 0, 0, 1, 1], dtype=float)
logging.getLogger().setLevel(logging.WARNING);
test_solve(a, b)
```

We used `linalg.solve` of NumPy to compute the "true" solution.
The documentation of this function tells that _solutions are computed using LAPACK routine gesv_. ([LAPACK](https://en.wikipedia.org/wiki/LAPACK) is a standard software library for numerical linear algebra.) The documentation of this routine can be found for example in Reference for Intel oneAPI Math Kernel Library, see [gesv](https://software.intel.com/content/www/us/en/develop/documentation/onemkl-developer-reference-c/top/lapack-routines/lapack-linear-equation-routines/lapack-linear-equation-driver-routines/gesv.html#gesv). We find that _LU
decomposition with partial pivoting and row interchanges is used_.

Let's now study what this means. (Pivoting below means partial pivoting in the above sense of row interchanges, see the Wikipedia article [Pivot element](https://en.wikipedia.org/wiki/Pivot_element#Partial_and_complete_pivoting) for description of complete pivoting.)

# LU factorization

Let us express the operation

1. Add a multiple of a row to another row

by using matrix multiplication. Let $E^{(rs)}$ be the matrix whose only nonzero element is 1 at row $r$, column $s$. Let $A$ be a matrix with elements $a_{ij}$. Then

$$
(E^{(rs)} A)_{ij} = \sum_{k=1}^n (E^{(rs)})_{ik} a_{kj}
= \sum_{k=1}^n \delta_{ri}\delta_{sk} a_{kj}
= \delta_{ri} a_{sj}
$$

This is the matrix whose row $r$ is the row $s$ of $A$, and all the other rows are zero. It follows that $(I + \mu E^{(rs)}) A$ is the matrix obtained from $A$ by adding $\mu$ times its row $s$ to its row $r$.

If we don't need to swap rows in Gaussian elimination, then the algorithm gives an upper triangular matrix $U$ satisfying

$$
L_{(N)} L_{(N-1)} \dots L_{(1)} A = U
$$

where each $L_{(j)}$ is a matrix of the form $(I + \mu E^{(rs)}) A$ with some $\mu$ and $r > s$.

There holds $(I + \mu E^{(rs)})^{-1} = (I - \mu E^{(rs)})$, since this undoes the addition. In particular,

$$
A = LU, \qquad L = L_{(1)}^{-1} \dots L_{(N-1)}^{-1} L_{(N)}^{-1}.
$$

## Lemma: product of lower triangular matrices

> The product of two lower triangular matrices is lower triangular.
If the diagonal elements of both the factors are all equal to one, the same holds for product.
In particular, $L$ is lower triangular with all its diagonal elements equal to one.

_Proof._ Consider two lower triangular matrices $A$ and $B$ with elements $a_{ij}$ and $b_{ij}$. Then $a_{ij} = b_{ij} = 0$ if $j > i$, and

$$
(AB)_{ij} = \sum_{k=1}^n a_{ik} b_{kj} = \sum_{k=j}^i a_{ik} b_{kj}.
$$

In particular, $(AB)_{ij} = 0$ if $j > i$ since in this case we are summing over an empty set of indices.

Rest of the proof is left as an exercise.
$\blacksquare$

# Pivoting

Let us now turn to the pivoting operation

2. Swap the position of two rows

## Definition: permutation matrix

> A square matrix is a permutation matrix if each of its elements is either one or zero and if each of its rows and colums contain exactly one nonzero element.

Let $P^{(rs)}$ be the permutation matrix obtained by swapping rows $r$ and $s$ of the identity matrix. Then $P^{(rs)} A$ is the matrix obtained by swapping rows $r$ and $s$ of $A$. (For the short proof see e.g. Lemma 2.5.13 of Pekka Pankka's notes on linear algebra.)

## Lemma: product of permutation matrices

> The product of two permutation matrices is a permutation matrix.

## Lemma: inverse of a permutation matrix

> If $P$ is a permutation matrix then $P^{-1}$ exists.

_Proof of the product of permutation matrices lemma._
Let $P \in \mathbb R^{n \times n}$ be permutation matrix and write $\tau(j)$ for the row index of the unique nonzero element in column $j$. As there is exactly one nonzero element on each row, we see that $\tau$ is a permutation of the set $\{1,\dots,n\}$. Write $e_1,\dots,e_n$ for the standard basis of $\mathbb R^n$. Then $P$ has the columns $e_{\tau(1)}, \dots, e_{\tau(n)}$ and so $P e_j = e_{\tau(j)}$.

Let $Q \in \mathbb R^{n \times n}$ be another permutation matrix and write $\sigma$ for the corresponding permutation. Then
$$
Q P e_j = Q e_{\tau(j)} = e_{\sigma(\tau(j))}.
$$
It follows that $QP$ is the permutation matrix corresponding to the permutation $\sigma \circ \tau$.
$\blacksquare$

Using the language of group theory, $n \times n$ permutation matrices give a faithful representation of the group of permutations of a set with $n$ elements.

Proof of the _inverse of a permutation matrix_ lemma is left as an exercise.

## Theorem: LU factorization with pivoting

> For a square matrix $A$ there are lower and upper triangular $L$ and $U$ and a permutation matrix $P$ such that $PA = LU$.

For a proof, see Theorem 2.3 in [the book](#thebook). From the computational point of view, a good pivoting strategy is to swap rows so that `d` in `forward_solve` is as large as possible. The proof of the theorem reflects this choice.

Let's see how the factorization can be used when solving $A x = b$.
As $P$ and $L$ are always invertible, $A$ is invertible if and only if $U$ is. Suppose that this is the case. We write $y = U x$. Then $A x = b$ is equivalent with $L y = P b$. As $L$ is triangular, $y$ can be solved easily. Then $x$ can be solved easily form $y = U x$.

```python
# LU factorization is implemented in SciPy
from scipy import linalg as la
a = np.array([
[1, 0, 0, 0.1, 0],
[0, 1, 0, 0.1, 0],
[0, 0, 1, 0.1, 0],
[1, 1, 1, 0.3, 1],
[0, 0, 0, 1.0, 0]
])
b1 = np.array([0, 0, 0, 1, 1], dtype=float)
b2 = np.array([1, 1, 1, 0, 0], dtype=float)
# As the diagonal of L consists of ones, SciPy does not store it
# Both L and U are stored in the same matrix
# The permutation matrix is represented by the pivot indices:
# row i of matrix was interchanged with row piv[i]
lu, piv = la.lu_factor(a)
# Once we have the LU factorization, we can solve Ax = b
# for several right-hand sides very quickly
print(f'{la.lu_solve((lu, piv), b1) = }')
print(f'{la.lu_solve((lu, piv), b2) = }')
```

# Condition number

The norm of a vector $v \in \mathbb R^n$ with elements $v_1, \dots, v_n$ is

$$
|v| = \sqrt{|v_1|^2 + \dots + |v_n|^2}
$$

and the norm of $A \in \mathbb R^{n \times n}$ is

$$
|A| = \max_{v \in \mathbb R^n \setminus \{0\}} \frac{|Av|}{|v|}.
$$

If $A$ is invertible, then its _condition number_ is

$$
\kappa(A) = |A| |A^{-1}|.
$$

Consider the equation $Ax = b$ and suppose that $b$ is known only up to an error $e$.
That is, we have the data $\tilde b = b + e$.
Then, instead of finding $x = A^{-1} b$, the best that we can do is to compute $\tilde x = A^{-1} \tilde b$.

The ratio of the relative error in the solution to the relative error in the data is

$$
\frac{|\tilde x - x|/|x|}{|\tilde b - b|/|b|}
=
\frac{|A^{-1} e|/|x|}{|e|/|b|}
=
\frac{|A^{-1} e|}{|e|} \frac{|A x|}{|x|}.
$$

Maximizing this ratio in both $e \ne 0$ and $x \ne 0$ gives $\kappa(A)$.
The condition number measures how much the solution $x$ to $Ax = b$ can change for a small change in the data $b$. If $\kappa(A)$ is large then $Ax = b$ is hard to solve computationally.

## Lemma: norm of a product

> For $A, B \in \mathbb R^{n \times n}$
>
>$$
|AB| \le |A|\, |B|
$$
>
>In particular, $\kappa(A) \ge 1$ for invertible $A$.

For a proof, see Theorem 2.10 in [the book](#thebook).

The square roots of the eigenvalues of $A^{\mathsf T} A$ are called the _singular values_ of $A$.

## Lemma: condition number and singular values

> The norm $|A|$ is equal to the largest singular value of $A$.
For invertible $A$, the condition number $\kappa(A)$ is the ratio of the largest singular value of $A$ to the smallest one.
>
> If $A \in \mathbb R^{n \times n}$ is symmetric and positive definite (that is, $A^{\mathsf T} = A$ and $x^{\mathsf T} A x > 0$ for all nonzero $x \in \mathbb R^n$),
then the condition number $\kappa(A)$ is the ratio of the largest eigenvalue of $A$ to the smallest one.

_Proof_. See Theorem 2.9 in [the book](#thebook) for a proof of the claim on $|A|$.
As in that proof, let $w_i$, $i=1,\dots,n$, be an orthonormal basis of eigenvectors of $A^{\mathsf T} A$ corresponding to eigenvalues $\lambda_i$, given in increasing order. Then for a vector $u$ given in the eigenbasis

$$
u = c_1 w_1 + \dots c_n w_n
$$

there holds

$$
|u|^2 = c_1^2 + \dots c_n^2,
\qquad
|Au|^2 = c_1^2 \lambda_1 + \dots c_n^2 \lambda_n \ge \lambda_1 |u|^2,
$$

with equality when $u = w_1$. Suppose that $v \in \mathbb R^n$ is nonzero and set $u = A^{-1} v$. Then

$$
|A^{-1} v| = |u| \le \lambda_1^{-1/2} |Au| = \lambda_1^{-1/2} |v|
$$

and $|A^{-1}| \le \lambda_1^{-1/2}$. Equality is achieved with $v = A w_1$.
Hence $|A^{-1}| = \lambda_1^{-1/2}$ and the claim follows.

The claim on the positive definite case follows from the fact that $\lambda_i = \mu_i^2$ for the eigenvalues $\mu_i$ of $A$. To prove this equality we write $w_i$ for a basis of eigenvectors of $A$ (the computation below shows that that this is also a basis of eigenvectors of $A^{\mathsf T} A$, so there is no conflict in notation). Then, using $A^{\mathsf T} = A$,

$$
A^T A w_i = A^2 w_i = A (\mu_i w_i) = \mu_i^2 w_i.
$$
$\blacksquare$

## Example: Hilbert matrix

The Hilbert matrix $H_{(n)} \in \mathbb R^{n \times n}$ has the elements

$$
h_{ij} = \frac{1}{i + j - 1}
$$

Let's find the condition number $\kappa(H_{(n)})$ for some small values of $n$.

```python
def element(i0, j0):
    i, j = i0 + 1, j0 + 1
    return 1 / (i + j - 1)

def hilbert(n):
    '''Construct the Hilbert matrix of size n x n'''
    return np.fromfunction(element, (n, n))

hilbert(4)
```

```python
def kappa_demo(a):
    '''Compute the largest and smallest eigenvalues
       and the condition number of a positive definite matrix'''
    ls = np.linalg.eigvals(a)
    lmax, lmin = np.max(ls), np.min(ls)
    kappa = lmax/lmin
    return lmax, lmin, kappa

ns = range(2,13,2)
data = [kappa_demo(hilbert(n)) for n in ns]
data
```

```python
# Let's use pandas to display the data prettily
import pandas as pd
df = pd.DataFrame(data)
df.columns = ['$\lambda_{max}$', '$\lambda_{min}$', '$\kappa$']
df.index = ns
df.index.name = 'n'
df.style.format('{:.1e}')
```

```python
# We can also compute the condition number directly using NumPy
kappas = [np.linalg.cond(hilbert(n)) for n in ns]
df = pd.DataFrame({
    '$\kappa$ demo': np.array(data)[:,2],
    '$\kappa$ NumPy': kappas,
    })
df.index = ns
df.index.name = 'n'
df.style.format('{:.1e}')
```

Consider the problem to approximate a function $f$ by a polynomial $p$ of degree $n$ so that the following distance, that is, the distance in $L^2(0,1)$, is minimized

$$
\|f - p\|_{L^2(0,1)} = \left(\int_0^1 |f(x) - p(x)|^2 dx \right)^{1/2}.
$$

We write $p(x) = c_0 + c_1 x + \dots + c_n x^n$ and note that the gradient of

$$
\|f - p\|_{L^2(0,1)}^2 = \|f\|_{L^2(0,1)}^2 - 2 \sum_{i = 0}^n c_i \int_0^1 f(x) x^i dx + \sum_{i,j=0}^n c_i c_j \int_0^1 x^{i + j} dx,
$$

with respect to $c = (c_0, \dots, c_n) \in \mathbb R^{n+1}$, must vanish when the distance is minimized. This leads to the linear system $M c = b$
where

$$
M_{ij} = \int_0^1 x^{i + j} dx = \frac{1}{i + j + 1},
\quad
b_i = \int_0^1 f(x) x^i dx, \qquad i,j=0,\dots,n.
$$

Here $M$ is the Hilbert matrix $H_{(n+1)}$, with the elements indexed starting from zero.

# Least squares method

Linear _least squares problem_ is of the form:

> Given a matrix $A \in \mathbb R^{m \times n}$ and a vector $y \in \mathbb R^m$, find a vector $x \in \mathbb R^n$
that minimizes the squared distance $|Ax - y|^2$.

Write $f(x) = |Ax - y|^2$. If $f$ achieves its minimum at $x$ then $\nabla f(x) = 0$.
This is equivalent with

\begin{align}\tag{1}
A^{\mathsf T} A x = A^{\mathsf T} y.
\end{align}

We can solve this system by usign the LU factorization, but this is not the best option.

## Example: condition number of $A^{\mathsf T} A$

Consider the matrix

$$
A =
\begin{pmatrix}
\epsilon & 0
\\
0 & 1
\end{pmatrix}
$$

for small $\epsilon > 0$, then $\kappa(A) = \epsilon^{-1}$ and $\kappa(A^{\mathsf T} A) = \epsilon^{-2}$. Hence inverting $A^{\mathsf T} A$ can be much harder than inverting $A$. A better way to solve the least squares problem is to use the [QR factorization](https://en.wikipedia.org/wiki/QR_decomposition), see Theorem 2.12 in [the book](#thebook).

## Example: linear regression

Given points $(x_j,y_j)$, $j=1,2,\dots,N$,
that are
generated by a process obeying the equation for a line
$y=kx + b$, but that are corrupted by noise.
Find $k$ and $b$. The parameters $k$ and $b$ match with the data
$(x_j,y_j)$, $j=1,2,\dots,N$, in the best possible way when they minimize the squared distances

\begin{equation*}
\sum_{j=1}^N |kx_j + b - y_j|^2.
    \end{equation*}

But this sum coincides with $|Ax -y|^2$
where

\begin{equation*}
A = \begin{pmatrix}
1 & x_1
\\
1 & x_2
\\\vdots & \vdots\\
1 & x_N
\end{pmatrix},
\quad x = \begin{pmatrix}
b \\ k
\end{pmatrix},
\quad y = \begin{pmatrix}
y_1 \\ y_2 \\\vdots\\ y_N
\end{pmatrix}.
    \end{equation*}

We arrive to the least squares problem to minimize $|Ax-y|^2$.

```python
from scipy import linalg as la
rng = np.random.default_rng()
def generate_data(N):
    k = 1.1
    b = 0.2
    xs = rng.uniform(0, 1, size=N)
    noise = rng.normal(scale=0.1, size=N)
    ys = k*xs + b + noise
    return xs, ys

xs, ys = generate_data(100)

a = np.ones((np.size(xs), 2))
a[:, 1] = xs
x, _, _, _ = la.lstsq(a, ys)
b, k = x
print(f'{b = :.2f}, {k = :.2f}')
```

```python
from matplotlib import pyplot as plt
plt.scatter(xs, ys)
xs_plot = np.linspace(0,1)
plt.plot(xs_plot, k*xs_plot + b, 'r');
```

# Further reading

Numerical linear algebra is a vast subject. Chapters 3 and 5 of [the book](#thebook) contain important topics that we won't cover. If you want to learn more about these topics, you can exercise your [flexible study right](https://studies.helsinki.fi/instructions/article/flexible-study-right-joo) and take the numerical matrix computations course at Aalto.

For an overview of linear algebra capabilities of SciPy see the [tutorial](https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html). (The tutorial mentions the [outdated](https://numpy.org/devdocs/reference/generated/numpy.matrix.html) matrix class of NumPy, and you can just skip the section on this.)

```python

```


# MAT 221 Calculus I

## April 9, 2020

Today's Agenda:
1. Continuous Function
2. Intermediate Value Theorem
3. Exercises

# Antiderivatives (Indefinite Integral)

Antiderivatives, which are also referred to as indefinite integrals or primitive functions, is essentially the opposite of a derivative (hence the name). More formally, an antiderivative $F$ is a function whose derivative is equivalent to the original function $f$, or stated more concisely: $F^\prime(x) = f(x)$.

The Fundamental Theorem of Calculus defines the relationship between differential and integral calculus. We will see later that an antiderivative can be thought of as a restatement of an indefinite integral. Therefore, the discussion of antiderivatives provides a nice segue from the differential to integral calculus.

## Finding Antiderivatives

The process of finding an antiderivative of a function is known as antidifferentiation and is the reverse of differentiating a function.

For example, consider the function $f(x) = x^3$. The antiderivative of this function, $F(x)$, is $\frac{1}{4} x^4 + C$. As antidifferentiation is the reverse operation of differentiation, we solve for the antiderivative by finding the original function before the differentiation took place. A constant $C$, known as the constant of integration, is added to the antideriative as the derivative of any constant is $0$. The addition of the constant does not affect the correspondence of the original antideriative.

The general antiderivative of $f(x) = x^n$, valid when $n \geq 0$, is:

$$ F(x) = \frac{x^{n+1}}{n+1} + C $$

```python
from sympy import symbols, limit, diff, sin, cos, log, tan, sqrt, init_printing, plot, integrate
from mpmath import ln, e, pi

init_printing()
x = symbols('x')
y = symbols('y')
```

We can confirm the antiderivative of the function $f(x) = x^3$ is $\frac{1}{4}x^4 + C$ with SymPy using the `integrate()` function.

```python
integrate(x ** 3, x)
```

## Examples

### Example 1: Find the antiderivative of $f(x) = x - 3$

Using the general antiderivative form above:

$$ F(x) = \frac{1}{2} x^2 - 3x + C $$

```python
integrate(x - 3, x)
```

### Example 2: Determine the antiderivative of the function $f(x) = \frac{1}{2} + \frac{3}{4} x^2 - \frac{4}{5} x^3$

Going left to right and performing the reverse of the differentiation operation:

$$ F(x) = \frac{1}{2} x + \frac{1}{4} x^4 - \frac{1}{5} x^4 + C $$

```python
integrate(x / 2 + (3 / 4) * x ** 2 - (4 / 5) * x ** 3, x)
```

### Example 3: Find the antiderivative of the function $f(x) = (x + 1)(2x - 1)$

Expanding the function then performing antidifferentiation:

$$ f(x) = 2x^2 - x - 1 $$

$$ F(x) = \frac{2}{3} x^3 + \frac{1}{2} x^2 - x + C $$

```python
integrate((x + 1) * (2 * x - 1), x)
```

### Example 4: Find the antiderivative of the function $f(x) = \frac{10}{x^9}$

Rewriting the function as $f(x) = \frac{10}{1} x^9$ and antidifferentiating:

$$ F(x) = -\frac{10}{8} x^{-8} = -\frac{5}{4} x^{-8} $$

```python
integrate(10 / x ** 9, x)
```

### Example 5: Find the antiderivative of $g(\theta) = \cos{\theta} - 5\sin{\theta}$

Antidifferentiate each piece of the function:

$$ G(\theta) = \sin{\theta} + 5\cos{\theta} $$

```python
integrate(cos(x) - 5 * sin(x))
```

## References

Antiderivative. (2017, December 29). In Wikipedia, The Free Encyclopedia. From https://en.wikipedia.org/w/index.php?title=Antiderivative&oldid=817529505

Stewart, J. (2007). Essential calculus: Early transcendentals. Belmont, CA: Thomson Higher Education.

Weisstein, Eric W. "Constant of Integration." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/ConstantofIntegration.html

Weisstein, Eric W. "Indefinite Integral." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/IndefiniteIntegral.html


```python
import matplotlib
matplotlib.rcParams['text.usetex'] = True
import matplotlib.pyplot as plt
import numpy as np
import sympy as sp

```

---
## $$ \vec{f}(x,y) = \sin(x)\,\hat{x} + \cos(y)\,\hat{y}$$

```python
x_range = y_range = np.linspace(0, 2*np.pi, 10)
x, y = np.meshgrid(x_range, y_range)

#fx, fy = np.sin(x), np.cos(y)
fx, fy = x, y
```

```python
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot()
Q = ax.quiver(x, y, fx, fy, pivot='middle')
ax.quiverkey(Q, .8, 1.02, 1, "= 1m/s", labelpos="E")
ax.axis("scaled")
ax.set_title("Flow")
ax.axvline(x=3, linestyle='--')
ax.axhline(y=3,color='r',linestyle='--')
plt.show()
```

```python
fig = plt.figure()
ax = fig.add_subplot()
ax.streamplot(x,y,fx,fy, density=2)
ax.axis("scaled")
plt.show()
```

```python
x_range = y_range = np.linspace(0, 2*np.pi, 100)
x, y = np.meshgrid(x_range, y_range)
fx, fy = x, y
#fx, fy = np.sin(x), np.cos(y)
f_mag = np.sqrt(fx**2 + fy**2)

fig = plt.figure()
ax = fig.add_subplot()
ax.imshow(f_mag, origin="lower", extent=(0,2*np.pi, 0, 2*np.pi), cmap="gray")
ax.quiver(x[5::10,::10],y[5::10,::10],fx[5::10,::10],fy[5::10,::10], pivot="middle", color="red")

plt.show()
```

---
## gradient $$ \vec{\nabla}f(x,y) = \frac{\partial{f}}{\partial{x}}\,\hat{x} + \frac{\partial{f}}{\partial{y}}\,\hat{y}$$

```python
def f(x,y):
    return x + y
    #return np.cos(x)**2 * np.cos(y)**2
```

```python
x_range = y_range = np.linspace(-np.pi/2, np.pi/2, 101)
x, y = np.meshgrid(x_range, y_range)

#dfdy, dfdx = np.gradient(f(x,y))
dfdx, dfdy = np.gradient(f(x,y))
#dfdy, dfdx = np.gradient(f(x,y), np.pi/100)
```

```python
fig = plt.figure()
ax = fig.add_subplot()
ax.contour(f(x,y), origin="lower", extent=(-np.pi/2, np.pi/2, -np.pi/2, np.pi/2))
ax.quiver(x[5::10,5::10],y[5::10,5::10],dfdx[5::10,5::10],dfdy[5::10,5::10], pivot="middle")
plt.show()
```

```python

```

```python

```

```python

```


## Exercise 6.1 in JT: Privately known private benefit and market breakdown

Let us start with a brief outline of the setup (from section 6.2), compared to exercise 3.5 (a lot of is the same, will not be repeated here):

* Two types of entrepreneurs: Good and bad types with private benefits of not behaving $B_H>B_L$. (good type has $B_L$)

* No equity (A=0),

* Investment is not continuous, but either 0 or I.

* Investment is either successfull (return R) or not (return 0).

* Capital markets put probability $\alpha\in(0,1)$ on type 'good' and $1-\alpha$ on type 'bad'.

* Regulary conditions:

$$ \begin{align}
    p_H\left(R-\dfrac{B_H}{\Delta p}\right)<I<p_H\left(R-\dfrac{B_L}{\Delta p}\right), && \text{and} && p_LR<I.
\end{align} $$

Recall that the IC condition for an entrpreneur was **behave if**: $\Delta p R_b \geq B$.

The regularity conditions thus state that:
1. It is not profitable for lenders to invest in project with 'bad' entrepreneurs **and** making sure that they behave. (first inequality)

2. It is profitable to invest in 'good type' entrepreneur **and** making sure that he behaves (second inequality).

3. It is not profitable to invest in **any** project, where the entrepreneur mis-behaves (third inequality).

## Q1: How does the level of rewards $R_b$ in case of success, change the equilibrium in various ranges?

Generally, there are three possible outcomes:

1. If $R_b<B_L/\Delta p$: Both IC constraints are violated $\Rightarrow$ all entrepreneurs misbehave $\Rightarrow$ no loan contracts are offered (third regularity condition)

2. If $R_b>B_H/\Delta p$: If the lender offers $R_b$ both types of entrepreneurs will state that they are of type 'bad' and receive the higher income. This is not profitable (first regularity condition). No loan contracts are offered.

3. If $B_L/\Delta p < R_b < B_H/\Delta p$: A range of values that induces 'good types' to work and may be profitable for the lender. Bad types identify themselves as 'good' types, but do not behave (as the IC constraint for them is not satisfied).

Thus only pooling equilibria in the 3. range are possible.

## Q2: Show that there exists a threshold $\alpha^*$, of good types, where financing occurs only if $\alpha>\alpha^*$

In Q1 we conluded that only pooling equilibria are feasible. Now we add the IR constraint for lender (no negative profits), given we know that only the pooling equilibria are feasible:

$$\begin{align}
    \underbrace{\left[\alpha p_H + (1-\alpha) p_L\right]}_{\text{probability of success}} \underbrace{\left(R-R_b\right)}_{\text{Revenue if success}} \geq I.
\end{align} $$

To find the threshold of $\alpha$ where this holds note:

* The left-hand side is increasing in $\alpha$,
* The left-hand side is decreasing in $R_b$; i.e. the contract (that is still feasible according to Q1) that offers the highest profits are $R_b=B_L/\Delta p$.

The threshold $\alpha^*$ is then defined by:

$$\begin{align}
    \left(\alpha^*p_H+(1-\alpha^*)p_L\right)\left(R-\dfrac{B_L}{\Delta p}\right) = I.
\end{align}$$

## Q3: Describe the 'cross-subsidies' between types that occur when borrowing is feasible.

Repetition of JT textbook terms on bullet form:

* **Adverse selection** ( pooling equilibrium $(\alpha\geq \alpha^*)$),
* **Over investment:** All entrepreneurs borrow (only 'good type' was optimal under perfect information)
* the quality of average lender drops,
* **Cross-subsidization:** 'Good type' subsidizes the 'bad type'.
* **Credit-rationing:** If $\alpha<\alpha^*$ the only pooling equilibrium is a collapse of the market; if the degree of adverse selection is too strong.

Simplest extensions to this setup, see: Signalling (Leland and Pyle (1977)), more general equilibrium descriptions (JT 13.1).


# Ages and transit time distribution from Century
This notebook shows how to use the LAPM package to compute system level metrics for the Century model.

```python
from sympy import *
from LAPM import *
from LAPM.linear_autonomous_pool_model import LinearAutonomousPoolModel
```

In the second line above, we imported also the linear_autonomous_pool_model module which contains most of the functions required for the examples in this notebook.
We will create now a compartmental representation of Century using the original parameter values described in Parton et al. (1987).

```python
B=Matrix([[-2.927714,   0,  0,   0,  0,  0,  0.000000],
           [0, -14.560,  0,   0,  0,  0,  0],
           [0,   0, -3.6211195,   0,  0,  0,  0],
           [0,   0,  0, -18.20,  0,  0,  0],
           [1.449218, 6.552, 1.4665534, 8.19, -3.731, 0.082992,  0.003042],
           [0.204940, 0,  0.2534784,   0,  2.193828, -0.1976,  0],
           [0,   0,  0,   0,  0.014924,  0.005928, -0.006760]])
u=Matrix(7,1,[3.58800, 18.97804, 4.68, 24.75396, 0, 0, 0])
```

```python
M=LinearAutonomousPoolModel(u, B, True)
M.A_expected_value
```

$\displaystyle 79.9346769131668$

Other useful system diagnostics are:

```python
M.A_quantile(0.5) # Median (50% quantile) of the system age distribution
```

    16.45239468609507

```python
M.T_expected_value #Mean transit time
```

$\displaystyle 4.14079989282277$

```python
M.T_quantile(0.95) # Median (50% quantile) of the transit time distribution
```

    13.765558504224721

```python
M.a_expected_value # Mean age vector of individual pools
```

$\displaystyle \left[\begin{matrix}0.341563417738208\\0.0686813186813198\\0.276157691012412\\0.0549450549450574\\4.23873284307247\\9.18117385427482\\156.219108831317\end{matrix}\right]$

```python
M.a_quantile(0.95) # 95% quantile of individual pools
```

    array([1.02323255e+00, 2.05750843e-01, 8.27294508e-01, 1.64600674e-01,
           1.36322290e+01, 2.25574777e+01, 4.55638688e+02])

```python
M.a_quantile(0.05) # 5% quantiles of individual pools
```

    array([1.75199129e-02, 3.52289113e-03, 1.41650377e-02, 2.81831283e-03,
           6.06130386e-02, 6.78460345e-01, 1.31186571e+01])

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
from OceanLab import EOF,DYN
import seawater as sw
```

```python
z = Symbol('z')
g,rho0 = 10,1025
```

```python
rho = tanh(1e-2*z-3)+rho0
n2 = (g/rho0)*Derivative(rho,z).doit()
rho
```

$\displaystyle \tanh{\left(0.01 z - 3 \right)} + 1025$

## Vertical Modes

```python
zi = np.arange(1000)
rhoi = lambdify(z,rho)(zi)
n2i = lambdify(z,n2)(zi)

nm = 3
Si,radii,Fi = DYN.eqmodes(n2i,-zi,nm,lat=-2,pmodes=True)
Fi2,radii2 = DYN.vmodes(n2i,-zi,nm,lat=-30)
```

```python
fig,ax = plt.subplots(1,4,figsize=(10,8))

ax[0].plot(rhoi-1000,-zi)
ax[1].plot(n2i*1e5,-zi,'r')

for i in range(nm):
    ax[2].plot(Fi[:,i],-zi,label='{:.0f}$\,$km'.format(radii[i]))
    ax[3].plot(Fi2[:,i],-zi,label='{:.0f}$\,$km'.format(radii2[i]))

ax[2].legend()
ax[3].legend()

ax[0].set(ylabel='z [m]',xlabel='$\sigma_\\theta$ [kg$\,$m$^{-1}$]')
ax[1].set(xlabel='N$^2$ 10$^{-5}\\times$ [s$^{-1}$]')
ax[2].set(title='2$^\circ$S (eqmodes)')
ax[3].set(title='30$^\circ$S (vmodes)')

_ = [a.set(yticklabels=[]) for a in ax[1:]]
_ = [a.grid(True) for a in ax]
_ = [a.set(ylim=[(-zi).min(),(-zi).max()]) for a in ax]
```

## EOF

Creating a synthetic currentmeter mooring using the vertical modes:

```python
n = 50
ti = np.arange(n)

np.random.seed(0)
ampe = np.array([10,5,3])
amp = np.random.randn(nm,n)*ampe[:,None]
noise = np.random.randn(zi.size,n)/1000

mooring = np.dot(Fi,amp)/100+noise

dl = 0.05
kw = {
    'vmin':-0.4,
    'vmax':0.4,
    'levels':np.arange(-0.4,0.4+dl,dl),
    'cmap':'bwr',
    'extend':'both',
}

fig,ax = plt.subplots(figsize=(13,4))
C = ax.contourf(ti,-zi,mooring,**kw)
ax.set(xlabel='time [days]',ylabel='z [m]')
cbar = fig.colorbar(C)
cbar.set_label('u [m/s]')
```

```python
evals_perc,evecs_norm,amp = EOF.eoft(mooring)
```

```python
fig,ax = plt.subplots(figsize=(4,8))
for i in range(nm):
    ax.plot(evecs_norm[:,i],-zi,label='{:.1f}%'.format(100*abs(evals_perc[i])))

ax.legend()
ax.grid(True)
_ = ax.set(ylabel='z [m]',ylim=[(-zi).min(),(-zi).max()])
```

Note that, despite being similar, statistical modes (EOFs) must not be confused with pressure vertical modes (QG/Equatorial), and in each EOF we may have variance from different vertical modes.

## zeta

```python
x = Symbol('x')
y = Symbol('y')

ly,lx = 10,10
u = cos(2*pi*y/ly)
v = cos(2*pi*x/lx)

vort = (Derivative(v,x).doit()-Derivative(u,y).doit())/112e3
vort
```

$\displaystyle - 1.78571428571429 \cdot 10^{-6} \pi \sin{\left(\frac{\pi x}{5} \right)} + 1.78571428571429 \cdot 10^{-6} \pi \sin{\left(\frac{\pi y}{5} \right)}$

```python
n = 100
xi,yi = np.meshgrid(np.linspace(0,15,q),np.linspace(0,15,q))

vorti = DYN.zeta(xi,yi,lambdify((x,y),u)(xi,yi),lambdify((x,y),v)(xi,yi))

kw = {
    'zeta':{
        'vmin':-1.2,
        'vmax':1.2,
        'levels':np.arange(-1.2,1.2+0.05,0.05),
        'cmap':'RdBu_r',
        'extend':'both',
    },
    'error':{
        'vmin':-0.1,
        'vmax':0.1,
        'levels':np.arange(-0.1,0.1+0.01,0.01),
        'cmap':'PiYG',
        'extend':'both',
    },
}

fig,ax = plt.subplots(1,3,figsize=(13,4))
C = ax[0].contourf(xi,yi,1e5*lambdify((x,y),vort)(xi,yi),**kw['zeta'])
ax[1].contourf(xi,yi,1e5*vorti,**kw['zeta'])
cbar = fig.colorbar(C,ax=ax[:2])
cbar.set_label('10$^{-5}\\times$ [s$^{-1}$]')

C = ax[2].contourf(xi,yi,1e5*(vorti-lambdify((x,y),vort)(xi,yi)),**kw['error'])
cbar = fig.colorbar(C,ax=ax[-1])
cbar.set_label('10$^{-5}\\times$ [s$^{-1}$]')

ax[0].set(title='Analytical')
ax[1].set(title='Numerical')
ax[2].set(title='Absolute Error')

_ = ax[0].set(ylabel='Latitude [$^\circ$]')
_ = [a.set(xlabel='Longitude [$^\circ$]') for a in ax]
```

## psi2uv

```python
x = Symbol('x')
y = Symbol('y')

amp = 2e5
L = 20
psi = amp*sin(2*pi*x/L)*sin(2*pi*y/(0.5*L))

u = -Derivative(psi,y).doit()/112e3
v = Derivative(psi,x).doit()/112e3

psi
```

$\displaystyle 200000.0 \sin{\left(\frac{\pi x}{10} \right)} \sin{\left(0.2 \pi y \right)}$

```python
xi,yi = np.mgrid[0:11,0:11]

ui,vi = DYN.psi2uv(xi,yi,lambdify((x,y),psi)(xi,yi))
err = np.sqrt((ui-lambdify((x,y),u)(xi,yi))**2+(vi-lambdify((x,y),v)(xi,yi))**2)

kw = {
    'psi':{
        'vmin':-2,
        'vmax':2,
        'levels':np.arange(-2,2+0.01,0.01),
        'cmap':'RdBu_r',
        'extend':'both',
    },
    'error':{
        'vmin':0,
        'vmax':1e-1,
        'levels':np.arange(0,1e-1+1e-3,1e-3),
        'cmap':'RdPu',
        'extend':'both',
    },
    'quiver':{
        'scale':10,
    },
}

fig,ax = plt.subplots(1,2,figsize=(13,4))
C = ax[0].contourf(xi,yi,1e-5*lambdify((x,y),psi)(xi,yi),**kw['psi'])
Q = ax[0].quiver(xi,yi,ui,vi,**kw['quiver'])
ax[0].quiverkey(Q,0.5,1.05,1,'1$\,$m/s',labelpos='E')
cbar = fig.colorbar(C,ax=ax[0])
cbar.set_label('10$^{5}\\times$ [m$^2\,$s$^{-1}$]')

C = ax[1].contourf(xi,yi,err,**kw['error'])
cbar = fig.colorbar(C,ax=ax[-1])
cbar.set_label('[m/s]')

_ = ax[0].set(xlabel='Longitude [$^\circ$]',ylabel='Latitude [$^\circ$]')
_ = ax[1].set(xlabel='Longitude [$^\circ$]',ylabel='Latitude [$^\circ$]',title='|$\\vec{u_{err}}$|')
```


Exercise material of the MSc-level course **Numerical Methods in Geotechnical Engineering**.
Held at Technische Universität Bergakademie Freiberg.

Comments to:

*Prof. Dr. Thomas Nagel
Chair of Soil Mechanics and Foundation Engineering
Geotechnical Institute
Technische Universität Bergakademie Freiberg.*

https://tu-freiberg.de/en/soilmechanics

```python
import numpy as np
import matplotlib.pyplot as plt

#Some plot settings
plt.style.use('seaborn-deep')
plt.rcParams['lines.linewidth']= 2.0
plt.rcParams['lines.color']= 'black'
plt.rcParams['legend.frameon']=True
plt.rcParams['font.family'] = 'serif'
plt.rcParams['legend.fontsize']=14
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.left'] = True
plt.rcParams['axes.spines.bottom'] = True
plt.rcParams['axes.axisbelow'] = True
plt.rcParams['figure.figsize'] = (12, 6)
```

# Exercise 6 - Linear FEM: soil column under gravity and top load

## Governing differential equation

Consider the equilibrium conditions on an infinitesimal element of the soil column:

$$
    \downarrow: \quad F_z + \frac{\partial F_z}{\partial z}\text{d}z - F_z + \varrho g A \text{d}z = 0
$$

The vertical force is determined by $F_z = \sigma_{zz}A = E_\text{s} A \epsilon_{zz} = -E_\text{s} A u_{z,z}$. Therefore, the equilibrium conditions read:

$$
    0 = \left[ \frac{\partial}{\partial z} \left(E_\text{s} A \frac{\partial u_z}{\partial z}\right) - \varrho g A \right]\text{d}z
$$

Considering, that the equation has to hold for an arbitrary infinitesimal non-zero $\text{d}z$, we find the ordinary differential equation

$$
    0 = \frac{\partial}{\partial z} \left(E_\text{s} A \frac{\partial u_z}{\partial z}\right) - \varrho g A
$$

Nota bene: compare this to the differential equation for a rod ('Zugstab'): $(EAu')' + n = 0$. While the cross section of a rod can vary along its length, the cross-sectional area of a soil column is considered a constant, simplifying the equation further:

$$
    0 = \frac{\partial}{\partial z} \left(E_\text{s} \frac{\partial u_z}{\partial z}\right) - \varrho g
$$

## Weak form

The displacements can have (the essential/Dirichlet) boundary conditions in the form:

$$
    u = \bar{u}\ \forall z \in \partial \Omega_\mathrm{D}% \qquad \text{and} \qquad - E_\text{s}A \frac{\partial u_z}{\partial z} = \bar{F} \ \forall z \in \partial \Omega_\mathrm{N}
$$

We now introduce a test function $\delta u$ (virtual displacement) which vanishes where the displacement is given

$$
    \delta u = 0\ \forall z \in \partial \Omega_\mathrm{D}
$$

and construct the weak form (using integration by parts):

\begin{align}
   0 &= \int \limits_0^H \left[\frac{\partial}{\partial z} \left(E_\text{s} \frac{\partial u_z}{\partial z}\right) - \varrho g\right] \delta u\, \text{d}z
   \\
   &= \int \limits_0^H \left[\frac{\partial}{\partial z} \left(E_\text{s} \frac{\partial u_z}{\partial z} \delta u \right) - \frac{\partial \delta u}{\partial z} E_\text{s} \frac{\partial u_z}{\partial z} - \varrho g \delta u \right] \, \text{d}z
\end{align}

Integrating the first term yields the natural/Neumann boundary conditions, so that
$$
\left[ \underbrace{E_\text{s} \frac{\partial u_z}{\partial z}}_{\displaystyle -\bar{\sigma}_z} \delta u \right]_0^H - \int \limits_0^H \varrho g \delta u \text{d}z = \int \limits_0^H \frac{\partial \delta u}{\partial z} E_\text{s} \frac{\partial u_z}{\partial z} \, \text{d}z
$$
which is recognised as the principal of virtual work: $\delta W_\text{ext} = \delta W_\text{int}$.

## Finite elements in 1D

We have a soil column of $H=20$ m on top of the bed rock at $z=0$.

We first create an element class. An element knows the number of nodes it has, their IDs in the global node vector, and the coordinates of its nodes. Linear elements have 2 nodes and 2 quadrature points, quadratic elements 3 nodes and 3 quadrature points. The natural coordinates of the element run from -1 to 1, and the quadrature points and weights are directly taken from Numpy.

```python
#element class
class line_element():#local coordinates go from -1 to 1
    #takes number of nodes, global nodal coordinates, global node ids
    def __init__(self, nnodes=2, ncoords=[0.,1.], nids=[0,1]):
        self.__nnodes = nnodes
        if (len(ncoords) != self.__nnodes):
            raise Exception("Number of coordinates does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(ncoords)))
        else:
            self.__coords = np.array(ncoords)

        self.__natural_coords = (self.__coords-self.__coords[0])/(self.__coords[-1]-self.__coords[0])*2. - 1.

        if (len(nids) != self.__nnodes):
            raise Exception("Number of node IDs does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(nids)))
        else:
            self.__global_ids = np.array(nids)
        self.__quad_degree = self.__nnodes
        self.__quad_points, self.__quad_weights = np.polynomial.legendre.leggauss(self.__quad_degree)

```

Next, we wish to generate a one-dimensional mesh by specifying the length of a line, the number of elements into which the mesh is to be split, and the number of nodes per element.

```python
def number_of_nodes(nelems,nodes_per_elem):
    return nelems*nodes_per_elem - (nelems - 1)

def generate_mesh(domain_length,nelems,nodes_per_elem):
    nn = number_of_nodes(nelems,nodes_per_elem)
    #coordinate vector of global nodes
    global_nodal_coordinates = np.linspace(0.,domain_length,nn)
    global_solution = np.array([0.]*nn)

    #generate elements
    element_vector = []
    for i in range(nelems):
        node_start = (nodes_per_elem-1)*i
        element_vector.append(
            line_element(nodes_per_elem,
                         global_nodal_coordinates[node_start:node_start+nodes_per_elem],
                         list(range(node_start,node_start+nodes_per_elem))))

    return global_nodal_coordinates, element_vector, global_solution
```

Let's put this to test.

```python
number_of_elements = 30
L = 10.
nodes_per_element = 3

nodes,elements,solution=generate_mesh(L,number_of_elements,nodes_per_element)
```

```python
plt.xlabel('$z$')
plt.ylabel('element ID')
plt.title('Finite element mesh')
for i,elem in enumerate(elements):
    if (i==0):
        plt.plot(elem._line_element__coords,[i]*elem._line_element__nnodes, 'bs-', label='elements')
    else:
        plt.plot(elem._line_element__coords,[i]*elem._line_element__nnodes, 'bs-')
plt.plot(nodes, [0]*len(nodes), 'ro ', label='nodes')
plt.legend();
```

## Shape functions in 1D

Now we have a mesh consisting of a vector of elements, and a global node vector. The next ingredient required are the shape functions for the elements in order to interpolate variables and determine gradients.

### Linear elements

Linear elements have two nodes with $\xi_1 = -1$ and $\xi_2 = 1$. The linear shape functions are

$$
    N_1(\xi) = \frac{1-\xi}{2} \qquad \text{and} \qquad N_2(\xi) = \frac{1+\xi}{2}
$$

Their gradients in natural coordinates follow as

$$
    \frac{\text{d} N_1}{\text{d}\xi} = - \frac{1}{2} \qquad \text{and} \qquad \frac{\text{d} N_2}{\text{d}\xi} = \frac{1}{2}
$$

### Quadratic elements

Quadratic elements have three nodes with $\xi_1 = -1$, $\xi_2 = 0$ and $\xi_3 = 1$. The quadratic shape functions are

$$
    N_1(\xi) = \frac{\xi (\xi - 1)}{2}, \qquad N_2(\xi) = (1-\xi)(1+\xi) \qquad \text{and} \qquad N_3(\xi) = \frac{\xi(1+\xi)}{2}
$$

Their gradients in natural coordinates follow as

$$
    \frac{\text{d} N_1}{\text{d}\xi} = \xi - \frac{1}{2}, \qquad \frac{\text{d} N_2}{\text{d}\xi} = -2\xi \qquad \text{and} \qquad \frac{\text{d} N_3}{\text{d}\xi} = \xi + \frac{1}{2}
$$

### Jacobians and gradients

An unknown is approximated as $u \approx N_i \hat{u}_i$ where $i$ runs over all nodes of a given element and $\hat{u}_i$ are the nodal degrees of freedom. To peform a gradient calculation (here in 1D)

$$
    \text{grad} u = \frac{\text{d} u}{\text{d} z} = \frac{\text{d}N_i}{\text{d} z} \hat{u}_i = \frac{\text{d}N_i}{\text{d} \xi} \frac{\text{d}\xi}{\text{d} z} \hat{u}_i = \frac{\text{d}N_i}{\text{d} \xi} J^{-1} \hat{u}_i
$$

The element Jacobian follows from the isoparametric coordinate approximation $z \approx N_i \hat{z}_i$ where $\hat{z}_i$ are the nodal coordinates:

$$
    J = \frac{\text{d} z}{\text{d}\xi} = \frac{\text{d} N_i}{\text{d}\xi} \hat{z}_i
$$

We use the following shorthand:

$$
    \nabla N_i = \frac{\text{d}N_i}{\text{d} z} = \frac{\text{d}N_i}{\text{d} \xi} J^{-1}
$$

```python
#N
def shape_function(element_order,xi):
    if (element_order == 2): #-1,1
            return np.array([(1.-xi)/2., (1.+xi)/2.])
    elif (element_order == 3): #-1, 0, 1
            return np.array([(xi - 1.)*xi/2., (1-xi)*(1+xi), (1+xi)*xi/2.])
#dN_dxi
def dshape_function_dxi(element_order,xi):
    if (element_order == 2): #-1,1
        return np.array([-0.5*xi/xi, 0.5*xi/xi])  #xi only later for plotting dimensions
    elif (element_order == 3):#-1,0,1
        return np.array([xi - 0.5,-2.*xi,xi + 0.5])

#dz_dxi
def element_jacobian(element,xi):
    element_order = element._line_element__nnodes
    Jacobian = 0.
    Jacobian += dshape_function_dxi(element_order,xi).dot(element._line_element__coords)
    return Jacobian

#dN_dz
def grad_shape_function(element,xi):
    element_order = element._line_element__nnodes
    Jac = element_jacobian(element,xi)
    return dshape_function_dxi(element_order,xi)/Jac

```

Now let's plot the shape functions to see if they were implemented correctly.

```python
def plot_shape_functions(order):
    xi = np.linspace(-1.,1.,100)
    fig, ax = plt.subplots(ncols=2)
    ax[0].set_xlabel('$\\xi$')
    ax[0].set_ylabel('$N(\\xi)$')
    for i in range(order):
        ax[0].plot(xi,shape_function(order,xi)[i],label='$N_{%i}$' %i)
    ax[0].legend()
    ax[1].set_xlabel('$\\xi$')
    ax[1].set_ylabel('$\\mathrm{d}N(\\xi)/\\mathrm{d}\\xi$')
    for i in range(order):
        ax[1].plot(xi,dshape_function_dxi(order,xi)[i],label='$\\mathrm{d}N_{%i}/\\mathrm{d}\\xi$' %i)
    ax[1].legend()
    fig.tight_layout()
```

```python
plot_shape_functions(2)
```

```python
plot_shape_functions(3)
```

## Discretization of virtual work (weak form)

Now let's write the discretized version of the principal of virtual work in the following form:
$$
\left[ \bar{\sigma}_z \delta u \right]_0^H - \int \limits_0^H \varrho g \delta u \text{d}z = \int \limits_0^H \frac{\partial \delta u}{\partial z} E_\text{s} \frac{\partial u_z}{\partial z} \, \text{d}z
$$

using

$$
    u \approx N_i \hat{u}_i, \quad \delta u \approx N_i \delta \hat{u}_i, \quad \frac{\partial u_z}{\partial z} \approx \nabla N_i \hat{u}_i, \quad \frac{\partial \delta u_z}{\partial z} \approx \nabla N_i \delta \hat{u}_i
$$

This yields

$$
\begin{aligned}
    \int \limits_0^H \varrho g \delta u\, \text{d}z &= \int \limits_0^H \varrho g \delta u\, \text{d}z = \int \limits_0^H \varrho g N_i\, \text{d}z\ \delta \hat{u}_i= \bigcup \limits_{e=1}^{n_\text{el}} \int \limits_{z_e} \varrho g N_i \det J \, \text{d}\xi\ \delta \hat{u}_i
    \\
    \int \limits_0^H \frac{\partial \delta u}{\partial z} E_\text{s} \frac{\partial u_z}{\partial z} \, \text{d}z &=  \int \limits_0^H \nabla N_i E_\text{s} \nabla N_k  \, \text{d}z \ \hat{u}_k \delta \hat{u}_i = \bigcup \limits_{e=1}^{n_\text{el}} \int \limits_{z_e} \nabla N_i E_\text{s} \nabla N_k \det J  \, \text{d}\xi \ \hat{u}_k \delta \hat{u}_i
    \\
    \left[ \bar{\sigma}_z \delta u \right]_0^H &= \bar{\sigma}_z(z=H) N_{n_\text{n}}(\xi_{n_\text{n}}) \delta \hat{u}_{n_\text{n}} - \bar{\sigma}_z(z=0) N_{0}(\xi_0) \delta \hat{u}_0 = \left[ \bar{\sigma}_z(z=H) N_{n_\text{n}}(\xi_{n_\text{n}}) \delta_{in_\text{n}} - \bar{\sigma}_z(z=0) N_{0}(\xi_0) \delta_{i0} \right] \delta \hat{u}_i
\end{aligned}
$$

where in the last step we assumed that the node at $z=0$ has the node id $0$, while the node at $z=H$ has the node id $n_\text{n}$.

The discretized system

$$
    \left[ \bar{\sigma}_z(z=H) N_{n_\text{n}}(\xi_{n_\text{n}}) \delta_{in_\text{n}} - \bar{\sigma}_z(z=0) N_{0}(\xi_0) \delta_{i0} - \bigcup \limits_{e=1}^{n_\text{el}} \int \limits_{z_e} \varrho g N_i \det J \, \text{d}\xi \right] \delta \hat{u}_i = \bigcup \limits_{e=1}^{n_\text{el}} \int \limits_{z_e} \nabla N_i E_\text{s} \nabla N_k \det J  \, \text{d}\xi \ \hat{u}_k \delta \hat{u}_i
$$

can be simplified by realizing that the nodal virtual displacements are arbitrary and thus

$$
    \bar{\sigma}_z(z=H) N_{n_\text{n}}(\xi_{n_\text{n}}) \delta_{in_\text{n}} - \bar{\sigma}_z(z=0) N_{0}(\xi_0) \delta_{i0} - \bigcup \limits_{e=1}^{n_\text{el}} \int \limits_{z_e} \varrho g N_i \det J \, \text{d}\xi = \bigcup \limits_{e=1}^{n_\text{el}} \int \limits_{z_e} \nabla N_i E_\text{s} \nabla N_k \det J  \, \text{d}\xi \ \hat{u}_k
$$

which leaves us with $n_\text{n}$ equations for the $n_\text{n}$ unknown nodal displacements $\hat{u}_k$. This is the discretized force balance with

$$
    F_i^\text{surf} + F_i^\text{body} = K_{ik} \hat{u}_k
$$

What we require now is the local assembler to calculate the stiffness matrix and the local right-hand side. Local integration is performed by Gauss quadrature:

$$
    \int \limits_{-1}^1 f(\xi)\,\text{d}\xi \approx \sum \limits_{i=1}^{n_\text{gp}} f(\xi_i) w_i
$$

## Local assember

```python
def Stiffness(z):
    E0 = 1.e8 #Pa
    return E0

def Density(z):
    rho0 = 2600.*(1.-0.38) #kg/m³
    return rho0

def BodyForce(z):#N/m
    g = 9.81
    return g*Density(z)
```

```python
def local_assembler(elem):
    element_order = elem._line_element__nnodes
    K_loc = np.zeros((element_order,element_order))
    b_loc = np.zeros(element_order)
    z_nodes = elem._line_element__coords
    for i in range(elem._line_element__quad_degree):
        #local integration point coordinate
        xi = elem._line_element__quad_points[i]
        #shape function
        N = shape_function(element_order,xi)
        #gradient of shape function
        dN_dX = grad_shape_function(elem,xi)
        #determinant of Jacobian
        detJ = np.abs(element_jacobian(elem,xi))
        #integration weight
        w = elem._line_element__quad_weights[i]

        #global integration point coordinate (for spatially varying properties)
        z_glob = np.dot(N,z_nodes)
        #evaluation of local material/structural properties
        E = Stiffness(z_glob)
        #evaluation of local body force
        rho_g = BodyForce(z_glob)

        #assembly of local stiffness matrix
        K_loc += np.outer(dN_dX,dN_dX) * E * w * detJ

        #assembly of local RHS
        b_loc += N * rho_g * w * detJ
    return K_loc,b_loc
```

```python
local_assembler(elements[3])
```

    (array([[ 7.0e+08, -8.0e+08,  1.0e+08],
            [-8.0e+08,  1.6e+09, -8.0e+08],
            [ 1.0e+08, -8.0e+08,  7.0e+08]]),
     array([ 878.54, 3514.16,  878.54]))

## Global assembly

Now we can construct the global matrix system $\mathbf{K}\mathbf{u} = \mathbf{f}$ or $\mathbf{A}\mathbf{x}=\mathbf{b}$ (see lecture script).

```python
def global_assembler(nodes,elements,solution):
    K_glob = np.zeros((len(nodes),len(nodes)))
    b_glob = np.zeros(len(nodes))
    for i,elem in enumerate(elements):
        K_i, b_i = local_assembler(elem)
        start_id = elem._line_element__global_ids[0]
        end_id = elem._line_element__global_ids[-1]
        K_glob[start_id:end_id+1,start_id:end_id+1] += K_i
        b_glob[start_id:end_id+1] += b_i
    return K_glob, b_glob
```

## Application of boundary conditions

Now we apply the natural (Neumann, nodal force) and the essential (Dirichlet, nodal displacement) boundary conditions.

A nodal force $F_\text{bc}$ at node $m$ is simply added by
$$
    F^\text{ext}_i = \left(
        \begin{array}{c}
            F_0\\ F_1\\ \vdots\\ F_m \\ \vdots \\ F_{n_\text{n}}
        \end{array}
    \right)
     = F_i^\text{body} + F_\text{bc} \delta_{im} = F_i^\text{body} + F_\text{bc}
     \left(
        \begin{array}{c}
            0\\ 0\\ \vdots \\ 1 \\ \vdots \\ 0
        \end{array}
    \right)
$$

```python
def apply_Neumann_bc(b_glob,node_id,value):
    b_glob[node_id] += value
    return b_glob
```

A Dirichlet boundary condition $\bar{u}$ at node $m$ is introduced into the system $K_{ik} u_k = f_i$ via

$$
    \left(
    \begin{array}{c}
        K_{11} & K_{12} & \dots & K_{1m} & \dots & K_{1n_\text{n}}
        \\
        K_{21} & K_{22} & \dots & K_{2m} & \dots & K_{2n_\text{n}}
        \\
        \vdots & \vdots & \vdots & \vdots & \vdots & \vdots
        \\
        K_{m1}:=0 & K_{m2}:=0 & \dots & K_{mm}:=1 & \dots & K_{mn_\text{n}}:=0
        \\
        \vdots & \vdots & \vdots & \vdots & \vdots & \vdots
        \\
        K_{n_\text{n}1} & K_{n_\text{n}2} & \dots & K_{n_\text{n}m} & \dots & K_{n_\text{n}n_\text{n}}
    \end{array}
    \right)
    \left(
    \begin{array}{c}
            u_0\\ u_1\\ \vdots\\ u_m \\ \vdots \\ u_{n_\text{n}}
        \end{array}
    \right)
    =
    \left(
    \begin{array}{c}
            F_0\\ F_1\\ \vdots\\ F_m:=\bar{u} \\ \vdots \\ F_{n_\text{n}}
        \end{array}
    \right)
$$

```python
def apply_Dirichlet_bc(K_glob,b_glob,node_id,value):
    K_glob[node_id,:] = 0.# = K_glob[:,node_id] = 0.
    K_glob[node_id,node_id] = 1.
    b_glob[node_id] = value
    return K_glob, b_glob
```

## Problem solution

We now perform the global assembly, apply a vanishing traction on the top and constrain the displacement at the bottom to zero.

```python
number_of_elements = 20
L = 10.
nodes_per_element = 3

nodes,elements,solution=generate_mesh(L,number_of_elements,nodes_per_element)
```

```python
K, f = global_assembler(nodes,elements,solution)
plt.spy(K,marker='.')
plt.title('sparsity pattern')
plt.tight_layout()
```

```python
f = apply_Neumann_bc(f,len(nodes)-1,0)
```

```python
K, f = apply_Dirichlet_bc(K, f, 0, 0.)
```

```python
solution = np.linalg.solve(K,f)
```

```python
plt.xlabel('$x$')
plt.ylabel('$u$ / mm')
plt.title('Finite element solution')
plt.plot(nodes, solution*1e3, 'ro-', label='$u$')
plt.legend();
```

### Convergence study

Let's do a simple convergence study.

```python
plt.xlabel('$x$')
plt.ylabel('$u$ / mm')
plt.title('Mesh convergence')
for i in [3,5,11]:
    number_of_elements = i
    nodes_per_element = 2
    nodes,elements,solution=generate_mesh(L,number_of_elements,nodes_per_element)
    K, f = global_assembler(nodes,elements,solution)
    f = apply_Neumann_bc(f,len(nodes)-1,0)
    K, f = apply_Dirichlet_bc(K, f, 0, 0.)
    solution = np.linalg.solve(K,f)
    plt.plot(nodes, solution*1e3, marker = 'o', label='%i elements' %i)
plt.legend();
```

That's quite boring. Let's make the case a little bit more interesting, by giving the stiffness a variation with depth and by applying a compressive traction at the surface.

```python
Stiffness = lambda x: 1.e8+0.8e8*np.sin(x) #Pa
sig_top = - 300e3#Pa
```

```python
fig, ax = plt.subplots(ncols=2)
ax[0].set_xlabel('$x$')
ax[1].set_xlabel('$x$')
ax[0].set_ylabel('$u$ / mm')
ax[1].set_ylabel('$E$ / GPa')
nodes_per_element = 2
finest = 1000
for i in [finest,5,10,20]:
    number_of_elements = i
    nodes,elements,solution=generate_mesh(L,number_of_elements,nodes_per_element)
    K, f = global_assembler(nodes,elements,solution)
    f = apply_Neumann_bc(f,len(nodes)-1,sig_top)
    K, f = apply_Dirichlet_bc(K, f, 0, 0.)
    solution = np.linalg.solve(K,f)
    if (i == finest):
        ax[1].plot(nodes, Stiffness(nodes)/1e9, label='reference',color='black')
        ax[0].plot(nodes, solution*1e3, label='reference',color='black')
    else:
        ax[1].plot(nodes, Stiffness(nodes)/1e9, marker = 'o', label='%i elements' %i)
        ax[0].plot(nodes, solution*1e3, marker = 'o', label='%i elements' %i)
ax[0].legend();
ax[1].legend();
fig.tight_layout()
```

Something similar can be done to demonstrate the effect of higher-order elements.

```python
def run_sim(npe,ne):
    nodes,elements,solution=generate_mesh(L,ne,npe)
    K, f = global_assembler(nodes,elements,solution)
    f = apply_Neumann_bc(f,len(nodes)-1,sig_top)
    K, f = apply_Dirichlet_bc(K, f, 0, 0.)
    solution = np.linalg.solve(K,f)
    return nodes, solution, K
```

```python
fig, ax = plt.subplots(ncols=2)
ax[0].set_xlabel('$x$')
ax[1].set_xlabel('$x$')
ax[0].set_ylabel('$u$ / mm')
ax[1].set_ylabel('$E$ / GPa')
#reference
nodes, solution, matrix = run_sim(2,1000)
ax[0].plot(nodes, solution*1e3, label='reference',color='black')
ax[1].plot(nodes, Stiffness(nodes)/1e9, label='reference',color='black')
#10 linear elements
nodes, solution, matrix = run_sim(2,6)
ax[0].plot(nodes, solution*1e3, label='6 elements with 2 nodes',marker='o')
ax[1].plot(nodes, Stiffness(nodes)/1e9, label='6 elements with 2 nodes',marker='o')
#20 linear elements
nodes, solution, matrix = run_sim(2,12)
ax[0].plot(nodes, solution*1e3, label='12 elements with 2 nodes',marker='o')
ax[1].plot(nodes, Stiffness(nodes)/1e9, label='12 elements with 2 nodes',marker='o')
#10 quadratic elements
nodes, solution, matrix = run_sim(3,6)
ax[0].plot(nodes, solution*1e3, label='6 elements with 3 nodes',marker='o')
ax[1].plot(nodes, Stiffness(nodes)/1e9, label='6 elements with 3 nodes',marker='o')

ax[0].legend()
fig.tight_layout();
```

We see that with the same number of nodes, the higher-order approximation is better. *Note* that the lines connecting the nodal values are linear here, whereas the actual approximation between the nodes of one element is quadratic and thus smoother than shown here!

## Interactive playground

```python
from ipywidgets import widgets
from ipywidgets import interact
#Compute reference solution with 100 cells
nodes_ref, solution_ref, matrix_ref = run_sim(2,1000)

@interact(num_nodes=widgets.IntSlider(min=3, max=41, value=9, step=2, description='nodes'),
         npes=widgets.RadioButtons(options=[2,3], value=2, description='element order',))
def plot(num_nodes=5,npes=2):
    fig,ax = plt.subplots(ncols=2)
    ax[0].set_xlabel('$x$')
    ax[0].set_ylabel('$u$ / mm')
    num_cells = int((num_nodes - 1)/(npes - 1))
    x, u, matrix = run_sim(npes,num_cells)
    ax[0].plot(nodes_ref, solution_ref*1e3, '--', color='k', label='reference solution');
    ax[0].plot(x, u*1e3, 'o-', label=str(len(x)) + ' nodes');
    ax[0].legend(loc='lower left')
    ax[1].spy(matrix,marker='.')
    plt.show()
```

    interactive(children=(IntSlider(value=9, description='nodes', max=41, min=3, step=2), RadioButtons(description…

And here's a convergence plot in terms of number of nodes for linear (blue) and quadratic (red) elements. We see that, despite the more complicated problem, the observed convergence order of the FEM scheme is much higher than with the FDM scheme for this type of problem.

```python
#HIDDEN
fig, ax = plt.subplots()
disc_n = np.logspace(1,2,10)
z, analytical, K = run_sim(3,1000)
analytical_top = analytical[-1]
for nn in disc_n:
    z, numerical, K = run_sim(2,int(nn))
    numerical_top = numerical[-1]
    relerr = np.abs((numerical_top - analytical_top)/analytical_top)
    ax.plot(len(z),relerr,ls='',marker='d',color='blue')
    z, numerical, K = run_sim(3,int(nn))
    numerical_top = numerical[-1]
    relerr = np.abs((numerical_top - analytical_top)/analytical_top)
    ax.plot(len(z),relerr,ls='',marker='d',color='red')
ax.plot([5e1,1e2],[1e-3,1e-3/2],ls='-',color='black')
ax.text(1.1e2,1e-3/2,'1')
ax.plot([5e1,1e2],[1e-3,1e-3/4],ls='-',color='black')
ax.text(1.1e2,1e-3/4,'2')
ax.plot([5e1,1e2],[1e-3,1e-3/8],ls='-',color='black')
ax.text(1.1e2,1e-3/8,'3')
ax.plot([5e1,1e2],[1e-3,1e-3/16],ls='-',color='black')
ax.text(1.1e2,1e-3/16,'4')
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel('$n_\\mathrm{nodes}$')
ax.set_ylabel('$\\left| \\frac{u_z^\\mathrm{numerical} - u_z^\\mathrm{analytical}}{u_z^\\mathrm{analytical}} \\right|_{z=H}$',size=24)
fig.tight_layout();
```


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
import scipy.io
from matplotlib import cm
from scipy.optimize import minimize
from scipy.interpolate import griddata
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['ytick.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['xtick.top'] = False
```

### Low-rank receptive field models
In part 1 and 2, we introduced a generalized linear model using Poisson regression together with a soft-thresholding nonlinearity $(f)$. We saw that the parameters $\mathbf{w}$ of the model could be found by minimizing the negative log-likelihood $(nll)$, and that we could incorporate prior knowledge through a regularization term. Here, we will further incorporate the nuclear norm regularization term introduced in Part 3, as this provides a way for estimating low-rank spatial-temporal receptive field (RF) models.

The spatial-temporal RF is usually thought of as a two-or three-dimensional matrix: one time dimension and one or two spatial dimensions. However, it is often not possible to estimate the full spatial-temporal RF, as it might require more data that what is available. One common approximation is then to assume space-time separability, which implies that the full RF matrix can be constructed as an outer product of a spatial and a temporal filter. This approximation usually works well for retinal ganglion cells, especially if they are not motion sensitive, but it might also approximate away interesting details. This is were nuclear norm regularization steps into the picture, as it comes with a parameter $\lambda$ that can be used to adjust the rank of the estimated RF. A high value of $\lambda$ results in a rank 1 RF, that is a time-space separable RF, whereas slightly lower values will provide low-rank RFs with additional structure.

### Methods
Following part 1 and 2, we define our smooth function $g$ as:

\begin{align}
 g(\mathbf{w}) &= \sum_i^N f(z_i) - y_i\log(f(z_i)), \\
 f(z_i) &=
 \begin{cases}
  \exp(z_i),& \text{if  } z_i \leq 0,\\
  z_i + 1,& \text{if  } z_i > 0,\\
 \end{cases} \\
 z_i &= \mathbf{w}^T\mathbf{x}_i,
\end{align}

where $\mathbf{w}$ and $\mathbf{x}_i$ represent the vectorized RF and the vectorized sample matrix, respectively. The sample matrix $\mathbf{X}_i$ is created from the spatial stimulus $\mathbf{s}_i$ at the current and preceding time points as:

$$
\mathbf{X}_i = \begin{bmatrix}
    s_{i, 1} & s_{i, 2} & s_{i, 3} & \dots  & s_{i, P} \\
    s_{i-1, 1} & s_{i-1, 2} & s_{i-1, 3} & \dots  & s_{i-1, P} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    s_{i-T-1, 1} & s_{i-T-1, 2} & s_{i-T-1, 3} & \dots  & s_{i-T-1, P}
\end{bmatrix}\in\mathbb{R}^{T\times P}.
$$

The sample matrix $\mathbf{X}_i$ as well as the RF ($\mathbf{W}$) thus have one time dimension and one spatial dimension. Following part 3, we thus define our regularization term ($h$) as:

$$
h(W)=\lambda||W||_{tr},
$$

and the full objective function as:

$$
\mathrm{ObjFun}(\mathbf{w}) = g(\mathbf{w}) + h(\mathbf{w}).
$$

The derivative of $g$ is (see part 1):

\begin{align}
 \nabla g &= \sum_i^N \frac{\partial f(z_i) - y_i\log(f(z_i))}{\partial \mathbf{w}}, \\
 \frac{\partial f(z_i) - y_i\log(f(z_i))}{\partial \mathbf{w}} &=
 \begin{cases}
  (\exp(z_i) - y_i)\mathbf{x}_i,& \text{if  } z_i \leq 0\\
  \left( 1-\frac{y_i}{z_i+1} \right) \mathbf{x}_i,& \text{if  } z_i > 0\\
 \end{cases},
\end{align}

whereas the proximal operator for $h$ is (see part 3):
\begin{align}
 \operatorname{prox}_{\eta\lambda}(B) &= S_{\eta \lambda}(B), \\
 S_{\eta \lambda} &= U\Sigma_{\eta \lambda} V^T,
\end{align}

with $(\Sigma_{\eta \lambda})_{ii}=\operatorname{max}\{\Sigma_{ii}-\eta \lambda,0\}$ and $U \Sigma V^T$ the singular value decomposition of $B$.

```python
def vectorToMatrix(w, wShape):
    return w[1:].reshape(wShape[0], wShape[1])

def matrixToVector(W, w0):
    w = np.empty(W.size+1)
    w[0] = w0
    w[1:] = W.ravel()
    return w

def invLinkFun(X, w):
    z = np.dot(X, w)
    mu = np.zeros(z.shape)
    mu[z<0] = np.exp(z[z<0])
    mu[z>=0] = z[z>=0] + 1
    mu[mu==0] = 1e-10 # Protection against log(0)
    return mu

def gFunDer(X, y, w):
    z = np.dot(X, w)
    der = np.dot(np.exp(z[z<0])-y[z<0], X[z<0, :])
    der += np.dot(1-y[z>=0]/(z[z>=0]+1), X[z>=0, :])
    return der

def gFun(X, y, w):
    return np.sum(invLinkFun(X, w) - y*np.log(invLinkFun(X, w)))

def hFun(w, regLambda, wShape):
    W = vectorToMatrix(w, wShape)
    U, Sigma, VT = np.linalg.svd(W, full_matrices=False)
    return regLambda*np.linalg.norm(Sigma, 1)

def negLogLikFun(X, y, w, reg_lambda):
    return gFun(X, y, w) + hFun(w, reg_lambda)

def matrixSoftThresholdingFun(W, th):
    U, Sigma, VT = np.linalg.svd(W, full_matrices=False)
    Sigma = Sigma - th
    Sigma[Sigma<0] = 0
    return np.dot(U * Sigma, VT)

def prox(w, th, wShape):
    WOld = vectorToMatrix(w, wShape)
    WNew = matrixSoftThresholdingFun(WOld, th)
    return matrixToVector(WNew, w[0])

# Proximal gradient with backtracking and accerlation
def minimizeObjFun(X, y, w0, wShape, regLambda):

    i = 2
    eta = 1
    beta = 0.8
    epsilon = 1e-5
    wNew = np.copy(w0)
    wOld = np.copy(w0)
    wTmp = np.copy(w0)

    objFunVals = []
    objFunVals.append(gFun(X, y, wNew) + hFun(wNew, regLambda, wShape))

    converged = False
    while not converged:
        gradTmp = gFunDer(X, y, wTmp)
        gTmp = gFun(X, y, wTmp)
        # Backtracking loop
        while True:
            wNew = prox(wTmp - eta*gradTmp, regLambda*eta, wShape)
            diff = wNew - wTmp
            if gFun(X, y, wNew) > gTmp + np.sum(np.multiply(gradTmp, diff)) + np.sum(diff**2)/(2*eta):
                eta *= beta
            else:
                break
        objFunVals.append(gFun(X, y, wNew) + hFun(wNew, regLambda, wShape))
        wTmp = wNew + (i-2)/(i-1)*(wNew-wOld)
        wOld = wNew
        i += 1

        # Converge check, mean over previous iterations due to Nesterow ripples
        if len(objFunVals) > 12:
            if (np.mean(objFunVals[-12:-2]) - objFunVals[-1]) / objFunVals[-1] < epsilon:
                converged = True
            if i > 300:
                converged = True

    return wNew, np.array(objFunVals)
```

### Estimation of a low-rank RF from real data
Load example data and create $\mathbf{X}$ and $\mathbf{y}$ based on the wanted time duration of the RF.

```python
# Load real example data
rawData = scipy.io.loadmat('data/20181211Dcc3SpatialRF.mat')
stimulus = rawData['data']['SpatialRFData'][0][0][0,0]['X']
spikeCounts = rawData['data']['SpatialRFData'][0][0][0,0]['Y']
stimPositions = rawData['data']['SpatialRFData'][0][0][0,0]['FlashPositions']
binWidth = rawData['data']['SpatialRFData'][0][0][0,0]['BinWidth']

# Define the the shape of the RF
nLags = 30
rfShape = [nLags, stimPositions.shape[0]]
nParams = rfShape[0]*rfShape[1]

# Create X and y data
y = np.copy(spikeCounts[nLags-1:].ravel()) # ignore the first bins that lack a whole stimulus matrix
# Create X by stacking time shifted stimulus matrices horizontally
X = np.ones((y.size, nParams+1))
for i in range(nLags-1):
    startIdx = i*stimPositions.shape[0]+1
    endIdx = (i+1)*stimPositions.shape[0]+1
    X[:,startIdx:endIdx] = stimulus[i:-nLags+i+1, :]
startIdx = (nLags-1)*stimPositions.shape[0]+1
X[:,startIdx:] = stimulus[nLags-1:, :]
```

### Example analyses using the low-rank RF
We first set $\lambda$ to be just high enough to ensure that we only get one non-zero singular value. This ensures that we end up with a time-space separable RF.

```python
# Run proximal gradient to find the RF parameters
w0 = np.zeros(X.shape[1])
wRF1, objFunVals = minimizeObjFun(X, y, w0, rfShape, 50)
# Transform the vector into a time-space separable spatial-temporal RF
WRF1 = vectorToMatrix(wRF1, rfShape)
U,Sigma,VT = np.linalg.svd(WRF1, full_matrices=False)
print('Rank(B):', np.sum(Sigma > 1e-10))

# Computing the spike triggered average for comparison
wSTA = np.dot(X.T - X[:, 1:].mean(), y)
WSTA = vectorToMatrix(wSTA, rfShape)

# Plotting
fig = plt.figure(figsize=(15, 3))
# Objective function
ax = fig.add_subplot(1, 4, 1)
ax.plot(range(objFunVals.size), objFunVals, c=[0.5, 0.5, 0.5])
ax.set_xlabel('Iteration')
ax.set_ylabel('Obj. fun.')
# Singular values
ax = fig.add_subplot(1, 4, 2)
ax.plot(range(Sigma.size), Sigma, 'o', ms=6, c=[0.5, 0.5, 0.5])
ax.set_xlabel('Singular value')
ax.set_ylabel('Magnitude')
# W
ax = fig.add_subplot(1, 4, 3)
ax.imshow(WRF1.T, cmap=cm.coolwarm, clim=[-np.max(np.abs(WRF1)), np.max(np.abs(WRF1))])
ax.set_title('Estimated RF')
ax.set_xlabel('Time')
ax.set_ylabel('Space (x and y)')
ax.set_xticks([])
ax.set_yticks([]);
# W
ax = fig.add_subplot(1, 4, 4)
ax.imshow(WSTA.T, cmap=cm.coolwarm, clim=[-np.max(np.abs(WSTA)), np.max(np.abs(WSTA))])
ax.set_title('STA, for comparison')
ax.set_xlabel('Time')
ax.set_ylabel('Space (x and y)')
ax.set_xticks([])
ax.set_yticks([]);
```

The RF above is not ideal for conveying an understandable description of the cells stimulus-response mapping: the reason being that both spatial dimensions have been combined along the same axis. However, as the RF is a rank one matrix, we can plot the spatial and the temporal filters separately. Next, we therefore analyze both separately.

```python
# Make sure the spatial RF center is positive
maxIdx = np.argmax(np.abs(VT[0, :]))
if VT[0, maxIdx] < 0:
    U[:, 0] *= -1
    VT[0, :] *= -1

# Cubic interpolation of the spatial RF onto a grid, for visualization purposes only
grid_points = np.linspace(-250, 250, 41)
grid_x, grid_y = np.meshgrid(grid_points, grid_points)
grid_z = griddata(stimPositions, VT[0, :], (grid_x, grid_y), method='cubic')
grid_z = np.ma.masked_where(np.isnan(grid_z), grid_z)
cLim = np.max(np.abs(grid_z))
levels = np.linspace(-cLim, cLim, 11)

# Find the RF center by fitting a symmetric Gaussian to the spatial RF
gaussianFun = lambda amplitude, center, sigma, positions: amplitude*np.exp(-np.sum((positions-center)**2 / (2*sigma**2), axis=1))
mseFun = lambda e: np.dot(e, e) / (2*e.size)
def gaussianObjFun(params, positions, heights):
    predictions = gaussianFun(params[0], params[1:3], params[-1], positions)
    errors = heights - predictions
    mes = mseFun(errors)
    return mes
params = minimize(gaussianObjFun, np.array([1, 0, 0, 100]), (stimPositions, VT[0, :]))
print('RF center: {:3.1f}, {:3.1f}\nsigma: {:3.1f}'.format(params.x[1], params.x[2], params.x[3]))

# Sort the all spatial positions based on their distance to the RF center
distance = stimPositions-params.x[1:3]
radius = np.sqrt(np.sum(distance**2, axis=1))
sortOrder = np.argsort(radius)

fig = plt.figure(figsize=(15, 3))
# Spatial filter
ax = fig.add_subplot(1, 4, 1)
ax.contour(grid_x, grid_y, grid_z, levels, cmap=cm.coolwarm, vmin=-cLim, vmax=cLim)
plt.scatter(stimPositions[:,0],stimPositions[:,1], 50, VT[0, :], cmap=cm.coolwarm, vmin=-cLim, vmax=cLim)
ax.set_title('Spatial filter')
ax.set_xticks([-250, -125, 0, 125, 250])
ax.set_yticks([-250, -125, 0, 125, 250])
ax.set_xlabel('X-dimension (um)')
ax.set_ylabel('Y-dimension (um)')
# Temporal filter
dt = 1./60
time = np.arange(-nLags*dt, 0, dt)
ax = fig.add_subplot(1, 4, 2)
ax.plot(time, U[:, 0]/np.max(np.abs(U[:, 0])), 'ko-')
ax.set_title('Temporal filter')
ax.set_yticks([-1, 0, 1]);
ax.set_xlabel('Time (s)')
# W, sorted
ax = fig.add_subplot(1, 4, 3)
ax.imshow(WRF1[:, sortOrder].T, cmap=cm.coolwarm, clim=[-np.max(np.abs(WRF1)), np.max(np.abs(WRF1))])
ax.set_title('Estimated RF')
ax.set_xticks([])
ax.set_yticks([]);
ax.set_xlabel('Time (s)')
ax.set_ylabel('Distance from center');
# STA, sorted
ax = fig.add_subplot(1, 4, 4)
ax.imshow(WSTA[:, sortOrder].T, cmap=cm.coolwarm, clim=[-np.max(np.abs(WSTA)), np.max(np.abs(WSTA))])
ax.set_title('STA, for comparison')
ax.set_xticks([])
ax.set_yticks([]);
ax.set_xlabel('Time (s)')
ax.set_ylabel('Distance from center');
```

Next, we estimate a new low-rank RF model to with a $\lambda$ value just high enough to get two non-zero singular values. The difference between the rank 1 and the rank 2 RF should then show the structure that was approximated away when we initially assumed a time-space separable RF.

```python
# Run proximal gradient to find the RF parameters
w0 = np.zeros(X.shape[1])
wRF, objFunVals = minimizeObjFun(X, y, w0, rfShape, 35)
# Transform the vector into a time-space separable spatial-temporal RF
WRF = vectorToMatrix(wRF, rfShape)
U,Sigma,VT = np.linalg.svd(WRF, full_matrices=False)
print('Rank(B):', np.sum(Sigma > 1e-10))

# Plotting
fig = plt.figure(figsize=(15, 3))
# Objective function
ax = fig.add_subplot(1, 4, 1)
ax.plot(range(objFunVals.size), objFunVals, c=[0.5, 0.5, 0.5])
ax.set_xlabel('Iteration')
ax.set_ylabel('Obj. fun.')
# Singular values
ax = fig.add_subplot(1, 4, 2)
ax.plot(range(Sigma.size), Sigma, 'o', ms=6, c=[0.5, 0.5, 0.5])
ax.set_xlabel('Singular value')
ax.set_ylabel('Magnitude')
# W, new
ax = fig.add_subplot(1, 4, 3)
ax.imshow(WRF[:, sortOrder].T, cmap=cm.coolwarm, clim=[-np.max(np.abs(WRF)), np.max(np.abs(WRF))])
ax.set_title('Rank {:1d}, RF'.format(np.sum(Sigma > 1e-10)))
ax.set_xticks([])
ax.set_yticks([])
ax.set_xlabel('Time (s)')
ax.set_ylabel('Distance from center');
# W, rank 1
ax = fig.add_subplot(1, 4, 4)
ax.imshow(WRF1[:, sortOrder].T, cmap=cm.coolwarm, clim=[-np.max(np.abs(WRF1)), np.max(np.abs(WRF1))])
ax.set_title('Rank 1, RF')
ax.set_xticks([])
ax.set_yticks([])
ax.set_xlabel('Time (s)')
ax.set_ylabel('Distance from center');
```

ConvexOptimization/Part 4, Low-rank receptive field models.ipynb