```python
### The Import Department
import networkx as nx
import math
import scipy
import scipy.integrate as spi
import numpy as np
import sympy as sm
sm.init_printing(use_latex='mathjax')
import matplotlib.pyplot as plt
import itertools
import random
%matplotlib inline
```

# Graphs and Linear Algebra

Recall that the adjacency matrix of an undirected graph has $A_{i,j} = 1$ if and only if nodes $i$ and $j$ are adjacent. Also, recall that a graph is **regular** with degree $k$ if every node has $k$ neighbors. We also say that the graph is $k$-regular. Finally, for shorthand, we say that the eigenvalues of a graph are the eigenvalues of its adjacency matrix.

**a)** Find three graphs with more than five nodes that are 2-regular, 3-regular, and 4-regular. Represent these in `networkx`, draw them, and find their adjacency matrices. These will be running examples for this problem.

**b)** Find the eigenvalues of the three examples, along with the multiplicities of the eigenvalues.

**c)** Show that if $G$ is $k$-regular, then $k$ is an eigenvalue of $G$.

**d)** Show that $G$ is $k$-regular and connected, then the eigenvalue $k$ of $G$ has multiplicity one.

**e)** Show that $G$ is $k$-regular then $|\lambda|\leq k$ for any eigenvalue $\lambda$ of $G$.

**f)** Let $J$ be the matrix of all ones and $A$ be the adjacency matrix of a $k$-regular graph. Show that $AJ = JA=kJ$.

**g)** Show by construction that there exists regular graph with least eigenvalue equal to $-2$.

**h)** Show that the following graph, called the Petersen Graph, is $3$-regular by finding its eigenvalues.

**i)** Show that if both $n \geq k+1$ and $nk$ is even, then there exists a $k$-regular graph of size $n$.

### Question (a)

```python
G = nx.Graph()
H = nx.Graph()
I = nx.Graph()

num_list = [1,2,3,4,5,6]
edge_two = [(1,2), (1,6), (2,3), (3,4), (4,5),(5,6)]
edge_three = [(1,2), (1,6),(1,4),(2,3),(2,5), (3,4),(3,6),(4,5),(5,6)]
edge_four = [(1,2),(1,3),(1,5),(1,6),(2,3),(2,6),(2,4),(3,4),(3,5),(4,5),(4,6),(5,6)]

#adding all the numbers as node: 2-Regular
G.add_nodes_from(num_list)
G.add_edges_from(edge_two)

#adding all the numbers as node: 3-Regular
H.add_nodes_from(num_list)
H.add_edges_from(edge_three)

#adding all the numbers as node: 4-Regular
I.add_nodes_from(num_list)
I.add_edges_from(edge_four)

basic_graph,ax = plt.subplots(1,3,figsize = (15,5))
nx.draw(G,
        ax=ax[0],
        pos=nx.kamada_kawai_layout(G),
        with_labels=True,
        node_color='#444444',
        font_color="white")

ax[0].set_title("2-Regular", size = 15)

nx.draw(H,
        ax=ax[1],
        pos=nx.kamada_kawai_layout(H),
        with_labels=True,
        node_color='#444444',
        font_color="white")
ax[1].set_title("3-Regular", size = 15)

nx.draw(I,
        ax=ax[2],
        pos=nx.kamada_kawai_layout(I),
        with_labels=True,
        node_color='#444444',
        font_color="white")
ax[2].set_title("4-Regular", size = 15)
```

**Adjacency Matrix for 2-Regular graph with 6 nodes:**
$$
A_2 = \begin{pmatrix}
0 & 1 & 0 & 0 & 0 & 1 \\
1 & 0 & 1 & 0 & 0 & 0 \\
0 & 1 & 0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 & 0 & 1 \\
1 & 0 & 0 & 0 & 1 & 0 \\
\end{pmatrix}
$$

**Adjacency Matrix for 3-Regular graph with 6 nodes:**
$$
A_3 = \begin{pmatrix}
0 & 1 & 0 & 1 & 0 & 1 \\
1 & 0 & 1 & 0 & 1 & 0 \\
0 & 1 & 0 & 1 & 0 & 1 \\
1 & 0 & 1 & 0 & 1 & 0 \\
0 & 1 & 0 & 1 & 0 & 1 \\
1 & 0 & 1 & 0 & 1 & 0 \\
\end{pmatrix}
$$

**Adjacency Matrix for 4-Regular graph with 6 nodes:**
$$
A_4 = \begin{pmatrix}
0 & 1 & 1 & 0 & 1 & 1 \\
1 & 0 & 1 & 1 & 0 & 1 \\
1 & 1 & 0 & 1 & 1 & 0 \\
0 & 1 & 1 & 0 & 1 & 1 \\
1 & 0 & 1 & 1 & 0 & 1 \\
1 & 1 & 0 & 1 & 1 & 0 \\
\end{pmatrix}
$$

### Question (b)

```python
def multiplicity(alist):
    """
    takes list as input, count repeatation of each number in the list.
    returns dictionary of form (eig: multiplicity)
    """
    temp = np.unique(np.round(alist))
    print("Unique eigen values are: ",temp)
    apna_dict = dict.fromkeys(temp, [])
    for i in range(len(temp)):
        count = 0                     # temporary count of eigen value
        for j in range(len(alist)):
            if (temp[i] == alist[j]):
                count = count + 1
        apna_dict[temp[i]] = count
    return apna_dict

def sym(a):
    """
    check if matrix is symmetric
    """
    b = a.T
    if b.all()==a.all():
        print("Symmetric")
    else:
        print('Not Symmetric')

## adjacency matrix of 2-regular graph with 6 nodes
adj_matrix_two = np.matrix([
    [0,1,0,0,0,1],
    [1,0,1,0,0,0],
    [0,1,0,1,0,0],
    [0,0,1,0,1,0],
    [0,0,0,1,0,1],
    [1,0,0,0,1,0]])

## adjacency matrix of 3-regular graph with 6 nodes
adj_matrix_three = np.matrix([
    [0,1,0,1,0,1],
    [1,0,1,0,1,0],
    [0,1,0,1,0,1],
    [1,0,1,0,1,0],
    [0,1,0,1,0,1],
    [1,0,1,0,1,0]])

## adjacency matrix of 4-regular graph with 6 nodes
adj_matrix_four = np.matrix([
    [0,1,1,0,1,1],
    [1,0,1,1,0,1],
    [1,1,0,1,1,0],
    [0,1,1,0,1,1],
    [1,0,1,1,0,1],
    [1,1,0,1,1,0]])

### Calculate eigen values of the Adjacency Matrix
eig_two = np.linalg.eigvals(adj_matrix_two)           # eigen value of 2-regular graph
eig_three = np.linalg.eigvals(adj_matrix_three)       # eigen value of 3-regular graph
eig_four = np.linalg.eigvals(adj_matrix_four)         # eigen value of 4-regular graph

### rounding eigen values to get more legible information
eig_two = np.round(eig_two)
eig_three = np.round(eig_three)
eig_four = np.round(eig_four)

print('\033[1m'"For 2-regular graph with 6 nodes"'\033[0m')
print("Eigen values are: ",eig_two)
print("Multiplicty of eigen values:",multiplicity(eig_two))

print('\033[1m'"\nFor 3-regular graph with 6 nodes"'\033[0m')
print("Eigen values are: ",eig_three)
print("Multiplicty of eigen values:",multiplicity(eig_three))

print('\033[1m'"\nFor 4-regular graph with 6 nodes"'\033[0m')
print("Eigen values are: ",eig_four)
print("Multiplicty of eigen values:",multiplicity(eig_four))
```

    For 2-regular graph with 6 nodes
    Eigen values are:  [-2.  2.  1. -1. -1.  1.]
    Unique eigen values are:  [-2. -1.  1.  2.]
    Multiplicty of eigen values: {-2.0: 1, -1.0: 2, 1.0: 2, 2.0: 1}

    For 3-regular graph with 6 nodes
    Eigen values are:  [-3. -0.  3.  0.  0.  0.]
    Unique eigen values are:  [-3. -0.  3.]
    Multiplicty of eigen values: {-3.0: 1, -0.0: 4, 3.0: 1}

    For 4-regular graph with 6 nodes
    Eigen values are:  [ 4. -2. -2.  0. -0.  0.]
    Unique eigen values are:  [-2.  0.  4.]
    Multiplicty of eigen values: {-2.0: 2, 0.0: 3, 4.0: 1}

### Question (c)
Show that if $G$ is $k$-regular, then $k$ is an eigenvalue of $G$.

**Answer:**

Let us take any eigen value $\lambda$ of adjacency matrix $A_G$of graph G, and, $v = (v_1, v_2, ...,v_n)$ be corresponding eigen vector. Among which $v_k$ is the largest cordinate of the vector v. Also, $\Delta(G)$ is maximum degree of nodes in the graph.

From basic equation, $$\lambda v = Av $$
We have, $$|\lambda.v_i| = \left|(a_{i1},a_{i2},a_{i3}...,a_{in})(v_1,v_2...,v_n)\right|$$\
$$|\lambda.v_i|=|\lambda||v_i| = \left|\sum_{j=1}^{n} a_{ij}v_j \right|$$

as $v_k$ is the largest coordinate in vector $v$,
$$|\lambda||v_i| = \left|\sum_{j=1}^{n} a_{ij}v_j  \right| \leq deg(G) \left|v_k\right| $$

For equality to hold, two condition need to be true, $deg(G) = \Delta(G) \; \&\; \forall i, v_i = v_k $,

For a k-regular graph, $deg(G) = \Delta(G) = k $. Also, for a graph to be regular (1,1,1,..1) must be eigen vector of the adjacency matrix $A_G$ that implies, we have $v_i = v_k \;\forall\; i$.

As both the conditions $deg(G) = \Delta(G) = k\; $ and $v_i = v_k \;\forall\; i$ are satisfied. we can say that k is eigenvalue of the the graph $G$.

```python
### For the graphs constructed in Question(a)
print("Eigen value of 2-regular matrix:",eig_two)
print("Eigen value of 3-regular matrix:",eig_three)
print("Eigen value of 4-regular matrix:",eig_four)

print("""
In all the three cases, we can see that k is part of eigen value list.
""")
```

    Eigen value of 2-regular matrix: [-2.  2.  1. -1. -1.  1.]
    Eigen value of 3-regular matrix: [-3. -0.  3.  0.  0.  0.]
    Eigen value of 4-regular matrix: [ 4. -2. -2.  0. -0.  0.]

    In all the three cases, we can see that k is part of eigen value list.

### Question (d)
Show that if $G$ is $k$-regular and connected, then the eigenvalue $k$ of $G$ has multiplicity one.

#### Answer
From the proof in question (c), we know that for a k-regular graph, eigen vector $(x_1,x_2,..,x_n)$ satisfies $x_1 = x_2 = ... = x_n $ for eigen value k. Where $k$ is eigen value of the graph $G$.

With this observation, we can say that the space of the eigen vectors of the eigen value $k$ is of dimension 1. That implies, k has multiplicity of 1.

To elaborate, for a graph to be connected, max value of spectral radius must be eigen value of the the graph. It should also satisfy the condition of multiplicity equal to 1. In this case, we have a connected k-regular graph. This implies, maximum value of spectral radius is k and multiplicity will be 1.

The idea of connected graph can be proved using contradiction between cauchy interlacing theorm and theorm that states, spectral radius of a graph is always greater than the spectral radius of its sub-graph.

```python
print('\033[1m'"For 2-regular graph with 6 nodes"'\033[0m')
print("Eigen values are: ",eig_two)
print("Multiplicty of eigen value k = 2:",multiplicity(eig_two)[2])

print('\033[1m'"\nFor 3-regular graph with 6 nodes"'\033[0m')
print("Eigen values are: ",eig_three)
print("Multiplicty of eigen value k = 3:",multiplicity(eig_three)[3])

print('\033[1m'"\nFor 4-regular graph with 6 nodes"'\033[0m')
print("Eigen values are: ",eig_four)
print("Multiplicty of eigen value k = 4:",multiplicity(eig_four)[4])
```

    For 2-regular graph with 6 nodes
    Eigen values are:  [-2.  2.  1. -1. -1.  1.]
    Unique eigen values are:  [-2. -1.  1.  2.]
    Multiplicty of eigen value k = 2: 1

    For 3-regular graph with 6 nodes
    Eigen values are:  [-3. -0.  3.  0.  0.  0.]
    Unique eigen values are:  [-3. -0.  3.]
    Multiplicty of eigen value k = 3: 1

    For 4-regular graph with 6 nodes
    Eigen values are:  [ 4. -2. -2.  0. -0.  0.]
    Unique eigen values are:  [-2.  0.  4.]
    Multiplicty of eigen value k = 4: 1

### Question (e)
Show that $G$ is $k$-regular then $|\lambda|\leq k$ for any eigenvalue $\lambda$ of $G$.

#### Answer
Let us take any eigen value $\lambda$ of adjacency matrix $A_G$ of graph G, and, $v = (v_1, v_2, ...,v_n)$ be corresponding eigen vector. If we consider, $v_k$ be the largest co-ordinate in eigen vector v ensuring that $v_k = 1$(can be scaled if required).

Considering everything above, we need to prove that $\lambda \leq \Delta(G)$. Here, $\Delta(G)$ is maximum degree of nodes in the graph.

From basic equation, $$\lambda v = Av $$
We have, $$|\lambda.v_k| = \left|(a_{k1},a_{k2},a_{k3}...,a_{kn})(v_1,v_2...,v_n)\right|$$\
$$|\lambda.v_k| = \left|\sum_{j=1}^{n} a_{kj}v_j  \right|$$

As $v_k$ is the largest coordinate in vector $v$,
$$|\lambda.v_k| = \left|\sum_{j=1}^{n} a_{kj}v_j  \right| \leq \left| \sum_{j=1}^{n} a_{kj}v_k\right| $$\
$$|\lambda.v_k| \leq \left| \sum_{j=1}^{n} a_{kj}v_k\right| \leq \left| v_k \sum_{j=1}^{n} a_{kj}\right| \;\;\;\text{...eq(A)}$$

Considering that graph is k-regular matrix, it is clear that summation of elements in each row of adjacency matrix will be $\Delta(G)$ which is eventually k. from $\text{eq(A) and} |v_k| = 1$

$$\therefore |\lambda| \leq  \Delta(G) = k $$

```python
print('\033[1m'"For 2-regular graph with 6 nodes"'\033[0m')
print("Absolute eigen values are: ",abs(eig_two))

print('\033[1m'"\nFor 3-regular graph with 6 nodes"'\033[0m')
print("Absolute eigen values are: ",abs(eig_three))

print('\033[1m'"\nFor 4-regular graph with 6 nodes"'\033[0m')
print("Absolute eigen values are: ",abs(eig_four))

print("\nFor all the three cases, we can see that eigen values are less the k")
```

    For 2-regular graph with 6 nodes
    Absolute eigen values are:  [2. 2. 1. 1. 1. 1.]

    For 3-regular graph with 6 nodes
    Absolute eigen values are:  [3. 0. 3. 0. 0. 0.]

    For 4-regular graph with 6 nodes
    Absolute eigen values are:  [4. 2. 2. 0. 0. 0.]

    For all the three cases, we can see that eigen values are less the k

### Question (f)
Let $J$ be the matrix of all ones and $A$ be the adjacency matrix of a $k$-regular graph. Show that $AJ = JA=kJ$.

#### Answer
Let us start with these observations:
1. Adjacency Matrix  $A$ is a symmetric matrix.
2. $J$ is a matrix with all ones. It is of same size as size of $A$. This will also be symmetric.
3. $G$ is the $k$-regular graph.

So, with the above conditions, we can say that,
$kJ$ = each element of the resultant matrix will be equal to $k$. By definition, $k$ is number of neighbors that each nodes will have.

As both of the matrix $A$ and $J$ are symmetric. We can say that swapping places will not have impact on the results. and, $AJ = JA$.

To compare this with $kJ$, for every row $J_i$ in Matrix $J$, we will be multipying with column $A_i$ and output will essentially be count of number of 1's in column $A_i$ of the matix A. This is equal to k.

$$J_i.A_i =  (1,1,...,1).(a_{i1},a_{i2},...a_{in})^T $$\
$$J_i.A_i =  \sum_{j = 1}^{n}(a_{ij}) $$\
$$J_i.A_i =  k $$

This implies, each element of resultant matrix AJ or JA is equal to k. Hence, AJ = JA = kJ

```python
J = np.ones((adj_matrix_two).shape)

## for adjacency matrix of 2-regular graph with 6 nodes defined question 1.
out1 = J*adj_matrix_two
print("Output of JA:\n", out1)
out2 = adj_matrix_two*J
print("Output of AJ:\n", out2)
out3 = 2*J
print("Output of kJ:\n", out3)
print("all the three matrices are equal.")
```

    Output of JA:
     [[2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]]
    Output of AJ:
     [[2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]]
    Output of kJ:
     [[2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]
     [2. 2. 2. 2. 2. 2.]]
    all the three matrices are equal.

### Question (g)
Show by construction that there exists regular graph with least eigenvalue equal to $-2$.

```python
G = nx.Graph()

num_list = [1,2,3,4]
edge = [(1,2), (2,3), (3,4), (4,1)]

#adding all the numbers as node: 2-Regular
G.add_nodes_from(num_list)
G.add_edges_from(edge)

basic_graph,ax = plt.subplots(1,1,figsize = (5,5))
nx.draw(G,
        ax=ax,
        pos=nx.kamada_kawai_layout(G),
        with_labels=True,
        node_color='#444444',
        font_color="white")
```

```python
## for the above 2-regular graph, adjacency matrix will be as below:
adj_G = np.matrix([[0,1,0,1],
                   [1,0,1,0],
                   [0,1,0,1],
                   [1,0,1,0]])

eig_G = np.round(np.linalg.eigvals(adj_G))
print("Eigen values of graph adj_G: ",eig_G)
print("Mininum eigen value of adj_G:", min(eig_G))
if min(eig_G) == -2:
    print("Voila, You did it!!")
else:
    print("Try Again! You can do it!")
```

    Eigen values of graph adj_G:  [-2.  0.  2.  0.]
    Mininum eigen value of adj_G: -2.0
    Voila, You did it!!

### Question (h)
Show that the following graph, called the Petersen Graph, is $3$-regular by finding its eigenvalues.

```python
G = nx.Graph()
num_list = [0,1,2,3,4,5,6,7,8,9]
edge = [(0,1),(1,2),(2,3),(3,4),(4,0),(4,9),(0,5),(1,6),
        (2,7),(3,8),(5,7),(5,8),(6,9),(6,8),(7,9)]

#adding all the numbers as node: 2-Regular
G.add_nodes_from(num_list)
G.add_edges_from(edge)

basic_graph,ax = plt.subplots(1,1,figsize = (5,5))
shells = [[5,9,8,7,6],[4,3,2,1,0]]
nx.draw(G,
    ax=ax,
    pos=nx.shell_layout(G,shells),
    with_labels=True,
    node_color='#444444',
    font_color="white")
```

```python
adj_H = np.matrix([[0,1,0,0,1,1,0,0,0,0],
                   [1,0,1,0,0,0,1,0,0,0],
                   [0,1,0,1,0,0,0,1,0,0],
                   [0,0,1,0,1,0,0,0,1,0],
                   [1,0,0,1,0,0,0,0,0,1],
                   [1,0,0,0,0,0,0,1,1,0],
                   [0,1,0,0,0,0,0,0,1,1],
                   [0,0,1,0,0,1,0,0,0,1],
                   [0,0,0,1,0,1,1,0,0,0],
                   [0,0,0,0,1,0,1,1,0,0]])

### for verification
for i in range(0,10):
    l = adj_H[i].tolist()
    print('For node %d:' %i, [i for i, e in enumerate(l[0]) if e == 1])

### print max eigen value of adjacency matrix of petersen graph and multiplicity of that eigen value.
eig_H = np.linalg.eigvals(adj_H)
print('\n')
m = multiplicity(eig_H)
eig_H = np.linalg.eigvals(adj_H)
print("Maximum eigen value of Petersen Graph is: ", max(eig_H))
print("Multiplicity of max eigen value: ",m[max(eig_H)])
print('\033[1m' 'Therefore, Petersen Graph is a 3-regular graph.')
```

    For node 0: [1, 4, 5]
    For node 1: [0, 2, 6]
    For node 2: [1, 3, 7]
    For node 3: [2, 4, 8]
    For node 4: [0, 3, 9]
    For node 5: [0, 7, 8]
    For node 6: [1, 8, 9]
    For node 7: [2, 5, 9]
    For node 8: [3, 5, 6]
    For node 9: [4, 6, 7]

    Unique eigen values are:  [-2.  1.  3.]
    Maximum eigen value of Petersen Graph is:  3.0
    Multiplicity of max eigen value:  1
    Therefore, Petersen Graph is a 3-regular graph.

### Question (i)
Show that if both $n \geq k+1$ and $nk$ is even, then there exists a $k$-regular graph of size $n$.

**Answer**

If $nk$ is even, and $k$ will be driving $n$, we have following cases:
1. n = 0 or k = 0
2. k is even $\Rightarrow$ n is odd or even
3. k is odd $\Rightarrow$ n is even

Based on above combinations, we will filter out possible cases for explanation.

**Case 1:**

For $n = 0:$ Graph doesn't exits. Also, $k$ will be negative, which is not possible.

For $k = 0: n \geq 1$, Graph will just have nodes with no neighbors. Graphs can be classified as 0-regular graphs. Thus in this case, **k-regular graph of size n exists**.

**Case 2**

$k$ is even, with $k \geq 2$

Suppose, $G_{n}(k)$ is a connected k-regular graph with vertex set $\{1,2,3,...,n\}$, for some $n \geq k+1 $.

Using the property: if minimum degree of a vertex in a graph X is $\delta \geq 2$, then there exists a path in G containing $\delta$ edges.

We can assume that, path P is a path in $G_{n}(k)$ containing k edges. Since, k is even, there are $\frac{k}{2}$ vertex disjoint edges $e_i = (u_i,v_i), 1\leq i \leq \frac{k}{2} \;\text{in}\;P$.

$\therefore$ $\bigcup_{1\leq i \leq \frac{k}{2}} \left\{ u_i,v_i \right\}\;$ has k distict verices. Remove the edges, $e_i, 1\leq i\leq \frac{k}{2}$ and add $k$ new edges.

$\therefore$ $\bigcup_{i=1}^{\frac{k}{2}}\left\{(n+1,u_i)\right\} \bigcup \left\{(n+1,v_i)\right\} $

the resulting graph is a connected k-regular graph with vertex set $\{1,2,...,n+1\}$ and is defined to be $G_{n+1}(k)$

**Case 3**
$k$ is odd.

**Base case:** $k = 1$, this implies, $n \geq 2$ and is even.

$\therefore n = \{2z:$ where z is natural number$\}$

In this case, as number of nodes are always even. They can always be distributed in two disjoint sets. and each node can be connected one element of set A is connected to exactly one element of set B.

This can be understood from a simple example, $n = 4$.
set $A = \{1,2\}$ and set $B = \{3,4\}$

edges are from the set, $\{(1,3), (2,4)\}$

**Now for case $k \geq 3$**

Suppose, $G_{n}(k)$ is a connected k-regular graph with vertex set $\{1,2,3,...,n\}$, for some even $n \geq k+1 $. We can use $G_{n}(k)$ to construct a connected k-regular graph $G_{n+2}(k)$ with vertex set $\{1,2,..,n+2\}$ with below process.

Using the property: if minimum degree of a vertex in a graph X is $\delta \geq 2$, then there exists a path in G containing $\delta$ edges.

The graph $G_{n}(k)$ contains a path $R_{n}(k) = (r_1,r_2,...,r_{k+1})$ and has k edges $\{(r_j,r_{j+1})\}_{1\leq j \leq k}$. If we remove k-1 edges $\{(r_j,r_{j+1})\}_{1\leq j \leq k-1}$ and add following edges:

1. For $1\leq j \leq k-1, j $ odd, add the edges $\{(n+1, r_j),(n+1, q_{j+1})\}$

2. For $1\leq j \leq k-1, j $ even, add the edges $\{(n+2, r_j),(n+2, q_{j+1})\}$

3. Add the edge $(n+1, n+2)$

since k is odd, the total number of edges added in $1$ is $k-1$ and so, there are $k-1$ edges with $n+1$ as end vertex after $1$. Simillarly, after step $2\;$ there are $k-1$ edges with $n+2$ as end vertex. and hence, the resulting graph $3$, is a connected k-regular graph with vertex set $\{1,2,...,n+1,n+2\}$ and can be called $G_{n+2}(k)$.


# Nonlinear Optimization methods - Gradient Method (Gradient Descent)

Pedro Blöß Braga, 2021 - February

## Contents
- Gradient Descent (1)
- Stochastic Gradient Descent (2)
- Underdetermined linear systems (3)
- Least Squares and the Normal System (4)

----------------------------------------------------------

### Gradient Descent - Algorithm (1.1)
- Given $x^0 \in \mathbb{R}^n$
- $k=0$
- WHILE $\nabla f(x^k)\neq0$
  - define $d^k = - \nabla f(x^k)$ (a descent direction)
  - obtain $t_k >0$ such that $f(x^k+t_kd^k)< f(x^k)$
  - $x^{k+1}=x^k+t_kd^k$
  - $k=k+1$

Basicly, in this method, you start at a given point $x^0$, and you update your location in a direction $d^k$ and step size $t_k$, changing the direction by the conjugate gradient, i.e. $\langle d^{k+1}, d \rangle=0$ (1)

(1) Proof

Let us assume a auxiliary function $\psi:\mathbb{R} \rightarrow \mathbb{R}$, given by

\begin{equation}
\psi(t) = f(x^k + t_kd^k)
\end{equation}

\begin{equation}
\psi'(t) = \nabla f(x^k+t_kd^k)^Td^k =  \nabla f(x^{k+1})^Td^k
\end{equation}

Then,

\begin{equation}
\langle d^{k+1}, d \rangle = - \nabla f(x^{k+1})^T d^k = - \psi'(t) =0 \quad \square
\end{equation}

This algorithm is an optimization method (can be nonlinear, for nonconstrained optimization), i.e., to minimize a certain cost function $J(\theta)$ (training a model in order to maximize accuracy).

For instance, with the cost function:

\begin{equation}
J(\theta) = \frac{1}{m} \sum_{i=1}^m (\theta^T x^{(i)} - y^{(i)})^2
\end{equation}

\begin{equation}
\frac{\partial J(\theta)}{\partial \theta_j} = \frac{2}{m} \sum_{i=1}^m (\theta^T x^{(i)} - y^{(i)}) x^{(i)}_j
\end{equation}

And our gradient vector would look like

\begin{equation}
\nabla_{\theta} J(\theta) = \pmatrix{ \frac{\partial J(\theta)}{\partial \theta_0} \\
\frac{\partial J(\theta)}{\partial \theta_1} \\
... \\
\frac{\partial J(\theta)}{\partial \theta_n}
}  = \frac{2}{m} X^T (X .\theta - y)
\end{equation}

We would have the updating function:

\begin{equation}
\theta_{j} \leftarrow \theta_j - \eta \frac{\partial}{\partial \theta_j} J(\theta)
\end{equation}
or more generally:
\begin{equation}
\theta \leftarrow \theta- \eta \nabla_{\theta} J(\theta)
\end{equation}

OBS: Note that the Gradient descent method uses $X$, so it uses the full training set, resulting in bad computational cost for large datasets. But, if you had thounsands of features, Gradient Descent would probally train faster then a Least Squares method.

## Implementation

```
import numpy as np
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt
```

```
def gradient_descent(X_b, y, theta, eta=0.05, epochs=100, m=100):

  for _ in range(epochs):
    gradients = (2/m)* X_b.T.dot(X_b.dot(theta) - y)
    theta = theta - eta * gradients

  return theta
```

```
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)
theta = np.random.randn(2,1)

X_b = np.c_[np.ones((100, 1)), X]  # add x0 = 1 to each instance
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y) # \theta = (X^TX)^(-1)(X^Ty) # normal equation

gradient_descent(X_b, y, theta)
```

    array([[3.94023509],
           [3.15237157]])

```
X_new = np.array([[0], [2]])
X_new_b = np.c_[np.ones((2, 1)), X_new]  # add x0 = 1 to each instance
y_predict = X_new_b.dot(theta_best) # <X, \theta>
y_predict
```

    array([[ 3.88972648],
           [10.28384337]])

```
def plot_gradient_descent(X_b, X, y, theta, eta, theta_path=None, epochs=1000):
    m = len(X_b)
    plt.plot(X, y, "b.")
    for iteration in range(epochs):
        if iteration < 10:
            y_predict = X_new_b.dot(theta)
            style = "b-" if iteration > 0 else "r--"
            plt.plot(X_new, y_predict, style)

        gradients = 2/m * X_b.T.dot(X_b.dot(theta) - y) # \frac{\partial J(\theta)}{\partial \theta_j} = \frac{2}{m} \sum_{i=1}^m (\theta^T x^{(i)} - y^{(i)}) x^{(i)}_j

        theta = theta - eta * gradients # \theta_{next} = \theta- \eta \nabla_{\theta} J(\theta)
        if theta_path is not None:
            theta_path.append(theta)
    plt.xlabel("$x_1$", fontsize=18)
    plt.axis([0, 2, 0, 15])
    plt.title(r"$\eta = {}$".format(eta), fontsize=16)
    return plt

plot_gradient_descent(X_b, X, y, theta, eta=0.05)
```

```
plt.figure(figsize=(18,4))
plt.suptitle('Changing the learning rate $\eta$', y=1.05, fontsize=15)
plt.subplot(1,3,1)
p1 = plot_gradient_descent(X_b, X, y, theta, eta=0.01)
plt.subplot(1,3,2)
p2 = plot_gradient_descent(X_b, X, y, theta, eta=0.05)
plt.subplot(1,3,3)
p3 = plot_gradient_descent(X_b, X, y, theta, eta=0.1)
plt.show()
```

## Stochastic Gradient Descent (SGD) (2)

The main issue with Gradient Descent is that it uses the intire training set to compute the gradients in each step, increasing computational complexity. In the other hand, Stochastic Gradient Descent utilizes a batch of $B$ data points to compute the mean gradient. The algorithm becomes much faster and enables working with large datasets.
As a counter point, since it inserts randomness to the process, the direction may not be always in decay in the cost surfice (we loose the property ($f(x_{k+1}) \leq f(x_k)$).

#### Algorithm
- Initialize weights randomly: $W \sim \mathcal{N}(0, \sigma^2)$
- Loop until convergence:
  - Select $B$ points
  - compute the gradient
      \begin{equation}
      \frac{\partial J(W)}{\partial W} = \frac{1}{B} \sum_{k=1}^B \frac{\partial J_k(W)}{\partial W}
      \end{equation}
      - update weigths
      \begin{equation}
      W \leftarrow W - \eta \nabla_W J(W)
      \end{equation}
- return $W$

```
m = len(X_b)
n_epochs = 50
t0, t1 = 5, 50  # learning schedule hyperparameters

def learning_schedule(t):
    return t0 / (t + t1)

theta = np.random.randn(2,1)  # random initialization

for epoch in range(n_epochs):
    for i in range(m):
        if epoch == 0 and i < 20:
            y_predict = X_new_b.dot(theta)
            style = "b-" if i > 0 else "r--"
            plt.plot(X_new, y_predict, style)
        random_index = np.random.randint(m)

        xi = X_b[random_index:random_index+1]
        yi = y[random_index:random_index+1]

        gradients = 2 * xi.T.dot(xi.dot(theta) - yi) # \frac{\partial J(\theta)}{\partial \theta_j} = \frac{2}{m} \sum_{i=1}^m (\theta^T x^{(i)} - y^{(i)}) x^{(i)}_j

        eta = learning_schedule(epoch * m + i) #eta = 5/(epoch * m + i + 50)

        theta = theta - eta * gradients # \theta_{next} = \theta- \eta \nabla_{\theta} J(\theta)

plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.show()
```

## Underdetermined Linear Systems (3)

- A matrix $A$, $m$x$n$ and $m>n$, can be associated with a linear transformation of the spaces $\mathbb{R}^n$ and $\mathbb{R}^m$
- Its Image is a vector subspace of $\mathbb{R}^m$, since $m>n$
- since $b \in \mathbb{R}^m$, $b$ is not generally containg in $A$'s image
- therefore there will not usually exist $x \in \mathbb{R}^n$ such that $Ax=b$, and this equation will have solution only if $b \in Im(A)$

- So we can determine the vector $\bar{x}$ in $Im(A)$ that is closest to $b$.

Using the euclidian distance:
\begin{equation}
  || b - A \bar{x} || \leq || b - AX ||, \forall x \in \mathbb{R}^n
\end{equation}

\begin{equation}
  ||b - Ax || = \sqrt{ \sum_{i=1}^m [b_i - (Ax)_i]^2}
\end{equation}

- Geometrically, $\bar{x} = proj_{Im(A)}(b)$
- This problem always haves a solution, and if the columns of $A$ are linearely independent of $\mathbb{R}^m$, then the solution is unique

## Least Squares (4)
- It consists of minimizing the cost function of the squared errors

Given the input matrix $X$ and the target vector $y$, the value $\hat{\theta}$ minimizes the cost function:

\begin{equation}
\hat{\theta} = (X^T. X)^{-1}X^T.y
\end{equation}

- Normal equation deduction

Given a undetermined linear system $Ax=b$, $A$ is $m$x$n$, $m>n$, $b\in \mathbb{R}^n$, and $x \in \mathbb{R}$, we have the ortogonality relation:
\begin{equation}
\langle a^{(j)}, b-Ax \rangle =0, \quad 1 \leq j \leq n
\end{equation}
where

\begin{equation}
a_{(j)} = \pmatrix{ a_{1j} \\ a_{2j} \\ ... \\ a_{mj} }
\end{equation}

is the $j$-th column vector of the matrix $A = a_i^{(j)} = a_{ij}$

\begin{equation}
\langle a^{(j)}, b-Ax \rangle = \sum_{i=1}^n a_i^{(j)} [b_i - (Ax)_i] = \sum_{i=1}^n a_{ij}[b_i - (Ax)_i]
\end{equation}

\begin{equation}
= \sum_{i=1}^n a_{ij}(b_i - \sum_{k=1}^n a_{ik}x_k)
\end{equation}

\begin{equation}
= \sum_{i=1}^n a_{ij}b_i - \sum_{k=1}^n(\sum_{i=1}^m a_{ij} a_{ik})x_k
\end{equation}

\begin{equation}
= (A^Tb)_j - \sum_{k=1}^n (A^T A)_{jk}x_k = (A^T b)_j - [(A^T A)x]_j
\end{equation}

\begin{equation}
\Rightarrow (A^T b)_j - [(A^T A)x]_j =0
\end{equation}

\begin{equation}
\therefore A^T Ax = A^T b \quad \square
\end{equation}

```
plt.title('Generated data')
plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([0, 2, 0, 15])
plt.show()
```

```
## using the generated linear function from the normal equation

plt.plot(X_new, y_predict, "r-", linewidth=2, label="MMQ predictions")
plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.show()
```

### References

- Richard L. Burden e J.Douglas Faires. "Numerical Analysis".
- Strang, Gilbert. "Linear Algebra and its applications".
- Otimização Contínua - KARAS, Elizabeth.
- https://github.com/ageron/handson-ml2/blob/master/04_training_linear_models.ipynb

#### License
- MIT

#### Contact
- https://www.linkedin.com/in/pedro-bl%C3%B6%C3%9F-braga-3263a1136/
- https://github.com/pedroblossbraga


# PyDSGE Example - Small New Keynesian Model
## Author: Gustavo Amarante
This jupyter notebook is an example on how to implement the basic new keynesian model. The details of the model and
parameters interpretations can be found in [Galí (2008)](https://www.amazon.com/Monetary-Policy-Inflation-Business-Cycle/dp/0691133166)
and a full step-by-step derivation of the model equations is in
[Bergholt (2012)](https://bergholt.weebly.com/uploads/1/1/8/4/11843961/the_basic_new_keynesian_model_-_drago_bergholt.pdf).

The linearized version of the model is given by the following equations:

$
\begin{align*}
\tilde{y}_{t} & =E_{t}\left(\tilde{y}_{t+1}\right)-\frac{1}{\sigma}\left[\hat{i}_{t}-E_{t}\left(\pi_{t+1}\right)\right]+\psi_{ya}^{n}\left(\rho_{a}-1\right)a_{t}\\
\pi_{t} & =\beta E_{t}\left(\pi_{t+1}\right)+\kappa\tilde{y}_{t}+\sigma_{\pi}\varepsilon_{t}^{\pi}\\
\hat{i}_{t} & =\phi_{\pi}\pi_{t}+\phi_{y}\tilde{y}_{t}+v_{t}\\
a_{t} & =\rho_{a}a_{t-1}+\sigma_{a}\varepsilon_{t}^{a}\\
v_{t} & =\rho_{v}v_{t-1}+\sigma_{v}\varepsilon_{t}^{v}
\end{align*}
$

where the following parameters are given by:

$
\psi_{ya}^{n}=\frac{1+\varphi}{\sigma\left(1-\alpha\right)+\varphi+\alpha}\qquad\kappa=\frac{\left(1-\theta\right)\left(1-\theta\beta\right)\left[\sigma\left(1-\alpha\right)+\varphi+\alpha\right]}{\theta\left(1-\alpha+\alpha\epsilon\right)}
$

We also need to write the expectations of time $t+1$ variables as time-$t$ variables. The relationship between variable
expectations and the effective value is just the expectational error.

$
\begin{align*}
\tilde{y}_{t+1} & =ex_{t}^{y}+\eta_{t+1}^{y}\\
\pi_{t+1} & =ex_{t}^{\pi}+\eta_{t+1}^{\pi}
\end{align*}
$

where $ex_{t}^{y}=E_{t}\left(\tilde{y}_{t+1}\right)$.

So we have 7 endogenous variables, 3 exogenous shocks $\varepsilon$ and 2 expectational errors $\eta$ that compose the
state equations of the model. We also need to specify the set of observation equations, which we will assume that only
the outputgap, inflation and interest rate are observable variables.

$
\begin{align*}
output\_gap_{t} & =\tilde{y}_{t}\\
inflation_{t} & =\pi_{t}\\
interest\_rate_{t} & =\left(\frac{1}{\beta}-1\right)+\hat{i}_{t}\\
\end{align*}
$

These are the sections of the notebook:
* Especification
* Calibration
* Simulation
* Estimation
* Analysis

---
# Imports
Your imports should include `symbols` and `Matrix` from the [`sympy` library](https://www.sympy.org/en/index.html), as
these are required for variables and model declaration.

(Obs: the need for libraries should be eliminated in future versions)

```python
import numpy as np
from pydsge import DSGE
import matplotlib.pyplot as plt
from sympy import symbols, Matrix
```

---
# Model Especification
## Variable Declaration
Declare all of the endogenous variables at time-$t$ as symbols and create a `Matrix` object with them.

```python
y, pi, i, a, v, exp_y, exp_pi = symbols('y, pi, i, a, v, exp_y, exp_pi')
endog = Matrix([y, pi, i, a, v, exp_y, exp_pi])
```

Declare all of the endogenous variables at time-$t-1$ as symbols and create a `Matrix` object with them. The convention
in this notebook is to  add an `l` (for "lagged") at the end of the variable name.

```python
yl, pil, il, al, vl, exp_yl, exp_pil = symbols('yl, pil, il, al, vl, exp_yl, exp_pil')
endogl = Matrix([yl, pil, il, al, vl, exp_yl, exp_pil])
```

Declare all of the exogenous shocks $\varepsilon$ as symbols and create a `Matrix` object with them.

```python
eps_a, eps_v, eps_pi = symbols('eps_a, eps_v, eps_pi')
exog = Matrix([eps_a, eps_v, eps_pi])
```

Declare all of the expectational errors $\eta$ as symbols and create a `Matrix` object with them.

```python
eta_y, eta_pi = symbols('eta_y, eta_pi')
expec = Matrix([eta_y, eta_pi])

```

Declare all of the parameter as symbols and create a `Matrix` object with them.
Summary parameters (functions of other parameters) do not need to be inside the matrix object, they only need to be
defined by their formula.

```python
sigma, varphi, alpha, beta, theta, phi_pi, phi_y, rho_a, sigma_a, rho_v, sigma_v, sigma_pi = \
    symbols('sigma, varphi, alpha, beta, theta, phi_pi, phi_y, rho_a, sigma_a, rho_v, sigma_v, sigma_pi')

psi_nya = (1 + varphi) / (sigma*(1-alpha) + varphi + alpha)
kappa = (1 - theta)*(1 - theta * beta)*(sigma*(1-alpha) + varphi + alpha)
```

## State Equations
Given the set of linearized equilibrium conditions described in the beggining of thie notebook, put all of the terms
to one side of the equation so that the system of equations is equal to zero. Since all of our variable and parameter
names are symbols, the equations will be correctly interpreted as symbolic expressions. These expressions should also
go into a `Matrix` object. This set of equations should include only the state equations of the system.

```python
eq1 = y - exp_y + (1/sigma)*(i - exp_pi) - psi_nya * (rho_a - 1) * a
eq2 = pi - beta * exp_pi - kappa * y - sigma_pi * eps_pi
eq3 = i - phi_pi * pi - phi_y * y - v
eq4 = a - rho_a * al - sigma_a * eps_a
eq5 = v - rho_v * vl - sigma_v * eps_v
eq6 = y - exp_yl - eta_y
eq7 = pi - exp_pil - eta_pi

equations = Matrix([eq1, eq2, eq3, eq4, eq5, eq6, eq7])
```

## Observation Equations
Using the observation equations described in the beggining of thie notebook, the expression for the observable variables
(linear function of the state variables) should be on the right side of the equation. These expressions should also go
into a `Matrix` object. This set of equations  should include only the observation equations of the system.

```python
obs01 = y
obs02 = pi
obs03 = 1/beta - 1 + i

obs_equations = Matrix([obs01, obs02, obs03])
```

---
# Model Calibration
Now that we have decribed the full set of equilibrium conditions of the models, we can calibrate the parameters. We will
save the simulated data to illuestrate the estimation processo later on. To calibrate a model, all you need is a
dictionary where the keys are the parameter symbols (*not* the string with their names) and the values are their
respective calibration value. The values below will be our "true" parameters. Later in this notebook we will estimate
them.

```python
calib_dict = {sigma: 1.3,
              varphi: 1,
              alpha: 0.4,
              beta: 0.997805,
              theta: 0.75,
              phi_pi: 1.5,
              phi_y: 0.2,
              rho_a: 0.9,
              sigma_a: 1.1,
              rho_v: 0.5,
              sigma_v: 0.3,
              sigma_pi: 0.8}
```

to build the DSGE object we just have to pass the `Matrix` objects that we created and the calibration dictionary.

```python
dsge_simul = DSGE(endog=endog,
                  endogl=endogl,
                  exog=exog,
                  expec=expec,
                  state_equations=equations,
                  obs_equations=obs_equations,
                  calib_dict=calib_dict)
```

Since we passed a calibration dictionary, the model already knows that it only has to substitute the calibration values
in their respective symbolic representaions and solves the model. Just by writing the statement above, the `dsge_simul`
object already computes everything.

To check if the calbration values yield a solution or not, the model has the `.eu` attibute, mirrors the definition from
the original *Sims (2002)* `gensys.m` function. It represents the existance and uniqueness of the solaution of the model.
If the first entry is equal to 1 it means the model has a solution and if the second entry is equal to 1 it means that
the solution is unique.

```python
print(dsge_simul.eu)
```

    [1, 1]

---
# Simulation

There is a method to simulate data from the calibrated model. Here we are using a random seed for replication purposes.
We will generate the data and plot the observed variables.

```python
df_obs, df_states = dsge_simul.simulate(n_obs=250, random_seed=1)

df_obs.plot(figsize=(15, 9))
```

---
# Estimation
The estimation DSGEs usually comes with the problem of identificiationm of some of their parameters. To handle this
problem, the parameters that are not identified usually are calibrated prior to estimation. In the cell below, we first
create a dictionary of the parameters we want to estimate and then create a `Matrix` object of the parameters that are
going to be estimated.

```python
calib_param = {varphi: 1, alpha: 0.4, beta: 0.997805}
estimate_param = Matrix([sigma, theta, phi_pi, phi_y, rho_a, sigma_a, rho_v, sigma_v, sigma_pi])
```

Since we use bayesian estimation with MCMC we need to define the priors of the parameters that are going to be
estimated. The available distributions for priors are Beta, Gamma, Inverse Gamma, Uniform and Normal. For each prior you
have to choose a mean and a standard deviation. The table below shows how the mean and standard deviations are related
to the distribution parameters. There is also a `label` entry for each parameter, which will be used for legends and
titles on the output charts, and notice that it accepts $\LaTeX$ entries.

|Distribution|Parameter $a$|Parameter $b$|Support           |Mean           |Variance                                           |
|------------|-------------|-------------|------------------|---------------|---------------------------------------------------|
|Beta        |$a>0$ (shape)|$b>0$ (shape)|$\left[0,1\right]$|$\frac{a}{a+b}$|$\frac{ab}{\left(a+b\right)^{2}\left(a+b+1\right)}$|
|Gamma	     |$a>0$ (shape)|$b>0$ (scale)|$\mathbb{R}^{+}$  |$ab$           |$ab^{2}$                                           |
|Inverse Gamma|$a>0$ (shape)|$b>0$ (scale)|$\mathbb{R}^{+}$ |$\frac{\beta}{\alpha-1}$|$\frac{\beta^{2}}{\left(\alpha-1\right)^{2}\left(\alpha-2\right)}$|
|Uniform     |$a\in\mathbb{R}$ (lower bound)|b>a (upper bound)|$\left[a,b\right]$|$\frac{a+b}{2}$|$\frac{\left(b-a\right)^{2}}{12}$|
|Normal      |$a\in\mathbb{R}$ (mean)|$b>0$ (std)|$\mathbb{R}$|$a$|$b^{2}$|

```python
prior_dict = {sigma:    {'dist': 'normal',   'mean':  1.30, 'std': 0.20, 'label': '$\\sigma$'},
              theta:    {'dist': 'beta',     'mean':  0.60, 'std': 0.20, 'label': '$\\theta$'},
              phi_pi:   {'dist': 'normal',   'mean':  1.50, 'std': 0.35, 'label': '$\\phi_{\\pi}$'},
              phi_y:    {'dist': 'gamma',    'mean':  0.25, 'std': 0.10, 'label': '$\\phi_{y}$'},
              rho_a:    {'dist': 'beta',     'mean':  0.50, 'std': 0.25, 'label': '$\\rho_a$'},
              sigma_a:  {'dist': 'invgamma', 'mean':  0.50, 'std': 0.25, 'label': '$\\sigma_a$'},
              rho_v:    {'dist': 'beta',     'mean':  0.50, 'std': 0.25, 'label': '$\\rho_v$'},
              sigma_v:  {'dist': 'invgamma', 'mean':  0.50, 'std': 0.25, 'label': '$\\sigma_v$'},
              sigma_pi: {'dist': 'invgamma', 'mean':  0.50, 'std': 0.25, 'label': '$\\sigma_{\\pi}$'}}

dsge = DSGE(endog=endog,
            endogl=endogl,
            exog=exog,
            expec=expec,
            state_equations=equations,
            estimate_params=estimate_param,
            calib_dict=calib_param,
            obs_equations=obs_equations,
            prior_dict=prior_dict,
            obs_data=df_obs,
            verbose=True)
```

After the model is declared, you can to pass three arguments to the `estimate` method.

The argument `file_path` is the address for where you would like to save the HDF file with the generated MCMC chains.
This filetype was chosen because of its fast read/write speed (on the other hand, the file is very large). Every 100
iterations, the chains will be saved to this file, which ensure that the MCMC chains are not lost on the event that the
execution is interrupted. If you pass the adress of a file that already exists, it will read the file and continue the
chain from where it stoped.

The `nsim` (default: 1000) is the size of the chain and `ck` (deafult: 0.2) is the scale parameter for the
Accpetance-Rejection part of the metropolis-hasting algorithm, which should be set in order to match your desired
acceptance rate.

```python
dsge.estimate(file_path='example_snkm.h5', nsim=2000, ck=0.3)
```

    Metropolis-Hastings:   3%|▎         | 63/1999 [00:36<17:41,  1.82it/s]/Users/gustavoamarante/anaconda3/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py:1: PerformanceWarning:
    your performance may suffer as PyTables will pickle object types that it cannot
    map directly to c-types [inferred_type->floating,key->block0_values] [items->Index(['sigma', 'theta', 'phi_pi', 'phi_y', 'rho_a', 'sigma_a', 'rho_v',
           'sigma_v', 'sigma_pi'],
          dtype='object')]

      """Entry point for launching an IPython kernel.
    Metropolis-Hastings: 100%|██████████| 1999/1999 [18:13<00:00,  1.83it/s]

    Acceptance rate: 28.349999999999998 percent

The `eval_chain` method generates two charts. The first one contains the subplots of the MCMC chains and the second one
is the prior (countinous line) and posterior (histogram) for each parameter. The `burnin` parameter defines how much of
the beggining of the chain left out of the analysis. If the input is a `float` between 0 and 1, that percentage of the
beggining of the chain will be dropped, and if it is an `int`, that many observations are dropped.

```python
dsge.eval_chains(burnin=0, show_charts=True)
```

After the estimation, the DSGE model obsject earns the `posterior_table` attribute, which contains a pandas DataFrame
with the tradional measures reported on academic papers.

```python
dsge.posterior_table
```

      prior dist
      prior mean
      prior std
      posterior mode
      posterior mean
      posterior 5%
      posterior 95%

      sigma
      normal
      1.3
      0.2
      1.225852
      1.235096
      1.226802
      1.242856

      theta
      beta
      0.6
      0.2
      0.673278
      0.678953
      0.674504
      0.682882

      phi_pi
      normal
      1.5
      0.35
      1.496686
      1.500633
      1.489683
      1.512747

      phi_y
      gamma
      0.25
      0.1
      0.177505
      0.175000
      0.171605
      0.178876

      rho_a
      beta
      0.5
      0.25
      0.744787
      0.747697
      0.739498
      0.754306

      sigma_a
      invgamma
      0.5
      0.25
      0.529376
      0.530659
      0.524724
      0.536755

      rho_v
      beta
      0.5
      0.25
      0.377100
      0.377956
      0.371537
      0.384737

      sigma_v
      invgamma
      0.5
      0.25
      0.529523
      0.533092
      0.527101
      0.539630

      sigma_pi
      invgamma
      0.5
      0.25
      0.675716
      0.668371
      0.663087
      0.673798

ANALIZAR A TABELA DAS POSTERIORIS
REVISAR COMO AS CADEIAS SÃO FORMADAS


# Can A Sprinter Run a Loop-the-Loop?

```
red,lime,blue,green,purple = "#CF5970","#94B848","#878CC0","#5EBAA1","#BD5DC7"
```

## Constants and Initial Conditions

```
R = 3.0 # m
g = 9.8 # m/s^2
dt = 0.001 #s
f = 10 # N/kg
sigma = 0.9 #1/s
theta0 = 0.0 # rad
omega0 = 3.0 # rad/s
```

## The Equations of Motion
\begin{align}
\alpha &= \frac{1}{R}( f - \sigma R \omega - g \sin \theta) \\
\frac{F_N}{m g} &= \cos \theta + \frac{R \omega^2}{g}
\end{align}

```
from scipy.integrate import odeint
```

```
def eom(u,t,R):
    ''' The equation of motion (eom) for runner in plane polar coordinates.'''
    omega = u[0]
    theta = u[1]
    return array([(f - sigma*R*omega - g * sin(theta))/R,omega])
```

### Setup the attempt

```
t = arange(0,5.0,dt)
init = [omega0,theta0]
```

### Integrate the equation of motion

```
omega = []
theta = []
time = []
R = [2.5,3.0]
for n,cR in enumerate(R):
    co = []
    ctheta = []
    ct = []
    u = odeint(eom,init,t,args=tuple([cR]))
    index = where(u[:,1]< 2.0*pi)
    omega.append(u[index,0][0])
    theta.append(u[index,1][0])
    time.append(t[index])
```

### Plot the results

```
figure(figsize=(10,5))
plot(time[0],theta[0]/(2.0*pi),'-', color=blue, linewidth=3, label=r'$R=%3.2f$ m'%R[0])
plot(time[1],theta[1]/(2.0*pi),'-', color=red, linewidth=3, label=r'$R=%3.2f$ m'%R[1])

ylabel(r'$\theta\, (2\pi)$')
xlabel(r'$t$ (s)')
legend(loc='upper left', frameon=False, fontsize=20)
tight_layout()
savefig('theta_vs_t.pdf')
```

```
figure(figsize=(10,5))

axhline(y=sqrt(g*R[0]), linestyle=':',color=blue, linewidth=2, label=r'$v_{\mathrm{min}}(2.5)$')
plot(theta[0]/(2.0*pi),omega[0]*R[0],'-', color=blue, linewidth=3, label=r'$R=%3.2f$ m'%R[0])
axhline(y=sqrt(g*R[1]), linestyle=':',color=red, linewidth=2, label=r'$v_{\mathrm{min}}(3)$')
plot(theta[1]/(2.0*pi),omega[1]*R[1],'-', color=red, linewidth=3, label=r'$R=%3.2f$ m'%R[1])

xlabel(r'$\theta\, (2\pi)$')
ylabel('$v$ (m/s)')
legend(loc='upper left', frameon=False, fontsize=20, ncol=2)
tight_layout()
savefig('v_vs_theta.pdf')
```

### Plot the normal force

```
figure(figsize=(10,5))

Nomg = cos(theta[0]) + R[0]*omega[0]**2/g
plot(theta[0]/(2.0*pi),Nomg,'-', color=blue, linewidth=3, label=r'$R=%3.2f$ m'%R[0])

Nomg = cos(theta[1]) + R[1]*omega[1]**2/g
plot(theta[1]/(2.0*pi),Nomg,'-', color=red, linewidth=3, label=r'$R=%3.2f$ m'%R[1])

xlabel(r'$\theta\, (2\pi)$')
ylabel('$F_N/m g$')
legend(loc='upper left', frameon=False, fontsize=20, ncol=1)
ylim(0,8)
tight_layout()
savefig('normal_vs_theta.pdf')
```

```

```


# Introduction of basic concepts

## Linear versus nonlinear equations

### Algebraic equations

A linear, scalar, algebraic equation in $x$ has the form

$$
ax + b = 0,
$$

for arbitrary real constants $a$ and $b$. The unknown is a number $x$.
All other algebraic equations, e.g., $x^2 + ax + b = 0$, are nonlinear.
The typical feature in a nonlinear algebraic equation is that the unknown
appears in products with itself, like $x^2$ or $e^x = 1 + x +\frac{1}{2} x^2 +
\frac{1}{3!}x^3 + \cdots$.

We know how to solve a linear algebraic equation, $x=-b/a$, but there are
no general methods for finding the exact solutions of
nonlinear algebraic equations, except for very special cases (quadratic
equations constitute a primary example). A nonlinear algebraic equation
may have no solution, one solution, or many solutions. The tools for
solving nonlinear algebraic equations are *iterative methods*, where
we construct a series of linear equations, which we know how to solve,
and hope that the solutions of the linear equations converge to a
solution of the nonlinear equation we want to solve.
Typical methods for nonlinear algebraic equation equations are
Newton's method, the Bisection method, and the Secant method.

### Differential equations

The unknown in a differential equation is a function and not a number.
In a linear differential equation, all terms involving the unknown function
are linear in the unknown function or its derivatives. Linear here means that
the unknown function, or a derivative of it, is multiplied by a number or
a known function. All other differential equations are non-linear.

The easiest way to see if an equation is nonlinear, is to spot nonlinear terms
where the unknown function or its derivatives are multiplied by
each other. For example, in

$$
u^{\prime}(t) = -a(t)u(t) + b(t),
$$

the terms involving the unknown function $u$ are linear: $u^{\prime}$ contains
the derivative of the unknown function multiplied by unity, and $au$ contains
the unknown function multiplied by a known function.
However,

$$
u^{\prime}(t) = u(t)(1 - u(t)),
$$

is nonlinear because of the term $-u^2$ where the unknown function is
multiplied by itself. Also

$$
\frac{\partial u}{\partial t} + u\frac{\partial u}{\partial x} = 0,
$$

is nonlinear because of the term $uu_x$ where the unknown
function appears in a product with its derivative.
(Note here that we use different notations for derivatives: $u^{\prime}$
or $du/dt$ for a function $u(t)$ of one variable,
$\frac{\partial u}{\partial t}$ or $u_t$ for a function of more than one
variable.)

Another example of a nonlinear equation is

$$
u^{\prime\prime} + \sin(u) =0,
$$

because $\sin(u)$ contains products of $u$, which becomes clear
if we expand the function in a Taylor series:

$$
\sin(u) = u - \frac{1}{3} u^3 + \ldots
$$

**Mathematical proof of linearity.**

To really prove mathematically that some differential equation
in an unknown $u$ is linear,
show for each term $T(u)$ that with $u = au_1 + bu_2$ for
constants $a$ and $b$,

$$
T(au_1 + bu_2) = aT(u_1) + bT(u_2)\thinspace .
$$

For example, the term $T(u) = (\sin^2 t)u'(t)$ is linear because

$$
\begin{align*}
T(au_1 + bu_2) &= (\sin^2 t)(au_1(t) + b u_2(t))\\
& = a(\sin^2 t)u_1(t) + b(\sin^2 t)u_2(t)\\
& =aT(u_1) + bT(u_2)\thinspace .
\end{align*}
$$

However, $T(u)=\sin u$ is nonlinear because

$$
T(au_1 + bu_2) = \sin (au_1 + bu_2) \neq a\sin u_1 + b\sin u_2\thinspace .
$$

## A simple model problem

A series of forthcoming examples will explain how to tackle
nonlinear differential equations with various techniques.
We start with the (scaled) logistic equation as model problem:

$$
\begin{equation}
u^{\prime}(t) = u(t)(1 - u(t)) \thinspace .
\label{nonlin:timediscrete:logistic:eq} \tag{1}
\end{equation}
$$

This is a nonlinear ordinary differential equation (ODE)
which will be solved by
different strategies in the following.
Depending on the chosen
time discretization of ([1](#nonlin:timediscrete:logistic:eq)),
the mathematical problem to be solved at every time level will
either be a linear algebraic equation or a nonlinear
algebraic equation.
In the former case, the time discretization method transforms
the nonlinear ODE into linear subproblems at each time level, and
the solution is straightforward to find since linear algebraic equations
are easy to solve. However,
when the time discretization leads to nonlinear algebraic equations, we
cannot (except in very rare cases) solve these without turning to
approximate, iterative solution methods.

The next subsections introduce various methods
for solving nonlinear differential equations,
using ([1](#nonlin:timediscrete:logistic:eq)) as model. We shall go through
the following set of cases:

 * explicit time discretization methods (with no need to
   solve nonlinear algebraic equations)

 * implicit Backward Euler time discretization, leading to nonlinear
   algebraic equations solved by

  * an exact analytical technique

  * Picard iteration based on manual linearization

  * a single Picard step

  * Newton's method

 * implicit Crank-Nicolson time discretization and linearization
   via a geometric mean formula

Thereafter, we compare the performance of the various approaches. Despite
the simplicity of ([1](#nonlin:timediscrete:logistic:eq)), the conclusions
reveal typical features of the various methods in much more complicated
nonlinear PDE problems.

## Linearization by explicit time discretization

Time discretization methods are divided into explicit and implicit
methods. Explicit methods lead to a closed-form formula for
finding new values of the unknowns, while implicit methods give
a linear or nonlinear system of equations that couples (all) the
unknowns at a new time level. Here we shall demonstrate that
explicit methods constitute an efficient way to deal with nonlinear
differential equations.

The Forward Euler
method is an explicit method. When applied to
([1](#nonlin:timediscrete:logistic:eq)), sampled at $t=t_n$, it results in

$$
\frac{u^{n+1} - u^n}{\Delta t} = u^n(1 - u^n),
$$

which is a *linear* algebraic
equation for the unknown value $u^{n+1}$ that we can easily solve:

$$
u^{n+1} = u^n + \Delta t\,u^n(1 - u^n)\thinspace .
$$

In this case, the nonlinearity in the original equation poses no difficulty
in the discrete algebraic equation.
Any other explicit scheme in time will also give only linear
algebraic equations
to solve. For example, a typical 2nd-order Runge-Kutta method
for ([1](#nonlin:timediscrete:logistic:eq)) leads to the following
formulas:

$$
\begin{align*}
u^* &= u^n + \Delta t u^n(1 - u^n),\\
u^{n+1} &= u^n + \Delta t \frac{1}{2} \left(
u^n(1 - u^n) + u^*(1 - u^*))
\right)\thinspace .
\end{align*}
$$

The first step is linear in the unknown $u^*$. Then $u^*$ is
known in the next step, which is linear in the unknown $u^{n+1}$ .

## Exact solution of nonlinear algebraic equations

Switching to a Backward Euler scheme for
([1](#nonlin:timediscrete:logistic:eq)),

$$
\begin{equation}
\frac{u^{n} - u^{n-1}}{\Delta t} = u^n(1 - u^n),
\label{nonlin:timediscrete:logistic:eq:BE} \tag{2}
\end{equation}
$$

results in a nonlinear algebraic equation for the unknown value $u^n$.
The equation is of quadratic type:

$$
\Delta t (u^n)^2 + (1-\Delta t)u^n - u^{n-1} = 0,
$$

and may be solved exactly by the well-known formula for such equations.
Before we do so, however, we will
introduce a shorter, and often cleaner, notation for
nonlinear algebraic equations at a given time level. The notation is
inspired by the natural notation (i.e., variable names) used in a
program, especially in more advanced partial differential equation
problems. The unknown in the algebraic equation is denoted by $u$,
while $u^{(1)}$ is the value of the unknown at the previous time level
(in general, $u^{(\ell)}$ is the value of the unknown $\ell$ levels
back in time). The notation will be frequently used in later
sections. What is meant by $u$ should be evident from the context: $u$
may either be 1) the exact solution of the ODE/PDE problem,
2) the numerical approximation to the exact solution, or 3) the unknown
solution at a certain time level.

The quadratic equation for the unknown $u^n$ in
([2](#nonlin:timediscrete:logistic:eq:BE)) can, with the new
notation, be written

$$
\begin{equation}
F(u) = \Delta t u^2 + (1-\Delta t)u - u^{(1)} = 0\thinspace .
\label{nonlin:timediscrete:logistic:eq:F} \tag{3}
\end{equation}
$$

The solution is readily found to be

$$
\begin{equation}
u = \frac{1}{2\Delta t}
\left(-1+\Delta t \pm \sqrt{(1-\Delta t)^2 - 4\Delta t u^{(1)}}\right)
\thinspace .
\label{nonlin:timediscrete:logistic:eq:roots} \tag{4}
\end{equation}
$$

Now we encounter a fundamental challenge with nonlinear
algebraic equations:
the equation may have more than one solution. How do we pick the right
solution? This is in general a hard problem.
In the present simple case, however, we can analyze the roots mathematically
and provide an answer. The idea is to expand the roots
in a series in $\Delta t$ and truncate after the linear term since
the Backward Euler scheme will introduce an error proportional to
$\Delta t$ anyway. Using `sympy`, we find the following Taylor series
expansions of the roots:

```
import sympy as sym
dt, u_1, u = sym.symbols('dt u_1 u')
r1, r2 = sym.solve(dt*u**2 + (1-dt)*u - u_1, u)  # find roots
r1
```

```
r2
```

```
print r1.series(dt, 0, 2)    # 2 terms in dt, around dt=0
```

```
print r2.series(dt, 0, 2)
```

We see that the `r1` root, corresponding to
a minus sign in front of the square root in
([4](#nonlin:timediscrete:logistic:eq:roots)),
behaves as $1/\Delta t$ and will therefore
blow up as $\Delta t\rightarrow 0$! Since we know that $u$ takes on
finite values, actually it is less than or equal to 1,
only the `r2` root is of relevance in this case: as $\Delta t\rightarrow 0$,
$u\rightarrow u^{(1)}$, which is the expected result.

For those who are not well experienced with approximating mathematical
formulas by series expansion, an alternative method of investigation
is simply to compute the limits of the two roots as $\Delta t\rightarrow 0$
and see if a limit appears unreasonable:

```
print r1.limit(dt, 0)
```

```
print r2.limit(dt, 0)
```

## Linearization

When the time integration of an ODE results in a nonlinear algebraic
equation, we must normally find its solution by defining a sequence
of linear equations and hope that the solutions of these linear equations
converge to the desired solution of the nonlinear algebraic equation.
Usually, this means solving the linear equation repeatedly in an
iterative fashion.
Alternatively, the nonlinear equation can sometimes be approximated by one
linear equation, and consequently there is no need for iteration.

Constructing a linear equation from a nonlinear one requires
*linearization* of each nonlinear term. This can be done manually
as in Picard iteration, or fully algorithmically as in Newton's method.
Examples will best illustrate how to linearize nonlinear problems.

## Picard iteration

Let us write ([3](#nonlin:timediscrete:logistic:eq:F)) in a
more compact form

$$
F(u) = au^2 + bu + c = 0,
$$

with $a=\Delta t$, $b=1-\Delta t$, and $c=-u^{(1)}$.
Let $u^{-}$ be an available approximation of the unknown $u$.
Then we can linearize the term $u^2$ simply by writing
$u^{-}u$. The resulting equation, $\hat F(u)=0$, is now linear
and hence easy to solve:

$$
F(u)\approx\hat F(u) = au^{-}u + bu + c = 0\thinspace .
$$

Since the equation $\hat F=0$ is only approximate, the solution $u$
does not equal the exact solution $\uex$ of the exact
equation $F(\uex)=0$, but we can hope that $u$ is closer to
$\uex$ than $u^{-}$ is, and hence it makes sense to repeat the
procedure, i.e., set $u^{-}=u$ and solve $\hat F(u)=0$ again.
There is no guarantee that $u$ is closer to $\uex$ than $u^{-}$,
but this approach has proven to be effective in a wide range of
applications.

The idea of turning a nonlinear equation into a linear one by
using an approximation $u^{-}$ of $u$ in nonlinear terms is
a widely used approach that goes under many names:
*fixed-point iteration*, the method of *successive substitutions*,
*nonlinear Richardson iteration*, and *Picard iteration*.
We will stick to the latter name.

Picard iteration for solving the nonlinear equation
arising from the Backward Euler discretization of the logistic
equation can be written as

$$
u = -\frac{c}{au^{-} + b},\quad u^{-}\ \leftarrow\ u\thinspace .
$$

The $\leftarrow$ symbols means assignment (we set $u^{-}$ equal to
the value of $u$).
The iteration is started with the value of the unknown at the
previous time level: $u^{-}=u^{(1)}$.

Some prefer an explicit iteration counter as superscript
in the mathematical notation. Let $u^k$ be the computed approximation
to the solution in iteration $k$. In iteration $k+1$ we want
to solve

$$
au^k u^{k+1} + bu^{k+1} + c = 0\quad\Rightarrow\quad u^{k+1}
= -\frac{c}{au^k + b},\quad k=0,1,\ldots
$$

Since we need to perform the iteration at every time level, the
time level counter is often also included:

$$
au^{n,k} u^{n,k+1} + bu^{n,k+1} - u^{n-1} = 0\quad\Rightarrow\quad u^{n,k+1}
= \frac{u^n}{au^{n,k} + b},\quad k=0,1,\ldots,
$$

with the start value $u^{n,0}=u^{n-1}$ and the final converged value
$u^{n}=u^{n,k}$ for sufficiently large $k$.

However, we will normally apply a mathematical notation in our
final formulas that is as close as possible to what we aim to write
in a computer code and then it becomes natural to use $u$ and $u^{-}$
instead of $u^{k+1}$ and $u^k$ or $u^{n,k+1}$ and $u^{n,k}$.

### Stopping criteria

The iteration method can typically be terminated when the change
in the solution is smaller than a tolerance $\epsilon_u$:

$$
|u - u^{-}| \leq\epsilon_u,
$$

or when the residual in the equation is sufficiently small ($< \epsilon_r$),

$$
|F(u)|= |au^2+bu + c| < \epsilon_r\thinspace .
$$

### A single Picard iteration

Instead of iterating until a stopping criterion is fulfilled, one may
iterate a specific number of times. Just one Picard iteration is
popular as this corresponds to the intuitive idea of approximating a
nonlinear term like $(u^n)^2$ by $u^{n-1}u^n$. This follows from the
linearization $u^{-}u^n$ and the initial choice of $u^{-}=u^{n-1}$ at
time level $t_n$. In other words, a single Picard iteration
corresponds to using the solution at the previous time level to
linearize nonlinear terms. The resulting discretization becomes (using
proper values for $a$, $b$, and $c$)

$$
\begin{equation}
\frac{u^{n} - u^{n-1}}{\Delta t} = u^n(1 - u^{n-1}),
\label{nonlin:timediscrete:logistic:BE:Picard:1it} \tag{5}
\end{equation}
$$

which is a linear algebraic equation in the unknown $u^n$, making
it easy to solve for $u^n$ without any need for
an alternative notation.

We shall later refer to the strategy of taking one Picard step, or
equivalently, linearizing terms with use of the solution at the
previous time step, as the *Picard1* method. mathcal{I}_t is a widely used
approach in science and technology, but with some limitations if
$\Delta t$ is not sufficiently small (as will be illustrated later).

**Notice.**

Equation ([5](#nonlin:timediscrete:logistic:BE:Picard:1it)) does not
correspond to a "pure" finite difference method where the equation
is sampled at a point and derivatives replaced by differences (because
the $u^{n-1}$ term on the right-hand side must then be $u^n$). The
best interpretation of the scheme
([5](#nonlin:timediscrete:logistic:BE:Picard:1it)) is a Backward Euler
difference combined with a single (perhaps insufficient) Picard
iteration at each time level, with the value at the previous time
level as start for the Picard iteration.

## Linearization by a geometric mean

We consider now a Crank-Nicolson discretization of
([1](#nonlin:timediscrete:logistic:eq)). This means that the
time derivative is approximated by a centered
difference,

$$
[D_t u = u(1-u)]^{n+\frac{1}{2}},
$$

written out as

$$
\begin{equation}
\frac{u^{n+1}-u^n}{\Delta t} = u^{n+\frac{1}{2}} -
(u^{n+\frac{1}{2}})^2\thinspace .
\label{nonlin:timediscrete:logistic:geometric:mean:scheme} \tag{6}
\end{equation}
$$

The term $u^{n+\frac{1}{2}}$ is normally approximated by an arithmetic
mean,

$$
u^{n+\frac{1}{2}}\approx \frac{1}{2}(u^n + u^{n+1}),
$$

such that the scheme involves the unknown function only at the time levels
where we actually intend to compute it.
The same arithmetic mean applied to the nonlinear term gives

$$
(u^{n+\frac{1}{2}})^2\approx \frac{1}{4}(u^n + u^{n+1})^2,
$$

which is nonlinear in the unknown $u^{n+1}$.
However, using a *geometric mean* for $(u^{n+\frac{1}{2}})^2$
is a way of linearizing the nonlinear term in
([6](#nonlin:timediscrete:logistic:geometric:mean:scheme)):

$$
(u^{n+\frac{1}{2}})^2\approx u^nu^{n+1}\thinspace .
$$

Using an arithmetic mean on the linear $u^{n+\frac{1}{2}}$ term in
([6](#nonlin:timediscrete:logistic:geometric:mean:scheme)) and a geometric
mean for the second term, results in a linearized equation for the
unknown $u^{n+1}$:

$$
\frac{u^{n+1}-u^n}{\Delta t} =
\frac{1}{2}(u^n + u^{n+1}) + u^nu^{n+1},
$$

which can readily be solved:

$$
u^{n+1} = \frac{1 + \frac{1}{2}\Delta t}{1+\Delta t u^n - \frac{1}{2}\Delta t}
u^n\thinspace .
$$

This scheme can be coded directly, and since
there is no nonlinear algebraic equation to iterate over,
we skip the simplified notation with $u$ for $u^{n+1}$
and $u^{(1)}$ for $u^n$. The technique with using
a geometric average is an example of transforming a nonlinear
algebraic equation to a linear one, without any need for iterations.

The geometric mean approximation is often very effective for
linearizing quadratic nonlinearities. Both the arithmetic and geometric mean
approximations have truncation errors of order $\Delta t^2$ and are
therefore compatible with the truncation error $\Oof{\Delta t^2}$
of the centered difference approximation for $u^\prime$ in the Crank-Nicolson
method.

Applying the operator notation for the means and finite differences,
the linearized Crank-Nicolson scheme for the logistic equation can be
compactly expressed as

$$
[D_t u = \overline{u}^{t} + \overline{u^2}^{t,g}]^{n+\frac{1}{2}}\thinspace .
$$

**Remark.**

If we use an arithmetic instead of a geometric mean
for the nonlinear term in
([6](#nonlin:timediscrete:logistic:geometric:mean:scheme)),
we end up with a nonlinear term $(u^{n+1})^2$.
This term can be linearized as $u^{-}u^{n+1}$ in a Picard iteration
approach and in particular as
$u^nu^{n+1}$ in a Picard1 iteration approach.
The latter gives a scheme almost identical to the one arising from
a geometric mean (the difference in $u^{n+1}$
being $\frac{1}{4}\Delta t u^n(u^{n+1}-u^n)\approx \frac{1}{4}\Delta t^2
u^\prime u$, i.e., a difference of size $\Delta t^2$).

## Newton's method

The Backward Euler scheme ([2](#nonlin:timediscrete:logistic:eq:BE))
for the logistic equation leads to a nonlinear algebraic equation
([3](#nonlin:timediscrete:logistic:eq:F)). Now we write any nonlinear
algebraic equation in the general and compact form

$$
F(u) = 0\thinspace .
$$

Newton's method linearizes this equation by approximating $F(u)$ by
its Taylor series expansion around a computed value $u^{-}$
and keeping only the linear part:

$$
\begin{align*}
F(u) &= F(u^{-}) + F^{\prime}(u^{-})(u - u^{-}) + {\frac{1}{2}}F^{\prime\prime}(u^{-})(u-u^{-})^2
+\cdots\\
& \approx F(u^{-}) + F^{\prime}(u^{-})(u - u^{-}) = \hat F(u)\thinspace .
\end{align*}
$$

The linear equation $\hat F(u)=0$ has the solution

$$
u = u^{-} - \frac{F(u^{-})}{F^{\prime}(u^{-})}\thinspace .
$$

Expressed with an iteration index in the unknown, Newton's method takes
on the more familiar mathematical form

$$
u^{k+1} = u^k - \frac{F(u^k)}{F^{\prime}(u^k)},\quad k=0,1,\ldots
$$

mathcal{I}_t can be shown that the error in iteration $k+1$ of Newton's method is
proportional to
the square of the error in iteration $k$, a result referred to as
*quadratic convergence*. This means that for
small errors the method converges very fast, and in particular much
faster than Picard iteration and other iteration methods.
(The proof of this result is found in most textbooks on numerical analysis.)
However, the quadratic convergence appears only if $u^k$ is sufficiently
close to the solution. Further away from the solution the method can
easily converge very slowly or diverge. The reader is encouraged to do
[nonlin:exer:Newton:problems1](#nonlin:exer:Newton:problems1) to get a better understanding
for the behavior of the method.

Application of Newton's method to the logistic equation discretized
by the Backward Euler method is straightforward
as we have

$$
F(u) = au^2 + bu + c,\quad a=\Delta t,\ b = 1-\Delta t,\ c=-u^{(1)},
$$

and then

$$
F^{\prime}(u) = 2au + b\thinspace .
$$

The iteration method becomes

$$
\begin{equation}
u = u^{-} + \frac{a(u^{-})^2 + bu^{-} + c}{2au^{-} + b},\quad
u^{-}\ \leftarrow u\thinspace .
\label{nonlin:timediscrete:logistic:Newton:alg1} \tag{7}
\end{equation}
$$

At each time level, we start the iteration by setting $u^{-}=u^{(1)}$.
Stopping criteria as listed for the Picard iteration can be used also
for Newton's method.

An alternative mathematical form, where we write out $a$, $b$, and $c$,
and use a time level counter $n$ and an iteration counter $k$, takes
the form

$$
\begin{equation}
u^{n,k+1} = u^{n,k} +
\frac{\Delta t (u^{n,k})^2 + (1-\Delta t)u^{n,k} - u^{n-1}}
{2\Delta t u^{n,k} + 1 - \Delta t},\quad u^{n,0}=u^{n-1},
\label{nonlin:timediscrete:logistic:Newton:alg2} \tag{8}
\end{equation}
$$

for $k=0,1,\ldots$.
A program implementation is much closer to ([7](#nonlin:timediscrete:logistic:Newton:alg1)) than to ([8](#nonlin:timediscrete:logistic:Newton:alg2)), but
the latter is better aligned with the established mathematical
notation used in the literature.

## Relaxation

One iteration in Newton's method or
Picard iteration consists of solving a linear problem $\hat F(u)=0$.
Sometimes convergence problems arise because the new solution $u$
of $\hat F(u)=0$ is "too far away" from the previously computed
solution $u^{-}$. A remedy is to introduce a relaxation, meaning that
we first solve $\hat F(u^*)=0$ for a suggested value $u^*$ and
then we take $u$ as a weighted mean of what we had, $u^{-}$, and
what our linearized equation $\hat F=0$ suggests, $u^*$:

$$
u = \omega u^* + (1-\omega) u^{-}\thinspace .
$$

The parameter $\omega$
is known as a *relaxation parameter*, and a choice $\omega < 1$
may prevent divergent iterations.

Relaxation in Newton's method can be directly incorporated
in the basic iteration formula:

$$
\begin{equation}
u = u^{-} - \omega \frac{F(u^{-})}{F^{\prime}(u^{-})}\thinspace .
\label{nonlin:timediscrete:logistic:relaxation:Newton:formula} \tag{9}
\end{equation}
$$

## Implementation and experiments

The program [`logistic.py`](${src_nonlin}/logistic.py) contains
implementations of all the methods described above.
Below is an extract of the file showing how the Picard and Newton
methods are implemented for a Backward Euler discretization of
the logistic equation.

```
def BE_logistic(u0, dt, Nt, choice='Picard',
                eps_r=1E-3, omega=1, max_iter=1000):
    if choice == 'Picard1':
        choice = 'Picard'
        max_iter = 1

    u = np.zeros(Nt+1)
    iterations = []
    u[0] = u0
    for n in range(1, Nt+1):
        a = dt
        b = 1 - dt
        c = -u[n-1]

        if choice == 'Picard':

            def F(u):
                return a*u**2 + b*u + c

            u_ = u[n-1]
            k = 0
            while abs(F(u_)) > eps_r and k < max_iter:
                u_ = omega*(-c/(a*u_ + b)) + (1-omega)*u_
                k += 1
            u[n] = u_
            iterations.append(k)

        elif choice == 'Newton':

            def F(u):
                return a*u**2 + b*u + c

            def dF(u):
                return 2*a*u + b

            u_ = u[n-1]
            k = 0
            while abs(F(u_)) > eps_r and k < max_iter:
                u_ = u_ - F(u_)/dF(u_)
                k += 1
            u[n] = u_
            iterations.append(k)
    return u, iterations
```

The Crank-Nicolson method utilizing a linearization based on the
geometric mean gives a simpler algorithm:

```
def CN_logistic(u0, dt, Nt):
    u = np.zeros(Nt+1)
    u[0] = u0
    for n in range(0, Nt):
        u[n+1] = (1 + 0.5*dt)/(1 + dt*u[n] - 0.5*dt)*u[n]
    return u
```

We may run experiments with the model problem
([1](#nonlin:timediscrete:logistic:eq)) and the different strategies for
dealing with nonlinearities as described above. For a quite coarse
time resolution, $\Delta t=0.9$, use of a tolerance $\epsilon_r=0.1$
in the stopping criterion introduces an iteration error, especially in
the Picard iterations, that is visibly much larger than the
time discretization error due to a large $\Delta t$. This is illustrated
by comparing the upper two plots in
[Figure](#nonlin:timediscrete:logistic:impl:fig:u). The one to
the right has a stricter tolerance $\epsilon = 10^{-3}$, which causes
all the curves corresponding to Picard and Newton iteration to be
on top of each other (and no changes can be visually observed by
reducing $\epsilon_r$ further). The reason why Newton's method does
much better than Picard iteration in the upper left plot is that
Newton's method with one step comes far below the $\epsilon_r$ tolerance,
while the Picard iteration needs on average 7 iterations to bring the
residual down to $\epsilon_r=10^{-1}$, which gives insufficient
accuracy in the solution of the nonlinear equation. mathcal{I}_t is obvious
that the Picard1 method gives significant errors in addition to
the time discretization unless the time step is as small as in
the lower right plot.

The *BE exact* curve corresponds to using the exact solution of the
quadratic equation at each time level, so this curve is only affected
by the Backward Euler time discretization.  The *CN gm* curve
corresponds to the theoretically more accurate Crank-Nicolson
discretization, combined with a geometric mean for linearization.
This curve appears more accurate, especially if we take the plot in
the lower right with a small $\Delta t$ and an appropriately small
$\epsilon_r$ value as the exact curve.

When it comes to the need for iterations, [Figure](#nonlin:timediscrete:logistic:impl:fig:iter) displays the number of
iterations required at each time level for Newton's method and
Picard iteration. The smaller $\Delta t$ is, the better starting value
we have for the iteration, and the faster the convergence is.
With $\Delta t = 0.9$ Picard iteration requires on average 32 iterations
per time step, but this number is dramatically reduced as $\Delta t$
is reduced.

However, introducing relaxation and a parameter $\omega=0.8$
immediately reduces the average of 32 to 7, indicating that for the large
$\Delta t=0.9$, Picard iteration takes too long steps. An approximately optimal
value for $\omega$ in this case is 0.5, which results in an average of only
2 iterations! An even more dramatic impact of $\omega$ appears when
$\Delta t = 1$: Picard iteration does not convergence in 1000 iterations,
but $\omega=0.5$ again brings the average number of iterations down to 2.

Impact of solution strategy and time step length on the solution.

Comparison of the number of iterations at various time levels for Picard and Newton iteration.

[hpl 1: Is this remark really relevant now? Compare with text.]

**Remark.**
The simple Crank-Nicolson method with a geometric mean for the quadratic
nonlinearity gives visually more accurate solutions than the
Backward Euler discretization. Even with a tolerance of $\epsilon_r=10^{-3}$,
all the methods for treating the nonlinearities in the Backward Euler
discretization give graphs that cannot be distinguished. So for
accuracy in this problem, the time discretization is much more crucial
than $\epsilon_r$. Ideally, one should estimate the error in the
time discretization, as the solution progresses, and set $\epsilon_r$
accordingly.

## Generalization to a general nonlinear ODE

Let us see how the various methods in the previous sections
can be applied to the more generic model

$$
\begin{equation}
u^{\prime} = f(u, t),
\label{nonlin:ode:generic:model} \tag{10}
\end{equation}
$$

where $f$ is a nonlinear function of $u$.

### Explicit time discretization

Explicit ODE methods like the Forward Euler scheme, Runge-Kutta methods and
Adams-Bashforth methods all evaluate $f$ at time levels where
$u$ is already computed, so nonlinearities in $f$ do not
pose any difficulties.

### Backward Euler discretization

Approximating $u^{\prime}$ by a backward difference leads to a Backward Euler
scheme, which can be written as

$$
F(u^n) = u^{n} - \Delta t\, f(u^n, t_n) - u^{n-1}=0,
$$

or alternatively

$$
F(u) = u - \Delta t\, f(u, t_n) - u^{(1)} = 0\thinspace .
$$

A simple Picard iteration, not knowing anything about the nonlinear
structure of $f$, must approximate $f(u,t_n)$ by $f(u^{-},t_n)$:

$$
\hat F(u) = u - \Delta t\, f(u^{-},t_n) - u^{(1)}\thinspace .
$$

The iteration starts with $u^{-}=u^{(1)}$ and proceeds with repeating

$$
u^* = \Delta t\, f(u^{-},t_n) + u^{(1)},\quad u = \omega u^* + (1-\omega)u^{-},
\quad u^{-}\ \leftarrow\ u,
$$

until a stopping criterion is fulfilled.

**Explicit vs implicit treatment of nonlinear terms.**

Evaluating $f$ for a known $u^{-}$ is referred to as *explicit* treatment of
$f$, while if $f(u,t)$ has some structure, say $f(u,t) = u^3$, parts of
$f$ can involve the unknown $u$, as in the manual linearization
$(u^{-})^2u$, and then the treatment of $f$ is "more implicit"
and "less explicit". This terminology is inspired by time discretization
of $u^{\prime}=f(u,t)$, where evaluating $f$ for known $u$ values gives
explicit schemes, while treating $f$ or parts of $f$ implicitly,
makes $f$ contribute to the unknown terms in the equation at the new
time level.

Explicit treatment of $f$ usually means stricter conditions on
$\Delta t$ to achieve stability of time discretization schemes. The same
applies to iteration techniques for nonlinear algebraic equations: the "less"
we linearize $f$ (i.e., the more we keep of $u$ in the original formula),
the faster the convergence may be.

We may say that $f(u,t)=u^3$ is treated explicitly if we evaluate $f$
as $(u^{-})^3$, partially implicit if we linearize as $(u^{-})^2u$
and fully implicit if we represent $f$ by $u^3$. (Of course, the
fully implicit representation will require further linearization,
but with $f(u,t)=u^2$ a fully implicit treatment is possible if
the resulting quadratic equation is solved with a formula.)

For the ODE $u^{\prime}=-u^3$ with $f(u,t)=-u^3$ and coarse
time resolution $\Delta t = 0.4$, Picard iteration with $(u^{-})^2u$
requires 8 iterations with $\epsilon_r = 10^{-3}$ for the first
time step, while $(u^{-})^3$ leads to 22 iterations. After about 10
time steps both approaches are down to about 2 iterations per time
step, but this example shows a potential of treating $f$ more
implicitly.

A trick to treat $f$ implicitly in Picard iteration is to
evaluate it as $f(u^{-},t)u/u^{-}$. For a polynomial $f$, $f(u,t)=u^m$,
this corresponds to $(u^{-})^{m}u/u^{-}=(u^{-})^{m-1}u$. Sometimes this more implicit
treatment has no effect, as with $f(u,t)=\exp(-u)$ and $f(u,t)=\ln (1+u)$,
but with $f(u,t)=\sin(2(u+1))$, the $f(u^{-},t)u/u^{-}$ trick
leads to 7, 9, and 11 iterations during the first three steps, while
$f(u^{-},t)$ demands 17, 21, and 20 iterations.
(Experiments can be done with the code [`ODE_Picard_tricks.py`](${src_nonlin}/ODE_Picard_tricks.py).)

Newton's method applied to a Backward Euler discretization of
$u^{\prime}=f(u,t)$
requires computation of the derivative

$$
F^{\prime}(u) = 1 - \Delta t\frac{\partial f}{\partial u}(u,t_n)\thinspace .
$$

Starting with the solution at the previous time level, $u^{-}=u^{(1)}$,
we can just use the standard formula

$$
\begin{equation}
u = u^{-} - \omega \frac{F(u^{-})}{F^{\prime}(u^{-})}
= u^{-} - \omega \frac{u^{-} - \Delta t\, f(u^{-}, t_n) - u^{(1)}}{1 - \Delta t
\frac{\partial}{\partial u}f(u^{-},t_n)}
\thinspace .
\label{nonlin:ode:generic:Newton} \tag{11}
\end{equation}
$$

### Crank-Nicolson discretization

The standard Crank-Nicolson scheme with arithmetic mean approximation of
$f$ takes the form

$$
\frac{u^{n+1} - u^n}{\Delta t} = \frac{1}{2}(f(u^{n+1}, t_{n+1})
+ f(u^n, t_n))\thinspace .
$$

We can write the scheme as a nonlinear algebraic equation

$$
\begin{equation}
F(u) = u - u^{(1)} - \Delta t{\frac{1}{2}}f(u,t_{n+1}) -
\Delta t{\frac{1}{2}}f(u^{(1)},t_{n}) = 0\thinspace .
\label{nonlin:ode:generic:Newton2} \tag{12}
\end{equation}
$$

A Picard iteration scheme must in general employ the linearization

$$
\hat F(u) = u - u^{(1)} - \Delta t{\frac{1}{2}}f(u^{-},t_{n+1}) -
\Delta t{\frac{1}{2}}f(u^{(1)},t_{n}),
$$

while Newton's method can apply the general formula
([11](#nonlin:ode:generic:Newton)) with $F(u)$ given in
([12](#nonlin:ode:generic:Newton2)) and

$$
F^{\prime}(u)= 1 - \frac{1}{2}\Delta t\frac{\partial f}{\partial u}(u,t_{n+1})\thinspace .
$$

## Systems of ODEs

We may write a system of ODEs

$$
\begin{align*}
\frac{d}{dt}u_0(t) &= f_0(u_0(t),u_1(t),\ldots,u_N(t),t),\\
\frac{d}{dt}u_1(t) &= f_1(u_0(t),u_1(t),\ldots,u_N(t),t),\\
&\vdots\\
\frac{d}{dt}u_m(t) &= f_m(u_0(t),u_1(t),\ldots,u_N(t),t),
\end{align*}
$$

as

$$
\begin{equation}
u^{\prime} = f(u,t),\quad u(0)=U_0,
\label{_auto1} \tag{13}
\end{equation}
$$

if we interpret $u$ as a vector $u=(u_0(t),u_1(t),\ldots,u_N(t))$
and $f$ as a vector function with components
$(f_0(u,t),f_1(u,t),\ldots,f_N(u,t))$.

Most solution methods for scalar ODEs, including
the Forward and Backward Euler schemes and the
Crank-Nicolson method, generalize in a
straightforward way to systems of ODEs simply by using vector
arithmetics instead of scalar arithmetics, which corresponds to
applying the scalar scheme to each component of the system.  For
example, here is a backward difference scheme applied to each
component,

$$
\begin{align*}
\frac{u_0^n- u_0^{n-1}}{\Delta t} &= f_0(u^n,t_n),\\
\frac{u_1^n- u_1^{n-1}}{\Delta t} &= f_1(u^n,t_n),\\
&\vdots\\
\frac{u_N^n- u_N^{n-1}}{\Delta t} &= f_N(u^n,t_n),
\end{align*}
$$

which can be written more compactly in vector form as

$$
\frac{u^n- u^{n-1}}{\Delta t} = f(u^n,t_n)\thinspace .
$$

This is a *system of algebraic equations*,

$$
u^n - \Delta t\,f(u^n,t_n) - u^{n-1}=0,
$$

or written out

$$
\begin{align*}
u_0^n - \Delta t\, f_0(u^n,t_n) - u_0^{n-1} &= 0,\\
&\vdots\\
u_N^n - \Delta t\, f_N(u^n,t_n) - u_N^{n-1} &= 0\thinspace .
\end{align*}
$$

### Example

We shall address the $2\times 2$ ODE system for
oscillations of a pendulum
subject to gravity and air drag. The system can be written as

$$
\begin{equation}
\dot\omega = -\sin\theta -\beta \omega |\omega|,
\label{_auto2} \tag{14}
\end{equation}
$$

$$
\begin{equation}
\dot\theta = \omega,
\label{_auto3} \tag{15}
\end{equation}
$$

where $\beta$ is a dimensionless parameter (this is the scaled, dimensionless
version of the original, physical model). The unknown components of the
system are the
angle $\theta(t)$ and the angular velocity $\omega(t)$.
We introduce $u_0=\omega$ and $u_1=\theta$, which leads to

$$
\begin{align*}
u_0^{\prime} = f_0(u,t) &= -\sin u_1 - \beta u_0|u_0|,\\
u_1^{\prime} = f_1(u,t) &= u_0\thinspace .
\end{align*}
$$

A Crank-Nicolson scheme reads

$$
\frac{u_0^{n+1}-u_0^{n}}{\Delta t} = -\sin u_1^{n+\frac{1}{2}}
- \beta u_0^{n+\frac{1}{2}}|u_0^{n+\frac{1}{2}}|\nonumber
$$

$$
\begin{equation}
 \approx -\sin\left(\frac{1}{2}(u_1^{n+1} + u_1n)\right)
- \beta\frac{1}{4} (u_0^{n+1} + u_0^n)|u_0^{n+1}+u_0^n|,
\label{_auto4} \tag{16}
\end{equation}
$$

$$
\begin{equation}
\frac{u_1^{n+1}-u_1^n}{\Delta t} = u_0^{n+\frac{1}{2}}\approx
\frac{1}{2} (u_0^{n+1}+u_0^n)\thinspace .
\label{_auto5} \tag{17}
\end{equation}
$$

This is a *coupled system* of two nonlinear algebraic equations
in two unknowns $u_0^{n+1}$ and $u_1^{n+1}$.

Using the notation $u_0$ and $u_1$ for the unknowns $u_0^{n+1}$ and
$u_1^{n+1}$ in this system, writing $u_0^{(1)}$ and
$u_1^{(1)}$ for the previous values $u_0^n$ and $u_1^n$, multiplying
by $\Delta t$ and moving the terms to the left-hand sides, gives

$$
\begin{equation}
u_0 - u_0^{(1)} + \Delta t\,\sin\left(\frac{1}{2}(u_1 + u_1^{(1)})\right)
+ \frac{1}{4}\Delta t\beta (u_0 + u_0^{(1)})|u_0 + u_0^{(1)}| =0,
\label{nonlin:ode:generic:sys:pendulum:u0} \tag{18}
\end{equation}
$$

$$
\begin{equation}
u_1 - u_1^{(1)} -\frac{1}{2}\Delta t(u_0 + u_0^{(1)}) =0\thinspace .
\label{nonlin:ode:generic:sys:pendulum:u1} \tag{19}
\end{equation}
$$

Obviously, we have a need for solving systems of nonlinear algebraic
equations, which is the topic of the next section.

# Systems of nonlinear algebraic equations

Implicit time discretization methods for a system of ODEs, or a PDE,
lead to *systems* of nonlinear algebraic equations, written
compactly as

$$
F(u) = 0,
$$

where $u$ is a vector of unknowns $u=(u_0,\ldots,u_N)$, and
$F$ is a vector function: $F=(F_0,\ldots,F_N)$.
The system at the end of the section [Systems of ODEs](#nonlin:ode:generic:sys:pendulum) fits
this notation with $N=1$, $F_0(u)$ given by the left-hand side of
([18](#nonlin:ode:generic:sys:pendulum:u0)), while $F_1(u)$ is
the left-hand side of ([19](#nonlin:ode:generic:sys:pendulum:u1)).

Sometimes the equation system has a special structure because of the
underlying problem, e.g.,

$$
A(u)u = b(u),
$$

with $A(u)$ as
an $(N+1)\times (N+1)$ matrix function of $u$ and $b$ as a vector
function: $b=(b_0,\ldots,b_N)$.

We shall next explain how Picard iteration and Newton's method
can be applied to systems like $F(u)=0$ and $A(u)u=b(u)$.
The exposition has a focus on ideas and practical computations.
More theoretical considerations, including quite general results
on convergence properties
of these methods, can be found in Kelley [[Kelley_1995]](#Kelley_1995).

## Picard iteration

We cannot apply Picard iteration to nonlinear equations unless there is
some special structure. For the commonly arising case
$A(u)u=b(u)$ we can linearize the
product $A(u)u$ to $A(u^{-})u$ and $b(u)$ as $b(u^{-})$.
That is, we use the most previously
computed approximation in $A$ and $b$ to arrive at a *linear system* for
$u$:

$$
A(u^{-})u = b(u^{-})\thinspace .
$$

A relaxed iteration takes the form

$$
A(u^{-})u^* = b(u^{-}),\quad u = \omega u^* + (1-\omega)u^{-}\thinspace .
$$

In other words, we solve a system of nonlinear algebraic equations as
a sequence of linear systems.

**Algorithm for relaxed Picard iteration.**

Given $A(u)u=b(u)$ and an initial guess $u^{-}$, iterate until convergence:

1. solve $A(u^{-})u^* = b(u^{-})$ with respect to $u^*$

2. $u = \omega u^* + (1-\omega) u^{-}$

3. $u^{-}\ \leftarrow\ u$

"Until convergence" means that the iteration is stopped when the
change in the unknown, $||u - u^{-}||$, or the residual $||A(u)u-b||$,
is sufficiently small, see the section [Stopping criteria](#nonlin:systems:alg:terminate) for
more details.

## Newton's method

The natural starting point for Newton's method is the general
nonlinear vector equation $F(u)=0$.
As for a scalar equation, the idea is to approximate $F$
around a known value $u^{-}$ by a linear function $\hat F$,
calculated from the first two terms of a Taylor expansion of
$F$. In the multi-variate case these two terms become

$$
F(u^{-}) + J(u^{-}) \cdot (u - u^{-}),
$$

where $J$ is the *Jacobian* of $F$, defined by

$$
J_{i,j} = \frac{\partial F_i}{\partial u_j}\thinspace .
$$

So, the original nonlinear system is approximated by

$$
\hat F(u) = F(u^{-}) + J(u^{-}) \cdot (u - u^{-})=0,
$$

which is linear in $u$ and can be solved in a two-step procedure:
first solve $J\delta u = -F(u^{-})$ with respect to the vector $\delta u$
and then update $u = u^{-} + \delta u$.
A relaxation parameter can easily be incorporated:

$$
u = \omega(u^{-} +\delta u)
+ (1-\omega)u^{-} = u^{-}  + \omega\delta u\thinspace .
$$

**Algorithm for Newton's method.**

Given $F(u)=0$ and an initial guess $u^{-}$, iterate until convergence:

1. solve $J\delta u = -F(u^{-})$ with respect to $\delta u$

2. $u = u^{-} + \omega\delta u$

3. $u^{-}\ \leftarrow\ u$

For the special system with structure $A(u)u=b(u)$,

$$
F_i = \sum_k A_{i,k}(u)u_k - b_i(u),
$$

one gets

$$
\begin{equation}
J_{i,j} = \sum_k \frac{\partial A_{i,k}}{\partial u_j}u_k
+ A_{i,j} -
\frac{\partial b_i}{\partial u_j}\thinspace .
\label{_auto6} \tag{20}
\end{equation}
$$

We realize that the Jacobian needed in Newton's method consists of
$A(u^{-})$ as in the Picard iteration plus two additional terms
arising from the differentiation. Using the notation $A^{\prime}(u)$ for
$\partial A/\partial u$ (a quantity with three indices: $\partial
A_{i,k}/\partial u_j$), and $b^{\prime}(u)$ for $\partial b/\partial u$ (a
quantity with two indices: $\partial b_i/\partial u_j$), we can write
the linear system to be solved as

$$
(A + A^{\prime}u + b^{\prime})\delta u = -Au + b,
$$

or

$$
(A(u^{-}) + A^{\prime}(u^{-})u^{-} + b^{\prime}(u^{-}))\delta u
= -A(u^{-})u^{-} + b(u^{-})\thinspace .
$$

Rearranging the terms demonstrates the difference from the system
solved in each Picard iteration:

$$
\underbrace{A(u^{-})(u^{-}+\delta u) - b(u^{-})}_{\hbox{Picard system}}
+\, \gamma (A^{\prime}(u^{-})u^{-} + b^{\prime}(u^{-}))\delta u
= 0\thinspace .
$$

Here we have inserted a parameter $\gamma$ such that $\gamma=0$
gives the Picard system and $\gamma=1$ gives the Newton system.
Such a parameter can be handy in software to easily switch between
the methods.

**Combined algorithm for Picard and Newton iteration.**

Given $A(u)$, $b(u)$, and an initial guess $u^{-}$, iterate until convergence:

1. solve $(A + \gamma(A^{\prime}(u^{-})u^{-} +
   b^{\prime}(u^{-})))\delta u = -A(u^{-})u^{-} + b(u^{-})$
   with respect to $\delta u$

2. $u = u^{-} + \omega\delta u$

3. $u^{-}\ \leftarrow\ u$

$\gamma =1$ gives a Newton method while $\gamma =0$ corresponds to
Picard iteration.

## Stopping criteria

Let $||\cdot||$ be the standard Euclidean vector norm. Four termination
criteria are much in use:

 * Absolute change in solution: $||u - u^{-}||\leq \epsilon_u$

 * Relative change in solution: $||u - u^{-}||\leq \epsilon_u ||u_0||$,
   where $u_0$ denotes the start value of $u^{-}$ in the iteration

 * Absolute residual: $||F(u)|| \leq \epsilon_r$

 * Relative residual: $||F(u)|| \leq \epsilon_r ||F(u_0)||$

To prevent divergent iterations to run forever,
one terminates the iterations when
the current number of iterations $k$ exceeds a maximum value $k_{\max}$.

The relative criteria are most used since they are not sensitive to
the characteristic size of $u$. Nevertheless, the relative criteria
can be misleading when the initial start value for the iteration is
very close to the solution, since an unnecessary reduction in
the error measure is enforced. In such cases the absolute criteria
work better. mathcal{I}_t is common to combine the absolute and relative measures
of the size of the residual,
as in

$$
\begin{equation}
||F(u)|| \leq \epsilon_{rr} ||F(u_0)|| + \epsilon_{ra},
\label{_auto7} \tag{21}
\end{equation}
$$

where $\epsilon_{rr}$ is the tolerance in the relative criterion
and $\epsilon_{ra}$ is the tolerance in the absolute criterion.
With a very good initial guess for the iteration
(typically the solution of a differential
equation at the previous time level), the term $||F(u_0)||$ is small
and $\epsilon_{ra}$ is the dominating tolerance. Otherwise,
$\epsilon_{rr} ||F(u_0)||$ and the relative criterion dominates.

With the change in solution as criterion we can formulate a combined
absolute and relative measure of the change in the solution:

$$
\begin{equation}
||\delta u|| \leq \epsilon_{ur} ||u_0|| + \epsilon_{ua},
\label{_auto8} \tag{22}
\end{equation}
$$

The ultimate termination criterion, combining the residual and
the change in solution with a test on the maximum number
of iterations, can be expressed as

$$
\begin{equation}
||F(u)|| \leq \epsilon_{rr} ||F(u_0)|| + \epsilon_{ra}
\quad\hbox{or}\quad
||\delta u|| \leq \epsilon_{ur} ||u_0|| + \epsilon_{ua}
\quad\hbox{or}\quad
k>k_{\max}\thinspace .
\label{_auto9} \tag{23}
\end{equation}
$$

## Example: A nonlinear ODE model from epidemiology

A very simple model for the spreading of a disease, such as a flu, takes
the form of a $2\times 2$ ODE system

$$
\begin{equation}
S^{\prime} = -\beta SI,
\label{_auto10} \tag{24}
\end{equation}
$$

$$
\begin{equation}
I^{\prime} = \beta SI - \nu I,
\label{_auto11} \tag{25}
\end{equation}
$$

where $S(t)$ is the number of people who can get ill (susceptibles)
and $I(t)$ is the number of people who are ill (infected).
The constants $\beta >0$ and $\nu >0$ must be given along with
initial conditions $S(0)$ and $I(0)$.

### Implicit time discretization

A Crank-Nicolson scheme leads to a $2\times 2$ system of nonlinear
algebraic equations in the unknowns $S^{n+1}$ and $I^{n+1}$:

$$
\begin{equation}
\frac{S^{n+1}-S^n}{\Delta t} = -\beta [SI]^{n+\frac{1}{2}}
\approx -\frac{\beta}{2}(S^nI^n + S^{n+1}I^{n+1}),
\label{_auto12} \tag{26}
\end{equation}
$$

$$
\begin{equation}
\frac{I^{n+1}-I^n}{\Delta t} = \beta [SI]^{n+\frac{1}{2}} -
\nu I^{n+\frac{1}{2}}
\approx \frac{\beta}{2}(S^nI^n + S^{n+1}I^{n+1}) -
\frac{\nu}{2}(I^n + I^{n+1})\thinspace .
\label{_auto13} \tag{27}
\end{equation}
$$

Introducing $S$ for $S^{n+1}$, $S^{(1)}$ for $S^n$, $I$ for $I^{n+1}$ and
$I^{(1)}$ for $I^n$, we can rewrite the system as

$$
\begin{equation}
F_S(S,I) = S - S^{(1)} +
\frac{1}{2}\Delta t\beta(S^{(1)}I^{(1)} + SI) = 0,
\label{nonlin:systems:alg:SI:CN:FS} \tag{28}
\end{equation}
$$

$$
\begin{equation}
F_I(S,I) = I - I^{(1)} -
\frac{1}{2}\Delta t\beta(S^{(1)}I^{(1)} + SI) +
\frac{1}{2}\Delta t\nu(I^{(1)} + I) =0\thinspace .
\label{nonlin:systems:alg:SI:CN:FI} \tag{29}
\end{equation}
$$

### A Picard iteration

We assume that we have approximations $S^{-}$ and $I^{-}$ to $S$ and
$I$, respectively.  A way of linearizing the only nonlinear term $SI$
is to write $I^{-}S$ in the $F_S=0$ equation and $S^{-}I$ in the
$F_I=0$ equation, which also *decouples* the equations.  Solving the
resulting linear equations with respect to the unknowns $S$ and $I$
gives

$$
\begin{align*}
S &= \frac{S^{(1)} - \frac{1}{2}\Delta t\beta S^{(1)}I^{(1)}}
{1 + \frac{1}{2}\Delta t\beta I^{-}},
\\
I &= \frac{I^{(1)} + \frac{1}{2}\Delta t\beta S^{(1)}I^{(1)} - \frac{1}{2}\Delta t\nu I^{(1)}}
{1 - \frac{1}{2}\Delta t\beta S^{-} + \frac{1}{2}\Delta t \nu}\thinspace .
\end{align*}
$$

Before a new iteration, we must update $S^{-}\ \leftarrow\ S$ and
$I^{-}\ \leftarrow\ I$.

### Newton's method

The nonlinear system
([28](#nonlin:systems:alg:SI:CN:FS))-([29](#nonlin:systems:alg:SI:CN:FI))
can be written as $F(u)=0$ with $F=(F_S,F_I)$ and $u=(S,I)$.  The
Jacobian becomes

$$
J = \left(\begin{array}{cc}
\frac{\partial}{\partial S} F_S & \frac{\partial}{\partial I}F_S\\
\frac{\partial}{\partial S} F_I & \frac{\partial}{\partial I}F_I
\end{array}\right)
= \left(\begin{array}{cc}
1 + \frac{1}{2}\Delta t\beta I & \frac{1}{2}\Delta t\beta S\\
- \frac{1}{2}\Delta t\beta I & 1 - \frac{1}{2}\Delta t\beta S +
\frac{1}{2}\Delta t\nu
\end{array}\right)
\thinspace .
$$

The Newton system $J(u^{-})\delta u = -F(u^{-})$
to be solved in each iteration is then

$$
\begin{align*}
&
\left(\begin{array}{cc}
1 + \frac{1}{2}\Delta t\beta I^{-} & \frac{1}{2}\Delta t\beta S^{-}\\
- \frac{1}{2}\Delta t\beta I^{-} & 1 - \frac{1}{2}\Delta t\beta S^{-} +
\frac{1}{2}\Delta t\nu
\end{array}\right)
\left(\begin{array}{c}
\delta S\\
\delta I
\end{array}\right)
=\\
& \qquad\qquad
\left(\begin{array}{c}
S^{-} - S^{(1)} + \frac{1}{2}\Delta t\beta(S^{(1)}I^{(1)} + S^{-}I^{-})\\
I^{-} - I^{(1)} - \frac{1}{2}\Delta t\beta(S^{(1)}I^{(1)} + S^{-}I^{-}) +
\frac{1}{2}\Delta t\nu(I^{(1)} + I^{-})
\end{array}\right)
\end{align*}
$$

**Remark.**
For this particular system of ODEs, explicit time integration methods work very
well. Even a Forward Euler scheme is fine, but (as also experienced more
generally)
the 4-th order Runge-Kutta method is an excellent
balance between high accuracy, high efficiency, and simplicity.