```python
import numpy as np
import qutip as qt
import matplotlib.pyplot as plt
%matplotlib inline
```

# The Jaynes-Cummings Model

$$
\begin{equation}
H = \hbar\omega_{c}a^{\dagger}a + \hbar\omega_{a}\sigma_{-}^{\dagger}\sigma_{-}+\hbar g(a^{\dagger}\sigma_{-}+a\sigma_{-}^{\dagger})
\end{equation}
$$

This is shorthand notation. If the cavity is system $A$ and atom system $B$ then formally:

$$
\begin{align}
a &= a \otimes \mathbb{I}_{2} \\
\sigma_{-} &= \mathbb{I}_{N} \otimes \sigma_{-}
\end{align}
$$

Part #1

```python
N = 15                 # number of cavity fock states
wc = 1.0  * 2 * np.pi  # cavity frequency
wa = 1.0  * 2 * np.pi  # atom frequency
g  = 0.05 * 2 * np.pi  # coupling strength

# operators
a  = qt.tensor(qt.destroy(N), qt.qeye(2)) # qeye is short for identity
sm = qt.tensor(qt.qeye(N), qt.sigmam())

# Hamiltonian
H = wc * a.dag() * a + wa * sm.dag() * sm + g * (a.dag() * sm + a * sm.dag())
```

 ## **Note that there is no $\hbar$ anywhere**  The Hamiltonians that are used in QuTiP have units of (typically angular) frequency.  We also do not need to specify the exact units for the frequency (e.g. GHz, MHz, etc.)

All numerical equations should be unitless.

## When doing time-evolution, the actual units of frequency set the timescale, for example if $\omega = 2\pi\ \mathrm{GHz}$ then the units of time are $\mathrm{ns}$.

What does the full Hamiltonian look like?

```python
plt.figure(figsize=(8,6))
plt.spy(H.data);
```

What is the density of non-zero elements?

```python
H.data.nnz / H.shape[0]**2
```

    0.06333333333333334

Part #2

```python
# intial state
psi0 = qt.tensor(qt.basis(N,0), qt.basis(2,0))    # start with an excited atom, cavity ground

# Times over which to evaluate system
tlist = np.linspace(0,25,101)

# Make list of expectation value operators
e_ops = [a.dag() * a, sm.dag() * sm]

#Perform evolution
output = qt.sesolve(H, psi0, tlist, e_ops)
```

Lets see what `output` is:

```python
output
```

    Result object with sesolve data.
    --------------------------------
    expect = True
    num_expect = 2, num_collapse = 0

The expectation value results are stored in the `expect` attribute of the result class, i.e. `output.expect`.  To see this, lets plot the results.

```python
plt.figure(figsize=(8,6))
plt.plot(tlist,output.expect[0], lw=3)
plt.plot(tlist,output.expect[1], lw=3)
plt.legend(['$\langle a^{\dagger}a\\rangle$',
            '$\langle \sigma_{-}^{\dagger}\sigma_{-}\\rangle$'],
           loc=1,fontsize=16)
plt.xlabel('Time', fontsize=16)
plt.ylabel('Occupation Number', fontsize=16);
```

If instead we want to return the state of the system at time $t$, then we call the same function without the `e_ops`, or setting `e_ops = []`.

```python
#Perform evolution returning state vectors
output = qt.sesolve(H, psi0, tlist, e_ops=[])
```

```python
output
```

    Result object with sesolve data.
    --------------------------------
    states = True
    num_collapse = 0

Part #3

## Partial trace and entropy

States are stored in the `output.states` attribute. To take the partial trace of a state or operator we use the `Q.ptrace(sel)` method of the Qobj class.  Here `sel` is the index of the desired subsystem(s) that you want to look at.

```python
entropy = np.zeros_like(tlist) # Making an array of zeros
for kk, state in enumerate(output.states):
    rho_cavity = state.ptrace(0)
    entropy[kk] = qt.entropy_vn(rho_cavity)
```

```python
plt.figure(figsize=(8,6))
plt.plot(tlist,entropy,lw=3)
plt.xlabel('Time', fontsize=16)
plt.ylabel('Entropy', fontsize=16);
```

```python
rho_atom = output.states[np.where(tlist==2.5)[0][0]].ptrace(1)
```

```python
rho_atom
```

Quantum object: dims = [[2], [2]], shape = (2, 2), type = oper, isherm = True\begin{equation*}\left(\begin{array}{*{11}c}0.500 & 0.0\\0.0 & 0.500\\\end{array}\right)\end{equation*}

Verify that the entropy of $\rho_{c}$ is the same as $\rho_{a}$.

```python
np.all([(qt.entropy_vn(state.ptrace(0)) == qt.entropy_vn(state.ptrace(1)))
        for state in output.states])
```

    True

At times $t=2.5, 7.5, \dots$, atom is in maximally mixed state, and thus entangled with the cavity. This should not be too surprising as the single-exciation in the system is begin eqully shared by the cavity and atom at those times.

## State of the cavity

```python
t_idx = np.where([tlist == t for t in [0.0, 5, 15, 25]])[1]
t_idx
```

    array([  0,  20,  60, 100])

```python
cavity_rho_list = [output.states[kk].ptrace(0) for kk in t_idx]
```

```python
# loop over the list of density matrices

xvec = np.linspace(-3,3,200)

fig, axes = plt.subplots(1,len(cavity_rho_list), sharex=True,
                         figsize=(4*len(cavity_rho_list),3))

for idx, rho in enumerate(cavity_rho_list):
    # calculate its wigner function
    W = qt.wigner(rho, xvec, xvec)
    # plot its wigner function
    cf = axes[idx].contourf(xvec, xvec, W, 100, vmin=-0.4, vmax=0.4, cmap='RdBu')
    axes[idx].set_title(r"$t = %.1f$" % tlist[t_idx][idx], fontsize=16)
cbar = fig.colorbar(cf, ax=axes.ravel().tolist())
```

```python
W.sum()*np.diff(xvec)[0]**2
```

    0.99919386620323025

The Wigner distribution of the cavity at times $t=5, 15, 25,\dots$ looks very similar to the $|n\rangle = |1\rangle$ Fock state that we saw earlier.

```python
W_fock1 = qt.wigner(qt.basis(N,1), xvec, xvec)
plt.figure(figsize=(7,6))
plt.contourf(xvec, xvec, W_fock1, 100, vmin=-0.4, vmax=0.4, cmap='RdBu')
plt.colorbar();
```

If we want to make this observation quantitative we can use one of several metrics (distance measures) to see how close our observed state is to a given target state:

#### Fidelity
$$
\begin{equation}
F(\rho, \sigma) = \mathrm{Tr}\left[\sqrt{\sqrt{\rho}\sigma\sqrt{\rho}}\right]
\end{equation}
$$

#### Trace Distance

$$
\begin{equation}
T(\rho, \sigma) = \frac{1}{2}\mathrm{Tr}\left[\sqrt{(\rho-\sigma)^{2}}\right] = \frac{1}{2}\sum_{i} |\lambda_{i}|
\end{equation}
$$

```python
qt.fidelity(qt.basis(N,1),cavity_rho_list[-1])
```

    0.9999999973704953

```python
qt.tracedist(qt.basis(N,1),cavity_rho_list[-1])
```

    0.0

```python

```


# Quantization of Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Oversampling

[Oversampling](https://en.wikipedia.org/wiki/Oversampling) is a technique which is applied in [analog-to-digital converters](https://en.wikipedia.org/wiki/Analog-to-digital_converter) to lower the average power of the quantization error. It requires a joint consideration of sampling and quantization.

### Ideal Analog-to-Digital Conversion

Let's consider the ideal sampling of a signal followed by its quantization, as given by the following block diagram

Ideal sampling is modeled by multiplying the continuous signal $x(t)$ with a series of equidistant Dirac impulse, resulting in the discrete signal $x[k] = x(k T)$ where $T$ denotes the sampling interval. The discrete signal $x[k]$ is then quantized. The output of the ideal analog-to-digital converter is the quantized discrete signal $x_\text{Q}[k]$.

### Nyquist Sampling

Sampling of the continuous signal $x(t)$ leads to repetitions of the spectrum $X(j \omega) = \mathcal{F} \{ x(t) \}$ at multiples of $\omega_\text{S} = \frac{2 \pi}{T}$. We limit ourselves to a continuous real-valued $x(t) \in \mathbb{R}$, band-limited signal $| X(j \omega) | = 0$ for $|\omega| > \omega_\text{C}$ where $\omega_\text{C}$ denotes its cut-off frequency. The spectral repetitions due to sampling do not overlap if the [sampling theorem](https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem) $\omega_\text{S} \geq  2 \cdot \omega_\text{C}$ is fulfilled. In the case of Nyquist (critical) sampling, the sampling frequency is chosen as $\omega_\text{S} =  2 \cdot \omega_\text{C}$.

### Oversampling

The basic idea of oversampling is to sample the input signal at frequencies which are significantly higher than the Nyquist criterion dictates. After quantization, the signal is low-pass filtered by a discrete filter $H_\text{LP}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and resampled back to the Nyquist rate. In order to avoid aliasing due to the resampling this filter has to be chosen as an ideal low-pass

\begin{equation}
H_\text{LP}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \text{rect} \left( \frac{\Omega}{2 \, \Omega_\text{C}} \right)
\end{equation}

where $\Omega_\text{C} = \omega_\text{C} \cdot T$. For an oversampling of factor $L \in \mathbb{N}$ we have $\omega_\text{S} = L \cdot 2 \omega_\text{C}$. For this case, the resampling can be realized by keeping only every $L$-th sample which is known as decimation. The following block diagram illustrates the building blocks of oversampled digital-to-analog conversion, $\downarrow L$ denotes decimation by a factor of $L$

In order the conclude on the benefits of oversampling we have to derive the average power of the overall quantization error. According to our [model of the quantization error](linear_uniform_quantization_error.ipynb#Model-for-the-Quantization-Error), the quantization error $e[k]$ can be modeled as uniformly distributed white noise. Its power spectral density (PSD) is given as

\begin{equation}
\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{Q^2}{12}
\end{equation}

where $Q$ denotes the quantization step. Before the discrete low-pass filter $H_\text{LP}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, the power of the quantization error is uniformly distributed over the entire frequency range $-\pi < \Omega \leq \pi$. However, after the ideal low-pass filter its frequency range is limited to $- \frac{\pi}{L} < \Omega \leq \frac{\pi}{L}$. The average power of the quantization error is then given as

\begin{equation}
\sigma_{e, \text{LP}}^2 = \frac{1}{2 \pi} \int\limits_{- \frac{\pi}{L}}^{\frac{\pi}{L}} \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})  \; \mathrm{d}\Omega = \frac{1}{L} \cdot \frac{Q^2}{12}
\end{equation}

The average power $\sigma_x^2$ of the sampled signal $x[k]$ is not affected, since the cutoff frequency of the low-pass filter has been chosen as the upper frequency limit $\omega_\text{C}$ of the input signal $x(t)$.

In order to calculate the SNR of the oversampled analog-to-digital converter we assume that the input signal is drawn from a wide-sense stationary (WSS) uniformly distributed zero-mean random process with $x_\text{min} \leq x[k] < x_\text{max}$. With the results from our discussion of [linear uniform quantization](linear_uniform_quantization_error.ipynb#Uniformly-Distributed-Signal) and $\sigma_{e, \text{LP}}^2$ from above we get

\begin{equation}
SNR = 10 \cdot \log_{10} \left( 2^{2 w} \right) + 10 \cdot \log_{10} \left( L \right) \approx 6.02 \, w + 10 \cdot \log_{10} \left( L \right) \quad \text{in dB}
\end{equation}

where $w$ denotes the number of bits used for a binary representation of the quantization index. Hence, oversampling by a factor of $L$ brings a plus of $10 \cdot \log_{10} \left( L \right)$ dB in terms of SNR. For instance, an oversampling by a factor of $L = 4$ results in a SNR which is approximately 6 dB higher. For equal SNR the quantization step $Q$ can be chosen larger. In terms of the wordlength of a quantizer this accounts to a reduction by one bit. Consequently, there is a trade-off between accuracy of the quantizer and its sampling frequency.

### Example

The following numerical simulation illustrates the benefit in terms of SNR for an oversampled linear uniform quantizer with $w = 16$ for the quantization of the harmonic signal $x[k] = \cos[\Omega_0 k]$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

w = 16  # wordlength of the quantized signal
L = 2**np.arange(1,10)  # oversampling factors

N = 8192  # length of signal
Om0 = 100*2*np.pi/N  # frequency of harmonic signal
Q = 1/(2**(w-1))  # quantization step

def uniform_midtread_quantizer(x, Q):
    # limiter
    x = np.copy(x)
    idx = np.where(x <= -1)
    x[idx] = -1
    idx = np.where(x > 1 - Q)
    x[idx] = 1 - Q
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

def SNR_oversampled_ADC(L):
    x = (1-Q)*np.cos(Om0*np.arange(N))
    xu = (1-Q)*np.cos(Om0*np.arange(N*L)/L)
    # quantize signal
    xQu = uniform_midtread_quantizer(xu, Q)
    # low-pass filtering and decimation
    xQ = sig.resample(xQu, N)
    # estimate SNR
    e = xQ - x

    return 10*np.log10((np.var(x)/np.var(e)))

# compute SNR for oversampled ADC
SNR = [SNR_oversampled_ADC(l) for l in L]

# plot result
plt.figure(figsize=(10, 4))
plt.semilogx(L, SNR, label='SNR with oversampling')
plt.plot(L, (6.02*w+1.76)*np.ones(L.shape), label='SNR w/o oversampling' )
plt.xlabel(r'oversampling factor $L$')
plt.ylabel(r'SNR in dB')
plt.legend(loc='upper left')
plt.grid()
```

**Exercise**

* What SNR can be achieved for an oversampling factor of $L=16$?
* By how many bits could the word length $w$ be reduced in order to gain the same SNR as without oversampling?

Solution: The SNR for the quantization of a uniformly distributed input signal without oversampling is $\text{SNR} \approx 6.02 w \approx 96$ dB and with 16 times oversampling $\text{SNR}_{L} \approx 6.02 w + 10 \cdot \log_{10} (16)  \approx 96 + 12$ dB. Since the [quantization of a harmonic signal](linear_uniform_quantization_error.ipynb#Harmonic-Signal) is considered an offset of $1.76$ dB has to added to both. The wordlength could be reduced by 2 bits according to these numbers.

### Anti-Aliasing Filter

Besides an increased SNR, oversampling has also another benefit. In order to ensure that the input signal $x(t)$ is band-limited before sampling, a low-pass filter $H_\text{LP}(\mathrm{j}\,\omega)$ is applied in typical analog-to-digital converters. This is illustrated in the following

The filter $H_\text{LP}(\mathrm{j}\,\omega)$ is also known as [anti-aliasing filter](https://en.wikipedia.org/wiki/Anti-aliasing_filter). The ideal low-pass filter is given as $H_\text{LP}(\mathrm{j}\,\omega) = \text{rect}\left( \frac{\omega}{\omega_\text{S}}  \right)$. The ideal $H_\text{LP}(\mathrm{j}\,\omega)$ can only be approximated in the analog domain. Since the sampling rate is higher than the Nyquist rate, there is no need for a steep slope of the filter in order to avoid aliasing. However, the pass-band of the filter within $|\omega| < |\omega_\text{C}|$ has to be flat.

Before decimation, the discrete filter $H_\text{LP}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ has to remove the spectral contributions that may lead to aliasing. However, a discrete filter $H_\text{LP}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ with steep slope can be realized much easier than in the analog domain.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


```python
%matplotlib inline
from IPython.display import display, Math
from sympy import *
init_session()
```

    IPython console for SymPy 1.8 (Python 3.7.10-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.8/

```python
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython import display

import random
import numpy as npc
```

```python
# バブルソート
n = 10
a = [3,1,4,1,5,9,2,6,5,3] # 配列の定義
for i in range(n): # range(t) で 0 から t-1 まで
    for k in range(1,n-i): # range(s,t) で s から t-1 まで
        if a[n-k-1] > a[n-k]: # 左の方が大きければ交換
            a[n-k-1],a[n-k]=a[n-k],a[n-k-1]
            print(a)
```

    [3, 1, 4, 1, 5, 9, 2, 6, 3, 5]
    [3, 1, 4, 1, 5, 9, 2, 3, 6, 5]
    [3, 1, 4, 1, 5, 2, 9, 3, 6, 5]
    [3, 1, 4, 1, 2, 5, 9, 3, 6, 5]
    [3, 1, 1, 4, 2, 5, 9, 3, 6, 5]
    [1, 3, 1, 4, 2, 5, 9, 3, 6, 5]
    [1, 3, 1, 4, 2, 5, 9, 3, 5, 6]
    [1, 3, 1, 4, 2, 5, 3, 9, 5, 6]
    [1, 3, 1, 4, 2, 3, 5, 9, 5, 6]
    [1, 3, 1, 2, 4, 3, 5, 9, 5, 6]
    [1, 1, 3, 2, 4, 3, 5, 9, 5, 6]
    [1, 1, 3, 2, 4, 3, 5, 5, 9, 6]
    [1, 1, 3, 2, 3, 4, 5, 5, 9, 6]
    [1, 1, 2, 3, 3, 4, 5, 5, 9, 6]
    [1, 1, 2, 3, 3, 4, 5, 5, 6, 9]

```python
N = 20
data = [random.randint(0,100) for i in range(N)]

# First set up the figure, the axes, and the plot element
fig, ax = plt.subplots()
plt.close()

#plt.title("The Unsorted data")
#plt.bar(np.arange(len(data)), data)
#plt.show()
#fig = plt.figure()

datalist = [data]
def bubble_sort(data):
    N = len(data)
    done = False
    while not done:
        swapped = False
        for i in range(N-1):
            if data[N-i-2] > data[N-i-1]:
                data[N-i-2], data[N-i-1] = data[N-i-1], data[N-i-2]
                swapped = True
                datalist.append(data.copy())
        if not swapped:
            done = True
    return data,datalist

data,datalist = bubble_sort(data)
print(len(datalist))

def init():
    return plt.bar(np.arange(len(data)), data)

def update(i,datalist):
    mycolor='blue'
    #print(datalist[i])
    if i != 0:
        ax.cla()
    if i == len(datalist) -1:
        mycolor='orange'
    return (ax.bar(np.arange(len(datalist[i])), datalist[i],color=mycolor),)

ani = animation.FuncAnimation(fig, update, init_func=init, fargs = [datalist],
                              interval=150, repeat=False,frames=len(datalist))

```

    98

```python
# アニメーションによる表示
from IPython.display import HTML
from matplotlib import animation, rc

from IPython.display import HTML

ax.cla()
rc('animation', html='jshtml')
HTML(ani.to_html5_video())
```

```python
# クイックソート
N = 100
data = [random.randint(0,100) for i in range(N)]
plt.title("The Unsorted data")
plt.bar(np.arange(len(data)), data)
plt.show()

# First set up the figure, the axes, and the plot element
fig, ax = plt.subplots()
plt.close()

#fig = plt.figure()

datalist = [data]
def quick_sort(data,i,j,datalist=[],swap=[0]):
    if j-i<=1:
        if data[i] > data[j]:
            data[i],data[j]=data[j],data[i]
            swap[0]=swap[0]+1
            datalist.append(data.copy())
        return
    f = 0
    c = data[i]
    for k in range(i+1,j+1):
        if c != data[k]:
            f = 1
            c = (c+data[k])/2
            break
    if f==0:
        return
    else:
        #print(data)
        #print (i,j,c)
        k1 = i
        k2 = j
        while(k1<=j and k2>=i and k1<k2):
            while(k1<=k2 and data[k1]<c):
                k1=k1+1
            while(k2>=k1 and data[k2]>=c):
                k2=k2-1
            if (k1<k2):
                data[k1],data[k2]=data[k2],data[k1]
                swap[0]=swap[0]+1
                datalist.append(data.copy())
        if data[k1]<c:
            i2=k1
        else:
            i2=k1-1
        quick_sort(data,i,i2,datalist=datalist,swap=swap)
        quick_sort(data,i2+1,j,datalist=datalist,swap=swap)
swap = [0]
datalist = []
quick_sort(data,0,N-1,datalist,swap)
print(swap[0],len(datalist))
def update(i,datalist):
    mycolor='blue'
    if i != 0:
        ax.cla()
    if i == len(datalist) -1:
        mycolor='orange'
    return (ax.bar(np.arange(len(datalist[i])), datalist[i],color=mycolor),)

ani = animation.FuncAnimation(fig, update, fargs = ([datalist]),interval=10**4/swap[0],
                              repeat=False,frames=len(datalist))
#ani.save("quicksort.mp4")
#plt.title("All sorted")
#plt.bar(np.arange(len(data)), data)
#plt.show()
```

```python
# アニメーションによる表示
from IPython.display import HTML
from matplotlib import animation, rc

from IPython.display import HTML

ax.cla()
rc('animation', html='jshtml')
HTML(ani.to_html5_video())
```

```python

```

```python

```


# Krylov Methods and Matrix Conditioning

## Contents

- [Krylov Methods and Matrix Conditioning](#Krylov-Methods-and-Matrix-Conditioning)
  - [Overview](#Overview)
  - [Ill-Conditioned Matrices](#Ill-Conditioned-Matrices)
  - [Stationary Iterative Algorithms for Linear Systems](#Stationary-Iterative-Algorithms-for-Linear-Systems)
  - [Krylov Methods](#Krylov-Methods)
  - [Iterative Methods for Linear Least Squares](#Iterative-Methods-for-Linear-Least-Squares)
  - [Iterative Methods for Eigensystems](#Iterative-Methods-for-Eigensystems)
  - [Krylov Methods for Markov-Chain Dynamics](#Krylov-Methods-for-Markov-Chain-Dynamics)

## Overview

This lecture takes the structure of [numerical methods for linear algebra](numerical_linear_algebra.html) and builds further
toward working with large, sparse matrices.  In the process, we will examine foundational numerical analysis such as
ill-conditioned matrices.

### Setup

```julia
using InstantiateFromURL
# optionally add arguments to force installation: instantiate = true, precompile = true
github_project("QuantEcon/quantecon-notebooks-julia", version = "0.8.0")
```

```julia
using LinearAlgebra, Statistics, BenchmarkTools, Random
Random.seed!(42);  # seed random numbers for reproducibility
```

### Applications

In this section, we will consider variations on classic problems

1. Solving a linear system for a square $ A $ where we will maintain throughout that there is a unique solution to
  $$
  A x = b
  $$
1. [Linear least-squares](https://en.wikipedia.org/wiki/Linear_least_squares) solution, for a rectangular $ A $
  $$
  \min_x \| Ax -b \|^2
  $$
  From theory, we know that if $ A $ has linearly independent columns, then the solution is the [normal equation](https://en.wikipedia.org/wiki/Linear_least_squares#Derivation_of_the_normal_equations)
  $$
  x = (A'A)^{-1}A'b
  $$
1. In the case of a square matrix $ A $, the  eigenvalue problem is that of finding $ x $ and $ \lambda $ such that
  $$
  A x = \lambda x
  $$
  For eigenvalue problems, keep in mind that you do not always require all of the $ \lambda $, and sometimes the largest (or smallest) would be enough.  For example, calculating the spectral radius requires only the eigenvalue with maximum absolute value.

## Ill-Conditioned Matrices

An important consideration in numerical linear algebra, and iterative methods in general, is the [condition number](https://en.wikipedia.org/wiki/Condition_number#Matrices).

An ill-conditioned matrix is one where the basis eigenvectors are close to, but not exactly, collinear.  While this poses no problem on pen and paper,
or with infinite-precision numerical methods, it is important in practice, for two reasons:

1. Ill-conditioned matrices introduce numerical errors roughly in proportion to the base-10 log of the condition number.
1. The convergence speed of many iterative methods is based on the spectral properties of the matrices (e.g., the basis formed by the eigenvectors), and hence ill-conditioned systems can converge slowly.

The solutions to these problems are to

- be careful with operations which introduce error based on the condition number (e.g., matrix inversions when the condition number is high)
- choose, where possible, alternative representations which have less collinearity (e.g., an orthogonal polynomial basis rather than a monomial one)
- use a preconditioner for iterative methods, which changes the spectral properties to increase convergence speed

### Condition Number

First, let’s define and explore the condition number $ \kappa $

$$
\kappa(A) \equiv \|A\| \|A^{-1}\|
$$

where you can use the Cauchy–Schwarz inequality to show that $ \kappa(A) \geq 1 $.  While the condition number can be calculated with any norm, we will focus on the 2-norm.

First, a warning on calculations: Calculating the condition number for a matrix can be an expensive operation (as would calculating a determinant)
and should be thought of as roughly equivalent to doing an eigendecomposition.  So use it for detective work judiciously.

Let’s look at the condition number of a few matrices using the `cond` function (which allows a choice of the norm, but we’ll stick with the default 2-norm).

```julia
A = I(2)
cond(A)
```

    1.0

Here we see an example of the best-conditioned matrix, the identity matrix with its completely orthonormal basis, which has a condition number of 1.

On the other hand, notice that

```julia
ϵ = 1E-6
A = [1.0 0.0
     1.0 ϵ]
cond(A)
```

    2.0000000000005004e6

has a condition number of order `10E6` - and hence (taking the base-10 log) you would expect to be introducing numerical errors of about 6 significant digits if you
are not careful.  For example, note that the inverse has both extremely large and extremely small negative numbers

```julia
inv(A)
```

    2×2 Array{Float64,2}:
      1.0    0.0
     -1.0e6  1.0e6

Since we know that the determinant of nearly collinear matrices is close to zero, this shows another symptom of poor conditioning

```julia
det(A)
```

    1.0e-6

However, be careful since the determinant has a scale, while the condition number is dimensionless.  That is,

```julia
@show det(1000 * A)
@show cond(1000 * A);
```

    det(1000A) = 1.0
    cond(1000A) = 2.0000000000005001e6

In that case, the determinant of `A` is 1, while the condition number is unchanged.  This example also provides some
intuition that ill-conditioned matrices typically occur when a matrix has radically different scales (e.g., contains both `1` and `1E-6`, or `1000` and `1E-3`).  This can occur frequently with both function approximation and linear least squares.

### Condition Numbers and Matrix Operations

Multiplying a matrix by a constant does not change the condition number.  What about other operations?

For this example, we see that the inverse has the same condition number (though this will not always be the case).

```julia
@show cond(A)
@show cond(inv(A));
```

    cond(A) = 2.0000000000005004e6
    cond(inv(A)) = 2.0000000002463197e6

The condition number of the product of two matrices can change radically and lead things to becoming
even more ill-conditioned.

This comes up frequently when calculating the product of a matrix and its transpose (e.g., forming the covariance matrix).  A classic example is the [Läuchli matrix](https://link.springer.com/article/10.1007%2FBF01386022).

```julia
lauchli(N, ϵ) = [ones(N)'; ϵ * I(N)]'
ϵ = 1E-8
L = lauchli(3, ϵ) |> Matrix
```

    3×4 Array{Float64,2}:
     1.0  1.0e-8  0.0     0.0
     1.0  0.0     1.0e-8  0.0
     1.0  0.0     0.0     1.0e-8

Note that the condition number increases substantially

```julia
@show cond(L)
@show cond(L' * L);
```

    cond(L) = 1.732050807568878e8
    cond(L' * L) = 5.345191558726545e32

You can show that the analytic eigenvalues of this are $ \{3 + \epsilon^2, \epsilon^2, \epsilon^2\} $ but the poor conditioning
means it is difficult to distinguish these from $ 0 $.

This comes up when conducting [Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis#Singular_value_decomposition), which
requires calculations of the eigenvalues of the covariance matrix

```julia
sort(sqrt.(Complex.(eigen(L*L').values)), lt = (x,y) -> abs(x) < abs(y))
```

    3-element Array{Complex{Float64},1}:
                       0.0 + 4.870456104375987e-9im
     4.2146848510894035e-8 + 0.0im
        1.7320508075688772 + 0.0im

Note that these are significantly different than the known analytic solution and, in particular, are difficult to distinguish from 0.

```julia
sqrt.([3 + ϵ^2, ϵ^2, ϵ^2]) |> sort
```

    3-element Array{Float64,1}:
     1.0e-8
     1.0e-8
     1.7320508075688772

Alternatively, we could calculate these by taking the square of the singular values of $ L $ itself, which is much more accurate
and lets us clearly distinguish from zero

```julia
svd(L).S  |> sort
```

    3-element Array{Float64,1}:
     9.999999999999997e-9
     1.0e-8
     1.7320508075688774

Similarly, we are better off calculating least squares directly rather than forming the normal equation (i.e., $ A' A x = A' b $) ourselves

```julia
N = 3
A = lauchli(N, 1E-7)' |> Matrix
b = rand(N+1)
x_sol_1 = A \ b  # using a least-squares solver
x_sol_2 = (A' * A) \ (A' * b)  # forming the normal equation ourselves
norm(x_sol_1 - x_sol_2)
```

    2502.05373776057

### Why a Monomial Basis Is a Bad Idea

A classic example of poorly conditioned matrices is using a monomial basis of a polynomial with interpolation.

Take a grid of points, $ x_0, \ldots x_N $ and values $ y_0, \ldots y_N $ where we want to calculate the
interpolating polynomial.

If we were to use the simplest, and most obvious, polynomial basis, then the calculation consists of finding the coefficients $ c_1, \ldots c_n $ where

$$
P(x) = \sum_{i=0}^N c_i x^i
$$

To solve for the coefficients, we notice that this is a simple system of equations

$$
\begin{array}
    \,y_0 = c_0 + c_1 x_0 + \ldots c_N x_0^N\\
    \,\ldots\\
    \,y_N = c_0 + c_1 x_N + \ldots c_N x_N^N
\end{array}
$$

Or, stacking $ c = \begin{bmatrix} c_0 & \ldots & c_N\end{bmatrix}, y = \begin{bmatrix} y_0 & \ldots & y_N\end{bmatrix} $ and

$$
A = \begin{bmatrix} 1 & x_0 & x_0^2 & \ldots &x_0^N\\
                    \vdots & \vdots & \vdots & \vdots & \vdots \\
                    1 & x_N & x_N^2 & \ldots & x_N^N
    \end{bmatrix}
$$

We can then calculate the interpolating coefficients as the solution to

$$
A c = y
$$

Implementing this for the interpolation of the $ exp(x) $ function

```julia
N = 5
f(x) = exp(x)
x = range(0.0, 10.0, length = N+1)
y = f.(x)  # generate some data to interpolate

A = [x_i^n for x_i in x, n in 0:N]
A_inv = inv(A)
c = A_inv * y
norm(A * c - f.(x), Inf)
```

    1.356966095045209e-9

The final step just checks the interpolation vs. the analytic function at the nodes.  Keep in mind that this should be very close to zero
since we are interpolating the function precisely at those nodes.
In our example, the Inf-norm (i.e., maximum difference) of the interpolation errors at the nodes is around `1E-9`, which
is reasonable for many problems.

But note that with $ N=5 $ the condition number is already of order `1E6`.

```julia
cond(A)
```

    564652.3214053963

What if we increase the degree of the polynomial with the hope of increasing the precision of the
interpolation?

```julia
N = 10
f(x) = exp(x)
x = range(0.0, 10.0, length = N+1)
y = f.(x)  # generate some data to interpolate

A = [x_i^n for x_i in x, n in 0:N]
A_inv = inv(A)
c = A_inv * y
norm(A * c - f.(x), Inf)
```

    8.61171429278329e-7

Here, we see that hoping to increase the precision between points by adding extra polynomial terms is backfiring.  By going to a 10th-order polynomial, we have
introduced an error of about `1E-5`, even at the interpolation points themselves.

This blows up quickly

```julia
N = 20
f(x) = exp(x)
x = range(0.0, 10.0, length = N+1)
y = f.(x)  # generate some data to interpolate

A = [x_i^n for x_i in x, n in 0:N]
A_inv = inv(A)
c = A_inv * y
norm(A * c - f.(x), Inf)
```

    19978.410967681375

To see the source of the problem, note that the condition number is astronomical.

```julia
cond(A)
```

    2.0386741019186427e24

At this point, you should be suspicious of the use of `inv(A)`, since we have considered solving
linear systems by taking the inverse as verboten.  Indeed, this made things much worse.  The
error drops dramatically if we solve it as a linear system

```julia
c = A \ y
norm(A * c - f.(x), Inf)
```

    1.864464138634503e-10

But an error of `1E-10` at the interpolating nodes themselves can be a problem in many applications, and if you increase `N`
then the error will become non-trivial eventually - even without taking the inverse.

The heart of the issue is that the monomial basis leads to a [Vandermonde matrix](https://en.wikipedia.org/wiki/Vandermonde_matrix), which
is especially ill-conditioned.

#### Aside on Runge’s Phenomenon

The monomial basis is also a good opportunity to look at a separate type of error due to [Runge’s Phenomenon](https://en.wikipedia.org/wiki/Runge%27s_phenomenon).    It is an important
issue in approximation theory, albeit not one driven by numerical approximation errors.

It turns out that using a uniform grid of points is, in general, the worst possible choice of interpolation nodes for a polynomial approximation.  This phenomenon can be seen with the interpolation of the seemingly innocuous Runge’s function, $ g(x) = \frac{1}{1 + 25 x^2} $.

Let’s calculate the interpolation with a monomial basis to find the $ c_i $ such that

$$
\frac{1}{1 + 25 x^2} \approx \sum_{i=0}^N c_i x^i,\, \text{ for } -1 \leq x \leq 1
$$

First, interpolate with $ N = 5 $ and avoid taking the inverse.  In that case, as long as we avoid taking an inverse, the numerical errors from the ill-conditioned matrix are manageable.

```julia
using Plots
N_display = 100
g(x) = 1/(1 + 25x^2)
x_display = range(-1, 1, length = N_display)
y_display = g.(x_display)

# interpolation
N = 5
x = range(-1.0, 1.0, length = N+1)
y = g.(x)
A_5 = [x_i^n for x_i in x, n in 0:N]
c_5 = A_5 \ y

# use the coefficients to evaluate on x_display grid
B_5 = [x_i^n for x_i in x_display, n in 0:N]   # calculate monomials for display grid
y_5 = B_5 * c_5  # calculates for each in x_display_grid
plot(x_display, y_5, label = "P_5(x)")
plot!(x_display, y_display, w = 3, label = "g(x)")
```

Note that while the function, $ g(x) $, and the approximation with a 5th-order polynomial, $ P_5(x) $, coincide at the 6 nodes, the
approximation has a great deal of error everywhere else.

The oscillations near the boundaries are the hallmarks of Runge’s Phenomenon.  You might guess that increasing the number
of grid points and the order of the polynomial will lead to better approximations:

```julia
N = 9
x = range(-1.0, 1.0, length = N+1)
y = g.(x)
A_9 = [x_i^n for x_i in x, n in 0:N]
c_9 = A_9 \ y

# use the coefficients to evaluate on x_display grid
B_9 = [x_i^n for x_i in x_display, n in 0:N]   # calculate monomials for display grid
y_9 = B_9 * c_9  # calculates for each in x_display_grid
plot(x_display, y_9, label = "P_9(x)")
plot!(x_display, y_display, w = 3, label = "g(x)")
```

While the approximation is better near `x=0`, the oscillations near the boundaries have become worse.  Adding on extra polynomial terms will not
globally increase the quality of the approximation.

#### Using an Orthogonal Polynomial Basis

We can minimize the numerical problems of an ill-conditioned basis matrix by choosing a different basis for the polynomials.

For example, [Chebyshev polynomials](https://en.wikipedia.org/wiki/Chebyshev_polynomials) form an orthonormal basis under an appropriate inner product, and we can form precise high-order approximations, with very little numerical error

```julia
using ApproxFun
N = 10000
S = Chebyshev(-1.0..1.0)  # form Chebyshev basis
x = points(S, N)  # chooses Chebyshev nodes
y = g.(x)
g_approx = Fun(S,ApproxFun.transform(S,y))  # transform fits the polynomial
@show norm(g_approx.(x) - g.(x), Inf)
plot(x_display, g_approx.(x_display), label = "P_10000(x)")
plot!(x_display, g.(x_display), w = 3, label = "g(x)")
```

    norm(g_approx.(x) - g.(x), Inf) = 4.440892098500626e-16

Besides the use of a different polynomial basis, we are approximating at different nodes (i.e., [Chebyshev nodes](https://en.wikipedia.org/wiki/Chebyshev_nodes)).  Interpolation with Chebyshev polynomials at the Chebyshev nodes ends up minimizing (but not eliminating) Runge’s Phenomenon.

#### Lessons for Approximation and Interpolation

To summarize:

1. Check the condition number on systems you suspect might be ill-conditioned (based on intuition of collinearity).
1. If you are working with ill-conditioned matrices, be especially careful not to take the inverse or multiply by the transpose.
1. Avoid a monomial polynomial basis.  Instead, use polynomials (e.g., Chebyshev or Lagrange) orthogonal under an appropriate inner product, or use a non-global basis such as cubic splines.
1. If possible, avoid using a uniform grid for interpolation and approximation, and choose nodes appropriate for the basis.

However, sometimes you can’t avoid ill-conditioned matrices. This is especially common with discretization of PDEs and with linear least squares.

## Stationary Iterative Algorithms for Linear Systems

As before, consider solving the equation

$$
A x = b
$$

We will now
focus on cases where $ A $ is both massive (e.g., potentially millions of equations) and sparse, and sometimes ill-conditioned - but where there is always a unique solution.

While this may seem excessive, it occurs in practice due to the curse of dimensionality, discretizations
of PDEs, and when working with big data.

The methods in the previous lectures (e.g., factorization and approaches similar to Gaussian elimination) are called direct methods, and are able
in theory to converge to the exact solution in a finite number of steps while directly working with the matrix in memory.

Instead, iterative solutions start with a guess on a solution and iterate until convergence.  The benefit will be that
each iteration uses a lower-order operation (e.g., an $ O(N^2) $ matrix-vector product) which will make it possible to

1. solve much larger systems, even if done less precisely.
1. define linear operators in terms of the matrix-vector products, rather than storing as a matrix.
1. get approximate solutions in progress prior to the completion of all algorithm steps, unlike the direct methods, which provide a solution only at the end.

Of course, there is no free lunch, and the computational order of the iterations themselves would be comparable to the direct methods for a given level of tolerance (e.g., $ O(N^3) $ operations may be required to solve a dense unstructured system).

There are two types of iterative methods we will consider.  The first type is stationary methods, which iterate on a map in a way that’s similar to fixed-point problems, and the second type is [Krylov](https://en.wikipedia.org/wiki/Krylov_subspace) methods, which iteratively solve using left-multiplications of the linear operator.

For our main examples, we will use the valuation of the continuous-time Markov chain from the [numerical methods for linear algebra](numerical_linear_algebra.html) lecture.  That is, given a payoff vector $ r $, a
discount rate $ \rho $, and the infinitesimal generator of the Markov chain $ Q $, solve the equation

$$
\rho v = r + Q v
$$

With the sizes and types of matrices here, iterative methods are inappropriate in practice, but they will help us understand
the characteristics of convergence and how they relate to matrix conditioning.

### Stationary Methods

First, we will solve with a direct method, which will give the solution to machine precision.

```julia
using LinearAlgebra, IterativeSolvers, Statistics
α = 0.1
N = 100
Q = Tridiagonal(fill(α, N-1), [-α; fill(-2α, N-2); -α], fill(α, N-1))

r = range(0.0, 10.0, length=N)
ρ = 0.05

A = ρ * I - Q
v_direct = A \ r
mean(v_direct)
```

    100.00000000000004

Without proof, consider that given the discount rate of $ \rho > 0 $, this problem could be set up as a contraction for solving the Bellman
equation through methods such as value-function iteration.

The condition we will examine here is called [**diagonal dominance**](https://en.wikipedia.org/wiki/Diagonally_dominant_matrix).

$$
|A_{ii}| \geq \sum_{j\neq i} |A_{ij}| \quad\text{for all } i = 1\ldots N
$$

That is, in every row, the diagonal element is weakly greater in absolute value than the sum of all of the other elements in the row.  In cases
where it is strictly greater, we say that the matrix is strictly diagonally dominant.

With our example, given that $ Q $ is the infinitesimal generator of a Markov chain, we know that each row sums to 0, and hence
it is weakly diagonally dominant.

However, notice that when $ \rho > 0 $, and since the diagonal of $ Q $ is negative,  $ A = ρ I - Q $ makes the matrix strictly diagonally dominant.

### Jacobi Iteration

For matrices that are **strictly diagonally dominant**, you can prove that a simple decomposition and iteration procedure
will converge.

To solve a system $ A x = b $, split the matrix $ A $ into its diagonal and off-diagonal elements.  That is,

$$
A = D + R
$$

where

$$
D = \begin{bmatrix} A_{11} & 0 & \ldots & 0\\
                    0    & A_{22} & \ldots & 0\\
                    \vdots & \vdots & \vdots & \vdots\\
                    0 & 0 &  \ldots & A_{NN}
    \end{bmatrix}
$$

and

$$
R = \begin{bmatrix} 0 & A_{12}  & \ldots & A_{1N} \\
                    A_{21}    & 0 & \ldots & A_{2N} \\
                    \vdots & \vdots & \vdots & \vdots\\
                    A_{N1}  & A_{N2}  &  \ldots & 0
    \end{bmatrix}
$$

Rearrange the $ (D + R)x = b $ as

$$
\begin{align}
D x &= b - R x\\
x &= D^{-1} (b - R x)
\end{align}
$$

where, since $ D $ is diagonal, its inverse is trivial to calculate with $ O(N) $ complexity.

To solve, take an iteration $ x^k $, starting from $ x^0 $, and then form a new guess with

$$
x^{k+1} = D^{-1}(b - R x^k)
$$

The complexity here is $ O(N^2) $ for the matrix-vector product, and $ O(N) $ for the vector subtraction and division.

The package [IterativeSolvers.jl](https://github.com/JuliaMath/IterativeSolvers.jl) package implements this method.

For our example, we start with a guess and solve for the value function and iterate

```julia
using IterativeSolvers, LinearAlgebra, SparseArrays
v = zeros(N)
jacobi!(v, A, r, maxiter = 40)
@show norm(v - v_direct, Inf)
```

    norm(v - v_direct, Inf) = 0.022858373200932647

    0.022858373200932647

With this, after 40 iterations we see that the error is in the order of `1E-2`

### Other Stationary Methods

In practice, there are many methods that are better than Jacobi iteration. For example [Gauss-Siedel](https://en.wikipedia.org/wiki/Gauss%E2%80%93Seidel_method)., which
splits the matrix $ A = L + U $ into a lower-triangular matrix $ L $ and an upper-triangular matrix $ U $ without the diagonal.

The iteration becomes

$$
L x^{k+1} = b - U x^k
$$

In that case, since the $ L $ matrix is triangular, the system can be solved in $ O(N^2) $ operations after $ b - U x^k $ is formed

```julia
v = zeros(N)
gauss_seidel!(v, A, r, maxiter = 40)
@show norm(v - v_direct, Inf);
```

    norm(v - v_direct, Inf) = 1.5616376089155892e-5

The accuracy increases substantially. After 40 iterations, we see that the error is of the order of `1E-5`

Another example is [Successive Over-relaxation (SOR)](https://en.wikipedia.org/wiki/Successive_over-relaxation), which takes a relaxation parameter $ \omega > 1 $ and decomposes the matrix as $ A = L + D + U $, where $ L, U $ are strictly upper- and lower-diagonal matrices and $ D $ is diagonal.

Decompose the $ A $ matrix, multiply the system by $ \omega $, and rearrange to find

$$
(D + \omega L) x^{k+1} = \omega b - \left(\omega U +(\omega - 1)D \right)x^k
$$

In that case, $ D + \omega L $ is a triangular matrix, and hence the linear solution is $ O(N^2) $.

```julia
v = zeros(N)
sor!(v, A, r, 1.1, maxiter = 40)
@show norm(v - v_direct, Inf);
```

    norm(v - v_direct, Inf) = 3.745356593753968e-7

The accuracy is now `1E-7`.  If we change the parameter to $ \omega = 1.2 $, the accuracy further increases to `1E-9`.

This technique is common with iterative methods:  Frequently, adding a damping or a relaxation parameter will counterintuitively speed up the convergence process.

**Note:** The stationary iterative methods are not always used directly, but are sometimes used as a “smoothing” step (e.g., running 5-10 times) prior to using other Krylov methods.

## Krylov Methods

A more commonly used set of iterative methods is based on [Krylov subspaces](https://en.wikipedia.org/wiki/Krylov_subspace), which involve iterating the $ A^k x $ matrix-vector product, and orthogonalizing to ensure that the resulting iteration is not too collinear.

The prototypical Krylov method is [Conjugate Gradient](https://en.wikipedia.org/wiki/Conjugate_gradient_method), which requires the $ A $ matrix to be
symmetric and positive definite.

Solving an example:

```julia
N = 100
A = sprand(100, 100, 0.1)   # 10 percent non-zeros
A = A * A'  # easy way to generate a symmetric positive-definite matrix
@show isposdef(A)
b = rand(N)
x_direct = A \ b  # sparse direct solver more appropriate here
cond(Matrix(A * A'))
```

    isposdef(A) = true

    3.5791585364800934e10

Notice that the condition numbers tend to be large for large random matrices.

Solving this system with the conjugate gradient method:

```julia
x = zeros(N)
sol = cg!(x, A, b, log=true, maxiter = 1000)
sol[end]
```

    Converged after 174 iterations.

### Introduction to Preconditioning

If you tell a numerical analyst that you are using direct methods, their first question may be, “which factorization?” But if you tell them you
are using an iterative method, they may ask “which preconditioner?”.

As discussed at the beginning of the lecture, the spectral properties of matrices determine the rate of convergence
of iterative methods. In particular, ill-conditioned matrices can converge slowly with iterative methods, for the same
reasons that naive value-function iteration will converge slowly if the discount rate is close to `1`.

Preconditioning solves this problem by adjusting the spectral properties of the matrix, at the cost of some extra computational
operations.

To see an example of a right-preconditioner, consider a matrix $ P $ which has a convenient and numerically stable inverse.  Then

$$
\begin{align}
A x &= b\\
A P^{-1} P x &= b\\
A P^{-1} y &= b\\
P x &= y
\end{align}
$$

That is, solve $ (A P^{-1})y = b $ for $ y $, and then solve $ P x = y $ for $ x $.

There are all sorts of preconditioners, and they are specific to the particular problem at hand. The key features are that they have convenient (and lower-order!) ways to solve the
resulting system and they lower the condition number of the matrix.  To see this in action, we can look at a simple preconditioner.

The diagonal precondition is simply `P = Diagonal(A)`.  Depending on the matrix, this can change the condition number a little or a lot.

```julia
AP = A * inv(Diagonal(A))
@show cond(Matrix(A))
@show cond(Matrix(AP));
```

    cond(Matrix(A)) = 189186.6473381337
    cond(Matrix(AP)) = 175174.59095330362

But it may or may not decrease the number of iterations

```julia
using Preconditioners
x = zeros(N)
P = DiagonalPreconditioner(A)
sol = cg!(x, A, b, log=true, maxiter = 1000)
sol[end]
```

    Converged after 174 iterations.

Another classic preconditioner is the incomplete LU decomposition

```julia
using IncompleteLU
x = zeros(N)
P = ilu(A, τ = 0.1)
sol = cg!(x, A, b, Pl = P, log=true, maxiter = 1000)
sol[end]
```

    Converged after 86 iterations.

The `τ` parameter determines the degree of the LU decomposition to conduct, providing a tradeoff between preconditioner and solve speed.

A good rule of thumb is that you should almost always be using a preconditioner with iterative methods, and you should experiment to find preconditioners that are appropriate for your problem.

Finally, naively trying another preconditioning approach (called [Algebraic Multigrid](https://en.wikipedia.org/wiki/Multigrid_method#Algebraic_MultiGrid_%28AMG%29)) gives us a further drop in the number of iterations.

```julia
x = zeros(N)
P = AMGPreconditioner{RugeStuben}(A)
sol = cg!(x, A, b, Pl = P, log=true, maxiter = 1000)
sol[end]
```

    Converged after 59 iterations.

*Note:* Preconditioning is also available for stationary, iterative methods (see [this example](https://en.wikipedia.org/wiki/Preconditioner#Preconditioned_iterative_methods)), but
is frequently not implemented since such methods are not often used for the complete solution.

### Methods for General Matrices

There are many algorithms which exploit matrix structure (e.g., the conjugate gradient method for positive-definite matrices, and  MINRES for matrices that are only symmetric/Hermitian).

On the other hand, if there is no structure to a sparse matrix, then GMRES is a good approach.

To experiment with these methods, we will use our ill-conditioned interpolation problem with a monomial basis.

```julia
using IterativeSolvers

N = 10
f(x) = exp(x)
x = range(0.0, 10.0, length = N+1)
y = f.(x)  # generate some data to interpolate
A = sparse([x_i^n for x_i in x, n in 0:N])
c = zeros(N+1)  # initial guess required for iterative solutions
results = gmres!(c, A, y, log=true, maxiter = 1000)
println("cond(A) = $(cond(Matrix(A))), $(results[end]) Norm error $(norm(A*c - y, Inf))")
```

    cond(A) = 4.462833495403007e12, Converged after 11 iterations. Norm error 7.62520357966423e-8

That method converged in 11 iterations.  Now if we try it with an incomplete LU preconditioner, we see that it converges immediately.

```julia
N = 10
f(x) = exp(x)
x = range(0.0, 10.0, length = N+1)
y = f.(x)  # generate some data to interpolate
A = [x_i^n for x_i in x, n in 0:N]
P = ilu(sparse(A), τ = 0.1)
c = zeros(N+1)  # initial guess required for iterative solutions
results = gmres!(c, A, y, Pl = P,log=true, maxiter = 1000)
println("$(results[end]) Norm error $(norm(A*c - y, Inf))")
```

    Converged after 1 iterations. Norm error 4.5034175855107605e-7

With other preconditioners (e.g., `DiagonalPreconditioner`), we may save only one or two iterations.  Keep in mind,
however, to consider the cost of the preconditioning process in your problem.

### Matrix-Free Methods

First, lets use a Krylov method to solve our simple valuation problem

```julia
α = 0.1
N = 100
Q = Tridiagonal(fill(α, N-1), [-α; fill(-2α, N-2); -α], fill(α, N-1))

r = range(0.0, 10.0, length=N)
ρ = 0.05

A = ρ * I - Q
v = zeros(N)
results = gmres!(v, A, r, log=true)
v_sol = results[1]
println("$(results[end])")
```

    Converged after 20 iterations.

While the `A` matrix was important to be kept in memory for direct methods, Krylov methods such as GMRES are built on matrix-vector products, i.e., $ A x $ for iterations on the $ x $.

This product can be written directly for a given $ x $,

$$
A x = \begin{bmatrix} (\rho + \alpha) x_1 - \alpha x_2\\
                    - \alpha x_1 + (\rho + 2 \alpha) x_2 - \alpha x_3\\
                    \vdots\\
                    - \alpha x_{N-2} + (\rho + 2 \alpha) x_{N-1} - \alpha x_{N}\\
                    - \alpha x_{N-1} + (\rho + \alpha) x_N
    \end{bmatrix}
$$

This can be implemented as a function (either in-place or out-of-place) which calculates $ y = A x $

```julia
A_mul(x) = [ (ρ + α) * x[1] - α * x[2];
             [-α * x[i-1] + (ρ + 2*α) * x[i] - α * x[i+1] for i in 2:N-1];  # comprehension
             - α * x[end-1] + (ρ + α) * x[end]]

x = rand(N)
@show norm(A * x - A_mul(x))  # compare to matrix;
```

    norm(A * x - A_mul(x)) = 0.0

The final line verifies that the `A_mul` function provides the same result as the matrix multiplication with our original `A` for a random vector.

In abstract mathematics, a finite-dimensional [linear operator](https://en.wikipedia.org/wiki/Linear_map) is a mapping $ A : R^N \to R^N $
that satisfies a number of criteria such as $ A (c_1 x_1 + c_2 x_2) = c_1 A x_1 + c_2 A x_2 $ for scalars $ c_i $ and vectors $ x_i $.

Moving from abstract mathematics to [generic programming](../more_julia/generic_programming.html), we can think of a linear operator
as a map that satisfies a number of requirements (e.g., it has a left-multiply to apply the map `*`, an in-place left-multiply `mul!`, an associated `size`).  A Julia matrix
is just one possible implementation of the abstract concept of a linear operator.

Convenience wrappers can provide some of the boilerplate which turns the `A_mul` function into something that behaves like a matrix.  One
package is [LinearMaps.jl](https://github.com/Jutho/LinearMaps.jl) and another is [LinearOperators.jl](https://github.com/JuliaSmoothOptimizers/LinearOperators.jl)

```julia
using LinearMaps
A_map = LinearMap(A_mul, N)  # map uses the A_mul function
```

    LinearMaps.FunctionMap{Float64}(A_mul, 100, 100; ismutating=false, issymmetric=false, ishermitian=false, isposdef=false)

Now, with the `A_map` object, we can fulfill many of the operations we would expect from a matrix

```julia
x = rand(N)
@show norm(A_map * x  - A * x)
y = similar(x)
mul!(y, A_map, x) # in-place multiplication
@show norm(y - A * x)
@show size(A_map)
@show norm(Matrix(A_map) - A)
@show nnz(sparse(A_map));
```

    norm(A_map * x - A * x) = 0.0
    norm(y - A * x) = 0.0
    size(A_map) = (100, 100)
    norm(Matrix(A_map) - A) = 0.0
    nnz(sparse(A_map)) = 298

**Note:**  In the case of `sparse(A_map)` and `Matrix(A_map)`, the code is using the left-multiplication operator with `N` standard basis vectors to construct
the full matrix.  This should be used only for testing purposes.

But notice that as the linear operator does not have indexing operations, it is not an array or a matrix.

```julia
typeof(A_map) <: AbstractArray
```

    false

As long as algorithms using linear operators are written generically (e.g., using the matrix-vector `*` or `mul!` functionss) and the types of functions are not
unnecessarily constrained to be `Matrix` or `AbstractArray` when it isn’t strictly necessary, then the `A_map` type can work in places which would otherwise require a matrix.

For example, the Krylov methods in `IterativeSolvers.jl` are written for generic left-multiplication

```julia
results = gmres(A_map, r, log = true)  # Krylov method using the matrix-free type
println("$(results[end])")
```

    Converged after 20 iterations.

These methods are typically not competitive with sparse, direct methods unless the problems become very large.  In that case,
we often want to work with pre-allocated vectors.  Instead of using `y = A * x` for matrix-vector products,
we would use the in-place `mul!(y, A, x)` function.  The wrappers for linear operators all support in-place non-allocating versions for this purpose.

```julia
function A_mul!(y, x)  # in-place version
    y[1] = (ρ + α) * x[1] - α * x[2]
    for i in 2:N-1
        y[i] = -α * x[i-1] + (ρ + 2α) * x[i] -α * x[i+1]
    end
    y[end] = - α * x[end-1] + (ρ + α) * x[end]
    return y
end
A_map_2 = LinearMap(A_mul!, N, ismutating = true)  # ismutating == in-place

v = zeros(N)
@show norm(A_map_2 * v - A * v)  # can still call with * and have it allocate
results = gmres!(v, A_map_2, r, log=true)  # in-place gmres
println("$(results[end])")
```

    norm(A_map_2 * v - A * v) = 0.0
    Converged after 20 iterations.

Finally, keep in mind that the linear operators can compose, so that $ A (c_1 x) + B (c_2 x) + x  = (c_1 A + c_2 B + I) x $ is well defined for any linear operators - just as
it would be for matrices $ A, B $ and scalars $ c_1, c_2 $.

For example, take $ 2 A x + x = (2 A + I) x \equiv B x $ as a new linear map,

```julia
B = 2.0 * A_map  + I  # composite linear operator
B * rand(N)  # left-multiply works with the composition
typeof(B)
```

    LinearMaps.LinearCombination{Float64,Tuple{LinearMaps.CompositeMap{Float64,Tuple{LinearMaps.FunctionMap{Float64,typeof(A_mul),Nothing},LinearMaps.UniformScalingMap{Float64}}},LinearMaps.UniformScalingMap{Bool}}}

The wrappers, such as `LinearMap` wrappers, make this composition possible by keeping the composition
graph of the expression (i.e., `LinearCombination`) and implementing the left-multiply recursively using the rules of linearity.

Another example is to solve the $ \rho v = r + Q v $ equation for $ v $ with composition of matrix-free methods for $ L $
rather than by creating the full $ A = \rho - Q $ operator, which we implemented as `A_mul`

```julia
Q_mul(x) = [ -α * x[1] +     α * x[2];
            [α * x[i-1] - 2*α * x[i] + α*x[i+1] for i in 2:N-1];  # comprehension
            α * x[end-1] - α * x[end];]
Q_map = LinearMap(Q_mul, N)
A_composed = ρ * I - Q_map   # map composition, performs no calculations
@show norm(A - sparse(A_composed))  # test produces the same matrix
gmres(A_composed, r, log=true)[2]
```

    norm(A - sparse(A_composed)) = 0.0

    Converged after 20 iterations.

In this example, the left-multiply of the `A_composed` used by `gmres` uses the left-multiply of `Q_map` and `I` with the rules
of linearity.  The `A_composed = ρ * I - Q_map` operation simply creates the `LinearMaps.LinearCombination` type, and doesn’t perform any calculations on its own.

## Iterative Methods for Linear Least Squares

In theory, the solution to the least-squares problem, $ \min_x \| Ax -b \|^2 $, is simply the solution to the normal equations $ (A'A) x  = A'b $.

We saw, however, that in practice, direct methods use a QR decomposition - in part because an ill-conditioned matrix $ A $ becomes even worse when $ A' A $ is formed.

For large problems, we can also consider Krylov methods for solving the linear least-squares problem.  One formulation is the [LSMR](https://stanford.edu/group/SOL/software/lsmr/LSMR-SISC-2011.pdf) algorithm,
which can solve the regularized

$$
\min_x \| Ax -b \|^2 + \| \lambda x\|^2
$$

The purpose of the $ \lambda \geq 0 $ parameter is to dampen the iteration process and/or regularize the solution.  This isn’t required, but can help convergence for ill-conditioned matrices $ A $.  With the
damping parameter, the normalized equations would become $ (A'A + \lambda^2 I) x  = A'b $.

We can compare solving the least-squares problem with LSMR and direct methods

```julia
M = 1000
N = 10000
σ = 0.1
β = rand(M)
# simulate data
X = sprand(N, M, 0.1)
y = X * β + σ * randn(N)
β_direct = X \ y
results = lsmr(X, y, log = true)
β_lsmr = results[1]
@show norm(β_direct - β_lsmr)
println("$(results[end])")
```

    norm(β_direct - β_lsmr) = 9.139228893911292e-6
    Converged after 14 iterations.

Note that rather than forming this version of the normal equations, the LSMR algorithm uses the $ A x $ and $ A' y $ (i.e., the matrix-vector product and the matrix-transpose vector product) to implement an iterative
solution.  Unlike the previous versions, the left-multiply is insufficient since the least squares also deals with the transpose of the operator.  For this reason, in order to use
matrix-free methods, we need to define the `A * x` and `transpose(A) * y` functions separately.

```julia
# Could implement as matrix-free functions.
X_func(u) = X * u  # matrix-vector product
X_T_func(v) = X' * v  # i.e., adjoint-vector product

X_map = LinearMap(X_func, X_T_func, N, M)
results = lsmr(X_map, y, log = true)
println("$(results[end])")
```

    Converged after 14 iterations.

## Iterative Methods for Eigensystems

When you use `eigen` on a dense matrix, it calculates an eigendecomposition and provides all the eigenvalues and eigenvectors.

While this is sometimes necessary, a spectral decomposition of a dense, unstructured matrix is one of the costliest $ O(N^3) $ operations (i.e., it has
one of the largest constants).  For large matrices, it is often infeasible.

Luckily, we frequently need only a few eigenvectors/eigenvalues (in some cases just one), which enables a different set of algorithms.

For example, in the case of a discrete-time Markov chain, in order to find the stationary distribution, we are looking for the
eigenvector associated with the eigenvalue 1.  As usual, a little linear algebra goes a long way.

From the [Perron-Frobenius theorem](https://en.wikipedia.org/wiki/Perron%E2%80%93Frobenius_theorem#Stochastic_matrices), the largest eigenvalue of an irreducible stochastic matrix is 1 - the same eigenvalue we are looking for.

Iterative methods for solving eigensystems allow targeting the smallest magnitude, the largest magnitude, and many others.  The easiest library
to use is [Arpack.jl](https://julialinearalgebra.github.io/Arpack.jl/latest/).

As an example,

```julia
using Arpack, LinearAlgebra
N = 1000
A = Tridiagonal([fill(0.1, N-2); 0.2], fill(0.8, N), [0.2; fill(0.1, N-2);])
A_adjoint = A'

λ, ϕ = eigs(A_adjoint, nev=1, which=:LM, maxiter=1000)  # Find 1 of the largest magnitude eigenvalue
ϕ = real(ϕ) ./ sum(real(ϕ))
@show λ
@show mean(ϕ);
```

    λ = Complex{Float64}[1.0000000000000189 + 0.0im]
    mean(ϕ) = 0.0010000000000000002

Indeed, the `λ` is equal to `1`.  If we choose `nev = 2`, it will provide the eigenpairs with the two eigenvalues of largest absolute value.

*Hint*: If you get errors using `Arpack`, increase the `maxiter` parameter for your problems.

Iterative methods for eigensystems rely on matrix-vector products rather than decompositions, and are amenable to matrix-free approaches.  For example,
take the Markov chain for a simple counting process:

1. The count starts at $ 1 $ and has a maximum of $ N $.
1. With probability $ \theta \geq 0 $, an existing count is lost with probability $ \zeta \geq 0 $ such that $ \theta + \zeta \leq 1 $.
1. If the count is at $ 1 $, then the only transition is to add a count with probability $ \theta $.
1. If the current count is $ N $, then the only transition is to lose the count with probability $ \zeta $.

First, finding the transition matrix $ P $ and its adjoint directly as a check

```julia
θ = 0.1
ζ = 0.05
N = 5
P = Tridiagonal(fill(ζ, N-1), [1-θ; fill(1-θ-ζ, N-2); 1-ζ], fill(θ, N-1))
P'
```

    5×5 Tridiagonal{Float64,Array{Float64,1}}:
     0.9  0.05   ⋅     ⋅     ⋅
     0.1  0.85  0.05   ⋅     ⋅
      ⋅   0.1   0.85  0.05   ⋅
      ⋅    ⋅    0.1   0.85  0.05
      ⋅    ⋅     ⋅    0.1   0.95

Implementing the adjoint-vector product directly, and verifying that it gives the same matrix as the adjoint

```julia
P_adj_mul(x) = [ (1-θ) * x[1] + ζ * x[2];
                [θ * x[i-1] + (1-θ-ζ) * x[i] + ζ * x[i+1] for i in 2:N-1];  # comprehension
            θ * x[end-1] + (1-ζ) * x[end];]
P_adj_map = LinearMap(P_adj_mul, N)
@show norm(P' - sparse(P_adj_map))
```

    norm(P' - sparse(P_adj_map)) = 0.0

    0.0

Finally, solving for the stationary distribution using the matrix-free method (which could be verified against the decomposition approach of $ P' $)

```julia
λ, ϕ = eigs(P_adj_map, nev=1, which=:LM, maxiter=1000)
ϕ = real(ϕ) ./ sum(real(ϕ))
@show λ
@show ϕ
```

    λ = Complex{Float64}[1.0 + 0.0im]
    ϕ = [0.03225806451612657; 0.06451612903225695; 0.1290322580645172; 0.25806451612903425; 0.516129032258065]

    5×1 Array{Float64,2}:
     0.03225806451612657
     0.06451612903225695
     0.1290322580645172
     0.25806451612903425
     0.516129032258065

Of course, for a problem this simple, the direct eigendecomposition will be significantly faster.  Use matrix-free iterative methods only for large systems where
you do not need all of the eigenvalues.

## Krylov Methods for Markov-Chain Dynamics

This example applies the methods in this lecture to a large continuous-time Markov chain, and provides some practice working with arrays of arbitrary dimensions.

Consider a version of the Markov-chain dynamics in [[Per19]](../zreferences.html#perla2019), where a firm has a discrete number of customers of different types.  To keep things as simple as possible, assume that there are $ m=1, \ldots M $ types of customers and that the firm may have $ n = 1, \ldots N $ customers of each type.

To set the notation, let $ n_m \in \{1, \ldots N\} $ be the number of customers of type $ m $, so that the state of a firm is $ \{n_1, \ldots n_m \ldots, n_M\} $.  The cardinality of possible states is then $ \mathbf{N}\equiv N^M $, which can blow up quickly as the number of types increases.

The stochastic process is a simple counting/forgetting process, as follows:

1. For every $ 1 \leq n_m(t) < N $, there is a $ \theta $ intensity of arrival of a new customer, so that $ n_m(t+\Delta) = n_m(t) + 1 $.
1. For every $ 1 < n_m(t) \leq N $, there is a $ \zeta $ intensity of losing a customer, so that $ n_m(t+\Delta) = n_m(t) - 1 $.

### Matrix-free Infinitesimal Generator

In order to define an intensity matrix $ Q $ of size $ \mathbf{N}\times \mathcal{N} $, we need to choose a consistent ordering of the states.  But
before we enumerate them linearly, take a $ v\in R^{\mathbf{N}} $ interpreted as a multidimensional array and look at the left product of the linear operator $ Q v \to R^{\mathbf{N}} $.

For example, if we were implementing the product at the row of $ Q $ corresponding to the $ (n_1, \ldots, n_M) $ state, then

$$
\begin{align}
    Q_{(n_1, \ldots n_M)} \cdot v &=
\theta \sum_{m=1}^M (n_m < N)  v(n_1, \ldots, n_m + 1, \ldots, n_M)\\
                                        &+ \zeta \sum_{m=1}^M (1 < n_m)  v(n_1, \ldots, n_m - 1, \ldots, n_M)\\
                                        &-\left(\theta\, \text{Count}(n_m < N) + \zeta\, \text{Count}( n_m > 1)\right)v(n_1, \ldots, n_M)
\end{align}
$$

Here:

- the first term includes all of the arrivals of new customers into the various $ m $
- the second term is the loss of a customer for the various $ m $
- the last term is the intensity of all exits from this state (i.e., counting the intensity of all other transitions, to ensure that the row will sum to $ 0 $)

In practice, rather than working with the $ f $ as a multidimensional type, we will need to enumerate the discrete states linearly, so that we can iterate $ f $ between $ 1 $ and $ \mathbf{N} $.  An especially convenient
approach is to enumerate them in the same order as the $ K $-dimensional Cartesian product of the $ N $ states in the multi-dimensional array above.

This can be done with the `CartesianIndices` function, which is used internally in Julia for the `eachindex` function.  For example,

```julia
N = 2
M = 3
shape = Tuple(fill(N, M))
v = rand(shape...)
@show typeof(v)
for ind in CartesianIndices(v)
    println("v$(ind.I) = $(v[ind])")  # .I gets the tuple to display
end
```

    typeof(v) = Array{Float64,3}
    v(1, 1, 1) = 0.639089412234831
    v(2, 1, 1) = 0.4302368488000152
    v(1, 2, 1) = 0.21490768283644002
    v(2, 2, 1) = 0.7542051014748841
    v(1, 1, 2) = 0.4330861190374067
    v(2, 1, 2) = 0.07556766967902084
    v(1, 2, 2) = 0.2143739072351467
    v(2, 2, 2) = 0.43231874437572815

The added benefit of this approach is that it will be the most efficient way to iterate through vectors in the implementation.

For the counting process with arbitrary dimensions, we will frequently be incrementing or decrementing the $ m $ unit vectors of the `CartesianIndex` type with

```julia
e_m = [CartesianIndex((1:M .== i)*1...)  for i in 1:M]
```

    3-element Array{CartesianIndex{3},1}:
     CartesianIndex(1, 0, 0)
     CartesianIndex(0, 1, 0)
     CartesianIndex(0, 0, 1)

and then use the vector to increment.  For example, if the current count is `(1, 2, 2)` and we want to add a count of `1` to the first index and remove a count
of `1` from the third index, then

```julia
ind = CartesianIndex(1, 2, 2)  # example counts coming from CartesianIndices
@show ind + e_m[1]  # increment the first index
@show ind - e_m[3];  # decrement the third index
```

    ind + e_m[1] = CartesianIndex(2, 2, 2)
    ind - e_m[3] = CartesianIndex(1, 2, 1)

This works, of course, because the `CartesianIndex` type is written to support efficient addition and subtraction.  Finally, to implement the operator, we need to count the indices in the states where increment and decrement occurs.

```julia
@show ind
@show count(ind.I .> 1)
@show count(ind.I .< N);
```

    ind = CartesianIndex(1, 2, 2)
    count(ind.I .> 1) = 2
    count(ind.I .< N) = 1

With this, we are now able to write the $ Q $ operator on the $ f $ vector, which is enumerated by the Cartesian indices.  First, collect the
parameters in a named tuple generator

```julia
using Parameters, BenchmarkTools
default_params = @with_kw (θ = 0.1, ζ = 0.05, ρ = 0.03, N = 10, M = 6,
                           shape = Tuple(fill(N, M)),  # for reshaping vector to M-d array
                           e_m = ([CartesianIndex((1:M .== i)*1...)  for i in 1:M]))
```

    ##NamedTuple_kw#256 (generic function with 2 methods)

Next, implement the in-place matrix-free product

```julia
function Q_mul!(dv, v, p)
    @unpack θ, ζ, N, M, shape, e_m = p
    v = reshape(v, shape)  # now can access v, dv as M-dim arrays
    dv = reshape(dv, shape)

    @inbounds for ind in CartesianIndices(v)
        dv[ind] = 0.0
        for m in 1:M
            n_m = ind[m]
            if(n_m < N)
                dv[ind] += θ * v[ind + e_m[m]]
            end
            if(n_m > 1)
                dv[ind] += ζ *v[ind - e_m[m]]
            end
        end
        dv[ind] -= (θ * count(ind.I .< N) + ζ * count(ind.I .> 1)) * v[ind]
    end
end

p = default_params()
v = zeros(p.shape)
dv = similar(v)
@btime Q_mul!($dv, $v, $p)
```

      53.485 ms (0 allocations: 0 bytes)

From the output of the benchmarking, note that the implementation of the left-multiplication takes less than 100 milliseconds, and allocates little or no memory, even though the Markov chain has a million possible states (i.e., $ N^M = 10^6 $).

### Solving a Valuation Problem

As before, we could use this Markov chain to solve a Bellman equation.  Assume that the firm discounts at rate $ \rho > 0 $ and gets a flow payoff of a different $ z_m $ per
customer of type $ m $.  For example, if the state of the firm is $ (n_1, n_2, n_3) = (2,3,2) $, then it gets $ \begin{bmatrix}2 & 3 & 2\end{bmatrix} \cdot \begin{bmatrix}z_1& z_2 & z_3\end{bmatrix} $ in flow profits.

Given this profit function, we can write the simple Bellman equation in our standard form of $ \rho v = r + Q v $, defining the appropriate payoff $ r $.  For example, if $ z_m = m^2 $, then

```julia
function r_vec(p)
    z = (1:p.M).^2  # payoffs per type m
    r = [0.5 * dot(ind.I, z)  for ind in CartesianIndices(p.shape)]
    return reshape(r, p.N^p.M)  # return as a vector
end
@show typeof(r_vec(p))
r_vec(p) |> mean
```

    typeof(r_vec(p)) = Array{Float64,1}

    250.25

Note that the returned $ r $ is a vector, enumerated in the same order as the $ n_m $ states.

Since the ordering of $ r $ is consistent with that of $ Q $, we can solve $ (\rho - Q) v = r $ as a linear system.

Below, we create a linear operator and compare the algorithm for a few different iterative methods [(GMRES, BiCGStab(l), IDR(s), etc.)](https://juliamath.github.io/IterativeSolvers.jl/dev/#What-method-should-I-use-for-linear-systems?-1) with a small problem
of only 10,000 possible states.

```julia
p = default_params(N=10, M=4)
Q = LinearMap((df, f) -> Q_mul!(df, f, p), p.N^p.M, ismutating = true)
A = p.ρ * I - Q
A_sparse = sparse(A)  # expensive: use only in tests
r = r_vec(p)
v_direct = A_sparse \ r
iv = zero(r)

@btime $A_sparse \ $r  # direct
@show norm(gmres(A, r) - v_direct)
@btime gmres!(iv, $A, $r) setup = (iv = zero(r))

@show norm(bicgstabl(A, r) - v_direct)
@btime bicgstabl!(iv, $A, $r) setup = (iv = zero(r))

@show norm(idrs(A, r) - v_direct)
@btime idrs($A, $r);
```

      679.258 ms (75 allocations: 181.85 MiB)
    norm(gmres(A, r) - v_direct) = 2.390837794373907e-5
      5.272 ms (226 allocations: 3.37 MiB)
    norm(bicgstabl(A, r) - v_direct) = 1.108634674599104e-5
      23.129 ms (432 allocations: 9.86 MiB)
    norm(idrs(A, r) - v_direct) = 4.992408654237078e-8
      7.443 ms (312 allocations: 4.91 MiB)

Here, we see that even if the $ A $ matrix has been created, the direct sparse solver (which uses a sparse LU or QR) is at least an order of magnitude slower and allocates over an order of magnitude more memory.  This is in addition to the allocation for the `A_sparse` matrix itself, which is not needed for iterative methods.

The different iterative methods have tradeoffs when it comes to accuracy, speed, convergence rate, memory requirements, and usefulness of preconditioning.  Going much above $ \mathbf{N} = 10^4 $, the direct methods quickly become infeasible.

Putting everything together to solving much larger systems with GMRES as our linear solvers

```julia
function solve_bellman(p; iv = zeros(p.N^p.M))
    @unpack ρ, N, M = p
    Q = LinearMap((df, f) -> Q_mul!(df, f, p), N^M, ismutating = true)
    A = ρ * I - Q
    r = r_vec(p)

    sol = gmres!(iv, A, r, log = false)  # iterative solver, matrix-free
    return sol
end
p = default_params(N=10, M=6)
@btime solve_bellman($p);
```

      1.581 s (270 allocations: 366.23 MiB)

This solves a value function with a Markov chain of a million states in a little over a second!  This general approach seems to scale roughly linearly.  For example, try $ N=10, M=8 $
to solve an equation with a Markov chain with 100 million possible states, which can be solved in about 3-4 minutes.  Above that order of magnitude, you may need to tinker with the linear solver parameters to ensure that you are not memory limited (e.g., change the `restart` parameter of GMRES).

### Markov Chain Steady State and Dynamics

Recall that given an $ N $-dimensional intensity matrix $ Q $ of a CTMC, the evolution of the pdf from an initial condition $ \psi(0) $ is the system of linear differential equations

$$
\dot{\psi}(t) = Q^T \psi(t)
$$

If $ Q $ is a matrix, we could just take its transpose to find the adoint.  However, with matrix-free methods, we need to implement the
adjoint-vector product directly.

The logic for the adjoint is that for a given $ n = (n_1,\ldots, n_m, \ldots n_M) $, the $ Q^T $ product for that row has terms enter when

1. $ 1 < n_m \leq N $, entering into the identical $ n $ except with one less customer in the :math:`m`th position
1. $ 1 \leq n_m < N $, entering into the identical $ n $ except with one more customer in the :math:`m`th position

Implementing this logic, first in math and then in code,

$$
\begin{align}
    Q^T_{(n_1, \ldots, n_M)} \cdot \psi &=
\theta \sum_{m=1}^M (n_m > 1)  \psi(n_1, \ldots, n_m - 1, \ldots, n_M)\\
                                        &+ \zeta \sum_{m=1}^M (n_m < N)  \psi(n_1, \ldots, n_m + 1, \ldots, n_M)\\
                                        &-\left(\theta\, \text{Count}(n_m < N) + \zeta\, \text{Count}( n_m > 1)\right)\psi(n_1, \ldots, n_M)
\end{align}
$$

```julia
function Q_T_mul!(dψ, ψ, p)
    @unpack θ, ζ, N, M, shape, e_m = p
    ψ = reshape(ψ, shape)
    dψ = reshape(dψ, shape)

    @inbounds for ind in CartesianIndices(ψ)
        dψ[ind] = 0.0
        for m in 1:M
            n_m = ind[m]
            if(n_m > 1)
                dψ[ind] += θ * ψ[ind - e_m[m]]
            end
            if(n_m < N)
                dψ[ind] += ζ *ψ[ind + e_m[m]]
            end
        end
        dψ[ind] -= (θ * count(ind.I .< N) + ζ * count(ind.I .> 1)) * ψ[ind]
    end
end
```

    Q_T_mul! (generic function with 1 method)

The `sparse` function for the operator is useful for testing that the function is correct, and is the adjoint of
our `Q` operator.

```julia
p = default_params(N=5, M=4)  # sparse is too slow for the full matrix
Q = LinearMap((df, f) -> Q_mul!(df, f, p), p.N^p.M, ismutating = true)
Q_T = LinearMap((dψ, ψ) -> Q_T_mul!(dψ, ψ, p), p.N^p.M, ismutating = true)
@show norm(sparse(Q)' - sparse(Q_T));  # reminder: use sparse only for testing!
```

    norm((sparse(Q))' - sparse(Q_T)) = 0.0

As discussed previously, the steady state can be found as the eigenvector associated with the zero eigenvalue (i.e., the one that solves $ Q^T \psi = 0 \psi $).  We could
do this with a dense eigenvalue solution for relatively small matrices

```julia
p = default_params(N=5, M=4)
eig_Q_T = eigen(Matrix(Q_T))
vec = real(eig_Q_T.vectors[:,end])
direct_ψ = vec ./ sum(vec)
@show eig_Q_T.values[end];
```

    eig_Q_T.values[end] = -4.163336342344337e-16 + 0.0im

This approach relies on a full factorization of the underlying matrix, delivering the entire spectrum.  For our purposes, this is not necessary.

Instead, we could use the `Arpack.jl` package to target the eigenvalue of smallest absolute value, which relies on an iterative method.

A final approach in this case is to notice that the $ \mathbf{N}\times\mathbf{N} $ matrix is of
rank $ \mathbf{N} - 1 $ when the Markov chain is irreducible.  The stationary solution is a vector in the $ 1 $-dimensional nullspace
of the matrix.

Using Krylov methods to solve a linear system with the right-hand side all $ 0 $min_x ||A x - 0||_2` solved
iteratively from a non-zero initial condition will converge to a point in the nullspace.

We can use various Krylov methods for this trick (e.g., if the matrix is symmetric and positive definite, we could use Conjugate Gradient) but in our case we will
use GMRES since we do not have any structure.

```julia
p = default_params(N=5, M=4)  # sparse is too slow for the full matrix
Q_T = LinearMap((dψ, ψ) -> Q_T_mul!(dψ, ψ, p), p.N^p.M, ismutating = true)
ψ = fill(1/(p.N^p.M), p.N^p.M) # can't use 0 as initial guess
sol = gmres!(ψ, Q_T, zeros(p.N^p.M))  # i.e., solve Ax = 0 iteratively
ψ = ψ / sum(ψ)
@show norm(ψ - direct_ψ);
```

    norm(ψ - direct_ψ) = 6.098250476301026e-11

The speed and memory differences between these methods can be orders of magnitude.

```julia
p = default_params(N=4, M=4)  # Dense and sparse matrices are too slow for the full dataset.
Q_T = LinearMap((dψ, ψ) -> Q_T_mul!(dψ, ψ, p), p.N^p.M, ismutating = true)
Q_T_dense = Matrix(Q_T)
Q_T_sparse = sparse(Q_T)
b = zeros(p.N^p.M)
@btime eigen($Q_T_dense)
@btime eigs($Q_T_sparse, nev=1, which=:SM, v0 = iv) setup = (iv = fill(1/(p.N^p.M), p.N^p.M))
@btime gmres!(iv, $Q_T, $b) setup = (iv = fill(1/(p.N^p.M), p.N^p.M));
```

      38.993 ms (270 allocations: 2.28 MiB)
      4.485 ms (341 allocations: 1.21 MiB)
      180.634 μs (359 allocations: 66.03 KiB)

The differences become even more stark as the matrix grows.  With `default_params(N=5, M=5)`, the `gmres` solution is at least 3 orders of magnitude faster, and uses close to 3 orders of magnitude less memory than the dense solver.  In addition, the `gmres` solution is about an order of magnitude faster than the iterative sparse eigenvalue solver.

The algorithm can solve for the steady state of $ 10^5 $ states in a few seconds

```julia
function stationary_ψ(p)
    Q_T = LinearMap((dψ, ψ) -> Q_T_mul!(dψ, ψ, p), p.N^p.M, ismutating = true)
    ψ = fill(1/(p.N^p.M), p.N^p.M) # can't use 0 as initial guess
    sol = gmres!(ψ, Q_T, zeros(p.N^p.M))  # i.e., solve Ax = 0 iteratively
    return ψ / sum(ψ)
end
p = default_params(N=10, M=5)
@btime stationary_ψ($p);
```

      3.183 s (4880 allocations: 19.32 MiB)

As a final demonstration, consider calculating the full evolution of the $ ψ(t) $ Markov chain.  For the constant
$ Q' $ matrix, the solution to this system of equations is $ \psi(t) = \exp(Q') \psi(0) $

Matrix-free Krylov methods using a technique called [exponential integration](https://en.wikipedia.org/wiki/Exponential_integrator) can solve this for high-dimensional problems.

For this, we can set up a `MatrixFreeOperator` for our `Q_T_mul!` function (equivalent to the `LinearMap`, but with some additional requirements for the ODE solver) and use the [LinearExponential](http://docs.juliadiffeq.org/latest/solvers/ode_solve.html#Exponential-Methods-for-Linear-and-Affine-Problems-1) time-stepping method.

```julia
using OrdinaryDiffEq, DiffEqOperators

function solve_transition_dynamics(p, t)
    @unpack N, M = p

    ψ_0 = [1.0; fill(0.0, N^M - 1)]
    O! = MatrixFreeOperator((dψ, ψ, p, t) -> Q_T_mul!(dψ, ψ, p), (p, 0.0), size=(N^M,N^M), opnorm=(p)->1.25)

    # define the corresponding ODE problem
    prob = ODEProblem(O!,ψ_0,(0.0,t[end]), p)
    return solve(prob, LinearExponential(krylov=:simple), tstops = t)
end
t = 0.0:5.0:100.0
p = default_params(N=10, M=6)
sol = solve_transition_dynamics(p, t)
v = solve_bellman(p)
plot(t, [dot(sol(tval), v) for tval in t], xlabel = "t", label = ["E_t(v)"])
```

The above plot (1) calculates the full dynamics of the Markov chain from the $ n_m = 1 $ for all $ m $ initial condition; (2) solves the dynamics of a system of a million ODEs; and (3) uses the calculation of the Bellman equation to find the expected valuation during that transition.  The entire process takes less than 30 seconds.


# Short Interest Rate Model Calibration
[Goutham Balaraman](http://gouthamanbalaraman.com)

I have talked about Hull-White model in my earlier blog posts. The focus of those posts was to see how to use the model classes. The model parameters were assumed to be given. However in practice, the model parameters need to calibrated from market data. Typically instruments such as swaptions, caps or floors and their market prices / volatilities are taken as inputs. Then the model parameters are fit in such a way that the model prices these options close enough. The goodness of fit depends, apart from the choice of the numerical methods, on the type of model itself. This is because models such as Hull-White 1 factor cannot fit some of the humped volatility term structures observed in the market. Never the less, Hull-White is usually a good starting point to understand calibration process.

Here we will discuss Hull-White model in detail. Then we will also show how the same procedure can be applied to calibrate other short rate models.

```python
import QuantLib as ql
from collections import namedtuple
import math
```

## Hull-White 1 Factor Model

Hull-White model was one of the first practical exogenous models that attempted to fit to the market interest rate term structures. The model is described as:

\begin{equation}
dr_t = (\theta(t) - a r_t) dt + \sigma dW_t
\end{equation}

where $a$ is the mean reversion constant, $\sigma$ is the volatility parameter. The parameter $\theta(t)$ is chosen in order to fit the input term structure of interest rates.

What is the "right" value for parameters $a$ and $\sigma$? This is the question that we address by calibrating to market instruments.

```python
today = ql.Date(15, ql.February, 2002);
settlement= ql.Date(19, ql.February, 2002);
ql.Settings.instance().evaluationDate = today;
term_structure = ql.YieldTermStructureHandle(
    ql.FlatForward(settlement,0.04875825,ql.Actual365Fixed())
    )
index = ql.Euribor1Y(term_structure)
```

In this example we are going to calibrate to the swaption volatilities as shown below.

```python
CalibrationData = namedtuple("CalibrationData",
                             "start, length, volatility")
data = [CalibrationData(1, 5, 0.1148),
        CalibrationData(2, 4, 0.1108),
        CalibrationData(3, 3, 0.1070),
        CalibrationData(4, 2, 0.1021),
        CalibrationData(5, 1, 0.1000 )]
```

```python
# <!-- collapse=True -->
def create_swaption_helpers(data, index, term_structure, engine):
    swaptions = []
    fixed_leg_tenor = ql.Period(1, ql.Years)
    fixed_leg_daycounter = ql.Actual360()
    floating_leg_daycounter = ql.Actual360()
    for d in data:
        vol_handle = ql.QuoteHandle(ql.SimpleQuote(d.volatility))
        helper = ql.SwaptionHelper(ql.Period(d.start, ql.Years),
                                   ql.Period(d.length, ql.Years),
                                   vol_handle,
                                   index,
                                   fixed_leg_tenor,
                                   fixed_leg_daycounter,
                                   floating_leg_daycounter,
                                   term_structure
                                   )
        helper.setPricingEngine(engine)
        swaptions.append(helper)
    return swaptions

def calibration_report(swaptions, data):
    print "-"*82
    print "%15s %15s %15s %15s %15s" % \
    ("Model Price", "Market Price", "Implied Vol", "Market Vol", "Rel Error")
    print "-"*82
    cum_err = 0.0
    for i, s in enumerate(swaptions):
        model_price = s.modelValue()
        market_vol = data[i].volatility
        black_price = s.blackPrice(market_vol)
        rel_error = model_price/black_price - 1.0
        implied_vol = s.impliedVolatility(model_price,
                                          1e-5, 50, 0.0, 0.50)
        rel_error2 = implied_vol/market_vol-1.0
        cum_err += rel_error2*rel_error2

        print "%15.5f %15.5f %15.5f %15.5f %15.5f" % \
        (model_price, black_price, implied_vol, market_vol, rel_error)
    print "-"*82
    print "Cumulative Error : %15.5f" % math.sqrt(cum_err)
```

### Calibrating Reversion and Volaitility

Here we use the `JamshidianSwaptionEngine` to value the swaptions as part of calibration. The `JamshidianSwaptionEngine` requires one-factor affine models as input. For other interest rate models, we need a pricing engine that is more suited to those models.

```python
model = ql.HullWhite(term_structure);
engine = ql.JamshidianSwaptionEngine(model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma = model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)
```

    a = 0.04642, sigma = 0.00580

```python

calibration_report(swaptions, data)
```

    ----------------------------------------------------------------------------------
        Model Price    Market Price     Implied Vol      Market Vol       Rel Error
    ----------------------------------------------------------------------------------
            0.00878         0.00949         0.10620         0.11480        -0.07485
            0.00967         0.01008         0.10629         0.11080        -0.04061
            0.00866         0.00872         0.10634         0.10700        -0.00614
            0.00649         0.00623         0.10644         0.10210         0.04237
            0.00354         0.00332         0.10661         0.10000         0.06582
    ----------------------------------------------------------------------------------
    Cumulative Error :         0.11614

```python
model = ql.HullWhite(term_structure);
engine = ql.JamshidianSwaptionEngine(model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma = model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)
calibration_report(swaptions, data)
```

    a = 0.04642, sigma = 0.00580
    ----------------------------------------------------------------------------------
        Model Price    Market Price     Implied Vol      Market Vol       Rel Error
    ----------------------------------------------------------------------------------
            0.00878         0.00949         0.10620         0.11480        -0.07485
            0.00967         0.01008         0.10629         0.11080        -0.04061
            0.00866         0.00872         0.10634         0.10700        -0.00614
            0.00649         0.00623         0.10644         0.10210         0.04237
            0.00354         0.00332         0.10661         0.10000         0.06582
    ----------------------------------------------------------------------------------
    Cumulative Error :         0.11614

### Calibrating Volatility With Fixed Reversion

Some times we need to calibrate with one parameter held fixed. This can be done in the QuantLib libraries. However, this ability is not exposed in the SWIG wrappers as of version 1.6. I have created a [github issue](https://github.com/lballabio/quantlib/issues/336) and provided a patch to address this issue. You will need this patch to execute the following cells. Below, the model is calibrated with a fixed reversion value of 5%.

```python
constrained_model = ql.HullWhite(term_structure, 0.05, 0.001);
engine = ql.JamshidianSwaptionEngine(constrained_model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
constrained_model.calibrate(swaptions, optimization_method, end_criteria, ql.NoConstraint(), [], [True, False])

a, sigma = constrained_model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)
```

    a = 0.05000, sigma = 0.00586

```python
calibration_report(swaptions, data)
```

    ----------------------------------------------------------------------------------
        Model Price    Market Price     Implied Vol      Market Vol       Rel Error
    ----------------------------------------------------------------------------------
            0.00878         0.00949         0.10621         0.11480        -0.07474
            0.00967         0.01008         0.10628         0.11080        -0.04068
            0.00866         0.00872         0.10633         0.10700        -0.00626
            0.00649         0.00623         0.10644         0.10210         0.04231
            0.00354         0.00332         0.10663         0.10000         0.06595
    ----------------------------------------------------------------------------------
    Cumulative Error :         0.11615

## Black Karasinski Model

The Black Karasinski model is described as:

\begin{equation}
d\ln(r_t) = (\theta_t - a \ln(r_t)) dt + \sigma dW_t
\end{equation}

In order to calibrate, we use the `TreeSwaptionEngine` which will work with all short rate models. The calibration is shown below.

```python
model = ql.BlackKarasinski(term_structure);
engine = ql.TreeSwaptionEngine(model, 100)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(10000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma =  model.params()
print "a = %6.5f, sigma = %6.5f" % (a, sigma)
```

    a = 0.03902, sigma = 0.11695

```python
calibration_report(swaptions, data)
```

    ----------------------------------------------------------------------------------
        Model Price    Market Price     Implied Vol      Market Vol       Rel Error
    ----------------------------------------------------------------------------------
            0.00872         0.00949         0.10550         0.11480        -0.08095
            0.00967         0.01008         0.10631         0.11080        -0.04045
            0.00868         0.00872         0.10654         0.10700        -0.00430
            0.00650         0.00623         0.10666         0.10210         0.04446
            0.00355         0.00332         0.10676         0.10000         0.06733
    ----------------------------------------------------------------------------------
    Cumulative Error :         0.12163

## G2++ Model

As a final example, let us look at a calibration example of the 2-factor G2++ model.
\begin{equation}
dr_t = \varphi(t) + x_t + y_t
\end{equation}

where $ x_t $ and $ y_t $ are defined by

\begin{eqnarray}
dx_t &=& -a x_t dt + \sigma dW^1_t\nonumber \\
dy_t &=& -b y_t dt + \eta dW^2_t \nonumber \\
\left<dW^1_t dW^2_t\right> &=& \rho dt
\end{eqnarray}

Once again, we use the `TreeSwaptionEngine` to value the swaptions in the calibration step.

```python
model = ql.G2(term_structure);
engine = ql.TreeSwaptionEngine(model, 25)
# engine = ql.G2SwaptionEngine(model, 10, 400)
# engine = ql.FdG2SwaptionEngine(model)
swaptions = create_swaption_helpers(data, index, term_structure, engine)

optimization_method = ql.LevenbergMarquardt(1.0e-8,1.0e-8,1.0e-8)
end_criteria = ql.EndCriteria(1000, 100, 1e-6, 1e-8, 1e-8)
model.calibrate(swaptions, optimization_method, end_criteria)

a, sigma, b, eta, rho = model.params()
print "a = %6.5f, sigma = %6.5f, b = %6.5f, eta = %6.5f, rho = %6.5f " % (a, sigma, b, eta, rho)
```

    a = 0.04050, sigma = 0.00474, b = 0.04438, eta = 0.00301, rho = 0.03541

```python
calibration_report(swaptions, data)
```

    ----------------------------------------------------------------------------------
        Model Price    Market Price     Implied Vol      Market Vol       Rel Error
    ----------------------------------------------------------------------------------
            0.00870         0.00949         0.10535         0.11480        -0.08228
            0.00967         0.01008         0.10633         0.11080        -0.04029
            0.00868         0.00872         0.10651         0.10700        -0.00456
            0.00650         0.00623         0.10665         0.10210         0.04438
            0.00355         0.00332         0.10680         0.10000         0.06770
    ----------------------------------------------------------------------------------
    Cumulative Error :         0.12265

## Conclusion

In this post, we saw some simple examples of calibrating the model to the swaption volatilities.

```python

```

```python

```

```python

```