# Global Search Algorithms

The solution of the unconstrained optimization problem

\begin{equation}
\min f\left(\vec x\right)
\end{equation}

can always be found by solving

\begin{equation}
\nabla f\left(\vec x^\star\right) = \vec 0\,.
\end{equation}

We choose an initial point and the generate a sequence of iterates. Typically, the best we can hope for is that the sequence converges to a local minimizer. For this reason, it is often desirable for the initial point to be close to a global optimizer. Moreover, these methods require first derivatives (and als second derivatives in the case of Newton's method).

In this lecture we discuss a method that is global in nature in the sense that they attempt to search throughout the entire feasible set.

The _Nelder-Mead simplex_ method uses only objective function values and do not require derivatives. Consequently, it is applicable to a much wider class of optimization problems.

## The Nelder-Mead Simplex Algorithm

The Nelder-Mead algorithm uses the concept of a _simplex_. A simplex is a geometric object determined by an assembly of $n+1$ points, $\vec{p}_{0},\vec{p}_{1},\dots,\vec{p}_{n}$, in the n-dimensional space such that

\begin{equation}
\det\begin{bmatrix}
\vec{p}_{0} & \vec{p}_{1} & \cdots & \vec{p}_{n}\\
1 & 1 & \cdots & 1
\end{bmatrix}\neq0\,.
\end{equation}

This condition ensures that two points in $\mathbb R$ do not coincide, three points in $\mathbb R^{2}$ are not collinear, four points in $\mathbb R^{3}$ are not coplanar, and so on. Thus, a simplex in $\mathbb R$ is a line segment, in $\mathbb R^{2}$ it is a triangle, while a simplex in $\mathbb R^{3}$ is a tetrahedron; in each case it encloses a finite $n$-dimensional volume.

Suppose that we wish to minimize $f\left(\vec{x}\right),\,\vec{x}\in\mathbb R^{n}$.
To start the algorithm, we initialize a simplex of $n+1$ points.
A possible way to set up a simplex is to start with an initial point
$\vec{x}^{\left(0\right)}=\vec{p}_{0}$ and generate the remaining
points of the initial simplex as follows:

\begin{equation}
\vec{p}_{i}=\vec{p}_{0}+\lambda_{i}\vec{e}_{i},\quad i=1,2,\dots,n\:,
\end{equation}

where the $\vec{e}_{i}$ are unit vectors constituting the natural
basis of $\mathbb R^{n}$. The positive constant coefficients $\lambda_{i}$
are selected in such a way that their magnitudes reflect the length
scale of the optimization problem.

Our objective is to modify the initial simplex stage by stage so that
the resulting simplices converge toward the minimizer. At each iteration
we evaluate the function $f$ at each point of the simplex. In the
function minimization process, the point with the largest function
value is replaced with another point. The process for modifying the
simplex continues until it converges toward the function minimizer.

We now present the rules for modifying the simplex stage by stage.
To aid in our presentation, we use a two-dimensional example to illustrate
the rules. We begin by selecting the initial set of $n+1$ points
that are to form the initial simplex. We next evaluate $f$ at each
point and order the $n+1$ vertices to satisfy

\begin{equation}
f\left(\vec{p}_{0}\right)\leq f\left(\vec{p}_{1}\right)\leq\cdots\leq f\left(\vec{p}_{n}\right)\,.
\end{equation}

For the two-dimensional case we let $\vec{p}_{l}$, $\vec{p}_{n}$
and $\vec{p}_{s}$ denote the points of the simplex for which $f$
is largest, next largest and smallest. We next compute $\vec{p}_{g}$,
the _centroid_ (center of gravity) of the best $n$ points:

\begin{equation}
\vec{p}_{g}=\frac{1}{n}\sum_{i=0}^{n-1}\vec{p}_{i}\,.
\end{equation}

We then reflect the worst vertex, $\vec{p}_{n}$, in $\vec{p}_{g}$
using a _reflection coefficient_ $\rho>0$ to obtain the reflection
point

\begin{equation}
\vec{p}_{r}=\vec{p}_{g}+\rho\left(\vec{p}_{g}-\vec{p}_{n}\right)\,.
\end{equation}

```julia
using NativeSVG

Drawing(width = "12cm", height = "6cm", viewBox = "0 0 600 300", preserveAspectRatio="none") do
    circle(cx = "52", cy = "150", r = "5", fill = "black")
    circle(cx = "250", cy = "50", r = "5", fill = "black")
    circle(cx = "250", cy = "250", r = "5", fill = "black")
    circle(cx = "250", cy = "150", r = "5", fill = "black")
    circle(cx = "450", cy = "150", r = "5", fill = "black")
    polyline(
        points="50, 150 250, 150 250, 50 50, 150 250, 250 250, 150 450 150",
        fill="none", stroke="black", stroke_width="2.5"
    )
    latex("\\vec p_l", x = "40", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_n", x = "260", y = "40", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_s", x = "260", y = "240", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_g", x = "210", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_r", x = "440", y = "160", width = "60", height = "60", font_size = "16pt")
end
```

A typical value is $\rho=1$.

We proceed to evaluate $f$ at $\vec{p}_{r}$. If $f\left(\vec{p}_{0}\right)\leq f\left(\vec{p}_{r}\right)<f\left(\vec{p}_{n-1}\right)$,
then the point $\vec{p}_{r}$ replaces $\vec{p}_{l}$ to form a new
simplex and we terminate the iteration. We proceed to repeat the process.

If , however, $f\left(\vec{p}_{r}\right)<f\left(\vec{p}_{0}\right)$,
so that the point $\vec{p}_{r}$ yields the smallest function value
among the points of the simplex, we argue that this direction is a
good one. In this case we increase the distance traveled using an
_expansion coefficient_ $\chi>1$, e.g. $\chi=2$, to obtain

\begin{equation}
\vec{p}_{e}=\vec{p}_{g}+\chi\left(\vec{p}_{r}-\vec{p}_{g}\right)\,.
\end{equation}

```julia
Drawing(width = "16cm", height = "6cm", viewBox = "0 0 800 300", preserveAspectRatio="none") do
    circle(cx = "50", cy = "150", r = "5", fill = "black")
    circle(cx = "250", cy = "50", r = "5", fill = "black")
    circle(cx = "250", cy = "250", r = "5", fill = "black")
    circle(cx = "250", cy = "150", r = "5", fill = "black")
    circle(cx = "450", cy = "150", r = "5", fill = "black")
    circle(cx = "650", cy = "150", r = "5", fill = "black")
    polyline(
        points="50, 150 250, 150 250, 50 50, 150 250, 250 250, 150 650 150",
        fill="none", stroke="black", stroke_width="2.5"
    )
    latex("\\vec p_l", x = "40", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_n", x = "260", y = "40", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_s", x = "260", y = "240", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_g", x = "210", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_r", x = "440", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_e", x = "640", y = "160", width = "60", height = "60", font_size = "16pt")
end
```

This operation yields a new point on the line $\vec{p}_{n}\vec{p}_{g}\vec{p}_{r}$
extend beyond $\vec{p}_{r}$. If $f\left(\vec{p}_{e}\right)<f\left(\vec{p}_{r}\right)$,
the expansion is declared a success and $\vec{p}_{e}$ replaces $\vec{p}_{n}$
in the next simplex. If, on the other hand, $f\left(\vec{p}_{e}\right)\geq f\left(\vec{p}_{r}\right)$,
the expansion is a failure and $\vec{p}_{r}$ replaces $\vec{p}_{n}$.

Finally, if $f\left(\vec{p}_{r}\right)\geq f\left(\vec{p}_{n-1}\right)$,
the reflected point $\vec{p}_{r}$ would constitute the point with
the largest function value in the new simplex. Then in the next step
it would be reflected in $\vec{p}_{g}$, probably an unfruitful operation.
Instead, this case is dealt with by a _contraction_ operation
in one of two ways. First, if $f\left(\vec{p}_{r}\right)<f\left(\vec{p}_{n}\right)$,
then we contract $\left(\vec{p}_{r}-\vec{p}_{g}\right)$ with a _contraction
coefficient_ $0<\gamma<1$, e.g. $\gamma=\frac{1}{2}$, to obtain

```julia
Drawing(width = "16cm", height = "6cm", viewBox = "0 0 800 300", preserveAspectRatio="none") do
    circle(cx = "50", cy = "150", r = "5", fill = "black")
    circle(cx = "250", cy = "50", r = "5", fill = "black")
    circle(cx = "250", cy = "250", r = "5", fill = "black")
    circle(cx = "250", cy = "150", r = "5", fill = "black")
    circle(cx = "450", cy = "150", r = "5", fill = "black")
    circle(cx = "350", cy = "150", r = "5", fill = "black")
    polyline(
        points="50, 150 250, 150 250, 50 50, 150 250, 250 250, 150 450 150",
        fill="none", stroke="black", stroke_width="2.5"
    )
    latex("\\vec p_l", x = "40", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_n", x = "260", y = "40", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_s", x = "260", y = "240", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_g", x = "210", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_r", x = "440", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_c", x = "340", y = "160", width = "60", height = "60", font_size = "16pt")
end
```

\begin{equation}
\vec{p}_{c}=\vec{p}_{g}+\gamma\left(\vec{p}_{r}-\vec{p}_{g}\right)\,.
\end{equation}

We refer to this operation as the _outside contraction_. If,
on the other hand, $f\left(\vec{p}_{r}\right)\geq f\left(\vec{p}_{n}\right)$,
then $\vec{p}_{n}$ replaces $\vec{p}_{r}$ in the contraction operation
and we get

\begin{equation}
\vec{p}_{c}=\vec{p}_{g}+\gamma\left(\vec{p}_{n}-\vec{p}_{g}\right)\,.
\end{equation}

This operation is referred to as the _inside contraction_.

```julia
Drawing(width = "16cm", height = "6cm", viewBox = "0 0 800 300", preserveAspectRatio="none") do
    circle(cx = "50", cy = "150", r = "5", fill = "black")
    circle(cx = "250", cy = "50", r = "5", fill = "black")
    circle(cx = "250", cy = "250", r = "5", fill = "black")
    circle(cx = "250", cy = "150", r = "5", fill = "black")
    circle(cx = "150", cy = "150", r = "5", fill = "black")
    polyline(
        points="50, 150 250, 150 250, 50 50, 150 250, 250 250, 150",
        fill="none", stroke="black", stroke_width="2.5"
    )
    latex("\\vec p_l", x = "40", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_n", x = "260", y = "40", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_s", x = "260", y = "240", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_g", x = "210", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_c", x = "140", y = "160", width = "60", height = "60", font_size = "16pt")
end
```

If , in either case, $f\left(\vec{p}_{c}\right)\leq f\left(\vec{p}_{n}\right)$,
the contraction is considered a success, and we replace $\vec{p}_{n}$
with $\vec{p}_{c}$ in the new simplex. If, however, $f\left(\vec{p}_{c}\right)>f\left(\vec{p}_{n}\right)$,
the contraction is a failure, and in this case a new simplex can be
formed by retaining $\vec{p}_{0}$ only and halving the distance from
$\vec{p}_{0}$ to every point in the simplex. We can refer to this
event as a _shrinkage operation_. In general, the shrink step
produces the $n$ new vertices of the new simplex according to the
formula

\begin{equation}
\vec{v}_{i}=\vec{p}_{0}+\sigma\left(\vec{p}_{i}-\vec{p}_{0}\right)\,,\quad i=1,2,\dots,n\,,
\end{equation}

```julia
Drawing(width = "16cm", height = "6cm", viewBox = "0 0 800 300", preserveAspectRatio="none") do
    circle(cx = "50", cy = "150", r = "5", fill = "black")
    circle(cx = "250", cy = "50", r = "5", fill = "black")
    circle(cx = "250", cy = "250", r = "5", fill = "black")
    circle(cx = "250", cy = "150", r = "5", fill = "black")
    circle(cx = "150", cy = "200", r = "5", fill = "black")
    polyline(
        points="250, 150 250, 50 50, 150 250, 250 250, 150 150, 200",
        fill="none", stroke="black", stroke_width="2.5"
    )
    latex("\\vec p_l", x = "40", y = "160", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_n", x = "260", y = "40", width = "60", height = "60", font_size = "16pt")
    latex("\\vec p_s", x = "260", y = "240", width = "60", height = "60", font_size = "16pt")
end
```

where $\sigma=\frac{1}{2}$. Hence, the vertices of the new simplex
are $\vec{p}_{0},\vec{v}_{1},\dots,\vec{v}_{n}$.

When implementing the simplex algorithm, we need a tie-breaking rule
to order points in the case of equal function values. Often, the highest
possible index consistent with the relation

\begin{equation}
f\left(\vec{p}_{0}\right)\leq f\left(\vec{p}_{1}\right)\leq\cdots\leq f\left(\vec{p}_{n}\right)
\end{equation}

is assigned to the new vertex.

## Example

To show how the `Optim` package can be used, we minimize the Rosenbrock function, a classical test problem for numerical optimization. We'll assume that you've already installed the Optim package using Julia's package manager. First, we load Optim and define the Rosenbrock function:

```julia
#using Pkg
#pkg"add Optim"

using Optim
using Plots

f(x)= (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2;

x= -1:0.007:2; y= -1:0.007:2; z= Surface((x,y)->log10.(f([x,y])), x, y);

p = contour(x,y,z)
```

Once we've defined this function, we can find the minimizer (the input that minimizes the objective) and the minimum (the value of the objective at the minimizer) using any of our favorite optimization algorithms. With a function defined, we just specify an initial point x and call optimize with a starting point `x0`:

```julia
x0 = [0.0, 0.0]
optimize(f, x0, NelderMead())
```

Other local solvers are available. Below, we use BFGS, a quasi-Newton method that requires a gradient. If we pass f alone, Optim will construct an approximate gradient for us using central finite differencing:

```julia
optimize(f, x0, BFGS())
```

For better performance and greater precision, you can pass your own gradient function. If your objective is written in all Julia code with no special calls to external (that is non-Julia) libraries, you can also use automatic differentiation, by using the `autodiff` keyword and setting it to `:forward`:

```julia
optimize(f, x0, BFGS(); autodiff = :forward)
```

For the Rosenbrock example, the analytical gradient can be shown to be:

```julia
function g!(G, x)
G[1] = -2.0 * (1.0 - x[1]) - 400.0 * (x[2] - x[1]^2) * x[1]
G[2] = 200.0 * (x[2] - x[1]^2)
end
```

Note, that the functions we're using to calculate the gradient (and later the Hessian `h!`) of the Rosenbrock function mutate a fixed-sized storage array, which is passed as an additional argument called `G` (or `H` for the Hessian) in these examples. By mutating a single array over many iterations, this style of function definition removes the sometimes considerable costs associated with allocating a new array during each call to the `g!` or `h!` functions.

```julia
optimize(f, g!, x0, BFGS())
```

In addition to providing gradients, you can provide a Hessian function `h!` as well. In our current case this is

```julia
function h!(H, x)
    H[1, 1] = 2.0 - 400.0 * x[2] + 1200.0 * x[1]^2
    H[1, 2] = -400.0 * x[1]
    H[2, 1] = -400.0 * x[1]
    H[2, 2] = 200.0
end
```

Now we can use Newton's method for optimization by running:

```julia
optimize(f, g!, h!, x0)
```

Which defaults to `Newton()` since a Hessian function was provided. Like gradients, the Hessian function will be ignored if you use a method that does not require it.

Note that `Optim` will not generate approximate Hessians using finite differencing because of the potentially low accuracy of approximations to the Hessians. Other than Newton's method, none of the algorithms provided by the Optim package employ exact Hessians.

## Algorithms

The first choice to be made is that of the order of the method. Zeroth-order methods do not have gradient information, and are very slow to converge, especially in high dimension. First-order methods do not have access to curvature information and can take a large number of iterations to converge for badly conditioned problems. Second-order methods can converge very quickly once in the vicinity of a minimizer. Of course, this enhanced performance comes at a cost: the objective function has to be differentiable, you have to supply gradients and Hessians, and, for second order methods, a linear system has to be solved at each step.

If you can provide analytic gradients and Hessians, and the dimension of the problem is not too large, then second order methods are very efficient. The _Newton method with trust region_ is the method of choice.

When you do not have an explicit Hessian or when the dimension becomes large enough that the linear solve in the Newton method becomes the bottleneck, first order methods should be preferred. _BFGS_ is a very efficient method, but also requires a linear system solve. _L-BFGS_ usually has a performance very close to that of BFGS, and avoids linear system solves (the parameter `m` can be tweaked: increasing it can improve the convergence, at the expense of memory and time spent in linear algebra operations). The _conjugate gradient method_ usually converges less quickly than L-BFGS, but requires less memory. _Gradient descent_ should only be used for testing.

When the objective function is non-differentiable or you do not want to use gradients, use zeroth-order methods. _Nelder-Mead_ is currently the most robust.

## Linesearches

Linesearches are used in every first- and second-order method except for the trust-region Newton method. Linesearch routines attempt to locate quickly an approximate minimizer of the univariate function

\begin{equation}
\alpha \rightarrow f\left(\vec x + \alpha \vec d\right)\,,
\end{equation}

where $\vec d$ is the descent direction computed by the algorithm. They vary in how accurate this minimization is. Two good linesearches are BackTracking and HagerZhang, the former being less stringent than the latter. For well-conditioned objective functions and methods where the step is usually well-scaled (such as L-BFGS or Newton), a rough linesearch such as BackTracking is usually the most performant. For badly behaved problems or when extreme accuracy is needed (gradients below the square root of the machine epsilon, about `10−8` with Float64), the HagerZhang method proves more robust. An exception is the conjugate gradient method which requires an accurate linesearch to be efficient, and should be used with the HagerZhang linesearch.

## Summary

As a very crude heuristic:

For a low-dimensional problem with analytic gradients and Hessians, use the Newton method with trust region. For larger problems or when there is no analytic Hessian, use L-BFGS, and tweak the parameter `m` if needed. If the function is non-differentiable, use Nelder-Mead.

Use the `HagerZhang` linesearch for robustness and `BackTracking` for speed.

Requires only a function handle:

- `NelderMead()`
- `SimulatedAnnealing()`, see next year
- `ParticleSwarm()`, see next year

Requires a function and gradient (will be approximated if omitted):

- `BFGS()`
- `LBFGS()`
- `ConjugateGradient()`
- `GradientDescent()`

Requires a function, a gradient, and a Hessian (cannot be omitted):

- `Newton()`
- `NewtonTrustRegion()`

Special methods for bounded univariate optimization:

- `Brent()`
- `GoldenSection()`


## Numerical Differentiation using finite-differences

```python
import numpy as np
import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib import cm
import seaborn as sns
from scipy.integrate import quad
sns.set_context("talk", font_scale=1.5, rc={"lines.linewidth": 2.5})
sns.set_style("whitegrid")
import sympy as sp
from IPython.display import HTML
from matplotlib import animation
%matplotlib inline

# Don't tinker, or do
#%matplotlib nbagg
# from matplotlib import rcParams
#rcParams['font.family']='sans-serif'
#rcParams('font', serif='Helvetica Neue')
# rcParams['text.usetex']= True
#rcParams.update({'font.size': 22})
```

## Numerical differentiation schemes
Differentiation is another fundamental operation in calculus, and involves finding the slope of a certain curve (geometrically) at some point. This corresponds to the rate of change (or expected growth or decay of the function). We look at some schemes/algorithms to do estimate the slope, similar to our time-stepper lecture. The problem we deal with is then to approximate
$$  \frac{df}{dx}(x) \approx \sum_{i=1}^{n} \omega_i f(x_i) $$
given at least a \( C^{1} \) function $f : [a,b] \to \mathbb{R}$ with *nodes* ($x_i$) and *weights* ($\omega_i$). Schemes that we will be discussing involve careful choices of these nodes and weights.

To get started, let us define a class that helps us implement any quadrature rule easily.

```python
class SpatialDifferentiator(object):
    """ Class for wrapping a diff scheme with other goodies
    """
    def __init__(self, i_a, i_b, i_h):
        """ Initialize the Differentiator
        """
        # What forcing function are we using?
        self.forcing = None

        # What differentation algorithm are we using?
        self.differentiator = None

        self.a = i_a
        self.b = i_b
        self.h = i_h
        self.nsteps = int((i_b - i_a)/i_h)

    def set_forcing_function(self, t_func):
        """ Set forcing function to be used
        """
        if type(t_func) is not str:
            try:
                # If not string, try and evaluate the function
                t_func(0.0)
                # If the function works, set this function as forcing
                self.forcing = t_func
            except:
                raise RuntimeError('Provided function cannot be evaluated')

    def analytic(self, x):
        """  For testing convergence, defined as a special function """
        analytical_differential = (self.forcing(x)[1])
        return analytical_differential

    def diff_using(self, differentiator, renderer):
        """  Differentiates the function and spits out the relative error
        """
        self.differentiator = differentiator.__name__
        # Periodic domain, b not included
        all_x = np.linspace(self.a, self.b, self.nsteps, endpoint=False)
        differential = differentiator(self.forcing, all_x, self.h)

        analytic_differential = self.analytic(all_x)
        # print("Integral is {:20.16f}, Analytical integral is {:20.16f}, Error is {:20.16f}".format(integral, analytical_integral, np.abs(integral - analytical_integral)))

        # Pass as (2xn) arrays. While reshaping do T to get (nx2) arrays
        self.draw(renderer, np.hstack((all_x, [self.b])), np.hstack((differential, differential[0])))
        err = np.max(np.abs((analytic_differential - differential))) / np.max(np.abs(analytic_differential))
        return err

    def draw(self, renderer, eval_x, num_diff):
        """ Draw the matplotlib canvas with the portrait we want
        """
        # If there is a timestepper, then there is numerical data
        # Plot them
        renderer.plot(eval_x, self.forcing(eval_x)[0], 'g-', label=r'$f(x)$')
        renderer.plot(eval_x, self.forcing(eval_x)[1], 'r-', label=r'$\frac{df(x)}{dx}$')
        renderer.plot(eval_x, num_diff, label=r'Numerical')
        renderer.legend()
        renderer.set_xlabel(r'$x$')
        renderer.set_ylabel(r'$y$')
        renderer.set_title('{}'.format(self.differentiator))
        # renderer.set_aspect('equal')
```

## Finite-difference differentiation

A variety of rules for differentiation exist, obtained by mainpulating the Taylor series expansion around the point of interest. We look to approximate the slope (first order of diff) and the curvature (second order of differentiation) using many types of schemes: (a) Forward (b) Backward and (c) Central differences, although others (spectral) will be discussed on the way. We will attempt to compare these methods in terms of their ease (in understanding/implementation), order of accuracy (in comparison to the discretization $h$) and function evaluations for each step $h$.

## What do these rules do?
They approximate the slope of the curve in a smart way---using local polynomials (in the case of finite-differences), you approximate the curve and differentiate these polynomials instead. We'll first look at some  schemes for obtaining first order derivatives.

We first look at two point derivatives (i.e. only two function evaluations to estimate the derivative)

```python
def fdiff_oneorder_twopoint(func, x, h):
    """Does forward differences to approximate the slope
    in a periodic domain

    Parameters
    ----------
    func : ufunc
        A function, f, that takes one argument and
        returns the function value and exact differential
        as a tuple
    x : float/array-like
        Points at which the function is sampled
    h : float/array-like
        The spacing between nodes

    Returns
    -------
    differential : float/array-like
        Approximation of the diff using quadrature
    """
    # Evaluate function
    fx = func(x)[0]

    # Periodic domain
    # (f(i+1)-f(i))/h
    df_dx = (np.roll(fx, -1) - fx) / (h)

    return df_dx

def bdiff_oneorder_twopoint(func, x, h):
    """Does backward differences to approximate the slope
    in a periodic domain

    Parameters
    ----------
    func : ufunc
        A function, f, that takes one argument and
        returns the function value and exact differential
        as a tuple
    x : float/array-like
        Points at which the function is sampled
    h : float/array-like
        The spacing between nodes

    Returns
    -------
    differential : float/array-like
        Approximation of the diff using quadrature
    """
    # Evaluate function
    fx = func(x)[0]

    # Periodic domain
    # (f(i)-f(i-1))/h
    df_dx = (fx - np.roll(fx, 1)) / (h)

    return df_dx

def cdiff_oneorder_twopoint(func, x, h):
    """Does centered differences to approximate the slope
    in a periodic domain

    Parameters
    ----------
    func : ufunc
        A function, f, that takes one argument and
        returns the function value and exact differential
        as a tuple
    x : float/array-like
        Points at which the function is sampled
    h : float/array-like
        The spacing between nodes

    Returns
    -------
    differential : float/array-like
        Approximation of the diff using quadrature
    """
    # Evaluate function
    fx = func(x)[0]

    # Periodic domain
    # (f(i+1)-f(i-1))/2h
    df_dx = (np.roll(fx, -1) - np.roll(fx, 1)) / (2*h)

    return df_dx
```

## Testing diff?
Let's test these rules out between $[a,b]$ for simple periodic functions (trigonometric) curves and draw inferences from them (the error and so on...). The choice of periodic domain is not arbitrary---we do not need to worry about boundary conditions and so on...

```python
# Bounds of function definition
a = 0
b = 1

# Step size of differentiation - change
# h = (b-a)/1
h = (b-a)/16
# h = (b-a)/50
# h = (b-a)/100

# Define the periodic function to be differentiated and
# its exact differential - change
def periodic_func(x):
#     # Periodic function
#     return np.sin(2.*np.pi*x), 2.*np.pi*np.cos(2.*np.pi*x)

    # Periodic function
    k = 5
    return np.sin(2.*np.pi*k*x), 2.*np.pi*k*np.cos(2.*np.pi*k*x)
```

```python
fig, ax = plt.subplots(1,1, figsize=(8, 8))
si = SpatialDifferentiator(a, b, h)
si.set_forcing_function(periodic_func)

# Differnt schemes, change
#si.diff_using(fdiff_oneorder_twopoint, ax)
#si.diff_using(bdiff_oneorder_twopoint, ax)
si.diff_using(cdiff_oneorder_twopoint, ax)
```

### Things to observe
- How does the spacing $h$ affect the derivative (visually)
- What about the choice of function?
- How do the central/backward/forward schemes compare?

## Higher order approximations
We then look at six point schemes to estimate the first derivative (i.e. six function evaluations to estimate the derivative). Look at https://en.wikipedia.org/wiki/Finite_difference_coefficient or http://web.media.mit.edu/~crtaylor/calculator.html for obtaining the coefficients in such schemes.

```python
def fdiff_oneorder_sixpoint(func, x, h):
    # Evaluate function
    fx = func(x)[0]

    # Periodic domain
    df_dx = (-137.*fx + 300.*np.roll(fx, -1) - 300.*np.roll(fx, -2) + 200.*np.roll(fx, -3) - 75.*np.roll(fx, -4) + 12.*np.roll(fx, -5)) / (60.*h)

    return df_dx

def bdiff_oneorder_sixpoint(func, x, h):
    # Evaluate function
    fx = func(x)[0]

    # Periodic domain
    df_dx = (137.*fx - 300.*np.roll(fx, 1) + 300.*np.roll(fx, 2) - 200.*np.roll(fx, 3) + 75.*np.roll(fx, 4) - 12.*np.roll(fx, 5)) / (60.*h)

    return df_dx

def cdiff_oneorder_sixpoint(func, x, h):
    # Evaluate function
    fx = func(x)[0]

    # Periodic domain
    df_dx = (-np.roll(fx, 3) + 9. * np.roll(fx, 2) - 45. * np.roll(fx, 1) + 45. * np.roll(fx, -1) - 9. * np.roll(fx, -2) + np.roll(fx, -3) ) / (60.0 * h)

    return df_dx
```

```python
# Bounds of function definition
a = 0
b = 1

# Step size of differentiation
h = (b-a)/16
```

```python
fig, ax = plt.subplots(1,1, figsize=(8, 8))
si = SpatialDifferentiator(a, b, h)
si.set_forcing_function(periodic_func)
si.diff_using(cdiff_oneorder_sixpoint, ax)
```

### Things to observe
- How does the relative error change when more point stencils are used?

## Other ways to approximate derivatives

Since we live in a periodic domain, we can spectrally differentiate functions using Fourier transforms. This is just for demonstration, and will not be covered in detail in the class.

```python
def spectral_oneorder(func, x, h):
    # Evaluate function
    fx = func(x)[0]
    f_hat = np.fft.fft(fx)

    n_samples = x.shape[0]
    w_hat = 1j*np.zeros(n_samples)
    k_prefac = 2*np.pi/(n_samples*h)
    # 0 represents 0, then postive and then negative
    w_hat[:n_samples//2] = 1j*np.arange(0, n_samples//2, 1)*k_prefac
    # Middle one is zero, so leave as is
    w_hat[n_samples//2:] = 1j*np.arange(-n_samples//2, 0, 1)*k_prefac

    df_dx_hat = w_hat*f_hat
    df_dx = np.real(np.fft.ifft(df_dx_hat))

    return df_dx
```

```python
# Bounds and step size
a = 0
b = 1
h = (b-a)/16
```

```python
fig, ax = plt.subplots(1,1, figsize=(10, 10))
si = SpatialDifferentiator(a, b, h)
si.set_forcing_function(periodic_func)
si.diff_using(spectral_oneorder, ax)
```

Isn't that awesome?

## Order of accuracy of finite-difference schemes
What's **order** of accuracy? Order of accuracy quantifies the rate of convergence of a numerical approximation of a differential to the exact differential.
The numerical solution ${u}$ is said to be $n^{\text{th}}$-order accurate if the error, $e(h):=\lVert\tilde{{u}}-{u} \rVert$ is proportional to the discretization-size $ h $, to the $n^{\text{th}}$ power. That is

$$ e(h)=\lVert\tilde{{u}}-{u} \rVert\leq C(h)^{n} $$

Details of this are given in the slides. Here, we focus on differentiating a periodic function and figure out the order of convergence. The model problem that we deal with is to find
$$ y = \frac{d \sin(2\pi kx)}{dx} $$
which as we know has the analytical solution $ \tilde{y} = 2\pi k \cdot \cos(2\pi kx) $, and so error can be calculated.

```python
a = 0
b = 1

# All functions that you coded up
# impl_list = [fdiff_oneorder_twopoint, bdiff_oneorder_twopoint, cdiff_oneorder_twopoint, fdiff_oneorder_sixpoint, bdiff_oneorder_sixpoint, cdiff_oneorder_sixpoint, spectral_oneorder]
# impl_list = [fdiff_oneorder_twopoint, bdiff_oneorder_twopoint, cdiff_oneorder_twopoint]
impl_list = [fdiff_oneorder_twopoint, bdiff_oneorder_twopoint, cdiff_oneorder_twopoint, fdiff_oneorder_sixpoint, bdiff_oneorder_sixpoint, cdiff_oneorder_sixpoint]

# Time steps and associated errors from 2^0 to 2^(-10)
h_steps = np.arange(1, 9, dtype=np.int16)
errors_list = [[None for i in h_steps] for impl in impl_list]

# Run simulations and collect errors
for i_impl, impl in enumerate(impl_list):
    for i_step, step in enumerate(h_steps):
        new_h = (b-a)/(8*(step))
        si = SpatialDifferentiator(a, b, new_h)
        si.set_forcing_function(periodic_func)
        errors_list[i_impl][i_step] = si.diff_using(impl, ax)
```

```python
# Draw error plots in a log-log plot
fig, ax = plt.subplots(1,1, figsize=(10, 10))

# x axis is time, y axis is error
for i_impl, impl in enumerate(impl_list):
    ax.plot((b-a)/(8.*h_steps), errors_list[i_impl], 'o-', label=impl.__name__)

# Draw helpful slope lines to compare
# x_ax = (b-a)/(8.*h_steps)
# ax.plot(x_ax, 1.2 * x_ax**2, 'k--')
# ax.plot(x_ax, 0.5 * x_ax**2, 'k--')
# ax.plot(x_ax, 0.015 * x_ax**4, 'k--')

# Make it readable
ax.set_xlabel(r'$h$')
ax.set_ylabel(r'$e(h)$')
ax.set_title('Order of accuracy')
ax.set_yscale('log')
ax.set_xscale('log')
ax.legend(loc='best')
# fig.savefig('ooa.pdf')
```

```python
a = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
```

```python
a
```

```python
aplusone = np.roll(a, -1)
aplusone
```

```python
np.roll(a, 1)
```

```python

```

```python
a = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
a
```

```python
b = np.pad(a, (2,2), 'constant', constant_values=(10,15))
b
```

```python

```


## Informal Introduction

Recursion Relation is also know as **Recursive Formula**. Informal definition of recursion relation is the way to define series that each item defined by the previous items. But, to construct a complete recursion series, we need to define initial terms as precondition at first place. By its construction, it is easier to define rather than to answer what value of n in the series.
* Fibonacci recurrence relation is:

$$F(0) = 0, F(1) = 1, F(n) = F(n-1) + F(n-2) \ \text{if} \ n \geq 2$$

Formula for \\(F(n)\\) which involve only n is:

$$F(n) = \bigl(\frac{\Phi^n - (-\phi)^n)}{\sqrt5}\bigr) \ \text{where} \ \Phi = \frac{(\sqrt5+1)}{2} \text{and} \ \phi=\frac{(\sqrt5-1)}{2}$$

* Find unique **derangements** of \\(n\\) people sit in \\(\{s_i, s_j, ..., s_n\}\\) where \\(i \neq j\\).
This problem has direct factorial formula (defined in Taylor series):

$$D(n) = n!(\frac{1}{2!} - \frac{1}{3!} + \frac{1}{4!} - ... \frac{(-1)^n}{n!}), \ if \ n > 2$$

The recursion formula is:

$$D(n) = (n - 1) (D(n-1) + D(n-2))$$

Formula for \\(D(n)\\) is:

$$D(n) = n D(n-1) + (-1)^n$$

Further introductory can be explored in [Recurrence Relations & Generating Functions by R. Knott](http://www.maths.surrey.ac.uk/hosted-sites/R.Knott/Fibonacci/LRGF.html)

## Geometric Series

For \\(r \neq 1\\), the sum of the first n of geometric series is:

$$a+ar+ar^{2}+ar^{3}+ \cdots +ar^{n-1}=\sum_{k=0}^{n-1}ar^{k}=a\left({\frac{1-r^{n}}{1-r}}\right)$$

**PROOF**: *a* is initial term and *r* is common ratio.

$$\eqalign{
s&=a+ar+ar^{2}+ar^{3}+\cdots +ar^{n-1},\\
rs&=ar+ar^{2}+ar^{3}+ar^{4}+\cdots +ar^{n},\\
s-rs&=a-ar^{n},\\s(1-r)&=a(1-r^{n}),
}$$

So:

$${\displaystyle s=a\left({\frac {1-r^{n}}{1-r}}\right)\quad {\text{(if }}r\neq 1{\text{)}}.}$$

As *n* goes to infinity, the absolute value of *r* must be less than one for the series to converge. The sum then becomes:

$$a+ar+ar^{2}+ar^{3}+ar^{4}+\cdots =\sum _{k=0}^{\infty }ar^{k}={\frac {a}{1-r}},{\text{ for }}|r|<1.$$

When *a = 1*, this can be simplified to:

$$1\,+\,r\,+\,r^{2}\,+\,r^{3}\,+\,\cdots \;=\;{\frac {1}{1-r}},$$

the left-hand side being a geometric series with common ratio *r*.

The formula also holds for complex *r*, with the corresponding restriction, the modulus of *r* is strictly less than one.

**Proof of Convergence**

We can prove that the geometric series converges using the sum formula for a geometric progression:

$${\displaystyle {\begin{aligned}1+r+r^{2}+r^{3}+\cdots \ &=\lim _{n\rightarrow \infty }\left(1+r+r^{2}+\cdots +r^{n}\right)\\&=\lim _{n\rightarrow \infty }{\frac {1-r^{n+1}}{1-r}}.\end{aligned}}}$$

Since \\((1 + r + r^2 + ... + r^n)(1−r) = 1−r^{n+1}\\) and \\(r^{n+1} \rightarrow 0 \ \text{for} \ |r| < 1\\)

Convergence of geometric series can also be demonstrated by rewriting the series as an equivalent telescoping series. Consider the function,

$$g(K)={\frac {r^{K}}{1-r}}$$

Note that

$${\displaystyle 1=g(0)-g(1)\ ,\ r=g(1)-g(2)\ ,\ r^{2}=g(2)-g(3)\ ,\ldots }$$

Thus,

$${\displaystyle S=1+r+r^{2}+r^{3}+\cdots =(g(0)-g(1))+(g(1)-g(2))+(g(2)-g(3))+\cdots .}$$

If,

$$|r| < 1$$

then

$$g(K)\longrightarrow 0{\text{ as }}K\to \infty$$

so *S* converges to:

$$g(0)={\frac {1}{1-r}}$$

## Binomial Theorem (Factorial Method)

Let \\(n\\) be a positive integer, and \\(x\\) and \\(y\\) real numbers (or complex numbers, or polynomials). The coefficient of \\(x^ky^{n-k}\\), in \\(k^{th}\\) term in the expansion of \\((x+y)^n\\), is equal to \\({n \choose k}\\), where:

$${n \choose k} = \frac{n!}{(n-k)!k!}$$

So the **Binomial Expansion** is:

$$(x+y)^n = \sum_{r=0}^n {n \choose r} x^{n-r} y^r = \sum_{r = 0}^n {n \choose r} x^r y^{n-r}$$

**PROOF**:

$$
\begin{align}
    (x+y)^n &= (x+y)(x+y)^{n-1} \\
            &= (x+y)\bigg(\binom{n-1}{0} x^{n-1} + \binom{n-1}{1} x^{n-2}y + \cdots + \binom{n-1}{n-1}y^{n-1}\bigg) \\
            &= x^n + \left( \binom{n-1}{0} + \binom{n-1}{1} \right) x^{n-1}y + \left( \binom{n-1}{1} + \binom{n-1}{2} \right) x^{n-2}y^2 \phantom{=} + \cdots + \left(\binom{n-1}{n-2} + \binom{n-1}{n-1} \right) xy^{n-1} + y^n \\
\end{align}
$$

and now Pascal's identity applies:

$$\binom{n-1}{k-1}+\binom{n-1}{k} = \binom{n}{k}$$

So the right side simplifies to:

$$x^n + \binom{n}{1} x^{n-1}y + \binom{n}{2} x^{n-2}y^2 + \cdots + \binom{n}{n-1}xy^{n-1} + y^n$$

## First Order Recurrences

The simplest type of recurrence reduce immediately to a product. The recurrence:

$$a_n = x_na_{n-1} \qquad \text{for} \ n>0 \ \text{with} \ a_0 = 1$$

equivalent to:

$$a_n = \prod_{1 \leq k \leq n} x_k$$

**Theorem 2.1 (First Order linear Recurrences)**

The recurrence

$$a_n = x_na_{n-1} + y_n \qquad \text{for} \ n>0 \ \text{with} \ a_0 = 0$$

has the explicit solution:

$$a_n = y_n + \sum_{1 \leq j < n}y_jx_{j+1}x_{j+2}...x_n$$

**PROOF**. Dividing both sides by \\(x_nx_{n-1}...x_1\\) and iterating, we have:

$$
\eqalign{
a_n &= x_nx_{n-1}...x_1 \sum_{1 \leq j \leq n}\frac{y_j}{x_jx_{j-1}...x_1}\\
    &= y_n + \sum_{1 \leq j < n}y_jx_{j+1}x_{j+2}...x_n
}
$$

## Nonlinear First-Order Recurrences

**Simple convergence**. One convincing reason to calculate initial values is that many recurrences with a complicated appearance simply converge to a constant. For example:

$$a_n = \frac{a}{(1+a_{n-1})} \qquad for \ n>0 \ with \ a_0 = 1$$

This is a so-called continued fraction equation, which is discussed in §2.5. By calculating initial values, we can guess that the recurrence converges to a constant:

Each iteration increases the number of significant digits available by a constant number of digits (about half a digit). is is known as simple convergence. If we assume that the recurrence does converge to a constant, we know that the constant must satisfy \\(\alpha = \frac{1}{(1+\alpha)}\\), or \\(1 - \alpha - \alpha^2 = 0\\), which leads to the solution \\(\alpha = \frac{(\sqrt{5}-1)}{2} \approx .6180334\\).

## Higher-Order Recurrences

Recurrences where the right-hand side of the equation for \\(a_n\\) is a linear combination of \\(a_{n-1}, a_{n-2},\\) and so on.

For example:

$$a_n = 3a_{n-1} - 2a_{n-2} \qquad for \ n>1 \ with \ a_0 = 0 \ and \ a_1=1$$

Solution:

$$
\eqalign{
a_n - a_{n-1}   &= 2(a_{n-1} - 2_{n-2})\\
                &= 2^{n-1}\\
a_n             &= 2^{n}-1\\
}
$$

Another example:

$$a_n = 5a_{n-1} - 6a_{n-1} \qquad for \ n>1 \ with \ a_0 = 0 \ and \ a_1=1$$

Has solution:

$$
\eqalign{
a_n - 3a_{n-1}   &= 2(a_{n-1} - 3_{n-2})\\
                 &= 2^{n-1}\\
a_n              &= 3^n - 2^n
}
$$

**Theorem 2.2 (Linear recurrences with constant coefficients)**

$$a_n = x_1a_{n-1}+x_2a_{n-2}+...+x_ta_{n-t} \qquad for \ n \geq \ t$$

Can be expressed as a linear combination (with coefficients depending on the initial conditions \\(a_0,a_1,...,a_{t-1}\\)):

$$q(z) \equiv z^t - x_1z^{t-1} - x_2z^{t-2} - ... - x_t$$

**PROOF**. For \\(a_n = \beta^n\\), where \\(\beta\\) is a root of the "characteristic polynomial":

$$\beta^n = x_1\beta^{n-1} + x_2\beta^{n-2} + ... + x_t\beta^{n-t} \qquad for \ n \geq t$$

or, equivalently,

$$\beta^{n-t}q(\beta) = 0$$

That is, \\(\beta^n\\) is a solution to the recurrence for any root \\(\beta\\) of the characteristic polynomial.

**Finding the coefficients**. Develop theorem 2.2 to create a system of simultaneously equations that can be solved to yield the constants in the linear combination. For example:

$$a_n = 5a_{n-1} - 6a_{n-2} \qquad for \ n \geq 2 \ with a_0 = 0 \ and \ a_1 = 1$$

The characteristic equation is \\(z^2 - 5z + 6 = (z-3)(z-2)\\), so:

$$a_n = c_03^n + c_12^n$$

Matching this formula against the values at \\(n = 0\\) and \\(n = 1\\), we have:

$$a_0 = 0 = c_0 + c_1$$
$$a_1 = 1 = 3c_0 + 2c_1$$

The solution to these simultaneous equations is \\(c_0 = 1\\) and \\(c_1 = -1\\), so \\(a_n = 3^n - 2^n\\)

**Degenerate cases**. When the coefficients turn out to be zero and/or some roots have the same modulus, the result can be somewhat counterintuitive. For example:

$$a_n = 2a_{n-1} - a_{n-2} \qquad for \ n \geq 2 \ with \ q_0 = 1 \ and \ a_1 = 2$$

The characteristic equation is \\(z^2 - 2z + 1 = (z-1)^2\\), so:

$$a_n = c_01^n + c_1n1^n$$

We have:

$$a_0 = 1 = c_0$$
$$a_1 = 2= c_0 + c_1$$

gives \\(c_0 = c_1 = 1\\), so \\(a_n = n + 1\\).

**Fibonacci numbers**. \\({0, 1, 1, 2, 3, 4, 8, 13, ...}\\) is defined by prototypical second-order recurrence:

$$F_n = F_{n-1} + F_{n-2} \qquad for \ n>1 \ with \ F_0 = 0 \ and \ F_1 = 1$$

roots of \\(u^2-u-1 = \phi = (1+\sqrt(5)/2 = 1.61803..\\) and \\(\hat{\phi} = (1-\sqrt(5)/2 = -.61803...)\\), Theorem 2.2 says that the solution is:

$$F_N = c_0\phi^N+c_1\hat{\phi}^N$$

for some constants \\(c_0\\) and \\(c1\\):

$$F_0 = 0 = c_0 + c_1$$
$$F_1 = 1 = c_0\phi + c_1\hat{\phi}$$

yield the solution:

$$F_N = \frac{1}{\sqrt{(5)}}(\phi^N - \hat{\phi}^N)$$

**Nonconstant coefficients**. If the coefficients are not constants, then more advanced technique are needed because Theorem 2.2 does not apply. For example:

$$\eqalign{
a_n &= na_{n-1} + n(n-1)a_{n-2} \qquad for \ n>1 \ with \ a_1 = 1 \ and \ a_0 = 0\\
\frac{a_n}{n!} &= \frac{na_{n-1} + n(n-1)a_{n-2}}{n!}\\
a_n &= n!F_n\\
}$$

## Method for Solving Recurrences

**Change of Variables**

For example:

$$a_n = \sqrt{a_{n-1}a_{n-2}} \qquad for \ n>1 \ with \ a_0 = 1 \ and \ a_1 = 2$$

Let \\(b_n = lga_n\\):

$$b_n = \frac{1}{2}(b_{n-1} + b_{n-2}) \qquad for \ n>1 \ with \ b_0 = 0 \ and \ b_1 = 1$$

Another example, for recurrence to continued fractions:

$$a_n = \frac{1}{(1+a_{n-1})} \qquad for \ n>0 \ with \ a_0 = 1$$

Iterating this recurrence gives the sequence:

$$\eqalign{
    a_0 &= 1\\
    a_1 &= \frac{1}{1+1} = \frac{1}{2}\\
    a_2 &= \frac{1}{1 + \frac{1}{1+1}} =  \frac{1}{1+\frac{1}{2}} = \frac{2}{3}\\
    a_3 &= \frac{1}{1+\frac{1}{1 + \frac{1}{1+1}}} = \frac{1}{1+\frac{2}{3}} = \frac{3}{5}
}$$

That form, reveals the Fibonacci numbers. Let \\(a_n = b_{n-1} / b_n\\):

$$\frac{b_{n-1}}{b_n} = 1 / (1 + \frac{b_{n-2}}{b_{n-1}}) \qquad for \ n > 1 \ with \ b_0 = b_1 = 1$$

Dividng both sides by \\(b_{n-1}\\) gives:

$$\frac{1}{b_n} = \frac{1}{b_{n-1} + b_{n-2}} \qquad for \ n > 1 \ with \ b_0 = b_1 = 1$$

**Repertoire**

$$a_n = (n-1)a_{n-1}-na_{n-2}+n+1 \qquad for \ n>1 \ with \ a_0 = a_1 = 1$$

Introducing a quantity \\(f(n)\\) to the right-hand side:

$$a_n = (n-1)a_{n-1}-na_{n-2}+f(n)$$

We arrive at the table:

|  \\(a_n\\)  |  \\(f(n)\\)  |
|-------------|--------------|
|      1      |       2      |
|  \\(n\\)    |     n - 1    |
|  \\(n^2\\)  |     n + 1    |

Subtracting the first row form the third gives the result that means that:

$$a_n = n^2 - 1 \qquad when \qquad f(n) = n-1\\
initial \ condition \ a_0 = -1 \ and \ a_1 = 0$$

Now, we have two (linearly independent) solutions for \\(f(n) = n - 1\\), which we combine to get the right initial values, yielding the result:

$$a_n = n^2 - n + 1$$

**Bootstapping**

Given Fibonacci recurrence:

$$a_n = a_{n-1} + a_{n-2} \qquad for \ n > 1 \ with a_0 = 0 \ and a_1 = 1$$

Note, that \\(a_n\\) is increasing. Therefore, \\(a_{n-1} > a_{n-2}\\) and \\(a_n > 2a_{n-2}\\). Iterating this inequality imply the lower bound:

$$a_n > 2^{n/2}$$

On the other hand, \\(a_{n-2} > a_{n-1}\\) and \\(a_n < 2a_{n-1}\\). Iterating this inequality imply the upper bound:

$$a_n < 2^{n}$$

"guessing" a solution of the form \\(a_n \sim c_0\alpha^n\\), with \\(\sqrt(2) < \alpha < 2\\)

**Perturbation**

$$a_{n+1} = 2a_n + \frac{a_{n-1}}{n^2} \qquad for \ n>1 \ with \ a_0=1 \ and \ a_1 = 2$$

Eliminate \\(\frac{1}{n^2}\\) because makes only a small contribution to recurrence, so:

$$a_{n+1} \approx 2a_n$$

This a growth of the rough form \\(a_n \approx 2^n\\) is anticipated. To make this precise, consider the simpler sequence:

$$b_{n+1} = 2b_n  \qquad for \ n>0 \ with \ b_0=1$$

(so that \\(b_n = 2^n\\)) and compare the two recurrence by forming the ratio:

$$\rho_n = \frac{a_n}{b_n} = \frac{a_n}{2^n}$$

From the recurrences, we have:

$$\rho_{n+1} = \rho_n + \frac{1}{4n^2} \rho_{n-1} \qquad for \ n>0 \ with \ \rho_0=1$$

Clearly, the \\(\rho_n\\) are increasing. To prove they tend to a constant, note that:

$$\rho_{n+1} \leq \rho_n (1+\frac{1}{4n^2}) \qquad for \ n \geq 1 \ so \ that \qquad \rho_{n+1} \leq \Pi_{k=1}^n (1+\frac{1}{4k^2})$$

But the infinite product corresponding to the right-hand side converges monotonically to:

$$\alpha_0 = \Pi_{k=1}^\infty (1 + \frac{1}{4k^2}) = 1.46505...$$

This, \\(\rho_n\\) is bounded from above by \\(\alpha_0\\) and, as it is increasing, it must converge to a constant. We have thus proved that:

$$a_n \sim \alpha \cdot 2^n$$

for some constant \\(\alpha \le 1.46505...\\)

## Binary Divide-and-Conquer Recurrences and Binary Numbers

The number of comparisons used by mergesort is given by the solution to the recurrence:

$$C_N=C_{\lfloor{N/2}\rfloor}+C_{\lceil{N/2}\rceil}+N \qquad for \ N > 1 \ with \ C_1=0 \qquad (4)$$

**Binary Search**

The number comparisons used during an unsuccessful search with binary search in a table of size N in the worst case is equal to the number of bits in the binary representation of N.

$$B_N = B_{\lfloor{N/2}\rfloor}+1 \qquad for \ N \geq 2 \ with \ B_1 = 1$$

Which exact solution is:

$$B_N = \lfloor{lgN}\rfloor + 1$$

**Exact solution of mergesort recurrence**

The mergesort recurrence (4) is easily solved by differencing: if \\(D_N\\) is defined to be \\(C_{N+1} - C_N\\), the \\(D_N\\) satisfied the recurrence:

$$D_N = D_{\lfloor{N/2}\rfloor}+1 \qquad for \ N \geq 2 \ with \ D_1 = 2$$

which iterates to:

$$D_N = {\lfloor{lgN}\rfloor}+2$$

and, therefore,

$$C_N = N - 1 + \sum_{1 \leq k < N} ({\lfloor{lgk}\rfloor}+1)$$

**Theorem 2.4 (Mergesort)**. The number of comparison used by mergesort is equal to \\(N - 1\\) plus the number of bits in the binary representations of all the numbers less than \\(N\\). Both quantities are described by the recurrence:

$$C_N=C_{\lfloor{N/2}\rfloor}+C_{\lceil{N/2}\rceil}+N \qquad for \ N \geq 2 \ with \ C_1=0$$

Which has the exact solution \\(C_N = N{\lfloor{lgN}\rfloor} + 2N - 2^{{\lfloor{lgN}\rfloor} + 1}\\).

**PROOF**

$$
\eqalign{
C_N &= (N-1) + (N-1) + (N-2) + (N-4) + ... + (N - 2^{{\lfloor{lgN}\rfloor} + 1})\\
&= (N-1) + N({\lfloor{lgN}\rfloor} + 1) - (1 + 2 + 4 + ... + 2^{{\lfloor{lgN}\rfloor} + 1})\\
&= N{\lfloor{lgN}\rfloor} + 2N - 2^{{\lfloor{lgN}\rfloor} + 1}
}
$$

## General Divide-and-Conquer Recurrences

A variety of "devide-and-conquer" recurrence qrise that depend on the number and relative size of subproblems, the extent to which they overlap, and the cost of recombining them for the solution.

In pursuit of general solution, we start with the recursive formula:

$$\alpha(x) = \alpha a (x/\beta) + f(x) \qquad for \ x>1 \ with \ a(x) = 0 \ for \ x \leq 1$$

defining a function over the positive real numbers. In essence, this corresponds to a divide-and-conquer algorithm that divides a problem of size x into \\(\alpha\\) subproblems of size \\(x/\beta\\) and recombines them at a cost of \\(f(x)\\).

For example, consider the case where \\(f(x)=x\\) and we restrict ourself to the integers \\(N=\beta^n\\). We have:

$$\alpha_{\beta n} = \alpha a_{\beta n - 1} + \beta^n \qquad for \ n > 0 \ with \ a_1 = 0$$

Dividing both sides by \\(\alpha^n\\) and iterating, we have the solution:

$$\alpha_{\beta n} = \alpha^n \sum_{1 \leq j \leq n} \left(\frac{\beta}{\alpha}\right)^j$$

**Theorem 2.5 (Divide-and-conquer functions)**. If the function \\(a(x)\\) satisfies the recurrence:

$$a(x) = \alpha a (x/\beta) + x \qquad for \ x > 1 \ with \ a(x) = 0 \ for \ x \leq 1$$

then

$$
\eqalign{
if \ \alpha < \beta &\qquad a(x) \sim \frac{\beta}{\beta-\alpha}x\\
if \ \alpha = \beta &\qquad a(x) \sim xlog_{\beta}x\\
if \ \alpha > \beta &\qquad a(x) \sim \frac{\alpha}{\alpha-\beta}\left(\frac{\beta}{\alpha}\right)^{{log_{\beta}\alpha}} x^{log_{\beta}\alpha}
}
$$

## Master Theorem

### Divide-and-conquer algorithms

Suppose that an algorithm attacks a problem of size \\(N\\) by:
* Dividing into \\(\alpha\\) parts of size about \\(N/\beta\\)
* Solving recursively
* Combining solutions with extra cost \\(\theta(N^{\gamma}(log N)^{\delta})\\)

1. Mergesort (\\(\alpha = 2, \ \beta = 2, \ \gamma = 1, \ \delta = 0\\))

$$C_N = 2C_{N/2}+N$$

2. Batcher network (\\(\alpha = 2, \ \beta = 2, \ \gamma = 1, \ \delta = 1\\))

$$C_N = 2C_{N/2} + NlgN$$

3. Karatsuba multiplication (\\(\alpha = 3, \ \beta = 2, \ \gamma = 1, \ \delta = 0\\))

$$C_N = 3C_{N/2}+N$$

4. Strasses matrix multiply (\\(\alpha = 7, \ \beta = 2, \ \gamma = 1, \ \delta = 0\\))

$$C_N = 7C_{N/2}+N$$


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license © 2015 L.A. Barba, G.F. Forsyth, B. Knaepen

# Relax and hold steady

This is the fourth and last notebook of **Module 5** (*"Relax and hold steady"*), dedicated to elliptic PDEs. In the [previous notebook](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_03_Iterate.This.ipynb), we examined how different algebraic formulations can speed up the iterative solution of the Laplace equation, compared to the simplest (but slowest) Jacobi method.  The Gauss-Seidel and successive-over relaxation methods both provide faster algebraic convergence than Jacobi. But there is still room for improvement.

In this lesson, we'll take a look at the very popular [conjugate gradient](https://en.wikipedia.org/wiki/Conjugate_gradient_method) (CG) method.
The CG method solves linear systems with coefficient matrices that are symmetric and positive-definite. It is either used on its own, or in conjunction with multigrid—a technique that we'll explore later on its own (optional) course module.

For a real understanding of the CG method, there is no better option than studying the now-classic monograph by Jonathan Shewchuck: *"An introduction to the conjugate gradient method without the agonizing pain"* (1994). Here, we try to give you a brief summary to explain the implementation in Python.

### Test problem

Let's return to the Poisson equation example from [Lesson 2](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_02_2D.Poisson.Equation.ipynb).

$$
\begin{equation}
\nabla^2 p = -2\left(\frac{\pi}{2}\right)^2\sin\left( \frac{\pi x}{L_x} \right) \cos\left(\frac{\pi y}{L_y}\right)
\end{equation}
$$

in the domain

$$
\left\lbrace \begin{align*}
0 &\leq x\leq 1  \\
-0.5 &\leq y \leq 0.5
\end{align*} \right.
$$

where $L_x = L_y = 1$ and with boundary conditions

$$
p=0 \text{ at } \left\lbrace
\begin{align*}
x&=0\\
y&=0\\
y&=-0.5\\
y&=0.5
\end{align*} \right.
$$

We will solve this equation by assuming an initial state of $p=0$ everywhere, and applying boundary conditions to relax via the Laplacian operator.

## Head in the right direction!

Recall that in its discretized form, the Poisson equation reads,

$$
\frac{p_{i+1,j}^{k}-2p_{i,j}^{k}+p_{i-1,j}^{k}}{\Delta x^2}+\frac{p_{i,j+1}^{k}-2 p_{i,j}^{k}+p_{i,j-1}^{k}}{\Delta y^2}=b_{i,j}^{k}
$$

The left hand side represents a linear combination of the values of $p$ at several grid points and this linear combination has to be equal to the value of the source term, $b$, on the right hand side.

Now imagine you gather the values $p_{i,j}$ of $p$ at all grid points into a big vector ${\bf p}$ and you do the same for $b$ using the same ordering. Both vectors ${\bf p}$ and ${\bf b}$ contain $N=nx*ny$ values and thus belong to $\mathbb{R}^N$. The discretized Poisson equation corresponds to the following linear system:

$$
\begin{equation}
A{\bf p}={\bf b},
\end{equation}
$$

where $A$ is an $N\times N$ matrix. Although we will not directly use the matrix form of the system in the CG algorithm, it is useful to examine the problem this way to understand how the method works.

All iterative methods start with an initial guess, $\mathbf{p}^0$, and modify it in a way such that we approach the solution. This can be viewed as modifying the vector of discrete $p$ values on the grid by adding another vector, i.e., taking a step of magnitude $\alpha$ in a direction $\mathbf{d}$, as follows:

$$
\begin{equation}
{\bf p}^{k+1}={\bf p}^k + \alpha {\bf d}^k
\end{equation}
$$

The iterations march towards the solution by taking steps along the direction vectors ${\bf d}^k$, with the scalar $\alpha$ dictating how big a step to take at each iteration.  We *could* converge faster to the solution if we just knew how to carefully choose the direction vectors and the size of the steps. But how to do that?

## The residual

One of the tools we use to find the right direction to step to is called the *residual*.  What is the residual?  We're glad you asked!

We know that, as the iterations proceed, there will be some error between the calculated value, $p^k_i$, and the exact solution $p^{exact}_i$.  We may not know what the exact solution is, but we know it's out there.  The error is:

$$
\begin{equation}
e^k_i = p^k_i - p^{exact}_i
\end{equation}
$$

**Note:** We are talking about error at a specific point $i$, not a measure of error across the entire domain.

What if we recast the Poisson equation in terms of a not-perfectly-relaxed $\bf p^k$?

$$
\begin{equation}
A \bf p^k \approx b
\end{equation}
$$

We write this as an approximation because $\bf p^k \neq p$.  To "fix" the equation, we need to add an extra term to account for the difference in the Poisson equation $-$ that extra term is called the residual.  We can write out the modified Poisson equation like this:

$$
\begin{equation}
{\bf r^k} + A \bf p^k = b
\end{equation}
$$

## The method of steepest descent

Before considering the more-complex CG algorithm, it is helpful to introduce a simpler approach called the *method of steepest descent*. At iteration $0$, we choose an initial guess. Unless we are immensely lucky, it will not satisfy the Poisson equation and we will have,

$$
\begin{equation}
{\bf b}-A{\bf p}^0={\bf r}^0\ne {\bf 0}
\end{equation}
$$

The vector ${\bf r}^0$ is the initial residual and measures how far we are from satisfying the linear system. We can monitor the residual vector at each iteration, as it gets (hopefully) smaller and smaller:

$$
\begin{equation}
{\bf r}^k={\bf b}-A{\bf p}^k
\end{equation}
$$

We make two choices in the method of steepest descent:

1. the direction vectors are the residuals ${\bf d}^k = {\bf r}^k$, and
2. the length of the step makes the $k+1^{th}$ residual orthogonal to the $k^{th}$ residual.

There are good (not very complicated) reasons to justify these choices and you should read one of the references to understand them. But since we want you to converge to the end of the notebook in a shorter time, please accept them for now.

Choice 2 requires that,

$$
\begin{align}
{\bf r}^{k+1}\cdot {\bf r}^{k} = 0 \nonumber \\
\Leftrightarrow ({\bf b}-A{\bf p}^{k+1}) \cdot {\bf r}^{k} = 0 \nonumber \\
\Leftrightarrow ({\bf b}-A({\bf p}^{k}+\alpha {\bf r}^k)) \cdot {\bf r}^{k} = 0 \nonumber \\
\Leftrightarrow ({\bf r}^k-\alpha A{\bf r}^k) \cdot {\bf r}^{k} = 0 \nonumber \\
\alpha = \frac{{\bf r}^k \cdot {\bf r}^k}{A{\bf r}^k \cdot {\bf r}^k}.
\end{align}
$$

We are now ready to test this algorithm.

To begin, let's import libraries and some helper functions and set up our mesh.

```python
import numpy
from helper import l2_norm, poisson_2d_jacobi, poisson_solution
```

```python
# Set parameters.
nx = 101  # number of points in the x direction
ny = 101  # number of points in the y direction
xmin, xmax = 0.0, 1.0  # limits in the x direction
ymin, ymax = -0.5, 0.5  # limits in the y direction
Lx = xmax - xmin  # domain length in the x direction
Ly = ymax - ymin  # domain length in the y direction
dx = Lx / (nx - 1)  # grid spacing in the x direction
dy = Ly / (ny - 1)  # grid spacing in the y direction

# Create the gridline locations and the mesh grid.
x = numpy.linspace(xmin, xmax, num=nx)
y = numpy.linspace(ymin, ymax, num=ny)
X, Y = numpy.meshgrid(x, y)

# Create the source term.
b = (-2.0 * (numpy.pi / Lx) * (numpy.pi / Ly) *
     numpy.sin(numpy.pi * X / Lx) *
     numpy.cos(numpy.pi * Y / Ly))

# Set the initial conditions.
p0 = numpy.zeros((ny, nx))

# Compute the analytical solution.
p_exact = poisson_solution(x, y, Lx, Ly)
```

### Time to code steepest descent!

Let's quickly review the solution process:

1. Calculate the residual, $\bf r^k$, which also serves as the direction vector, $\bf d^k$
2. Calculate the step size $\alpha$
3. Update ${\bf p}^{k+1}={\bf p}^k + \alpha {\bf d}^k$

##### How do we calculate the residual?

We have an equation for the residual above:

$$
\begin{equation}
{\bf r}^k={\bf b}-A{\bf p}^k
\end{equation}
$$

Remember that $A$ is just a stand-in for the discrete Laplacian, which taking $\Delta x=\Delta y$ is:

$$
\begin{equation}
\nabla^2 p^k = \frac{-4p^k_{i,j} + \left(p^{k}_{i,j-1} + p^k_{i,j+1} + p^{k}_{i-1,j} + p^k_{i+1,j} \right)}{\Delta x^2}
\end{equation}
$$

##### What about calculating $\alpha$?

The calculation of $\alpha$ is relatively straightforward, but does require evaluating the term $A{\bf r^k}$, but we just wrote the discrete $A$ operator above. You just need to apply that same formula to $\mathbf{r}^k$.

```python
def poisson_2d_steepest_descent(p0, b, dx, dy,
                                maxiter=20000, rtol=1e-6):
    """
    Solves the 2D Poisson equation on a uniform grid,
    with the same grid spacing in both directions,
    for a given forcing term
    using the method of steepest descent.

    The function assumes Dirichlet boundary conditions with value zero.
    The exit criterion of the solver is based on the relative L2-norm
    of the solution difference between two consecutive iterations.

    Parameters
    ----------
    p0 : numpy.ndarray
        The initial solution as a 2D array of floats.
    b : numpy.ndarray
        The forcing term as a 2D array of floats.
    dx : float
        Grid spacing in the x direction.
    dy : float
        Grid spacing in the y direction.
    maxiter : integer, optional
        Maximum number of iterations to perform;
        default: 20000.
    rtol : float, optional
        Relative tolerance for convergence;
        default: 1e-6.

    Returns
    -------
    p : numpy.ndarray
        The solution after relaxation as a 2D array of floats.
    ite : integer
        The number of iterations performed.
    conv : list
        The convergence history as a list of floats.
    """
    def A(p):
        # Apply the Laplacian operator to p.
        return (-4.0 * p[1:-1, 1:-1] +
                p[1:-1, :-2] + p[1:-1, 2:] +
                p[:-2, 1:-1] + p[2:, 1:-1]) / dx**2
    p = p0.copy()
    r = numpy.zeros_like(p)  # initial residual
    Ar = numpy.zeros_like(p)  # to store the mat-vec multiplication
    conv = []  # convergence history
    diff = rtol + 1  # initial difference
    ite = 0  # iteration index
    while diff > rtol and ite < maxiter:
        pk = p.copy()
        # Compute the residual.
        r[1:-1, 1:-1] = b[1:-1, 1:-1] - A(p)
        # Compute the Laplacian of the residual.
        Ar[1:-1, 1:-1] = A(r)
        # Compute the step size.
        alpha = numpy.sum(r * r) / numpy.sum(r * Ar)
        # Update the solution.
        p = pk + alpha * r
        # Dirichlet boundary conditions are automatically enforced.
        # Compute the relative L2-norm of the difference.
        diff = l2_norm(p, pk)
        conv.append(diff)
        ite += 1
    return p, ite, conv
```

Let's see how it performs on our example problem.

```python
# Compute the solution using the method of steepest descent.
p, ites, conv_sd = poisson_2d_steepest_descent(p0, b, dx, dy,
                                               maxiter=20000,
                                               rtol=1e-10)
print('Method of steepest descent: {} iterations '.format(ites) +
      'to reach a relative difference of {}'.format(conv_sd[-1]))
```

    Method of steepest descent: 2 iterations to reach a relative difference of 1.3307695446303778e-16

```python
# Compute the relative L2-norm of the error.
l2_norm(p, p_exact)
```

    8.225076220929745e-05

Not bad! it took only *two* iterations to reach a solution that meets our exit criterion. Although this seems great, the steepest descent algorithm is not too good when used with large systems or more complicated right-hand sides in the Poisson equation (we'll examine this below!). We can get better performance if we take a little more care in selecting the direction vectors, $\bf d^k$.

## The method of conjugate gradients

With steepest descent, we know that two **successive** jumps are orthogonal, but that's about it.  There is nothing to prevent the algorithm from making several jumps in the same (or a similar) direction.  Imagine you wanted to go from the intersection of 5th Avenue and 23rd Street to the intersection of 9th Avenue and 30th Street. Knowing that each segment has the same computational cost (one iteration), would you follow the red path or the green path?

#### Figure 1. Do you take the red path or the green path?

The method of conjugate gradients reduces the number of jumps by making sure the algorithm never selects the same direction twice. The size of the jumps is now given by:

$$
\begin{equation}
\alpha = \frac{{\bf r}^k \cdot {\bf r}^k}{A{\bf d}^k \cdot {\bf d}^k}
\end{equation}
$$

and the direction vectors by:

$$
\begin{equation}
{\bf d}^{k+1}={\bf r}^{k+1}+\beta{\bf d}^{k}
\end{equation}
$$

where $\beta = \frac{{\bf r}^{k+1} \cdot {\bf r}^{k+1}}{{\bf r}^k \cdot {\bf r}^k}$.

The search directions are no longer equal to the residuals but are instead a  linear combination of the residual and the previous search direction. It turns out that CG converges to the exact solution (up to machine accuracy) in a maximum of $N$ iterations! When one is satisfied with an approximate solution, many fewer steps are needed than with any other method. Again, the derivation of the algorithm is not immensely difficult and can be found in Shewchuk (1994).

### Implementing Conjugate Gradients

We will again update $\bf p$ according to

$$
\begin{equation}
{\bf p}^{k+1}={\bf p}^k + \alpha {\bf d}^k
\end{equation}
$$

but use the modified equations above to calculate $\alpha$ and ${\bf d}^k$.

You may have noticed that $\beta$ depends on both ${\bf r}^{k+1}$ and ${\bf r}^k$ and that makes the calculation of ${\bf d}^0$ a little bit tricky.  Or impossible (using the formula above).  Instead we set ${\bf d}^0 = {\bf r}^0$ for the first step and then switch for all subsequent iterations.

Thus, the full set of steps for the method of conjugate gradients is:

Calculate ${\bf d}^0 = {\bf r}^0$ (just  once), then

1. Calculate $\alpha = \frac{{\bf r}^k \cdot {\bf r}^k}{A{\bf d}^k \cdot {\bf d}^k}$
2. Update ${\bf p}^{k+1}$
3. Calculate ${\bf r}^{k+1} = {\bf r}^k - \alpha A {\bf d}^k$ $\ \ \ \ $(see Shewchuk (1994))
4. Calculate $\beta = \frac{{\bf r}^{k+1} \cdot {\bf r}^{k+1}}{{\bf r}^k \cdot {\bf r}^k}$
5. Calculate ${\bf d}^{k+1}={\bf r}^{k+1}+\beta{\bf d}^{k}$
6. Repeat!

```python
def poisson_2d_conjugate_gradient(p0, b, dx, dy,
                                  maxiter=20000, rtol=1e-6):
    """
    Solves the 2D Poisson equation on a uniform grid,
    with the same grid spacing in both directions,
    for a given forcing term
    using the method of conjugate gradients.

    The function assumes Dirichlet boundary conditions with value zero.
    The exit criterion of the solver is based on the relative L2-norm
    of the solution difference between two consecutive iterations.

    Parameters
    ----------
    p0 : numpy.ndarray
        The initial solution as a 2D array of floats.
    b : numpy.ndarray
        The forcing term as a 2D array of floats.
    dx : float
        Grid spacing in the x direction.
    dy : float
        Grid spacing in the y direction.
    maxiter : integer, optional
        Maximum number of iterations to perform;
        default: 20000.
    rtol : float, optional
        Relative tolerance for convergence;
        default: 1e-6.

    Returns
    -------
    p : numpy.ndarray
        The solution after relaxation as a 2D array of floats.
    ite : integer
        The number of iterations performed.
    conv : list
        The convergence history as a list of floats.
    """
    def A(p):
        # Apply the Laplacian operator to p.
        return (-4.0 * p[1:-1, 1:-1] +
                p[1:-1, :-2] + p[1:-1, 2:] +
                p[:-2, 1:-1] + p[2:, 1:-1]) / dx**2
    p = p0.copy()
    r = numpy.zeros_like(p)  # initial residual
    Ad = numpy.zeros_like(p)  # to store the mat-vec multiplication
    conv = []  # convergence history
    diff = rtol + 1  # initial difference
    ite = 0  # iteration index
    # Compute the initial residual.
    r[1:-1, 1:-1] = b[1:-1, 1:-1] - A(p)
    # Set the initial search direction to be the residual.
    d = r.copy()
    while diff > rtol and ite < maxiter:
        pk = p.copy()
        rk = r.copy()
        # Compute the Laplacian of the search direction.
        Ad[1:-1, 1:-1] = A(d)
        # Compute the step size.
        alpha = numpy.sum(r * r) / numpy.sum(d * Ad)
        # Update the solution.
        p = pk + alpha * d
        # Update the residual.
        r = rk - alpha * Ad
        # Update the search direction.
        beta = numpy.sum(r * r) / numpy.sum(rk * rk)
        d = r + beta * d
        # Dirichlet boundary conditions are automatically enforced.
        # Compute the relative L2-norm of the difference.
        diff = l2_norm(p, pk)
        conv.append(diff)
        ite += 1
    return p, ite, conv
```

```python
# Compute the solution using the method of conjugate gradients.
p, ites, conv_cg = poisson_2d_conjugate_gradient(p0, b, dx, dy,
                                                 maxiter=20000,
                                                 rtol=1e-10)
print('Method of conjugate gradients: {} iterations '.format(ites) +
      'to reach a relative difference of {}'.format(conv_cg[-1]))
```

    Method of conjugate gradients: 2 iterations to reach a relative difference of 1.2982770796281907e-16

```python
# Compute the relative L2-norm of the error.
l2_norm(p, p_exact)
```

    8.225076220929585e-05

The method of conjugate gradients also took two iterations to reach a solution that meets our exit criterion. But let's compare this to the number of iterations needed for the Jacobi iteration:

```python
# Compute the solution using Jacobi relaxation.
p, ites, conv_jacobi = poisson_2d_jacobi(p0, b, dx, dy,
                                         maxiter=40000,
                                         rtol=1e-10)
print('Jacobi relaxation: {} iterations '.format(ites) +
      'to reach a relative difference of {}'.format(conv_jacobi[-1]))
```

    Jacobi relaxation: 31227 iterations to reach a relative difference of 9.997923503623598e-11

For our test problem, we get substantial gains in terms of computational cost using the method of steepest descent or the conjugate gradient method.

## More difficult Poisson problems

The conjugate gradient method really shines when one needs to solve more difficult Poisson problems. To get an insight into this, let's solve the Poisson problem using the same boundary conditions as the previous problem but with the following right-hand side,

$$
\begin{equation}
b = \sin\left(\frac{\pi x}{L_x}\right) \cos\left(\frac{\pi y}{L_y}\right) + \sin\left(\frac{6\pi x}{L_x}\right) \cos\left(\frac{6\pi y}{L_y}\right)
\end{equation}
$$

```python
# Modify the source term of the Poisson system.
b = (numpy.sin(numpy.pi * X / Lx) *
     numpy.cos(numpy.pi * Y / Ly) +
     numpy.sin(6.0 * numpy.pi * X / Lx) *
     numpy.sin(6.0 * numpy.pi * Y / Ly))
```

```python
maxiter, rtol = 40000, 1e-10
p, ites, conv = poisson_2d_jacobi(p0, b, dx, dy,
                                  maxiter=maxiter, rtol=rtol)
print('Jacobi relaxation: {} iterations'.format(ites))
p, ites, conv = poisson_2d_steepest_descent(p0, b, dx, dy,
                                            maxiter=maxiter,
                                            rtol=rtol)
print('Method of steepest descent: {} iterations'.format(ites))
p, ites, conv = poisson_2d_conjugate_gradient(p0, b, dx, dy,
                                              maxiter=maxiter,
                                              rtol=rtol)
print('Method of conjugate gradients: {} iterations'.format(ites))
```

    Jacobi relaxation: 31226 iterations
    Method of steepest descent: 27059 iterations
    Method of conjugate gradients: 3 iterations

Now we can really appreciate the marvel of the CG method!

## References

Shewchuk, J. (1994). [An Introduction to the Conjugate Gradient Method Without the Agonizing Pain (PDF)](http://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf)

Ilya Kuzovkin, [The Concept of Conjugate Gradient Descent in Python](http://ikuz.eu/2015/04/15/the-concept-of-conjugate-gradient-descent-in-python/)

---
###### The cell below loads the style of this notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>