 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
import warnings
import sympy
sympy.init_printing()
```

# Root Finding and Optimization

Our goal in this section is to develop techniques to approximate the roots of a given function $f(x)$.  That is find solutions $x$ such that $f(x)=0$. At first glance this may not seem like a meaningful exercise, however, this problem arises  in a wide variety of circumstances.

For example, suppose that you are trying to find a solution to the equation

$$
    x^2 + x = \sin{x}.
$$

Simply rearranging, the expression can be rewritten in the form

$$
    f(x) =  x^2 + x -\sin{x} = 0.
$$

Determining the roots of the function $f(x)$ is now equivalent to determining the solution to the original expression. Unfortunately, a number of other issues arise. In particular,  with non-linear equations, there may be multiple solutions, or no real solutions at all.

```python
x = numpy.linspace(-100, 100, 201)
f = x**2 + x - numpy.sin(x)

plt.plot(x, f, 'bo')
plt.grid()
plt.show()

```

The task of approximating the roots of a function can be a deceptively difficult thing to do. For much of the treatment here we will ignore many details such as existence and uniqueness, but you should keep in mind that they are important considerations.

**GOAL:**
For this section we will focus on multiple techniques for efficiently and accurately solving the fundamental problem $f(x)=0$ for functions of a single variable.

### Example:  Future Time Annuity

Can I ever retire?

$$ A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] $$

* $A$ total value after $n$ years
* $P$ is payment amount per compounding period
* $m$ number of compounding periods per year
* $r$ annual interest rate
* $n$ number of years to retirement

#### Question:

For a fix monthly Payment $P$, what does the minimum interest rate $r$ need to be so I can retire in 20 years with \$1M.

Set $P = \frac{\$18,000}{12} = \$1500, \quad m=12, \quad n=20$.

$$
    A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]
$$

```python
def total_value(P, m, r, n):
    """Total value of portfolio given parameters

    Based on following formula:

    A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n}
                - 1 \right ]

    :Input:
     - *P* (float) - Payment amount per compounding period
     - *m* (int) - number of compounding periods per year
     - *r* (float) - annual interest rate
     - *n* (float) - number of years to retirement

     :Returns:
     (float) - total value of portfolio

    """

    return P / (r / float(m)) * ( (1.0 + r / float(m))**(float(m) * n)
                                 - 1.0)

P = 1500.0
m = 12
n = 20.0

r = numpy.linspace(0.05, 0.15, 100)
goal = 1e6

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, total_value(P, m, r, 10),label='10 years')
axes.plot(r, total_value(P, m, r, 15),label='15 years')
axes.plot(r, total_value(P, m, r, n),label='20 years')
axes.plot(r, numpy.ones(r.shape) * goal, 'r--')
axes.set_xlabel("r (interest rate)", fontsize=16)
axes.set_ylabel("A (total value)", fontsize=16)
axes.set_title("When can I retire?",fontsize=18)
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.set_xlim((r.min(), r.max()))
axes.set_ylim((total_value(P, m, r.min(), 10), total_value(P, m, r.max(), n)))
axes.legend(loc='best')
axes.grid()
plt.show()
```

## Fixed Point Iteration

How do we go about solving this?

Could try to solve at least partially for $r$:

$$ A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] ~~~~ \Rightarrow ~~~~~$$

$$ r = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] ~~~~ \Rightarrow ~~~~~$$

$$ r = g(r)$$
or
$$ g(r) - r = 0$$

#### Plot these

$$ r = g(r) = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]$$

```python
def g(P, m, r, n, A):
    """Reformulated minimization problem

    Based on following formula:

    g(r) = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]

    :Input:
     - *P* (float) - Payment amount per compounding period
     - *m* (int) - number of compounding periods per year
     - *r* (float) - annual interest rate
     - *n* (float) - number of years to retirement
     - *A* (float) - total value after $n$ years

     :Returns:
     (float) - value of g(r)

    """

    return P * m / A * ( (1.0 + r / float(m))**(float(m) * n)
                                 - 1.0)

P = 1500.0
m = 12
n = 20.0

r = numpy.linspace(0.00, 0.1, 100)
goal = 1e6

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, g(P, m, r, n, goal),label='$g(r)$')
axes.plot(r, r, 'r--',label='$r$')
axes.set_xlabel("r (interest rate)",fontsize=16)
axes.set_ylabel("$g(r)$",fontsize=16)
axes.set_title("Minimum rate for a 20 year retirement?",fontsize=18)
axes.set_ylim([0, 0.12])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.set_xlim((0.00, 0.1))
axes.set_ylim((g(P, m, 0.00, n, goal), g(P, m, 0.1, n, goal)))
axes.legend()
axes.grid()
plt.show()
```

Guess at $r_0$ and check to see what direction we need to go...

1. $r_0 = 0.0800, \quad g(r_0) - r_0 = -0.009317550125425428$
1. $r_1 = 0.0850, \quad g(r_1) - r_1 = -0.00505763375972$
1. $r_2 = 0.0875, \quad g(r_2) - r_2 = -0.00257275331014$

A bit tedious, we can also make this algorithmic:

```python
r_values = numpy.linspace(0.08, 0.1, 11)
g_values = g(P,m,r_values,n,goal)
residual = numpy.abs(g_values - r_values)
```

```python
print('     r\t\t  g(r)\t\tresidual')
print('------------------------------------------------')
for i,r in enumerate(r_values):
    print('{:8.3f}\t{:10.8f}\t{:10.8f}\t'.format(r,g_values[i],residual[i]))
```

         r		  g(r)		residual
    ------------------------------------------------
       0.080	0.07068245	0.00931755
       0.082	0.07427690	0.00772310
       0.084	0.07801640	0.00598360
       0.086	0.08190680	0.00409320
       0.088	0.08595414	0.00204586
       0.090	0.09016473	0.00016473
       0.092	0.09454513	0.00254513
       0.094	0.09910215	0.00510215
       0.096	0.10384290	0.00784290
       0.098	0.10877473	0.01077473
       0.100	0.11390533	0.01390533

### Example 2:

Let $f(x) = x - e^{-x}$, solve $f(x) = 0$

Equivalent to $x = e^{-x}$ or $x = g(x)$ where $g(x) = e^{-x}$

```python
x = numpy.linspace(0.2, 1.0, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.exp(-x), 'r',label='$f(x)=exp(-x)$')
axes.plot(x, x, 'b',label='$x$')
axes.set_xlabel("x",fontsize=16)
axes.legend()
plt.grid()
plt.show()
```

#### Consider the iterative scheme

set $x_0$ then compute

$$
    x_i = g(x_{i-1})\quad \mathrm{for}\quad i=1,2,3\ldots
$$

or in code
```python
x = x0
for i in range(N):
    x = g(x)
```

```python
x = numpy.linspace(0.2, 1.0, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.exp(-x), 'r',label='$f(x)=exp(-x)$')
axes.plot(x, x, 'b',label='$x$')
axes.set_xlabel("x",fontsize=16)
axes.legend()

x = 0.4
print('\tx\t      exp(-x)\t    residual')
for steps in range(6):
    residual = numpy.abs(numpy.exp(-x) - x)
    print("{:12.7f}\t{:12.7f}\t{:12.7f}".format(x, numpy.exp(-x), residual))
    axes.plot(x, numpy.exp(-x),'kx')
    axes.text(x+0.01, numpy.exp(-x)+0.01, steps, fontsize="15")
    x = numpy.exp(-x)

plt.grid()
plt.show()
```

### Example 3:

Let $f(x) = \ln x + x$ and solve $f(x) = 0$ or $x = -\ln x$.

Note that this problem is equivalent to $x = e^{-x}$.

```python
x = numpy.linspace(0.1, 1.0, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, -numpy.log(x), 'r',label='$f(x)=-\log(x)$')
axes.plot(x, x, 'b',label='$x$')
axes.set_xlabel("x",fontsize=16)
axes.set_ylabel("f(x)",fontsize=16)
axes.set_ylim([0, 1.5])
axes.legend(loc='best')

x = 0.55
print('\tx\t      -log(x)\t    residual')

for steps in range(5):
    residual = numpy.abs(numpy.log(x) + x)
    print("{:12.7f}\t{:12.7f}\t{:12.7f}".format(x, -numpy.log(x), residual))
    axes.plot(x, -numpy.log(x),'kx')
    axes.text(x + 0.01, -numpy.log(x) + 0.01, steps, fontsize="15")
    x = -numpy.log(x)

plt.grid()
plt.show()
```

### These are equivalent problems!

Something is awry...

## Analysis of Fixed Point Iteration

Existence and uniqueness of fixed point problems

*Existence:*

Assume $g \in C[a, b]$, if the range of the mapping $y = g(x)$ satisfies $y \in [a, b] \quad \forall \quad x \in [a, b]$ then $g$ has a fixed point in $[a, b]$.

```python
x = numpy.linspace(0.0, 1.0, 100)

# Plot function and intercept
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.exp(-x), 'r',label='$g(x)$')
axes.plot(x, x, 'b',label='$x$')
axes.set_xlabel("x",fontsize=16)
axes.legend(loc='best',fontsize=14)
axes.set_title('$g(x) = e^{-x}$',fontsize=24)

# Plot domain and range
axes.plot(numpy.ones(x.shape) * 0.4, x, '--k')
axes.plot(numpy.ones(x.shape) * 0.8, x, '--k')
axes.plot(x, numpy.ones(x.shape) * numpy.exp(-0.4), '--k')
axes.plot(x, numpy.ones(x.shape) * numpy.exp(-0.8), '--k')
axes.plot(x, numpy.ones(x.shape) * 0.4, '--',color='gray',linewidth=.5)
axes.plot(x, numpy.ones(x.shape) * 0.8, '--',color='gray',linewidth=.5)

axes.set_xlim((0.0, 1.0))
axes.set_ylim((0.0, 1.0))

plt.show()
```

```python
x = numpy.linspace(0.1, 1.0, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, -numpy.log(x), 'r',label='$g(x)$')
axes.plot(x, x, 'b',label='$x$')
axes.set_xlabel("x",fontsize=16)
axes.set_xlim([0.1, 1.0])
axes.set_ylim([0.1, 1.0])
axes.legend(loc='best',fontsize=14)
axes.set_title('$g(x) = -\ln(x)$',fontsize=24)

# Plot domain and range
axes.plot(numpy.ones(x.shape) * 0.4, x, '--k')
axes.plot(numpy.ones(x.shape) * 0.8, x, '--k')
axes.plot(x, numpy.ones(x.shape) * -numpy.log(0.4), '--k')
axes.plot(x, numpy.ones(x.shape) * -numpy.log(0.8), '--k')
axes.plot(x, numpy.ones(x.shape) * 0.4, '--',color='gray',linewidth=.5)
axes.plot(x, numpy.ones(x.shape) * 0.8, '--',color='gray',linewidth=.5)
plt.show()
```

```python
r = numpy.linspace(0.06, 0.1, 100)
goal = 1e6

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, g(P, m, r, n, goal))
axes.plot(r, r, 'r--')
axes.set_xlabel("r")
axes.set_ylabel("$g(r)$")
axes.set_xlim([0.06, 0.1])
axes.set_ylim([g(P, m, 0.06, n, goal), g(P, m, 0.1, n, goal)])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

axes.plot([0.08, 0.08], [g(P, m, 0.06, n, goal), g(P, m, 0.1, n, goal)], '--k')
axes.plot([0.095, 0.095], [g(P, m, 0.06, n, goal), g(P, m, 0.1, n, goal)], '--k')
axes.plot(r, numpy.ones(r.shape) * g(P, m, 0.08, n, goal), '--k')
axes.plot(r, numpy.ones(r.shape) * g(P, m, 0.095, n, goal), '--k')

plt.show()
```

*Uniqueness:*

Additionally, suppose $g'(x)$ is defined on $x \in [a, b]$ and $\exists K < 1$ such that

$$
    |g'(x)| \leq K < 1 \quad \forall \quad x \in (a,b)
$$

then $g$ has a unique fixed point $P \in [a,b]$

Note: Lipschitz continuity

```python
x = numpy.linspace(0.4, 0.8, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.abs(-numpy.exp(-x)), 'r')
axes.plot(x, numpy.ones(x.shape), 'k--')
axes.set_xlabel("$x$",fontsize=18)
axes.set_ylabel("$g\,'(x)$",fontsize=18)
axes.set_ylim((0.0, 1.1))
axes.set_title("$g(x) = e^{-x}$",fontsize=20)
axes.grid()
plt.show()
```

*Asymptotic convergence*: Behavior of fixed point iterations

$$x_{k+1} = g(x_k)$$

Assume that a fixed point $x^\ast$ exists, such that
$$
x^\ast = g(x^\ast)
$$

Then define
$$
    x_{k+1} = x^\ast + e_{k+1} \quad \quad  x_k = x^\ast + e_k
$$
Where $e_k$ is the correction (error) for $x_k$

substituting
$$
    x^\ast + e_{k+1} = g(x^\ast + e_k)
$$

Evaluate $$
    g(x^\ast + e_k)
$$

Taylor expand $g(x)$ about $x^\ast$ and substitute $$x = x_k = x^\ast + e_k$$

$$
    g(x^\ast + e_k) = g(x^\ast) + g'(x^\ast) e_k + \frac{g''(x^\ast) e_k^2}{2} + O(e_k^3)
$$

Note: assumption is that $g'(x)$ at fixed point $\neq$ 0.

from our definition  $$x^\ast + e_{k+1} = g(x^\ast + e_k)$$ we have
$$
    x^\ast + e_{k+1} = g(x^\ast) + g'(x^\ast) e_k + \frac{g''(x^\ast) e_k^2}{2} +  O(e_k^3)
$$

Note that because $x^* = g(x^*)$ (fixed point) these terms cancel leaving

$$e_{k+1} = g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2}$$

So if $|g'(x^*)| \leq K < 1$ we can conclude that

$$|e_{k+1}| = K |e_k|$$

which shows convergence and fixed point iteration is stable.  Also note that $K$ is related to $|g'(x^*)|$.

### Convergence of iterative schemes

Given any iterative scheme where

$$|e_{k+1}| = C |e_k|^n$$

If $C < 1$ and:
 - $n=1$ then the scheme is **linearly convergent**
 - $n=2$ then the scheme is **quadratically convergent**
 - $n > 1$ the scheme can also be called **superlinearly convergent**

If $C > 1$ then the scheme is **divergent**

### Examples Revisited

* Example 1:

$$
g(x) = e^{-x}\quad\mathrm{with}\quad x^* \approx 0.56
$$

   $$|g'(x^*)| = |-e^{-x^*}| \approx 0.56$$

* Example 2:

$$g(x) = - \ln x \quad \text{with} \quad x^* \approx 0.56$$

   $$|g'(x^*)| = \frac{1}{|x^*|} \approx 1.79$$

* Example 3: The retirement problem

$$
    r = g(r) = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]
$$

```python
r, P, m, A, n = sympy.symbols('r P m A n')
g_sym = P * m / A * ((1 + r /m)**(m * n) - 1)
g_prime = g_sym.diff(r)
r_star = 0.08985602484084668
print("g(r) = ", g_sym)
print("g'(r) = ", g_prime)
print()
print("g'(r*) = ", g_prime.subs({P: 1500.0, m: 12, n:20, A: 1e6, r: r_star}))
print("g(r*) - r* = {}".format(g_sym.subs({P: 1500.0, m: 12, n:20, A: 1e6, r: r_star}) - r_star))
```

    g(r) =  P*m*((1 + r/m)**(m*n) - 1)/A
    g'(r) =  P*m*n*(1 + r/m)**(m*n)/(A*(1 + r/m))

    g'(r*) =  2.14108802539073
    g(r*) - r* = 7.00606239689705E-12

* Example 3: The retirement problem

$$
    r = g(r) = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]
$$

```python
f = sympy.lambdify(r, g_prime.subs({P: 1500.0, m: 12, n:20, A: 1e6}))
g = sympy.lambdify(r, g_sym.subs({P: 1500.0, m: 12, n:20, A: 1e6}))
r = numpy.linspace(-0.01, 0.1, 100)

fig = plt.figure(figsize=(7,5))
fig.set_figwidth(2. * fig.get_figwidth())
axes = fig.add_subplot(1, 2, 1)
axes.plot(r, g(r),label='$g(r)$')
axes.plot(r, r, 'r--',label='$r$')
axes.set_xlabel("r (interest rate)",fontsize=14)
axes.set_ylabel("$g(r)$",fontsize=14)
axes.set_title("Minimum rate for a 20 year retirement?",fontsize=14)
axes.set_ylim([0, 0.12])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.set_xlim((0.00, 0.1))
axes.set_ylim(g(0.00), g(0.1))
axes.legend()
axes.grid()

axes = fig.add_subplot(1, 2, 2)
axes.plot(r, f(r))
axes.plot(r, numpy.ones(r.shape), 'k--')
axes.plot(r_star, f(r_star), 'ro')
axes.plot(0.0, f(0.0), 'ro')
axes.set_xlim((-0.01, 0.1))
axes.set_xlabel("$r$",fontsize=14)
axes.set_ylabel("$g'(r)$",fontsize=14)
axes.grid()

plt.show()
```

## Better ways for root-finding/optimization

If $x^*$ is a fixed point of $g(x)$ then $x^*$ is also a *root* of $f(x^*) = g(x^*) - x^*$ s.t. $f(x^*) = 0$.

For instance:

$$f(r) = r - \frac{m P}{A} \left [ \left (1 + \frac{r}{m} \right)^{m n} - 1 \right ] =0 $$

or

$$f(r) = A - \frac{m P}{r} \left [ \left (1 + \frac{r}{m} \right)^{m n} - 1 \right ] =0 $$

## Classical Methods
 - Bisection (linear convergence)
 - Newton's Method (quadratic convergence)
 - Secant Method (super-linear)

## Combined Methods
 - RootSafe (Newton + Bisection)
 - Brent's Method (Secant + Bisection)

### Bracketing and Bisection

A **bracket** is an interval $[a,b]$ that contains exactly one zero or minima/maxima of interest.

In the case of a zero the bracket should satisfy
$$
    \text{sign}(f(a)) \neq \text{sign}(f(b)).
$$

In the case of minima or maxima we need
$$
    \text{sign}(f'(a)) \neq \text{sign}(f'(b))
$$

**Theorem**:

Let
$$
    f(x) \in C[a,b] \quad \text{and} \quad \text{sign}(f(a)) \neq \text{sign}(f(b))
$$

then there exists a number
$$
    c \in (a,b) \quad \text{s.t.} \quad f(c) = 0.
$$
(proof uses intermediate value theorem)

**Example**:  The retirement problem again.  For fixed $A, P, m, n$

$$
    f(r) = A - \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]
$$

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.1, 100)
f = lambda r, A, m, P, n: A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r, A, m, P, n), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')
axes.set_xlabel("r", fontsize=16)
axes.set_ylabel("f(r)", fontsize=16)
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.grid()

a = 0.075
b = 0.095
axes.plot(a, f(a, A, m, P, n), 'ko')
axes.plot([a, a], [0.0, f(a, A, m, P, n)], 'k--')
axes.plot(b, f(b, A, m, P, n), 'ko')
axes.plot([b, b], [f(b, A, m, P, n), 0.0], 'k--')

plt.show()
```

Basic bracketing algorithms shrink the bracket while ensuring that the root/extrema remains within the bracket.

What ways could we "shrink" the bracket so that the end points converge to the root/extrema?

#### Bisection Algorithm

Given a bracket $[a,b]$ and a function $f(x)$ -
1. Initialize with bracket
2. Iterate
   1. Cut bracket in half and check to see where the zero is
   2. Set bracket to new bracket based on what direction we went

##### basic code
```python
def bisection(f,a,b,tol):
    c = a + delta_x / 2.0
    f_a = f(a)
    f_b = f(b)
    f_c = f(c)

    for step in range(1, MAX_STEPS + 1):
        if numpy.abs(f_c) < tol:
            break
        if numpy.sign(f_a) != numpy.sign(f_c):
            b = c
            f_b = f_c
        else:
            a = c
            f_a = f_c
        delta_x = b - a
        c = a + delta_x / 2.0
        f_c = f(c)
    return c
```

```python
# real code with standard bells and whistles

def bisection(f,a,b,tol = 1.e-6):
    """ uses bisection to isolate a root x of a function of a single variable f such that f(x) = 0.
        the root must exist within an initial bracket a < x < b
        returns when f(x) at the midpoint of the bracket < tol

    Parameters:
    -----------

    f: function of a single variable f(x) of type float
    a: float
        left bracket a < x
    b: float
        right bracket x < b

        Note:  the signs of f(a) and f(b) must be different to insure a bracket
    tol: float
        tolerance.  Returns when |f((a+b)/2)| < tol

    Returns:
    --------
    x: float
        midpoint of final bracket
    x_array: numpy array
        history of bracket centers (for plotting later)

    Raises:
    -------
    ValueError:
        if initial bracket is invalid
    Warning:
        if number of iterations exceed MAX_STEPS

    """
    MAX_STEPS = 1000

    # initialize
    delta_x = b - a
    c = a + delta_x / 2.0
    c_array = [ c ]

    f_a = f(a)
    f_b = f(b)
    f_c = f(c)

    # check bracket
    if numpy.sign(f_a) == numpy.sign(f_b):
        raise ValueError("no bracket: f(a) and f(b) must have different signs")

    # Loop until we reach the TOLERANCE or we take MAX_STEPS
    for step in range(1, MAX_STEPS + 1):
        # Check tolerance - Could also check the size of delta_x
        # We check this first as we have already initialized the values
        # in c and f_c
        if numpy.abs(f_c) < tol:
            break

        if numpy.sign(f_a) != numpy.sign(f_c):
            b = c
            f_b = f_c
        else:
            a = c
            f_a = f_c
        delta_x = b - a
        c = a + delta_x / 2.0
        f_c = f(c)
        c_array.append(c)

    if step == MAX_STEPS:
        warnings.warn('Maximum number of steps exceeded')

    return c, numpy.array(c_array)
```

```python
# set up function as an inline lambda function
P = 1500.0
m = 12
n = 20.0
A = 1e6
f = lambda r: A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

# Initialize bracket
a = 0.07
b = 0.10
```

```python
# find root
r_star, r_array = bisection(f, a, b, tol=1e-8)
print('root at r = {}, f(r*) = {}, {} steps'.format(r_star,f(r_star),len(r_array)))
```

    root at r = 0.08985602483470759, f(r*) = -9.080395102500916e-09, 40 steps

```python
r = numpy.linspace(0.05, 0.11, 100)
# Setup figure to plot convergence
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')
axes.set_xlabel("r", fontsize=16)
axes.set_ylabel("f(r)", fontsize=16)
# axes.set_xlim([0.085, 0.091])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.plot(a, f(a), 'ko')
axes.plot([a, a], [0.0, f(a)], 'k--')
axes.text(a, f(a), str(0), fontsize="15")

axes.plot(b, f(b), 'ko')
axes.plot([b, b], [f(b), 0.0], 'k--')
axes.text(b, f(b), str(1), fontsize="15")
axes.grid()

# plot out the first N steps
N = 5
for k,r in enumerate(r_array[:N]):
    # Plot iteration
    axes.plot(r, f(r),'kx')
    axes.text(r, f(r), str(k + 2), fontsize="15")

axes.plot(r_star, f(r_star), 'go', markersize=10)
axes.set_title('Bisection method:  first {} steps'.format(N), fontsize=20)
plt.show()
```

What is the smallest tolerance that can be achieved with this routine?  Why?

```python
# find root
r_star, r_array = bisection(f, a, b, tol=1e-8 )
print('root at r = {}, f(r*) = {}, {} steps'.format(r_star,f(r_star),len(r_array)))
```

    root at r = 0.08985602483470759, f(r*) = -9.080395102500916e-09, 40 steps

```python
# this might be useful
print(numpy.diff(r_array))
```

    [ 7.50000000e-03 -3.75000000e-03  1.87500000e-03 -9.37500000e-04
      4.68750000e-04 -2.34375000e-04 -1.17187500e-04  5.85937500e-05
     -2.92968750e-05  1.46484375e-05  7.32421875e-06  3.66210937e-06
     -1.83105469e-06 -9.15527344e-07 -4.57763672e-07 -2.28881836e-07
     -1.14440918e-07 -5.72204590e-08  2.86102295e-08 -1.43051147e-08
     -7.15255737e-09  3.57627869e-09 -1.78813934e-09  8.94069666e-10
      4.47034840e-10  2.23517413e-10  1.11758713e-10 -5.58793567e-11
      2.79396783e-11 -1.39698392e-11  6.98492653e-12  3.49245632e-12
     -1.74622816e-12 -8.73121020e-13 -4.36553571e-13  2.18283724e-13
      1.09134923e-13  5.45674617e-14  2.72837308e-14]

#### Convergence of Bisection

Generally have
$$
    |e_{k+1}| = C |e_k|^n
$$
where we need $C < 1$ and $n > 0$.

Letting $\Delta x_k$ be the width of the $k$th bracket we can then estimate the error with
$$
    e_k \approx \Delta x_k
$$
and therefore
$$
    e_{k+1} \approx \frac{1}{2} \Delta x_k.
$$
Due to the relationship then between $x_k$ and $e_k$ we then know
$$
    |e_{k+1}| = \frac{1}{2} |e_k|
$$
so therefore the method is linearly convergent.

### Newton's Method (Newton-Raphson)
 - Given a bracket, bisection is guaranteed to converge linearly to a root
 - However bisection uses almost no information about $f(x)$ beyond its sign at a point
 - Can we do "better"? Newton's method, *when well behaved* can achieve quadratic convergence.

**Basic Ideas**: There are multiple interpretations we can use to derive Newton's method
* Use Taylor's theorem to estimate a correction to minimize the residual $f(x)=0$
* A geometric interpretation that approximates $f(x)$ locally as a straight line to  predict where $x^*$ might be.
* As a special case of a fixed-point iteration

Given current location $x_k$, we have $f(x_k)$ and $f'(x_k)$ and form a line through the point $(x_k, f(x_k))$:

Form equation for the line:

$$y = f'(x_k) x + b$$

Solve for the y-intercept value $b$

$$f(x_k) = f'(x_k) x_k + b$$

$$b = f(x_k) - f'(x_k) x_k$$

and simplify.

$$y = f'(x_k) x + f(x_k) - f'(x_k) x_k$$

$$y = f'(x_k) (x - x_k) + f(x_k)$$

Now find the intersection of our line and the x-axis (i.e. when $y = 0$) and use the resulting value of $x$ to set $x_{k+1}$

$$
    0 = f'(x_k) (x_{k+1}-x_k) + f(x_k)
$$

$$
    x_{k+1} = x_k-\frac{f(x_k)}{f'(x_k)}
$$

Perhaps the simplest derivation uses Taylor series.  Consider an initial guess at point $x_k$.  For arbitrary $x_k$, it's unlikely $f(x_k)=0$.  However we can hope there is a correction $\delta_k$ such that at
$$x_{k+1} = x_k + \delta_k$$
and
$$
    f(x_{k+1}) = 0
$$

expanding in a Taylor series around point $x_k$

$$
    f(x_k + \delta_k) \approx  f(x_k) + f'(x_k) \delta_k  + O(\delta_k^2)
$$

substituting into $f(x_{k+1})=0$ and dropping the higher order terms gives

$$
    f(x_k) + f'(x_k) \delta_k  =0
$$

substituting into $f(x_{k+1})=0$ and dropping the higher order terms gives

$$
    f(x_k) + f'(x_k) \delta_k  =0
$$

or solving for the correction
$$
    \delta_k = -f(x_k)/f'(x_k)
$$

which leads to the update for the next iteration
$$
    x_{k+1} = x_k + \delta_k
$$

or

$$
    x_{k+1} = x_k -f(x_k)/f'(x_k)
$$

rinse and repeat, as it's still unlikely that $f(x_{k+1})=0$ (but we hope the error will be reduced)

### Algorithm

1. Initialize $x = x_0$
1. While ( $f(x) > tol$ )
  - solve $\delta = -f(x)/f'(x)$
  - update $x \leftarrow x + \delta$

### Geometric interpretation

By truncating the taylor series at first order,  we are locally approximating $f(x)$ as a straight line tangent to the point $f(x_k)$.  If the function was linear at that point, we could find its intercept such that $f(x_k+\delta_k)=0$

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2

# Initial guess
x_k = 0.06

# Setup figure to plot convergence
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

# Plot x_k point
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_k, f(x_k), 'ko')
axes.text(x_k, -5e4, "$x_k$", fontsize=16)
axes.plot(x_k, 0.0, 'xk')
axes.text(x_k, f(x_k) + 2e4, "$f(x_k)$", fontsize=16)
axes.plot(r, f_prime(x_k) * (r - x_k) + f(x_k), 'k')

# Plot x_{k+1} point
x_k = x_k - f(x_k) / f_prime(x_k)
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_k, f(x_k), 'ko')
axes.text(x_k, 1e4, "$x_{k+1}$", fontsize=16)
axes.plot(x_k, 0.0, 'xk')
axes.text(0.0873, f(x_k) - 2e4, "$f(x_{k+1})$", fontsize=16)

axes.set_xlabel("r",fontsize=16)
axes.set_ylabel("f(r)",fontsize=16)
axes.set_title("Newton-Raphson Steps",fontsize=18)
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.grid()

plt.show()
```

### Some code

```python
def newton(f,f_prime,x0,tol = 1.e-6):
    """ uses newton's method to find a root x of a function of a single variable f

    Parameters:
    -----------
    f: function f(x)
        returns type: float
    f_prime: function f'(x)
        returns type: float
    x0: float
        initial guess
    tolerance: float
        Returns when |f(x)| < tol

    Returns:
    --------
    x: float
        final iterate
    x_array: numpy array
        history of iteration points

    Raises:
    -------
    Warning:
        if number of iterations exceed MAX_STEPS
    """
    MAX_STEPS = 200

    x = x0
    x_array = [ x0 ]
    for k in range(1, MAX_STEPS + 1):
        x = x  - f(x) / f_prime(x)
        x_array.append(x)
        if numpy.abs(f(x)) < tol:
            break

    if k == MAX_STEPS:
        warnings.warn('Maximum number of steps exceeded')

    return x, numpy.array(x_array)
```

### Set the problem up

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2
```

### and solve

```python
x0 = 0.06
x, x_array = newton(f, f_prime, x0, tol=1.e-8)
print('x = {}, f(x) = {}, Nsteps = {}'.format(x, f(x), len(x_array)))
print(f_prime(x)*numpy.finfo('float').eps)
```

    x = 0.08985602483470316, f(x) = 5.122274160385132e-09, Nsteps = 6
    -2.819760168457753e-09

```python
r = numpy.linspace(0.05, 0.10, 100)

# Setup figure to plot convergence
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

for n, x in enumerate(x_array):
    axes.plot(x, f(x),'kx')
    axes.text(x, f(x), str(n), fontsize="15")

axes.set_xlabel("r", fontsize=16)
axes.set_ylabel("f(r)", fontsize=16)
axes.set_title("Newton-Raphson Steps", fontsize=18)
axes.grid()
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

axes = fig.add_subplot(1, 2, 2)
axes.semilogy(numpy.arange(len(x_array)), numpy.abs(f(x_array)), 'bo-')
axes.grid()
axes.set_xlabel('Iterations', fontsize=16)
axes.set_ylabel('Residual $|f(r)|$', fontsize=16)
axes.set_title('Convergence', fontsize=18)

plt.show()
```

What is the smallest tolerance that can be achieved with this routine?  Why?

### Example:

$$f(x) = x - e^{-x}$$

$$f'(x) = 1 + e^{-x}$$

$$x_{k+1} = x_k - \frac{f(x_k)}{f'(x_k)} = x_k - \frac{x_k - e^{-x_k}}{1 + e^{-x_k}}$$

#### setup in sympy

```python
x = sympy.symbols('x')
f = x - sympy.exp(-x)
f_prime = f.diff(x)
f, f_prime
```

#### and solve

```python
f = sympy.lambdify(x,f)
f_prime = sympy.lambdify(x,f_prime)
x0 = 0.
x, x_array = newton(f, f_prime, x0, tol = 1.e-9)
print('x = {}, f(x) = {}, Nsteps = {}'.format(x, f(x), len(x_array)))
```

    x = 0.5671432904097811, f(x) = -4.440892098500626e-15, Nsteps = 5

```python
xa = numpy.linspace(-1,1,100)
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1,2,1)
axes.plot(xa,f(xa),'b')
axes.plot(xa,numpy.zeros(xa.shape),'r--')
axes.plot(x,f(x),'go', markersize=10)
axes.plot(x0,f(x0),'kx',markersize=10)
axes.grid()
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_title('$f(x) = x - e^{-x}$', fontsize=18)

axes = fig.add_subplot(1, 2, 2)
axes.semilogy(numpy.arange(len(x_array)), numpy.abs(f(x_array)), 'bo-')
axes.grid()
axes.set_xlabel('Iterations', fontsize=16)
axes.set_ylabel('Residual $|f(r)|$', fontsize=16)
axes.set_title('Convergence', fontsize=18)

plt.show()
```

### Asymptotic Convergence of Newton's Method

Newton's method can be also considered a fixed point iteration

$$x_{k+1} = g(x_k)$$

with  $g(x) = x - \frac{f(x)}{f'(x)}$

Again if $x^*$ is the fixed point and $e_k$ the error at iteration $k$:

$$x_{k+1} = x^* + e_{k+1} \quad \quad x_k = x^* + e_k$$

Taylor Expansion around $x^*$

$$
    x^* + e_{k+1} = g(x^* + e_k) = g(x^*) + g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2!} + O(e_k^3)
$$

Note that as before $x^*$ and $g(x^*)$ cancel:

$$e_{k+1} = g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2!} + \ldots$$

What about $g'(x^*)$ though?

$$\begin{aligned}
    g(x) &= x - \frac{f(x)}{f'(x)} \\
    g'(x) & = 1 - \frac{f'(x)}{f'(x)} + \frac{f(x) f''(x)}{(f'(x))^2} = \frac{f(x) f''(x)}{(f'(x))^2}
\end{aligned}$$

which evaluated at $x = x^*$ becomes

$$
    g'(x^*) = \frac{f(x^*)f''(x^*)}{f'(x^*)^2} = 0
$$

since $f(x^\ast) = 0$ by definition (assuming $f''(x^\ast)$ and $f'(x^\ast)$ are appropriately behaved).

Back to our expansion we have again

$$
    e_{k+1} = g'(x^*) e_k + \frac{g''(x^*) e_k^2}{2!} + \ldots
$$

which simplifies to

$$
    e_{k+1} = \frac{g''(x^*) e_k^2}{2!} + \ldots
$$

which leads to
$$
    |e_{k+1}| < \left | \frac{g''(x^*)}{2!} \right | |e_k|^2
$$

Newton's method is therefore quadratically convergent where  the constant is controlled by the second derivative.

#### Example:  Convergence for a non-simple root

Consider our first problem

$$
    f(x) = x^2 + x - \sin(x)
$$

the case is, unfortunately, not as rosey.  Why might this be?

#### Setup the problem

```python
f = lambda x: x*x + x - numpy.sin(x)
f_prime = lambda x: 2*x + 1. - numpy.cos(x)
x0 = .9

x, x_array = newton(f, f_prime, x0, tol= 1.e-16)
print('x = {}, f(x) = {}, Nsteps = {}'.format(x, f(x), len(x_array)))
```

    x = 7.608160088543346e-09, f(x) = 5.788409914373972e-17, Nsteps = 28

```python
xa = numpy.linspace(-2,2,100)
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1,2,1)
axes.plot(xa,f(xa),'b')
axes.plot(xa,numpy.zeros(xa.shape),'r--')
axes.plot(x,f(x),'go', markersize=10)
axes.plot(x0,f(x0),'kx', markersize=10)
axes.grid()
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_title('$f(x) = x^2 +x - sin(x)$', fontsize=18)

axes = fig.add_subplot(1, 2, 2)
axes.semilogy(numpy.arange(len(x_array)), numpy.abs(f(x_array)), 'bo-')
axes.grid()
axes.set_xlabel('Iterations', fontsize=16)
axes.set_ylabel('Residual $|f(r)|$', fontsize=16)
axes.set_title('Convergence', fontsize=18)

plt.show()
```

### Convergence appears linear, can you show this?:
$$f(x) = x^2 + x -\sin (2 \pi x)$$

### Example: behavior of Newton with multiple roots
$f(x) = \sin (2 \pi x)$

$$x_{k+1} = x_k - \frac{\sin (2 \pi x)}{2 \pi \cos (2 \pi x)}= x_k - \frac{1}{2 \pi} \tan (2 \pi x)$$

```python
x = numpy.linspace(0, 2, 1000)
f = lambda x: numpy.sin(2.0 * numpy.pi * x)
f_prime = lambda x: 2.0 * numpy.pi * numpy.cos(2.0 * numpy.pi * x)
x_kp = lambda x: x - f(x)/f_prime(x)

fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(x, f(x),'b')
axes.plot(x, f_prime(x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_title("Comparison of $f(x)$ and $f'(x)$")
axes.set_ylim((-2,2))
axes.set_xlim((0,2))
axes.plot(x, numpy.zeros(x.shape), 'k--')

x_k = 0.3
axes.plot([x_k, x_k], [0.0, f(x_k)], 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x, f_prime(x_k) * (x - x_k) + f(x_k), 'k')

x_k = x_k - f(x_k) / f_prime(x_k)
axes.plot([x_k, x_k], [0.0, f(x_k)], 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')

axes = fig.add_subplot(1, 2, 2)
axes.plot(x, f(x),'b')
axes.plot(x, x_kp(x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_title("Comparison of $f(x)$ and $x_{k+1}(x)$",fontsize=18)
axes.set_ylim((-2,2))
axes.set_xlim((0,2))
axes.plot(x, numpy.zeros(x.shape), 'k--')

plt.show()
```

### Basins of Attraction

Given a point $x_0$ can we determine if Newton-Raphson converges and to **which root** it converges to?

A *basin of attraction* $X$ for Newton's methods is defined as the set such that $\forall x \in X$ Newton iterations converges to the same root.  Unfortunately this is far from a trivial thing to determine and even for simple functions can lead to regions that are complicated or even fractal.

```python
# calculate the basin of attraction for f(x) = sin(2\pi x)

x_root = numpy.zeros(x.shape)
N_steps = numpy.zeros(x.shape)
for i,xk in enumerate(x):
    x_root[i], x_root_array = newton(f, f_prime, xk)
    N_steps[i] = len(x_root_array)

```

```python
y = numpy.linspace(-2,2)
X,Y = numpy.meshgrid(x,y)
X_root = numpy.outer(numpy.ones(y.shape),x_root)

plt.figure(figsize=(8, 6))
plt.pcolor(X, Y, X_root,vmin=-5, vmax=5,cmap='seismic')
cbar = plt.colorbar()
cbar.set_label('$x_{root}$', fontsize=18)
plt.plot(x, f(x), 'k-')
plt.plot(x, numpy.zeros(x.shape),'k--', linewidth=0.5)
plt.xlabel('x', fontsize=16)
plt.title('Basins of Attraction: $f(x) = \sin{2\pi x}$', fontsize=18)
#plt.xlim(0.25-.1,0.25+.1)
plt.show()
```

### Fractal Basins of Attraction

If $f(x)$ is complex (for $x$ complex), then the basins of attraction can be beautiful and fractal

Plotted below are two fairly simple equations which demonstrate the issue:
1. $f(x) = x^3 - 1$
2. Kepler's equation $\theta - e \sin \theta = M$

```python
f = lambda x: x**3 - 1
f_prime = lambda x: 3 * x**2

N = 1001
x = numpy.linspace(-2, 2, N)
X, Y = numpy.meshgrid(x, x)
R = X + 1j * Y

for i in range(30):
    R = R - f(R) / f_prime(R)

roots = numpy.roots([1., 0., 0., -1])
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
fig.set_figheight(fig.get_figheight() * 2)
axes = fig.add_subplot(1, 1, 1, aspect='equal')
#axes.contourf(X, Y, numpy.sign(numpy.imag(R))*numpy.abs(R),vmin = -10, vmax = 10)
axes.contourf(X, Y, R, vmin = -8, vmax= 8.)
axes.scatter(numpy.real(roots), numpy.imag(roots))
axes.set_xlabel("Real")
axes.set_ylabel("Imaginary")
axes.set_title("Basin of Attraction for $f(x) = x^3 - 1$")
axes.grid()
plt.show()
```

```python
def f(theta, e=0.083, M=1):
    return theta - e * numpy.sin(theta) - M
def f_prime(theta, e=0.083):
    return 1 - e * numpy.cos(theta)

N = 1001
x = numpy.linspace(-30.5, -29.5, N)
y = numpy.linspace(-17.5, -16.5, N)
X, Y = numpy.meshgrid(x, y)
R = X + 1j * Y

for i in range(30):
    R = R - f(R) / f_prime(R)
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
fig.set_figheight(fig.get_figheight() * 2)
axes = fig.add_subplot(1, 1, 1, aspect='equal')
axes.contourf(X, Y, R, vmin = 0, vmax = 10)
axes.set_xlabel("Real")
axes.set_ylabel("Imaginary")
axes.set_title("Basin of Attraction for $f(x) = x - e \sin x - M$")
plt.show()
```

#### Other Issues

Need to supply both $f(x)$ and $f'(x)$, could be expensive

Example:  FTV equation $f(r) = A - \frac{m P}{r} \left[ \left(1 + \frac{r}{m} \right )^{m n} - 1\right]$

Can use symbolic differentiation (`sympy`)

### Secant Methods

Is there a method with the convergence of Newton's method but without the extra derivatives?  What way would you modify Newton's method so that you would not need $f'(x)$?

Given $x_k$ and $x_{k-1}$ represent the derivative as the approximation

$$f'(x) \approx \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}}$$

Combining this with the Newton approach leads to

$$x_{k+1} = x_k - \frac{f(x_k) (x_k - x_{k-1}) }{f(x_k) - f(x_{k-1})}$$

This leads to superlinear convergence and not quite quadratic as the exponent on the convergence is $\approx 1.7$.

Alternative interpretation, fit a line through two points and see where they intersect the x-axis.

$$(x_k, f(x_k)) ~~~~~ (x_{k-1}, f(x_{k-1})$$

$$y = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x - x_k) + b$$

$$b = f(x_{k-1}) - \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x_{k-1} - x_k)$$

$$ y = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x - x_k) + f(x_k)$$

Now solve for $x_{k+1}$ which is where the line intersects the x-axies ($y=0$)

$$0 = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x_{k+1} - x_k) + f(x_k)$$

$$x_{k+1} = x_k - \frac{f(x_k)  (x_k - x_{k-1})}{f(x_k) - f(x_{k-1})}$$

#### Secant Method

$$x_{k+1} = x_k - \frac{f(x_k)  (x_k - x_{k-1})}{f(x_k) - f(x_{k-1})}$$

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

# Initial guess
x_k = 0.07
x_km = 0.06

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

axes.plot(x_k, 0.0, 'ko')
axes.plot(x_k, f(x_k), 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_km, 0.0, 'ko')
axes.plot(x_km, f(x_km), 'ko')
axes.plot([x_km, x_km], [0.0, f(x_km)], 'k--')

axes.plot(r, (f(x_k) - f(x_km)) / (x_k - x_km) * (r - x_k) + f(x_k), 'k')
x_kp = x_k - (f(x_k) * (x_k - x_km) / (f(x_k) - f(x_km)))
axes.plot(x_kp, 0.0, 'ro')
axes.plot([x_kp, x_kp], [0.0, f(x_kp)], 'r--')
axes.plot(x_kp, f(x_kp), 'ro')

axes.set_xlabel("r", fontsize=16)
axes.set_ylabel("f(r)", fontsize=14)
axes.set_title("Secant Method", fontsize=18)
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.grid()

plt.show()
```

What would the algorithm look like for such a method?

#### Algorithm

Given $f(x)$,   a `TOLERANCE`, and a `MAX_STEPS`

1. Initialize two points $x_0$, $x_1$, $f_0 = f(x_0)$, and $f_1 = f(x_1)$
2. Loop for k=2, to  `MAX_STEPS` is reached or `TOLERANCE` is achieved
   1. Calculate new update
   $$x_{2} = x_1 - \frac{f(x_1)  (x_1 - x_{0})}{f(x_1) - f(x_{0})}$$
   2. Check for convergence and break if reached
   3. Update parameters $x_0 = x_1$, $x_1 = x_{2}$, $f_0 = f_1$ and $f_1 = f(x_1)$

#### Some Code

```python
def secant(f, x0, x1, tol = 1.e-6):
    """ uses a linear secant method to find a root x of a function of a single variable f

    Parameters:
    -----------
    f: function f(x)
        returns type: float
    x0: float
        first point to initialize the algorithm
    x1: float
        second point to initialize the algorithm x1 != x0
    tolerance: float
        Returns when |f(x)| < tol

    Returns:
    --------
    x: float
        final iterate
    x_array: numpy array
        history of iteration points

    Raises:
    -------
    ValueError:
        if x1 is too close to x0
    Warning:
        if number of iterations exceed MAX_STEPS
    """
    MAX_STEPS = 200

    if numpy.isclose(x0, x1):
        raise ValueError('Initial points are too close (preferably should be a bracket)')

    x_array = [ x0, x1 ]
    for k in range(1, MAX_STEPS + 1):
        x2 = x1 - f(x1) * (x1 - x0) / (f(x1) - f(x0))
        x_array.append(x2)
        if numpy.abs(f(x2)) < tol:
            break
        x0 = x1
        x1 = x2

    if k == MAX_STEPS:
        warnings.warn('Maximum number of steps exceeded')

    return x2, numpy.array(x_array)
```

### Set the problem up

```python
P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

```

### and solve

```python
x0 = 0.06
x1 = 0.07

x, x_array = secant(f, x0, x1, tol= 1.e-7)
print('x = {}, f(x) = {}, Nsteps = {}'.format(x, f(x), len(x_array)))
```

    x = 0.08985602483470356, f(x) = 9.66247171163559e-09, Nsteps = 8

```python
r = numpy.linspace(0.05, 0.10, 100)

# Setup figure to plot convergence
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

for n, x in enumerate(x_array):
    axes.plot(x, f(x),'kx')
    axes.text(x, f(x), str(n), fontsize="15")

axes.set_xlabel("r", fontsize=16)
axes.set_ylabel("f(r)", fontsize=16)
axes.set_title("Secant Method Steps", fontsize=18)
axes.grid()
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

axes = fig.add_subplot(1, 2, 2)
axes.semilogy(numpy.arange(len(x_array)), numpy.abs(f(x_array)), 'bo-')
axes.grid()
axes.set_xlabel('Iterations', fontsize=16)
axes.set_ylabel('Residual $|f(r)|$', fontsize=16)
axes.set_title('Convergence', fontsize=18)

plt.show()
```

#### Comments

 - Secant method as shown is equivalent to linear interpolation
 - Can use higher order interpolation for higher order secant methods
 - Convergence is not quite quadratic
 - Not guaranteed to converge
 - Does not preserve brackets
 - Almost as good as Newton's method if your initial guess is good.

### Hybrid Methods

Combine attributes of methods with others to make one great algorithm to rule them all (not really)

#### Goals
1. Robustness:  Given a bracket $[a,b]$, maintain bracket
1. Efficiency:  Use superlinear convergent methods when possible

#### Options
 - Methods requiring $f'(x)$
   - NewtSafe (RootSafe, Numerical Recipes)
   - Newton's Method within a bracket, Bisection otherwise
 - Methods not requiring $f'(x)$
   - Brent's Algorithm (zbrent, Numerical Recipes)
     - Combination of bisection, secant and inverse quadratic interpolation
   - `scipy.optimize` package

```python
from scipy.optimize import brentq
a = 0.07
b = 0.1
x, res  = brentq(f, a, b, full_output=True)
print('x = {}, f(x) = {}'.format(x, f(x)))
print(res)
#brentq?
```

    x = 0.08985602483470466, f(x) = 2.1886080503463745e-08
          converged: True
               flag: 'converged'
     function_calls: 8
         iterations: 7
               root: 0.08985602483470466

## Optimization (finding extrema)

I want to find the extrema of a function $f(x)$ on a given interval $[a,b]$.

A few approaches:
 - Interpolation Algorithms: Repeated parabolic interpolation
 - Bracketing Algorithms:  Golden-Section Search (linear)
 - Hybrid Algorithms

Note: for continuous functions, root finding is much simpler.

### Interpolation Approach

Successive parabolic interpolation - *similar to secant method*

Basic idea:  Fit polynomial to function using three points, find its minima, and guess new points based on that minima

Good *if* you know 3 points that are close to the extremum

1. What do we need to fit a polynomial $p_n(x)$ of degree $n \geq 2$?

2. How do we construct the polynomial $p_2(x)$?

3. Once we have constructed $p_2(x)$ how would we find the minimum?

#### Algorithm

Given $f(x)$ and $[x_0,x_1]$ - Note that unlike a bracket these will be a sequence of better approximations to the minimum.
1. Initialize $x = [x_0, x_1, (x_0+x_1)/2]$
1. Loop
   1. Evaluate function $f(x)$ at the three points
   1. Find the quadratic polynomial that interpolates those points:

      $$p(x) = p_0 x^2 + p_1 x + p_2$$

   3. Calculate the minimum:

      $$p'(x) = 2 p_0 x + p_1 = 0 \quad \Rightarrow \quad x^\ast = -p_1 / (2 p_0)$$

   1. New set of points $x = [x_1, (x_0+x_1)/2, x^\ast]$
   1. Check tolerance

Not guaranteed to converge, or find global minimum/maximum

```python
def f(t):
    """Simple function for minimization demos"""
    return -3.0 * numpy.exp(-(t - 0.3)**2 / (0.1)**2) \
           +      numpy.exp(-(t - 0.6)**2 / (0.2)**2) \
           +      numpy.exp(-(t - 1.0)**2 / (0.2)**2) \
           +      numpy.sin(t)                        \
           -      2.0
```

```python
MAX_STEPS = 100
TOLERANCE = 1e-4

x = numpy.array([0.5, 0.2, (0.7) / 2.0])

t = numpy.linspace(0, 2, 200)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")
axes.plot(x[0], f(x[0]), 'ko')
axes.plot(x[1], f(x[1]), 'ko')

success = False
for n in range(1, MAX_STEPS + 1):
    axes.plot(x[2], f(x[2]), 'ko')
    poly = numpy.polyfit(x, f(x), 2)
    axes.plot(t, poly[0] * t**2 + poly[1] * t + poly[2], 'r--')
    x[0] = x[1]
    x[1] = x[2]
    x[2] = -poly[1] / (2.0 * poly[0])
    if numpy.abs(x[2] - x[1]) / numpy.abs(x[2]) < TOLERANCE:
        success = True
        break

if success:
    print("Success!")
    print("  t* = %s" % x[2])
    print("  f(t*) = %s" % f(x[2]))
    print("  number of steps = %s" % n)
else:
    print("Reached maximum number of steps!")

axes.set_ylim((-5, 0.0))
axes.grid()
plt.show()
```

#### Some Code
Using numpy.polyfit(x, y, degree))

```python
def parabolic_interpolation(f, bracket, tol = 1.e-6):
    """ uses repeated parabolic interpolation to refine a local minimum of a function f(x)
    this routine uses numpy functions polyfit and polyval to fit and evaluate the quadratics

    Parameters:
    -----------
    f: function f(x)
        returns type: float
    bracket: array
        array [x0, x1] containing an initial bracket that contains a minimum
    tolerance: float
        Returns when relative error of last two iterates < tol

    Returns:
    --------
    x: float
        final estimate of the minima
    x_array: numpy array
        history of iteration points

    Raises:
    -------
    Warning:
        if number of iterations exceed MAX_STEPS
    """
    MAX_STEPS = 100

    x = numpy.zeros(3)
    x[:2] = bracket
    x[2] = (x[0] + x[1])/2.

    x_array = [ x[2] ]
    for k in range(1, MAX_STEPS + 1):
        poly = numpy.polyfit(x, f(x), 2)
        x[0] = x[1]
        x[1] = x[2]
        x[2] = -poly[1] / (2.0 * poly[0])
        x_array.append(x[2])
        if numpy.abs(x[2] - x[1]) / numpy.abs(x[2]) < tol:
            break

    if k == MAX_STEPS:
        warnings.warn('Maximum number of steps exceeded')

    return x[2], numpy.array(x_array)
```

#### set up problem

```python
bracket = numpy.array([0.5, 0.2])
x, x_array = parabolic_interpolation(f, bracket, tol = 1.e-6)
print("Extremum f(x) = {}, at x = {}, N steps = {}".format(f(x), x, len(x_array)))
```

    Extremum f(x) = -4.604285452397024, at x = 0.29588830243766273, N steps = 8

```python
t = numpy.linspace(0, 2, 200)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.plot(x_array, f(x_array),'ro')
axes.plot(x, f(x), 'go')
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")
axes.grid()
plt.show()
```

### Bracketing Algorithm (Golden Section Search)

Given $f(x) \in C[x_0,x_3]$ that is convex (concave) over an interval $x \in [x_0,x_3]$ reduce the interval size until it brackets the minimum (maximum).

Note that we no longer have the $x=0$ help we had before so bracketing and doing bisection is a bit trickier in this case.  In particular choosing your initial bracket is important!

#### Bracket Picking

Say we start with a bracket $[x_0, x_3]$ and pick two new points $x_1 < x_2 \in [x_0, x_3]$.  We want to pick a new bracket that guarantees that the extrema exists in it.  We then can pick this new bracket with the following rules:
 - If $f(x_1) < f(x_2)$ then we know the minimum is between $x_0$ and $x_2$.
 - If $f(x_1) > f(x_2)$ then we know the minimum is between $x_1$ and $x_3$.

*Bracket point restriction*: f(x) for 1st and 3rd point must be higher (for minimum) or lower (maximum) than middle point (2nd point).

```python
f = lambda x: x**2

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
fig.set_figheight(fig.get_figheight() * 2)

search_points = [-1.0, -0.5, 0.75, 1.0]
axes = fig.add_subplot(2, 2, 1)
x = numpy.linspace(search_points[0] - 0.1, search_points[-1] + 0.1, 100)
axes.plot(x, f(x), 'b')
for (i, point) in enumerate(search_points):
    axes.plot(point, f(point),'or')
    axes.text(point + 0.05, f(point), str(i))
axes.plot(0, 0, 'sk')
axes.set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
axes.set_title("$f(x_1) < f(x_2) \Rightarrow [x_0, x_2]$")

search_points = [-1.0, -0.75, 0.5, 1.0]
axes = fig.add_subplot(2, 2, 2)
x = numpy.linspace(search_points[0] - 0.1, search_points[-1] + 0.1, 100)
axes.plot(x, f(x), 'b')
for (i, point) in enumerate(search_points):
    axes.plot(point, f(point),'or')
    axes.text(point + 0.05, f(point), str(i))
axes.plot(0, 0, 'sk')
axes.set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
axes.set_title("$f(x_1) > f(x_2) \Rightarrow [x_1, x_3]$")

search_points = [-1.0, 0.25, 0.75, 1.0]
axes = fig.add_subplot(2, 2, 3)
x = numpy.linspace(search_points[0] - 0.1, search_points[-1] + 0.1, 100)
axes.plot(x, f(x), 'b')
for (i, point) in enumerate(search_points):
    axes.plot(point, f(point),'or')
    axes.text(point + 0.05, f(point), str(i))
axes.plot(0, 0, 'sk')
axes.set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
axes.set_title("$f(x_1) < f(x_2) \Rightarrow [x_0, x_2]$")

search_points = [-1.0, -0.75, -0.25, 1.0]
axes = fig.add_subplot(2, 2, 4)
x = numpy.linspace(search_points[0] - 0.1, search_points[-1] + 0.1, 100)
axes.plot(x, f(x), 'b')
for (i, point) in enumerate(search_points):
    axes.plot(point, f(point),'or')
    axes.text(point + 0.05, f(point), str(i))
axes.plot(0, 0, 'sk')
axes.set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
axes.set_title("$f(x_1) > f(x_2) \Rightarrow [x_1, x_3]$")

plt.show()
```

#### Picking Brackets and Points

Again say we have a bracket $[x_0,x_3]$ and suppose we have two new search points $x_1$ and $x_2$ that separates $[x_0,x_3]$ into two new overlapping brackets.

Define:  the length of the line segments in the interval
\begin{aligned}
    a &= x_1 - x_0, \\
    b &= x_2 - x_1,\\
    c &= x_3 - x_2 \\
\end{aligned}
and the total bracket length
\begin{aligned}
    d &= x_3 - x_0. \\
\end{aligned}

```python
f = lambda x: (x - 0.25)**2 + 0.5
phi = (numpy.sqrt(5.0) - 1.) / 2.0

x = [-1.0, None, None, 1.0]
x[1] = x[3] - phi * (x[3] - x[0])
x[2] = x[0] + phi * (x[3] - x[0])

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
t = numpy.linspace(-2.0, 2.0, 100)
axes.plot(t, f(t), 'k')
# First set of intervals
axes.plot([x[0], x[1]], [0.0, 0.0], 'g',label='a')
axes.plot([x[1], x[2]], [0.0, 0.0], 'r', label='b')
axes.plot([x[2], x[3]], [0.0, 0.0], 'b', label='c')
axes.plot([x[0], x[3]], [2.5, 2.5], 'c', label='d')

axes.plot([x[0], x[0]], [0.0, f(x[0])], 'g--')
axes.plot([x[1], x[1]], [0.0, f(x[1])], 'g--')
axes.plot([x[1], x[1]], [0.0, f(x[1])], 'r--')
axes.plot([x[2], x[2]], [0.0, f(x[2])], 'r--')
axes.plot([x[2], x[2]], [0.0, f(x[2])], 'b--')
axes.plot([x[3], x[3]], [0.0, f(x[3])], 'b--')
axes.plot([x[0], x[0]], [2.5, f(x[0])], 'c--')
axes.plot([x[3], x[3]], [2.5, f(x[3])], 'c--')

points = [ (x[0] + x[1])/2., (x[1] + x[2])/2., (x[2] + x[3])/2., (x[0] + x[3])/2. ]
y      = [ 0., 0., 0., 2.5]
labels = [ 'a', 'b', 'c', 'd']

for (n, point)  in enumerate(points):
    axes.text(point, y[n] + 0.1, labels[n], fontsize=15)

for (n, point) in enumerate(x):
    axes.plot(point, f(point), 'ok')
    axes.text(point, f(point)+0.1, n, fontsize='15')

axes.set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
axes.set_ylim((-1.0, 3.0))

plt.show()
```

For **Golden Section Search** we require two conditions:
 - The two new possible brackets are of equal length.  i.e $[x_0, x_2] =  [x_1, x_3]$ or

 $$
     a + b = b + c
 $$

 or simply $a = c$

```python
f = lambda x: (x - 0.25)**2 + 0.5
phi = (numpy.sqrt(5.0) - 1.) / 2.0

x = [-1.0, None, None, 1.0]
x[1] = x[3] - phi * (x[3] - x[0])
x[2] = x[0] + phi * (x[3] - x[0])

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
t = numpy.linspace(-2.0, 2.0, 100)
axes.plot(t, f(t), 'k')
# First set of intervals
axes.plot([x[0], x[1]], [0.0, 0.0], 'g',label='a')
axes.plot([x[1], x[2]], [0.0, 0.0], 'r', label='b')
axes.plot([x[2], x[3]], [0.0, 0.0], 'b', label='c')
axes.plot([x[0], x[3]], [2.5, 2.5], 'c', label='d')

axes.plot([x[0], x[0]], [0.0, f(x[0])], 'g--')
axes.plot([x[1], x[1]], [0.0, f(x[1])], 'g--')
axes.plot([x[1], x[1]], [0.0, f(x[1])], 'r--')
axes.plot([x[2], x[2]], [0.0, f(x[2])], 'r--')
axes.plot([x[2], x[2]], [0.0, f(x[2])], 'b--')
axes.plot([x[3], x[3]], [0.0, f(x[3])], 'b--')
axes.plot([x[0], x[0]], [2.5, f(x[0])], 'c--')
axes.plot([x[3], x[3]], [2.5, f(x[3])], 'c--')

points = [ (x[0] + x[1])/2., (x[1] + x[2])/2., (x[2] + x[3])/2., (x[0] + x[3])/2. ]
y      = [ 0., 0., 0., 2.5]
labels = [ 'a', 'b', 'c', 'd']

for (n, point)  in enumerate(points):
    axes.text(point, y[n] + 0.1, labels[n], fontsize=15)

for (n, point) in enumerate(x):
    axes.plot(point, f(point), 'ok')
    axes.text(point, f(point)+0.1, n, fontsize='15')

axes.set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
axes.set_ylim((-1.0, 3.0))

plt.show()
```

 - The ratio of segment lengths is the same for every level of recursion so the problem is *self-similar* i.e.

 $$
     \frac{b}{a} = \frac{c}{a + b}
 $$

 These two requirements will allow *maximum reuse* of previous points and require adding only one new point $x^*$ at each iteration.

```python
f = lambda x: (x - 0.25)**2 + 0.5
phi = (numpy.sqrt(5.0) - 1.) / 2.0

x = [-1.0, None, None, 1.0]
x[1] = x[3] - phi * (x[3] - x[0])
x[2] = x[0] + phi * (x[3] - x[0])

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
axes = []
axes.append(fig.add_subplot(1, 2, 1))
axes.append(fig.add_subplot(1, 2, 2))
t = numpy.linspace(-2.0, 2.0, 100)
for i in range(2):
    axes[i].plot(t, f(t), 'k')

    # First set of intervals
    axes[i].plot([x[0], x[2]], [0.0, 0.0], 'g')
    axes[i].plot([x[1], x[3]], [-0.2, -0.2], 'r')
    axes[i].plot([x[0], x[0]], [0.0, f(x[0])], 'g--')
    axes[i].plot([x[2], x[2]], [0.0, f(x[2])], 'g--')
    axes[i].plot([x[1], x[1]], [-0.2, f(x[1])], 'r--')
    axes[i].plot([x[3], x[3]], [-0.2, f(x[3])], 'r--')
    for (n, point) in enumerate(x):
        axes[i].plot(point, f(point), 'ok')
        axes[i].text(point, f(point)+0.1, n, fontsize='15')

    axes[i].set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
    axes[i].set_ylim((-1.0, 3.0))

# Left new interval
x_new = [x[0], None, x[1], x[2]]
x_new[1] = phi * (x[1] - x[0]) + x[0]

#axes[0].plot([x_new[0], x_new[2]], [1.5, 1.5], 'b')
#axes[0].plot([x_new[1], x_new[3]], [1.75, 1.75], 'c')
#axes[0].plot([x_new[0], x_new[0]], [1.5, f(x_new[0])], 'b--')
#axes[0].plot([x_new[2], x_new[2]], [1.5, f(x_new[2])], 'b--')
#axes[0].plot([x_new[1], x_new[1]], [1.75, f(x_new[1])], 'c--')
#axes[0].plot([x_new[3], x_new[3]], [1.75, f(x_new[3])], 'c--')
axes[0].plot(x_new[1], f(x_new[1]), 'ko')
axes[0].text(x_new[1], f(x_new[1]) + 0.1, "*", fontsize='15')
for i in range(4):
    axes[0].text(x_new[i], -0.5, i, color='g',fontsize='15')

# Right new interval
x_new = [x[1], x[2], None, x[3]]
x_new[2] = (x[2] - x[1]) * phi + x[2]

#axes[1].plot([x_new[0], x_new[2]], [1.25, 1.25], 'b')
#axes[1].plot([x_new[1], x_new[3]], [1.5, 1.5], 'c')
#axes[1].plot([x_new[0], x_new[0]], [1.25, f(x_new[0])], 'b--')
#axes[1].plot([x_new[2], x_new[2]], [1.25, f(x_new[2])], 'b--')
#axes[1].plot([x_new[1], x_new[1]], [1.5, f(x_new[2])], 'c--')
#axes[1].plot([x_new[3], x_new[3]], [1.5, f(x_new[3])], 'c--')
axes[1].plot(x_new[2], f(x_new[2]), 'ko')
axes[1].text(x_new[2], f(x_new[2]) + 0.1, "*", fontsize='15')
for i in range(4):
    axes[1].text(x_new[i], -0.5, i, color='r',fontsize='15')
axes[0].set_title('Choose left bracket', fontsize=18)
axes[1].set_title('Choose right bracket', fontsize=18)

plt.show()
```

As the first rule implies that $a = c$,  we can substitute into the second rule to yield

$$
     \frac{b}{a} = \frac{a}{a + b}
$$

or inverting and rearranging
$$
    \frac{a}{b} = 1 + \frac{b}{a}
$$

if we let the ratio $b/a = x$,  then
$$
    x + 1 = \frac{1}{x} \quad \text{or} \quad x^2 + x - 1 = 0
$$

$$
     x^2 + x - 1 = 0
$$
has a single positive root for
$$
    x = \frac{\sqrt{5} - 1}{2} = \varphi = 0.6180339887498949
$$
where $\varphi$ is related to the  **"golden ratio"** (which in most definitions is given by $1+\varphi$, but either work as $ 1+\varphi = 1/\varphi $ )

*So* $x = \frac{b}{a} = \varphi = 0.6180339887498949$

Subsequent proportionality implies that the distances between the 4 points at one iteration is proportional to the next.  We can now use all of our information to find the points $x_1$ and $x_2$ given any overall bracket $[x_0, x_3]$

Given $b/a = \varphi$,   $a = c$, and the known width of the bracket $d$ it follows that

$$ d = a + b + c = (2 + \phi)a $$
or
$$ a = \frac{d}{2 + \varphi} = \frac{\varphi}{1 + \varphi} d$$
by the rather special properties of $\varphi$.

We could use this result immediately to find

\begin{align}
    x_1 &= x_0 + a \\
    x_2 &= x_3 - a \\
\end{align}

Equivalently, you can show that

$$a + b = (1 + \varphi)a = \varphi d$$

so, to get middle 2 points $x_1$ and $x_2$:

\begin{align}
    x_1 &= x_3 - \varphi d \\
    x_2 &= x_0 + \varphi d \\
\end{align}

```python
f = lambda x: (x - 0.25)**2 + 0.5
phi = (numpy.sqrt(5.0) - 1.) / 2.0

x = [-1.0, None, None, 1.0]
x[1] = x[3] - phi * (x[3] - x[0])
x[2] = x[0] + phi * (x[3] - x[0])

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
t = numpy.linspace(-2.0, 2.0, 100)
axes.plot(t, f(t), 'k')
# First set of intervals
axes.plot([x[0], x[1]], [0.0, 0.0], 'g',label='a')
axes.plot([x[1], x[2]], [0.0, 0.0], 'r', label='b')
axes.plot([x[2], x[3]], [0.0, 0.0], 'b', label='c')
axes.plot([x[0], x[3]], [2.5, 2.5], 'c', label='d')

axes.plot([x[0], x[0]], [0.0, f(x[0])], 'g--')
axes.plot([x[1], x[1]], [0.0, f(x[1])], 'g--')
axes.plot([x[1], x[1]], [0.0, f(x[1])], 'r--')
axes.plot([x[2], x[2]], [0.0, f(x[2])], 'r--')
axes.plot([x[2], x[2]], [0.0, f(x[2])], 'b--')
axes.plot([x[3], x[3]], [0.0, f(x[3])], 'b--')
axes.plot([x[0], x[0]], [2.5, f(x[0])], 'c--')
axes.plot([x[3], x[3]], [2.5, f(x[3])], 'c--')

points = [ (x[0] + x[1])/2., (x[1] + x[2])/2., (x[2] + x[3])/2., (x[0] + x[3])/2. ]
y      = [ 0., 0., 0., 2.5]
labels = [ 'a', 'b', 'c', 'd']

for (n, point)  in enumerate(points):
    axes.text(point, y[n] + 0.1, labels[n], fontsize=15)

for (n, point) in enumerate(x):
    axes.plot(point, f(point), 'ok')
    axes.text(point, f(point)+0.1, n, fontsize='15')

axes.set_xlim((search_points[0] - 0.1, search_points[-1] + 0.1))
axes.set_ylim((-1.0, 3.0))

plt.show()
```

#### Algorithm
1. Initialize bracket $[x_0,x_3]$
1. Initialize points $x_1 = x_3 - \varphi (x_3 - x_0)$ and $x_2 = x_0 + \varphi (x_3 - x_0)$
1. Loop
   1. Evaluate $f_1$ and $f_2$
   1. If $f_1 < f_2$ then we pick the left interval for the next iteration
   1. and otherwise pick the right interval
   1. Check size of bracket for convergence $x_3 - x_0 <$ `TOLERANCE`
   1. calculate the appropriate new point $x^*$ ($x_1$ on left, $x_2$ on right)

Golden section search for extrema: **linear convergence**

```python
def golden_section(f, bracket, tol = 1.e-6):
    """ uses golden section search to refine a local minimum of a function f(x)
    this routine uses numpy functions polyfit and polyval to fit and evaluate the quadratics

    Parameters:
    -----------
    f: function f(x)
        returns type: float
    bracket: array
        array [x0, x3] containing an initial bracket that contains a minimum
    tolerance: float
        Returns when | x3 - x0 | < tol

    Returns:
    --------
    x: float
        final estimate of the midpoint of the bracket
    x_array: numpy array
        history of midpoint of each bracket

    Raises:
    -------
    ValueError:
        If initial bracket is < tol or doesn't appear to have any interior points
        that are less than the outer points
    Warning:
        if number of iterations exceed MAX_STEPS
    """
    MAX_STEPS = 100
    phi = (numpy.sqrt(5.0) - 1.) / 2.0

    x = [ bracket[0], None, None, bracket[1] ]

    delta_x = x[3] - x[0]
    x[1] = x[3] - phi * delta_x
    x[2] = x[0] + phi * delta_x

    # check for initial bracket
    fx = f(numpy.array(x))
    bracket_min = min(fx[0], fx[3])
    if fx[1] > bracket_min and fx[2] > bracket_min:
        raise ValueError("interval does not appear to include a minimum")
    elif delta_x < tol:
        raise ValueError("interval is already smaller than tol")

    x_mid = (x[3] + x[0])/2.
    x_array = [ x_mid ]
    for k in range(1, MAX_STEPS + 1):
        f_1 = f(x[1])
        f_2 = f(x[2])

        if f_1 < f_2:
            # Pick the left bracket
            x_new = [x[0], None, x[1], x[2]]
            delta_x = x_new[3] - x_new[0]
            x_new[1] = x_new[3] - phi * delta_x
        else:
            # Pick the right bracket
            x_new = [x[1], x[2], None, x[3]]
            delta_x = x_new[3] - x_new[0]
            x_new[2] = x_new[0] + phi * delta_x
        x = x_new
        x_array.append((x[3] + x[0])/ 2.)

        if numpy.abs(x[3] - x[0]) < tol:
            break

    if k == MAX_STEPS:
        warnings.warn('Maximum number of steps exceeded')

    return x_array[-1], numpy.array(x_array)
```

```python
def f(t):
    """Simple function for minimization demos"""
    return -3.0 * numpy.exp(-(t - 0.3)**2 / (0.1)**2) \
           +      numpy.exp(-(t - 0.6)**2 / (0.2)**2) \
           +      numpy.exp(-(t - 1.0)**2 / (0.2)**2) \
           +      numpy.sin(t)                        \
           -      2.0
```

```python
x, x_array = golden_section(f,[0.2, 0.5], 1.e-4)
print('t* = {}, f(t*) = {}, N steps = {}'.format(x, f(x), len(x_array)-1))
```

    t* = 0.29588198925376136, f(t*) = -4.604285440148672, N steps = 17

```python
t = numpy.linspace(0, 2, 200)
fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.grid()
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")
axes.plot(x_array, f(x_array),'ko')
axes.plot(x_array[0],f(x_array[0]),'ro')
axes.plot(x_array[-1],f(x_array[-1]),'go')

plt.show()
```

## Scipy Optimization

Scipy contains a lot of ways for optimization!

```python
import scipy.optimize as optimize
print(optimize.golden(f, brack=(0.2, 0.25, 0.5)))
```

    0.29588830308853215

```python
optimize?
```

```python

```


# Realization of Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Quantization of Variables and Operations

As for [non-recursive filters](../nonrecursive_filters/quantization_effects.ipynb#Quantization-Effects), the practical realization of recursive filters may suffer from the quantization of variables and algebraic operations. The effects of [coefficient quantization](quantization_of_coefficients.ipynb) were already discussed. This section takes a look at the quantization of variables.  We limit the investigations to the recursive part of a second-order section (SOS), since any recursive filter of order $N \geq 2$ can be [decomposed into SOSs](cascaded_structures.ipynb).

The computation of the output signal $y[k] = \mathcal{H}\{ x[k] \}$ by a difference equation involves a number of multiplications and additions. As discussed already for [non-recursive filters](../nonrecursive_filters/quantization_effects.ipynb#Quantization-of-Signals-and-Operations), multiplying two numbers in a binary representation (e.g. [two's complement](https://en.wikipedia.org/wiki/Two's_complement) or [floating point](https://en.wikipedia.org/wiki/Floating_point)) requires requantization of the result to keep the word length constant. The addition of two numbers may fall outside the maximum/minimum values of the representation and may suffer from clipping.

The resulting round-off and clipping errors depend on the number and sequence of algebraic operations. These depend on the structure used for implementation of the SOSs. For ease of illustration we limit our discussion to the [direct form I and II](direct_forms.ipynb). Similar insights can be achieved in a similar manner for other structures.

### Analysis of Round-Off Errors

Round-off errors are a consequence of reducing the word length after a multiplication. In order to investigate the influence of these errors on a recursive filter, the statistical model for [round-off errors in multipliers](../nonrecursive_filters/quantization_effects.ipynb#Model-for-round-off-errors-in-multipliers) as introduced for non-recursive filters is used. We furthermore neglect clipping.

The difference equation for the recursive part of a SOS realized in direct form I or II is given as

\begin{equation}
y[k] =  x[k] - a_1 \, y[k-1] - a_2 \, y[k-2]
\end{equation}

where $a_0 = 1$, $a_1$ and $a_2$ denote the coefficients of the recursive part. Introducing the requantization after the multipliers into the difference equation yields the output signal $y_Q[k]$

\begin{equation}
y_Q[k] = x[k] - \mathcal{Q} \{ a_{1} \, y[k-1] \} - \mathcal{Q} \{ a_{2} \, y[k-2] \}
\end{equation}

where $\mathcal{Q} \{ \cdot \}$ denotes the requantizer. Requantization is a non-linear process which results in a requantization error. If the value to be requantized is much larger that the quantization step $Q$, the average statistical properties of this error can be modeled as additive uncorrelated white noise. Introducing the error into above difference equation gives

\begin{equation}
y_Q[k] = x[k] - a_1 \, y[k-1] - e_1[k] - a_2 \, y[k-2] - e_2[k]
\end{equation}

where the two white noise sources $e_1[k]$ and $e_2[k]$ are assumed to be uncorrelated to each other. This difference equation can be split into a set of two difference equations

\begin{align}
y_Q[k] &= y[k] + e[k] \\
y[k] &=  x[k] - a_1 \, y[k-1] - a_2 \, y[k-2] \\
e[k] &= - e_1[k] - e_2[k] - a_1 \, e[k-1] - a_2 \, e[k-2]
\end{align}

The first difference equation computes the desired output signal $y[k]$ as a result of the input signal $x[k]$. The second one the additive error $e[k]$ due to requantization as a result of the requantization error $- (e_1[k] + e_2[k])$ injected into the recursive filter.
The power spectral density (PSD) $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the error $e[k]$ is then given as

\begin{equation}
\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = | H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 \cdot (\Phi_{e_1 e_1}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{e_2 e_2}(\mathrm{e}^{\,\mathrm{j}\,\Omega}))
\end{equation}

According to the model for the requantization errors, their PSDs are given as $\Phi_{e_1 e_1}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{e_2 e_2}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{Q^2}{12}$. Introducing this together with the transfer function of the SOS yields

\begin{equation}
\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \left| \frac{1}{1 + a_1 \, \mathrm{e}^{\,-\mathrm{j}\,\Omega} + a_2 \, \mathrm{e}^{\,-\mathrm{j}\,2\,\Omega}} \right|^2 \cdot \frac{Q^2}{6}
\end{equation}

#### Example - Round-off error of a SOS

The following example evaluates the error $e[k] = y_Q[k] - y[k]$ for a SOS which only consists of a recursive part. The desired system response $y[k]$ is computed numerically by floating point operations with double precision, $y_Q[k]$ is computed by applying a uniform midtread quantizer after the multiplications. The system is excited by uniformly distributed white noise. Besides the PSD $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, the signal-to-noise ratio (SNR) $10 \cdot \log_{10} \left( \frac{\sigma_y^2}{\sigma_e^2} \right)$ in dB of the filter is evaluated.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 8192  # length of signals
w = 8  # wordlength for requantization of multiplications

def uniform_midtread_quantizer(x):
    '''Uniform mid-tread quantizer w/o limiter.'''
    return Q * np.floor(x/Q + 1/2)

def no_quantizer(x):
    '''Dummy quantizer.'''
    return x

def sos_df1(x, a, requantize=None):
    '''Realization of a recursive SOS with round-off of multiplications.'''
    y = np.zeros(len(x)+2)  # initial value appended
    for k in range(len(x)):
        y[k] = x[k] - requantize(a[1]*y[k-1]) - requantize(a[2]*y[k-2])

    return y[0:-2]

# cofficients of the SOS
p = 0.90*np.array([np.exp(1j*np.pi/3), np.exp(-1j*np.pi/3)])
a = np.poly(p)
# quantization step
Q = 1/(2**(w-1))

# compute input signal
x = np.random.uniform(low=-1, high=1, size=N)
# compute output signals w and w/o requantization
yQ = sos_df1(x, a, requantize=uniform_midtread_quantizer)
y = sos_df1(x, a, requantize=no_quantizer)
# compute requantization error
e = yQ-y
# Signal-to-noise ratio
SNR = 10*np.log10(np.var(y)/np.var(e))
print('SNR due to requantization: %f dB' % SNR)

# estimate PSD of requantization error
nf, Pxx = sig.welch(e, window='hamming', nperseg=256, noverlap=128)
Pxx = .5*Pxx  # due to normalization in scipy.signal
Om = 2*np.pi*nf
# compute frequency response of system
w, H = sig.freqz([1, 0, 0], a)

# plot results
plt.figure(figsize=(10, 4))
plt.plot(Om, Pxx/Q**2 * 12, 'b', label=r'$\hat{\Phi}_{ee}(e^{j \Omega})$')
plt.plot(w, np.abs(H)**2 * 2, 'g', label=r'$|H(e^{j \Omega})|^2$')
plt.title('Estimated PSD and transfer function of requantization noise')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$Q^2/12$')
plt.axis([0, np.pi, 0, 100])
plt.legend()
plt.grid()
```

    SNR due to requantization: 45.442600 dB

### Small Limit Cycles

Besides the requantization noise, recursive filters may be subject to periodic oscillations present at the output. These undesired oscillations are termed *limit cycles*. Small limit cycles emerge from the additive round-off noise due to requantization after a multiplication. The feedback in a recursive filter leads to a feedback of the requantization noise. This may lead to a periodic output signal with an amplitude range of some quantization steps $Q$, even after the input signal is zero. The presence, amplitude and frequency of small limit cycles depends on the location of poles and the structure of the filter. A detailed treatment of this phenomenon is beyond the scope of this notebook and can be found in the literature.

#### Example - Small limit cycles of a SOS

The following example illustrates small limit cycles for the system investigated in the previous example. The input signal is uniformly distributed white noise till time-index $k=256$ and zero for the remainder.

```python
# compute input signal
x = np.random.uniform(low=-1, high=1, size=256)
x = np.concatenate((x, np.zeros(1024)))
# compute output signal
yQ = sos_df1(x, a, requantize=uniform_midtread_quantizer)

# plot results
np.seterr(divide='ignore')
plt.figure(figsize=(10, 3))
plt.plot(20*np.log10(np.abs(yQ)))
plt.title('Level of output signal')
plt.xlabel(r'$k$')
plt.ylabel(r'$|y_Q[k]|$ in dB')
plt.grid()

plt.figure(figsize=(10, 3))
k = np.arange(1000, 1050)
plt.stem(k, yQ[k]/Q, use_line_collection=True)
plt.title('Output signal for zero input')
plt.xlabel(r'$k$')
plt.ylabel(r'$y_Q[k] / Q$ ')
plt.axis([k[0], k[-1], -3, 3])
plt.grid()
```

**Exercise**

* Estimate the period of the small limit cycles. How is it related to the poles of the system?
* What amplitude range is spanned?

Solution: The period of the small limit cycles can be estimated from the second illustration as $P = 6$. The normalized frequency of a harmonic exponential signal with the same periodicity is given as $\Omega_0 = \frac{2 \pi}{P} = \frac{\pi}{3}$. The poles of the system can be extracted from the code of the first example as $z_{\infty 0,1} = 0.9 \cdot e^{\pm j \frac{\pi}{3}}$. The periodicity of the small limit cycles is hence linked to the normalized frequency of the poles. The amplitude range spanned by the small limit cycles is $\pm 2 Q$.

### Large Limit Cycles

Large limit cycles are periodic oscillations of a recursive filter due to overflows in the multiplications/additions. As for small limit cycles, large limit cycles may be present even after the input signal is zero. Their level is typically in the range of the minimum/maximum value of the requantizer. Large limit cycles should therefore be avoided in a practical implementation. The presence of large limit cycles depends on the scaling of input signal and coefficients, as well as the strategy used to cope for clipping. Amongst others, they can be avoided by proper scaling of the coefficients to prevent overflow. Again, a detailed treatment of this phenomenon is beyond the scope of this notebook and can be found in the literature.

#### Example - Large limit cycles of a SOS

The following example illustrates large limit cycles for the system investigated in the first example. In order to trigger large limit cycles, the coefficients of the filter have been doubled. The input signal is uniformly distributed white noise till time-index $k=256$ and zero for the remainder.

```python
def uniform_midtread_quantizer(x, xmin=1):
    '''Uniform mid-tread quantizer with limiter.'''
    # limiter
    x = np.copy(x)
    if x <= -xmin:
        x = -1
    if x > xmin - Q:
        x = 1 - Q
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

# compute input signal
x = np.random.uniform(low=-1, high=1, size=256)
x = np.concatenate((x, np.zeros(1024)))
# compute output signal
yQ = sos_df1(x, 2*a, requantize=uniform_midtread_quantizer)

# plot results
plt.figure(figsize=(10, 3))
plt.plot(20*np.log10(np.abs(yQ)))
plt.title('Level of output signal')
plt.xlabel(r'$k$')
plt.ylabel(r'$|y_Q[k]|$ in dB')
plt.grid()

plt.figure(figsize=(10, 3))
k = np.arange(1000, 1050)
plt.stem(k, yQ[k], use_line_collection=True)
plt.title('Output signal for zero input')
plt.xlabel(r'$k$')
plt.ylabel(r'$y_Q[k]$ ')
plt.grid()
```

**Exercise**

* Determine the period of the large limit cycles. How is it related to the poles of the system?

Solution: The period of the large limit cycles can be estimated from the second illustration as $P = 6$. The normalized frequency of a harmonic exponential signal with the same periodicity is given as $\Omega_0 = \frac{2 \pi}{P} = \frac{\pi}{3}$. The poles of the system can be extracted from the code of the first example as $z_{\infty 0,1} = 0.9 \cdot e^{\pm j \frac{\pi}{3}}$. The periodicity of the large limit cycles is hence linked to the normalized frequency of the poles.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# FFM234, Klassisk fysik och vektorfält - Veckans tal

**[Christian Forssén](http://fy.chalmers.se/subatom/nt/), Institutionen för fysik, Chalmers**

Date: **Aug 10, 2019**

## Uppgift 11.8.8

Bestäm $\vec{E}$-fältet överallt i rummet från en sfäriskt symmetrisk laddningsfördelning bestående av två delar, nämligen en rymdladdning $\rho(r) = \rho_0$ för $r a/2$ och en ytladdning $\sigma = -\rho_0 a/24$ på sfären $r = a$.

**Hint.**
* Utnyttja den sfäriska symmetrin i problemet som gör att $\vec{E} = E_r \hat{r}$.

* Använd Maxwells första ekvation som säger att $\nabla \cdot \vec{E} = \rho / \epsilon_0$, där $\rho$ är laddningstätheten.

* En volymsintegral av laddningstätheten (t.ex. över en sfär med radien $r$) ger den inneslutna laddningen.

* Motsvarande volymsintegral över VL av Maxwells första ekv (dvs över divergensen av $\vec{E}$-fältet) kan skrivas om med Gauss sats.

* En ytladdning har enheten laddning/area. Dvs den totala laddningen på det yttre skalet fås genom att integrera över ytan.

**Answer.**

$$
\begin{equation}
E\left(r\right) = \frac{\rho_0}{3 \epsilon_0} r \quad  \mathrm{för~} r < \frac{a}{2}
\label{_auto1} \tag{1}
\end{equation}
$$

$$
\begin{equation}
E\left(r\right) =\frac{\rho_0}{24\epsilon_0} \frac{a^3}{r^2} \quad  \mathrm{för~} \frac{a}{2} < r  a
\label{_auto3} \tag{3}
\end{equation}
$$

**Solution.**
Att göra

### Remarks

Uppgiften illustrerar användandet av Maxwells ekvationer och hur vi kan utnyttja en integralsats.


```python
import random
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import sympy
from sklearn.linear_model import LinearRegression
from scipy.spatial import Voronoi, voronoi_plot_2d

```

```python
N = 70
M = 10

Matrix = [(random.random()*100,random.random()*100)  for x in range(M)]
points = np.array(Matrix)

vor = Voronoi(points)
voronoi_plot_2d(vor)
plt.show()

```

```python
vor.regions
```

    [[-1, 2],
     [3, 1, -1, 2],
     [8, 6, 5, 7],
     [7, -1, 0, 5],
     [8, 1, 3, 4, 6],
     [8, 1, -1, 7],
     [10, 9, -1],
     [9, 4, 3, 2, -1],
     [],
     [10, 0, 5, 6, 4, 9],
     [10, 0, -1]]

```python
MAX_NUMBER = 10000
primes = list(sympy.primerange(0, MAX_NUMBER))

X = []
y = []

for n in range(MAX_NUMBER):
    valid_primes = [p for p in primes if p <= n ** 0.5]
    remainder = sum([n % p for p in valid_primes])
    y.append(remainder)
    X.append([n])

model = LinearRegression(fit_intercept = False).fit(X, y)
1 / model.coef_[0]
```

    17.841987719271046

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# 隐式马尔可夫模型基础

## 摘要

我们如何将机器学习应用在基于时间序列的观测数据上？举个例子，比如我们可能想要从某人演讲的音频记录中得到他使用的词语序列；或者根据词性标注对一系列单词做注解等等。在本文中，我们将对马尔可夫模型（在时间序列上的状态的形式化推理）以及隐式马尔可夫模型（从观测数据的序列中找回原来的状态序列）的概念从数学方面做一个全面的介绍。本文的最后一节包含了本文中某些观点的来源并提供深入学习的材料。

## 1. 马尔可夫模型

对于给定的状态集$S=\left\{s_1,s_2,\cdots,s_{\lvert S\rvert}\right\}$，我们可以沿时间序列观测到一些列数据$\vec z\in S^T$。举个例子，一个从天气系统中得到的状态集$S=\left\{sun,cloud,rain\right\}$，集合的大小为$\lvert S\rvert=3$，我们也在天气系统中得到一组观测数据$\left\{z_1=s_{sun},z_2=s_{cloud},z_3=s_{cloud},z_4=s_{rain},z_5=s_{cloud}\right\}$，观测数据集大小为$T=5$。

天气例子的的观测状态向我们展示了一个在时间上的随机过程的输出。如果不做进一步假设，$t$时刻的状态$s_j$可能是一个关于任意多个变量的函数，其中包括了从时刻$1$到$t-1$的所有状态、以及一些我们甚至都没有建模的变量。不过，我们可以做出两个能够使时间序列更已处理的**马尔可夫假设（Markov Assumptions）**。

**限制时域假设（Limited Horizon Assumption）**指的是到达$t$时刻状态的概率只与$t-1$时刻的状态有关。这条假设的直观理解就是$t$时刻的状态“足以概括”之前所有状态，并可以合理的预测未来的状态，形式化的表示为：
$$
P\left(z_t\mid z_{t-1},z_{t-2},\cdots,z_1\right)=P\left(z_t\mid z_{t-1}\right)
$$

**平稳过程假设（Stationary Process Assumption）**指的是在当前状态下，下个状态的条件分布不随时间变化而改变，形式化的表示为：
$$
P\left(z_t\mid z_{t-1}\right)=P\left(z_2\mid z_1\right);\ t\in2\cdots T
$$

按照习惯，我们也会假设初始状态和初始观测数据满足$z_0\equiv s_0$，$s_0$表示在$0$时刻的状态的初始概率分布。这个习惯可以使我们将“观测第一个真实状态$z_1$的先验概率”的观点通过$P\left(z_1\mid z_0\right)$表达出来。因为我们定义对于任意状态序列都有$z_0=s_0$，所以$P\left(z_t\mid z_{t-1},z_{t-2},\cdots,z_1\right)=P\left(z_t\mid z_{t-1},z_{t-2},\cdots,z_1,z_0\right)$。（其他HMM的表示方法有时通过向量$\pi\in\mathbb R^{\lvert S\rvert}$表达其先验的观点。）

我们定义一个状态转换矩阵$\displaystyle A\in\mathbb R^{(\lvert S\rvert+1)\times(\lvert S\rvert+1)}$来参数化这些转换，$A_{ij}$就表示在任意时刻$t$下从状态$i$转换到状态$j$的概率。（可以参考[线性代数中马尔科夫矩阵的例子](http://nbviewer.jupyter.org/github/zlotus/notes-linear-algebra/blob/master/chapter24.ipynb)）在天气系统的例子中，就可以定义如下的状态转换矩阵：
$$
A=\begin{matrix}
&s_0&s_{sun}&s_{cloud}&s_{rain}\\
s_0&0&.33&.33&.33\\
s_{cloud}&0&.2&.6&.2\\
s_{rain}&0&.1&.2&.7
\end{matrix}
$$

容易观察到这些数字（虽然是编造的）表示的直观的天气的自相关性：如果当前是晴天，则天气倾向于保持晴朗；如果是多云则倾向于保持多云，等等。这种模式经常在马尔可夫模型中出现，体现在状态转移矩阵中就是对角线元素远大于其他元素。在本例中，初始状态$s_0$是转换到这三个天气状态上的均匀概率分布。

### 1.1 马尔可夫模型的两个问题

结合马尔可夫假设和状态转换参数$A$，我们可以回答在马尔科夫链中关于状态序列的两个基本问题：
* 状态$\vec z$对应的一个特定序列的概率是什么？
* 对于观测到的序列$\vec z$，如何估计模型中的参数$A$的才能最大化其似然性？

#### 1.1.1 状态序列的概率

我们可以利用概率论中的[链式法则](https://en.wikipedia.org/wiki/Chain_rule_(probability))状态$\vec z$对应的一个特定序列的概率：
$$
\begin{align}
P\left(\vec z\right)&=P\left(z_t,z_{t-1},\cdots,z_1;A\right)\\
&=P\left(z_t,z_{t-1},\cdots,z_1,z_0;A\right)\\
&=P\left(z_t\mid z_{t-1},z_{t-2},\cdots,z_1;A\right)P\left(z_{t-1}\mid z_{t-2},\cdots,z_1;A\right)\cdots P(z_1\mid z_0;A)\\
&=P\left(z_t\mid z_{t-1};A\right)P\left(z_{t-1}\mid z_{t-2};A\right)\cdots P\left(z_2\mid z_1;A\right)P\left(z_1\mid z_0;A\right)\\
&=\prod_{t=1}^TP\left(z_{t}\mid z_{t-1};A\right)\\
&=\prod_{t=1}^TA_{z_{t-1}\ z_t}
\end{align}
$$

式中的第二步，我们根据前面关于$z_0$的定义，将$z_0$引入联合分布。第三步是由对联合分布应用链式法则，或重复迭代贝叶斯发展推导出的。第四步使用了马尔可夫模型的限制时域假设。而最后一步是将这些乘积项用其对应在状态转换矩阵$A$中的元素表示出来。

我们来计算一下天气系统例子中的序列：如果要计算序列$P\left(z_1=s_{sun},z_2=s_{cloud},z_3=s_{rain},z_4=s_{rain},z_5=s_{cloud}\right)$的概率，则可以分解为$P\left(s_{sun}\mid s_0\right)P\left(s_{cloud}\mid s_{sun}\right)P\left(s_{rain}\mid s_{cloud}\right)P\left(s_{rain}\mid s_{rain}\right)P\left(s_{cloud}\mid s_{rain}\right)=.33\times.1\times.2\times.7\times.2$。

#### 1.1.2 参数的最大似然估计

从学习的角度，我们想要知道能够“最大化观测到的序列$\vec z$的对数似然性”的参数$A$的取值。这相当于比较“从晴到阴”与“从晴到晴”等状态的似然性，以使得某个观测序列更有可能发生。我们定义马尔可夫模型的对数似然函数：

$$
\begin{align}
l(A)&=\log P\left(\vec z;A\right)\\
&=\log\prod_{t=1}^TA_{z_{t-1}\ z_t}\\
&=\sum_{t=1}^T\log A_{z_{t-1}\ z_t}\\
&=\sum_{i=1}^{\lvert S\rvert}\sum_{j=1}^{\lvert S\rvert}\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}\log A_{ij}
\end{align}
$$

我们在式中的最后一步使用了指示函数（其参数成立时函数值取$1$，否则取$0$）来从状态转移矩阵$A$中选出从时间序列中观测到的每一步状态转移对应的概率。在解这个优化问题时要注意，解出的参数$A$必须满足一个状态转移矩阵的要求——于每个状态$i$对应的分布，其概率之和必须为$1$；矩阵的所有元素均为非负数。我们可以使用拉格朗日乘数法来求解这个优化问题：

$$
\begin{align}
\max_A&\ l(A)\\
\mathrm{s.t.}&\ \sum_{j=1}^{\lvert S\rvert}A_{ij}=1,\ i=1,\cdots,\lvert S\rvert\\
&\ A_{ij}\geq0,\ i,j=1,\cdots,\lvert S\rvert
\end{align}
$$

这个带有约束条件的优化问题可以使用拉格朗日乘数法直接求出其解析解。我们会将等式约束条件（即第一个约束条件）引入拉格朗日算子，而不等式约束条件则可以方向的忽略——因为最优解中的$A_{ij}$的元素都是非负的。于是，构造拉格朗日算子：

$$
\mathcal L(A,\alpha)=\sum_{i=1}^{\lvert S\rvert}\sum_{j=1}^{\lvert S\rvert}\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}\log A_{ij}+\sum_{i=1}^{\lvert S\rvert}\alpha_i\left(1-\sum_{j=1}^{\lvert S\rvert}A_{ij}\right)
$$

对拉格朗日算子求关于$A_{ij}$的偏导，并将偏导置为$0$得到：

$$
\begin{align}
\frac{\partial\mathcal L(A,\alpha)}{\partial A_{ij}}&=\frac{\partial}{\partial A_{ij}}\left(\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}\log A_{ij}\right)+\frac{\partial}{\partial A_{ij}}\alpha_i\left(1-\sum_{j=1}^{\lvert S\rvert}A_{ij}\right)\\
&=\frac{1}{A_{ij}}\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}-\alpha_{i}\equiv0\\
&\Downarrow\\
A_{ij}&=\frac{1}{\alpha_i}\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}
\end{align}
$$

再求关于$\alpha$的偏导，并将偏导置为$0$且将上式代入得到：

$$
\begin{align}
\frac{\partial\mathcal L(A,\alpha)}{\partial\alpha_i}&=1-\sum_{j=1}^{\lvert S\rvert}A_{ij}\\
&=1-\sum_{j=1}^{\lvert S\rvert}\frac{1}{\alpha_i}\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}\equiv0\\
\Downarrow\\
\alpha_i&=\sum_{j=1}^{\lvert S\rvert}\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}\\
&=\sum_{t=1}^T1\left\{z_{t-1}=s_i\right\}
\end{align}
$$

再将得到的$\alpha_i$代回前面$A_{ij}$的表达式，就能得到参数$\hat A_{ij}$的最大似然估计：

$$
\hat A_{ij}=\frac{\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_{t}=s_j\right\}}{\sum_{t=1}^T1\left\{z_{t-1}=s_i\right\}}
$$

这个式子的意义非常直观——系统从状态$i$转移到状态$j$的最大似然概率为“系统从状态$i$转移到状态$j$的次数”除以“系统处于状态$i$的次数”。换句话说，最大似然参数应为系统处于状态$i$时转移到状态$j$的概率。

## 2. 隐式马尔可夫模型

马尔可夫模型对基于时间序列的数据来说是一种强大的抽象，但是却不能表达一些很常见的场景。我们如何在“无法观测到状态本身，只能得到关于这些状态的某些概率性函数”的情况下，对状态序列进行讨论？这样的场景包括：我们只能观测到单词序列，而不能得到其词性的情况下如何进行词性标注？或是在只能得到音频序列的情况下，如何找出生成这些音频序列的单词序列？我们使用Jason Eisner推荐的例子——冰淇淋气候学——来展开讲解：

> 假设我们是生活在2799年的气候学家，正在研究全球气候变暖的历史，但是找不到巴尔的摩的气象记录。不过却找到了Jason Eisner的日记——JE在日记中坚持不懈的记录了他每天吃掉的冰淇淋的数量。*那么，通过这本日记，我们能够得到关于天气的什么信息呢？*

隐式马尔可夫模型就可以应用在这一类场景中——我们并不能观测到实际的状态序列（例子中每天的天气），而只能得到由这些状态导致的某种结果（每天吃掉的冰淇淋数量）。

形式化的表达就是，HMM是这样一种马尔可夫模型——我们能够得到的是*可观测*输出序列$x=\left\{x_1,x_2,\cdots,x_T\right\}$，它们来自输出集合$V=\left\{v_1,v_2,\cdots,v_{\lvert V\rvert}\right\}$，即$x_t\in V,t=1\cdots T$。在上一节中，我们也假设存在状态序列$z=\left\{z_1,z_2,\cdots,z_T\right\}$，它们来自状态集合$S=\left\{s_1,s_2,\cdots,s_{\lvert S\rvert}\right\}$，即$z_t\in S,t=1\cdots T$，不过在这个场景中，这些状态是*不可观测*的。而状态$i,j$之间的转换同样表达在一个状态转换矩阵的元素$A_{ij}$中。

我们同样会将输出观测值建模为一个关于隐藏状态的函数，不过，我们先要做一个**输出独立性假设**——定义$P\left(x_t=v_k\mid z_t=s_j\right)=P\left(x_t=v_k\mid x_1,\cdots,x_T,z_1,\cdots,z_T\right)=B_{jk}$。矩阵$B$表示系统在某时刻处于状态$s_j$的条件下生成可观测输出$v_k$的概率。回到天气的例子中，假设我们得到了一个四天内的冰淇淋消耗日志：$\vec x=\left\{x_1=v_3,x_2=v_2,x_3=v_1,x_4=v_2\right\}$。其中输出集合中有这些观测值代码所表示的意义，即$V=\left\{v_1=1\textit{ice cream},v_2=2\textit{ice cream},v_3=3\textit{ice cream}\right\}$。那么，HMM可以让我们解答什么问题呢？

### 2.1 隐式马尔可夫模型的三个问题

关于HMM，我们有三个基本问题想要知道：
* 可观测序列的概率是什么（我们有多大可能得到一个$3,2,1,2$的冰淇淋消耗记录）？
* 最可能生成这个可观测序列的状态序列是什么样的（这四天到底是什么样的天气）？
* 如何从给定观测到的数据中学到HMM模型中的参数$A,B$？

### 2.2 可观测序列的概率：前向算法

在HMM中，我们假设得到的观测数据的生成过程为：假设在时间序列上存在一组状态$\vec z$，而我们认为这组状态序列是由以参数$A$为状态转移矩阵的马尔可夫模型生成的。其中，对于每一个时刻$t$，我们都将$x_t$看做是一个关于$z_t$的函数。于是，要计算观测值序列的概率，就需要将在每一个可能的状态序列下生成观测值序列$\vec x$的概率相加。

$$
\begin{align}
P\left(\vec x; A, B\right)&=\sum_{\vec z}P\left(\vec x, \vec z; A, B\right)\\
&=\sum_{\vec z}P\left(\vec x\mid\vec z; A, B\right)P\left(\vec z; A, B\right)
\end{align}
$$

这个式子在任意概率分布下恒成立，而且，如果我们使用HMM假设，就能够进一步化简该式：

$$
\begin{align}
P\left(\vec x; A, B\right)&=\sum_{\vec z}P\left(\vec x\mid\vec z; A, B\right)P\left(\vec z; A, B\right)\\
&=\sum_{\vec z}\left(\prod_{t=1}^T P\left(\vec x_t\mid \vec z_t; B\right)\right)\left(\prod_{t=1}^T P\left(\vec z_t\mid \vec z_{t-1}; A\right)\right)\\
&=\sum_{t=1}^T\left(\prod_{t=1}^T B_{z_t\ x_t}\right)\left(\prod_{t=1}^T A_{z_{t-1}\ z_t}\right)
\end{align}
$$

好消息是，该式是一个由参数项组成的简单表达式。上面的推导使用的HMM假设有：输出独立性假设、马尔可夫假设、及平稳过程假设，式子的第二步就是由这些假设推导出的。坏消息是，这个求和是对于所有可能的$\vec z$，而对于每一个时刻，$z_t$可以取到$\lvert S\rvert$种不同的值，于是这个求和就需要消耗$O\left(\lvert S\rvert^T\right)$次操作。

幸运的是，一个称为**前向算法（Forward Procedure）**的动态规划算法可以让我们快速计算$P\left(\vec x; A, B\right)$。首先我们定义$\alpha_i(t)=P(x_1,x_2,\cdots,x_t,z_t=s_i; A, B)$，这个量表示：系统在$t$时刻时处于$s_i$状态时，从初始时刻到$t$时刻中对应的所有观测值的总概率（计算时$t$时刻可以赋任意状态）。如果我们能够得到这个量，则观测值全集的概率$P\left(\vec x\right)$可以表示为：

$$
\begin{align}
P\left(\vec x; A, B\right)&=P\left(x_1,x_2,\cdots,x_T; A, B\right)\\
&=\sum_{i=1}^{\lvert S\rvert}P\left(x_1,x_2,\cdots,x_T,z_T=s_i; A, B\right)\\
&=\sum_{i=1}^{\lvert S\rvert}\alpha_i(T)
\end{align}
$$

该算法介绍了一种高效计算$\alpha_i(t)$的方法，对于系统的每一个时刻只需要进行$O\left(\lvert S\rvert\right)$次操作。那么，对于计算可观测序列的总概率$P\left(\vec x; A, B\right)$，这个算法的总体复杂度为$O\left(\lvert S\rvert\cdot T\right)$。

$$
\begin{array}
{l}
\hline
\textbf{算法 1 }\textrm{使用前向算法计算}\alpha_i(t)\\
\hline
\textrm{1. 初始状态: }\alpha_i(0)=A_{0i},\ i=1,\cdots,\lvert S\rvert\\
\textrm{2. 递推公式: }\displaystyle\alpha_j(t)=\sum_{i=1}^{\lvert S\rvert}\alpha_i(t-1)A_{ij}B_{jx_t},\ j=1,\cdots,\lvert S\rvert, t=1,\cdots,T\\
\hline
\end{array}
$$

还有一个类似的算法称为**后向算法（Backward Procedure）**，可以通过与前向算法相反的次序来计算相应的概率$\beta_i(t)=P(x_T,x_{T-1},\cdots,x_{t+1},z_t=s_i; A, B)$。

### 2.3 寻找最大似然状态序列：Viterbi算法

关于隐式马尔可夫模型，最常见的问题就是：对于给定的可观测序列$\vec x\in V^T$，生成这个可观测序列的最可能的状态序列$\vec z\in S^T$是什么？用公式表达就是：

$$
\arg\max_{\vec z} P\left(\vec z\mid\vec x; A, B\right)=\arg\max_{\vec z}\frac{P\left(\vec x,\vec z; A, B\right)}{\sum_{\vec z} P\left(\vec x, \vec z; A, B\right)}=\arg\max_{\vec z} P\left(\vec x,\vec z; A, B\right)
$$

式子的第一步化简使用了贝叶斯定理，第二步化简则是通过观察得知——分母并不直接依靠$\vec z$得出。简单的说就是我们将所有可能的状态序列$\vec z$代入模型，然后选择联合概率最大的那个状态序列。不过这又将导致穷举所有可能的状态序列，而使得计算复杂度上升到$O\left(\lvert S\rvert^T\right)$。此时，我们会自然的想到上一节前向算法中介绍的动态规划解决方案，而且容易发现，如果将式中的$\displaystyle\arg\max_{\vec z}$换成$\displaystyle\sum_{\vec z}$，则当前的问题就真的变成了前向算法中要解决的问题了。

$$
\begin{array}
{l}
\hline
\textbf{算法 2 }\textrm{将EM算法应用在HMM上}\\
\hline
\textrm{重复直到收敛\{}\\
\qquad\textrm{（E步骤）对每个可能的状态标识}\vec z\in S^T\textrm{令：}\\
\qquad\qquad Q\left(\vec z\right):=p\left(\vec z\mid\vec x; A, B\right)\\
\qquad\textrm{（M步骤）令：}\\
\qquad\qquad
\begin{eqnarray}
A,B&:=\arg\max_{A,B}&\sum_{\vec z}Q\left(\vec z\right)\log\frac{P\left(\vec x,\vec z; A,B\right)}{Q\left(\vec z\right)}\\
&\mathrm{s.t.}&\sum_{j=1}^{\lvert S\rvert}A_{ij}=1,i=1,\cdots,\lvert S\rvert;\ A_{ij}\geq0,\ i,j=1,\cdots,\lvert S\rvert\\
& &\sum_{j=1}^{\lvert S\rvert}B_{ik}=1,i=1,\cdots,\lvert S\rvert;\ B_{ik}\geq0,\ i=1,\cdots,\lvert S\rvert,\ k=1,\cdots,\lvert V\rvert
\end{eqnarray}\\
\mathrm{\}}\\
\hline
\end{array}
$$

**Viterbi算法**与前向算法十分相似，只不过现在我们不再跟踪观测到某时刻$t$的状态序列的总概率（到时刻$t$为止的整个状态序列生成观测值序列的概率），而仅跟踪最大概率并记录其对应的状态序列。

### 2.4 参数学习：EM算法在HMM上的应用

关于HMM的最后一个问题就是，对于给定的观测值序列，在模型中对应的状态转换概率矩阵$A$、和可观测值的输出生成概率矩阵$B$最可能是什么？举个例子，在语音识别问题中，用语音识别数据集解出最大似然参数，可以让我们在不需要知道“某段音频信号对应的的最可能状态序列”的情况下有效的训练HMM。

在本节，我们将推导出应用在隐式马尔可夫模型上的最大期望算法。这个推导将使用本课程（CS229）[讲义](chapter13.ipynb)中给出的EM算法的公司。上一节的算法给出了基础的EM算法的思路，可以看出M步骤中已经包含了关于合理的$A,B$的概率约束条件。类似在1.1.3节中介绍的（非隐式）马尔可夫模型的最大似然解法，而对于现在的优化问题，我们仍然可以使用拉格朗日乘数法解决。还需要留意的是，在E步骤和M步骤中，都需要穷举所有$\lvert S\rvert^T$个可能的状态序列标识$\vec z$。我们将会使用前面提到的前向/后向算法求出E步骤和M步骤的充分统计量。

首先，使用马尔可夫假设重新整理目标函数：

$$
\begin{eqnarray}
A,B&=&\arg\max_{A,B}\sum_{\vec z}Q\left(\vec z\right)\log\frac{P\left(\vec x,\vec z; A, B\right)}{Q\left(\vec z\right)}\\
&=&\arg\max_{A,B}\sum_{\vec z}Q\left(\vec z\right)\log P\left(\vec x,\vec z;A,B\right)\\
&=&\arg\max_{A,B}\sum_{\vec z}Q\left(\vec z\right)\log\left(\prod_{t=1}^TP\left(\vec x_t\mid\vec z_t;B\right)\right)\left(\prod_{t=1}^TP\left(\vec z_t\mid\vec z_{t-1};A\right)\right)\\
&=&\arg\max_{A,B}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T\log B_{z_t\ x_t}+\log A_{z_{t-1}\ z_t}\\
&=&\arg\max_{A,B}\sum_{\vec z}Q\left(\vec z\right)\sum_{i=1}^{\lvert S\rvert}\sum_{j=1}^{\lvert S\rvert}\sum_{k=1}^{\lvert V\rvert}\sum_{t=1}^{\lvert T\rvert}1\left\{z_t=s_j\land x_t=v_k\right\}\log B_{jk}+1\left\{z_{t-1}=s_i\land z_t=s_j\right\}\log A_{ij}
\end{eqnarray}
$$

我们将第一步$\log$中的除法转为减法并发现参数$A,B$并不影响该项的取值，所以在求最大化参数的过程中可以舍弃；而在第三步中应用了马尔可夫假设；第五步化为指示函数，用状态来索引矩阵$A,B$。与显式马尔可夫模型中最大似然参数的推导一样，我们可以舍弃不等式约束，因为求出的解会自然符合这些非负约束。构造拉格朗日算子：

$$
\begin{eqnarray}
\mathcal L(A,B,\sigma,\epsilon)&=&\sum_{\vec z}Q\left(\vec z\right)\sum_{i=1}^{\lvert S\rvert}\sum_{j=1}^{\lvert S\rvert}\sum_{k=1}^{\lvert V\rvert}\sum_{t=1}^{\lvert T\rvert}1\left\{z_t=s_j\land x_t=v_k\right\}\log B_{jk}+1\left\{z_{t-1}=s_i\land z_t=s_j\right\}\log A_{ij}\\
&&+\sum_{j=1}^{\lvert S\rvert}\epsilon_j\left(1-\sum_{k=1}^{\lvert V\rvert}B_{jk}\right)+\sum_{i=1}^{\lvert S\rvert}\delta_i\left(1-\sum_{j=1}^{\lvert S\rvert}A_{ij}\right)
\end{eqnarray}
$$

求关于参数$A,B$的偏导并将其置为零：

$$
\begin{eqnarray}
\frac{\partial\mathcal L(A,B,\sigma,\epsilon)}{\partial A_{ij}}&=&\sum_{\vec z}Q\left(\vec z\right)\frac{1}{A_{ij}}\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_t=s_j\right\}-\sigma_i\equiv0\\
A_{ij}&=&\frac{1}{\sigma_i}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_t=s_j\right\}\\
\frac{\partial\mathcal L(A,B,\sigma,\epsilon)}{\partial B_{jk}}&=&\sum_{\vec z}Q\left(\vec z\right)\frac{1}{B_{jk}}\sum_{t=1}^T1\left\{z_t=s_j\land x_t=v_k\right\}-\epsilon_j\equiv0\\
B_{jk}&=&\frac{1}{\epsilon_j}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\land x_t=v_k\right\}
\end{eqnarray}
$$

再求关于拉格朗日乘数的偏导并将其置为零，然后再带入刚才求得的$A_{ij},B_{jk}$：

$$
\begin{eqnarray}
\frac{\partial\mathcal L(A,B,\sigma,\epsilon)}{\partial \sigma_i}&=&1-\sum_{j=1}^{\lvert S\rvert}A_{ij}\\
&=&1-\sum_{j=1}^{\lvert S\rvert}\frac{1}{\sigma_i}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_t=s_j\right\}\equiv0\\
\sigma_i&=&\sum_{j=1}^{\lvert S\rvert}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_t=s_j\right\}\\
&=&\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\right\}\\
\frac{\partial\mathcal L(A,B,\sigma,\epsilon)}{\partial \epsilon_j}&=&1-\sum_{k=1}^{\lvert V\rvert}B_{jk}\\
&=&1-\sum_{k=1}^{\lvert V\rvert}\frac{1}{\epsilon_j}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\land x_t=v_k\right\}\equiv0\\
\epsilon_j&=&\sum_{k=1}^{\lvert V\rvert}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\land x_t=v_k\right\}\\
&=&\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\right\}
\end{eqnarray}
$$

再将求得的拉格朗日乘数表达式带回$A_{ij},B_{jk}$的表达式，便可求得能够最大化目标函数的参数$\hat A,\hat B$：

$$
\begin{eqnarray}
\hat A_{ij}&=&\frac{\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_t=s_j\right\}}{\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\right\}}\\
\hat B_{jk}&=&\frac{\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\land x_t=v_k\right\}}{\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\right\}}
\end{eqnarray}
$$

不幸的是，式中关于$\vec z$的求和都需要计算所有可能的状态序列$\vec z\in S^T$。回忆$Q\left(\vec z\right)$项在E步骤中定义为$P\left(\vec z\mid\vec x; A,B\right)$，它出现在状态序列的最后一步，其参数为$A,B$。现在来考虑如何用前向/后向算法中的$\alpha_i(t),\beta_j(t)$将$\hat A_{ij}$的分子表示出来。

$$
\begin{eqnarray}
\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_t=s_j\right\}&=&\sum_{t=1}^T\sum_{\vec z}1\left\{z_{t-1}=s_i\land z_t=s_j\right\}Q\left(\vec z\right)\\
&=&\sum_{t=1}^T\sum_{\vec z}1\left\{z_{t-1}=s_i\land z_t=s_j\right\}P\left(\vec z\mid\vec x; A,B\right)\\
&=&\frac{1}{P\left(\vec x; A,B\right)}\sum_{t=1}^T\sum_{\vec z}1\left\{z_{t-1}=s_i\land z_t=s_j\right\}P\left(\vec z, \vec x; A,B\right)\\
&=&\frac{1}{P\left(\vec x; A,B\right)}\sum_{t=1}^T\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)
\end{eqnarray}
$$

在前两步中，我们交换了求和顺序，并代入了$Q$的定义；在第三步中使用了贝叶斯公式；第四步中代入了$\alpha,\beta,A,B$的定义。类似的，分母也可以表示为分子在$j$到$\lvert S\rvert$上的求和：

$$
\begin{eqnarray}
\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\right\}&=&\sum_{j=1}^{\lvert S\rvert}\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_{t-1}=s_i\land z_t=s_j\right\}\\
&=&\frac{1}{P\left(\vec x;A,B\right)}\sum_{j=1}^{\lvert S\rvert}\sum_{t=1}^T\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)
\end{eqnarray}
$$

将上面求得的表达式结合起来，就可以描述出最大似然状态转换参数$\hat A_{ij}$，而且不需要穷举所有可能的状态序列：

$$
\hat A_{ij}=\frac{\sum_{t=1}^T\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)}{\sum_{j=1}^{\lvert S\rvert}\sum_{t=1}^T\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)}
$$

类似的，我们可以将$\hat B_{jk}$的分子表示为：

$$
\begin{eqnarray}
\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\land x_t=v_k\right\}&=&\frac{1}{P\left(\vec x;A,B\right)}\sum_{t=1}^T\sum_{\vec z}1\left\{z_t=s_j\land x_t=v_k\right\}P\left(\vec z,\vec x;A,B\right)\\
&=&\frac{1}{P\left(\vec x;A,B\right)}\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T\sum_{\vec z}1\left\{z_{t-1}=s_i\land z_t=s_j\land x_t=v_k\right\}P\left(\vec z,\vec x;A,B\right)\\
&=&\frac{1}{P\left(\vec x;A,B\right)}\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T1\left\{x_t=v_k\right\}\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)
\end{eqnarray}
$$

$\hat B_{jk}$的分母表示为：

$$
\begin{eqnarray}
\sum_{\vec z}Q\left(\vec z\right)\sum_{t=1}^T1\left\{z_t=s_j\right\}&=&\frac{1}{P\left(\vec x;A,B\right)}\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T\sum_{\vec z}1\left\{z_{t-1}=s_i\land z_t=s_j\right\}P\left(\vec z,\vec x;A,B\right)\\
&=&\frac{1}{P\left(\vec x;A,B\right)}\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)
\end{eqnarray}
$$

结合上面的表达式，就可以得到最大似然可观测值生成参数$\hat B_{jk}$：

$$
\hat B_{jk}=\frac{\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T1\left\{x_t=v_k\right\}\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)}{\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)}
$$

$$
\begin{array}
{l}
\hline
\textbf{算法 3 }\textrm{HMM参数学习上的前向-后向算法}\\
\hline
\textrm{初始化：令}A,B\textrm{为随机的有效概率矩阵，其中}A_{i0}=0,B_{0k}=0,\ i=1,\cdots,\lvert S\rvert,k=1,\cdots,\lvert V\rvert\textrm{。}\\
\textrm{重复直到收敛\{}\\
\qquad\textrm{（E步骤）运行前向及后向算法，计算}\alpha_i,\beta_i,\ i=1,\cdots,\lvert S\rvert\textrm{令：}\\
\qquad\qquad \gamma_t(i,j):=\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)\\
\qquad\textrm{（M步骤）重新计算参数的最大似然估计：}\\
\qquad\qquad
\begin{eqnarray}
A_{ij}&=&\frac{\sum_{t=1}^T\gamma_t(ij)}{\sum_{j=1}^{\lvert S\rvert}\sum_{t=1}^T\gamma_t(ij)}\\
B_{jk}&=&\frac{\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T1\left\{x_t=v_k\right\}\gamma_t(ij)}{\sum_{i=1}^{\lvert S\rvert}\sum_{t=1}^T\gamma_t(ij)}
\end{eqnarray}\\
\mathrm{\}}\\
\hline
\end{array}
$$

这个算法是另一种形式的前向-后向算法，也被称作**Baum-Welch算法**。在E步骤中，我们并不会直接计算每一个可能的$\vec z\in S^T$对应的$Q\left(\vec z\right)$，而是求出一个充分统计量$\gamma_t(ij)=\alpha_i(t)A_{ij}B_{jx_t}\beta_j(t+1)$。这个量对应的是“在$t$时刻，对于所有给定的观测序列$\vec x$，从状态$s_i$转换到状态$s_j$”的概率。而且，算法中推导出的关于$A_{ij},B_{jk}$的表达式的意义也很直观。其中$A_{ij}$是“从状态$s_i$转换到状态$s_j$的次数”除以“状态$s_i$出现的次数”；$B_{jk}$是“由状态$s_j$生成可观测值$v_k$的次数”除以“状态s_j出现的次数”。

同其他EM算法处理的问题一样，HMM上的参数学习也是一个非凸问题，存在着很多局部最大值。EM算法可能会根据初始值的不同，收敛于不同的最大值处。所以，用不同的初始值多运行几次是一种比较常见的做法。另外，对由$A,B$表示的概率分布做平滑处理也很重要，这样可以避免某个“状态转换的过程”或“状态生成观测值的”是$0$概率。

### 2.5 深入学习材料

网上有很多隐式马尔可夫模型的学习资源，对于自然语言处理（NLP）参考Jurafsky & Martin的《Speech and Language Processing》第二版；或是Manning & Schütze的《Foundations of Statistical Natural Language Processing》；或是Eisner的《HMM in a spreadsheet》——一个仅靠spreadsheet建立的轻量级交互式HMM（主要用于为前向-后向算法提供一个直观的例子）。