# Chapter 4
# Finite-Dimensional Optimization

In this chapter we examine methods for optimizing a function with respect to a finite
number of variables. In the finite-dimensional optimization problem, one is given a
real-valued function $f$ defined on $X \subset R^n$ and asked to find an $x^* \in X$ such that
$f(x^*) \geq f(x)$ for all $x \in X$. We denote this problem

$$\max_{x \in X} f(x)$$

and call $f$ the objective function, $X$ the feasible set, and $x^*$, if it exists, a maximum.

There is a close relationship between the finite-dimensional optimization problems
discussed in this chapter and the rootfinding and complementarity problems
discussed in the previous chapter. The first-order necessary conditions of an unconstrained
problem pose a rootfinding problem; the Karush-Kuhn-Tucker first-order
necessary conditions of a constrained optimization problem pose a complementarity
problem. The rootfinding and complementarity problems associated with optimization
problems are special in that they possess a natural merit function, the objective
function itself, which may be used to determine whether iterations are converging on
a solution.

Over the years, numerical analysts have studied finite-dimensional optimization
problems extensively and have devised a variety of algorithms for solving them quickly
and accurately. We begin our discussion with derivative-free methods, which are useful
if the objective function is rough or if its derivatives are expensive to compute.
We then turn to Newton-type methods for unconstrained optimization, which employ
derivatives or derivative estimates to locate an optimum. Univariate unconstrained
optimization methods are of particular interest because many multivariate optimization
algorithms use the strategy of first determining a linear direction to move in,
and then finding the optimal point in that direction. We conclude with a discussion
of how to solve constrained optimization problems.

## 4.1 Derivative-Free Methods

As was the case with univariate rootfinding, optimization algorithms exist that will
place progressively smaller brackets around a local maximum of a univariate function.
Such methods are relatively slow, but do not require the evaluation of function
derivatives and are guaranteed to find a local optimum to a prescribed tolerance in a
known number of steps.

The most widely-used derivative-free method is the **golden search** method.

Suppose
we wish to find a local maximum of a continuous univariate function $f(x)$ on
the interval $[a; b]$.

Pick any two numbers in the interior of the interval, say $x_1$ and $x_2$
with $x_1 < x_2$.

Evaluate the function and replace the original interval with $[a; x2]$ if
$f(x_1) > f(x_2)$ or with $[x_1; b]$ if $f(x_2) \geq f(x_1)$.

A key issue is how to pick the interior evaluation points.

Two simple criteria lead
to the most widely-used strategy.

First, the length of the new interval should be
independent of whether the upper or lower bound is replaced.

Second, on successive
iterations, one should be able to reuse an interior point from the previous iteration so
that only one new function evaluation is performed per iteration.

These conditions
are uniquely satisfied by selecting $x_i = a + \alpha_i (b - a)$, where

$$\alpha_1 = \frac{3-\sqrt 5}{2}$$

$$\alpha_2 = \frac{\sqrt 5 -1}{2}$$

The value $\alpha_2$ is known as the golden ratio, a number dear to the hearts of Greek
philosophers and Renaissance artists.

```python
import numpy as np
from numpy import append, array, diagonal, tril, triu
from numpy.linalg import inv
from scipy.linalg import lu
#from scipy.linalg import solve
from pprint import pprint
from numpy import array, zeros, diag, diagflat, dot

from sympy import *
import sympy as sym
init_printing()
```

```python
%matplotlib notebook
from matplotlib import pyplot as plt
```

```python
maxit = 1000
tol = 1/10000
x0= np.array([0,3])
f = lambda x: x * np.cos(x ** 2)

a,b = 0,3
```

```python
x = np.linspace(0,3, 100)
y = f(x)
plt.plot(x,y)
plt.scatter( np.array([0.8083,2.5234]), f(np.array([0.8083,2.5234])) , c='r' )
plt.title("Figure 4.1 Maximization of $x cos(x^2)$ via golden search")
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x7f73f848bcf8>

```python

```

```python
alpha1 = (3 - np.sqrt(5)) / 2
alpha2 = (np.sqrt(5) - 1) / 2
if a > b:
    a, b = b, a

x1 = a + alpha1 * (b - a)
x2 = a + alpha2 * (b - a)

f1, f2 = f(x1), f(x2)

d = (alpha1 * alpha2)*(b - a)
```

```python
while d > tol:
    d = d * alpha2
    if f2 < f1: # x2 is new upper bound
        x2, x1 = x1, x1 - d
        f2, f1 = f1, f(x1)
    else:  # x1 is new lower bound
        x1, x2 = x2, x2 + d
        f1, f2 = f2, f(x2)
```

```python
#x1 if f1 > f2 else x2
```

```python
if f1>f2:
    x = x2
else:
    x = x1
x
```

```python
def mygolden(f,a, b, maxit = 1000, tol = 1/10000):
    alpha1 = (3 - np.sqrt(5)) / 2
    alpha2 = (np.sqrt(5) - 1) / 2
    if a > b:
        a, b = b, a

    x1 = a + alpha1 * (b - a)
    x2 = a + alpha2 * (b - a)

    f1, f2 = f(x1), f(x2)

    d = (alpha1 * alpha2)*(b - a) # initial d
    while d > tol:
        d = d * alpha2 # alpha2 is the golden ratio
        if f2 < f1: # x2 is new upper bound
            x2, x1 = x1, x1 - d
            f2, f1 = f1, f(x1)
        else:  # x1 is new lower bound
            x1, x2 = x2, x2 + d
            f1, f2 = f2, f(x2)

    if f1>f2:
        x = x2
    else:
        x = x1
    return x
```

```python
mygolden(f, 0, 3)
```

Execution of this script yields the result $x = 0.8083$. As can be seen in Figure 4.1,
this point is a local maximum, but not a global maximum in $[0; 3]$. The golden search
method is guaranteed to find the global maximum when the function is concave.
However, as the present example makes clear, this need not be true when the optimand
is not concave.

## Nelder-Mead algorithm

```python

```

Another widely-used derivative-free optimization method for multivariate functions
is the **Nelder-Mead algorithm**.

The Nelder-Mead algorithm is simple, but slow and unreliable. However, if a
problem involves only a single optimization or costly function and derivative evaluations,
the Nelder-Mead algorithm is worth trying. In many problems an optimization
problem that is embedded in a larger problem must be solved repeatedly, with the
function parameters perturbed slightly with each iteration. For such problems, which
are common is dynamic models, one generally will want to use a method that moves
more quickly and reliably to the optimum, given a good starting point.

(source: https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method)

The Nelder–Mead method or downhill simplex method or amoeba method is a commonly applied numerical method used to find the minimum or maximum of an objective function in a multidimensional space. It is applied to nonlinear optimization problems for which derivatives may not be known. However, the Nelder–Mead technique is a heuristic search method that can converge to non-stationary points[1] on problems that can be solved by alternative methods.

(source: http://www.scholarpedia.org/article/Nelder-Mead_algorithm)

The Nelder-Mead algorithm or simplex search algorithm, originally published in 1965 (Nelder and Mead, 1965), is one of the best known algorithms for multidimensional unconstrained optimization without derivatives. This method should not be confused with Dantzig's simplex method for linear programming, which is completely different, as it solves a linearly constrained linear problem.

The basic algorithm is quite simple to understand and very easy to use. For these reasons, it is very popular in many fields of science and technology, especially in chemistry and medicine.

The method does not require any derivative information, which makes it suitable for problems with non-smooth functions. It is widely used to solve parameter estimation and similar statistical problems, where the function values are uncertain or subject to noise. It can also be used for problems with discontinuous functions, which occur frequently in statistics and experimental mathematics.

```python
#https://github.com/fchollet/nelder-mead/blob/master/nelder_mead.py

'''
    Pure Python/Numpy implementation of the Nelder-Mead algorithm.
    Reference: https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method
'''
import copy

def nelder_mead(f, x_start,
                step=0.1, no_improve_thr=10e-6,
                no_improv_break=10, max_iter=0,
                alpha=1., gamma=2., rho=-0.5, sigma=0.5):
    '''
    @param f (function): function to optimize, must return a scalar score
        and operate over a numpy array of the same dimensions as x_start
    @param x_start (numpy array): initial position
    @param step (float): look-around radius in initial step
    @no_improv_thr,  no_improv_break (float, int): break after no_improv_break iterations with
        an improvement lower than no_improv_thr
    @max_iter (int): always break after this number of iterations.
        Set it to 0 to loop indefinitely.
    @alpha, gamma, rho, sigma (floats): parameters of the algorithm
        (see Wikipedia page for reference)
    return: tuple (best parameter array, best score)
    '''
    # init
    dim = len(x_start)
    prev_best = f(x_start)
    no_improv = 0
    res = [[x_start, prev_best]]

    for i in range(dim):
        x = copy.copy(x_start)
        x[i] = x[i] + step
        score = f(x)
        res.append([x, score])

    # simplex iter
    iters = 0
    while 1:
        # order
        res.sort(key=lambda x: x[1])
        best = res[0][1]

        # break after max_iter
        if max_iter and iters >= max_iter:
            return res[0]
        iters += 1

        # break after no_improv_break iterations with no improvement
        print('...best so far:', best)

        if best < prev_best - no_improve_thr:
            no_improv = 0
            prev_best = best
        else:
            no_improv += 1

        if no_improv >= no_improv_break:
            return res[0]

        # centroid
        x0 = [0.] * dim
        for tup in res[:-1]:
            for i, c in enumerate(tup[0]):
                x0[i] += c / (len(res)-1)

        # reflection
        xr = x0 + alpha*(x0 - res[-1][0])
        rscore = f(xr)
        if res[0][1] <= rscore < res[-2][1]:
            del res[-1]
            res.append([xr, rscore])
            continue

        # expansion
        if rscore < res[0][1]:
            xe = x0 + gamma*(x0 - res[-1][0])
            escore = f(xe)
            if escore < rscore:
                del res[-1]
                res.append([xe, escore])
                continue
            else:
                del res[-1]
                res.append([xr, rscore])
                continue

        # contraction
        xc = x0 + rho*(x0 - res[-1][0])
        cscore = f(xc)
        if cscore < res[-1][1]:
            del res[-1]
            res.append([xc, cscore])
            continue

        # reduction
        x1 = res[0][0]
        nres = []
        for tup in res:
            redx = x1 + sigma*(tup[0] - x1)
            score = f(redx)
            nres.append([redx, score])
        res = nres

```

```python
import math
import numpy as np

# def f(x):
#     return math.sin(x[0]) * math.cos(x[1]) * (1. / (abs(x[2]) + 1))

#f(x,y) = x^2 - 4*x + y^2 - y - x*y;
# f = lambda x: x[0]**2- 4*x[0] + x[1]**2- x[1] - x[0]*x[1]

def f(x):
    return x[0]**2- 4*x[0] + x[1]**2- x[1] - x[0]*x[1]
```

```python
nelder_mead(f, np.array([0., 0.]))
```

    ...best so far: -0.39
    ...best so far: -0.7275
    ...best so far: -1.393125
    ...best so far: -2.35265625
    ...best so far: -3.5309765625
    ...best so far: -5.22336914063
    ...best so far: -5.22336914063
    ...best so far: -5.4678515625
    ...best so far: -6.54388916016
    ...best so far: -6.54388916016
    ...best so far: -6.79
    ...best so far: -6.79
    ...best so far: -6.82644058228
    ...best so far: -6.89778457642
    ...best so far: -6.94423038483
    ...best so far: -6.98128607035
    ...best so far: -6.98128607035
    ...best so far: -6.99655470744
    ...best so far: -6.99655470744
    ...best so far: -6.99655470744
    ...best so far: -6.99880626416
    ...best so far: -6.99950646219
    ...best so far: -6.99950646219
    ...best so far: -6.99972928513
    ...best so far: -6.99991771801
    ...best so far: -6.99995652906
    ...best so far: -6.99995652906
    ...best so far: -6.99998705445
    ...best so far: -6.99999239547
    ...best so far: -6.99999584097
    ...best so far: -6.99999751173
    ...best so far: -6.99999837535
    ...best so far: -6.99999975505
    ...best so far: -6.99999975505
    ...best so far: -6.99999975505
    ...best so far: -6.99999994341
    ...best so far: -6.99999994341
    ...best so far: -6.99999995028
    ...best so far: -6.99999997999
    ...best so far: -6.99999999846
    ...best so far: -6.99999999846

    [array([ 2.99996614,  2.00000911]), -6.9999999984625303]

```python
#https://codesachin.wordpress.com/2016/01/16/nelder-mead-optimization/
from IPython.display import YouTubeVideo
# Evaluates the function:
# f(x,y) = x^2 - 4*x + y^2 - y - x*y;
YouTubeVideo("HUqLxHfxWqU")
```

##### Scipy implementation

http://www.scipy-lectures.org/advanced/mathematical_optimization/

In scipy, scipy.optimize.fmin() implements the Nelder-Mead approach:

```python
from scipy import optimize

optimize.fmin(f, [2, 2])
```

    Optimization terminated successfully.
             Current function value: -7.000000
             Iterations: 40
             Function evaluations: 73

    array([ 2.99998082,  2.00001514])

https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html

```python
optimize.minimize(f, [2, 2],method='Nelder-Mead')
```

     final_simplex: (array([[ 2.99998082,  2.00001514],
           [ 2.9999984 ,  1.99993767],
           [ 2.99993967,  1.99992527]]), array([-7., -7., -7.]))
               fun: -6.9999999991122497
           message: 'Optimization terminated successfully.'
              nfev: 73
               nit: 40
            status: 0
           success: True
                 x: array([ 2.99998082,  2.00001514])

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## 4.2 Newton-Raphson Method

banana " f = ('-100*(x(2)-x(1)^2)^2-(1-x(1))^2')", so-called because its contours resemble bananas.

```python
f = lambda x,y:(-100*(y-x**2)**2-(1-x)**2)

# def f(x,y):
#     # the height function
#     return (1 - x / 2 + x**5 + y**3) * np.exp(-x**2 -y**2)

n = 256
x = np.linspace(-0.25, 1.25, n)
y = np.linspace(-0.25, 1.25, n)
X,Y = np.meshgrid(x, y)

plt.figure()
x0,y0 = 0,0
# use plt.contourf to filling contours
# X, Y and value for (X,Y) point
plt.contourf(X, Y, f(X, Y), 38, alpha=.75,cmap='bone')# cmap=plt.cm.hot)

# use plt.contour to add contour lines
C = plt.contour(X, Y, f(X, Y), 38, colors='black', linewidth=.5)

plt.clabel(C, inline=True, fontsize=10)
# plt.xticks(())
# plt.yticks(())

# set dot styles
plt.scatter([x0, ], [y0, ], s=50, color='b')

plt.xlim(-0.25, 1.25)
plt.ylim(-0.25, 1.25)
```

The Newton-Raphson method for maximizing an objective function uses successive
quadratic approximations to the objective in the hope that the maxima of the approximants
will converge to the maximum of the objective. The Newton-Raphson
method is intimately related to the Newton method for solving rootfinding problems.

Indeed, the Newton-Raphson method is identical to applying Newton's method to
compute the root of the gradient of the objective function.

The Taylor series of $f(x)$ about the point  $x=x_0 + \epsilon$ is given by

$$f(x_0 + \epsilon) = f(x_0)+ f'(x_0) \epsilon +\frac{1}{2} f''(x_0) \epsilon^2 $$

$$f(x) = f(x^{(k)})+ f'(x^{(k)}) (x-x^{(k)}) + \frac{1}{2}(x-x^{(k)})^T  f''(x^{(k)}) (x-x^{(k)}) $$

```python

```

Solving the first order condition

```python

```

$$f'(x^{(k)})+ f''(x^{(k)}) (x-x^{(k)}) = 0$$

yields the iteration rule

$$x^{(k+1)} \leftarrow x^{(k)} -  [f''(x^{(k)})]^{-1} f'(x^{(k)}) $$

In theory, the Newton-Raphson method converges if $f$ is twice continuously difierentiable
and if the initial value of x supplied by the analyst is sufficiently close to a
local maximum of $f$ at which the **Hessian $f''$** is negative definite. There is, however,
no generally practical formula for determining what sufficiently close is.

```python

```

The Newton-Raphson method can be robust to the starting
value if $f$ is well behaved, for example, if f is **globally concave**. The Newton-Raphson
method, however, can be very sensitive to starting value if the function is not globally
concave. Also, in practice, the **Hessian $f''$**  must be well-conditioned at the optimum,
otherwise rounding errors in the vicinity of the optimum can make it difficult to
compute a precise approximate solution.

The Newton-Raphson algorithm has numerous drawbacks.

First, the algorithm
requires computation of both the first and second derivatives of the objective function.

Second, the Newton-Raphson algorithm offers no **guarantee** that the objective function
value may be increased in the direction of the Newton step. Such a guarantee is
available only if the Hessian **Hessian $f''(x^k)$** is **negative definite**; otherwise, one may actually
move towards a saddle point of f (if the Hessian is indefinite) or even a minimum (if
Hessian is **positive definite**).

For this reason, the Newton-Raphson method is rarely
used in practice, and then only if the objective function is **globally concave**.

## 4.3 Quasi-Newton Methods

Quasi-Newton methods employ a similar strategy to the Newton-Raphson method,
but **replace the Hessian of the objective function (or its inverse) with a negative
definite approximation, guaranteeing that function value can be increased in the direction
of the Newton step**.

The most efficient quasi-Newton algorithms employ an
approximation to the inverse Hessian, rather than the Hessian itself, in order to avoid
performing a linear solve, and employ updating rules that do **not require second
derivative information** to ease the burden of implementation and the cost of computation.

In analogy with the Newton-Raphson method, quasi-Newton methods use a search
direction of the form

$$d^{(k)} = -B^{(k)} f'(x^{(k)})$$

where $B^{(k)}$ is an approximation to the **inverse Hessian** of f at the kth iterate $x^{(k)}$.
The vector $d^{(k)}$ is called the **Newton or quasi-Newton step**.

The more robust quasi-Newton methods do not necessarily take the full Newton
step, but rather shorten it or lengthen it in order to obtain improvement in the
objective function. This is accomplished by performing a line-search in which one
seeks a **step length $s > 0$** that maximizes or nearly maximizes $f (x^{(k)} + sd^{(k)})$. Given
the computed step length $s^{(k)}$, one updates the iterate as follows:

$$x^{(k+1)}=  x^{(k)} + s^{(k)}  d^{(k)}$$

Quasi-Newton method differ in how the inverse Hessian approximation Bk is constructed
and updated. The simplest quasi-Newton method sets

$$B^{(k)} = - I $$,

where I is  the identity matrix. This leads to a Newton step that is identical to the gradient of
the objective function at the current iterate:

$$d^{(k)} = f'(x^{(k)})$$

The choice of gradient as a step direction is intuitively appealing because the gradient
always points in the direction which, to a first order, promises the greatest increase in
f. For this reason, this quasi-Newton method is called the method of *steepest ascent.*'

The steepest ascent method is simple to implement, but is numerically *less efiicient*
in practice than competing quasi-Newton methods that *incorporate* information regarding
the **curvature of the objective function**.

The **most widely-used** quasi-Newton methods that employ **curvature information**
produce a sequence of inverse Hessian estimates that satisfy two conditions.

**First,**
given that

$$d^{(k)} \approx f''^{-1}(x^{(k)})( f'(x^{(k)}+ d^{(k)} ) - f'(x^{(k)})  )$$

the **inverse Hessian estimate** $A^{(-k)}$ is required to satisfy the so-called **quasi-Newton condition:**

$$d^{(k)} = B^{(k)} (x^{(k)})( f'(x^{(k)}+ d^{(k)} ) - f'(x^{(k)})  )$$

**Second,** the inverse Hessian estimate $A^{(-k)}$  is required to be both **symmetric and
negative-definite**, as must be true of the inverse Hessian at a local maximum. The
negative definiteness of the Hessian estimate assures that the objective function value
can be inreased in the **direction of the Newton step**.

Two methods that satisfy the quasi-Newton and negative definiteness conditions
are the **Davidson-Fletcher-Powell (DFP)** and **Broyden-Fletcher-Goldfarb-Shano (BFGS)**
updating methods. The **DFP** method uses the updating scheme

$$B \leftarrow B + \frac{d d^T}{d^T u}  - \frac{B u u^T B}{u^T B u} $$

where

$$d = x^{(k+1)} - x^{(k)}$$

and

$$u = f'(x^{(k+1)}) - f'(x^{(k)})$$

The **BFGS** method uses the update scheme

$$B \leftarrow B + \frac{1}{d^T u}( w d^T + d w^T  - \frac{w^T u}{d^T u}) d d^T $$

where

$$w = d - B u$$

The BFGS algorithm is generally considered superior to DFP, although there
are problems for which DFP outperforms BFGS. However, except for the updating
formulae, the two methods are identical, so it is easy to implement both and give
users the choice.

```python
step_methods = ['none','bhhh','bt','golden']
search_methods = ['steepest','dfp','bfgs']
```

```python
# step_methods = {'none': _step_none,
#                      'bhhh': _step_bhhh,
#                      'bt': _step_bt,
#                      'golden': _step_golden
#                      }
# search_methods = {'steepest': _search_steepest,
#                        'bfgs': _search_bfgs,
#                        'dfp': _search_dfp
#                        }
```

```python
# def _search_bfgs(f, ff=None, u=None, d=None):
#         ud = np.inner(u, d)
#         w = d - B.dot(u)
#         wd = np.outer(w, d)
#         return  B+ ((wd + wd.T) - (np.inner(u, w) * np.outer(d, d)) / ud) / ud
#         # self.reset = False

# def _search_dfp(self, ff=None, u=None, d=None):
#         ud = np.inner(u, d)
#         v = B.dot(u)
#         return B+ np.outer(d, d) / ud - np.outer(v, v) / np.inner(u, v)
#         #self.reset = False

# def _search_steepest(self, ff, u=None, d=None):
#     return   -np.identity(k) / np.maximum(abs(fx0), 1)
```

```python

```

```python
# this function optstep is not covered in textbook. Only supporing implementation of qnewton.
errcode = False
def optstep(stepmeth,func, x0, fx0, g0, d, maxstep = 1000):
    # take multiple output of function
    A = func(x)
    _is_there_jacobian = (type(A) is tuple) and (len(A) == 2)

    if _is_there_jacobian:
        #print('Jacobian was provided by user!')
        f = lambda z:  func(z)[0]

    # several step search method
    def _step_none(f, x0, fx0, d,maxstep):
        fx = f(x0 + d)
        if fx < fx0:
            s = 1
            errcode = False
            return s, f
        else:
            return _step_golden(f, x0, fx0, d,maxstep)

    def _step_bhhh(f, x0, fx0, g0, d,maxstep):
        # Intializations
        delta = 0.0001
        dg = -np.inner(g0, d)  # directional derivative
        tol1 = dg * delta
        tol0 = dg * (1 - delta)
        s, ds = 1, 1
        errcode = False

        # Bracket the cone
        for it in range(maxstep):
            x = x0 + s * d
            fs = f(x)
            temp = (fx0 - fs) / s
            if temp < tol0:
                ds *= 2
                s += ds
            else:
                break

        if (tol0 <= temp) and (temp <=tol1):
            return s, fs

        ds /= 2
        s -= ds
        it0 = it + 1

        # Then use bisection to get inside it
        for it in range(it0, maxstep):
            ds /= 2
            x = x0 + s * d
            fs =  f(x)
            temp = (fx0 - fs) / s
            if temp > tol1:
                s -= ds
            elif temp < tol0:
                s += ds
            else:
                return s, fs

        # If it has not returned yet, call _step_golden!
        return _step_golden(f, x0, fx0, d, maxstep)

    def _step_bt(f, x0, fx0, g0, d, maxstep):
        delta = 1e-4 # Defines cone of convergence; must be on (0,1/2)
        ub = 0.5     # Upper bound on acceptable reduction in s.
        lb = 0.1     # Lower bound on acceptable reduction in s.
        errcode = 0
        dg = -np.inner(d, g0)  # directional derivative
        tol1 = delta * dg
        tol0 = (1 - delta) * dg

        # full step
        s = 1
        fs = f(x0+d)
        if (fx0 - fs)   <= tol1:
            return s, fs

        # quadratic approximation
        s2, fs2 = s, fs
        s = -0.5 * dg / (-fs + fx0 - dg)
        s = max(s, lb)
        fs = f(x0 + s * d)
        temp = (-fs + fx0) / s
        if (tol0 <= temp) and (temp <= tol1):
            return s, fs

        # cubic approximation
        for it in range(3, maxstep):
            temp = (s - s2) * np.array([s * s, s2 * s2])
            temp = np.array([- fs + fx0 - dg * s, -fs2 + fx0 - dg * s2]) / temp
            a = temp[0] - temp[1]
            b = s * temp[1] - s2 * temp[0]
            s2 = s
            fs2 = fs
            if np.all(a == 0):  # quadratic fits exactly
                s = -0.5 * dg / b
            else:
                disc = b * b - 3 * a * dg
                if np.all(disc < 0):
                    errcode = 2
                    return s, fs  # complex root
                s = (np.sqrt(disc) - b) / (3 * a)

            s = np.maximum(np.minimum(s, ub * s2), lb * s2)  # ensures acceptable step size; cp(f, lb, up)
            fs = f(x0 + s * d)
            temp = (-fs + fx0) / s
            if np.all(tol0 <= temp) and np.all(temp <= tol1):
                return s, fs

        # If it has not returned yet, call _step_golden instead
        return _step_golden(f, x0, fx0, d,maxstep)

    def _step_golden(f, x0, fx0, d,maxstep):
        alpha1 = (3 - np.sqrt(5)) / 2
        alpha2 = (np.sqrt(5) - 1) / 2
        tol = 1.e-4
        tol *= alpha1*alpha2
        s = 1
        errcode = True
        niter = 0
        s0 = 0
        it = 0

        # Find a bracketing interval
        fs = f(x0 + d)
        if fx0 >= fs:
            lenght = alpha1
        else:
            for it in range(maxstep):
                s *= 2
                fl = fs
                fs = f(x0 + s*d)
                if fs <=fl:
                    lenght = alpha1 * (s - s0)
                    break
                else:
                    s0 /= 2

            if (it + 1) >= maxstep:
                s /= 2
                fs = fl
                return s, fs

        xl = x0 + (s + lenght) * d
        xs = x0 + (s - lenght) * d

        s -= lenght
        lenght *= alpha2  # lenght now measures relative distance between xl and xs

        fs = f(xs)
        fl = f(xl)

        # Golden search to find minimum
        while it < maxstep:
            it += 1
            if fs < fl:
                s -= lenght
                lenght *= alpha2
                xs = xl
                xl -= lenght * d
                fs = fl
                fl = f(xl)
            else:
                lenght *= alpha2
                s += lenght
                xl = xs
                xs += lenght * d
                fl = fs
                fs = f(xs)

            if lenght < tol:
                errcode = False
                break
        if fl > fs:
            fs = fl
            s -= lenght
        return s, fs

    # return resulted s and fx
    if stepmeth == None:
        return _step_none(f, x0, fx0, d,maxstep)
    elif stepmeth == "bhhh":
        return _step_bhhh(f, x0, fx0, g0, d,maxstep)
    elif stepmeth == "bt":
        return _step_bt(f, x0, fx0, g0, d,maxstep)
    elif stepmeth == "golden":
        return _step_golden(f, x0, fx0, d,maxstep)

```

```python
def f(x):
    y = (-100*(x[1]-x[0]**2)**2-(1-x[0])**2)
    dy = np.array([2*(1-x[0])+400*(x[1]-x[0]**2)*x[0],  -200*(x[1]-x[0]**2)])
    return y,dy
```

```python
s, fx = optstep("golden" ,f, x, fx0, g0, d, maxstep)
```

```python
s,fx
```

The script assumes that the user
has written a Python routine f that evaluates the function at an arbitrary point and
that the user has specified a starting point x, an initial guess for the inverse Hessian
A, a convergence tolerance tol, and a limit on the number of iterations maxit. The
script uses an auxiliary algorithm optstep to determine the step length (discussed
in the next section). The algorithm also offers the user a choice on how to select the
search direction, searchmeth (1-steepest ascent, 2-DFP, 3-BFGS).

https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html

```python

```

```python
#         if self.x0 is None or self.x0[0] is None:
#             raise ValueError('Initial value is required to solve a OP, none provided!')
```

```python
x_list = list()#  sequence of solutions of x for ploting

x0 = np.array([1.,0.]) # initial value for x

maxit, maxstep, tol,eps0, eps1,all_x  = 10000, 10000, 1/10000,1.0,1.e-12 ,False # keyword arguments

x_list = [x0] # first x
searchmeth =2 #  pick a search method.
stepmeth = "bt"
```

```python
x = x0 # initialize
k = x.shape[0] # number of variables
eps = np.spacing(1) # epsolin

A = f(x) # tuble of multiple outputs from object function

_is_there_jacobian = (type(A) is tuple) and (len(A) == 2)

# get first fx and g. object value and gradient/hessian value.
if _is_there_jacobian:
    print('Jacobian was provided by user!')
    fx0,g0 = f(x)
else:
    print('Jacobian was not provided by user!')
    fx0 = f(x)
    try:
        g0 = jacobian(f,x) # customized jacobian function
    except NameError:
        print("jacobian function Not in scope!\n Using identity matrix as jacobian matrix")
        g0 = np.identity(k)
    else:
        print("jacobian function In scope!")

B = None # inversed Hessian matrix

if B is None:
    B =  -np.identity(k) / np.maximum(abs(fx0), 1)   # using identity matrix as Hessian
    print("Hessian is not provide and reset as normailized identity matrix! so steepest ascent") # steepest ascent
```

    Jacobian was provided by user!
    Hessian is not provide and reset as normailized identity matrix! so steepest ascent

```python
import warnings
```

```python
if np.linalg.norm(g0) < eps: # similar to np.all(g0<eps)
    #break #return x
    print("g0 is less than eps")
if np.all(g0 < eps): # check conditions
    #break #return x
    print("g0 is less than eps")
print("Solving nonlinear equations by using {} search method and {} step method".format(search_methods[searchmeth-1].capitalize(), stepmeth))

print("Start iteration......")

for it in range(maxit):

    d = -np.dot(B, g0)  # search direction

    if (np.inner(d, g0) / (np.inner(d, d))) < eps1:  # must go uphill
        B =  -np.identity(k) / np.maximum(abs(fx0), 1) # otherwise use
        d = g0 / np.maximum(np.abs(fx0), 1)  # steepest ascent

    s, fx = optstep("bt" ,f, x, fx0, g0, d, maxstep)

    if fx <= fx0:

        warnings.warn('Iterations stuck in qnewton')
        # break #x # return x
        # reset Hessian and d.
        B =  -np.identity(k) / np.maximum(abs(fx0), 1) # otherwise use
        d = g0.T / np.maximum(abs(fx0), 1)  # steepest ascent
        s, fx = optstep("bt" ,f, x, fx0, g0, d, maxstep)
        if errcode:
            warnings.warn('Cannot find suitable step in qnewton')
            # return x
            # reset to 1 and fx0
            s, fx =  1, fx0

    d *= s
    x = x + d

    x_list.append(x.copy())

    if np.any(np.isnan(x) | np.isinf(x)):
        raise ValueError('NaNs or Infs encountered')

    # update fx and g
    if _is_there_jacobian:
        #print('Jacobian was provided by user!')
        fx,g = f(x)
    else:
        print('Jacobian was not provided by user!')
        fx = f(x)
        try:
            g = jacobian(f,x)
        except NameError:
            print("jacobian function Not in scope!\n Using identity matrix as jacobian matrix")
            g = np.identity(k)
        else:
            print("jacobian function In scope!")

    # Test convergence using Marquardt's criteria and gradient test
    if ((fx - fx0) / (abs(fx) + eps0) < tol and
            np.all(np.abs(d) / (np.abs(x) + eps0) < tol)) or\
            np.all(np.abs(g) < eps):
            print("Meet the tol. x: ", x)
            break
#         #return x

#     if np.all( np.abs(d)/(np.abs(x) + eps0)< tol) or np.all(np.abs(g) < eps):
#         print("Meet the tol. x: ", x)
#         break

    # Update inverse Hessian
    u = g - g0  # change in Jacobian
    ud = np.inner(u, d)

    #print("Please specify one search method: 1:steepest ascen;2: DFP;3:BFGS")
    if np.all(np.abs(ud) < eps):
        B =  -np.identity(k) / np.maximum(abs(fx0), 1) # otherwise use
    else:
        if searchmeth == 1 and np.abs(ud) < eps: # steepest ascent
            B =  -np.identity(k) / np.maximum(abs(fx), 1)
        elif searchmeth == 2: # DFP
            v = B.dot(u)
            B += np.outer(d, d) / ud - np.outer(v, v) / np.inner(u, v)
        elif searchmeth == 3: # BFGS
            w = d - B.dot(u)
            wd = np.outer(w, d)
            B += ((wd + wd.T) - (np.inner(u, w) * np.outer(d, d)) / ud) / ud
#         else:
#             print("Please specify one search method: 1:steepest ascen;2: DFP;3:BFGS")

    # Update iteration
    fx0 = fx
    g0 = g
    print("finish {}th iteration...".format(it))

#print("x list: " +  for str(x) in x_list)
if it > maxit:
    warnings.warn('Maximum iterations exceeded in qnewton')
```

    Solving nonlinear equations by using Dfp search method and bt step method
    Start iteration......
    finish 0th iteration...
    finish 1th iteration...
    finish 2th iteration...
    finish 3th iteration...
    finish 4th iteration...
    finish 5th iteration...
    finish 6th iteration...
    finish 7th iteration...
    finish 8th iteration...
    finish 9th iteration...
    finish 10th iteration...
    finish 11th iteration...
    finish 12th iteration...
    finish 13th iteration...
    finish 14th iteration...
    finish 15th iteration...
    finish 16th iteration...
    finish 17th iteration...
    finish 18th iteration...
    finish 19th iteration...
    finish 20th iteration...
    finish 21th iteration...
    finish 22th iteration...
    finish 23th iteration...
    finish 24th iteration...
    finish 25th iteration...
    finish 26th iteration...
    finish 27th iteration...
    finish 28th iteration...
    finish 29th iteration...
    finish 30th iteration...
    Meet the tol. x:  [ 0.99999993  0.99999986]

```python
x_list

```

    [array([ 1.,  0.]),
     array([ 0.41314554,  0.29342723]),
     array([ 0.39580618,  0.1727269 ]),
     array([ 0.39675149,  0.15838385]),
     array([ 0.40058059,  0.16006693]),
     array([ 0.44685739,  0.18183245]),
     array([ 0.43732885,  0.17841517]),
     array([ 0.43639767,  0.17942195]),
     array([ 0.43473828,  0.18607608]),
     array([ 0.43680863,  0.18965204]),
     array([ 0.4480958 ,  0.20703251]),
     array([ 0.45412053,  0.21461039]),
     array([ 0.47763513,  0.24324066]),
     array([ 0.49988381,  0.26943615]),
     array([ 0.60901205,  0.39721893]),
     array([ 0.62360754,  0.38653846]),
     array([ 0.65591217,  0.42617689]),
     array([ 0.70613869,  0.49067417]),
     array([ 0.79844428,  0.61783675]),
     array([ 0.77015994,  0.58500286]),
     array([ 0.78330409,  0.60878183]),
     array([ 0.83733992,  0.70089268]),
     array([ 0.92482331,  0.84647259]),
     array([ 0.90279767,  0.81240971]),
     array([ 0.91516948,  0.83493489]),
     array([ 0.9689266 ,  0.93422914]),
     array([ 0.9676163 ,  0.93401163]),
     array([ 0.97753087,  0.95539111]),
     array([ 0.98854753,  0.97740207]),
     array([ 0.99970154,  0.99931034]),
     array([ 0.9998261,  0.9996212]),
     array([ 1.00000204,  1.00000394]),
     array([ 0.99999993,  0.99999986])]

```python

```

```python
def myqnewton(f, x0, B, searchmeth = 3,stepmeth = "bt" ,maxit = 10000, maxstep = 10000,tol = 1/100000,\
              eps = np.spacing(1),eps0 =1.0, eps1 = 1.e-12, all_x = False):
    '''
    maxit, maxstep, tol,eps0, eps1  = 10000, 10000, 1/10000,1.0,1.e-12
    f: object function and jacobian
    x0: initial value
    all_x: if we collect x value for plotting
    '''
    x = x0
    if all_x:
        x_list = [x0]

    A = f(x)
    _is_there_jacobian = (type(A) is tuple) and (len(A) == 2)

    if _is_there_jacobian:
        print('Jacobian was provided by user!')
        fx0,g0 = f(x)
    else:
        print('Jacobian was not provided by user!')
        fx0 = f(x)
        try:
            g0 = jacobian(f,x)
        except NameError:
            print("jacobian function Not in scope!\n Using identity matrix as jacobian matrix")
            g0 = np.identity(k)
        else:
            print("jacobian function In scope!")

    if np.all(np.abs(g0) < eps): # similar to np.all(g0<eps)
        print("abs(g0)< eps...")
        return x

    print("Solving nonlinear equations by using {} search method and {} step method".format(search_methods[searchmeth-1].capitalize(), stepmeth))

    print("Start iteration......")

    for it in range(maxit):

        d = -np.dot(B, g0)  # search direction, initial d

        # https://github.com/randall-romero/CompEcon-python/blob/master/compecon/optimize.py
        if (np.inner(d, g0) / (np.inner(d, d))) < eps1:  # must go uphill
            B =  -np.identity(k) / np.maximum(abs(fx0), 1) # otherwise use
            d = g0 / np.maximum(np.abs(fx0), 1)  # steepest ascent
        # optimize search step length
        s, fx = optstep(stepmeth ,f, x, fx0, g0, d, maxstep)

        if fx <= fx0:

            warnings.warn('Iterations stuck in qnewton')
            #return x
            # reset Hessian and d.
            B =  -np.identity(k) / np.maximum(abs(fx0), 1) # otherwise use
            d = g0.T / np.maximum(abs(fx0), 1)  # steepest ascent
            s, fx = optstep("bt" ,f, x, fx0, g0, d, maxstep)
            if errcode:
                warnings.warn('Cannot find suitable step in qnewton')
                # return x
                # reset to 1 and fx0
                s, fx =  1, fx0

        # update d and x
        d *= s
        x = x + d
        # keep record of x sequence in list
        if all_x:
            x_list.append(x.copy())

        if np.any(np.isnan(x) | np.isinf(x)):
            raise ValueError('NaNs or Infs encountered')

        #  update fx and g again
        if _is_there_jacobian:
            #print('Jacobian was provided by user!')
            fx,g = f(x)
        else:
            print('Jacobian was not provided by user!')
            fx = f(x)
            try:
                g = jacobian(f,x)
            except NameError:
                print("jacobian function Not in scope!\n Using identity matrix as jacobian matrix")
                g = np.identity(k)
            else:
                print("jacobian function In scope!")

        # Test convergence using Marquardt's criteria and gradient test
        if ((fx - fx0) / (abs(fx) + eps0) < tol and
                np.all(np.abs(d) / (np.abs(x) + eps0) < tol)) or\
                np.all(np.abs(g) < eps):
                print("Meet the tol. x: ", x)
                #break
                if all_x:
                    return x, x_list
                else:
                    return x

        # Update inverse Hessian
        u = g - g0  # change in Jacobian
        ud = np.inner(u, d)

        # pick a search method
        #print("Please specify one search method: 1:steepest ascen;2: DFP;3:BFGS")
        if np.all(np.abs(ud) < eps):
            B =  -np.identity(k) / np.maximum(abs(fx0), 1) # otherwise use
        else:
            if searchmeth == 1 and np.abs(ud) < eps: # steepest ascent
                B =  -np.identity(k) / np.maximum(abs(fx), 1)
            elif searchmeth == 2: # DFP
                v = B.dot(u)
                B += np.outer(d, d) / ud - np.outer(v, v) / np.inner(u, v)
            elif searchmeth == 3: # BFGS
                w = d - B.dot(u)
                wd = np.outer(w, d)
                B += ((wd + wd.T) - (np.inner(u, w) * np.outer(d, d)) / ud) / ud

        # Update iteration
        fx0 = fx
        g0 = g
        print("finish {}th iteration...".format(it))

    # end of iteration if exceed the maxit
    if it >= maxit:
        warnings.warn('Maximum iterations exceeded in qnewton')
        return x
```

```python
myqnewton(f, x0, B, searchmeth = 3,stepmeth = "bt" ,maxit = 10000, maxstep = 10000,tol = 1/100000,\
              eps = np.spacing(1),eps0 =1.0, eps1 = 1.e-12, all_x = False)
```

    Jacobian was provided by user!
    Solving nonlinear equations by using Bfgs search method and bt step method
    Start iteration......
    finish 0th iteration...
    finish 1th iteration...
    finish 2th iteration...
    finish 3th iteration...
    finish 4th iteration...
    finish 5th iteration...
    finish 6th iteration...
    finish 7th iteration...
    finish 8th iteration...
    finish 9th iteration...
    finish 10th iteration...
    Meet the tol. x:  [ 1.00000061  1.00000117]

    array([ 1.00000061,  1.00000117])

```python
myqnewton(f, x0, B, searchmeth = 2,stepmeth = "bt" ,maxit = 10000, maxstep = 10000,tol = 1/100000,\
              eps = np.spacing(1),eps0 =1.0, eps1 = 1.e-12, all_x = False)
```

    Jacobian was provided by user!
    Solving nonlinear equations by using Dfp search method and bt step method
    Start iteration......
    finish 0th iteration...
    finish 1th iteration...
    finish 2th iteration...
    finish 3th iteration...
    finish 4th iteration...
    finish 5th iteration...
    finish 6th iteration...
    finish 7th iteration...
    finish 8th iteration...
    finish 9th iteration...
    finish 10th iteration...
    finish 11th iteration...
    finish 12th iteration...
    finish 13th iteration...
    finish 14th iteration...
    finish 15th iteration...
    finish 16th iteration...
    finish 17th iteration...
    finish 18th iteration...
    finish 19th iteration...
    finish 20th iteration...
    finish 21th iteration...
    finish 22th iteration...
    finish 23th iteration...
    finish 24th iteration...
    finish 25th iteration...
    finish 26th iteration...
    Meet the tol. x:  [ 0.99999807  0.99999585]

    array([ 0.99999807,  0.99999585])

```python
myqnewton(f, x0, B, searchmeth = 2,stepmeth = "bt" ,maxit = 10000, maxstep = 10000,tol = 1/100000,\
              eps = np.spacing(1),eps0 =1.0, eps1 = 1.e-12, all_x = True)
```

    Jacobian was provided by user!
    Solving nonlinear equations by using Dfp search method and bt step method
    Start iteration......
    finish 0th iteration...
    finish 1th iteration...
    finish 2th iteration...
    finish 3th iteration...
    finish 4th iteration...
    finish 5th iteration...
    finish 6th iteration...
    finish 7th iteration...
    finish 8th iteration...
    finish 9th iteration...
    finish 10th iteration...
    finish 11th iteration...
    finish 12th iteration...
    finish 13th iteration...
    finish 14th iteration...
    finish 15th iteration...
    finish 16th iteration...
    finish 17th iteration...
    finish 18th iteration...
    finish 19th iteration...
    finish 20th iteration...
    finish 21th iteration...
    finish 22th iteration...
    finish 23th iteration...
    finish 24th iteration...
    finish 25th iteration...
    finish 26th iteration...
    finish 27th iteration...
    finish 28th iteration...
    finish 29th iteration...
    finish 30th iteration...
    Meet the tol. x:  [ 1.  1.]

    (array([ 1.,  1.]),
     [array([ 1.,  0.]),
      array([ 0.62610675, -0.45538251]),
      array([ 0.80942433,  0.73890693]),
      array([ 0.80926495,  0.65836064]),
      array([ 0.81029476,  0.65658064]),
      array([ 0.81149645,  0.65795954]),
      array([ 0.81937217,  0.66750197]),
      array([ 0.82112266,  0.67005007]),
      array([ 0.84119489,  0.69949451]),
      array([ 0.83921082,  0.69679031]),
      array([ 0.83394945,  0.68989746]),
      array([ 0.83252062,  0.68833605]),
      array([ 0.83030783,  0.68639339]),
      array([ 0.83004821,  0.68680305]),
      array([ 0.83034079,  0.6889422 ]),
      array([ 0.83128484,  0.69133142]),
      array([ 0.83394548,  0.69736556]),
      array([ 0.83614526,  0.70186326]),
      array([ 0.84123512,  0.71195744]),
      array([ 0.84518495,  0.71952602]),
      array([ 0.85314137,  0.73456449]),
      array([ 0.86088506,  0.74901455]),
      array([ 0.87586691,  0.77680972]),
      array([ 0.91582748,  0.85079014]),
      array([ 0.92108449,  0.84802548]),
      array([ 0.95707049,  0.91414103]),
      array([ 0.98409515,  0.96593609]),
      array([ 0.98863641,  0.97619571]),
      array([ 0.9968118,  0.993516 ]),
      array([ 0.99961717,  0.99919731]),
      array([ 0.99997711,  0.99995547]),
      array([ 1.00000171,  1.00000325]),
      array([ 1.,  1.])])

```python
myqnewton(f, x0, B, searchmeth =1,stepmeth = "bt" ,maxit = 10000, maxstep = 10000,tol = 1/100000,\
              eps = np.spacing(1),eps0 =1.0, eps1 = 1.e-12, all_x = False)
```

    Jacobian was provided by user!
    Solving nonlinear equations by using Steepest search method and bt step method
    Start iteration......
    finish 0th iteration...
    finish 1th iteration...
    finish 2th iteration...
    finish 3th iteration...
    finish 4th iteration...
    finish 5th iteration...
    finish 6th iteration...
    finish 7th iteration...
    finish 8th iteration...
    finish 9th iteration...
    finish 10th iteration...
    finish 11th iteration...
    finish 12th iteration...
    finish 13th iteration...
    finish 14th iteration...
    finish 15th iteration...
    finish 16th iteration...
    finish 17th iteration...
    finish 18th iteration...
    finish 19th iteration...
    finish 20th iteration...
    finish 21th iteration...
    finish 22th iteration...
    finish 23th iteration...
    finish 24th iteration...
    finish 25th iteration...
    finish 26th iteration...
    finish 27th iteration...
    finish 28th iteration...
    finish 29th iteration...
    finish 30th iteration...
    finish 31th iteration...
    finish 32th iteration...
    finish 33th iteration...
    finish 34th iteration...
    finish 35th iteration...
    finish 36th iteration...
    finish 37th iteration...
    finish 38th iteration...
    finish 39th iteration...
    finish 40th iteration...
    finish 41th iteration...
    finish 42th iteration...
    finish 43th iteration...
    finish 44th iteration...
    finish 45th iteration...
    finish 46th iteration...
    finish 47th iteration...
    finish 48th iteration...
    finish 49th iteration...
    finish 50th iteration...
    finish 51th iteration...
    finish 52th iteration...
    finish 53th iteration...
    finish 54th iteration...
    finish 55th iteration...
    finish 56th iteration...
    finish 57th iteration...
    finish 58th iteration...
    finish 59th iteration...
    finish 60th iteration...
    finish 61th iteration...
    finish 62th iteration...
    finish 63th iteration...
    finish 64th iteration...
    finish 65th iteration...
    finish 66th iteration...
    finish 67th iteration...
    finish 68th iteration...
    finish 69th iteration...
    finish 70th iteration...
    finish 71th iteration...
    finish 72th iteration...
    finish 73th iteration...
    finish 74th iteration...
    Meet the tol. x:  [ 1.00000592  1.00001275]

    array([ 1.00000592,  1.00001275])

```python

```

## 4.4 Line Search Methods
Just as was the case with rootfinding problems, it is not always best to take a full
Newton step. In fact, it may be better to either stop short or move past the Newton
step. If we view the Newton step as defining a *search direction*, performing a onedimensional
search in that direction will generally produce improved results.

http://reference.wolfram.com/language/tutorial/UnconstrainedOptimizationLineSearchMethods.html

https://en.wikipedia.org/wiki/Line_search

A number of diffierent line
search methods are used in practice, including the golden search method.

The **golden
search** algorithm is very reliable, but computationally inefficient. Two alternative
schemes are typically used in practice to perform line searches.

The first, known as
the **Armijo search**, is similar to the backstepping algorithm used in rootfinding and

complementarity problems. The idea is to find the minimum power j such that

```python
# https://github.com/smwade/ACME-2/blob/master/line_search/solutions.py

def backtracking(f, slope, x, p, a=1, rho=.9, c=10e-4):
    """Perform a backtracking line search to satisfy the Armijo Conditions.
    Parameters:
        f (function): the twice-differentiable objective function.
        slope (float): The value of grad(f)^T p.
        x (ndarray of shape (n,)): The current iterate.
        p (ndarray of shape (n,)): The current search direction.
        a (float): The intial step length. (set to 1 in Newton and
            quasi-Newton methods)
        rho (float): A number in (0,1).
        c (float): A number in (0,1).

    Returns:
        (float) The computed step size satisfying the Armijo condition.
    """
    while f(x + a*p) > f(x) + c * a * slope:
        a = float(rho * a)
    return a

```

Another widely-used approach, known as **Goldstein search**, is to find any value of
s that satisfies

A simple strategy for locating an acceptable point is to first find a point in or
above the cone using step doubling (doubling the value of s at each iteration). If a
point above the cone is found first, we have a bracket within which points in the cone
must lie. We can then narrow the bracket using the golden search method. We call this the bhhhstep approach.

Another approach, stepbt, checks to see if s = 1 is in the cone and, if so, maximizes
a quadratic approximation to the objective function in the Newton direction
constructed from knowledge of f(x), f0(x)d and f(x + d). If the computed step s is
acceptable, it is taken. Otherwise, the algorithm iterates until an acceptable step is
found using a cubic approximation to the objective function in the Newton direction
constructed from knowledge of f(x), f0(x)d, f(x + s(j􀀀1)d) and f(x + s(j)d). stepbt
is fast and generally gives good results. It is recommended as the default lines search
procedure for general maximization algorithms.

```python
#

```

```python

```

```python

```

##  4.5 Special Cases
Two special cases arise often enough in economic practice (especially in econometrics)
to warrant additional discussion. Nonlinear least squares and the maximum likelihood
problems have objective functions with special structures that give rise to their
own special quasi-Newton methods. The special methods differ from other Newton
and quasi-Newton methods only in the choice of the matrix used to approximate the
Hessian. Because these problems generally arise in the context of statistical applications,
we alter our notation to conform with the conventions for those applications.
The optimization takes place with respect to a k-dimensional parameter vector $\theta$ and
n will refer to the number of observations.

```python

```

```python

```

## Reference

- Optimization and Solving Systems of Equations in Julia

https://github.com/pkofod/JC2017

https://www.youtube.com/watch?v=E_UlaGoObTw


```python
import sympy as sm
from sympy import *
```

**Analytical solution**

The model project concerns the Stackelberg duopoly model (later extended to an oligopoly).

The opening model is a sequential move model, where two firms compete over quantities of some good. Firm 1 chooses a quantity, *$q_1\geq0$*, of the good. Firm 2 then observes this quantity and sets their own quantity, *$q_2\geq0$*, accordingly.

The model has complete information.

Each firm want to maximize profits w.r.t. quantity supplied using the profit function, known to both parties:
$$\pi_i(q_i,q_j)=q_i[P(Q)-c]$$
*c* is the marginal cost of production, identical for both firms, and *P(Q)* is the market clearing price for the good:
$$P(Q)=a-Q$$
*$Q=q_1+q_2$* is the total production of the good in the economy.

Profit function:

$$
\pi_2(q_1, q_2) = q_2(a - q_1 - q_2 - c)
$$

This must be maximized w.r.t. the quantity of firm 2, q_2

$$
\max_{q_{2\ge0}} \pi_2(q_1, q_2)
$$

```python
# Define the symbols
q_1 = sm.symbols('q_1')
q_2 = sm.symbols('q_2')
a = sm.symbols('a')
c = sm.symbols('c')
```

The model is solved using backwards induction, first firm 2 maximize their profit and find their response function $R_2(q_1)$ that way.

```python
# Define the profit function for firm 2.
profit_2 = q_2 * (a - q_1 - q_2 - c)
```

To get the reaction function for $q_1$, the derivative is taken w.r.t. $q_2$ and isolating for $q_2$

```python
R_2 = sm.solve(format(Derivative(profit_2, q_2).doit()),q_2)[0]
print(f"R_2 = {R_2}")
R_2
```

    R_2 = a/2 - c/2 - q_1/2

$\displaystyle \frac{a}{2} - \frac{c}{2} - \frac{q_{1}}{2}$

Now having found firm 2's response function for a quantity set by firm 1, firm 1 is now able to input this into their profit function

$$\pi_1(q_1, R_2(q_1)) = q_1(a - q_1 - R_2(q_1) - c)$$

$$\max_{q_{1\ge0}} \pi_1(q_1, R_2(q_1))=q_1(a - q_1 - (a/2 - c/2 - q_1/2) - c)$$

```python
profit_1 = q_1 * (a - q_1 - R_2 - c)
```

Again, the derivative is is taken and isolated:

```python
q_1_max = sm.solve(format(Derivative(profit_1, q_1).doit()),q_1)[0]
print(f"q_1* = {q_1_max}")
q_1_max
```

    q_1* = a/2 - c/2

$\displaystyle \frac{a}{2} - \frac{c}{2}$

Now that we have the optimal q_1, we can insert it in the optimal q_2 (the reaction function):

$$R_2(q_1) = a/2 - c/2 - q_1/2$$

$$
R_2(q_1) = a/2 - c/2 - (a/2 - c/2)/2
$$

This gives the optimal q_2:

```python
q_2_max = a/2 - c/2 - (a/2 - c/2)/2
print(f"q_2* = {q_1_max}")
q_2_max
```

    q_2* = a/2 - c/2

$\displaystyle \frac{a}{4} - \frac{c}{4}$

The takeaway from this result is that in the firm 1 have a first mover advantage over firm 2 in the Staceklberg competition.
This is due to the fact, that it is a sequential move model with full information, therefore firm 2's only credible action is to commit to whatever quantity firm 1 sets. Knowing this firm 1 can set their quantity with the response from firm 2 as given.

This is a different result from the Cournot model where both firm would move at the same time and therefore split the production evenly.

Now that the optimal q_1 and q_2 are found, they can be inserted in the respective profit functions for firm 1 and firm 2:

```python
profit_1 = q_1_max * (a - q_1_max - q_2_max - c)
profit_1
```

$\displaystyle \left(\frac{a}{4} - \frac{c}{4}\right) \left(\frac{a}{2} - \frac{c}{2}\right)$

```python
profit_2 = q_2_max * (a - q_1_max - q_2_max - c)
profit_2
```

$\displaystyle \left(\frac{a}{4} - \frac{c}{4}\right)^{2}$

Obviously as the two firms sell their good at the same price, and both have the same marginal cost of production, firm 1 will have a higher profit than firm 2 due to their first mover advantage.

**Extension**

In the following we extend the model to allow more firms than just 2, going from a duopoly to an oligopoly. The new entrant, firm 3, will follow firm 2 in choosing their quantity.

```python
q_3 = sm.symbols('q_3')
```

```python
profit_3 = q_3 * (a - q_1 - q_2 - q_3 - c)
```

```python
R_3 = sm.solve(format(Derivative(profit_3, q_3).doit()),q_3)[0]
R_3
```

$\displaystyle \frac{a}{2} - \frac{c}{2} - \frac{q_{1}}{2} - \frac{q_{2}}{2}$

```python
profit_2_new = q_2 * (a - q_1 - q_2 - R_3 - c)
```

```python
R_2_new = sm.solve(format(Derivative(profit_2_new, q_2).doit()),q_2)[0]
R_2_new
```

$\displaystyle \frac{a}{2} - \frac{c}{2} - \frac{q_{1}}{2}$

This again can be inserted in R_3_new:

```python
R_3_new = a/2-c/2-q_1/2-(a/2-c/2-q_1/2)/2
```

```python
profit_1_new = q_1 * (a - q_1 - R_2_new - R_3_new - c)
```

```python
q_1_max_new = sm.solve(format(Derivative(profit_1_new, q_1).doit()),q_1)[0]
q_1_max_new
```

$\displaystyle \frac{a}{2} - \frac{c}{2}$

```python
q_2_max_new = a/2 - c/2 - (a/2 - c/2)/2
q_2_max_new
```

$\displaystyle \frac{a}{4} - \frac{c}{4}$

```python
q_3_max = a/2 - c/2 - (a/2 - c/2)/2 - (a/4 - c/4)/2
q_3_max
```

$\displaystyle \frac{a}{8} - \frac{c}{8}$

```python
profit_1_new = q_1_max_new * (a - q_1_max_new - q_2_max_new - q_3_max - c)
profit_1_new
```

$\displaystyle \left(\frac{a}{8} - \frac{c}{8}\right) \left(\frac{a}{2} - \frac{c}{2}\right)$

```python
profit_2_new = q_2_max_new * (a - q_1_max_new - q_2_max_new - q_3_max - c)
profit_2_new
```

$\displaystyle \left(\frac{a}{8} - \frac{c}{8}\right) \left(\frac{a}{4} - \frac{c}{4}\right)$

```python
profit_3_new = q_3_max * (a - q_1_max_new - q_2_max_new - q_3_max - c)
profit_3_new
```

$\displaystyle \left(\frac{a}{8} - \frac{c}{8}\right)^{2}$

This augmented model shows, that the later you get to pick your quantity the worse of you get. This is due to the same intuition as the case for 2 firms, the other firms knowing the strategy that there must be commited to hurts the firms going in the later stages.
Another way to exxapnd the amount of firms would be to still have two periods and let the remaining firms compete in a Cournot competition in stage 2.


# Music Machine Learning - Bayesian inference

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. An introduction to [Bayesian inference](#bayesian)
2. A formal introduction to [Variational Auto-Encoders](#vae) (VAEs)
3. An explanation of the [implementation](#implem) of VAEs
4. Some [modifications and tips to improve the reconstruction](#improve) of VAEs **(exercise)**

<a id="recap"> </a>

## Introduction on Bayesian inference

Here, we discuss *Bayesian inference* and how to use the **Bayes theorem** to perform classification. First, we will see how to derive *estimators* for the different properties of the distributions, and verify that these are *unbiased*. Then, we will implement the **Maximum Likelihood** Estimators (MLE) in order to perform classification of a dataset. First, we will assess the case where parameters are known to implement the discriminant function and decision rule. Then, we will perform the MLE to obtain the means and covariance matrix for each class.

To understand these concepts graphically, we will rely on both the `scikit-learn` and `Pytorch` libraries (with the `probability` package).

```python
import torch
import torch.distributions as distribution
import torch.distributions.transforms as transform
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from helper_plot import hdr_plot_style, plot_gaussian_ellipsoid
hdr_plot_style()
```

### Bayesian framework

Two alternative interpretations of probability can be considered

  * **Frequentist** (*classical* approach), assumes that probability is the *long-term frequency of events*. This becomes harder to interpret when events have no long-term frequency (eg. probability in an election, which happens only once). In that case, frequentists consider the *frequency of occurrences across alternative realities*, which defines the probability.
  * **Bayesian** interprets probability as measure of *believability in an event*. Therefore, a probability is measure of *belief*, or confidence, of an event occurring. This definition leaves room for conflicting beliefs based on the different *information* about the world. Hence, bayesian inference is mostly based on *updating your beliefs* after considering new *evidence*.

To align with probability notation, we denote a belief about event $a$ as $p\left(a\right)$, called the *prior probability* of an event to occur. We denote the updated belief as $ p\left( a \mid e \right) $, interpreted as the probability of $a$ *given* the new evidence $e$, called the *posterior probability*. The prior belief is not completely removed, but we *re-weight this prior* to incorporate new evidence $e$ (i.e. we put more weight, or confidence, on some beliefs versus others). By introducing prior uncertainty about events, we admit that any guess we make can be wrong. As we gather an *infinite* amount of evidence $N \rightarrow \infty$, the Bayesian results (often) align with frequentist results. Hence for small $N$, inference is *unstable*, where frequentist estimates have more variance and larger confidence intervals. However, by introducing a prior, and returning probabilities, we *preserve the uncertainty* that reflects the instability on a small $N$ dataset.

Updating the *prior belief* to obtain our *posterior belief* is done via the the Bayes' Theorem

$$
\begin{equation}
 p\left( a \mid e \right) \propto \frac{ p\left(e \mid a\right) p\left(a\right) } {p\left(e\right) }
\end{equation}
$$

We see that our posterior belief of event $a$ given the new evidence $e$ is proportional to ($\propto$) the *likelihood* of observing this particular evidence $e$ given the event $a$ ($p\left(e \mid a\right)$) multiplied by our prior belief in that particular event $a$ ($p\left(a\right)$).

### Using Bayesian classification (in `scikit-learn`)

The Bayesian classification methods rely on Bayes's theorem, where we are interested in finding the probability of a label $y$ given some observed features, which we can write as $p(y~|~{\rm features})$. Bayes's theorem tells us how to express this in terms of quantities that we can compute more directly

$$
p(y~|~{\rm features}) = \frac{p({\rm features}~|~y)p(y)}{p({\rm features})}
$$

If we are trying to decide between two labels ($y_1$ and $y_2$), then one way to make this decision is to compute the ratio of the posterior probabilities for each label

$$
\frac{p(y_1~|~{\rm features})}{p(y_2~|~{\rm features})} = \frac{p({\rm features}~|~y_1)}{p({\rm features}~|~y_2)}\frac{p(y_1)}{p(y_2)}
$$

All we need now is some model by which we can compute $p({\rm features}~|~y_i)$ for each label.
Such a model is called a *generative model* because it specifies the hypothetical random process that generates the data.
Specifying this generative model for each label is the main piece of the training of such a Bayesian classifier.
The general version of such a training step is a very difficult task, but we can make it simpler through the use of some simplifying assumptions about the form of this model. If we make *very naive assumptions* (called *Naive Bayes*) about the generative model for each label, we can find a rough approximation of the generative model for each class, and then proceed with the Bayesian classification.

#### Gaussian Naive Bayes

Perhaps the easiest naive Bayes classifier to understand is Gaussian naive Bayes. In this classifier, the assumption is that *data from each label is drawn from a simple Gaussian distribution*. Imagine that we have the following data

```python
hdr_plot_style()
from sklearn.datasets import make_blobs
X, y = make_blobs(200, 2, centers=2, random_state=2, cluster_std=2.2)
plt.figure(figsize=(10, 8)); plt.scatter(X[:, 0], X[:, 1], c=y, s=80, cmap='RdBu', edgecolor='w'); plt.grid(True)
```

One extremely fast way to create a simple model is to assume that the data is described by a Gaussian distribution with no covariance between dimensions. This model can be fit by simply finding the mean and standard deviation of the points within each label, which is all you need to define such a distribution.

With this generative model for each class, we have a simple recipe to compute the likelihood $P({\rm features}~|~L_1)$ for any data point, and thus we can quickly compute the posterior ratio and determine which label is the most probable for a given point. This procedure is implemented in Scikit-Learn's ``sklearn.naive_bayes.GaussianNB`` estimator

```python
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X, y);
```

Now let's generate some new data and predict the label:

```python
rng = np.random.RandomState(0)
Xnew = [-6, -14] + [14, 18] * rng.rand(2000, 2)
ynew = model.predict(Xnew)
```

Now we can plot the data and see where the decision boundary is

```python
fig = plt.figure(figsize=(10,8)); ax = fig.add_subplot(111)
# predict the classification probabilities on a grid
xlim = X[:, 0].min() - .5, X[:, 0].max() + .5
ylim = X[:, 1].min() - .5, X[:, 1].max() + .5
xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 71), np.linspace(ylim[0], ylim[1], 81))
Z = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
Z = Z.reshape(xx.shape)
ax.contourf(xx, yy, Z, cmap=plt.cm.RdBu, alpha=.8)
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu', edgecolor='w');
```

We see a slightly curved boundary in the classifications (generally, the boundary in Gaussian naive Bayes is quadratic). A nice piece of this Bayesian formalism is that it naturally allows for probabilistic classification, which we can compute using the ``predict_proba`` method:

```python
yprob = model.predict_proba(Xnew)
yprob[-8:].round(2)
```

The columns give the posterior probabilities of the first and second label, respectively. If you are looking for estimates of uncertainty in your classification, Bayesian approaches like this can be a useful approach. Of course, the final classification will only be as good as the model assumptions that lead to it, which is why Gaussian naive Bayes often does not produce very good results.

## Bayesian inference

Suppose we have coin and want to estimate the probability of heads ($p$) for it. The coin is Bernoulli distributed:

$$
\begin{equation}
\phi(x)= p^x (1-p)^{(1-x)}
\end{equation}
$$

where $x$ is the outcome, *1* for heads and *0* for tails. Based on $n$ *independent* flips, we have the likelihood:

$$
\begin{equation}
\mathcal{L}(p|\mathbf{x})= \prod_{i=1}^n p^{ x_i }(1-p)^{1-x_i}
\end{equation}
$$

(the independent-trials assumption allows us to just substitute everything into $ \phi(x)$).

The idea of *maximum likelihood* will be to maximize this as the function of $p$ after given all of the $x_i$ data. This means that our estimator, $\hat{p}$, is a function of the observed $x_i$ data, and as such, is a random variable with its own distribution.

**Defining the estimator**

The only way to know for sure that our estimator is correctly defined is to check if the estimator is unbiased, namely, if

$$ \mathbb{E}(\hat{p}) = p $$

### Understanding this behavior with `pytorch`

Without solving for the previous exercise, we can  Let's say that we have 100 samples from a Bernoulli distribution.

```python
bernoulli = distribution.Bernoulli(0.2)
sample = bernoulli.sample((100, ))
sns.distplot(sample)
plt.title("Samples from a Bernoulli (p = .2)")
torch.mean(sample)
```

Now we can use our estimator, and define our problem more formally, by defining our input samples as a dataset of observations $x$, and we are trying to model this set, as a `Bernoulli` distribution, for which the $p$ parameter is unknown (here defined as a `Variable`).

```python
from torch.autograd import Variable
x = Variable(sample).type(torch.FloatTensor)
p = Variable(torch.rand(1), requires_grad=True)
```

Now we can use our estimator to gradually compute the Maximum Likelihood, in order to uncover the estimated probability of the underlying distribution, solely based on observing the samples.

```python
learning_rate = 0.00002
for t in range(1000):
    NLL = -torch.sum(x*torch.log(p) + (1-x)*torch.log(1-p))
    NLL.backward()
    if t % 200 == 0:
        print("loglik = %.4f - p = %.4f - dL/dp = %.4f"%(NLL.data.numpy(), p.data.numpy(), p.grad.data.numpy()))
    p.data -= learning_rate * p.grad.data
    p.grad.data.zero_()
print('Final probability p =', p.data[0])
```

***

**Exercise**

  1. Compute the *log-likelihood* $J=\log(\mathcal{L}(p \mid \mathbf{x}))$ of our given problem
  2. Based on this, compute its derivative $ \frac{dJ}{dp} $
  3. Solve it to find the estimator $\hat{p}$
  4. Verify that this estimator is unbiased $ \mathbb{E}(\hat{p}) = p $
  5. Compute the variance of the estimator $ \mathbb{E}\left(\hat{p}^2\right) $

***

```python
x = Variable(sample).type(torch.FloatTensor)
p = torch.rand(1)
sample = bernoulli.sample((100, ))

######################
# YOUR CODE GOES HERE
######################
# Questions :
# - Pourquoi est-ce que la formule du haut marche quand même ?
# - Koman on fé si l'on ne connaît pas la loi que suit la distribution ?
# - _Model boosting_ : méthode où l'on choisit entre plusieurs modèle selon
#   leur pertinence au problème a posteriori.
ll = lambda p, x: np.sum(x*np.log(p) + (1-x)*np.log(1-p))
dll_dp = lambda p, x: np.sum(x/p - (1-x)/(1-p))

learning_rate = 0.00002
for t in range(1000):
    nll = -ll(p.data.numpy(),x.data.numpy())
    grad = -dll_dp(p.data.numpy(), x.data.numpy())
    if t % 200 == 0:
        print("loglik = %.4f - p = %.4f - dL/dp = %.4f"%(nll, p, grad))
    p.data -= learning_rate * grad
print('Final probability p =', p.data[0])
bias = torch.abs(p - torch.mean(sample))
diff = torch.pow(p-x, 2)
variance = torch.sum(diff)
print(f"Variance: {variance}")
print(f"Bias: {bias}")
```

### Full estimator density

In general, computing the mean and variance of the estimator is insufficient to characterize the underlying probability density of $\hat{p}$, except if we knew that $\hat{p}$ were normally distributed. This is where the [*central limit theorem*](http://mathworld.wolfram.com/CentralLimitTheorem.html). Indeed, the form of the estimator, implies that $\hat{p}$ is normally distributed, but only *asymptotically*, which doesn't quantify how many samples $n$ we need. Unfortunately, in the real world, each sample may be precious. Hence, to write out the full density for $\hat{p}$, we first have to ask what is the probability that the estimator will equal a specific value such as

$$
\begin{equation}
\hat{p} = \frac{1}{n}\sum_{i=1}^n x_i  = 0
\end{equation}
$$

This can only happen when $x_i=0$,  $\forall i$. The corresponding probability can be computed from the density

$$
\begin{equation}
f(\mathbf{x},p)= \prod_{i=1}^n \left(p^{x_i} (1-p)^{1-x_i}  \right)
\end{equation}
$$

$$
\begin{equation}
f\left(\sum_{i=1}^n x_i  = 0,p\right)= \left(1-p\right)^n
\end{equation}
$$

Likewise, if $\lbrace x_i \rbrace$ has one $i^{th}$ value equal to one, then

$$
\begin{equation}
f\left(\sum_{i=1}^n x_i  = 1,p\right)= n p \prod_{i=1}^{n-1} \left(1-p\right)
\end{equation}
$$

where the $n$ comes from the $n$ ways to pick one value equal to one from the $n$ elements $x_i$. Continuing this way, we can construct the entire density as

$$
\begin{equation}
f\left(\sum_{i=1}^n x_i  = k,p\right)= \binom{n}{k} p^k  (1-p)^{n-k}
\end{equation}
$$

where the term on the left is the binomial coefficient of $n$ things taken $k$ at a time. This is the binomial distribution and it's not the density for $\hat{p}$, but rather for $n\hat{p}$. We'll leave this as-is because it's easier to work with below. We just have to remember to keep track of the $n$ factor.

## Maximum Likelihood implementation

Maximum Likelihood Estimate (MLE) allows to perform typical statistical pattern classification tasks. In the cases where **probabilistic models and parameters are known**, the design of a Bayes' classifier is rather easy. However, in real applications, we are rarely given this information and this is where the MLE comes into play.

MLE still **requires partial knowledge** about the problem. We have to assume that the **model of the class conditional densities is known** (usually Gaussian distributions). Hence, Using MLE, we want to estimate the values of the parameters of a given distribution for the class-conditional densities, for example, the *mean* and *variance* assuming that the class-conditional densities are *normal*  distributed (Gaussian) with

$$
\begin{equation}
p(\pmb x   \mid   y_i) \sim N(\mu, \sigma^2)
\end{equation}
$$

### Parameters known

Imagine that we want to classify data consisting of two-dimensional patterns, $\pmb{x} = [x_1, x_2] \in \mathbb{R}^{2}$ that could belong to 1 out of 3 classes $y_1,y_2,y_3$.

Let's assume the following information about the model where we use continuous univariate normal (Gaussian) model for the class-conditional densities

$$
\begin{equation}
p(\pmb x \mid y_j) \sim N(\pmb \mu \mid \Sigma) = \frac{1}{(2\pi)^{d/2}   \mid \Sigma|^{1/2}} exp \bigg[ -\frac{1}{2}(\pmb x - \pmb \mu)^t \Sigma^{-1}(\pmb x - \pmb \mu) \bigg]
\end{equation}
$$

Furthermore, we consider for this first problem that we know the distributions of the classes, ie. their mean and covariances.

$$
\begin{equation}
p([x_1, x_2]^t \mid y_1) ∼ N([0,0],3I), \\
p([x_1, x_2]^t \mid y_2) ∼ N([9,0],3I), \\
p([x_1, x_2]^t \mid y_3) ∼ N([6,6],4I),
\end{equation}
$$

Therefore, the means of the sample distributions for 2-dimensional features are defined as

$$
\begin{equation}
\pmb{\mu}_{\,1} = \bigg[ 0, 0 \bigg],    \pmb{\mu}_{\,2} = \bigg[ 9,  0 \bigg],    \pmb{\mu}_{\,3} = \bigg[ 6,  6 \bigg]
\end{equation}
$$

The **covariance matrices** for the statistically independent and identically distributed ('i.i.d') features

$$
\begin{array}{ccc}
\Sigma_1 = \bigg[
\begin{array}{cc}
3 & 0\\
0 & 3 \\
\end{array} \bigg],
\Sigma_2 = \bigg[
\begin{array}{cc}
3 & 0\\
0 & 3 \\
\end{array} \bigg],
\Sigma_3 = \bigg[
\begin{array}{cc}
4 & 0\\
0 & 4 \\
\end{array} \bigg]
\end{array}$$

Finally, we consider that all classes have an **equal prior probability**

$$p(y_1)   =    p(y_2)   =   p(y_3)   = \frac{1}{3}$$

***

**Exercise**

  1. Generate some data (samples from the multivariate Gaussians) following classes distributions
  2. Plot the class-dependent data

***

```python
# 7.0 - Generate data with known parameters

nb_patterns = 100
# Generate random patterns for class 1
mu1 = np.array([0,0])
cov1 = np.array([[1,0],[0,1]])
# Generate random patterns for class 2
mu2 = np.array([9,0])
cov2 = np.array([[3,0],[0,3]])
# Generate random patterns for class 3
mu3 = np.array([6,6])
cov3 = np.array([[7,0],[0,7]])

######################
# YOUR CODE GOES HERE
######################

# Prepare concatenated versions of class properties
mu_vals = {};
mu_vals[0] = np.array([mu1]).transpose()
mu_vals[1] = np.array([mu2]).transpose()
mu_vals[2] = np.array([mu3]).transpose()
cov_vals = {}
cov_vals[0] = cov1
cov_vals[1] = cov2
cov_vals[2] = cov3
# Plot the corresponding data
plt.figure(figsize=(12, 8))
plt.scatter(x1samples[:, 0], x1samples[:, 1], s=60, marker='o', c=[0, 0.8, 0], edgecolors='w')
plt.scatter(x2samples[:, 0], x2samples[:, 1], s=60, marker='o', c=[0, 0, 0.8], edgecolors='w')
plt.scatter(x3samples[:, 0], x3samples[:, 1], s=60, marker='o', c=[0.8, 0, 0], edgecolors='w')
h = plot_gaussian_ellipsoid(mu1, cov1, 2, color=[0, 0.8, 0]);
h = plot_gaussian_ellipsoid(mu2, cov2, 2, color=[0, 0, 0.8]);
h = plot_gaussian_ellipsoid(mu3, cov3, 2, color=[0.8, 0, 0]);
plt.title('Training Dataset')
plt.ylabel('x2')
plt.xlabel('x1')
```

Here, our **objective function** is to maximize the discriminant function $g_i(\pmb x)$, which we define as the posterior probability to perform a **minimum-error classification** (Bayes classifier).

$$
\begin{equation}
g_1(\pmb x) = p(y_1 \mid   \pmb{x}), \quad  g_2(\pmb{x}) = p(y_2 \mid   \pmb{x}), \quad  g_3(\pmb{x}) = p(y_3 \mid   \pmb{x})
\end{equation}
$$

So that our decision rule is to choose the class $\omega_i$ for which $g_i(\pmb x)$ is max., where

 $$
 \begin{equation}
 \quad g_i(\pmb{x}) = \pmb{x}^{\,t} \bigg( - \frac{1}{2} \Sigma_i^{-1} \bigg) \pmb{x} + \bigg( \Sigma_i^{-1} \pmb{\mu}_{\,i}\bigg)^t \pmb x + \bigg( -\frac{1}{2} \pmb{\mu}_{\,i}^{\,t}  \Sigma_{i}^{-1} \pmb{\mu}_{\,i} -\frac{1}{2} ln(\left|\Sigma_i\right|)\bigg)
 \end{equation}
 $$

***

**Exercise**

  1. Implement the discriminant function
  2. Implement the decision rule (classifier)
  3. Classify the data generated in the previous exercise
  4. Plot the confusion matrix
  5. Calculate the empirical error

***

```python
import operator

def discriminant_function(x_vec, mu_vec, cov_mat):
    # Calculates the value of the discriminant function for a dx1 dimensional
    # sample given the covariance matrix and mean vector.
    #
    #   x_vec: A dx1 dimensional numpy array representing the sample.
    #   cov_mat: numpy array of the covariance matrix.
    #   mu_vec: dx1 dimensional numpy array of the sample mean.
    #
    #   Returns a float value g as result of the discriminant function.

    ######################
    # YOUR CODE GOES HERE
    ######################

    return float(g)

def classify_data(x_vec, g, mu_vecs, cov_mats):
    #    Classifies an input sample into 1 out of 3 classes determined by
    #    maximizing the discriminant function g_i().

    #    Keyword arguments:
    #        x_vec: A dx1 dimensional numpy array representing the sample.
    #        g: The discriminant function.
    #        mu_vecs: A list of mean vectors as input for g.
    #        cov_mats: A list of covariance matrices as input for g.
    #
    #    Returns the max probability and class id.

    ######################
    # YOUR CODE GOES HERE
    ######################

    return maxVal, maxId

x1classes = np.zeros((nb_patterns, 1))
conf_matrix = np.zeros((3, 3))
for i in range(nb_patterns):
    x, g = classify_data(x1samples[i, :], discriminant_function, mu_vals, cov_vals);
    x1classes[i] = g;
    conf_matrix[0, g] = conf_matrix[0, g] + 1;
x2classes = np.zeros((nb_patterns, 1))
for i in range(nb_patterns):
    x, g = classify_data(x2samples[i, :], discriminant_function, mu_vals, cov_vals);
    x2classes[i] = g;
    conf_matrix[1, g] = conf_matrix[1, g] + 1;
x3classes = np.zeros((nb_patterns, 1))
for i in range(nb_patterns):
    x, g = classify_data(x3samples[i, :], discriminant_function, mu_vals, cov_vals);
    x3classes[i] = g;
    conf_matrix[2, g] = conf_matrix[2, g] + 1;

print('%16s \t %s \t %s \t %s\n' % (' ', 'class 1', 'class 2', 'class 3'));
print('%16s \t %f \t %f \t %f\n' % ('class 1', conf_matrix[0, 0], conf_matrix[0, 1], conf_matrix[0, 2]));
print('%16s \t %f \t %f \t %f\n' % ('class 2', conf_matrix[1, 0], conf_matrix[1, 1], conf_matrix[1, 2]));
print('%16s \t %f \t %f \t %f\n' % ('class 3', conf_matrix[2, 0], conf_matrix[2, 1], conf_matrix[2, 2]));
```

### Unknown parameters

In contrast to the previous case, let us assume that we only know the number of parameters for the class conditional densities $p (\pmb x \mid  y_i)$, and we want to use a Maximum Likelihood Estimation (MLE) to estimate the quantities of these parameters from the training data.

Given the information about the our model (the data is normal distributed) the 2 parameters to be estimated for each class are $\pmb \mu_i$ and $\pmb \Sigma_i$, which are summarized by the parameter vector

$$
\begin{equation}
\pmb \theta_i = \bigg[ \begin{array}{c}
\ \theta_{i1} \\
\ \theta_{i2} \\
\end{array} \bigg]=
\bigg[ \begin{array}{c}
\pmb \mu_i \\
\pmb \Sigma_i \\
\end{array} \bigg]
\end{equation}
$$

For the Maximum Likelihood Estimate (MLE), we assume that we have a set of samples $D = \left\{ \pmb x_1, \pmb x_2,..., \pmb x_n \right\} $ that are *i.i.d.* (independent and identically distributed, drawn with probability $p(\pmb x \mid   y_i, \pmb \theta_i) )$.  Thus, we can **work with each class separately** and omit the class labels, so that we write the probability density as $p(\pmb x \mid \pmb \theta)$

**Likelihood of $ \pmb \theta $**

Thus, the probability of observing $D = \left\{ \pmb x_1, \pmb x_2,..., \pmb x_n \right\} $ is

$$
\begin{equation}
p(D  \mid    \pmb \theta ) = p(\pmb x_1   \mid   \pmb \theta )  \cdot   p(\pmb x_2   \mid  \pmb \theta )   \cdot  ...    p(\pmb x_n   \mid   \pmb \theta ) = \prod_{k=1}^{n}   p(\pmb x_k \pmb   \mid   \pmb \theta  )
\end{equation}
$$

Where $p(D  \mid    \pmb  \theta )$ is also called the ***likelihood of $\pmb\ \theta$***

We know that $p([x_1,x_2]^t)  ∼   N(\pmb \mu,\pmb \Sigma) $ (remember that we dropped the class labels, since we are working with every class separately). And the mutlivariate normal density is given as

$$
\begin{equation}
\quad \quad p(\pmb x) = \frac{1}{(2\pi)^{d/2}   |\Sigma|^{1/2}} exp \bigg[ -\frac{1}{2}(\pmb x - \pmb \mu)^t \Sigma^{-1}(\pmb x - \pmb \mu) \bigg]
\end{equation}
$$

Therefore, we obtain

$$
\begin{equation}
p(D  \mid    \pmb \theta ) = \prod_{k=1}^{n}   p(\pmb x_k \pmb   \mid   \pmb \theta  ) =  \prod_{k=1}^{n}   \frac{1}{(2\pi)^{d/2}   |\Sigma|^{1/2}} exp \bigg[ -\frac{1}{2}(\pmb x - \pmb \mu)^t \Sigma^{-1}(\pmb x - \pmb \mu) \bigg]
\end{equation}
$$
and the log of the multivariate density

$$
\begin{equation}
l(\pmb\theta) =  \sum\limits_{k=1}^{n} - \frac{1}{2}(\pmb x - \pmb \mu)^t \pmb \Sigma^{-1}   (\pmb x - \pmb \mu) - \frac{d}{2}   ln   2\pi - \frac{1}{2}  ln   |\pmb\Sigma|
\end{equation}
$$

In order to obtain the MLE $\boldsymbol{\hat{\theta}}$, we maximize $l (\pmb  \theta)$, which can be done via differentiation

$$
\begin{equation}
\nabla_{\pmb \theta} \equiv \begin{bmatrix}
\frac{\partial   }{\partial   \theta_1} \\
\frac{\partial   }{\partial   \theta_2}
\end{bmatrix} = \begin{bmatrix}
\frac{\partial   }{\partial   \pmb \mu} \\
\frac{\partial   }{\partial   \pmb \sigma}
\end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
\nabla_{\pmb \theta} l = \sum\limits_{k=1}^n \nabla_{\pmb \theta}  ln  p(\pmb x| \pmb \theta) = 0
\end{equation}
$$

***

**Exercise**
<div markdown = "1">

  1. Perform the differentiation for $\frac{\partial   \mathcal{L}}{\partial   \pmb \mu}$ to obtain the estimator of the mean $\hat{\mu}$
  2. Perform the differentiation for $\frac{\partial   \mathcal{L}}{\partial   \pmb \Sigma}$ to obtain the estimator of the covariance matrix $\hat{\Sigma}$
  3. Implement the two estimators as functions based on a set of data
  4. Apply these estimators (MLE) in order to obtain estimated parameters
  5. Re-compute the classification errors on the previous dataset

***

```python
########################################################
# YOUR CODE GOES HERE (Perform mu estimates)
########################################################

# mu_est_1 = ?
# mu_est_2 = ?
# mu_est_3 = ?

print('%16s \t %s \t %s \t %s \t %s \t %s \t %s' % ('', 'mu1_1   ', 'mu1_2  ', 'mu2_1  ', 'mu2_2  ', 'mu3_1  ', 'mu3_2  '));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('MLE', mu_est_1[0], mu_est_1[1], mu_est_2[0], mu_est_2[1], mu_est_3[0], mu_est_3[1]));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('Truth', mu1[0], mu1[1], mu2[0], mu2[1], mu3[0], mu3[1]));

def mle_covariance(x_samples, mu_est):
    #    Calculates the Maximum Likelihood Estimate for the covariance matrix.
    #
    #    Keyword Arguments:
    #        x_samples: np.array of the samples for 1 class, n x d dimensional
    #        mu_est: np.array of the mean MLE, d x 1 dimensional
    #
    #    Returns the MLE for the covariance matrix as d x d numpy array.

    ######################
    # YOUR CODE GOES HERE
    ######################

    return cov_est_

cov_est_1 = mle_covariance(x1samples, np.array([mu_est_1]).transpose());
cov_est_2 = mle_covariance(x2samples, np.array([mu_est_2]).transpose());
cov_est_3 = mle_covariance(x3samples, np.array([mu_est_3]).transpose());

print('%16s \t %s \t %s \t %s \t %s \t %s \t %s' % ('', 'cov1_1   ', 'cov1_2  ', 'cov2_1  ', 'cov2_2  ', 'cov3_1  ', 'cov3_2  '));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('MLE', cov_est_1[0, 0], cov_est_1[1, 0], cov_est_2[0, 0], cov_est_2[1, 0], cov_est_3[0, 0], cov_est_3[1, 0]));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('Truth', cov1[0, 0], cov1[1, 0], cov2[0, 0], cov2[1, 0], cov3[0, 0], cov3[1, 0]));

mu_estimates = {};
mu_estimates[0] = np.array([mu_est_1]).transpose()
mu_estimates[1] = np.array([mu_est_2]).transpose()
mu_estimates[2] = np.array([mu_est_3]).transpose()
cov_estimates = {};
cov_estimates[0] = cov_est_1;
cov_estimates[1] = cov_est_2;
cov_estimates[2] = cov_est_3;

# Plot the corresponding data
plt.figure(figsize=(12, 8))
plt.scatter(x1samples[:,0], x1samples[:,1], s=40, marker='o', c=[0, 0.8, 0], edgecolors='w');
plt.scatter(x2samples[:,0], x2samples[:,1], s=40, marker='s', c=[0, 0, 0.8], edgecolors='w');
plt.scatter(x3samples[:,0], x3samples[:,1], s=40, marker='^', c=[0.8, 0, 0], edgecolors='w');
h = plot_gaussian_ellipsoid(mu1, cov1, 2, color=[0.2, 0.6, 0.2]);
h = plot_gaussian_ellipsoid(mu2, cov2, 2, color=[0.2, 0.2, 0.6]);
h = plot_gaussian_ellipsoid(mu3, cov3, 2, color=[0.6, 0.2, 0.2]);
h = plot_gaussian_ellipsoid(mu_est_1, cov_est_1, 2, color=[0.1, 0.99, 0.1]);
h = plot_gaussian_ellipsoid(mu_est_2, cov_est_2, 2, color=[0.1, 0.1, 0.99]);
h = plot_gaussian_ellipsoid(mu_est_3, cov_est_3, 2, color=[0.99, 0.1, 0.1]);
plt.title('Comparing estimated MLE Gaussians');
```

## Audio source separation

The maximum likelihood estimator (MLE) is widely used in practical signal modeling and we can show that the MLE is equivalent to the least squares estimator for a wide class of problems, including well resolved sinusoids in white noise. We are going to consider a model consisting of a complex sinusoid in additive white (complex) noise:

$$
\displaystyle x(n) = {\cal A}e^{j\omega_{0} n} + v(n)
$$

Here, $ {\cal A}= A e^{j\phi} $ is the complex amplitude of the sinusoid, and $ v(n) $ is white noise that we assume to be Gaussian distributed with zero mean. Hence, we assume that its probability density function is given by

$$
\displaystyle p_{v}(\nu) = \frac{1}{\pi \sigma_{v}^2} e^{-\frac{\vert\nu\vert^2}{\sigma_{v}^2}}.
$$

We express the zero-mean Gaussian assumption by writing

$$
\displaystyle v(n) \sim {\cal N}(0,\sigma_{v}^2)
$$

The parameter  $ \sigma_{v}^2 $ is the *variance* of the random process $ v(n) $ , and $ \sigma_{v} $ is its standard deviation. It turns out that when Gaussian random variables $ v(n) $ are uncorrelated (i.e., when $ v(n) $ is white noise), they are also independent. This means that the probability of observing particular values of $ v(n) $ and $ v(m) $ is given by the product of their respective probabilities. We will now use this fact to compute an explicit probability for observing any data sequence $ x(n) $

Since the sinusoidal part of our signal model,  $ {\cal A}e^{j\omega_{0}n}$ , is deterministic; i.e., it does not including any random components; it may be treated as the time-varying mean of a Gaussian random process $ x(n) $ . That is, our signal model can be rewritten as

$$
\displaystyle x(n) \sim {\cal N}({\cal A}e^{j\omega_{0} n},\sigma_{v}^2)
$$

and the probability density function for the whole set of observations $ x(n) $ ,  $ n=0,1,2,\ldots,N-1 $ is given by

$$
\displaystyle p(x) = p[x(0)] p[x(1)]\cdots p[x(N-1)] = \left(\frac{1}{\pi \sigma_v^2}\right)^N e^{-\frac{1}{\sigma_v^2}\sum_{n=0}^{N-1} \left\vert x(n) - {\cal A}e^{j\omega_0 n}\right\vert^2}
$$

Thus, given the noise variance  $ \sigma_v^2 $ and the three sinusoidal parameters $ A,\phi,\omega_0 $ (remember that $ {\cal A}= A e^{j\phi} $ ), we can compute the relative probability of any observed data samples $ x(n) $ .

We can generalize this approach in order to perform a complete blind audio source separation algorithm, such as detailed in the following paper

Févotte, C., & Cardoso, J. F. "Maximum likelihood approach for blind audio source separation using time-frequency Gaussian source models". *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics*, 2005. (pp. 78-81). IEEE.

You can try to implement this by relying on the [full paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.640.6981&rep=rep1&type=pdf) that details this method.

***

**Exercise**
<div markdown = "1">

  1. Implement the single sinusoid extraction
  2. Apply this approach to multiple sinusoids
  3. Follow the paper to implement blind source separation

***

```python

```


# Lecture 10: Expectation Continued

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## A Proof of Linearity (discrete case)

Let $T = X + Y$, and show that $\mathbb{E}(T) = \mathbb{E}(X) + \mathbb{E}(Y)$.

We will also show that $\mathbb{E}(cX) = c \mathbb{E}(X)$.

In general, we'd like to be in a position where

\begin{align}
  \sum_{t} t P(T=t) \stackrel{?}{=} \sum_{x} x P(X=x) + \sum_{y} y P(Y=y)
\end{align}

so, let's try attacking this from the l.h.s.

Considering the image above of a discrete r.v. in Pebble World, note that

\begin{align}
  \mathbb{E}(X) &= \sum_{x} x P(X=x) & &\text{grouping the pebbles per X value; weighted average} \\
  &= \sum_{s}X(s)P(\{s\}) & &\text{ungrouped; sum each pebble separately} \\
  \\
  \\
  \Rightarrow \mathbb{E}(T) &= \sum_{s} (X+Y)(s)P(\{s\}) \\
  &= \sum_{s}X(s)P(\{s\}) + \sum_{s}Y(s)P(\{s\}) \\
  &= \sum_{x} x P(X=x) + \sum_{y} y P(Y=y) \\
  &= \mathbb{E}(X) + \mathbb{E}(Y) ~~~~ \blacksquare \\
  \\
  \\
  \Rightarrow \mathbb{E}(cX) &= \sum_{x} cx P(X=x) \\
  &= c \sum_{x} x P(X=x) \\
  &= c \mathbb{E}(X) ~~~~ \blacksquare
\end{align}

----

## Negative Binomial Distribution

### Description

A misnomer: this distribution is actually non-negative, and neither is it binomial.

The Negative Binomial is a generalization of the Geometric distribution, where we have a series of independent $Bern(p)$ trials and we want to know # failures before the rth success.

We can codify this using a bit string:

\begin{align}
  & \text{1000100100001001} & \text{0 denotes failure, 1 denotes success} & \\
  & r = 5 \\
  & n = 11 & \text{failures}
\end{align}

Note that the very last bit position is, of course, a success.

Note also that we can permutate the preceding $r-1$ successes amongst the $n+r-1$ slots that come before that final rth success.

### Notation

$X \sim \operatorname{NB}(r,p)$

### Parameters

* $r$ - the total number of successes before we stop counting
* $p$ - probability of success

### Probability mass function

\begin{align}
  P(X=n) &= \binom{n+r-1}{r-1}  p^r (1-p)^n & &\text{for } n = 0,1,2,\dots\\
  &= \binom{n+r-1}{n}  p^r (1-p)^n & &\text{or conversely}\\
\end{align}

### Expected value

Let $X_j$ be the # failures before the $(j-1)^{\text{st}}$ and $j^{\text{th}}$ success. Then we could write

\begin{align}
  \mathbb{E}(X) &= \mathbb{E}(X_1 + X_2 + \dots + X_r) \\
                &= \mathbb{E}(X_1) + \mathbb{E}(X_2) + \dots + \mathbb{E}(X_r) & &\text{by Linearity} \\
                &= r \mathbb{E}(X_1) & &\text{by symmetry} \\
                &= r \frac{q}{p} ~~~~ \blacksquare
\end{align}

----

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
from scipy.stats import nbinom

%matplotlib inline

plt.xkcd()
_, ax = plt.subplots(figsize=(12,8))

# seme Negative Binomial parameters
r_values = [1, 2, 4, 8]
p_values = [0.25]*len(r_values)
#p_values = [0.25, 0.24, 0.23, 0.22]
params = list(zip(r_values, p_values))

# colorblind-safe, divergent color scheme
colors = ['#018571', '#80cdc1', '#dfc27d', '#a6611a']

for i,(r,p) in enumerate(params):
    x = np.arange(nbinom.ppf(0.01, r, p), nbinom.ppf(0.99, r, p))
    pmf = nbinom.pmf(x, r, p)
    ax.plot(x, pmf, 'o', color=colors[i], ms=8, label='r={}, p={}'.format(r,p))
    ax.vlines(x, 0, pmf, lw=2, color=colors[i], alpha=0.3)

# legend styling
legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

# y-axis
ax.set_ylim([0.0, 0.251])
ax.set_ylabel(r'$P(X=n)$')

# x-axis
ax.set_xlim([0, 55])
ax.set_xlabel(r'total # of failures $n$ before seeing $r^{th}$ success')

# x-axis tick formatting
majorLocator = MultipleLocator(5)
majorFormatter = FormatStrFormatter('%d')
minorLocator = MultipleLocator(1)
ax.xaxis.set_major_locator(majorLocator)
ax.xaxis.set_major_formatter(majorFormatter)
ax.xaxis.set_minor_locator(minorLocator)

ax.grid(color='grey', linestyle='-', linewidth=0.3)

plt.suptitle(r'Negative Binomial PMF: $P(X=n) = \binom{n+r-1}{r-1}  p^r (1-p)^n$')

plt.show()
```

## Revisting the Geometric: the First Success Distribution

$X \sim \operatorname{FS}(p)$ is the geometric distribution that counts the trials until first success, *including that first success*.

Let $Y = X - 1$.

Then $Y \sim \operatorname{Geom}(p)$

Expected value of $\operatorname{FS}(p)$ is

\begin{align}
  \mathbb{E}(X) &= E(Y) + 1 \\
                &= \frac{q}{p} + 1 \\
                &= \boxed{\frac{1}{p}}
\end{align}

----

## Putnam Problem

Consider a random permutation of $1, 2, 3, \dots , n$, where $n \ge 2$.

Find expected # local maxima. For example, given the permuation $\boxed{3} ~~ 2 ~~ 1 ~~ 4 ~~ \boxed{7} ~~ 5 ~~ \boxed{6}$ we have 3 local maxima:

- $\boxed{3} \gt 2$
- $4 \lt \boxed{7} \gt 5$
- $ 5 \lt \boxed{6}$

Now, there are 2 kinds of cases we need to consider:

- non-edge case: $4 ~~ \boxed{7} ~~ 5$ has probability of $\frac{1}{3}$ that the largest number is in the middle position
- edge case: in both left-edge $\boxed{3} ~~ 2$ and right-edge $5 ~~ \boxed{6}$, the probability that the larger number is in the right position is $\frac{1}{2}$

Let $I_j$ be the indicator r.v. of position $j$ having a local maximum, $1 \le j \le n$.

Using Linearity, we can say that the expected number of local maxima is given by

\begin{align}
  \mathbb{E}(I_j) &= \mathbb{E}(I_1 + I_2 + \dots + I_n) \\
                  &= \mathbb{E}(I_1) + \mathbb{E}(I_2) + \dots + \mathbb{E}(I_n) & &\text{by Linearity} \\
                  &= (n-2) \frac{1}{3} + 2 \frac{1}{2} \\
                  &= \boxed{\frac{n+1}{3}}
\end{align}

Idiot-checking this, we have:

\begin{align}
  \mathbb{E}(I_{n=2}) &= \frac{2+1}{3} & &\text{... case where } n=2 \\
  &= 1 \\
  \\
  \\
  \mathbb{E}(I_{n=\infty}) &= \frac{\infty+1}{3} & &\text{... case where } n= \infty \\
  &= \infty \\
\end{align}

----

## St. Petersburg Paradox

Consider a game of chance involving a fair coin. We will flip the coin until the very first heads shows (hypergeometric distribution).

- If heads shows on the very first flip, you get $\$2$.
- If the first heads shows on the second flip, you get $\$4$.
- If the first heads shows on the third flip, you get $\$8$.

So you will get $\$2^n$ if the first heads shows up on the nth trial, including the heads flip.

_How much would you be willing to pay to play this game?_

Let's tackle this by thinking about the expected amount of $\$\$\$$ we stand to make.

Given $Y = 2^n$, find $\mathbb{E}(Y)$:

\begin{align}
  \mathbb{E}(Y) &= \sum_{k=1}^\infty 2^k \frac{1}{2^{k-1}} ~ \frac{1}{2}\\
                &= \sum_{k=1}^\infty 2^k \frac{1}{2^k}\\
                &= \sum_{k=1}^\infty 1\\
  \\
  \\
  \mathbb{E}(Y_{k=40}) &= \sum_{k=1}^{40} 1 \\
                       &= 40
\end{align}

So, the "paradox" here is that even if we capped the payout to $2^{40} \approx \$1000000000$, Linearity shows us we would only pay $40. It is very hard to grasp this, but the truth is that if you were offered this game at any price, you should take it.

----

View [Lecture 10: Expectation Continued | Statistics 110](http://bit.ly/2vXxPsj) on YouTube.