# Minimum Energy Double Pendulum Swing Up

This is a simple demonstration of trajectory optimization using opty. Given two link pendulum attached to a sliding cart that can be actuated by a lateral force, we look for a minimal energy method to swing the pendulum to a vertical position in a specific time interval.

The following diagram shows an N link pendulum.

Our system is a two link version of this.

# Setup

First, import the necessary packages, modules, functions, and classes.

```python
import sympy as sm
import sympy.physics.mechanics as me
import numpy as np
import matplotlib.pyplot as plt
from IPython.core.pylabtools import figsize
from pydy.models import n_link_pendulum_on_cart
from opty.direct_collocation import Problem
from opty.utils import parse_free

import utils
```

    :0: FutureWarning: IPython widgets are experimental and may change in the future.

The following will initialize pretty printing of the symbolics.

```python
me.init_vprinting(use_latex="mathjax")
```

# Model

PyDy includes a canned model for this system. We can create a two link version like so:

```python
pendulum_sys = n_link_pendulum_on_cart(n=2, cart_force=True)
```

    /home/moorepants/anaconda/envs/inverted-pendulum-id/lib/python2.7/site-packages/pydy/system.py:102: PyDyFutureWarning: PyDy System is experimental and may change in the future.
      warnings.warn(msg, PyDyFutureWarning)

The system has six states, one exogenous specified input, six and constants.

```python
num_states = len(pendulum_sys.states)
```

```python
pendulum_sys.states
```

$$\left [ q_{0}, \quad q_{1}, \quad q_{2}, \quad u_{0}, \quad u_{1}, \quad u_{2}\right ]$$

```python
pendulum_sys.specifieds_symbols
```

$$\left\{F\right\}$$

```python
pendulum_sys.constants_symbols
```

$$\left\{g, l_{0}, l_{1}, m_{0}, m_{1}, m_{2}\right\}$$

opty only requires that the equations of motion be provided in implicit form. We will make use of the "full" mass matrix and forcing vector.

```python
M = sm.trigsimp(pendulum_sys.eom_method.mass_matrix_full)
M
```

$$\left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & m_{0} + m_{1} + m_{2} & - l_{0} \left(m_{1} + m_{2}\right) \operatorname{sin}\left(q_{1}\right) & - l_{1} m_{2} \operatorname{sin}\left(q_{2}\right)\\0 & 0 & 0 & - l_{0} \left(m_{1} + m_{2}\right) \operatorname{sin}\left(q_{1}\right) & l_{0}^{2} m_{1} + l_{0}^{2} m_{2} & l_{0} l_{1} m_{2} \operatorname{cos}\left(q_{1} - q_{2}\right)\\0 & 0 & 0 & - l_{1} m_{2} \operatorname{sin}\left(q_{2}\right) & l_{0} l_{1} m_{2} \operatorname{cos}\left(q_{1} - q_{2}\right) & l_{1}^{2} m_{2}\end{matrix}\right]$$

```python
f = sm.trigsimp(pendulum_sys.eom_method.forcing_full)
f
```

$$\left[\begin{matrix}u_{0}\\u_{1}\\u_{2}\\l_{0} m_{1} u^{2}_{1} \operatorname{cos}\left(q_{1}\right) + l_{0} m_{2} u^{2}_{1} \operatorname{cos}\left(q_{1}\right) + l_{1} m_{2} u^{2}_{2} \operatorname{cos}\left(q_{2}\right) + F\\- l_{0} \left(g m_{1} \operatorname{cos}\left(q_{1}\right) + g m_{2} \operatorname{cos}\left(q_{1}\right) + l_{1} m_{2} u^{2}_{2} \operatorname{sin}\left(q_{1} - q_{2}\right)\right)\\l_{1} m_{2} \left(- g \operatorname{cos}\left(q_{2}\right) + l_{0} u^{2}_{1} \operatorname{sin}\left(q_{1} - q_{2}\right)\right)\end{matrix}\right]$$

The implicit expression for the right hand side of the equations of motion can be computed as such:

$$
\mathbf{0} = \mathbf{g}(\dot{\mathbf{x}}, \mathbf{x}, \mathbf{r}, \mathbf{p}, t) = \mathbf{M} \dot{\mathbf{x}} - \mathbf{f}
$$

```python
eom_expr = sm.trigsimp(M * sm.Matrix(pendulum_sys.states).diff() - f)
eom_expr
```

$$\left[\begin{matrix}- u_{0} + \dot{q}_{0}\\- u_{1} + \dot{q}_{1}\\- u_{2} + \dot{q}_{2}\\- l_{0} m_{1} u^{2}_{1} \operatorname{cos}\left(q_{1}\right) - l_{0} m_{2} u^{2}_{1} \operatorname{cos}\left(q_{1}\right) - l_{0} \left(m_{1} + m_{2}\right) \operatorname{sin}\left(q_{1}\right) \dot{u}_{1} - l_{1} m_{2} u^{2}_{2} \operatorname{cos}\left(q_{2}\right) - l_{1} m_{2} \operatorname{sin}\left(q_{2}\right) \dot{u}_{2} + \left(m_{0} + m_{1} + m_{2}\right) \dot{u}_{0} - F\\l_{0} l_{1} m_{2} \operatorname{cos}\left(q_{1} - q_{2}\right) \dot{u}_{2} - l_{0} \left(m_{1} + m_{2}\right) \operatorname{sin}\left(q_{1}\right) \dot{u}_{0} + l_{0} \left(g m_{1} \operatorname{cos}\left(q_{1}\right) + g m_{2} \operatorname{cos}\left(q_{1}\right) + l_{1} m_{2} u^{2}_{2} \operatorname{sin}\left(q_{1} - q_{2}\right)\right) + \left(l_{0}^{2} m_{1} + l_{0}^{2} m_{2}\right) \dot{u}_{1}\\l_{1} m_{2} \left(g \operatorname{cos}\left(q_{2}\right) - l_{0} u^{2}_{1} \operatorname{sin}\left(q_{1} - q_{2}\right) + l_{0} \operatorname{cos}\left(q_{1} - q_{2}\right) \dot{u}_{1} + l_{1} \dot{u}_{2} - \operatorname{sin}\left(q_{2}\right) \dot{u}_{0}\right)\end{matrix}\right]$$

# NLP Problem Setup

Here we set some numerical values for the constant parameters:

```python
pendulum_sys.constants = {p: 9.81 if p.name == 'g' else 1.0 for p in pendulum_sys.constants_symbols}
pendulum_sys.constants
```

$$\left \{ g : 9.81, \quad l_{0} : 1.0, \quad l_{1} : 1.0, \quad m_{0} : 1.0, \quad m_{1} : 1.0, \quad m_{2} : 1.0\right \}$$

We will try to swing the pendulum up in 10 seconds and will discretize time with 500 nodes.

```python
duration = 10.0
num_nodes = 500
interval_value = duration / (num_nodes - 1)
```

## Objective

Our objective is to use minimal energy to swing the pendulum to a vertical stationary state by 10 seconds. One way to approach this is to minimize the square of the applied force:

$$ J(\theta) = h \sum F_i^2 $$

Here we create a function for the objective and its gradient:

```python
def obj(free):
    """Minimize the sum of the squares of the control force."""
    F = free[num_states * num_nodes:]
    return interval_value * np.sum(F**2)

def obj_grad(free):
    grad = np.zeros_like(free)
    grad[num_states * num_nodes:] = 2.0 * interval_value * free[num_states * num_nodes:]
    return grad
```

## Task Constraints

We also need some constraints to specify what state we want the pendulum to be in at the start and end of the task. At the beginning and end we want the pendulum to be stationary, i.e. the velocities should be zero. At the beginning the pendulum should be hanging down, i.e. $q(10)=-\pi/2$ and at the end the pendulum should vertical, i.e. $q(0) = \pi/2$. These constraints are specified by "instance constraints", i.e. constraints specified for a single instance of time.

```python
target_angle = sm.pi / 2
```

```python
q0, q1, q2, u0, u1, u2 = [x.__class__ for x in pendulum_sys.states]
instance_constraints = (q0(0.0),
                        q1(0.0) + target_angle,
                        q1(duration) - target_angle,
                        q2(0.0) + target_angle,
                        q2(duration) - target_angle,
                        u0(0.0),
                        u0(duration),
                        u1(0.0),
                        u1(duration),
                        u2(0.0),
                        u2(duration))

instance_constraints
```

$$\left ( \operatorname{q0}\left(0.0\right), \quad \operatorname{q1}\left(0.0\right) + \frac{\pi}{2}, \quad \operatorname{q1}\left(10.0\right) - \frac{\pi}{2}, \quad \operatorname{q2}\left(0.0\right) + \frac{\pi}{2}, \quad \operatorname{q2}\left(10.0\right) - \frac{\pi}{2}, \quad \operatorname{u0}\left(0.0\right), \quad \operatorname{u0}\left(10.0\right), \quad \operatorname{u1}\left(0.0\right), \quad \operatorname{u1}\left(10.0\right), \quad \operatorname{u2}\left(0.0\right), \quad \operatorname{u2}\left(10.0\right)\right )$$

## Construct the Problem

The Problem object available in opty is now initialized with all of the problem setup information.

```python
prob = Problem(obj, obj_grad, eom_expr, pendulum_sys.states,
               num_nodes, interval_value,
               known_parameter_map=pendulum_sys.constants,
               instance_constraints=instance_constraints)
```

IPOPT options can be added. The following sets the linear solver.

```python
prob.addOption('linear_solver', 'ma57')
```

# Solve the Problem

We then supply a random initial guess.

```python
initial_guess = np.random.random(prob.num_free)
```

Alternatively, you can load a previously found optimal solution.

```python
initial_guess = np.load('dp-solution.npy')
```

The problem can then be solved by calling the `solve()` method with the initial guess. The solution is returned along with some IPOPT info and stats.

```python
solution, info = prob.solve(initial_guess)
```

```python
prob.obj(solution)
```

$$616.919724446$$

Uncomment this if you want to save an optimal solution.

```python
#np.save('dp-solution', solution)
```

# Plot the Results

```python
x, r, p = parse_free(solution, prob.collocator.num_states,
                     prob.collocator.num_unknown_input_trajectories, num_nodes)
```

```python
time = np.linspace(0.0, duration, num=num_nodes)
```

```python
%matplotlib inline
figsize(12, 9)
```

## Trajectories

```python
fig, axes = plt.subplots(5)

axes[0].set_title('State and Control Trajectories')

axes[0].plot(time, r)
axes[0].set_ylabel('Force [N]')

axes[1].plot(time, x[0])
axes[1].set_ylabel('Distance [m]')
axes[1].legend(pendulum_sys.states[0:1])

axes[2].plot(time, np.rad2deg(x[1:3].T))
axes[2].set_ylabel('Angle [deg]')
axes[2].legend(pendulum_sys.states[1:3])

axes[3].plot(time, x[3])
axes[3].set_ylabel('Speed [m/s]')
axes[3].legend(pendulum_sys.states[3:4])

axes[4].plot(time, np.rad2deg(x[4:].T))
axes[4].set_ylabel('Angular Rate [deg/s]')
axes[4].legend(pendulum_sys.states[4:])
axes[4].set_xlabel('Time [S]')
```

## Constraint violations

```python
con_violations = prob.con(solution)
con_nodes = range(2, num_nodes + 1)
N = len(con_nodes)

fig, axes = plt.subplots(5)

axes[0].set_title('Constraint Violations')
axes[0].plot(con_nodes, con_violations[:N])
axes[0].set_ylabel('Distance [m]')

axes[1].plot(con_nodes, con_violations[N:3 * N].reshape(N, 2))
axes[1].set_ylabel('Angle [rad]')

axes[2].plot(con_nodes, con_violations[3 * N:4 * N])
axes[2].set_ylabel('Speed [m/s]')

axes[3].plot(con_nodes, con_violations[4 * N:6 * N].reshape(N, 2))
axes[3].set_ylabel('Angular Rate [rad/s]')
axes[3].set_xlabel('Node Number')

axes[4].plot(con_violations[6 * N:])
axes[4].set_ylabel('Instance')
```

# Motion Visualization

```python
scene = utils.pydy_n_link_pendulum_scene(pendulum_sys)
```

    /home/moorepants/anaconda/envs/inverted-pendulum-id/lib/python2.7/site-packages/pydy/viz/camera.py:73: PyDyUserWarning: Rotation of Perspective Camera does not work properly in the visualiser.
      warnings.warn(msg, PyDyUserWarning)

```python
scene.states_trajectories = x.T
```

```python
scene.frames_per_second = 1.0 / interval_value
```

```python
scene.display_ipython()
```

# Version Information

```python
%install_ext version_information.py
%load_ext version_information
%version_information numpy, sympy, scipy, matplotlib, pydy, opty
```

    Installed version_information.py. To use it, type:
      %load_ext version_information

Software	Version

Python	2.7.10 64bit [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

IPython	3.2.0

OS	Linux 3.13.0 55 generic x86_64 with debian jessie sid

numpy	1.9.2

sympy	0.7.7.dev

scipy	0.15.1

matplotlib	1.4.3

pydy	0.3.0

opty	0.1.0

Wed Jul 08 09:20:11 2015 CDT


```python
def pow(x, n, I, mult):
    # https://sahandsaba.com/five-ways-to-calculate-fibonacci-numbers-with-python-code.html
    """
    Returns x to the power of n. Assumes I to be identity relative to the
    multiplication given by mult, and n to be a positive integer.
    """
    if n == 0:
        return I
    elif n == 1:
        return x
    else:
        y = pow(x, n // 2, I, mult)
        y = mult(y, y)
        if n % 2:
            y = mult(x, y)
        return y

```

```python
def identity_matrix(n):
    """Returns the n by n identity matrix."""
    r = list(range(n))
    return [[1 if i == j else 0 for i in r] for j in r]

```

```python
def matrix_multiply(A, B):
    BT = list(zip(*B))
    return [[sum(a * b
                 for a, b in zip(row_a, col_b))
            for col_b in BT]
            for row_a in A]
```

```python
def fib(n):
    F = pow([[1, 1], [1, 0]], n, identity_matrix(2), matrix_multiply)
    return F[0][1]

```

```python
# github.com/nsc9's code begins here
from sympy import *
from sympy.abc import x,y,z
fib2 = []
poly1 = []
poly2 = []
poly3 = []
poly4 = []
poly5 = []
poly6 = []
poly7 = []
for i in range(0,2000):
    #print(-fib(i),"---------------------------------------------------------------------------------------------------------",fib(i))
    fib2.append(fib(i))
    poly1.append(x+fib2[i])
    poly2.append(fib2[i]*x+fib2[i])
    poly3.append(fib2[i]*x**2+fib2[i]*x+fib2[i])
    poly4.append(fib2[i]*x**3+fib2[i]*x**2+fib2[i]*x+fib2[i])
    poly5.append(fib2[i]*x**4+fib2[i]*x**3+fib2[i]*x**2+fib2[i]*x+fib2[i])
    poly6.append(fib2[i]*x**5+fib2[i]*x**4+fib2[i]*x**3+fib2[i]*x**2+fib2[i]*x+fib2[i])
    poly7.append(fib2[i]*x**8+fib2[i]*x**7+fib2[i]*x**6+fib2[i]*x**5+fib2[i]*x**4+fib2[i]*x**3+fib2[i]*x**2+fib2[i]*x**1+fib2[i])
print(poly7[0:10])
eqs = []
for i in range(1,100,2):
    #print(i)
    #plot(poly3[i]/10)
    #print(poly3[i])
    eqs.append(poly7[i])
    #print("   ")
    #'--Fibonacci numbers------------------------------------------------------------',fib2,'--x + Fibnumbr------------------------------------------------------------ ',poly1,'--Fibnumr*x+Fibnumbr-----------------------------------------------------------------  ',poly2,'--Fibnumbr*x**2+Fibnumbrx+Fibnumbr-----------------------------------------------------------------  ',poly3
```

    [0, x**8 + x**7 + x**6 + x**5 + x**4 + x**3 + x**2 + x + 1, x**8 + x**7 + x**6 + x**5 + x**4 + x**3 + x**2 + x + 1, 2*x**8 + 2*x**7 + 2*x**6 + 2*x**5 + 2*x**4 + 2*x**3 + 2*x**2 + 2*x + 2, 3*x**8 + 3*x**7 + 3*x**6 + 3*x**5 + 3*x**4 + 3*x**3 + 3*x**2 + 3*x + 3, 5*x**8 + 5*x**7 + 5*x**6 + 5*x**5 + 5*x**4 + 5*x**3 + 5*x**2 + 5*x + 5, 8*x**8 + 8*x**7 + 8*x**6 + 8*x**5 + 8*x**4 + 8*x**3 + 8*x**2 + 8*x + 8, 13*x**8 + 13*x**7 + 13*x**6 + 13*x**5 + 13*x**4 + 13*x**3 + 13*x**2 + 13*x + 13, 21*x**8 + 21*x**7 + 21*x**6 + 21*x**5 + 21*x**4 + 21*x**3 + 21*x**2 + 21*x + 21, 34*x**8 + 34*x**7 + 34*x**6 + 34*x**5 + 34*x**4 + 34*x**3 + 34*x**2 + 34*x + 34]

```python
diff(x**x)
```

$\displaystyle x^{x} \left(\log{\left(x \right)} + 1\right)$

```python
poly7[11]
```

$\displaystyle 89 x^{8} + 89 x^{7} + 89 x^{6} + 89 x^{5} + 89 x^{4} + 89 x^{3} + 89 x^{2} + 89 x + 89$

```python
plot(eqs[0],eqs[2],eqs[3],eqs[4],eqs[5],eqs[6],eqs[7],eqs[8],eqs[9],E**x)
```

```python
plot(eqs[6]*0.85,E**x,legend = True)
```

```python
plot(eqs[6]*0.851,E**x,legend = True, xlim = (0,11),size = (13,4))
```

```python
plot(eqs[0],eqs[1],eqs[2],eqs[3],eqs[4],eqs[5],E**x,x**x,legend = True,size = (23,4),yscale='log',ylim=(0.1,10**11),xlim = (-10,10))
```

```python
from sympy import Sum
from sympy.abc import k,m
print(Sum(eqs[0], (x, 1, 10)).doit(), "=")
Sum(eqs[0], (x, 1, 10))

```

    188039796 =

$\displaystyle \sum_{x=1}^{10} \left(x^{8} + x^{7} + x^{6} + x^{5} + x^{4} + x^{3} + x^{2} + x + 1\right)$

```python
print(Sum(eqs[1], (x, 1, 10)).doit(), "=")
Sum(eqs[1], (x, 1, 10))
```

    376079592 =

$\displaystyle \sum_{x=1}^{10} \left(2 x^{8} + 2 x^{7} + 2 x^{6} + 2 x^{5} + 2 x^{4} + 2 x^{3} + 2 x^{2} + 2 x + 2\right)$

```python
print(Sum(eqs[2], (x, 1, 10)).doit(), "=")
Sum(eqs[2], (x, 1, 10))
```

    940198980 =

$\displaystyle \sum_{x=1}^{10} \left(5 x^{8} + 5 x^{7} + 5 x^{6} + 5 x^{5} + 5 x^{4} + 5 x^{3} + 5 x^{2} + 5 x + 5\right)$

```python
print(Sum(eqs[3], (x, 1, 10)).doit(), "=")
Sum(eqs[3], (x, 1, 10))
```

    2444517348 =

$\displaystyle \sum_{x=1}^{10} \left(13 x^{8} + 13 x^{7} + 13 x^{6} + 13 x^{5} + 13 x^{4} + 13 x^{3} + 13 x^{2} + 13 x + 13\right)$

```python
print(Sum(eqs[4], (x, 1, 10)).doit(), "=")
Sum(eqs[4], (x, 1, 10))
```

    6393353064 =

$\displaystyle \sum_{x=1}^{10} \left(34 x^{8} + 34 x^{7} + 34 x^{6} + 34 x^{5} + 34 x^{4} + 34 x^{3} + 34 x^{2} + 34 x + 34\right)$

```python
6393353064/2444517348
```

    2.6153846153846154

```python
2444517348/940198980
```

    2.6

```python
940198980/376079592
```

    2.5

```python
print(Sum(eqs[5], (x, 1, 10)).doit(), "=")
Sum(eqs[5], (x, 1, 10))
```

    16735541844 =

$\displaystyle \sum_{x=1}^{10} \left(89 x^{8} + 89 x^{7} + 89 x^{6} + 89 x^{5} + 89 x^{4} + 89 x^{3} + 89 x^{2} + 89 x + 89\right)$

```python
16735541844/6393353064
```

    2.6176470588235294

```python
Sum(eqs[5], (x, 1, 10)).doit()/Sum(eqs[5-1], (x, 1, 10)).doit()
```

$\displaystyle \frac{89}{34}$

```python
for i in range(1,25):
    print((Sum(eqs[i], (x, 1, i)).doit()/Sum(eqs[i-1], (x, 1, i)).doit()).evalf())
```

    2.00000000000000
    2.50000000000000
    2.60000000000000
    2.61538461538462
    2.61764705882353
    2.61797752808989
    2.61802575107296
    2.61803278688525
    2.61803381340013
    2.61803396316671
    2.61803398501736
    2.61803398820533
    2.61803398867044
    2.61803398873830
    2.61803398874820
    2.61803398874965
    2.61803398874986
    2.61803398874989
    2.61803398874989
    2.61803398874989
    2.61803398874989
    2.61803398874989
    2.61803398874989
    2.61803398874989

```python
for i in range(1,50):
    print(i,"   ",fib2[i+1]/fib2[i])
```

    1     1.0
    2     2.0
    3     1.5
    4     1.6666666666666667
    5     1.6
    6     1.625
    7     1.6153846153846154
    8     1.619047619047619
    9     1.6176470588235294
    10     1.6181818181818182
    11     1.6179775280898876
    12     1.6180555555555556
    13     1.6180257510729614
    14     1.6180371352785146
    15     1.618032786885246
    16     1.618034447821682
    17     1.6180338134001253
    18     1.618034055727554
    19     1.6180339631667064
    20     1.6180339985218033
    21     1.618033985017358
    22     1.6180339901755971
    23     1.618033988205325
    24     1.618033988957902
    25     1.6180339886704431
    26     1.6180339887802426
    27     1.618033988738303
    28     1.6180339887543225
    29     1.6180339887482036
    30     1.6180339887505408
    31     1.6180339887496482
    32     1.618033988749989
    33     1.618033988749859
    34     1.6180339887499087
    35     1.6180339887498896
    36     1.618033988749897
    37     1.618033988749894
    38     1.6180339887498951
    39     1.6180339887498947
    40     1.618033988749895
    41     1.618033988749895
    42     1.618033988749895
    43     1.618033988749895
    44     1.618033988749895
    45     1.618033988749895
    46     1.618033988749895
    47     1.618033988749895
    48     1.618033988749895
    49     1.618033988749895

```python
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
fig, ax = plt.subplots()  # Create a figure containing a single axes.

digsumfibs = []
def sum_digits(n):
    #https://stackoverflow.com/questions/14939953/sum-the-digits-of-a-number
    s = 0
    while n:
        s += n % 10
        n //= 10
    return s

for i in range(0,26):
    print("The digit sum of fib number",fib2[i], "=",sum_digits(fib2[i]))
    digsumfibs.append(sum_digits(fib2[i]))
ax.plot([digsumfibs[0],digsumfibs[1],digsumfibs[2],digsumfibs[3],digsumfibs[4],digsumfibs[5],digsumfibs[6],digsumfibs[7],
         digsumfibs[8],digsumfibs[9],digsumfibs[10],digsumfibs[11],digsumfibs[12],digsumfibs[13],digsumfibs[14],digsumfibs[15],
         digsumfibs[16],digsumfibs[17],digsumfibs[18],digsumfibs[19],digsumfibs[20],digsumfibs[21],digsumfibs[22],digsumfibs[23],digsumfibs[24],digsumfibs[25]])
```

```python

```


# Quantum Fourier Transforms

The **"QFT (Quantum Fourier Transform)"** quantum kata is a series of exercises designed
to teach you the basics of quantum Fourier transform (QFT). It covers implementing QFT and using
it to perform simple state transformations.

Each task is wrapped in one operation preceded by the description of the task.
Your goal is to fill in the blank (marked with the `// ...` comments)
with some Q# code that solves the task. To verify your answer, run the cell using Ctrl+Enter (⌘+Enter on macOS).

Within each section, tasks are given in approximate order of increasing difficulty;
harder ones are marked with asterisks.

## Part I. Implementing Quantum Fourier Transform

This sequence of tasks uses the implementation of QFT described in Nielsen & Chuang.
All numbers in this kata use big endian encoding: most significant bit of the number
 is stored in the first (leftmost) bit/qubit.

### Task 1.1. 1-qubit QFT

**Input:**

  A qubit in state $|\psi\rangle = x_0 |0\rangle + x_1 |1\rangle$.

**Goal:**

Apply QFT to this qubit, i.e., transform it to a state
$\frac{1}{\sqrt{2}} \big((x_0 + x_1) |0\rangle + (x_0 - x_1) |1\rangle\big)$.

In other words, transform a basis state $|j\rangle$ into a state $\frac{1}{\sqrt{2}} \big(|0\rangle + e^{2\pi i \cdot \frac{j}{2}}|1\rangle\big)$ .

```qsharp
%kata T11_OneQubitQFT_Test

operation OneQubitQFT (q : Qubit) : Unit is Adj+Ctl {
    // ...
}
```

### Task 1.2. Rotation gate

**Inputs:**

  1. A qubit in state $|\psi\rangle = \alpha |0\rangle + \beta |1\rangle$.

  2. An integer k $\geq$ 0.

**Goal:**

Change the state of the qubit to $\alpha |0\rangle + \beta \cdot e^{\frac{2\pi i}{2^{k}}} |1\rangle$.

> Be careful about not introducing an extra global phase!
This is going to be important in the later tasks.

```qsharp
%kata T12_Rotation_Test

operation Rotation (q : Qubit, k : Int) : Unit is Adj+Ctl {
    // ...
}
```

### Task 1.3. Prepare binary fraction exponent (classical input)

**Inputs:**

  1. A qubit in state $|\psi\rangle = \alpha |0\rangle + \beta |1\rangle$.

  2. An array of $n$ bits $[j_1, j_2, ..., j_n]$, stored as `Int[]` ($ j_k \in \{0,1\}$).

**Goal:**

Change the state of the qubit to $\alpha |0\rangle + \beta \cdot e^{2\pi i \cdot 0.j_1 j_2 ... j_n} |1\rangle$,
where $0.j_1 j_2 ... j_n$ is a binary fraction, similar to decimal fractions:

$$0.j_1 j_2 ... j_n = j_1 \cdot \frac{1}{2^1} + j_2 \cdot \frac{1}{2^2} + ... j_n \cdot \frac{1}{2^n}$$

```qsharp
%kata T13_BinaryFractionClassical_Test

operation BinaryFractionClassical (q : Qubit, j : Int[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 1.4. Prepare binary fraction exponent (quantum input)

**Inputs:**

  1. A qubit in state $|\psi\rangle = \alpha |0\rangle + \beta |1\rangle$.

  2. A register of $n$ qubits in state $|j_1 j_2 ... j_n\rangle$.

**Goal:**

Change the state of the input
from $(\alpha |0\rangle + \beta |1\rangle) \otimes |j_1 j_2 ... j_n\rangle$
to $(\alpha |0\rangle + \beta \cdot e^{2\pi i \cdot 0.j_1 j_2 ... j_n} |1\rangle) \otimes |j_1 j_2 ... j_n\rangle$,

where $0.j_1 j_2 ... j_n$ is a binary fraction corresponding to the basis state $j_1 j_2 ... j_n$ of the register.

> The register of qubits can be in superposition as well;
the behavior in this case is defined by behavior on the basis states and the linearity of unitary transformations.

```qsharp
%kata T14_BinaryFractionQuantum_Test

operation BinaryFractionQuantum (q : Qubit, jRegister : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 1.5. Prepare binary fraction exponent in place (quantum input)

**Input:**

 A register of $n$ qubits in state $|j_1 j_2 ... j_n \rangle$.

**Goal:**

Change the state of the register
from $|j_1\rangle \otimes |j_2 ... j_n\rangle$
to $\frac{1}{\sqrt{2}} \big(|0\rangle + e^{2\pi i \cdot 0.j_1 j_2 ... j_n} |1\rangle \otimes |j_2 ... j_n\rangle\big)$.

> The register of qubits can be in superposition as well;
the behavior in this case is defined by behavior on the basis states and the linearity of unitary transformations.

  Need a hint? Click here
  This task is very similar to task 1.4, but the digit $j_1$ is encoded in-place, using task 1.1.

```qsharp
%kata T15_BinaryFractionQuantumInPlace_Test

operation BinaryFractionQuantumInPlace (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 1.6. Reverse the order of qubits

**Input:**

 A register of $n$ qubits in state $|x_1 x_2 ... x_n \rangle$.

**Goal:**

Reverse the order of qubits, i.e., convert the state of the register to $|x_n ... x_2 x_1\rangle$.

```qsharp
%kata T16_ReverseRegister_Test

operation ReverseRegister (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 1.7. Quantum Fourier transform

**Input:**

 A register of $n$ qubits in state $|j_1 j_2 ... j_n \rangle$.

**Goal:**

Apply quantum Fourier transform to the input register, i.e., transform it to a state

$$\frac{1}{\sqrt{2^{n}}} \sum_{k=0}^{n-1} e^{2\pi i \cdot \frac{jk}{2^{n}}} |k\rangle = $$
$$\begin{align}= &\frac{1}{\sqrt{2}} \big(|0\rangle + e^{2\pi i \cdot 0.j_n} |1\rangle\big) \otimes \\
\otimes &\frac{1}{\sqrt{2}} \big(|0\rangle + e^{2\pi i \cdot 0.j_{n-1} j_n} |1\rangle\big) \otimes ... \\
\otimes &\frac{1}{\sqrt{2}} \big(|0\rangle + e^{2\pi i \cdot 0.j_1 j_2 ... j_{n-1} j_n} |1\rangle\big)\end{align}$$

> The register of qubits can be in superposition as well;
the behavior in this case is defined by behavior on the basis states and the linearity of unitary transformations.

> You can do this with a library call, but we recommend
implementing the algorithm yourself for learning purposes, using the previous tasks.

  Need a hint? Click here
Consider preparing a different state first and transforming it to the goal state:

$\frac{1}{\sqrt{2}} (|0\rangle + exp(2\pi i * 0.j_1 j_2 ... j_{n-1} j_n) |1\rangle) \otimes ... \otimes$
$\otimes \frac{1}{\sqrt{2}} (|0\rangle + exp(2\pi i * 0.j_{n-1} j_n) |1\rangle) \otimes$
$\otimes \frac{1}{\sqrt{2}} (|0\rangle + exp(2\pi i * 0.j_n) |1\rangle) \otimes$

```qsharp
%kata T17_QuantumFourierTransform_Test

operation QuantumFourierTransform (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 1.8. Inverse QFT

**Input:**

 A register of $n$ qubits in state $|j_1 j_2 ... j_n \rangle$.

**Goal:**

Apply inverse QFT to the input register, i.e., transform it to a state
$\frac{1}{\sqrt{2^{n}}} \sum_{k=0}^{n-1} e^{-2\pi i \cdot \frac{jk}{2^{n}}} |k\rangle$.

  Need a hint? Click here
Inverse QFT is literally the inverse transformation of QFT.
Do you know a quantum way to express this?

```qsharp
%kata T18_InverseQFT_Test

operation InverseQFT (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

## Part II. Using the Quantum Fourier Transform

This section offers you tasks on state preparation and state analysis
that can be solved using QFT (or inverse QFT). It is possible to solve them
without QFT, but we recommend that you to try and come up with a QFT-based solution.

### Task 2.1. Prepare an equal superposition of all basis states

**Input:**

A register of $n$ qubits in state $|0...0\rangle$.

**Goal:**

Prepare an equal superposition of all basis vectors from $|0...0\rangle$ to $|1...1\rangle$
(i.e., state $\frac{1}{\sqrt{2^{n}}} \big(|0...0\rangle + ... + |1...1\rangle\big)$.

```qsharp
%kata T21_PrepareEqualSuperposition_Test

operation PrepareEqualSuperposition (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 2.2. Prepare a periodic state

**Inputs:**

  1. A register of $n$ qubits in state $|0...0\rangle$.

  2. An integer frequency F ($0 \leq F \leq 2^{n}-1$).

**Goal:**

Prepare a periodic state with frequency F on this register:

$$\frac{1}{\sqrt{2^{n}}} \sum_k e^{2\pi i \cdot \frac{Fk}{2^{n}}} |k\rangle$$

> For example, for $n = 2$ and $F = 1$ the goal state is $\frac{1}{2}\big(|0\rangle + i|1\rangle - |2\rangle - i|3\rangle\big)$.

> If you're using `DumpMachine` to debug your solution,
remember that this kata uses big endian encoding of states,
while `DumpMachine` uses little endian encoding.

  Need a hint? Click here
Which basis state can be mapped to this state using QFT?

```qsharp
%kata T22_PreparePeriodicState_Test

operation PreparePeriodicState (register : Qubit[], F : Int) : Unit is Adj+Ctl {
    // ...
}
```

### Task 2.3. Prepare a periodic state with alternating $1$ and $-1$ amplitudes

**Input:**

A register of $n$ qubits in state $|0...0\rangle$.

**Goal:**

Prepare a periodic state with alternating $1$ and $-1$ amplitudes of basis states:

$$\frac{1}{\sqrt{2^{n}}} \big(|0\rangle - |1\rangle + |2\rangle - |3\rangle + ... - |2^{n}-1\rangle\big)$$

> For example, for $n = 2$ the goal state is $\frac{1}{2} \big(|0\rangle - |1\rangle + |2\rangle - |3\rangle\big)$.

  Need a hint? Click here
Which basis state can be mapped to this state using QFT?  Which frequency would allow you to use task 2.2 here?

```qsharp
%kata T23_PrepareAlternatingState_Test

operation PrepareAlternatingState (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 2.4. Prepare an equal superposition of all even basis states

**Input:**

A register of $n$ qubits in state $|0...0\rangle$.

**Goal:**

Prepare an equal superposition of all even basis vectors:
$\frac{1}{\sqrt{2^{n-1}}} \big(|0\rangle + |2\rangle + ... + |2^{n}-2\rangle\big)$.

  Need a hint? Click here
Which superposition of two basis states can be mapped to this state using QFT?
    Use the solutions to tasks 2.1 and 2.3 to figure out the answer.

```qsharp
%kata T24_PrepareEqualSuperpositionOfEvenStates_Test

operation PrepareEqualSuperpositionOfEvenStates (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 2.5. Prepare a square-wave signal

**Input:**

A register of $n\geq2$ qubits in state $|0...0\rangle$.

**Goal:**

Prepare a periodic state with alternating $1, 1, -1, -1$ amplitudes of basis states:
$$\frac{1}{\sqrt{2^{n}}} \big(|0\rangle + |1\rangle - |2\rangle - |3\rangle + ... - |2^{n}-2\rangle - |2^{n}-1\rangle\big)$$

  Need a hint? Click here
Which superposition of two basis states can be mapped to this state using QFT?
    Remember that sum of two complex amplitudes can be a real number if their imaginary parts cancel out.

```qsharp
%kata T25_PrepareSquareWaveSignal_Test

operation PrepareSquareWaveSignal (register : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 2.6. Get the frequency of a signal

**Input:**

A register of $n\geq2$ qubits in state
$\frac{1}{\sqrt{2^{n}}} \sum_k e^{2\pi i \cdot \frac{Fk}{2^{n}}} |k\rangle$, $0\leq F\leq 2^{n}-1$.

**Goal:**

Return the frequency F of the "signal" encoded in this state.

```qsharp
%kata T26_Frequency_Test

operation Frequency (register : Qubit[]) : Int {
    // ...

    return -1;
}
```

## Part III. Powers and roots of the QFT

### Task 3.1 Implement powers of the QFT

**Inputs:**

  1. A register of $n$ qubits in an arbitrary state.

  2. An integer $P$ ($ 0 \leq P \leq 2^{20} - 1 $).

**Goal:**

Transform state $|x\rangle$ into state $ QFT^{P} |x\rangle$, where $QFT$ is the quantum Fourier transform implemented in part I.

**Note:**

Your solution has to run fast for any $P$ in the given range!

```qsharp
%kata T31_QFTPower_Test

operation QFTPower (P : Int, inputRegister : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```

### Task 3.2. Implement roots of the QFT
**Inputs:**

  1. A register of $n$ qubits in an arbitrary state.

  2. An integer $P$ ($ 2 \leq P \leq 8 $).

**Goal:**

Transform state $|x\rangle$ into state $V |x\rangle$ where $V^{P} = QFT$. In other words, implement a $P^{th}$ root of the $QFT$. You can implement the required unitary up to a global phase.

```qsharp
%kata T32_QFTRoot_Test

operation QFTRoot (P : Int, inputRegister : Qubit[]) : Unit is Adj+Ctl {
    // ...
}
```


# Data Analysis and Machine Learning: Logistic Regression

**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

Date: **Sep 26, 2018**

Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license

## Logistic Regression

In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable $y_i$ is based on some
independent variables $\hat{x}_i$. Linear regression resulted in
analytical expressions (in terms of matrices to invert) for several
quantities, ranging from the variance and thereby the confidence
intervals of the parameters $\hat{\beta}$ to the mean squared
error. If we can invert the product of the design matrices, linear
regression gives then a simple recipe for fitting our data.

Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients' brains, figure out if there is a tumor or
not; or given a specific physical system, we'd like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.

The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.

## Optimization and Deep learning

Logistic regression will also serve as our stepping stone towards neural
network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters $\hat{\beta}$. The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms.

We note also that many of the topics discussed here
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.

## Basics

We consider the case where the dependent variables, also called the
responses or the outcomes, $y_i$ are discrete and only take values
from $k=0,\dots,K-1$ (i.e. $K$ classes).

The goal is to predict the
output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
made of $n$ samples, each of which carries $p$ features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.

Let us specialize to the case of two classes only, with outputs $y_i=0$ and $y_i=1$. Our outcomes could represent the status of a credit card user who could default or not on her/his credit card debt. That is

$$
y_i = \begin{bmatrix} 0 & \mathrm{no}\\  1 & \mathrm{yes} \end{bmatrix}.
$$

## Linear classifier

Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if $y_i > 0.5$ and the no default case $y_i \leq 0.5$.

We would then have our
weighted linear combination, namely

$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} +  \hat{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
$$

where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.

## Some selected properties

The main problem with our  function is that it
takes values on the entire real axis. In the case of
logistic regression, however, the labels $y_i$ are discrete
variables.

One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values $\{0,1\}$,
$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft" classifier that outputs
the probability of a given category. This leads us to the logistic function.

The code for plotting the perceptron can be seen here. This si nothing but the standard [Heaviside step function](https://en.wikipedia.org/wiki/Heaviside_step_function).

## The logistic function

The perceptron is an example of a ``hard classification" model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a "soft"
classifier that outputs the probability of a given category rather
than a single value. For example, given $x_i$, the classifier
outputs the probability of being in a category $k$.  Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point $x_i$
belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,

$$
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$

Note that $1-p(t)= p(-t)$.
The following code plots the logistic function.

## Two parameters

We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities

$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
\end{align*}
$$

where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.

Note that we used

$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
$$

## Maximum likelihood

In order to define the total likelihood for all possible outcomes from a
dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels
$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is

$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$

from which we obtain the log-likelihood and our **cost/loss** function

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
$$

## The cost function rewritten

Reordering the logarithms, we can rewrite the **cost/loss** function as

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that

$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.

## Minimizing the cross entropy

The cross entropy is a convex function of the weights $\hat{\beta}$ and,
therefore, any local minimizer is a global minimizer.

Minimizing this
cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$

and

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$

## A more compact expression

Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
derivative of cost function as

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
$$

If we in addition define a diagonal matrix $\hat{W}$ with elements
$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as

$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
$$

## Extending to more predictors

## Including more classes

## Optimizing the cost function

Newton's method and gradient descent methods

## A **scikit-learn** example

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
iris = datasets.load_iris()
list(iris.keys())
['data', 'target_names', 'feature_names', 'target', 'DESCR']
X = iris["data"][:, 3:] # petal width
y = (iris["target"] == 2).astype(np.int) # 1 if Iris-Virginica, else 0

from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression()
log_reg.fit(X, y)

X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
y_proba = log_reg.predict_proba(X_new)
plt.plot(X_new, y_proba[:, 1], "g-", label="Iris-Virginica")
plt.plot(X_new, y_proba[:, 0], "b--", label="Not Iris-Virginica")
plt.show()
```

## A simple classification problem

```python
import numpy as np
from sklearn import datasets, linear_model
import matplotlib.pyplot as plt

def generate_data():
    np.random.seed(0)
    X, y = datasets.make_moons(200, noise=0.20)
    return X, y

def visualize(X, y, clf):
    # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)
    # plt.show()
    plot_decision_boundary(lambda x: clf.predict(x), X, y)
    plt.title("Logistic Regression")

def plot_decision_boundary(pred_func, X, y):
    # Set min and max values and give it some padding
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole gid
    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
    plt.show()

def classify(X, y):
    clf = linear_model.LogisticRegressionCV()
    clf.fit(X, y)
    return clf

def main():
    X, y = generate_data()
    # visualize(X, y)
    clf = classify(X, y)
    visualize(X, y, clf)

if __name__ == "__main__":
    main()
```


```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import CurrentDistribution as current

currentPath = current.Solenoid(10, 10, 1, sp.pi/64)
offset=5
testPointNum = 16

xPoints = np.linspace(int(currentPath.xmin-offset), int(currentPath.xmax+offset), num=testPointNum)
yPoints = np.linspace(int(currentPath.ymin-offset), int(currentPath.ymax+offset), num=testPointNum)
zPoints = np.linspace(int(currentPath.zmin-offset), int(currentPath.zmax+offset), num=testPointNum)

xx, yy, zz = np.meshgrid(xPoints, yPoints, zPoints)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(currentPath.segments[:,0],currentPath.segments[:,1],currentPath.segments[:,2],s=1)
ax.scatter(xx,yy,zz,s=1, c="Red")
plt.show()
```

```python

```


# Learning from data: Bayesian Parameter Estimation

**Christian Forssén**, Department of Physics, Chalmers University of Technology, Sweden

Date: **Sep 16, 2019**

Copyright 2018-2019, Christian Forssén. Released under CC Attribution-NonCommercial 4.0 license

Joint pdf for the masses of two black holes merging obtained from the data analysis of a gravitational wave signal. This representation of a joint pdf is known as a corner plot.

# Inference With Parametric Models
Inductive inference with parametric models is a very important tool in the natural sciences.
* Consider $N$ different models $M_i$ ($i = 1, \ldots, N$), each with parameters $\boldsymbol{\theta}_i$. Each of them implies a sampling distribution for possible data

$$
p(D|\boldsymbol{\theta}_i, M_i)
$$

* The likelihood function is the pdf of the actual, observed data ($D_\mathrm{obs}$) given a set of parameters $\boldsymbol{\theta}_i$:

$$
\mathcal{L}_i (\boldsymbol{\theta}_i) \equiv p(D_\mathrm{obs}|\boldsymbol{\theta}_i, M_i)
$$

* We may be uncertain about $M_i$ (model uncertainty),

* or uncertain about $\boldsymbol{\theta}_i$ (parameter uncertainty).

Parameter Estimation:
  :
  Premise = We have chosen a model (say $M_1$)
  $\Rightarrow$ What can we say about its parameters $\boldsymbol{\theta}_1$?

Model comparison:
  :
  Premise = We have a set of different models $\{M_i\}$
  $\Rightarrow$ How do they compare with each other? Do we have evidence to say that, e.g. $M_1$, is better than the other models?

Model adequacy:
  :
  Premise = We have a model $M_1$
  $\Rightarrow$ Is $M_1$ adequate?

Hybrid Uncertainty:
  :
  Models share some common params: $\boldsymbol{\theta}_i = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\}$
  $\Rightarrow$ What can we say about $\boldsymbol{\varphi}$? (Systematic error is an example)

## Parameter estimation
Overview comments:
* In general terms, "parameter estimation" in physics means obtaining values for parameters (constants) that appear in a theoretical model which describes data (exceptions to this general definition exist of course).

* Conventionally this process is known as "parameter fitting" and the goal is to find the "best fit".

* We will make particular interpretations of these phrases from our Bayesian point of view.

* We will also see how familiar ideas like "least-squares optimization" show up from a Bayesian perspective.

## Bayesian parameter estimation
We will now consider the very important task of model parameter estimation using statistical inference.

Let us first remind ourselves what can go wrong in a fit. We have encountered both **underfitting** (model is not complex enough to describe the variability in the data) and **overfitting** (model tunes to data fluctuations, or terms are underdetermined causing them playing off each other). Bayesian methods can prevent/identify both these situations.

# Example: Measured flux from a star (single parameter)
Adapted from the blog [Pythonic Perambulations](http://jakevdp.github.io) by Jake VanderPlas.

Imagine that we point our telescope to the sky, and observe the light coming from a single star. Our physics model will be that the star's true flux is constant with time, i.e. that  it has a fixed value $F_\mathrm{true}$ (we'll also ignore effects like sky noise and other sources of systematic error). Thus, we have a single model parameter: $F_\mathrm{true}$.

We'll assume that we perform a series of $N$ measurements with our telescope, where the i:th measurement reports an observed photon flux $F_i$ and is accompanied by an error model given by $e_i$[^errors].
The question is, given this set of measurements $D = \{F_i\}_{i=0}^{N-1}$, and the statistical model $F_i = F_\mathrm{true} + e_i$, what is our best estimate of the true flux $F_\mathrm{true}$?

[^errors]: We'll make the reasonable assumption that errors are Gaussian. In a Frequentist perspective, $e_i$ is the standard deviation of the results of a single measurement event in the limit of repetitions of *that event*. In the Bayesian perspective, $e_i$ is the standard deviation of the (Gaussian) probability distribution describing our knowledge of that particular measurement given its observed value.

Because the measurements are number counts, a Poisson distribution is a good approximation to the measurement process:

```python
%matplotlib inline

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import emcee
```

```python
np.random.seed(1)      # for repeatability
F_true = 1000          # true flux, say number of photons measured in 1 second
N = 50                 # number of measurements
F = stats.poisson(F_true).rvs(N)
                       # N measurements of the flux
e = np.sqrt(F)         # errors on Poisson counts estimated via square root
```

Now let's make a simple visualization of the "observed" data, see Fig. [fig:flux](#fig:flux).

```python
fig, ax = plt.subplots()
ax.errorbar(F, np.arange(N), xerr=e, fmt='ok', ecolor='gray', alpha=0.5)
ax.vlines([F_true], 0, N, linewidth=5, alpha=0.2)
ax.set_xlabel("Flux");ax.set_ylabel("measurement number");
```

Single photon counts (flux measurements).

These measurements each have a different error $e_i$ which is estimated from Poisson statistics using the standard square-root rule. In this toy example we know the true flux that was used to generate the data, but the question is this: given our measurements and statistical model, what is our best estimate of $F_\mathrm{true}$?

Let's take a look at the frequentist and Bayesian approaches to solving this.

### Simple Photon Counts: Frequentist Approach

We'll start with the classical frequentist maximum likelihood approach. Given a single observation $D_i = F_i$, we can compute the probability distribution of the measurement given the true flux $F_\mathrm{true}$ given our assumption of Gaussian errors

$$
\begin{equation}
p(D_i | F_\mathrm{true}, I) = \frac{1}{\sqrt{2\pi e_i^2}} \exp \left( \frac{-(F_i-F_\mathrm{true})^2}{2e_i^2} \right).
\label{_auto1} \tag{1}
\end{equation}
$$

This should be read "the probability of $D_i$ given $F_\mathrm{true}$
equals ...". You should recognize this as a normal distribution with mean $F_\mathrm{true}$ and standard deviation $e_i$.

We construct the *likelihood function* by computing the product of the probabilities for each data point

$$
\begin{equation}
\mathcal{L}(F_\mathrm{true}) = \prod_{i=1}^N p(D_i | F_\mathrm{true}, I),
\label{_auto2} \tag{2}
\end{equation}
$$

here $D = \{D_i\}$ represents the entire set of measurements. Because the value of the likelihood can become very small, it is often more convenient to instead compute the log-likelihood.

**Notice.**

In the following we will use $\log$ to denote the natural logarithm. We will write $\log_{10}$ if we specifically mean the logarithm with base 10.

Combining the previous two equations and computing the log, we have

$$
\begin{equation}
\log\mathcal{L} = -\frac{1}{2} \sum_{i=1}^N \left[ \log(2\pi e_i^2) +  \frac{(F_i-F_\mathrm{true})^2}{e_i^2} \right].
\label{_auto3} \tag{3}
\end{equation}
$$

What we'd like to do is determine $F_\mathrm{true}$ such that the likelihood is maximized. At this pont we can note that that problem of maximizing the likelihood is equivalent to the minimization of the sum

$$
\begin{equation}
\sum_{i=1}^N \frac{(F_i-F_\mathrm{true})^2}{e_i^2},
\label{_auto4} \tag{4}
\end{equation}
$$

which you should recognize as the chi-squared function encountered in the linear regression model.

Therefore, it is not surprising that this particular maximization problem can be solved analytically (i.e. by setting $d\log\mathcal{L}/d F_\mathrm{true} = 0$). This results in the following observed estimate of $F_\mathrm{true}$

$$
\begin{equation}
F_\mathrm{est} = \frac{ \sum_{i=1}^N w_i F_i }{ \sum_{i=1}^N w_i}, \quad w_i = 1/e_i^2.
\label{_auto5} \tag{5}
\end{equation}
$$

Notice that in the special case of all errors $e_i$ being equal, this reduces to

$$
\begin{equation}
F_\mathrm{est} = \frac{1}{N} \sum_{i=1} F_i.
\label{_auto6} \tag{6}
\end{equation}
$$

That is, in agreement with intuition, $F_\mathrm{est}$ is simply the mean of the observed data when errors are equal.

We can go further and ask what the error of our estimate is. In the frequentist approach, this can be accomplished by fitting a Gaussian approximation to the likelihood curve at maximum; in this simple case this can also be solved analytically (the sum of Gaussians is also a Gaussian). It can be shown that the standard deviation of this Gaussian approximation is $\sigma_\mathrm{est}$, which is given by

$$
\begin{equation}
\frac{ 1 } {\sigma_\mathrm{est}^2} = \sum_{i=1}^N w_i .
\label{_auto7} \tag{7}
\end{equation}
$$

These results are fairly simple calculations; let's evaluate them for our toy dataset:

```python
w=1./e**2
print(f"""
F_true = {F_true}
F_est = {(w * F).sum() / w.sum():.0f} +/- { w.sum() ** -0.5:.0f} (based on {N} measurements) """)
```

`F_true = 1000`
`F_est = 998 +/- 4 (based on 50 measurements)`

We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value.

### Simple Photon Counts: Bayesian Approach

The Bayesian approach, as you might expect, begins and ends with probabilities. Our hypothesis is that the star has a constant flux $F_\mathrm{true}$. It recognizes that what we fundamentally want to compute is our knowledge of the parameter in question given the data and other information (such as our knowledge of uncertainties for the observed values), i.e. in this case, $p(F_\mathrm{true} | D,I)$.
Note that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like $F_\mathrm{true}$. Nevertheless, within the Bayesian philosophy this is perfectly acceptable.

To compute this pdf, Bayesians next apply Bayes' Theorem.
If we set the prior $p(F_\mathrm{true}|I) \propto 1$ (a flat prior), we find
$p(F_\mathrm{true}|D,I) \propto p(D | F_\mathrm{true},I) \equiv \mathcal{L}(F_\mathrm{true})$
and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent.

### A note about priors

The prior allows inclusion of other information into the computation, which becomes very useful in cases where multiple measurement strategies are being combined to constrain a single model. The necessity to specify a prior, however, is one of the more controversial pieces of Bayesian analysis.
A frequentist will point out that the prior is problematic when no true prior information is available. Though it might seem straightforward to use a noninformative prior like the flat prior mentioned above, there are some [surprisingly subtleties](http://normaldeviate.wordpress.com/2013/07/13/lost-causes-in-statistics-ii-noninformative- priors/comment-page-1/) involved. It turns out that in many situations, a truly noninformative prior does not exist! Frequentists point out that the subjective choice of a prior which necessarily biases your result has no place in statistical data analysis.
A Bayesian would counter that frequentism doesn't solve this problem, but simply skirts the question. Frequentism can often be viewed as simply a special case of the Bayesian approach for some (implicit) choice of the prior: a Bayesian would say that it's better to make this implicit choice explicit, even if the choice might include some subjectivity.

### Simple Photon Counts: Bayesian approach in practice

Leaving these philosophical debates aside for the time being, let's address how Bayesian results are generally computed in practice. For a one parameter problem like the one considered here, it's as simple as computing the posterior probability $p(F_\mathrm{true} | D,I)$ as a function of $F_\mathrm{true}$: this is the distribution reflecting our knowledge of the parameter $F_\mathrm{true}$.
But as the dimension of the model grows, this direct approach becomes increasingly intractable. For this reason, Bayesian calculations often depend on sampling methods such as Markov Chain Monte Carlo (MCMC). For this practical example, let us apply an MCMC approach using Dan Foreman-Mackey's [emcee](http://dan.iel.fm/emcee/current/) package. Keep in mind here that the goal is to generate a set of points drawn from the posterior probability distribution, and to use those points to determine the answer we seek.
To perform this MCMC, we start by defining Python functions for the prior $p(F_\mathrm{true} | I)$, the likelihood $p(D | F_\mathrm{true},I)$, and the posterior $p(F_\mathrm{true} | D,I)$, noting that none of these need be properly normalized. Our model here is one-dimensional, but to handle multi-dimensional models we'll define the model in terms of an array of parameters $\boldsymbol{\theta}$, which in this case is $\boldsymbol{\theta} = [F_\mathrm{true}]$

```python
def log_prior(theta):
    if theta>0 and theta<10000:
        return 0 # flat prior
    else:
        return -np.inf

def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * e ** 2) \
                             + (F - theta[0]) ** 2 / e ** 2)

def log_posterior(theta, F, e):
    return log_prior(theta) + log_likelihood(theta, F, e)
```

Now we set up the problem, including generating some random starting guesses for the multiple chains of points.

```python
ndim = 1      # number of parameters in the model
nwalkers = 50 # number of MCMC walkers
nwarm = 1000  # "warm-up" period to let chains stabilize
nsteps = 2000 # number of MCMC steps to take
# we'll start at random locations between 0 and 2000
starting_guesses = 2000 * np.random.rand(nwalkers, ndim)
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F,e])
sampler.run_mcmc(starting_guesses, nsteps)
# Shape of sampler.chain  = (nwalkers, nsteps, ndim)
# Flatten the sampler chain and discard warm-in points:
samples = sampler.chain[:, nwarm:, :].reshape((-1, ndim))
```

If this all worked correctly, the array sample should contain a series of 50,000 points drawn from the posterior. Let's plot them and check. See results in Fig. [fig:flux-bayesian](#fig:flux-bayesian).

```python
fig, ax = plt.subplots()
ax.hist(samples, bins=50, histtype="stepfilled", alpha=0.3, density=True)
ax.set_xlabel(r'$F_\mathrm{est}$')
ax.set_ylabel(r'$p(F_\mathrm{est}|D,I)$');
```

Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements.

## Aside: Best estimates and confidence intervals
The posterior distribution from our Bayesian data analysis is the key quantity that encodes our inference about the values of the model parameters, given the data and the relevant background information. Often, however, we wish to summarize this result with just a few numbers: the best estimate and a measure of its reliability.

There are a few different options for this. The choice of the most appropriate one depends mainly on the shape of the posterior distribution:

### Symmetric posterior pdfs

Since the probability (density) associated with any particular value of the parameter is a measure of how much we believe that it lies in the neighbourhood of that point, our best estimate is given by the maximum of the posterior pdf. If we denote the quantity of interest by $\theta$, with a posterior pdf $P =p(\theta|D,I)$, then the best estimate of its value $\theta_0$ is given by the condition $dP/d\theta|_{\theta=\theta_0}=0$. Strictly speaking, we should also check the sign of the second derivative to ensure that $\theta_0$ represents a maximum.

To obtain a measure of the reliability of this best estimate, we need to look at the width or spread of the posterior pdf about $\theta_0$. When considering the behaviour of any function in the neighbourhood of a particular point, it is often helpful to carry out a Taylor series expansion; this is simply a standard tool for (locally) approximating a complicated function by a low-order polynomial. The linear term is zero at the maximum and the quadratic term is often the dominating one determining the width of the posterior pdf. Ignoring all the higher-order terms we arrive at the Gaussian approximation (see more details below)

$$
\begin{equation}
p(\theta|D,I) \approx \frac{1}{\sigma\sqrt{2\pi}} \exp \left[ -\frac{(\theta-\mu)^2}{2\sigma^2} \right],
\label{_auto8} \tag{8}
\end{equation}
$$

where the mean $\mu = \theta_0$ and the variance $\sigma = \left( - \left. \frac{d^2L}{d\theta^2} \right|_{\theta_0} \right)^{-1/2}$, where $L$ is the logarithm of the posterior $P$. Our inference about the quantity of interest is conveyed very concisely, therefore, by the 67% Bayesian confidence interval $\theta = \theta_0 \pm \sigma$, and

$$
p(\theta_0-\sigma < \theta < \theta_0+\sigma | D,I) = \int_{\theta_0-\sigma}^{\theta_0+\sigma} p(\theta|D,I) d\theta \approx 0.67.
$$

### Asymmetric posterior pdfs

While the maximum of the posterior ($\theta_0$) can still be regarded as giving the best estimate, the true value is now more likely to be on one side of this rather than the other. Alternatively one can compute the mean value, $\langle \theta \rangle = \int \theta p(\theta|D,I) d\theta$, although this tends to overemphasise very long tails. The best option is probably a compromise that can be employed when having access to a large sample from the posterior (as provided by an MCMC), namely to give the median of this ensemble.

Furthermore, the concept of an error-bar does not seem appropriate in this case, as it implicitly entails the idea of symmetry. A good way of expressing the reliability with which a parameter can be inferred, for an asymmetric posterior pdf, is rather through a *confidence interval*. Since the area under the posterior pdf between $\theta_1$ and $\theta_2$ is proportional to how much we believe that $\theta$ lies in that range, the shortest interval that encloses 67% of the area represents a sensible measure of the uncertainty of the estimate. Obviously we can choose to provide some other degree-of-belief that we think is relevant for the case at hand. Assuming that the posterior pdf has been normalized, to have unit area, we need to find $\theta_1$ and $\theta_2$ such that:

$$
p(\theta_1 < \theta < \theta_2 | D,I) = \int_{\theta_1}^{\theta_2} p(\theta|D,I) d\theta \approx 0.67,
$$

where the difference $\theta_2 - \theta_1$ is as small as possible. The region $\theta_1 < \theta < \theta_2$ is then called the shortest 67% confidence interval.

### Multimodal posterior pdfs

We can sometimes obtain posteriors which are multimodal; i.e. contains several disconnected regions with large probabilities. There is no difficulty when one of the maxima is very much larger than the others: we can simply ignore the subsidiary solutions, to a good approximation, and concentrate on the global maximum. The problem arises when there are several maxima of comparable magnitude. What do we now mean by a best estimate, and how should we quantify its reliability? The idea of a best estimate and an error-bar, or even a confidence interval, is merely an attempt to summarize the posterior with just two or three numbers; sometimes this just can’t be done, and so these concepts are not valid. For the bimodal case we might be able to characterize the posterior in terms of a few numbers: two best estimates and their associated error-bars, or disjoint confidence intervals. For a general multimodal pdf, the most honest thing we can do is just display the posterior itself.

Two options for assigning credible intervals to asymmetric and multimodal pdfs:
* Equal-tailed interval: the probability area above and below the interval are equal.

* Highest posterior density (HPD) interval: The posterior density for any point within the interval is larger than the posterior density for any point outside the interval.

### Different views on confidence intervals

A Bayesian confidence interval, or credible interval, or degree-of-belief (DOB) interval is the following: Given this data and other information there is $d \%$ probability that this interval contains the true value of the parameter. E.g. a 95% DOB interval implies that the Baysian data analyser would bet 20-to-1 that the true result is inside the interval.

A frequentist 95% confidence interval should be understood as follows:
"There is a 95% probability that when I compute a confidence interval from data of this sort that he true value of the parameter will fall within the (hypothetical) space of observations". So the parameter is fixed (no pdf) and the confidence interval is based on random sampling of data.

Let's try again to understand this: If we make a large number of repeated samples, then 95% of the intervals extracted in this way will include the true value of the parameter.

### Simple Photon Counts: Best estimates and confidence intervals

To compute these numbers for our example, you would run:

```python
sampper=np.percentile(samples, [2.5, 16.5, 50, 83.5, 97.5],axis=0).flatten()
print(f"""
F_true = {F_true}
Based on {N} measurements the posterior point estimates are:
...F_est = { np.mean(samples):.0f} +/- { np.std(samples):.0f}
or using credibility intervals:
...F_est = {sampper[2]:.0f}          (posterior median)
...F_est in [{sampper[1]:.0f}, {sampper[3]:.0f}] (67% credibility interval)
...F_est in [{sampper[0]:.0f}, {sampper[4]:.0f}] (95% credibility interval) """)
```

`F_true = 1000`
`Based on 50 measurements the posterior point estimates are:`
`...F_est = 998 +/- 4`
`or using credibility intervals:`
`...F_est = 998          (posterior median)`
`...F_est in [993, 1002] (67% credibility interval)`
`...F_est in [989, 1006] (95% credibility interval)`

In this particular example, the posterior pdf is actually a Gaussian (since it is constructed as a product of Gaussians), and the mean and variance from the quadratic approximation will agree exactly with the frequentist approach.

From this final result you might come away with the impression that the Bayesian method is unnecessarily complicated, and in this case it certainly is. Using an MCMC sampler to characterize a one-dimensional normal distribution is a bit like using the Death Star to destroy a beach ball, but we did this here because it demonstrates an approach that can scale to complicated posteriors in many, many dimensions, and can provide nice results in more complicated situations where an analytic likelihood approach is not possible.

Furthermore, as data and models grow in complexity, the two approaches can diverge greatly.

# Example: Gaussian noise and averages
The example in the demonstration notebook is from Sivia's book. How do we infer the mean and standard deviation of a Gaussian distribution from $M$ measurements $D \in \{ x_k \}_{k=0}^{M-1}$ that should be distributed according to a normal distribution $p( D | \mu,\sigma,I)$?

Start from Bayes theorem

$$
p(\mu,\sigma | D, I) = \frac{p(D|\mu,\sigma,I) p(\mu,\sigma|I)}{p(D|I)}
$$

* Remind yourself about the names of the different terms.

* It should become intuitive what the different probabilities (pdfs) describe.

* Bayes theorem tells you how to flip from (hard-to-compute) $p(\mu,\sigma | D, I) \Leftrightarrow p(D|\mu,\sigma,I)$ (easier-to-compute).

Aside on the denominator, which is known as the "data probability" or "marginalized likelihood" or "evidence".
* With $\theta$ denoting a general vector of parameters we must have

$$
p(D|I) = \int d\theta p(D|\theta,I) p(\theta|I).
$$

* This integration (or marginalization) over all parameters is often difficult to perform.

* Fortunately, for **parameter estimation** we don't need $p(D|I)$ since it doesn't depend on $\theta$. We usually only need relative probabilities, or we can determine the normalization $N$ after we have computed the unnormalized posterior

$$
p(\theta | D,I) = \frac{1}{N} p(D|\theta,I) p(\theta|I).
$$

If we use a uniform prior $p(\theta | I ) \propto 1$ (in a finite volume), then the posterior is proportional to the **likelihood**

$$
p(\theta | D,I) \propto p(D|\theta,I) = \mathcal{L}(\theta)
$$

In this particular situation, the mode of the likelihood (which would correspond to the point estimate of maximum likelihood) is equivalent to the mode of the posterior pdf in the Bayesian analysis.

The real use of the prior, however, is to include into the analysis any additional information that you might have. The prior statement makes such additional assumptions and information very explicit.

But how do we actually compute the posterior in practice. Most often we won't be able to get an analytical expression, but we can sample the distribution using a method known as Markov Chain Monte Carlo (MCMC).

# Example: Fitting a straight line
The next example that we will study is the well known fit of a straight line.

* Here the theoretical model is

$$
y_\mathrm{th}(x; \theta) = m x + b,
$$

with parameters $\theta = [b,m]$.

* The statistical model for the data is

$$
y_{\mathrm{exp},i} = y_{\mathrm{th},i} + \delta y_{\mathrm{exp},i},
$$

where we often assume that the experimental errors are independent and normally distributed so that

$$
y_i = \mathcal{N} \left( y_\mathrm{th}(x_i; \theta), e_i^2 \right).
$$

* Is independent errors always a good approximation?

* An even better statistical model for theoretical models with a quantified, finite resolution would be

$$
y_\mathrm{exp} = y_\mathrm{th} + \delta y_\mathrm{exp} + \delta y_\mathrm{th}.
$$

### Why normal distributions?

Let us give a quick motivation why Gaussian distributions show up so often. Say that we have a pdf $p(\theta | D,I)$. Our best estimate from this pdf will be $\theta_0$ where

$$
\left.
\frac{ \partial p }{ \partial \theta }
\right|_{\theta_0} = 0, \qquad
\left. \frac{ \partial^2 p }{ \partial \theta^2 }
\right|_{\theta_0} < 0.
$$

The distribution usually varies very rapidly so we study $L(\theta) \equiv \log p$ instead.
Near the peak, it behaves as

$$
L(\theta) = L(\theta_0) + \frac{1}{2} \left. \frac{\partial^2 L}{\partial \theta^2} \right|_{\theta_0} \left( \theta - \theta_0 \right)^2 + \ldots,
$$

where the first-order term is zero since we are expanding around a maximum and $\partial L / \partial\theta = 0$.

If we neglect higher-order terms we find that

$$
p(\theta|D,I) \approx A \exp \left[ \frac{1}{2} \left. \frac{\partial^2 L}{\partial \theta^2} \right|_{\theta_0} \left( \theta - \theta_0 \right)^2  \right],
$$

which is a Gaussian $\mathcal{N}(\mu,\sigma^2)$ with

$$
\mu = \theta_0, \qquad \sigma^2 = \left( - \left. \frac{\partial^2 L}{\partial \theta^2} \right|_{\theta_0} \right)^{-1/2}.
$$

## Correlations
In the "fitting a straight-line" example you should find that the joint pdf for the slope and the intercept $[m, b]$ corresponds to a slanted ellipse. That result implies that the model parameters are **correlated**.

* Try to understand the correlation that you find in this example.

Let us explore correlations by studying the behavior of the pdf at the maximum.
A Taylor expansion for a bivariate pdf $p(x,y)$ around the mode $(x_0,y_0)$ gives

$$
p(x,y) \approx p(x_0,y_0) + \frac{1}{2} \begin{pmatrix} x-x_0 & y-y_0 \end{pmatrix}
H
\begin{pmatrix} x-x_0 \\ y-y_0 \end{pmatrix},
$$

where $H$ is the symmetric Hessian matrix

$$
\begin{pmatrix}
A & C \\ C & B
\end{pmatrix},
$$

with elements

$$
A = \left. \frac{\partial^2 p}{\partial x^2} \right|_{x_0,y_0}, \quad
B = \left. \frac{\partial^2 p}{\partial y^2} \right|_{x_0,y_0}, \quad
C = \left. \frac{\partial^2 p}{\partial x \partial y} \right|_{x_0,y_0}.
$$

* So in this quadratic approximation the contour is an ellipse centered at $(x_0,y_0)$ with orientation and eccentricity determined by $A,B,C$.

* The principal axes are found from the eigenvectors of $H$.

* Depending on the skewness of the ellipse, the parameters are either (i) not correlated, (ii) correlated, or (iii) anti-correlated.

* Take a minute to consider what that implies.


## Interactive Bootstrap Demonstration

### Boostrap for Uncertainty in Sample Statistics Tutorial

* interactive plot demonstration with ipywidget package

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.  Let's try it out.

#### Objective

Provide an example and demonstration for:

1. interactive plotting in Jupyter Notebooks with Python packages matplotlib and ipywidgets
2. provide an intuitive hands-on example of statistical boostrap

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

#### Load the Required Libraries

The following code loads the required libraries.

```python
%matplotlib inline
from ipywidgets import interactive                        # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
import matplotlib.pyplot as plt                           # plotting
import numpy as np                                        # working with arrays
import pandas as pd                                       # working with DataFrames
from scipy.stats import triang                            # parametric distributions
from scipy.stats import binom
from scipy.stats import norm
from scipy.stats import uniform
from scipy.stats import triang
from scipy import stats                                   # statistical calculations
import random                                             # random drawing / bootstrap realizations of the data
```

#### Make a Synthetic Dataset

This is an interactive method to:

* select a parametric distribution

* select the distribution parameters

* select the number of samples and visualize the synthetic dataset distribution

```python
# parameters for the synthetic dataset
bins = np.linspace(0,1000,1000)

# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value='                                      Boostrap Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
dist = widgets.Dropdown(
    options=['Triangular', 'Uniform', 'Gaussian'],
    value='Gaussian',
    description='Dataset Distribution:',
    disabled=False,
    layout=Layout(width='200px', height='30px')
)
a = widgets.FloatSlider(min=0.0, max = 100.0, value = 0.5, description = 'Dataset: Mean / Mode',orientation='vertical',layout=Layout(width='170px', height='200px'))
a.style.handle_color = 'blue'
d = widgets.FloatSlider(min=0.01, max = 30.0, value = 5.0, step = 1.0, description = 'Dataset: St. Deviation',orientation='vertical',layout=Layout(width='130px', height='200px'))
d.style.handle_color = 'green'
b = widgets.FloatSlider(min = 0, max = 100.0, value = 0.5, description = 'Dataset: Minimum',orientation='vertical',layout=Layout(width='130px', height='200px'))
b.style.handle_color = 'red'
c = widgets.IntSlider(min = 0, max = 100, value = 100, description = 'Dataset: Maximum',orientation='vertical',layout=Layout(width='130px', height='200px'))
c.style.handle_color = 'orange'
n = widgets.IntSlider(min = 2, max = 1000, value = 100, description = 'Dataset: Number Samples',orientation='vertical',layout=Layout(width='180px', height='200px'))
n.style.handle_color = 'gray'

ui = widgets.HBox([dist,a,d,b,c,n],)                      # basic widget formatting
ui2 = widgets.VBox([l,ui],)

def f_make(dist,a, b, c, d, n):                           # function to take parameters, make sample and plot
    dataset = make_data(dist,a, b, c, d, n)

    plt.subplot(111)
    plt.hist(
        dataset,
        alpha=0.2,
        color="red",
        edgecolor="black",
        bins=bins)
    plt.xlim(0.0,100.0); plt.title('Synthetic Dataset'); plt.ylabel('Frequency'); plt.xlabel('Data Values')

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.6, wspace=0.2, hspace=0.3)
    plt.show()

def make_data(dist,a, b, c, d, n):                        # function to check parameters and make sample
    if dist == 'Uniform':
        if b >= c:
            print('Invalid uniform distribution parameters')
            return None
        dataset = uniform.rvs(size=n, loc = b, scale = c, random_state = 73073).tolist()
        return dataset
    elif dist == 'Triangular':
        interval = c - b
        if b >= a or a >= c or interval <= 0:
            print('Invalid triangular distribution parameters')
            return None
        dataset = triang.rvs(size=n, loc = b, c = (a-b)/interval, scale = interval, random_state = 73073).tolist()
        return dataset
    elif dist == 'Gaussian':
        dataset = norm.rvs(size=n, loc = a, scale = d, random_state = 73073).tolist()
        return dataset

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'dist': dist,'a': a, 'd': d, 'b': b, 'c': c, 'n': n})
interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

#### Display the GUI for Building the Synthetic Dataset

We display the GUI now.  Select the desired parametric distribution and associated parameters.

* if the parameters are invalid (e.g. traingular mode > max) an error message should display.

```python
display(ui2, interactive_plot)                            # display the interactive plot
```

    VBox(children=(Text(value='                                      Boostrap Demonstration, Michael Pyrcz, Associ…

    Output()

We now have a synthetic dataset to work with. Now we can:

* assign the samples to a 1D ndarray

* make a DataFrame with the samples

* check the summary statistics

This is our sample set that we will apply ot bootstrap.

```python
data = make_data(dist.value,a.value, b.value, c.value, d.value, n.value)
df = pd.DataFrame({'DataSet':data})
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      DataSet
      27.0
      57.671481
      11.604119
      22.400407
      51.761746
      58.371597
      67.016427
      76.082631

#### Perform Bootstrap

Now we take our synthetic dataset, sampled from the parametric distributioin above, and apply it to statistical bootstrap.

* we calculate the sampling distributions / uncertainty in the mean and standard deviation

```python
L = 1000                                                  # set the number of realizations
mean = np.zeros(L); stdev = np.zeros(L)                   # declare arrays to hold the realizations of the statistics
for l in range(0, L):                                     # loop over realizations
    samples = random.choices(df['DataSet'].values, weights=None, cum_weights=None, k=len(df))
    mean[l] = np.average(samples)
    stdev[l] = np.std(samples)

plt.subplot(121)                                          # plot the distribution for uncertainty in the mean
plt.hist(mean,alpha=0.2,color="red",edgecolor="black",bins=np.linspace(0,100,40)); plt.xlim(0,100); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Mean')

plt.subplot(122)                                          # plot the distribution for uncertainty in the standard deviation
plt.hist(stdev,alpha=0.2,color="red",edgecolor="black",bins=np.linspace(0,40,40)); plt.xlim(0,40); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Standard Deviation')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

# provide summary statistics, P10, P50 and P90
print('Summary Statistics for Bootstrap for Uncertainty in the Mean:')
print(stats.describe(mean))
print('P10: ' + str(round(np.percentile(mean,10),3)) + ', P50: ' + str(round(np.percentile(mean,50),3)) + ', P90: ' + str(round(np.percentile(mean,90),3)))

print('\nSummary Statistics for Bootstrap for Uncertainty in the Standard Deviation:')
print(stats.describe(stdev))
print('P10: ' + str(round(np.percentile(stdev,10),3)) + ', P50: ' + str(round(np.percentile(stdev,50),3)) + ', P90: ' + str(round(np.percentile(stdev,90),3)))
```

#### Change the Number of Data

Let's change the number of data drawn to observe the change in uncertainty

* we will assume the same dataset and not recalculate it each time

* we will just sample with replacement with the new number of samples for each bootstrap realization

```python
# parameters for the synthetic dataset
bins = np.linspace(0,1000,1000)

l = widgets.Text(value='                           Boostrap Demonstration with Modified Number of Data, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

n = widgets.IntSlider(min = 2, max = 1000, value = 100, description = 'New Number Samples',orientation='horizontal',layout=Layout(width='800px', height='20px'))
n.style.handle_color = 'gray'

ui3 = widgets.VBox([l,n],)

def f_rerun(n):
    L = 1000                                 # set the number of realizations
    mean2 = np.zeros(L); stdev2 = np.zeros(L)    # declare arrays to hold the realizations of the statistics
    for l in range(0, L):                      # loop over realizations
        samples = random.choices(df['DataSet'].values, weights=None, cum_weights=None, k=n)
        mean2[l] = np.average(samples)
        stdev2[l] = np.std(samples)

    plt.subplot(121)
    plt.hist(mean,alpha=0.2,color="red",edgecolor="black",bins=np.linspace(0,100,40),label='Original'); plt.xlim(0,100); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Mean')
    plt.hist(mean2,alpha=0.2,color="blue",edgecolor="black",bins=np.linspace(0,100,40),label='New'); plt.xlim(0,100); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Mean')
    plt.legend()

    plt.subplot(122)
    plt.hist(stdev,alpha=0.2,color="red",edgecolor="black",bins=np.linspace(0,40,40),label='Original'); plt.xlim(0,40); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Standard Deviation')
    plt.hist(stdev2,alpha=0.2,color="blue",edgecolor="black",bins=np.linspace(0,40,40),label='New'); plt.xlim(0,40); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Standard Deviation')
    plt.legend()

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
    plt.show()

interactive_plot3 = widgets.interactive_output(f_rerun, {'n': n})
interactive_plot3.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

#### Display the GUI for Modifying the Number of Data

We display the GUI now. Select the desired number of data

* observed the change (new) vs. the original (original) bootstrap uncertainty as the number of data ranges from less than to greater than the original number of data

```python
display(ui3, interactive_plot3)                            # display the interactive plot
```

    VBox(children=(Text(value='                           Boostrap Demonstration with Modified Number of Data, Mic…

    Output()

#### Observations

Some observations form changing the number of data:

* with less data the uncertainty increases

* with more data the uncertianty decreases

* there is a bias low in standard deviation with too few samples, as we fail to observe the dispersion well

#### Comments

This was a simple demonstration of interactive plots in Jupyter Notebook Python with the ipywidgets and matplotlib packages.

I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


**SVM Example**

**A hyperplane in space is defined as**

$$\mathbf{w}^T\mathbf{x} + b = 0$$

**SVM Margin**

$$margin = \underset{x}{min}\; \frac{y_n(\mathbf{w}^Tx_n+b)}{{\left \| \mathbf{w} \right \|}_2}$$

_SVM optimization is the problem of finding $\mathbf{w}$ and $b$ such that $margin$ is maximized._
\begin{equation}
\tag{1}
(\mathbf{w},b) = arg\;\underset{\mathbf{w},b}{max}\left \{ \underset{n}{min}\;\frac{y_n(\mathbf{w}^Tx_n+b)}{{\left \| \mathbf{w} \right \|}_2} \right \}= arg\;\underset{\mathbf{w},b}{max}\left \{\frac{1}{{{\left \| \mathbf{w} \right \|}_2}} \underset{n}{min}\;y_n(\mathbf{w}^Tx_n+b) \right \}
\end{equation}

Note that: $\forall n, \; y_n(\mathbf{w}^Tx_n + b) \geq 1$

Eq. (1) can be turned to a constrained optimization problem as follows:
\begin{equation}
\tag{2}
(\mathbf{w},b) = arg\;\underset{\mathbf{w},b}{min} \frac{1}{2} {\left \| \mathbf{w} \right \|}_2^2
\end{equation}
subject to
$$1-y_n(\mathbf{w}^Tx_n+b) \leq 0,\; \forall n=1,2,...,N$$

### Lagrangian of SVM

Lagrangian of (2) is:
\begin{equation}
\tag{3}
\mathbf{L}(\mathbf{w},b,\lambda) = \frac{1}{2} {\left \| \mathbf{w} \right \|}_2^2 \;+\; \sum_{n=1}^{N}\lambda_n(1-y_n(\mathbf{w}^Tx_n+b))
\end{equation}
with
$\lambda=[\lambda_1,\lambda_2,...,\lambda_N]^T$ and $\lambda_n \geq 0,\;\forall n=1,2,...,N$

### SVM Dual Lagrangian function

\begin{equation}
\tag{4}
g(\lambda) = \underset{\mathbf{w},b}{min}\;\mathbf{L}(\mathbf{w},b,\lambda)
\end{equation}
with $$\lambda\succeq0$$

It's equivalent:
\begin{equation}
\nabla_{w,b,\lambda} \mathbf{L}(\mathbf{w},b,\lambda) = 0
\end{equation}

\begin{cases}
\frac{\partial\mathbf{L}(\mathbf{w}, b, \lambda)}{\partial_{w}} = \mathbf{w} - \sum_{n=1}^{N}\lambda_n\\
\frac{\partial\mathbf{L}(\mathbf{w}, b, \lambda)}{\partial_{b}}
\end{cases}

By solving (4), we get
\begin{equation}
\tag{5}
g(\lambda)=\sum_{n=1}^{N}\lambda_n - \frac{1}{2}\sum_{n=1}^{N}\sum_{m=1}^{N}\lambda_n\lambda_my_ny_m\mathbf{x}_n^T\mathbf{x}_m
\end{equation}

**Matrix Representation**

Set $\mathbf{V}=[y_1x_1,y_2x_2,...,y_Nx_N]$
and vector $\mathbf{1}=[1,1,...,1]^T$, $g(\lambda)$ in (5) can then be represented as:
\begin{equation}
\tag{6}
g(\lambda)=-\frac{1}{2}\lambda^T\mathbf{V}^T\mathbf{V}\lambda + \mathbf{1}^T\lambda
\end{equation}

Let $\mathbf{K}=\mathbf{V}^T\mathbf{V}$, we can prove that $\mathbf{K} \succeq 0$

(6) can be then represented as:
\begin{equation}
\tag{7}
g(\lambda) = -\frac{1}{2}\lambda^T\mathbf{K}\lambda + \mathbf{1}^T\lambda
\end{equation}

$g(\lambda)$ is a concave function

### SVM Dual Lagrangian problem

We want to solve the dual problem of (7), which is:
\begin{equation}
\tag{8}
\lambda=arg\; \underset{\lambda}{max}\;g(\lambda)
\end{equation}

subject to: $$\lambda \succeq 0$$
$$\sum_{n=1}^{N}\lambda_ny_n=0$$

We can prove that
\begin{equation}
\tag{9}
b = \frac{1}{N_{S}}\sum_{n\in S}(y_n-\mathbf{w}^Tx_n) = \frac{1}{N_{S}}\sum_{n\in S} \left ( y_n - \sum_{m \in S} \lambda_my_mx_m^Tx_n \right )
\end{equation}

\begin{equation}
\tag{10}
\mathbf{w} = \sum_{m \in S} \lambda_my_mx_m
\end{equation}

### Self-programming

```python
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import cdist
np.random.seed(22)

#Generate simulation data
means = [[2, 2], [4, 2]]
cov = [[.3, .2], [.2, .3]]
N = 10
X0 = np.random.multivariate_normal(means[0], cov, N) # class 1
X1 = np.random.multivariate_normal(means[1], cov, N) # class -1
X = np.concatenate((X0.T, X1.T), axis = 1) # all data
y = np.concatenate((np.ones((1, N)), -1*np.ones((1, N))), axis = 1) # labels
```

##### Solving equation (8) by using CVXOPT

```python
from cvxopt import matrix, solvers
# build K
V = np.concatenate((X0.T, -X1.T), axis = 1)

K = matrix(V.T.dot(V)) # see definition of V, K near eq (6)
p = matrix(-np.ones((2*N, 1))) # all-one vector
# build A, b, G, h

G = matrix(-np.eye(2*N)) # for all lambda_n >= 0
h = matrix(np.zeros((2*N, 1)))
A = matrix(y) # the equality constrain is actually y^T lambda = 0
print(A)
b = matrix(np.zeros((1, 1)))
print(b)
solvers.options['show_progress'] = False
sol = solvers.qp(K, p, G, h, A, b)

l = np.array(sol['x'])
print('lambda = ')
print(l.T)
```

    [ 1.00e+00  1.00e+00  1.00e+00  1.00e+00  1.00e+00  1.00e+00  1.00e+00 ... ]

    [ 0.00e+00]

    lambda =
    [[8.54018321e-01 2.89132533e-10 1.37095535e+00 6.36030818e-10
      4.04317408e-10 8.82390106e-10 6.35001881e-10 5.49567576e-10
      8.33359230e-10 1.20982928e-10 6.86678649e-10 1.25039745e-10
      2.22497367e+00 4.05417905e-09 1.26763684e-10 1.99008949e-10
      2.13742578e-10 1.51537487e-10 3.75329509e-10 3.56161975e-10]]

```python
epsilon = 1e-6 # just a small number, greater than 1e-9
print(np.where(l > epsilon))
S = np.where(l > epsilon)[0]
print(S)

VS = V[:, S]
print(V)
print(VS)
XS = X[:, S]
yS = y[:, S]
lS = l[S]
# calculate w and b
w = VS.dot(lS)
b = np.mean(yS.T - w.T.dot(XS))

print('w = ', w.T)
print('b = ', b)
```

    (array([ 0,  2, 12]), array([0, 0, 0]))
    [ 0  2 12]
    [[ 2.37319011  1.51261889  2.4696794   1.78736889  1.81231157  2.03717355
       1.53790057  2.29312867  1.38805594  1.57279694 -3.42746579 -4.24760864
      -3.33595491 -3.69420104 -4.53897645 -3.3071994  -4.13924705 -4.47383468
      -4.00512009 -4.28205624]
     [ 1.71875981  1.40558943  2.02144973  1.29380961  1.56119497  1.93397133
       1.87434722  2.76537389  1.86419379  0.90707347 -0.71254431 -2.39846497
      -1.61731637 -1.94273986 -2.54957308 -0.19362396 -2.09561534 -2.41269466
      -1.89290099 -1.79675607]]
    [[ 2.37319011  2.4696794  -3.33595491]
     [ 1.71875981  2.02144973 -1.61731637]]
    w =  [[-2.00984381  0.64068336]]
    b =  4.668560633868063

### Using scikit-learn

```python
from sklearn.svm import SVC

y1 = y.reshape((2*N,))
X1 = X.T # each sample is one row
clf = SVC(kernel = 'linear', C = 1e5) # just a big number

clf.fit(X1, y1)

w = clf.coef_
b = clf.intercept_
print('w = ', w)
print('b = ', b)
```

    w =  [[-2.00971102  0.64194082]]
    b =  [4.66595309]

```python

```