```python
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
%matplotlib inline
```

# Basic Concepts

## What is "learning from data"?

> In general **Learning from Data** is a scientific discipline that is concerned with the design and development of algorithms that allow computers to infer (from data) a  model that allows *compact representation* (unsupervised learning) and/or *good generalization* (supervised learning).

This is an important technology because it enables computational systems to adaptively improve their performance with experience accumulated from the observed data.

Most of these algorithms are based on the *iterative solution* of a mathematical problem that involves data and model. If there was an analytical solution to the problem, this should be the adopted one, but this is not the case for most of the cases.

So, the most common strategy for **learning from data** is based on solving a system of equations as a way to find a series of parameters of the model that minimizes a mathematical problem. This is called **optimization**.

The most important technique for solving optimization problems is **gradient descend**.

## Preliminary: Nelder-Mead method for function minimization.

The most simple thing we can try to minimize a function $f(x)$ would be to sample two points relatively near each other, and just repeatedly take a step down away from the largest value. This simple algorithm has a severe limitation: it can't get closer to the true minima than the step size.

The Nelder-Mead method dynamically adjusts the step size based off the loss of the new point. If the new point is better than any previously seen value, it **expands** the step size to accelerate towards the bottom. Likewise if the new point is worse it **contracts** the step size to converge around the minima. The usual settings are to half the step size when contracting and double the step size when expanding.

This method can be easily extended into higher dimensional examples, all that's required is taking one more point than there are dimensions. Then, the simplest approach is to replace the worst point with a point reflected through the centroid of the remaining n points. If this point is better than the best current point, then we can try stretching exponentially out along this line. On the other hand, if this new point isn't much better than the previous value, then we are stepping across a valley, so we shrink the step towards a better point.

> See "An Interactive Tutorial on Numerical Optimization": http://www.benfrederickson.com/numerical-optimization/

## Gradient descend (for *hackers*) for function minimization: 1-D

Let's suppose that we have a function $f: \Re \rightarrow \Re$. For example:

$$f(x) = x^2$$

Our objective is to find the argument  $x$ that minimizes this function (for maximization, consider $-f(x)$). To this end, the critical concept is the **derivative**.

The derivative of $f$ of a variable $x$, $f'(x)$ or $\frac{\mathrm{d}f}{\mathrm{d}x}$,  is a measure of the rate at which the value of the function changes with respect to the change of the variable. It is defined as the following limit:

$$ f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x)}{h} $$

The derivative specifies how to scale a small change in the input in order to obtain the corresponding change in the output:

$$ f(x + h) \approx f(x) + h f'(x)$$

```python
# numerical derivative at a point x

def f(x):
    return x**2

def fin_dif(x,
            f,
            h = 0.00001):
    '''
    This method returns the derivative of f at x
    by using the finite difference method
    '''
    return (f(x+h) - f(x))/h

x = 2.0
print "{:2.4f}".format(fin_dif(x,f))
```

The limit as $h$ approaches zero, if it exists, should represent the **slope of the tangent line** to $(x, f(x))$.

For values that are not zero it is only an approximation.

```python
for h in np.linspace(0.0, 1.0 , 5):
    print "{:3.6f}".format(f(5+h)), "{:3.6f}".format(f(5)+h*fin_dif(5,f))
```

```python
x = np.linspace(-1.5,-0.5, 100)
f = [i**2 for i in x]

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,f, 'r-')
plt.plot([-1.5, -0.5], [2, 0.0], 'k-', lw=2)
plt.plot([-1.4, -1.0], [1.96, 1.0], 'b-', lw=2)
plt.plot([-1],[1],'o')
plt.plot([-1.4],[1.96],'o')
plt.text(-1.0, 1.2, r'$x,f(x)$')
plt.text(-1.4, 2.2, r'$(x-h),f(x-h)$')
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

It can be shown that the “centered difference formula" is better when computing numerical derivatives:

$$ \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h} $$

The error in the "finite difference" approximation can be derived from Taylor's theorem and, assuming that $f$ is differentiable, is $O(h)$. In the case of “centered difference" the error is $O(h^2)$.

The derivative tells how to chage $x$ in order to make a small improvement in $f$.

Then, we can follow these steps to decrease the value of the function:

+ Start from a random $x$ value.
+ Compute the derivative $f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h}$.
+ Walk a small step (possibly weighted by the derivative module) in the **opposite** direction of the derivative, because we know that $f(x - h \mbox{ sign}(f'(x))$ is less than $f(x)$ for  small enough $h$.

The search for the minima ends when the derivative is zero because we have no more information about which direction to move. $x$ is a critical o stationary point if $f'(x)=0$.

 + A **minimum (maximum)** is a critical point where $f(x)$ is lower (higher) than at all neighboring points.
 + There is a third class of critical points: **saddle points**.

If $f$ is a **convex function**, this should be the minimum (maximum) of our functions. In other cases it could be a local minimum (maximum) or a saddle point.

```python
x = np.linspace(-15,15,100)
y = x**2

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        20,
        'Minimum',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

```python
x = np.linspace(-15,15,100)
y = -x**2

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-250,10])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        -30,
        'Maximum',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

```python
x = np.linspace(-15,15,100)
y = x**3

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-3000,3000])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        400,
        'Saddle Point',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

There are two problems with numerical derivatives:
+ It is approximate.
+ It is very slow to evaluate (two function evaluations: $f(x + h) , f(x - h)$ ).

Our knowledge from Calculus could help!

We know that we can get an **analytical expression** of the derivative for **some** functions.

For example, let's suppose we have a simple quadratic function, $f(x)=x^2−6x+5$, and we want to find the minimum of this function.

#### First approach

We can solve this analytically using Calculus, by finding the derivate $f'(x)  =  2x-6$ and setting it to zero:

\begin{equation}
\begin{split}
2x-6  & = & 0 \\
2x  & = & 6 \\
x & = & 3 \\
\end{split}
\end{equation}

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([3],[3**2 - 6*3 + 5],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(3,
        10,
        'Min: x = 3',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

#### Second approach

To find the local minimum using **gradient descend**: you start at a random point, and move into the direction of steepest **descent** relative to the derivative:

+ Start from a random $x$ value.
+ Compute the derivative $f'(x)$ analitically.
+ Walk a small step in the **opposite** direction of the derivative.

In this example, let's suppose we start at $x=15$. The derivative at this point is $2×15−6=24$.

Because we're using gradient descent, we need to subtract the gradient from our $x$-coordinate: $f(x - f'(x))$. However, notice that $15−24$ gives us $−9$, clearly overshooting over target of $3$.

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5
start = 15

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([start],[start**2 - 6*start + 5],'o')
ax.text(start,
        start**2 - 6*start + 35,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )

d = 2 * start - 6
end = start - d

plt.plot([end],[end**2 - 6*end + 5],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(end,
        start**2 - 6*start + 35,
        'End',
        ha='center',
        color=sns.xkcd_rgb['green'],
       )
plt.show
```

To fix this, we multiply the gradient by a step size. This step size (often called **alpha**) has to be chosen carefully, as a value too small will result in a long computation time, while a value too large will not give you the right result (by overshooting) or even fail to converge.

In this example, we'll set the step size to 0.01, which means we'll subtract $24×0.01$ from $15$, which is $14.76$.

This is now our new temporary local minimum: We continue this method until we either don't see a change after we subtracted the derivative step size, or until we've completed a pre-set number of iterations.

```python
old_min = 0
temp_min = 15
step_size = 0.01
precision = 0.0001

def f(x):
    return x**2 - 6*x + 5

def f_derivative(x):
    import math
    return 2*x -6

mins = []
cost = []

while abs(temp_min - old_min) > precision:
    old_min = temp_min
    gradient = f_derivative(old_min)
    move = gradient * step_size
    temp_min = old_min - move
    cost.append((3-temp_min)**2)
    mins.append(temp_min)

# rounding the result to 2 digits because of the step size
print "Local minimum occurs at {:3.6f}.".format(round(temp_min,2))
```

An important feature of gradient descent is that **there should be a visible improvement over time**: In this example, we simply plotted the squared distance from the local minima calculated by gradient descent and the true local minimum,  ``cost``, against the iteration during which it was calculated. As we can see, the distance gets smaller over time, but barely changes in later iterations.

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

x, y = (zip(*enumerate(cost)))

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-', alpha=0.7)
plt.ylim([-10,150])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.plot(mins,cost,'o', alpha=0.3)
ax.text(start,
        start**2 - 6*start + 25,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
ax.text(mins[-1],
        cost[-1]+20,
        'End (%s steps)' % len(mins),
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
plt.show
```

## From derivatives to gradient: $n$-dimensional function minimization.

Let's consider a $n$-dimensional function $f: \Re^n \rightarrow \Re$. For example:

$$f(\mathbf{x}) = \sum_{n} x_n^2$$

Our objective is to find the argument  $\mathbf{x}$ that minimizes this function.

The **gradient** of $f$ is the vector whose components are the $n$ partial derivatives of $f$. It is thus a vector-valued function.

The gradient points in the direction of the greatest rate of **increase** of the function.

$$\nabla {f} = (\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n})$$

```python
def f(x):
    return sum(x_i**2 for x_i in x)

def fin_dif_partial_centered(x,
                             f,
                             i,
                             h=1e-6):
    '''
    This method returns the partial derivative of the i-th component of f at x
    by using the centered finite difference method
    '''
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def fin_dif_partial_old(x,
                        f,
                        i,
                        h=1e-6):
    '''
    This method returns the partial derivative of the i-th component of f at x
    by using the (non-centered) finite difference method
    '''
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(x))/h

def gradient_centered(x,
                      f,
                      h=1e-6):
    '''
    This method returns the gradient vector of f at x
    by using the centered finite difference method
    '''
    return[round(fin_dif_partial_centered(x,f,i,h), 10) for i,_ in enumerate(x)]

def gradient_old(x,
                 f,
                 h=1e-6):
    '''
    This method returns the the gradient vector of f at x
    by using the (non-centered)ç finite difference method
    '''
    return[round(fin_dif_partial_old(x,f,i,h), 10) for i,_ in enumerate(x)]

x = [1.0,1.0,1.0]

print '{:.6f}'.format(f(x)), gradient_centered(x,f)
print '{:.6f}'.format(f(x)), gradient_old(x,f)
```

The function we have evaluated, $f({\mathbf x}) = x_1^2+x_2^2+x_3^2$, is $3$ at $(1,1,1)$ and the gradient vector at this point is $(2,2,2)$.

Then, we can follow this steps to maximize (or minimize) the function:

+ Start from a random $\mathbf{x}$ vector.
+ Compute the gradient vector.
+ Walk a small step in the opposite direction of the gradient vector.

> It is important to be aware that this gradient computation is very expensive: if $\mathbf{x}$ has dimension $n$, we have to evaluate $f$ at $2*n$ points.

### How to use the gradient.

$f(x) = \sum_i x_i^2$, takes its mimimum value when all $x$ are 0.

Let's check it for $n=3$:

```python
def euc_dist(v1,v2):
    import numpy as np
    import math
    v = np.array(v1)-np.array(v2)
    return math.sqrt(sum(v_i ** 2 for v_i in v))
```

Let's start by choosing a random vector and then walking a step in the opposite direction of the gradient vector. We will stop when the difference between the new solution and the old solution is less than a tolerance value.

```python
# choosing a random vector

import random
import numpy as np

x = [random.randint(-10,10) for i in range(3)]
x
```

```python
def step(x,
         grad,
         alpha):
    '''
    This function makes a step in the opposite direction of the gradient vector
    in order to compute a new value for the target function.
    '''
    return [x_i - alpha * grad_i for x_i, grad_i in zip(x,grad)]

tol = 1e-15
alpha = 0.01
while True:
    grad = gradient_centered(x,f)
    next_x = step(x,grad,alpha)
    if euc_dist(next_x,x) < tol:
        break
    x = next_x
print [round(i,10) for i in x]
```

### Alpha

The step size, **alpha**, is a slippy concept: if it is too small we will slowly converge to the solution, if it is too large we can diverge from the solution.

There are several policies to follow when selecting the step size:

+ Constant size steps. In this case, the size step determines the precision of the solution.
+ Decreasing step sizes.
+ At each step, select the optimal step.

The last policy is good, but too expensive. In this case we would consider a fixed set of values:

```python
step_size = [100, 10, 1, 0.1, 0.01, 0.001, 0.0001, 0.00001]
```

## Learning from data

In general, we have:

+ A dataset $(\mathbf{x},y)$ of $n$ examples.
+ A target function $f_\mathbf{w}$, that we want to minimize, representing the **discrepancy between our data and the model** we want to fit. The model is represented by a set of parameters $\mathbf{w}$.
+ The gradient of the target function, $g_f$.

In the most common case $f$ represents the errors from a data representation model $M$. To fit the model is to find the optimal parameters $\mathbf{w}$ that minimize the following expression:

$$ f_\mathbf{w} = \frac{1}{n} \sum_{i} (y_i - M(\mathbf{x}_i,\mathbf{w}))^2 $$

For example, $(\mathbf{x},y)$ can represent:

+ $\mathbf{x}$: the behavior of a "Candy Crush" player; $y$: monthly payments.
+ $\mathbf{x}$: sensor data about your car engine; $y$: probability of engine error.
+ $\mathbf{x}$: finantial data of a bank customer; $y$: customer rating.

> If $y$ is a real value, it is called a *regression* problem.

> If $y$ is binary/categorical, it is called a *classification* problem.

Let's suppose that our model is a one-dimensional linear model $M(\mathbf{x},\mathbf{w}) = w \cdot x $.

### Batch gradient descend

We can implement **gradient descend** in the following way (*batch gradient descend*):

```python
import numpy as np
import random

# f = 2x
x = np.arange(10)
y = np.array([2*i for i in x])

# f_target = 1/n Sum (y - wx)**2
def target_f(x,y,w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,y,w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def step(w,grad,alpha):
    return w - alpha * grad

def BGD_multi_step(target_f,
                   gradient_f,
                   x,
                   y,
                   toler = 1e-6):
    '''
    Batch gradient descend by using a multi-step approach
    '''
    alphas = [100, 10, 1, 0.1, 0.001, 0.00001]
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_ws = [step(w, gradient, alpha) for alpha in alphas]
        next_vals = [target_f(x,y,w) for w in next_ws]
        min_val = min(next_vals)
        next_w = next_ws[next_vals.index(min_val)]
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
```

```python
print '{:.6f}'.format(BGD_multi_step(target_f, gradient_f, x, y))
```

```python
%%timeit
BGD_multi_step(target_f, gradient_f, x, y)
```

```python
def BGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        alpha=0.01):
    '''
    Batch gradient descend by using a given step
    '''
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_w = step(w, gradient, alpha)
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
```

```python
print '{:.6f}'.format(BGD(target_f, gradient_f, x, y))
```

```python
%%timeit
BGD(target_f, gradient_f, x, y)
```

### Stochastic Gradient Descend

The last function evals the whole dataset $(\mathbf{x}_i,y_i)$ at every step.

If the dataset is large, this strategy is too costly. In this case we will use a strategy called **SGD** (*Stochastic Gradient Descend*).

When learning from data, the cost function is additive: it is computed by adding sample reconstruction errors.

Then, we can compute the estimate the gradient (and move towards the minimum) by using only **one data sample** (or a small data sample).

Thus, we will find the minimum by iterating this gradient estimation over the dataset.

A full iteration over the dataset is called **epoch**. During an epoch, data must be used in a random order.

If we apply this method we have some theoretical guarantees to find a good minimum:
+ SGD essentially uses the inaccurate gradient per iteration. Since there is no free food, what is the cost by using approximate gradient? The answer is that the convergence rate is slower than the gradient descent algorithm.
+ The convergence of SGD has been analyzed using the theories of convex minimization and of stochastic approximation: it converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum.

```python
import numpy as np
x = np.arange(10)
y = np.array([2*i for i in x])
data = zip(x,y)

for (x_i,y_i) in data:
    print '{:3d} {:3d}'.format(x_i,y_i)
print

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

for (x_i,y_i) in in_random_order(data):
    print '{:3d} {:3d}'.format(x_i,y_i)
```

```python
import numpy as np
import random

def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):
    '''
    Stochastic gradient descend with automatic step adaptation (by
    reducing the step to its 95% when there are iterations with no increase)
    '''
    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    epoch = 0
    iteration_no_increase = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w
```

```python
print 'w: {:.6f}'.format(SGD(target_f, gradient_f, x, y))
```

## Exercise: Stochastic Gradient Descent and Linear Regression

The linear regression model assumes a linear relationship between data:

$$ y_i = w_1 x_i + w_0 $$

Let's generate a more realistic dataset (with noise), where $w_1 = 2$ and $w_0 = 0$.

```python
%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

```python
# x: input data
# y: noisy output data

x = np.random.uniform(0,1,20)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$f(x)$', fontsize=15)
plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

Complete the following code in order to:
+ Compute the value of $w$ by using a estimator based on minimizing the squared error.
+ Get from SGD function a vector, `target_value`, representing the value of the target function at each iteration.

```python
# Write your target function as f_target 1/n Sum (y - wx)**2
def target_f(x,y,w):
    # your code here

# Write your gradient function
def gradient_f(x,y,w):
    # your code here

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

# Modify the SGD function to return a 'target_value' vector
def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):

    # Insert your code among the following lines

    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    iteration_no_increase = 0
    epoch = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w
```

```python
# Print the value of the solution

w, target_value = SGD(target_f, gradient_f, x, y)
print 'w: {:.6f}'.format(w)
```

```python
# Visualize the solution regression line

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x, y, 'o', label='t')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)', alpha=0.5)
plt.plot([0, 1], [0*w, 1*w], 'r-', label='fitted line', alpha=0.5, linestyle='--')
plt.xlabel('input x')
plt.ylabel('target t')
plt.title('input vs. target')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

```python
# Visualize the evolution of the target function value during iterations.

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(np.arange(target_value.size), target_value, 'o', alpha = 0.2)
plt.xlabel('Iteration')
plt.ylabel('Cost')
plt.grid()
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show()
```

## Mini-batch Gradient Descent

In code, general batch gradient descent looks something like this:

```python
nb_epochs = 100
for i in range(nb_epochs):
    grad = evaluate_gradient(target_f, data, w)
    w = w - learning_rate * grad
```

For a pre-defined number of epochs, we first compute the gradient vector of the target function for the whole dataset w.r.t. our parameter vector.

**Stochastic gradient descent** (SGD) in contrast performs a parameter update for each training example and label:

```python
nb_epochs = 100
for i in range(nb_epochs):
    np.random.shuffle(data)
    for sample in data:
        grad = evaluate_gradient(target_f, sample, w)
        w = w - learning_rate * grad
```

Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of $n$ training examples:

```python
nb_epochs = 100
for i in range(nb_epochs):
  np.random.shuffle(data)
  for batch in get_batches(data, batch_size=50):
    grad = evaluate_gradient(target_f, batch, w)
    w = w - learning_rate * grad
```

Minibatch SGD has the advantage that it works with a slightly less noisy estimate of the gradient. However, as the minibatch size increases, the number of updates done per computation done decreases (eventually it becomes very inefficient, like batch gradient descent).

There is an optimal trade-off (in terms of computational efficiency) that may vary depending on the data distribution and the particulars of the class of function considered, as well as how computations are implemented.

```python
def get_batches(iterable,
                num_elem_batch = 1):
    '''
    Generator of batches from an iterable that contains data
    '''
    current_batch = []
    for item in iterable:
        current_batch.append(item)
        if len(current_batch) == num_elem_batch:
            yield current_batch
            current_batch = []
    if current_batch:
        yield current_batch

x = np.array(range(0, 10))
y = np.array(range(10, 20))
data = zip(x,y)
np.random.shuffle(data)
for x in get_batches(data, 3):
    print x

print

for batch in get_batches(data, 3):
    print np.array(zip(*batch)[0]), np.array(zip(*batch)[1])
```

```python
%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

```python
# x: input data
# y: noisy output data

x = np.random.uniform(0,1,2000)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$t$', fontsize=15)
plt.ylim([0,2])
plt.title('inputs (x) vs targets (y)')
plt.grid()
plt.legend(loc=2)
plt.gcf().set_size_inches((10,3))
plt.show()
```

```python
# f_target = 1/n Sum (y - wx)**2
def target_f(x,
             y,
             w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,
               y,
               w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

def get_batches(iterable,
                num_elem_batch = 1):
    '''
    Generator of batches from an iterable that contains data
    '''
    current_batch = []
    for item in iterable:
        current_batch.append(item)
        if len(current_batch) == num_elem_batch:
            yield current_batch
            current_batch = []
    if current_batch:
        yield current_batch

def SGD_MB(target_f, gradient_f, x, y, epochs=100, alpha_0=0.01):
    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    epoch = 0
    while epoch < epochs:
        val = target_f(x, y, w)
        if val < min_val:
            min_w, min_val = w, val
            alpha = alpha_0
        else:
            alpha *= 0.9
        np.random.shuffle(data)
        for batch in get_batches(data, num_elem_batch = 100):
            x_batch = np.array(zip(*batch)[0])
            y_batch = np.array(zip(*batch)[1])
            gradient = gradient_f(x_batch, y_batch, w)
            w = w - (alpha *  gradient)
        epoch += 1
    return min_w
```

```python
w = SGD_MB(target_f, gradient_f, x, y)
print 'w: {:.6f}'.format(w)
```

```python
plt.plot(x, y, 'o', label='t')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)', alpha=0.5)
plt.plot([0, 1], [0*w, 1*w], 'r-', label='fitted line', alpha=0.5, linestyle='--')
plt.xlabel('input x')
plt.ylabel('target t')
plt.ylim([0,2])
plt.title('input vs. target')
plt.grid()
plt.legend(loc=2)
plt.gcf().set_size_inches((10,3))
plt.show()
```

## Loss Funtions

Loss functions $L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \ell(y_i, f(\mathbf{x_i}))$ represent the price paid for inaccuracy of predictions in classification/regression problems.

In classification this function is often the **zero-one loss**, that is, $ \ell(y_i, f(\mathbf{x_i}))$ is zero when $y_i = f(\mathbf{x}_i)$ and one otherwise.

This function is discontinuous with flat regions and is thus extremely hard to optimize using gradient-based methods. For this reason it is usual to consider a proxy to the loss called a *surrogate loss function*. For computational reasons this is usually convex function. Here we have some examples:

### Square / Euclidean Loss

In regression problems, the most common loss function is the square loss function:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (y_i - f(\mathbf{x}_i))^2  $$

The square loss function can be re-written and utilized for classification:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (1 - y_i f(\mathbf{x}_i))^2  $$

### Hinge / Margin Loss (i.e. Suport Vector Machines)

The hinge loss function is defined as:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \mbox{max}(0, 1 - y_i f(\mathbf{x}_i))  $$

The hinge loss provides a relatively tight, convex upper bound on the 0–1 Loss.

### Logistic Loss (Logistic Regression)

This function displays a similar convergence rate to the hinge loss function, and since it is continuous, simple gradient descent methods can be utilized.

$$ L(y, f(\mathbf{x})) = \frac{1}{n} log(1 + exp(-y_i f(\mathbf{x}_i))) $$

### Sigmoid Cross-Entropy Loss (Softmax classifier)

Cross-Entropy is a loss function that is very used for training **multiclass problems**. We'll focus on models that assume that classes are mutually exclusive.

In this case, our labels have this form $\mathbf{y}_i =(1.0,0.0,0.0)$. If our model predicts a different distribution, say  $ f(\mathbf{x}_i)=(0.4,0.1,0.5)$, then we'd like to nudge the parameters so that $f(\mathbf{x}_i)$ gets closer to $\mathbf{y}_i$.

C.Shannon showed that if you want to send a series of messages composed of symbols from an alphabet with distribution $y$ ($y_j$  is the probability of the  $j$-th symbol), then to use the smallest number of bits on average, you should assign  $\log(\frac{1}{y_j})$  bits to the  $j$-th symbol.

The optimal number of bits is known as **entropy**:

$$ H(\mathbf{y}) = \sum_j y_j \log\frac{1}{y_j} = - \sum_j y_j \log y_j$$

**Cross entropy** is the number of bits we'll need if we encode symbols by using a wrong distribution $\hat y$:

$$ H(y, \hat y) =   - \sum_j y_j \log \hat y_j $$

In our case, the real distribution is $\mathbf{y}$ and the "wrong" one is $f(\mathbf{x}_i)$. So, minimizing **cross entropy** with respect our model parameters will result in the model that best approximates our labels if considered as a probabilistic distribution.

Cross entropy is used in combination with **Softmax** classifier. In order to classify $\mathbf{x}_i$ we could take the index corresponding to the max value of $f(\mathbf{x}_i)$, but Softmax gives a slightly more intuitive output (normalized class probabilities) and also has a probabilistic interpretation:

$$ P(\mathbf{y}_i = j \mid \mathbf{x_i}) = - log \left( \frac{e^{f_j(\mathbf{x_i})}}{\sum_k e^{f_k(\mathbf{x_i})} } \right) $$

where $f_k$ is a linear classifier.

## Advanced gradient descend

### Momentum

SGD has trouble navigating ravines, i.e. areas where the surface curves much more steeply in one dimension than in another, which are common around local optima. In these scenarios, SGD oscillates across the slopes of the ravine while only making hesitant progress along the bottom towards the local optimum.

Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations. It does this by adding a fraction of the update vector of the past time step to the current update vector:

$$ v_t = m v_{t-1} + \alpha \nabla_w f $$
$$ w = w - v_t    $$

The momentum $m$ is commonly set to $0.9$.

### Nesterov

However, a ball that rolls down a hill, blindly following the slope, is highly unsatisfactory. We'd like to have a smarter ball, a ball that has a notion of where it is going so that it knows to slow down before the hill slopes up again.

Nesterov accelerated gradient (NAG) is a way to give our momentum term this kind of prescience. We know that we will use our momentum term $m v_{t-1}$ to move the parameters $w$. Computing
$w - m v_{t-1}$ thus gives us an approximation of the next position of the parameters (the gradient is missing for the full update), a rough idea where our parameters are going to be. We can now effectively look ahead by calculating the gradient not w.r.t. to our current parameters $w$ but w.r.t. the approximate future position of our parameters:

$$ w_{new} = w - m v_{t-1} $$
$$ v_t = m v_{t-1} + \alpha \nabla_{w_{new}} f $$
$$ w = w - v_t $$

### Adagrad

All previous approaches manipulated the learning rate globally and equally for all parameters. Tuning the learning rates is an expensive process, so much work has gone into devising methods that can adaptively tune the learning rates, and even do so per parameter.

Adagrad is an algorithm for gradient-based optimization that does just this: It adapts the learning rate to the parameters, performing larger updates for infrequent and smaller updates for frequent parameters.

$$ c = c + (\nabla_w f)^2 $$
$$ w = w - \frac{\alpha}{\sqrt{c}} $$

### RMProp

RMSProp update adjusts the Adagrad method in a very simple way in an attempt to reduce its aggressive, monotonically decreasing learning rate. In particular, it uses a moving average of squared gradients instead, giving:

$$ c = \beta c + (1 - \beta)(\nabla_w f)^2 $$
$$ w = w - \frac{\alpha}{\sqrt{c}} $$

where $\beta$ is a decay rate that controls the size of the moving average.

(Image credit: Alec Radford)

(Image credit: Alec Radford)

```python
%reset

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

```python
# the function that I'm going to plot
def f(x,y):
    return x**2 + 5*y**2

x = np.arange(-3.0,3.0,0.1)
y = np.arange(-3.0,3.0,0.1)
X,Y = np.meshgrid(x, y, indexing='ij') # grid of point
Z = f(X, Y) # evaluation of the function on the grid

plt.pcolor(X, Y, Z, cmap=plt.cm.gist_earth)
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.gca().set_aspect('equal', adjustable='box')
plt.gcf().set_size_inches((6,6))
plt.show()
```

```python
def target_f(x):
    return x[0]**2.0 + 5*x[1]**2.0

def part_f(x,
           f,
           i,
           h=1e-6):
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def gradient_f(x,
               f,
               h=1e-6):
    return np.array([round(part_f(x,f,i,h), 10) for i,_ in enumerate(x)])
```

```python
def SGD(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001):
    alpha = alpha_0
    min_val = float('inf')
    steps = 0
    iteration_no_increase = 0
    trace = []
    while iteration_no_increase < 100:
        val = target_f(x)
        if min_val - val > toler:
            min_val = val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            alpha *= 0.95
            iteration_no_increase += 1
        trace.append(x)
        gradient_i = gradient_f(x, target_f)
        x = x - (alpha *  gradient_i)
        steps += 1
    return x, val, steps, trace

x = np.array([2,-2])
x, val, steps, trace = SGD(target_f, gradient_f, x)
print x
print 'Val: {:.6f}, steps: {:.0f}'.format(val, steps)
```

```python
def SGD_M(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001,
        m = 0.9):

    alpha = alpha_0
    min_val = float('inf')
    steps = 0
    iteration_no_increase = 0
    v = 0.0
    trace = []
    while iteration_no_increase < 100:
        val = target_f(x)
        if min_val - val > toler:
            min_val = val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            alpha *= 0.95
            iteration_no_increase += 1
        trace.append(x)
        gradient_i = gradient_f(x, target_f)
        v = m * v + (alpha *  gradient_i)
        x = x - v
        steps += 1
    return x, val, steps, trace

x = np.array([2,-2])
x, val, steps, trace2 = SGD_M(target_f, gradient_f, x)

print '\n',x
print 'Val: {:.6f}, steps: {:.0f}'.format(val, steps)
```

```python
x2 = np.array(range(len(trace)))
x3 = np.array(range(len(trace2)))
plt.xlim([0,len(trace)])
plt.gcf().set_size_inches((10,3))
plt.plot(x3, trace2)
plt.plot(x2, trace, '-')
```


Text provided under a Creative Commons Attribution license, CC-BY.  All code is made available under the FSF-approved BSD-3 license.  (c) Lorena A. Barba, Gilbert F. Forsyth 2017. Thanks to NSF for support via CAREER award #1149784.
[@LorenaABarba](https://twitter.com/LorenaABarba)

12 steps to Navier-Stokes
=====
***

You see where this is going ... we'll do 2D diffusion now and next we will combine steps 6 and 7 to solve Burgers' equation. So make sure your previous steps work well before continuing.

Step 7: 2D Diffusion
----
***

And here is the 2D-diffusion equation:

$$\frac{\partial u}{\partial t} = \nu \frac{\partial ^2 u}{\partial x^2} + \nu \frac{\partial ^2 u}{\partial y^2}$$

You will recall that we came up with a method for discretizing second order derivatives in Step 3, when investigating 1-D diffusion.  We are going to use the same scheme here, with our forward difference in time and two second-order derivatives.

$$\frac{u_{i,j}^{n+1} - u_{i,j}^n}{\Delta t} = \nu \frac{u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n}{\Delta x^2} + \nu \frac{u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n}{\Delta y^2}$$

Once again, we reorganize the discretized equation and solve for $u_{i,j}^{n+1}$

\begin{align}
u_{i,j}^{n+1} = u_{i,j}^n &+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n) \\
&+ \frac{\nu \Delta t}{\Delta y^2}(u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n)
\end{align}

```python
import numpy
from matplotlib import pyplot, cm
from mpl_toolkits.mplot3d import Axes3D ##library for 3d projection plots
%matplotlib inline
```

```python
###variable declarations
nx = 31
ny = 31
nt = 17
nu = .05
dx = 2.0 / (nx - 1)
dy = 2.0 / (ny - 1)
sigma = .25
dt = sigma * dx * dy / nu

x = numpy.linspace(0, 2, nx)
y = numpy.linspace(0, 2, ny)

u = numpy.ones((ny, nx))  # create a 1xn vector of 1's
un = numpy.ones((ny, nx))

###Assign initial conditions
# set hat function I.C. : u(.5<=x<=1 && .5<=y<=1 ) is 2
u[int(.5 / dy):int(1 / dy + 1),int(.5 / dx):int(1 / dx + 1)] = 2
```

```python
fig = pyplot.figure()
ax = fig.gca(projection='3d')
X, Y = numpy.meshgrid(x, y)
surf = ax.plot_surface(X, Y, u, rstride=1, cstride=1, cmap=cm.viridis,
        linewidth=0, antialiased=False)

ax.set_xlim(0, 2)
ax.set_ylim(0, 2)
ax.set_zlim(1, 2.5)

ax.set_xlabel('$x$')
ax.set_ylabel('$y$');
```

\begin{align}
u_{i,j}^{n+1} = u_{i,j}^n &+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n) \\
&+ \frac{\nu \Delta t}{\Delta y^2}(u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n)
\end{align}

```python
###Run through nt timesteps
def diffuse(nt):
    u[int(.5 / dy):int(1 / dy + 1),int(.5 / dx):int(1 / dx + 1)] = 2

    for n in range(nt + 1):
        un = u.copy()
        u[1:-1, 1:-1] = (un[1:-1,1:-1] +
                        nu * dt / dx**2 *
                        (un[1:-1, 2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                        nu * dt / dy**2 *
                        (un[2:,1: -1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))
        u[0, :] = 1
        u[-1, :] = 1
        u[:, 0] = 1
        u[:, -1] = 1

    fig = pyplot.figure()
    ax = fig.gca(projection='3d')
    surf = ax.plot_surface(X, Y, u[:], rstride=1, cstride=1, cmap=cm.viridis,
        linewidth=0, antialiased=True)
    ax.set_zlim(1, 2.5)
    ax.set_xlabel('$x$')
    ax.set_ylabel('$y$');

```

```python
diffuse(10)
```

```python
diffuse(14)
```

```python
diffuse(50)
```

## Learn More

The video lesson that walks you through the details for Steps 5 to 8 is **Video Lesson 6** on You Tube:

```python
from IPython.display import YouTubeVideo
YouTubeVideo('tUg_dE3NXoY')
```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#CD2305;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #CD2305;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

> (The cell above executes the style for this notebook.)


# Übungsblatt 4: MCMC

 * [Aufgabe 1](#Aufgabe-1)
 * [Aufgabe 2](#Aufgabe-2)

---

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
from functools import partial

plt.style.use('ggplot')
```

---
## Aufgabe 1
Erzeugen Sie Paare von Zufallszahlen (x, y) die gemäß der zweidimensionalen Dichte

\begin{equation}
    f(x, y) \propto \exp\left(-((x-2)x + y^2)/10\right)\cdot\sin^2(x\cdot y + x)
\end{equation}

verteilt sind. Bestimmen Sie daraus die Verteilung $\rho(r)$ mit $r^2 = x^2 + y^2$.

---

```python
def mcmc_step(pdf, xs):
    ds = np.random.uniform(-3, 3, size=len(xs))
    r = np.random.uniform()
    nxs = xs + ds
    p0 = pdf(xs)
    p1 = pdf(nxs)
    if p1 > p0 or r < (p1 / p0):
        return nxs
    return xs

def mcmc_sample(pdf, xs, size=100000):
    xss = []
    for _ in range(size):
        xs = mcmc_step(pdf, xs)
        xss.append(xs)
    return np.array(xss).T

def mcmc_generate(pdf, xs, size=100000):
    ds = np.random.uniform(-3, 3, size=(size, len(xs)))
    r = np.random.uniform(size=size)
    p0 = pdf(xs)
    xss = []
    for i in range(size):
        nxs = xs + ds[i]
        p1 = pdf(nxs)
        if p1 > p0 or r[i] < (p1 / p0):
            xs = nxs
            p0 = p1
        xss.append(xs)
    return np.array(xss).T

def pdf(xs):
    x, y = xs
    return np.exp(-((x - 2) * x + y ** 2) / 10) * np.sin(x * y + x) ** 2

%time xs, ys = mcmc_sample(pdf, np.array([0, 0]))
%time xs, ys = mcmc_generate(pdf, np.array([0, 0]))
```

    CPU times: user 1.34 s, sys: 33.3 ms, total: 1.37 s
    Wall time: 1.41 s
    CPU times: user 633 ms, sys: 18.3 ms, total: 652 ms
    Wall time: 687 ms

```python
plt.hist2d(xs, ys, bins=40)
plt.show()

plt.hist(np.sqrt(xs ** 2 + ys ** 2), bins=40)
plt.show()
```

---
## Aufgabe 2
Ein Experiment beobachtet $n$ Ereignisse. Die Nachweiswahrscheinlichkeit für ein Ereignis sei $ε$. Die wahre Anzahl ist Poisson-verteilt mit einem Mittelwert $μ$, der eine Funktion zweier Theorieparameter $a$ und $b$ ist, $μ = a^b$. Aus Kalibrationsmessungen ist bekannt dass $ε$ Gauss-verteilt ist mit $ε = 0.75 \pm 0.05$. Vom Parameter $b$ ist bekannt dass er nahe bei $b = 1$ liegt. Für ihn wird eine gleichförmige Verteilung im Bereich $0.9 < b < 1.1$ angenommen. Der Parameter $a$ muss positiv sein. Verwenden Sie bayes’sche Statistik und MCMC, und bestimmen Sie für $n = 1$ und $n = 10$ die posterior Verteilung von $a$ nach Integration über die Nuissance-Parameter $ε$ und $b$.

---

Zunächst müssen wir berechnen, wie die Effizienz die Verteilung der $n$ beeinflusst. Es sei $q_n$ die Wahrscheinlichkeit $n$ Ereignisse zu beobachten, wenn die $n$ um einen Mittelwert $\mu$ Poisson-verteilt sind und mit einer Effizienz $\varepsilon$ beobachtet werden. Es kann also sein, dass genau $n$ Ereignisse auftreten und alle beobachtet werden, oder $n+1$ beobachtet werden und eins nicht usw. Das bedeutet

\begin{align}
    q_n &= \sum_{k=n}^\infty \mathrm{Poisson}(k, \mu) \binom{k}{n} \varepsilon^n (1-\varepsilon)^{k-n} \\
        &= \sum_{k=n}^\infty \exp(-\mu) \frac{\mu^k}{k!} \frac{k!}{n!(k-n)!} \varepsilon^n (1-\varepsilon)^{k-n} \\
        &= \exp(-\mu) \frac{\mu^n}{n!} \sum_{k=n}^\infty \frac{\mu^k (1-\varepsilon)^{k-n}}{(k-n)!} \\
        &= \exp(-\mu) \frac{(\varepsilon\mu)^n}{n!} \sum_{k=n}^\infty \frac{(\mu(1-\varepsilon)^{k-n}}{(k-n)!} \\
        &= \exp(-\mu) \frac{(\varepsilon\mu)^n}{n!} \sum_{k=0}^\infty \frac{(\mu(1-\varepsilon)^{k}}{(k)!} \\
        &= \exp(-\mu) \frac{(\varepsilon\mu)^n}{n!} \exp(\mu(1-\varepsilon)) \\
        &= \exp(-\varepsilon\mu) \frac{(\varepsilon\mu)^n}{n!} \\
        &= \mathrm{Poisson}(n, \varepsilon\mu) \\
\end{align}

Damit ist

\begin{equation}
    p(n\,|\,a,b,\varepsilon) \propto \exp\left(\mu\varepsilon\right)\cdot\left(\mu\varepsilon\right)^n \quad\text{mit}\quad \mu = a^b \,.
\end{equation}

Mit Hilfe von Bayes Theorem können wir nun die Posteriorverteilung der unbekannten Parameter $\left\{a,b,\varepsilon\right\}$ ausdrücken. Es ist

\begin{equation}
    p(a, b, \varepsilon\,|\,n) \propto p(n\,|\,a, b, \varepsilon) \cdot p(a, b, \varepsilon)
\end{equation}

mit der Priorwahrscheinlichkeitsdichte

\begin{equation}
    p(a, b, \varepsilon) \propto \Theta(a) \cdot \Theta(b - 0.9) \cdot \Theta(1.1 - b) \cdot \exp\left(-200(\varepsilon - 0.75)^2\right) \,.
\end{equation}

Dieser Ausdruck wird jetzt per MCMC über $\varepsilon$ und $b$ integriert um die Verteilung von $p(a\,|\,n)$ zu erhalten.

```python
uniform_b = scipy.stats.uniform(0.9, 1.1).pdf
gaussian_eps = scipy.stats.norm(0.75, 0.05).pdf

def p(a, n, eps, b):
    if a <= 0:
        return 0
    epsab = eps * a**b
    return scipy.stats.poisson.pmf(n, epsab) * uniform_b(b) * gaussian_eps(eps)
```

Diese können wir dann bequem mit unseren, in Aufgabe 1 formulierten, MCMC-Funktionen verwenden. Wir müssen uns nur einen kleinen Wrapper schreiben.

```python
def p_n(n, pars):
    a, eps, b = pars
    return p(a, n, eps, b)

p_1 = partial(p_n, 1)
as1, _, _ = mcmc_generate(p_1, np.array([0.01, 0.75, 1]), size=1000000)

plt.hist(as1, bins=40)
plt.title('$n = 1$')
plt.xlabel('$a$')
plt.ylabel('Absolute Häufigkeit')
plt.show()
```

```python
p_10 = partial(p_n, 10)
as10, _, _ = mcmc_generate(p_10, np.array([0.01, 0.75, 1]), size=1000000)

plt.hist(as10, bins=40)
plt.title('$n = 10$')
plt.xlabel('$a$')
plt.ylabel('Absolute Häufigkeit')
plt.show()
```


# HW 1: Theory

This particular homework will not require you to turn in extraneous code so just write in your answers into this Jupyter notebook.  Please note that there is a bit of literature searching that I am requiring you to do though and this will **absolutely** require you to cite what you find.  Citations must be done properly, not just URLs.  If you have any issues doing so please reach out to [me](http://www.columbia.edu/~ktm2132/) or your [local friendly librarian](https://library.columbia.edu).

**Name: Guanhua Yu**

**Uni:gy2278**

## 1 - Amdahl's Law

**(a)** [15] Consider the following bit of code and estimate the proportion of serial vs. parallelizable code.
```python
a = empty(N)
b = empty(N)
for i in range(N):
    a[i] = f(i)

for i in range(N):
    b[i] = g(i, a)
```
Assume in this case that $g(\cdot)$ is a function that can be run independently from other functions while $f(\cdot)$ cannot.  Your solution may need to depend on $N$.

Explain your answer by annotating the code above.

From the condition above we know that the first two lines of the code that initialize the array and the first  $for$  loop which calculates each entry of $a[]$ is the serial part of this piece of code. The second  $for$  loop which calculates the value of $b[]$ is the parallelizable part.

For the time of the initialize part, we can suppose it to be contant like 2(=1+1) units time . The first  $for$  loop takes $N*C_f$ time, where $C_f$ is the time calculating $f$ once. The second loop takes $N*C_g$ time.

Thus the proportion of serial part:
\begin{equation}
    s = \frac{2+C_f*N}{2+C_f*N+C_g*N},
\end{equation}

the proportion of parallelizable part:
\begin{equation}
    1-s = \frac{C_g*N}{2+C_f*N+C_g*N},
\end{equation}

**(b)** [15] Plot the number of processes vs. the resulting expectation of efficiency/speedup and compare to the limits of Amdahl's law.

In this part for plotting, we will take $C_f = 1$, in other words, calculating function f once takes one unit time (for convinence). And $C_g=N+1$, as compared to $f$, $g$ has N more variables. Under these assumptions,
\begin{equation}
    s = \frac{2+N}{2+N+N(N+1)} = \frac{2+N}{N^2+2N+2},
\end{equation}
and
\begin{equation}
    1-s = \frac{N^2+N}{2+N+N(N+1)} = \frac{N^2+N}{N^2+2N+2}.
\end{equation}

We will give the plot in which $x-$axis is the number of processes $p$ while $y-$axis is the expectation speedup $es$, when $N$ from 1 to 10.

```python
import numpy as np
from matplotlib import pyplot as plt

def s(n):
    return (2+n)/(n*n+2*n+2)

def speedup(s,p):
    return 1/(s+(1-s)/p)

plt.title("Amdahl's Law")
plt.xlabel("Number of Processes")
plt.ylabel("Expectation Speedup")

p = [1]
temp = 1
for _ in range(8):
    temp = temp*2
    p.append(temp)

for n in range(1,11):
    serial = s(n)
    print("1/s = %.2f, for N = %d"%(1/serial,n))
    es = []
    for pi in p:
        es.append(speedup(serial,pi))
    plt.plot(p,es)

plt.show()

```

Form the printed result and the plot we can find that the upper bound of speedup is limited by $1/s$ as Amdal's law says.

**(c)** [10] Confirm the importance of reducing the amount of serial $S$ code by plotting $S$ vs. efficiency/speedup using Amdahl's Law.

In this part we will take serial code $S$ from 100% to 10% with different number of processes.

```python
s = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]
p = 1

plt.title("Serial VS. Speedup")
plt.xlabel("Serial proportion")
plt.ylabel("Expectation Speedup")
while p<=65536:
    es = []
    for si in s:
        es.append(speedup(si,p))
    plt.plot(s,es)
    p =p*2
plt.show()
```

From the plot we know that by decreasing the proportion of serial part of a program, the speedup will become better and better.

## 2 - Roofline Models

**(a)** [15] Pick a _specific_ chip and compute a roof-line model for it.  Feel free to make some assumptions as you will probably need to get the true memory bandwidth to main-memory.  Plot this roof-line model and justify the model.

For this part we will choose NVIDIA GeForce GTX 1080 Ti: maximum FLOPs per second:$11.3TFLOP/s$, maximum memory acess per second: $484GB/s$.
So in this case:
\begin{equation}
    \pi = 11.3TFLOP/s,\qquad \beta = 484GB/s.
\end{equation}
So,
\begin{equation}
    I = \frac{\pi}{\beta} = 23.9074
\end{equation}

```python
plt.title("Roofline Model for Nvidia GTX 1080ti")
plt.xlabel("Operation Intensity (FLOPS/byte)")
plt.ylabel("Performance(TFLOPs)")

def roofline(x):
    res = []
    sep = 11.3*1024/484.0
    for xi in x:
        if xi<=sep:
            res.append(11.3/sep*xi)
        else:
            res.append(11.3)

    return res

x = np.linspace(0,50,1000)

y = roofline(x)
plt.plot(x,y)
sep = 11.3*1024/484.0
plt.annotate('(23.9074,11.3)', xy=(sep, 11.3), xytext=(30, 6),arrowprops=dict(facecolor='black', shrink=0.05,width = 1))

plt.show()
```

**(b)** [10] Estimate the arithmetic intensity of the matrix vector product and plot it on your roof-line model.

In this case consider a matirx with the size of M\*N and a vector N\*1.

We will estimate the arithmetic intensity of a multiplication operation between them.

So first are the read and write operations. The total amount of data transferred is M\*N+N\*1+M\*1.

Then, computing each entry of the result vector needs N multiplications and N-1 additions. So the total amount of calculation is M\*(2N-1).

Thus, the arithmetic instensity is $\frac{M(2N-1)}{MN+N+M}$.

Thus when M=N and as they go to infinity:
\begin{equation}
    \lim_{M,N\rightarrow\infty}I = 2
\end{equation}

```python
plt.title("Roofline Model for Nvidia GTX 1080ti")
plt.xlabel("Operation Intensity (FLOPS/byte)")
plt.ylabel("Performance(TFLOPs)")

def roofline(x):
    res = []
    sep = 11.3*1024/484.0
    for xi in x:
        if xi<=sep:
            res.append(11.3/sep*xi)
        else:
            res.append(11.3)

    return res

x = np.linspace(0,50,1000)

y = roofline(x)
plt.plot(x,y)
sep = 11.3*1024/484.0
plt.annotate('(23.9074,11.3)', xy=(sep, 11.3), xytext=(30, 6),arrowprops=dict(facecolor='black', shrink=0.05,width = 1))
plt.annotate('Limitation of the case\n (2,0.9453)', xy=(2,11.3/sep*2), xytext=(20, 2),arrowprops=dict(facecolor='black', shrink=0.05,width = 1))
plt.show()
```

**(c)** [15] Look through the scientific and engineering literature for articles that discuss their estimate for their arithmetic intensity and plot it on your model above.  Discuss how they came to their conclusion.

I choose one article Data-type Aware Arithmetic Intensity for Deep Neural Networks$^{[1]}$ in which it gives the arithmetic intensities of MobileNet-v1 and MobileNet-V2 which are machine learning frameworks.

MobileNet-v1:18, MobileNet-V2:32

[1] Jha, N. K., Mittal, S., & Avancha, S. Data-type Aware Arithmetic Intensity for Deep Neural Networks. Energy, 120, x109.

```python
plt.title("Roofline Model for Nvidia GTX 1080ti")
plt.xlabel("Operation Intensity (FLOPS/byte)")
plt.ylabel("Performance(TFLOPs)")

def roofline(x):
    res = []
    sep = 11.3*1024/484.0
    for xi in x:
        if xi<=sep:
            res.append(11.3/sep*xi)
        else:
            res.append(11.3)

    return res

x = np.linspace(0,50,1000)

y = roofline(x)
plt.plot(x,y)
sep = 11.3*1024/484.0

plt.annotate('MobileNet-v1\n (2,8.5078)', xy=(18,11.3/sep*18), xytext=(20, 2),arrowprops=dict(facecolor='black', shrink=0.05,width = 1))
plt.annotate('MobileNet-v2\n (32,11.3)', xy=(32, 11.3), xytext=(35, 6),arrowprops=dict(facecolor='black', shrink=0.05,width = 1))
plt.show()
```

## 3 - Scalability

[20] Explore the scientific and engineering literature for discussions of approaches to problems with an emphasis on scalability.  For instance, if you wanted to solve for the first few eigenvalues of a matrix, what is the most scalable way to do so (do not do this one).  You can pick from any field.

I choose the article Performance modeling and analysis of parallel Gaussian elimination on multi-core computers$^{[2]}$ by Fadi N.Sibai.

Here is the abstract of it:

Gaussian elimination is used in many applications and in particular in the solution of systems of linear equations. This paper presents mathematical performance models and analysis of four parallel Gaussian Elimination methods (precisely the Original method and the new Meet in the Middle –MiM– algorithms and their variants with SIMD vectorization) on multi-core systems. Analytical performance models of the four methods are formulated and presented followed by evaluations of these models with modern multi-core systems' operation latencies. Our results reveal that the four methods generally exhibit good performance scaling with increasing matrix size and number of cores. SIMD vectorization only makes a large difference in performance for low number of cores. For a large matrix size (n ⩾ 16 K), the performance difference between the MiM and Original methods falls from 16× with four cores to 4× with 16 K cores. The efficiencies of all four methods are low with 1 K cores or more stressing a major problem of multi-core systems where the network-on-chip and memory latencies are too high in relation to basic arithmetic operations. Thus Gaussian Elimination can greatly benefit from the resources of multi-core systems, but higher performance gains can be achieved if multi-core systems can be designed with lower memory operation, synchronization, and interconnect communication latencies, requirements of utmost importance and challenge in the exascale computing age.

[2] Sibai F N. Performance modeling and analysis of parallel Gaussian elimination on multi-core computers[J]. Journal of King Saud University-Computer and Information Sciences, 2014, 26(1): 41-54.


$\newcommand{\xbf}{{\bf x}}
\newcommand{\ybf}{{\bf y}}
\newcommand{\wbf}{{\bf w}}
\newcommand{\Ibf}{\mathbf{I}}
\newcommand{\Xbf}{\mathbf{X}}
\newcommand{\Rbb}{\mathbb{R}}
\newcommand{\vec}[1]{\left[\begin{array}{c}#1\end{array}\right]}
$

# Introduction aux réseaux de neurones
Matériel de cours rédigé par Pascal Germain, 2018
************

```python
import numpy as np
import aidecours
from matplotlib import pyplot as plt
from sklearn.datasets import make_blobs

from aidecours import code_button as CB
aidecours.center_images(); CB()
```

    voir/cacher le code.

# Cacher ce neurone que je ne saurais voir

Un *réseau de neurones artificiels* n'est qu'un simple *graphe de calcul*, qui exprime une fonction (possiblement) complexe comme une succession d'opérations simples.

Commencons par examiner un réseau de neurones très simple, possédant deux neurones d'entrées et un neurone de sortie. Ce réseau représente une fonction $R_\wbf: \Rbb^2 \to \Rbb$, que nous illustrons ainsi:

Le réseau $R_\wbf$ ci-dessus reçoit deux valeurs en entrée: $x_1$ et $x_2$. Ces valeurs sont respectivement multipliées par les poids $w_1$ et $w_2$. Le neurone de droite reçoit les valeurs $w_1 x_1$ et $w_2 x_2$, et les combine en appliquant la fonction $f$. La valeur obtenue correspond à la sortie du réseau.

$$R_\wbf\left({\small\vec{x_1\\[-3mm]x_2}}\right) = f(w_1 x_1+w_2 x_2)\,.$$

## Régression: Le neurone linéaire

Par exemple, si $f$ est la fonction identitée $f(x)=x$, ce réseau représente l'opération:

$$R_\wbf\left({\small\vec{x_1\\[-3mm]x_2}}\right) = f(w_1 x_1 +  w_2 x_2) = w_1 x_1 + w_2 x_2\,.$$

Nous avons donc ici d'un réseau de neurones dans sa plus simple expression, représentant un produit scalaire entre un vecteur $\xbf = (x_1,x_2)$ et un vecteur $\wbf = (w_1,w_2)$ :
$$R_\wbf({\xbf}) = \wbf\cdot \xbf\,.$$

En réalité, un réseau de neuronnes possédera généalement plus de 2 entrées. On notera $d$ la dimension de l'espace d'entré du réseau (le nombre de neurones d'entrée). On a donc $\xbf\in\Rbb^d$ et $\wbf\in\Rbb^d$.

$$R_\wbf({\xbf}) = \sum_{i=1}^d w_i x_i = \wbf\cdot \xbf\,.$$

### Entraînement du réseau
Typiquement, l'entraînement d'un réseau de neurones consistera a présenter au réseau un ensemble d'apprentissage afin qui «apprenne» les poids ${\bf w}$.

Cet **ensemble d'apprentissage** sera noté $S$ et contiendra $n$ observations,

$$S = \{ ({\bf x}_1, y_1), ({\bf x}_2, y_2), \ldots, ({\bf x}_n, y_n) \}\,,$$

où une *observation* est un couple entrée-sortie $({\bf x}, y)$.

### Fonction de perte

Afin de guider le processus d'apprentissage, nous devons choisir une fonction de perte $L\big(y',y\big)$. L'apprentissage consistera alors à résoudre le problème suivant:

$$\min_\wbf \left[\frac1n \sum_{i=1}^n L\Big(R_\wbf(\xbf_i), y_i\Big)\right].$$

### Perte quadratique

Prenons l'exemple de la perte quadratique:

$$L_{\rm quad}\big(y', y\big) = (y'-y)^2\,.$$

Ici, on note $y'$ l'étiquette prédite par le réseau de neurones et $y$ la véritable étiquette d'une observation $\xbf$.

```python
def perte_quadratique(y_prime, y):
    return (y_prime-y)**2

for y in (-1, 2, 3):
    aidecours.show_1d_function(lambda a: perte_quadratique(a, y), -2, 4, label=f'$y={y}$');

plt.xlim(-2,4), plt.ylim(0,5)
plt.title('Perte quadratique'); plt.xlabel('$y\'$'); plt.ylabel('$L_{quad}(y,y\')$'); CB()
```

Le problème d'apprentissage devient alors
$$\min_\wbf \left[\frac1n \sum_{i=1}^n (\wbf\cdot\xbf_i- y_i)^2\right].$$

Imaginons un ensemble d'entraîment à 3 observations:

$$
\begin{array}{rcl}
S =  \{ & ((1,1),& {-}1),\\
& ((0,-1),& 3),\\
&((2,\frac12),& 2)\quad\}
\end{array}$$

Illustrons la fonction objectif correspondante:

```python
x = np.array([(1,1),(0,-1),(2,.5)])
y = np.array([-1,3,2])
def calc_perte_quadratique(w): return np.mean((x @ w - y) ** 2)

aidecours.show_2d_function(calc_perte_quadratique, -10, 10, .5)
plt.title('Fonction objectif'); plt.xlabel('$w_1$'); plt.ylabel('$w_2$'); CB()
```

### C'est la méthode des moindres carrés!

On sait comment trouver la réponse exacte au problème d'optimisation.

En représentant les observations $S$ sous forme matricielle,

$$
\Xbf = \left[ \begin{array}{cc}
1& 1\\
0& -1\\
2& \frac12
\end{array} \right],
\quad\ybf = \left[ \begin{array}{c}
-1\\
3\\
2
\end{array} \right]
$$

On sait que
$\wbf^* = (\Xbf^T \Xbf)^{-1} \Xbf^T \ybf \approx (1.76, -2.90)$

voir: <https://fr.wikipedia.org/wiki/M%C3%A9thode_des_moindres_carr%C3%A9s>

```python
w_opt = np.linalg.inv(x.T @ x) @ x.T @ y
aidecours.show_2d_function(calc_perte_quadratique, -10, 10, .5)
plt.scatter(*w_opt, s=200, marker='*', c='r')
plt.title('Fonction objectif'); plt.xlabel('$w_1$'); plt.ylabel('$w_2$'); CB()
```

### Régularisation

Comme tout algorithme d'apprentissage, un réseau de neurone est susceptible de *surapprendre* les données d'apprentissage. Nous verrons dans ce cours quelques techniques pour se prémunir les réseaux de neurones contre le surapprentisage.

### Régularisation $\mathrm{L}^2$

La première technique est d'ajouter un régularisateur à la fonction objectif. Il est fréquent d'utiliser la norme Euclidienne des poids $\wbf$ au carré comme fonction de régularisation:

$$\min_\wbf \left[\frac1n \sum_{i=1}^n L\big(R_\wbf(\xbf_i), y_i\big) + \frac\lambda2\|\wbf\|^2\right].$$
Ici, $\lambda>0$ est un *hyperparamètre* de la procédure d'entraîment du réseau de neurones.

```python
lambdas = (0., 1., 10.)
fig, axes = plt.subplots(1, len(lambdas), figsize=(16, 4))
for param_lambda, ax in zip(lambdas, axes):
    def calc_perte_quadratique_regularisee(w):
        return np.mean((x @ w - y) ** 2) + param_lambda * (w @ w) / 2
    aidecours.show_2d_function(calc_perte_quadratique_regularisee, -10, 10, .5, ax=ax)
    w_opt = np.linalg.inv(x.T @ x + param_lambda * np.eye(2) * 2) @ x.T @ y
    ax.scatter(*w_opt, s=200, marker='*', c='r')
    plt.title(f'Fonction objectif ($\lambda = {param_lambda}$)'); plt.xlabel('$w_1$'); plt.ylabel('$w_2$');
CB()
```

### C'est la régression de Ridge (c.-à-d. les moindres carrés régularisés)

On sait comment trouver la réponse exacte au problème d'optimisation.

En représentant les observations $S$ sous forme matricielle,

$$
\Xbf = \left[ \begin{array}{cc}
1& 1\\
0& -1\\
2& \frac12
\end{array} \right],
\quad\ybf = \left[ \begin{array}{c}
-1\\
3\\
2
\end{array} \right],
$$
on sait que
$\wbf^* = (\Xbf^T \Xbf+\lambda \Ibf d)^{-1} \Xbf^T \ybf\,.$

$\lambda$ |$\longrightarrow$  | $w_1^*$ | $w_2^*$
----------|-|---------|--------
0         |$\longrightarrow$  | 1.76 | -2.90
1         |$\longrightarrow$   | 0.73 | -1.05
10        |$\longrightarrow$   | 0.13 | -0.15

voir: <https://fr.wikipedia.org/wiki/R%C3%A9gularisation_de_Tikhonov>

## Classification: Le neurone sigmoïdale

La combinaison de le neurone de sortie linéaire et de la perte quadratique convient bien aux problèmes de régression (où les étiquettes sont des valeurs réelles ($y\in\Rbb$).

Imaginons maintenant que nous voulons résoudre un problème de classification binaire:

$$y\in\{0,1\}$$

Reprenons notre réseau de neurones simplifié.

Nous allons considérer que le neurone de sortie applique la **fonction sigmoidale**, habituellement notée $\sigma$:

$$f(x) = \sigma(x) = \frac{1}{1+e^{-x}}\,.$$

```python
def sigmoid(x):
    return 1 / (1+np.exp(-x))

aidecours.show_1d_function(sigmoid, -6, 6, constante_x=.5);
plt.title('Fonction sigmoidale'); plt.xlabel('$x$'); plt.ylabel('$\sigma(x)$'); CB()
```

La sortie du réseau de neurones sera donc comprise entre $0$ et $1$.
On considère que l'étiquette prédite est $y=0$ lorsque

$$R_\wbf(\xbf) = \sigma(\wbf\cdot\xbf) < 0.5\,,$$
et $y=1$ sinon.

Une sortie près de $0.5$ est interprétée comme une grande incertitude envers le résultat. Inversement, plus la sortie est près de $0$ ou de $1$, plus on considère que le réseau est *confiant* envers sa décision.

Une interprétation *bayésienne* de la sortie du neurone sigmoïdal est de la voir comme la probabilité, selon le réseau $R_\wbf$, que $y=1$ pour une  certaine observation $\xbf$:

$$P(y = 1\,|\,\xbf; \wbf) \ = \ \sigma(\wbf\cdot\xbf) \,.$$
Conséquemment:
$$P(y = 0\,|\,\xbf; \wbf)  \ = \ 1 - P(y = 1\,|\,\xbf; \wbf) \ =\ 1 - \sigma(\wbf\cdot\xbf) \,.$$

Lors de l'apprentissage, on désire pénaliser le réseau d'autant plus que la probabilité attribuée à l'étiquette d'une observation s'éloigne de la véritable étiquette. La fonction de perte utilisée dans ce contexte est nommé la perte du **négatif log vraisemblance**:

$$\begin{align}
L_{\rm nlv}\big(y', y\big) = &- y \log(y') - (1-y)\log(1-y') \\[2mm]
= &\begin{cases}
-\log(1-y') & \mbox{si $y=0$ ,} \\[1mm]
-\log(y') & \mbox{si $y=1$  .}
\end{cases}\end{align}$$

```python
def perte_nlv_y0(y_prime):
    return - np.log(1. - y_prime)
def perte_nlv_y1(y_prime):
    return - np.log(y_prime)

aidecours.show_1d_function(perte_nlv_y0, 0, 0.999, .01, label='$y=0$');
aidecours.show_1d_function(perte_nlv_y1, 0.001, 1, .01, label='$y=1$');
plt.title('Négatif Log Vraisemblance'); plt.xlabel('$y\'$'); plt.ylabel('$L_{nlv}(y,y\')$'); CB()
```

#### Récapitulons.

1. Le neurone de sortie applique la fonction sigmoïdale à la somme de ses entrées:

$$\sigma(\wbf\cdot\xbf) = \frac{1}{1+e^{-\wbf\cdot\xbf}}\,.$$
2. Si $\sigma(\wbf\cdot\xbf) > 0.5$, on déclare que $y=1$.

3. Lors de l'apprentissage, on pénalise selon

$$
\begin{align}
L_{\rm nlv}\Big(\sigma(\wbf\cdot\xbf), y\Big)
&= - y \log(\sigma(\wbf\cdot\xbf)) - (1-y)\log(1-\sigma(\wbf\cdot\xbf))\\
&= \quad\vdots\\[1mm]
&= - y \wbf\cdot\xbf + \log(1+e^{\wbf\cdot\xbf})
\end{align}$$

La fonction objectif est alors:
$$
\min_\wbf \left[\frac1n \sum_{i=1}^n - y_i \wbf\cdot\xbf_i + \log(1+e^{\wbf\cdot\xbf_i})+ \frac\lambda2\|\wbf\|^2\right].
$$

### Il s'agit de la régression logistique!

**À démontrer en exercice**

```python
aidecours.show_1d_function(lambda a: perte_nlv_y0(sigmoid(a)), label='$y=0$');
aidecours.show_1d_function(lambda a: perte_nlv_y1(sigmoid(a)), label='$y=1$');
plt.title('Perte logistique'); plt.xlabel('$x$'); plt.ylabel('$L_{nlv}(1,\sigma(x))$'); CB()
```

### Exemple de classification

Ci-dessous, nous générons au hasard un ensemble de 100 observations en deux dimensions à l'aide de la fonction
`sklearn.datasets.make_blobs`

(voir: http://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_blobs.html)

```python
x, y = make_blobs(n_samples=100, centers=2, n_features=2, cluster_std=1, random_state=0)

aidecours.show_2d_dataset(x, y); CB()
```

Illustrons la fonction à optimiser (avec $\lambda=0.01$):

$$
\frac1n \sum_{i=1}^n - y_i \wbf\cdot\xbf_i + \log(1+e^{\wbf\cdot\xbf_i})+ \frac\lambda2\|\wbf\|^2\,.
$$

```python
def calc_perte_logistique(w):
    pred = sigmoid(x @ w)
    pred[y==0] = 1-pred[y==0]
    return np.mean(-np.log(pred)) + .01*w @ w/2

aidecours.show_2d_function(calc_perte_logistique, -4, 4, .05)
w_opt =[ 1.59526315, -1.09703947]
plt.scatter(*w_opt, s=200, marker='*', c='r')
plt.title('Fonction objectif'); plt.xlabel('$w_1$'); plt.ylabel('$w_2$'); CB()
```

L'optimum se trouve en $\wbf^* \approx \vec{1.60\\ -1.10}\,$.

Ce qui correspond au prédicteur suivant.

```python
from sklearn.linear_model import LogisticRegression
algo = LogisticRegression(C=100./len(y), fit_intercept=False)
algo.fit(x, y)
coord=aidecours.show_2d_predictions(x, y, algo.predict)
plt.plot([coord[0],0,0], [0,0,coord[2]], '--k');
#print('vecteur w :', algo.coef_[0])
#print('biais b   :', algo.intercept_)
CB()
```

## Ajout d'un biais

Pour éviter de restreindre la fonction de prédiction à passer par l'origine, on ajoute un **biais**:

Dans le cas de notre réseau simple, la fonction de prédiction devient alors:

$$R_{\wbf,b}({\xbf}) = f\left(\sum_{i=1}^d w_i x_i + b\right) = f\big(\wbf\cdot \xbf+b\big).$$

Retournons à notre exemple de régression logistique (c'est-à-dire le réseau de neurone où le neurone de sortie $f$ est la **fonction sigmoïdale** $\sigma$ et a perte du **négatif log vraisemblance** $L_{\rm nlv}$. Le problème d'optimisation **avec biais** s'exprime ainsi:

$$
\min_{\wbf,b} \left[\frac1n \sum_{i=1}^n - y_i (\wbf\cdot\xbf_i+b) + \log(1+e^{\wbf\cdot\xbf_i+b})+ \frac\lambda2\|\wbf\|^2\right].
$$

```python
algo = LogisticRegression(C=100/len(y), fit_intercept=True)
algo.fit(x, y)
coord=aidecours.show_2d_predictions(x, y, algo.predict)
plt.plot([coord[0],0,0], [0,0,coord[2]], '--k'); CB()
#print('vecteur w :', algo.coef_[0])
#print('biais b   :', algo.intercept_[0])
CB()
```

L'optimum se trouve en $\wbf^* \approx \vec{0.97\\ -1.52}, b \approx 2.33\,$.

notebooks/01 - Commencons simplement.ipynb


```python
%pylab inline
from sympy.interactive import init_printing
init_printing()
from sympy import sqrt, sin, pi, var, integrate, Symbol, S, Integral
var("x y z")
n = 3 * ((x-1)**2 + (y-1)**2 + (z-1)**2 - 1) / pi
Vh = -((x-1)**4 + (y-1)**4 + (z-1)**4) + 2*(x**2+y**2+z**2)
def laplace(f):
    return f.diff(x, 2) + f.diff(y, 2) + f.diff(z, 2)
print "n ="
n
```

```python
print "Vh ="
Vh
```

Let's check charge neutrality:

```python
integrate(n, (x, 0, 2), (y, 0, 2), (z, 0, 2))
```

$V_H$ is produced by the charge density $n$:

```python
laplace(Vh) / (-4*pi)
```

```python
_.simplify()
```

The Hartree energy $E_H$ is:

```python
integrate(Vh*n, (x, 0, 2), (y, 0, 2), (z, 0, 2))/2
```

We are solving a Poisson equation:
$$\nabla^2 V_H(x, y, z) = -4\pi n(x, y, z)$$
with
$$n(x, y, z) = {3\over\pi} ((x-1)^2 + (y-1)^2 + (z-1)^2 - 1)$$
on the domain $[0, 2] \times [0, 2] \times [0, 2]$ with periodic boundary condition. This charge density is net neutral. The solution is given by:
$$
V_H({\bf x}) = \int {n({\bf y})\over |{\bf x}-{\bf y}|} d^3 y
$$
where ${\bf x}=(x, y, z)$. In reciprocal space
$$
V_H({\bf G}) = 4\pi {n({\bf G})\over G^2}
$$
where ${\bf G}$ are the reciprocal space vectors. I am interested in the Hartree energy:
$$
E_H = {1\over 2} \int {n({\bf x}) n({\bf y})\over |{\bf x}-{\bf y}|} d^3 x d^3 y
= {1\over 2} \int V_H({\bf x}) n({\bf x}) d^3 x
$$
In reciprocal space this becomes (after discretization):
$$
E_H = 2\pi \sum_{{\bf G}\ne 0} {|n({\bf G})|^2\over G^2}
$$
The ${\bf G}=0$ term is omitted, which effectively makes the charge density net neutral (and since it is already neutral, then everything is consistent).

For the test problem above, this can be evaluated analytically and one gets:
$$
E_H = {128\over 35\pi} = 1.16410...
$$
Now we implement the FFT solver in NumPy below and plot the convergence graphs with respect to this exact energy.

This problem has also been discussed at:

http://scicomp.stackexchange.com/questions/7097/convergence-rate-of-fft-poisson-solver/7169

[Computational Science StackExchange](http://scicomp.stackexchange.com/) is a great site to ask these questions!

```python
from numpy import empty, pi, meshgrid, linspace, sum
from numpy.fft import fftn, fftfreq
E_exact = 128/(35*pi)
print "Hartree Energy (exact):      %.15f" % E_exact
f = open("conv.txt", "w")
N = 1
for i in range(1, 9):
    N *= 2
    print "N =", N
    L = 2.
    x1d = linspace(0, L, N+1)[:-1]
    x, y, z = meshgrid(x1d, x1d, x1d)

    nr = 3 * ((x-1)**2 + (y-1)**2 + (z-1)**2 - 1) / pi
    ng = fftn(nr) / N**3

    G1d = N * fftfreq(N) * 2*pi/L
    kx, ky, kz = meshgrid(G1d, G1d, G1d)
    G2 = kx**2+ky**2+kz**2
    G2[0, 0, 0] = 1  # omit the G=0 term

    tmp = 2*pi*abs(ng)**2 / G2
    tmp[0, 0, 0] = 0  # omit the G=0 term
    E = sum(tmp) * L**3
    print "Hartree Energy (calculated): %.15f" % E
    f.write("%d %.15f\n" % (N, E))
f.close()
```

```python
from pylab import plot, semilogy
from numpy import loadtxt, pi
D = loadtxt("conv.txt")
N = D[:, 0]
E = D[:, 1]
E_exact = 128/(35*pi)
```

```python
figure(figsize=(8, 6), dpi=80)
semilogy(N, E-E_exact, "k-")
grid()
title("Convergence of an FFT Poisson solver (semilogy)")
xlabel("N (number of PW in each direction)")
ylabel("E  -  E_exact  [a.u.]")
savefig("fft_convergence_semilogy.png")
```

```python
figure(figsize=(8, 6), dpi=80)
calculated = E-E_exact
loglog(N, E-E_exact, "ko", label="calculated")
predicted = 1/N**2
predicted = predicted * (calculated[-1] /  predicted[-1])
loglog(N, predicted, "g-", label="$1/N^2$")
grid()
title("Convergence of an FFT Poisson solver (loglog)")
xlabel("N (number of PW in each direction)")
ylabel("E  -  E_exact  [a.u.]")
legend()
savefig("fft_convergence_loglog.png")
```

## Solution

```python
from numpy import empty, pi, meshgrid, linspace, sum, exp, sin
from numpy.fft import fftn, fftfreq
f = open("conv2.txt", "w")
for N in range(3, 30, 2):
    print "N =", N
    L = 2.
    x1d = linspace(0, L, N+1)[:-1]
    x, y, z = meshgrid(x1d, x1d, x1d)

    nr = 3*pi*exp(sin(pi*x)*sin(pi*y)*sin(pi*z))/4
    ng = fftn(nr) / N**3

    G1d = N * fftfreq(N) * 2*pi/L
    kx, ky, kz = meshgrid(G1d, G1d, G1d)
    G2 = kx**2+ky**2+kz**2
    G2[0, 0, 0] = 1  # omit the G=0 term

    tmp = 2*pi*abs(ng)**2 / G2
    tmp[0, 0, 0] = 0  # omit the G=0 term
    E = sum(tmp) * L**3
    print "Hartree Energy (calculated): %.15f" % E
    f.write("%d %.15f\n" % (N, E))
f.close()
```

```python
from pylab import plot, semilogy
from numpy import loadtxt, pi
D = loadtxt("conv2.txt")
N = D[:, 0]
E = D[:, 1]
E_exact = E[-1]

figure(figsize=(8, 6), dpi=80)
calculated = E-E_exact
semilogy(N, E-E_exact, "ko-", label="calculated")
grid()
title("Convergence of an FFT Poisson solver (loglog)")
xlabel("N (number of PW in each direction)")
ylabel("E  -  E_exact  [a.u.]")
legend()
savefig("fft_convergence_loglog.png")
```


```python
#https://github.com/kyamaz/openql-notes/blob/master/docs/20180219/sympy_programing_1_handout.ipynb

#0. [準備]必要なライブラリをインポートする

from sympy import *
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import Qubit,QubitBra
from sympy.physics.quantum.gate import X,Y,Z,H,S,T,CNOT,SWAP, CPHASE
from sympy.physics.quantum.gate import IdentityGate as _I

```

```python
#↓SymPy で良い感じに表示するためのおまじない

from sympy.printing.dot import dotprint
init_printing()
```

```python

#１. 計算に必要な量子ビット（量子レジスタ）を準備して、その値を初期化する
#計算に必要な量子ビットは、sympy.physics.quantum.qubit.Qubitクラス（実態はケット・ベクトル）を必要なビット数を
#初期化して作成します。

#例１：変数 $a$ に、２量子ビット $ \vert 00 \rangle $ を準備します。

a = Qubit('00')
print(a)
a

```

```python
#例３：8量子ビットをすべて $0$ として準備します。

q_8 = Qubit('0'*8)
pprint(q_8)
q_8
```

```python
#例４：（後述で説明する計算方法を使いますが）計算を行って、変数 $c$ の状態ベクトルを準備します。
#２量子ビットの重ね合わせ
c = qapply(H(1)*H(0)*a)
print(c)

```

    |00>/2 + |01>/2 + |10>/2 + |11>/2

```python
c
```

```python
#Qubitクラスには次のような関数やプロパティが定義されています。

q = Qubit('000')

# サイズ（量子ビット数）を返す
print("nqubits=%d"% q.nqubits)
print("len=%d" % len(q))
print("dimension=%d" % q.dimension)
print(q.qubit_values)

q.flip(1)  # 引数で指定された位置(*)の右から2番目の量子ビットを反転します.
```

```python
#2. 量子計算をユニタリ行列（ゲート演算子）で記述する
#作用させるユニタリ行列を標準で準備されているユニタリ行列のテンソル積で表します。
#
#まず最初に、標準で定義されているユニタリ行列を見て見ましょう。
#
#１量子ビット操作
#引数に渡す数字は、作用させる量子ビットの位置を表します。
#
#パウリ演算子
X(0).get_target_matrix() # σ_x, get_target_matrix() で行列表現を取り出します

```

$$\left[\begin{matrix}0 & 1\\1 & 0\end{matrix}\right]$$

```python
Y(0).get_target_matrix() # σ_y
```

$$\left[\begin{matrix}0 & - i\\i & 0\end{matrix}\right]$$

```python
Z(0).get_target_matrix()  # σ_z
```

$$\left[\begin{matrix}1 & 0\\0 & -1\end{matrix}\right]$$

```python
H(0).get_target_matrix()
```

$$\left[\begin{matrix}\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\\frac{1}{\sqrt{2}} & - \frac{\sqrt{2}}{2}\end{matrix}\right]$$

```python
S(0).get_target_matrix()
```

$$\left[\begin{matrix}1 & 0\\0 & i\end{matrix}\right]$$

```python
represent(S(0), nqubits=1) # represent は、行列を nqubits で表現します。
```

$$\left[\begin{matrix}1 & 0\\0 & i\end{matrix}\right]$$

```python
Dagger(S(0))
```

```python
Dagger(S(0).get_target_matrix())  # get_target_matrix() はDaggerカッコの中
```

$$\left[\begin{matrix}1 & 0\\0 & - i\end{matrix}\right]$$

```python
represent(Dagger(S(0)), nqubits=1)
```

$$\left[\begin{matrix}1 & 0\\0 & - i\end{matrix}\right]$$

```python
def Sdg(n): return S(n)**(-1)   # S^{\dagger} としては、この演算子を使います。
represent(Sdg(0),nqubits=1)
```

$$\left[\begin{matrix}1 & 0\\0 & - i\end{matrix}\right]$$

```python
T(0)
```

```python
represent(T(0),nqubits=1)
```

$$\left[\begin{matrix}1 & 0\\0 & e^{\frac{i \pi}{4}}\end{matrix}\right]$$

```python
Dagger(T(0))
```

```python
represent(Dagger(T(0)),nqubits=1)
```

$$\left[\begin{matrix}1 & 0\\0 & e^{- \frac{i \pi}{4}}\end{matrix}\right]$$

```python
Dagger(T().get_target_matrix())  # get_target_matrix() はDaggerカッコの中
```

$$\left[\begin{matrix}1 & 0\\0 & e^{- \frac{i \pi}{4}}\end{matrix}\right]$$

```python
def Tdg(n): return T(n)**(-1)   # T^{\dagger} としては、この演算子を使います。
represent(Tdg(0),nqubits=1)
```

$$\left[\begin{matrix}1 & 0\\0 & e^{- \frac{i \pi}{4}}\end{matrix}\right]$$

```python
"""
２量子ビット操作
引数に渡す数字は２つ、作用させる量子ビットの位置を表します。省略すると 0 のみとなるため、そのあとの計算がうまくできません。
必ず、引数を指定するようにしましょう。

CNOT操作
次の $ 4 \times 4 $ の行列です。 get_target_matrix() では表示できません。represent() を使います。
$ \left( \begin{array}{cccc} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 \end{array} \right) $
"""
```

    '\n２量子ビット操作\n引数に渡す数字は２つ、作用させる量子ビットの位置を表します。省略すると 0 のみとなるため、そのあとの計算がうまくできません。\n必ず、引数を指定するようにしましょう。\n\nCNOT操作\n次の $ 4 \times 4 $ の行列です。 get_target_matrix() では表示できません。represent() を使います。\n$ \\left( \x08egin{array}{cccc} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 \\end{array} \right) $\n'

```python
CX=CNOT(1,0)
print("controls=%s,targets=%s,gate=%s" % (CX.controls, CX.targets, CX.gate))  # CNOTのプロパティ
# CX.get_target_matrix() は、XGate になってしまいます。
# pprint(CX.get_target_matrix())
represent(CX,nqubits=2) # 行列形式で直接表現する方法
```

    controls=(1,),targets=(0,),gate=X(0)

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\\0 & 0 & 1 & 0\end{matrix}\right]$$

```python
"""
SWAP操作
指定された量子ビットを入れ替える操作です。
"""
pprint(SWAP(0,1).get_target_matrix())
represent(SWAP(0,1),nqubits=2) # 行列形式で直接表現する方法
```

    ⎡1  0  0  0⎤
    ⎢          ⎥
    ⎢0  0  1  0⎥
    ⎢          ⎥
    ⎢0  1  0  0⎥
    ⎢          ⎥
    ⎣0  0  0  1⎦

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 0 & 1 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$$

```python
"""
CPHASE操作
制御Zゲート、Control-Z、CZ と呼ばれ、次の行列です。

$ \left( \begin{array}{cccc} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & -1 \end{array} \right) $

"""

CPHASE(0,1)

```

```python
represent(CPHASE(0,1),nqubits=2)
```

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & -1\end{matrix}\right]$$

```python
Op1 = X(1)*_I(0)
pprint(TensorProduct(X().get_target_matrix(),_I().get_target_matrix())) # Op1 を行列形式で直接プリントできる方法があるかも？なかったら作ります
Op1
```

```python
# 標準の１量子ビット操作のユニタリ演算は、その掛け算の順番よりも、指定した量子ビット（引数）がポイントとなります。
# 順番が大切になるのは、同じ位置の量子ビットに作用する場合で、積の交換はできません。
Op1_ = _I(0)*X(1)
Op1_
```

```python
Op2 = _I(1)*X(0)
pprint(TensorProduct(_I().get_target_matrix(),X().get_target_matrix())) # Op2 を行列形式で直接プリントできる方法があるかも？なかったら作ります
Op2
```

```python
# !pip install graphviz
import graphviz
from sympy.printing.dot import dotprint
from graphviz import Source
Source(dotprint(Op1)) # Jupyter NotebookやQtConsoleでは、ツリー構造が表示できます。
```

```python
Source(dotprint(Op2))  # 内部的な構造のため、通常の量子計算では不要です。
```

```python
def hadamard(s,n):
    h = H(s)
    for i in range(s+1,n+s):
       h = H(i)*h
    return h

h_8 = hadamard(0,8)
print(h_8)
print("\n")
pprint(h_8)    # = pretty_print(h_8)
# h8.get_target_matrix() は実装がなく、エラーになります
h_8
```

```python
h_4  =  hadamard(0,4)
represent(h_4,nqubits=4)
```

$$\left[\begin{array}{cccccccccccccccc}\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4}\\\frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & \frac{1}{4} & - \frac{1}{4} & - \frac{1}{4} & \frac{1}{4}\end{array}\right]$$

```python
#量子回路図
#ユニタリ行列のテンソル積は、量子回路として、量子回路図で表現できます。
#SymPy には、量子回路図を描画するための仕組みが備わっています。量子回路図を描いてみましょう。
# そのために、まずは、必要なライブラリを読み込んで準備します。
%matplotlib inline
import matplotlib.pyplot as plt
from sympy.physics.quantum.circuitplot import CircuitPlot,labeller, Mz,CreateOneQubitGate
```

```python
#CNOTゲートを描いてみましょう。
CircuitPlot(CNOT(1,0), 2, labels=labeller(2,'\phi')[::-1])  # labellerでつくられる添え字の上下が逆？
#（ダガー）付きのゲート演算子など、ゲートの自作もできます。
```

```python
Sdag = CreateOneQubitGate('Sdg','S^{\dagger}')
Tdag = CreateOneQubitGate('Tdg','T^{\dagger}')
CircuitPlot(Sdag(0)*Tdag(0),nqubits=1)
```

```python
CircuitPlot(CNOT(1,0)*S(1), 2, labels=labeller(2)[::-1])  # 操作の順番に注意してください。
```

```python
CircuitPlot(S(1)*H(0)*CNOT(1,0)*S(1)*H(1), 2, labels=labeller(2)[::-1])
```

```python
#（参考）Z基底での測定を描くときは、Mz を使って表現します。
CircuitPlot(Mz(0)*Mz(1)*S(1)*H(0)*CNOT(1,0)*S(1)*H(1), 2, labels=labeller(2)[::-1])
```

```python
#3. ユニタリ行列を量子ビットに作用する
#ここまでのユニタリ行列の準備では、実際の行列計算は行われておりません。
#SymPy の特徴である「変数を変数のまま」扱ってきました。変数は単なる記号でしかありませんでした。

#実際に変数を計算するのは、 qapply() という関数で行います。計算したい記述を qapply 関数の引数に指定して計算します。

#直前で回路図を描いた計算をしてみましょう。

qapply(S(1)*H(0)*CNOT(1,0)*S(1)*H(1)*Qubit('00'))

```

```python
hadamard_8=qapply(h_8*q_8)
hadamard_8
```

```python
"""
4. 測定する
ここまでの量子計算（qapply）では、量子状態のまま計算されます。
シミュレーション上の理論的な計算では、qapplyまで行えば、目的が達成することが多いです。

SymPy には、観測のためのメソッドも備わっています。
measure_all()では、引数で渡された量子ビットのすべての状態の測定確率を計算で求めて出力します。

そのほかにも幾つか観測のためのメソッドが提供されます。例えば、measure_partial()は、第一引数で指定された量子ビットに対して、第二引数で指定された部分的な振幅の測定を実行します。

"""
```

    '\n4. 測定する\nここまでの量子計算（qapply）では、量子状態のまま計算されます。\nシミュレーション上の理論的な計算では、qapplyまで行えば、目的が達成することが多いです。\n\nSymPy には、観測のためのメソッドも備わっています。\nmeasure_all()では、引数で渡された量子ビットのすべての状態の測定確率を計算で求めて出力します。\n\nそのほかにも幾つか観測のためのメソッドが提供されます。例えば、measure_partial()は、第一引数で指定された量子ビットに対して、第二引数で指定された部分的な振幅の測定を実行します。\n\n'

```python
from sympy.physics.quantum.qubit import measure_all, measure_partial, measure_all_oneshot, measure_partial_oneshot
```

```python
measure_all(hadamard_8)
```

```python
for i in range(8):
    pprint(measure_all_oneshot(hadamard_8))
```

    ❘10011010⟩
    ❘11001101⟩
    ❘00100000⟩
    ❘01000111⟩
    ❘10011011⟩
    ❘01100011⟩
    ❘11100100⟩
    ❘10100111⟩

```python
#（例題１）Toffoli（CCX）, CCCX, CCCCX ...
from sympy.physics.quantum.gate import CGateS

```

```python
def CCX(c1,c2,t):  return CGateS((c1,c2),X(t))
def Toffoli(c1,c2,t):  return CGateS((c1,c2),X(t))
CircuitPlot(CCX(1,2,0),3,labels=labeller(3)[::-1])
```

```python
represent(CCX(1,2,0),nqubits=3)
```

$$\left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\0 & 0 & 0 & 0 & 0 & 0 & 1 & 0\end{matrix}\right]$$

```python
def CCCX(c1,c2,c3,t): return CGateS((c1,c2,c3),X(t))
CircuitPlot(CCCX(1,2,3,0),4,labels=labeller(4)[::-1])
```

```python
def CCCCX(c1,c2,c3,c4,t): return CGateS((c1,c2,c3,c4),X(t))
CircuitPlot(CCCCX(1,2,3,4,0),5,labels=labeller(5)[::-1])
```

```python
"""

（例題２）重ね合わせ状態の中の、ある状態のみマーキングする
重ね合わせ状態の中の、ある状態 $ s_t $ のみに、−１掛ける操作（ここでは、マーキングを呼びます）を考えます。

たとえば、２量子ビットの重ね合わせ状態 $\displaystyle \frac{1}{2} \left(\ |\ 00\ \rangle\ +\ |\ 01\ \rangle\ +\ |\ 10\ \rangle\ +\ |\ 11\ \rangle\ \right) $ を考えるとき、
状態 $\;|\ 10\ \rangle\;$ をマーキングするとします。

マーキング後の状態は、$\displaystyle  \frac{1}{2} \left(\ |\ 00\ \rangle\ +\ |\ 01\ \rangle\ -\ |\ 10\ \rangle\ +\ |\ 11\ \rangle\ \right) $ となることを目指します。

３量子ビットのマーキング操作を試してみましょう

"""
h_3 = hadamard(0,3)
target_state_3 = qapply(h_3*Qubit('000'))          #３量子ビットの重ね合わせ状態を準備します。
def CCZ(c1,c2,t): return (H(t)*CCX(c1,c2,t)*H(t)) # ＣＣＺ演算子を定義します。

```

```python
mark_7 = CCZ(1,2,0)
qapply(mark_7*target_state_3)
```

```python
mark_6 = X(0)*CCZ(1,2,0)*X(0)
qapply(mark_6*target_state_3)
```

```python
mark_5 = X(1)*CCZ(1,2,0)*X(1)
qapply(mark_5*target_state_3)
```

```python
mark_4 = X(1)*X(0)*CCZ(1,2,0)*X(1)*X(0)
qapply(mark_4*target_state_3)
```

```python
mark_3 = X(2)*CCZ(1,2,0)*X(2)
qapply(mark_3*target_state_3)
```

```python
mark_2 = X(2)*X(0)*CCZ(1,2,0)*X(2)*X(0)
qapply(mark_2*target_state_3)
```

```python
mark_1 = X(2)*X(1)*CCZ(1,2,0)*X(2)*X(1)
qapply(mark_1*target_state_3)
```

```python
mark_0 = X(2)*X(1)*X(0)*CCZ(1,2,0)*X(2)*X(1)*X(0)
qapply(mark_0*h_3*Qubit('000'))
```

```python
"""
（例題３）重ね合わせ状態の中に、マーキングした状態があるかを見る
著名な操作「Grover のアルゴリズム」を試してみましょう。

（ヒント１）平均値周りの反転操作：$\displaystyle D_{n} = H_{n} \cdot \Big( 2\ |\ 0\ \rangle\langle\ 0\ |_{n}\ -\ I_{n} \Big) \cdot H_{n} $ を使います。

（ヒント２）試行回数は、$\displaystyle \mathcal{O}(\sqrt{n}) $
"""

# d_3 = h_3 * X(0)*X(1)*X(2) * H(0)*CCX(1,2,0)*H(0) * X(0)*X(1)*X(2) * h_3    # グローバル位相（絶対位相）の差に注意
# d_3 = h_3 * X(0)*X(1)*X(2) * CGateS((1,2), Z(0)) * X(0)*X(1)*X(2) * h_3
def DOp(n): return (Qubit('0'*n)*QubitBra('0'*n)*2-_I(0))     # ゲート操作で計算するには、上記コメントのような演算になります。
d_3 = h_3 * DOp(3) * h_3    # 平均値周りの反転操作
represent(d_3,nqubits=3)

```

$$\left[\begin{matrix}- \frac{3}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & - \frac{3}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & - \frac{3}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{3}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{3}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{3}{4} & \frac{1}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{3}{4} & \frac{1}{4}\\\frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & - \frac{3}{4}\end{matrix}\right]$$

```python
#３量子ビットで状態｜７＞を探す

ret1=qapply(d_3*mark_7*target_state_3)
measure_all(ret1)
```

```python
ret2=qapply(d_3*mark_7*ret1)  # シミュレーター計算ですから、上で一度観測した ret1 を使っても観測の影響なく次の計算に利用可能。
measure_all(ret2)
```

```python

over_1=qapply(d_3*mark_7*ret2)# 試行回数が多いとどうなるでしょう。# 試行回数が多
measure_all(over_1)

```

```python
over_2=qapply(d_3*mark_7*over_1)    # 試行回数が多いとどうなるでしょう。# 試行回数が多
measure_all(over_2)
```

```python
#遠しで計算すると、回路が長くなってきています。少し時間かかります。
from sympy.physics.quantum.gate import gate_simp
search_7_in_3qubit = gate_simp(d_3*mark_7*d_3*mark_7)
ret3=qapply(search_7_in_3qubit*target_state_3)
ret3
```

```python
print(measure_all(ret3))
for i in range(10):
    pprint(measure_all_oneshot(ret3))
```

    [(|000>, 1/128), (|001>, 1/128), (|010>, 1/128), (|011>, 1/128), (|100>, 1/128), (|101>, 1/128), (|110>, 1/128), (|111>, 121/128)]
    ❘111⟩
    ❘111⟩
    ❘111⟩
    ❘111⟩
    ❘111⟩
    ❘111⟩
    ❘111⟩
    ❘111⟩
    ❘111⟩
    ❘111⟩

```python
d_3_gate = h_3 * X(0)*X(1)*X(2) * H(0)*CCX(1,2,0)*H(0) * X(0)*X(1)*X(2) * h_3
CircuitPlot(gate_simp(d_3_gate*mark_7*d_3_gate*mark_7*h_3),nqubits=3,labels=labeller(3)[::-1])
```

```python
"""
OpenQL 発のPythonライブラリ QuantPy を紹介します
QuantPy は、フロントエンドを SymPy を使っています。
「量子ビットの準備」「量子操作（ゲート演算子）の記述」は SymPy で行います。
「量子ビットにゲート演算子を作用」する qapply() を置き換えて動作します。
"""
from quantpy.sympy.qapply import qapply
# sympy.physics.quantum.qapply を上書きします。sympy.physics.quantum のインポートの後に読み込んでください。

```

```python
from quantpy.sympy.executor.classical_simulation_executor import ClassicalSimulationExecutor
classical_simulator = ClassicalSimulationExecutor()
for i in range(8):
    print(qapply(H(2)*H(1)*H(0)*Qubit('000'), executor=classical_simulator))
```

    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>
    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>
    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>
    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>
    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>
    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>
    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>
    (0.353553390593+0j)|000> + (0.353553390593+0j)|100> + (0.353553390593+0j)|010> + (0.353553390593+0j)|110> + (0.353553390593+0j)|001> + (0.353553390593+0j)|101> + (0.353553390593+0j)|011> + (0.353553390593+0j)|111>

```python
from quantpy.sympy.executor.ibmq_executor import IBMQExecutor
# APItoken = 'hogehoge'
# ibmq_simulator = IBMQExecutor(api_key=APItoken, backend='ibmqx2')
ibmq_simulator = IBMQExecutor(backend='local_qasm_simulator')
qpy_ret=qapply(H(2)*H(1)*H(0)*Qubit('000'), executor=ibmq_simulator)
print(qpy_ret)
```

```python
from quantpy.sympy.expr_extension import *
sympy_expr_add_operators()
circuit = X(0)>>X(1)>>X(2)
circuit
```

```python
#（Homework）Toffoli 再び
#課題１）　Toffoli ゲートを、基本的な量子ゲートだけで表してください。
def ToffoliGate_plot(q0,q1,q2):
    return T(q0)*S(q1)*CNOT(q0,q1)*Tdag(q1)*CNOT(q0,q1)\
            *H(q2)*Tdag(q1)*T(q2)*CNOT(q0,q2)*Tdag(q2)*CNOT(q1,q2)\
            *T(q2)*CNOT(q0,q2)*Tdag(q2)*CNOT(q1,q2)*H(q2)
CircuitPlot(ToffoliGate_plot(2,1,0), nqubits=3)
```

```python
def Tdg(q): return (T(q)**(-1))
def ToffoliGate(q0,q1,q2):
    return T(q0)*S(q1)*CNOT(q0,q1)*Tdg(q1)*CNOT(q0,q1)\
            *H(q2)*Tdg(q1)*T(q2)*CNOT(q0,q2)*Tdg(q2)*CNOT(q1,q2)\
            *T(q2)*CNOT(q0,q2)*Tdg(q2)*CNOT(q1,q2)*H(q2)
ToffoliGate(2,1,0)
# qapply(ToffoliGate(2,1,0)*Qubit('110'), dagger=True)   # 残念ながら、計算ができませんでした。
```

```python
def CCCX_with_ancilla(q0,q1,q2,t,a0,a1,a2):
    return CNOT(q0,a0)*Toffoli(q1,a0,a1)*Toffoli(q2,a1,a2)*CNOT(a2,t)*Toffoli(q2,a1,a2)*Toffoli(q1,a0,a1)*CNOT(q0,a0)
CircuitPlot(CCCX_with_ancilla(6,5,4,3,2,1,0),nqubits=7,labels=(labeller(3,'a')+['t']+labeller(3,)))
```

```python
qapply(CCCX_with_ancilla(6,5,4,3,2,1,0)*Qubit('1111000'))
```

```python
qapply(CCCX_with_ancilla(6,5,4,3,2,1,0)*Qubit('1110000'))
```

```python
"""
付録Ａ：ブラ・ベクトル
複素共役を表すための dagger（ $ \dagger $ ） をQubitオプジェクトに作用させると、ブラ・ベクトルになります。
また、内積も計算できます。
シンボルのままで内積にするには、単にブラ・ベクトル QubitBra とケット・ベクトル Qubit を掛ける（*）だけです。
実際の値を計算するには、この内積オブジェクトのdoit()メソッドで計算します。
"""

from sympy.physics.quantum.dagger import Dagger
Dagger(q)

```

```python
type(Dagger(q))
```

    sympy.physics.quantum.qubit.QubitBra

```python
ip = Dagger(q)*q
ip
```

```python
ip.doit()
```

```python
"""
付録Ｂ：量子フーリエ変換
"""
from sympy.physics.quantum.qft import QFT
fourier_3 = QFT(0,3).decompose()
fourier_3

```

```python
figure_qft3 = CircuitPlot(fourier_3, nqubits=3)
qapply(fourier_3 * Qubit('000'))
```

```python
"""
付録Ｃ：QASM
SymPy独自のQASMでゲートを記述できます。
ただし、OpenQASM とは異なるフォーマットです。注意してください。

"""
from sympy.physics.quantum.qasm import Qasm
qprog1 = Qasm(
    'qubit q0',
    'qubit q1',
    'h q0',
    'cnot q0,q1')
CircuitPlot(qprog1.get_circuit(),2,labels=labeller(2))

```

```python
qprog2 = Qasm("def	c-S,1,'S'",
        "def	c-T,1,'T'",
        "qubit	j_0",
        "qubit	j_1",
        "qubit	j_2",
        "h	j_0",
        "c-S	j_1,j_0",
        "c-T	j_2,j_0",
        "nop	j_1",
        "h	j_1",
        "c-S	j_2,j_1",
        "h	j_2",
        "swap	j_0,j_2")
qprog2.plot()
qapply(qprog2.get_circuit()*Qubit('000'))
```

```python
qasm_lines = """\
	def	CU,1,'U'
	def	CU2,1,'U^2'
	def	CU4,1,'U^4'
	def	c-S,1,'S'
	def	c-T,1,'T'
	qubit	j_0,0	# QFT qubits
	qubit	j_1,0
	qubit	j_2,0
	qubit	s_0	# U qubits
	h	j_0	# equal superposition
	h	j_1
	h	j_2
	CU4	j_0,s_0	# controlled-U
	CU2	j_1,s_0
	CU	j_2,s_0
	h	j_0	# QFT
	c-S	j_0,j_1
	h	j_1
	nop	j_0
	c-T	j_0,j_2
	c-S	j_1,j_2
	h	j_2
	nop	j_0
	nop	j_0
	nop	j_1
	measure	j_0	# final measurement
	measure	j_1
	measure	j_2"""

qprog3 = Qasm(*qasm_lines.splitlines())
qprog3.plot()
```

```python

```