```python
# imports

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
%matplotlib inline
```

# Basic Concepts

## What is "learning from data"?

> In general **Learning from Data** is a scientific discipline that is concerned with the design and development of algorithms that allow computers to infer (from data) a  model that allows *compact representation* (unsupervised learning) and/or *good generalization* (supervised learning).

This is an important technology because it enables computational systems to adaptively improve their performance with experience accumulated from the observed data.

Most of these algorithms are based on the *iterative solution* of a mathematical problem that involves data and model. If there was an analytical solution to the problem, this should be the adopted one, but this is not the case for most of the cases.

So, the most common strategy for **learning from data** is based on solving a system of equations as a way to find a series of parameters of the model that minimizes a mathematical problem. This is called **optimization**.

The most important technique for solving optimization problems is **gradient descend**.

## Preliminary: Nelder-Mead method for function minimization.

The most simple thing we can try to minimize a function $f(x)$ would be to sample two points relatively near each other, and just repeatedly take a step down away from the largest value. This simple algorithm has a severe limitation: it can't get closer to the true minima than the step size.

The Nelder-Mead method dynamically adjusts the step size based off the loss of the new point. If the new point is better than any previously seen value, it **expands** the step size to accelerate towards the bottom. Likewise if the new point is worse it **contracts** the step size to converge around the minima. The usual settings are to half the step size when contracting and double the step size when expanding.

This method can be easily extended into higher dimensional examples, all that's required is taking one more point than there are dimensions. Then, the simplest approach is to replace the worst point with a point reflected through the centroid of the remaining n points. If this point is better than the best current point, then we can try stretching exponentially out along this line. On the other hand, if this new point isn't much better than the previous value, then we are stepping across a valley, so we shrink the step towards a better point.

> See "An Interactive Tutorial on Numerical Optimization": http://www.benfrederickson.com/numerical-optimization/

## Gradient descend (for *hackers*) for function minimization: 1-D

Let's suppose that we have a function $f: \mathbb{R} \rightarrow \mathbb{R}$. For example:

$$f(x) = x^2$$

Our objective is to find the argument  $x$ that minimizes this function (for maximization, consider $-f(x)$). To this end, the critical concept is the **derivative**.

The derivative of $f$ of a variable $x$, $f'(x)$ or $\frac{\partial f}{\partial x}$,  is a measure of the rate at which the value of the function changes with respect to the change of the variable.

It is defined as the following limit:

$$ f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x)}{h} $$

The derivative specifies how to scale a small change in the input in order to obtain the corresponding change in the output:

$$ f(x + h) \approx f(x) + h f'(x)$$

```python
# numerical derivative at a point x

def f(x):
    return x**2

def fin_dif(x,
            f,
            h = 0.00001):
    '''
    This method returns the derivative of f at x
    by using the finite difference method
    '''
    return (f(x+h) - f(x))/h

x = 2.0
print "{:2.4f}".format(fin_dif(x,f))
```

    4.0000

The limit as $h$ approaches zero, if it exists, should represent the **slope of the tangent line** to $(x, f(x))$.

For values that are not zero it is only an approximation.

> **NOTE**: It can be shown that the “centered difference formula" is better when computing numerical derivatives:

> $$ \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h} $$

> The error in the "finite difference" approximation can be derived from Taylor's theorem and, assuming that $f$ is differentiable, is $O(h)$. In the case of “centered difference" the error is $O(h^2)$.

The derivative tells how to chage $x$ in order to make a small improvement in $f$.

Then, we can follow these steps to decrease the value of the function:

+ Start from a random $x$ value.
+ Compute the derivative $f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h}$.
+ Walk a small step (possibly weighted by the derivative module) in the **opposite** direction of the derivative, because we know that $f(x - h \mbox{ sign}(f'(x)) < f(x)$ for  small enough $h$.

The search for the minima ends when the derivative is zero because we have no more information about which direction to move. $x$ is a critical o stationary point if $f'(x)=0$.

 + A **minimum (maximum)** is a critical point where $f(x)$ is lower (higher) than at all neighboring points.
 + There is a third class of critical points: **saddle points**.

If $f$ is a **convex function**, this should be the minimum (maximum) of our functions. In other cases it could be a local minimum (maximum) or a saddle point.

```python
x = np.linspace(-15,15,100)
y = x**2

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        20,
        'Minimum',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

```python
x = np.linspace(-15,15,100)
y = -x**2

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-250,10])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        -30,
        'Maximum',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

```python
x = np.linspace(-15,15,100)
y = x**3

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-3000,3000])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        400,
        'Saddle Point',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

There are two problems with numerical derivatives:
+ It is approximate.
+ It is very slow to evaluate (two function evaluations: $f(x + h) , f(x - h)$ ).

Our knowledge from Calculus could help!

We know that we can get an **analytical expression** of the derivative for **some** functions.

For example, let's suppose we have a simple quadratic function, $f(x)=x^2−6x+5$, and we want to find the minimum of this function.

#### First approach

We can solve this analytically using Calculus, by finding the derivate $f'(x)  =  2x-6$ and setting it to zero:

\begin{equation}
\begin{split}
2x-6  & = & 0 \\
2x  & = & 6 \\
x & = & 3 \\
\end{split}
\end{equation}

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([3],[3**2 - 6*3 + 5],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(3,
        10,
        'Min: x = 3',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

#### Second approach

To find the local minimum using **gradient descend**: you start at a random point, and move into the direction of steepest **descent** relative to the derivative:

+ Start from a random $x$ value.
+ Compute the derivative $f'(x)$ analitically.
+ Walk a small step in the **opposite** direction of the derivative.

In this example, let's suppose we start at $x=15$. The derivative at this point is $2×15−6=24$.

Because we're using gradient descent, we need to subtract the gradient from our $x$-coordinate: $f(x - f'(x))$. However, notice that $15−24$ gives us $−9$, clearly overshooting over target of $3$.

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5
start = 15

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([start],[start**2 - 6*start + 5],'o')
ax.text(start,
        start**2 - 6*start + 35,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )

d = 2 * start - 6
end = start - d

plt.plot([end],[end**2 - 6*end + 5],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(end,
        start**2 - 6*start + 35,
        'End',
        ha='center',
        color=sns.xkcd_rgb['green'],
       )
plt.show
```

To fix this, we multiply the gradient by a step size. This step size (often called **alpha**) has to be chosen carefully, as a value too small will result in a long computation time, while a value too large will not give you the right result (by overshooting) or even fail to converge.

In this example, we'll set the step size to 0.01, which means we'll subtract $24×0.01$ from $15$, which is $14.76$.

This is now our new temporary local minimum: We continue this method until we either don't see a change after we subtracted the derivative step size (or until we've completed a pre-set number of iterations).

```python
old_min = 0
temp_min = 15
step_size = 0.01
precision = 0.0001

def f(x):
    return x**2 - 6*x + 5

def f_derivative(x):
    import math
    return 2*x -6

mins = []
cost = []

while abs(temp_min - old_min) > precision:
    old_min = temp_min
    gradient = f_derivative(old_min)
    move = gradient * step_size
    temp_min = old_min - move
    cost.append((3-temp_min)**2)
    mins.append(temp_min)

# rounding the result to 2 digits because of the step size
print "Local minimum occurs at {:3.6f}.".format(round(temp_min,2))
```

    Local minimum occurs at 3.000000.

An important feature of gradient descent is that **there should be a visible improvement over time**: In this example, we simply plotted the squared distance from the local minima calculated by gradient descent and the true local minimum,  ``cost``, against the iteration during which it was calculated. As we can see, the distance gets smaller over time, but barely changes in later iterations.

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

x, y = (zip(*enumerate(cost)))

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-', alpha=0.7)
plt.ylim([-10,150])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.plot(mins,cost,'o', alpha=0.3)
ax.text(start,
        start**2 - 6*start + 25,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
ax.text(mins[-1],
        cost[-1]+20,
        'End (%s steps)' % len(mins),
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
plt.show
```

## From derivatives to gradient: $n$-dimensional function minimization.

Let's consider a $n$-dimensional function $f: \Re^n \rightarrow \Re$. For example:

$$f(\mathbf{x}) = \sum_{n} x_n^2$$

Our objective is to find the argument  $\mathbf{x}$ that minimizes this function.

The **gradient** of $f$ is the vector whose components are the $n$ partial derivatives of $f$. It is thus a vector-valued function.

The gradient points in the direction of the greatest rate of **increase** of the function.

$$\nabla {f} = (\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n})$$

```python
def f(x):
    return sum(x_i**2 for x_i in x)

def fin_dif_partial_centered(x,
                             f,
                             i,
                             h=1e-6):
    '''
    This method returns the partial derivative of the i-th
    component of f at x
    by using the centered finite difference method
    '''
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def fin_dif_partial_old(x,
                        f,
                        i,
                        h=1e-6):
    '''
    This method returns the partial derivative of the i-th
    component of f at x
    by using the (non-centered) finite difference method
    '''
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(x))/h

def gradient_centered(x,
                      f,
                      h=1e-6):
    '''
    This method returns the gradient vector of f at x
    by using the centered finite difference method
    '''
    return[round(fin_dif_partial_centered(x,f,i,h), 10) for i,_ in enumerate(x)]

def gradient_old(x,
                 f,
                 h=1e-6):
    '''
    This method returns the the gradient vector of f at x
    by using the (non-centered)ç finite difference method
    '''
    return[round(fin_dif_partial_old(x,f,i,h), 10) for i,_ in enumerate(x)]

x = [1.0,1.0,1.0]

print '{:.6f}'.format(f(x)), gradient_centered(x,f)
print '{:.6f}'.format(f(x)), gradient_old(x,f)
```

    3.000000 [2.0000000001, 2.0000000001, 2.0000000001]
    3.000000 [2.0000009999, 2.0000009999, 2.0000009999]

The function we have evaluated, $f({\mathbf x}) = x_1^2+x_2^2+x_3^2$, is $3$ at $(1,1,1)$ and the gradient vector at this point is $(2,2,2)$.

Then, we can follow this steps to maximize (or minimize) the function:

+ Start from a random $\mathbf{x}$ vector.
+ Compute the gradient vector.
+ Walk a small step in the opposite direction of the gradient vector.

> It is important to be aware that gradient computation is very expensive: if $\mathbf{x}$ has dimension $n$, we have to evaluate $f$ at $2*n$ points.

### How to use the gradient.

$f(x) = \sum_i x_i^2$, takes its mimimum value when all $x$ are 0.

Let's check it for $n=3$:

```python
def euc_dist(v1,v2):
    import numpy as np
    import math
    v = np.array(v1)-np.array(v2)
    return math.sqrt(sum(v_i ** 2 for v_i in v))
```

Let's start by choosing a random vector and then walking a step in the opposite direction of the gradient vector. We will stop when the difference between the new solution and the old solution is less than a tolerance value.

```python
# choosing a random vector

import random
import numpy as np

x = [random.randint(-10,10) for i in range(3)]
x
```

    [5, -7, 7]

```python
def step(x,
         grad,
         alpha):
    '''
    This function makes a step in the opposite direction of
    the gradient vector
    in order to compute a new value for the target function.
    '''
    return [x_i - alpha * grad_i for x_i, grad_i in zip(x,grad)]

tol = 1e-15
alpha = 0.01
while True:
    grad = gradient_centered(x,f)
    next_x = step(x,grad,alpha)
    if euc_dist(next_x,x) < tol:
        break
    x = next_x
print [round(i,10) for i in x]
```

    [0.0, -0.0, 0.0]

### Alpha

The step size, **alpha**, is a slippy concept: if it is too small we will slowly converge to the solution, if it is too large we can diverge from the solution.

There are several policies to follow when selecting the step size:

+ Constant size steps. In this case, the size step determines the precision of the solution.
+ Decreasing step sizes.
+ At each step, select the optimal step.

The last policy is good, but too expensive. In this case we would consider a fixed set of values:

```python
step_size = [100, 10, 1, 0.1, 0.01, 0.001, 0.0001, 0.00001]
```

## Learning from data

In general, we have:

+ A dataset $(\mathbf{x},y)$ of $n$ examples.
+ A target function $f_\mathbf{w}$, that we want to minimize, representing the **discrepancy between our data and the model** we want to fit. The model is represented by a set of parameters $\mathbf{w}$.
+ The gradient of the target function, $g_f$.

In the most common case $f$ represents the errors from a data representation model $M$. To fit the model is to find the optimal parameters $\mathbf{w}$ that minimize the following expression:

$$ f_\mathbf{w} = \frac{1}{n} \sum_{i} (y_i - M(\mathbf{x}_i,\mathbf{w}))^2 $$

For example, $(\mathbf{x},y)$ can represent:

+ $\mathbf{x}$: the behavior of a "Candy Crush" player; $y$: monthly payments.
+ $\mathbf{x}$: sensor data about your car engine; $y$: probability of engine error.
+ $\mathbf{x}$: finantial data of a bank customer; $y$: customer rating.

> If $y$ is a real value, it is called a *regression* problem.

> If $y$ is binary/categorical, it is called a *classification* problem.

Let's suppose that our model is a one-dimensional linear model $M(\mathbf{x},\mathbf{w}) = w \cdot x $.

### Batch gradient descend

We can implement **gradient descend** in the following way (*batch gradient descend*):

```python
import numpy as np
import random

# f = 2x
x = np.arange(10)
y = np.array([2*i for i in x])

# f_target = 1/n Sum (y - wx)**2
def target_f(x,y,w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,y,w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def step(w,grad,alpha):
    return w - alpha * grad

def BGD_multi_step(target_f,
                   gradient_f,
                   x,
                   y,
                   toler = 1e-6):
    '''
    Batch gradient descend by using a multi-step approach
    '''
    alphas = [100, 10, 1, 0.1, 0.001, 0.00001]
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_ws = [step(w, gradient, alpha) for alpha in alphas]
        next_vals = [target_f(x,y,w) for w in next_ws]
        min_val = min(next_vals)
        next_w = next_ws[next_vals.index(min_val)]
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
```

```python
print '{:.6f}'.format(BGD_multi_step(target_f, gradient_f, x, y))
```

    1.999635

```python
%%timeit
BGD_multi_step(target_f, gradient_f, x, y)
```

    100 loops, best of 3: 4.27 ms per loop

```python
def BGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        alpha=0.01):
    '''
    Batch gradient descend by using a given step
    '''
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_w = step(w, gradient, alpha)
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
```

```python
print '{:.6f}'.format(BGD(target_f, gradient_f, x, y))
```

    2.000097

```python
%%timeit
BGD(target_f, gradient_f, x, y)
```

    The slowest run took 5.62 times longer than the fastest. This could mean that an intermediate result is being cached.
    10000 loops, best of 3: 101 µs per loop

### Stochastic Gradient Descend

The last function evals the whole dataset $(\mathbf{x}_i,y_i)$ at every step.

If the dataset is large, this strategy is too costly. In this case we will use a strategy called **SGD** (*Stochastic Gradient Descend*).

When learning from data, the cost function is additive: it is computed by adding sample reconstruction errors.

Then, we can compute the estimate the gradient (and move towards the minimum) by using only **one data sample** (or a small data sample).

Thus, we will find the minimum by iterating this gradient estimation over the dataset.

A full iteration over the dataset is called **epoch**. During an epoch, data must be used in a random order.

If we apply this method we have some theoretical guarantees to find a good minimum:
+ SGD essentially uses the inaccurate gradient per iteration. Since there is no free food, what is the cost by using approximate gradient? The answer is that the convergence rate is slower than the gradient descent algorithm.
+ The convergence of SGD has been analyzed using the theories of convex minimization and of stochastic approximation: it converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum.

```python
import numpy as np
x = np.arange(10)
y = np.array([2*i for i in x])
data = zip(x,y)

for (x_i,y_i) in data:
    print '{:3d} {:3d}'.format(x_i,y_i)
print

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

for (x_i,y_i) in in_random_order(data):
    print '{:3d} {:3d}'.format(x_i,y_i)
```

      0   0
      1   2
      2   4
      3   6
      4   8
      5  10
      6  12
      7  14
      8  16
      9  18

      5  10
      2   4
      1   2
      0   0
      3   6
      4   8
      7  14
      6  12
      9  18
      8  16

```python
import numpy as np
import random

def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):
    '''
    Stochastic gradient descend with automatic step adaptation (by
    reducing the step to its 95% when there are iterations with
    no increase)
    '''
    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    epoch = 0
    iteration_no_increase = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w
```

```python
print 'w: {:.6f}'.format(SGD(target_f, gradient_f, x, y))
```

    w: 2.000000

## Exercise: Stochastic Gradient Descent and Linear Regression

The linear regression model assumes a linear relationship between data:

$$ y_i = w_1 x_i + w_0 $$

Let's generate a more realistic dataset (with noise), where $w_1 = 2$ and $w_0 = 0$.

```python
%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

    Once deleted, variables cannot be recovered. Proceed (y/[n])? y

```python
# x: input data
# y: noisy output data

x = np.random.uniform(0,1,20)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$f(x)$', fontsize=15)
plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

Complete the following code in order to:
+ Compute the value of $w$ by using a estimator based on minimizing the squared error.
+ Get from SGD function a list, `target_value`, representing the value of the target function at each iteration.

```python
# Write your target function as f_target 1/n Sum (y - wx)**2
def target_f(x,y,w):
    # your code here
    return

# Write your gradient function
def gradient_f(x,y,w):
    # your code here
    return

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

# Modify the SGD function to return a 'target_value' vector
def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):

    # Insert your code among the following lines

    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    iteration_no_increase = 0
    epoch = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w
```

```python
# Print the value of the solution

w, target_value = SGD(target_f, gradient_f, x, y)
print 'w: {:.6f}'.format(w)
```

```python
# Visualize the solution regression line

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x, y, 'o', label='t')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)', alpha=0.5)
plt.plot([0, 1], [0*w, 1*w], 'r-', label='fitted line', alpha=0.5, linestyle='--')
plt.xlabel('input x')
plt.ylabel('target t')
plt.title('input vs. target')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

```python
# Visualize the evolution of the target function value during iterations.

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(np.arange(target_value.size), target_value, 'o', alpha = 0.2)
plt.xlabel('Iteration')
plt.ylabel('Cost')
plt.grid()
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show()
```

## Mini-batch Gradient Descent

In code, general batch gradient descent looks something like this:

```python
nb_epochs = 100
for i in range(nb_epochs):
    grad = evaluate_gradient(target_f, data, w)
    w = w - learning_rate * grad
```

For a pre-defined number of epochs, we first compute the gradient vector of the target function for the whole dataset w.r.t. our parameter vector.

**Stochastic gradient descent** (SGD) in contrast performs a parameter update for each training example and label:

```python
nb_epochs = 100
for i in range(nb_epochs):
    np.random.shuffle(data)
    for sample in data:
        grad = evaluate_gradient(target_f, sample, w)
        w = w - learning_rate * grad
```

Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of $n$ training examples:

```python
nb_epochs = 100
for i in range(nb_epochs):
  np.random.shuffle(data)
  for batch in get_batches(data, batch_size=50):
    grad = evaluate_gradient(target_f, batch, w)
    w = w - learning_rate * grad
```

Minibatch SGD has the advantage that it works with a slightly less noisy estimate of the gradient. However, as the minibatch size increases, the number of updates done per computation done decreases (eventually it becomes very inefficient, like batch gradient descent).

There is an optimal trade-off (in terms of computational efficiency) that may vary depending on the data distribution and the particulars of the class of function considered, as well as how computations are implemented.

```python
def get_batches(iterable,
                num_elem_batch = 1):
    '''
    Generator of batches from an iterable that contains data
    '''
    current_batch = []
    for item in iterable:
        current_batch.append(item)
        if len(current_batch) == num_elem_batch:
            yield current_batch
            current_batch = []
    if current_batch:
        yield current_batch

x = np.array(range(0, 10))
y = np.array(range(10, 20))
data = zip(x,y)
np.random.shuffle(data)
for x in get_batches(data, 3):
    print x

print

for batch in get_batches(data, 3):
    print np.array(zip(*batch)[0]), np.array(zip(*batch)[1])
```

    [(5, 15), (2, 12), (4, 14)]
    [(9, 19), (8, 18), (7, 17)]
    [(1, 11), (6, 16), (3, 13)]
    [(0, 10)]

    [5 2 4] [15 12 14]
    [9 8 7] [19 18 17]
    [1 6 3] [11 16 13]
    [0] [10]

```python
%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

    Once deleted, variables cannot be recovered. Proceed (y/[n])? y

```python
# x: input data
# y: noisy output data

x = np.random.uniform(0,1,2000)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$t$', fontsize=15)
plt.ylim([0,2])
plt.title('inputs (x) vs targets (y)')
plt.grid()
plt.legend(loc=2)
plt.gcf().set_size_inches((10,3))
plt.show()
```

```python
# f_target = 1/n Sum (y - wx)**2
def target_f(x,
             y,
             w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,
               y,
               w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

def get_batches(iterable,
                num_elem_batch = 1):
    '''
    Generator of batches from an iterable that contains data
    '''
    current_batch = []
    for item in iterable:
        current_batch.append(item)
        if len(current_batch) == num_elem_batch:
            yield current_batch
            current_batch = []
    if current_batch:
        yield current_batch

def SGD_MB(target_f, gradient_f, x, y, epochs=100, alpha_0=0.01):
    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    epoch = 0
    while epoch < epochs:
        val = target_f(x, y, w)
        if val < min_val:
            min_w, min_val = w, val
            alpha = alpha_0
        else:
            alpha *= 0.9
        np.random.shuffle(data)
        for batch in get_batches(data, num_elem_batch = 100):
            x_batch = np.array(zip(*batch)[0])
            y_batch = np.array(zip(*batch)[1])
            gradient = gradient_f(x_batch, y_batch, w)
            w = w - (alpha *  gradient)
        epoch += 1
    return min_w
```

```python
w = SGD_MB(target_f, gradient_f, x, y)
print 'w: {:.6f}'.format(w)
```

    w: 2.003238

```python
plt.plot(x, y, 'o', label='t')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)', alpha=0.5)
plt.plot([0, 1], [0*w, 1*w], 'r-', label='fitted line', alpha=0.5, linestyle='--')
plt.xlabel('input x')
plt.ylabel('target t')
plt.ylim([0,2])
plt.title('input vs. target')
plt.grid()
plt.legend(loc=2)
plt.gcf().set_size_inches((10,3))
plt.show()
```

## Loss Funtions

Loss functions $L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \ell(y_i, f(\mathbf{x_i}))$ represent the price paid for inaccuracy of predictions in classification/regression problems.

In classification this function is often the **zero-one loss**, that is, $ \ell(y_i, f(\mathbf{x_i}))$ is zero when $y_i = f(\mathbf{x}_i)$ and one otherwise.

This function is discontinuous with flat regions and is thus extremely hard to optimize using gradient-based methods. For this reason it is usual to consider a proxy to the loss called a *surrogate loss function*. For computational reasons this is usually convex function. Here we have some examples:

### Square / Euclidean Loss

In regression problems, the most common loss function is the square loss function:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (y_i - f(\mathbf{x}_i))^2  $$

The square loss function can be re-written and utilized for classification:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (1 - y_i f(\mathbf{x}_i))^2  $$

### Hinge / Margin Loss (i.e. Suport Vector Machines)

The hinge loss function is defined as:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \mbox{max}(0, 1 - y_i f(\mathbf{x}_i))  $$

The hinge loss provides a relatively tight, convex upper bound on the 0–1 Loss.

### Logistic Loss (Logistic Regression)

This function displays a similar convergence rate to the hinge loss function, and since it is continuous, simple gradient descent methods can be utilized.

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i log(1 + exp(-y_i f(\mathbf{x}_i))) $$

### Sigmoid Cross-Entropy Loss (Softmax classifier)

Cross-Entropy is a loss function that is very used for training **multiclass problems**. We'll focus on models that assume that classes are mutually exclusive.

In this case, our labels have this form $\mathbf{y}_i =(1.0,0.0,0.0)$. If our model predicts a different distribution, say  $ f(\mathbf{x}_i)=(0.4,0.1,0.5)$, then we'd like to nudge the parameters so that $f(\mathbf{x}_i)$ gets closer to $\mathbf{y}_i$.

C.Shannon showed that if you want to send a series of messages composed of symbols from an alphabet with distribution $y$ ($y_j$  is the probability of the  $j$-th symbol), then to use the smallest number of bits on average, you should assign  $\log(\frac{1}{y_j})$  bits to the  $j$-th symbol.

The optimal number of bits is known as **entropy**:

$$ H(\mathbf{y}) = \sum_j y_j \log\frac{1}{y_j} = - \sum_j y_j \log y_j$$

**Cross entropy** is the number of bits we'll need if we encode symbols by using a wrong distribution $\hat y$:

$$ H(y, \hat y) =   - \sum_j y_j \log \hat y_j $$

In our case, the real distribution is $\mathbf{y}$ and the "wrong" one is $f(\mathbf{x}_i)$. So, minimizing **cross entropy** with respect our model parameters will result in the model that best approximates our labels if considered as a probabilistic distribution.

Cross entropy is used in combination with **Softmax** classifier. In order to classify $\mathbf{x}_i$ we could take the index corresponding to the max value of $f(\mathbf{x}_i)$, but Softmax gives a slightly more intuitive output (normalized class probabilities) and also has a probabilistic interpretation:

$$ P(\mathbf{y}_i = j \mid \mathbf{x_i}) = - log \left( \frac{e^{f_j(\mathbf{x_i})}}{\sum_k e^{f_k(\mathbf{x_i})} } \right) $$

where $f_k$ is a linear classifier.

## Advanced gradient descend

### Momentum

SGD has trouble navigating ravines, i.e. areas where the surface curves much more steeply in one dimension than in another, which are common around local optima. In these scenarios, SGD oscillates across the slopes of the ravine while only making hesitant progress along the bottom towards the local optimum.

Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations. It does this by adding a fraction of the update vector of the past time step to the current update vector:

$$ v_t = m v_{t-1} + \alpha \nabla_w f $$
$$ w = w - v_t    $$

The momentum $m$ is commonly set to $0.9$.

### Nesterov

However, a ball that rolls down a hill, blindly following the slope, is highly unsatisfactory. We'd like to have a smarter ball, a ball that has a notion of where it is going so that it knows to slow down before the hill slopes up again.

Nesterov accelerated gradient (NAG) is a way to give our momentum term this kind of prescience. We know that we will use our momentum term $m v_{t-1}$ to move the parameters $w$. Computing
$w - m v_{t-1}$ thus gives us an approximation of the next position of the parameters (the gradient is missing for the full update), a rough idea where our parameters are going to be. We can now effectively look ahead by calculating the gradient not w.r.t. to our current parameters $w$ but w.r.t. the approximate future position of our parameters:

$$ w_{new} = w - m v_{t-1} $$
$$ v_t = m v_{t-1} + \alpha \nabla_{w_{new}} f $$
$$ w = w - v_t $$

### Adagrad

All previous approaches manipulated the learning rate globally and equally for all parameters. Tuning the learning rates is an expensive process, so much work has gone into devising methods that can adaptively tune the learning rates, and even do so per parameter.

Adagrad is an algorithm for gradient-based optimization that does just this: It adapts the learning rate to the parameters, performing larger updates for infrequent and smaller updates for frequent parameters.

$$ c = c + (\nabla_w f)^2 $$
$$ w = w - \frac{\alpha}{\sqrt{c}} $$

### RMProp

RMSProp update adjusts the Adagrad method in a very simple way in an attempt to reduce its aggressive, monotonically decreasing learning rate. In particular, it uses a moving average of squared gradients instead, giving:

$$ c = \beta c + (1 - \beta)(\nabla_w f)^2 $$
$$ w = w - \frac{\alpha}{\sqrt{c}} $$

where $\beta$ is a decay rate that controls the size of the moving average.

(Image credit: Alec Radford)

(Image credit: Alec Radford)

```python
%reset

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

    Once deleted, variables cannot be recovered. Proceed (y/[n])? y

```python
# the function that I'm going to plot
def f(x,y):
    return x**2 + 5*y**2

x = np.arange(-3.0,3.0,0.1)
y = np.arange(-3.0,3.0,0.1)
X,Y = np.meshgrid(x, y, indexing='ij') # grid of point
Z = f(X, Y) # evaluation of the function on the grid

plt.pcolor(X, Y, Z, cmap=plt.cm.gist_earth)
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.gca().set_aspect('equal', adjustable='box')
plt.gcf().set_size_inches((6,6))
plt.show()
```

```python
def target_f(x):
    return x[0]**2.0 + 5*x[1]**2.0

def part_f(x,
           f,
           i,
           h=1e-6):
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def gradient_f(x,
               f,
               h=1e-6):
    return np.array([round(part_f(x,f,i,h), 10) for i,_ in enumerate(x)])
```

```python
def SGD(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001):
    alpha = alpha_0
    min_val = float('inf')
    steps = 0
    iteration_no_increase = 0
    trace = []
    while iteration_no_increase < 100:
        val = target_f(x)
        if min_val - val > toler:
            min_val = val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            alpha *= 0.95
            iteration_no_increase += 1
        trace.append(x)
        gradient_i = gradient_f(x, target_f)
        x = x - (alpha *  gradient_i)
        steps += 1
    return x, val, steps, trace

x = np.array([2,-2])
x, val, steps, trace = SGD(target_f, gradient_f, x)
print x
print 'Val: {:.6f}, steps: {:.0f}'.format(val, steps)
```

    [  8.73043864e-04  -4.00026038e-12]
    Val: 0.000001, steps: 475

```python
def SGD_M(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001,
        m = 0.9):

    alpha = alpha_0
    min_val = float('inf')
    steps = 0
    iteration_no_increase = 0
    v = 0.0
    trace = []
    while iteration_no_increase < 100:
        val = target_f(x)
        if min_val - val > toler:
            min_val = val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            alpha *= 0.95
            iteration_no_increase += 1
        trace.append(x)
        gradient_i = gradient_f(x, target_f)
        v = m * v + (alpha *  gradient_i)
        x = x - v
        steps += 1
    return x, val, steps, trace

x = np.array([2,-2])
x, val, steps, trace2 = SGD_M(target_f, gradient_f, x)

print '\n',x
print 'Val: {:.6f}, steps: {:.0f}'.format(val, steps)
```

    [  1.90552394e-05   1.27080227e-05]
    Val: 0.000000, steps: 241

```python
x2 = np.array(range(len(trace)))
x3 = np.array(range(len(trace2)))
plt.xlim([0,len(trace)])
plt.gcf().set_size_inches((10,3))
plt.plot(x3, trace2)
plt.plot(x2, trace, '-')
```


# Solução dos exercícios propostos na aula 4

Ao fim da aula 4, foram propostos alguns exercícios envolvendo interpolação. A seguir estão os exercícios. Em seguida, vem uma breve recapitulação da aula. A solução está logo após.

## Exercícios

### Problema 1

Interpole a função de Runge com $-1 \le x \le 1$:
$$
f(x) = \frac{1}{1 + 25x^2}
$$

 1. Use 11 pontos uniformemente distribuídos
 2. Aumente o número de pontos
 3. Tente usar os pontos $x_k = \cos\left(\frac{k\pi}{N}\right)$ para $k = 0\ldots N$.
 4. Brinque com o número de pontos

### Problema 2

Procure na Net o método de diferenças divididas de Newton a interpole a função anterior nos mesmos pontos. Este método é simplesmente um jeito inteligente de resolver a matriz apresentada lá em cima.

### Problema 3

Use a biblioteca Interpolations.jl e Dierckx.jl para fazer as interpolações. Compare a interpolação linear com os splines.

<!-- TEASER_END -->

# Interpolação

Dado um conjunto de n pontos $(x_i, y_i)$, qual o poliômio que passa por todos?

$$
y_i = a_0 + a_1 x_i + a_2 x_i^2 + \ldots a_n x_i^n \qquad i=1, \ldots, m
$$

## Vandermonde

Com n+1 pontos distintos, se o polinômio for de grau n, pode-se montar o seguinte sistema linear:

$$
\begin{bmatrix}
1 & x_0 & x_0^2 & \cdots & x_0^n \\
1 & x_1 & x_1^2 & \cdots & x_1^n \\
\vdots & \vdots & \vdots & \ddots & \vdots\\
1 & x_n & x_n^2 & \cdots & x_n^n\\
\end{bmatrix}\cdot
\left\{ \begin{matrix} a_0 \\ a_1 \\ \vdots \\ a_n\\ \end{matrix}\right\}
= \left\{\begin{matrix} y_0 \\ y_1 \\ \vdots \\ y_n\\\end{matrix}\right\}
$$

Mas esta é uma operação cara, $\mathcal{O}(n^3)$!

## Outra possibilidade

$$
y = f(x) \approx a_0 + a_1(x-x_0) + a_2(x-x_0)(x-x_1) + \cdots + a_n(x-x_0)(x-x_1)\cdots(x - x_{n-1})
$$

Com isso chegamos ao seguinte sistema linear triangular:
$$
\begin{bmatrix}
1 & 0 & 0 &  0 &\cdots & 0\\
1 & (x_1-x_0) & 0 & 0 &  \cdots & 0\\
1 & (x_2 - x_0) &  (x_2 - x_0)(x_2 - x_1) & 0 & \cdots & 0\\
\vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
1 & (x_n - x_0) &  (x_n - x_0)(x_n - x_1) & (x_n - x_0)(x_n - x_1)(x_n-x_2) & \cdots &(x_n-x_0)(x_n-x_1)\cdots(x_n-x_{n-1})\\
\end{bmatrix}\cdot\left\{\begin{matrix} a_0\\ a_1 \\ a_2 \\ \vdots \\ a_n\end{matrix}\right\} =
\left\{\begin{matrix} y_0\\ y_1 \\ y_2 \\ \vdots \\ y_n\end{matrix}\right\}
$$

Resolver este sistema é muito mais barato: $\mathcal{O}(n^2)$

## Interpolação de Lagrange:

$$
y(x) = \sum_{i=1}^n y_i h_i(x)
$$

onde $h_i(x)$ é o interpolador de Lagrange:

$$
h_k(x) = \prod_{i=1\ldots n,}^n \frac{x - x_i}{x_k - x_i} \qquad i\ne k
$$

Propriedade:
$$
h_i(x_i) = \delta_{ij} \quad \text{onde} \quad \delta_{ij} = \left\{\begin{matrix}1, \: i=j \\ 0, i\ne j\\ \end{matrix}\right.
$$

### Vamos organizar a interpolação de Lagrange

```julia
struct Lagrange
    x::Vector{Float64}
    y::Vector{Float64}
    Lagrange(x, y) = new(copy(x), copy(y))
end
Base.Broadcast.broadcastable(lgr::Lagrange) = Ref(lgr)

function lagrange(k, z, x)
 h = 1.0
    n = length(z)
    for i = 1:(k-1)
        h *= (x - z[i]) / (z[k] - z[i])
    end

    for i = (k+1):n
        h *= (x - z[i]) / (z[k] - z[i])
    end
    return h
end

function interp(lgr::Lagrange, x)

    y = lgr.y[1] * lagrange(1, lgr.x, x)

    for i = 2:length(lgr.x)
        y += lgr.y[i] * lagrange(i, lgr.x, x)
    end

    return y
end

(lgr::Lagrange)(x) = interp(lgr, x)
```

## Interpolação Linear

```julia
struct LinearInterp
    x::Vector{Float64}
    y::Vector{Float64}
    LinearInterp(x, y) = new(copy(x), copy(y))
end
Base.Broadcast.broadcastable(lin::LinearInterp) = Ref(lin)

function interp1(lin::LinearInterp, x)

    if x < lin.x[1] || x > lin.x[end]
        error("Fora do Range")
    end

    index = 2
    n = length(lin.x)
    for i = 2:n
        if lin.x[i] >= x
            index = i
            break
        end
    end
    i1 = index-1
    return lin.y[i1] + (lin.y[index] - lin.y[i1]) * (x - lin.x[i1]) / (lin.x[index] - lin.x[i1])

end

function interp2(lin::LinearInterp, x)
    index = searchsortedfirst(lin.x, x)
    if index == 1
        index = 2
    end
    i1 = index-1
    return lin.y[i1] + (lin.y[index] - lin.y[i1]) * (x - lin.x[i1]) / (lin.x[index] - lin.x[i1])

end

(lin::LinearInterp)(x) = interp1(lin, x)

```

# Pacotes

 * [Interpolations](https://github.com/JuliaMath/Interpolations.jl)
 * [Dierckx](https://github.com/kbarbary/Dierckx.jl)
 * [GridInterpolations](https://github.com/sisl/GridInterpolations.jl)

```julia
using PyPlot
using BenchmarkTools
using LinearAlgebra
```

# Problema 1

```julia
f(x) = 1.0 / (1.0 + 25x^2)

xx = -1:0.001:1
yy = f.(xx);
```

## Pontos igualmente espaçados

```julia
function ploteqspacing(N, dx=0.001)
    xp = range(-1.0, 1.0, length=N)
    yp = f.(xp)
    lgr = Lagrange(xp, yp)
    x1 = -1:dx:1
    y1 = lgr.(x1)
    y0 = f.(x1)
    plot(xp, yp, "ro")
    plot(x1, y0, "r-")
    plot(x1, y1)
end

```

```julia
ploteqspacing(11)

```

```julia
ploteqspacing(5)
```

```julia
ploteqspacing(8)
```

```julia
ploteqspacing(15)
```

```julia
ploteqspacing(16)
```

## Distribuição do cosseno

```julia
function plotcosspacing(N, dx=0.001)
    xp = [cos(k*π/N) for k in 0:N]
    yp = f.(xp)
    lgr = Lagrange(xp, yp)
    x1 = -1:dx:1
    y1 = lgr.(x1)
    y0 = f.(x1)
    plot(xp, yp, "ro")
    plot(x1, y0, "r-")
    plot(x1, y1)
end
```

```julia
plotcosspacing(10)
```

```julia
plotcosspacing(4)
```

```julia
plotcosspacing(8)
```

```julia
plotcosspacing(16)
```

```julia
plotcosspacing(20)
```

# Problema 2

Diferenças divididas

```julia
function divdiffmat(x)
    n = length(x)

    A = zeros(n,n)
    for i = 1:n
        A[i,1] = 1.0
        for k in 2:i
            A[i, k] = A[i,k-1] * (x[i] - x[k-1])
        end
    end

    return A

end

```

```julia
struct DividedDiff
    x::Vector{Float64}
    a::Vector{Float64}
end
function divideddiff(x, y)
    A = LowerTriangular(divdiffmat(x))
    a = A\y
    return DividedDiff(copy(x), a)
end
Base.Broadcast.broadcastable(ddif::DividedDiff) = Ref(ddif)

function interp(ddif, x)
    xx = ddif.x
    a = ddif.a

    y = a[end]

    for i in (lastindex(a)-1):-1:1
        y = a[i] + (x-xx[i])*y
    end
    return y
end
(ddif::DividedDiff)(x) = interp(ddif, x)
```

```julia
fun2(x) = 1.0 + 2.0*x + 3.0*x*(x-1.0) + 4.0*x*(x-1.0)*(x-2.0) + 5.0*x*(x-1.0)*(x-2.0)*(x-3.0)
x1 = [0.0, 1.0, 2.0, 3.0, 4.0]
y1 = fun2.(x1);
```

```julia
diff = divideddiff(x1, y1)
```

```julia
x2 = 0:0.01:4
y2 = fun2.(x2)
y2b = diff.(x2);
```

```julia
plot(x1, y1, "rs")
plot(x2, y2, "r-")
plot(x2, y2b, "b:")

```

```julia
x = -1:0.4:1
y = sin.(π*x);
```

```julia
diff = divideddiff(x, y)
```

```julia
y1 = diff.(x);
```

```julia
xx = -1:0.01:1
yy = sin.(π*xx)
yy1 = diff.(xx);
```

```julia
plot(x, y, "rs")
plot(xx, yy, "r-")
plot(xx, yy1, "b:")
```

## Algorítmo de Newton

Um jeito interessante de calcular a interpolação usando diferenças divididas. Ao se resolver o sistema linear, pode-se escrever a solução como:

$$
\begin{align}
a_0 &= f(x_0)\\
a_1 & = \frac{f(x_1) - f(y_0)}{x_1 - x_0}\\
a_2 &= \frac{\frac{f(x_2)-f(x_0)}{x_2-x_0} - \frac{f(x_1) - f(x_0)}{x_1-x_0} }{x_2 - x_1}\\
\vdots &=  \vdots\\
\end{align}
$$

o k-ésimo coeficiente vale:
$$
a_k = \mathcal{F}\left(x_0, x_1, \ldots, x_k\right)
$$

com
$$
\mathcal{F}\left(x_0, x_1, \ldots, x_k\right) = \frac{\mathcal{F}\left(x_0, x_1, \ldots, x_{k-1}\right) -\mathcal{F}\left(x_1, x_1, \ldots, x_k\right)}{x_0-x_k}
$$

```julia
function newton_divdiff(x, y)

    n = length(x)
    F = zeros(n,n)

    for i in 1:n
        F[i,1] = y[i]
    end

    for i = 2:n
        for k in 2:i
            F[i,k] = (F[i,k-1] - F[i-1,k-1]) / (x[i] - x[i-k+1])
        end
    end
    a = [F[i,i] for i in 1:n]

    return DividedDiff(copy(x), a)

end

```

```julia
fun2(x) = 1.0 + 2.0*x + 3.0*x*(x-1.0) + 4.0*x*(x-1.0)*(x-2.0) + 5.0*x*(x-1.0)*(x-2.0)*(x-3.0)
x1 = [0.0, 1.0, 2.0, 3.0, 4.0]
y1 = fun2.(x1);
diff2 = newton_divdiff(x1, y1)

```

## Uma abordagem que aloca menos memória

Mesmo algorítmo anterior mas não aloca a matriz inteira.

```julia
function newton_divdiff2(x, y)

    n = length(x)
    F = zeros(n)
    F0 = zeros(n)
    F1 = zeros(n)

    F0[1] = y[1]
    F1[1] = y[1]
    F[1] = y[1]

    for i = 2:n
        F1[1] = y[i]
        for k in 2:i
            F1[k] = (F1[k-1] - F0[k-1]) / (x[i] - x[i-k+1])
        end
        F[i] = F1[i]
        for k in 1:i
            F0[k] = F1[k]
        end
    end

    return DividedDiff(copy(x), F)

end

```

```julia
fun2(x) = 1.0 + 2.0*x + 3.0*x*(x-1.0) + 4.0*x*(x-1.0)*(x-2.0) + 5.0*x*(x-1.0)*(x-2.0)*(x-3.0)
x1 = [0.0, 1.0, 2.0, 3.0, 4.0]
y1 = fun2.(x1);
diff2 = newton_divdiff2(x1, y1)

```

# Problema 3

Usando a biblioteca Interpolations

```julia
using Interpolations
```

```julia
x = -1:0.2:1
y = f.(x);
```

# Interpolação com malha (*grid*)

```julia
itp1 = interpolate((x,),  y, Gridded(Constant()));
itp2 = interpolate((x,),  y, Gridded(Linear()));
```

```julia
xx = -1:0.001:1
yy = f.(xx)
yy1 = itp1.(xx);
yy2 = itp2.(xx);
plot(x, y, "rs")
plot(xx, yy, "r-")
plot(xx, yy1, "b-")
plot(xx, yy2, "b--")

```

```julia
itp3 = CubicSplineInterpolation((x,), y; bc=Line(OnGrid()));

yy3 = itp3.(xx)

plot(x, y, "rs")
plot(xx, yy, "r-")
plot(xx, yy3, "b:")
```

```julia
function compare(h, x)
    xp = -1:h:1
    yp = f.(xp)

    itp1 =  interpolate((xp,),  yp, Gridded(Constant()));
    itp2 =  interpolate((xp,),  yp, Gridded(Linear()));
    itp3 = CubicSplineInterpolation((xp,), yp, bc=Flat(OnGrid()));
    fx = f(x)
    return itp1(x)-fx, itp2(x)-fx, itp3(x)-fx
end
```

```julia
hstep = [1.0, 0.5, 0.4, 0.2, 0.1, 0.05, 0.02, 0.01, 0.001]
nn = 2.0 ./ hstep
erra = [compare(h, -1+h/4) for h in hstep]
errb = [compare(h, h/4) for h in hstep];
errc = [compare(h, -0.9905) for h in hstep];
errd = [compare(h, 0.0005) for h in hstep];
```

```julia
ea1 = [abs(e[1]) for e in erra]
ea2 = [abs(e[2]) for e in erra]
ea3 = [abs(e[3]) for e in erra]

eb1 = [abs(e[1]) for e in errb]
eb2 = [abs(e[2]) for e in errb]
eb3 = [abs(e[3]) for e in errb];

ec1 = [abs(e[1]) for e in errc]
ec2 = [abs(e[2]) for e in errc]
ec3 = [abs(e[3]) for e in errc];

ed1 = [abs(e[1]) for e in errd]
ed2 = [abs(e[2]) for e in errd]
ed3 = [abs(e[3]) for e in errd];

```

```julia
loglog(nn, ea1, "ro-")
loglog(nn, ea2, "bo-")
loglog(nn, ea3, "go-")

```

```julia
loglog(nn, eb1, "rs-")
loglog(nn, eb2, "bo-")
loglog(nn, eb3, "g^-")

```

```julia
loglog(nn, ec1, "rs-")
loglog(nn, ec2, "bo-")
loglog(nn, ec3, "g^-")

```

```julia
loglog(nn, ed1, "rs-")
loglog(nn, ed2, "bo-")
loglog(nn, ed3, "g^-")

```


The SymPy tutorial (1.3) in Julia

Table of contents

Introduction

Gotchas

Basic Operations

Simplification

Calculus

Solvers

Matrices

Advanced Expression Manipulation

About SymPy.jl

The SymPy package for Julia allows Julia users to interact with  python's SymPy module in a mostly seamless manner, thanks to the power of the PyCall package for Julia. The following pages reexpress the SymPy tutorial illustrating the associated Julia commands. There are some changes, but mostly modest ones.

The SymPy package for Julia essentially does:

it provides a type for symbolic objects, a simple wrapper around the underlying PyCall objects.

Such objects may be created using the SymPy syntax:

```julia
using SymPy
x, y = symbols("x, y")
```

    (x, y)

Alternatively, the @vars macro may be used:

```julia
@vars a b real=true
```

    (a, b)

Symbolic objects are subtypes of numbers, SymPy provides methods for the generic interface for Julia numbers. For example, elementary operations +, -, *, /, ^ are defined as well as more Julia-specific methodssuch as ones, zeros, eps, etc.

SymPy is organized around various modules, from which functions are imported for use. A common SymPy command would be:

```julia
from sympy import *
```

This imports all functions and modules from the base sympy module into the user's namespace. By default, SymPy imports all functions from sympy – but no modules, as there would be several conflicts with fundamental Julia objects (e.g., Matrix). As such, modules must be qualified, as in sympy.Matrix. (SymPy exports the underlying sympy object.) For other modules, SymPy provides the function import_from to load functions from a given module.

The functions imported by SymPy are specialized so that their first argument is of symbolic type. This is not always appropriate, as some SymPy functions naturally take integer values for the first entry. An example is ones. Either the integer can be made symbolic (as with Sym(1)) or the function can be qualified, as in sympy.ones.

SymPy and Julia have matrix (and vector) objects. These may be expressed as elements of type Matrix{Sym} (using Julia's generic matrix type to store symbolic objects. SymPy methods for matrices are available using the getproperty notation (the dot), whereas, Julia methods would just use the fact that Matrix{Sym} will have many generic methods already defined for it.

A quick example:

```julia

@vars x
A = [1 x; x x^3]  # Matrix{Sym}
using LinearAlgebra
A.shape, size(A)
```

    ((2, 2), (2, 2))


# Nyquist

```python
import numpy as np
import sympy as sy
import itertools
import matplotlib.pyplot as plt
import control.matlab as cm
init_printing()
%matplotlib inline
```

## Proportional control of the normalized DC-motor
Zero-order hold sampling of the DC motor with transfer function $G(s)=\frac{1}{s(s+1)}$ gives the discrete time system
\begin{equation} H(z) = \frac{\big(h-1+e^{-h}\big)z + \big(1-e^{-h}-he^{-h}\big)}{z^2 -\big(1+e^{-h}\big)z + e^{-h}} \end{equation}

Let $h=\ln 2 \approx 0.693$. This gives the pulse-transfer function
\begin{equation} H(z) = \frac{B(z)}{A(z)} = \frac{0.19z + 0.15}{z^2 - 1.5z + 0.5} \end{equation}

Proportional control gives the closed loop system
\begin{equation}
H_c(z) = \frac{K H(z)}{KH(z) + 1} = \frac{K B(z)}{A(z) + KB(z)}.
\end{equation}
The characteristic equation of the closed loop system is
\begin{equation}
z^2 + (-1.5+0.19K)z + 0.5+0.15K
\end{equation}

```python
z,h = sy.symbols('z,h')
eh = sy.exp(-h)
H = ( (h-1+eh)*z + (1-eh-h*eh) )/( z*z - (1+eh)*z + 0.5)

B,A = sy.fraction(H)
print A
```

    z**2 - z*(1 + exp(-h)) + 0.5

```python
Hs = {}
Hs2 = {}
H0 = cm.tf([1.0], [1, 1, 0])
Hs[0] = (H0,[1.0],[1.0, 1.0, 0])
Hs2[0] = (H0,[1.0],[1.0, 1.0, 0])
for hh in [0.01, 0.1, 0.2, 0.4, 0.8, 1]:
    Bp = sy.Poly(B.subs(h,hh))
    Ap = sy.Poly(A.subs(h,hh))
    a = []
    for el in Ap.coeffs():
        a.append(float(sy.N(el)))
    b = []
    for el in Bp.coeffs():
        b.append(float(sy.N(el)))
    Hs[hh] = (cm.tf(b,a,hh), b,a)
    Hs2[hh] = (cm.c2d(H0, hh), b,a)

```

```python
z = np.exp(1j*np.linspace(0.02,np.pi, 800))
s = 1j*np.linspace(0.3,100,800)

zz = Hs[0][0](s);

xy = np.column_stack((np.real(zz),np.imag(zz)))
np.savetxt('nyquistH0150908.out', xy, fmt='%10.5f')

for (hh,HH) in Hs2.iteritems():
    if hh > 0:
        zz = HH[0](z);
        xy = np.column_stack((np.real(zz),np.imag(zz)))
        np.savetxt('nyquistHzz%d150908.out' % (100*hh), xy, fmt='%10.5f')

zz = Hs2[0][0](s);
zz2 = Hs2[0.01][0](z);
zz3 = Hs2[0.1][0](z);
zz4 = Hs2[0.2][0](z);
zz5 = Hs2[0.8][0](z);

plt.figure()
plt.plot(np.real(zz), np.imag(zz))
plt.plot(np.real(zz2), np.imag(zz2))
plt.plot(np.real(zz3), np.imag(zz3))
plt.plot(np.real(zz4), np.imag(zz4))
plt.plot(np.real(zz5), np.imag(zz5))
plt.ylim(-4,1)
plt.xlim(-2,1)

```

```python
Hln2 = H.subs(h, np.log(2))
Hln2 = cm.tf([0.19, 0.15], [1, -1.5, 0.5], np.log(2))
Hln2
```

      0.19 z + 0.15
    -----------------
    z^2 - 1.5 z + 0.5

    dt = 0.69314718056

```python
cm.rlocus(Hln2)
plt.ylim(-3,3)
plt.xlim(-3,3)
```

```python

```

### Apply Jury's criterion

```python

```

```python

```

```python

```

```python

```

```python

```


# 2.3 Layer-wise Relevance Propagation Part 1.

## Introduction

**This section corresponds to Sections 4.3 and 5.1 of the original paper.**

Recall how in Section 2.1 we defined a relevance function $R$ that takes in an ith pixel $x_i$ of image $x$ as an argument and returns a scalar value $R(x_i)$ which indicates how much positive or negative contribution $x_i$ has had on the final decision $f(x)$. Then as we progressed throughout Sections 2.1 and 2.2, we added several constraints to $R(x_i)$ such that it better described the rationale behind the DNN's decision. I will restate the constraints below so we can refer back to it later.

##### Constraint 1. $R(x_i) > 0$ should indicate positive contribution and $R(x_i) < 0$ negative contribution. If the range of $R$ is limited to nonnegative numbers, $R(x_i) = 0$ indicates no contribution and $R(x_i) > 0$ indicates positive contribution (Section 2.1).

##### Constraint 2. Relevance Conservation Constraint (Section 2.2):

\begin{equation}
f(x) = \sum^V_{i=1} R(x_i)
\end{equation}

By adding *Constraint (1)* in the Sensitivity Analysis framework, we quantified the amount of influence individual pixel $x_i$ has on $f(x)$. However, sensitivity analysis did not provide an straightforward explanation of the function value  $f(x)$, but rather suggestions. By introducing *Contraint (2)* in the Simple Taylor Decomposition framework, we created an explicit relationship between $R(x_i)$ and $f(x)$. We aim to improve $R$ by attaching a third constraint that exploits the feed-forward graph structure of the DNN.

**Notation Warning: **
so far, we defined the relevance score $R(x_i)$ as a function that takes in pixels only. However, we are now going to generalize the concept of relevance score such that it can be applied to all layers of a DNN. The $l$th layer of a DNN is modeled as a vector of pre-activations $z = (z^{(l)}_d)^{V(l)}_{d = 1}$ with dimensionality $V(l)$. That is, $z = (z^{(l)}_1, z^{(l)}_2, z^{(l)}_3, ..., z^{(l)}_{V(l)})$ where $V(l)$ is the number of nodes/neurons in the $l$th layer. Then we can define a relevance score $R^{(l)}_d$ for each pre-activation $z^{(l)}_d$ of the vector $z$ at layer $l$. See that we have now incorporated both the layer index and the neuron index into the notation of the relevance score. With this new notation, we can indicate the relevance of any neuron at any layer, even including the input layer! For example, the previous notation $R(x_i)$ is now equivalent to $R_i^{(1)}$ for $i = 1, 2, 3, ..., V(1)$. They both indicate the relevance of $i$th neuron at the first layer.

## Layer-wise Relevance Propagation

As its name implies, LRP makes explicit use of the feed-forward graph structure of a DNN. The first layer are the inputs, the pixels of the image, and the last layer is the real-valued prediction output of the classifier $f$. LRP assumes that we have a relevance score $R^{(l+1)}_d$ for each pre-activation $z^{(l+1)}_d$ of the vector $z$ at layer $l + 1$. The idea is to find a relevance score $R^{(l)}_d$ for each dimension $z^{(l+1)}_d$ of the vector $z$ at the next layer $l$ which is close to the input layer such that the following holds:

\begin{equation}
f(x) = \cdots = \sum_{d = 1}^{V(l+1)} R^{(l+1)}_d = \sum_{d = 1}^{V(l)} R^{(l)}_d = \cdots = \sum_{i = 1}^{V(1)} R^{(1)}_d
\end{equation}

Iterating this equation from the last layer which is the classifier output $f(x)$ down to the input layer $x$ consisting of image pixels naturally leads to *Constraint (2)*. With this, we can refine *Relevance Conservation Constraint* into *Layer-wise Relevance Conservation Constraint*.

##### Constraint 3. Layer-wise Relevance Conservation Constraint: total relevance must be preserved throughout layers.

Given a DNN where $j$ and $k$ are indices for neurons at two successive layers $l$ and $(l+1)$. We can define $R_{j \leftarrow k}^{(l,l+1)}$ as the portion of relevance that flows from neuron $k$ to neuron $j$. The portion is determined by the amount of contribution of neuron $j$ to $R_{k}^{(l+1)}$, subject to the *Layer-wise Relevance Conservation Constraint*:

\begin{equation}
\sum_j R_{j \leftarrow k}^{(l,l+1)} = R_{k}^{(l+1)}
\end{equation}

Same can be said for $R_{j}^{(l)}$:

\begin{equation}
\sum_k R_{j \leftarrow k}^{(l,l+1)} = R_{j}^{(l)}
\end{equation}

## Propagation Rules for DNNs

Let the neurons of the DNN be described by the equation

\begin{equation}
a_k = \sigma \left( \sum_j a_j w_{jk} + b_k \right)
\end{equation}

with $a_k$ the neuron activation, $a_j$ the activations from the previous layer, $w_{jk}$ the weight and $b_k$ the bias parameters of the neuron. The activation function $\sigma$ is a positive and monotonically increasing activation function (e.g. tanh, ReLU).

One propagation rule that fulfills the three constraints mentioned above is the $\alpha \beta$-rule. Let $z_{jk}^{+} = a_j w_{jk}^{+}$ and $z_{k}^{+} = \sum_j a_j w_{jk}^{+} + b_{k}^{+} = \sum_j z_{jk}^{+} + b_{k}^{+}$. $()^{+}$ and $()^{-}$ denote the positive and negative parts respectively. Same applies to $z_{jk}^{-}$ and $z_{k}^{-}$. Then the $\alpha \beta$-rule is given by

\begin{equation}
R_{j \leftarrow k}^{(l,l+1)} = R_{k}^{(l+1)} \cdot \left(\alpha \cdot \frac{z_{jk}^{+}}{z_{k}^{+}} - \beta \cdot \frac{z_{jk}^{-}}{z_{k}^{-}} \right)
\end{equation}

where the parameters $\alpha$ and $\beta$ satisfy the constraint $\alpha - \beta = 1$ and $\beta \geq 0$. Then, the *Layer-wise Relevance Conservation Constraint* becomes:

\begin{align}
\sum_j R_{j \leftarrow k}^{(l,l+1)} & = \sum_j R_{k}^{(l+1)} \cdot \left( \alpha \cdot \frac{z_{jk}^{+}}{z_{k}^{+}} - \beta \cdot \frac{z_{jk}^{-}}{z_{k}^{-}} \right) \\
& = R_{k}^{(l+1)} \cdot \left( \alpha \cdot \frac{\sum_j z_{jk}^{+}}{z_{k}^{+}} - \beta \cdot \frac{\sum_j z_{jk}^{-}}{z_{k}^{-}} \right) \\
& = R_{k}^{(l+1)} \cdot \left( \alpha \cdot \frac{z_{k}^{+} - b_{k}^{+}}{z_{k}^{+}} - \beta \cdot \frac{z_{k}^{-} - b_{k}^{-}}{z_{k}^{-}} \right) \\
& = R_{k}^{(l+1)} \cdot \left( (\alpha - \beta) - \alpha \cdot \frac{b_{k}^{+}}{z_{k}^{+}} + \beta \cdot \frac{b_{k}^{-}}{z_{k}^{-}} \right) \\
& = R_{k}^{(l+1)} \cdot \left( 1 - \alpha \cdot \frac{b_{k}^{+}}{z_{k}^{+}} + \beta \cdot \frac{b_{k}^{-}}{z_{k}^{-}} \right) \\
\end{align}

For different combinations of $\alpha$ and $\beta$, we name the corresponding propagation rule by subscripting $\alpha$ and $\beta$ with corresponding values. For example, choosing the parameters $\alpha = 2$ and $\beta = 1$ will give us the LRP-$\alpha_2 \beta_1$ rule.

Choosing LRP-$\alpha_1 \beta_0$ simplifies the propagation rule to and the *Layer-wise Relevance Conservation Constraint* to:

\begin{equation}
R_{j \leftarrow k}^{(l,l+1)} = \frac{z_{jk}^{+}}{z_{k}^{+}} \cdot R_{k}^{(l+1)}
\end{equation}

\begin{equation}
\sum_j R_{j \leftarrow k}^{(l,l+1)} = R_{k}^{(l+1)} \cdot \left( 1 - \frac{b_{k}^{+}}{z_{k}^{+}} \right)
\end{equation}

In addition, if all the biases are constrained or set to be $0$, the propagation rule and the *Layer-wise Relevance Conservation Constraint* can be further simplified into:

\begin{equation}
R_{j \leftarrow k}^{(l,l+1)} = R_{k}^{(l+1)} \cdot \frac{a_j w_{jk}^{+}}{\sum_j a_j w_{jk}^{+}}
\end{equation}

\begin{equation}
\sum_j R_{j \leftarrow k}^{(l,l+1)} = R_{k}^{(l+1)}
\end{equation}

In this tutorial, we are going to apply the above rule to a ReLU network without bias (which is basically equivalent to a network with $0$ bias).

## Tensorflow Implementation Details

Consider the LRP-$\alpha_1 \beta_0$ propagation rule of *Eq. (12)*:

\begin{equation}
R_{j}^{(l)} = a_j \sum_k \frac{w_{jk}^{+}}{\sum_j a_j w_{jk}^{+} + b_{k}^{+}} R_{k}^{(l+1)}
\end{equation}

This rule can be written as four elementary computations, all of which can also be expressed in vector form:

##### Element-wise

\begin{align*}
z_k & \leftarrow \sum_j a_j w_{jk}^{+} \\
s_k & \leftarrow R_k / z_k \\
c_j & \leftarrow \sum_k w_{jk}^{+} s_k \\
R_j & \leftarrow a_j c_j
\end{align*}

##### Vector Form

\begin{align*}
\mathbf{z} & \leftarrow W_{+}^{\top} \cdot \mathbf{a} \\
\mathbf{s} & \leftarrow \mathbf{R} \oslash \mathbf{z} \\
\mathbf{c} & \leftarrow W_{+} \cdot \mathbf{s} \\
\mathbf{R} & \leftarrow \mathbf{a} \odot \mathbf{c}
\end{align*}

By applying the same procedure to negative parts, we obtain the LRP implementation for all cases of $\alpha$ and $\beta$. In fully-connected dense layers, LRP can be implemented by the following sequence of Tensorflow operations:

```python
def backprop_dense(activation, kernel, bias, relevance):
    W_p = tf.maximum(0., kernel)
    b_p = tf.maximum(0., bias)
    z_p = tf.matmul(activation, W_p) + b_p
    s_p = relevance / z_p
    c_p = tf.matmul(s_p, tf.transpose(W_p))

    W_n = tf.minimum(0., kernel)
    b_n = tf.minimum(0., bias)
    z_n = tf.matmul(activation, W_n) + b_n
    s_n = relevance / z_n
    c_n = tf.matmul(s_n, tf.transpose(W_n))

    return activation * (self.alpha * c_p + (1 - self.alpha) * c_n)
```

In convolution layers, the matrix-vector multiplications can be more efficiently implemented by `backprop` methods used for gradient propagation.

```python
def backprop_conv(self, activation, kernel, bias, relevance, strides, padding='SAME'):
    W_p = tf.maximum(0., kernel)
    b_p = tf.maximum(0., bias)
    z_p = nn_ops.conv2d(activation, W_p, strides, padding) + b_p
    s_p = relevance / z_p
    c_p = nn_ops.conv2d_backprop_input(tf.shape(activation), W_p, s_p, strides, padding)

    W_n = tf.minimum(0., kernel)
    b_n = tf.minimum(0., bias)
    z_n = nn_ops.conv2d(activation, W_n, strides, padding) + b_n
    s_n = relevance / z_n
    c_n = nn_ops.conv2d_backprop_input(tf.shape(activation), W_n, s_n, strides, padding)

    return activation * (self.alpha * c_p + (1 - self.alpha) * c_n)
```

In max-pooling layers, the original paper by [Bach et al.](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140) uses a winner-take-all redistribution policy where all relevance goes to the most activated neuron in the pool. On the other hand, the paper by [Montavon et al.](https://www.sciencedirect.com/science/article/pii/S0031320316303582) uses the proportional redistribution rule where the redistribution is proportional to neuron activations in the pool:

\begin{equation}
R_{j}^{(l)} = \frac{x_j}{\sum_j x_j} R_{k}^{(l+1)}
\end{equation}

Like the convolution layers, redistribution in pooling layers can also be efficiently implemend by `backprop` methods.

```python
# Bach et al.'s redistribution rule
def backprop_pool(self, activation, relevance, ksize, strides, pooling_type, padding='SAME'):
    z = nn_ops.max_pool(activation, ksize, strides, padding) + 1e-10
    s = relevance / z
    c = gen_nn_ops._max_pool_grad(activation, z, s, ksize, strides, padding)
    return activation * c

# Montavon et al.'s redistribution rule
def backprop_pool(self, activation, relevance, ksize, strides, pooling_type, padding='SAME'):
    z = nn_ops.avg_pool(activation, ksize, strides, padding) + 1e-10
    s = relevance / z
    c = gen_nn_ops._avg_pool_grad(tf.shape(activation), s, ksize, strides, padding)
    return activation * c
```

Now that we have all the tools for LRP-$\alpha \beta$, we will see its application to a DNN trained on MNIST in the next part of the tutorial.

```python

```

2.3 Layer-wise Relevance Propagation (1).ipynb


# Chapter 2
> Linear Algebra and Machine Learning

## Lecture 9
___

### Review of Linear Algebra

Reference Books: Matrix Cookbook by Kaare Brandt Petersen & Michael Syskind Pedersen, 2012

$A \in \mathbb{R}^{n \times m}, n\text{ rows and } m\text{ columns}$

range($A$):=span{\underline{a}$_1$,...,\underline{a}$_m$}

null($A$):={\underline{x} $\in \mathbb{R}^{m}$|$A$\underline{x}=0}

Column rank = Row rank = number of linearly independent vectors.

Full rank: rank(A) = min{m,n}.

A \textbf{nonsingular} or \textbf{ invertible} matrix is a square matrix of full rank

Angle between two vectors: $\displaystyle \alpha = cos^{-1}(\frac{\mathbf{x}^T\mathbf{y}}{||\mathbf{x}||\cdot ||\mathbf{y}|| })$

$Q$ matrix is \textbf{ unitary} or \textbf{ orthogonal} if $Q^T=Q^{-1}$:
$||Q\mathbf{x}||=||\mathbf{x}||$ rotation

\textbf{ Norms}:

$\displaystyle ||\mathbf{x}||_p:=(\sum_{j=1}^n|\mathbf{x}_j|^p)^{1/p}$

$\displaystyle ||\mathbf{x}||_1:=\sum_{j=1}^n|\mathbf{x}_j|$

$\displaystyle ||\mathbf{x}||_2:=(\mathbf{x}^T\mathbf{x})^{1/2}$

$\displaystyle ||\mathbf{x}||_\infty:=\max_{1\le j\le n}|\mathbf{x}_j|$

$\displaystyle
||A||_{(m,n)}:=\sup_{x\in \mathbf{R}^m, \mathbf{x} \ne 0}\frac{||A\mathbf{x}||_n}{||\mathbf{x}||_m}=\sup_{||\mathbf{x}||_m=1}||A\mathbf{x}||_n$

$\displaystyle ||A||_1 = \text{ max column in } A$

$\displaystyle ||A||_\infty = \text{ max row in } A$

Frobenins (Hilbert-Schmidt norm):

$\displaystyle ||A||_F = (\sum_{i=1}^n\sum_{j=1}^m |a_{i,j}|^2)^{1/2}=\sqrt{\text{ Tr}(A^TA)}=\sqrt{\text{Tr}(AA^T)}\text{ , where Tr}(B) = \sum_j b_{jj}, \text{ sum of diagonal entries.}$

$||QA||_2=||A||_2 \,\&\, ||QA||_F=||A||_F $

\textbf{Singular Value Decomposition} (SVD): rotation & stretching of a basis

$U$ - left singular matrix, $V$- right singular matrix, $\Sigma$ is the diagonal entry matrix.

\begin{align}
\text{To find the } V, \text{we note that } A^TA &= V\Sigma U^{-1}U\Sigma V^T \\
& = V\Sigma^2V^T \\
&= V\left[
\begin{array}{ccc}
\sigma_1^2 &  & \\
& \sigma_2^2 &  \\
& & ... \\
& & \sigma_n^2  \\
\end{array}
\right]V^T
\end{align}

The eigenvectors of this matrix $A^TA$ will give us the vectors $\mathbf{v}_i$, and the eigenvalues will give the numbers $\sigma_i$.

Similarly, the matrix $AA^T$ gives us info for $U$. Examples can be seen here [SVD examples](
https://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebra-fall-2011/positive-definite-matrices-and-applications/singular-value-decomposition/MIT18_06SCF11_Ses3.5sum.pdf#:~:text=The%20singular%20value%20decomposition%20of%20a%20matrix%20is,its%20eigenvectors%20are%20orthogonal%20and%20we%20can%20write)

## Lecture 10
___

### PCA (principal component analysis)

$\overline{x} $ center data by subtraction the mean

$\frac{1}{N-1}XX^T$ is simply the empirical approximation to Cov(X) which henceforth we denote as $C_x$. $C_x$ is non-negative definite & Symmetric (NDS). Hence it has an eigen decomposition $C_x = Q\Lambda Q^T$. The eigen-vectors of $C_x$ are called the principal components or the Rarhumen-Loere modes or PCA modes of $C_x$.

At the same time write $X=U\Sigma V^T$ then,
\begin{align}
C_x=\frac{1}{N-1}XX^T&=\frac{1}{N-1}U\Sigma V^TV\Sigma U^T\\
&=\frac{1}{N-1}U\Sigma^2U^T
\end{align}

Thus, columns of U,\textbf{ the left singular vectors of X are precisely the Principal Components of} $C_x$

But why do we care about the PCA?

A random varibale is called Gaussian if its PDF has the form
$$\Pi(x)=\frac{1}{\sqrt{(2\pi)^ddet(C)}}exp(-\frac{1}{2}(x-m)^TC^{-1}(x-m))$$, where m is mean

\begin{lemma}
Suppose $x \sim \mathcal{N}(m,C),m \in \mathbb{R}^d, C\in \mathbb{R}^{d\times{d}}. \text{ Let } b\in \mathbb{R}^n \& A\in \mathbb{R}^{n\times d} \text{ then } z=Ax+b \text{ is also Gaussian & } z\in \mathcal{N}(b+Am, ACA^T)  $
\end{lemma}

PCA explains the covariance and directions of maximum singular values in our dataset. It is good for low-dimension compression.

## Lecture 11
___
### From PCA & SVD to Proper Orthogonal Decomposition & Dynamic Mode Decoposition

POD(proper orthogonal decomposition)

$\displaystyle f(x,t):=\sum_{k=0}^{\infty}c_k(t)\psi_k(x)$

In the context, $\psi_k(x)$ are called the POD modes & $c_k(t)$ are called the POD coefficient. Luckily, there is an easy way to compute POD using SVD.

Suppose dynamics of f(x,t) are observed over u discrete set, ie (x_j,t_k) ,j=0...N-1, k=0,...,T-1. Now put it into a matrix D:

$$D:=\begin{bmatrix}
\\
	d_0=f(x_j,t_0) \;\; | \;\; d_1=f(x_j,t_1) \;\; | \;\; ....\;\; |\;\; d_{T-1}=f(x_j,t_{T-1})\\
    \\
	\end{bmatrix}$$

Then, compute the SVD of D.

$D=U\Sigma V^T$, $U$ doesn't change even if the matrix is shuffled around, the direction of time is not important. Then the columns of $U$ are precisely the POD modes & the rows of $\Sigma V^T$ are precisely the POD coefficients.

\textbf{ The power of this method is also important that it could be used to predict the future behavior based on the DMD}. Book:Data-Driven Science and Engineering by Steven

```python
#import numpy as np
#import matplotlib.pyplot as plt
#
# first load frames of cylinder flow simulation
#from google.colab import drive
# drive.mount('/content/drive')
#
data = np.load(
    "/content/drive/MyDrive/Courses/AMATH482582-WIN2022/Notebooks/CylData/cyldata.npy")
Nx = 200
Ny = 50
```

```python
# ax0 is the flattened frames and ax1 are the time frames

# compute svd of data

import numpy.matlib as matlib

mean_data = np.mean(data, 1)

centered_data = data - np.transpose(matlib.repmat(mean_data, 200, 1))

dU, ds, dVt = np.linalg.svd(centered_data, full_matrices=False)
```

```python
# plot some of the singular values and some of the principal modes

fig, ax = plt.subplots(1, 1, figsize=(8, 8))

ax.plot(np.log(ds[0:100]))
ax.set_xlabel('index $j$')
ax.set_ylabel('$\log(\sigma_j)$')
#ax.set_xlim(0, 150)

fig, ax = plt.subplots(8, 2, figsize=(20, 20))

for j in range(8):

    ax[j][0].imshow(np.reshape(dU[:, j], (Ny, Nx)), cmap='bwr')
    ax[j][0].get_xaxis().set_visible(False)
    ax[j][0].get_yaxis().set_visible(False)
    ax[j][0].set_aspect('equal')

    ax[j][1].plot(np.abs(dVt[j, :]))
    ax[j][1].set_xlabel('Time step')

    if j == 0:
        ax[j][0].set_title('POD Modes', fontsize=30)
        ax[j][1].set_title('|POD Coeff.|', fontsize=30)

plt.show()
```

```python
# approximate the dynamics using the POD modes only keeping the first 40 and compare to original data.

dss = np.copy(ds)

dss[10:None] = 0

# low rank approx
approx_centered_data = np.dot(dU, np.dot(np.diag(dss), dVt))

print(approx_centered_data.shape)

# add the mean back
approx_data = approx_centered_data + \
    np.transpose(matlib.repmat(mean_data, 200, 1))
```

```python
# side by side comparison of original and approximate dynamics

fig, ax = plt.subplots(4, 2, figsize=(20, 15))

frm_indx = [10, 100, 150, 199]

for j in range(4):

    ax[j][0].imshow(np.reshape(data[:, frm_indx[j]], (Ny, Nx)), cmap='bwr')
    ax[j][0].get_xaxis().set_visible(False)
    ax[j][0].get_yaxis().set_visible(False)
    ax[j][0].set_aspect('equal')

    ax[j][1].imshow(np.reshape(
        approx_data[:, frm_indx[j]], (Ny, Nx)), cmap='bwr')
    ax[j][1].get_xaxis().set_visible(False)
    ax[j][1].get_yaxis().set_visible(False)
    ax[j][1].set_aspect('equal')

    if j == 0:
        ax[j][0].set_title('Original', fontsize=30)
        ax[j][1].set_title('POD Approximation', fontsize=30)
```

### DMD (Dynamic Modes Decomposition) - developed 2010s, some work done at UW:

Look back to the POD, the POD modes are smooth & structured, the POD coefficients are rough & chaotic. This is because POD only sees spatial structure & ignores temporal dynamics. DMD attempts to address this issue by modeling the dynamics of the data:

$$d_{k+1}=Ad_k$$
$$f(x,t_{k+1}){\leftarrow}A\leftarrow f(x,t_k)$$

Lets reconstruct our data as follows, one keeping the T-1 data from 0 to T-2, the other keeping the T-1 data from 1 to T-1.

$$D_1:=\begin{bmatrix}
\\
	d_0 \;\; | \;\; d_1\;\; | \;\; ....\;\; |\;\; d_{T-2}\\
    \\
	\end{bmatrix}$$

$$D_2:=\begin{bmatrix}
\\
	d_1 \;\; | \;\; d_2\;\; | \;\; ....\;\; |\;\; d_{T-1}\\
    \\
	\end{bmatrix}$$

Then DMD seeks to find a matrix $A$ such that
$$D_2 \approx AD_1 $$

This approx is then done in a best-fit

$$A=\min_{B\in \mathbb{R}^{N\times N},\text{ rank(B)}\le r}||BD_1-D_2||_F^2$$

So what does DMD actually do?

\begin{align}
d_k&=Ad_{k-1}=A(Ad_{k-2})=...\\
d_k&=A^{k-1}d_0=Q\Lambda^{k-1}Q^{-1}d_0 \\
&=Q\Lambda^{k-1}b=\sum_{j=1}^rq_{j}\lambda_{j}^{k-1}b_j
\end{align}

```python
from pydmd import DMD
```

```python
# first we create a DMD object
dmd = DMD(svd_rank=10)  # a rank 10 approximation to the dynamic matrix A

# simply pass centered data set of snapshots to the dmd class
dmd.fit(centered_data)

dmd.plot_eigs(show_axes=True)
```

```python
# now plot the DMD modes and the temporal dynamics as we did for POD

fig, ax = plt.subplots(8, 4, figsize=(30, 18))

for j in range(8):

    ax[j][0].imshow(np.reshape(np.real(dmd.modes[:, j]), (Ny, Nx)), cmap='bwr')
    ax[j][0].get_xaxis().set_visible(False)
    ax[j][0].get_yaxis().set_visible(False)
    ax[j][0].set_aspect('equal')

    ax[j][1].plot(np.real(dmd.dynamics[j, :]))
    ax[j][1].set_xlabel('Time step')

    ax[j][2].imshow(np.reshape(np.imag(dmd.modes[:, j]), (Ny, Nx)), cmap='bwr')
    ax[j][2].get_xaxis().set_visible(False)
    ax[j][2].get_yaxis().set_visible(False)
    ax[j][2].set_aspect('equal')

    ax[j][3].plot(np.imag(dmd.dynamics[j, :]))
    ax[j][3].set_xlabel('Time step')

    if j == 0:
        ax[j][0].set_title('Real DMD Modes', fontsize=30)
        ax[j][1].set_title('Real DMD Dynamics', fontsize=30)
        ax[j][2].set_title('Imag DMD Modes', fontsize=30)
        ax[j][3].set_title('Imag DMD Dynamics', fontsize=30)

plt.show()
```

 ## Lecture 12
 ___
 ### Introduction to Machine Learning

 Book: The Elements of Statistical Learning by Hastie

\textbf{Supervised Learning}: With labels.
Predict/classify data given a training dataset. Eg: REgression, classification, function approximation, etc.

\textbf{Un-supervised Learning}:
No labels.
Find meaningful structure in dataset. Eg: Clustering, dimensionality reduction, Generation modeling.

### Supervised Learning

The function model assumes there exists a function
$\displaystyle f^+:X\rightarrow y$ so that $y_j=f^+(x_j)+\epsilon_j$, where $\epsilon_j$ are some noise that may be in the output or our observation of the $f^+(x_j)$

By far the most common assumption is Gaussian noise

$$\epsilon_j \approx \mathcal{N}(0,\sigma^2)$$

This implies $\displaystyle y_i|x_j \approx \mathcal{N}(f^+(x_j),\sigma^2)$

$$\Pi(y_j|x_j)\propto exp(-\frac{1}{2\sigma^2}|f^+(x_j)-y_j|^2)$$, $\Pi$ is the PDF of $y$ for fixed $x_j$

For Euclidern norm, this is called a maximum likelihood estimate (MLE):

$$f_{MLE}=arg\min_f \frac{1}{2\sigma^2}||f(X)-Y||^2$$

At this moment, it is useless without a model, since there are many solutions. One of the most simple model is \textbf{ linear regression}.

$$f_{MLE}\equiv\beta_{MLE}=argmin \frac{1}{2\sigma^2}||A\beta-Y||^2$$, where A=$
\left[
\begin{array}{ccc}
1& & x_0^T \\
.& & . \\
.& & . \\
1& & x_{N-1}^T
\end{array}
\right]
$.

Therefore, MLE is nothing but a least square solution to the problem. Typically, the system is over-determined. Solution is given by solving the normal equations,

$$\frac{\partial}{\partial\beta}(\frac{1}{2\sigma^2}(A\beta-y)^T(A\beta-y))=\frac{1}{\sigma^2}A^T(A\beta-y)=0$$

\begin{align}\implies A^T(A\beta-y)&=0\\
\beta&=(A^TA)^{-1}A^Ty
\end{align}

## Lecture 13
___
### Evaluating SL models

\textbf{ Regularization/penalization/shrinkage}:
we consider $\displaystyle \hat{\beta}=argmin \frac{1}{2\sigma^2}||A\beta-y||^2+\frac{\lambda}{2}||\beta||_p^p$

$\lambda\ge 0$ is called the regularization/penalty parameter & $p \ge 1$ denotes the choice. $p=2 $ for Ridge regression.

$$\beta=(\frac{1}{\sigma^2}A^TA+\lambda I)^{-1}A^Ty$$

So doing SVD of A,

$$\frac{1}{\sigma^2}A^TA+\lambda I=V(\frac{1}{\sigma^2}\Sigma^2+ \lambda I)V^T$$, the diagonals are non-negative, eliminating zeros so A matrix can be invertible.

Again, the choice of $\lambda$ is important for stability and accuracy.

### Training & Testing Errors

\textbf{Training mean squared error (MSE)}

${X,Y}$ - training set, used to find $\hat{f}(\equiv \hat{\beta})$

${X',Y'}$ - testing set, used for validation.

Analyzing the MSE doesn't mean which model is always better. It is still important for choice of $\lambda$

## Lecture 14
___

### Model tuning with Cross Validation

$\lambda$ too small, the model is basically memorizing all the train data. The test error is large. (over-fitting, high-variance)

$\lambda $ too large, model is too simply biased.()

We want the best test error which is the smallest. However, in real life, we don't know the test error.

\textbf{Cross Validation}: Split the train data to k parts, or k-fold.

Randomly permute the data pairs-- $\mathbf{x}=\{x_{10},x_{-1},...,x{_13}\}$ and responding $\mathbf{y}$.

Then split the data $\mathbf{x} \,\&\, \mathbf{y} $ into K-subsets. Iterate over $k=0,...k=K-1$ and fit the model to the training data with the k-th fold removed.

Finally, calculate the CV prediction error (CV cost) with changing $\lambda$

$$CV(\hat{f},\lambda) := \frac{1}{N}\sum_{k=0}^{K-1}||\hat{f}(\mathbf{x}_k,\lambda)-\mathbf{y}_k||^2$$

## Lecture 15
___
### Introduction to Kernel Methods

\textbf{Kernel NDS}: We say K is non-negative definite & symmetric (NDS) if

\begin{itemize}
\item $K(\mathbf{x},\mathbf{x'})=K(\mathbf{x'},\mathbf{x})$
\item For any set of points $(x_0,...x_n)$, the matrix
$(K)_{ij}=K(x_i,x_j) $ is NDS.
\item $\displaystyle K(\mathbf{x},\mathbf{x'})=\sum_{j=0}^\infty\lambda_j\psi_j(\mathbf{x})\psi_j(\mathbf{x'}) \text{, the number }\lambda_j\ge 0 $ are the eigenvalues, while the $\psi_j$ are eigenfunctions.

\end{itemize}

Mercer Theorem says if a matrix $A$ is NDS then $A=Q\Lambda Q^{-1}$

$$\implies A=\sum_j\lambda_jq_jq_j^T$$

Define the functions,

$$F_j(x)=\sqrt{\lambda_j}\psi_j(x)$$

along with the mapping,

$$F(\mathbf{x})=(F_0(\mathbf{x}),F_1(\mathbf{x}),....)$$

Functions in RKHS(Reproducing Kernel to Hilbert Space) have the nice properties:

$\displaystyle K (\mathbf{x},\mathbf{x'})=\sum_j F_j(\mathbf{x})F_j(\mathbf{x'}) \,\&\, \text{ so, for a fixed }\mathbf{x}$ we have that $\displaystyle f(\mathbf{x})=\sum_j c_jF_j(\mathbf{x})=(\{c_j\}_{j=0}^\infty,\{F_j(\mathbf{x})\}_{j=0}^\infty)_{l^2}=<f,K(\mathbf{x},)>_{H_k}$

### Kernel Interpolation
Suppose we have an interpolation problem, given x datapoints and y datapoints. we wish to find $\displaystyle f=\sum_{j=0}^\infty c_jF_j$ so we need to solve $\displaystyle \sum_{j=0}^\infty c_jF_j（x_i)=y_i, i=0,...,N-1$ We want the solution with minimized $H_k$ norm. Therefore, we wish to have

$$minimize \sum_{j=0}^\infty c_j^2$$

$$s.t \sum_{j=0}^\infty c_jF_j(x_i)=y_i$$

What it reduces to is a solution of the form

$$f(\mathbf{x})=\sum_{j=0}^{N-1}a_jK(\mathbf{x_j},\mathbf{x})$$

The interpolation constraints then tell us that $\displaystyle \sum_{j=0}^{N-1}a_jK(\mathbf{x}_j,\mathbf{x}_i)=y_i$

\begin{align}
\quad\quad\quad\quad\quad\quad\quad\quad \mathbf{a}&=(a_0,....,a_{N-1})\\
\implies \Theta \mathbf{a}=\mathbf{y},\quad\quad\mathbf{y}&=(y_0,....y_{N-1}) \\\quad\quad \Theta_{ji}&=K(\mathbf{x}_j,\mathbf{x}_i)
\end{align}

Thus, $\mathbf{a}=\Theta^{-1}\mathbf{y}$, now the matrix is invertible provided that it is NDS, and the $x_j$ are distinct.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
# define function to be interpolated

def f(x):

  val = np.exp( - x**2/0.05 )*np.cos(np.pi*20*x) + 0.5*np.tanh(5*(x - 0.5))

  return val
```

```python
N = 100
N_p = 5

x = np.linspace(0,1, N) # grid used for plotting

x_p = np.linspace(0,1, N_p) # set of points for interpolation

# plot the function

fig,ax = plt.subplots(1,1, figsize=(10,8))

ax.plot(x, f(x))
ax.scatter(x_p, f(x_p), color='r')
ax.set_xlabel('x', fontsize=20)
ax.set_ylabel('f(x)', fontsize=20)
```

```python
# define the kernel we wish to use
# Gaussian Kernel

def K(x1, x2, l):

  val = np.exp( - (np.abs(x1 - x2)**2)/(l**2) )

  return val

# plot K(x, 0.5) for illustration

l = [0.05, 0.1, 0.25]

fig,ax = plt.subplots(1,1, figsize=(10,8))

for i in range(3):

  ax.plot(x, K(x, 0.5, l[i]), label='l = '+str(l[i]))

ax.set_xlabel('x', fontsize=20)
ax.set_ylabel('K(x, 0.5)', fontsize=20)
plt.legend(fontsize=20)
```

```python
# compute kernel interpolands and plot them for each choice of l

# construct y vector, just for uniform notation with notes
y = f(x_p)

Theta = np.zeros( (N_p, N_p) )

fig,ax = plt.subplots(1,1, figsize=(10,8))

ax.plot(x, f(x), label='f')

for i in range(3):
  # construct kernel matrix (maybe not the most efficient way but ok)
  for j in range(N_p):
    Theta[j,:] = K( x_p[j], x_p, l[i] )

  # compute kernel interpolant

  a = np.linalg.solve(Theta, y)

  # plot the interpolant along with original function and interpolation data

  KV = np.zeros( (N_p, N) )

  for j in range(N_p):
    KV[j, :] = K( x_p[j], x, l[i] )

  f_interp = np.dot(a, KV)

  ax.plot(x, f_interp, label='l = ' + str(l[i]))

ax.scatter(x_p, y, color='r' )

ax.set_xlabel('x', fontsize=20)
plt.legend(fontsize=20)

```

## Lecture 16
___
### Kernel Ridge Regression

Mercers Theorem:
$$K(\mathbf{x},\mathbf{x'})=\sum_{j=0}^\infty F_j(\mathbf{x})F_j(\mathbf{x'})$$
,where $F_j$ are the features of $K$, $F_j\rightarrow 0 \text{ as } j\rightarrow \infty$. This regression is capable of solving cases with infinite number of features but at the cost of constructing a finite $n\times n$ matrix.

\textbf{Ridge Regression}:
$$\min_{\beta}||A\beta-y||^2+\lambda||\beta||^2$$

where A is our feature matrix often of the form

$$A=\left[\begin{array}{ccc}
F_0(x_0) &F_1(x_1)&...\\
F_0(x_1) & F_1(x_1)&...\\
.. & .. \\
F_0(x_{N-1}) &F_1(x_{N-1})&..
\end{array}\right]$$

while we see $||f||^2_{H_k}=\sum_{j=0}^{J-1}\beta_j^2=||\beta||^2$

\textbf{Kernel Ridge Regression}:
$$\min_{f}||f(X)-Y||^2+\lambda||f||_{H_k}^2$$

\textbf{Representer Theorem}:
$$\hat{f}(\mathbf{x})=\sum_{n=0}^{N-1}\hat{a}_nK(\mathbf{x}_n,\mathbf{x})$$

,with $\hat{a}=(\hat{a}_0,...,\hat{a}_{N-1})$ being minimizer of
$$minimize \;||\Theta\mathbf{a}-Y||^2+\lambda\mathbf{a}^T\Theta\mathbf{a}$$

The solution is found by differentiating it, and it is exactly $\displaystyle \mathbf{a}=(\Theta+\lambda I)^{-1}y$

### Example: House Pricing in Taiwan

Steps to do:

\begin{itemize}
\item Create corresponding train and test data.
\item Normalize the train and test data: (data-mean)/std
\item Choose and fit Kernel functions (example: "rbf"-Gaussian Kernel)
\item Create discrete $\lambda \text{ and } \gamma $ parameters.
\item Double for-loop to calculate cross-validation scores (scoring="neg_mean_squared_error") and store the scores.
\item Find the best score for choosing parameters
\item Predict
\end{itemize}

## Lecture 17
___
### Kernel Perspective vs. Feature Perspective

FR: $minimize \;||A\beta-Y||^2+\lambda||\beta||^2$ \\

KR: $minimize \; ||\Theta a-Y||^2 +\lambda a^T\Theta a$

where $A_{ij}=F_j(x_i)$ matrix of feature values,
$\Theta_{ij}=K(x_i.x_j)$

The size of computation is different in that, provided a large data set with data points far more than feature points, we might want to use FR because KR would generate a matrix of NXN dimension which costs too much.

For KR, we have infinite many features and more freedom to choose.

### Connecting Kernels to Fourier Series & Wavelets

From discrete Fourier Series we note that $\displaystyle f(t)=\sum_{k=-N/2}^{N/2-1}\hat{f}_kexp(ikt)$. We than identify a kernel called the \textbf{ Dirichlet Kernel}:
$$K(t,s)=\sum_{k=-N/2}^{N/2-1}exp(ik(t-s))$$
This gives the identity that

$$\int_0^{2\pi}K(t,s)f(s)\,ds=\sum_{k=-N/2}^{N/2-1}\hat{f}_kexp(ikt)$$

Then, we also note that linear projection is not the only transformation, non-linear is also possible.


# Create named function from an expression

This is not implemented inside `fun_expr`. Sometimes it is useful to have named functions that know about their name and can use this

Initialisations:

```python
from sympy import *
init_printing()

import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format='retina'

import numpy as np

from IPython.display import display, Math, display_latex, Latex

from fun_expr import Function_from_Expression as FE, Named_Function_from_Expression as NFE
```

To create named functions, subclass FE:

```python
class Named_Function_from_Expression(FE):
    """
    Create a named function from an expression.

    Example fo use:
    >>> from sympy import *
    >>> from fun_expr import (
            Function_from_Expression as FE,
            Named_Function_from_Expression as NFE,
            )
    >>> f = NFE("f", x, x**2)
    >>> f.displ()
    '$f(x) = x^{2}$'
    >>> f_1 = f.diff(x,name="f'")
    >>> f_1.displ()
    "$f'(x) = 2 x$"
    >>> f_1 = f.diff(x)
    '((x) \mapsto 2\,x)'
    >>> print(type(f_1))
    fun_expr.function_from_expression.Function_from_Expression
    >>> f.displ(2)
    '$f(2) = 4$'
    """

    def __new__(cls, name, variables, expr):
        obj = FE.__new__(cls, variables, expr)
        obj.name = name
        return obj

    def displ(self, *values, par="$", n=False, simplify=False, **kwargs):
        """
        This is a convenience function to display the equation

          $f(x_1,x_2,...) = rhs$
          or
          $$f(x_1,x_2,...) = rhs$$ # if par='$$'

        in latex-notation.

        If values == (): replace rhs by self.expr
        instead. If values are specified, exactly one
        value for every variable of this function is
        expected.

        If n ist True, replace the rhs by rhs.n().
        If n is an integer number replace the rhs by rhs.n(n).

        If simplify is true, simplify rhs.

        Here rhs always is the right hand side of the equation.
        """

        # there are exactly two possibilities:
        #    either there is no value at all, values == ()
        #    or there are exactly len(self.variables) values.
        #    if there are no values, set rhs to self.expr
        #    else calculate rhs = f(*values)
        if values is not ():
            vals = values
            rhs = super().__call__(*values, **kwargs)
        else:
            vals = self.variables
            rhs = self.expr

        # n can be None, bool or int.
        # if n is None do nothing
        # if n is bool and n is True, replace rhs by rhs.n()
        # if n is bool and n is False, do nothing
        # if n is int replace rhs by rhs.n(n)
        if n is not None:
            if isinstance(n, bool):
                if n:
                    rhs = rhs.n()
            else:
                rhs = rhs.n(n)

        # simplify can be True or False
        # if simplfiy ist True, the rhs is simplified
        if simplify:
            rhs = rhs.simplify()

        # create output as
        #    f(x_1,x_2,...) = rhs
        # in latex notation
        ret_val = par
        ret_val += latex(self.name)
        ret_val += r"\left({vars}\right) = ".format(vars=','.join(latex(v) for v in vals))
        ret_val += latex(rhs)
        ret_val += par
        return ret_val

    def diff(self, *symbols, **assumptions):
        """
        If a name is specified, as_function is set to True.
        In this case, a Named_Function_of_Expression is retured.
        If no name is specified, the result depends on as_function
        """
        name = assumptions.pop('name',None)
        as_function = assumptions.pop('as_function', True) or name is not None
        # let my parent do the hard part
        res = super().diff(*symbols, as_function=as_function, **assumptions)
        if name is not None:
            return Named_Function_from_Expression(name, res.variables, res.expr)
        else:
            return res

    def integrate(self, *args, **kwargs):
        """
        If a name is specified, as_function is set to True.
        In this case, a Named_Function_of_Expression is returned.
        If no name is specified, the result depends on as_function
        """
        name = kwargs.pop('name',None)
        as_function = kwargs.pop('as_function', False) or name is not None
        # let my parent do the hard part
        res = super().integrate(*args, as_function=as_function, **kwargs)
        if name is not None:
            return Named_Function_from_Expression(name, res.variables, res.expr)
        else:
            return res

    def subs(self, *args, **kwargs):
        obj = Named_Function_from_Expression(self.name, self.variables, self.expr.subs(*args, **kwargs))
        if False in [u==v for u,v in zip(self.variables,obj.variables)]:
            raise ValueError(
                    "Substitution of variables is not valid: {}".format(arg)
                    )
        return obj
```

```python
a,x = symbols('a,x')
f = Named_Function_from_Expression('f', x, a*x**2)
Math(f.displ())
#f = FE(x,a*x**2)
```

$$f\left(x\right) = a x^{2}$$

```python
display(f.subs(x,1))
display(type(f))
```

```python
f = Named_Function_from_Expression('f',(a,x),a*x**2)
f
```

```python
f.subs(a,x)
```

```python
f = Lambda((a,x), a*x**2)
f
```

```python
f.subs(a,x)
```

```python
a_0 = Symbol('a_0')
f.subs(a,a_0)
```

```python
f = Named_Function_from_Expression('f', (a,x), a*x)
Math(f.displ())
```

$$f\left(a,x\right) = a x$$

```python
f.subs(a,a_0)
```

```python
class A():
    name = 'A'

    def ret_name(self):
        return (A.name, self.name)

class B(A):
    name = 'B'

    def ret_name_too(self):
        return super().ret_name()

```

```python
b = B()
print(b.ret_name())
b.ret_name_too()
```

    ('A', 'B')

    ('A', 'B')

```python
a = A()
print(a.ret_name())
```

    ('A', 'A')

```python
a,x = symbols('a,x')
g = Lambda(x,a*x**2)
g
```

```python
g.subs(a,1)
```

```python
g.subs(a,1)
```

```python
f = NFE('f', x, a*x**2)
f.displ()
```

    '$f\\left(x\\right) = a x^{2}$'

```python
f.subs({a:1})
```

Create a simple function to test `Named_Function_from_Expression`:

```python
x = Symbol('x')
f = Named_Function_from_Expression("f",x,x**2)
f.displ()
```

```python
x = Symbol('x')
f = NFE('f', x, x**2)
display(f.displ())
type(f)
```

```python
f.displ(par='$$')
```

```python
# to have nice output:
Math(f.displ())
```

```python
display(f.displ(2))
Math(f.displ(2))
```

Plot the graph $y=f(x)$ for $-1 \le x \le 1$:

```python
fig, ax = plt.subplots()

lx = np.linspace(-1,1)

ax.plot(lx,f.lambdified(lx),label=f.displ())
ax.axhline(0,c='k')
ax.axvline(0,c='k')
ax.grid(True)
ax.legend(loc='upper center')
ax.set_title("Graph of the square function {f}".format(f=f.displ()))
```

Determine the derivative of $f(x)$ as named function:

```python
f_1 = f.diff(x,name="f'")
display(f_1.displ())
display(Math(f_1.displ()))
display(type(f_1))
```

If no name is specified for `f.diff(x)`, a function of class `FE` is returned:

```python
f_1 = f.diff(x)
display(f_1)
display(type(f_1))
```

If no name is specified for `f.diff(x)` and `is_function=False`, an expression is returned:

```python
expr = f.diff(x,as_function=False)
display(expr)
type(expr)
```

Integration leads to an expression, not to a function:

```python
f_0 = f.integrate(x)
f_0
```

If `as_function=True` is set, the result is of type `FE`:

```python
f_0 = f.integrate(x,as_function=True)
display(f_0)
display(type(f_0))
```

If a name is specified `as_function` is set to `True`, the result is of type `Named_Function_from_Expression`:

```python
f_0 = f.integrate(x,name="f_0")
display(Math(f_0.displ()))
display(type(f_0))
```

```python
f_0 = f.integrate(x,name='f_0',as_function=False)
display(Math(f_0.displ()))
type(f_0)
```

```python
Math(f_0.displ(3))
```

Named functions of more then one variable are possible, too.

```python
a,x,y,z = symbols('a,x,y,z')

g = NFE('g',(x,y,z), a*x*y*z)
Math(g.displ())
```

```python
Math(g.diff(z,name=r"\frac{\partial}{\partial z}\,g").displ())
```

Names of functions can be latex expressions like `r'\mu'`:

```python
mu = NFE(r'\mu', (x,y), x+y)
display(mu.displ())
Math(mu.displ())
```

Evaluation of a function works the expected way:

```python
display(mu(x/y,y/x))
Math(mu.displ(x/y,y/x))
```

Evaluation of a function needs exactly one value for every variable:

```python
try:
    mu(1)
except TypeError as e:
    print("Error: " + str(e))
```

```python
try:
    mu.displ(1)
except TypeError as e:
    print("Error: " + str(e))
```

```python
try:
    mu.displ(1,2,3)
except TypeError as e:
    print("Error: " + str(e))
```

```python
try:
    mu.displ(1,2,3)
except TypeError as e:
    print("Error: " + str(e))
```