```python
# imports

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
%matplotlib inline
```

# Basic Concepts

## What is "learning from data"?

> In general **Learning from Data** is a scientific discipline that is concerned with the design and development of algorithms that allow computers to infer (from data) a  model that allows *compact representation* (unsupervised learning) and/or *good generalization* (supervised learning).

This is an important technology because it enables computational systems to adaptively improve their performance with experience accumulated from the observed data.

Most of these algorithms are based on the *iterative solution* of a mathematical problem that involves data and model. If there was an analytical solution to the problem, this should be the adopted one, but this is not the case for most of the cases.

So, the most common strategy for **learning from data** is based on solving a system of equations as a way to find a series of parameters of the model that minimizes a mathematical problem. This is called **optimization**.

The most important technique for solving optimization problems is **gradient descend**.

## Preliminary: Nelder-Mead method for function minimization.

The most simple thing we can try to minimize a function $f(x)$ would be to sample two points relatively near each other, and just repeatedly take a step down away from the largest value. This simple algorithm has a severe limitation: it can't get closer to the true minima than the step size.

The Nelder-Mead method dynamically adjusts the step size based off the loss of the new point. If the new point is better than any previously seen value, it **expands** the step size to accelerate towards the bottom. Likewise if the new point is worse it **contracts** the step size to converge around the minima. The usual settings are to half the step size when contracting and double the step size when expanding.

This method can be easily extended into higher dimensional examples, all that's required is taking one more point than there are dimensions. Then, the simplest approach is to replace the worst point with a point reflected through the centroid of the remaining n points. If this point is better than the best current point, then we can try stretching exponentially out along this line. On the other hand, if this new point isn't much better than the previous value, then we are stepping across a valley, so we shrink the step towards a better point.

> See "An Interactive Tutorial on Numerical Optimization": http://www.benfrederickson.com/numerical-optimization/

## Gradient descend (for *hackers*) for function minimization: 1-D

Let's suppose that we have a function $f: \mathbb{R} \rightarrow \mathbb{R}$. For example:

$$f(x) = x^2$$

Our objective is to find the argument  $x$ that minimizes this function (for maximization, consider $-f(x)$). To this end, the critical concept is the **derivative**.

The derivative of $f$ of a variable $x$, $f'(x)$ or $\frac{\partial f}{\partial x}$,  is a measure of the rate at which the value of the function changes with respect to the change of the variable.

It is defined as the following limit:

$$ f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x)}{h} $$

The derivative specifies how to scale a small change in the input in order to obtain the corresponding change in the output:

$$ f(x + h) \approx f(x) + h f'(x)$$

```python
# numerical derivative at a point x

def f(x):
    return x**2

def fin_dif(x,
            f,
            h = 0.00001):
    '''
    This method returns the derivative of f at x
    by using the finite difference method
    '''
    return (f(x+h) - f(x))/h

x = 2.0
print("{:2.4f}".format(fin_dif(x,f)))
```

    4.0000

The limit as $h$ approaches zero, if it exists, should represent the **slope of the tangent line** to $(x, f(x))$.

For values that are not zero it is only an approximation.

> **NOTE**: It can be shown that the “centered difference formula" is better when computing numerical derivatives:

> $$ \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h} $$

> The error in the "finite difference" approximation can be derived from Taylor's theorem and, assuming that $f$ is differentiable, is $O(h)$. In the case of “centered difference" the error is $O(h^2)$.

The derivative tells how to chage $x$ in order to make a small improvement in $f$.

Then, we can follow these steps to decrease the value of the function:

+ Start from a random $x$ value.
+ Compute the derivative $f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h}$.
+ Walk a small step (possibly weighted by the derivative module) in the **opposite** direction of the derivative, because we know that $f(x - h \mbox{ sign}(f'(x)) < f(x)$ for  small enough $h$.

The search for the minima ends when the derivative is zero because we have no more information about which direction to move. $x$ is a critical o stationary point if $f'(x)=0$.

 + A **minimum (maximum)** is a critical point where $f(x)$ is lower (higher) than at all neighboring points.
 + There is a third class of critical points: **saddle points**.

If $f$ is a **convex function**, this should be the minimum (maximum) of our functions. In other cases it could be a local minimum (maximum) or a saddle point.

```python
x = np.linspace(-15,15,100)
y = x**2

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        20,
        'Minimum',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

```python
x = np.linspace(-15,15,100)
y = -x**2

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-250,10])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        -30,
        'Maximum',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

```python
x = np.linspace(-15,15,100)
y = x**3

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([0],[0],'o')
plt.ylim([-3000,3000])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(0,
        400,
        'Saddle Point',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

There are two problems with numerical derivatives:
+ It is approximate.
+ It is very slow to evaluate (two function evaluations: $f(x + h) , f(x - h)$ ).

Our knowledge from Calculus could help!

We know that we can get an **analytical expression** of the derivative for **some** functions.

For example, let's suppose we have a simple quadratic function, $f(x)=x^2−6x+5$, and we want to find the minimum of this function.

#### First approach

We can solve this analytically using Calculus, by finding the derivate $f'(x)  =  2x-6$ and setting it to zero:

\begin{equation}
\begin{split}
2x-6  & = & 0 \\
2x  & = & 6 \\
x & = & 3 \\
\end{split}
\end{equation}

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([3],[3**2 - 6*3 + 5],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(3,
        10,
        'Min: x = 3',
        ha='center',
        color=sns.xkcd_rgb['pale red'],
       )
plt.show
```

#### Second approach

To find the local minimum using **gradient descend**: you start at a random point, and move into the direction of steepest **descent** relative to the derivative:

+ Start from a random $x$ value.
+ Compute the derivative $f'(x)$ analitically.
+ Walk a small step in the **opposite** direction of the derivative.

In this example, let's suppose we start at $x=15$. The derivative at this point is $2×15−6=24$.

Because we're using gradient descent, we need to subtract the gradient from our $x$-coordinate: $f(x - f'(x))$. However, notice that $15−24$ gives us $−9$, clearly overshooting over target of $3$.

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5
start = 15

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([start],[start**2 - 6*start + 5],'o')
ax.text(start,
        start**2 - 6*start + 35,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )

d = 2 * start - 6
end = start - d

plt.plot([end],[end**2 - 6*end + 5],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(end,
        start**2 - 6*start + 35,
        'End',
        ha='center',
        color=sns.xkcd_rgb['green'],
       )
plt.show
```

To fix this, we multiply the gradient by a step size. This step size (often called **alpha**) has to be chosen carefully, as a value too small will result in a long computation time, while a value too large will not give you the right result (by overshooting) or even fail to converge.

In this example, we'll set the step size to 0.01, which means we'll subtract $24×0.01$ from $15$, which is $14.76$.

This is now our new temporary local minimum: We continue this method until we either don't see a change after we subtracted the derivative step size (or until we've completed a pre-set number of iterations).

```python
old_min = 0
temp_min = 15
step_size = 0.01
precision = 0.0001

def f(x):
    return x**2 - 6*x + 5

def f_derivative(x):
    import math
    return 2*x -6

mins = []
cost = []

while abs(temp_min - old_min) > precision:
    old_min = temp_min
    gradient = f_derivative(old_min)
    move = gradient * step_size
    temp_min = old_min - move
    cost.append((3-temp_min)**2)
    mins.append(temp_min)

# rounding the result to 2 digits because of the step size
print("Local minimum occurs at {:3.6f}.".format(round(temp_min,2)))
```

    Local minimum occurs at 3.000000.

An important feature of gradient descent is that **there should be a visible improvement over time**: In this example, we simply plotted the squared distance from the local minima calculated by gradient descent and the true local minimum,  ``cost``, against the iteration during which it was calculated. As we can see, the distance gets smaller over time, but barely changes in later iterations.

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

x, y = (zip(*enumerate(cost)))

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-', alpha=0.7)
plt.ylim([-10,150])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

```python
x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.plot(mins,cost,'o', alpha=0.3)
ax.text(start,
        start**2 - 6*start + 25,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
ax.text(mins[-1],
        cost[-1]+20,
        'End (%s steps)' % len(mins),
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
plt.show
```

## From derivatives to gradient: $n$-dimensional function minimization.

Let's consider a $n$-dimensional function $f: \Re^n \rightarrow \Re$. For example:

$$f(\mathbf{x}) = \sum_{n} x_n^2$$

Our objective is to find the argument  $\mathbf{x}$ that minimizes this function.

The **gradient** of $f$ is the vector whose components are the $n$ partial derivatives of $f$. It is thus a vector-valued function.

The gradient points in the direction of the greatest rate of **increase** of the function.

$$\nabla {f} = (\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n})$$

```python
def f(x):
    return sum(x_i**2 for x_i in x)

def fin_dif_partial_centered(x,
                             f,
                             i,
                             h=1e-6):
    '''
    This method returns the partial derivative of the i-th
    component of f at x
    by using the centered finite difference method
    '''
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def fin_dif_partial_old(x,
                        f,
                        i,
                        h=1e-6):
    '''
    This method returns the partial derivative of the i-th
    component of f at x
    by using the (non-centered) finite difference method
    '''
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(x))/h

def gradient_centered(x,
                      f,
                      h=1e-6):
    '''
    This method returns the gradient vector of f at x
    by using the centered finite difference method
    '''
    return[round(fin_dif_partial_centered(x,f,i,h), 10) for i,_ in enumerate(x)]

def gradient_old(x,
                 f,
                 h=1e-6):
    '''
    This method returns the the gradient vector of f at x
    by using the (non-centered)ç finite difference method
    '''
    return[round(fin_dif_partial_old(x,f,i,h), 10) for i,_ in enumerate(x)]

x = [1.0,1.0,1.0]

print('{:.6f}'.format(f(x)), gradient_centered(x,f))
print('{:.6f}'.format(f(x)), gradient_old(x,f))
```

    3.000000 [2.0000000001, 2.0000000001, 2.0000000001]
    3.000000 [2.0000009999, 2.0000009999, 2.0000009999]

The function we have evaluated, $f({\mathbf x}) = x_1^2+x_2^2+x_3^2$, is $3$ at $(1,1,1)$ and the gradient vector at this point is $(2,2,2)$.

Then, we can follow this steps to maximize (or minimize) the function:

+ Start from a random $\mathbf{x}$ vector.
+ Compute the gradient vector.
+ Walk a small step in the opposite direction of the gradient vector.

> It is important to be aware that gradient computation is very expensive: if $\mathbf{x}$ has dimension $n$, we have to evaluate $f$ at $2*n$ points.

### How to use the gradient.

$f(x) = \sum_i x_i^2$, takes its mimimum value when all $x$ are 0.

Let's check it for $n=3$:

```python
def euc_dist(v1,v2):
    import numpy as np
    import math
    v = np.array(v1)-np.array(v2)
    return math.sqrt(sum(v_i ** 2 for v_i in v))
```

Let's start by choosing a random vector and then walking a step in the opposite direction of the gradient vector. We will stop when the difference between the new solution and the old solution is less than a tolerance value.

```python
# choosing a random vector

import random
import numpy as np

x = [random.randint(-10,10) for i in range(3)]
x
```

    [-9, -2, -9]

```python
def step(x,
         grad,
         alpha):
    '''
    This function makes a step in the opposite direction of
    the gradient vector
    in order to compute a new value for the target function.
    '''
    return [x_i - alpha * grad_i for x_i, grad_i in zip(x,grad)]

tol = 1e-15
alpha = 0.01
while True:
    grad = gradient_centered(x,f)
    next_x = step(x,grad,alpha)
    if euc_dist(next_x,x) < tol:
        break
    x = next_x
print([round(i,10) for i in x])
```

    [-0.0, -0.0, -0.0]

### Alpha

The step size, **alpha**, is a slippy concept: if it is too small we will slowly converge to the solution, if it is too large we can diverge from the solution.

There are several policies to follow when selecting the step size:

+ Constant size steps. In this case, the size step determines the precision of the solution.
+ Decreasing step sizes.
+ At each step, select the optimal step.

The last policy is good, but too expensive. In this case we would consider a fixed set of values:

```python
step_size = [100, 10, 1, 0.1, 0.01, 0.001, 0.0001, 0.00001]
```

## Learning from data

In general, we have:

+ A dataset $(\mathbf{x},y)$ of $n$ examples.
+ A target function $f_\mathbf{w}$, that we want to minimize, representing the **discrepancy between our data and the model** we want to fit. The model is represented by a set of parameters $\mathbf{w}$.
+ The gradient of the target function, $g_f$.

In the most common case $f$ represents the errors from a data representation model $M$. To fit the model is to find the optimal parameters $\mathbf{w}$ that minimize the following expression:

$$ f_\mathbf{w} = \frac{1}{n} \sum_{i} (y_i - M(\mathbf{x}_i,\mathbf{w}))^2 $$

For example, $(\mathbf{x},y)$ can represent:

+ $\mathbf{x}$: the behavior of a "Candy Crush" player; $y$: monthly payments.
+ $\mathbf{x}$: sensor data about your car engine; $y$: probability of engine error.
+ $\mathbf{x}$: finantial data of a bank customer; $y$: customer rating.

> If $y$ is a real value, it is called a *regression* problem.

> If $y$ is binary/categorical, it is called a *classification* problem.

Let's suppose that our model is a one-dimensional linear model $M(\mathbf{x},\mathbf{w}) = w \cdot x $.

### Batch gradient descend

We can implement **gradient descend** in the following way (*batch gradient descend*):

```python
import numpy as np
import random

# f = 2x
x = np.arange(10)
y = np.array([2*i for i in x])

# f_target = 1/n Sum (y - wx)**2
def target_f(x,y,w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,y,w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def step(w,grad,alpha):
    return w - alpha * grad

def BGD_multi_step(target_f,
                   gradient_f,
                   x,
                   y,
                   toler = 1e-6):
    '''
    Batch gradient descend by using a multi-step approach
    '''
    alphas = [100, 10, 1, 0.1, 0.001, 0.00001]
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_ws = [step(w, gradient, alpha) for alpha in alphas]
        next_vals = [target_f(x,y,w) for w in next_ws]
        min_val = min(next_vals)
        next_w = next_ws[next_vals.index(min_val)]
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
```

```python
print('{:.6f}'.format(BGD_multi_step(target_f, gradient_f, x, y)))
```

    1.999618

```python
%%timeit
BGD_multi_step(target_f, gradient_f, x, y)
```

    4.88 ms ± 353 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
def BGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        alpha=0.01):
    '''
    Batch gradient descend by using a given step
    '''
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_w = step(w, gradient, alpha)
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
```

```python
print('{:.6f}'.format(BGD(target_f, gradient_f, x, y)))
```

    2.000065

```python
%%timeit
BGD(target_f, gradient_f, x, y)
```

    127 µs ± 9.75 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

### Stochastic Gradient Descend

The last function evals the whole dataset $(\mathbf{x}_i,y_i)$ at every step.

If the dataset is large, this strategy is too costly. In this case we will use a strategy called **SGD** (*Stochastic Gradient Descend*).

When learning from data, the cost function is additive: it is computed by adding sample reconstruction errors.

Then, we can compute the estimate the gradient (and move towards the minimum) by using only **one data sample** (or a small data sample).

Thus, we will find the minimum by iterating this gradient estimation over the dataset.

A full iteration over the dataset is called **epoch**. During an epoch, data must be used in a random order.

If we apply this method we have some theoretical guarantees to find a good minimum:
+ SGD essentially uses the inaccurate gradient per iteration. Since there is no free food, what is the cost by using approximate gradient? The answer is that the convergence rate is slower than the gradient descent algorithm.
+ The convergence of SGD has been analyzed using the theories of convex minimization and of stochastic approximation: it converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum.

```python
import numpy as np
x = np.arange(10)
y = np.array([2*i for i in x])
data = zip(x,y)

for (x_i,y_i) in data:
    print('{:3d} {:3d}'.format(x_i,y_i))
print()

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

for (x_i,y_i) in in_random_order(data):
    print('{:3d} {:3d}'.format(x_i,y_i))
```

      0   0
      1   2
      2   4
      3   6
      4   8
      5  10
      6  12
      7  14
      8  16
      9  18

```python
import numpy as np
import random

def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):
    '''
    Stochastic gradient descend with automatic step adaptation (by
    reducing the step to its 95% when there are iterations with
    no increase)
    '''
    data = list(zip(x,y))
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    epoch = 0
    iteration_no_increase = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w
```

```python
print('w: {:.6f}'.format(SGD(target_f, gradient_f, x, y)))
```

    w: 2.000000

## Exercise: Stochastic Gradient Descent and Linear Regression

The linear regression model assumes a linear relationship between data:

$$ y_i = w_1 x_i + w_0 $$

Let's generate a more realistic dataset (with noise), where $w_1 = 2$ and $w_0 = 0$.

```python
%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

    Once deleted, variables cannot be recovered. Proceed (y/[n])? y

```python
# x: input data
# y: noisy output data

x = np.random.uniform(0,1,20)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$f(x)$', fontsize=15)
plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

Complete the following code in order to:
+ Compute the value of $w$ by using a estimator based on minimizing the squared error.
+ Get from SGD function a list, `target_value`, representing the value of the target function at each iteration.

```python
# Write your target function as f_target 1/n Sum (y - wx)**2
def target_f(x,y,w):
    # your code here
    return

# Write your gradient function
def gradient_f(x,y,w):
    # your code here
    return

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

# Modify the SGD function to return a 'target_value' vector
def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):

    # Insert your code among the following lines

    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    iteration_no_increase = 0
    epoch = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w
```

```python
# Print the value of the solution

w, target_value = SGD(target_f, gradient_f, x, y)
print('w: {:.6f}'.format(w))
```

```python
# Visualize the solution regression line

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x, y, 'o', label='t')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)', alpha=0.5)
plt.plot([0, 1], [0*w, 1*w], 'r-', label='fitted line', alpha=0.5, linestyle='--')
plt.xlabel('input x')
plt.ylabel('target t')
plt.title('input vs. target')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show
```

```python
# Visualize the evolution of the target function value during iterations.

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(np.arange(target_value.size), target_value, 'o', alpha = 0.2)
plt.xlabel('Iteration')
plt.ylabel('Cost')
plt.grid()
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show()
```

## Mini-batch Gradient Descent

In code, general batch gradient descent looks something like this:

```python
nb_epochs = 100
for i in range(nb_epochs):
    grad = evaluate_gradient(target_f, data, w)
    w = w - learning_rate * grad
```

For a pre-defined number of epochs, we first compute the gradient vector of the target function for the whole dataset w.r.t. our parameter vector.

**Stochastic gradient descent** (SGD) in contrast performs a parameter update for each training example and label:

```python
nb_epochs = 100
for i in range(nb_epochs):
    np.random.shuffle(data)
    for sample in data:
        grad = evaluate_gradient(target_f, sample, w)
        w = w - learning_rate * grad
```

Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of $n$ training examples:

```python
nb_epochs = 100
for i in range(nb_epochs):
  np.random.shuffle(data)
  for batch in get_batches(data, batch_size=50):
    grad = evaluate_gradient(target_f, batch, w)
    w = w - learning_rate * grad
```

Minibatch SGD has the advantage that it works with a slightly less noisy estimate of the gradient. However, as the minibatch size increases, the number of updates done per computation done decreases (eventually it becomes very inefficient, like batch gradient descent).

There is an optimal trade-off (in terms of computational efficiency) that may vary depending on the data distribution and the particulars of the class of function considered, as well as how computations are implemented.

## Loss Funtions

Loss functions $L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \ell(y_i, f(\mathbf{x_i}))$ represent the price paid for inaccuracy of predictions in classification/regression problems.

In classification this function is often the **zero-one loss**, that is, $ \ell(y_i, f(\mathbf{x_i}))$ is zero when $y_i = f(\mathbf{x}_i)$ and one otherwise.

This function is discontinuous with flat regions and is thus extremely hard to optimize using gradient-based methods. For this reason it is usual to consider a proxy to the loss called a *surrogate loss function*. For computational reasons this is usually convex function. Here we have some examples:

### Square / Euclidean Loss

In regression problems, the most common loss function is the square loss function:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (y_i - f(\mathbf{x}_i))^2  $$

The square loss function can be re-written and utilized for classification:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (1 - y_i f(\mathbf{x}_i))^2  $$

### Hinge / Margin Loss (i.e. Suport Vector Machines)

The hinge loss function is defined as:

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \mbox{max}(0, 1 - y_i f(\mathbf{x}_i))  $$

The hinge loss provides a relatively tight, convex upper bound on the 0–1 Loss.

### Logistic Loss (Logistic Regression)

This function displays a similar convergence rate to the hinge loss function, and since it is continuous, simple gradient descent methods can be utilized.

$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i log(1 + exp(-y_i f(\mathbf{x}_i))) $$

### Sigmoid Cross-Entropy Loss (Softmax classifier)

Cross-Entropy is a loss function that is very used for training **multiclass problems**. We'll focus on models that assume that classes are mutually exclusive.

In this case, our labels have this form $\mathbf{y}_i =(1.0,0.0,0.0)$. If our model predicts a different distribution, say  $ f(\mathbf{x}_i)=(0.4,0.1,0.5)$, then we'd like to nudge the parameters so that $f(\mathbf{x}_i)$ gets closer to $\mathbf{y}_i$.

C.Shannon showed that if you want to send a series of messages composed of symbols from an alphabet with distribution $y$ ($y_j$  is the probability of the  $j$-th symbol), then to use the smallest number of bits on average, you should assign  $\log(\frac{1}{y_j})$  bits to the  $j$-th symbol.

The optimal number of bits is known as **entropy**:

$$ H(\mathbf{y}) = \sum_j y_j \log\frac{1}{y_j} = - \sum_j y_j \log y_j$$

**Cross entropy** is the number of bits we'll need if we encode symbols by using a wrong distribution $\hat y$:

$$ H(y, \hat y) =   - \sum_j y_j \log \hat y_j $$

In our case, the real distribution is $\mathbf{y}$ and the "wrong" one is $f(\mathbf{x}_i)$. So, minimizing **cross entropy** with respect our model parameters will result in the model that best approximates our labels if considered as a probabilistic distribution.

Cross entropy is used in combination with **Softmax** classifier. In order to classify $\mathbf{x}_i$ we could take the index corresponding to the max value of $f(\mathbf{x}_i)$, but Softmax gives a slightly more intuitive output (normalized class probabilities) and also has a probabilistic interpretation:

$$ P(\mathbf{y}_i = j \mid \mathbf{x_i}) = - log \left( \frac{e^{f_j(\mathbf{x_i})}}{\sum_k e^{f_k(\mathbf{x_i})} } \right) $$

where $f_k$ is a linear classifier.

## Advanced gradient descend

### Momentum

SGD has trouble navigating ravines, i.e. areas where the surface curves much more steeply in one dimension than in another, which are common around local optima. In these scenarios, SGD oscillates across the slopes of the ravine while only making hesitant progress along the bottom towards the local optimum.

Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations. It does this by adding a fraction of the update vector of the past time step to the current update vector:

$$ v_t = m v_{t-1} + \alpha \nabla_w f $$
$$ w = w - v_t    $$

The momentum $m$ is commonly set to $0.9$.

### Nesterov

However, a ball that rolls down a hill, blindly following the slope, is highly unsatisfactory. We'd like to have a smarter ball, a ball that has a notion of where it is going so that it knows to slow down before the hill slopes up again.

Nesterov accelerated gradient (NAG) is a way to give our momentum term this kind of prescience. We know that we will use our momentum term $m v_{t-1}$ to move the parameters $w$. Computing
$w - m v_{t-1}$ thus gives us an approximation of the next position of the parameters (the gradient is missing for the full update), a rough idea where our parameters are going to be. We can now effectively look ahead by calculating the gradient not w.r.t. to our current parameters $w$ but w.r.t. the approximate future position of our parameters:

$$ w_{new} = w - m v_{t-1} $$
$$ v_t = m v_{t-1} + \alpha \nabla_{w_{new}} f $$
$$ w = w - v_t $$

### Adagrad

All previous approaches manipulated the learning rate globally and equally for all parameters. Tuning the learning rates is an expensive process, so much work has gone into devising methods that can adaptively tune the learning rates, and even do so per parameter.

Adagrad is an algorithm for gradient-based optimization that does just this: It adapts the learning rate to the parameters, performing larger updates for infrequent and smaller updates for frequent parameters.

$$ c = c + (\nabla_w f)^2 $$
$$ w = w - \frac{\alpha}{\sqrt{c}} $$

### RMProp

RMSProp update adjusts the Adagrad method in a very simple way in an attempt to reduce its aggressive, monotonically decreasing learning rate. In particular, it uses a moving average of squared gradients instead, giving:

$$ c = \beta c + (1 - \beta)(\nabla_w f)^2 $$
$$ w = w - \frac{\alpha}{\sqrt{c}} $$

where $\beta$ is a decay rate that controls the size of the moving average.

(Image credit: Alec Radford)

(Image credit: Alec Radford)

```python
%reset

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline
```

    Once deleted, variables cannot be recovered. Proceed (y/[n])? y

```python
# the function that I'm going to plot
def f(x,y):
    return x**2 + 5*y**2

x = np.arange(-3.0,3.0,0.1)
y = np.arange(-3.0,3.0,0.1)
X,Y = np.meshgrid(x, y, indexing='ij') # grid of point
Z = f(X, Y) # evaluation of the function on the grid

plt.pcolor(X, Y, Z, cmap=plt.cm.gist_earth)
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.gca().set_aspect('equal', adjustable='box')
plt.gcf().set_size_inches((6,6))
plt.show()
```

```python
def target_f(x):
    return x[0]**2.0 + 5*x[1]**2.0

def part_f(x,
           f,
           i,
           h=1e-6):
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def gradient_f(x,
               f,
               h=1e-6):
    return np.array([round(part_f(x,f,i,h), 10) for i,_ in enumerate(x)])
```

```python
def SGD(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001):
    alpha = alpha_0
    min_val = float('inf')
    steps = 0
    iteration_no_increase = 0
    trace = []
    while iteration_no_increase < 100:
        val = target_f(x)
        if min_val - val > toler:
            min_val = val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            alpha *= 0.95
            iteration_no_increase += 1
        trace.append(x)
        gradient_i = gradient_f(x, target_f)
        x = x - (alpha *  gradient_i)
        steps += 1
    return x, val, steps, trace

x = np.array([2,-2])
x, val, steps, trace = SGD(target_f, gradient_f, x)
print(x)
print('Val: {:.6f}, steps: {:.0f}'.format(val, steps))
```

    [ 8.73043864e-04 -4.00026038e-12]
    Val: 0.000001, steps: 475

```python
def SGD_M(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001,
        m = 0.9):

    alpha = alpha_0
    min_val = float('inf')
    steps = 0
    iteration_no_increase = 0
    v = 0.0
    trace = []
    while iteration_no_increase < 100:
        val = target_f(x)
        if min_val - val > toler:
            min_val = val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            alpha *= 0.95
            iteration_no_increase += 1
        trace.append(x)
        gradient_i = gradient_f(x, target_f)
        v = m * v + (alpha *  gradient_i)
        x = x - v
        steps += 1
    return x, val, steps, trace

x = np.array([2,-2])
x, val, steps, trace2 = SGD_M(target_f, gradient_f, x)

print('\n',x)
print('Val: {:.6f}, steps: {:.0f}'.format(val, steps))
```

     [1.90552394e-05 1.27080227e-05]
    Val: 0.000000, steps: 241

```python
x2 = np.array(range(len(trace)))
x3 = np.array(range(len(trace2)))
plt.xlim([0,len(trace)])
plt.gcf().set_size_inches((10,3))
plt.plot(x3, trace2)
plt.plot(x2, trace, '-')
```


# Setting up a Demand Function

## 1. Why do we need it?

paminco allows to compute parametric minimum cost flows (MCF) w.r.t. to demand multiplier $\lambda$. I.e., the goal is to find a MCF for all values of $\lambda$ in the interval $\left[0, \lambda^{\max }\right]$:

$$
\begin{align*}
    \min \text { } & C(\mathbf{x}) \\
    \text {s.t.} \quad \mathbf{\Gamma} \mathbf{x} &= \mathbf{h}(\lambda), \\
    \mathbf{x} &\geq \mathbf{0},
\end{align*}
$$

where $\mathbf{h}(\lambda)$ is a ``demand function`` that maps $\lambda$ to a ``demand vector`` $\mathbf{b}$.

## 2. Demand vectors and matrices

### 2.1. Demand vector

For single commodity problems a demand vector is a (n, ) array that encodes the inflow and outflow rate per node, negative values indicating *sources* and positive values *sinks*.

Consider the following example.
The network consists of four vertices and five edges.
We want to send 10 units from vertex $A$ to vertex $D$, indicated by the wiggly green arrow

```{figure} ./img/graph_demand.jpg
  :name: simple network
  :width: 60%

  Simple network with demand.
```

#### Network setup

We setup this network (without demand) as follows:

```python
import paminco
import numpy as np

edge_data = np.array([
    ["A", "B"],
    ["A", "C"],
    ["B", "C"],
    ["B", "D"],
    ["C", "D"],
])
graph = paminco.Network(edge_data)
graph.nodes.lbl2id
```

    {'A': 0, 'B': 1, 'C': 2, 'D': 3}

#### Building demand vector

We now build a demand vector:

```python
demand_vector = paminco.net.demand.demand_vector(
    ("A", "D", 10),  # send 10 units from A to D
    shared=graph.shared,  # each network element must know about shared data ...
)
demand_vector.map_node_label_to_id()
```

#### Retrieving sparse representation

This demand vector consists of one CommoditySingleSourceSink:

```python
demand_vector.commodities
```

    [CommoditySingleSourceSink @ 0x2b484184a90
     'A' (0) → 'D' (3) | 10.]

Algorithms in paminco expect the demand vector to be sparse matrix, which can be accessed by:

```python
demand_vector()
```

    <4x1 sparse matrix of type '<class 'numpy.float64'>'
    	with 2 stored elements in Compressed Sparse Column format>

```python
demand_vector().toarray()
```

    array([[-10.],
           [  0.],
           [  0.],
           [ 10.]])

#### Setting demand function

We equip the network with a linear demand function that scales this demand vector by the demand multiplier $\lambda$:

```python
graph.set_demand(demand_vector)
graph.demand
```

    <paminco.net.demand.LinearDemandFunction at 0x2b484184790>

For a linear demand function with $\lambda = 1$ (no scaling), we expect to retrieve the specified demand vector:

```python
graph.demand(1).toarray()
```

    array([[-10.],
           [  0.],
           [  0.],
           [ 10.]])

For other values of $\lambda$, the input demand vector is scaled:

```python
graph.demand(0.32).toarray()
```

    array([[-3.2],
           [ 0. ],
           [ 0. ],
           [ 3.2]])

```python
graph.demand(1.63).toarray()
```

    array([[-16.3],
           [  0. ],
           [  0. ],
           [ 16.3]])

#### All in one go

By default, the network constructor builds a linear demand function from the ``demand_data``:

```python
graph.set_demand(("A", "D", 10))
graph.demand(1.63).toarray()
```

    array([[-16.3],
           [  0. ],
           [  0. ],
           [ 16.3]])

### 2.2. Demand matrix

For certain problems, the demand may consist of multiple commodities.
E.g., in a traffic network some drivers want to go from one location to another, and others vice versa.
Consider an adaption of the graph above:

```{figure} ./img/graph_demand_multi.jpg
  :name: demand with multiple sources and/or sinks
  :width: 60%

  Demand as in/outflow per node in the graph.
```
Here, we set five commodities

$$
\begin{align*}
  A & \overset{2}{\rightarrow} B \quad (\text{blue}), \\
  A & \overset{5}{\rightarrow} D \quad (\text{orange}), \\
  B & \overset{7}{\rightarrow} A \quad (\text{light green}), \\
  C & \overset{5}{\rightarrow} B \quad (\text{dark green}), \\
  D & \overset{5}{\rightarrow} C \quad (\text{red}).
\end{align*}
$$

```python
commodities = [
    ("A", "B", 2),
    ("A", "D", 5),
    ("B", "A", 7),
    ("C", "B", 5),
    ("D", "C", 5),
]
demand_matrix = paminco.net.demand_vector(commodities, shared=graph.shared)
demand_matrix.map_node_label_to_id()
```

We encoded the commodites in a demand matrix, where each column is a demand vector:

```python
import pandas as pd
d = pd.DataFrame(demand_matrix().toarray())
d.set_index(graph.nodes.labels)
```

      0
      1
      2
      3
      4

      A
      -2.0
      -5.0
      7.0
      0.0
      0.0

      B
      2.0
      0.0
      -7.0
      5.0
      0.0

      C
      0.0
      0.0
      0.0
      -5.0
      5.0

      D
      0.0
      5.0
      0.0
      0.0
      -5.0

### 2.3. Commodity with multiple sinks and sources

So far, we considered only commodities that have a single source and single sink.
However, for certain applications -- such as a gas network -- a commodity may consits of ``multiple sources and/or multiple sinks``.
This CommodityMultiSourceSink is encoded as a vector or dictionary that specfiies in and outflow rate of each node:

```python
com = {"A": -5, "B": -2, "C": 1, "D": 6}
demand_vector = paminco.net.demand_vector(com, shared=graph.shared)
demand_vector.commodities
```

    [CommodityMultiSourceSink @ 0x2b485e8c790
     'A' : -5.0
     'B' : -2.0
     'C' : 1.0
     'D' : 6.0]

In a sense, hhis commodity specifies that we input 5 units at $A$ and 2 at $B$, which are consumed by $D (6)$ and $C (1)$, i.e., $(A, B) \overset{7}{\rightarrow} (C, D)$.'

## 3. Function types

paminco has allows for two demand function types (other can be implemented by inheritance of paminco.demand.DemandFunction): linear demand functions and affine demand functions.
For notational simplicity, we only consider demands that consists of a single commodity for the rest of this section.

### 3.1. Linear demand function

A linear demand function is of the form:

$$
\begin{equation}
    \mathbf{h}_{\text{lin}}(\lambda) = \lambda \mathbf{b},
\end{equation}
$$

where $\mathbf{b}$ is a demand vector (see 1.1). In a sense, the demand is simply scaled by the parameter $\lambda$.

```python
import paminco
import numpy as np

edge_data = np.array([
    ["A", "B"],
    ["A", "C"],
    ["B", "C"],
    ["B", "D"],
    ["C", "D"],
])
demand_data = ("A", "D", 1)

graph = paminco.Network(edge_data, demand_data=demand_data)
```

The demand function of the network for $\lambda = 2.5$ -- i.e., sending 2.5 times the original demand (1) from $A$ to $D$ -- is called by:

```python
graph.demand(2.5)   # or graph.h(2.5)
```

    <4x1 sparse matrix of type '<class 'numpy.float64'>'
    	with 2 stored elements in Compressed Sparse Column format>

which is a $n \times k$ sparse matrix. This matrix encodes the in and outflow rates for all nodes (row) for all commodities (column). It is easily converted to a numpy.ndarray:

```python
graph.demand(2.5).toarray()
```

    array([[-2.5],
           [ 0. ],
           [ 0. ],
           [ 2.5]])

For example, the -2.5 indicates that the node at index 0 (node $A$) has a total inflow of -2.5 units, negative rates thus marking *source*. All the inflow in node $A$ is then "consumed" by the node at index 3 (node $D$, a *sink*).

### 3.2. Affine demand function

In contrast, an *affine demand function* consists of two parts: a base demand $\mathbf{b}_0$ that is invariant of $\lambda$ and a part that scales linearly with the demand multiplier:

$$
\begin{equation}
    \mathbf{h}_{\text{affine}}(\lambda) = \mathbf{b}_0 + \lambda \mathbf{b}.
\end{equation}
$$

This is of interest for problems where there exsists a trivial solution to problems with linearly scaled demands.
Using the data from the above example, we can setup an affine demand function as follows:

```python
b0 = ("B", "D", 1)
b = ("A", "D", 1)
demand_data = (b0, b)

graph2 = paminco.Network(
    edge_data,
    demand_data=demand_data,
    kw_demand={"mode": "affine"}
)
```

This graph now has a demand part that is independent of the demand multiplier:

```python
graph2.h(0).toarray()
```

    array([[ 0.],
           [-1.],
           [ 0.],
           [ 1.]])

and one that scales linearly with it:

```python
graph2.h(7).toarray()
```

    array([[-7.],
           [-1.],
           [ 0.],
           [ 8.]])

We can think of calling this affine demand function with $\lambda = 7$ as sending 7 units from $A$ to $D$ and 1 unit from $B$ to $D$, resuilting in total 8 units in the sink $D$.


# Thinking probabilistically - Continuous variables
> A Summary of lecture "Statistical Thinking in Python (Part 1)", via datacamp

- toc: true
- badges: true
- comments: true
- author: Chanseok Kang
- categories: [Python, Datacamp, Data_Science, Statistics]
- image: images/std-cdf.png

## Probability density functions
- Continuous variables
    - Quantities that can take any value, not just discrete values
- Probability Density function (PDF)
    - Continuous analog to the PMF
    - Mathematical description of the relative likelihood of observing a value of a continuous variable

## Introduction to the Normal distribution
- Normal distribution
    - Describes a continuous variable whose PDF has a single symmetric peak.

    $$ \begin{align} \text{mean of a Normal distribution} & \neq \text{mean computed from data} \\ \text{st. dev of a Normal distribution} & \neq \text{st. dev computed from data} \end{align} $$

### The Normal PDF
In this exercise, you will explore the Normal PDF and also learn a way to plot a PDF of a known distribution using hacker statistics. Specifically, you will plot a Normal PDF for various values of the variance.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

sns.set()
```

```python
# Draw 100000 samples from Normal distribution
# with stds of interest: samples_std1, samples_std3, stamples_std10
samples_std1 = np.random.normal(20, 1, 100000)
samples_std3 = np.random.normal(20, 3, 100000)
samples_std10 = np.random.normal(20, 10, 100000)

# Make histograms
_ = plt.hist(samples_std1, histtype='step', density=True, bins=100)
_ = plt.hist(samples_std3, histtype='step', density=True, bins=100)
_ = plt.hist(samples_std10, histtype='step', density=True,bins=100)

# Make a legend, set limits
_ = plt.legend(('std = 1', 'std = 3', 'std = 10'))
plt.ylim(-0.01, 0.42)
```

### The Normal CDF
Now that you have a feel for how the Normal PDF looks, let's consider its CDF. Using the samples you generated in the last exercise (in your namespace as ```samples_std1```, ```samples_std3```, and ```samples_std10```), generate and plot the CDFs.

```python
def ecdf(data):
    """Compute ECDF for a one-dimensional array of measurements."""
    # Number of data points: n
    n = len(data)

    # x-data for the ECDF: x
    x = np.sort(data)

    # y-data for the ECDF: y
    y = np.arange(1, n + 1) / n

    return x, y
```

```python
# Generate CDFs
x_std1, y_std1 = ecdf(samples_std1)
x_std3, y_std3 = ecdf(samples_std3)
x_std10, y_std10 = ecdf(samples_std10)

# Plot CDFs
_ = plt.plot(x_std1, y_std1, marker='.', linestyle='none')
_ = plt.plot(x_std3, y_std3, marker='.', linestyle='none')
_ = plt.plot(x_std10, y_std10, marker='.', linestyle='none')

# Make a legend
_ = plt.legend(('std = 1', 'std = 3', 'std = 10'), loc='lower right')
plt.savefig('../images/std-cdf.png')
```

## The Normal distribution: Properties and warnings

### Are the Belmont Stakes results Normally distributed?
Since 1926, the Belmont Stakes is a 1.5 mile-long race of 3-year old thoroughbred horses. [Secretariat](https://en.wikipedia.org/wiki/Secretariat_(horse) ) ran the fastest Belmont Stakes in history in 1973. While that was the fastest year, 1970 was the slowest because of unusually wet and sloppy conditions. With these two outliers removed from the data set, compute the mean and standard deviation of the Belmont winners' times. Sample out of a Normal distribution with this mean and standard deviation using the ```np.random.normal()``` function and plot a CDF. Overlay the ECDF from the winning Belmont times. Are these close to Normally distributed?

Note: Justin scraped the data concerning the Belmont Stakes from the [Belmont Wikipedia page](https://en.wikipedia.org/wiki/Belmont_Stakes).

```python
df = pd.read_csv('./dataset/belmont.csv')

belmont_no_outliers = np.array([148.51, 146.65, 148.52, 150.7 , 150.42, 150.88, 151.57, 147.54,
       149.65, 148.74, 147.86, 148.75, 147.5 , 148.26, 149.71, 146.56,
       151.19, 147.88, 149.16, 148.82, 148.96, 152.02, 146.82, 149.97,
       146.13, 148.1 , 147.2 , 146.  , 146.4 , 148.2 , 149.8 , 147.  ,
       147.2 , 147.8 , 148.2 , 149.  , 149.8 , 148.6 , 146.8 , 149.6 ,
       149.  , 148.2 , 149.2 , 148.  , 150.4 , 148.8 , 147.2 , 148.8 ,
       149.6 , 148.4 , 148.4 , 150.2 , 148.8 , 149.2 , 149.2 , 148.4 ,
       150.2 , 146.6 , 149.8 , 149.  , 150.8 , 148.6 , 150.2 , 149.  ,
       148.6 , 150.2 , 148.2 , 149.4 , 150.8 , 150.2 , 152.2 , 148.2 ,
       149.2 , 151.  , 149.6 , 149.6 , 149.4 , 148.6 , 150.  , 150.6 ,
       149.2 , 152.6 , 152.8 , 149.6 , 151.6 , 152.8 , 153.2 , 152.4 ,
       152.2 ])
```

```python
# Compute mean and standard deviation: mu, sigma
mu = np.mean(belmont_no_outliers)
sigma = np.std(belmont_no_outliers)

# Sample out of a normal distribution with this mu and sigma: samples
samples = np.random.normal(mu, sigma, size=10000)

# Get the CDF of the samples and of the data
x_theor, y_theor = ecdf(belmont_no_outliers)
x, y = ecdf(samples)

# Plot the CDFs
_ = plt.plot(x_theor, y_theor)
_ = plt.plot(x, y, marker='.', linestyle='none')
_ = plt.xlabel('Belmont winning time (sec.)')
_ = plt.ylabel('CDF')
```

### What are the chances of a horse matching or beating Secretariat's record?
Assume that the Belmont winners' times are Normally distributed (with the 1970 and 1973 years removed), what is the probability that the winner of a given Belmont Stakes will run it as fast or faster than Secretariat?

```python
# Take a million samples out of the Normal distribution: samples
samples = np.random.normal(mu, sigma, size=1000000)

# Compute the fraction that are faster than 144 seconds: prob
prob = np.sum(samples < 144) / len(samples)

# Print the result
print('Probability of besting Secretariat:', prob)
```

    Probability of besting Secretariat: 0.00057

## The Exponential distribution
- The waiting time between arrivals of a Poisson process is Exponentially distributed

### If you have a story, you can simulate it!
Sometimes, the story describing our probability distribution does not have a named distribution to go along with it. In these cases, fear not! You can always simulate it. We'll do that in this and the next exercise.

In earlier exercises, we looked at the rare event of no-hitters in Major League Baseball. Hitting the cycle is another rare baseball event. When a batter hits the cycle, he gets all four kinds of hits, a single, double, triple, and home run, in a single game. Like no-hitters, this can be modeled as a Poisson process, so the time between hits of the cycle are also Exponentially distributed.

How long must we wait to see both a no-hitter and then a batter hit the cycle? The idea is that we have to wait some time for the no-hitter, and then after the no-hitter, we have to wait for hitting the cycle. Stated another way, what is the total waiting time for the arrival of two different Poisson processes? The total waiting time is the time waited for the no-hitter, plus the time waited for the hitting the cycle.

Now, you will write a function to sample out of the distribution described by this story.

```python
def successive_poisson(tau1, tau2, size=1):
    """Compute time for arrival of 2 successive Poisson processes."""
    # Draw samples out of first exponential distribution: t1
    t1 = np.random.exponential(tau1, size)

    # Draw samples out of second exponential distribution: t2
    t2 = np.random.exponential(tau2, size)

    return t1 + t2
```

### Distribution of no-hitters and cycles
Now, you'll use your sampling function to compute the waiting time to observe a no-hitter and hitting of the cycle. The mean waiting time for a no-hitter is 764 games, and the mean waiting time for hitting the cycle is 715 games.

```python
# Draw samples of waiting times: waiting_times
waiting_times = successive_poisson(764, 715, size=100000)

# Make the histogram
_ = plt.hist(waiting_times, bins=100, density=True, histtype='step')

# Label axes
_ = plt.xlabel('waiting times')
_ = plt.ylabel('PDF')
```


# On Eigendecomposition of Asset Returns
### Francisco A. Ibanez

## Part 1. $\Sigma$ v. $R$
To test some of the outstanding points on the stability of the eigendecomposition of asset returns, we will analyze simulated returns to avoid drawing conclusions from a specific dataset, which might not serve the general case. Because the eigendocmposition of a 2-by-2 covariance matrix is trivial, we need to work with at least the three-assets case.

Let's assume a market where the asset price dynamics can be described by the following stochastic differential equation:

\begin{align}
d P_{t} &= \mu_{i} P_{t} dt + \sigma P_{t} dW_{t} \\
\end{align}

where $P_{i,t}$ is the price of asset $i$ at time $t$, $\mu$ and $\sigma$ are the mean and standard deviation of the asset's returns and $W_{t}$ is a Wiener process with Gaussian increments. Using the Cholesky decomposition, the previous equation can be expanded to a two-asset framework with prices $S_{1}$ and $S_{2}$ with non-zero correlation $\rho$ as follows:

\begin{align}
d S_{1,t} &= \mu_{1} S_{1, t} dt + \sigma_{1} S_{1, t} dW_{t} \\
d S_{2, t} &= \mu_{2} S_{2, t} dt + \sigma_{2} S_{2, t} \rho dW_{t} + \sigma_{2} S_{2, t} \sqrt{1 - \rho^{2}} dZ_{t}
\end{align}

We can generalize the system above for the case with multiple assets, by summarizing their interaction in the correlation matrix $R$:

\begin{align}
R = \begin{bmatrix} 1 & \rho_{1, 2} & \cdots & \rho_{1, n} \\ \rho_{2, 1} & 1 & \cdots & \rho_{2, n} \\ \vdots & \vdots & \ddots & \vdots \\ \rho_{n, 1} & \rho_{n, 2} & \cdots & \rho_{n, n}\end{bmatrix}
\end{align}

where where $\rho_{i, j}$ is the correlation coefficient between asset $i$ and asset $j$. We can now apply again the Cholesky decomposition on $R$

\begin{align}
R &= LL^{T} \\
\end{align}

and expand to describe the dynamics of $N$ correlated assets:

\begin{align}
\frac{dP}{P} &= \mu dt + S L W_{t}
\end{align}

where S is a diagonal matrix that has the standard deviation of the asset in the diagonal entries, and zero everywhere else. The expression applied to the three-asset case with no drift term yields:

\begin{align}
\begin{bmatrix}
\frac{dP_{1,t}}{P_{1,t}} \\ \frac{dP_{2,t}}{P_{2,t}} \\ \frac{dP_{3,t}}{P_{3,t}}
\end{bmatrix} = \begin{bmatrix} \sigma_{1} & 0 & 0 \\ 0 & \sigma_{2} & 0 \\ 0 & 0 & \sigma_{3} \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ \rho_{2,1} & \sqrt{1 - \rho_{2,1}^{2}} & 0 \\ \rho_{3,1} & \frac{\rho_{3,2} - \rho_{3,1}\rho_{2,1}}{\sqrt{1 - \rho_{2,1}^{2}}} & \sqrt{1 - \rho_{2,1}^{2} - \left( \frac{\rho_{3,2} - \rho_{3,1}\rho_{2,1}}{\sqrt{1 - \rho_{2,1}^{2}}} \right)^{2}} \end{bmatrix}
\begin{bmatrix} dW_{1, t} \\ dW_{2, t} \\ dW_{3, t} \end{bmatrix}
\end{align}

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import string

n = 1E4
np.random.seed(0)
dWt = np.random.randn(int(n), 3)
Rho = np.array([
    [1, .5, .5],
    [.5, 1, .5],
    [.5, .5, 1]
])
L = np.linalg.cholesky(Rho)

sigma = [np.diag([x, 1, 1]) for x in range(1, 3)]

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(12.5, 5), sharey=True, sharex=True)
colors = ['darkorange', 'dodgerblue']
for i in range(2):
    S = sigma[i]
    r_t = S @ L @ dWt.T
    u, s, v_t = np.linalg.svd(r_t.T, full_matrices=False, compute_uv=True)

    for asset in range(v_t.shape[1]):
        xy = v_t[:2, asset]
        axs[i].arrow(*(0, 0), *xy, color=colors[i], width=0.0025)
        axs[i].annotate(string.ascii_uppercase[asset], xy, fontsize=12.5)
    axs[i].grid(ls='--')

axs[0].set_title('Balanced variance case')
axs[1].set_title('Unbalanced variance case')
plt.show()
```

## Part 2. Impact of outliers

http://faculty.baruch.cuny.edu/jgatheral/JumpDiffusionModels.pdf

\begin{align}
d S_{1,t} &= \mu_{1} S_{1, t} dt + \sigma_{1} S_{1, t} dW_{t} + S_{1, t} d J_{1, t}\\
d S_{2, t} &= \mu_{2} S_{2, t} dt + \sigma_{2} S_{2, t} \rho dW_{t} + \sigma S_{2, t} \sqrt{1 - \rho^{2}} dZ_{t}  + S_{2, t} d J_{2, t}
\end{align}

Points to make:
1. Financial time series are rarely distributed normal (left tail events)
2. Add jump difussion to the exercise above, show the impact of the jumps in the resulting eigenvalues (paint the outliers differently)
3. Is it possible to measure the impact? (even if it's through simulations)

```python
n = 10000
np.random.seed(0)
dWt = np.random.randn(n, 2)
mu = np.array([.1, .15])
sigma = np.array([.125, .25])
rho = .6
r1 = mu[0] + sigma[0] * dWt[:, 0]
r2 = mu[1] + sigma[1] * rho * dWt[:, 0] + sigma[1] * np.sqrt(1 - rho**2) * dWt[:, 1]

np.random.seed(0)
jump_location = np.random.poisson(lam=.25, size=(n, 2))
jump_magnitude = np.random.normal(loc=-.1, scale=.15, size=(n, 2))
dJt = np.multiply(jump_location, jump_magnitude)

r1j = r1 + dJt[:, 0].flatten()
r2j = r2 + dJt[:, 1].flatten()

matrices = {
    'Normal': np.cov(r1, r2),
    'Normal + Jumps': np.cov(r2j, r2j)
}

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(12.5, 5), sharey=True, sharex=True)
for i, m in enumerate(matrices):
    L, V = np.linalg.eig(matrices[m])
    order = L.argsort()[::-1]
    L = L[order]
    V = V[:, order]
    flip_sign = np.sign(V[:, 0].sum())

    axs[i].set_title(m)
    if i == 1:
        axs[i].scatter(r1j, r2j, color='red', alpha=.5)
    axs[i].scatter(r1, r2, color='dodgerblue', alpha=.5)
    axs[i].set_xlim(-1, 1)
    axs[i].set_ylim(-1, 1)
    axs[i].quiver(*mu, *V[:, 0] * flip_sign, color=['r'], scale=5)
    axs[i].quiver(*mu, *V[:, 1], color=['orange'], scale=5)  # Scale should be proportional to the eigenvalue
    axs[i].grid(ls='--')
plt.show()
```

```python
import seaborn as sns

data = pd.DataFrame([r1, r2j], index=['Normal', 'Normal + Jumps']).T
g = sns.jointplot(x=data['Normal'], y=data['Normal + Jumps'])
# g.plot_joint(sns.kdeplot, color="r", zorder=100, levels=10, fill=True, bw=.10)

```

## Part 3. Robust Principal Component Analysis

The Singular Value Decomposition is highly susceptible to outliers and corrupted data, making it fragile with respect to non-normal disturbances. Candes et al. (2011) developed an approach to robust principal component analysis that allows one to recover the principal components of a data matrix even though a positive fractivon of its entries are arbitrarly corrupted, and even missing, with applications in the area of face recognition.

The methodology seeks to decompose the data matrix $X$ into a structured low-rank matrix $L$ and a sparse matrix $S$ containing outliers and corrupt data:

\begin{aligned}
M=L_0+S_0
\end{aligned}

The principal components of $L$ are robust to outliers and corrupt data in $S$. Mathematically, the goal is to find $L$ and $S$ that satisfy the following:

\begin{aligned}
\min_{L,S} ||L||_{*} + \lambda ||S||_{1} \\
\text{subject to} \\ L + S = M
\end{aligned}

The authors recommend, for practical problems, to set $\lambda = 1 / \sqrt{\max (N, T)}$ as a good rule of thumb, which can then be adjusted slightly to obtain the best possible result.

From https://www.hindawi.com/journals/acisc/2017/2520301/

For a scalar $\tau$, the shrinkage operator $S_{\tau}(x)$ for $\ell_{1}$ norm minimization problem is defined as follows:

\begin{aligned}
S_{\tau}(X) = \text{sgn}(x)max(|x| - \tau, 0)
\end{aligned}

where sgn is a signum function; $|.|$ calculates the absolute value.

```python
import yfinance as yf
import pandas_datareader.data as web

# ETF prices
meta = pd.read_csv('etfs_comprehensive.csv', index_col=0)
#meta = meta.iloc[:4]
collect = []
for n in meta.index:
    ticker = yf.Ticker(n)
    px = ticker.history(period="max")['Close']
    px.name = n
    collect += [px]

prices = pd.concat(collect, axis=1)

# Risk free rate (Fama-French)
ff = web.DataReader(
    'F-F_Research_Data_Factors_daily',
    'famafrench',
    start='12/31/1999'
)
rf = ff[0]['RF'] * .01  # Originally expressed in percentage points

# Subtract the risk-free rate
total_return = prices.pct_change().dropna()
excess_return = total_return.sub(rf.reindex(total_return.index, method='ffill'), axis=0)
```

```python
from rpca import RobustPCA
X = excess_return.sub(excess_return.mean()).div(excess_return.std()).values
n1, n2 = X.shape
lambda_ = 1 / np.sqrt(max(n1, n2)) * 4
rob = RobustPCA(lmb=lambda_, max_iter=int(1E6))
L, S = rob.fit(X)
```

```python
er_robust = pd.DataFrame(L, index=excess_return.index, columns=excess_return.columns)
er_noise = pd.DataFrame(S, index=excess_return.index, columns=excess_return.columns)
print('Original')
excess_return.plot()
print('Noise')
er_noise.plot()
print('Robust')
er_robust.plot()
```

```python
'''
Truncation
----------
Sources:
    - https://arxiv.org/pdf/1305.5870.pdf
    - http://www.pyrunner.com/weblog/2016/08/01/optimal-svht/
    - https://github.com/erichson/optht
'''
from scipy.linalg import svd
from optht import optht

u, s, vh = svd(er_robust, full_matrices=False, compute_uv=True, lapack_driver='gesdd')
pd.Series(s / s.sum()).plot(title='Singular Values', color='blue', lw=2)
k = optht(er_robust.values, sv=s, sigma=None)
print('Optimal truncation (robust): ', k)
plt.gca().axvline(k, color='blue', ls='--')

u, s, vh = svd(excess_return, full_matrices=False, compute_uv=True, lapack_driver='gesdd')
pd.Series(s / s.sum()).plot(title='Singular Values', color='red')
k = optht(excess_return.values, sv=s, sigma=None)
print('Optimal truncation (raw): ', k)
plt.gca().axvline(k, color='red', ls='--')

# The low rank structure of the data becomes more aparaent when the data is cleaned with RPCA
```

```python
# M = excess_return.sub(excess_return.mean()).div(excess_return.std())
# M = M.values

# def shrink(X, tau):
#     shrinked_X = np.multiply(np.sign(X), (np.abs(S) - tau).clip(0))
#     return shrinked_X

# def SVT(X, tau):
#     U, S, Vt = np.linalg.svd(X, full_matrices=False, compute_uv=True)
#     res = U @ shrink(np.diag(S), tau) @ Vt
#     return res

# # RPCA
# X = M.copy()
# n1, n2 = M.shape
# mu = n1 * n2 / (4 * np.abs(M).sum())
# tau = 1 / mu
# lambda_ = 1 / np.sqrt(max(n1, n2))
# thresh = 1E-7 * np.linalg.norm(X, ord='fro')

# L = np.zeros((n1, n2), dtype=float)
# S = np.zeros((n1, n2), dtype=float)
# Y = np.zeros((n1, n2), dtype=float)
# count = 0

# while (np.linalg.norm(X - S - S, 'fro')) > thresh and (count < 1000):
#     L = SVT(X=X - S + (1 / mu) * Y, tau=1 / mu)
#     S = shrink(X - L + (1 / mu), tau=lambda_ / mu)
#     Y = Y + mu * (X - L - S)
#     count = count + 1
```

## Part 4. Sparse Representation of the Portfolio Risk

Points to make:
1. Noise might also be captured by the last eigenvalues
2. Given the ordered nature of the eigenvalues, main drivers of return are identified in the first N eigenvectors
3. More eigenvectors to equalize (SRP) more computation time
3. Dropping last eigenvectors should reduce noise in the projection and speed up the computation time

## Part 5. The Eigenvector Sign Issue ($J$)

Points to make:
1. How can I select a $j$ within the $2^{N-1}$ columns of $J$?


```python
from sympy import *
from IPython.display import display
```

### Path length of LoS in galactic ellipsoid
We want to calculate the path length of an LoS that intersects an ellipsoid with two identical axes, $1 = \frac{x^2}{a^2} + \frac{y^2}{b^2} + \frac{z^2}{b^2}$, which represents an axisymmetric galaxy.
The LoS is determined by an inclination angle $\theta$ w.r.t. the individual axis and two offset parameters y and z from the center of the galaxy. While z moves perpendiclar to the axis of inclination, the y parameter moves along one of the identical axes. Hence, y defines a plane parallel to x-z, whose intersection with the ellipsoid defines an ellipse which is intersected by the LoS. We first determine the axes of this ellipse and then obtain the path-length of the intersecting LoS within the ellipse.

The y coordinate on the ellipsoid is determined by the offset parameter, hence the ellipse equation can be rewritten as $1 = \frac{x^2}{a^2 \left( 1 - \frac{y^2}{b^2} \right) }  + \frac{z^2}{b^2\left( 1 - \frac{y^2}{b^2} \right)}$, determining the axes of the ellipse to be $a_{new} = a \sqrt{\left( 1 - \frac{y^2}{b^2} \right)}$ and $b_{new} = b \sqrt{\left( 1 - \frac{y^2}{b^2} \right)}$

By parametrizing the ellipse $r(\Theta)$ and the LoS $L(t|\theta)$, we determine their intersections and calculate the length of the vector between the two solutions, $|| r(\Theta_1) - r(\Theta_2) ||$.

```python
Th, th, t, z, a, b = symbols("Theta theta t z a b")

r = [ b*sin(Th), a*cos(Th) ]
L = [ t*sin(th) + z*cos(th), t*cos(th) - z*sin(th) ]

## find solutions for Th or t, where r = L
```

```python
t1=solve( r[0]-L[0], t)[0]
t2=solve( r[1]-L[1], t)[0]
display(t1,t2)
```

$\displaystyle \frac{b \sin{\left(\Theta \right)} - z \cos{\left(\theta \right)}}{\sin{\left(\theta \right)}}$

$\displaystyle \frac{a \cos{\left(\Theta \right)}}{\cos{\left(\theta \right)}} + z \tan{\left(\theta \right)}$

```python
#solve(t1-t2,Th)
## ... takes aeons ...
```

works faster when reduced by hand to the following, where $a=a ~\sin(\theta)$ and $b=b ~\cos(\theta)$ (use trigonometric identity with z )

```python
sol = solve( a*cos(Th) - b*sin(Th) + z, Th )
```

```python
print("solutions for Theta")
for s in sol:
    display(s.subs(b,b*cos(th)).subs(a,a*sin(th)))
    print(latex(s.subs(b,b*cos(th)).subs(a,a*sin(th))))
```

    solutions for Theta

$\displaystyle - 2 \operatorname{atan}{\left(\frac{b \cos{\left(\theta \right)} - \sqrt{a^{2} \sin^{2}{\left(\theta \right)} + b^{2} \cos^{2}{\left(\theta \right)} - z^{2}}}{a \sin{\left(\theta \right)} - z} \right)}$

    - 2 \operatorname{atan}{\left(\frac{b \cos{\left(\theta \right)} - \sqrt{a^{2} \sin^{2}{\left(\theta \right)} + b^{2} \cos^{2}{\left(\theta \right)} - z^{2}}}{a \sin{\left(\theta \right)} - z} \right)}

$\displaystyle - 2 \operatorname{atan}{\left(\frac{b \cos{\left(\theta \right)} + \sqrt{a^{2} \sin^{2}{\left(\theta \right)} + b^{2} \cos^{2}{\left(\theta \right)} - z^{2}}}{a \sin{\left(\theta \right)} - z} \right)}$

    - 2 \operatorname{atan}{\left(\frac{b \cos{\left(\theta \right)} + \sqrt{a^{2} \sin^{2}{\left(\theta \right)} + b^{2} \cos^{2}{\left(\theta \right)} - z^{2}}}{a \sin{\left(\theta \right)} - z} \right)}

```python
Th1, Th2 = symbols('Theta_1 Theta_2')

rr = [[0,0],[0,0]]
#for i, s in enumerate(sol):
#    ss = s.subs(b,b*cos(th)).subs(a,a*sin(th))
for i, ss in enumerate([Th1,Th2]):
    rr[i][0] = r[0].subs(Th, ss)
    rr[i][1] = r[1].subs(Th, ss)
print( "path length of LoS within ellipse" )
solution = sqrt(  (rr[0][0] - rr[1][0] )**2 + (rr[0][1] - rr[1][1] )**2 )
display(simplify(solution))
print(latex(simplify(solution)))

## unluckily, cannot be simplified further..
```

    path length of LoS within ellipse

$\displaystyle \sqrt{a^{2} \left(\cos{\left(\Theta_{1} \right)} - \cos{\left(\Theta_{2} \right)}\right)^{2} + b^{2} \left(\sin{\left(\Theta_{1} \right)} - \sin{\left(\Theta_{2} \right)}\right)^{2}}$

    \sqrt{a^{2} \left(\cos{\left(\Theta_{1} \right)} - \cos{\left(\Theta_{2} \right)}\right)^{2} + b^{2} \left(\sin{\left(\Theta_{1} \right)} - \sin{\left(\Theta_{2} \right)}\right)^{2}}

 for the final solution, all $a$ and $b$ have to be replaced by the expressions derived in the very beginning,
$a_{new} = a \sqrt{\left( 1 - \frac{y^2}{b^2} \right)}$ and $b_{new} = b \sqrt{\left( 1 - \frac{y^2}{b^2} \right)}$.

```python

```

```python
## test results  !!! z has to be small enough to be within the ellipse, otherwise solution is complex
##               !!! doesnt work if both exact theta=0 and z=0
solution_ = solution.subs(Th1,sol[0].subs(b,b*cos(th)).subs(a,a*sin(th))).subs(Th2,sol[1].subs(b,b*cos(th)).subs(a,a*sin(th)))

solution_.subs(th,0).subs(a,1).subs(b,40).subs(z,0.01)

```

$\displaystyle 1.9999999375$

```python

```

Here we provide the implementation to be used in the Monte-Carlo simulation performed by Luiz F. S. Rodrigues.

```python
def PathLength( inclination=0, z_off=0., y_off=0, major_axis=40, minor_axis=1 ): ## inclination in rad, rest in kpc
    a = minor_axis * np.sqrt(1 - (y_off/major_axis)**2 )
    b = major_axis * np.sqrt(1 - (y_off/major_axis)**2 )
    t = inclination
    theta_1 = -2*np.arctan2( b*np.cos(t) - np.sqrt( (a*np.sin(t))**2 + (b*np.cos(t))**2 - z_off**2  ) , a*np.sin(t) - z_off )
    theta_2 = -2*np.arctan2( b*np.cos(t) + np.sqrt( (a*np.sin(t))**2 + (b*np.cos(t))**2 - z_off**2  ) , a*np.sin(t) - z_off )
    length = np.sqrt( a**2 * ( np.cos(theta_1) - np.cos(theta_2) )**2 + b**2 * (np.sin(theta_1) - np.sin(theta_2))**2 )
    return length

def LoSHostProbability( inclination=0, z_off=0., y_off=0, major_axis=40, minor_axis=1 ): ## inclination in rad, rest in kpc
    return PathLength( inclination=inclination, z_off=z_off, y_off=y_off, major_axis=major_axis, minor_axis=minor_axis ) / (2*major_axis)
```

check results

```python
import matplotlib.pyplot as plt, numpy as np

thetas = np.linspace(0.0001, np.pi, 91)
sols = LoSHostProbability( inclination=thetas, z_off=0.001, y_off=0., major_axis=40, minor_axis=1 )
plt.plot(thetas, sols)
plt.hlines([1./40,1], 0., 3.14)
```

valid. The maximum is reached for $\theta = \frac{\pi}{2}$, i. e. an edge on galaxy, while for $\theta \leftarrow 0$ or $\pi$, i. e. face on galaxy, the minimum is approached fast, according to expectation.

```python

```


## Problem Sheet Question 2a

The general form of the population growth differential equation
\begin{equation} y^{'}=t-y, \ \ (0 \leq t \leq 4) \end{equation}
with the initial condition
\begin{equation}y(0)=1\end{equation}
For N=4
with the analytic (exact) solution
\begin{equation} y= 2e^{-t}+t-1.\end{equation}

# 3-step Adams Bashforth

The 3-step Adams Bashforth difference equation is
\begin{equation}w_{i+1} = w_{i} + \frac{h}{12}(23f(t_i,w_i)-16f(t_{i-1},w_{i-1})+5f(t_{i-2},w_{i-2})) \end{equation}
where
\begin{equation}w_{i+1} = w_{i} + \frac{h}{12}(23(t_i-w_i)-16(t_{i-1}-w_{i-1})+5(t_{i-2}-w_{i-2})) \end{equation}

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

```python
def myfun_ty(t,y):
    return t-y

#PLOTS
def Adams_Bashforth_3step(N,IC):

    x_end=4
    x_start=0
    INTITIAL_CONDITION=IC
    h=x_end/(N)
    N=N+2;
    k_list=np.zeros(N)
    t=np.zeros(N)
    w=np.zeros(N)
    k_mat=np.zeros((4,N-1))
    Analytic_Solution=np.zeros(N)
    k=0
    w[0]=INTITIAL_CONDITION
    Analytic_Solution[0]=INTITIAL_CONDITION
    t[0]=x_start
    t[1]=x_start+1*h
    t[2]=x_start+2*h
    w[1]=2*math.exp(-t[1])+t[1]-1
    w[2]=2*math.exp(-t[2])+t[2]-1
    Analytic_Solution[1]=2*math.exp(-t[1])+t[1]-1
    Analytic_Solution[1+1]=2*math.exp(-t[2])+t[2]-1

    for k in range (2,N-1):
        w[k+1]=w[k]+h/12.0*(23*myfun_ty(t[k],w[k])-16*myfun_ty(t[k-1],w[k-1])+5*myfun_ty(t[k-2],w[k-2]))
        t[k+1]=t[k]+h
        Analytic_Solution[k+1]=2*math.exp(-t[k+1])+t[k+1]-1

    fig = plt.figure(figsize=(10,4))
    # --- left hand plot
    ax = fig.add_subplot(1,3,1)
    plt.plot(t,w,color='black')
    #ax.legend(loc='best')
    plt.title('Numerical Solution h=%s'%(h))

    # --- right hand plot
    ax = fig.add_subplot(1,3,2)
    plt.plot(t,Analytic_Solution,':o',color='blue')
    plt.title('Analytic Solution')

    ax = fig.add_subplot(1,3,3)
    plt.plot(t,Analytic_Solution-w,':o',color='red')
    plt.title('Error')
 # --- title, explanatory text and save

    # --- title, explanatory text and save
    fig.suptitle(r"$y'=t-y$", fontsize=20)
    plt.tight_layout()
    plt.subplots_adjust(top=0.85)
    print(t)
    print(Analytic_Solution)
    print(w)
```

```python
Adams_Bashforth_3step(4,1)
```

```python

```

Chapter 04 - Multistep Methods/4_Problem_Sheet/401a_Problem_Sheet.ipynb