# Optimizer tweaks

```
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```
#export
from exp.nb_08 import *
```

## Imagenette data

We grab the data from the previous notebook.

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=3917)

```
path = datasets.untar_data(datasets.URLs.IMAGENETTE_160)
```

```
tfms = [make_rgb, ResizeFixed(128), to_byte_tensor, to_float_tensor]
bs=128

il = ImageList.from_files(path, tfms=tfms)
sd = SplitData.split_by_func(il, partial(grandparent_splitter, valid_name='val'))
ll = label_by_func(sd, parent_labeler, proc_y=CategoryProcessor())
data = ll.to_databunch(bs, c_in=3, c_out=10, num_workers=4)
```

Then a model:

```
nfs = [32,64,128,256]
```

```
cbfs = [partial(AvgStatsCallback,accuracy), CudaCallback,
        partial(BatchTransformXCallback, norm_imagenette)]
```

This is the baseline of training with vanilla SGD.

```
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```
run.fit(1, learn)
```

    train: [1.7649338173278268, tensor(0.3866, device='cuda:0')]
    valid: [1.69968408203125, tensor(0.4180, device='cuda:0')]

## Refining the optimizer

In PyTorch, the base optimizer in `torch.optim` is just a dictionary that stores the hyper-parameters and references to the parameters of the model we want to train in parameter groups (different groups can have different learning rates/momentum/weight decay... which is what lets us do discriminative learning rates).

It contains a method `step` that will update our parameters with the gradients and a method `zero_grad` to detach and zero the gradients of all our parameters.

We build the equivalent from scratch, only ours will be more flexible. In our implementation, the step function loops over all the parameters to execute the step using stepper functions that we have to provide when initializing the optimizer.

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=4074)

Jeremy uses param groups in order to tweak the hyper params or add annealing, etc. They also call the parameters groups as layer groups.

In linear layer, we have wieght and bias tensor, so each one of these is a parameter tensor. so to Optimize something, we need to know what the parameters tensors are in the model torch, we have model.params.

We want to be able to say the the LR for the last 2 layers in the model should be X - discriminative learning rates.

```
class Optimizer():
    def __init__(self, params, steppers, **defaults):
        # might be a generator
        self.param_groups = list(params)
        # ensure params is a list of lists
        if not isinstance(self.param_groups[0], list): self.param_groups = [self.param_groups]
        self.hypers = [{**defaults} for p in self.param_groups] # for each param group, we have a dict that holds all the hyperparam values
        self.steppers = listify(steppers)

    def grad_params(self):
        return [(p,hyper) for pg,hyper in zip(self.param_groups,self.hypers)
            for p in pg if p.grad is not None]

    def zero_grad(self): # zero params and reset gradient history
        for p,hyper in self.grad_params():
            p.grad.detach_()
            p.grad.zero_()

    def step(self):
        for p,hyper in self.grad_params(): compose(p, self.steppers, **hyper)
```

To do basic SGD, this what a step looks like:

```
#export
def sgd_step(p, lr, **kwargs):
    p.data.add_(-lr, p.grad.data)
    return p
```

```
opt_func = partial(Optimizer, steppers=[sgd_step])
```

Now that we have changed the optimizer, we will need to adjust the callbacks that were using properties from the PyTorch optimizer: in particular the hyper-parameters are in the list of dictionaries `opt.hypers` (PyTorch has everything in the the list of param groups).

```
#export
class Recorder(Callback):
    def begin_fit(self): self.lrs,self.losses = [],[]

    def after_batch(self):
        if not self.in_train: return
        self.lrs.append(self.opt.hypers[-1]['lr'])
        self.losses.append(self.loss.detach().cpu())

    def plot_lr  (self): plt.plot(self.lrs)
    def plot_loss(self): plt.plot(self.losses)

    def plot(self, skip_last=0):
        losses = [o.item() for o in self.losses]
        n = len(losses)-skip_last
        plt.xscale('log')
        plt.plot(self.lrs[:n], losses[:n])

class ParamScheduler(Callback): # Updating ParamScheduler to use our new Optimizer
    _order=1
    def __init__(self, pname, sched_funcs):
        self.pname,self.sched_funcs = pname,listify(sched_funcs)

    def begin_batch(self):
        if not self.in_train: return
        fs = self.sched_funcs
        if len(fs)==1: fs = fs*len(self.opt.param_groups)
        pos = self.n_epochs/self.epochs
        for f,h in zip(fs,self.opt.hypers): h[self.pname] = f(pos)

class LR_Find(Callback):
    _order=1
    def __init__(self, max_iter=100, min_lr=1e-6, max_lr=10):
        self.max_iter,self.min_lr,self.max_lr = max_iter,min_lr,max_lr
        self.best_loss = 1e9

    def begin_batch(self):
        if not self.in_train: return
        pos = self.n_iter/self.max_iter
        lr = self.min_lr * (self.max_lr/self.min_lr) ** pos
        for pg in self.opt.hypers: pg['lr'] = lr

    def after_step(self):
        if self.n_iter>=self.max_iter or self.loss>self.best_loss*10:
            raise CancelTrainException()
        if self.loss < self.best_loss: self.best_loss = self.loss
```

So let's check we didn't break anything and that recorder and param scheduler work properly.

```
sched = combine_scheds([0.3, 0.7], [sched_cos(0.3, 0.6), sched_cos(0.6, 0.2)])
```

```
cbfs = [partial(AvgStatsCallback,accuracy),
        CudaCallback, Recorder,
        partial(ParamScheduler, 'lr', sched)]
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs, opt_func=opt_func)
```

```
%time run.fit(1, learn)
```

    train: [1.7421688685144252, tensor(0.4011, device='cuda:0')]
    valid: [1.589141357421875, tensor(0.4820, device='cuda:0')]
    CPU times: user 4.72 s, sys: 2.29 s, total: 7.01 s
    Wall time: 11.2 s

```
run.recorder.plot_loss()
```

```
run.recorder.plot_lr()
```

## Weight decay

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=4623)

By letting our model learn high parameters, it might fit all the data points in the training set with an over-complex function that has very sharp changes, which will lead to overfitting.

Weight decay comes from the idea of L2 regularization, which consists in adding to your loss function the sum of all the weights squared. Why do that? Because when we compute the gradients, it will add a contribution to them that will encourage the weights to be as small as possible.

Limiting our weights from growing too much is going to hinder the training of the model, but it will yield to a state where it generalizes better. Going back to the theory a little bit, weight decay (or just `wd`) is a parameter that controls that sum of squares we add to our loss:
``` python
loss_with_wd = loss + (wd/2) * (weights**2).sum()
```

In practice though, it would be very inefficient (and maybe numerically unstable) to compute that big sum and add it to the loss. If you remember a little bit of high school math, the derivative of `p**2` with respect to `p` is `2*p`. So adding that big sum to our loss is exactly the same as doing:
``` python
weight.grad += wd * weight
```

for every weight in our model, which in the case of vanilla SGD is equivalent to updating the parameters with:
``` python
weight = weight - lr*(weight.grad + wd*weight)
```

This technique is called "weight decay", as each weight is decayed by a factor `lr * wd`, as it's shown in this last formula.

This only works for standard SGD, as we have seen that with momentum, RMSProp and Adam, the update has some additional formulas around the gradient. In those cases, the formula that comes from L2 regularization:
``` python
weight.grad += wd * weight
```
is different than weight decay
``` python
new_weight = weight - lr * weight.grad - lr * wd * weight
```

Most libraries use the first one, but as it was pointed out in [Decoupled Weight Regularization](https://arxiv.org/pdf/1711.05101.pdf) by Ilya Loshchilov and Frank Hutter, it is better to use the second one with the Adam optimizer, which is why fastai made it its default.

Weight decay is subtracting `lr*wd*weight` from the weights. We need this function to have an attribute `_defaults` so that we are sure there is an hyper-parameter of the same name in our `Optimizer`.

```
#export
def weight_decay(p, lr, wd, **kwargs): # adding weight decay step, just like we added the SGD step. So we are not adding WD optimizers, but adding WD step as an optimizer
    p.data.mul_(1 - lr*wd)
    return p
weight_decay._defaults = dict(wd=0.)
```

L2 regularization is adding `wd*weight` to the gradients.

```
#export
def l2_reg(p, lr, wd, **kwargs):
    p.grad.data.add_(wd, p.data)
    return p
l2_reg._defaults = dict(wd=0.)
```

Let's allow steppers to add to our `defaults` (which are the default values of all the hyper-parameters). This helper function adds in `dest` the key/values it finds while going through `os` and applying `f` when they was no `key` of the same name.

```
#export
def maybe_update(os, dest, f):
    for o in os:
        for k,v in f(o).items():
            if k not in dest: dest[k] = v

def get_defaults(d): return getattr(d,'_defaults',{})
```

This is the same as before, we just take the default values of the steppers when none are provided in the kwargs.

```
#export
class Optimizer():
    def __init__(self, params, steppers, **defaults):
        self.steppers = listify(steppers)
        maybe_update(self.steppers, defaults, get_defaults)
        # might be a generator
        self.param_groups = list(params)
        # ensure params is a list of lists
        if not isinstance(self.param_groups[0], list): self.param_groups = [self.param_groups]
        self.hypers = [{**defaults} for p in self.param_groups]

    def grad_params(self):
        return [(p,hyper) for pg,hyper in zip(self.param_groups,self.hypers)
            for p in pg if p.grad is not None]

    def zero_grad(self):
        for p,hyper in self.grad_params():
            p.grad.detach_()
            p.grad.zero_()

    def step(self):
        for p,hyper in self.grad_params(): compose(p, self.steppers, **hyper)
```

```
#export
sgd_opt = partial(Optimizer, steppers=[weight_decay, sgd_step]) # sgd step
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs, opt_func=sgd_opt)
```

Before trying to train, let's check the behavior works as intended: when we don't provide a value for `wd`, we pull the corresponding default from `weight_decay`.

```
model = learn.model
```

```
opt = sgd_opt(model.parameters(), lr=0.1)
test_eq(opt.hypers[0]['wd'], 0.)
test_eq(opt.hypers[0]['lr'], 0.1)
```

But if we provide a value, it overrides the default.

```
opt = sgd_opt(model.parameters(), lr=0.1, wd=1e-4)
test_eq(opt.hypers[0]['wd'], 1e-4)
test_eq(opt.hypers[0]['lr'], 0.1)
```

Now let's fit.

```
cbfs = [partial(AvgStatsCallback,accuracy), CudaCallback]
```

```
learn,run = get_learn_run(nfs, data, 0.3, conv_layer, cbs=cbfs, opt_func=partial(sgd_opt, wd=0.01))
```

```
run.fit(1, learn)
```

    train: [1.8518677231173415, tensor(0.3537, device='cuda:0')]
    valid: [1.947130615234375, tensor(0.3100, device='cuda:0')]

This is already better than the baseline!

## With momentum

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=4872)

Momentum requires to add some state. We need to save the moving average of the gradients to be able to do the step and store this inside the optimizer state. To do this, we introduce statistics. Statistics are object with two methods:
- `init_state`, that returns the initial state (a tensor of 0. for the moving average of gradients)
- `update`, that updates the state with the new gradient value

We also read the `_defaults` values of those objects, to allow them to provide default values to hyper-parameters.

DP: For momentum, we need to have the state saved.

```
#export
class StatefulOptimizer(Optimizer):
    def __init__(self, params, steppers, stats=None, **defaults):
        self.stats = listify(stats)
        maybe_update(self.stats, defaults, get_defaults)
        super().__init__(params, steppers, **defaults)
        self.state = {} #we need to have the state saved

    def step(self):
        for p,hyper in self.grad_params():
            if p not in self.state:
                #Create a state for p and call all the statistics to initialize it.
                self.state[p] = {}
                maybe_update(self.stats, self.state[p], lambda o: o.init_state(p))
            state = self.state[p]
            for stat in self.stats: state = stat.update(p, state, **hyper)
            compose(p, self.steppers, **state, **hyper)
            self.state[p] = state
```

```
#export
class Stat():
    _defaults = {}
    def init_state(self, p): raise NotImplementedError
    def update(self, p, state, **kwargs): raise NotImplementedError
```

Here is an example of `Stat`:

```
class AverageGrad(Stat):
    _defaults = dict(mom=0.9)

    def init_state(self, p): return {'grad_avg': torch.zeros_like(p.grad.data)}
    def update(self, p, state, mom, **kwargs):
        state['grad_avg'].mul_(mom).add_(p.grad.data)
        return state
```

Then we add the momentum step (instead of using the gradients to perform the step, we use the average).

```
#export
def momentum_step(p, lr, grad_avg, **kwargs):
    p.data.add_(-lr, grad_avg)
    return p
```

```
sgd_mom_opt = partial(StatefulOptimizer, steppers=[momentum_step,weight_decay],
                  stats=AverageGrad(), wd=0.01)
```

```
learn,run = get_learn_run(nfs, data, 0.3, conv_layer, cbs=cbfs, opt_func=sgd_mom_opt)
```

```
run.fit(1, learn)
```

    train: [1.7929072509500543, tensor(0.3878, device='cuda:0')]
    valid: [1.6677803955078125, tensor(0.3960, device='cuda:0')]

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=5115) for discussion about weight decay interaction with batch normalisation

### Momentum experiments

What does momentum do to the gradients exactly? Let's do some plots to find out!

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=5487)

```
x = torch.linspace(-4, 4, 200)
y = torch.randn(200) + 0.3
betas = [0.5, 0.7, 0.9, 0.99]
```

```
def plot_mom(f):
    _,axs = plt.subplots(2,2, figsize=(12,8))
    for beta,ax in zip(betas, axs.flatten()):
        ax.plot(y, linestyle='None', marker='.')
        avg,res = None,[]
        for i,yi in enumerate(y):
            avg,p = f(avg, beta, yi, i)
            res.append(p)
        ax.plot(res, color='red')
        ax.set_title(f'beta={beta}')
```

This is the regular momentum.

```
def mom1(avg, beta, yi, i):
    if avg is None: avg=yi
    res = beta*avg + yi
    return res,res
plot_mom(mom1)
```

As we can see, with a too high value, it may go way too high with no way to change its course.

Another way to smooth noisy data is to do an exponentially weighted moving average. In this case, there is a dampening of (1-beta) in front of the new value, which is less trusted than the current average. We'll define `lin_comb` (*linear combination*) to make this easier (note that in the lesson this was named `ewma`).

```
#export
def lin_comb(v1, v2, beta): return beta*v1 + (1-beta)*v2 # or exponentially weighted moving avg
```

```
def mom2(avg, beta, yi, i):
    if avg is None: avg=yi
    avg = lin_comb(avg, yi, beta)
    return avg, avg
plot_mom(mom2)
```

We can see it gets to a zero-constant when the data is purely random. If the data has a certain shape, it will get that shape (with some delay for high beta).

```
y = 1 - (x/3) ** 2 + torch.randn(200) * 0.1
```

```
y[0]=0.5
```

```
plot_mom(mom2)
```

Debiasing is here to correct the wrong information we may have in the very first batch. The debias term corresponds to the sum of the coefficient in our moving average. At the time step i, our average is:

$\begin{align*}
avg_{i} &= \beta\ avg_{i-1} + (1-\beta)\ v_{i} = \beta\ (\beta\ avg_{i-2} + (1-\beta)\ v_{i-1}) + (1-\beta)\ v_{i} \\
&= \beta^{2}\ avg_{i-2} + (1-\beta)\ \beta\ v_{i-1} + (1-\beta)\ v_{i} \\
&= \beta^{3}\ avg_{i-3} + (1-\beta)\ \beta^{2}\ v_{i-2} + (1-\beta)\ \beta\ v_{i-1} + (1-\beta)\ v_{i} \\
&\vdots \\
&= (1-\beta)\ \beta^{i}\ v_{0} + (1-\beta)\ \beta^{i-1}\ v_{1} + \cdots + (1-\beta)\ \beta^{2}\ v_{i-2} + (1-\beta)\ \beta\  v_{i-1} + (1-\beta)\ v_{i}
\end{align*}$

and so the sum of the coefficients is

$\begin{align*}
S &=(1-\beta)\ \beta^{i} + (1-\beta)\ \beta^{i-1} + \cdots + (1-\beta)\ \beta^{2} + (1-\beta)\ \beta + (1-\beta) \\
&= (\beta^{i} - \beta^{i+1}) + (\beta^{i-1} - \beta^{i}) + \cdots + (\beta^{2} - \beta^{3}) + (\beta - \beta^{2}) + (1-\beta) \\
&= 1 - \beta^{i+1}
\end{align*}$

since all the other terms cancel out each other.

By dividing by this term, we make our moving average a true average (in the sense that all the coefficients we used for the average sum up to 1).

```
def mom3(avg, beta, yi, i):
    if avg is None: avg=0
    avg = lin_comb(avg, yi, beta)
    return avg, avg/(1-beta**(i+1))
plot_mom(mom3)
```

## Adam and friends

ADAM = (dampenend debiased momentum) / (dampened debiased RMS gradient)

In Adam, we use the gradient averages but with dampening (not like in SGD with momentum), so let's add this to the `AverageGrad` class.

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=5889)

```
#export
class AverageGrad(Stat):
    _defaults = dict(mom=0.9)

    def __init__(self, dampening:bool=False): self.dampening=dampening # made dampening optional
    def init_state(self, p): return {'grad_avg': torch.zeros_like(p.grad.data)}
    def update(self, p, state, mom, **kwargs):
        state['mom_damp'] = 1-mom if self.dampening else 1.
        state['grad_avg'].mul_(mom).add_(state['mom_damp'], p.grad.data)
        return state
```

We also need to track the moving average of the gradients squared.

```
#export
class AverageSqrGrad(Stat):
    _defaults = dict(sqr_mom=0.99)

    def __init__(self, dampening:bool=True): self.dampening=dampening
    def init_state(self, p): return {'sqr_avg': torch.zeros_like(p.grad.data)}
    def update(self, p, state, sqr_mom, **kwargs):
        state['sqr_damp'] = 1-sqr_mom if self.dampening else 1.
        state['sqr_avg'].mul_(sqr_mom).addcmul_(state['sqr_damp'], p.grad.data, p.grad.data)
        return state
```

We will also need the number of steps done during training for the debiasing.

```
#export
class StepCount(Stat):
    def init_state(self, p): return {'step': 0}
    def update(self, p, state, **kwargs):
        state['step'] += 1
        return state
```

This helper function computes the debias term. If we dampening, `damp = 1 - mom` and we get the same result as before. If we don't use dampening, (`damp = 1`) we will need to divide by `1 - mom` because that term is missing everywhere.

```
#export
def debias(mom, damp, step): return damp * (1 - mom**step) / (1-mom)
```

Then the Adam step is just the following:

```
#export
def adam_step(p, lr, mom, mom_damp, step, sqr_mom, sqr_damp, grad_avg, sqr_avg, eps, **kwargs):
    debias1 = debias(mom,     mom_damp, step)
    debias2 = debias(sqr_mom, sqr_damp, step)
    p.data.addcdiv_(-lr / debias1, grad_avg, (sqr_avg/debias2).sqrt() + eps)
    return p
adam_step._defaults = dict(eps=1e-5)
```

```
#export
def adam_opt(xtra_step=None, **kwargs):
    return partial(StatefulOptimizer, steppers=[adam_step,weight_decay]+listify(xtra_step),
                   stats=[AverageGrad(dampening=True), AverageSqrGrad(), StepCount()], **kwargs)
```

```
learn,run = get_learn_run(nfs, data, 0.001, conv_layer, cbs=cbfs, opt_func=adam_opt())
```

```
run.fit(3, learn)
```

    train: [1.7606317618659841, tensor(0.3931, device='cuda:0')]
    valid: [1.4439287109375, tensor(0.5320, device='cuda:0')]
    train: [1.246226745482395, tensor(0.5872, device='cuda:0')]
    valid: [1.113287353515625, tensor(0.6260, device='cuda:0')]
    train: [0.9583190188992943, tensor(0.6872, device='cuda:0')]
    valid: [0.9859400634765625, tensor(0.6720, device='cuda:0')]

## LAMB

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=6038)

It's then super easy to implement a new optimizer. This is LAMB from a [very recent paper](https://arxiv.org/pdf/1904.00962.pdf):

$\begin{align}
g_{t}^{l} &= \nabla L(w_{t-1}^{l}, x_{t}) \\
m_{t}^{l} &= \beta_{1} m_{t-1}^{l} + (1-\beta_{1}) g_{t}^{l} \\ EWMA
v_{t}^{l} &= \beta_{2} v_{t-1}^{l} + (1-\beta_{2}) g_{t}^{l} \odot g_{t}^{l} \\ EWMA with grad^2
m_{t}^{l} &= m_{t}^{l} / (1 - \beta_{1}^{t}) \\ debiasing
v_{t}^{l} &= v_{t}^{l} / (1 - \beta_{2}^{t}) \\
r_{1} &= \|w_{t-1}^{l}\|_{2} \\ weight decay
s_{t}^{l} &= \frac{m_{t}^{l}}{\sqrt{v_{t}^{l} + \epsilon}} + \lambda w_{t-1}^{l} \\ ADAM Step
r_{2} &= \| s_{t}^{l} \|_{2} \\
\eta^{l} &= \eta * r_{1}/r_{2} \\
w_{t}^{l} &= w_{t}^{l-1} - \eta_{l} * s_{t}^{l} \\
\end{align}$

```
def lamb_step(p, lr, mom, mom_damp, step, sqr_mom, sqr_damp, grad_avg, sqr_avg, eps, wd, **kwargs):
    debias1 = debias(mom,     mom_damp, step)
    debias2 = debias(sqr_mom, sqr_damp, step)
    r1 = p.data.pow(2).mean().sqrt()
    step = (grad_avg/debias1) / ((sqr_avg/debias2).sqrt()+eps) + wd*p.data
    r2 = step.pow(2).mean().sqrt()
    p.data.add_(-lr * min(r1/r2,10), step)
    return p
lamb_step._defaults = dict(eps=1e-6, wd=0.)
```

```
lamb = partial(StatefulOptimizer, steppers=lamb_step, stats=[AverageGrad(dampening=True), AverageSqrGrad(), StepCount()])
```

```
learn,run = get_learn_run(nfs, data, 0.003, conv_layer, cbs=cbfs, opt_func=lamb)
```

```
run.fit(3, learn)
```

    train: [1.8728660132619823, tensor(0.3419, device='cuda:0')]
    valid: [1.484662841796875, tensor(0.4860, device='cuda:0')]
    train: [1.3553239394291918, tensor(0.5506, device='cuda:0')]
    valid: [1.537447021484375, tensor(0.4840, device='cuda:0')]
    train: [1.0624497101364976, tensor(0.6537, device='cuda:0')]
    valid: [1.11101806640625, tensor(0.6460, device='cuda:0')]

Other recent variants of optimizers:
- [Large Batch Training of Convolutional Networks](https://arxiv.org/abs/1708.03888) (LARS also uses weight statistics, not just gradient statistics. Can you add that to this class?)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://arxiv.org/abs/1804.04235) (Adafactor combines stats over multiple sets of axes)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://arxiv.org/abs/1902.09843)

## Export

```
!python notebook2script.py 09_optimizers.ipynb
```

```

```


# Aplicando Python para análisis de precios: simulación de escenarios futuros de precios

> En la clase anterior vimos como importar datos de activos de la base de datos de Yahoo Finance usando el paquete pandas-datareader. En esta clase, veremos como pronosticar escenarios de evolución de precios, suponiendo que los rendimientos diarios se distribuyen normalmente. Como esta evolución de precios es aleatoria, utilizaremos la simulación montecarlo (hacer muchas simulaciones de escenarios de evolución de precios) para obtener probabilidades de que los precios de cierre estén encima de un valor umbral y tomar decisiones con base en estas probabilidades.

**Referencias:**
- http://pandas.pydata.org/
- http://www.learndatasci.com/python-finance-part-yahoo-finance-api-pandas-matplotlib/

## 1. Recordemos como descargar datos...

Antes que nada, para poder hacer simular escenarios de predicción de precios, vamos a recordar lo que hicimos en la clase pasada de descargar los datos de Yahoo Finance, utilizando el paquete `data` de la librería `pandas_datareader`.

Esta vez, utilizaremos los datos de precios de cierre ajustados de activos de la compañía Apple en el año 2016 para nuestra aplicación.

```python
# Importamos librerías
from pandas_datareader import data as web
import pandas as pd
import numpy as np
```

```python
# Función para descargar precios de cierre ajustados de varios activos a la vez:
def get_closes(tickers, start_date=None, end_date=None):
    # Fecha inicio por defecto (start_date='2010-01-01') y fecha fin por defecto (end_date=today)
    # Frecuencia de muestreo por defecto (freq='d')
    closes = web.DataReader(tickers, start=start_date, end=end_date, data_source='yahoo')['Adj Close']
    closes = closes.sort_index()
    return closes
```

```python
# Descargamos datos...
# Instrumento: Apple
name = ['AAPL']
# Fechas de interés (inicio y fin): 2010-2016
start = '2010-01-01'
end = '2016-12-31'
# Función DataReader
closes = get_closes(tickers=name,
                    start_date=start,
                    end_date=end)
```

```python
closes.head()
```

      Symbols
      AAPL

      Date

      2010-01-04
      6.583586

      2010-01-05
      6.594968

      2010-01-06
      6.490066

      2010-01-07
      6.478067

      2010-01-08
      6.521136

```python
# Graficamos
closes.plot();
```

## 2. Simulación de rendimientos diarios

Recordemos que los precios diarios de cierre ajustados no son un proceso estocástico estacionario, pero los rendimientos diarios si lo son. Por tanto calculamos los rendimientos a partir de los precios de cierre, obtenemos sus propiedades estadísticas muestrales y proyectamos los rendimientos. Luego, obtenemos la proyección de los precios.

Para una sucesión de precios $\{S_t\}_{t=0}^{n}$, el rendimiento simple $R_t$ se define como el el cambio porcentual

$$
R_t=\frac{S_t-S_{t-1}}{S_{t-1}}\approx \ln\left(\frac{S_t}{S_{t-1}}\right)=r_t.
$$
para $t=1,\ldots,n$.

Para el ejemplo en curso, ¿cómo calcular esto?

Además, supusimos que los rendimientos diarios eran una variable aleatoria con distribución normal (que se caracteriza con su media y varianza). Por tanto obtenemos la media y desviación estandar muestrales. Hagamos una función que retorne lo anterior.

```python
# Calcular rendimientos diarios y graficarlos
ret = closes.pct_change().dropna()
```

```python
ret.head()
```

      Symbols
      AAPL

      Date

      2010-01-05
      0.001729

      2010-01-06
      -0.015906

      2010-01-07
      -0.001849

      2010-01-08
      0.006648

      2010-01-11
      -0.008821

```python
len(ret)
```

    1761

```python
ret.plot();
```

Entonces, suponemos que la diferencia logaritmica de los precios (rendimientos diarios) tiene una distribución normal.

¿Cómo se caracteriza una [distribución normal](https://es.wikipedia.org/wiki/Distribuci%C3%B3n_normal)?

```python
# Calculamos media y desviación estándar
mu = ret.mean()['AAPL']
sigma = ret.std()['AAPL']
mu, sigma
```

    (0.000946043424708954, 0.016507169383872627)

Habiendo caracterizado los rendimientos diarios como una variable aleatoria normal con la media y la varianza muestral obtenida de los datos del 2016, podemos generar números aleatorios con estas características para simular el comportamiento de los precios de cierre de las acciones en el 2017 (hay un supuesto de que las cosas no cambiarán fundamentalmente).

Sin embargo, cada simulación que hagamos nos conducirá a distintos resultados (los precios siguen evolucionando aleatoriamente). Entonces, lo que haremos es simular varios escenarios para así ver alguna tendencia y tomar decisiones.

Hagamos una una función que simule varios escenarios de rendimientos diarios rendimientos diarios y que devuelva un dataframe con esta simulación.

```python
# Ayuda en la función np.random.randn
help(np.random.randn)
```

    Help on built-in function randn:

    randn(...) method of numpy.random.mtrand.RandomState instance
        randn(d0, d1, ..., dn)

        Return a sample (or samples) from the "standard normal" distribution.

        .. note::
            This is a convenience function for users porting code from Matlab,
            and wraps `standard_normal`. That function takes a
            tuple to specify the size of the output, which is consistent with
            other NumPy functions like `numpy.zeros` and `numpy.ones`.

        .. note::
            New code should use the ``standard_normal`` method of a ``default_rng()``
            instance instead; see `random-quick-start`.

        If positive int_like arguments are provided, `randn` generates an array
        of shape ``(d0, d1, ..., dn)``, filled
        with random floats sampled from a univariate "normal" (Gaussian)
        distribution of mean 0 and variance 1. A single float randomly sampled
        from the distribution is returned if no argument is provided.

        Parameters
        ----------
        d0, d1, ..., dn : int, optional
            The dimensions of the returned array, must be non-negative.
            If no argument is given a single Python float is returned.

        Returns
        -------
        Z : ndarray or float
            A ``(d0, d1, ..., dn)``-shaped array of floating-point samples from
            the standard normal distribution, or a single such float if
            no parameters were supplied.

        See Also
        --------
        standard_normal : Similar, but takes a tuple as its argument.
        normal : Also accepts mu and sigma arguments.
        Generator.standard_normal: which should be used for new code.

        Notes
        -----
        For random samples from :math:`N(\mu, \sigma^2)`, use:

        ``sigma * np.random.randn(...) + mu``

        Examples
        --------
        >>> np.random.randn()
        2.1923875335537315  # random

        Two-by-four array of samples from N(3, 6.25):

        >>> 3 + 2.5 * np.random.randn(2, 4)
        array([[-4.49401501,  4.00950034, -1.81814867,  7.29718677],   # random
               [ 0.39924804,  4.68456316,  4.99394529,  4.84057254]])  # random

```python
# Función que simula varios escenarios de rendimientos diarios
def ret_sim(mu, sigma, n_days, n_esc, start):
    dates = pd.date_range(start=start, periods=n_days)
    return pd.DataFrame(data=sigma * np.random.randn(n_days, n_esc) + mu, index=dates)
```

```python
# Simulamos 100 escenarios para todo el 2017
n_esc = 1000000
n_days = 255
sim_ret = ret_sim(mu, sigma, n_days, n_esc, start='2017-01-01')
sim_ret.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      999990
      999991
      999992
      999993
      999994
      999995
      999996
      999997
      999998
      999999

      2017-01-01
      0.036537
      -0.005990
      -0.016395
      0.007986
      0.005297
      0.019306
      -0.013699
      -0.011789
      0.024795
      0.004934
      ...
      -0.020908
      -0.015728
      0.004612
      0.010736
      -0.004749
      0.009232
      0.024915
      0.023263
      -0.002839
      0.011655

      2017-01-02
      -0.001527
      -0.003874
      -0.014072
      0.013740
      -0.006203
      -0.026455
      0.008172
      -0.004785
      0.001172
      -0.007976
      ...
      0.010559
      0.017211
      0.000909
      0.009722
      0.010219
      0.013265
      0.009697
      -0.007922
      -0.015835
      0.012817

      2017-01-03
      -0.012792
      0.000097
      0.009589
      0.037698
      0.000833
      0.024115
      0.003037
      0.003182
      -0.005037
      0.007961
      ...
      0.003544
      -0.037546
      -0.005589
      0.001489
      -0.006732
      0.012166
      0.009932
      0.033082
      0.018128
      -0.012980

      2017-01-04
      0.033052
      -0.001008
      0.005649
      -0.012305
      -0.004740
      0.012646
      -0.001396
      0.049358
      -0.003362
      -0.017953
      ...
      -0.033124
      -0.003309
      0.008640
      -0.008735
      -0.001633
      0.005835
      0.012575
      0.005049
      -0.028284
      0.028532

      2017-01-05
      -0.034045
      0.002133
      0.000679
      -0.002243
      0.002306
      0.007379
      0.014718
      -0.029282
      -0.019564
      -0.030599
      ...
      0.003357
      0.000369
      -0.003881
      0.029838
      0.001091
      0.003877
      -0.032864
      0.033261
      0.021127
      0.010730

5 rows × 1000000 columns

```python
# Mostrar
#sim_ret.plot(legend=False);
```

```python
ret.plot();
```

## 3. Proyección de precios de cierre

Por tanto, para calcular los precios, tenemos:

$$\begin{align}
p_i&=p_{i-1}(R_i+1)\\
p_{i+1}&=p_i(R_{i+1}+1)=p_{i-1}(R_i+1)(R_{i+1}+1)\\
&\vdots\\
p_{i+k}&=p_{i-1}(R_i+1)\cdots(R_{i+k}+1).
\end{align}$$

Si hacemos $i=0$ en la última ecuación, tenemos que $p_{k}=p_{-1}\exp(r_0+\cdots+r_{k})$, donde $p_{-1}$ es el último precio reportado en el 2016.

Con los rendimientos, calculamos los precios de cierre...

```python
closes.tail()
```

      Symbols
      AAPL

      Date

      2016-12-23
      27.547411

      2016-12-27
      27.722363

      2016-12-28
      27.604153

      2016-12-29
      27.597061

      2016-12-30
      27.381918

```python
closes.loc['2016-12-27']
```

    Symbols
    AAPL    27.722363
    Name: 2016-12-27 00:00:00, dtype: float64

```python
closes.iloc[-1]
```

    Symbols
    AAPL    27.381918
    Name: 2016-12-30 00:00:00, dtype: float64

```python
# Obtenemos los precios
ultimo_precio = closes.iloc[-1]['AAPL']
sim_closes = ultimo_precio * (1 + sim_ret).cumprod()
```

```python
sim_closes.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      999990
      999991
      999992
      999993
      999994
      999995
      999996
      999997
      999998
      999999

      2017-01-01
      28.382384
      27.217895
      26.933004
      27.600594
      27.526958
      27.910549
      27.006819
      27.059119
      28.060862
      27.517028
      ...
      26.809413
      26.951262
      27.508201
      27.675891
      27.251893
      27.634709
      28.064130
      28.018913
      27.304192
      27.701042

      2017-01-02
      28.339045
      27.112455
      26.554004
      27.979823
      27.356207
      27.172175
      27.227532
      26.929651
      28.093749
      27.297552
      ...
      27.092497
      27.415108
      27.533200
      27.944960
      27.530372
      28.001297
      28.336266
      27.796956
      26.871822
      28.056081

      2017-01-03
      27.976543
      27.115079
      26.808640
      29.034598
      27.378999
      27.827434
      27.310231
      27.015347
      27.952242
      27.514878
      ...
      27.188508
      26.385767
      27.379324
      27.986575
      27.345025
      28.341948
      28.617694
      28.716539
      27.358968
      27.691923

      2017-01-04
      28.901213
      27.087739
      26.960080
      28.677334
      27.249225
      28.179344
      27.272103
      28.348781
      27.858264
      27.020897
      ...
      26.287916
      26.298463
      27.615876
      27.742125
      27.300378
      28.507322
      28.977550
      28.861519
      26.585149
      28.482022

      2017-01-05
      27.917257
      27.145517
      26.978377
      28.613010
      27.312053
      28.387288
      27.673498
      27.518679
      27.313249
      26.194079
      ...
      26.376166
      26.308179
      27.508694
      28.569882
      27.330149
      28.617833
      28.025242
      29.821493
      27.146805
      28.787621

5 rows × 1000000 columns

Concatenamos y graficamos...

```python
# Concatenar y graficar
#sim_closes_cat = pd.concat([closes.iloc[-255:], sim_closes])
#sim_closes_cat.plot(legend=False);
```

## 4. Probabilidad Precio-Umbral

Ya que tenemos muchos escenarios de precios proyectados, podemos ver varias cosas. Por ejemplo, ¿cuál es la probabilidad de que el precio de cierre sobrepase algún valor umbral en algún momento?

```python
# Umbral de 10% del ultimo precio
umbral = 1.15 * ultimo_precio
# Fechas
dates = sim_ret.index
# DataFrame de Strikes
umbrales = pd.DataFrame(index=dates, data=umbral * np.ones(len(dates)))
# Concatenar y graficar
#sim_closes_cat_strikes = pd.concat([sim_closes_cat.T, umbrales.T]).T
#sim_closes_cat_strikes.plot(legend=False);
```

```python
# Comparar cada escenario en cada fecha
conteo = sim_closes > umbral
# Sumamos para cada fecha y dividimos entre el número de escenarios
prob_est = conteo.sum(axis=1) / n_esc
# Gráfico de probabilidad
prob_est.plot();
```

```python
closes_AAPL_2017 = get_closes(tickers=['AAPL'],
                              start_date='2017-01-01',
                              end_date='2017-12-31')
```

```python
umbral
```

    31.48920564651489

```python
closes_AAPL_2017.plot();
```

___
Entonces, ya aprendimos a bajar datos con pandas-datareader. En específico, a partir de los precios de cierre ajustados obtuvimos los rendimientos diarios.

Suponiendo que los rendimientos diarios son un proceso estocástico estacionario de distribución normal, pudimos caracaterizarlo y proyectar varios escenarios de evolución de los precios (montecarlo).

Con estas proyecciones pudimos calcular probabilidades de sobrepasar cierto precio umbral: toma de decisiones.

Created with Jupyter by Esteban Jiménez Rodríguez.


# Illustration of PKPD model for Erlotinib and Gefitinib

Following [[1]]

- [[1]] Eigenmann et. al., Combining Nonclinical Experiments with Translational PKPD Modeling to Differentiate
Erlotinib and Gefitinib

## PK model

The PK model according to [[1]] is defined as

\begin{align}
\frac{\text{d}A_d}{\text{d}t} &= -k_{a}A_d\\
\frac{\text{d}A_p}{\text{d}t} &= k_{a} A_d - k_e A_p\\
C_p &= \frac{A_p}{V},
\end{align}

where $A_i$ is the amount of drug in the dose compartment $i=d$ and in the plasma $i=p$. $C_p$ is the plasma, $V$ the volume of distribution in the plasma, $k_a$ the absorption rate and $k_e$ the elimination rate.

### Computing time-concentration profiles:

```python
import myokit
import numpy as np

# build model
model = myokit.Model()

# add dosing compartment and central compartment
dose_comp = model.add_component('dose')
central_comp = model.add_component('plasma')

# add relevant variables and constants to dose compartment
# (the dose and regimen will be set with an additional dose_rate
# and regimen variable)
amount_d = dose_comp.add_variable('amount')
k_a = dose_comp.add_variable('k_a')
dose_rate = dose_comp.add_variable('dose_rate')
regimen = dose_comp.add_variable('regimen')

# add relevant variables and constants to central compartment
amount_p = central_comp.add_variable('amount')
volume = central_comp.add_variable('volume')
k_e = central_comp.add_variable('k_e')
conc_p = central_comp.add_variable('drug_concentration')  # to be able to use this with pyqt PKPDApp

# bind time
time = central_comp.add_variable('time')
time.set_binding('time')

# bind regimen to protocol pacer
regimen.set_binding('pace')

# set intial values (some default values)
time.set_rhs(0)

amount_d.set_rhs(0)
k_a.set_rhs(0)
dose_rate.set_rhs(0)
regimen.set_rhs(0)

amount_p.set_rhs(0)
volume.set_rhs(1)
k_e.set_rhs(0)
conc_p.set_rhs(0)

# set units
time.set_unit('day')  # time in days

amount_d.set_unit('mg')  # miligram
k_a.set_unit('1 / day')  # 1 / day
dose_rate.set_unit('mg / day')  # miligram / day
regimen.set_unit(myokit.units.dimensionless)  # dimensionless

amount_p.set_unit('mg')  # miligram
volume.set_unit('L')  # liter
k_e.set_unit('1 / day')  # 1 / day
conc_p.set_unit('mg / L')  # miligram / liter

# set preferred representation of units
# time days
unit = myokit.parse_unit('day')
myokit.Unit.register_preferred_representation('day', unit)
# rates in 1 / day
unit = myokit.parse_unit('1/day')
myokit.Unit.register_preferred_representation('1/day', unit)
# amount in mg
unit = myokit.parse_unit('mg')
myokit.Unit.register_preferred_representation('mg', unit)
# dose rate in mg / day
unit = myokit.parse_unit('mg/day')
myokit.Unit.register_preferred_representation('mg/day', unit)
# concentration mg / L
unit = myokit.parse_unit('mg/L')
myokit.Unit.register_preferred_representation('mg/L', unit)

# set rhs of state variables (dose_rate * regimen - k_a * A_d)
amount_d.promote()
amount_d.set_rhs(
    myokit.Minus(
        myokit.Multiply(myokit.Name(dose_rate), myokit.Name(regimen)),
        myokit.Multiply(myokit.Name(k_a), myokit.Name(amount_d))
    )
)

amount_p.promote()
amount_p.set_rhs(
    myokit.Minus(
        myokit.Multiply(myokit.Name(k_a), myokit.Name(amount_d)),
        myokit.Multiply(myokit.Name(k_e), myokit.Name(amount_p))
    )
)

# set algebraic relation between drug and concentration
conc_p.set_rhs(myokit.Divide(myokit.Name(amount_p), myokit.Name(volume)))

print(model.code(line_numbers=True))
model.check_units(mode=myokit.UNIT_TOLERANT)
model.validate()
```

    1 [[model]]
     2 # Initial values
     3 dose.amount   = 0
     4 plasma.amount = 0
     5
     6 [dose]
     7 dot(amount) = dose_rate * regimen - k_a * amount
     8     in [mg]
     9 dose_rate = 0
    10     in [mg/day]
    11 k_a = 0
    12     in [1/day]
    13 regimen = 0 bind pace
    14     in [1]
    15
    16 [plasma]
    17 dot(amount) = dose.k_a * dose.amount - k_e * amount
    18     in [mg]
    19 drug_concentration = amount / volume
    20     in [mg/L]
    21 k_e = 0
    22     in [1/day]
    23 time = 0 bind time
    24     in [day]
    25 volume = 1
    26     in [L]

### Run simulation for given parameters, dose and regimen

```python
# set parameter values for Erlotinib (according to [1])
k_a.set_rhs(55.0)  # [1 / d]

volume.set_rhs(0.127)  # 0.127 [L]
k_e.set_rhs(7.56)  # 7.56 [1 / d]

# define dosing regimen
duration = 0.001  # [1 / d] how long does it take for dose to be in dosing compartment?
dose_rate.set_rhs(25 * 0.2 / duration)  # 100 mg / kg / d let's estimate weight of mouse 0.2 kg
dosing_regimen = myokit.Protocol()
dosing_regimen.schedule(level=1, start=0, duration=duration)

# run simulation
simulation = myokit.Simulation(model, dosing_regimen)
result = simulation.run(1)
```

```python
import matplotlib.pyplot as plt

time_key = 'plasma.time'
conc_key = 'plasma.drug_concentration'

time = result[time_key]
drug_conc = result[conc_key]

# plot result
fig = plt.figure(figsize=(12, 6))

# semilog plot
plt.semilogy(time, drug_conc)

# set x and y label
unit = model.get(time_key).unit()
plt.xlabel('time in %s' % str(unit))
unit = model.get(conc_key).unit()
plt.ylabel('plasma conc. in %s' % str(unit))

# set y lim
plt.ylim([0.01,100])

plt.show()
```

## Target Modulation PKPD (TM/PKPD)

The above PK model is complemented by a target modulation model to model the pErk profiles [[1]].

\begin{align}
C_{\text{pErk}} = C_{\text{pErk},0}\left( 1 - \frac{I_{\text{max}}C_p}{C_{I50}+C_p}\right) ,
\end{align}

where $C_{\text{pErk}}$ is the pErk concentration in the plasma, $C_{\text{pErk},0}$ the pErk base level, $I_{\text{max}}$ the maximal inhibitory effect in units of $C_{\text{pErk},0}$, and $C_{I50}$ the concentration where half of the inhibitory effect is achieved.

### Time-concentration profile of plasma drud and plasma pErk

```python
import myokit
import numpy as np

# build model
model = myokit.Model()

# add dosing compartment and central compartment
dose_comp = model.add_component('dose')
central_comp = model.add_component('plasma')

# add relevant variables and constants to dose compartment
# (the dose and regimen will be set with an additional dose_rate
# and regimen variable)
amount_d = dose_comp.add_variable('amount')
k_a = dose_comp.add_variable('k_a')
dose_rate = dose_comp.add_variable('dose_rate')
regimen = dose_comp.add_variable('regimen')

# add PK variables to central compartment
amount_p = central_comp.add_variable('amount')
volume = central_comp.add_variable('volume')
k_e = central_comp.add_variable('k_e')
conc_p = central_comp.add_variable('drug_concentration')  # to be able to use this with pyqt PKPDApp

# add PD variables to central compartment
conc_perk = central_comp.add_variable('pErk_concentration')
initial_perk = central_comp.add_variable('init_pErk')
i_max = central_comp.add_variable('i_max')
ic_50 = central_comp.add_variable('ic_50')

# bind time
time = central_comp.add_variable('time')
time.set_binding('time')

# bind regimen to protocol pacer
regimen.set_binding('pace')

# set intial values (some default values)
time.set_rhs(0)

amount_d.set_rhs(0)
k_a.set_rhs(0)
dose_rate.set_rhs(0)
regimen.set_rhs(0)

amount_p.set_rhs(0)
volume.set_rhs(1)
k_e.set_rhs(0)
conc_p.set_rhs(0)

conc_perk.set_rhs(0)
initial_perk.set_rhs(0)
i_max.set_rhs(0)
ic_50.set_rhs(1)

# set units
time.set_unit('day')  # time in days

amount_d.set_unit('mg')  # miligram
k_a.set_unit('1 / day')  # 1 / day
dose_rate.set_unit('mg / day')  # miligram / day
regimen.set_unit(myokit.units.dimensionless)  # dimensionless

amount_p.set_unit('mg')  # miligram
volume.set_unit('L')  # liter
k_e.set_unit('1 / day')  # 1 / day
conc_p.set_unit('mg / L')  # miligram / liter

conc_perk.set_unit('pg / mL')  # picogram
initial_perk.set_unit('pg / mL')  # picogram
i_max.set_unit(myokit.units.dimensionless)  # dimensionless
ic_50.set_unit('mg / L')  # miligram / liter

# set preferred representation of units
# time days
unit = myokit.parse_unit('day')
myokit.Unit.register_preferred_representation('day', unit)
# rates in 1 / day
unit = myokit.parse_unit('1/day')
myokit.Unit.register_preferred_representation('1/day', unit)
# amount in mg
unit = myokit.parse_unit('mg')
myokit.Unit.register_preferred_representation('mg', unit)
# dose rate in mg / day
unit = myokit.parse_unit('mg/day')
myokit.Unit.register_preferred_representation('mg/day', unit)
# drug concentration mg / L
unit = myokit.parse_unit('mg/L')
myokit.Unit.register_preferred_representation('mg/L', unit)
# pErk concentration mg / L
unit = myokit.parse_unit('pg/mL')
myokit.Unit.register_preferred_representation('pg/mL', unit)

# set rhs of state variables (dose_rate * regimen - k_a * A_d)
amount_d.promote()
amount_d.set_rhs(
    myokit.Minus(
        myokit.Multiply(myokit.Name(dose_rate), myokit.Name(regimen)),
        myokit.Multiply(myokit.Name(k_a), myokit.Name(amount_d))
    )
)

amount_p.promote()
amount_p.set_rhs(
    myokit.Minus(
        myokit.Multiply(myokit.Name(k_a), myokit.Name(amount_d)),
        myokit.Multiply(myokit.Name(k_e), myokit.Name(amount_p))
    )
)

# set algebraic equation
# drug and concentration
conc_p.set_rhs(myokit.Divide(myokit.Name(amount_p), myokit.Name(volume)))
# pErk concentration
conc_perk.set_rhs(myokit.Multiply(
    myokit.Name(initial_perk),
    myokit.Minus(
        myokit.Number(1),
        myokit.Divide(
            myokit.Multiply(myokit.Name(i_max), myokit.Name(conc_p)),
            myokit.Plus(myokit.Name(ic_50), myokit.Name(conc_p))
        )
    )
))

print(model.code(line_numbers=True))
model.check_units(mode=myokit.UNIT_TOLERANT)
model.validate()
```

    1 [[model]]
     2 # Initial values
     3 dose.amount   = 0
     4 plasma.amount = 0
     5
     6 [dose]
     7 dot(amount) = dose_rate * regimen - k_a * amount
     8     in [mg]
     9 dose_rate = 0
    10     in [mg/day]
    11 k_a = 0
    12     in [1/day]
    13 regimen = 0 bind pace
    14     in [1]
    15
    16 [plasma]
    17 dot(amount) = dose.k_a * dose.amount - k_e * amount
    18     in [mg]
    19 drug_concentration = amount / volume
    20     in [mg/L]
    21 i_max = 0
    22     in [1]
    23 ic_50 = 1
    24     in [mg/L]
    25 init_pErk = 0
    26     in [pg/mL]
    27 k_e = 0
    28     in [1/day]
    29 pErk_concentration = init_pErk * (1 - i_max * drug_concentration / (ic_50 + drug_concentration))
    30     in [pg/mL]
    31 time = 0 bind time
    32     in [day]
    33 volume = 1
    34     in [L]

### Run simulation for given parameters, dose and regimen

```python
# set parameter values for Erlotinib (according to [1] Sup. Table S2)
k_a.set_rhs(55.0)  # [1 / d]

volume.set_rhs(0.326)  # [L]
k_e.set_rhs(4.47)  # [1 / d]

initial_perk.set_rhs(668)
i_max.set_rhs(0.72)
ic_50.set_rhs(0.513)

# define dosing regimen
duration = 0.001  # [1 / d] how long does it take for dose to be in dosing compartment?
dose_rate.set_rhs(25 * 0.2 / duration)  # 100 mg / kg / d let's estimate weight of mouse 0.2 kg
dosing_regimen = myokit.Protocol()
dosing_regimen.schedule(level=1, start=0, duration=duration)

# run simulation
simulation = myokit.Simulation(model, dosing_regimen)
result = simulation.run(1)
```

```python
import matplotlib.pyplot as plt

time_key = 'plasma.time'
conc_key = 'plasma.drug_concentration'
pErk_key = 'plasma.pErk_concentration'

time = result[time_key]
drug_conc = result[conc_key]
pErk_conc = result[pErk_key]

# plot drug time-concentration curve
fig = plt.figure(figsize=(12, 6))

# semilog plot
plt.semilogy(time, drug_conc)

# set x and y label
unit = model.get(time_key).unit()
plt.xlabel('time in %s' % str(unit))
unit = model.get(conc_key).unit()
plt.ylabel('plasma conc. in %s' % str(unit))

# set y lim
plt.ylim([0.01,100])

plt.show()

# plot pErk time-concentration curve
fig = plt.figure(figsize=(12, 6))

# semilog plot
plt.plot(time, pErk_conc)

# set x and y label
unit = model.get(time_key).unit()
plt.xlabel('time in %s' % str(unit))
unit = model.get(pErk_key).unit()
plt.ylabel('pErk conc. in %s' % str(unit))

plt.show()
```

## Tumor Growth Inhibition PKPD (TGI/PKPD)

The above PK model is complemented by a tumor growth inhibition model to model the tumor volume over time [[1]].

\begin{align}
\frac{\text{d}V_T}{\text{d}t} = \frac{2\lambda _0\lambda _1 V_T}{2\lambda _0 V_T + \lambda _1} - \kappa C_pV_T,
\end{align}

where $V_T$ is the volume of the tumor, $\lambda _0 $ the exponential and $\lambda _1$ the linear growth rate. $\kappa $ represents the potency of the compound.

### Time-concentration profile of plasma drug and tumor volume

```python
import myokit
import numpy as np

# build model
model = myokit.Model()

# add dosing compartment and central compartment
dose_comp = model.add_component('dose')
central_comp = model.add_component('plasma')

# add relevant variables and constants to dose compartment
# (the dose and regimen will be set with an additional dose_rate
# and regimen variable)
amount_d = dose_comp.add_variable('amount')
k_a = dose_comp.add_variable('k_a')
dose_rate = dose_comp.add_variable('dose_rate')
regimen = dose_comp.add_variable('regimen')

# add PK variables to central compartment
amount_p = central_comp.add_variable('amount')
volume_p = central_comp.add_variable('volume_plasma')
k_e = central_comp.add_variable('k_e')
conc_p = central_comp.add_variable('drug_concentration')  # to be able to use this with pyqt PKPDApp

# add PD variables to central compartment
volume_T = central_comp.add_variable('volume_tumor')
lambda_0 = central_comp.add_variable('lambda_0')
lambda_1 = central_comp.add_variable('lambda_1')
kappa = central_comp.add_variable('kappa')

# bind time
time = central_comp.add_variable('time')
time.set_binding('time')

# bind regimen to protocol pacer
regimen.set_binding('pace')

# set intial values (some default values)
time.set_rhs(0)

amount_d.set_rhs(0)
k_a.set_rhs(0)
dose_rate.set_rhs(0)
regimen.set_rhs(0)

amount_p.set_rhs(0)
volume_p.set_rhs(1)
k_e.set_rhs(0)
conc_p.set_rhs(0)

volume_T.set_rhs(0)
lambda_0.set_rhs(0)
lambda_1.set_rhs(1)  # avoid dividing by zero
kappa.set_rhs(0)

# set units
time.set_unit('day')  # time in days

amount_d.set_unit('mg')  # miligram
k_a.set_unit('1 / day')  # 1 / day
dose_rate.set_unit('mg / day')  # miligram / day
regimen.set_unit(myokit.units.dimensionless)  # dimensionless

amount_p.set_unit('mg')  # miligram
volume_p.set_unit('L')  # liter
k_e.set_unit('1 / day')  # 1 / day
conc_p.set_unit('mg / L')  # miligram / liter

volume_T.set_unit('mm^3')  # milimeter cubed
lambda_0.set_unit('1 / day')  # per day
lambda_1.set_unit('mm^3 / day')  # milimiter cubed per day
kappa.set_unit('L / mg / day')  # in reference L / ug / day,
# so values here are 10^3 larger.

# set preferred representation of units
# time days
unit = myokit.parse_unit('day')
myokit.Unit.register_preferred_representation('day', unit)
# rates in 1 / day
unit = myokit.parse_unit('1/day')
myokit.Unit.register_preferred_representation('1/day', unit)
# amount in mg
unit = myokit.parse_unit('mg')
myokit.Unit.register_preferred_representation('mg', unit)
# dose rate in mg / day
unit = myokit.parse_unit('mg/day')
myokit.Unit.register_preferred_representation('mg/day', unit)
# drug concentration mg / L
unit = myokit.parse_unit('mg/L')
myokit.Unit.register_preferred_representation('mg/L', unit)
# tumor volume
unit = myokit.parse_unit('mm^3')
myokit.Unit.register_preferred_representation('mm^3', unit)
# linear growth
unit = myokit.parse_unit('mm^3/day')
myokit.Unit.register_preferred_representation('mm^3/day', unit)
# potency
unit = myokit.parse_unit('L/mg/day')
myokit.Unit.register_preferred_representation('L/mg/day', unit)

# set rhs of state variables (dose_rate * regimen - k_a * A_d)
amount_d.promote()
amount_d.set_rhs(
    myokit.Minus(
        myokit.Multiply(myokit.Name(dose_rate), myokit.Name(regimen)),
        myokit.Multiply(myokit.Name(k_a), myokit.Name(amount_d))
    )
)

amount_p.promote()
amount_p.set_rhs(
    myokit.Minus(
        myokit.Multiply(myokit.Name(k_a), myokit.Name(amount_d)),
        myokit.Multiply(myokit.Name(k_e), myokit.Name(amount_p))
    )
)

volume_T.promote()
volume_T.set_rhs(
    myokit.Minus(
        myokit.Divide(
            myokit.Multiply(
                myokit.Number(2),
                myokit.Multiply(
                    myokit.Name(lambda_0),
                    myokit.Multiply(
                        myokit.Name(lambda_1),
                        myokit.Name(volume_T)
                    )
                )
            ),
            myokit.Plus(
                myokit.Multiply(
                    myokit.Number(2),
                    myokit.Multiply(
                        myokit.Name(lambda_0),
                        myokit.Name(volume_T)
                    )
                ),
                myokit.Name(lambda_1)
            )
        ),
        myokit.Multiply(
            myokit.Name(kappa),
            myokit.Multiply(
                myokit.Name(conc_p),
                myokit.Name(volume_T)
            )
        )
    )
)

# set algebraic equation
# drug and concentration
conc_p.set_rhs(myokit.Divide(myokit.Name(amount_p), myokit.Name(volume_p)))

print(model.code(line_numbers=True))
model.check_units(mode=myokit.UNIT_TOLERANT)
model.validate()
```

    1 [[model]]
     2 # Initial values
     3 dose.amount         = 0
     4 plasma.amount       = 0
     5 plasma.volume_tumor = 0
     6
     7 [dose]
     8 dot(amount) = dose_rate * regimen - k_a * amount
     9     in [mg]
    10 dose_rate = 0
    11     in [mg/day]
    12 k_a = 0
    13     in [1/day]
    14 regimen = 0 bind pace
    15     in [1]
    16
    17 [plasma]
    18 dot(amount) = dose.k_a * dose.amount - k_e * amount
    19     in [mg]
    20 drug_concentration = amount / volume_plasma
    21     in [mg/L]
    22 k_e = 0
    23     in [1/day]
    24 kappa = 0
    25     in [L/mg/day]
    26 lambda_0 = 0
    27     in [1/day]
    28 lambda_1 = 1
    29     in [mm^3/day]
    30 time = 0 bind time
    31     in [day]
    32 volume_plasma = 1
    33     in [L]
    34 dot(volume_tumor) = 2 * (lambda_0 * (lambda_1 * volume_tumor)) / (2 * (lambda_0 * volume_tumor) + lambda_1) - kappa * (drug_concentration * volume_tumor)
    35     in [mm^3]

### Run simulation for given parameters, dose and regimen (Erlotinib, A677)

```python
# set parameter values for Erlotinib (according to [1] Sup. Table S2)
k_a.set_rhs(55.0)  # [1 / d]

volume_p.set_rhs(0.127)  # [L]
k_e.set_rhs(7.56)  # [1 / d]

volume_T.set_state_value(122)  # mm^3
lambda_0.set_rhs(0.0971)  # per day
lambda_1.set_rhs(127)  # mm^3 / day
kappa.set_rhs(0.117)  # L / mg / day

# define dosing regimen
duration = 0.001  # [1 / d] how long does it take for dose to be in dosing compartment?
dose_rate.set_rhs(100 * 0.2 / duration)  # 100 mg / kg / d let's estimate weight of mouse 0.2 kg
dosing_regimen = myokit.Protocol()
dosing_regimen.schedule(level=1, start=2, duration=duration, period=1, multiplier=15)

# run simulation
simulation = myokit.Simulation(model, dosing_regimen)
result = simulation.run(30)
```

```python
import matplotlib.pyplot as plt

time_key = 'plasma.time'
conc_key = 'plasma.drug_concentration'
tumor_key = 'plasma.volume_tumor'

time = result[time_key]
drug_conc = result[conc_key]
volume_tumor = result[tumor_key]

# plot drug time-concentration curve
fig = plt.figure(figsize=(12, 6))

# semilog plot
plt.semilogy(time, drug_conc)

# set x and y label
unit = model.get(time_key).unit()
plt.xlabel('time in %s' % str(unit))
unit = model.get(conc_key).unit()
plt.ylabel('plasma conc. in %s' % str(unit))

# # set y lim
# plt.ylim([0.01,100])

plt.show()

# plot time-tumor volume curve
fig = plt.figure(figsize=(12, 6))

# semilog plot
plt.semilogy(time, volume_tumor)

# set x and y label
unit = model.get(time_key).unit()
plt.xlabel('time in %s' % str(unit))
unit = model.get(tumor_key).unit()
plt.ylabel('tumor volume in %s' % str(unit))

plt.show()
```

```python

```


```python
### PREAMBLE
# Chapter 2 - linear models
# linear.svg

import numpy as np
import matplotlib.pyplot as plt
import matplotlib

%matplotlib inline
%config InlineBackend.figure_format = 'svg'
```

**any bullet points are comments I've made to help for understanding!**

## Chapter 2: Linear models

Before we dive into the discussion of adversarial attacks and defenses on deep networks, it is worthwhile considering the situation that arises when the hypothesis class is linear.  That is, for the multi-class setting $h_\theta : \mathbb{R}^n \rightarrow \mathbb{R}^k$, we consider a classifier of the form
\begin{equation}
h_\theta(x) = W x + b
\end{equation}
where $\theta = \{W \in \mathbb{R}^{k \times n}, b \in \mathbb{R}^k\}$.  We will also shortly consider a binary classifier of a slightly different form, as many of the ideas are a bit easer to describe in this setting, before returning back to the multi-class case.

Substituting this hypothesis back into our robust optimization framework, and also focusing on the case where the pertrubation set $\Delta$ is a norm ball $\Delta = \{\delta : \|\delta\| \leq \epsilon\}$, where we don't actually specify the type of norm, so this could be $\ell_\infty$, $\ell_2$, $\ell_1$, etc, we arrive at the mix-max problem
\begin{equation}
\DeclareMathOperator*{\minimize}{minimize}
\minimize_{W,b} \frac{1}{|D|} \sum_{x,y \in D} \max_{\|\delta\| \leq \epsilon}\ell(W(x+\delta) + b, y).
\end{equation}
**The key point we will emphasize in this section, is that under this formulation, we can solve the inner maximization $exactly$ for the case of binary optimization, and provide a relatively tight upper bound for the case of multi-class classification.  Futhermore, because the resulting minimization problem is still convex in $\theta$ (we will see shortly that it remains convex even after maximizing over $\delta$, the resulting robust training procedure can _also_ be solved optimally, and thus we can achieve the globally optimal robust classifier (at least for the case of binary classification).  This is in stark constrast to the deep network case, where neither the inner maximization problem nor the outer minmimization problem can be solved globally (in the case of the outer minimization, this holds _even_ if we assume exact solutions of the inner problem, due to the non-convexity of the network itself).**

However, understanding the linear case provides important insights into the theory and practice of adversarial robustness, and also provides connections to more commonly-studied methods in machine learning such as support vector machines.

## Binary classification

Let's begin first by considering the case of binary classification, i.e., k=2 in the multi-class setting we desribe above.  In this case, rather than use multi-class cross entropy loss, we'll be adopting the more common approach and using the binary cross entropy, or logistic loss.  In this setting, we have our hypothesis function
\begin{equation}
h_\theta(x) = w^T x + b
\end{equation}
for $\theta = \{w \in \mathbb{R}^n, b \in \mathbb{R}\}$, class label $y \in \{+1,-1\}$, and loss function
\begin{equation}
\ell(h_\theta(x), y) = \log(1+\exp(-y\cdot h_\theta(x))) \equiv L(y \cdot h_\theta(x))
\end{equation}
where for convience below we define the function $L(z) = \log(1+\exp(-z))$ which we will use below when discussing how to solve the optimization problems involving this loss.  The semantics of this setup are that for a data point $x$, the classifier predicts class $+1$ with probability
\begin{equation}
p(y=+1|x) = \frac{1}{1 + \exp(-h_\theta(x))}.
\end{equation}

**Aside:** Again, for those who may be unfamiliar with how this setting relates to the multiclass case we saw before, note that if we use the traditional mutlticlass cross entropy loss with two classes, of class 1 would be given by
\begin{equation}
\frac{\exp(h_\theta(x)_1)}{\exp(h_\theta(x)_1) + \exp(h_\theta(x)_2)} =
\frac{1}{1 + \exp(h_\theta(x)_2 - h_\theta(x)_1)}
\end{equation}
and similarly the probaiblity of predicting class 2
\begin{equation}
\frac{\exp(h_\theta(x)_2)}{\exp(h_\theta(x)_1) + \exp(h_\theta(x)_2)} =
\frac{1}{1 + \exp(h_\theta(x)_1 - h_\theta(x)_2)}.
\end{equation}
We can thus define a single scalar-valued hypothesis
\begin{equation}
h'_\theta(x) \equiv h_\theta(x)_1 - h_\theta(x)_2
\end{equation}
with the associated probabilities
\begin{equation}
p(y|x) = \frac{1}{1 + \exp(-y\cdot h'_\theta(x))}
\end{equation}
for $y$ defined as $y \in \{+1,-1\}$ as written above.  Taking the negative log of this quantity gives
\begin{equation}
-\log \frac{1}{1 + \exp(-y\cdot h'_\theta(x))} = \log(1 + \exp(-y\cdot h'_\theta(x)))
\end{equation}
which is exactly the logistic loss we define above.

### Solving the inner maximization problem

* showing how this can be done in the binary case for a linear model

Now let's return to the robust optimization problem, and consider the inner maximization problem, which in this case takes the form
\begin{equation}
\DeclareMathOperator*{\maximize}{maximize}
\maximize_{\|\delta\| \leq \epsilon} \ell(w^T (x+\delta), y) \equiv \maximize_{\|\delta\| \leq \epsilon} L(y \cdot (w^T(x+\delta) + b)).
\end{equation}
The key point we need to make here is that in this setting, it is actually possible to solve this inner maximization problem exactly.  To show this, first note the $L$ as we described it earlier is a scalar function that is monotonically decreasing, and looks like the following:

```python
x = np.linspace(-4,4)
plt.plot(x, np.log(1+np.exp(-x)))
```

    [<matplotlib.lines.Line2D at 0x11d3c2470>]

Because the function is monotoic decreasing, if we want to maximize this function applied to a scalar, that is equivalent to just minimizing the scalar quantity.  That is
\begin{equation}
\begin{split}
\DeclareMathOperator*{\minimize}{minimize}
\max_{\|\delta\| \leq \epsilon} L \left(y \cdot (w^T(x+\delta) + b) \right) & =
L\left( \min_{\|\delta\| \leq \epsilon}  y \cdot (w^T(x+\delta) + b) \right) \\
& = L\left(y\cdot(w^Tx + b) + \min_{\|\delta\| \leq \epsilon} y \cdot w^T\delta  \right)
\end{split}
\end{equation}
where we get the second line by just distributing out the linear terms.

So we need to consider how to solve the problem
\begin{equation}
\min_{\|\delta\| \leq \epsilon} y \cdot w^T\delta.
\end{equation}
To get the intuition here, let's just consider the case that $y = +1$, and consider an $\ell_\infty$ norm constraint $\|\delta\|_\infty \leq \epsilon$.  **Since the $\ell_\infty$ norm says that each element in $\delta$ must have magnitude less than or equal $\epsilon$, we clearly minimize this quantity when we set $\delta_i = -\epsilon$ for $w_i \geq 0$ and $\delta_i = \epsilon$ for $w_i < 0$.  For $y = -1$, we would just flip these quantities.**  That is, the optimal solution to the above optimization problem for the $\ell_\infty$ norm is given by
\begin{equation}
\delta^\star = - y \epsilon \cdot \mathrm{sign}(w)
\end{equation}

Furthermore, we can also determine the function valued achieved by this solution,
\begin{equation}
y \cdot w^T\delta^\star = y \cdot \sum_{i=1} -y \epsilon \cdot \mathrm{sign}(w_i) w_i = -y^2 \epsilon \sum_{i} |w_i| = -\epsilon \|w\|_1.
\end{equation}
Thus, we can actually _analytically_ compute the solution of the inner maximization problem, which just has the form
\begin{equation}
\max_{\|\delta\|_\infty \leq \epsilon} L \left(y \cdot (w^T(x+\delta) + b) \right) =
L \left(y \cdot (w^Tx + b) - \epsilon \|w\|_1 \right ).
\end{equation}
**Therefore, instead of solving the robust min-max problem as an actual min-max problem, we have been able to convert it to a pure minimization problem,** given by
\begin{equation}
\minimize_{w,b} \frac{1}{D} \sum_{(x,y) \in D} L \left(y \cdot (w^Tx + b) - \epsilon \|w\|_1 \right ).
\end{equation}
**This problem is still convex in $w,b$, so can be solved exactly, or e.g., SGD will also approach the globally optimal solution.**  A little more generally, it turns out that in general the optimization problem
\begin{equation}
\min_{\|\delta\| \leq \epsilon} y \cdot w^T\delta = -\epsilon \|w\|_*
\end{equation}
where $\|\cdot\|_*$ denotes the the dual norm of our original norm bound on $\theta$ ($\|\cdot\|_p$ and $\|\cdot\|_q$ are dual norms for $1/p + 1/q = 1$).  So regardless of our norm constraint, we can actually solve the robust optimization problem via a single minimization problem (and find the analytical solution to the worse-case adversarial attack), without the need to explicitly solve a min-max problem.

Note that the final robust optimization problem (now adopting the general form),
\begin{equation}
\minimize_{w,b} \frac{1}{D}\sum_{(x,y) \in D} L \left(y \cdot (w^Tx + b) - \epsilon \|w\|_* \right )
\end{equation}
looks an awful lot like the typical norm-regularized objective we commonly consider in machine learning
\begin{equation}
\minimize_{w,b} \frac{1}{D}\sum_{(x,y) \in D} L (y \cdot (w^Tx + b)) + \epsilon \|w\|_*
\end{equation}
with the except that the regularization term is _inside_ the loss function.  **Intuitively, this means that in the robust optimization case, if a point is far from the decision boundary, we _don't_ penalize the norm of the parameters, but we _do_ penalize the norm of the parameters (transformed by the loss function) for a point where we close to the decision boundary.**  The connections between such formulations and e.g. support vector machines, has been studied extensively [\cite XuMannor].

### Illustration of binary classification setting

Let's see what this looks like for an actual linear classifier.  In doing so, we can also get a sense of how well traditional linear models might work to also prevent adversarial examples (spoiler: not very well, unless you do regularize).  To do so, we're going to consider the MNIST data set, which will actually serve as a running example for the vast majority of the rest of this tutorial.  **MNIST is actually a fairly poor choice of problem for many reasons: in addition to being very small for modern ML, it also has the property that it can easily be "binarized", i.e., because the pixel values are essentially just black and white, we can remove more $\ell_\infty$ noise by just rounding to 0 or 1, and then classifying the resulting image.**  But presuming we _don't_ use such strategies, it is still a reasonable choice for initial experiments, and small enough that some of the more complex methods we discuss in further sections still can be run in a reasonable amount of time.
* explains why MNIST is not best to use
* the problem with MNIST appears to relate mostly to $\ell_\infty$

Since we're in the binary classification setting for now, let's focus on the even easier problem of just classifying between 0s and 1s in the MNIST data (we'll return back to the multi-class setting for linear models shortly).  Let's first load the data using the PyTorch library and build a simple linear classifier using gradient descent.  Note that we're going to do this a bit more explicitly to replicate the logic above (i.e., using labels of +1/-1, using the direct computation of the $L$ function, etc) instead of reverse-engineering it from the typical PyTorch functions.

Let's first load the MNIST data reduced to the 0/1 examples.

```python
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

mnist_train = datasets.MNIST("./data", train=True, download=True, transform=transforms.ToTensor())
mnist_test = datasets.MNIST("./data", train=False, download=True, transform=transforms.ToTensor())

train_idx = mnist_train.train_labels <= 1
mnist_train.train_data = mnist_train.train_data[train_idx]
mnist_train.train_labels = mnist_train.train_labels[train_idx]

test_idx = mnist_test.test_labels <= 1
mnist_test.test_data = mnist_test.test_data[test_idx]
mnist_test.test_labels = mnist_test.test_labels[test_idx]

train_loader = DataLoader(mnist_train, batch_size = 100, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size = 100, shuffle=False)
```

Now let's build a simple linear classifier (the `nn.Linear` module does this, containing the weights in the `.weight` object and the bias in the `.bias` object).  The `nn.Softplus` function implement $L$ function above (though without negating the input), and does so in a more numerically stable way than using the exp or log functions directly.

```python
import torch
import torch.nn as nn
import torch.optim as optim

# do a single pass over the data
def epoch(loader, model, opt=None):
    total_loss, total_err = 0.,0.
    for X,y in loader:
        yp = model(X.view(X.shape[0], -1))[:,0]
        loss = nn.BCEWithLogitsLoss()(yp, y.float())
        if opt:
            opt.zero_grad()
            loss.backward()
            opt.step()

        total_err += ((yp > 0) * (y==0) + (yp < 0) * (y==1)).sum().item()
        total_loss += loss.item() * X.shape[0]
    return total_err / len(loader.dataset), total_loss / len(loader.dataset)
```

We'll train the classifier for 10 epochs, though note that MNIST 0/1 binary classification is a _very_ easy problem, and after one epoch we basically have converged to the final test error (though test loss still decreases).  It eventually reaches as error of 0.0004, which in this case actually just making one mistake on the test set.

```python
model = nn.Linear(784, 1)
opt = optim.SGD(model.parameters(), lr=1.)
print("Train Err", "Train Loss", "Test Err", "Test Loss", sep="\t")
for i in range(10):
    train_err, train_loss = epoch(train_loader, model, opt)
    test_err, test_loss = epoch(test_loader, model)
    print(*("{:.6f}".format(i) for i in (train_err, train_loss, test_err, test_loss)), sep="\t")
```

    Train Err	Train Loss	Test Err	Test Loss
    0.007501	0.015405	0.000946	0.003278
    0.001342	0.005392	0.000946	0.002892
    0.001342	0.004438	0.000473	0.002560
    0.001105	0.003788	0.000946	0.002495
    0.000947	0.003478	0.000946	0.002297
    0.000947	0.003251	0.000946	0.002161
    0.000711	0.002940	0.000473	0.002159
    0.000790	0.002793	0.000946	0.002109
    0.000711	0.002650	0.000946	0.002107
    0.000790	0.002529	0.000946	0.001997

In case you're curious, we can actually look at the one test example that the classifier makes a mistake on, which indeed seems a bit odd relative to most 0s and 1s.

```python
X_test = (test_loader.dataset.test_data.float()/255).view(len(test_loader.dataset),-1)
y_test = test_loader.dataset.test_labels
yp = model(X_test)[:,0]
idx = (yp > 0) * (y_test == 0) + (yp < 0) * (y_test == 1)
plt.imshow(1-X_test[idx][0].view(28,28).numpy(), cmap="gray")
plt.title("True Label: {}".format(y_test[idx].item()))
```

    Text(0.5,1,'True Label: 0')

Hopefully you've already noticed something else about the adversarial examples we generate in the linear case: because the optimal perturbation is equal to
\begin{equation}
\delta^\star = - y \epsilon \cdot \mathrm{sign}(w),
\end{equation}
**which doesn't depend on $x$, this means that the best perturbation to apply is the _same_ across all examples.  Note however, that to get the best valid perturbation, here we should really be constraining $x + \delta$ to be in $[0,1]$, which _doesn't_ hold for this case.**  For simplicity, we'll ignore this for now, and go ahead and add this same $\delta$ anyway (even if it gives us a technically invalid image).  After all, for the classifier, the inputs are just numerical value, so we can always have values greater than one or less than zero; the performance we'll see also applies to the case where we clip values, it just adds a bit of unnecessary hassle.

Let's look at the actual perturbation, to try to get a sense of it.

```python
epsilon = 0.2
delta = epsilon * model.weight.detach().sign().view(28,28)
plt.imshow(1-delta.numpy(), cmap="gray")
```

    <matplotlib.image.AxesImage at 0x128527978>

**It's perhaps not all that obvious, but if you squint you can see that maybe there is a vertical line (like a 1) in black pixels, and a cirlce (like a 0) in in white.  The intuition here is that moving in the black direction, we make the classifier think the image is more like a 1, while moving in the white direction, more like a 0.**  But the picture here is not perfect, and it you didn't know to look for this, it may not be obviously.  We'll shortly

Let's next see what happens when we evaluate the test accuracy when we make this (optimal) adverarial attack on the images in the test set.

```python
def epoch_adv(loader, model, delta):
    total_loss, total_err = 0.,0.
    for X,y in loader:
        # this yp (y prediction) is changing the X's in the test to account for the adversarial attack
        yp = model((X-(2*y.float()[:,None,None,None]-1)*delta).view(X.shape[0], -1))[:,0]
        loss = nn.BCEWithLogitsLoss()(yp, y.float())
        total_err += ((yp > 0) * (y==0) + (yp < 0) * (y==1)).sum().item()
        total_loss += loss.item() * X.shape[0]
    return total_err / len(loader.dataset), total_loss / len(loader.dataset)
print(epoch_adv(test_loader, model, delta[None,None,:,:]))
```

    (0.8458628841607565, 3.4517438034075654)

**So allowing perturbations within the $\ell_\infty$ ball of size $\epsilon=0.2$, the classifier we go from essentially zero error to 84.5% error.  Unlike the ImageNet case, the perturbed images here, _are_ recognizably different (we just overlay the noise you saw above), but this would definitely not be sufficient to fool most humans in recognizing the image.**

```python
f,ax = plt.subplots(5,5, sharey=True)
for i in range(25):
    ax[i%5][i//5].imshow(1-(X_test[i].view(28,28) - (2*y_test[i]-1)*delta).numpy(), cmap="gray")
    ax
```

### Training robust linear models

We've now seen that a standard linear model suffers from a lot of the same problems as deep models (though it should be said, they are still slightly more resilient than standard training for deep networks, for which an $\ell_\infty$ ball with $\epsilon=0.2$ could easily create 100% error).  But we also know that we can easily perform exact robust optimization (i.e., solving the equivalent of the min-max problem) by simply incorporating the $\ell_1$ norm into the objective.  Putting this into the standard binary cross entropy loss that PyTorch implements (which uses labels of 0/1 by default, not -1/+1), takes a bit of munging, but the training procedure is still quite simple: we just subtract $\epsilon(2y-1)\|w\|_1$ from the predictions (the $2y-1$ scales the 0/1 entries to -1/+1).

```python
# do a single pass over the data
# training now to handle adversarial attack
def epoch_robust(loader, model, epsilon, opt=None):
    total_loss, total_err = 0.,0.
    for X,y in loader:
        yp = model(X.view(X.shape[0], -1))[:,0] - epsilon*(2*y.float()-1)*model.weight.norm(1)
        loss = nn.BCEWithLogitsLoss()(yp, y.float())
        if opt:
            opt.zero_grad()
            loss.backward()
            opt.step()

        total_err += ((yp > 0) * (y==0) + (yp < 0) * (y==1)).sum().item()
        total_loss += loss.item() * X.shape[0]
    return total_err / len(loader.dataset), total_loss / len(loader.dataset)
```

```python
model = nn.Linear(784, 1)
opt = optim.SGD(model.parameters(), lr=1e-1)
epsilon = 0.2
print("Rob. Train Err", "Rob. Train Loss", "Rob. Test Err", "Rob. Test Loss", sep="\t")
for i in range(20):
    train_err, train_loss = epoch_robust(train_loader, model, epsilon, opt)
    test_err, test_loss = epoch_robust(test_loader, model, epsilon)
    print(*("{:.6f}".format(i) for i in (train_err, train_loss, test_err, test_loss)), sep="\t")
```

    Rob. Train Err	Rob. Train Loss	Rob. Test Err	Rob. Test Loss
    0.147414	0.376791	0.073759	0.228654
    0.073352	0.223381	0.053901	0.176481
    0.062929	0.197301	0.043026	0.154818
    0.057008	0.183879	0.038298	0.139773
    0.052981	0.174964	0.040662	0.143639
    0.050059	0.167973	0.037352	0.132365
    0.048164	0.162836	0.032624	0.119755
    0.046190	0.158340	0.033570	0.123211
    0.044769	0.154719	0.029787	0.118066
    0.043979	0.152048	0.027423	0.118974
    0.041058	0.149381	0.026478	0.110074
    0.040268	0.147034	0.027423	0.114998
    0.039874	0.145070	0.026950	0.109395
    0.038452	0.143232	0.026950	0.109015
    0.037663	0.141919	0.027896	0.113093
    0.036715	0.140546	0.026478	0.103066
    0.036321	0.139162	0.026478	0.107028
    0.035610	0.138088	0.025059	0.104717
    0.035215	0.137290	0.025059	0.104803
    0.034741	0.136175	0.025059	0.106629

We say it above, but we should **emphasize that all the numbers reported above are the _robust_ (i.e., worst case adversarial) errors and losses.  So by training with the robust optimization problem, we're able to train a model such that for $\epsilon=0.2$, no adversarial attack can lead to more then 2.5% error on the test set.**  Quite an improvement from the ~85% that the standard training had.  But how well does it do on the _non-adversarial_ training set?

```python
train_err, train_loss = epoch(train_loader, model)
test_err, test_loss = epoch(test_loader, model)
print("Train Err", "Train Loss", "Test Err", "Test Loss", sep="\t")
print(*("{:.6f}".format(i) for i in (train_err, train_loss, test_err, test_loss)), sep="\t")
```

    Train Err	Train Loss	Test Err	Test Loss
    0.006080	0.015129	0.003783	0.008186

We're getting 0.3% error on the test set.  This is good, but _not_ as good as we were doing with standard training; we're now making 8 mistakes on the test set, instead of the 1 that we were making before.  And this is not just a random effect of this particular problem, or the fact that it is relatively easy.  **Rather, perhaps somewhat surprisingly, there is a _fundamental_ tradeoff between clean accuracy and robust accuracy, and doing better on the robust error leads to higher clean error.**  We will return to this point in much more detail later.

* something to note here where adversarial training typically leads to higher overall error when compared to no adversarial training when looking at standard training set

Finally, let's look at the image of the optimal perturbation for this robust model.

```python
delta = epsilon * model.weight.detach().sign().view(28,28)
plt.imshow(1-delta.numpy(), cmap="gray")
```

    <matplotlib.image.AxesImage at 0x128d39240>

That looks substantially more like a zero than what we saw before.  Thus, we have some (admittedly, at this point, fairly weak) evidence that robsut training may also lead to "adversarial directions" that are inherrently more meaningful.  Rather than fooling the classifier by just adding "random noise" we actually need to start moving the image in the direction of an actual new image (and even doing so, at least with this size epsilon, we aren't very successful at fooling the classifier).  This idea will also come up later.

## Multi-class classification

Before moving on to the deep learning setting, let's briefly consider the multi-class extension of what we presented above.  After all, most deep classifiers that we care about are actually multi-class classifiers, using the cross entropy loss or something similar.  Recalling what we defined before, this means we are considering the linear hypothesis function
\begin{equation}
h_\theta(x) = W x + b
\end{equation}
which results in an inner maximization problem of the form
\begin{equation}
\max_{\|\delta\| \leq \epsilon}\ell(W(x+\delta) + b, y).
\end{equation}
Unforutnately in the binary case, it turns out that it is no longer possible to optimally solve the inner maximization problem.  Specifcally, if we consider the cross entropy loss plugged into the above expression
\begin{equation}
\max_{\|\delta\| \leq \epsilon} \left (\log \left ( \sum_{j=1}^k \exp(w_j^T (x + \delta) + b_i) \right ) - (w_y^T(x + \delta) + b_y) \right ).
\end{equation}
Here, unlike the binary case, we cannot push the max over $\delta$ inside the nonlinear function (the log-sum-exp function is convex, so maximizing over it is difficult in general).

```python

```


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

```python
#
# Pierre Proulx
#
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
%matplotlib inline
```

### Section 2.4, écoulement d'un film de fluide Newtonien dans une conduite annulaire

>Le développement fait dans Transport Phenomena sera répété ici en développant les solutions avec le calculateur formel sympy et en traçant la solution avec sympy.plot.
>Vous pouvez suivre le développement détaillé à l'aide du livre et voir comment les équations de bilans initiales deviennent les profils de vitesse, de force, débit, etc...

```python
# Paramètres, variables et fonctions, presque identique à la section 2.3
#
r,delta_r,L,mu,R,P_0,P_L,kappa,rho,g=sp.symbols('r,delta_r,L,mu,R,P_0,P_L,kappa,rho,g')
C1=sp.symbols('C1')
phi_rz,phi_zz=sp.symbols('phi_rz,phi_zz')
```

```python
#
# équation identique à la section 2.3
#
#
# Équation du bilan des forces
#
eq=2*sp.pi*r*L*phi_rz(r)      -2*sp.pi*(r+delta_r)*L*phi_rz(r+delta_r)
eq=eq+2*sp.pi*delta_r*r*phi_zz(0)-2*sp.pi*r*delta_r*phi_zz(L)
eq=eq+2*sp.pi*r*delta_r*L*rho*g
#eq=r*phi_rz(r)/delta_r-(r+delta_r)*phi_rz(r+delta_r)/delta_r+r*P_0/L-r*P_L/L
#
# pose de la limite quand dr tends vers 0
#
eq=eq/(2*sp.pi*L*delta_r)
display(eq)
```

```python
eq1=sp.limit(eq,delta_r,0).doit()
display(eq1)
```

```python
eq1=eq1.subs(phi_zz(0),P_0)     # les termes de convection s'annulent comme en 2.2
eq1=eq1.subs(phi_zz(L),P_L)     # mais il reste le terme de pression
display(eq1)
```

```python
eq1=sp.limit(eq1,delta_r,0).doit()
display(eq1)
```

```python
#
# encore identique à 2.3
#
tau_rz=sp.symbols('tau_rz')
eq2=sp.Eq(eq1.subs(phi_rz(r),tau_rz(r)))
display(eq2)
```

```python
eq2=sp.dsolve(eq2)
v_z=sp.Function('v_z')(r)
newton=-mu*sp.Derivative(v_z,r)
eq3=eq2.subs(tau_rz(r),newton)
display(eq3)
```

## Maintenant la différence avec la section 2.3 arrive, la condition limite pour $C_1$ n'est pas la même car r=0 n'est pas partie du domaine.

```python
eq4=sp.dsolve(eq3,v_z)
display(eq4)
```

Maintenant on peut trouver les deux constantes inconnues en solutionnant les deux équations à deux inconnues provenant de la pose des conditions aux limites:

```python
display(sp.Eq(eq4.subs(r,kappa*R),0),sp.Eq(eq4.subs(r,R      ),0))
# juste pour voir le terme de gauche de l'équation avec le terme de droite, on éliminera le terme de gauche pour solutionner
condition_1=sp.Eq(eq4.rhs.subs(r,kappa*R))
condition_2=sp.Eq(eq4.rhs.subs(r,R      ))
display(condition_1,condition_2)
constante=sp.solve([condition_1,condition_2], sp.symbols('C1,C2'))
display(constante)
```

```python
v_z=eq4.subs(constante)
display(v_z.simplify())
```

```python
#
# Maintenant traçons le profil en donnant des valeurs de paramètres réalistes
#
# Remarquez ici que la gravité est négative sur la figure 2.4-1
#
# Changez la hauteur, la pression en 0, etc... pour voir le changement de direction de
# l'écoulement. Quelle est la pression nécessaire pour pousser le fluide vers le haut
# si la hauteur de la colonne est de 10 mètres?
#
dictionnaire={'rho':1000, 'mu':0.001,'g':-9.81,
              'R':0.05, 'L':10,'P_0':100000,'P_L':0,'kappa':0.5}
v_z=v_z.rhs.subs(dictionnaire).evalf()
R=R.subs(dictionnaire)
kappa=kappa.subs(dictionnaire)
display(v_z)
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize']=6,4
sp.plot(v_z,(r,kappa*R,R),ylim=[0,20],ylabel='v_z');
```

#### * 1 Exemple de calcul: trouvez la position du maximum de vitesse, c'est le $ \lambda R$ de la figure 2.4-1

#### La solution est donc l'endroit ou la dérivée de v est nulle, donc on cherche $\lambda R $ tel que $ \frac {dv_z} {dr} |_{r = \lambda R} = 0$

```python
lam=sp.solve(sp.diff(v_z,r))      # Je cherche l'endroit ou la dérivée est nulle.
print('Position du maximum (le lambda R de Bird)')
maxi=lam[1]                     # la deuxième racine car la première est négative.
display(maxi)
```

#### * 2 Calculez la pression $P_0$ requise pour pour que l'écoulement soit nul en gardant une hauteur de 10 mètres. Vérifiez ensuite graphiquement que l'écoulement est bien nul en tracant le graphe de la vitesse.