```
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```
#export
from exp.nb_06 import *
```

## ConvNet

Let's get the data and training interface from where we left in the last notebook.

```
x_train,y_train,x_valid,y_valid = get_data()

x_train,x_valid = normalize_to(x_train,x_valid)
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)

nh,bs = 50,512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```
mnist_view = view_tfm(1,28,28)
cbfs = [Recorder,
        partial(AvgStatsCallback,accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```
nfs = [8,16,32,64,64]
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```
%time run.fit(2, learn)
```

    train: [0.88018953125, tensor(0.7081, device='cuda:0')]
    valid: [0.17246004638671875, tensor(0.9443, device='cuda:0')]
    train: [0.14102130859375, tensor(0.9570, device='cuda:0')]
    valid: [0.10157955932617188, tensor(0.9701, device='cuda:0')]
    CPU times: user 5.13 s, sys: 1.63 s, total: 6.76 s
    Wall time: 8.03 s

## Batchnorm

### Custom

Let's start by building our own `BatchNorm` layer from scratch.

```
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.ones(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
        if self.training:
            with torch.no_grad(): m,v = self.update_stats(x)
        else: m,v = self.means,self.vars
        x = (x-m) / (v+self.eps).sqrt()
        return x*self.mults + self.adds
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

```
#export
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None: m.bias.data.zero_()
    for l in m.children(): init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None, uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
```

```
with Hooks(learn.model, append_stats) as hooks:
    run.fit(1, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```
learn,run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

```
%time run.fit(3, learn)
```

    train: [0.26143421875, tensor(0.9176, device='cuda:0')]
    valid: [0.12511885986328125, tensor(0.9612, device='cuda:0')]
    train: [0.083749794921875, tensor(0.9745, device='cuda:0')]
    valid: [0.219935791015625, tensor(0.9334, device='cuda:0')]
    train: [0.062450517578125, tensor(0.9809, device='cuda:0')]
    valid: [0.17669794921875, tensor(0.9466, device='cuda:0')]
    CPU times: user 4.63 s, sys: 624 ms, total: 5.25 s
    Wall time: 5.36 s

### Builtin batchnorm

```
#export
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

```
%time run.fit(3, learn)
```

    train: [0.28877712890625, tensor(0.9121, device='cuda:0')]
    valid: [0.1067998046875, tensor(0.9680, device='cuda:0')]
    train: [0.0775843115234375, tensor(0.9765, device='cuda:0')]
    valid: [0.07021278076171875, tensor(0.9800, device='cuda:0')]
    train: [0.053029921875, tensor(0.9840, device='cuda:0')]
    valid: [0.06344912109375, tensor(0.9817, device='cuda:0')]
    CPU times: user 5.67 s, sys: 915 ms, total: 6.59 s
    Wall time: 8.88 s

### With scheduler

Now let's add the usual warm-up/annealing.

```
sched = combine_scheds([0.3, 0.7], [sched_lin(0.6, 2.), sched_lin(2., 0.1)])
```

```
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched)])
```

```
run.fit(8, learn)
```

    train: [0.32286591796875, tensor(0.9060, device='cuda:0')]
    valid: [0.1767323486328125, tensor(0.9450, device='cuda:0')]
    train: [0.09884525390625, tensor(0.9682, device='cuda:0')]
    valid: [0.17165784912109375, tensor(0.9475, device='cuda:0')]
    train: [0.0699253076171875, tensor(0.9774, device='cuda:0')]
    valid: [0.06874525756835938, tensor(0.9787, device='cuda:0')]
    train: [0.042292138671875, tensor(0.9869, device='cuda:0')]
    valid: [0.06281668090820312, tensor(0.9808, device='cuda:0')]
    train: [0.02978388671875, tensor(0.9907, device='cuda:0')]
    valid: [0.061725439453125, tensor(0.9817, device='cuda:0')]
    train: [0.01906186279296875, tensor(0.9946, device='cuda:0')]
    valid: [0.0569135498046875, tensor(0.9834, device='cuda:0')]
    train: [0.01300778564453125, tensor(0.9967, device='cuda:0')]
    valid: [0.048958990478515624, tensor(0.9857, device='cuda:0')]
    train: [0.009002083129882812, tensor(0.9983, device='cuda:0')]
    valid: [0.04764816589355469, tensor(0.9860, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

```
class LayerNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))

    def forward(self, x):
        m = x.mean((1,2,3), keepdim=True)
        v = x.var ((1,2,3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

```
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

```
%time run.fit(3, learn)
```

    train: [nan, tensor(0.1184, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 9.16 s, sys: 1.82 s, total: 11 s
    Wall time: 14.3 s

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

```
class InstanceNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))

    def forward(self, x):
        m = x.mean((2,3), keepdim=True)
        v = x.var ((2,3), keepdim=True)
        res = (x-m) / ((v+self.eps).sqrt())
        return res*self.mults + self.adds
```

```
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

```
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 8.74 s, sys: 1.6 s, total: 10.3 s
    Wall time: 13.6 s

*Question*: why can't this classify anything?

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) have you covered:

### Group norm

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectors of size `num_channels` if
`affine` is `True`.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    `num_groups (int)`: number of groups to separate the channels into
-    `num_channels (int)`: number of channels expected in input
-    `eps`: a value added to the denominator for numerical stability. Default: `1e-5`
-    `affine`: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

```
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```
%time run.fit(1, learn)
```

    train: [2.33641984375, tensor(0.1839, device='cuda:0')]
    valid: [3050.7212, tensor(0.2348, device='cuda:0')]
    CPU times: user 1min 45s, sys: 8.65 s, total: 1min 54s
    Wall time: 1min 54s

### Running Batch Norm

To solve this problem we introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std.

```
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step', tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        s = x.sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel()/nc)
        mom1 = 1 - (1-self.mom)/math.sqrt(bs-1)
        self.mom1 = self.dbias.new_tensor(mom1)
        self.sums.lerp_(s, self.mom1)
        self.sqrs.lerp_(ss, self.mom1)
        self.count.lerp_(c, self.mom1)
        self.dbias = self.dbias*(1-self.mom1) + self.mom1
        self.batch += bs
        self.step += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        c = self.count
        if self.step<100:
            sums = sums / self.dbias
            sqrs = sqrs / self.dbias
            c    = c    / self.dbias
        means = sums/c
        vars = (sqrs/c).sub_(means*means)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div_((vars.add_(self.eps)).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

NB: the calculation of `self.dbias` in the version in the lesson video was incorrect. The correct version is in the cell above. Also, we changed how we calculated `self.mom1` to something that it more mathematically appropriate. These two changes improved the accuracy from 91% (in the video) to 97%+ (shown below)!

```
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(RunningBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_rbn, cbs=cbfs)
```

```
%time run.fit(1, learn)
```

    train: [0.3415529296875, tensor(0.9098, device='cuda:0')]
    valid: [0.13382496337890626, tensor(0.9723, device='cuda:0')]
    CPU times: user 4min 32s, sys: 24 s, total: 4min 56s
    Wall time: 4min 56s

This solves the small batch size issue!

### What can we do in a single epoch?

Now let's see with a decent batch size what result we can get.

```
data = DataBunch(*get_dls(train_ds, valid_ds, 32), c)
```

```
learn,run = get_learn_run(nfs, data, 0.8, conv_rbn, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

```
%time run.fit(1, learn)
```

    train: [0.155741728515625, tensor(0.9518, device='cuda:0')]
    valid: [0.10425401611328125, tensor(0.9705, device='cuda:0')]
    CPU times: user 17.5 s, sys: 1.6 s, total: 19.1 s
    Wall time: 19.1 s

### Simplified RunningBatchNorm

It turns out we don't actually need to debias - because, for instance, dividing a debiased sum by a debiased count is the same as dividing a *biased* sum by a *biased* count! So we can remove all the debiasing stuff and end up with a simpler class. Also, we should save `eps` as a buffer since it impacts the calculation. (Thanks to Stas Bekman for noticing these.) Also we can slightly change the final calculation in `forward` with one that uses `factor` and `offset` to reduce the amount of broadcasting required. (Thanks to Tom Viehmann for this suggestion.)

```
#export
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom, self.eps = mom, eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('factor', tensor(0.))
        self.register_buffer('offset', tensor(0.))
        self.batch = 0

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        s    = x    .sum(dims, keepdim=True)
        ss   = (x*x).sum(dims, keepdim=True)
        c    = s.new_tensor(x.numel()/nc)
        mom1 = s.new_tensor(1 - (1-self.mom)/math.sqrt(bs-1))
        self.sums .lerp_(s , mom1)
        self.sqrs .lerp_(ss, mom1)
        self.count.lerp_(c , mom1)
        self.batch += bs
        means = self.sums/self.count
        varns = (self.sqrs/self.count).sub_(means*means)
        if bool(self.batch < 20): varns.clamp_min_(0.01)
        self.factor = self.mults / (varns+self.eps).sqrt()
        self.offset = self.adds - means*self.factor

    def forward(self, x):
        if self.training: self.update_stats(x)
        return x*self.factor + self.offset
```

```
learn,run = get_learn_run(nfs, data, 0.8, conv_rbn, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

```
%time run.fit(1, learn)
```

    train: [0.15630783203125, tensor(0.9518, device='cuda:0')]
    valid: [0.07094555053710938, tensor(0.9787, device='cuda:0')]
    CPU times: user 15.6 s, sys: 1.57 s, total: 17.1 s
    Wall time: 17.2 s

## Export

```
nb_auto_export()
```

    <IPython.core.display.Javascript object>


## Dealing with Unbalanced Classes in Machine Learning

In many real-world classification problems, we stumble upon training data with unbalanced classes. This means that the individual classes do not contain the same number of elements. For example, if we want to build an [image-based skin cancer detection system](https://www.nature.com/nature/journal/v542/n7639/full/nature21056.html#tables) using [convolutional neural networks](https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/), we might encounter a dataset with about 95% negatives and 5% positives. This is for good reasons: Images associated with a negative diagnosis are way more common than images with a positive diagnosis. Rather than regarding this as a flaw in the dataset, we should leverage the additional information that we get. This blog post will show you how.

Unbalanced classes create two problems:

1. The accuracy (i.e. ratio of test samples for which we predicted the correct class) is no longer a good measure of the model performance. A model that just predicts “not cancer” everytime will yield a 95% accuracy, even though it is a bad (and even dangerous) model that does not yield any insight or scientific advancement, despite the fact that “95% accuracy” sounds like something good. In addition, it’s hard to get an intuition for how good a model with 96%, 97% or 98% accuracy really is.
2. The training process might arrive at a local optimum that always predicts “not cancer”, making it hard to further improve the model.

Fortunately, these problems are not so difficult to solve. Here are a few ways to tackle them.

### 1. Collect more data

If possible, you could collect more data for the underrepresented classes to match the number of samples in the overrepresented classes. This is probably the most rewarding approach, but it is also the hardest and most time-consuming, if not downright impossible. In the cancer example, there is a good reason that we have way more non-cancer samples than cancer samples: These are easier to obtain, since there are more people in the world who haven’t developed cancer.

### 2. Create copies of training samples

Artificially increase the number of training samples for the underrepresented classes by creating copies. While this is the easiest solution, it wastes time and computing resources. In the cancer example, we would almost have to double the size of the dataset in order to achieve a 50:50 share between the classes, which also doubles training time without adding any new information.

### 3. Create augmented copies of training samples

Similar to 2, but create augmented copies of the underrepresented classes. For example, in the case of images, create slightly rotated, shifted or flipped versions of the original images. This has the positive side-effect of making the model more robust to unseen examples. However, it only does so for the underrepresented classes. Ideally, you would want to do this for all classes, but then the classes are unbalanced again and we’re back where we started.

### 4. (Remove training samples)

Remove training samples from the overrepresented classes so that the number of training samples for all classes is the same. This solves our problem and reduces training time, but it makes our model worse. After all, we want to use as much labelled data as we possibly can, even if this causes unbalanced classes. I don’t recommend this solution.

### 5. Train for sensitivity and specificity

The sensitivity tells us the probability that we detect cancer, given that the patient really has cancer. It is thus a measure of how good we are at correctly diagnosing people who have cancer.

```latex
%%latex
\begin{align}
sensitivity = Pr(detect\ cancer\ |\ cancer) = \frac{true\ positives}{positives}
\end{align}
```

\begin{align}
sensitivity = Pr(detect\ cancer\ |\ cancer) = \frac{true\ positives}{positives}
\end{align}

The specificity tells us the probability that we do not detect cancer, given that the patient doesn’t have cancer. It measures how good we are at not causing people to believe that they have cancer if in fact they do not.

```latex
%%latex
\begin{align}
specificity = Pr(\neg detect\ cancer\ |\ \neg cancer) = \frac{true\ negatives}{negatives}
\end{align}
```

\begin{align}
specificity = Pr(\neg detect\ cancer\ |\ \neg cancer) = \frac{true\ negatives}{negatives}
\end{align}

A model that always predicts cancer will have a sensitivity of 1 and a specificity of 0. A model that never predicts cancer will have a sensitivity of 0 and a specificity of 1. An ideal model should have both a sensitivity of 1 and a specificity of 1. In reality, however, this is unlikely to be achievable. Therefore, we should look for a model that achieves a good tradeoff between specificity and sensitivity. So which one of the two is more important? This can’t be said in general. It highly depends on the application.

If you build a photo-based skin cancer detection app, then a high sensitivity is probably more important than a high specificity, since you want to cause people who might have cancer to get themselves checked by a doctor. Specificity is a little less important here, but still, if you detect cancer too often, people might stop using your app since they unnecessarily get annoyed and scared.

Now suppose that our desired tradeoff between sensitivity and specificity is given by a number t∈[0,1] where t=1 means that we only pay attention to sensitivity, t=0 means we only pay attention to specificity and t=0.5 means that we regard both to be equally important. In order to incorporate the desired tradeoff into the training process, we need the samples of the different classes to have a different contribution to the loss. To achieve this, we can simply multiply the contribution of the cancer samples to the loss by

```latex
%%latex
\begin{align}
\frac{number\ of\ non-cancer\ samples}{number\ of\ cancer\ samples} \times t
\end{align}
```

\begin{align}
\frac{number\ of\ non-cancer\ samples}{number\ of\ cancer\ samples} \times t
\end{align}

### Keras Implementation

In [Keras](https://keras.io/), the class weights can easily be incorporated into the loss by adding the following parameter to the fit function (assuming that 1 is the cancer class):

```python
class_weight={
    1: n_non_cancer_samples / n_cancer_samples * t
}
```

Now, while we train, we want to monitor the sensitivity and specificity. Here is how to do this in Keras. In other frameworks, the implementation should be similar (for instance, you could replace all the K calls by numpy calls).

```python
from keras import backend as K

def sensitivity(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    return true_positives / (possible_positives + K.epsilon())

def specificity(y_true, y_pred):
    true_negatives = K.sum(K.round(K.clip((1-y_true) * (1-y_pred), 0, 1)))
    possible_negatives = K.sum(K.round(K.clip(1-y_true, 0, 1)))
    return true_negatives / (possible_negatives + K.epsilon())

model.compile(
    loss='binary_crossentropy',
    optimizer=RMSprop(0.001),
    metrics=[sensitivity, specificity]
)
```

### Generalizing to more than 2 classes

If we have more than two classes, we can generalize sensitivity and specificity to a “per-class accuracy”:

```latex
%%latex
\begin{align}
perClassAccuracy(C) = Pr(detect\ C\ |\ C)
\end{align}
```

\begin{align}
perClassAccuracy(C) = Pr(detect\ C\ |\ C)
\end{align}

In order to train for maximum per-class accuracy, we have to specify class weights that are inversely proportional to the size of the class:

```python
class_weight={
    0: 1.0/n_samples_0,
    1: 1.0/n_samples_1,
    2: 1.0/n_samples_2,
    ...
}
```

Here is a Keras implementation of the per-class accuracy, which I adopted from [jdehesa at Stackoverflow](https://stackoverflow.com/questions/41458859/keras-custom-metric-for-single-class-accuracy).

```python
INTERESTING_CLASS_ID = 0  # Choose the class of interest

def single_class_accuracy(y_true, y_pred):
    class_id_true = K.argmax(y_true, axis=-1)
    class_id_preds = K.argmax(y_pred, axis=-1)
    accuracy_mask = K.cast(K.equal(class_id_preds, INTERESTING_CLASS_ID), 'int32')
    class_acc_tensor = K.cast(K.equal(class_id_true, class_id_preds), 'int32') * accuracy_mask
    class_acc = K.sum(class_acc_tensor) / K.maximum(K.sum(accuracy_mask), 1)
    return class_acc
```

Source: [http://www.deepideas.net/unbalanced-classes-machine-learning/](http://www.deepideas.net/unbalanced-classes-machine-learning/)


# 损失函数

### **KL散度**
$D_{\mathrm{KL}}(P \| Q)=-\sum_{i} P(i) \ln \frac{Q(i)}{P(i)}=\sum_{i} P(i) \ln \frac{P(i)}{Q(i)}=\sum_{i}P(i)(\log{P(i)}-\log{Q(i)})$

其中$Q(i)>0,P(i)>0$, 其中P为原始分布，Q为近似分布，KL散度是两个分布对数差值的期望, 并非距离度量，没有对称性

### **交叉熵** = 熵+KL散度

衡量的是同个变量的不同分布

样本集的两个概率分布 p(x) 和 q(x)，其中 p(x) 为真实分布， q(x)非真实分布

$\mathbf{H}(p, q)=\mathbf{E}_{p}[-\log q]=\mathbf{H}(p)+D_{\mathrm{KL}}(p \| q)=\sum_{x} p(x) \log \frac{1}{q(x)}$

### **条件熵：**

已知随机变量 X 的条件下随机变量 Y 的不确定性

$\begin{aligned} H(Y | X) &=\sum_{x} p(x) H(Y | X=x) \\ &=-\sum_{x} p(x) \sum_{y} p(y | x) \log p(y | x) \\ &=-\sum_{x} \sum_{y} p(x, y) \log p(y | x) \\ &=-\sum_{x, y} p(x, y) \log p(y | x) \end{aligned}$

### **联合熵**：

衡量的是不同的变量

$H(X, Y)=-\sum_{x, y} p(x, y) \log p(x, y)=-\sum_{i=1}^{n} \sum_{j=1}^{m} p\left(x_{i}, y_{i}\right) \log p\left(x_{i}, y_{i}\right)$

# 支撑向量机（SVM）

一般优化表示，

\begin{equation}
\begin{aligned}
&\min \quad f_{0}(x)\\
&\text { s.t. } \quad f_{i}(x) \leq 0, \quad i=1, \cdots, m\\
&\quad \quad \quad h_{i}(x)=0,  i=1, \cdots, p
\end{aligned}
\end{equation}

1.   不管原命题形式如何，其对偶问题都是凸函数
2.   不管什么问题，都可以转化为拉格朗日对偶问题求解
3.   对偶问题是原问题最优解的下确界
4.   slater条件：存在一点 $x \in$relint $D$ ($D$ 的相对内点集）满足$f_{i}(x)<0, i=1, \ldots, m, A x=b$, 这样的点称为严格可行的点
5.   slater定理：满足slater条件且原问题是凸优化问题时，强对偶性成立
6.   slater条件确保了鞍点的存在，KKT条件是鞍点是最优解的充分条件。当原问题是凸优化问题时，KKT条件是充要条件。
7.   KKT条件，其中$h_{i}(x)$是等式约束，$g_{i}(x)$是不等式约束

$$
\begin{array}{c}
\nabla f(x)+\sum_{i=1}^{n} \lambda_{i} \nabla h_{i}(x)+\sum_{i=1}^{n} \mu_{i} \nabla g_{i}(x)=0 \\
\mu_{i} g(x)_{i}=0 \\
\mu_{i} \geq 0 \\
h_{i}(x)=0 \\
g_{i}(x) \leq 0 \\
i=1,2, \ldots, n
\end{array}
$$
8.   凸优化就是这个优化问题的优化函数是凸函数，并且可行域是凸集

#GDBT

```python

```