# Frequentist Inference Case Study - Part A

## 1. Learning objectives

Welcome to part A of the Frequentist inference case study! The purpose of this case study is to help you apply the concepts associated with Frequentist inference in Python. Frequentist inference is the process of deriving conclusions about an underlying distribution via the observation of data. In particular, you'll practice writing Python code to apply the following statistical concepts:
* the _z_-statistic
* the _t_-statistic
* the difference and relationship between the two
* the Central Limit Theorem, including its assumptions and consequences
* how to estimate the population mean and standard deviation from a sample
* the concept of a sampling distribution of a test statistic, particularly for the mean
* how to combine these concepts to calculate a confidence interval

## Prerequisites

To be able to complete this notebook, you are expected to have a basic understanding of:
* what a random variable is (p.400 of Professor Spiegelhalter's *The Art of Statistics, hereinafter AoS*)
* what a population, and a population distribution, are (p. 397 of *AoS*)
* a high-level sense of what the normal distribution is (p. 394 of *AoS*)
* what the t-statistic is (p. 275 of *AoS*)

Happily, these should all be concepts with which you are reasonably familiar after having read ten chapters of Professor Spiegelhalter's book, *The Art of Statistics*.

We'll try to relate the concepts in this case study back to page numbers in *The Art of Statistics* so that you can focus on the Python aspects of this case study. The second part (part B) of this case study will involve another, more real-world application of these tools.

For this notebook, we will use data sampled from a known normal distribution. This allows us to compare our results with theoretical expectations.

## 2. An introduction to sampling from the normal distribution

First, let's explore the ways we can generate the normal distribution. While there's a fair amount of interest in [sklearn](https://scikit-learn.org/stable/) within the machine learning community, you're likely to have heard of [scipy](https://docs.scipy.org/doc/scipy-0.15.1/reference/index.html) if you're coming from the sciences. For this assignment, you'll use [scipy.stats](https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/stats.html) to complete your work.

This assignment will require some digging around and getting your hands dirty (your learning is maximized that way)! You should have the research skills and the tenacity to do these tasks independently, but if you struggle, reach out to your immediate community and your mentor for help.

```python
from scipy.stats import norm
from scipy.stats import t
import numpy as np
import pandas as pd
from numpy.random import seed
import matplotlib.pyplot as plt
```

__Q1:__ Call up the documentation for the `norm` function imported above. (Hint: that documentation is [here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html)). What is the second listed method?

```python
#norm?
```

__A:__

The second-listed method for the `norm` object is the probability denstity function: `norm.pdf(x, loc=0, scale=1)`.

__Q2:__ Use the method that generates random variates to draw five samples from the standard normal distribution.

__A:__  Using the norm.rvs() method:

```python
seed(47)
# draw five samples here
norm_samples = norm.rvs(size=5)
print(norm_samples)
```

    [-0.84800948  1.30590636  0.92420797  0.6404118  -1.05473698]

__Q3:__ What is the mean of this sample? Is it exactly equal to the value you expected? Hint: the sample was drawn from the standard normal distribution. If you want a reminder of the properties of this distribution, check out p. 85 of *AoS*.

__A:__

I didn't expect any particular value because the sample size was small, but, yes, the mean is relatively close to zero.

```python
# Calculate and print the mean here, hint: use np.mean()
norm_samples_mean = norm_samples.mean()
print(norm_samples_mean)
```

    0.19355593334131074

__Q4:__ What is the standard deviation of these numbers? Calculate this manually here as $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}}$ (This is just the definition of **standard deviation** given by Professor Spiegelhalter on p.403 of *AoS*). Hint: np.sqrt() and np.sum() will be useful here and remember that numPy supports [broadcasting](https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).

__A:__  The standard deviation should be around 1:

```python
manual_std = np.sqrt(
    np.sum(
        ((norm_samples - norm_samples_mean) ** 2)
    ) /
    len(norm_samples)
)
assert manual_std == np.std(norm_samples)
print(manual_std)
```

    0.9606195639478641

Here we have calculated the actual standard deviation of a small data set (of size 5). But in this case, this small data set is actually a sample from our larger (infinite) population. In this case, the population is infinite because we could keep drawing our normal random variates until our computers die!

In general, the sample mean we calculate will not be equal to the population mean (as we saw above). A consequence of this is that the sum of squares of the deviations from the _population_ mean will be bigger than the sum of squares of the deviations from the _sample_ mean. In other words, the sum of squares of the deviations from the _sample_ mean is too small to give an unbiased estimate of the _population_ variance. An example of this effect is given [here](https://en.wikipedia.org/wiki/Bessel%27s_correction#Source_of_bias). Scaling our estimate of the variance by the factor $n/(n-1)$ gives an unbiased estimator of the population variance. This factor is known as [Bessel's correction](https://en.wikipedia.org/wiki/Bessel%27s_correction). The consequence of this is that the $n$ in the denominator is replaced by $n-1$.

You can see Bessel's correction reflected in Professor Spiegelhalter's definition of **variance** on p. 405 of *AoS*.

__Q5:__ If all we had to go on was our five samples, what would be our best estimate of the population standard deviation? Use Bessel's correction ($n-1$ in the denominator), thus $\sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}}$.

__A:__  To calculate the unbiased estimator, we set `ddof=1` or normalize the sum of squares by N-1:

```python
manual_sample_std = np.sqrt(
    np.sum(
        ((norm_samples - norm_samples_mean) ** 2)
    ) /
    (len(norm_samples) - 1)
)
assert manual_sample_std == np.std(norm_samples, ddof=1)
print(manual_sample_std)
```

    1.0740053227518152

__Q6:__ Now use numpy's std function to calculate the standard deviation of our random samples. Which of the above standard deviations did it return?

__A:__ Numpy's default is to use the biased estimator (`ddof=0`). Note that this behavior is **diffferent** from Pandas, which normalizes by N-1 by default! In a way, this makes sense: a numeric library should not guess at what you are trying to do, but in Pandas, we are typically working with real-world samples, so the sample standard deviation is what we usually want.

```python
print(np.std(norm_samples))
```

    0.9606195639478641

__Q7:__ Consult the documentation for np.std() to see how to apply the correction for estimating the population parameter and verify this produces the expected result.

__A:__

```python
np.std(norm_samples, ddof=1)
```

    1.0740053227518152

```python
if manual_sample_std == np.std(norm_samples, ddof=1):
    print("Yes, setting ddof=1 normalizes by N-1!")
```

    Yes, setting ddof=1 normalizes by N-1!

### Summary of section

In this section, you've been introduced to the scipy.stats package and used it to draw a small sample from the standard normal distribution. You've calculated the average (the mean) of this sample and seen that this is not exactly equal to the expected population parameter (which we know because we're generating the random variates from a specific, known distribution). You've been introduced to two ways of calculating the standard deviation; one uses $n$ in the denominator and the other uses $n-1$ (Bessel's correction). You've also seen which of these calculations np.std() performs by default and how to get it to generate the other.

You use $n$ as the denominator if you want to calculate the standard deviation of a sequence of numbers. You use $n-1$ if you are using this sequence of numbers to estimate the population parameter. This brings us to some terminology that can be a little confusing.

The population parameter is traditionally written as $\sigma$ and the sample statistic as $s$. Rather unhelpfully, $s$ is also called the sample standard deviation (using $n-1$) whereas the standard deviation of the sample uses $n$. That's right, we have the sample standard deviation and the standard deviation of the sample and they're not the same thing!

The sample standard deviation
\begin{equation}
s = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n-1}} \approx \sigma,
\end{equation}
is our best (unbiased) estimate of the population parameter ($\sigma$).

If your dataset _is_ your entire population, you simply want to calculate the population parameter, $\sigma$, via
\begin{equation}
\sigma = \sqrt{\frac{\sum_i(x_i - \bar{x})^2}{n}}
\end{equation}
as you have complete, full knowledge of your population. In other words, your sample _is_ your population. It's worth noting that we're dealing with what Professor Spiegehalter describes on p. 92 of *AoS* as a **metaphorical population**: we have all the data, and we act as if the data-point is taken from a population at random. We can think of this population as an imaginary space of possibilities.

If, however, you have sampled _from_ your population, you only have partial knowledge of the state of your population. In this case, the standard deviation of your sample is not an unbiased estimate of the standard deviation of the population, in which case you seek to estimate that population parameter via the sample standard deviation, which uses the $n-1$ denominator.

Great work so far! Now let's dive deeper.

## 3. Sampling distributions

So far we've been dealing with the concept of taking a sample from a population to infer the population parameters. One statistic we calculated for a sample was the mean. As our samples will be expected to vary from one draw to another, so will our sample statistics. If we were to perform repeat draws of size $n$ and calculate the mean of each, we would expect to obtain a distribution of values. This is the sampling distribution of the mean. **The Central Limit Theorem (CLT)** tells us that such a distribution will approach a normal distribution as $n$ increases (the intuitions behind the CLT are covered in full on p. 236 of *AoS*). For the sampling distribution of the mean, the standard deviation of this distribution is given by

\begin{equation}
\sigma_{mean} = \frac{\sigma}{\sqrt n}
\end{equation}

where $\sigma_{mean}$ is the standard deviation of the sampling distribution of the mean and $\sigma$ is the standard deviation of the population (the population parameter).

This is important because typically we are dealing with samples from populations and all we know about the population is what we see in the sample. From this sample, we want to make inferences about the population. We may do this, for example, by looking at the histogram of the values and by calculating the mean and standard deviation (as estimates of the population parameters), and so we are intrinsically interested in how these quantities vary across samples.

In other words, now that we've taken one sample of size $n$ and made some claims about the general population, what if we were to take another sample of size $n$? Would we get the same result? Would we make the same claims about the general population? This brings us to a fundamental question: _when we make some inference about a population based on our sample, how confident can we be that we've got it 'right'?_

We need to think about **estimates and confidence intervals**: those concepts covered in Chapter 7, p. 189, of *AoS*.

Now, the standard normal distribution (with its variance equal to its standard deviation of one) would not be a great illustration of a key point. Instead, let's imagine we live in a town of 50,000 people and we know the height of everyone in this town. We will have 50,000 numbers that tell us everything about our population. We'll simulate these numbers now and put ourselves in one particular town, called 'town 47', where the population mean height is 172 cm and population standard deviation is 5 cm.

```python
seed(47)
pop_heights = norm.rvs(172, 5, size=50000)
```

```python
_ = plt.hist(pop_heights, bins=30)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in entire town population')
_ = plt.axvline(172, color='r')
_ = plt.axvline(172+5, color='r', linestyle='--')
_ = plt.axvline(172-5, color='r', linestyle='--')
_ = plt.axvline(172+10, color='r', linestyle='-.')
_ = plt.axvline(172-10, color='r', linestyle='-.')
```

Now, 50,000 people is rather a lot to chase after with a tape measure. If all you want to know is the average height of the townsfolk, then can you just go out and measure a sample to get a pretty good estimate of the average height?

```python
def townsfolk_sampler(n):
    return np.random.choice(pop_heights, n)
```

Let's say you go out one day and randomly sample 10 people to measure.

```python
seed(47)
daily_sample1 = townsfolk_sampler(10)
```

```python
_ = plt.hist(daily_sample1, bins=10)
_ = plt.xlabel('height (cm)')
_ = plt.ylabel('number of people')
_ = plt.title('Distribution of heights in sample size 10')
```

The sample distribution doesn't resemble what we take the population distribution to be. What do we get for the mean?

```python
np.mean(daily_sample1)
```

    173.47911444163503

And if we went out and repeated this experiment?

```python
daily_sample2 = townsfolk_sampler(10)
```

```python
np.mean(daily_sample2)
```

    173.7317666636263

__Q8:__ Simulate performing this random trial every day for a year, calculating the mean of each daily sample of 10, and plot the resultant sampling distribution of the mean.

__A:__

```python
def townsfolk_sample_mean(n):
    return np.mean(townsfolk_sampler(n))
```

Note that the supplied function above `townsfolk_sampler()` samples *with* replacement, which is probably not what we want here. The CLT still applies, of course; it's just always good to be aware of defaults!

```python
seed(47)
# take your samples here
townsfolk_sample_means = np.array([townsfolk_sample_mean(10) for _ in range(365)])
```

```python
plt.figure(figsize=(10,8))
_ = plt.hist(townsfolk_sample_means, bins='auto', histtype='step')
_ = plt.xlabel('mean sample height (cm)')
_ = plt.ylabel('Number of samples')
_ = plt.title('Distribution of 365 sample means')
_ = plt.axvline(172, color='r', label='pop. mean')
_ = plt.axvline(172+5, color='r', linestyle='--')
_ = plt.axvline(172-5, color='r', linestyle='--', label='+/- 1 pop. sd')
_ = plt.axvline(np.mean(townsfolk_sample_means), color='dodgerblue', label='sample mean')
_ = plt.legend()
```

The above is the distribution of the means of samples of size 10 taken from our population. The Central Limit Theorem tells us the expected mean of this distribution will be equal to the population mean, and standard deviation will be $\sigma / \sqrt n$, which, in this case, should be approximately 1.58.

__Q9:__ Verify the above results from the CLT.

__A:__

```python
print(f"""
The mean of our sample means is {np.mean(townsfolk_sample_means):.2f} cm.
The standard deviation of our sample means is {np.std(townsfolk_sample_means):.2f} cm.
""")
```

    The mean of our sample means is 171.87 cm.
    The standard deviation of our sample means is 1.58 cm.

Remember, in this instance, we knew our population parameters, that the average height really is 172 cm and the standard deviation is 5 cm, and we see some of our daily estimates of the population mean were as low as around 168 and some as high as 176.

__Q10:__ Repeat the above year's worth of samples but for a sample size of 50 (perhaps you had a bigger budget for conducting surveys that year)! Would you expect your distribution of sample means to be wider (more variable) or narrower (more consistent)? Compare your resultant summary statistics to those predicted by the CLT.

__A:__

```python
seed(47)
# calculate daily means from the larger sample size here
townsfolk_sample_means_50 = np.array([townsfolk_sample_mean(50) for _ in range(365)])
```

```python
print(f"""
Given 365 samples of 50 people:
The mean of our sample means is {np.mean(townsfolk_sample_means_50):.2f}.
The standard deviation of our sample means is {np.std(townsfolk_sample_means_50):.2f}.
""")
```

    Given 365 samples of 50 people:
    The mean of our sample means is 171.94.
    The standard deviation of our sample means is 0.67.

What we've seen so far, then, is that we can estimate population parameters from a sample from the population, and that samples have their own distributions. Furthermore, the larger the sample size, the narrower are those sampling distributions.

### Normally testing time!

All of the above is well and good. We've been sampling from a population we know is normally distributed, we've come to understand when to use $n$ and when to use $n-1$ in the denominator to calculate the spread of a distribution, and we've  seen the Central Limit Theorem in action for a sampling distribution. All seems very well behaved in Frequentist land. But, well, why should we really care?

Remember, we rarely (if ever) actually know our population parameters but we still have to estimate them somehow. If we want to make inferences to conclusions like "this observation is unusual" or "my population mean has changed" then we need to have some idea of what the underlying distribution is so we can calculate relevant probabilities. In frequentist inference, we use the formulae above to deduce these population parameters. Take a moment in the next part of this assignment to refresh your understanding of how these probabilities work.

Recall some basic properties of the standard normal distribution, such as that about 68% of observations are within plus or minus 1 standard deviation of the mean. Check out the precise definition of a normal distribution on p. 394 of *AoS*.

__Q11:__ Using this fact, calculate the probability of observing the value 1 or less in a single observation from the standard normal distribution. Hint: you may find it helpful to sketch the standard normal distribution (the familiar bell shape) and mark the number of standard deviations from the mean on the x-axis and shade the regions of the curve that contain certain percentages of the population.

__A:__ The probability of observing a value of 1 or less in a single observation from the standard normal distribution is about **84%** (50% probability of less than 0 + 68% / 2 = 34% probabliity of within one standard deviation above the mean.

Calculating this probability involved calculating the area under the curve from the value of 1 and below. To put it in mathematical terms, we need to *integrate* the probability density function. We could just add together the known areas of chunks (from -Inf to 0 and then 0 to $+\sigma$ in the example above). One way to do this is to look up tables (literally). Fortunately, scipy has this functionality built in with the cdf() function.

__Q12:__ Use the cdf() function to answer the question above again and verify you get the same answer.

__A:__

```python
print(f"The probability observing a value less than 1 in a single observation from the standard normal distribution is {norm.cdf(1) * 100:.1f}%.")
```

    The probability observing a value less than 1 in a single observation from the standard normal distribution is 84.1%.

__Q13:__ Using our knowledge of the population parameters for our townsfolks' heights, what is the probability of selecting one person at random and their height being 177 cm or less? Calculate this using both of the approaches given above.

__A:__  The probability of selecting one person at random and their height being 177 cm or less is the probability of that observation being less than or equal to one standard deviation above the mean, which is about 84%, using the same reasoning as above.

```python
print(f"""The probability of selecting one person at random
and their height being 177 cm or less is {norm.cdf(177, loc=172, scale=5) * 100:.1f}%.
""")
```

    The probability of selecting one person at random
    and their height being 177 cm or less is 84.1%.

__Q14:__ Turning this question around — suppose we randomly pick one person and measure their height and find they are 2.00 m tall. How surprised should we be at this result, given what we know about the population distribution? In other words, how likely would it be to obtain a value at least as extreme as this? Express this as a probability.

__A:__

```python
print(f"""
The probability of selecting one person at random
and their height being 200 cm or more is {(1 - norm.cdf(200, loc=172, scale=5))};
in other words, an outlier.""")
```

    The probability of selecting one person at random
    and their height being 200 cm or more is 1.0717590259723409e-08;
    in other words, an outlier.

What we've just done is calculate the ***p-value*** of the observation of someone 2.00m tall (review *p*-values if you need to on p. 399 of *AoS*). We could calculate this probability by virtue of knowing the population parameters. We were then able to use the known properties of the relevant normal distribution to calculate the probability of observing a value at least as extreme as our test value.

We're about to come to a pinch, though. We've said a couple of times that we rarely, if ever, know the true population parameters; we have to estimate them from our sample and we cannot even begin to estimate the standard deviation from a single observation.

This is very true and usually we have sample sizes larger than one. This means we can calculate the mean of the sample as our best estimate of the population mean and the standard deviation as our best estimate of the population standard deviation.

In other words, we are now coming to deal with the sampling distributions we mentioned above as we are generally concerned with the properties of the sample means we obtain.

Above, we highlighted one result from the CLT, whereby the sampling distribution (of the mean) becomes narrower and narrower with the square root of the sample size. We remind ourselves that another result from the CLT is that _even if the underlying population distribution is not normal, the sampling distribution will tend to become normal with sufficiently large sample size_. (**Check out p. 199 of AoS if you need to revise this**). This is the key driver for us 'requiring' a certain sample size, for example you may frequently see a minimum sample size of 30 stated in many places. In reality this is simply a rule of thumb; if the underlying distribution is approximately normal then your sampling distribution will already be pretty normal, but if the underlying distribution is heavily skewed then you'd want to increase your sample size.

__Q15:__ Let's now start from the position of knowing nothing about the heights of people in our town.
* Use the random seed of 47, to randomly sample the heights of 50 townsfolk
* Estimate the population mean using np.mean
* Estimate the population standard deviation using np.std (remember which denominator to use!)
* Calculate the (95%) [margin of error](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/hypothesis-testing/margin-of-error/#WhatMofE) (use the exact critial z value to 2 decimal places - [look this up](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/find-critical-values/) or use norm.ppf()) Recall that the ***margin of error*** is mentioned on p. 189 of the *AoS* and discussed in depth in that chapter).
* Calculate the 95% Confidence Interval of the mean (***confidence intervals*** are defined on p. 385 of *AoS*)
* Does this interval include the true population mean?

__A:__

```python
seed(47)
# take your sample now
single_sample_50 = townsfolk_sampler(50)
n = len(single_sample_50)
```

```python
estimate = np.mean(single_sample_50)
print(f"Our estimate of the population mean is {estimate:.2f} cm.")
```

    Our estimate of the population mean is 172.78 cm.

```python
sd = np.std(single_sample_50, ddof=1)
print(f"Our estimate of the population standard deviation is {sd:.2f} cm.")
```

    Our estimate of the population standard deviation is 4.20 cm.

```python
se = sd / np.sqrt(n)
print(f"The standard error of our estimate is {se:.3f} cm.")
print(f"The 95% critical z-value is {norm.ppf(0.975):.2f}.")
print(f"Using the normal distribution, the 95% confidence interval is {np.round(estimate + norm.ppf([0.025, 0.975]) * se, 2)} cm.")
```

    The standard error of our estimate is 0.593 cm.
    The 95% critical z-value is 1.96.
    Using the normal distribution, the 95% confidence interval is [171.62 173.94] cm.

__Q16:__ Above, we calculated the confidence interval using the critical z value. What is the problem with this? What requirement, or requirements, are we (strictly) failing?

__A:__ By using the z-value, we are failing to account for the uncertainty in using 50 data points to estimate the standard error. By using the t-distribution, as below, which has heavier tails than the normal distribution for low degrees of freedom, we can account for this uncertainty in our estimate.

__Q17:__ Calculate the 95% confidence interval for the mean using the _t_ distribution. Is this wider or narrower than that based on the normal distribution above? If you're unsure, you may find this [resource](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/confidence-interval/) useful. For calculating the critical value, remember how you could calculate this for the normal distribution using norm.ppf().

__A:__

```python
se = sd / np.sqrt(n)
print(f"The standard error of our estimate is {se:.3f} cm.")
print(f"The 95% critical t-value for {n-1} degrees of freedom is {t.ppf(0.975, n-1):.2f}.")
print(f"Using the t distribution, the 95% confidence interval is {np.round(estimate + t.ppf([0.025, 0.975], n-1) * se, 2)} cm.")
```

    The standard error of our estimate is 0.593 cm.
    The 95% critical t-value for 49 degrees of freedom is 2.01.
    Using the t distribution, the 95% confidence interval is [171.59 173.97] cm.

This is slightly wider than the previous confidence interval. This reflects the greater uncertainty given that we are estimating population parameters from a sample.

## 4. Learning outcomes

Having completed this project notebook, you now have hands-on experience:
* sampling and calculating probabilities from a normal distribution
* identifying the correct way to estimate the standard deviation of a population (the population parameter) from a sample
* with sampling distribution and now know how the Central Limit Theorem applies
* with how to calculate critical values and confidence intervals


# Exercise 5
* Due: 10.2. at 14:00
* Max points: 4

## General rules
* **Submit your answers by using Moodle.** The link is https://moodle.jyu.fi/mod/assign/view.php?id=689680&forceview=1.
* Name your file as *lastname_ex5.xxx*. If you have multiple files, use *lastname_ex5_1.xxx, lastname_ex5_2.xxx* etc.
* **Remember to add comments to your answers and include also some testing part!**

## Exercises

1. **(2 points)** Solve the problem
$$
\begin{align}
\min   \  & x_1^2 + x_2^2\\
\text{s.t. } & x_1 + x_2 \geq 1.
\end{align}$$ by using just the optimality conditions.

2. **(2 points)** Consider a problem
$$\begin{align}
\min   \  & f(x)\\
\text{s.t. } & h_k(x)=0, \text{ for all } k=1,\dots,K,
\end{align}
$$

where all the functions are twice differentiable. Show, that the *gradient of the augmented Lagrangian function* is zero in the minimizer $x^*$ of the above problem. In other words, show that $\nabla_xL_c(x^*,\lambda^*)=0$, where $\lambda^*\in R^n$ is the corresponding optimal Lagrange multiplier vector.


# imports

```
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from lets_plot import *
```

```
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import scale
```

```
from scipy.stats import norm, uniform
```

```
from copy import copy
```

```
import random
```

```
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import roc_curve
```

```
def gg_confusion_matrix(y, y_hat):
    conf_mat = confusion_matrix(y, y_hat)[::-1]
    confusion_dat = pd.DataFrame(conf_mat)
    observed = confusion_dat.columns.values
    actual = confusion_dat.index.values
    xx, yy = np.meshgrid(actual, observed)
    xx = xx.reshape(-1)
    yy = yy.reshape(-1)
    zz = conf_mat.reshape(-1)
    dat = {'predicted':xx, 'actual':yy[::-1], 'z':zz}
    p = ggplot(dat, aes('predicted', 'actual', fill='z')) \
        + geom_raster() \
        + geom_text(aes(label='z'), color='white')\
        + theme(legend_position='none', axis_ticks='blank', axis_line='blank')\
        + ggsize(600, 600) + scale_x_discrete() + scale_y_discrete()\
        + ggtitle('Confusion matrix')
    return p
```

```
def intersect(a, b):
    return list(set(a) & set(b))
```

```
def plot_digit(digit, caption=None):
    digit = digit.reshape(28, 28)
    digit = (digit - np.min(digit))/(np.max(digit) - np.min(digit))
    p = ggplot() + geom_image(image_data=digit) + labs(x='', y='') \
        + theme(axis_line='blank', axis_title='blank', axis_ticks='blank', axis_text='blank')
    if caption:
        p += ggtitle(caption)
    return p
```

$\newcommand{\E}{\mathbb{E}}$
$\newcommand{\D}{\mathbb{D}}$
$\newcommand{\pr}{\mathbb{P}}$
$\newcommand{\I}{\mathbb{I}}$

$\newcommand{\Er}{\mathcal{E}}$
$\newcommand{\Xset}{\mathcal{X}}$
$\newcommand{\Yset}{\mathcal{Y}}$
$\newcommand{\L}{\mathcal{L}}$
$\newcommand{\l}{\mathcal{l}}$
$\newcommand{\pr}{\mathbb{P}}$
$\newcommand{\E}{\mathbb{E}}$
$\newcommand{\D}{\mathbb{D}}$
$\newcommand{\w}{\mathbf{w}}$
$\newcommand{\X}{\mathbf{X}}$
$\newcommand{\y}{\mathbf{y}}$
$\newcommand{\x}{\mathbf{x}}$
$\newcommand{\J}{\mathbf{J}}$
$\newcommand{\I}{\mathbf{I}}$
$\newcommand{\X}{\mathbf{X}}$
$\newcommand{\S}{\mathbf{S}}$
$\newcommand{\e}{\mathbf{e}}$
$\newcommand{\C}{\mathbf{C}}$
$\newcommand{\K}{\mathbf{K}}$
$\newcommand{\R}{\mathbb{R}}$
$\newcommand{\v}{\mathbf{v}}$
$\newcommand{\phivec}{\boldsymbol{\phi}}$
$\newcommand{\sign}{\mathrm{sign}}$
$\newcommand{\F}{\mathcal{F}}$

$\newcommand{\tg}{\mathrm{tg}}$
$\newcommand{\ctg}{\mathrm{ctg}}$
$\newcommand{\arctg}{\mathrm{arctg}}$

# Logistic regression

### Logistic regression and decision boundary
In this section we are going to consider the logistic regression from the point of view of construction decision boundaries
separating classes.
Suppose in the binary classification task
$\begin{align}
&\mathbb{P}\left(y=1 \;|\; x;\, \theta \right) = a_{\theta}(x)\\
&\mathbb{P}\left(y=0 \;|\; x;\, \theta \right) = 1 - a_{\theta}(x)
\end{align}$
Suppose we are going to find decision boundary that separates two classes:
$$\pr\left(y=0\;|\;x;\theta\right)=\pr\left(y=1\;|\;x;\theta\right)$$
Let's rewrite this statement in the logarithmic form
$$\log\left(\frac{\pr\left(y=0\;|\;x:\theta\right)}{\pr\left(y=1\;|\;x;\theta\right)}\right)=0$$
Consider the linear equation for the decision boundary:
$$\log\left(\frac{\pr\left(y=0\;|\;x:\theta\right)}{\pr\left(y=1\;|\;x;\theta\right)}\right)=\theta^Tx$$
Using the sigmoid function $g(z)$ where $z=\theta^Tx$ and $z^{(i)}=\theta^Tx^{(i)}$ we get that
$$\pr\left(y=1\;|\;x;\theta\right)=g(z)\\
\pr\left(y=0\;|\;x;\theta\right)=1-g(z)$$
Consider the log-likelihood function
$$\ell(\theta)=\sum_{i=1}^m\log p(y^{(i)}\;|\;x^{(i)};\theta)=\sum_{i=1}^m\log\left( g(z^{(i)})^{y^{(i)}}\left(1-g(z^{(i)})\right)^{1-y^{(i)}}\right)=\\
\sum_{i=1}^m\left(y^{(i)}\log g(z^{(i)}) + (1-y^{(i)})\log\left(1-g(z^{(i)})\right)\right)$$

**Gradient of the loss function**
Consider the loss function $J(a,y;\theta)$ as a function of $\theta$: $J(a,y;\theta)=\ell(\theta)$
Let $a_{\theta}(x)=g(z)=g(\theta^Tx)$, where $g(z)$ - sigmoid function.
Set $m=1$ for simplicity and the function takes form
$$\displaystyle \ell(\theta)=y\log g(\theta^Tx)+(1-y)\log(1-g(\theta^Tx))$$
Take the derivative:
$$
\frac{\partial \ell(\theta)}{\partial \theta_j}=\left(\frac{y}{g(z)}g(z)(1-g(z))-\frac{1-y}{1-g(z)}g(z)(1-g(z))\right)\frac{\partial \theta^Tx}{\partial \theta_j}
=\left(y-g(\theta^Tx)\right)x_j
$$

**Stochastic gradient ascent**
As far as we are working with the likelihood function - our aim is to find it's maximum and therefore we
are going to implement gradient ascent:
$$\boxed{\;\theta_j \leftarrow \theta_j+\eta\left(y^{(i)}-g(\theta^Tx^{(i)})\right)x_j^{(i)}\;}$$

```
def sigmoid(x, slope=1.0):
    return 1.0 / (1.0 + np.exp(-slope * x))
```

```
def sigmoid_prime(x, slope=1.0):
    return slope * sigmoid(x, slope=slope) * (1.0 - sigmoid(x, slope=slope))
```

```
def check_gradients(f, grad, z, eps=1e-4, type='central', rtol=1e-5):
    if type == 'forward':
        num_grad = (f(z + eps) - f(z)) / (eps)
    elif type == 'backward':
        num_grad = (f(z) - f(z - eps)) / (eps)
    elif type == 'central':
        num_grad = (f(z + eps) - f(z - eps)) / (2 * eps)
    else:
        raise NotImplementedError
    rel_tol = np.abs(num_grad - grad(z))/(1. + np.minimum(np.abs(num_grad), np.abs(grad(z))))
    return np.all(rel_tol < rtol), np.max(rel_tol)
```

```
z = np.random.normal(size=10000)
check_gradients(sigmoid, sigmoid_prime, z, type='central', rtol=1e-5)
```

    (True, 1.6717289512841335e-10)

```
def step_function(x, margin = 0, label=[0,1]):
    return np.where(x >= margin, label[1], label[0])
```

### Problem 12 (0.5)
Recap: binary cross-entropy between `target` $y$ and `output` $a$:
$$\displaystyle J(a, y;\theta)=-\sum_{i=1}^m\left(y^{(i)}\log(a_{\theta}^{(i)})+(1-y^{(i)})\log(1-a_{\theta}^{(i)})\right)$$
where `target` $y^{(i)}$ is 0 or 1 and `output` is $a_{\theta}^{(i)}\in [0,1]$
The implementation of the binary cross entropy function used in the lecture notes had a computational drawback.
Describe the problem and fix it using some small value $\varepsilon$.
Fill in the implementation below.

```
def binary_cross_entropy(output, target, eps=1e-15):
    """ Binary cross-entropy loss function
    Args:
        output - the predicted model output
        target - targer values
        eps - tolerance parameter
    Returns:
        bce - loss function values
    """
    #### your code (start)
    sum_score = sum(target[i] * np.log(eps + output[i]) for i in range(len(target)))
    mean_sum_score = 1.0 / len(target) * sum_score
    #### your code (end)
    return -mean_sum_score
```

### Problem 13 (0.5)
Explain, what happens to the trained weights $\theta$ of the logistic regression, when the classes are linearly separable.
Offer how to fix the logistic regession model in this case. Fill in the implementation of the `logistic_regression`
function.
Do we have to regularize the bias term? Write why not or why yes.

```
def add_intercept(X_):
    m, _ = X_.shape
    X = np.hstack((np.ones(m)[:,np.newaxis], X_))
    return X
```

```

def logistic_regression(X, y, eta=0.001, iter=100, alpha=0.5):
    """Logistic regression
    Args:
        X - feature matrix
        y - target vector
        eta - learning rate
        iter - number of iterations
        alpha - regularization parameter
    Returns:
        theta - weight estimates
        losses - historical loss data
        thetas - historical thetas data
    """
    _, n = X.shape
    theta = np.zeros(n)
    losses = np.zeros(iter)
    thetas = np.zeros(iter * 3).reshape(iter, 3)
    #### your code (start)

    for i in range(iter):
        s = sigmoid(X.dot(theta))
        gradient = X.T.dot(y - s)
        gradient[1:] = [gradient[i] + alpha * theta[i] for i in range(1, len(gradient))]
        theta = [theta[i] + gradient[i] * eta for i in range(len(theta))]
        thetas[i] = np.array(theta, copy=True)
        losses[i] = binary_cross_entropy(s, y)

    #### your code (end)
    return theta, losses, thetas
```

```
def predict(X, theta):
    y_hat = sigmoid(X.dot(theta))
    return step_function(y_hat, 0.5)
```

```
# add simulation for toy example
m = 1000
xi1 = norm.rvs(1, 1, size=2 * m).reshape(m, 2)
xi2 = norm.rvs(10, 5, size=2 * m).reshape(m, 2)
y1 = [0] * m
y2 = [1] * m
X = np.vstack((xi1, xi2))
y = np.hstack((y1, y2))
```

```
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)
dat = {'x1': X_train.T[0], 'x2': X_train.T[1], 'variable': y_train}
```

```
dat = pd.DataFrame(dat)
dat['color'] = ['red' if variable == 0 else 'blue' for variable in dat['variable']]
```

```
p = ggplot(dat, aes(x='x1', y='x2', group='variable', color='color')) \
        + geom_point() + theme(legend_position='none')
p
```

```
X_train = add_intercept(X_train)
```

```
theta, losses, thetas = logistic_regression(X_train, y_train, eta=0.001, iter=100, alpha=20.)
```

```
X_test = add_intercept(X_test)
```

### Problem 14 (0.5)
Derive the expression for the decision boundary. Compute `intercept` and `slope` values,
where `x2 = intercept + slope*x1` in terms of `theta`.

```
#### Write your code here (start)
intercept = -theta[0] / theta[2]
slope = -theta[1] / theta[2]
#### Write your code here (end)
```

```
p += geom_abline(intercept=intercept, slope=slope, color='red', linetype='dashed', size=2)
p
```

```
thetas = np.array(thetas)
```

```
theta_dat = pd.DataFrame(thetas, columns = ['theta0', 'theta1', 'theta2'])
theta_dat['t'] = np.arange(len(theta_dat['theta0']))
theta_names = list(theta_dat.columns[:-1])
```

```
def plot_theta(name):
    return ggplot(theta_dat)+ geom_line(aes('t', name))
```

```
bunch = GGBunch()
x0, y0 = 0, 0
row = 0
col = 0
width = 350
height = 250
n_cols = 3
for name in theta_names:
    if col == n_cols:
        col = 0
        row += 1
    plot = plot_theta(name)
    bunch.add_plot(plot, x0 + col*width, y0 + row*height, width, height)
    col += 1
bunch.show()
```

### Problem 15 (1.0)
Let $\theta\in\mathbb{R}$ and $J(\theta)\in\mathbb{R}$. Consider two-sided numerical gradient
$$\frac{d J(\theta)}{d\theta}\approx\frac{J(\theta+\varepsilon)-J(\theta-\varepsilon)}{2\varepsilon}$$
We can generalize the above formula for the case of $\theta\in\mathbb{R}^{n}$:
$$\frac{\partial}{\partial\theta_1}J(\theta)\approx\frac{J(\theta_1+\varepsilon,\theta_2,\ldots,\theta_n)-J(\theta_1-\varepsilon,\theta_2,\ldots,\theta_n)}{2\varepsilon};\\
\frac{\partial}{\partial\theta_2}J(\theta)\approx\frac{J(\theta_1,\theta_2+\varepsilon,\ldots,\theta_n)-J(\theta_1,\theta_2-\varepsilon,\ldots,\theta_n)}{2\varepsilon};\\
\vdots\\
\frac{\partial}{\partial\theta_n}J(\theta)\approx\frac{J(\theta_1,\theta_2,\ldots,\theta_n+\varepsilon)-J(\theta_1,\theta_2,\ldots,\theta_n-\varepsilon)}{2\varepsilon}$$
Implement the `gradient_checker` function, which takes some function `J` and it's gardient `grad_J` as functions of `theta` and
checks if the relative tolerance `rel_tol` greater than prespecified threshold value `rtol` for each each axis.
This function has to correctly process not only cases when $\theta\in\mathbb{R}^n$, but also the case $\theta\in\mathbb{R}$.

Relative tolerance for two values $x$ and $y$ is $\displaystyle\boxed{\;\frac{|x-y|}{\min\left(|x|, |y|\right)}\;}$

```
def to_scalar(x, i):
    """Returns i-th element from an array x if x is array, else returns x
    Args:
        x - input array
        i - index to return
    Returns:
        x[i] if x is an array and x else
    """
    if isinstance(x, (np.ndarray, list)):
        return x[i]
    return x
```

Use `np.nditer` to iterate over dimensions. If you are going to perform an operation `theta + eps` along some axis,
don't forget to make a copy of `theta`:
```
theta_ = np.array(theta, copy=True)
```

```
def gradient_checker(J, grad_J, theta, eps=1e-4, rtol=1e-5):
    """Gradient checker for scalar and vector functions
    Args:
        J - function of theta
        grad_J - gradient of function J
        theta - the point for which to compute the numerical gradient
        eps - step value in numerical gradient
        rtol - relative tolerance threshold value
    Returns:
        error message if the relative tolerance is greater for some axis
        or "Gradient check passed" else
    """
    it = np.nditer(theta, flags=['multi_index'], op_flags=['readwrite'])

    while not it.finished:
        ix = it.multi_index

        #### your code (start)
        r_grad = to_scalar(grad_J(np.array(theta, copy=True)), ix)
        arr = np.arrange(theta.shape[0])
        ieps = eps * (arr == ix)
        js = J(theta + ieps) - J(theta - ieps)
        js /=  eps * 2
        l_grad = to_scalar(js, ix)
        rel_tol = np.abs(l_grad - r_grad) / np.min([l_grad, r_grad])
        #### your code (end)

        if rel_tol > rtol:
            print(f'Incorrect gradient for the axis {str(ix)}')
            return
        it.iternext()
    print(f'Gradient check passed')
```

Check the gradient of sigmoid function.

```
z = np.random.normal(size=5)
gradient_checker(sigmoid, sigmoid_prime, z, eps=1e-4, rtol=1e-5)
```

Check the gradient of the binary cross entropy function.

Note that both `loss_function` and `loss_grad` are defined as functions of `theta`.

```
def loss_function(theta, y=y_train, X=X_train):
    y_hat = sigmoid(X.dot(theta))
    return binary_cross_entropy(y_hat, y)
```

```
def loss_grad(theta, y=y_train, X=X_train):
    y_hat = sigmoid(X.dot(theta))
    grad = (y - y_hat).dot(X)
    return -grad
```

```
m, n = X_train.shape
theta = np.random.normal(scale=0.1, size=n)
gradient_checker(loss_function, loss_grad, theta)
```

    Incorrect gradient for the axis (0,)

# Sigmoid

### Problem 16 (0.5)
Show that
$$y\log g(\theta^Tx)+(1-y)\log\left(1-g(\theta^Tx)\right)=y\theta^Tx-\log\left(1+\exp(\theta^Tx)\right),$$
where $\displaystyle g(z)=\frac{1}{1+\exp(-z)}$ - sigmoid function.

### Solution:

If $a = \theta^T x$, \
then $(1 - y) \log (1 - g(a)) + y \log(g(a)) = ya - \log (1 + \exp(a))$

$ya - \log(1 + \exp(a)) = (1 - y) \log \left({1 \over a + \exp(-1)}\right) + y \log \left({1 \over a + \exp(-1)}\right) = $

$ = (1 - y) \log \left({\exp(-a) \over a + \exp(-1)}\right) - y (\log(1) - \log(1 + \exp(-a)) =$

$ = (1 - y) (-a - \log(1 + \exp(-a))) - y \log(1 + \exp(-a))$.

$ya - \log (1 + \exp(a)) = (1 - y) (-a - \log(1 + \exp(-a))) - y \log(1 + \exp(-a))$

$-\log(1 + \exp(a)) = (1 - y) (- \log(1 + \exp(-a))) - y \log(1 + \exp(-a)) - a = -\log(1 + \exp(-a)) - a$

$\exp(a) = \log\left({1 + \exp(a) \over 1 + \exp(-a)}\right)$

$\exp(a) + 1 = 1 + \exp(a)$.

# Adversarial examples

### Adversarial examples: Fast Gradient Sign Method
See the article [EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES](https://arxiv.org/pdf/1412.6572.pdf) for the details.
Let the $\theta$ be the parameters of the model with inputs $x$ and target values $y$. The loss function of the model
denote as $J(\theta,x,y)$ then the FGS method of construction of the adversarial examples is:
$$x\leftarrow \mathrm{clip}\left(x+\eta\;\mathrm{sign}\nabla_x J(\theta,x,y)\right)$$
where the function $\mathrm{clip}$ is used to clip each dimension of $x$ to the range $[0,1]$.
Below we have the trained softmax regression.

```
def sigmoid(x,slope=1.0):
    return 1.0 / (1.0 + np.exp(-slope * x))
```

```
def step_function(x, margin = 0, label=[0, 1]):
    return np.where(x >= margin, label[1], label[0])
```

```
def mini_batches(X, y, batch_size):
    m, _ = X.shape
    rand_index = np.random.choice(m, size=m)
    X_, y_ = X[rand_index], y[rand_index]
    pos = 0
    while pos < m:
        X_batch, y_batch = X_[pos:pos+batch_size], y_[pos:pos+batch_size]
        yield (X_batch, y_batch)
        pos += batch_size
```

```
def softmax(x):
    a = np.amax(x, axis=1)[:, np.newaxis]
    ex = np.exp(x - a)
    ex_sum = np.sum(ex, axis=1)[:, np.newaxis]
    x = ex / ex_sum
    return x
```

```
def one_hot_encoder(labels, n_classes=10):
    encoded = np.zeros((labels.size, n_classes))
    encoded[np.arange(labels.size), labels.astype(int)] = 1
    return encoded
```

```
def cross_entropy_loss(output, labels, eps=1e-5):
    logs = np.log(output + eps)
    return np.multiply(-labels, logs).sum(axis=1).mean()
```

```
X = pd.read_csv('mnist_data.csv').values[:, 1:]
y = pd.read_csv('mnist_target.csv').values[:, 1]
```

```
X = X / 255
X_train, X_test, y_train, y_test = train_test_split(X, y)
```

```
def softmax_regression(X, y, n_classes=10, eta=0.001, iter=100):
    _, n = X.shape
    theta = np.zeros((n, n_classes))
    losses = np.zeros(iter)
    for i in range(iter):
        loss = 0
        for batch in mini_batches(X, y, 1000):
            X_, y_ = batch
            y_ = one_hot_encoder(y_, n_classes)
            y_hat = softmax(X_.dot(theta))
            grad = X_.T.dot(y_ - y_hat)
            theta += grad*eta
            loss += cross_entropy_loss(y_hat, y_)
        losses[i] = loss
    return theta, losses
```

```
def predict(X, theta):
    y_hat = softmax(X.dot(theta))
    return np.argmax(y_hat, axis=1), y_hat
```

```
theta, losses = softmax_regression(X_train, y_train, eta=0.0001, iter=100)
```

```
def softmax(x):
    if len(x.shape) > 1:
        a = np.amax(x, axis=1)[:, np.newaxis]
        ex = np.exp(x - a)
        ex_sum = np.sum(ex, axis=1)[:, np.newaxis]
        x = ex / ex_sum
    else:
        a = np.amax(x)
        ex = np.exp(x - a)
        ex_sum = np.sum(ex)
        x = ex / ex_sum
    return x
```

```
def predict(x, theta):
    y_hat = softmax(theta.T.dot(x))
    return np.argmax(y_hat), y_hat
```

### Problem 17 (1.0)
Implement the fast gradient sign example function below. Explain what you see looking at the confusion matrix after applying the attack.

```
def fast_adversarial_example(dig, theta, n=784, n_classes=10, eta=0.01, iters=10):
    y = np.zeros(n_classes)
    y[dig] = 1
    x = np.random.normal(loc=0, scale=0.01, size=n)
    for _ in range(iters):
        # write your code here (start)
        s = np.sign((y - softmax(theta.T @ x)).T @ theta.T)
        x = np.clip(x + eta * s, 0, 1)
        # write your code here (end)
    return x
```

```
example = fast_adversarial_example(8, theta, eta=0.1, iters=5)
```

```
def predict(X, theta):
    y_hat = softmax(X.dot(theta))
    return np.argmax(y_hat, axis=1), y_hat
```

```
alpha = 0.5
```

```
X_spoiled = np.maximum(X_test*(1-alpha), alpha*example)
```

```
y_test_model, test_score = predict(X_spoiled, theta)
```

```
gg_confusion_matrix(y_test, y_test_model)
```

```
x_target = X_spoiled[0]
```

```
def predict(x, theta):
    y_hat = softmax(theta.T.dot(x))
    return np.argmax(y_hat), y_hat
```

```
y_pred, score = predict(x_target.T, theta)
```

```
bunch = GGBunch()
plot = plot_digit(x_target, 'Predicted digit: '+str(y_pred))
bunch.add_plot(plot, 0, 0)
digit_labels = np.arange(10)
dat = pd.DataFrame({'labels':digit_labels, 'score':score})
plot = ggplot(dat) + geom_histogram(aes('labels', 'score'), stat='identity') + scale_x_discrete()
bunch.add_plot(plot, 500, 0)
bunch.show()
```

# Fisher information

## Problem 18 (1.0)
#### Fisher information matrix
Show, that under the regularity conditions (that we can interchange differentiation and integration) we have, that $$\displaystyle \boxed{I(\theta;\mathrm{X})=\E\left(\frac{\partial}{\partial\theta}\log L(\theta;\mathrm{X})\right)^2=-\E\left(\frac{\partial^2}{\partial\theta^2}\log L(\theta;\mathrm{X})\right)}$$

### Solution:

# Dataset distillation

### Dataset distillation
Based on the paper [Dataset Distillation](https://arxiv.org/abs/1811.10959).
Consider training set $\{x^{(i)}\}_{i=1}^m$ and some loss function $\ell(x^{(i)},\theta)$, where $\theta$ - model parameters. Find empirical error minimizer:
$$\theta^{*}=\arg\min_{\theta}\frac{1}{m}\sum_{i=1}^m\ell(x^{(i)},\theta)$$

Find distilled training data $\{\tilde{x}^{(i)}\}_{i=1}^{\tilde{m}}$, such that $\tilde{m}\ll m$ with the help of gradient descent:
$$\theta_1=\theta_0-\tilde{\eta}\nabla_{\theta_0}\ell(\tilde{x},\theta_0)$$
define $\displaystyle \mathcal{L}(x,\eta\;|\;\theta_0)=\ell(x,\theta_1)$ and set distilled data to be
$$\tilde{x}^*,\tilde{\eta}^*=\arg\min_{\tilde{x},\tilde{\eta}}\mathcal{L}\left(\tilde{x},\tilde{\eta}\;|\;\theta_0\right)=\arg\min_{\tilde{x},\tilde{\eta}}\ell(x,\theta_1)=\arg\min_{\tilde{x},\tilde{\eta}}\ell\left(x,\theta_0-\tilde{\eta}\nabla_{\theta_0}\ell(\tilde{x},\theta_0)\right)$$
Actually you can perform several SGD steps, updating distilled data $\tilde{x}$ and corresponding learning rate $\tilde{\eta}$:
$$
\tilde{x} \leftarrow \tilde{x} - \alpha\nabla_{\tilde{x}}\mathcal{L}\\
\tilde{\eta} \leftarrow \tilde{\eta} - \alpha\nabla_{\tilde{\eta}}\mathcal{L},
$$
where $\alpha$ - learning rate of your distillation algorithm.

### Algorithm:
**Input:** $\tilde{m}$ - distilled dataset size, $\alpha$ - learning rate, $T$ - number of iterations, $\theta_0$ - initial value for $\theta$ (taken from some distribution $p(\theta_0)$), $\tilde{\eta}_0$ - initial value for $\tilde{\eta}$
1. Initialize $\tilde{x}$ randomly, set $\tilde{\eta} \leftarrow \tilde{\eta}_0$
2. For each step $t$ in $[1, T]$:
3. Sample a batch of real training data $x_t$
4. Compute updated parameters with SGD on distilled data: $\displaystyle \theta_1 = \theta_0-\tilde{\eta}\nabla_{\theta_0}\ell(\tilde{x},\theta_0)$
5. Evaluate the objective function on real training data: $\displaystyle \mathcal{L}=\ell(x_t,\theta_1)$
6. Update distilled data parameters:
$$
\tilde{x} \leftarrow \tilde{x} - \alpha\nabla_{\tilde{x}}\mathcal{L}\\
\tilde{\eta} \leftarrow \tilde{\eta} - \alpha\nabla_{\tilde{\eta}}\mathcal{L},
$$

### Gradients
* We are dealing with a classification task, so we can take cross-entropy as an objective function
$$\ell(x,\theta)=-\sum_jy^{(j)}\log(\hat{y}^{(j)}),$$
where $\displaystyle \hat{y}=\mathrm{softmax}(z)$ and $z=\theta^Tx$. We already know, that $\displaystyle \boxed{\;\frac{\partial \ell}{\partial \theta}=X^T(Y-\hat{Y})\;}$

* We also have to take derivatives $\displaystyle \nabla_{x}\mathcal{L}$ and $\displaystyle \nabla_{\eta}\mathcal{L}$
Consider $\mathcal{L}(x,\eta\;|\;\theta_0)=\ell(x,\theta_1)$ and rewrite the cross-entropy loss in the next form:
$$-\sum_jy^{(j)}\log\hat{y}^{(j)}=-\sum_jy^{(j)}\log\left(\mathrm{softmax}\left(\underbrace{\left(\underbrace{\theta_0-\tilde{\eta}\tilde{x}^T(\tilde{y}-\hat{\tilde{y}})}_{=\theta_1}\right)^Tx}_{=z}\right)\right)$$
and we can notice that
$$
\frac{\partial \mathcal{L}}{\partial \tilde{x}}=\frac{\partial\mathcal{L}}{\partial z}\frac{\partial z}{\partial \tilde{x}}\\
\frac{\partial \mathcal{L}}{\partial \tilde{\eta}}=\frac{\partial\mathcal{L}}{\partial z}\frac{\partial z}{\partial \tilde{\eta}}
$$
where, as we already know $\displaystyle \frac{\partial \mathcal{L}}{\partial z}=\hat{y}-y$
Consider
$$
\frac{\partial z}{\partial \tilde{x}}=\frac{\partial}{\partial \tilde{x}}\left(\left(\theta_0-\tilde{\eta}\tilde{x}^T(\tilde{y}-\hat{\tilde{y}})\right)^Tx\right)
$$

### Problem 19 (2.5)

Derive expressions for $\displaystyle \nabla_{\tilde{x}}\mathcal{L}$ and $\displaystyle \nabla_{\tilde{\eta}}\mathcal{L}$

### Solution:

### Read MNIST dataset and perform train/test split

```
X = pd.read_csv('mnist_data.csv').values[:, 1:]
y = pd.read_csv('mnist_target.csv').values[:, 1]
```

```
X = X / 255
```

```
X_train, X_test, y_train, y_test = train_test_split(X, y)
```

### Problem 19 (cont)
Fill in the gaps in the implementation of the distillation algorithm below:

```
def dataset_distill(X, y, n=784, n_classes=10, alpha=0.1, eta=0.01, num_epochs=10):
    xd = np.random.normal(loc=0, scale=0.01, size=(n, n_classes))
    yd = np.eye(n_classes)
    theta = np.random.normal(loc=0, scale=0.01, size=(n, n_classes))

    grad_x = np.zeros_like(xd)
    grad_eta = 0

    for _ in range(num_epochs):
        for batch in mini_batches(X, y, 1000):
            # compute updated parameter on distilled data
            yd_hat = softmax(xd.T.dot(theta))
            grad = xd.dot(yd - yd_hat)
            theta += grad*eta
            # evaluate the objective function on real training data
            X_, y_ = batch
            y_ = one_hot_encoder(y_, n_classes)
            y_hat = softmax(X_.dot(theta))

            # write your code here (start)

            # write your code here (end)
            xd -= alpha * grad_x
            eta -= alpha * grad_eta
    return xd, eta
```

```
xd, eta = dataset_distill(X_train, y_train, eta=0.001, num_epochs=100)
```

```
images = xd.T
bunch = GGBunch()
size = 200
cols = 5
col = 0
for i, image in enumerate(images):
    col = i // cols
    row = i % cols
    plot = plot_digit(image.reshape(28, 28)) + ggsize(size, size)
    bunch.add_plot(plot, row * size, col * size)
bunch.show()
```

```
yd = np.arange(10)
```

```
def softmax_regression(X, y, n_classes=10, eta=0.001, num_epochs=100):
    _, n = X.shape
    theta = np.zeros((n, n_classes))
    losses = np.zeros(num_epochs)
    for i in range(num_epochs):
        y_ = one_hot_encoder(y, n_classes)
        y_hat = softmax(X.dot(theta))
        grad = X.T.dot(y_ - y_hat)
        theta += grad * eta
        loss = cross_entropy_loss(y_hat, y_)
        losses[i] = loss
    return theta, losses
```

```
theta, losses = softmax_regression(xd.T, yd, eta=eta, num_epochs=400)
```

```
num_epochs = len(losses)
```

```
losses_data = pd.DataFrame({'epoch': np.arange(num_epochs),'loss':losses})
```

```
other_options = xlim(0, num_epochs) + ggsize(500, 300) + ggtitle('Loss')
ggplot(losses_data) + geom_area(aes(x='epoch',y='loss'), alpha=0.2) + other_options
```

```
def predict(X, theta):
    y_hat = softmax(X.dot(theta))
    return np.argmax(y_hat, axis=1), y_hat
```

```
y_test_model, test_score = predict(X_test, theta)
```

```
gg_confusion_matrix(y_test, y_test_model)
```

```
def f1_score_micro(conf_matrix):
    num_tags = conf_matrix.shape[0]
    score = 0.
    pr, p, r = 0., 0., 0.
    for tag in range(num_tags):
        pr += conf_matrix[tag, tag]
        p += sum(conf_matrix[tag, :])
        r += sum(conf_matrix[:, tag])
    try:
        score = 2 * pr / (p + r)
    except ZeroDivisionError:
        pass
    return score
```

```
cm = confusion_matrix(y_test, y_test_model)
score = f1_score_micro(cm)
print(f'F1 score micro: {score}')
```

    F1 score micro: 0.040342857142857146

### Let's take some random digits for each class

```
def first(a, value):
    return next(i for i, x in enumerate(a) if x == value)
```

```
def find_first(a, values):
    ids = []
    for value in values:
        ids += [first(a, value)]
    return ids
```

```
values = np.arange(10)
```

```
ids = find_first(y_train, values)
```

```
ys = y_train[ids]
xs = X_train[ids]
```

```
theta, losses = softmax_regression(xs, ys, num_epochs=400)
```

```
num_epochs = len(losses)
```

```
losses_data = pd.DataFrame({'epoch': np.arange(num_epochs),'loss':losses})
```

```
other_options = xlim(0, num_epochs) + ggsize(500, 300) + ggtitle('Loss')
ggplot(losses_data) + geom_area(aes(x='epoch',y='loss'), alpha=0.2) + other_options
```

```
y_test_model, test_score = predict(X_test, theta)
```

```
gg_confusion_matrix(y_test, y_test_model)
```

```
cm = confusion_matrix(y_test, y_test_model)
score = f1_score_micro(cm)
print(f'F1 score micro: {score}')
```

    F1 score micro: 0.3958857142857143

# info

### The due date is $\color{blue}{9}$ of March 2022 23:59:59

### Instructions
* To submit the assignment share your workbook to me. Make sure that you provide me "Can edit" access.

* Check, that you found all the tasks in the workbook.


# Physics 256
## Lecture 10 - More Numpy and Matplotlib

```python
import style
style._set_css_style('../include/bootstrap.css')
```

## Last Time

### [Notebook Link: 09_NumpyMatplotlib.ipynb](./09_NumpyMatplotlib.ipynb)

- multidimensional arrays with numpy
- introduction to plotting with matplotlib

## Today
- plot customization
- different plot types

## Plotting with Matplotlib

Here we load a *style* file that includes many nice plotting defaults.  Have a look at it and try changing some of the values.

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

plt.style.use('../include/notebook.mplstyle');
%config InlineBackend.figure_format = 'svg'
```

 Individual Programming Challenge

Make a plot of the Gaussian distribution

\begin{equation}
y(x) = \frac{1}{\sqrt{2\pi}} \mathrm{e}^{-x^2/2}
\end{equation}

### Customizing

There are multitudinous ways to customize the look of your plots
Note the use of LaTeX in the legend.  This works in all captions and labels by using raw strings with `r'$$'`

### Plot options

We can add keyword options to control the properties of lines and markers in our plots.  Check out:

```python
from IPython.display import HTML
HTML('')
```

```python
# For all the details
HTML('')
```

### Putting it all together

Note the use of LaTeX to add math labels.  I've included many options here so you can come back to this code and see how to change things in your own plots.

```python
theta = np.linspace(0,2*np.pi,20)
sinlabel = r'$\sin(\theta)$'
coslabel = r'$\cos(\theta)$'
plt.figure(figsize=(8,6))
plt.plot(theta,np.sin(theta),label=sinlabel,color=(0.0,0.0,0.6),linestyle='-',linewidth=2.5,marker='o',markeredgecolor='green',\
    markerfacecolor='#0FDDAF',markersize=12.0,markeredgewidth=0.8)
plt.plot(theta,np.cos(theta),label=coslabel,color='red',linestyle='None',\
         linewidth=0.5,marker='s',markeredgecolor='None',\
         markerfacecolor='pink',markersize=15.0)
plt.xlabel(r'$\theta$')
plt.xlim(0,2*np.pi)
plt.legend(loc='lower left', fontsize=15)
```

 Team programming challenge

 Given a list of all marker types produce a plot of a family parabolas showing them off.

`marker_list = ['o','s','p','^','v','>','<','D','*','H']`

```python
marker_list = ['o','s','p','^','v','>','<','D','*','H']
x = np.arange(-1,1,0.1)

# generate and plot the parabola
for i,marker in enumerate(marker_list):
    # generate the parabola

    # generate a label

    # plot the parabola for each marker type
    plt.plot()

# some labels
plt.xlabel('$x$')
plt.ylabel('$y(x)$')
plt.title('A Family of Parabolas $y=x^2+a$')

# add a legend
plt.legend(loc='upper center', ncol=3)
```

## Color Cycle

You can get access to the default color cycle via the following dictionary

```python
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
print(colors)
```

```python

```


# Telescope design

How do we design a telescope that magnifies a deflected laser beam to illuminate an objective?

```python
from matplotlib import pyplot as plt

plt.style.use('bodo')

colors = plt.get_cmap('tab20').colors
```

Given focal length $f_1$ of the first telescope lens, focal length $f_2$ of the second telescope lens and effective focal length $f_o$ of the objective, the ABCD transfer matrix of this system reads,
$$
\begin{pmatrix}
-f_2/f_1 & -f_of_1/f_2 \\
f_2/(f_of_1)& 0
\end{pmatrix},
$$
by introducting the magnification factor $M=-f_2/f_1$ we can rewrite the matrix as,
$$
\begin{pmatrix}
M & -f_o/M \\
M/f_o & 0
\end{pmatrix}.
$$

```python
from optics.geometrical import ThinLens, Propagation

def system_abcd(f1, f2, fo):
    P1 = Propagation(f1).abcd()
    L1 = ThinLens(f1).abcd()

    P2 = Propagation(f2).abcd()
    L2 = ThinLens(f2).abcd()

    Po = Propagation(fo).abcd()
    Lo = ThinLens(fo).abcd()

    return Lo * Po * P2 * L2 * P2 * P1 * L1 * P1
```

For a geometrical ray with distance $y=0$ from the optical axis and angle $\theta$ with the optical axis, we can read off the position offset at the objective,
$$y^\prime=-\frac{f_0}{M}\theta.$$

```python
def magnification(f1, f2):
    return - f2 / f1

def offset(f, M, theta):
    return f * theta / M
```

We consider two deflector types,

* ecousto-optic

$$
\begin{align}
    a_a= 5\text{ mm}, && \theta_a=8\text{ mrad},
\end{align}
$$

* electro-optic

$$
\begin{align}
    a_e=0.5\text{ mm}, && \theta_e=20\text{ mrad},
\end{align}
$$

wherein $a$ denotes the aperture radius and $\theta$ the maximum deflection angle of the respective deflector type. We fix the effective focal length of the objective to
$$f_o=25\text{ mm},$$
and assume an aperture radius of the objective of
$$a_o=20\text{ mm}.$$

```python
# numerical aperture of objective
NA = 0.8

# effective focal length of objective
fo = 25e-3

# aperture radius of objective
ao = 20e-3

# maximum aperture radius of acousto-optic and electro-optic deflector
aa = 5.0e-3
ae = 0.5e-3

# maximum deflection angle of acousto-optic and electro-optic deflector
thetaa = 8e-3
thetae = 20e-3

# laser wavelength
lamda = 532e-9
```

The spot size in the focal plane can be estimated by,
$$s(w_0,a_o)=\frac{0.97\lambda a_o}{2 w_0 \mathit{NA}}\sqrt{\frac{e}{1-\exp(-a_o^2/w_0^2)}-1}.$$

```python
def spot_size(w0):
    T = w0 / ao

    return 0.97 * lamda * np.sqrt(np.exp(1)/(1 - np.exp(-1 / T**2)) - 1) / (2 * T * NA)
```

We need to specify a value range for the magnification factor $M$. From the other notebook we know that it is desirable to illuminate the complete lens surface in order to achieve a small focal spot size.

```python
M = np.linspace(1, 40)

plt.figure(figsize=(12, 6))
plt.xlabel(r'$M$')
plt.ylabel(r'${w_0}^\prime$ (\si{\milli\meter})')
plt.plot(M, M * 0.5, label=r'$w_0=\SI{0.5}{\milli\meter}$', color=colors[0])
plt.plot(M, M * 1.0, label=r'$w_0=\SI{1.0}{\milli\meter}$', color=colors[1])
plt.plot(M, M * 2.5, label=r'$w_0=\SI{2.5}{\milli\meter}$', color=colors[2])
plt.plot(M, M * 5.0, label=r'$w_0=\SI{5.0}{\milli\meter}$', color=colors[3])
plt.axhline(20, linestyle='dotted')
plt.ylim(0, 30)
plt.legend(loc='upper center', bbox_to_anchor=(.5, -0.12), ncol=4)
plt.show()
```

In order to illuminate the complete objective surface we need a magnification of $M=40$ for the electro-optic deflector aperture. For the acousto-optic deflector we can assume a beam waist in between the orange lines, thus we would require a magnification of $M=10$ for the acousto-optic deflector to illuminate the complete objective surface.

```python
M = np.linspace(2, 20)

plt.figure(figsize=(12, 6))

plt.xlabel(r'$M$')
plt.ylabel(r'$s$ (\si{\micro\meter})')
plt.plot(M, spot_size(M * 0.5e-3) / 1e-6, label=r'$w_0=\SI{0.5}{\milli\meter}$', color=colors[0])
plt.plot(M, spot_size(M * 1.0e-3) / 1e-6, label=r'$w_0=\SI{1.0}{\milli\meter}$', color=colors[1])
plt.plot(M, spot_size(M * 2.5e-3) / 1e-6, label=r'$w_0=\SI{2.5}{\milli\meter}$', color=colors[2])
plt.plot(M, spot_size(M * 5.0e-3) / 1e-6, label=r'$w_0=\SI{5.0}{\milli\meter}$', color=colors[3])
plt.axhline(1, linestyle='dotted')
plt.ylim(0, 5)
plt.legend(loc='upper center', bbox_to_anchor=(.5, -0.12), ncol=4)
plt.show()
```

The focus spot should be about $1\mu m$. This would require a magnification of $M=17$ for the electro-optic defelector and $M=3$ for the acousto-optic deflector.

```python
M = np.linspace(2, 20)

plt.figure(figsize=(12, 6))

plt.xlabel(r'$M$')
plt.ylabel(r'$y^\prime$ (\si{\micro\meter})')
plt.plot(M, offset(fo, M, thetae) / 1e-6, label=r'EOD ($\theta_0=\SI{20}{\milli\radian}$)', color=colors[0])
plt.plot(M, offset(fo, M, thetaa) / 1e-6, label=r'AOD ($\theta_a=\SI{8}{\milli\radian}$)', color=colors[2])
#plt.ylim(0, 400)
plt.legend(loc='upper center', bbox_to_anchor=(.5, -0.12), ncol=4)
plt.show()
```

From the diagrams we can limit the magnification range for both cases.

```python
Ma = np.linspace(3, 28)
Me = np.linspace(15, 40)

fig, ax1 = plt.subplots(figsize=(12, 6))

ax1.set_xlabel(r'$M$')
ax1.set_ylabel(r'$s$ (\si{\micro\meter})')
ax1.plot(Ma, spot_size(Ma * 3.7e-3) / 1e-6, label=r'AOD ($w_0=\SI{3.7}{\milli\meter}$)', color=colors[0])
ax1.plot(Me, spot_size(Me * 0.5e-3) / 1e-6, label=r'EOD ($w_0=\SI{0.5}{\milli\meter}$)', color=colors[1])
ax1.axhline(1, label=r'\SI{1}{\micro\meter}', linestyle='dotted', color=colors[0])
ax1.yaxis.label.set_color(colors[0])

plt.legend(loc='upper center', bbox_to_anchor=(.5, +1.12), ncol=3)

ax2 = ax1.twinx()
ax2.set_ylabel(r'$2y^\prime$ (\si{\micro\meter})')
ax2.plot(Ma, 2*offset(fo, Ma, thetaa) / 1e-6, label=r'AOD ($w_0=\SI{3.7}{\milli\meter}$)', color=colors[2])
ax2.plot(Me, 2*offset(fo, Me, thetae) / 1e-6, label=r'EOD ($w_0=\SI{3.7}{\milli\meter}$)', color=colors[3])
ax2.axhline(30, label=r'\SI{30}{\micro\meter}', linestyle='dotted', color=colors[2])
ax2.yaxis.label.set_color(colors[2])

plt.legend(loc='upper center', bbox_to_anchor=(.5, -0.12), ncol=3)

plt.show()
```

We will only consider the AOD case as the EOD would require a too large magnification. Furthermore we will limit ourselves to the case $w_0=3.7\text{ mm}$ inicdent beam waist.

```python
w0 = 3.7e-3
```

Now we want to check what diameter the telescope lenses should have to avoid clipping of the Gaussian beams. The transfer matrix of to the first telescope lense is $$\begin{pmatrix}0 & f_1\\ -1/f_1 & 1\end{pmatrix},$$ therefore the offset and angle are given by $$\begin{align}y^\prime&=f_1\theta\\\theta^\prime&=\theta-y/f_1\end{align}.$$

Using these, we can estimate the lens diameter of the first telescope lens through, $$D_1=2f_1\theta_\text{max}+4Mw_0,$$ where the first term can be obtained from the transfer matrix from deflector to first telescope lens and the second term should ensure large enough spacing to avoid diffraction.

```python
from optics.beam import GaussianBeam

def diameter1(f1, theta, w0):
    T = ThinLens(f1).abcd() * Propagation(f1).abcd()

    return 2 * f1 * theta + 4 * w0

diameter1 = np.vectorize(diameter1)
```

```python
diameter1(25e-3, thetaa, w0), diameter1(50e-3, thetaa, w0), diameter1(100e-3, thetaa, w0)
```

    (array(0.0152), array(0.0156), array(0.0164))

For the second telescope lens the transfer matrix reads $$\begin{pmatrix}-f_2/f_1 & f_1\\0 &-f_1/f_2\end{pmatrix},$$ therefore the lens diameter is $$D_2=2f_1\theta_\text{max}+4Mw_0.$$

```python
def diameter2(f1, f2, theta, w0):
    T = ThinLens(f2).abcd() * Propagation(f1 + f2).abcd() * ThinLens(f1).abcd() * Propagation(f1).abcd()

    return 2 * f1 * theta + 4 * w0 * f2 / f1

diameter2 = np.vectorize(diameter2)
```

```python
diameter2(25e-3, 150e-3, thetaa, w0), diameter2(50e-3, 300e-3, thetaa, w0), diameter2(100e-3, 600e-3, thetaa, w0)
```

    (array(0.0892), array(0.0896), array(0.0904))