# Flipping Coins

A classic statistics experiment is simply counting how many "heads" and "tails" you observe when flipping a coin repeatedly. With a perfectly unbiased coin in a statistically perfect world, one might expect to count an equal number of heads and tails by flipping a coin hundreds of times.

However, the world we live in is far from statistically perfect. The real world is plagued with **statistical fluctuations**, meaning that measured data are not always equal to what you would expect. For example, if you were to flip a coin 100 times, you may observe something like 45 heads and 55 tails. The reason for this is most of our statistical expectations represent an upper boundary of probability. In other words, if we were to flip a coin infinitely many times, we would expect exactly half of those trials to be heads, and exactly half of those trials to be tails. Unfortunately (or fortunately depending on if you have hobbies or not), you cannot flip a coin infinitely many times. You can only flip a coin a given amount of times. In mathematical terms, the results of any coin flipping experiment will have a _discrete_ number of heads and a _discrete_ number of tails. Or, you'll have counted heads and tails from a set number of trials.

This sort of statistical problem where you run a number of trials is easily simulated using Python. However, modeling trials of real world situations subject to statistical fluctuations requires something included in Python known as the **random number generator**. So, before we move on to how we can simulate statistical fluctuations, we introduce random numbers and the random number generator.

## Random Numbers

First, let's take a look at some of those "random" numbers.

```python
# This is the python module which contains the random number generator
import random

# This prints 10 random numbers between 0 and 1
for i in range(10):
    print(random.uniform(0,1))
```

Run the above cell a few times, are the numbers the same? They probably aren't. Is the fact that those numbers aren't the same enough to say they are random? Certainly they appear to be from looking at them. Unfortunately, this is math, meaning we need to understand and define the properties that random number should have in order to determine if the random numbers from Python satisfy these requirements.

## Properties of Random Numbers

Suppose you have a sequence of $N$ random numbers $\{R\}$ with contents $\{r_1, r_2, ... , r_N\}$ where each element is a random number. What sort of properties should this sequence of numbers have? If this is truly a sequence of random numbers, it _must_ satisfy the following properties, which we will explain in greater detail:

1. Drawing any $r_i$ is equally probable and independent.
2. The sequence of random numbers is uniformly distributed.

"Drawing a value" in this scope means we're picking a number from our sequence of random numbers, but not removing it from the sequence (the sequence remains unchanged, we're simply "observing" the random number).

Let's look at these two properties in a little more detail.

### All Values Are Equally Probable and Independent

What this means is that if you were to select a number from your sequence of random numbers $\{r_1, r_2, ... , r_N\}$ at random, the probability of drawing any of those numbers is
$p(r_i) = \frac{1}{N}$
where $p(r_i)$ is the probability of selecting a number $r_i$. This probability is identical for all numbers within your set. More explicitly:

$p(r_1) = p(r_2) = ... = p(r_N) = \frac{1}{N}$

The independence property means that the if you draw a number from the set, it does not affect the probability of drawing other numbers, or even itself at a later time. This is because the sequence remains unchanged after you draw (observe) a number. This property leads directly to the second important properties of random numbers, discussed below.

### The Numbers are Uniformly Distributed

This simply means that there is no "trend" within the set of random numbers. If you were to plot the histogram of all your random numbers, they would produce a flat rectangle, or the uniform distribution:

\begin{equation}
P(r)  = \left\{
\begin{matrix}
p & a \leq x \leq b \\
0 & \text{otherwise}
\end{matrix}
\right.
\end{equation}

where $P(r)$ is the probability _distribution_ and $p$ is the probability of drawing any number between $a$ and $b$. It is important to note that this probability $p$ is the **same** probability for all numbers between $a$ and $b$. This function simply defines a horizontal line, meaning that all values from this distribution are equally probable.

So we have some definitions of random numbers, let's test Python's random number generator to see if these "random" numbers satisfy our two requirements. Below is a python widget that you can use to visualize random numbers generated with Python. This widget will draw $N$ random numbers (you get to define $N$) and bin those values into a histogram. If you're unsure what a histogram is, think of it as a bar graph where the bars are defined by counting how many number land within a specific range or "bin".

Certianly, the question remains of how exactly we're going to translate the definitions above into something that we can use in order to create such a histogram. Let's break down how we'll acomplish this into steps with the following flowchart

>
>A schematic diagram for the process required in order to test if the numbers created by the random number generator are uniformly distributed.

This flowchart explains the process required in order to create the histogram in the widget below. Essentially all we need to do is create those $N$ random numbers, place them in the appropriate bind, and then count the number of entires that each bin has. These counts will define the size of each bar in the histogram. We also note that we have converted these raw counts into a percentage by dividing the counts in each bin by $N$.

```python
N = 1000

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from IPython import display

def Random_Histogram(N):
    fig = plt.gcf()
    fig.show()
    fig.canvas.draw()
    run = 0
    plot_step = 1
    random_numbers = []

    for i in range(N):
        run += 1
        r = random.uniform(0,1)
        random_numbers.append(r)

        if run >= plot_step or i == N-1:
            # Speed the plot up as we go so we can see the variations at the beginning
            plot_step = i/2
            plt.gca().cla()
            plt.ylabel('Probability')
            plt.ylim([0,0.2])
            plt.xlim([-0.05,1.05])
            # This is to scale our boxes from counts to probability
            weights = np.ones_like(random_numbers)/float(len(random_numbers))
            plt.hist(random_numbers,
                     bins=10,
                     weights=weights,
                     label="Drawn Numbers",
                     edgecolor='black', linewidth=1.2)

            plt.title(" ".join([str(i+1), "Numbers From Set"]))
            plt.plot([0,0 ,1,1],[0,.1,.1,0], linewidth = 3, label = "Expected")
            plt.legend(loc='center left', bbox_to_anchor=(1, .8))
            # This helps us get ready for the creation of the new frame
            display.clear_output(wait=True)
            display.display(plt.gcf())
            # reset counter
            run = 0
    display.clear_output(wait=True)
# call the function we just defined
Random_Histogram(N)
```

That animated a histogram which fills with random numbers drawn from the set. The line shown on the plot is the "expected" output, a uniform distribution, that the random numbers should follow. You can change the value of $N$ and run the cell again to generate a different amount of random numbers.

### Questions
1. Does the histogram of random numbers line up with what you expect after drawing
    - 100 numbers?
    - 1000 numbers?
    - 75000 numbers?
    - Run those three test cases again, are they identical to the first time you created the histogram?
2. If the numbers you draw don't line up with the expected distribution does that mean our random numbers don't fit the expected curve using a small number of random numbers, does that mean the random number generator isn't behaving correctly? What if it doesn't line up when you include lots of points?
3. Define statistical variation in your own words and whether or not you see that in the animations above.
4. Do the numbers returned from the random number generator satisfy our requirements for random numbers? Why or why not?

### Relevance to Coin Flipping

The connection between generating random numbers in Python and flipping a coin isn't necessarily obvious. Why do we need to generate random numbers to write a code that simulates flipping coins for us? Well, the answer is we need some metric in order to decide which "coins" are heads, and which are tails. Unfortunately to model a real world situation, we need statistical variation, we can't just use our expected probabilities to decide the outcome of the coin toss. If we did that we would only be modeling the expected result after an infinite number of trials.

So, the next question is how can we use these random numbers generated by Python in order to simulate something like a coin toss? To explain, it is important to keep the uniform distribution in mind. If you were to flip a fair coin, unbiased for heads or tails, there is a 50% probability of observing heads, and a 50% probability of observing tails after any given coin toss. In terms of the uniform distribution and the random number generator, we could make the following plot to describe a coin toss

```python
plt.text(0.15, 5,"Heads",size=20)
plt.text(0.65, 5,"Tails", size=20)
plt.ylim([0,20])
plt.ylabel("Probability (%)", size=20)
plt.xlabel("Random Number Value",size=20)
plt.plot([0,0 ,1,1],[0,10,10,0], linewidth = 3, label = "Uniform Distribution")
plt.plot([0.5,0.5], [0,20], linewidth = 3, c='b')
plt.tick_params(axis='both', which='major', labelsize=18)
plt.show()
```

Let's take a moment to understand this plot. On the y-axis we have probability, and on the x-axis we have "Random Number Value". What this plot means is that for a given random number between 0 and 1 (the x value), we simply need to look at which side of the line our random number is on. If it's on the left, our simulated coin toss returns heads, and if it's on the right our simulated coin toss returns tails. As all random numbers between zero and one are equally probable, this graph shows that we have a 50% probability of either heads or tails, modeled well with the random number generator.

This idea can also be used to model a _biased_ coin, or a coin that favors either heads or tails. For example, suppose we have a coin with a 60 % probability to land with heads facing up. Using the uniform distribution we would have a plot that looks similar to this:

```python
plt.text(0.15, 5,"Heads",size=20)
plt.text(0.70, 5,"Tails", size=20)
plt.ylim([0,20])
plt.ylabel("Probability 100 (%)", size=20)
plt.xlabel("Random Number Value?",size=20)
plt.plot([0,0 ,1,1],[0,10,10,0], linewidth = 3, label = "Uniform Distribution")
plt.plot([0.6,0.6], [0,20], linewidth = 3, c='b')
plt.tick_params(axis='both', which='major', labelsize=18)
plt.show()
```

This plot is functionally identical to the one shown before it, but the probability of our simulated coin returning heads is 60%, and the probability of our simulated coin returning tails is 40%.

Any biased or unbiased coin can be simulated this way and that's exactly what the Python code below does. We have a known probability of returning heads in the range from 0 to 1. We draw a random number, and if that random number is less than the probability of heads we return heads, if it's greater we return tails. This process as a flowchart would appear as follows

>
>
> A potential workflow that would simulate flipping an unbiased coin $N$ times using the random number generator.

Rather than a flow chart, the portion of the above workflow in the green box is written as a simple python function below to demonstrate how straighforward this is to translate into code. hello world!

```python
def HeadsOrTails(probability_of_heads):
    # draw a random number between 0 and 1
    r = random.uniform(0,1)
    # check if that random number is greater than the input you provided
    if r > probability_of_heads:
        return 'tails'
    else:
        return 'heads'

HeadsOrTails(0.7)
```

Feel free to run the above code snippet as many times as you like for different coin tosses. Does it behave how you would expect?

Certainly however, this gets more interesting as we begin flipping the coin many times and counting the results to see if the coin is behaving as we'd expect. Below is a Python code that does exactly that by counting the amount of heads and tails it returns in a loop, and animating the results of all the coin tosses in an interactive widget. While the code below looks more complicated, it is doing exactly the same thing as the snippet above, except now it's counting and plotting.

```python
# Feel free to change these values
flips = 100
probability_of_heads = 0.5   # maximum is 1, minimum is 0

# define a coin tossing function
def CoinToss(flips, probability_of_heads):
    # initialize plot
    fig = plt.gcf()
    fig.show()
    fig.canvas.draw()
    head_count = 0
    tail_count = 0
    run = 0
    # Animating every flip would take too long, we'll only animate 100 frames
    plot_step = 0.05 * flips
    # just like previous example, but counting results
    for i in range(flips):
        r = random.uniform(0,1)
        if r > probability_of_heads:
            tail_count += 1
        else:
            head_count += 1
        run += 1
        # Plotting every frame would be slow,
        # we'll buffer it by only plotting so many and the last one
        if run >= plot_step or i == flips-1:
            # This is just for plotting, no coin flipping in this section.
            plt.gca().cla()
            plt.ylim([0, flips*max(probability_of_heads, 1-probability_of_heads) + flips/4])
            plt.ylabel("Count")
            plt.xticks(np.array([1,2]),["Heads", "Tails"])
            plt.bar([1,2],[head_count,tail_count],align='center',edgecolor='black',linewidth=2)
            plt.title(' '.join(["Trial Number:",str(i+1)]))
            # Put counts and percentages above the plot as it is animated
            plt.text(1-.2, head_count+.01*flips,
                     str(head_count)+" ("+ str(round(head_count/(i+1)*100,2))+"%)",fontweight='bold')
            plt.text(2-.2, tail_count+.01*flips,
                     str(tail_count)+" (" + str(round(tail_count/(i+1)*100,2))+"%)",fontweight='bold')
            # This helps us get ready for the creation of the new frame
            display.clear_output(wait=True)
            display.display(plt.gcf())
            # reset counter for if we want to plot or not
            run = 0
    display.clear_output(wait=True)
    plt.title(' '.join([str(flips),"Flips with",str(probability_of_heads),"Probability of Heads"]))
CoinToss(flips, probability_of_heads)
```

### Questions
1. Simulate a coin with:
    - 10 flips
    - 100 flips
    - 10000 flips
    - 100000 flips

Do you get the number of heads and tails you would expect? Why or why not? If there is a difference between the simulated value and your expected value, is that something to be concerned about? Why or why not?

2. Repeat the above with a biased coin, where the `probability_of_heads` is not 0.5.

### Conclusion

In this notebook we covered the idea of statistical fluctuations and how we can use the random number generator to model those statistical fluctuations. In doing so, we created a simulation of coin tosses for both fair and biased coins and observed the difference in counting statistics between the two. We learned that while the probability of a fair coin is 50/50, the observed counting statistics may vary slightly from this. This sort of model can be applied to nearly any statistical process, and can act as a primer for the idea of statistical simulations and Monte Carlo methods.

[](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)


```python
%load_ext autoreload
%autoreload 2

import matplotlib.pyplot as plt
%matplotlib inline

import torch
import torch.nn as nn
```

# Convolutions

Convolutions are a special kind of linear transformations. For each patch of neighboring pixels in an image, a convolution outputs a linear combination of the pixel values with the weights of the convolution filter *kernel*. Each of these outputs again represents a pixel value of the output image. One convolution kernel is of patch size $h \times w$. As we want one kernel for each input and output channel, the weights for a convolution layer are actually of size $C_{out} \times C_{in} \times h \times w$.

There are three properties which make convolutions very powerful:
- **parameter sharing:** The same parameters of the kernel are used in each patch of the image. This simplifies the model and additionally leads to a more faithful gradient on the weights.
- **locality:** The shape of the patches makes sure that we only use information from pixels located in the close neighborhood.
- **translational invariance:** No matter, on which patch we apply the convolution kernel, the weights stay the same and therefore the behaviour is independent of the absolute position of the patch in the image.

# Cross-correlation vs. Convolutions

In the context of Deep Learning people often refer to discrete cross-correlation as convolutions. In PyTorch the `torch.nn.Conv2d` module takes inputs of the size $C_{\text{in}} \times H \times W$ and calculates the discrete ross-correlation w.r.t. a kernel of size $C_{\text{out}} \times C_{\text{in}} \times h \times w$ to produce an output of size $C_{\text{out}} \times H' \times W'$. The spatial size of the output $H'$, $W'$ depends on the kernel size as well as on the padding and the stride values. For the case of no padding and stride the output values $z$ for input $x$ and kernel $k$ are calculated as
\begin{align}
z_{c, i, j} = \sum_{l=1}^{C_{\text{in}}} \sum_{m=0}^{h-1} \sum_{n=0}^{w-1} x_{l,i+m,j+n} \cdot k_{c,l,m+1,n+1}.
\end{align}
For the sake of notational simplicity we will use the term *'convolution'* in the following even though we actually mean cross-correlation.

# Convolutions in classical image processing task

The following tasks comprise a series of classical applications of convolutions and should give an impression of the their versatility and expressivenes.

## Edge detector [2 points]

We can implement a simple edge detector by approximating derivatives with Taylor's expansion formula and calculating the finite differences with a kernel filter. For an image pixel $u_{i,j}$ calculate the derivates $(u_x)_{i,j}$ as $u_{i,j+1}-u_{i,j}$ and $(u_y)_{i,j}$ as $u_{i+1,j}-u_{i,j}$ using dedicated convolution kernels.

```python
from torchvision.transforms.functional import to_tensor, to_grayscale
from PIL import Image
im_gray = to_tensor(to_grayscale(Image.open('foxes.png')))
im_gray = im_gray[None, ...]
print(im_gray.shape)
plt.imshow(im_gray[0, 0], cmap='gray', vmin=0., vmax=1.)
```

```python
########################################
# Set the values of the kernel in ker_x

ker_x = [[-1, 1], [0, 0]]
ker_x = torch.tensor(ker_x).float()
ker_x = ker_x[None, None, :, :]

########################################

conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2, 2), padding=(1, 1), bias=False)
conv.weight.data = ker_x

with torch.no_grad():
    res = conv(im_gray)

plt.imshow(res[0, 0], cmap='gray', vmin=-0.5, vmax=0.5)
torch.save(ker_x, 'ker_x')
```

```python
########################################
# Set the values of the kernel in ker_y

ker_y = [[-1, 0], [1, 0]]
ker_y = torch.tensor(ker_y).float()
ker_y = ker_y[None, None, :, :]

########################################

conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2, 2), padding=(1, 1), bias=False)
conv.weight.data = ker_y

with torch.no_grad():
    res = conv(im_gray)
print (res.shape)
plt.imshow(res[0, 0], cmap='gray', vmin=-0.5, vmax=0.5)
torch.save(ker_y, 'ker_y')
```

## Gaussian Blur Kernel [3 points]

A common method for removing noise from an image is applying a Gauss filter https://en.wikipedia.org/wiki/Gaussian_filter. Identifying the length of a pixel as one and setting $\sigma$ to 1, discretely approximate the Gauss kernel. Normalize your kernel's weights such that they sum up to $1$.

```python
im_noisy = im_gray + torch.randn_like(im_gray) * 25./255.

plt.imshow(im_noisy[0, 0], cmap='gray', vmin=0., vmax=1.)
```

```python
###########################################
# Set the values of the kernel in ker_gauss
# The spatial patch size is 5x5

sz = 2

lin = torch.linspace(-sz, sz, 2*sz+1)

x = lin
y = lin[..., None]

d = x ** 2 + y ** 2

ker_gauss = torch.exp(-(1./2) * d)
ker_gauss /= ker_gauss.sum()
ker_gauss = ker_gauss[None, None, :]
###########################################

conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(5, 5), padding=(1, 1), bias=False)
conv.weight.data = ker_gauss

with torch.no_grad():
    res = conv(im_noisy)

plt.imshow(res[0, 0], cmap='gray', vmin=0.0, vmax=1.0)
torch.save(ker_gauss, 'ker_gauss')
```

# Convolution implementation [6 points]

Implement a $2$-D convolution in `modules.py` and test your code below. Do not use pre-implemented code for calculating convolutions.

```python
from modules import Conv2d
```

```python
x = torch.randn(1, 3, 6, 8)
kernel = torch.randn(2, 3, 4, 5)

conv = Conv2d(kernel, padding=1, stride=3)
res = conv(x[0])
print(res.shape)

conv_gt = nn.Conv2d(3, 2, (4, 5), padding=1, stride=3, bias=False)
conv_gt.weight.data = kernel

print (conv_gt(x)[0].shape)
print (conv_gt(x).shape)
with torch.no_grad():
    res_gt = conv_gt(x)[0]
```

    torch.Size([2, 2, 2])
    torch.Size([2, 2, 2])
    torch.Size([1, 2, 2, 2])

```python
x = torch.ones(1,3,5,5)
kernel = torch.ones(2, 3, 3, 3)*2

conv = Conv2d(kernel, padding=1, stride=3)
res = conv(x[0])
print(res)

conv_gt = nn.Conv2d(3, 2, (3, 3), padding=1, stride=3, bias=False)
conv_gt.weight.data = kernel
print (conv_gt(x)[0].shape)
print (conv_gt(x)[0])

with torch.no_grad():
    res_gt = conv_gt(x)[0]

```

    tensor([[[24., 36.],
             [36., 54.]],

            [[24., 36.],
             [36., 54.]]])
    torch.Size([2, 2, 2])
    tensor([[[24., 36.],
             [36., 54.]],

            [[24., 36.],
             [36., 54.]]], grad_fn=<SelectBackward>)

```python
assert (res_gt - res).abs().max() < 1e-5
```

```python

```


# Simple model of epidemic dynamics: SIR
### Prof. Marco Arieli Herrera-Valdez,
#### Facultad de Ciencias, Universidad Nacional Autónoma de México
Created March 7, 2016

Let $x$, $y$, and $z$ represent the fraction of susceptibles, infected, and recovered individuals within a population. Assume homogeneous mixing with a probability of infection given a contact with an infected individual given by $\beta$ and an average removal time $\beta^{-1}$ from the infected group, by recovery or death due to infection. The population dynamics are given by
\begin{eqnarray}
\partial_t x &=& -\alpha xy
\\
\partial_t y &=& \left( \alpha x - \beta \right) y
\\
\partial_t x &=& \beta y
\end{eqnarray}

Notice that the population size does not matter because it is kept constant.

```python
#Import the necessary modules and perform the necessary tests
import scipy as sc
import pylab as gr
sc.test("all",verbose=0)
%matplotlib inline
```

    Running unit tests for scipy
    NumPy version 1.10.4
    NumPy relaxed strides checking option: False
    NumPy is installed in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/numpy
    SciPy version 0.17.0
    SciPy is installed in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scipy
    Python version 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
    nose version 1.3.7

    ----------------------------------------------------------------------
    Ran 1 test in 2.492s

    OK

Setup a python function that specifies the dynamics

```python
def SIR(U,t,p):
    x,y,z=U
    yNew= p["alpha"] * y * x
    zNew= p["beta"] * y
    dx = -yNew
    dy = yNew - zNew
    dz = zNew
    return dx, dy, dz
```

The function SIR above takes three arguments, $U$, $t$, and $p$ that represent the states of the system, the time and the parameters, respectively.

### Outbreak condition

The condition
\begin{equation}
\frac{\alpha}{\beta}x(t)>1 , \quad y>0
\end{equation}
defines a threshold for a full epidemic outbreak. An equivalent condition is
\begin{equation}
x>\frac{\beta}{\alpha }, \quad y>0
\end{equation}

Therefore, with the parameters $(\alpha,\beta)$=(0.5,0.1), there will be an outbreak if the initial condition for $x(t)>1/5$ with $y>0$.
Notice that the initial value for $z$ can be interpreted as the initial proportion of immune individuals within the population.

The dynamics related to the oubreak condition can be studied by defining a variable $B(t) = x(t) \alpha/\beta$, called by some authors "_effective reproductive number_". If $x(t)\approx 1$, the corresponding $B(t)$ is called "_basic reproductive number_", or $R_o$.

Let's define a python dictionary containing parameters and initial conditions to perform simulations.

```python
p={"alpha": 0.15, "beta":0.1, "timeStop":300.0, "timeStep":0.01 }
p["Ro"]=p["alpha"]/p["beta"]
p["sampTimes"]= sc.arange(0,p["timeStop"],p["timeStep"])
N= 1e4; i0= 1e1; r0=0; s0=N-i0-r0
x0=s0/N; y0=i0/N; z0=r0/N;
p["ic"]=[x0,y0,z0]
print("N=%g with initial conditions (S,I,R)=(%g,%g,%g)"%(N,s0,i0,r0))
print("Initial conditions: ", p["ic"])
print("B(0)=%g"%(p["ic"][0]*p["Ro"]))
```

    N=10000 with initial conditions (S,I,R)=(9990,10,0)
    Initial conditions:  [0.999, 0.001, 0.0]
    B(0)=1.4985

Integrate numerically and plot the results

```python
# Numerical integration
xyz= sc.integrate.odeint(SIR, p["ic"], p["sampTimes"], args=(p,)).transpose()
# Calculate the outbreak indicator
B= xyz[0]*p["alpha"]/p["beta"]
```

```python
# Figure
fig=gr.figure(figsize=(11,5))
gr.ioff()
rows=1; cols=2
ax=list()
for n in sc.arange(rows*cols):
    ax.append(fig.add_subplot(rows,cols,n+1))

ax[0].plot(p["sampTimes"], xyz[0], 'k', label=r"$(t,x(t))$")
ax[0].plot(p["sampTimes"], xyz[1], 'g', lw=3, label=r"$(t,y(t))$")
ax[0].plot(p["sampTimes"], xyz[2], 'b', label=r"$(t,z(t))$")
ax[0].plot(p["sampTimes"], B, 'r', label=r"$(t,B(t))$")
ax[0].plot([0, p["timeStop"]], [1,1], 'k--', alpha=0.4)
ax[1].plot(xyz[0], xyz[1], 'g', lw=3, label=r"$(x(t),y(t))$")
ax[1].plot(xyz[0], xyz[2], 'b', label=r"$(x(t),z(t))$")
ax[1].plot(xyz[0], B, 'r', label=r"$(x(t),B(t))$")
ax[1].plot([0, 1], [1,1], 'k--', alpha=0.4)
ax[0].legend(); ax[1].legend(loc="upper left")
gr.ion(); gr.draw()
```

Notice that $y$ reaches its maximum when $B(t)$ crosses 1. That is, the epidemic starts to wine down when the $B(t)<1$.

### Exercises:
Setup two simulations for which there is no outbreak, such that:

(a) The initial density of "immune" individuals is large enough to prevent an epidemic.

(b) The initial density of "immune" individuals is really small but there is no epidemic outbreak.

```python

```


# Programming and Data Analysis

> Homework 0

Kuo, Yao-Jen <yaojenkuo@ntu.edu.tw> from [DATAINPOINT](https://www.datainpoint.com)

## Instructions

- We've imported necessary modules at the beginning of each exercise.
- We've put necessary files(if any) in the working directory of each exercise.
- We've defined the names of functions/inputs/parameters for you.
- Write down your solution between the comments `### BEGIN SOLUTION` and `### END SOLUTION`.
- It is NECESSARY to `return` the answer, tests will fail by just printing out the answer.
- Do not use `input()` function, it will halt the notebook while running tests.
- Running tests to see if your solutions are right: Kernel -> Restart & Run All -> Restart and Run All Cells.
- You can run tests after each question or after finishing all questions.

```python
import unittest
```

## 01. Define a function named `convert_fahrenheit_to_celsius(x)` which converts Fahrenheit degrees to Celsius degrees.

\begin{equation}
Celsius^{\circ} C = (Fahrenheit^{\circ} F - 32) \times \frac{5}{9}
\end{equation}

- Expected inputs：a numeric `x`.
- Expected outputs：a numeric.

```python
def convert_fahrenheit_to_celsius(x):
    """
    >>> convert_fahrenheit_to_celsius(212)
    100.0
    >>> convert_fahrenheit_to_celsius(32)
    0.0
    """
    ### BEGIN SOLUTION
    C = (x - 32)*5/9
    return C
    ### END SOLUTION
```

## 02. Define a function named `calculate_bmi(height, weight)` which calculates BMI according to heights in meters and weights in kilograms.

\begin{equation}
BMI = \frac{weight_{kg}}{height_{m}^2}
\end{equation}

Source: <https://en.wikipedia.org/wiki/Body_mass_index>

- Expected inputs：2 numerics `height` and `weight`.
- Expected outputs：a numeric.

```python
def calculate_bmi(height, weight):
    """
    >>> calculate_bmi(216, 147) # Shaquille O'Neal in his prime
    31.507201646090532
    >>> calculate_bmi(206, 113) # LeBron James
    26.628334433028563
    >>> calculate_bmi(211, 110) # Giannis Antetokounmpo
    24.70744143213315
    """
    ### BEGIN SOLUTION
    BMI = weight / ((height * .01)**2)
    return BMI
    ### END SOLUTION
```

## 03. Define a function named `show_big_mac_index(country, currency, price)` which returns the Big Mac Index given a country, its currency, and the price of a Big Mac.

- Expected inputs：2 strings and a numeric.
- Expected outputs：a string.

```python
def show_big_mac_index(country, currency, price):
    """
    >>> show_big_mac_index('US', 'USD', 5.65)
    A Big Mac costs 5.65 USD in US.
    >>> show_big_mac_index('South Korea', 'Won', 6520)
    A Big Mac costs 6,520.00 Won in South Korea.
    >>> show_big_mac_index('Taiwan', 'NTD', 72)
    A Big Mac costs 72.00 NTD in Taiwan.
    """
    ### BEGIN SOLUTION
    sentence = "A Big Mac costs {:0,.2f} {} in {}.".format(price, currency, country)
    return sentence
    ### END SOLUTION
```

## 04. Define a function named `is_a_divisor(x, y)` which returns whether `x` is a is_a_divisor of `y` or not.

- Expected inputs：2 integers.
- Expected outputs：a boolean.

```python
def is_a_divisor(x, y):
    """
    >>> is_a_divisor(1, 3)
    True
    >>> is_a_divisor(2, 3)
    False
    >>> is_a_divisor(3, 3)
    True
    >>> is_a_divisor(1, 4)
    True
    >>> is_a_divisor(2, 4)
    True
    >>> is_a_divisor(3, 4)
    False
    >>> is_a_divisor(4, 4)
    True
    """
    ### BEGIN SOLUTION
    divisor = not(bool(y%x))
    return divisor
    ### END SOLUTION
```

## 05. Define a function named `contains_vowels(x)` which returns whether x contains one of the vowels: a, e, i, o, u or not.

- Expected inputs：a string.
- Expected outputs：a boolean.

```python
def contains_vowels(x):
    """
    >>> contains_vowels('pythn')
    False
    >>> contains_vowels('ncnd')
    False
    >>> contains_vowels('rtclt')
    False
    >>> contains_vowels('python')
    True
    >>> contains_vowels('anaconda')
    True
    >>> contains_vowels('reticulate')
    True
    """
    ### BEGIN SOLUTION
    if x.find('a') != -1:
        return True
    elif x.find('e') != -1:
        return True
    elif x.find('i') != -1:
        return True
    elif x.find('o') != -1:
        return True
    elif x.find('u') != -1:
        return True
    else:
        return False
    ### END SOLUTION
```

## Run tests!

Kernel -> Restart & Run All. -> Restart And Run All Cells.

```python
class TestHomeworkZero(unittest.TestCase):
    def test_01_convert_fahrenheit_to_celsius(self):
        self.assertAlmostEqual(convert_fahrenheit_to_celsius(212), 100.0)
        self.assertAlmostEqual(convert_fahrenheit_to_celsius(32), 0.0)
    def test_02_calculate_bmi(self):
        self.assertTrue(calculate_bmi(216, 147) > 31)
        self.assertTrue(calculate_bmi(216, 147) < 32)
        self.assertTrue(calculate_bmi(206, 113) > 26)
        self.assertTrue(calculate_bmi(206, 113) < 27)
        self.assertTrue(calculate_bmi(211, 110) > 24)
        self.assertTrue(calculate_bmi(211, 110) < 25)
    def test_03_show_big_mac_index(self):
        self.assertEqual(show_big_mac_index('US', 'USD', 5.65), 'A Big Mac costs 5.65 USD in US.')
        self.assertEqual(show_big_mac_index('South Korea', 'Won', 6520), 'A Big Mac costs 6,520.00 Won in South Korea.')
        self.assertEqual(show_big_mac_index('Taiwan', 'NTD', 72), 'A Big Mac costs 72.00 NTD in Taiwan.')
    def test_04_is_a_divisor(self):
        self.assertTrue(is_a_divisor(1, 2))
        self.assertTrue(is_a_divisor(2, 2))
        self.assertTrue(is_a_divisor(1, 3))
        self.assertFalse(is_a_divisor(2, 3))
        self.assertTrue(is_a_divisor(1, 4))
        self.assertTrue(is_a_divisor(2, 4))
        self.assertFalse(is_a_divisor(3, 4))
        self.assertTrue(is_a_divisor(4, 4))
    def test_05_contains_vowels(self):
        self.assertFalse(contains_vowels('pythn'))
        self.assertFalse(contains_vowels('ncnd'))
        self.assertFalse(contains_vowels('rtclt'))
        self.assertTrue(contains_vowels('python'))
        self.assertTrue(contains_vowels('anaconda'))
        self.assertTrue(contains_vowels('reticulate'))

suite = unittest.TestLoader().loadTestsFromTestCase(TestHomeworkZero)
runner = unittest.TextTestRunner(verbosity=2)
test_results = runner.run(suite)
number_of_failures = len(test_results.failures)
number_of_errors = len(test_results.errors)
number_of_test_runs = test_results.testsRun
number_of_successes = number_of_test_runs - (number_of_failures + number_of_errors)
```

    test_01_convert_fahrenheit_to_celsius (__main__.TestHomeworkZero) ... ok
    test_02_calculate_bmi (__main__.TestHomeworkZero) ... ok
    test_03_show_big_mac_index (__main__.TestHomeworkZero) ... ok
    test_04_is_a_divisor (__main__.TestHomeworkZero) ... ok
    test_05_contains_vowels (__main__.TestHomeworkZero) ... ok

    ----------------------------------------------------------------------
    Ran 5 tests in 0.006s

    OK

```python
print("You've got {} successes among {} questions.".format(number_of_successes, number_of_test_runs))
```

    You've got 5 successes among 5 questions.


# Complex numbers
*Roberto Di Remigio*, *Luca Frediani*

Complex numbers are extremely important in mathematics and physics. The concept of a
complex number was introduced as early as the 16th century in order to solve algebraic equations
without a real solution, such as:

\begin{equation}
    x^2 + 1 = 0
\end{equation}

If we _define_ the _imaginary unit_ as:

\begin{equation}
    \mathrm{i}^2 = -1
\end{equation}

then the quadratic equation above admits the two, purely imaginary, solutions $+\mathrm{i}$ and $-\mathrm{i}$.

In general, a complex number is any entity that can be written in the form:

\begin{equation}
 z = a +\mathrm{i}b = \Re(z) +\mathrm{i}\Im(z)
\end{equation}

where $a, b$ are real numbers and are called the _real_ and _imaginary_ part of the complex number $z$.
Apart from all the usual operations, i.e. addition, subtraction, multiplication and division, we can furthermore define _complex conjugation_:

\begin{equation}
 z^* = a - \mathrm{i} b = \Re(z) - \mathrm{i}\Im(z)
\end{equation}

## Exercise 1

Write down explicitly the form of the sum, difference, multiplication and division of two complex numbers $z = a + \mathrm{i}b$ and $w=c + \mathrm{i}d$.
Prove also that:

 \begin{alignat}{2}
  \Re(z) = a = \frac{z+z^*}{2};\quad&\quad \Im(z) = b = \frac{z-z^*}{2\mathrm{i}}
 \end{alignat}

We can draw a powerful analogy between points in the plane and complex numbers. Let's rename the $x$-axis the _real_ axis and the $y$-axis the _imaginary_ axis. Any complex
number can be represented in this plane, simply drawing the point with $x$ coordinate equal to its real part and $y$ coordinate equal to its imaginary part. This amounts to
specifying the components of a 2-dimensional vector:

\begin{equation}\label{eq:vector}
 \overrightarrow{v} = \begin{pmatrix}
      \Re(z) \\
      \Im(z)
     \end{pmatrix}
     = \begin{pmatrix}
      a \\
      b
     \end{pmatrix}
\end{equation}

## Exercise 2

Draw the following numbers in the complex plane: $z=\frac{\sqrt{2}}{2} + \mathrm{i}\frac{\sqrt{2}}{2}$, $w=\frac{\sqrt{3}}{2} + \mathrm{i}\frac{1}{2}$ and their complex conjugates $z^*$ and $w^*$.
The following example shows how to draw arrows using `matplotlib` and the `arrow` function. This function takes two four arguments, the $x$ and $y$ coordinates of the arrow start and the $x$ and $y$ coordinates of the arrow end.

```python
import matplotlib.pyplot as plt
%matplotlib inline

ax = plt.axes()
# Use the arrow function
# The arrow start at x=0.0, y=0.0 and ends at x=0.5, y=0.5
ax.arrow(x=0, y=0, dx=0.5, dy=0.5)
plt.show()
```

The analogy between complex numbers and 2-dimensional vectors can be brought a bit further. Using Pythagoras' theorem we can calculate the length of a 2-dimensional vector
(its _norm_ or its _magnitude_) in terms of its components:

\begin{equation}
 v = \sqrt{a^2 + b^2} = \sqrt{\Re(z)^2 + \Im(z)^2}
\end{equation}

This lets us introduce a third way of representing a complex number, the _polar_ representation. Instead of specifying the cartesian components of our point in the complex plane
(cartesian or _rectangular_ representation), we can specify the _length_ of the vector, $\rho$, and its _angle_ with respect to the real axis  $\theta$. We assume that
angles increase in an anti-clockwise sense, with the zero corresponding to the $x$-axis.

## Exercise 3

Using the Figure below

prove the following sets of relations between the rectangular and polar representations:

 \begin{alignat}{2}
  \left\{
  \begin{array}{lr}
    \rho = \sqrt{a^2 + b^2} \\
    \tan\theta = \frac{b}{a}
  \end{array}
\right.
\quad&\quad
\left\{
  \begin{array}{lr}
    a = \rho\cos\theta\\
    b = \rho\sin\theta
  \end{array}
\right.
 \end{alignat}
The tangent function is defined as $\tan\alpha=\frac{\sin\alpha}{\cos\alpha}$.

Using the second set among the equations above we have:

\begin{equation}
 z = \rho\cos\theta + \mathrm{i}\rho\sin\theta = \rho(\cos\theta + \mathrm{i}\sin\theta)
\end{equation}

which we will rewrite as:

\begin{equation}
 z = \rho \mathrm{e}^{\mathrm{i}\theta}
\end{equation}

The exponential above can be expanded as:

\begin{equation}
  \mathrm{e}^{\mathrm{i}\theta} = \cos\theta + \mathrm{i}\sin\theta
\end{equation}

known as _Euler's formula_ and $\mathrm{e}=2.71828\ldots$ is the _Napier's constant_. This formula is extremely important both from a theoretical and practical point of view.

## Exercise 4

Given $z=\rho \mathrm{e}^{\mathrm{i}\theta}$ how would you write the Euler representation of its complex conjugate $z^*$?

    Hint: what happens to $a+\mathrm{i} b$ when taking the complex conjugate?

Draw some more vectors in the complex plane using `matplotlib`: $\mathrm{e}^{\mathrm{i}\theta}$ where $\theta$ is $30^\circ$, $150^\circ$, $210^\circ$ and $330^\circ$.
What do all these vectors have in common?
You can write polar vectors in `matplotlib` in two ways. Either by using Euler's formula to find the arrow start and end coordinates or by using the proper syntax for polar plots.
We invite you to use both and define the appropriate functions for the calculations needed when using the Cartesian representation.
What follows is an example of polar plot.

```python
import matplotlib.pyplot as plt
# make sure we see it on this notebook
%matplotlib inline
import numpy as np

# Enable polar coordinates
ax = plt.axes(polar=True)
# Use the arrow function
# The arrow forms a 45° angle with the x axis and has length 1
ax.arrow(x=45/180.*np.pi, y=0.0, dx=0, dy=1)
# We can still use the Cartesian format
ax.arrow(x=0.0, y=0.0, dx=0.5, dy=0.5)
plt.show()
```

## Exercise 5

The Taylor expansion of $\mathrm{e}^{\mathrm{i}\theta}$ is given below. In this exercise we will try to ``prove'', by a convincing yet non-formal argument, that Euler's formula does indeed hold true.

\begin{equation}
\begin{aligned}
 e^{\mathrm{i}\theta} &= \sum_{n = 0}^\infty \frac{(\mathrm{i}\theta)^n}{n!} \\
 &= 1 + \mathrm{i}\theta + \frac{(\mathrm{i}\theta)^2}{2!} + \frac{(\mathrm{i}\theta)^3}{3!} + \frac{(\mathrm{i}\theta)^3}{3!} + \ldots \\
 &= \left( 1 - \frac{\theta^2}{2!} + \frac{\theta^4}{4!} - \frac{\theta^6}{6!} +\ldots\right)
 + \mathrm{i}\left( \theta - \frac{\theta^3}{3!} + \frac{\theta^5}{5!} - \frac{\theta^7}{7!} + \ldots \right) \\
 &= \cos\theta + \mathrm{i}\sin\theta
 \end{aligned}
\end{equation}

Using `matplotlib`, you will produce two plots.
The former will contain the plot of $\cos\theta$, $1$, $\left( 1 - \frac{\theta^2}{2!}\right)$,
$\left( 1 - \frac{\theta^2}{2!} + \frac{\theta^4}{4!}\right)$ and $\left( 1 - \frac{\theta^2}{2!} + \frac{\theta^4}{4!} - \frac{\theta^6}{6!} \right)$.
The latter will contain the plot of $\sin\theta$, $\theta$, $\left(\theta - \frac{\theta^3}{3!}\right)$,
$\left(\theta - \frac{\theta^3}{3!} + \frac{\theta^5}{5!}\right)$ and $\left(\theta - \frac{\theta^3}{3!} + \frac{\theta^5}{5!} - \frac{\theta^7}{7!} \right)$.
What is it you notice?

As a quick reminder, you should use **both** `numpy` and `matplotlib` to generate these plots. Defining functions will also help ;)

### Bonus

The general expression for a Taylor expansion at $x_0$ is:

\begin{equation}
 f(x) = \sum_{n=0}^\infty\frac{(x-x_0)^n}{n!}f^{(n)}(x_0)
\end{equation}

where $f^{(n)}(x_0) = \frac{\mathrm{d}^n f}{\mathrm{d}x^n}(x_0)$ is the $n$-th order derivative function of $f(x)$ evaluated at $x_0$.
Write down explicitly the first few terms for the expansion of $\cos(x)$ and $\sin(x)$ for $x_0=0$ and verify that the identification made in the Taylor expansion above is correct.


# Nonlinear Decision Boundaries
Load the data in `data.csv` as a data frame.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

dat = pd.read_csv("data.csv")
```

Plot the data with matplotlib as a 2D scatter plot, using different colors for the different classes.

```python

```

Which non-linear transformation $\phi$ will lead linearly separable data in the $\phi$-space? Compute this transformation, and plot the transformed data.

```python

```

# Kernel Density Estimators

In the lecture, you got to know kernel density estimatiors as a non-parametric method for representing a distribution. They represent the distribution of values $x$ as follows:

\begin{equation}
p(t) = \frac{1}{n \, h} \sum_{i=1}^n \varphi\left(\frac{t-x_i}{h}\right)
\end{equation}
Here,  $\varphi$ is a windowing function, e.g. the Gauss window:
\begin{equation}
\varphi(u) = \frac{1}{\sqrt{2\pi}} e^{-u^2/2}
\end{equation}

## Exercise 1
Implement the function `kde(t,h,x)` that estimates the kernel density estimate $p(t)$, given a point $t$, a window size $h$ and an array of training samples $x$.

```python

```

## Exercise 2

Implement the function `classify_kde(xnew,x,classes)` that performs a classification with a kernel density estimator. Specifically, this is a Bayes classifier where the likelihood is represented by a kernel density estimator.

```python

```


# Ordinary differential equations

## Introduction

Engineering systems are often represented as differential equations. ``sympy`` has robust support for the following types of ordinary differential equations (ODEs):

- 1st order separable differential equations
- 1st order differential equations whose coefficients or dx and dy are functions homogeneous of the same order.
- 1st order exact differential equations.
- 1st order linear differential equations
- 1st order Bernoulli differential equations.
- 2nd order Liouville differential equations.
- nth order linear homogeneous differential equation with constant coefficients.
- nth order linear inhomogeneous differential equation with constant coefficients using the method of undetermined coefficients.
- nth order linear inhomogeneous differential equation with constant coefficients using the method of variation of parameters.

In addition to ODE support, sympy can even solve separable (either multiplicative or additive) partial differential equations (PDEs).

The main functionality for ODE solving in sympy is the ``sympy.dsolve`` function. The call signature for this function is ``sympy.dsolve(eq, goal, **kwargs)``, where ``eq`` is the differential equation, ``goal`` is the function you want to end up with, and ``**kwargs`` is a placeholder for other arguments that could be passed to the function to help it out a bit. For more detail on what ``**kwargs`` could be, see the documentation.

```python
from sympy import *
```

```python
init_session()
plt.style.use(u"seaborn-notebook")
```

    IPython console for SymPy 1.0 (Python 2.7.13-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

To solve differential equations, use dsolve. First, create an undefined function by passing cls=Function to the symbols function.

```python
f, g = symbols('f g', cls=Function)
```

Derivatives of f(x) are unevaluated.

```python
f(x).diff(x)
```

To represent the differential equation $f′′(x)−2f′(x)+f(x)=sin(x)$, we would use

```python
ode = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x))
```

```python
ode
```

To solve the ODE, pass it and the function to solve for to ``dsolve``.

```python
sol = dsolve(ode, f(x))
display(sol)
```

``dsolve`` returns an instance of ``Eq``. This is because in general, solutions to differential equations cannot be solved explicitly for the function.

```python
dsolve(f(x).diff(x)*(1 - sin(f(x))), f(x))
```

The arbitrary constants in the solutions from ``dsolve`` are symbols of the form $C_1, C_2, C_3$, and so on.

## Applying Boundary conditions

This isn't implemented yet in ``dsolve`` stable version, but it will be available in the next release (probably!).

For now, solve for contants on your own. For example, if

$$f(0)=1\quad \left.\frac{d f}{d x}\right\vert_{x=0}=0,$$

solve the following equations:

```python
constants = solve([sol.rhs.subs(x,0) - 1, sol.rhs.diff(x,1).subs(x,0)- 0])
constants
```

```python
C1, C2 = symbols('C1,C2')
sol = sol.subs(constants)
sol
```

## Systems of equations

```python
g1, g2 = symbols('g1 g2', cls=Function)
```

```python
eq1 = g1(x).diff(x) - 2*g1(x) - g2(x)
eq2 = g2(x).diff(x) - 3*g1(x) - 4*g2(x)
```

```python
dsolve((eq1, eq2), fun=(g1(x), g2(x)))
```

```python

```

## Vector field

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

The '''Lotka–Volterra equations''', also known as the '''predator–prey equations''', are a pair of first-order, nonlinear, differential equations frequently used to describe the dynamics of biological systems in which two species interact, one as a predator and the other as prey. The populations change through time according to the pair of equations:

\begin{align}
\frac{dx}{dt} & = \alpha x - \beta x y \\[6pt]
\frac{dy}{dt} & = \delta x y  - \gamma y
\end{align}

where

* $x$ is the number of prey (for example, rabbits);
* $y$ is the number of some predator (for example, foxes);
* $\tfrac{dy}{dt}$ and $\tfrac{dx}{dt}$ represent the growth rates of the two populations over time;
* $t$  represents time; and
* $α, β, γ, δ$ are positive real parameters describing the interaction of the two species.

```python
y, x = np.mgrid[0:2.5:100j, 0:4:100j]
```

```python
alpha = 2/3
beta = 4/3
gamma = 1
delta = 1
dxdt = alpha*x - beta*x*y
dydt = delta*x*y - gamma*y
speed = np.sqrt(dxdt**2 + dydt**2)
```

In the first case we just plot the velocity field.

```python
fig0 = plt.figure(figsize=(8,5))
strm = plt.streamplot(x, y, dxdt, dydt, linewidth=2)
plt.xlabel("Preys")
plt.ylabel("Predators")
plt.xlim(0,4)
plt.ylim(0,2.5)
```

In this example we change the color of the line according to the magnitude of the
_speed_ at each point.

```python
fig1 = plt.figure(figsize=(8,5))
strm = plt.streamplot(x, y, dxdt, dydt, color=speed, linewidth=2, cmap="summer")
plt.colorbar(strm.lines)
plt.xlabel("Preys")
plt.ylabel("Predators")
plt.xlim(0,4)
plt.ylim(0,2.5)
```

In this case the densities in $x$ and $y$ directions are different.

```python
fig2 = plt.figure(figsize=(8,5))
plt.streamplot(x, y, dxdt, dydt, density=[0.5, 1])
plt.xlabel("Preys")
plt.ylabel("Predators")
plt.xlim(0,4)
plt.ylim(0,2.5)
```

In this last case we vary the width of the lines according to the magnitude
of the _speed_ at each point.

```python
fig3 = plt.figure(figsize=(8,5))
lw = 5*speed / speed.max()
plt.streamplot(x, y, dxdt, dydt, density=0.6, color='k', linewidth=lw)
plt.xlabel("Preys")
plt.ylabel("Predators")
plt.xlim(0,4)
plt.ylim(0,2.5)
```

## Power series approximation

```python
def picard_iteration(f, t, y0, n):
    y = y0
    for cont in range(n):
        y = y0 + f(t, y, n).integrate((t, 0, t))

    return y
```

```python
def fun(t, y, n):
    return Matrix([y[1],  t*y[0]])
```

```python
x0, x1 = symbols("x0 x1")
y = Matrix([airyai(0), airyaiprime(0)])
y_approx = picard_iteration(fun, t, y, 10)
```

```python
p0 = plot(airyai(t), (t, -4, 4), line_color="black");
p1 = plot(N(y_approx[0]), (t, -4, 4));
p0.extend(p1);
p0.show()
```

## References

1. SymPy Development Team (2016). [Sympy Tutorial: Matrices](http://docs.sympy.org/latest/tutorial/matrices.html)
2. Ivan Savov (2016). [Taming math and physics using SymPy](https://minireference.com/static/tutorials/sympy_tutorial.pdf)

The following cell change the style of the notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
/* Based on Lorena Barba template available at: https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css*/
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```


# day 12: Stochastic Gradient Descent

# Objectives

* See how Stochastic Gradient Descent can be used on a simple model (1-dim. linear regression)
* See how SGD can be used on a complex model (MLPClassifier)
* Understand impact of batch size and learning rate (aka step size)

# Outline
* [Review: Loss and Gradient for 1-dim. Linear Regression](#part1)
* [Part 1: Stochastic Estimates of the Loss](#part1)
* [Part 2: Stochastic Estimates of the Gradient](#part2)
* [Part 3: Stochastic Gradient Descent Algorithm in a few lines of Python](#part3)
* [Part 4: Using sklearn code to train MLPClassifier with SGD](#part4)

# Takeaways

* Stochastic estimates of loss functions are possible when the function is *additive* over training examples

* Stochastic gradient descent is a simple algorithm that can be implemented in a few lines of Python
* * Practical issues include selecting step size and batch size $B$

* Selecting batch size trades off two things:
* * Runtime cost of computing each gradient estimate (scales with $O(B)$, so smaller is better)
* * Quality of the estimate (larger $B$ leads to less variance)

```python
import numpy as np
```

```python
import sklearn.neural_network
```

```python
# import plotting libraries
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', font_scale=1.25, style='whitegrid')
```

# Create simple dataset:   y = 1.234 * x + noise

We will *intentionally* create a toy dataset where we know that a good solution has slope near 1.234.

We'll generate N = 1000 examples.

Naturally, the best slope for the finite dataset we create won't be exactly 1.234 (because of the noise added plus the fact that our dataset size is limited).

```python
def create_dataset(N=1000, slope=1.234, noise_stddev=0.1, random_state=0):
    random_state = np.random.RandomState(int(random_state))

    # input features
    x_N = np.linspace(-2, 2, N)

    # output features
    y_N = slope * x_N + random_state.randn(N) * noise_stddev

    return x_N, y_N
```

```python
xtrain_N, ytrain_N = create_dataset(N=1000, noise_stddev=0.3)
```

```python
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5,5))
plt.plot(xtrain_N, ytrain_N, 'k.');
plt.xlabel('x');
plt.ylabel('y');
```

# Review: Gradient Descent for 1-dim. Linear Regression

## Define prediction model

Consider the *simplest* linear regression model. A single weight parameter $w \in \mathbb{R}$ representing the slope of the prediction line. No bias/intercept.

To make predictions, we just compute the weight multiplied by the input feature
$$
\hat{y}(x) = w \cdot x
$$

## Define loss function

We want to minimize the total *squared error* across all N observed data examples (input features $x_n$, output responses $y_n$)

Given a full dataset of $N$ examples, we can compute the loss as:

\begin{align}
    \min_{w \in \mathbb{R}} ~~ &\ell(w)
    \\
    \text{calc_loss}(w) = \ell(w) &= \frac{1}{N} \sum_{n=1}^N \frac{1}{2} (y_n - w x_n)^2
\end{align}

Given a random minibatch of $B$ examples, we can *estimate* the loss as:

\begin{align}
    \text{estimate_loss}(w) = &= \frac{1}{B} \sum_{n=1}^B \frac{1}{2} (y_n - w x_n)^2
\end{align}

### Exercise 1A: Complete the code below

You should make it match the math expression above.

```python
def calc_loss(w, xbatch_B, ybatch_B):
    ''' Compute loss for slope-only least-squares linear regression

    Will compute the estimate of the loss given a minibatch of x,y values

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    loss : float
        Mean of squared error loss at provided w value
    '''
    B = xbatch_B.shape[0]
    yhat_B = xbatch_B * w
    half_mean_squared_error = 0.5 * np.mean(np.square(ybatch_B - yhat_B))
    return half_mean_squared_error
```

### Define the gradient function

Given a full dataset of $N$ examples, we can compute the gradient as:

\begin{align}
\text{calc_grad}(w) = \ell'(w) &= \frac{1}{N} \frac{\partial}{\partial w} [ \sum_{n=1}^N \frac{1}{2} (y_n - w x_n)^2]
\\
&= \frac{1}{N}  \sum_{n=1}^N (y_n - w x_n) (-x_n)
\\
&= \frac{1}{N} \sum_{n=1}^N (w x_n - y_n) (x_n)
\\
&= w \left( \frac{1}{N} \sum_{n=1}^N x_n^2 \right) - \frac{1}{N} \sum_{n=1}^N y_n x_n
\end{align}

Given a random minibatch of $B$ examples, we can *estimate* the gradient using:

\begin{align}
\text{estimate_grad}(w) &= \frac{1}{B} \frac{\partial}{\partial w} [ \sum_{n=1}^B \frac{1}{2} (y_n - w x_n)^2]
\\
&= w \left( \frac{1}{B} \sum_{n=1}^B x_n^2 \right) - \frac{1}{B} \sum_{n=1}^B y_n x_n
\end{align}

Below, we've implemented the gradient calculation in code for you

```python
def calc_grad(w, xbatch_B, ybatch_B):
    ''' Compute gradient for slope-only least-squares linear regression

    Will compute a deterministic estimate of the gradient given a minibatch of x,y values

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    g : float
        Value of derivative of loss function at provided w value
    '''
    g = w * np.mean(np.square(xbatch_B)) - np.mean(xbatch_B * ybatch_B)
    return g
```

# Part 1: Stochastic estimates of the loss

### Plot whole-dataset loss evaluated at each w from -3 to 8

We should see a "bowl" shape with one *global* minima, because our optimization problem is "convex"

```python
G = 101
w_grid = np.linspace(-3, 8, G) # create array of 300 values between -3 and 8
print(w_grid)
```

    [-3.   -2.89 -2.78 -2.67 -2.56 -2.45 -2.34 -2.23 -2.12 -2.01 -1.9  -1.79
     -1.68 -1.57 -1.46 -1.35 -1.24 -1.13 -1.02 -0.91 -0.8  -0.69 -0.58 -0.47
     -0.36 -0.25 -0.14 -0.03  0.08  0.19  0.3   0.41  0.52  0.63  0.74  0.85
      0.96  1.07  1.18  1.29  1.4   1.51  1.62  1.73  1.84  1.95  2.06  2.17
      2.28  2.39  2.5   2.61  2.72  2.83  2.94  3.05  3.16  3.27  3.38  3.49
      3.6   3.71  3.82  3.93  4.04  4.15  4.26  4.37  4.48  4.59  4.7   4.81
      4.92  5.03  5.14  5.25  5.36  5.47  5.58  5.69  5.8   5.91  6.02  6.13
      6.24  6.35  6.46  6.57  6.68  6.79  6.9   7.01  7.12  7.23  7.34  7.45
      7.56  7.67  7.78  7.89  8.  ]

```python
loss_grid = np.zeros(G)
for gg in range(G):
    loss_grid[gg] = calc_loss(w_grid[gg], xtrain_N, ytrain_N)

plt.plot(w_grid, loss_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('loss(w)');
plt.ylim([0, 35]);
```

### Discussion 1a: Visually, at what value of $w$ does the loss function have a minima? Is it near where you would expect (hint: look above for the "true" slope value used to generate the data)

```python

```

### Sampling a minibatch of size B

We have provided some starter code that samples a minibatch of B examples from a training set of N examples

Uses a provided `random_state` pseudo-random number generator, which defaults to numpy's if not specified.

```python
def draw_minibatch(xtrain_N, ytrain_N, batch_size=100, random_state=np.random):
    ''' Sample a minibatch of desired size from provided training set

    Returns
    -------
    xbatch_B : 1D array, size (B,)
        x values of minibatch
    ybatch_B : 1D array, size (B,)
        y values of minibatch
    '''
    N = ytrain_N.size
    selected_row_ids = random_state.choice(np.arange(N), size=batch_size, replace=False)
    xbatch_B = xtrain_N[selected_row_ids].copy()
    ybatch_B = ytrain_N[selected_row_ids].copy()
    return xbatch_B, ybatch_B
```

### Show several minibatches of size 1

```python
draw_minibatch(xtrain_N, ytrain_N, 1)
```

    (array([1.80780781]), array([2.30416787]))

```python
draw_minibatch(xtrain_N, ytrain_N, 1)
```

    (array([-0.44244244]), array([-0.49670565]))

```python
draw_minibatch(xtrain_N, ytrain_N, 1)
```

    (array([1.67167167]), array([1.98366164]))

### Show example minibatch of size 500

```python
xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, 500)

print("Showing first 5 entries of x array of shape %s" % str(xbatch_B.shape))
print(xbatch_B[:5])
print("Showing first 5 entries of y array of shape %s" % str(ybatch_B.shape))
print(ybatch_B[:5])
```

    Showing first 5 entries of x array of shape (500,)
    [-1.11111111 -0.27827828 -0.1021021   0.88688689 -0.16616617]
    Showing first 5 entries of y array of shape (500,)
    [-1.5101899  -0.48344906  0.39566781  0.9413306  -0.14525899]

### Exercise 1b: Would you expect the loss on the minibatch to be better at $w=1.0$ or at $w = -1.0$?

TODO write answer here to guess, then use code below to figure it out.

Show the loss at this random minibatch when $w = 1.0$

```python
calc_loss(1.0, xbatch_B, ybatch_B)
```

    0.07291125339530102

Show the loss at this random minibatch when w = -2.0

```python
calc_loss(-1.0, xbatch_B, ybatch_B)
```

    3.264773935155204

### Exercise 1c: Can you draw a random minibatch of size 25 and display *all* of it?

```python
xbatch_25, ybatch_25 = None, None # TODO use draw_minibatch

print("Showing x array")
print(xbatch_25)
print("Showing y array")
print(ybatch_25)

```

    Showing x array
    None
    Showing y array
    None

### Exercise 1d: What is the loss at your minibatch at $w=1.0$ and $w = 1.5$?

```python
# TODO your code here, call calc_loss
```

```python

```

## Plot: Compute the *stochastic estimate* of loss as a function of $w$, using batch size 50

We'll make 3 lines, for 3 separate *trials* of this procedure, so we can see how much each trial's curve might vary

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_loss_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=50, random_state=random_state)
        stoch_loss_grid[gg] = calc_loss(w_grid[gg], xbatch_B, ybatch_B)

    plt.plot(w_grid, stoch_loss_grid, '.-', label='Trial %d' % trial);
plt.xlabel('w');
plt.ylabel('loss(w)');
plt.ylim([0, 35]);
```

## Exercise 1e: Compute stochastic estimate of loss as function of w, for batch_size = 5

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_loss_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = None, None # TODO use draw_minibatch
        stoch_loss_grid[gg] = 0.0             # TODO use calc_loss at the current batch

    plt.plot(w_grid, stoch_loss_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('loss(w)');
```

## Exercise 1f: Compute stochastic estimate of loss as function of w, for batch_size = 1

Repeat the above for batch_size = 1

```python

```

## Discussion 1f: What can you say about the *variance* of the stochastic estimates

How does variance change as a function of batch size ?

## Part 2: Stochastic estimates of the gradient

## Sanity check: plot whole dataset gradient evaluated at each w from -3 to 8

```python
grad_grid = np.zeros(G)
for gg in range(G):
    grad_grid[gg] = calc_grad(w_grid[gg], xtrain_N, ytrain_N)

plt.plot(w_grid, grad_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('grad(w)');
plt.ylim([-7, 9]);
```

## Plot: Compute the *stochastic estimate* of grad as a function of $w$, using batch size 50

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_grad_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=50, random_state=random_state)
        stoch_grad_grid[gg] = calc_grad(w_grid[gg], xbatch_B, ybatch_B)

    plt.plot(w_grid, stoch_grad_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('grad(w)');
plt.ylim([-7, 9]);
```

## Exercise 2a: Repeat the above plot at batch_size = 5

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_grad_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = None, None # TODO use draw_minibatch
        stoch_grad_grid[gg] = 0.0 # TODO use calc_grad

    plt.plot(w_grid, stoch_grad_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('grad(w)');
plt.ylim([-7, 9]);
```

## Exercise 2c: Repeat the above plot at batch_size = 1

TODO copy code from above.

```python
# TODO
```

## Discussion 2d: What happens to the variance of the grad estimate as batch_size increases?

## Discussion 2d: What happens to the cost of computing the estimate as batch_size increases?

## Part 3: Stochastic Gradient Descent (GD) as an algorithm in Python

### Define minimize_via_sgd algorithm

Can you understand what each step of this algorithm does?

```python
def minimize_via_sgd(xtrain_N, ytrain_N, init_w=0.0, batch_size=10, step_size=0.001, max_iters=100, random_state=0):
    ''' Perform minimization of provided loss function via gradient descent

    Each "iteration" performs one or more gradient updates, until total training set has been "seen"

    Args
    ----
    xtrain_N : numpy array, shape (N,)
    ytrain_N : numpy array, shape (N,)
    init_w : float
    batch_size : int
    step_size : float
    max_iters : positive int

    Return
    ----
    wopt: float
        array of optimized weights that approximately gives the least error
    info_dict : dict
        Contains information about the optimization procedure useful for debugging
        Entries include:
        * trace_loss_list : list of loss values
        * trace_grad_list : list of gradient values
    '''
    N = int(ytrain_N.size)
    B = int(batch_size)

    if isinstance(random_state, int):
        random_state = np.random.RandomState(random_state)

    w = 1.0 * init_w

    # Create some lists to track progress over time (for debugging)
    trace_loss_list = []
    trace_w_list = []
    trace_grad_list = []

    step_id = 0
    for iter_id in range(max_iters):

        n_examples_seen_this_iter = 0
        while n_examples_seen_this_iter < N:
            xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size, random_state=random_state)
            n_examples_seen_this_iter += batch_size

            loss = calc_loss(w, xbatch_B, ybatch_B)
            grad = calc_grad(w, xbatch_B, ybatch_B)
            w = w - step_size * grad
            step_id += 1

        print("  iter %5d/%d done | step %5d | w  % 13.5f | loss % 13.4f | grad % 13.4f" % (
            iter_id, max_iters, step_id, w, loss, grad))

        trace_loss_list.append(loss)
        trace_w_list.append(w)
        trace_grad_list.append(grad)

    wopt = w
    info_dict = dict(
        trace_loss_list=trace_loss_list,
        trace_w_list=trace_w_list,
        trace_grad_list=trace_grad_list)

    return wopt, info_dict
```

### Discussion 2a: Which line of the above function does the *parameter update* happen?

Remember, in math, the parameter update of gradient descent is this:
$$
w \gets w - \alpha \nabla_w \ell(w)
$$

where $\alpha > 0$ is the step size.

In words, this math says *move* the parameter $w$ from its current value a *small step* in the "downhill" direction (indicated by gradient).

TODO write down here which line above *you* think it is, then discuss with your group

```python

```

### Try it! Run SGD with step_size = 0.01 and batch_size = 200

Running the cell below will have the following effects:

1) one line will be printed for every iteration, indicating the current w value and its associated loss

2) the "optimal" value of w will be stored in the variable named `wopt` returned by this function

3) a dictionary of information useful for debugging will be stored in the `info_dict` returned by this function

```python
wopt, info_dict = minimize_via_sgd(xtrain_N, ytrain_N, step_size=0.01, batch_size=200);
```

      iter     0/100 done | step     5 | w        0.08015 | loss        1.0488 | grad       -1.7202
      iter     1/100 done | step    10 | w        0.15328 | loss        0.8576 | grad       -1.4773
      iter     2/100 done | step    15 | w        0.22259 | loss        0.7444 | grad       -1.3535
      iter     3/100 done | step    20 | w        0.28735 | loss        0.6190 | grad       -1.2232
      iter     4/100 done | step    25 | w        0.35058 | loss        0.5804 | grad       -1.2300
      iter     5/100 done | step    30 | w        0.40936 | loss        0.5670 | grad       -1.2455
      iter     6/100 done | step    35 | w        0.46134 | loss        0.4426 | grad       -1.0366
      iter     7/100 done | step    40 | w        0.51053 | loss        0.3971 | grad       -0.9791
      iter     8/100 done | step    45 | w        0.55889 | loss        0.3367 | grad       -0.8819
      iter     9/100 done | step    50 | w        0.60231 | loss        0.3295 | grad       -0.8519
      iter    10/100 done | step    55 | w        0.64268 | loss        0.2927 | grad       -0.8156
      iter    11/100 done | step    60 | w        0.68140 | loss        0.2557 | grad       -0.7506
      iter    12/100 done | step    65 | w        0.71875 | loss        0.2472 | grad       -0.7419
      iter    13/100 done | step    70 | w        0.75133 | loss        0.1835 | grad       -0.5930
      iter    14/100 done | step    75 | w        0.78277 | loss        0.1722 | grad       -0.5639
      iter    15/100 done | step    80 | w        0.81064 | loss        0.1731 | grad       -0.5796
      iter    16/100 done | step    85 | w        0.83925 | loss        0.1716 | grad       -0.6222
      iter    17/100 done | step    90 | w        0.86445 | loss        0.1464 | grad       -0.5074
      iter    18/100 done | step    95 | w        0.88863 | loss        0.1269 | grad       -0.4665
      iter    19/100 done | step   100 | w        0.91101 | loss        0.1231 | grad       -0.4666
      iter    20/100 done | step   105 | w        0.93204 | loss        0.1004 | grad       -0.4243
      iter    21/100 done | step   110 | w        0.95136 | loss        0.0874 | grad       -0.3560
      iter    22/100 done | step   115 | w        0.97035 | loss        0.0987 | grad       -0.3765
      iter    23/100 done | step   120 | w        0.98704 | loss        0.0808 | grad       -0.2952
      iter    24/100 done | step   125 | w        1.00346 | loss        0.0852 | grad       -0.3169
      iter    25/100 done | step   130 | w        1.01760 | loss        0.0784 | grad       -0.3205
      iter    26/100 done | step   135 | w        1.03255 | loss        0.0701 | grad       -0.2819
      iter    27/100 done | step   140 | w        1.04555 | loss        0.0659 | grad       -0.2295
      iter    28/100 done | step   145 | w        1.05768 | loss        0.0687 | grad       -0.2576
      iter    29/100 done | step   150 | w        1.06847 | loss        0.0614 | grad       -0.2003
      iter    30/100 done | step   155 | w        1.07939 | loss        0.0661 | grad       -0.2469
      iter    31/100 done | step   160 | w        1.08989 | loss        0.0582 | grad       -0.2022
      iter    32/100 done | step   165 | w        1.10010 | loss        0.0630 | grad       -0.2164
      iter    33/100 done | step   170 | w        1.10851 | loss        0.0565 | grad       -0.1732
      iter    34/100 done | step   175 | w        1.11634 | loss        0.0552 | grad       -0.1632
      iter    35/100 done | step   180 | w        1.12382 | loss        0.0565 | grad       -0.1650
      iter    36/100 done | step   185 | w        1.12964 | loss        0.0494 | grad       -0.1079
      iter    37/100 done | step   190 | w        1.13630 | loss        0.0470 | grad       -0.1289
      iter    38/100 done | step   195 | w        1.14222 | loss        0.0478 | grad       -0.1077
      iter    39/100 done | step   200 | w        1.14832 | loss        0.0590 | grad       -0.1447
      iter    40/100 done | step   205 | w        1.15326 | loss        0.0419 | grad       -0.1074
      iter    41/100 done | step   210 | w        1.15853 | loss        0.0460 | grad       -0.0721
      iter    42/100 done | step   215 | w        1.16395 | loss        0.0509 | grad       -0.1238
      iter    43/100 done | step   220 | w        1.16902 | loss        0.0426 | grad       -0.1063
      iter    44/100 done | step   225 | w        1.17258 | loss        0.0459 | grad       -0.0702
      iter    45/100 done | step   230 | w        1.17556 | loss        0.0466 | grad       -0.0392
      iter    46/100 done | step   235 | w        1.17890 | loss        0.0428 | grad       -0.0586
      iter    47/100 done | step   240 | w        1.18149 | loss        0.0433 | grad       -0.0558
      iter    48/100 done | step   245 | w        1.18422 | loss        0.0477 | grad       -0.0653
      iter    49/100 done | step   250 | w        1.18750 | loss        0.0428 | grad       -0.0637
      iter    50/100 done | step   255 | w        1.19068 | loss        0.0525 | grad       -0.0635
      iter    51/100 done | step   260 | w        1.19410 | loss        0.0478 | grad       -0.0637
      iter    52/100 done | step   265 | w        1.19610 | loss        0.0371 | grad       -0.0345
      iter    53/100 done | step   270 | w        1.19875 | loss        0.0444 | grad       -0.0442
      iter    54/100 done | step   275 | w        1.20074 | loss        0.0458 | grad       -0.0389
      iter    55/100 done | step   280 | w        1.20295 | loss        0.0467 | grad       -0.0212
      iter    56/100 done | step   285 | w        1.20470 | loss        0.0410 | grad       -0.0131
      iter    57/100 done | step   290 | w        1.20604 | loss        0.0386 | grad       -0.0209
      iter    58/100 done | step   295 | w        1.20676 | loss        0.0387 | grad       -0.0180
      iter    59/100 done | step   300 | w        1.20872 | loss        0.0407 | grad       -0.0393
      iter    60/100 done | step   305 | w        1.20974 | loss        0.0391 | grad       -0.0075
      iter    61/100 done | step   310 | w        1.21074 | loss        0.0448 | grad       -0.0445
      iter    62/100 done | step   315 | w        1.21162 | loss        0.0469 | grad        0.0055
      iter    63/100 done | step   320 | w        1.21239 | loss        0.0385 | grad        0.0217
      iter    64/100 done | step   325 | w        1.21317 | loss        0.0404 | grad       -0.0250
      iter    65/100 done | step   330 | w        1.21514 | loss        0.0422 | grad       -0.0427
      iter    66/100 done | step   335 | w        1.21667 | loss        0.0505 | grad       -0.0682
      iter    67/100 done | step   340 | w        1.21734 | loss        0.0441 | grad       -0.0089
      iter    68/100 done | step   345 | w        1.21843 | loss        0.0487 | grad       -0.0520
      iter    69/100 done | step   350 | w        1.21968 | loss        0.0426 | grad       -0.0217
      iter    70/100 done | step   355 | w        1.22061 | loss        0.0451 | grad       -0.0162
      iter    71/100 done | step   360 | w        1.22148 | loss        0.0494 | grad       -0.0131
      iter    72/100 done | step   365 | w        1.22243 | loss        0.0432 | grad       -0.0476
      iter    73/100 done | step   370 | w        1.22310 | loss        0.0359 | grad       -0.0035
      iter    74/100 done | step   375 | w        1.22355 | loss        0.0445 | grad       -0.0733
      iter    75/100 done | step   380 | w        1.22387 | loss        0.0475 | grad       -0.0229
      iter    76/100 done | step   385 | w        1.22515 | loss        0.0418 | grad       -0.0054
      iter    77/100 done | step   390 | w        1.22519 | loss        0.0457 | grad        0.0189
      iter    78/100 done | step   395 | w        1.22512 | loss        0.0411 | grad        0.0211
      iter    79/100 done | step   400 | w        1.22509 | loss        0.0423 | grad       -0.0424
      iter    80/100 done | step   405 | w        1.22611 | loss        0.0435 | grad       -0.0184
      iter    81/100 done | step   410 | w        1.22761 | loss        0.0454 | grad       -0.0352
      iter    82/100 done | step   415 | w        1.22761 | loss        0.0444 | grad        0.0206
      iter    83/100 done | step   420 | w        1.22788 | loss        0.0434 | grad        0.0002
      iter    84/100 done | step   425 | w        1.22852 | loss        0.0393 | grad       -0.0144
      iter    85/100 done | step   430 | w        1.22787 | loss        0.0438 | grad       -0.0081
      iter    86/100 done | step   435 | w        1.22866 | loss        0.0451 | grad       -0.0259
      iter    87/100 done | step   440 | w        1.22834 | loss        0.0410 | grad        0.0073
      iter    88/100 done | step   445 | w        1.22975 | loss        0.0406 | grad       -0.0768
      iter    89/100 done | step   450 | w        1.23034 | loss        0.0427 | grad       -0.0344
      iter    90/100 done | step   455 | w        1.23006 | loss        0.0496 | grad       -0.0083
      iter    91/100 done | step   460 | w        1.23052 | loss        0.0438 | grad       -0.0056
      iter    92/100 done | step   465 | w        1.23073 | loss        0.0441 | grad        0.0013
      iter    93/100 done | step   470 | w        1.23094 | loss        0.0471 | grad        0.0267
      iter    94/100 done | step   475 | w        1.23106 | loss        0.0392 | grad        0.0114
      iter    95/100 done | step   480 | w        1.23164 | loss        0.0459 | grad        0.0033
      iter    96/100 done | step   485 | w        1.23252 | loss        0.0481 | grad       -0.0301
      iter    97/100 done | step   490 | w        1.23304 | loss        0.0368 | grad       -0.0023
      iter    98/100 done | step   495 | w        1.23335 | loss        0.0429 | grad       -0.0010
      iter    99/100 done | step   500 | w        1.23343 | loss        0.0513 | grad       -0.0131

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(12,3))

axes[0].plot(info_dict['trace_loss_list'], '.-');
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list'], '.-');
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list'], '.-');
axes[2].set_title('w');
```

### Discussion 2b: Does it appear from the *parameter* values in trace above that the SGD procedure converged?

### Discussion 2c: Does it appear from the *loss* values in trace above that the SGD procedure converged?

### Discussion 2d: Does it appear from the *grad* values in trace above that the SGD procedure converged?

```python

```

```python

```

## Try with smaller batch_size = 10

```python
wopt, info_dict = minimize_via_sgd(xtrain_N, ytrain_N, step_size=0.01, batch_size=10);
```

      iter     0/100 done | step   100 | w        0.91625 | loss        0.1191 | grad       -0.3511
      iter     1/100 done | step   200 | w        1.14951 | loss        0.0377 | grad        0.0505
      iter     2/100 done | step   300 | w        1.20424 | loss        0.0991 | grad        0.1091
      iter     3/100 done | step   400 | w        1.23031 | loss        0.0381 | grad        0.0042
      iter     4/100 done | step   500 | w        1.23886 | loss        0.0511 | grad       -0.2389
      iter     5/100 done | step   600 | w        1.24595 | loss        0.0859 | grad       -0.1187
      iter     6/100 done | step   700 | w        1.25205 | loss        0.0334 | grad        0.1019
      iter     7/100 done | step   800 | w        1.23007 | loss        0.0334 | grad       -0.0147
      iter     8/100 done | step   900 | w        1.22628 | loss        0.0573 | grad        0.0265
      iter     9/100 done | step  1000 | w        1.23810 | loss        0.0412 | grad        0.0452
      iter    10/100 done | step  1100 | w        1.22357 | loss        0.0227 | grad        0.0010
      iter    11/100 done | step  1200 | w        1.23342 | loss        0.0515 | grad       -0.0504
      iter    12/100 done | step  1300 | w        1.24008 | loss        0.0484 | grad       -0.2045
      iter    13/100 done | step  1400 | w        1.24485 | loss        0.0738 | grad       -0.0568
      iter    14/100 done | step  1500 | w        1.23728 | loss        0.0317 | grad        0.0817
      iter    15/100 done | step  1600 | w        1.22575 | loss        0.0404 | grad        0.0695
      iter    16/100 done | step  1700 | w        1.22235 | loss        0.0293 | grad        0.1196
      iter    17/100 done | step  1800 | w        1.22595 | loss        0.0330 | grad       -0.1241
      iter    18/100 done | step  1900 | w        1.23493 | loss        0.0149 | grad        0.0567
      iter    19/100 done | step  2000 | w        1.23858 | loss        0.0361 | grad       -0.0242
      iter    20/100 done | step  2100 | w        1.23371 | loss        0.0423 | grad        0.0176
      iter    21/100 done | step  2200 | w        1.23280 | loss        0.0428 | grad       -0.0151
      iter    22/100 done | step  2300 | w        1.24703 | loss        0.0544 | grad       -0.0138
      iter    23/100 done | step  2400 | w        1.24159 | loss        0.0544 | grad        0.0364
      iter    24/100 done | step  2500 | w        1.24189 | loss        0.0278 | grad       -0.0217
      iter    25/100 done | step  2600 | w        1.23713 | loss        0.0509 | grad        0.0451
      iter    26/100 done | step  2700 | w        1.23292 | loss        0.0502 | grad        0.1502
      iter    27/100 done | step  2800 | w        1.23131 | loss        0.0266 | grad       -0.0233
      iter    28/100 done | step  2900 | w        1.23236 | loss        0.0267 | grad        0.0038
      iter    29/100 done | step  3000 | w        1.22569 | loss        0.0350 | grad       -0.0734
      iter    30/100 done | step  3100 | w        1.22976 | loss        0.0349 | grad        0.0029
      iter    31/100 done | step  3200 | w        1.23033 | loss        0.0173 | grad        0.0927
      iter    32/100 done | step  3300 | w        1.23814 | loss        0.0387 | grad        0.0252
      iter    33/100 done | step  3400 | w        1.23415 | loss        0.0182 | grad       -0.0323
      iter    34/100 done | step  3500 | w        1.23714 | loss        0.0549 | grad        0.1117
      iter    35/100 done | step  3600 | w        1.23755 | loss        0.0294 | grad       -0.0801
      iter    36/100 done | step  3700 | w        1.23231 | loss        0.0303 | grad        0.1700
      iter    37/100 done | step  3800 | w        1.23270 | loss        0.0449 | grad        0.0136
      iter    38/100 done | step  3900 | w        1.22366 | loss        0.0325 | grad        0.0768
      iter    39/100 done | step  4000 | w        1.22075 | loss        0.0216 | grad       -0.0362
      iter    40/100 done | step  4100 | w        1.23983 | loss        0.0784 | grad       -0.1227
      iter    41/100 done | step  4200 | w        1.22596 | loss        0.0632 | grad        0.1253
      iter    42/100 done | step  4300 | w        1.23038 | loss        0.0280 | grad       -0.1657
      iter    43/100 done | step  4400 | w        1.23377 | loss        0.0390 | grad       -0.1455
      iter    44/100 done | step  4500 | w        1.22033 | loss        0.0512 | grad       -0.1874
      iter    45/100 done | step  4600 | w        1.22807 | loss        0.0153 | grad        0.0763
      iter    46/100 done | step  4700 | w        1.23235 | loss        0.0493 | grad       -0.2164
      iter    47/100 done | step  4800 | w        1.22289 | loss        0.0420 | grad       -0.1002
      iter    48/100 done | step  4900 | w        1.22318 | loss        0.0355 | grad        0.0695
      iter    49/100 done | step  5000 | w        1.22875 | loss        0.0320 | grad        0.0094
      iter    50/100 done | step  5100 | w        1.21265 | loss        0.0573 | grad        0.0005
      iter    51/100 done | step  5200 | w        1.23306 | loss        0.0531 | grad       -0.1133
      iter    52/100 done | step  5300 | w        1.23758 | loss        0.0500 | grad        0.0407
      iter    53/100 done | step  5400 | w        1.23312 | loss        0.0659 | grad        0.0761
      iter    54/100 done | step  5500 | w        1.23100 | loss        0.0128 | grad       -0.0745
      iter    55/100 done | step  5600 | w        1.23669 | loss        0.0392 | grad        0.0078
      iter    56/100 done | step  5700 | w        1.24015 | loss        0.0410 | grad       -0.0476
      iter    57/100 done | step  5800 | w        1.22657 | loss        0.0248 | grad        0.0298
      iter    58/100 done | step  5900 | w        1.22985 | loss        0.0573 | grad        0.0474
      iter    59/100 done | step  6000 | w        1.22832 | loss        0.0259 | grad       -0.0250
      iter    60/100 done | step  6100 | w        1.23766 | loss        0.0662 | grad       -0.0484
      iter    61/100 done | step  6200 | w        1.23839 | loss        0.0862 | grad       -0.2737
      iter    62/100 done | step  6300 | w        1.23734 | loss        0.0323 | grad       -0.0365
      iter    63/100 done | step  6400 | w        1.23474 | loss        0.0286 | grad       -0.0304
      iter    64/100 done | step  6500 | w        1.24034 | loss        0.0578 | grad        0.1138
      iter    65/100 done | step  6600 | w        1.23524 | loss        0.0212 | grad        0.0872
      iter    66/100 done | step  6700 | w        1.22606 | loss        0.0438 | grad        0.1876
      iter    67/100 done | step  6800 | w        1.23196 | loss        0.0306 | grad       -0.0178
      iter    68/100 done | step  6900 | w        1.23354 | loss        0.0525 | grad       -0.0704
      iter    69/100 done | step  7000 | w        1.22663 | loss        0.0442 | grad       -0.0648
      iter    70/100 done | step  7100 | w        1.22977 | loss        0.0419 | grad        0.0042
      iter    71/100 done | step  7200 | w        1.23995 | loss        0.0629 | grad        0.0045
      iter    72/100 done | step  7300 | w        1.22573 | loss        0.0197 | grad        0.0462
      iter    73/100 done | step  7400 | w        1.22610 | loss        0.0434 | grad        0.1277
      iter    74/100 done | step  7500 | w        1.22869 | loss        0.0187 | grad       -0.0309
      iter    75/100 done | step  7600 | w        1.22590 | loss        0.0532 | grad        0.0942
      iter    76/100 done | step  7700 | w        1.22963 | loss        0.0493 | grad        0.1311
      iter    77/100 done | step  7800 | w        1.23985 | loss        0.0852 | grad        0.0897
      iter    78/100 done | step  7900 | w        1.23384 | loss        0.0514 | grad        0.0544
      iter    79/100 done | step  8000 | w        1.23430 | loss        0.0295 | grad       -0.1374
      iter    80/100 done | step  8100 | w        1.22765 | loss        0.0141 | grad       -0.0433
      iter    81/100 done | step  8200 | w        1.23420 | loss        0.0216 | grad       -0.0395
      iter    82/100 done | step  8300 | w        1.22563 | loss        0.0665 | grad       -0.1015
      iter    83/100 done | step  8400 | w        1.24080 | loss        0.0364 | grad        0.0033
      iter    84/100 done | step  8500 | w        1.23774 | loss        0.0632 | grad       -0.0353
      iter    85/100 done | step  8600 | w        1.23484 | loss        0.0781 | grad       -0.0488
      iter    86/100 done | step  8700 | w        1.23275 | loss        0.0117 | grad       -0.0342
      iter    87/100 done | step  8800 | w        1.23411 | loss        0.0321 | grad        0.0455
      iter    88/100 done | step  8900 | w        1.22806 | loss        0.0688 | grad       -0.0674
      iter    89/100 done | step  9000 | w        1.23307 | loss        0.0222 | grad        0.1727
      iter    90/100 done | step  9100 | w        1.23491 | loss        0.0251 | grad       -0.0312
      iter    91/100 done | step  9200 | w        1.23106 | loss        0.0519 | grad        0.0098
      iter    92/100 done | step  9300 | w        1.23303 | loss        0.0437 | grad       -0.1625
      iter    93/100 done | step  9400 | w        1.23277 | loss        0.0708 | grad       -0.2593
      iter    94/100 done | step  9500 | w        1.22882 | loss        0.0636 | grad        0.0411
      iter    95/100 done | step  9600 | w        1.24039 | loss        0.0208 | grad       -0.0708
      iter    96/100 done | step  9700 | w        1.23499 | loss        0.0839 | grad        0.0251
      iter    97/100 done | step  9800 | w        1.23123 | loss        0.0467 | grad        0.0922
      iter    98/100 done | step  9900 | w        1.23677 | loss        0.0182 | grad        0.0165
      iter    99/100 done | step 10000 | w        1.22925 | loss        0.0492 | grad        0.0620

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(12,3))

axes[0].plot(info_dict['trace_loss_list'], '.-');
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list'], '.-');
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list'], '.-');
axes[2].set_title('w');
```

### Try with even smaller batch_size of 1

```python
# TODO write code here
```

### Discussion 3b: What happens here with this smaller batch size? Is it converging?

# Part 5: SGD for MLPClassifier

Let's revisit the XOR dataset from our previous lab, and try SGD for it.

```python
def make_xor_dataset(n_per_blob=50, stddev=0.4, random_state=0):
    random_state = np.random.RandomState(random_state)
    cov_22 = np.square(stddev) * np.eye(2)
    x_00 = random_state.multivariate_normal([-1, -1], cov_22, size=n_per_blob)
    x_01 = random_state.multivariate_normal([-1, +1], cov_22, size=n_per_blob)
    x_10 = random_state.multivariate_normal([+1, -1], cov_22, size=n_per_blob)
    x_11 = random_state.multivariate_normal([+1, +1], cov_22, size=n_per_blob)

    N = n_per_blob * 4
    x_N2 = np.vstack([x_00, x_11, x_01, x_10])
    assert x_N2.shape == (N, 2)

    y_N = np.hstack([np.ones(N//2), np.zeros(N//2)]).astype(np.int32)
    assert y_N.shape == (N,)

    # Shuffle the order
    perm_ids = random_state.permutation(N)
    x_N2 = x_N2[perm_ids].copy()
    y_N = y_N[perm_ids].copy()

    return x_N2, y_N
```

```python
x_tr_N2, y_tr_N = make_xor_dataset(n_per_blob=50)
```

```python
plt.figure(figsize=(4,4))
plt.plot(x_tr_N2[y_tr_N==0,0], x_tr_N2[y_tr_N==0,1], 'rx', label='y=0', mew=2);
plt.plot(x_tr_N2[y_tr_N==1,0], x_tr_N2[y_tr_N==1,1], 'b+', label='y=1', mew=2);

plt.xlabel('x_1');
plt.ylabel('x_2');
plt.legend(bbox_to_anchor=(1.0, 0.5));
```

# Setup: Create Utility function for visualizing classifier predictions

You do NOT need to understand the details of this function. We'll just use it as is.

```python
def plot_pretty_probabilities_for_clf(
        clf,
        ax=None,
        x1_grid=(-2.1, 2.1, 50), x2_grid=(-2.1, 2.1, 50),
        x1_ticks=[-1, 0, 1], x2_ticks=[-1, 0, 1],
        do_show_colorbar=False,
        c_ticks=np.asarray([0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]),
        c_num_levels=100,
        ):
    ''' Display predicted probabilities from classifier as color contour plot

    Args
    ----
    clf : sklearn object that implements classifier API
    ax : matplotlib axis handle, or None
        If provided, will use axis handle as primary axis to plot on.
        If None, will use the current handle, or make new one as necessary.
    x1_grid : tuple-like or array-like
        If tuple of length 3, interpreted as args to np.linspace
        Otherwise, cast to array and assumed to be a 1d grid of x1 values
    x2_grid : tuple-like or array-like
        If tuple of length 3, interpreted as args to np.linspace
        Otherwise, cast to array and assumed to be a 1d grid of x2 values

    '''
    # Activate the current axis, if necessary
    if ax is None:
        cur_ax = plt.gca()
    else:
        cur_ax = ax
        plt.sca(cur_ax)

    # Create dense grid of x1 and x2 values
    # useful for visualizing probabilities
    if isinstance(x1_grid, tuple) and len(x1_grid) == 3:
        x1_grid = np.linspace(x1_grid[0], x1_grid[1], x1_grid[2])
    if isinstance(x2_grid, tuple) and len(x2_grid) == 3:
        x2_grid = np.linspace(x2_grid[0], x2_grid[1], x2_grid[2])
    x1_grid = np.asarray(x1_grid).flatten()
    x2_grid = np.asarray(x2_grid).flatten()

    c_levels = np.linspace(0.0, 1.0, c_num_levels)

    # Get regular grid of G x H points, where each point is an (x1, x2) location
    G = x1_grid.size
    H = x2_grid.size
    x1_GH, x2_GH = np.meshgrid(x1_grid, x2_grid)

    # Combine the x1 and x2 values into one array
    # Flattened into M = G x H rows
    # Each row of x_M2 is a 2D vector [x_m1, x_m2]
    x_M2 = np.hstack([x1_GH.flatten()[:,np.newaxis], x2_GH.flatten()[:,np.newaxis]])

    # Predict proba for each point in the flattened grid
    yproba1_M = clf.predict_proba(x_M2)[:,1]
    # Reshape the M probas into the GxH 2D field
    yproba1_GH = np.reshape(yproba1_M, x1_GH.shape)
    # Contour plot
    cmap = plt.cm.RdYlBu
    my_contourf_h = plt.contourf(
            x1_GH, x2_GH, yproba1_GH,
            levels=c_levels, vmin=0, vmax=1.0,
            cmap=cmap, alpha=0.5)

    # Edit the ticks observed
    if x1_ticks is not None:
        plt.xticks(x1_ticks, x1_ticks);
    if x2_ticks is not None:
        plt.yticks(x2_ticks, x2_ticks);
    if do_show_colorbar:
        left, bottom, width, height = plt.gca().get_position().bounds
        cax = plt.gcf().add_axes([left + 1.1*width, bottom, 0.03, height])
        plt.colorbar(my_contourf_h, orientation='vertical', cax=cax, ticks=c_ticks);
        plt.sca(cur_ax);
```

## Try an MLP with 2 hidden units, batch size of 25

This time, we'll use SGD as built in solver

```python
mlp_2hidden_run5 = sklearn.neural_network.MLPClassifier(
    hidden_layer_sizes=[2],
    activation='relu',
    solver='sgd',
    learning_rate_init=0.1,
    random_state=5,
    batch_size=25,
    )
```

```python
# Fit the model to training data

mlp_2hidden_run5.fit(x_tr_N2, y_tr_N)
```

    MLPClassifier(batch_size=25, hidden_layer_sizes=[2], learning_rate_init=0.1,
                  random_state=5, solver='sgd')

### Visualize the results using our utility function

```python
plt.figure(figsize=(4,4))
plot_pretty_probabilities_for_clf(mlp_2hidden_run5, do_show_colorbar=True, ax=plt.gca());

plt.plot(x_tr_N2[y_tr_N==0,0], x_tr_N2[y_tr_N==0,1], 'rx', label='y=0', mew=2);
plt.plot(x_tr_N2[y_tr_N==1,0], x_tr_N2[y_tr_N==1,1], 'b+', label='y=1', mew=2);
```

```python
### Visualize the trace of the loss

plt.plot(mlp_2hidden_run5.loss_curve_, 'k.-')
```

# Try several trials with batch_size of 25

```python
for trial in [1, 2, 3, 4, 5]:
    mlp_2hidden_batchsize25 = sklearn.neural_network.MLPClassifier(
        hidden_layer_sizes=[2],
        activation='relu',
        solver='sgd',
        learning_rate_init=0.05,
        random_state=trial,
        batch_size=25,
        )

    # Fit the model to training data
    mlp_2hidden_batchsize25.fit(x_tr_N2, y_tr_N)

    # Visualize the trace of the loss
    plt.plot(mlp_2hidden_batchsize25.loss_curve_, '.-', label='trial %d' % trial)

plt.legend();
```

### Now try several trials with a batch_size of 1

```python
for trial in [1, 2, 3, 4, 5]:
    mlp_2hidden_batchsize1 = sklearn.neural_network.MLPClassifier(
        hidden_layer_sizes=[2],
        activation='relu',
        solver='sgd',
        learning_rate_init=0.05,
        random_state=trial,
        batch_size=1,
        )

    # Fit the model to training data
    mlp_2hidden_batchsize1.fit(x_tr_N2, y_tr_N)

    # Visualize the trace of the loss
    plt.plot(mlp_2hidden_batchsize1.loss_curve_, '.-', label='trial %d' % trial)

plt.legend();
```

### Discussion 4a: What is happening here?

```python

```

```python

```


# Corollary 0.0.4 in $\mathbb R^2$
In this worksheet we will run through the proof of Corollary 0.0.4 from Vershynin. We will "pythonize" the proof step-by-step in the case of a polytope in $\mathbb R^2$ and visualize it. Please fill in the code wherever indicated. Here is the corollary (slightly generalized) for reference:

**Corollary 0.0.4 (Generalized)**: Let $P$ be a polytope in $\mathbb R^n$ with $N$ vertices. Then $P$ can be covered by at most $N^{\lceil (\text{diam}(T)/\epsilon)^2 \rceil}$ Euclidean balls of radii $\epsilon > 0$.

```python
# Some useful imports:
import numpy as np
import matplotlib.pyplot as plt
import itertools as it
import math
```

## The Proof of Corollary 0.0.4 in $\mathbb R^2$

Fix $\epsilon > 0$ and a polytope $P$ in $\mathbb R^n$ (in our case $n = 2$). Denote by $T$ the set of vertices of $P$.

```python
# Set epsilon to your favorite positive number
epsilon = 1.0

# Represent a polytope P of your choice in R2 by listing its points clockwise
# Represent the points as numpy arrays of length 2
P =  [] # TODO: Write this!

# Set N to the number of vertices of P (don't hard-code it in if you want to be able to change P easily later)
N = -1 # TODO: Write this!
```

Let us define the centers of the balls as follows. \
Let $k := \lceil (\text{diam}(T)/\epsilon)^2 \rceil$. Recall that the $\text{diam}(T) = \sup_{x, y \in T} \lvert x - y \rvert$.

```python
# Compute diam(T)
# TODO: Write this!
diamT = 0

# Compute k
k = 0  # TODO: Write this!
```

Consider the set
\begin{equation}
\mathcal N := \left\{ \frac{1}{k} \sum_{j=1}^k x_j : x_j \text{ are vertices of } P \right\}
\end{equation}

```python
# Construct \mathcal N
calN = []

# This gives an iterator over all combinations
# of k elements of P with replacement.
combinations = it.combinations_with_replacement(P, k)

# Compute the vector average and append it to calN
for comb in combinations:
    # TODO: Write this!
    vec_avg = np.array([0,0])

    calN.append(vec_avg)
```

We claim that the family of $\epsilon$-balls centered at $\mathcal N$ satisfy the conclusion of the corollary. To check this, note that the polytope $P$ is the convex hull of the set of its vertices. Thus we apply Theorem 0.0.2 to any point $x \in P = \text{conv}(T)$ and deduce that $x$ is within distance $\text{diam(T)} / \sqrt k \leq \epsilon$ from some point in $\mathcal N$. This shows that the $\epsilon$-balls centered at $\mathcal N$ indeed cover $P$.

```python
# We visualize the covering here
# Feel free to play around with the visualization!
scale = 10
figure, axes = plt.subplots(figsize=(scale, scale))
axes.axis('equal')
axes.set_xlim([-1, 1.5])
axes.set_ylim([-1,1.5])
plt.fill([p[0] for p in P], [p[1] for p in P], 'y', fill = True)
plt.plot([p[0] for p in calN], [p[1] for p in calN], 'or')
for p in calN:
    axes.add_artist(plt.Circle((p[0], p[1]), epsilon, fill = False))
```

We can bound the cardinality of $\mathcal N$ by noting that there are $N^k$ ways to choose $k$ out of $N$ vertices with repetition. Thus $|\mathcal N| \leq N^k = N^{\lceil (\text{diam}(T)/\epsilon)^2 \rceil}$. In fact we can be more clever by noticing that the order in which we choose the elements does not matter (this is addressed in exercise 0.0.6).

## Further Questions
At least in $\mathbb R^2$, Corollary 0.0.4 is rather wasteful. How can we come up with a more efficient covering of a polytope?
Is there a way to cleverly construct a subset of $\mathcal N$ that gets the job done?

Copyright (c) 2020 TRIPODS/GradStemForAll 2020 Team