# Lecture 2 - Introduction to Probability Theory

> Probability theory is nothing but common sense reduced to calculation. P. Laplace (1812)

## Objectives
+ To use probability theory to represent states of knowledge.
+ To use probability theory to extend Aristotelian logic to reason under uncertainty.
+ To learn about the **pruduct rule** of probability theory.
+ To learn about the **sum rule** of probability theory.
+ What is a **random variable**?
+ What is a **discrete random variable**?
+ When are two random variable **independent**?
+ What is a **continuous random variable**?
+ What is the **cumulative distribution function**?
+ What is the **probability density function**?

## Readings

Before coming to class, please read the following:

+ [Chapter 1 of Probabilistic Programming and Bayesian Methods for Hackers](http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter1_Introduction/Chapter1.ipynb)
+ [Chapter 1](http://home.fnal.gov/~paterno/images/jaynesbook/cc01p.pdf) of (Jaynes, 2003).
+ [Chapter 2](http://home.fnal.gov/~paterno/images/jaynesbook/cc02p.pdf) of (Jaynes, 2003) (skim through).

## The basic desiderata of probability theory
It is actually possible to derive the rules of probability based on a system of common sense requirements.
Paraphrasing
[Chapter 1](http://home.fnal.gov/~paterno/images/jaynesbook/cc01p.pdf) of \cite{jaynes2003}),
we would like our system to satisfy the following desiderata:

1) *Degrees of plausibility are represented by real numbers.*

2) *The system should have a qualitative correspondance with common sense.*

3) *The system should be consistent in the sense that:*

   + *If a conclusion can be reasoned out in more than one way, then every possible way must lead to the same result.*

   + *All the evidence relevant to a question should be taken into account.*

   + *Equivalent states of knowledge must be represented by equivalent plausibility assignments.*

## How to speak about probabilities?
Let
+ A be a logical sentence,
+ B be another logical sentence, and
+ and I be all other information we know.

There is no restriction on what A and B may be as soon as none of them is a contradiction.
We write as a shortcut:
$$
\mbox{not A} \equiv \neg,
$$
$$
A\;\mbox{and}\;B \equiv A,B \equiv AB,
$$
$$
A\;\mbox{or}\;B \equiv A + B.
$$

We **write**:
$$
p(A|BI),
$$
and we **read**:
> the probability of A being true given that we know that B and I is true

or (assuming knowledge I is implied)

> the probability of A being true given that we know that B is true

or (making it even shorter)

> the probability of A given B.

$$
p(\mbox{something} | \mbox{everything known}) = \mbox{probability samething is true conditioned on what is known}.
$$

$p(A|B,I)$ is just a number between 0 and 1 that corresponds to the degree of plaussibility of A conditioned on B and I.
0 and 1 are special.

+ If
$$
p(A|BI) = 0,
$$
we say that we are certain that A is false if B is true.

+ If
$$
p(A|BI) = 1,
$$
we say that we are certain that A is false if B is false.

+ If
$$
p(A|BI) \in (0, 1),
$$
we say that we are uncertain about A given that B is false.
Depending on whether $p(A|B,I)$ is closer to 0 or 1 we beleive more on one possibiliy or another.
Complete ignorance corresponds to a probability of 0.5.

## The rules of probability theory

According to
[Chapter 2](http://home.fnal.gov/~paterno/images/jaynesbook/cc02m.pdf) of \cite{jaynes2003} the desiderata are enough
to derive the rules of probability.
These rules are:

+ The **obvious rule** (in lack of a better name):
$$
p(A | I) + p(\neg A | I) = 1.
$$

+ The **product rule** (also known as the Bayes rule or Bayes theorem):
$$
p(AB|I) = p(A|BI)p(B|I).
$$
or
$$
p(AB|I) = p(B|AI)p(A|I).
$$

These two rules are enough to compute any probability we want. Let us demonstrate this by a very simple example.

### Example: Drawing balls from a box without replacement
Consider the following example of prior information I:

> We are given a box with 10 balls 6 of which are red and 4 of which are blue.
The box is sufficiently mixed so that so that when we get a ball from it, we don't know which one we pick.
When we take a ball out of the box, we do not put it back.

Let A be the sentence:

> The first ball we draw is blue.

Intuitively, we would set the probability of A equal to:
$$
p(A|I) = \frac{4}{10}.
$$

This choice can actually be justified, but we will come to this later in this course.
From the "obvious rule", we get that the probability of not drawing a blue ball, i.e.,
the probability of drawing a red ball in the first draw is:
$$
p(\neg A|I) = 1 - p(A|I) = 1 - \frac{4}{10} = \frac{6}{10}.
$$

Now, let B be the sentence:

> The second ball we draw is red.

What is the probability that we draw a red ball in the second draw given that we drew a blue ball in the first draw?
Just before our second draw, there remain 9 bals in the box, 3 of which are blue and 6 of which are red.
Therefore:
$$
p(B|AI) = \frac{6}{9}.
$$

We have not used the product rule just yet. What if we wanted to find the probability that we draw a blue during the first draw and a red during the second draw? Then,
$$
p(AB|I) = p(A|I)p(B|AI) = \frac{4}{10}\frac{6}{9} = \frac{24}{90}.
$$

What about the probability o a red followed by a blue? Then,
$$
p(\neg AB|I) = p(\neg A|I)p(B|AI) = \left[1 - p(A|I) \right]p(B|\neg AI) = \frac{6}{10}\frac{5}{9} = \frac{30}{90}.
$$

### Other rules of probability theory

All the other rules of probability theory can be derived from these two rules.
To demonstrate this, let's prove that:
$$
p(A + B|I) = p(A|I) + p(B|I) - p(AB|I).
$$
Here we go:
\begin{eqnarray*}
p(A+B|I) &=& 1 - p(\neg A \neg B|I)\;\mbox{(obvious rule)}\\
         &=& 1 - p(\neg A|\neg BI)p(\neg B|I)\;\mbox{(product rule)}\\
         &=& 1 - [1 - p(A |\neg BI)]p(\neg B|I)\;\mbox{(obvious rule)}\\
         &=& 1 - p(\neg B|I) + p(A|\neg B I)p(\neg B|I)\\
         &=& 1 - [1 - p(B|I)] + p(A|\neg B I)p(\neg B|I)\\
         &=& p(B|I) + p(A|\neg B I)p(\neg B|I)\\
         &=& p(B|I) + p(A\neg B|I)\\
         &=& p(B|I) + p(\neg B|AI)p(A|I)\\
         &=& p(B|I) + [1 - p(B|AI)] p(A|I)\\
         &=& p(B|I) + p(A|I) - p(B|AI)p(A|I)\\
         &=& p(A|I) + p(B|I) - p(AB|I).
\end{eqnarray*}

### The sum rule
Now consider a finite set of logical sentences, $B_1,\dots,B_n$ such that:
1. One of them is definitely true:
    $$
    p(B_1+\dots+B_n|I) = 1.
    $$
2. They are mutually exclusive:
    $$
    p(B_iB_j|I) = 0,\;\mbox{if}\;i\not=j.
    $$

The **sum rule** states that:
    $$
    P(A|I) = \sum_i p(AB_i|I) = \sum_i p(A|B_i I)p(B_i|I).
    $$
We can prove this by induction, but let's just prove it for $n=2$:
$$
\begin{array}{ccc}
p(A|I) &=& p[A(B_1+B_2|I]\\
       &=& p(AB_1 + AB_2|I)\\
       &=& p(AB_1|I) + p(AB_2|I) - p(AB_1B_2|I)\\
       &=& p(AB_1|I) + p(AB_2|I),
\end{array}
$$
since
$$
p(AB_1B_2|I) = p(A|B_1B_2|I)p(B_1B_2|I) = 0.
$$

Let's go back to our example. We can use the sum rule to compute the probability of getting a red ball on the second draw independently of what we drew first. This is how it goes:
$$
\begin{array}{ccc}
p(B|I) &=& p(AB|I) + p(\neg AB|I)\\
       &=& p(B|AI)p(A|I) + p(B|\neg AI) p(\neg A|I)\\
       &=& \frac{6}{9}\frac{4}{10} + \frac{5}{9}\frac{6}{10}\\
       &=& \dots
\end{array}
$$

### Example: Medical Diagnosis

This example is a modified version of the one found in [Lecture 1](http://www.zabaras.com/Courses/BayesianComputing/IntroToProbabilityAndStatistics.pdf) of the Bayesian Scientific Computing course offered during Spring 2013 by Prof. N. Zabaras at Cornell University.

We are going to examine the usefullness of a new tuberculosis test. Let the prior information, I, be:

> The percentage of the population infected by tuberculosis is 0.4%. We have run several experiments and determined that:
 + If a tested patient has the disease, then 80% of the time the test comes out positive.
 + If a tested patient does not have the disease, then 90% of the time, the test comes out negative.

Suppose now that you administer this test to a patient and that the result is positive. How confident are you that the patient does indeed have the disease?

Let's use probability theory to answer this question. Let A be the event:
> The patient's test is positive.

Let B be the event:
> The patient has tuberculosis.

According to the prior information, we have:
$$
p(B|I) = p(\mbox{has tuberculosis}|I) = 0.004,
$$
and
$$
p(A|B,I) = p(\mbox{test is positive}|\mbox{has tuberculosis},I) = 0.8.
$$
Similarly,
$$
p(A|\neg B, I) = p(\mbox{test is positive}|\mbox{does not have tuberculosis}, I) = 0.1.
$$
We are looking for:
$$
\begin{array}{ccc}
p(\mbox{has tuberculosis}|\mbox{test is positive},I) &=& P(B|A,I)\\
&=& \frac{p(AB|I)}{p(A|I)} \\
&=& \frac{p(A|B,I)p(B|I)}{p(A|B,I)p(B|I) + p(A|\neg B, I)p(\neg B|I)}\\
&=& \frac{0.8\times 0.004}{0.8\times 0.004 + 0.1 \times 0.996}\\
&\approx& 0.031.
\end{array}
$$
How much would you pay for such a test?

## Conditional Independence

We say that $A$ and $B$ are **independent** (conditional on I), and write,
$$
A\perp B|I,
$$
if knowledge of one does not yield any information about the other. Mathematically, by $A\perp B|I$, we mean that:
$$
p(A|B,I) = p(A|I).
$$
Using the product rule, we can easily show that:
$$
A\perp B|I \iff p(AB|I) = p(A|I)p(B|I).
$$

### Question
+ Give an example of $I, A$ and $B$ so that $A\perp B|I$.

Now, let $C$ be another event. We say that $A$ and $B$ are **independent** conditional on $C$ (and I), and write:
$$
A\perp B|C,I,
$$
if knowlege of $C$ makes information about $A$ irrelevant to $B$ (and vice versa). Mathematically, we mean that:
$$
p(A|B,C,I) = p(A|C,I).
$$

### Question
+ Give an example of $I,A,B,C$ so that $A\perp B|C,I$.

## Random Variables

The formal mathematical definition of a random variable involves measure theory and is well beyond the scope of this course.
Fortunately, we do not have to go through that route to get a theory that is useful in applications.
For us, a **random variable** $X$ will just be a variable of our problem whose value is unknown to us.
Note that, you should not take the word "random" too literally.
If we could, we would change the name to **uncertain** or **unknown** variable.
A random variable could correspond to something fixed but unknown, e.g., the number of balls in a box,
or it could correspond to something truely random, e.g., the number of particles that hit a [Geiger counter](https://en.wikipedia.org/wiki/Geiger_counter) in a specific time interval.

### Discrete Random Variables
We say that a random variable $X$ is discrete, if the possible values it can take are discrete (possibly countably infinite).
We write:
$$
p(X = x|I)
$$
and we read "the probability of $X$ being $x$".
If it does not cause any ambiguity, sometimes we will simplify the notation to:
$$
p(x) \equiv p(X=x|I).
$$
Note that $p(X=x)$ is actually a discrete function of $x$ which depends on our beliefs about $X$.
The function $p(x) = p(X=x|I)$ is known as the probability density function of $X$.

Now let $Y$ be another random variable.
The **sum rule** becomes:
$$
p(X=x|I) = \sum_{y}p(X=x,Y=y|I) = \sum_y p(X=x|Y=y,I)p(Y=y|I),
$$
or in simpler notation:
$$
p(x) = \sum_y p(x,y) = \sum_y p(x|y)p(y).
$$
The function $p(X=x, Y=y|I) \equiv p(x, y)$ is known as the joint *probability mass function* of $X$ and $Y$.

The **product rule** becomes:
$$
p(X=x,Y=y|I) = p(X=x|Y=y,I)p(Y=y|I),
$$
or in simpler notation:
$$
p(x,y) = p(x|y)p(y).
$$

We say that $X$ and $Y$ are **independent** and write:
$$
X\perp Y|I,
$$
if knowledge of one does not yield any information about the other.
Mathematically, $Y$ gives no information about $X$ if:
$$
p(x|y) = p(x).
$$
From the product rule, however, we get that:
$$
p(x) = p(x|y) = \frac{p(x,y)}{p(y)},
$$
from which we see that the joint distribution of $X$ and $Y$ must factorize as:
$$
p(x, y) = p(x) p(y).
$$
It is trivial to show that if this factorization holds, then
$$
p(y|x) = p(y),
$$
and thus $X$ yields no information about $Y$ either.

### Continuous Random Variables
A random variable $X$ is continuous if the possible values it can take are continuous. The probability of a continuous variable getting a specific value is always zero. Therefore, we cannot work directly with probability mass functions as we did for discrete random variables. We would have to introduce the concepts of the **cumulative distribution function** and the **probability density function**. Fortunately, with the right choice of mathematical symbols, the theory will look exactly the same.

Let us start with a real continuous random variable $X$, i.e., a random variable taking values in the real line $\mathbb{R}$. Let $x \in\mathbb{R}$ and consider the probability of $X$ being less than or equal to $x$:
$$
F(x) := p(X\le x|I).
$$
$F(x)$ is known as the **cumulative distribution function** (CDF). Here are some properties of the CDF whose proof is
left as an excersise:

+ The CDF starts at zero and goes up to one:
$$
F(-\infty) = 0\;\mbox{and}\;F(+\infty) = 1.
$$

+ $F(x)$ is an increasing function of $x$, i.e.,
$$
x_1 \le x_2 \implies F(x_1)\le F(x_2).
$$

+ The probability of $X$ being in the interval $[x_1,x_2]$ is:
$$
p(x_1 \le X \le x_2|I) = F(x_2) - F(x_1).
$$

Now, assume that the derivative of $F(x)$ with respect to $x$ exists.
Let us call it $f(x)$:
$$
f(x) = \frac{dF(x)}{dx}.
$$
Using the fundamental theorem of calculus, it is trivial to show Eq. (\ref{eq:CDF_prob}) implies:
\begin{equation}
p(x_1 \le X \le x_2|I) = \int_{x_1}^{x_2}f(x)dx.
\end{equation}
$f(x)$ is known as the **probability density function** (PDF) and it is measured in probability per unit of $X$.
To see this note that:
$$
p(x \le X \le x + \delta x|I) = \int_{x}^{x+\delta x}f(x')dx' \approx f(x)\delta x,
$$
so that:
$$
f(x) \approx \frac{p(x \le X \le x + \delta x|I)}{\delta x}.
$$

The PDF should satisfy the following properties:

+ It should be positive
$$
f(x) \ge 0,
$$
+ It should integrate to one:
$$
\int_{-\infty}^{\infty} f(x) dx = 1.
$$

#### Notation about the PDF of continuous random variables
In order to make all the formulas of probability theory the same, we define for a continuous random variable $X$:
$$
p(x) := f(x) = \frac{dF(x)}{dx} = \frac{d}{dx}p(X \le x|I).
$$
But keep in mind, that if $X$ is continuous $p(x)$ is not a probability but a probability density.
That is, it needs a $dx$ to become a probability.

Let the PDF $p(x)$ of $X$ and the PDF $p(y)$ of $Y$ ($Y$ is another continuous random variable).
We can find the PDF of the random variable $X$ conditioned on $Y$, i.e., the PDF of $X$ if $Y$ is directly observed.
This is the **product rule** for continuous random variables:
\begin{equation}
\label{eq:continuous_bayes}
p(y|x) = \frac{p(x, y)}{p(y)},
\end{equation}
where $p(x,y)$ is the **joint PDF** of $X$ and $Y$.
The **sum rule** for continous random variables is:
\begin{equation}
\label{eq:continuous_sum}
p(x) = \int p(x, y) dy = \int p(x | y) p(y) dy.
\end{equation}

The similarity between these rules and the discrete ones is obvious.
We have prepared a table to help you remember it.

| Concept | Discrete Random Variables | Continuous Random Variables |
|---|---------------|-----------------|
|$p(x)$| in units of robability | in units of probability per unit of $X$|
|sum rule| $\sum_y p(x,y) = \sum_y p(x|y)p(y)$ | $\int_y p(x,y) dy = \int_y p(x|y) p(y)$|
|product rule| $p(x,y) = p(x|y)p(y)$ | $p(x,y) = p(x|y)p(y)$|

## Expectations

Let $X$ be a random variable. The expectation of $X$ is defined to be:
$$
\mathbb{E}[X] := \mathbb{E}[X | I] = \int x p(x) dx.
$$
Now let $g(x)$ be any function. The expectation of $g(X)$, i.e., the random variable defined after passing $X$ through $g(\cdot)$, is:
$$
\mathbb{E}[g(X)] := \mathbb{E}[g(X)|I] = \int g(x)p(x)dx.
$$
As usual, calling $\mathbb{E}[\cdot]$ is not a very good name.
You may think of $\mathbb{E}[g(X)]$ as the expected value of $g(X)$, but do not take it too far.
Can you think of an example in which the expected value is never actually observed?

### Conditional Expectation
Let $X$ and $Y$ be two random variables. The conditional expectation of $X$ given $Y=y$ is defined to be:
$$
\mathbb{E}[X|Y=y] := \mathbb{E}[X|Y=y,I] = \int xp(x|y)dx.
$$

### Properties of Expectations

The following properties of expectations of random variables are extremely helpful. In what follows, $X$ and $Y$ are random variables and $c$ is a constant:

+ Sum of random variable with a constant:
$$
\mathbb{E}[X+c] = \mathbb{E}[X] + c.
$$

+ Sum of two random variables:
$$
\mathbb{E}[X+Y] = \mathbb{E}[X] + \mathbb{E}[Y].
$$

+ Product of random variable with constant:
$$
\mathbb{E}[cX] = c\mathbb{E}[X].
$$

+ If $X\perp Y$, then:
$$
\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y].
$$
**NOTE**: This property does not hold if $X$ and $Y$ are not independent!

+ If $f(\cdot)$ is a convex function, then:
$$
f(\mathbb{E}[X]) \le \mathbb{E}[f(X)].
$$
**NOTE**: The equality holds only if $f(\cdot)$ is linear!

### Variance of a Random Variable

The variance of $X$ is defined to be:
$$
\mathbb{V}[X] = \mathbb{E}\left[X - \mathbb{E}[X])^2\right].
$$
It is easy to prove (and a very useful formulat to remember), that:
$$
\mathbb{V}[X] = \mathbb{E}[X^2] - \left(\mathbb{E}[X]\right)^2.
$$

### Covariance of Two Random Variables
Let $X$ and $Y$ be two random variables.
The covariance between $X$ and $Y$ is defined to be:
$$
\mathbb{C}[X, Y] = \mathbb{E}\left[\left(X - \mathbb{E}[X]\right)
\left(Y-\mathbb{E}[Y]\right)\right]
$$

### Properties of the Variance
Let $X$ and $Y$ be random variables and $c$ be a constant.
Then:
+ Sum of random variable with a constant:
$$
\mathbb{V}[X + c] = \mathbb{V}[X].
$$
+ Product of random variable with a constant:
$$
\mathbb{V}[cX] = c^2\mathbb{V}[X].
$$
+ Sum of two random variables:
$$
\mathbb{V}[X+Y] = \mathbb{V}[X] + \mathbb{V}[Y] + 2\mathbb{C}(X,Y).
$$
+ Sum of two independent random variables:
$$
\mathbb{V}[X+Y] = \mathbb{V}[X] + \mathbb{V}[Y].
$$

# References

(Jaynes, 2003) E T Jaynes, ``_Probability Theory: The Logic of Science_'',  2003.  [online](http://bayes.wustl.edu/etj/prob/book.pdf)


#### _Speech Processing Labs 2020: Module 1_

```python
## Run this first!
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

from dspMisc import *
```

# 5 Interpreting the  Discrete Fourier Transform

### Learning Outcomes
* Understand how sampling rate effects the DFT output
* Understand what the DFT leakage is.

### Need to know
* Topic Videos: Fourier Analysis, Frequency Domain
* [Digital Signals: Sampling sinusoids](./sp-m1-3-sampling-sinusoids.ipynb)
* [The Discrete Fourier Transform](./sp-m1-4-discrete-fourier-transform.ipynb) _(desirable)_

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 5.1 A Very Quick DFT recap
The previous worksheet went through the mechanics of the Discrete Fourier Transform (DFT).  The DFT input and output are broadly:

* **Input:**  $N$ amplitude samples over time
    * $x[n]$, for $n=1..N$ (i.e. a time series of $N$ samples)

* **Output:** the dot product (i.e., the similiarity) between the input and $N$ sinusoids with different frequencies
    * DFT[k] $= Me^{j\phi}$, i.e. a complex number with **magnitude** $M$ and **phase** angle $\phi$

The outputs are calculated using the following formula for $k=0,...N-1$.

$$
\begin{align}
DFT[k] &= \sum_{n=0}^{N-1} x[n] e^{-j \frac{2\pi nk}{N}} \\
&= \sum_{n=0}^{N-1} x[n]\big[\cos\big(\frac{2\pi nk}{N} \big) - j \sin\big(\frac{2\pi nk}{N} \big) \big]
\end{align}
$$

We saw how the complex numbers in this equation $e^{-j2\pi nk/N}$ for $k=1...N-1$ represent sinusoids with specific frequencies as projections of phasors. You can just think of a phasor as an analogue clock with one hand (vector) ticking (rotating) around a clockface (i.e. a circle), where the length of the hand is the amplitude of that wave, and how fast it goes around the clock is it's frequency.

Each DFT[k] output essentially tells us how strongly the input is repeats itself (i.e. is periodic) at the frequency of the $k$th DFT phasor. So, we talk about the DFT outputs as providing a **frequency response**.

Since the the DFT outputs are complex numbers, we can talk about them in terms of magnitude and phase angle: The magnitude of DFT[k] tells us how much we'd weight the $k$-th phasor if we were to try to reconstruct the original input by adding all the DFT phasors together.  The phase of DFT[k] tells use whether we need to shift that wave along the time axis.

## 5.2 The DFT Frequency Response:  Which Frequencies?

You will have noticed in the previous worksheet that we only talked about relative frequencies above, e.g. the $e^{-j2\pi nk/N}$ phasor has $k$ times the frequency of the $e^{-j2\pi n/N}$ phasor.

Since the frequencies of the DFT phasors are all multiples of the $k=1$ phasor, we just have to figure out the frequency of that phasor and we'll know all the rest. Now, the $k=1$ phasor takes $N$ steps to complete a full cycle of the unit circle.  So, it's frequency depends on how long each of these steps takes.  That is, the **sampling rate**.

Assume we are sampling at a rate of $f_s$ samples per second. Then, we can calculate:

* How long between each sample (i.e. the **sampling time**):
    * $t_s = 1/f_s$  (seconds)

* How long does it take to take $N$ samples?
    * $t_s \times N$  (seconds x samples = seconds)

* How long will it take the $k=1$ phasor to make 1 complete cycle?
    * $T_1 = t_s \times N$  (seconds)
    * This is the **period** or **wavelength** of the phasor

* What is the **frequency** of the $k=1$ phasor?
    * $f_{min} = 1/T_1 = 1/(t_s N) $ (cycles/second)

* What is the **frequency** of the $k$th phasor?
    * $kf_{min}$ (cycles/second)

So, the important thing to remember is that the DFT outputs depend on:

* The **number of samples** in the input sequence, $N$
* The **sampling rate**, $f_s$ samples/second

### Example
Assume we have a sampling rate of $f_s = 32$ samples/second, and an input length of $N=16$.

We can then calculate the $k=1$ DFT output frequency by noting the following:

* The sampling time is: $t_s = 1/f_s = 1/32$ seconds
* So, the time it takes to complete one cycle is: $T_1 = t_sN = 1/32 \times 16 = 1/2$ seconds
* So, the frequency of DFT[1] is $f_{min} = 1/1/2 = 2$ cycles/second

### Exercise

* What's the frequency of the $k=5$ phasor in the previous example?

### Notes

```python

```

### Exercise

Try running this code which sets the input to be a sine wave with the same frequency as the the 1st DFT component. That is, `f_in = 1/wavlen_in`.

```python
N=64

#sampling rate:
f_s = 128

## sample time
t_s = 1/f_s

## Notice that the DFT component frequencies as completely determined by the number of samples N
## and the sampling frequency f_s.
## So if we wanted the input frequency to exactly match the frequency of the 1st DFT component we
## could just do this:

wavlen_in = t_s*N
f_in = 1/wavlen_in

print("f_s: %f\nt_s: %f\nwavlen: %f\nf_in %f" % (f_s, t_s, wavlen_in, f_in))

## indices of input sequence of size N
nsteps = np.array(range(N))

samples_per_cycle = wavlen_in/t_s
print("samples per cycle %f" % samples_per_cycle)

## Generate a sine wave with f_in matching the frequency of the k=1 DFT phasor
## You can look at the code for this function in the file XXX.py

x, time_steps = gen_sinusoid(frequency=f_in, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.sin)

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x, color='magenta')
timedom.plot(time_steps, x, color='magenta')
timedom.set_xlabel("time (s)")
timedom.set_ylabel("Amplitude")
timedom.set_title("Input: $x[n]$ in the time domain")
```

```python
## Calculate the frequencies associated with each DFT output
def get_dft_freqs_all(sample_rate, seq_len):
    ## Get the minimum frequency
    f_min = sample_rate/seq_len

    ## All the other frequencies are just integer multiples of the minimum frequency
    ## We have 0,...,seq_len-1 outputs
    dft_outs = np.arange(seq_len)

    return f_min * dft_outs
```

```python
## Find the DFT output frequencies for our current sample rate and input length
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)
print(dft_freqs)

```

    [  0.   2.   4.   6.   8.  10.  12.  14.  16.  18.  20.  22.  24.  26.
      28.  30.  32.  34.  36.  38.  40.  42.  44.  46.  48.  50.  52.  54.
      56.  58.  60.  62.  64.  66.  68.  70.  72.  74.  76.  78.  80.  82.
      84.  86.  88.  90.  92.  94.  96.  98. 100. 102. 104. 106. 108. 110.
     112. 114. 116. 118. 120. 122. 124. 126.]

```python
## Apply the DFT to the input and return the magnitudes and phases for each DFT output
mags, phases = get_dft_mag_phase(x, seq_len=N)

## Plot the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.scatter(dft_freqs, mags)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Magnitude response")

#print(mags)

## Plot the phases
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.scatter(dft_freqs, phases)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Phase angle (radians)")
fdom.set_title("Phase response")

#print(phases)

```

### Exercise

* Is the output what you'd expect given the input?
* Why is the phase non-zero for the DFT output's 1 and 63?
* What happens if you change the `amplitude` of the input sine wave? What about the `phase`?
* What happens to the range of DFT output frequencies if you increase the sampling rate `f_s` but keep the input sequence length `N` fixed?
* What happens to the range of DFT output frequencies for $k > N/2$? Can the DFT actually model those frequencies?

### Notes

```python

```

## 5.3 DFT on a compound waveform

Now, let's see what happens when we use a compound waverform as input to the DFT.  First we need to set the length of the sequence and the sample rate.

```python
## Set the number of samples N, sampling rate f_s
N=64

#sampling rate:
f_s = 64

## sample time
t_s = 1/f_s

print("N: %d\nf_s: %f\nt_s: %f" % (N, f_s, t_s))
```

    N: 64
    f_s: 64.000000
    t_s: 0.015625

Let's create a compound waveform as input with frequency components of 4Hz and 12Hz:

```python
## Now let's create a compound waveform as input
## We're setting gen_function=np.cos, meaning that each of our sinusoids is a cosine wave
x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

## Let's shift the phase of this one by 90 degrees (pi/2 rads)
x2, time_steps = gen_sinusoid(frequency=12, phase=np.pi/2, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

x_compound = x1 + x2

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x_compound, color='magenta')
timedom.plot(time_steps, x_compound, color='magenta')
timedom.set_xlabel("time (s)")
timedom.set_ylabel("Amplitude")
timedom.set_title("A compound waveform: $x_compound$ in the time domain")
```

You should see a somewhat complicated waveform whose overall pattern repeats 4 times in a second.  Now we do the DFT and look at the magnitude and phase values for the different DFT outputs.

```python
## Do the DFT on the compound waveform as above:
mags, phases = get_dft_mag_phase(x_compound, seq_len=N)
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))
#timedom.scatter(dft_freqs[:round(N)], mags[:round(N)])
fdom.scatter(dft_freqs[:round(N/2)], mags[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Magnitude response")
#print(mags)

## Plot the phases - let's just skip this for the moment
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.scatter(dft_freqs[:round(N/2)], phases[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Phase (rad)")
fdom.set_title("Phase response")
#print(phases)

```

### Exercise

* Do you see what you would expect given the input?
* Why does the number of non-zero phases differ from the number of non-zero magnitudes?

### Notes

```python

```

### Exercise

Explore the outputs of the DFT when you change the following parameters
* sampling rate `f_s`
* sequence length `N`
* frequency of the input sequence(s) `frequency` (in the function `gen_sinusoid`)

**Question**
* What happens if you set the frequency of one of the input components to be greater than the Nyquist frequency?
* What happens when the frequency of the input components doesn't match any of the DFT component frequencies?
(e.g. for the example above what if one of the input frequencies is not a whole number?)

### Notes

```python

```

## 5.4  Leakage

One of the main things to remember about the DFT is that you're calculating the correlation between the input and phasors with specific frequencies.  If your input exactly matches one of those phasor frequencies the magnitude response will show a positive magnitude for that phasor and zero for everything else.  However, if the input frequency falls between phasor frequencies (i.e. doesn't correspond to a specific DFT output), then you'll see spectral **leakage**.  The DFT outputs close to the input frequency will also get positive magnitudes, with the DFT output closest to the input frequency getting the highest magnitude.

The following code gives an example

```python
N=64
f_s = 64
f_in1 = 4.5 ## In between DFT output frequencies
f_in2 = 12  ## One of the DFT outputs

x1, time_steps = gen_sinusoid(frequency=f_in1, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=f_in2, phase=np.pi/2, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x_compound = x1  + x2

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x_compound, color='magenta')
timedom.plot(time_steps, x_compound, color='magenta')
timedom.set_xlabel("time (s)")
timedom.set_ylabel("Amplitude")
timedom.set_title("Leakage example (time domain)")

```

```python
## Do the DFT on the compound waveform as above:
mags, phases = get_dft_mag_phase(x_compound, seq_len=N)
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))

## Just plot the first N/2 frequencies since we know that they are the mirrored for k>N/2
fdom.scatter(dft_freqs[:round(N/2)], mags[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Leakage example: Magnitude response")
#print(mags)

```

### Leakage as the normalized sinc function

Leakage makes the DFT harder to interpret. However, we can derive the shape that leakage will have from the the DFT equation and some algebra about rectangular functions.  It turns out that leakage for a particular frequency follows the normalized **sinc** function:

$$
\begin{align}
X(m) &= \Big|\frac{AN}{2} \cdot \mathop{sinc}(c-m)\Big|\\
&= \Big|\frac{AN}{2} \cdot \frac{\sin(\pi(c-m))}{2\pi(c-m))}\Big|\\
\end{align}
$$

Where $A$ is the peak amplitude of the input, $N$ is the input sequence length, $c$ is the number of cycles completed in the input sequence time.  If $c$ is a whole number we just get the usual DFT response, but if $c$ is not a whole number, we get a spread across output frequency bins.

Let's check whether the sinc function matches what we get in the DFT. First we write a function to evaluate the leakage function in between our DFT outputs.

```python
## Calculate the approximated leakage as the sinc function
def calc_leakage(freq,  sample_rate, seqlen, amplitude=1):
    sequence_time = (1/sample_rate)*seqlen

    ## number of cycles in input for component
    c = freq * sequence_time
    print("c=", c)

    ## Interpolate between DFT ouput indices
    ms = np.arange(0, seqlen, 0.05)

    ## Approximated response - we could actually just return a function here, but
    ## let's just keep things concrete for now.
    leakage = np.abs(amplitude * seqlen * 0.5 * np.sinc((c-ms)))

    return leakage, ms * (sample_rate/seqlen)

```

Now let's plot the leakage predicted for our two input components separately (top) and added together (bottom)

```python
## Calculate the leakage function for our know input wave components
leakage1, ms = calc_leakage(f_in1, f_s, N)
leakage2, ms = calc_leakage(f_in2, f_s, N)

## Plot the magnitude response and the leakage function for each of our input components
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2), ylim=(-1, N))
fdom.scatter(dft_freqs, mags)

fdom.plot(ms, leakage1)
fdom.plot(ms, leakage2)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Leakage function for each input component frequency")

## Plot the magnitude response and the sum of the leakage functions
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2), ylim=(-1, N))
fdom.scatter(dft_freqs, mags)
fdom.plot(ms, leakage1 + leakage2, color='C5')
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Sum of leakage functions for input components")

## It fits, though not perfectly!
```

In the top figure, you should see that peaks (**main lobes**) of each leakage function are aligned with our input component frequencies. The peaks are at the same points as the DFT outputs when the sinusoidal component has a frequency matching the DFT output frequency (i.e. 12 above Hz).  Otherwise we see the spread of leakage around the input component frequency (i.e. around 4.5 Hz).

You'll also notice that our DFT magnitudes points don't always line up perfectly with our sinc curves.  Part of this is because the leakage function is an _approximation_.  Nevertheless, it's a very good approximation!

### Exercise

* What happens if the frequencies of the two components making the compound waveform are very close together?
    * e.g. make `f_in2=6`
* What if one of the components has a relatively small amplitude?
    * e.g. change `amplitude` of the second input to 0.4

### Notes

```python

```

### Shaping the lobes
The leakage sinc function has a peak around a specific frequency. If we want our DFT to be better able to distinguish between close frequencies, we need that peak, the **main lobe** to be narrower.  We also want the other peaks, the **side lobes** to be flatter.  We can achieve this using different **windowing methods** on our input.  This is why you see 'Hanning' as the default option for window method in the advanced spectrogram settings in praat.

We'll see some more examples of this when we look at different types of filters later.  But for now the main thing to observe is that leakage might give you the impression that specific frequency components are present in your waveform, when what's actually happening is that your waveform has frequencies that don't match the DFT phasors.  Another thing that can happen is that the peak for a low amplitude component gets subsumed into the main lobe of a nearby frequency component.  This might make you miss frequency components in your input!

```python

```

## 5.6 Extra:

Here's the composition and decomposition for the compound waveform one you saw in Notebook 2!

```python
N=1028
f_s = 1028
f_in1 = 8 ## In between DFT output frequencies
f_in2 = 20  ## One of the DFT outputs
f_in3 = 36  ## One of the DFT outputs

x1, time_steps = gen_sinusoid(frequency=f_in1, phase=0, amplitude=0.5, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=f_in2, phase=np.pi/2, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x3, time_steps = gen_sinusoid(frequency=f_in3, phase=0, amplitude=0.3, sample_rate=f_s, seq_length=N, gen_function=np.cos)

x_compound = x1  + x2 + x3

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
#timedom.scatter(time_steps, x_compound, color='magenta')
#timedom.set(xlim=(0, 0.2))
timedom.plot(time_steps, x_compound, color='magenta')
```

```python
## Plot the input

fig = plt.figure(figsize=(15,15))
gs = fig.add_gridspec(3,2)

ymax=2
timedom = fig.add_subplot(gs[0, 0])
timedom.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
timedom.plot(time_steps, x_compound, color='magenta')
timedom.set_title("waveform made from adding 3 sine waves")

s1 = fig.add_subplot(gs[0, 1])
s1.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
s1.plot(time_steps, x1)
s1.set_title("component 1: 8 Hz")

s2 = fig.add_subplot(gs[1, 1])
s2.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
s2.plot(time_steps, x2)
s2.set_title("component 2: 20 Hz")

s3 = fig.add_subplot(gs[2, 1])
s3.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
s3.plot(time_steps, x3)
s3.set_title("component 3: 36 Hz")

fig.savefig("./fig/compound_waveform.png")
```

```python
## Do the DFT on the compound waveform as above:
mags, phases = get_dft_mag_phase(x_compound, seq_len=N)
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot the magnitudes
fig,fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2))
## Just plot the first N/2 frequencies since we know that they are the mirrored for k>N/2
fdom.scatter(dft_freqs[:round(N/2)], mags[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Magnitude Response")
#print(mags)

```

```python

```

```python

```


```
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```
#export
from exp.nb_06 import *
```

## ConvNet

```
x_train,y_train,x_valid,y_valid = get_data()

x_train,x_valid = normalize_to(x_train,x_valid)
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)

nh,bs = 50,512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```
mnist_view = view_tfm(1,28,28)
cbfs = [Recorder,
        partial(AvgStatsCallback,accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```
nfs = [8,16,32,32]
```

```
def get_cnn_layers(data, nfs, **kwargs):
    nfs = [1] + nfs
    return [conv2d(nfs[i], nfs[i+1], 5 if i==0 else 3, **kwargs)
            for i in range(len(nfs)-1)] + [
        nn.AdaptiveAvgPool2d(1), Lambda(flatten), nn.Linear(nfs[-1], data.c)]

def get_runner(model, data, lr=0.6, loss_func = F.cross_entropy):
    opt = optim.SGD(model.parameters(), lr=lr)
    learn = Learner(model, opt, loss_func, data)
    return learn, Runner(cb_funcs=cbfs)

def get_learn_run(nfs, data, lr, cbs=None, leak=0.1, sub=0.4, maxv=6.):
    model = nn.Sequential(*get_cnn_layers(data, nfs, leak=leak, sub=sub, maxv=maxv))
    init_cnn(model)
    return get_runner(model, data, lr=lr)
```

```
learn,run = get_learn_run(nfs, data, 0.8)
```

```
%time run.fit(1, learn)
```

    train: [0.5218110546875, tensor(0.8417, device='cuda:0')]
    valid: [0.15871717529296875, tensor(0.9545, device='cuda:0')]
    CPU times: user 2.77 s, sys: 802 ms, total: 3.57 s
    Wall time: 3.48 s

## Batchnorm

### Custom

```
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.zeros(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))
        self.steps = 0

    def update_stats(self, x):
        self.steps += 1
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.detach_()
        self.vars.detach_()
        mom = 1. if self.steps==1 else self.mom
        self.means.lerp_(m, mom)
        self.vars.lerp_ (v, mom)

    def forward(self, x):
        if self.training: self.update_stats(x)
        x = (x-self.means).div_((self.vars+self.eps).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```
def conv2d(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.9)
```

```
with Hooks(learn.model, append_stats) as hooks:
    run.fit(1, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```
learn,run = get_learn_run(nfs, data, 1.0)
```

```
%time run.fit(3, learn)
```

    train: [0.4270889453125, tensor(0.8674, device='cuda:0')]
    valid: [0.1574052978515625, tensor(0.9538, device='cuda:0')]
    train: [0.141321171875, tensor(0.9570, device='cuda:0')]
    valid: [0.1134663330078125, tensor(0.9656, device='cuda:0')]
    train: [0.097023193359375, tensor(0.9705, device='cuda:0')]
    valid: [0.10197774658203125, tensor(0.9700, device='cuda:0')]
    CPU times: user 3.75 s, sys: 449 ms, total: 4.19 s
    Wall time: 4.21 s

### Bias corrected

See XL file *mom_bias.xlsx*.

```
optim.SGD()
```

```
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.zeros(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))
        self.steps = 0

    def update_stats(self, x):
        self.steps += 1
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.detach_()
        self.vars.detach_()
        mom = self.mom
        self.means.lerp_(m, mom)
        self.vars.lerp_ (v, mom)

    def forward(self, x):
        if self.training: self.update_stats(x)
        means = self.means
        vars  = self.vars
        if self.steps<100:
            corr = 1-(1-self.mom)**self.steps
            means = means/corr
            vars  = vars /corr
        x = (x-means).div_((vars+self.eps).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```
learn,run = get_learn_run(nfs, data, 1.0)
```

```
%time run.fit(3, learn)
```

    train: [0.409007578125, tensor(0.8792, device='cuda:0')]
    valid: [0.16397078857421876, tensor(0.9487, device='cuda:0')]
    train: [0.140029892578125, tensor(0.9574, device='cuda:0')]
    valid: [0.123492578125, tensor(0.9648, device='cuda:0')]
    train: [0.09799765625, tensor(0.9704, device='cuda:0')]
    valid: [0.08464044189453125, tensor(0.9749, device='cuda:0')]
    CPU times: user 3.93 s, sys: 355 ms, total: 4.28 s
    Wall time: 4.28 s

### jit

```
#export
from torch.jit import ScriptModule, script_method, script
from typing import *
```

```
class BatchNorm(ScriptModule):
    __constants__ = ['mom', 'eps']
    def __init__(self, nf, mom=0.1, eps=1e-3):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.zeros(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))
        self.register_buffer('steps', tensor(0))

    @script_method
    def update_stats(self, x):
        self.steps += 1
        self.means.detach_()
        self.vars.detach_()
        if bool(self.steps>100) and bool(self.steps % 2): return
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        if bool(self.steps==1):
            self.means.copy_(m)
            self.vars .copy_(v)
        else:
            self.means.copy_(self.means*(1-self.mom) + m*self.mom)
            self.vars .copy_(self.vars *(1-self.mom) + v*self.mom)

    @script_method
    def forward(self, x):
        if self.training: self.update_stats(x)
        x = (x-self.means).div((self.vars+self.eps).sqrt())
        return x.mul(self.mults).add(self.adds)
```

```
learn,run = get_learn_run(nfs, 0.8)
```

```
%time run.fit(3, learn)
```

    train: [0.4505270703125, tensor(0.8677, device='cuda:0')]
    valid: [0.18385860595703124, tensor(0.9452, device='cuda:0')]
    train: [0.1617141015625, tensor(0.9501, device='cuda:0')]
    valid: [0.12416044921875, tensor(0.9642, device='cuda:0')]
    train: [0.114721123046875, tensor(0.9653, device='cuda:0')]
    valid: [0.09863848876953125, tensor(0.9700, device='cuda:0')]
    CPU times: user 3.87 s, sys: 171 ms, total: 4.04 s
    Wall time: 4.04 s

### jit

```
#export
from torch.jit import ScriptModule, script_method, script
from typing import *
```

```
class BatchNorm2(nn.Module):
    __constants__ = ['mom', 'eps']
    def __init__(self, nf, mom=0.01, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums',  torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs',  torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step',  tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        self.mom1 = 1 - (1-self.mom)**x.shape[0]
        s  = x.sum(dims, keepdim=True)
        ss = x.pow(2).sum(dims, keepdim=True)
        c  = x.numel()/nc
        self.sums.copy_ (self.sums .add(self.mom1, s -self.sums))
        self.sqrs.copy_ (self.sqrs .add(self.mom1, ss-self.sqrs))
        self.count.copy_(self.count.add(self.mom1, c-self.count))
        self.dbias.copy_(self.mom1 + self.dbias*(1-self.mom1))
        self.batch += bs
        self.step  += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        if self.step<100:
            sums = sums/self.dbias
            sqrs = sqrs/self.dbias
        means = self.sums/self.count
        vars  = self.sqrs/self.count - means.pow(2)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div((vars+self.eps).sqrt())
        return x*self.mults + self.adds
```

```
class BatchNorm2(ScriptModule):
    __constants__ = ['mom', 'eps']
    def __init__(self, nf, mom=0.1, eps=1e-3):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (1,nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(1,nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('coun', torch.zeros(1,nf,1,1))
        self.register_buffer('steps', tensor(0))

    @script_method
    def update_stats(self, x):
        nc = x.shape[1]
        self.steps += 1
        self.sqrs.detach_()
        self.sums.detach_()
        # if bool(self.steps>100) and bool(self.steps % 2): return
        dims = (0,2,3)
        s  = x    .sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c  = x.numel()/nc
        if bool(self.steps==1):
            self.sums.copy_(s)
            self.sqrs.copy_(ss)
            self.coun.copy_(c)
        else:
            self.sums.copy_(self.sums*(1-self.mom) + s *self.mom)
            self.sqrs.copy_(self.sqrs*(1-self.mom) + ss*self.mom)
            self.coun.copy_(self.coun*(1-self.mom) + c *self.mom)

    @script_method
    def forward(self, x):
        if self.training: self.update_stats(x)
        means = self.sums/self.coun
        vars  = self.sqrs/self.coun - means.pow(2)
        x = (x-means) / ((vars+self.eps).sqrt())
        return x*self.mults + self.adds
```

```
def conv2d(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm2(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.8)
```

```
%time run.fit(3, learn)
```

    train: [0.079270703125, tensor(0.9763, device='cuda:0')]
    valid: [0.08132650146484376, tensor(0.9758, device='cuda:0')]
    train: [0.0634197607421875, tensor(0.9816, device='cuda:0')]
    valid: [0.07836026000976562, tensor(0.9761, device='cuda:0')]
    train: [0.0550289306640625, tensor(0.9839, device='cuda:0')]
    valid: [0.09280201416015625, tensor(0.9725, device='cuda:0')]
    CPU times: user 4.25 s, sys: 516 ms, total: 4.77 s
    Wall time: 4.77 s

### Builtin batchnorm

```
#export
def conv2d(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-3))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 1.0)
```

```
%time run.fit(3, learn)
```

    train: [0.4437156640625, tensor(0.8695, device='cuda:0')]
    valid: [0.1359041748046875, tensor(0.9602, device='cuda:0')]
    train: [0.11120369140625, tensor(0.9661, device='cuda:0')]
    valid: [0.08841785888671876, tensor(0.9738, device='cuda:0')]
    train: [0.079300078125, tensor(0.9763, device='cuda:0')]
    valid: [0.107217041015625, tensor(0.9689, device='cuda:0')]
    CPU times: user 3.09 s, sys: 348 ms, total: 3.44 s
    Wall time: 3.45 s

### With scheduler

```
sched = combine_scheds([0.5, 0.5], [sched_lin(0.2, 1.), sched_lin(1., 0.1)])
```

```
learn,run = get_learn_run(nfs, data, 1., cbs=ParamScheduler('lr', sched))
```

```
run.fit(8, learn)
```

    train: [0.4314221484375, tensor(0.8787, device='cuda:0')]
    valid: [0.1140492431640625, tensor(0.9683, device='cuda:0')]
    train: [0.102120380859375, tensor(0.9695, device='cuda:0')]
    valid: [0.08423486938476563, tensor(0.9760, device='cuda:0')]
    train: [0.072295810546875, tensor(0.9787, device='cuda:0')]
    valid: [0.07796448974609375, tensor(0.9761, device='cuda:0')]
    train: [0.0582051318359375, tensor(0.9824, device='cuda:0')]
    valid: [0.0651952880859375, tensor(0.9801, device='cuda:0')]
    train: [0.0478931201171875, tensor(0.9857, device='cuda:0')]
    valid: [0.061673541259765624, tensor(0.9822, device='cuda:0')]
    train: [0.0414492431640625, tensor(0.9877, device='cuda:0')]
    valid: [0.05984327392578125, tensor(0.9823, device='cuda:0')]
    train: [0.0365991015625, tensor(0.9887, device='cuda:0')]
    valid: [0.0609909912109375, tensor(0.9839, device='cuda:0')]
    train: [0.0328167578125, tensor(0.9898, device='cuda:0')]
    valid: [0.0601783447265625, tensor(0.9836, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

```
class LayerNorm(ScriptModule):
    __constants__ = ['eps']
    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))

    @script_method
    def forward(self, x):
        m = x.mean((1,2,3), keepdim=True)
        v = x.std ((1,2,3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

```
def conv2d(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.5)
```

```
%time run.fit(3, learn)
```

    train: [0.4526175, tensor(0.8583, device='cuda:0')]
    valid: [0.1315213623046875, tensor(0.9623, device='cuda:0')]
    train: [0.1259929296875, tensor(0.9618, device='cuda:0')]
    valid: [0.108642041015625, tensor(0.9664, device='cuda:0')]
    train: [0.088629716796875, tensor(0.9735, device='cuda:0')]
    valid: [0.08789295043945312, tensor(0.9742, device='cuda:0')]
    CPU times: user 4.57 s, sys: 484 ms, total: 5.05 s
    Wall time: 5.06 s

*Thought experiment*: can this distinguish green frogs from red frogs (assuming you're using it before the first conv)?

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

```
class InstanceNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))

    def forward(self, x):
        m = x.mean((2,3), keepdim=True)
        v = x.std ((2,3), keepdim=True)
        res = (x-m) / ((v+self.eps).sqrt())
        return res*self.mults + self.adds
```

```
def conv2d(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.1)
```

```
%time run.fit(3, learn)
```

    train: [2.30253484375, tensor(0.1128, device='cuda:0')]
    valid: [2.3023453125, tensor(0.1064, device='cuda:0')]
    train: [2.301203125, tensor(0.1136, device='cuda:0')]
    valid: [2.30204765625, tensor(0.1064, device='cuda:0')]
    train: [2.30111546875, tensor(0.1136, device='cuda:0')]
    valid: [2.302075390625, tensor(0.1064, device='cuda:0')]
    CPU times: user 3.73 s, sys: 620 ms, total: 4.35 s
    Wall time: 4.36 s

*Question*: why can't this classify anything?

### Group norm

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: :math:`(N, num\_channels, *)`
- Output: :math:`(N, num\_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### Exponentially Weighted Layer Norm

```
class EWLayerNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('sum'  , tensor(0.))
        self.register_buffer('sqr'  , tensor(0.))
        self.register_buffer('count', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sum.detach_()
        self.sqr.detach_()
        mom = 1 if self.batch==0 else self.mom
        self.sum  .add_(mom, x.sum()     - self.sum  )
        self.sqr  .add_(mom, (x*x).sum() - self.sqr  )
        self.count.add_(mom, x.numel()   - self.count)
        self.batch += bs

    def forward(self, x):
        if self.training: self.update_stats(x)
        mean = self.sum/self.count
        var  = self.sqr/self.count - mean.pow(2)
        x = (x-mean)/((var+self.eps).sqrt())
        return x*self.mult + self.add
```

```
def conv2d(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(EWLayerNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, 0.5)
```

```
%time run.fit(3, learn)
```

    train: [0.074243193359375, tensor(0.9766, device='cuda:0')]
    valid: [0.08686154174804687, tensor(0.9731, device='cuda:0')]
    train: [0.059469072265625, tensor(0.9819, device='cuda:0')]
    valid: [0.096335546875, tensor(0.9706, device='cuda:0')]
    train: [0.053373134765625, tensor(0.9833, device='cuda:0')]
    valid: [0.07646878051757812, tensor(0.9772, device='cuda:0')]
    CPU times: user 3.68 s, sys: 280 ms, total: 3.96 s
    Wall time: 3.97 s

### Exponentially Weighted Batch Norm

```
class EWBatchNorm(nn.Module):
    __constants__ = ['mom', 'eps']
    def __init__(self, nf, mom=0.01, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums',  torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs',  torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step',  tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        self.mom1 = 1 - (1-self.mom)**x.shape[0]
        s  = x.sum(dims, keepdim=True)
        ss = x.pow(2).sum(dims, keepdim=True)
        c  = x.numel()/nc
        self.sums.copy_ (self.sums .add(self.mom1, s -self.sums))
        self.sqrs.copy_ (self.sqrs .add(self.mom1, ss-self.sqrs))
        self.count.copy_(self.count.add(self.mom1, c-self.count))
        self.dbias.copy_(self.mom1 + self.dbias*(1-self.mom1))
        self.batch += bs
        self.step  += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        if self.step<100:
            sums = sums/self.dbias
            sqrs = sqrs/self.dbias
        means = self.sums/self.count
        vars  = self.sqrs/self.count - means.pow(2)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div((vars+self.eps).sqrt())
        return x*self.mults + self.adds
```

```
def conv2d(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(EWBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 1.0)
```

```
%time run.fit(3, learn)
```

    train: [0.33104734375, tensor(0.9050, device='cuda:0')]
    valid: [0.111326513671875, tensor(0.9681, device='cuda:0')]
    train: [0.088771162109375, tensor(0.9726, device='cuda:0')]
    valid: [0.07529160766601563, tensor(0.9775, device='cuda:0')]
    train: [0.059317353515625, tensor(0.9819, device='cuda:0')]
    valid: [0.07531204833984376, tensor(0.9759, device='cuda:0')]
    CPU times: user 4.93 s, sys: 536 ms, total: 5.46 s
    Wall time: 5.47 s

```
sched = combine_scheds([0.5, 0.5], [sched_cos(0.3, 1.2), sched_cos(1., 0.1)])
```

```
learn,run = get_learn_run(nfs, data, 1., cbs=ParamScheduler('lr', sched))
```

```
run.fit(8, learn)
```

    train: [0.34060890625, tensor(0.9043, device='cuda:0')]
    valid: [0.29444873046875, tensor(0.9061, device='cuda:0')]
    train: [0.091703251953125, tensor(0.9726, device='cuda:0')]
    valid: [0.09838342895507812, tensor(0.9697, device='cuda:0')]
    train: [0.0651663623046875, tensor(0.9801, device='cuda:0')]
    valid: [0.06815178833007812, tensor(0.9798, device='cuda:0')]
    train: [0.051250078125, tensor(0.9844, device='cuda:0')]
    valid: [0.065077880859375, tensor(0.9811, device='cuda:0')]
    train: [0.0417406640625, tensor(0.9877, device='cuda:0')]
    valid: [0.07483850708007812, tensor(0.9760, device='cuda:0')]
    train: [0.03471643798828125, tensor(0.9903, device='cuda:0')]
    valid: [0.06753575439453124, tensor(0.9795, device='cuda:0')]
    train: [0.02933214111328125, tensor(0.9912, device='cuda:0')]
    valid: [0.05319308471679687, tensor(0.9832, device='cuda:0')]
    train: [0.024283916015625, tensor(0.9928, device='cuda:0')]
    valid: [0.053880108642578126, tensor(0.9845, device='cuda:0')]

## Export

```
!./notebook2script.py 07_batchnorm.ipynb
```

    Converted 07_batchnorm.ipynb to nb_07.py

```

```