# Assignment 5

This assignment has weighting $1.5$.

# Model tuning and evaluation

```python
# Added version check for recent scikit-learn 0.18 checks
from distutils.version import LooseVersion as Version
from sklearn import __version__ as sklearn_version
```

## Dataset

We will use the Wisconsin breast cancer dataset for the following questions

```python
import pandas as pd

wdbc_source = 'https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data'
#wdbc_source = '../datasets/wdbc/wdbc.data'

df = pd.read_csv(wdbc_source, header=None)
```

```python
from sklearn.preprocessing import LabelEncoder
X = df.loc[:, 2:].values
y = df.loc[:, 1].values
le = LabelEncoder()
y = le.fit_transform(y)
le.transform(['M', 'B'])
```

    array([1, 0])

```python
if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.20, random_state=1)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

# K-fold validation (20 points)

Someone wrote the code below to conduct cross validation.
Do you see anything wrong with it?
And if so, correct the code and provide an explanation.

```python
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import Perceptron
from sklearn.pipeline import Pipeline

if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import StratifiedKFold
else:
    from sklearn.model_selection import StratifiedKFold

scl = StandardScaler()
pca = PCA(n_components=2)
# original: clf = Perceptron(random_state=1)

# data preprocessing
X_train_std = scl.fit_transform(X_train)
X_test_std = scl.transform(X_test)

X_train_pca = pca.fit_transform(X_train_std)
X_test_pca = pca.transform(X_test_std)

# compute the data indices for each fold
if Version(sklearn_version) < '0.18':
    kfold = StratifiedKFold(y=y_train,
                            n_folds=10,
                            random_state=1)
else:
    kfold = StratifiedKFold(n_splits=10,
                            random_state=1).split(X_train, y_train)

num_epochs = 2
scores = [[] for i in range(num_epochs)]

enumerate_kfold = list(enumerate(kfold))
# new:
clfs = [Perceptron(random_state=1) for i in range(len(enumerate_kfold))]

for epoch in range(num_epochs):
    for k, (train, test) in enumerate_kfold:

# original:
#         clf.partial_fit(X_train_std[train], y_train[train], classes=np.unique(y_train))
#         score = clf.score(X_train_std[test], y_train[test])
#         scores.append(score)
# new:
        clfs[k].partial_fit(X_train_pca[train], y_train[train], classes=np.unique(y_train))
        score = clfs[k].score(X_train_pca[test], y_train[test])
        scores[epoch].append(score)

        print('Epoch: %s, Fold: %s, Class dist.: %s, Acc: %.3f' % (epoch,
                                                                   k,
                                                                   np.bincount(y_train[train]),
                                                                   score))
    print('')

# new:
for epoch in range(num_epochs):
    print('Epoch: %s, CV accuracy: %.3f +/- %.3f' % (epoch, np.mean(scores[epoch]), np.std(scores[epoch])))
```

    Epoch: 0, Fold: 0, Class dist.: [256 153], Acc: 0.935
    Epoch: 0, Fold: 1, Class dist.: [256 153], Acc: 0.957
    Epoch: 0, Fold: 2, Class dist.: [256 153], Acc: 0.957
    Epoch: 0, Fold: 3, Class dist.: [256 153], Acc: 0.913
    Epoch: 0, Fold: 4, Class dist.: [256 153], Acc: 0.935
    Epoch: 0, Fold: 5, Class dist.: [257 153], Acc: 0.956
    Epoch: 0, Fold: 6, Class dist.: [257 153], Acc: 0.978
    Epoch: 0, Fold: 7, Class dist.: [257 153], Acc: 0.956
    Epoch: 0, Fold: 8, Class dist.: [257 153], Acc: 0.978
    Epoch: 0, Fold: 9, Class dist.: [257 153], Acc: 0.978

    Epoch: 1, Fold: 0, Class dist.: [256 153], Acc: 0.913
    Epoch: 1, Fold: 1, Class dist.: [256 153], Acc: 0.957
    Epoch: 1, Fold: 2, Class dist.: [256 153], Acc: 0.957
    Epoch: 1, Fold: 3, Class dist.: [256 153], Acc: 0.913
    Epoch: 1, Fold: 4, Class dist.: [256 153], Acc: 0.935
    Epoch: 1, Fold: 5, Class dist.: [257 153], Acc: 0.978
    Epoch: 1, Fold: 6, Class dist.: [257 153], Acc: 0.978
    Epoch: 1, Fold: 7, Class dist.: [257 153], Acc: 0.956
    Epoch: 1, Fold: 8, Class dist.: [257 153], Acc: 1.000
    Epoch: 1, Fold: 9, Class dist.: [257 153], Acc: 0.911

    Epoch: 0, CV accuracy: 0.954 +/- 0.020
    Epoch: 1, CV accuracy: 0.950 +/- 0.029

## Answer
#### Problems with the original code:
1. Use partial fit for all folds, which results in dependent training and thus cumulative learning.

2. Due to problem 1, we the CV accuracy is incorrect, and not able to show score for every epoch.

#### There are two major changes:
1. We create a list of Perceptrons, each corresponds to one fold. Within each fold, we run partial fit for a given number of epochs.

2. Also, we create a list of accuracy scores, each corresponds to one epoch. Within each epoch, we append scores of all folds.

#### There is one minor change:
1. To enable PCA, we change X_train_std into X_train_pca.

# Precision-recall curve (40 points)

We have plotted ROC (receiver operator characteristics) curve for the breast cancer dataset.

Plot the precision-recall curve for the same data set using the same experimental setup.
What similarities and differences you can find between ROC and precision-recall curves?

You can find more information about precision-recall curve online such as: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html

# Answer

```python
from sklearn.metrics import roc_curve, precision_recall_curve, auc
from scipy import interp
from sklearn.linear_model import LogisticRegression

pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('pca', PCA(n_components=2)),
                    ('clf', LogisticRegression(penalty='l2',
                                               random_state=0,
                                               C=100.0))])

# intentionally use only 2 features to make the task harder and the curves more interesting
X_train2 = X_train[:, [4, 14]]
X_test2 = X_test[:, [4, 14]]

if Version(sklearn_version) < '0.18':
    cv = StratifiedKFold(y_train, n_folds=3, random_state=1)
else:
    cv = list(StratifiedKFold(n_splits=3, random_state=1).split(X_train, y_train))

fig = plt.figure(figsize=(7, 5))

# **************************************
#            ROC - 2 Features
# **************************************
print ("ROC Curve - 2 Features")
mean_tpr = 0.0
mean_fpr = np.linspace(0, 1, 100)
all_tpr = []

for i, (train, test) in enumerate(cv):
    probas = pipe_lr.fit(X_train2[train],
                         y_train[train]).predict_proba(X_train2[test])

    fpr, tpr, thresholds = roc_curve(y_train[test],
                                     probas[:, 1],
                                     pos_label=1)
    mean_tpr += interp(mean_fpr, fpr, tpr)
    #mean_tpr[0] = 0.0
    roc_auc = auc(fpr, tpr)
    plt.plot(fpr,
             tpr,
             lw=1,
             label='ROC fold %d (area = %0.2f)'
                   % (i+1, roc_auc))

mean_tpr /= len(cv)
mean_tpr[0] = 0.0
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)
plt.plot(mean_fpr, mean_tpr, 'k--',
         label='mean ROC (area = %0.2f)' % mean_auc, lw=2)

plt.plot([0, 1],
         [0, 1],
         linestyle='--',
         color=(0.6, 0.6, 0.6),
         label='random guessing')

plt.plot([0, 0, 1],
         [0, 1, 1],
         lw=2,
         linestyle=':',
         color='black',
         label='perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.legend(loc='lower left', bbox_to_anchor=(1, 0))

plt.tight_layout()
# plt.savefig('./figures/roc.png', dpi=300)
plt.show()

# **************************************
#            ROC - All Features
# **************************************
print ("ROC Curve - All Features")
mean_tpr = 0.0
mean_fpr = np.linspace(0, 1, 100)
all_tpr = []

for i, (train, test) in enumerate(cv):
    probas = pipe_lr.fit(X_train[train],
                         y_train[train]).predict_proba(X_train[test])

    fpr, tpr, thresholds = roc_curve(y_train[test],
                                     probas[:, 1],
                                     pos_label=1)
    mean_tpr += interp(mean_fpr, fpr, tpr)
    #mean_tpr[0] = 0.0
    roc_auc = auc(fpr, tpr)
    plt.plot(fpr,
             tpr,
             lw=1,
             label='ROC fold %d (area = %0.2f)'
                   % (i+1, roc_auc))

mean_tpr /= len(cv)
mean_tpr[0] = 0.0
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)
plt.plot(mean_fpr, mean_tpr, 'k--',
         label='mean ROC (area = %0.2f)' % mean_auc, lw=2)

plt.plot([0, 1],
         [0, 1],
         linestyle='--',
         color=(0.6, 0.6, 0.6),
         label='random guessing')

plt.plot([0, 0, 1],
         [0, 1, 1],
         lw=2,
         linestyle=':',
         color='black',
         label='perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.legend(loc='lower left', bbox_to_anchor=(1, 0))

plt.tight_layout()
# plt.savefig('./figures/roc.png', dpi=300)
plt.show()

# **************************************
#      Precision-Recall - 2 Features
# **************************************
print ("Precision Recall Curve - 2 Features")
mean_pre = 0.0
mean_rec = np.linspace(0, 1, 100)
all_pre = []

for i, (train, test) in enumerate(cv):
    probas = pipe_lr.fit(X_train2[train],
                         y_train[train]).predict_proba(X_train2[test])

    ## note that the return order is precision, recall
    pre, rec, thresholds = precision_recall_curve(y_train[test],
                                     probas[:, 1],
                                     pos_label=1)

    ## flip the recall and precison array
    mean_pre += interp(mean_rec, np.flipud(rec), np.flipud(pre))

    pr_auc = auc(rec, pre)
    plt.plot(rec,
             pre,
             lw=1,
             label='PR fold %d (area = %0.2f)'
                   % (i+1, pr_auc))

mean_pre /= len(cv)
mean_auc = auc(mean_rec, mean_pre)
plt.plot(mean_rec, mean_pre, 'k--',
         label='mean PR (area = %0.2f)' % mean_auc, lw=2)

# random classifier: a line of Positive/(Positive+Negative)
# here for simplicity, we set it to be Positive/(Positive+Negative) of fold 3
plt.plot(mean_rec,
         [pre[0] for i in range(len(mean_rec))],
         linestyle='--', label='random guessing of fold 3', color=(0.6, 0.6, 0.6))

# perfect performance: y = 1 for x in [0, 1]; when x = 1; down to the random guessing line
plt.plot([0, 1, 1],
         [1, 1, pre[0]],
         lw=2,
         linestyle=':',
         color='black',
         label='perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('recall')
plt.ylabel('precision')
plt.title('Precision-Recall')
plt.legend(loc='lower left', bbox_to_anchor=(1, 0))

plt.tight_layout()
# plt.savefig('./figures/roc.png', dpi=300)
plt.show()

# **************************************
#    Precision-Recall - All Features
# **************************************
print ("Precision Recall Curve - All Features")
mean_pre = 0.0
mean_rec = np.linspace(0, 1, 100)
all_pre = []

for i, (train, test) in enumerate(cv):
    probas = pipe_lr.fit(X_train[train],
                         y_train[train]).predict_proba(X_train[test])

    ## note that the return order is precision, recall
    pre, rec, thresholds = precision_recall_curve(y_train[test],
                                     probas[:, 1],
                                     pos_label=1)

    ## flip the recall and precison array
    mean_pre += interp(mean_rec, np.flipud(rec), np.flipud(pre))

    pr_auc = auc(rec, pre)
    plt.plot(rec,
             pre,
             lw=1,
             label='PR fold %d (area = %0.2f)'
                   % (i+1, pr_auc))

mean_pre /= len(cv)
mean_auc = auc(mean_rec, mean_pre)
plt.plot(mean_rec, mean_pre, 'k--',
         label='mean PR (area = %0.2f)' % mean_auc, lw=2)

# random classifier: a line of Positive/(Positive+Negative)
# here for simplicity, we set it to be Positive/(Positive+Negative) of fold 3
plt.plot(mean_rec,
         [pre[0] for i in range(len(mean_rec))],
         linestyle='--', label='random guessing of fold 3', color=(0.6, 0.6, 0.6))

# perfect performance: y = 1 for x in [0, 1]; when x = 1; down to the random guessing line
plt.plot([0, 1, 1],
         [1, 1, pre[0]],
         lw=2,
         linestyle=':',
         color='black',
         label='perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('recall')
plt.ylabel('precision')
plt.title('Precision-Recall')
plt.legend(loc='lower left', bbox_to_anchor=(1, 0))

plt.tight_layout()
# plt.savefig('./figures/roc.png', dpi=300)
plt.show()
```

## Explanation

### Definition
REC(recall) = TPR(true positive rate) = $\frac{TP}{TP+FN}$

PRE(precision) = $\frac{TP}{TP+FP}$

FPR(false positive rate) = $\frac{FP}{FP+TN}$

TP: Actual Class $A$, test result $A$

FP: Actual Class $B$, test result $A$

TN: Actual Class $B$, test result $B$

FN: Actual Class $A$, test result $B$

### Similarities
1. When number of features increases, in ROC curve, true positive rate is close to 1 regardless of false positive rate (not equals 0); in Precison-Recall curve, precision is close to 1 regardless of recall (not equals 1).

2. The training results are better than random guessing in this example.

### Differences

#### For monotonicity
* ROC curve: As false positive rate increase, the true positive rate generally increases, and the effect is huger when number of features is small.

This is because when avoid classifying class $B$ as class $A$, we actually increase the probability of successfully classifying class $A$ as class $A$, which increases the true positive rate.

Actually, the curve is almost concave. The image below illustrates the reason.

Please keep online to view the picture. Reference: https://upload.wikimedia.org/wikipedia/commons/5/5c/ROCfig.PNG

* Precision-Recall curve: As recall increases, the precision generally decreases, especially when number of features is small.

This is because when we try to increase recall (reduce false negative), we avoid classifying samples with acutal class $A$ as class $B$, that is, we classify samples similar to class $A$ as class $A$. However, whis will also increase the probability of classifying samples with acutal class $B$ (but tested to be similar to class $A$) as class $A$, which is FP.

Notice that when FP increases, it is _not necessary_ that precision decrease since TP also increases. Only when the ratio $\frac{TP}{TP+FP}$ drops, which is often the general case when number of features is small, precision drops. The image below illustrates the reason.

Please keep online to view the picture. Reference: http://numerical.recipes/CS395T/lectures2008/17-ROCPrecisionRecall.pdf

#### For random guessing
* ROC curve: true positive rate = false positive rate. This is because the propotion of TP in (TP+FN) equals the propotion of FP in (FP+TN), because declaring a result has no dependence on the actuall class label for random guessing.
* Precision-Recall curve: the random guessing is a horizontal line with y = P/(P+N) for given boundary of P and N, which varies with cases. (In this example, we choose P/(P+N) of fold 3 for simplicity).

#### For perfect performance
* ROC curve: The perfect performance is (0, 0) -> (0, 1) -> (1, 1). We aim to have result in the "top-left" corner to obtain low false positive rate and high true positive rate.
* Precision-Recall curve: The perfect performance is (0, 1) -> (1, 1) -> (1, P/(P+N)). We aim to have result in the "top-right" corner to obtain high recall and high precision.

# Ensemble learning

We have used the following code to compute and plot the ensemble error from individual classifiers for binary classification:

```python
from scipy.misc import comb
import math
import numpy as np

def ensemble_error(num_classifier, base_error):
    k_start = math.ceil(num_classifier/2)
    probs = [comb(num_classifier, k)*(base_error**k)*((1-base_error)**(num_classifier-k)) for k in range(k_start, num_classifier+1)]
    return sum(probs)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def plot_base_error(ensemble_error_func, num_classifier, error_delta):

    error_range = np.arange(0.0, 1+error_delta, error_delta)
    ensemble_errors = [ensemble_error_func(num_classifier=num_classifier, base_error=error) for error in error_range]

    plt.plot(error_range, ensemble_errors,
             label = 'ensemble error',
             linewidth=2)
    plt.plot(error_range, error_range,
             label = 'base error',
             linestyle = '--',
             linewidth=2)
    plt.xlabel('base error')
    plt.ylabel('base/ensemble error')
    plt.legend(loc='best')
    plt.grid()
    plt.show()
```

```python
num_classifier = 11
error_delta = 0.01
base_error = 0.25
```

```python
print(ensemble_error(num_classifier=num_classifier, base_error=base_error))
```

    0.034327507019

```python
plot_base_error(ensemble_error, num_classifier=num_classifier, error_delta=error_delta)
```

# Number of classifiers (40 points)

The function plot_base_error() above plots the ensemble error as a function of the base error given a fixed number of classifiers.

Write another function to plot ensembe error versus different number of classifiers with a given base error.

Does the ensemble error always go down with more classifiers?
Why or why not?

Can you improve the method ensemble_error() to produce a more reasonable plot?

## Answer

The code for plotting is below:

```python
def plot_num_classifier(ensemble_error_func, max_num_classifier, base_error):

    num_classifiers = range(1, max_num_classifier+1)
    ensemble_errors = [ensemble_error_func(num_classifier = num_classifier, base_error=base_error) for num_classifier in num_classifiers]

    plt.plot(num_classifiers, ensemble_errors,
             label = 'ensemble error',
             linewidth = 2)
    plt.plot(range(max_num_classifier), [base_error]*max_num_classifier,
             label = 'base error',
             linestyle = '--',
             linewidth=2)

    plt.xlabel('num classifiers')
    plt.ylabel('ensemble error')
    plt.xlim([1, max_num_classifier])
    plt.ylim([0, 1])
    plt.title('base error %.2f' % base_error)
    plt.legend(loc='best')
    plt.grid()
    plt.show()
```

```python
max_num_classifiers = 20
base_error = 0.25
```

```python
plot_num_classifier(ensemble_error,
                    max_num_classifier=max_num_classifiers,
                    base_error=base_error)
```

### Explanation
#### Observations:
The ensemble error __DOES NOT ALWAYS__ go down with more classifiers.

Overall, the ensemble error declines as the number of classifiers increases. However, when the number of calssifiers $N = 2k$, the error is much higher than $N = 2k-1$ and $2k+1$.
#### Reason:
This is because when a draw comes, i.e. $K$ subclassifiers are wrong and $K$ subclassifiers are correct, it is considered as "wrong prediction" by the original function.

Describe a better algorithm for computing the ensemble error.

```python
def better_ensemble_error(num_classifier, base_error):
    k_start = math.ceil(num_classifier/2)
    probs = [comb(num_classifier, k)*(base_error**k)*((1-base_error)**(num_classifier-k)) for k in range(k_start, num_classifier+1)]
    if num_classifier % 2 == 0:
        probs.append(-0.5*comb(num_classifier, k_start)*(base_error**k_start)*((1-base_error)**(num_classifier-k_start)))
    return sum(probs)
```

```python
plot_num_classifier(better_ensemble_error,
                    max_num_classifier=max_num_classifiers,
                    base_error=base_error)
```

### Discription
When there is a draw, i.e. $K$ subclassifiers are wrong and $K$ subclassifiers are correct, we take half of the probability of this condition to be wrong. That is, $\frac{1}{2} C\left(2K, K\right) \epsilon^K \left(1-\epsilon\right)^{K}$.

Notice that the result of $2K$ classifiers is the same as $2K-1$ classifier, Becase

$$
\begin{align}
P(2K\text{ failed}) = & \text{ }P(2K\text{ failed | }K\text{ in }2K-1\text{ failed})P(K\text{ in }2K-1\text{ failed}) \\
& + P(2K\text{ failed | }K-1\text{ in }2K-1\text{ failed})P(K-1\text{ in }2K-1\text{ failed}) \\
& + P(2K\text{ failed | less than }K-1\text{ in }2K-1\text{ failed})P(\text{less than }K-1\text{ in }2K-1\text{ failed}) \\
& + P(2K\text{ failed | more than }K\text{ in }2K-1\text{ failed})P(\text{more than }K\text{ in }2K-1\text{ failed}) \\
= & \text{ }(\epsilon + \frac{1-\epsilon}{2})C\left(2K-1, K\right)\epsilon^K \left(1-\epsilon\right)^{K-1}
+ (\frac{\epsilon}{2})C\left(2K-1, K-1\right)\epsilon^{K-1} \left(1-\epsilon\right)^{K} \\
& + 0 \times \sum_{i=0}^{K-2}C\left(2K-1, i\right)\epsilon^i \left(1-\epsilon\right)^{2K-1-i}
+ 1 \times \sum_{i=K+1}^{2K-1}C\left(2K-1, i\right)\epsilon^i \left(1-\epsilon\right)^{2K-1-i} \\
= & \text{ }C\left(2K-1, K\right)\epsilon^{K+1} \left(1-\epsilon\right)^{K-1} + C\left(2K-1, K\right)\epsilon^K \left(1-\epsilon\right)^{K} + \sum_{i=K+1}^{2K-1}C\left(2K-1, i\right)\epsilon^i \left(1-\epsilon\right)^{2K-1-i} \\
= & \text{ }C\left(2K-1, K\right) (\epsilon + 1 - \epsilon) \epsilon^{K} \left(1-\epsilon\right)^{K-1} + \sum_{i=K+1}^{2K-1}C\left(2K-1, i\right)\epsilon^i \left(1-\epsilon\right)^{2K-1-i} \\
= & \text{ }\sum_{i=K}^{2K-1}C\left(2K-1, i\right)\epsilon^i \left(1-\epsilon\right)^{2K-1-i} \\
= & \text{ }P(2K-1\text{ failed})
\end{align}
$$

Hence $P(2K$ failed) = $P(2K-1$ failed).


# Optimizer tweaks

```python
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```python
#export
from exp.nb_08 import *
```

## Imagenette data

Most libraries treat different optimizers as different entities. Every time a new optimizer comes out, the framework has to add a new set of parameters to accommodate the new method. J.H. says that there is *one optimizer* into which you can inject different bits of behavior in a small number of ways. We will start with a generic optimizer and we will end up with something similar to what is described in [this paper](https://arxiv.org/abs/1904.00962).

We grab the data from the previous notebook.

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=3917)

```python
path = datasets.untar_data(datasets.URLs.IMAGENETTE_160)
```

```python
tfms = [make_rgb, ResizeFixed(128), to_byte_tensor, to_float_tensor]
bs = 128

il = ImageList.from_files(path, tfms=tfms)
sd = SplitData.split_by_func(il, partial(
    grandparent_splitter, valid_name='val'))
ll = label_by_func(sd, parent_labeler, proc_y=CategoryProcessor())
data = ll.to_databunch(bs, c_in=3, c_out=10, num_workers=4)
```

Then a model:

```python
nfs = [32, 64, 128, 256]
```

```python
cbfs = [partial(AvgStatsCallback, accuracy), CudaCallback,
        partial(BatchTransformXCallback, norm_imagenette)]
```

This is the baseline of training with vanilla SGD.

```python
learn, run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
run.fit(1, learn)
```

    train: [1.8552763045860174, tensor(0.3513, device='cuda:0')]
    valid: [1.7813377040207006, tensor(0.3702, device='cuda:0')]

## Refining the optimizer

In most frameworks, optimizers are Python dictionaries. In notebook `05_anneal`, in the `Recorder` class, the `after_batch()` method uses `self.opt.param_groups`, which we have never defined or declared. This works because J.H. cheated, and used the optimizer even though the rules forbid it. We didn't have parameter groups, and we borrowed them from PyTorch.

In PyTorch, the base optimizer in `torch.optim` is just a dictionary that stores the hyper-parameters and references to the parameters of the model we want to train in parameter groups (different groups can have different learning rates/momentum/weight decay... which is what lets us do discriminative learning rates).

It contains a method `step` that will update our parameters with the gradients and a method `zero_grad` to detach and zero the gradients of all our parameters.

We build the equivalent from scratch, only ours will be more flexible. In our implementation, the step function loops over all the parameters to execute the step using stepper functions that we have to provide when initializing the optimizer.

The `step()` function below doesn't do anything, but rather uses composition to run some functions stored in `self.steppers` that are passed during initialization.

Parameters are stored in lists. Parameter groups are list of lists, one per group. This is how we tell the optimizer that different parameter groups should be treated differently, for example with discriminative learning rates. The `self.param_groups` attribute below contains the parameter groups.

Each parameter group can contain its own list of hyper-parameters. Each parameter group will store its hyper-parameters in a dictionary. The list `self.hypers` contains a dictionary for each parameter group.

The new bit is provided by the `steppers`. We define a basic SGD `stepper` below.

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=4074)

```python
class Optimizer():
    def __init__(self, params, steppers, **defaults):
        # might be a generator
        self.param_groups = list(params)
        # ensure params is a list of lists
        if not isinstance(self.param_groups[0], list):
            self.param_groups = [self.param_groups]
        self.hypers = [{**defaults} for p in self.param_groups]
        self.steppers = listify(steppers)

    def grad_params(self):
        return [(p, hyper) for pg, hyper in zip(self.param_groups, self.hypers)
                for p in pg if p.grad is not None]

    def zero_grad(self):
        for p, hyper in self.grad_params():
            p.grad.detach_()
            p.grad.zero_()

    def step(self):
        for p, hyper in self.grad_params():
            compose(p, self.steppers, **hyper)
```

A `stepper` is a function, like the one defined below, and we want to pass our hyper-parameters to this function. We do it via `compose`. When we introduced `compose` we said that it can contain keyword arguments. The `step()` method above contains a good example of why this can be useful. `p` is a parameter that we are going to update. The main idea is to move from "an SGD optimizer" to "an optimizer with an SGD step".

**Note**: in PyTorch, `t1.add_(t2)` adds in place tensor `t2` to tensor `t1`, **however**, if we pass a first scalar argument, `t1.add_(s, t2)`, it first *multiplies* `t2` by `s` and *then* adds.
Therefore, a basic SGD, this what a step looks like:

```python
#export
def sgd_step(p, lr, **kwargs):
    p.data.add_(-lr, p.grad.data)
    return p
```

```python
opt_func = partial(Optimizer, steppers=[sgd_step])
```

Now that we have changed the optimizer, we will need to adjust the callbacks that were using properties from the PyTorch optimizer: in particular the hyper-parameters are in the list of dictionaries `opt.hypers` (PyTorch has everything in the the list of param groups).

```python
#export
class Recorder(Callback):
    def begin_fit(self): self.lrs, self.losses = [], []

    def after_batch(self):
        if not self.in_train:
            return
        self.lrs.append(self.opt.hypers[-1]['lr'])
        self.losses.append(self.loss.detach().cpu())

    def plot_lr(self): plt.plot(self.lrs)
    def plot_loss(self): plt.plot(self.losses)

    def plot(self, skip_last=0):
        losses = [o.item() for o in self.losses]
        n = len(losses)-skip_last
        plt.xscale('log')
        plt.plot(self.lrs[:n], losses[:n])

class ParamScheduler(Callback):
    _order = 1

    def __init__(self, pname, sched_funcs):
        self.pname, self.sched_funcs = pname, listify(sched_funcs)

    def begin_batch(self):
        if not self.in_train:
            return
        fs = self.sched_funcs
        if len(fs) == 1:
            fs = fs*len(self.opt.param_groups)
        pos = self.n_epochs/self.epochs
        for f, h in zip(fs, self.opt.hypers):
            h[self.pname] = f(pos)

class LR_Find(Callback):
    _order = 1

    def __init__(self, max_iter=100, min_lr=1e-6, max_lr=10):
        self.max_iter, self.min_lr, self.max_lr = max_iter, min_lr, max_lr
        self.best_loss = 1e9

    def begin_batch(self):
        if not self.in_train:
            return
        pos = self.n_iter/self.max_iter
        lr = self.min_lr * (self.max_lr/self.min_lr) ** pos
        for pg in self.opt.hypers:
            pg['lr'] = lr

    def after_step(self):
        if self.n_iter >= self.max_iter or self.loss > self.best_loss*10:
            raise CancelTrainException()
        if self.loss < self.best_loss:
            self.best_loss = self.loss
```

So let's check we didn't break anything and that recorder and param scheduler work properly.

```python
sched = combine_scheds([0.3, 0.7], [sched_cos(0.3, 0.6), sched_cos(0.6, 0.2)])
```

```python
cbfs = [partial(AvgStatsCallback, accuracy),
        CudaCallback, Recorder,
        partial(ParamScheduler, 'lr', sched)]
```

```python
learn, run = get_learn_run(nfs, data, 0.4, conv_layer,
                           cbs=cbfs, opt_func=opt_func)
```

```python
%time run.fit(1, learn)
```

    train: [1.8353323179586019, tensor(0.3554, device='cuda:0')]
    valid: [1.653249900477707, tensor(0.4257, device='cuda:0')]
    CPU times: user 2.19 s, sys: 788 ms, total: 2.97 s
    Wall time: 4.28 s

```python
run.recorder.plot_loss()
```

```python
run.recorder.plot_lr()
```

## Weight decay

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=4623)

By letting our model learn high parameters, it might fit all the data points in the training set with an over-complex function that has very sharp changes, which will lead to overfitting.

Weight decay comes from the idea of L2 regularization, which consists in adding to your loss function the sum of all the weights squared. Why do that? Because when we compute the gradients, it will add a contribution to them that will encourage the weights to be as small as possible.

Limiting our weights from growing too much is going to hinder the training of the model, but it will yield to a state where it generalizes better. Going back to the theory a little bit, weight decay (or just `wd`) is a parameter that controls that sum of squares we add to our loss:
``` python
loss_with_wd = loss + (wd/2) * (weights**2).sum()
```

In practice though, it would be very inefficient (and maybe numerically unstable) to compute that big sum and add it to the loss. If you remember a little bit of high school math, the derivative of `p**2` with respect to `p` is `2*p`. So adding that big sum to our loss is exactly the same as doing:
``` python
weight.grad += wd * weight
```

for every weight in our model, which in the case of vanilla SGD is equivalent to updating the parameters with:
``` python
weight = weight - lr*(weight.grad + wd*weight)
```

This technique is called "weight decay", as each weight is decayed by a factor `lr * wd`, as it's shown in this last formula.

This only works for standard SGD, as we have seen that with momentum, RMSProp and Adam, the update has some additional formulas around the gradient. In those cases, the formula that comes from L2 regularization:
``` python
weight.grad += wd * weight
```
is different than weight decay
``` python
new_weight = weight - lr * weight.grad - lr * wd * weight
```

Most libraries use the first one, but as it was pointed out in [Decoupled Weight Regularization](https://arxiv.org/pdf/1711.05101.pdf) by Ilya Loshchilov and Frank Hutter, it is better to use the second one with the Adam optimizer, which is why fastai made it its default.

Weight decay is subtracting `lr*wd*weight` from the weights. We need this function to have an attribute `_defaults` so that we are sure there is an hyper-parameter of the same name in our `Optimizer`.

```python
#export
def weight_decay(p, lr, wd, **kwargs):
    p.data.mul_(1 - lr*wd)
    return p

weight_decay._defaults = dict(wd=0.)
```

The logic behind `_defaults` is that we don't want to manually set weight decay to zero every time we want to turn it off.

L2 regularization is adding `wd*weight` to the gradients.

```python
#export
def l2_reg(p, lr, wd, **kwargs):
    p.grad.data.add_(wd, p.data)
    return p

l2_reg._defaults = dict(wd=0.)
```

Let's allow steppers to add to our `defaults` (which are the default values of all the hyper-parameters). This helper function adds in `dest` the key/values it finds while going through `os` and applying `f` when they was no `key` of the same name.

```python
#export
def maybe_update(os, dest, f):
    for o in os:
        for k, v in f(o).items():
            if k not in dest:
                dest[k] = v

def get_defaults(d): return getattr(d, '_defaults', {})
```

This is the same as before, we just take the default values of the steppers when none are provided in the kwargs.

```python
#export
class Optimizer():
    def __init__(self, params, steppers, **defaults):
        self.steppers = listify(steppers)
        maybe_update(self.steppers, defaults, get_defaults)
        # might be a generator
        self.param_groups = list(params)
        # ensure params is a list of lists
        if not isinstance(self.param_groups[0], list):
            self.param_groups = [self.param_groups]
        self.hypers = [{**defaults} for p in self.param_groups]

    def grad_params(self):
        return [(p, hyper) for pg, hyper in zip(self.param_groups, self.hypers)
                for p in pg if p.grad is not None]

    def zero_grad(self):
        for p, hyper in self.grad_params():
            p.grad.detach_()
            p.grad.zero_()

    def step(self):
        for p, hyper in self.grad_params():
            compose(p, self.steppers, **hyper)
```

Let's create an SGD optimizer containing an SGD and a weight decay steppers.

```python
#export
sgd_opt = partial(Optimizer, steppers=[weight_decay, sgd_step])
```

```python
learn, run = get_learn_run(nfs, data, 0.4, conv_layer,
                           cbs=cbfs, opt_func=sgd_opt)
```

Before trying to train, let's check the behavior works as intended: when we don't provide a value for `wd`, we pull the corresponding default from `weight_decay`.

```python
model = learn.model
```

```python
opt = sgd_opt(model.parameters(), lr=0.1)
test_eq(opt.hypers[0]['wd'], 0.)
test_eq(opt.hypers[0]['lr'], 0.1)
```

But if we provide a value, it overrides the default.

```python
opt = sgd_opt(model.parameters(), lr=0.1, wd=1e-4)
test_eq(opt.hypers[0]['wd'], 1e-4)
test_eq(opt.hypers[0]['lr'], 0.1)
```

Now let's fit.

```python
cbfs = [partial(AvgStatsCallback, accuracy), CudaCallback]
```

```python
learn, run = get_learn_run(nfs, data, 0.3, conv_layer,
                           cbs=cbfs, opt_func=partial(sgd_opt, wd=0.01))
```

```python
run.fit(1, learn)
```

    train: [1.8711743996858168, tensor(0.3426, device='cuda:0')]
    valid: [1.8584231936703821, tensor(0.3811, device='cuda:0')]

This is already better than the baseline! (**hahaha! It's not!**)

## With momentum

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=4872)

Momentum requires to add some state. We need to save the moving average of the gradients to be able to do the step and store this inside the optimizer state. To do this, we introduce statistics. Statistics are object with two methods:
- `init_state`, that returns the initial state (a tensor of 0. for the moving average of gradients)
- `update`, that updates the state with the new gradient value

We also read the `_defaults` values of those objects, to allow them to provide default values to hyper-parameters.

```python
#export
class StatefulOptimizer(Optimizer):
    def __init__(self, params, steppers, stats=None, **defaults):
        self.stats = listify(stats)
        maybe_update(self.stats, defaults, get_defaults)
        super().__init__(params, steppers, **defaults)
        self.state = {}

    def step(self):
        for p, hyper in self.grad_params():
            if p not in self.state:
                # Create a state for p and call all the statistics to initialize it.
                self.state[p] = {}
                maybe_update(
                    self.stats, self.state[p], lambda o: o.init_state(p))
            state = self.state[p]
            for stat in self.stats:
                state = stat.update(p, state, **hyper)
            compose(p, self.steppers, **state, **hyper)
            self.state[p] = state
```

`Stats` are object not dissimilar to steppers, they are object that we pass in to calculate, for example, momentum. Momentum is calculated by averaging the gradient, as is done by the `AverageGrad` class. It's not enough to have updates, we also need an initial state `init_state`.

```python
#export
class Stat():
    _defaults = {}

    def init_state(self, p):
        raise NotImplementedError

    def update(self, p, state, **kwargs):
        raise NotImplementedError
```

Here is an example of `Stat`:

```python
class AverageGrad(Stat):
    _defaults = dict(mom=0.9)

    def init_state(self, p):
        return {'grad_avg': torch.zeros_like(p.grad.data)}

    def update(self, p, state, mom, **kwargs):
        state['grad_avg'].mul_(mom).add_(p.grad.data)
        return state
```

Then we add the momentum step (instead of using the gradients to perform the step, we use the average).

```python
#export
def momentum_step(p, lr, grad_avg, **kwargs):
    p.data.add_(-lr, grad_avg)
    return p
```

```python
sgd_mom_opt = partial(StatefulOptimizer, steppers=[momentum_step, weight_decay],
                      stats=AverageGrad(), wd=0.01)
```

```python
learn, run = get_learn_run(nfs, data, 0.3, conv_layer,
                           cbs=cbfs, opt_func=sgd_mom_opt)
```

```python
run.fit(1, learn)
```

    train: [1.8644938829997888, tensor(0.3522, device='cuda:0')]
    valid: [1.7255024631767515, tensor(0.4138, device='cuda:0')]

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=5115) for discussion about weight decay interaction with batch normalisation.

In a nutshell, [this paper](https://arxiv.org/abs/1706.05350) shows that adding L2 regularization on top of batch normalization, has no regularizing effect. Assume we have some weights $W$ used to generate a vector of activations $\mathbf{a}$ that are then passed to a BatchNorm layer that will do a number of multiplications and of additions. Let's assume that we apply weight decay of 1e06 (yes, a million), so that the sum of the squares of the weights is also multiplied by a million. This would destroy the loss function, we might expect. The BatchNorm, however will also multiply by a million (**check**), what now happens to have the same activations as before we need to divide the weights by a million and the weight decay is nothing (**terrible explanation - check the math**).

Bottom line, using weight decay and BatchNorm together is pointless. Van Laarhoven's paper came out in June 2017. Truth is L2 regularization does something, and if you change it, it does something. A more recent [paper by Roger Grosse](https://arxiv.org/abs/1810.12281) shows three different mechanisms by which weight decay actually does something. Check the [lesson notes](https://course.fast.ai/videos/?lesson=11) for the links to these and other papers.

In the end, we don't really know how these things work, and everyone can actually contribute something useful. Using this telemetry helps a lot.

### Momentum experiments

Momentum is another thing that we don't really understand. What does momentum do to the gradients exactly? Let's do some plots to find out!

We generate 200 numbers from -4 to +4 with some added random noise. We then create a function to plot momentum for 4 different values of the parameter $\beta$. With a small value of $\beta$ we get a very bumpy trend, but as the momentum grows larger, it gets more and more wrong. This is why, if your momentum is too high, you will shoot past the "good" region of the parameter space.

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=5487)

```python
x = torch.linspace(-4, 4, 200)
y = torch.randn(200) + 0.3
betas = [0.5, 0.7, 0.9, 0.99]
```

```python
def plot_mom(f):
    _, axs = plt.subplots(2, 2, figsize=(12, 8))
    for beta, ax in zip(betas, axs.flatten()):
        ax.plot(y, linestyle='None', marker='.')
        avg, res = None, []
        for i, yi in enumerate(y):
            avg, p = f(avg, beta, yi, i)
            res.append(p)
        ax.plot(res, color='red')
        ax.set_title(f'beta={beta}')
```

This is the regular momentum.

```python
def mom1(avg, beta, yi, i):
    if avg is None:
        avg = yi
    res = beta*avg + yi
    return res, res

plot_mom(mom1)
```

As we can see, with a too high value, it may go way too high with no way to change its course.

Another way to smooth noisy data is to do an **exponentially weighted moving average**. In this case, there is a dampening of $1-\beta$ in front of the new value, which is less trusted than the current average. We'll define `lin_comb` (*linear combination*) to make this easier (note that in the lesson this was named `ewma`).

```python
# export
def lin_comb(v1, v2, beta): return beta*v1 + (1-beta)*v2
```

```python
def mom2(avg, beta, yi, i):
    if avg is None:
        avg = yi
    avg = lin_comb(avg, yi, beta)
    return avg, avg

plot_mom(mom2)
```

We can see it gets to a zero-constant when the data is purely random. If the data has a certain shape, it will get that shape (with some delay for high beta).

```python
y = 1 - (x/3) ** 2 + torch.randn(200) * 0.1
```

```python
y[0] = 0.5
```

```python
plot_mom(mom2)
```

Debiasing is here to correct the wrong information we may have in the very first batch. The debias term corresponds to the sum of the coefficient in our moving average. At the time step i, our average is:

$\begin{align*}
avg_{i} &= \beta\ avg_{i-1} + (1-\beta)\ v_{i} = \beta\ (\beta\ avg_{i-2} + (1-\beta)\ v_{i-1}) + (1-\beta)\ v_{i} \\
&= \beta^{2}\ avg_{i-2} + (1-\beta)\ \beta\ v_{i-1} + (1-\beta)\ v_{i} \\
&= \beta^{3}\ avg_{i-3} + (1-\beta)\ \beta^{2}\ v_{i-2} + (1-\beta)\ \beta\ v_{i-1} + (1-\beta)\ v_{i} \\
&\vdots \\
&= (1-\beta)\ \beta^{i}\ v_{0} + (1-\beta)\ \beta^{i-1}\ v_{1} + \cdots + (1-\beta)\ \beta^{2}\ v_{i-2} + (1-\beta)\ \beta\  v_{i-1} + (1-\beta)\ v_{i}
\end{align*}$

and so the sum of the coefficients is

$\begin{align*}
S &=(1-\beta)\ \beta^{i} + (1-\beta)\ \beta^{i-1} + \cdots + (1-\beta)\ \beta^{2} + (1-\beta)\ \beta + (1-\beta) \\
&= (\beta^{i} - \beta^{i+1}) + (\beta^{i-1} - \beta^{i}) + \cdots + (\beta^{2} - \beta^{3}) + (\beta - \beta^{2}) + (1-\beta) \\
&= 1 - \beta^{i+1}
\end{align*}$

since all the other terms cancel out each other.

By dividing by this term, we make our moving average a true average (in the sense that all the coefficients we used for the average sum up to 1).

```python
def mom3(avg, beta, yi, i):
    if avg is None:
        avg = 0
    avg = lin_comb(avg, yi, beta)
    return avg, avg/(1 - beta**(i+1))

plot_mom(mom3)
```

## Adam and friends

In Adam, we use the gradient averages but with dampening (not like in SGD with momentum), so let's add this to the `AverageGrad` class.

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=5889)

```python
# export
class AverageGrad(Stat):
    _defaults = dict(mom=0.9)

    def __init__(self, dampening: bool = False):
        self.dampening = dampening

    def init_state(self, p):
        return {'grad_avg': torch.zeros_like(p.grad.data)}

    def update(self, p, state, mom, **kwargs):
        state['mom_damp'] = 1-mom if self.dampening else 1.
        state['grad_avg'].mul_(mom).add_(state['mom_damp'], p.grad.data)
        return state
```

We also need to track the moving average of the gradients squared.

```python
# export
class AverageSqrGrad(Stat):
    _defaults = dict(sqr_mom=0.99)

    def __init__(self, dampening: bool = True):
        self.dampening = dampening

    def init_state(self, p):
        return {'sqr_avg': torch.zeros_like(p.grad.data)}

    def update(self, p, state, sqr_mom, **kwargs):
        state['sqr_damp'] = 1-sqr_mom if self.dampening else 1.
        state['sqr_avg'].mul_(sqr_mom).addcmul_(
            state['sqr_damp'], p.grad.data, p.grad.data)
        return state
```

We will also need the number of steps done during training for the debiasing.

```python
# export
class StepCount(Stat):
    def init_state(self, p):
        return {'step': 0}

    def update(self, p, state, **kwargs):
        state['step'] += 1
        return state
```

This helper function computes the debias term. If we dampening, `damp = 1 - mom` and we get the same result as before. If we don't use dampening, (`damp = 1`) we will need to divide by `1 - mom` because that term is missing everywhere.

```python
# export
def debias(mom, damp, step):
    return damp * (1 - mom**step) / (1-mom)
```

Then the Adam step is just the following:

```python
# export
def adam_step(p, lr, mom, mom_damp, step, sqr_mom, sqr_damp, grad_avg, sqr_avg, eps, **kwargs):
    debias1 = debias(mom,  mom_damp, step)
    debias2 = debias(sqr_mom, sqr_damp, step)
    p.data.addcdiv_(-lr / debias1, grad_avg, (sqr_avg/debias2).sqrt() + eps)
    return p

adam_step._defaults = dict(eps=1e-5)
```

```python
# export
def adam_opt(xtra_step=None, **kwargs):
    return partial(StatefulOptimizer, steppers=[adam_step, weight_decay]+listify(xtra_step),
                   stats=[AverageGrad(dampening=True), AverageSqrGrad(), StepCount()], **kwargs)
```

```python
learn, run = get_learn_run(
    nfs, data, 0.001, conv_layer, cbs=cbfs, opt_func=adam_opt())
```

```python
run.fit(3, learn)
```

    train: [1.8092643181038124, tensor(0.3743, device='cuda:0')]
    valid: [1.549335439888535, tensor(0.4764, device='cuda:0')]
    train: [1.3413920795820573, tensor(0.5576, device='cuda:0')]
    valid: [1.3411971287818472, tensor(0.5557, device='cuda:0')]
    train: [1.0581666779424965, tensor(0.6535, device='cuda:0')]
    valid: [1.1864999253582802, tensor(0.6115, device='cuda:0')]

## LAMB

[Jump_to lesson 11 video](https://course.fast.ai/videos/?lesson=11&t=6038)

It's then super easy to implement a new optimizer. This is LAMB from a [very recent paper](https://arxiv.org/pdf/1904.00962.pdf):

$\begin{align}
g_{t}^{l} &= \nabla L(w_{t-1}^{l}, x_{t}) \\
m_{t}^{l} &= \beta_{1} m_{t-1}^{l} + (1-\beta_{1}) g_{t}^{l} \\
v_{t}^{l} &= \beta_{2} v_{t-1}^{l} + (1-\beta_{2}) g_{t}^{l} \odot g_{t}^{l} \\
m_{t}^{l} &= m_{t}^{l} / (1 - \beta_{1}^{t}) \\
v_{t}^{l} &= v_{t}^{l} / (1 - \beta_{2}^{t}) \\
r_{1} &= \|w_{t-1}^{l}\|_{2} \\
s_{t}^{l} &= \frac{m_{t}^{l}}{\sqrt{v_{t}^{l} + \epsilon}} + \lambda w_{t-1}^{l} \\
r_{2} &= \| s_{t}^{l} \|_{2} \\
\eta^{l} &= \eta * r_{1}/r_{2} \\
w_{t}^{l} &= w_{t}^{l-1} - \eta_{l} * s_{t}^{l} \\
\end{align}$

```python
def lamb_step(p, lr, mom, mom_damp, step, sqr_mom, sqr_damp, grad_avg, sqr_avg, eps, wd, **kwargs):
    debias1 = debias(mom,     mom_damp, step)
    debias2 = debias(sqr_mom, sqr_damp, step)
    r1 = p.data.pow(2).mean().sqrt()
    step = (grad_avg/debias1) / ((sqr_avg/debias2).sqrt()+eps) + wd*p.data
    r2 = step.pow(2).mean().sqrt()
    p.data.add_(-lr * min(r1/r2, 10), step)
    return p

lamb_step._defaults = dict(eps=1e-6, wd=0.)
```

```python
lamb = partial(StatefulOptimizer, steppers=lamb_step, stats=[
               AverageGrad(dampening=True), AverageSqrGrad(), StepCount()])
```

```python
learn, run = get_learn_run(
    nfs, data, 0.003, conv_layer, cbs=cbfs, opt_func=lamb)
```

```python
run.fit(3, learn)
```

    train: [1.9359589928318723, tensor(0.3193, device='cuda:0')]
    valid: [1.9230039560111465, tensor(0.3493, device='cuda:0')]
    train: [1.464564054414405, tensor(0.5175, device='cuda:0')]
    valid: [1.3850895700636943, tensor(0.5389, device='cuda:0')]
    train: [1.15418611059774, tensor(0.6260, device='cuda:0')]
    valid: [1.2176715515525478, tensor(0.6064, device='cuda:0')]

Other recent variants of optimizers:
- [Large Batch Training of Convolutional Networks](https://arxiv.org/abs/1708.03888) (LARS also uses weight statistics, not just gradient statistics. Can you add that to this class?)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://arxiv.org/abs/1804.04235) (Adafactor combines stats over multiple sets of axes)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://arxiv.org/abs/1902.09843)

## Export

```python
!python notebook2script.py 09_optimizers.ipynb
```

    Converted 09_optimizers.ipynb to exp/nb_09.py

```python

```


```python
%matplotlib inline
```

# 592B, Class 5.1 (10/01).  The Fourier Transform in Action

```python
import numpy as np
import matplotlib.pyplot as plt
import librosa
from scipy import fftpack, signal #new

from ipywidgets import interactive
from IPython.display import Audio, display
```

## Review questions

Remember the Fourier series of the square wave you computed? Grab that code and define the square wave approximation as a Fourier series for $N=6$ as you did in Problem Set 2, and plot your square wave approximation.

Draw the expected spectrum by hand.

---

Last time we ended up with the Fourier series defined as $T \rightarrow \infty$, expressed as a double integral:

\begin{equation}
g(t) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty}e^{-2\pi ift}g(t)dt e^{2\pi ift}df
\end{equation}

and then we said you could rewrite it as $G(f)$, where $G(f)$ is defined as the Fourier transform of a function $g(t)$:

\begin{equation}
G(f) = \int_{-\infty}^{\infty}e^{-2\pi ift} g(t)dt
\end{equation}

and the inverse Fourier transform $g(t)$ of a function $G(f)$ is:

\begin{equation}
g(t) = \int_{-\infty}^{\infty}e^{2\pi ift} G(f)df
\end{equation}

Show that $g(t)$ rewritten in terms of $G(f)$ is the same as the $g(t)$ written out as a double integral.

---

Today we'll be having some fun with Fourier transforms, after all the hard work of mathematically deriving Fourier series and transforms. We'll be getting acquainted with [`scipy.fftpack`](https://docs.scipy.org/doc/scipy/reference/tutorial/fftpack.html). A good portion of this notebook is inspired by [this fantastic chapter of **Elegant SciPy**](https://www.safaribooksonline.com/library/view/elegant-scipy/9781491922927/ch04.html).

## The Fourier Transform in action: a first look

Remember this function?

$$y(t) = \cos \left(\frac{9\pi}{2}t\right ) $$

We worked with it when we were studying aliasing. Let's sample it at 20 Hz, for a duration of 20 seconds.

```python
fs = 20 # Sampling rate of 20 Hz
t_start = 0; t_stop = 20
ns = int((t_stop - t_start) * fs)
#t = np.linspace(t_start,t_stop,ns)
t = np.linspace(t_start,t_stop,ns, endpoint=False)
x = np.cos(9/2*np.pi*t)

fig, ax = plt.subplots()
ax.plot(t, x)
ax.set_xlabel('Time [s]')
ax.set_ylabel('Signal amplitude')
plt.xlim(0,10)
plt.title("A familiar signal")
```

Now let's take the Fourier Transform of the signal!

```python
X = fftpack.fft(x)
freqs = fftpack.fftfreq(len(x)) * fs

fig, ax = plt.subplots()

ax.stem(freqs, np.abs(X), use_line_collection = True)
ax.set_xlabel('Frequency in Hertz [Hz]')
ax.set_ylabel('Magnitude')
ax.set_xlim(-fs/2, fs/2)
#ax.set_xlim(-3.5, 3.5)
ax.set_ylim(-5, 220)
ax.set_ylabel('Fourier transform of signal')
```

We end up with non-zero values at only -2.25 and 2.25 Hz. At the moment, let's not worry about why we also get negative frequencies.

---
***Discussion: why set the x-axis limits as [-fs/2,fs/2]?***

***Discussion: does it make sense that we're getting a peak at 2.25 Hz?***

---

***In-class exercise: what happens to the computed spectrum if you delete `endpoint=False` in the np.linspace command? Why? ***

---

***In-class exercise: repeat what we did above for the signal sampled at 1 Hz. What happens and why?***

---

### Again, a bit more slowly...

Let's walk through this step-by-step. Now let's define a complex wave with two components.

```python
fs = 100
t_stop = 10
t = np.linspace(0,t_stop, t_stop*fs, endpoint=False)
f1 = 5; f2= 2
x = 0.1 * np.sin(2 * np.pi * f1 * t) + 1* np.sin(2 * np.pi * f2 * t) + 50
```

Let's take the Fourier Transform and inspect the values we get.

```python
X = fftpack.fft(x)

print(X[0:5])
print(len(X))

plt.stem(X, use_line_collection = True)
```

Aha! We are getting **complex** values!

What we actually plot in the spectrum turns out to be the size of a complex number, its **modulus**, defined as:

$$Mod(a+bi) = \sqrt{a^2+b^2}$$

In python, we can work with complex numbers as follows:

```python
c = complex(3,4) # syntax: complex(a,b) for defining a+bi
print(c)
print(c.real) # the real component, a
print(c.imag) # the imaginary component, b
print(np.abs(c)) # the modulus
```

    (3+4j)
    3.0
    4.0
    5.0

So let's try plotting the Fourier transform again, this time plotting the modulus.

```python
print(np.abs(X[0:5]))
plt.stem(np.abs(X), use_line_collection = True)
```

But there are still some problems. First the x-axis is in terms of points. We have 1000 points, so we get an x-axis going up to 1000. How do we get this in terms of frequency? We need to set the x-axis to go from [$-f_{Nyquist}, f_{Nyquist}$], $\Delta f_s$ apart. This is where `fftpack.fftfreq` comes in.

***Discussion: What do you notice about the array `freqs`? How long is it? What numbers does it range over? How far are the numbers apart?***

```python
freqs = fftpack.fftfreq(len(x)) * fs
print(freqs)

#print(fftpack.fftfreq(len(x)))

```

    [  0.     0.05   0.1    0.15   0.2    0.25   0.3    0.35   0.4    0.45
       0.5    0.55   0.6    0.65   0.7    0.75   0.8    0.85   0.9    0.95
       1.     1.05   1.1    1.15   1.2    1.25   1.3    1.35   1.4    1.45
       1.5    1.55   1.6    1.65   1.7    1.75   1.8    1.85   1.9    1.95
       2.     2.05   2.1    2.15   2.2    2.25   2.3    2.35   2.4    2.45
       2.5    2.55   2.6    2.65   2.7    2.75   2.8    2.85   2.9    2.95
       3.     3.05   3.1    3.15   3.2    3.25   3.3    3.35   3.4    3.45
       3.5    3.55   3.6    3.65   3.7    3.75   3.8    3.85   3.9    3.95
       4.     4.05   4.1    4.15   4.2    4.25   4.3    4.35   4.4    4.45
       4.5    4.55   4.6    4.65   4.7    4.75   4.8    4.85   4.9    4.95
       5.     5.05   5.1    5.15   5.2    5.25   5.3    5.35   5.4    5.45
       5.5    5.55   5.6    5.65   5.7    5.75   5.8    5.85   5.9    5.95
       6.     6.05   6.1    6.15   6.2    6.25   6.3    6.35   6.4    6.45
       6.5    6.55   6.6    6.65   6.7    6.75   6.8    6.85   6.9    6.95
       7.     7.05   7.1    7.15   7.2    7.25   7.3    7.35   7.4    7.45
       7.5    7.55   7.6    7.65   7.7    7.75   7.8    7.85   7.9    7.95
       8.     8.05   8.1    8.15   8.2    8.25   8.3    8.35   8.4    8.45
       8.5    8.55   8.6    8.65   8.7    8.75   8.8    8.85   8.9    8.95
       9.     9.05   9.1    9.15   9.2    9.25   9.3    9.35   9.4    9.45
       9.5    9.55   9.6    9.65   9.7    9.75   9.8    9.85   9.9    9.95
     -10.    -9.95  -9.9   -9.85  -9.8   -9.75  -9.7   -9.65  -9.6   -9.55
      -9.5   -9.45  -9.4   -9.35  -9.3   -9.25  -9.2   -9.15  -9.1   -9.05
      -9.    -8.95  -8.9   -8.85  -8.8   -8.75  -8.7   -8.65  -8.6   -8.55
      -8.5   -8.45  -8.4   -8.35  -8.3   -8.25  -8.2   -8.15  -8.1   -8.05
      -8.    -7.95  -7.9   -7.85  -7.8   -7.75  -7.7   -7.65  -7.6   -7.55
      -7.5   -7.45  -7.4   -7.35  -7.3   -7.25  -7.2   -7.15  -7.1   -7.05
      -7.    -6.95  -6.9   -6.85  -6.8   -6.75  -6.7   -6.65  -6.6   -6.55
      -6.5   -6.45  -6.4   -6.35  -6.3   -6.25  -6.2   -6.15  -6.1   -6.05
      -6.    -5.95  -5.9   -5.85  -5.8   -5.75  -5.7   -5.65  -5.6   -5.55
      -5.5   -5.45  -5.4   -5.35  -5.3   -5.25  -5.2   -5.15  -5.1   -5.05
      -5.    -4.95  -4.9   -4.85  -4.8   -4.75  -4.7   -4.65  -4.6   -4.55
      -4.5   -4.45  -4.4   -4.35  -4.3   -4.25  -4.2   -4.15  -4.1   -4.05
      -4.    -3.95  -3.9   -3.85  -3.8   -3.75  -3.7   -3.65  -3.6   -3.55
      -3.5   -3.45  -3.4   -3.35  -3.3   -3.25  -3.2   -3.15  -3.1   -3.05
      -3.    -2.95  -2.9   -2.85  -2.8   -2.75  -2.7   -2.65  -2.6   -2.55
      -2.5   -2.45  -2.4   -2.35  -2.3   -2.25  -2.2   -2.15  -2.1   -2.05
      -2.    -1.95  -1.9   -1.85  -1.8   -1.75  -1.7   -1.65  -1.6   -1.55
      -1.5   -1.45  -1.4   -1.35  -1.3   -1.25  -1.2   -1.15  -1.1   -1.05
      -1.    -0.95  -0.9   -0.85  -0.8   -0.75  -0.7   -0.65  -0.6   -0.55
      -0.5   -0.45  -0.4   -0.35  -0.3   -0.25  -0.2   -0.15  -0.1   -0.05]

OK, now we're set!

```python
fig, ax = plt.subplots()

ax.stem(freqs, np.abs(X), use_line_collection = True)
ax.set_xlabel('Frequency in Hertz [Hz]')
ax.set_ylabel('Magnitude')
#ax.set_xlim(-fs/2, fs/2)
ax.set_xlim(-10, 10)
ax.set_ylim(-5, 300)
ax.set_ylabel('Fourier transform of signal')

```

***In-class exercise: Fourier transform of a square wave. Compute a spectrum for the square wave you defined in the Review Questions.***

---

## From spectrum to spectrogram

Let's have some fun with [Japanese vowels](http://www.phonetics.ucla.edu/course/chapter9/japanese/japanese.html) from soundfiles collected by Peter Ladefoged. First we'll use `librosa` to load an audio recording of [ma]. And just for kicks, let's listen to it resampled at different sampling rates.

```python
y, sr = librosa.load("ma.aiff", sr = None)
print(type(y)) # Note that it is a numpy array
print("The sampling rate is: ", sr, "Hz")

y_8k = librosa.resample(y, sr, 8000)
y_2k = librosa.resample(y, sr, 2000)
y_500 = librosa.resample(y, sr, 500)

display(Audio(data=y,rate = sr))
display(Audio(data=y_8k,rate = 8000))
display(Audio(data=y_2k,rate = 2000))
display(Audio(data=y_500,rate = 500))

```

    <class 'numpy.ndarray'>
    The sampling rate is:  22050 Hz

<audio  controls="controls" >
    <source src="data:audio/wav;base64,UklGRoQnAABXQVZFZm10IBAAAAABAAEAIlYAAESsAAACABAAZGF0YWAnAACz/7f/pf+//73/kf+v/6n/m/+5/6v/k/+j/7H/k/+b//n//f8HAAAACwAAAPn/+f8DAPn/AwAFAPv/9f/t//X/AQAAAOn/+f/3/+3/BwD5//X/+f/7//f/6f/x/+///f/z/+v/7//x/+//5f/1/+X/AQDz/+n/9//v/+3/5f/x//H/3//x/9X/9f8AAO3/6f/r//H//f/r/+v/9//z//P/6f/p/9//8f/r/+3/3f/f/+P/6//n/+n/6//t/+P/8f/Z/9v/8f/X/93/0//j/93/4f/X/+3/7//h//v/3//x/+//5f/5/+f//f/v//n/7//x/wAAAwD///n/5/8BAOn/9f8DAPv/CQAHAAcAFwD//wcABwAFAAMABQDn//3/8f/5/wAA6f/r/+P/7//d/7//3f/N/7//vf+5/7f/vf+t/6//p/+j/4v/lf+B/53/jf+T/6P/m/+N/5H/m/99/4n/if+B/43/h/+X/6H/q/+T/5v/n/+P/6v/of+3/8P/w//T/73/zf/f/93/3//r//P/3f/V/+v/5//5//v/BQAPABEAGQA3ADcAOwA3ADkAIwApACcAGQAlABsALwAlABsAGQAxAC0AJQAdACMAJQALABcAHwATAOv/AwDz/+//1//V/9n/uf+9/7X/m/+p/5n/kf+P/4X/f/9d/1f/V/9D/0X/Lf8t/0P/Kf8l/y//I/8r/yX/If8p/xv/L/89/z3/K/8//1P/Sf9h/2H/W/9j/23/hf+B/4P/n/+n/7n/vf/L/9//+f8FABUALQA1AFUASQBlAH0AgwCLAH8AmQCfAKUAnwDNALsAtQDBAMEAzQCzAM8AxwDLAKkAkwCNAH8AdwB1AG8AVwBNACUAFQAFAPn/7//v/8X/t/+f/3//bf9l/13/Sf8p/x//A//x/uf+3f7H/rX+rf6R/n3+l/6J/o3+f/53/mn+a/5n/nf+hf6J/nH+e/6l/o/+qf7N/s3+3/7h/gn/Pf9D/2H/g/+Z/8H/0/8BAC8AQQBtAJsArwDXAPsANwFJAYMBlwHPAesBDwIzAmECXwJ5Ap8CqQK3AqsCvQK5Ao8CnwKFAmkCWQI5Ah0C9QHTAbkBdQFRAR0B7wDLAH0AWQAPANf/r/9r/03/8f7T/pn+Vf4//gf+8/3F/Xv9e/07/Tv9F/3f/L/8mfyL/G38W/xV/E/8Rfw1/E/8Ufxf/IP8n/zJ/N/8//wh/UX9c/2D/c/99/0j/nH+s/77/j//nf8AAGsA1QA7Ab0BKQKhAh0DfwP3A38E5QQ6BZAF4gUaBoYGxAb4BjoHageAB5oHmAeUB44HXgceB7QGSgboBXgFKgW/BE8E6wNXAwEDdwIVAqUB/QCbANf/X//B/iP+w/0b/Zv8B/x/+yv7pvpI+uT5bPkS+Zb4Xvj296j3aPci9+D2rvaQ9pz2mPbC9tL2BPc49273xvcO+HL4sPgW+Xb51vlk+tj6e/sb/K/8d/0n/uv+x/+NAGkBWwJDAykEIAX4BbAGTgcOCMgIkAlFCgML3wuLDGENGQ6dDhkPTw9bD1EP5Q6bDiEOZQ3PDBMMdQv5Ck0KCArCCTIJvggICBwH/AVfBMsC7wAr/4n94/uY+or5svj092L3zvZG9qH15/Qh9EPzK/JB8UTwfO/Y7ljuQu5S7ozuDu+673LwKfHX8W3y/fKT8wf0ifQF9Yn1Fvaw9kb3FPj2+Mz56/oR/EH9cf63/w0BRQJnA3MEUgUyBvwGCAgkCVUKjwu1DOsNAQ88EGQRmhKoE5kUYRX3FVEWdxYxFpkVnRRkEwASvhD4D7IPvg/GD+MOpQ27C0YJOAcVBY8DRQLhAGv/cf27+2T6XvnC+OL3MvdI9ln1l/SV86/yofF88HLvXu6e7S7tRu2G7Y7tYu3c7Ijsduzc7I7tSu7q7lzvqu++787v9O848J7wIfHN8XnyR/Mx9CX1+vXG9nb3bPik+Rf7q/wb/pn/EQFZAp8D/wR+BhwIvAmFC0kN8Q6gECQSqBMVFUsWrxcbGX4a9hv8HLgd4B10HYQctBqnGLcWFxUsFNYTChOuEWoPYQyiCc4GKAXrA9MCAQK//2H9oPow+Az3ifWR9AvzU/FG8CTv2u5y7sLtRO0y7FPrnerB6qTrgOw67frsiuwu7DzsBO3a7cruWO+C75Dvju/I7+jvQPC98BvxifHp8bvysfNt9BH1QfWH9fb16vZM+LD5Cfs3/H/95f6HAJsCnQSCBk4I1glBC9UMsQ7MENQSsxRLFqkXJxkCGxwdFx+fIOUhhyK1Ikci1yDPHsob3xgbFhgU9BLSEaYQoQ4/DIQJCgdIBeMDzQL9ACn+qvom97/01/Ir8bjv1u2A7IPrRevW62Ls8Owg7dDsOuzi657s0u0G77Dvpu847xTv6u8T8Ynyv/N99O30p/Tt8xHzg/Fw8LTvNO+i7/bva/GN8qny2fIZ8s/x1/FH8j/zTfOf8/3zffQO9q734vn/+zH+zQAnA6QFHghJChkMMw2fDgIQCBKFFFUX+hnMG4AdAR/lIOUi2CS0JvAnpiioKEInpiT5IBAdaRllFn0UrBIcEWIPDQ2pClwHEwUVAyEBG/+b+xj4cfQ38Rrvluyh6rPoWeft5gvnL+hF6YPqluui65zrWesS7HDtxu4M8GTwq/D/8NHxTfPD9D72WPfQ91L3lPba9Tv1KfWF9Df0d/Od8pfyJfKn8vXyD/Mr83Hyq/LB8j/zS/Tf9If10PVi9sD3Wvmn+wX+mQDJAqkEzgakCO0KOQ1yD3ARLBNzFasXNBrGHDMfwSG9IxQmSihPKgMszSyHLA8qYibPIRYdPxm3FqsU6hLsEOEOPw15CpwIggZHBDECQ/70+U/13fAo7u/qX+gG5vrjmOO242TlPefH6IfqLetn6w/rLetE7CbtUu687ujuKu/U75vxP/MT9cT25veu+Lr4ePiE9xj2EfXt81fzD/Mz8//zI/Qb9Lfzq/Ib8uHxKfKV8qnyF/N18zn0S/XS9pj4YvqJ/Kf+2wAlA1oFigdGCQUL7QzTDlYRLBRZFyoachyuHuEgLSO2JVYoySrtLJourC9mLwkt+SiBI5YdOxkfFuoTuBIcEQIQow1LCjoIiAUfBDECPf6++c/zdO8u7OXofeZ8457hw+AR4fLi9uSl5+vp++q56svpA+rR6oTsyO007kTu0O3e7lrwTfJv9K/1EPf699D4CvlM+J73lvYh9ZHzj/KL8gPz0fP/82PzefL58afyU/Ph893znfPJ8130b/Ww9jj4KPoZ/Bv+QQDdAnQFGgiPCoMMEw6IDyoSSRVzGGQb6h1XIP8iKCZ5KYksHi84MYwyXDJSMKUscifPIcYc3RiPFcITehLIEaQQJQ4ZDEYJyAZPBEEAcfs39djvnutX5zrkP+FR35Xe+d7P4H7iHOWd5zHps+nz6MXos+gV6Wnq0epF60/rOuwE7tjvL/J39Gj2JPh0+TT69PkA+Rr4mvbp9PHzS/Nx88XzKfQh9DXzmfKh8sPy/fLt8v/y5fIp8//z7/Q09nD3DPm2+mv8of7TAHUDwAUQCDoK+QtHDggREBQTF9gZoBw7H/8hGiVAKGMrNC7+MAoz/TM9M5wwMyyAJr0g7Bv1F0sVXhOyEeIQqQ7xDPcKVAhwBm0Cs/3i97XxOO136Hrk/+Cj3QzcXNuJ3LXe6eDi4/blHecf57/mBec55xHog+jd6DXp++nY68TtNPCx8tf0Jvfm+Kr6Sfu2+qD5ZPeX9Ynzk/L18lPzR/R99FP02/Nj89/zrfM7853yy/Gp8cHxu/IH9Df1vvY6+BD68ftH/skACQNMBS4HEgk/CwcOjBHnFCcYIBvqHakghSOuJq0pbywiL8Yx7TPPNOEzljB1K5Al7R9eG6UXlRX8E+YSXBHzDvMMXQoUCCwFmQAp+830LO9/6trl9uE93pbbKNo02uLb993f4HrjNOW65YDlaOVa5RLm0eZX59nna+jt6bLroO3s76PxF/Mz9Nv0gfUU9n726vZq9r31OfU79L3zX/MF88Hy//Gh8U3xHfGF8fHxgfI98+fz2fQU9uT3FPpf/Nf+MQFLA70EjAayCNkKdQ0uEB4TFRYvGbQc8x9jI7gmUSrRLfYwJzSbNlY4XDhRNXIvPCeJHygacxYpFS0VexWjFVgTtBBxDRYKVAgFBRMAUPnP8dbroebu4snfjdyQ2vTZPNun3fHg0uRN563nTuZS5A7j6OJo5P7l2+Yx52/nZ+jV6SDsiu6G8JfxV/ID8zPzn/Pf8wX0o/N58hPy+fEl8tfyD/OP8jvxFvD27xbw8/B18o3zm/TI9ab3tvkv/E//VwKbBJYGfghhCjsMtw5kEfITgRaPGeIckSCCJJIokyz2L20zezYOObo6djqNN1IxOyn1INYatRYRFaUVSxbjFkMV1hLAD3EM4gn2BXMAkPkF8gzs++ZY4y3gD91M2+zaAtwh3l7hWOXH5yXoJOZM43rhAeHg4hLkFOWM5ablQecd6bDr1u0S78zvVPAp8Vnxs/HN8dvxPfJn8snyr/KW8Cnql+nd6nfrquyu7MDtcu9J8jn1ivcS+kv+GwSsCKkNkBGmE8MWSRk8G8IdNyGwJKIoZSxcMeU3UT7+RkpXumMdZxdocmXUWrZG2CizC8P1IuRt1qDH47/fwBfKgdUE2/ndQ+CH5qzuFfLB8ILsNuwf8Vz3kf2BArkKyxVPIL4n9yhSJxMjlhyEEusB1O5R3RHS8cqIxuLD0MIuxVvJms040ITQMtCm0PfSIda82IncDuNO7gj6iwJhCnAQmRUNGNEVrA+WBTP7U/MA7yjtWOxu7ePwlvYr/acDEwsYE94ZHh6tIFwkQyr4MWI5aT7wQnhHok3hVNRbk19AYkpm2WlXafxhyk7HKY73xMOLn42LlYAFgAWAqYvJrBHUxfM6CNcXaCVQLzAxKCggG0YTMhJ2FMkUyRSrGAsgqSP+G0sKjfWs4RjOlLpYpsWWPY4ukn+hObgd1PLtqwHHDBYR2Q7vCs8Erfx58WnnduS96U/0R/0RAmsBTf0e9/zu6OQM2nbQL8wezuvVDuPD87oGCxdrIg8pLyx0LpovKi/zLOkqvSx7M789SUhuT6lUrFkpXzJkV2frafZs/G6ubg9oElsgRIUZZ93amwOAB4AHgAeAB4D9ggW10O48HWc3TEWMT+FSj0lEMoEZAwx7CqMLrgcFBIwF/wsZDdX+LOQQxams8J3BlVSSPpJ3mZGr98fh5yUCphFDFjsOwfzi7Hbln+dx69zt6/C094sDWw4WEKQFKfNO4dfUjM21ypXLh9Ib3xzvkf95DlQcQCYZKhQoxiTUJVErQjH1M0szyTM8OdVAbEebSr5NV1QrXEhi02Z2bCNzU3UMb+tdiz6IBy3BB4AJgAWACYAHgAWAiawX6fMhUEXUUOJRDlCGRWIuBQ+U+Cf0+PmLABUBEQT9CqUOTwSt6AzHd6sCnmqaKJxXosCu88G82JLvFwJnDSUNVf/D6NfSK8qmz3/dC+se9uEBvQ4TGGkWNgek8ODbrM3+xg7GxcsI2Kfo/PluB/gR0hnHHt8eNhtzGdgdvCZgLooxKjHRMwA5zDx/Ptk/2EOFTIRXcmKybJ90zHk6eBxtbVatK+3odJsHgAeAB4AHgAmARJ6d1IoTVkbZXQxaGk1fP5MsQhPs98XqQO+5++IGnQpZDDsMNgWP8QDRV7Mvn0uYCpuMpDm3ns2M5Xb44QKGBmwFQwAn9EveEMcLvo7HD92s7876/QECCaUOOQut+4LlB9S9y9PL9tBw28nqN/u2CL8O0A+aECgUURhpGfIZth2qJdktWjIyMzMzezQ7NYk1XTctPy5NuV3sa3N1vXtvfWB5Z2dXQL3+tK8FgAWAB4AHgAeA+Zd2zesL10E7X1pipVb+RdwvLBR++aPpDuyW94MDxAlrDOcN+AZt8/XU77aUpZmgY6IXqUO3as876sX/CQs7DhoKyfvG5KHKCLkrt0zFq9wt84oFIhNmGwob+w4Q+nziuM58w6/BOcgz1qfpZ/6LDjEXgRldF3YTXg+ZDrITMB7DKZow0DDyLlAvKDG2MQoxOTOGPI9MD16Ba+R1R37Pf5V7IGRPNgPxs6IHgAmAB4AFgAWA2asV598iIVQubFlrMlowQ44oCQsu8MLhyORj8iMCvQywEOcNSQHz6ojNvbOEpU2iq6pauaLOkuRm9zEEjwpBCh0B5fB62O7CaLgNv9PSE+qj//UNwxWVFgIQSwF+7OnWD8guwz/ItdUL57j54AjqEj8XrRXkEZENzQwIEUob5ihWMmczNi+pKuwnoCRHI8wmwDALQoBX4Guies1/0X/Pf2Zuokd+CZy8BYAHgAmAB4AHgDeg5dSMEM5FNWf0bOlfu0pkMToUNveg5ODjgO+p/agISQ6LDnAFffH/1ZK7Wa2trV23/7/TymTZ4u4tARgKrAf3+x7uHd8AzwTDkMQh1W3q4vpyBnMOfhOeEHsDPO6I2I3KYMZpyH7P1tu87hsDthKoGpIaKRfMEckOSBA7F90i+SyUL+IoWSFbIG4kdicBKoYwhz3wUIRlLnYxf81/zX91c3tUmBqqz92DBYAHgAWABYCymgvKJQSEOp1fXGxcZR1T9DpEHbP/S+ms45jsmPixAYwGaAkgBUf1Cd2uxH+2z7XlvbLDicpH1t3qd/46ByIG2f1f8hzjktGHwlvCJtHB6N76kQR7C0QQ0hGkCCr2ceAO0BHJdckUzzrZs+lx/QEPERkeHIQbkxjTFvsWNBsdIhooPymnI/gc8hrfHmAkdyqEMV0+C1KjaL560X/Pf+F7Cm2hS/wPtsUFgAeAB4AHgAOAYKb71HENy0HYZGJtuV/PSA4wkxeVACDtV+f47n38xAcDCy4I/f1E7A7X8sJkuNS6PMUIzV3U990U7/f9SwMv/+3yX+co2oLNPciUz07jNvmQCcwSwRWTFHcN4/yv5pTReMUqxJXJJ9Xa48L2FAq1GMsfuB6sGgkXuRaeGtkgpifBKkMpXSJKHcIbDB27IAAmSDIkRBhbw3DpftF/zX+0eUVgQS2s5R6dBYAHgAWABYD+mmbHf/7vM0Jb+2oCYrlMgTToHbAGR/FU5Qfq/PhEByENUwuQBeT5pecQ0Tm/bLl5vZPIG9PT37zt/fodAkkArPhy7OncGs7qxrnMX9038esDXBAhF0kYbBPsBiPzld1/zILDdMSAzs3eQfGHAkYR4hsdIY0hWx9YHrcgHCZ9KsspuyKJGdkVpxcMGyUfJCbRM7pGgVwibzJ5a3zweXJwbFhvKe3nQqQJgAeABYAChuGp7s8N/1MtYE+fXTJXrEY2MnAcwAbT8kfpuO1Y+ZEEDgoHC8oHs/wj60vWYMe4w9HK3tF31f7X/d9E7Wv1ePfx8P/oS9/M12/V1NuD6k/7VAeFC4UMJwvcBgX7J+qS2o7QgM9r04ba+uOv8IEBbhFqHHUgxR8VH5sgbCU9Kk8rqCUgHTEW0hPfFIUZRSJQLos/qVKQYjtrDXGHclxwZGHnQakNVcwmkgGAB4AFgNKaB8G16eUV9DqJVVdcqVO3P3Yodg+0+KHocuR568L2LQPDCmUMJAWZ9aLjWdMdzEHLmczs0OHVH96H5hbt2O/27H/pmONN3InVyNeW5Z738AUDDGcMpQrQBlH/p/EQ4pvVRs+wzj3S/Nob6dr6RQqnFB4aGh75IHsj4CY1KWEppCUzH1kY8hJuEfsU9BygJ182Wkd8WBNnB3IqeBp2fWkpSx8Z09ZJlQmABYAHgBSTvb5t6hEZW0CwW6tgb1K4OSMg3Ajt8/rlUuLD6gz5lAecEHQRfwo3+03nV9RMxiXBjsS4zSbZJ+Hj51rtmfGX8zDwY+h53vPUK9T93TDwnQJoCV4JowQ3Aq//ovax5zPW5csizfnUfd/76tr39AWsEYgaix+PISsjAiUIJ3YmTyO4HfUWmhE3D7oTehwZKlQ5L0p+WVhkm2rMawFpIV11QhQUKdUllgeAB4AFgPicksWa7gAbuUERXXNgTFAHNuocAwUS7/3epNr65Ab2NgcsELwQhQoj/SHr/Nf3ytbE/MW1zPfVv98R6EbtWu7o7H7sI+rO5ffeiNiL3dDsuwDPC6gI7f4g9lH0QfJl6MjYR8zPyhXUBuLm72/8JghoE7IbnSCZIwwm/iecJqkglxgIEmgQahGYETwUnBylK889hE1UWsJjg2uCcMRsTFnTMzsBsMbXjgeAB4AHgIqkWdSr/yomekUNXZ5hfFGIMjoSSviD5iXdsd1l6Kb4AgmIEpgSkAiQ+H/nYtigzuXLhtA107/SpdNO2nrlaOwC76fqTuQr3qncZ99x6aD5wQpkEl8N9QNy+jXz3+es2WPL/sPyxrjRXeDw7+MA7BJXIconOCYnIsEfJx9CHcwZQRUQEYwPOw9aEXcW1yFeMvRCAlGRXsNorG4gbclg2UipIFTtFLEJgAeAB4AeicS5b+gsFEg6rFgfZ6NdKENtIjwGmu5v3FXT/tcF6dn+hA/rFUQSqgdU+O3motcgz8bPJtdH3Ezb+NmS2+jh+uP84r3dzNtO4jLtkPir/1IF+gguB0n+KfFs5EfdZtlG1yXVv9X42vbkHPB0+t0CLQtCE90XLBrSGw0gWiQuJLseCBR1Cu4FjgZdDCMWlia0OdpMlFp8Y6Fp1m5WcBZkqkXpFATYnpsHgAeABYD5jA/CL/USJIFJ8GO6bD9fP0HuHMH83ONT04DNR9Y76lECmRTYGjcV+AZU9kblXtdy0N7RONcg2eTZHtww4xPrxu7064bj69wz3Kzhw+kD9N/8tQOOBk0EE/6Z9PfqOd9X1E7N6cvw0ErbG+ry+RgKjRcDIAMitR/GHbAdVB7IGuwSBAqlBO0EugnmEr0eNS3UPDlL4FiuY+BrCXE+bTpblzdHA8LC14QFgAeABYD5oRrXGAmbNvZYSm4gbKlWDTX2EObvX9WIxzPKDNw99YsM8BpcHgsX2AZP8rfeqtGMzWjRMtj/3k7ifuPO4zTk9+bF5vLlmOJA5ADtSPqKB8EL/gllAu75sO3h3drOssSAxFnKj9ZS5PnzgQOkEDcZWBoDGa0VphMIEkQTCxcfGEcVHQ+LC0ML+w3oE94cbCh/N9lJX1wPab9wUXGdZihNjCXX8pO1BYAHgAeAB4AztFXrEBo/Qu9efm9xaOpNiifRAlLlBtEByCTNFeH1+kwS8R6DH7cVWAV98ivf99EGzqvSFNqr4HHg99ww29/dQOTh57Ppveec62X19wHSCd4HnQC29rLu0OUD3T/VGtEV08DZ/uNy7lr5SQIgCXkNEBBQE+0VCxilGKgbhSAVIjgdCBT/C24JlQ4XGQwlpjE3QjFWf2cabzRtOV+wQwAbTOExogmABYAHgIuVqtGwBu01LlpUcFNyflvfNvUNcOz/0vLEsMMB08Tu/ww3I5cqIiV9FJv+reeR1bPMhs2H1YXcduGc4kziYuLu4/vnt+pG7dLvxvZLAc8MCBKJDaUBv/QX6IzaGs6KxY/CCsUIz9fen/ENBJQSNhoiG9cYARU9D3oI3wNeBc8N3xdyG6EXSBM2FKYaEiRbLWU0SD0XS25ZOWGeYYBbo0rlLKv+0cHwhweAB4AlgRO1LuzAHHFJZ2cAdvNqzE2cJrUAZuH1yY2/2MOu2VT3uBNmJcUpbSK8EG376uXR1mrQs9Iu2PndkOTP6hXnhd2Z1QnX1OKG70X8cwLrDSocXCQaGkn8UdwqxnG/Y79DwRLFKNDU5WX+vhIwG54alxQHDcQFD/+P/ZMA2gfGEAgb5SJzI7segRf4EugT/BxZK9Y550dXVnBjE2l2ZABRLjEvA6/IA48FgAWABYARqxPrqSGIToJsQnpqcDZRgiZR+7rZlMSTvZjDltiC93kXXy0PNHkrWxZF/OTiCtF7yavLddSt3ofobO6B8VHzo/IU8EfqJeFy217i9veCET8h+yLbFU8Ddu8g2sjE8rL5rfe02cep4Bz6pBCTIK0pzijHH+AQEwBZ8WvpXOzK+D8Myx6KJ/glcSEnIjInvy1pM/c3nT/NS2xZe2CLXVZOIjIFBVzHZ4sFgAeACYA+sBvo8hl1SKJsxXzpcptSHyrHAhziNch+uSa8k9K59dUXyS01M/cq3RjtADPoGdU/zHvME9Kq2XLj1O9V/aAHQwqlBBD6D+mq2cvTddwt8TcEtBAKEsQPAArN+9DhxcEZq3KmTbThymTljQCaHLc07T9IOiwmBwxH87bhstm82+3pYQIiHK8s9TNBNps3kDhSODU2yzREOgBHW1PPVhROlTUIB7fHrYoHgAeA/40CvO/miBE7P29pxXxLcXpO/CXDAhLlUctQumC97dXC+g4bwS0KMtUq0hpJA4np89S5y5fLQM+p1GfeUu77AFkPGBNxDNcAKPcg79HpBev98hEAlglfDMAGU/yM72bZ27+5rKupmrgs0UzuFgdSHjIx2DiEMFgbRwNm7YHeydbt1vPeefLhDcwnATiSPDQ6IzQKMPssxyp5LE0zjT1eRAZGyUCkL9ENMts2ptGBCYBrmcfA4+buCNMtiFDsZa5jOUrAJhIGIO1s2OXJx8jU2fr2NhQMJz0s9CZ3GSAGyO7I2fTNPcxa0PvVB95D6oT6KgqWEjgRfgj5/uj4zvdL+7EB6gbFATT4DO+P6j3nXOFQ20HTu9KR1Uzam97t6O75Hwv3FqoZ5xbQDwwJ+/8r9brrP+eD6eDvm/vvCtYbUSywOK0+LD12OZk1qjHsLlEt+SxvK4on5B1sB17kcr1doTSbGa1ezr7t+AdDH2g5Ok4dVCZGDihMCX/x4uLg16vT2to+7pAG0RhhIGodbxXeCaD6jemY23PVs9Ve2c/d0OTl8H0Atw08EgsOAAaj/9X9U//3AdEDxwQRBDEBj/zF807k39QnyP7E98nX1kLm9/Gz/+wJjhJZFSAUhQ0NAgT4Pu/J6fTlAedn6wX1vQOtFGwk8C+oOI48tjzKOTU0dyzeI2AekBr/GE0X3BLlBFzsts6Dtbmsm7b/0Xnydg9KJp8230EuQ+w6liYaCkrujtrd0jvVzeAR8q4G1RhAJLgkohq1Cmj5S+rX3W/WcdQU17PdV+ez80sB/w2vFTcVAQ7FBHv+wfw//pkAHwHL/3v9gPog983y0u0D6Czi+d4X4ZnoivC++TUCmwprDJAFdPoE7fHmQOXh52fpROyh83f8lgXbCxYRUBTbGHoegyMuJuYn4ylbK6ksYyvmJkUfKRiSEr8MAQV4+b/oV9V0xqfBfcmO2tP0QBJrKTs3BzbJLXsg6xTaB6f0JuNi2bnekutl+0AIdBKmGVQbWBT5A8fyZuYd4R/fXd/84kPqR/Wp/zwHRQuXDRsOZQoRA+X6lvbe91v8dwBNAYf/xfzW+VL2xfHG7VHr/erI7VfzH/zEBgYRsRc3GB4Tdwr9AKj39O0s467ah9WX1Z/cO+im9ncAvAZDC3oQsxaJF5oTbQwkCYcKfQyRDMkLBQ6aE5obrSH6JFwknSHoG7sUOw7QBlv+y/Jf53vdMte71ircWua/8+0DvBm7LLQyqyoZF5wIUf8F/AnyTOIo3LTjAvabAdwFcAaXCuwPkw7xAlP0oO4R8Q31ffSX8+j2s/2ZA5MCm/2Y+pv82/45/LT3+PVG+qsBFAfKB6sEMwEB/n75NfSx8C/xM/Ty9z38oQH6CB4QAhTuEbsKkwJN+6P1yfB47dzsku8j9bH72wFMBsQJLwwpDScM9gmIB/kE9QLv/g3zVuYX4RHpBvjZAh4IxAffC6gSCRWBDfv/+fRA7hLtFPDU9vP+9AgYFFAeBCYqKFsinRZNCucBt/yo+VT41PZN9Vv0EfXe9XD2M/X38drtAO238nv+7A8CJak09jHuHacCRfFi7GTsxea02uDYOehDAvUUdhrVGJkXHxY3Dff6y+Ym3APdHOTq63P1oQK+DwEX1xUMEWMOQw7PCgkAjfJJ6vzrwfR//dkBTQH1/sn8J/uI+jP70/xR/g3/WwCXA0AJfw6ID50KUQHe9wnxFO0B6xnqLOzv8aT6VQMiCRIJ7QMF/HH16/Ti+AH/swEtAy4GSwxIE2MVvBKDDasK5wqPDJUMkQrsB+AFBwXxAhsA9fpf9ZzwnO0c7SzuZ/HE9X76lf6zA2EMeRm+KAQvrCTJDZj2NOxV69rspege5AvqLvqHCwQTYhJbD78N1glLACnz1en/6H7tSfN0+S8CWQsCEAsOvAfFAhcC+QJBAFD5x/IT8en06fsnAs0EIwN9/0/8nvoj+7v8cf5h/28AewJOBUoIwAlOCCcDzfsF9bfwHu9a7z/xUfXR+/MDSQtXD4APgw3bCjYIKAWJAYn93Pnk92b44vrr/S8AyQC5/xf+af2z/Sn+Of7p/c39hf4ZAB0CkwMpBMEDoQJnAZcAfwB1ANv///7P/s//owFnAxEELwQ/A/MCWQKPAl0CHwHd/xv+8/1f/mP/ef9X/3H/2/9LAP//6f6Z/Lj6+vma+uH7Uf0r/88A+QJwBZYHzggACSgI5gXfAmcAsf5N/aP73vmc+JT4tvkV+/P7cfwv/fv9b/6//kX/u/9hAC0BAwJvArUCjQKvAUEA9/4z/uf90/3p/Rn+j/6j/1UBHQN5BBUFOAURBfEEmwR1A40BNf8f/dX7GfsJ+1n78fvl/PP99/5t/4//X/8H/8n+v/4r/0cAwQF1AxUFLAYMB0oHxAYyBbkCLQDJ/Qf8tPrK+W75tvm6+gn8Zf2v/pf/NQB9AL8AFwG1ATMCsQLTAtsC/wLTAiUC8wB7/xX+zfwH/LX77fuZ/J39s/7Z//cAyQEFAtcBeQETAcEAlwBxAGkAhQC3ANkA7QCzADEAl/+p/vv9m/2R/ZX9Ef6x/qH/nQCPAUECxQLzArcCPQKZAREBowBbAO//k/9f/0X/Uf87/wv/2f65/rv+A/9P/7X/IwCXAC0BzQEbAiEC7wGZATEBsQAxAK//T/8B/9P+0f7H/gv/R/9b/3X/ef+P/7n/2f8JABMAXwCbANUA+wAFAc0AmQAzAMv/ef8v/+v+uf6l/rP+2/4V/2P/of/B/+H/HwBDAGsAgwCZAJ0AkwC3ALMAlQBvAEEAAwDV/63/if9j/2f/b/99/7H/+/8tAHUAnwCrAKMAvQDLALsApwB/AGEAUwBfAEMAMQAXAOf/wf+7/53/nf+h/5v/t//L//3/IQBZAGsAewBxAIEAbwBZAC8AEQDx/83/pf+d/4//i/9x/3X/e/9x/4v/if+j/6//2////wkAHQAtADMAKwA1AB8ACwD//+//v/+v/5n/h/+X/4v/lf+N/5P/p//P/9H/6////wMADQAlAD0ANQAnACcAMQAlABMA/f/1/+n/1//n/9X/yf/J/83/6//n/+//CQARAAAAFQANAAUA9//1//P/5f/T/8P/x/+3/7P/of+b/43/i/8=" type="audio/wav" />
    Your browser does not support the audio element.
</audio>

<audio  controls="controls" >
    <source src="data:audio/wav;base64,UklGRm4OAABXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YUoOAADQ/7X/sf+u/6//pf/v/woA+f8CAPz/9v/6//X//P/5//H/9P/y/+3/8//0/+//7f/p//L/8P/x//X/7P/r/+j/5P/t/+r/5f/g/97/4//t/+7/7v/1//P//v/6//L/AAAJAAcABgD6//b/9P/j/9b/xv++/7b/oP+W/6L/n/+W/4//mv+p/6H/q//H/87/3//r/+H/9v8IACMAOAAnAB4AIQAfACUAHgAXAAkA8f/d/8P/rf+d/4b/Yf9O/0L/Pv83/zn/Rf9Q/2b/cf+F/6L/wf/p/xoAQABnAH8AjgCpAK4AsgC2AI4AbgBeACEA/P/Z/5f/b/9E/w3/7v69/qv+p/6L/p7+of66/uT+D/9d/6D/8/9RAKcADAFvAc8BIAJZAnoCbAJIAggCtwFJAdQASwC8/zj/p/4//tL9ff0n/dD8rPyW/Kr87vxA/ZT9A/6S/kn/MgBKAWoCkwOkBHAFLAa4BuwG0gYTBgwFFwTuAtMBgwAJ/6/9UPwv+yf6KPlg+LL3afeb9wr4x/ic+Z/6Avyo/Zj/twERBBgG2QfACa4LYA3/DXQNDwxkCi8JKgghBlgC5v11+nv4DfdF9d7ysPDO74rwRPLh8zD1cfbo9+v5nvy4/9YCNwWHB4IKbQ1lEPcSVhTZE/QQdw45DgwLbAW2AZ/9HfpJ+A/2y/MF8QPvLe+V7nruR/Ae8W3xZvIt9Gf2aPhh+0P/ngIvBmIKtA51EtoVWxlCGzAZSRTwEaoOUQdqA8X/ivn39XXyNPAW767svO2u7gDu/O8N8SPxDPIF8w31Pvah9wj7Lv6RAm4HJwsZEHUUgxhRHZ4foR3EFm8RdQ4NCL0D+v7k9iTyQ+5Y7eHuze1+7xjx8vAO9OL1QvRl8RfxqPN782HzofS79Rj6w/8FBhsLZA5eFPsZMR4PIxslFCCaFiURngzdBa0A+/ju8MLr/ehy62Xtl+2T8B3y8PPO9+v3SPZO9bXz5vMh9NvzpfXs9hv6KABbBbIKBxBBFaQbZSHKJoYoSiD0Ff8Q0wumBlkApvXl7DLnueaF6/zsv+0B8N/wTfR++D355/ZS9OP06fRT86Pzi/Tu9pH73ACHBuoKtBDoF6odtyOdKeEqEiHIFMkQsQu2BZj/nfNx6rHkh+QQ60Ps2uyl7/3vz/P991z5GfhM9F/0h/SZ8+D08/Qs96P71ACiB14MHBKdGTAg+ie+LfUqjR7vE5sQNQwnBi/8s+6D5ZThFOXK6v/qdesE7ezuWPQi+Yf6wvel9Ab1evTS8yv0dvQR9776wv/iBQcLohHnGNMflCc2LoEtfiGFFagQYwxNBz79Ue/W5N/ew+EX6Djpwuny6lXtF/Or+Kb7b/gK9O70OvXC9P/z4/JJ9e742/3dA6wI7g8OGCwfjCZ/LXYv3CSUF2MSAw6DCOr+ePAR5frdjd9E5u3nJ+iw6f/rFfHm9Kn2kPcW9pv0ovOK8ivzyfSo9zv9xwJOB2YNzxQJHcUlAC6eM/oqjxhOE/wSKAwLBXH2nefw39DdGOZq6azl4uet6XTsE/Li8wr1RPQ28yL0APLK8dv0FviF/hEFsAmhD2oWUB8vKZAxfzWqKfkW3ROmEyQN9QOp8zvmo9893//nv+j44+3mQuge7f/wk/Ld8hn0uvHY60Tuw/B09zoACg1TFHMa7yJgLWpCKl3TWB4hROaDyCPS2OL37SjxN/PlBegb2iV6FDPtO859yY/PY9V41qDfufQbDDsVGQd28a3ufveXCfMZtiI9Ncs/WFKVWRliUiZAtV6HcKCs7hQieibuE2IQGx3sEafgrq4UmjvLrAQODjz+IOj69uoBo/Hk2IvTY/a/HVspzypWKDVBe042XrFgR2RKIbCimoS1mmwDH0wjOJ0SwQJRDKX3ULgFmlWrDu2fFerxzeoc9I4L5fpk0kHUie2UGkIi9ybcK+I1HEDOUPBcuGsWRfu3d4KwljcCP1RENEwES/YHDSP7NrpHoEW/DPKTDVHcctgv+tUT3vrAy4DVYfaHGfoWTyDJKRs15zapSiZgPnJtNden6YJDp0kn21ZHH3z1i/vaEYjqFa5GpWbdgQDGA+TR89Pj/OkLzPBAzWrjPgT3EVcTpB92Kpsyty1VTTBmMHdxFZ2SEIbuxqVJ3U6PDq7u2QS3DCzXOqjHuFjxsw7X6Ua9Sub9EsIPHdnvyFnt1hV6EVgQASWKLeEqaDOHVl9yPWTF06SCSJe+C0BojzSj9SbsyBGY+bS9Oa0d394E9wF7y3/NNwRdFC/yq8gZ3wsIpxWxCT8f3SwgJvge3EZCa7Z5Fxb1kM6HidPPVIJSeQob6LYD9wsi04G108zX91kGtN8by1nyOxAGAgrSWtCB9OsZBg9qFPwmniKBH9c1Ul4uetpMULUBgJOuNCrSZQgm+O9G8mYNWOmowBDFBunRBf3xCskh4+AIfQ1h5SrMyOQ3D54ZHhSgIrce8hs0K1NWMHU2Y7fXi4BNnb4Mj2WoMxH7Ye+XDMTyZsadx6fjYv8A8xfPneRgDjMRZuXNyA7iZA/1GzAUCiNEIswZAiLbTlRzuGu05pODsZvnCcFhwTS4+v3u/wyC+/vMhsR25Yb/K/TpzgPihw1TFAntn8hj3/gL/x2MHIclbBsSFbglWVJ+cXxeHN78gfCpGBI4Vu0scvlG8gEMB/sG0WzQ/d+M9DbuSNin8eAK0wbg40DUROfTDR8d6R+sJWEW4hOaMfdYW2o4RuO+S4P4w40qwFPpHnHtffLnDEvyQdHa01XktvFl5JTcHf15Dbz8pN2Z0qzwuRIfHUIlAiAyEvMWTzwSX/xuFBlRlcaRHvHhT6Q/Dv565z8ElQ2X4R7HPdtE7aT02t1m4q4FggXN8kjTwd08ADkYOyFnIp4YRQ5zJp9Oz2FZUC/W4IKKuVIlTVaZHhLnnOv1D3f+pdQDzMDksu5W7qDdhPLxB1T2p+RW0KrwVw4+HzAkBBevDssVQjubWlplrBiUmLuSTvPDTeRCDPiu4PwAxQ/B50nSVtaf4NzwGuGj6I4KgQUO6bHNzNe1BWMhhB8XGLcQVw6/KJNPl2SVRNzI24Gxxvc15FcRFEPc5OqSEyIABtmP2QbfaeSf4TrtLgb8ADHmS9lN3vb1kA1qF6ogSBb4BNoVxEPWXaJkdwkYj7WXawT4Xhs+/+to1foDzBXB6Q/XE9vB5tHtzN+t7WIDOQC75rfR3d0+ByIdqhxyFn4GlAvjLuxP8WfqQHK5cIE50r1GV11gC1nOW+ziG0cD69al2STlSui05tzsrAfCAp7guMkI328GchilEbwT0xE0CnYXtjuSYVNeDPswiZSgNhhIY5E1yt761HcLUBqW6wPUS+H73wjnq+sS/+MEnutq2XfbdveICScSDhfjHfQPSQ0kL+JWgWJfCIWQtJ0sGwxoSDPd2Y7QchDoIv/smdE24c/kbenB7wgF2wss60/ORMxQ9WUWTxUdBToOZBZsFb4r6EV8WlAyLrWrhLzmbVlBVhT7A8XG7ysmeArH25fYUOsd3g7lYgEMHlb25sIOy5HyRBvgCfv+/gqTIPMUjxhRPZ5bz0mc07CBMsj4TfRlVAuyw+nhwCa2HMLcGtJd6Yf0w+5w4CwN+xnW5ea5FcuRC1olgQyi6vgHoiOfH14vo0GvWZ4eSp9BjyUD3GnCSSzn78CAADgwTwK400fYnfLQCjLujtz7ArwSCOlXsrvQ1husNV8CZNrQAgou1DKzMfA9vE5j5a6FlL4SNUJrdBzvyz7WbCJFJKDpj8+53SoF7Qto78vyZAj8/UnH4LhO+CwuMRyy5Drdbg8kOboqCSqVNZk/eehmjSTJri3wWJEYB9Rr4NEdQCD16P3RdN6dA6QNFPnlANf9D+rh3n/WcenbC6sWk/7M7AnzIB3oNhszMilXKPgHc7Ewv7EMKkQ6MGLpGtvZAVof5/+r32ja2fAiDrAE4/5ABMD/z+JnymjkZwV7E2kCj+uJ64QHsC2vNjss9xhtFrbw3LXr3w0mJD1kExvZWebdFN0cA/ND2qHfA/7mEz4DRf5MANr5nO0845Lusge2AQ3qA+pk+cALtheyIaonpCWlFl0FpeGpxiz6oi4jJRP+/99w96YXbQss6VbhMvMCCIQMBvwT+/MAkvnF70PuvAA7FqIJP/Jo2gbjOv9ED/gSHwlCCzgQXiEkHLoNJvUu3tzfYgi/Kg4RB/Rk5br6Qw0jBSX02fLF/Ub/2f3e+MP+4QZe/KHzPfaqBbgQOATO8d/wwfumCosJDAhE+abnkP0dDB8PU/e672sECB6cHvYCMvmt9rv1YvVJ7kwZsidB+cblouUpD+gZm/7+4H/t7g0ZEbALzvIV9LoCRvvO/Fz+ggcqDFL3puxb8ZwGCwHb9gYAhAkjEskKdgpgB8cAwfeW7F73Dv1THOEfhvQ86Q3x/g1DD2D61+ur+TsNjQWwARr1nvlvBNn7L/3q/1kGfwaT9kPwL/kEC7EMNwUf/Fr5UACb/hL+G/4PAOoDmwGKAA3/KAHYA2sC9AGx/uf+kf8oAFn9j/pi/twCNgjZBe3/jftp+fr7vP0D/1AALALzAez+Cv6C/gYCrASPBN0Btfx++zj9Zv9L/+b+sgF6BXcG0AE6/Bf6yfsq/20AegGCAqwCxQAv/Tf8JP4HAbgByABkAKkArAAO/7j9W/53AHoCPgLvAOr/ZP80/9n+Wf97AL8BvQGVAHj/3P4y/3v/tv8RAJ0A6QA7AFv/zf78/pn/BQBrAJAAnABNAMX/fP+H/yIAjwCvAJYAUgA9AOT/rP+n/+X/UQByAFYA/v+x/43/gf+R/8b/DgAsACMA+f+x/5n/l/+5/+3/EgAuACkADwDr/9n/0//s/wsACAD//97/zP+p/wAA" type="audio/wav" />
    Your browser does not support the audio element.
</audio>

<audio  controls="controls" >
    <source src="data:audio/wav;base64,UklGRrgDAABXQVZFZm10IBAAAAABAAEA0AcAAKAPAAACABAAZGF0YZQDAADF/5j////2//H/6//o/+v/4P/a/+T/9v8CAPn/s/+A/3H/mv/a/yEAOAAVALn/Lv/3/iX/tf+AAPAAngC//77+Iv5O/n7/bQEgA8UCWwCp/cL71/si/i0DbAgmCEICQvul9Uj1wfoaBcMP6RByBmj2WuyV7tn4bAZBFlcY0wcB9Zfqsum/8O79NxQsIlUS0feV6OLoi+3F9aEIkCIWIRoDXOpO6U3wHe3R9MIMJCr2IkX9JeTJ6i/0Ke899TINjS1BJI76B+HP6r70ZO8q9dgOjzDkJGX4s94m6oL04+9V9bcQ4DOsJPz0uttI6fr0PvBZ9AgPWzQCJ0v1yNie5j31hPCm8gUMUTS4KRD3+dZF5Bvy7+/m9MkTgzk4I0PxrNVf5yntP/GS+K8lwzkhGBLkUtjw6SjpR/NxDwFm0PuV0TsgoNg5wUIYpeQvZEo1N6y3NuTMG9KhAQjdSDv2f6nI/AXsCISi0giM3tMP/3/yAxbgrx5Ar2vngvTG6dZyzibo2YgbwsfG0Hf7bN6TVhZGKNpzF1rdRsNJAPbZQUC0VcDjVRFW7XbAyP0+3Zotylwt6vYK9PgTwAD71d4iH2xh/vHrBPsBi8DY95fjDhWGYqr2OgEACFLBHPn75fkSzWDW+vkAzgn1xKL3C+luE89ekvdrALMJvsU9+IjobxmoWVjxNQWxAmPGDPo25aIhJlJE7a0Lcvm6yZH6E+N/LrlEo+c/EOPt/8+/983kHznpN9/lWBXg4ozVbPdZ5sJEdybx6ZMV8tp63rPx0+3FSZUaSu9PFLfVN+V27SD3Bk3lCov4PA6E0wvt/Ob7BOJJEACF/uQJfM9b81DjeRncQKTyJA+29l/a2fT23wYqbjS+7ZsTR+6D30PzR+StN4cjvvJRFXXlBurJ7lToej3PGPH0JRQP5Zzv7enz7jM+Swxw+i0Q2eUY92/mXvBWOV4I2/jsDp/nFvpq6pPrWjJSC0z5MQwe7Or6rvFo4/8nJBTj9ZcNE/BF+iX8PuaWBGQqEfThCPb73/RS/7/4/fb0+Jgn+/RXBZn8Y/md/F/9Jv/2/i3+4ACNEQL02wnf9lsBF/5P/nj91fWEFH35pQTKAN/7EwCF/V7+4f6eAuP8wAERAxr+2f8QAh78Lv+OAgoAiP0QBNL8swCy/3X/qgBn/4AB0v6ZAXX+/gCp/s4ArP+ZABgA7f8UAFj/UABF/4EAkf8rAAAA" type="audio/wav" />
    Your browser does not support the audio element.
</audio>

<audio  controls="controls" >
    <source src="data:audio/wav;base64,UklGRgoBAABXQVZFZm10IBAAAAABAAEA9AEAAOgDAAACABAAZGF0YeYAAAB77X4Rzu2VDJz1Bf1mCYPk4CMY0Do149DyJ8npw/+JGhrBq0VXm/JeJpwjOYzqteVjRueW234BgJ5xpKzjG/ANg65cW3iDLWEZqLUiYwLE4+soe+rtEUYT5C+r9aEwgOpgGZj32gmaDkcFpSCe/XIhjwTlHZsOvx+6ISsgLyI4G+8cCA/2GSELTAcFBjQBEwMF/j8BLARS/t0GWf09BssB2Ak2AxAE7grlBpwMowaIDHIAmgnz8jcLkPJJA+8AKPIeGGPb/yTg4g4ALRih2OklveWeBBoCaflNBs732Ahn+KIGQvoAAA==" type="audio/wav" />
    Your browser does not support the audio element.
</audio>

---
***In-class exercise: compute the spectrum of [a] in the recording.***

1. Plot the waveform, and use the waveform to determine what part of the recording has a steady-state [a].
2. Now that you've determined that, work with just that portion of the signal for computing the spectrum.
3. Try computing the spectrum for the [a] sampled at different sampling rates. How does this affect what you see in the resulting spectrum, if at all?
4. Do your computed spectra look as expected, given what you know about formants? You might try computing spectra for the other vowel samples at the webpage, too, e.g., [o], [i], etc.

---

Now let's work towards computing the spectrogram for the audio file. First let's plot the waveform of the audio file.

***Discussion: do you notice a technical problem with the recording revealed from plotting the waveform?***

```python
N = y.shape[0]

L = N / sr

print(f'Audio length: {L:.2f} seconds')

f, ax = plt.subplots()
ax.plot(np.arange(N) / sr, y)
ax.set_xlabel('Time [s]')
ax.set_ylabel('Amplitude [unknown]');
```

We can compute a spectrogram using [spectral analysis tools provided by `scipy.signal`](https://github.com/scipy/scipy/blob/master/scipy/signal/spectral.py), namely, `scipy.signal.spectrogram`.

```python
help(signal.spectrogram)
```

    Help on function spectrogram in module scipy.signal.spectral:

    spectrogram(x, fs=1.0, window=('tukey', 0.25), nperseg=None, noverlap=None, nfft=None, detrend='constant', return_onesided=True, scaling='density', axis=-1, mode='psd')
        Compute a spectrogram with consecutive Fourier transforms.

        Spectrograms can be used as a way of visualizing the change of a
        nonstationary signal's frequency content over time.

        Parameters
        ----------
        x : array_like
            Time series of measurement values
        fs : float, optional
            Sampling frequency of the `x` time series. Defaults to 1.0.
        window : str or tuple or array_like, optional
            Desired window to use. If `window` is a string or tuple, it is
            passed to `get_window` to generate the window values, which are
            DFT-even by default. See `get_window` for a list of windows and
            required parameters. If `window` is array_like it will be used
            directly as the window and its length must be nperseg.
            Defaults to a Tukey window with shape parameter of 0.25.
        nperseg : int, optional
            Length of each segment. Defaults to None, but if window is str or
            tuple, is set to 256, and if window is array_like, is set to the
            length of the window.
        noverlap : int, optional
            Number of points to overlap between segments. If `None`,
            ``noverlap = nperseg // 8``. Defaults to `None`.
        nfft : int, optional
            Length of the FFT used, if a zero padded FFT is desired. If
            `None`, the FFT length is `nperseg`. Defaults to `None`.
        detrend : str or function or `False`, optional
            Specifies how to detrend each segment. If `detrend` is a
            string, it is passed as the `type` argument to the `detrend`
            function. If it is a function, it takes a segment and returns a
            detrended segment. If `detrend` is `False`, no detrending is
            done. Defaults to 'constant'.
        return_onesided : bool, optional
            If `True`, return a one-sided spectrum for real data. If
            `False` return a two-sided spectrum. Defaults to `True`, but for
            complex data, a two-sided spectrum is always returned.
        scaling : { 'density', 'spectrum' }, optional
            Selects between computing the power spectral density ('density')
            where `Sxx` has units of V**2/Hz and computing the power
            spectrum ('spectrum') where `Sxx` has units of V**2, if `x`
            is measured in V and `fs` is measured in Hz. Defaults to
            'density'.
        axis : int, optional
            Axis along which the spectrogram is computed; the default is over
            the last axis (i.e. ``axis=-1``).
        mode : str, optional
            Defines what kind of return values are expected. Options are
            ['psd', 'complex', 'magnitude', 'angle', 'phase']. 'complex' is
            equivalent to the output of `stft` with no padding or boundary
            extension. 'magnitude' returns the absolute magnitude of the
            STFT. 'angle' and 'phase' return the complex angle of the STFT,
            with and without unwrapping, respectively.

        Returns
        -------
        f : ndarray
            Array of sample frequencies.
        t : ndarray
            Array of segment times.
        Sxx : ndarray
            Spectrogram of x. By default, the last axis of Sxx corresponds
            to the segment times.

        See Also
        --------
        periodogram: Simple, optionally modified periodogram
        lombscargle: Lomb-Scargle periodogram for unevenly sampled data
        welch: Power spectral density by Welch's method.
        csd: Cross spectral density by Welch's method.

        Notes
        -----
        An appropriate amount of overlap will depend on the choice of window
        and on your requirements. In contrast to welch's method, where the
        entire data stream is averaged over, one may wish to use a smaller
        overlap (or perhaps none at all) when computing a spectrogram, to
        maintain some statistical independence between individual segments.
        It is for this reason that the default window is a Tukey window with
        1/8th of a window's length overlap at each end.

        .. versionadded:: 0.16.0

        References
        ----------
        .. [1] Oppenheim, Alan V., Ronald W. Schafer, John R. Buck
               "Discrete-Time Signal Processing", Prentice Hall, 1999.

        Examples
        --------
        >>> from scipy import signal
        >>> import matplotlib.pyplot as plt

        Generate a test signal, a 2 Vrms sine wave whose frequency is slowly
        modulated around 3kHz, corrupted by white noise of exponentially
        decreasing magnitude sampled at 10 kHz.

        >>> fs = 10e3
        >>> N = 1e5
        >>> amp = 2 * np.sqrt(2)
        >>> noise_power = 0.01 * fs / 2
        >>> time = np.arange(N) / float(fs)
        >>> mod = 500*np.cos(2*np.pi*0.25*time)
        >>> carrier = amp * np.sin(2*np.pi*3e3*time + mod)
        >>> noise = np.random.normal(scale=np.sqrt(noise_power), size=time.shape)
        >>> noise *= np.exp(-time/5)
        >>> x = carrier + noise

        Compute and plot the spectrogram.

        >>> f, t, Sxx = signal.spectrogram(x, fs)
        >>> plt.pcolormesh(t, f, Sxx)
        >>> plt.ylabel('Frequency [Hz]')
        >>> plt.xlabel('Time [sec]')
        >>> plt.show()

        Note, if using output that is not one sided, then use the following:

        >>> f, t, Sxx = signal.spectrogram(x, fs, return_onesided=False)
        >>> plt.pcolormesh(t, np.fft.fftshift(f), np.fft.fftshift(Sxx, axes=0))
        >>> plt.ylabel('Frequency [Hz]')
        >>> plt.xlabel('Time [sec]')
        >>> plt.show()

We can use the example code from the help file to try to compute the spectrogram for [ma].

***Discussion: Does the spectrogram look as you expect? Fiddle with the settings of the plot as suits you. Also try using the code from [Elegant SciPy](https://www.safaribooksonline.com/library/view/elegant-scipy/9781491922927/ch04.html) used to produce figure 4-4. How does that look? Compare your computed spectrograms to the ones you get when you analyze the [ma] file in Praat.***

```python
f,t, Sxx = signal.spectrogram(y,sr)
plt.pcolormesh(t, f, Sxx)
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [sec]')
```

To better understand how scipy.signal.spectrogram works, let's look at how it is defined. We can do this [using the `inspect` module](https://stackoverflow.com/questions/427453/how-can-i-get-the-source-code-of-a-python-function#427533). We can also look at [the code on github](https://github.com/scipy/scipy/blob/master/scipy/signal/spectral.py), which is a little prettier.

```python
import inspect
lines = inspect.getsourcelines(signal.spectrogram)
print("".join(lines[0]))
```

    def spectrogram(x, fs=1.0, window=('tukey', .25), nperseg=None, noverlap=None,
                    nfft=None, detrend='constant', return_onesided=True,
                    scaling='density', axis=-1, mode='psd'):
        """
        Compute a spectrogram with consecutive Fourier transforms.

        Spectrograms can be used as a way of visualizing the change of a
        nonstationary signal's frequency content over time.

        Parameters
        ----------
        x : array_like
            Time series of measurement values
        fs : float, optional
            Sampling frequency of the `x` time series. Defaults to 1.0.
        window : str or tuple or array_like, optional
            Desired window to use. If `window` is a string or tuple, it is
            passed to `get_window` to generate the window values, which are
            DFT-even by default. See `get_window` for a list of windows and
            required parameters. If `window` is array_like it will be used
            directly as the window and its length must be nperseg.
            Defaults to a Tukey window with shape parameter of 0.25.
        nperseg : int, optional
            Length of each segment. Defaults to None, but if window is str or
            tuple, is set to 256, and if window is array_like, is set to the
            length of the window.
        noverlap : int, optional
            Number of points to overlap between segments. If `None`,
            ``noverlap = nperseg // 8``. Defaults to `None`.
        nfft : int, optional
            Length of the FFT used, if a zero padded FFT is desired. If
            `None`, the FFT length is `nperseg`. Defaults to `None`.
        detrend : str or function or `False`, optional
            Specifies how to detrend each segment. If `detrend` is a
            string, it is passed as the `type` argument to the `detrend`
            function. If it is a function, it takes a segment and returns a
            detrended segment. If `detrend` is `False`, no detrending is
            done. Defaults to 'constant'.
        return_onesided : bool, optional
            If `True`, return a one-sided spectrum for real data. If
            `False` return a two-sided spectrum. Defaults to `True`, but for
            complex data, a two-sided spectrum is always returned.
        scaling : { 'density', 'spectrum' }, optional
            Selects between computing the power spectral density ('density')
            where `Sxx` has units of V**2/Hz and computing the power
            spectrum ('spectrum') where `Sxx` has units of V**2, if `x`
            is measured in V and `fs` is measured in Hz. Defaults to
            'density'.
        axis : int, optional
            Axis along which the spectrogram is computed; the default is over
            the last axis (i.e. ``axis=-1``).
        mode : str, optional
            Defines what kind of return values are expected. Options are
            ['psd', 'complex', 'magnitude', 'angle', 'phase']. 'complex' is
            equivalent to the output of `stft` with no padding or boundary
            extension. 'magnitude' returns the absolute magnitude of the
            STFT. 'angle' and 'phase' return the complex angle of the STFT,
            with and without unwrapping, respectively.

        Returns
        -------
        f : ndarray
            Array of sample frequencies.
        t : ndarray
            Array of segment times.
        Sxx : ndarray
            Spectrogram of x. By default, the last axis of Sxx corresponds
            to the segment times.

        See Also
        --------
        periodogram: Simple, optionally modified periodogram
        lombscargle: Lomb-Scargle periodogram for unevenly sampled data
        welch: Power spectral density by Welch's method.
        csd: Cross spectral density by Welch's method.

        Notes
        -----
        An appropriate amount of overlap will depend on the choice of window
        and on your requirements. In contrast to welch's method, where the
        entire data stream is averaged over, one may wish to use a smaller
        overlap (or perhaps none at all) when computing a spectrogram, to
        maintain some statistical independence between individual segments.
        It is for this reason that the default window is a Tukey window with
        1/8th of a window's length overlap at each end.

        .. versionadded:: 0.16.0

        References
        ----------
        .. [1] Oppenheim, Alan V., Ronald W. Schafer, John R. Buck
               "Discrete-Time Signal Processing", Prentice Hall, 1999.

        Examples
        --------
        >>> from scipy import signal
        >>> import matplotlib.pyplot as plt

        Generate a test signal, a 2 Vrms sine wave whose frequency is slowly
        modulated around 3kHz, corrupted by white noise of exponentially
        decreasing magnitude sampled at 10 kHz.

        >>> fs = 10e3
        >>> N = 1e5
        >>> amp = 2 * np.sqrt(2)
        >>> noise_power = 0.01 * fs / 2
        >>> time = np.arange(N) / float(fs)
        >>> mod = 500*np.cos(2*np.pi*0.25*time)
        >>> carrier = amp * np.sin(2*np.pi*3e3*time + mod)
        >>> noise = np.random.normal(scale=np.sqrt(noise_power), size=time.shape)
        >>> noise *= np.exp(-time/5)
        >>> x = carrier + noise

        Compute and plot the spectrogram.

        >>> f, t, Sxx = signal.spectrogram(x, fs)
        >>> plt.pcolormesh(t, f, Sxx)
        >>> plt.ylabel('Frequency [Hz]')
        >>> plt.xlabel('Time [sec]')
        >>> plt.show()

        Note, if using output that is not one sided, then use the following:

        >>> f, t, Sxx = signal.spectrogram(x, fs, return_onesided=False)
        >>> plt.pcolormesh(t, np.fft.fftshift(f), np.fft.fftshift(Sxx, axes=0))
        >>> plt.ylabel('Frequency [Hz]')
        >>> plt.xlabel('Time [sec]')
        >>> plt.show()
        """
        modelist = ['psd', 'complex', 'magnitude', 'angle', 'phase']
        if mode not in modelist:
            raise ValueError('unknown value for mode {}, must be one of {}'
                             .format(mode, modelist))

        # need to set default for nperseg before setting default for noverlap below
        window, nperseg = _triage_segments(window, nperseg,
                                           input_length=x.shape[axis])

        # Less overlap than welch, so samples are more statisically independent
        if noverlap is None:
            noverlap = nperseg // 8

        if mode == 'psd':
            freqs, time, Sxx = _spectral_helper(x, x, fs, window, nperseg,
                                                noverlap, nfft, detrend,
                                                return_onesided, scaling, axis,
                                                mode='psd')

        else:
            freqs, time, Sxx = _spectral_helper(x, x, fs, window, nperseg,
                                                noverlap, nfft, detrend,
                                                return_onesided, scaling, axis,
                                                mode='stft')

            if mode == 'magnitude':
                Sxx = np.abs(Sxx)
            elif mode in ['angle', 'phase']:
                Sxx = np.angle(Sxx)
                if mode == 'phase':
                    # Sxx has one additional dimension for time strides
                    if axis < 0:
                        axis -= 1
                    Sxx = np.unwrap(Sxx, axis=axis)

            # mode =='complex' is same as `stft`, doesn't need modification

        return freqs, time, Sxx

Well that's not that helpful. Looks like we need to take a look at `signal._spectral_helper`, which we can look at in the github code.

```python
# This doesn't work!
#lines = inspect.getsourcelines(signal.spectral_helper)
#print("".join(lines[0]))
```

Now let's use the code from [Elegant Scipy](https://www.safaribooksonline.com/library/view/elegant-scipy/9781491922927/ch04.html) used to produce Figure 4.3, which breaks down spectrogram computation step by step.

```python
from skimage import util # See installation here: https://anaconda.org/anaconda/scikit-image

M = 1024

slices = util.view_as_windows(y, window_shape=(M,), step=100)
print(f'Audio shape: {y.shape}, Sliced audio shape: {slices.shape}')

win = np.hanning(M + 1)[:-1]
slices = slices * win

slices = slices.T
print('Shape of `slices`:', slices.shape)

spectrum = np.fft.fft(slices, axis=0)[:M // 2 + 1:-1]
spectrum = np.abs(spectrum)
```

    Audio shape: (5040,), Sliced audio shape: (41, 1024)
    Shape of `slices`: (1024, 41)

```python
f, ax = plt.subplots(figsize=(4.8, 2.4))

S = np.abs(spectrum)
S = 20 * np.log10(S / np.max(S))

ax.imshow(S, origin='lower', cmap='viridis',
          extent=(0, L, 0, sr / 2 / 1000))
ax.axis('tight')
ax.set_ylabel('Frequency [kHz]')
ax.set_xlabel('Time [s]');
```

## Working through DFT, FFT

Finally, let's work through the [Elegant Scipy](https://www.safaribooksonline.com/library/view/elegant-scipy/9781491922927/ch04.html) material on DFT and FFT together. This is the section that starts after Figure 4.4.