[source](../api/alibi_detect.cd.mmd.rst)

# Maximum Mean Discrepancy

## Overview

The [Maximum Mean Discrepancy (MMD)](http://jmlr.csail.mit.edu/papers/v13/gretton12a.html) detector is a kernel-based method for multivariate 2 sample testing. The MMD is a distance-based measure between 2 distributions *p* and *q* based on the mean embeddings $\mu_{p}$ and $\mu_{q}$ in a reproducing kernel Hilbert space $F$:

\begin{align}
MMD(F, p, q) & = || \mu_{p} - \mu_{q} ||^2_{F} \\
\end{align}

We can compute unbiased estimates of $MMD^2$ from the samples of the 2 distributions after applying the kernel trick. We use by default a [radial basis function kernel](https://en.wikipedia.org/wiki/Radial_basis_function_kernel), but users are free to pass their own kernel of preference to the detector. We obtain a $p$-value via a [permutation test](https://en.wikipedia.org/wiki/Resampling_(statistics)) on the values of $MMD^2$.

For high-dimensional data, we typically want to reduce the dimensionality before computing the permutation test. Following suggestions in [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://arxiv.org/abs/1810.11953), we incorporate Untrained AutoEncoders (UAE), black-box shift detection using the classifier's softmax outputs ([BBSDs](https://arxiv.org/abs/1802.03916)) and [PCA](https://en.wikipedia.org/wiki/Principal_component_analysis) as out-of-the box preprocessing methods. Preprocessing methods which do not rely on the classifier will usually pick up drift in the input data, while BBSDs focuses on label shift.

Detecting input data drift (covariate shift) $\Delta p(x)$ for text data requires a custom preprocessing step. We can pick up changes in the semantics of the input by extracting (contextual) embeddings and detect drift on those. Strictly speaking we are not detecting $\Delta p(x)$ anymore since the whole training procedure (objective function, training data etc) for the (pre)trained embeddings has an impact on the embeddings we extract. The library contains functionality to leverage pre-trained embeddings from [HuggingFace's transformer package](https://github.com/huggingface/transformers) but also allows you to easily use your own embeddings of choice. Both options are illustrated with examples in the [Text drift detection on IMDB movie reviews](../examples/cd_text_imdb.nblink) notebook.

## Usage

### Initialize

Parameters:

* `p_val`: p-value used for significance of the permutation test.

* `X_ref`: Data used as reference distribution.

* `preprocess_X_ref`: Whether to already apply the (optional) preprocessing step to the reference data at initialization and store the preprocessed data. Dependent on the preprocessing step, this can reduce the computation time for the predict step significantly, especially when the reference dataset is large. Defaults to *True*.

* `update_X_ref`: Reference data can optionally be updated to the last N instances seen by the detector or via [reservoir sampling](https://en.wikipedia.org/wiki/Reservoir_sampling) with size N. For the former, the parameter equals *{'last': N}* while for reservoir sampling *{'reservoir_sampling': N}* is passed.

* `preprocess_fn`: Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique.

* `preprocess_kwargs`: Keyword arguments for `preprocess_fn`. Again see the notebooks for [image](../examples/cd_mmd_cifar10.nblink) and [text](../examples/cd_text_imdb.nblink) data for concrete, detailed examples. The built-in *UAE*, *BBSDs* or text-specific preprocessing steps are passed here as well. See below for a brief example.

* `kernel`: Kernel function used when computing the MMD. Defaults to a Gaussian kernel.

* `kernel_kwargs`: Keyword arguments for the kernel function. For the Gaussian kernel this is the kernel bandwidth `sigma`. We can also sum over a number of different kernel bandwidths. `sigma` then becomes an array with different values. If `sigma` is not specified, the detector will infer it by computing the pairwise distances between each of the instances in the 2 samples and set `sigma` to the median distance.

* `n_permutations`: Number of permutations used in the permutation test.

* `chunk_size`: Used to optionally compute the MMD between the 2 samples in chunks using [dask](https://dask.org/) to avoid potential out-of-memory errors. In terms of speed, the optimal chunk size is application and hardware dependent, so it is often worth to test a few different values, including *None*. *None* means that the computation is done in-memory in NumPy.

* `data_type`: can specify data type added to metadata. E.g. *'tabular'* or *'image'*.

Initialized drift detector example:

```python
from alibi_detect.cd import MMDDrift
from alibi_detect.cd.preprocess import uae  # Untrained AutoEncoder

encoder_net = tf.keras.Sequential(
  [
      InputLayer(input_shape=(32, 32, 3)),
      Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),
      Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),
      Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu),
      Flatten(),
      Dense(32,)
  ]
)
uae = UAE(encoder_net=encoder_net)

cd = MMDDrift(
    p_val=.05,
    X_ref=X_ref,
    preprocess_X_ref=True,
    preprocess_kwargs={'model': uae, 'batch_size': 128},
    kernel=gaussian_kernel,
    kernel_kwargs={'sigma': np.array([.5, 1., 5.])},
    chunk_size=1000,
    n_permutations=1000
)
```

### Detect Drift

We detect data drift by simply calling `predict` on a batch of instances `X`. We can return the p-value and the threshold of the permutation test by setting `return_p_val` to *True* and the maximum mean discrepancy metric and threshold by setting `return_distance` to *True*.

The prediction takes the form of a dictionary with `meta` and `data` keys. `meta` contains the detector's metadata while `data` is also a dictionary which contains the actual predictions stored in the following keys:

* `is_drift`: 1 if the sample tested has drifted from the reference data and 0 otherwise.

* `p_val`: contains the p-value if `return_p_val` equals *True*.

* `threshold`: p-value threshold if `return_p_val` equals *True*.

* `distance`: MMD metric between the reference data and the new batch if `return_distance` equals *True*.

* `distance_threshold`: MMD metric value from the permutation test which corresponds to the the p-value threshold.

```python
preds_drift = cd.predict(X, return_p_val=True, return_distance=True)
```

### Saving and loading

The drift detectors can be saved and loaded in the same way as other detectors when using the built-in preprocessing steps (`alibi_detect.cd.preprocess.UAE` and `alibi_detect.cd.preprocess.HiddenOutput`) or no preprocessing at all:

```python
from alibi_detect.utils.saving import save_detector, load_detector

filepath = 'my_path'
save_detector(cd, filepath)
cd = load_detector(filepath)
```

A custom preprocessing step can be passed as follows:

```python
cd = load_detector(filepath, **{'preprocess_kwargs': preprocess_kwargs})
```

## Examples

### Image

[Drift detection on CIFAR10](../examples/cd_mmd_cifar10.nblink)

[Text drift detection on IMDB movie reviews](../examples/cd_text_imdb.nblink)


# Implementing FIR filters

In real-time filtering applications, filters are implemented by using some variation or other of their constant-coefficient difference equation (CCDE), so that one new output sample is generated for each new input sample. If all input data is available in advance, as in non-real-time (aka "offline") applications, then the CCDE-based algorithm is iteratively applied to all samples in the buffer.

In the case of FIR filters, the CCDE coefficients correspond to the impulse response and implementing the CCDE is equivalent to performing a convolution sum. In this notebook we will look at different ways to implement FIR filters.

```python
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
```

## Online implementation

The classic way to implement a filter is the one-in one-out approach. We will need to implement a persistent delay line. In Python we can either define a class or use function attributes; classes are tidier and reusable:

```python
class FIR_loop():
    def __init__(self, h):
        self.h = h
        self.ix = 0
        self.M = len(h)
        self.buf = np.zeros(self.M)

    def filter(self, x):
        y = 0
        self.buf[self.ix] = x
        for n in range(0, self.M):
            y += self.h[n] * self.buf[(self.ix+self.M-n) % self.M]
        self.ix = (self.ix + 1) % self.M
        return y
```

```python
# simple moving average:
h = np.ones(5)/5

f = FIR_loop(h)
for n in range(0, 10):
    print(round(f.filter(n), 3),  end='  ')
```

    0.0  0.2  0.6  1.2  2.0  3.0  4.0  5.0  6.0  7.0

While there's nothing wrong with the above implementation, when the data to be filtered is known in advance, it makes no sense to explicitly iterate over its element and it's better to use higher-level commands to perform the convolution. In Numpy, the command is `convolve`; before we use it, though, we need to take border effects into consideration.

## Offline implementations: border effects

When filtering a finite-length data vector with a finite-length impulse response, we need to decide what to do with the "invalid" shifts appearing in the terms of the convolution sum. Remember that, in the infinite-length case, the output is defined as

$$
    y[n] = \sum_{k=-\infty}^{\infty} h[k]x[n-k]
$$

Let's say that the impulse response is $M$ points long, so that $h[n]$ is nonzero only between $0$ and $M-1$; this means that the sum is reduced to

$$
    y[n] = \sum_{k=0}^{M-1} h[k]x[n-k]
$$

Now assume that $x[n]$ is a length-$N$ signal, so it is defined only for $0 \leq n \le N$ (we can safely consider $N > M$, otherwise exchange the roles of $x$ and $h$). In this case, the above sum is properly defined only for $M - 1 \le n \le N-1$; for any other value of $n$, the sum will contain an element $x[n-k]$ outside of the valid range of indices for the input.

So, if we start with an $N$-point input, we can only formally compute $N-M+1$ output samples. While this may not be a problem in some applications, it certainly is troublesome if repeated filtering operations end up "chipping away" at the signal little by little.

The solution is to "embed" the finite-length input data signal into an infinite-length sequence and, as always, the result will depend on the method we choose: finite support or periodization. (Note that the impulse response is already an infinite sequence since it's the response of the filter to the infinite sequence $\delta[n]$).

However, the embedding will create "artificial" data points that are dependent on the chosen embedding: these data points are said to suffer from **border effects**.

Let's build a simple signal and a simple FIR filter:

```python
# let's use a simple moving average:
M = 5
h = np.ones(M)/float(M)

# let's build a signal with a ramp and a plateau
x = np.concatenate((np.arange(1, 9), np.ones(5) * 8, np.arange(8,0,-1)))
plt.stem(x);
print('signal length: {}'.format(len(x)))
```

### 1) No border effects

We may choose to accept the loss of data points and use only the $N-M+1$ output samples that correspond to a full overlap between the input data and the impulse response. This can be achieved by selecting `mode='valid'` in `correlate`:

```python
y = np.convolve(x, h, mode='valid')
print('signal length: {}'.format(len(y)))
plt.stem(y);
```

### 2) finite-support extension

By embedding the input into a finite-support signal, the convolution sum is now well defined for all values of $n$, which now creates a new problem: the output will be nonzero for all values of $n$ for which $x[n-k]$ is nonzero, that is for $0 \le n \le N+M-1$: we end up with a *longer* support for the output sequence. This is the default in `correlate`  and corresponds to `mode='full'`:

```python
y = np.convolve(x, h, mode='full')
print('signal length: {}'.format(len(y)))
plt.stem(y);
```

If we want to preserve the same length for input and output, we need to truncate the result. You can keep the *first* $N$ samples and discard the tail; this corresponds to the online implementation of the FIR filter. Alternatively, you can discard half the extra samples from the beginning and half from the end of the output and distribute the border effect evenly; this is achieved in `correlate`  by setting `mode='same'`:

```python
y = np.convolve(x, h, mode='same')
print('signal length: {}'.format(len(y)))
plt.stem(y);
```

### 3) Periodic extension

As we know, the other way of embedding a finite-length signal is to build a periodic extension. The convolution in this case will return an $N$-periodic output:

$$
    \tilde{y}[n] = \sum_{k=0}^{M-1} h[k]\tilde{x}[n-k]
$$

We can easily implement a circular convolution using `convolve` like so: since the overlap between time-reversed impulse response and input is already good for the last $N-M$ points in the output, we just need to consider two periods of the input to compute the first $M$:

```python
def cconv(x, h):
    # as before, we assume len(h) < len(x)
    L = len(x)
    xp = np.concatenate((x,x))
    # full convolution
    y = np.convolve(xp, h)
    return y[L:2*L]
```

```python
y = cconv(x, h)
print('signal length: {}'.format(len(y)))
plt.stem(y);
```

OK, clearly the result is not necessarily what we expected; note however that in both circular and "normal" convolution, you still have $M-1$ output samples "touched" by border effects, it's just that the border effects act differently in the two cases.

Interestingly, you can still obtain a "normal" convolution using a circular convolution if you zero-pad the input signal with $M-1$ zeros:

```python
y = cconv(np.concatenate((x, np.zeros(M-1))), h)
print('signal length: {}'.format(len(y)))
plt.stem(y);
# plot in red the difference with the standard conv
plt.stem(y - np.convolve(x, h, mode='full'), markerfmt='ro');
```

Why is this interesting? Because of the DFT....

## Offline implementations using the DFT

The convolution theorem states that, for infinite sequences,

$$
    (x\ast y)[n] = \mbox{IDTFT}\{X(e^{j\omega})Y(e^{j\omega})\}[n]
$$

Can we apply this result to the finite-length case? In other words, what is the inverse DFT of the product of two DFTs? Let's see:

\begin{align}
    \sum_{k=0}^{N-1}X[k]Y[k]e^{j\frac{2\pi}{N}nk} &= \sum_{k=0}^{N-1}\sum_{p=0}^{N-1}x[p]e^{-j\frac{2\pi}{N}pk}\sum_{q=0}^{N-1}y[q]e^{-j\frac{2\pi}{N}qk} \,e^{j\frac{2\pi}{N}nk} \\
    &= \sum_{p=0}^{N-1}\sum_{q=0}^{N-1}x[p]y[q]\sum_{k=0}^{N-1}e^{j\frac{2\pi}{N}(n-p-q)k} \\
    &= N\sum_{p=0}^{N-1}x[p]y[(n-p) \mod N]
\end{align}

The results follows from the fact that $\sum_{k=0}^{N-1}e^{j\frac{2\pi}{N}(n-p-q)k}$ is nonzero only for $n-p-q$ multiple of $N$; as $p$ varies from $0$ to $N-1$, the corresponding value of $q$ between $0$ and $N$ that makes $n-p-q$ multiple of $N$ is $(n-p) \mod N$.

So the fundamental result is: **the inverse DFT of the product of two DFTs is the circular convolution of the underlying time-domain sequences!**

To apply this result to FIR filtering, the first step is to choose the space for the DFTs. In our case we have a finite-length data vector of length $N$ and a finite-support impulse response of length $M$ with $M<N$ so let's operate in $\mathbb{C}^N$ by zero-padding the impulse response to size $N$. Also, we most likely want the normal convolution, so let's zero-pad both signals by an additional $M-1$ samples

```python
def DFTconv(x, h, mode='full'):
    # we want the compute the full convolution
    N = len(x)
    M = len(h)
    X = np.fft.fft(x, n=N+M-1)
    H = np.fft.fft(h, n=N+M-1)
    # we're using real-valued signals, so drop the imaginary part
    y = np.real(np.fft.ifft(X * H))
    if mode == 'valid':
        # only N-M+1 points, starting at M-1
        return y[M-1:N]
    elif mode == 'same':
        return y[int((M-1)/2):int((M-1)/2+N)]
    else:
        return y
```

Let's verify that the results are the same

```python
y = np.convolve(x, h, mode='valid')
print('signal length: {}'.format(len(y)))
plt.stem(y);
y = DFTconv(x, h, mode='valid')
print('signal length: {}'.format(len(y)))
plt.stem(y, markerfmt='ro');
```

```python
y = np.convolve(x, h, mode='same')
print('signal length: {}'.format(len(y)))
plt.stem(y);
y = DFTconv(x, h, mode='same')
print('signal length: {}'.format(len(y)))
plt.stem(y, markerfmt='ro');
```

Of course the question at this point is: why go through the trouble of taking DFTs if all we want is the standard convolution? The answer is: **computational efficiency.**

If you look at the convolution sum, each output sample requires $M$ multiplications (and $M-1$ additions but let's just consider multiplications). In order to filter an $N$-point signal we will need $NM$ multiplications. Assume $N \approx M$ and you can see that the computational requirements are on the order of $M^2$. If we go the DFT route using an efficient FFT implementation we have approximately:

 * $M\log_2 M$ multiplication to compute $H[k]$
 * $M\log_2 M$ multiplication to compute $X[k]$
 * $M\log_2 M$ multiplication to compute $X[k]H[k]$
 * $M\log_2 M$ multiplication to compute the inverse DFT

Even considering that we now have to use complex multiplications (which will cost twice as much), we can estimate the cost of the DFT based convolution at around $8M\log_2M$, which is smaller than $M^2$ as soon as $M>44$.

In practice, the data vector is much longer than the impulse response so that filtering via standard convolution requires on the order of $MN$ operations. Two techniques, called [Overlap Add](https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method) and [Overlap Save](https://en.wikipedia.org/wiki/Overlap%E2%80%93save_method)
can be used to divide the convolution into $N/M$ independent convolutions between $h[n]$ and an $M$-sized piece of $x[n]$; FFT-based convolution can then be used on each piece. While the exact cost per sample of each technique is a bit complicated to estimate, as a rule of thumb **as soon as the impulse response is longer than 50 samples, it's more convenient to use DFT-based filtering.**

```python

```


# Assignment 1
Autor: Dimitri Muralt

```python
import sys
import nltk
import math
import itertools
import scipy.stats
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
from scipy.integrate import quad
import sympy
```

## Daten einlesen

```python
people_en = pd.read_csv('people_wiki_EN.csv', sep=',')
people_de = pd.read_csv('10k-people-raw.csv', sep=';')

char_array = [chr(c) for c in range(ord('a'), ord('z')+1)]
char_dict = {char_array[index]:index for index in range(0,26)}
```

## Hilfsfunktionen

```python
def box_plot(dist, title = None):
    chars = [chr(c) for c in range(ord('a'), ord('z') + 1)]
    print(chars)

    fig1, ax1 = plt.subplots(figsize = (18, 8))
    ax1.set_xlabel('Character')
    ax1.set_ylabel('Frequency')
    ax1.boxplot(dist, labels = chars)
    ax1.set_ylim([0, 0.3])
    if title is not None:
        plt.title(title)
    plt.show()
```

```python
def frequency_plot(mean_en, mean_de):
    chars = [chr(c) for c in range(ord('a'), ord('z')+1)]
    ind = np.arange(len(chars))
    width = 1
    frequency = np.log((np.array(mean_en) / np.array(mean_de)))
    color = ['g' if f > 0 else 'b' for f in frequency]

    fig, ax = plt.subplots(figsize = (18, 8))
    rects1 = ax.bar(ind, frequency, width, color = color, edgecolor = 'black')
    ax.set_xticklabels(chars)
    ax.set_xticks(ind + (width * 0.5) - 0.5)
    ax.set_xlabel('Character')
    ax.set_ylim([-2, 2])
    ax.set_ylabel('F')
    ax.set_xlim([-0.5, len(chars) - 0.5])
    ax.text(1, 1.5, 'More frequent in English ', bbox={'facecolor':'green', 'alpha':0.5, 'pad':10})
    ax.text(1, 1.2, 'More frequent in German', bbox={'facecolor':'blue', 'alpha':0.5, 'pad':10})
    ax.text(5.5, 1.35, r'$F = \ln(\frac{mean_e}{mean_d})$', fontsize=25)
    plt.show()
```

```python
def plot_confusion_matrix(cm, classes, title = 'Confusion matrix', cmap = plt.cm.Greens):
    plt.title(title)
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation = 45)
    plt.yticks(tick_marks, classes)

    cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
    cm_norm[np.isnan(cm_norm)] = 0
    cm_norm = np.around(cm_norm, decimals = 3)

    plt.imshow(cm_norm, interpolation='nearest', cmap = cmap)

    thresh = cm_norm.max() / 2.
    for i, j in itertools.product(range(cm_norm.shape[0]), range(cm_norm.shape[1])):
        plt.text(j, i, cm[i, j], horizontalalignment="center", color="white" if cm_norm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.show()
```

## Häufigkeitsverteilungen

```python
def character_distribution(people):
    # Implementieren Sie eine Funktion die die Buchstabenhäufigkeiten berechnet.
    # Die Funktion soll folgendes zurückgeben:
    #  mean: Durchschnittlicher relative Häufigkeit pro Buchstabe
    #  std: Relative Standardabweichung pro Buchstabe
    #  dist: Relative Häufigkeit pro Text und pro Buchstabe

    chars = [chr(c) for c in range(ord('a'), ord('z') + 1)]
    dist = np.zeros(shape = (len(people), 26))

    for item_index, item in enumerate(people):
        fdist = nltk.FreqDist(char.lower() for char in item if char.isalpha())

        for char_index, char in enumerate(chars):
            dist[item_index, char_index] = fdist[char]

        dist[item_index,:] = dist[item_index,:]/sum(dist[item_index])

    mean = np.mean(dist,axis=0)
    std = np.std(dist,axis=0)

    return mean, std, dist
```

```python
mean_de, std_de, dist_de = character_distribution(people_de['text'])
box_plot(dist_de, 'German')
```

```python
mean_en, std_en, dist_en = character_distribution(people_en['text'])
box_plot(dist_en, 'English')
```

```python
frequency_plot(mean_en, mean_de)
```

## Spracherkennung

### Auswahl der Buchstaben für den Sprachdetektor

```python
# integrand() Funktion übernommen von Aufgabe Übungsserie 2, Aufgabe 13
# entspricht der Fläche unterhalb des Produktes der beiden Dichtefunktionen
# je kleiner die Fläche, desto weniger überschneiden sich die Dichtefunktionen
def integrand(x):
    return 1/std_de[i]/std_en[i]*sympy.exp(-(x-mean_de[i])**2/2/std_de[i]**2-(x-mean_en[i])**2/2/std_en[i]**2)
```

```python
# für jeden Buchstaben die Fläche der Überscheidung berechnen
charOverlapArea = {}
for i in np.arange(0,26):
     charOverlapArea[char_array[i]] = quad(integrand,0,1)[0]

charOverlapAreaSorted = sorted((value, key) for (key,value) in charOverlapArea.items())
# sechs Buchstaben mit der tiefsten Fläche eignen sich potenziell ab besten für den Sprachdetektor
print("Chars with lowest overlap area:", "\n", charOverlapAreaSorted[0:6], "\n")
```

    Chars with lowest overlap area:
     [(0.9945622527746305, 'o'), (3.5069789980395134, 'e'), (34.0850941940391, 'a'), (38.70627766315792, 'z'), (59.61832993894152, 'y'), (85.74923006903875, 't')]

Nachfolgend werden für die sechs Buchstaben mit der kleinsten Fläche der Überschneidung weitere Metriken berechnet um deren Eignung für den Sprachdetektor zu genauer zu prüfen.

```python
for i in [char_dict['o'],char_dict['e'],char_dict['a'],char_dict['z'],char_dict['y'],char_dict['t']]:
    #plot
    avgMean = np.mean([mean_de[i],mean_en[i]])
    plt.plot([mean_de[i]-std_de[i]*1.5,  mean_de[i]+std_de[i]*1.5], [char_array[i],char_array[i]], 'bo--',
             [mean_en[i]-std_en[i]*1.5,  mean_en[i]+std_en[i]*1.5], [char_array[i],char_array[i]], 'go--',
             [avgMean], [char_array[i]], 'rs',)

    #print statistics
    if mean_de[i] < mean_en[i]:
        cdf_german = round(scipy.stats.norm.cdf(avgMean, mean_de[i], std_de[i]),4)
        cdf_english = round(1 - scipy.stats.norm.cdf(avgMean, mean_en[i], std_en[i]),4)
        print(char_array[i], '|',
              round(cdf_german*100,1), "% in scope",
              "| avgOfTwoMeans: ",     round(avgMean,6), "|",
              round(cdf_english*100,1), "% in scope")
    else:
        cdf_german = round(1 - scipy.stats.norm.cdf(avgMean, mean_de[i], std_de[i]),4)
        cdf_english = round(scipy.stats.norm.cdf(avgMean, mean_en[i], std_en[i]),4)
        print(char_array[i], '|',
              round(cdf_english*100,1), "% in scope",
              "| avgOfTwoMeans: ",     round(avgMean,6), "|",
              round(cdf_german*100,1), "% in scope")

print()
print("Blue:   German  chars within 1.5 sdt")
print("Green:  English chars within 1.5 sdt")
print("Red:    Average of the two means")
plt.show()
```

Der nachfolgende Sprachdetektor benötigt vier Buchstaben. Gemäss den Erläuterungen oben eignen sich die Buchstaben o, e, a, z, y, z (in dieser Reihenfolge) am besten für den Sprachdetektor.
Die Buchstaben x und y überschneiden sich zwar wenig, allerdigs kommen sich relativ selten vor, so dass wahrscheinlicher Abweichugen in kürzeren Texten auftreten können. Folglich wurden die Buchstaben x und y nicht für den Sprachdetektor verwendet.

Ausgewählte Buchstaben: *o, e, a, t*

### Klasse CharStats

Für die vier ausgewählte Buchstaben wird je eine Instanz der Klasse CharStats erstellt. Die Klasse CharStats enthält die relevanten Metriken eines Buchstabens. So könen die Metriken später einfach zugezogen werden.

```python
class CharStats():

    def __init__(self, char, char_index,  avg_mean, lang_higher_mean, upper_boundary):
        self.char             = char
        self.char_index       = char_index
        self.avg_mean         = avg_mean
        self.lang_higher_mean = lang_higher_mean
        self.upper_boundary   = upper_boundary

    def __str__(self):
        return "Char: %s | Average of two means: %f | Language higher mean: %s | " \
               "Upper boundary: %f | Index: %d" % \
            (self.char,
             self.avg_mean,
             self.lang_higher_mean,
             self.upper_boundary,
             self.char_index)
```

```python
char_stats = []

for i in [char_dict['o'],char_dict['e'],char_dict['a'],char_dict['t']]:

    # calculate relavant statistics for a char
    avg_mean = np.mean([mean_de[i],mean_en[i]])
    lang_higher_mean = 'G' if mean_de[i] > mean_en[i] else 'E'

    sizeSdt = 3 # over 99% of data are within 3 std
    if lang_higher_mean == 'G':
        upper_boundary = mean_de[i] + sizeSdt*std_de[i]
    else:
        upper_boundary = mean_en[i] + sizeSdt*std_en[i]

    # create instance CharStats
    current_char = CharStats(char_array[i], i, avg_mean, lang_higher_mean, upper_boundary)
    char_stats.append(current_char)

for c in char_stats:
    print(c)
```

    Char: o | Average of two means: 0.053039 | Language higher mean: E | Upper boundary: 0.102237 | Index: 14
    Char: e | Average of two means: 0.137985 | Language higher mean: G | Upper boundary: 0.191752 | Index: 4
    Char: a | Average of two means: 0.078075 | Language higher mean: E | Upper boundary: 0.126781 | Index: 0
    Char: t | Average of two means: 0.069298 | Language higher mean: E | Upper boundary: 0.106445 | Index: 19

### Erstellung Entscheidungsbaum

Klasse Node: Ein Node hat zwei Äste (left, right). Zusätzlich weiss ein Node auf welchem Level im Baum er liegt und enthält einen Buchstaben für die Entscheidung, ob die Sprache eher Deutsch oder Englisch ist. Der lang_code wird im nachfolgenden Snippet erklärt.

```python
class Node(object):

    def __init__(self,lang_code, level, char_stats):
        self.left = None
        self.right = None
        self.lang_code = lang_code
        self.level = level
        self.char_stats = char_stats

        if   lang_code > 1:   self.lang = 'E'
        elif lang_code < -1:  self.lang = 'G'
        else:                 self.lang = 'X'

    def __str__(self):
        return str(self.lang_code)
    def insert_left(self,node):
        self.left = node
    def insert_right(self,node):
        self.right = node
    def is_leaf(self):
        if self.left is None and self.right is None:
            return True
        else:
            return False
```
Gemäss der Aufgabenstellung dürfen nur vier True/False Entscheidungen getroffen werden.
Dadurch entsteht ein vierstufiger Baum. Für jede Stufe je ein Buchstabe aus den vier auserkorenen ausgewählt.
Bei jedem Node wird die Entscheidung gefällt, ob die Buchstabenverteilung eher Deutsch oder Englisch ist.

Eine Sprache gilt als gefunden, wenn die Entscheidung zwei mal in Folge für die gleiche Sprache fällt oder
insgesamt drei von vier mal die gleiche Sprache ausgewählt wird. Ansonsten gilt die Sprache als nicht gefunden.(X)

Dadurch entsteht der folgende Baum mit vier Stufen und zehn Blättern.

                     0                      ------ o (Buchstabe für Vergleich)
             /                 \
            /                   \
           /                     \
          /                       \
        -1                         1        ------ e
      /    \                    /     \
     /      \                  /       \
   -2       0                 0        2    ------ a
         /    \            /    \
       -1      1         -1      1          ------ t
       / \    / \        / \    / \
     -2   0  0   2     -2   0  0   2
   |  |   |  |   |      |   |  |   |    |
   |  |   |  |   |      |   |  |   |    |
   G  G   X  X   E      G   X  X   E    E  (ausgewählte Sprache)

```python
def generate_tree(node):

    if 2 > node.lang_code > -2 and node.level < 5:

        char_for_decision = char_stats[node.level] if node.level < 4 else None

        left  = generate_tree(Node(node.lang_code-1, node.level+1, char_for_decision))
        node.insert_left(left)

        right = generate_tree(Node(node.lang_code+1, node.level+1, char_for_decision))
        node.insert_right(right)

    return node

root = Node(0,1,char_stats[0])
lang_tree = generate_tree(root)
```

```python
def test_tree(lang_tree):
    print("Höhe des Baums: ",height(lang_tree))
    print("Anzahl Blätter:", leaf(lang_tree))
    for i in range(1,7): # zu viel drucken um Baumgrenzen zu zeigen
        print_tree_level(lang_tree, i)
        print()

def height(node):
    if node.is_leaf():
        return 0
    else:
        return max(height(node.left), height(node.right)) + 1

def leaf(node):
    if node.is_leaf():
        return 1
    else:
        return leaf(node.left) + leaf(node.right)

# der Baum muss zur nachfollziehbaren Darstellung pro Level gedruckt werden
# Die üblichen rekursiven Baumdurchläufe "Preorder" und "Postorder" sind nicht gut lesbar
def print_tree_level(node, level):
    if not node:
        print("_", end=" ")
        return
    if (level == 1):
        print("[",node.lang_code,
              node.lang,
              node.char_stats.char if node.char_stats else "_",
              "]",
              end=" ")
    elif (level > 1):
        print_tree_level(node.left, level-1);
        print_tree_level(node.right, level-1);

test_tree(lang_tree)
```

    Höhe des Baums:  4
    Anzahl Blätter: 10
    [ 0 X o ]
    [ -1 X e ] [ 1 X e ]
    [ -2 G a ] [ 0 X a ] [ 0 X a ] [ 2 E a ]
    _ _ [ -1 X t ] [ 1 X t ] [ -1 X t ] [ 1 X t ] _ _
    _ _ [ -2 G _ ] [ 0 X _ ] [ 0 X _ ] [ 2 E _ ] [ -2 G _ ] [ 0 X _ ] [ 0 X _ ] [ 2 E _ ] _ _
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

### Sprache bestimmen

```python
def language_tree(mean_de, mean_en, texts):

    lang = []
    _, _, dist_test = character_distribution(texts)

    for index in range(len(texts)):
        #für jeden Text mit dem generierten Baum die Sprache suchen
        current_lang = get_lang(dist_test[index], lang_tree)
        lang.append(current_lang)

    return lang
```

```python
# get_lang durchläuft rekursiv den erstellten Suchbaum.
# Sobald bei einem Blatt angekommen, wird die Sprache zurückgegeben.

def get_lang(text_dist, node):
    if node.is_leaf():
        return node.lang

    # Nebst dem avg_mean wird die upper_boundary (=dreifache Standartabweichung, siehe Generierung char_stats)
    # für die Entscheidung der Sprache zugezogen um Aussreisser auszuschliessen.
    if node.char_stats.upper_boundary > text_dist[node.char_stats.char_index] > node.char_stats.avg_mean:

        # den Entscheidungsbaum der Sprache entsprechend absteigen
        if node.char_stats.lang_higher_mean == 'G':
            return get_lang(text_dist, node.left)
        else:
            return get_lang(text_dist, node.right)

    else:
        if node.char_stats.lang_higher_mean == 'G':
            return get_lang(text_dist, node.right)
        else:
            return get_lang(text_dist, node.left)
```

```python
def accuracy(cm):
    i = np.eye(cm.shape[0])
    return (i * cm).sum() / cm.sum()
```

```python
people_test = pd.read_csv('Language_test.csv', sep=';', quotechar='|')
lang_true = np.array(people_test['language'])
```

```python
lang = language_tree(mean_de, mean_en, people_test['text'])
cm = confusion_matrix(lang_true, lang)
print('Accuracy:', accuracy(cm))
```

    Accuracy: 0.9901477832512315

```python
plot_confusion_matrix(cm, ['English', 'German', 'X'])
```

### Wertung

Mit dem erstellten Baum kann English und Deutsch sehr gut unterschieden werden. Die gut überlegte Auswahl der Buchstaben, sowie Erstellung des Baums zeigen am meisten Wirkung.
Ein Schwachpunkt scheint die Indentifizierung von anderen Sprachen zu sein. Die zwei falsch zugewiesene Sprachen sind unten noch detailliert festgehalten.

Hierzu wurde nochmals der weiter oben verwendete Plot genutzt (neu mit drei Standartabweichungen). Auf dem Plot ist neu mit dem schwarzen Dreieck der Wert des jeweiligen Texts in Relation zu den vorhandenen Verteilungen in den beiden Sprachen zu sehen.

In beiden Fällen entprichen die ersten beiden Vergleiche (o und a) des Textes recht genau der Englischen Sprache(grün).

Um diese Beiden Sprachen korrekt zuteilen zu können, könnte man beispielsweise andere Buchstaben verwenden oder mehr Abfragen durchführen, bevor eine Sprache festgelegt wird.

```python
_, _, dist_test = character_distribution(people_test['text'])

for index in range(len(people_test)):

    if lang_true[index] != lang[index]:
        desc = "True: " + lang_true[index] + " Found: "+lang[index] + " ..."+people_test['text'][index][:30]+"..."

        plt.figure()
        for i in [char_dict['o'],char_dict['e'],char_dict['a'],char_dict['t']]:
            plt.title(desc)
            avgMean = np.mean([mean_de[i],mean_en[i]])
            plt.plot([mean_de[i]-std_de[i]*3,  mean_de[i]+std_de[i]*3], [char_array[i],char_array[i]], 'bo--',
                     [mean_en[i]-std_en[i]*3,  mean_en[i]+std_en[i]*3], [char_array[i],char_array[i]], 'go--',
                     [avgMean], [char_array[i]], 'rs',
                      dist_test[index][i],[char_array[i]], 'k^')
        plt.show

```


# A Review of Yesterday's Content:

- Neural Networks are made up of layers of nodes or neurons.
- There are three kinds of layers: input, hidden, and output layers.

- Each layer is made up of individual neurons which have a corresponding weight and bias.

- Signal is passed from layer to layer through a network by:
 - Taking in inputs from the training data (or previous layer)
 - Multiplying each input by its corresponding weight (think arrow/connecting line)
 - Adding a bias to this weighted sum of inputs and weights
 - Activating this weighted sum + bias by squishifying it with sigmoid or some other activation function. With a single perceptron with three inputs, calculating the output from the node is done like so:
\begin{align}
 y = sigmoid(\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)
\end{align}
 - this final activated value is the signal that gets passed onto the next layer of the network.

## Multilayer Perceptrons AKA Feedforward Neural Networks

Yesterday we talked about the capabilities and limitations of individual perceptrons.

- Perceptrons are like a single neuron or layer of neurons in a neural network
- Perceptrons can only fit linear boundaries between classes

What is a Feedforward Neural Network? Well, it's a Neural Network made up of multiple perceptrons that has at least 1 hidden layer (a layerbetween the input and output layers). When we use the [Tensorflow Playground](https://playground.tensorflow.org) website to explore different neural network architectures, we're really just composing different architectures (topologies) of feedforward neural networks. However, when we give the network multiple hidden layers it should be designated as a "deep" feedforward neural network.

### Deep Feedforward Neural Network

# How do Neural Networks Learn?

## Calculating *"loss"* or *"error"*

We've talked about how in order to evaluate a network's performance, the data is "fed forward" until predictions are obtained and then the "loss" or "error" for a given observation is ascertained by looking at what the network predicted for that observation and comparing it to what it *should* have predicted.

The error for a given observation is calculated by taking the square of the difference between the predicted value and the actual value.

We can summarize the overal quality of a network's predictions by finding the average error across all observations. This gives us the "Mean Squared Error." which hopefully is a fairly familiar model evaluation metric by now. Graphing the MSE over each epoch (training cycle) is a common practice with Neural Networks. This is what you're seeing in the top right corner of the Tensorflow Playground website as the number of "epochs" climbs higher and higher.

We also graphed loss over a number of iterations when working with gradient descent. Neural Networks use gradient descent to find their optimal parameters.

## What is an "Epoch"?

An "Epoch" is one cycle of:
* passing our data forward through the network
* measuring error given our specified cost function
* updating weights within our network (via gradient descent) to hopefully improve the quality of our predictions on the next iteration

### Batch vs Minibatch vs Stochastic Gradient Descent Epochs

You may have heard these variations on the training process referenced in the 3Blue1Brown videos about backpropagation. "Minibatch" Gradient Descent means that instead of passing all of our data through the network for a given epoch (Batch GD), we just pass a randomized portion of our data through the network for each epoch.

**Stochastic Gradient Descent** is when we make updates to our weights after forward propagating each individual training observation.

### Why might we use something like Minibatch GD?

Neural Networks often require high quantities of training data. Consider the [MNIST Dataset](http://yann.lecun.com/exdb/mnist/):

This dataset is what has been used in the 3Blue1Brown videos for demonstrating important neural network concepts. The dataset has 60,000 training observations and 10,000 test observations. SInce there are 10 numeric digits that works out to be 6,000 training images per digit. Do you think our neural network needs to look at all 6000 number 3s in order to get an idea of how accurate or innacurate it is at recognizing the number 3? No, it doesn't. It's actually pretty computationally expensive to run the entire dataset through the network for each epoch, so we'll just randomly select a portion of the dataset for each training epoch.

How much of the data should you pass through for each epoch? Well, you'll probably get sick of hearing this response to questions like this, but `batch_size` is another hyperparameter for our model that we'll need to tune to try and get the best results. We'll try different amounts out and try to find an optimal "batch size."

### A note about Hyperparameters

Neural Networks have many more hyperparameters than other machine learning algorithms which is part of what makes them a beast to train. 1) You need more data to train them on. 2) They're complex so they take longer to train. 3) They have lots and lots of hyperparameters which we need to find the most optimal combination of, so we might end up training our model dozens or hundreds of times with different combinations of hyperparameters in order to try and squeeze out a few more tenths of a percent of accuracy.

# Backpropagation

Backpropagation is short for ["Backwards Propagation of errors"](https://en.wikipedia.org/wiki/Backpropagation) and refers to a specific (rather calculus intensive) algorithm for how weights in a neural network are updated in reverse order at the end of each training epoch. Our purpose today is to demonstrate the backpropagation algorithm on a simple Feedforward Neural Network and in so doing help you get a grasp on the main process. If you want to understand all of the underlying calculus of how the gradients are calculated then you'll need to dive into it yourself, [3Blue1Brown's video is a great starting place](https://www.youtube.com/watch?v=tIeHLnjs5U8). Watch this Welch Labs series [Neural Networks Demystified](https://www.youtube.com/watch?v=bxe2T-V8XRs) if you want a rapid yet orderly walkthrough of the main intuitions and math behind the backpropagation algorithm.

## FeedForward NN for predicting Test Scores

### Generate some Fake Data

```python
import numpy as np
np.random.seed(1)

# hours studying, hours sleeping
X = np.array([
    [2, 9],
    [1, 5],
    [3, 6]
], dtype=float)

# test scores
y = np.array([
    [92],
    [86],
    [89]
], dtype=float)
```

```python
# standardize by feature (divide by max to get relative to maximum)
X = X / np.amax(X, axis=0)
y = y / 100
```

```python
print("Studying, Sleeping\n", X)
print("Test Score\n", y)
```

    Studying, Sleeping
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Test Score
     [[0.92]
     [0.86]
     [0.89]]

### Lets create a Neural_Network class to contain this functionality
* How many random weights do we need to initialize? "Fully-connected Layers"
* Implement Feedforward Functionality
  * After this step our neural network should be able to generate an output even though it has not been trained.

```python
class NeuralNetwork(object):
    def __init__(self):
        self.inputs = 2
        self.hiddenNodes = 3
        self.outputNodes = 1
        # weights for each layer
        self.weights1 = np.random.randn(self.inputs, self.hiddenNodes) # 2x3
        self.weights2 = np.random.randn(self.hiddenNodes, self.outputNodes) # 3x1

    def sigmoid(self, s):
        return 1 / (1 + np.exp(-s))

    def feed_forward(self, X):
        """Calculate the NN inference using feed forwad."""
        self.hidden_sum = np.dot(X, self.weights1) # 3x3
        self.activated_hidden = self.sigmoid(self.hidden_sum)
        self.output_sum = np.dot(self.activated_hidden, self.weights2)
        self.activated_output = self.sigmoid(self.output_sum)
        return self.activated_output
```

```python
nn = NeuralNetwork()
print("Layer 1 weights: \n", nn.weights1)
print("Layer 2 weights: \n", nn.weights2)
```

    Layer 1 weights:
     [[ 1.62434536 -0.61175641 -0.52817175]
     [-1.07296862  0.86540763 -2.3015387 ]]
    Layer 2 weights:
     [[ 1.74481176]
     [-0.7612069 ]
     [ 0.3190391 ]]

### Can we generate an output?

```python
print(X[0])
output = nn.feed_forward(X[0])
print('output ', output)
```

    [0.66666667 1.        ]
    output  [0.60623806]

### Calculating Error

```python
output_all = nn.feed_forward(X)
error_all = y - output_all
print(output_all, error_all, sep='\n')
```

    [[0.60623806]
     [0.6168044 ]
     [0.71212205]]
    [[0.31376194]
     [0.2431956 ]
     [0.17787795]]

## Backpropagation

AKA assigning blame for bad predictions and delivering justice - repeatedly and a little bit at a time.

What in our model could be causing our predictions to suck so bad?

Well, we know that our inputs (X) and outputs (y) are correct, if they weren't then we would have bigger problems than understanding backpropagation.

We also know that our activation function (sigmoid) is working correctly. It can't be blamed because it just does whatever we tell it to and transforms the data in a known way.

So what are the potential culprits for these terrible predictions? The **weights** of our model. Here's the problem though. I have weights that exist in both layers of my model. How do I know if the weights in the first layer are to blame, or the second layer, or both?

Lets investigate. And see if we can just eyeball what should be updated.

```python
# Look at how a single observation is processed
print("Inputs", X[0],
      "Initial Weights", nn.weights1,
      "Weighted Sum 1", nn.hidden_sum,
      "Activations from first layer", nn.activated_hidden,
      "Second Layer Weights", nn.weights2,
      "Weighted Sum 2", nn.output_sum,
      "Predictions", output,
      "Correct Output", y[0],
      "Error", y[0]-output,
      sep="\n")
```

    Inputs
    [0.66666667 1.        ]
    Initial Weights
    [[ 1.62434536 -0.61175641 -0.52817175]
     [-1.07296862  0.86540763 -2.3015387 ]]
    Weighted Sum 1
    [[ 0.00992829  0.45757002 -2.6536532 ]
     [-0.05464522  0.27686321 -1.45468986]
     [ 0.90903295 -0.03481799 -2.06253088]]
    Activations from first layer
    [[0.50248205 0.61243756 0.0657642 ]
     [0.48634209 0.56877703 0.18928084]
     [0.71280223 0.49129638 0.11279232]]
    Second Layer Weights
    [[ 1.74481176]
     [-0.7612069 ]
     [ 0.3190391 ]]
    Weighted Sum 2
    [[0.43152625]
     [0.47600639]
     [0.90571269]]
    Predictions
    [0.60623806]
    Correct Output
    [0.92]
    Error
    [0.31376194]

Why is my error so big?

My error is so big because my prediction is low.

Why are my prediction low?

Because either:

  1. Second layer **weights** are low
  2. OR Activations coming from the first layer are low

How are activations from the first layer determined?

  1. By inputs - fixed
  * by **weights** - variable

The only thing that I have control over throughout this process in order to increase the value of my final predictions is to either increase weights in layer 2 or increase weights in layer 1.

Imagine that you could only change your weights by a fixed amount. Say you have .3 and you have to split that up and disperse it over your weights so as to increase your predictions as much as possible. (This isn't actually what happens, but it will help us identify which weights we would benefit the most from moving.)

I need to increase weights of my model somewhere, I'll get the biggest bang for my buck if I increase weights in places where I'm already seeing high activation values -because they end up getting multiplied together before being passed to the sigmoid function.

> "Neurons that fire together, wire together"

```python
# I want activations that correspond to negative weights to be lower
# and activations that correspond to positive weights to be higher
```

## Diagnosing Backpropagation issues by hand. (not recommended)

Our model has 9 total weights (6 in the first layer, 3 in the last layer) that could be off.

1. Calculate Error for a given each observation
* Does the error indicate that I'm overestimating or underestimating in my prediction?
* Look at final layer weights to get an idea for which weights are helping pass desireable signals and which are stifling desireable signals
* Also go to the previous layer and see what can be done to boost activations that are associated with helpful weights, and limit activations that are associated with unhelpful weights.

### However, we would prefer to do all this in a fancy automated, controlled calculus way.

5. Repeat steps 1-4 for every observation in a given batch, and then given the network's cost function, calculate its gradient using calculus and update weights associated with the (negative) gradient of the cost function.

Remember that we have 9 weights in our network therefore the gradient that comes from our gradient descent calculation will be the vector that takes us in the most downward direction along some function in 9-dimensional hyperspace.

\begin{align}
C(w1, w2, w3, w4, w5, w6, w7, w8, w9)
\end{align}

You should also know that with neural networks it is common to have gradients that are not convex (like what we saw when we applied gradient descent to linear regression). Due to the high complexity of these models and their nonlinearity, it is common for gradient descent to get stuck in a local minimum, but there are ways to combat this:

1) Stochastic Gradient Descent

2) More advanced Gradient-Descent-based "Optimizers" - See Stretch Goals on assignment.

## Adding BackPropogation to our implementation (non Gradient-Descent)

```python
class Neural_Network(object):
    def __init__(self):
        self.inputs = 2
        self.hiddenNodes = 3
        self.outputNodes = 1

        # Initlize Weights
        self.L1_weights = np.random.randn(self.inputs, self.hiddenNodes) # (3x2)
        self.L2_weights = np.random.randn(self.hiddenNodes, self.outputNodes) # (3x1)

    def feed_forward(self, X):
        # Weighted sum between inputs and hidden layer
        self.hidden_sum = np.dot(X, self.L1_weights)
        # Activations of weighted sum
        self.activated_hidden = self.sigmoid(self.hidden_sum)
        # Weighted sum between hidden and output
        self.output_sum = np.dot(self.activated_hidden, self.L2_weights)
        # final activation of output
        self.activated_output = self.sigmoid(self.output_sum)
        return self.activated_output

    def sigmoid(self, s):
        return 1/(1+np.exp(-s))

    def sigmoidPrime(self, s):
        """derivative of sigmoid"""
        return s * (1 - s)

    def backward(self, X, y, o):
        """backward propgate through the network"""
        self.o_error = y - o # error in output
        self.o_delta = self.o_error * self.sigmoidPrime(o) # applying derivative of sigmoid to error

        self.z2_error = self.o_delta.dot(self.L2_weights.T) # z2 error: how much our hidden layer weights contributed to output error
        self.z2_delta = self.z2_error * self.sigmoidPrime(self.activated_hidden) # applying derivative of sigmoid to z2 error

        self.L1_weights += X.T.dot(self.z2_delta) # adjusting first set (input --> hidden) weights
        self.L2_weights += self.activated_hidden.T.dot(self.o_delta) # adjusting second set (hidden --> output) weights

    def train(self, X, y):
        o = self.feed_forward(X)
        self.backward(X, y, o)
```

```python
NN = Neural_Network()
for i in range(1000): # trains the NN 1,000 times
    if i+1 in [1,2,3,4,5] or (i+1) % 50 == 0:
        print('+---------- EPOCH', i+1, '-----------+')
        print("Input: \n", X)
        print("Actual Output: \n", y)
        print("Predicted Output: \n" + str(NN.feed_forward(X)))
        print("Loss: \n" + str(np.mean(np.square(y - NN.feed_forward(X))))) # mean sum squared loss
        print("\n")
    NN.train(X, y)
```

    +---------- EPOCH 1 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.52433559]
     [0.51924096]
     [0.51502029]]
    Loss:
    0.13775894381073592

    +---------- EPOCH 2 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.61638089]
     [0.60060247]
     [0.60029617]]
    Loss:
    0.08113331898809754

    +---------- EPOCH 3 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.67975295]
     [0.65752584]
     [0.66008717]]
    Loss:
    0.05052477938399689

    +---------- EPOCH 4 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.72298247]
     [0.69707787]
     [0.70161669]]
    Loss:
    0.03361593234908083

    +---------- EPOCH 5 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.75354918]
     [0.72552618]
     [0.73144002]]
    Loss:
    0.02364344984289771

    +---------- EPOCH 50 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.89817372]
     [0.8693767 ]
     [0.88023095]]
    Loss:
    0.00021991447262439465

    +---------- EPOCH 100 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90403648]
     [0.87561831]
     [0.88678081]]
    Loss:
    0.00016970965466925458

    +---------- EPOCH 150 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90464459]
     [0.8761624 ]
     [0.88760286]]
    Loss:
    0.00016758601956212707

    +---------- EPOCH 200 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90470047]
     [0.87610621]
     [0.88781732]]
    Loss:
    0.00016608320206239755

    +---------- EPOCH 250 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.9046928 ]
     [0.87598343]
     [0.88795802]]
    Loss:
    0.00016465003512956114

    +---------- EPOCH 300 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90467833]
     [0.87585606]
     [0.88808705]]
    Loss:
    0.00016327587429809022

    +---------- EPOCH 350 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.9046637 ]
     [0.87573126]
     [0.88821179]]
    Loss:
    0.00016195754331315595

    +---------- EPOCH 400 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90464966]
     [0.87560978]
     [0.88833321]]
    Loss:
    0.00016069214593036286

    +---------- EPOCH 450 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90463627]
     [0.87549157]
     [0.88845152]]
    Loss:
    0.00015947696134234877

    +---------- EPOCH 500 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90462351]
     [0.87537654]
     [0.88856682]]
    Loss:
    0.0001583094313983074

    +---------- EPOCH 550 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90461136]
     [0.87526456]
     [0.88867921]]
    Loss:
    0.00015718714930283785

    +---------- EPOCH 600 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90459979]
     [0.87515553]
     [0.88878879]]
    Loss:
    0.0001561078491550748

    +---------- EPOCH 650 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90458878]
     [0.87504935]
     [0.88889563]]
    Loss:
    0.00015506939631380884

    +---------- EPOCH 700 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.9045783 ]
     [0.87494591]
     [0.88899983]]
    Loss:
    0.0001540697785221398

    +---------- EPOCH 750 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90456833]
     [0.87484512]
     [0.88910146]]
    Loss:
    0.0001531070977244867

    +---------- EPOCH 800 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90455885]
     [0.87474688]
     [0.8892006 ]]
    Loss:
    0.00015217956251462378

    +---------- EPOCH 850 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90454985]
     [0.87465113]
     [0.88929732]]
    Loss:
    0.00015128548115943705

    +---------- EPOCH 900 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.9045413 ]
     [0.87455776]
     [0.8893917 ]]
    Loss:
    0.00015042325514852734

    +---------- EPOCH 950 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.90453319]
     [0.8744667 ]
     [0.8894838 ]]
    Loss:
    0.00014959137322461912

    +---------- EPOCH 1000 -----------+
    Input:
     [[0.66666667 1.        ]
     [0.33333333 0.55555556]
     [1.         0.66666667]]
    Actual Output:
     [[0.92]
     [0.86]
     [0.89]]
    Predicted Output:
    [[0.9045255 ]
     [0.87437787]
     [0.88957368]]
    Loss:
    0.00014878840585406128

## A true GD-based implementation from [Welch Labs on YouTube](https://www.youtube.com/watch?v=bxe2T-V8XRs)

```python
class Neural_Network(object):
    def __init__(self):
        #Define Hyperparameters
        self.inputLayerSize = 2
        self.outputLayerSize = 1
        self.hiddenLayerSize = 3

        #Weights (parameters)
        self.W1 = np.random.randn(self.inputLayerSize,self.hiddenLayerSize)
        self.W2 = np.random.randn(self.hiddenLayerSize,self.outputLayerSize)

    def forward(self, X):
        #Propogate inputs though network
        self.z2 = np.dot(X, self.W1)
        self.a2 = self.sigmoid(self.z2)
        self.z3 = np.dot(self.a2, self.W2)
        yHat = self.sigmoid(self.z3)
        return yHat

    def sigmoid(self, z):
        #Apply sigmoid activation function to scalar, vector, or matrix
        return 1/(1+np.exp(-z))

    def sigmoidPrime(self,z):
        #Gradient of sigmoid
        return np.exp(-z)/((1+np.exp(-z))**2)

    def costFunction(self, X, y):
        #Compute cost for given X,y, use weights already stored in class.
        self.yHat = self.forward(X)
        J = 0.5*sum((y-self.yHat)**2)
        return J

    def costFunctionPrime(self, X, y):
        #Compute derivative with respect to W and W2 for a given X and y:
        self.yHat = self.forward(X)

        delta3 = np.multiply(-(y-self.yHat), self.sigmoidPrime(self.z3))
        dJdW2 = np.dot(self.a2.T, delta3)

        delta2 = np.dot(delta3, self.W2.T)*self.sigmoidPrime(self.z2)
        dJdW1 = np.dot(X.T, delta2)

        return dJdW1, dJdW2

    #Helper Functions for interacting with other classes:
    def getParams(self):
        #Get W1 and W2 unrolled into vector:
        params = np.concatenate((self.W1.ravel(), self.W2.ravel()))
        return params

    def setParams(self, params):
        #Set W1 and W2 using single paramater vector.
        W1_start = 0
        W1_end = self.hiddenLayerSize * self.inputLayerSize
        self.W1 = np.reshape(params[W1_start:W1_end], (self.inputLayerSize , self.hiddenLayerSize))
        W2_end = W1_end + self.hiddenLayerSize*self.outputLayerSize
        self.W2 = np.reshape(params[W1_end:W2_end], (self.hiddenLayerSize, self.outputLayerSize))

    def computeGradients(self, X, y):
        dJdW1, dJdW2 = self.costFunctionPrime(X, y)
        return np.concatenate((dJdW1.ravel(), dJdW2.ravel()))
```

```python
from scipy import optimize

class trainer(object):
    def __init__(self, N):
        #Make Local reference to network:
        self.N = N

    def callbackF(self, params):
        self.N.setParams(params)
        self.J.append(self.N.costFunction(self.X, self.y))

    def costFunctionWrapper(self, params, X, y):
        self.N.setParams(params)
        cost = self.N.costFunction(X, y)
        grad = self.N.computeGradients(X,y)

        return cost, grad

    def train(self, X, y):
        #Make an internal variable for the callback function:
        self.X = X
        self.y = y

        #Make empty list to store costs:
        self.J = []

        params0 = self.N.getParams()

        options = {'maxiter': 200, 'disp' : True}
        _res = optimize.minimize(self.costFunctionWrapper, params0, jac=True, method='BFGS', \
                                 args=(X, y), options=options, callback=self.callbackF)

        self.N.setParams(_res.x)
        self.optimizationResults = _res
```

```python
NN = Neural_Network()
```

```python
T = trainer(NN)
```

```python
T.train(X,y)
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 73
             Function evaluations: 81
             Gradient evaluations: 81

```python
print("Predicted Output: \n" + str(NN.forward(X)))
print("Loss: \n" + str(np.mean(np.square(y - NN.forward(X))))) # mean sum squared loss
```

    Predicted Output:
    [[0.92001045]
     [0.85999675]
     [0.88999721]]
    Loss:
    4.2507090512976365e-11

```python
import matplotlib.pyplot as plt
plt.plot(T.J)
plt.xlabel('Iterations')
plt.ylabel('Cost')
plt.show();
```


```python
from sklearn import metrics
import numpy as np
import sympy
from sympy.matrices import Matrix
import pandas as pd
import matplotlib.pyplot as plt

```

# 3-class

```python
x = np.array([[2, 0],
              [1, 1],
              [0, 2],
              [2, 0],
              [1, 2],
              [0, 0],
              [2, 0]])

df = pd.DataFrame(x, columns = ['y','yHat'])
df
```

      y
      yHat

      0
      2
      0

      1
      1
      1

      2
      0
      2

      3
      2
      0

      4
      1
      2

      5
      0
      0

      6
      2
      0

```python
print(metrics.classification_report(df['y'], df['yHat']))
```

                  precision    recall  f1-score   support

               0       0.25      0.50      0.33         2
               1       1.00      0.50      0.67         2
               2       0.00      0.00      0.00         3

        accuracy                           0.29         7
       macro avg       0.42      0.33      0.33         7
    weighted avg       0.36      0.29      0.29         7

```python
metrics.precision_score( df['y'], df['yHat'], average=None)
```

    array([0.25, 1.  , 0.  ])

```python
metrics.precision_score( df['y'], df['yHat'], average="micro")
```

    0.2857142857142857

```python
metrics.precision_score(df['y'], df['yHat'], average="macro")
```

    0.4166666666666667

```python
metrics.precision_score(df['y'], df['yHat'], average="weighted")
```

    0.35714285714285715

```python
metrics.recall_score( df['y'], df['yHat'], average="micro")
```

    0.2857142857142857

```python
metrics.recall_score(df['y'], df['yHat'], average="macro")
```

    0.3333333333333333

```python
metrics.recall_score(df['y'], df['yHat'], average="weighted")
```

    0.2857142857142857

```python
metrics.f1_score( df['y'], df['yHat'], average="micro")
```

    0.2857142857142857

```python
metrics.f1_score(df['y'], df['yHat'], average="macro")
```

    0.3333333333333333

```python
metrics.f1_score(df['y'], df['yHat'], average="weighted")
```

    0.2857142857142857

```python

```

```python
metrics.accuracy_score(df['y'], df['yHat'])
```

    0.2857142857142857

```python
pd.crosstab(index=df['y'], columns=df['yHat'], margins=True)
```

      yHat
      0
      1
      2
      All

      y

      0
      1
      0
      1
      2

      1
      0
      1
      1
      2

      2
      3
      0
      0
      3

      All
      4
      1
      2
      7

```python
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(df['y'], df['yHat'])
```

```python
confusion_matrix( df['yHat'],df['y'],)
```

    array([[1, 0, 3],
           [0, 1, 0],
           [1, 1, 0]])

```python
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay(cm, display_labels=[0,1,2]).plot()
```

```python
from sklearn.metrics import multilabel_confusion_matrix
mcm = multilabel_confusion_matrix(df['y'], df['yHat'], labels=[0, 1, 2])
mcm
```

    array([[[2, 3],
            [1, 1]],

           [[5, 0],
            [1, 1]],

           [[2, 2],
            [3, 0]]])

```python
multilabel_confusion_matrix(df['y'], df['yHat'])
```

    array([[[2, 3],
            [1, 1]],

           [[5, 0],
            [1, 1]],

           [[2, 2],
            [3, 0]]])

```python
#ConfusionMatrixDisplay(mcm, display_labels=[0,1,2]).plot()

# how to plot it
# https://stackoverflow.com/questions/62722416/plot-confusion-matrix-for-multilabel-classifcation-python
```

```python
from math import sqrt

def factorize_2d(value):
  res = []
  for x in range(1, int(sqrt(value)+1)):
    if not (value % x):
      res.append([x,value//x])
  return res

factorize_2d(24)
```

    [[1, 24], [2, 12], [3, 8], [4, 6]]

```python

def plot_mcm(mcm):
  n_classes = len(mcm)
  maybe_shapes = factorize_2d(n_classes)
  shape = maybe_shapes[0]
  for s in maybe_shapes:
    if (s[0] + s[1]) < (shape[0] + shape[1]):
      shape = s
  #proportions = 5*np.array(shape)
  proportions = (15, 6)
  fig, axes = plt.subplots(*shape, figsize=proportions)
  if shape[0] != 1 and shape[1] != 1:
    axes = [item for sublist in axes for item in sublist]
  for matrix, ax, i_class in zip(mcm, axes, range(n_classes)):
    ConfusionMatrixDisplay(matrix, display_labels=["others", i_class]).plot(ax=ax)

  plt.tight_layout()
  plt.show()

plot_mcm(mcm)
```

```python

```

#More classes

```python
num_classes = 3
y_true = np.random.randint(0, num_classes, 10)
y_pred = np.random.randint(0, num_classes, 10)
df = pd.DataFrame(data={"y_true": y_true, "y_pred": y_pred})
df.transpose()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9

      y_true
      0
      1
      1
      1
      1
      2
      1
      1
      0
      1

      y_pred
      1
      1
      0
      1
      2
      2
      1
      1
      2
      0

```python
cm = confusion_matrix(y_true, y_pred)
ConfusionMatrixDisplay(cm, display_labels=range(num_classes)).plot()
```

```python
mcm = multilabel_confusion_matrix(y_true, y_pred)
```

```python
plot_mcm(mcm)
```

```python
print(metrics.classification_report( y_true, y_pred))
```

                  precision    recall  f1-score   support

               0       0.00      0.00      0.00         2
               1       0.80      0.57      0.67         7
               2       0.33      1.00      0.50         1

        accuracy                           0.50        10
       macro avg       0.38      0.52      0.39        10
    weighted avg       0.59      0.50      0.52        10

Chapter 2/python/classification metrics/3-class metrics.ipynb


# Statistics

## 1.1 Sales for the past week was the following amounts: [3505, 2400, 3027, 2798, 3700, 3250, 2689]. Without using library functions, what is the mean, variance, and standard deviation of of sales from last week? (for extra bonus points, write your own function that can calculate these two values for any sized list)

```
import numpy as np
import pandas as pd
```

```
sales = [3505,2400,3027,2798,3700,3250,2689]
print('MEAN:',np.mean(sales))
print('VAR:',np.var(sales, ddof=1))
print('STD:',np.std(sales, ddof=1))
```

    MEAN: 3052.714285714286
    VAR: 214387.90476190473
    STD: 463.0204150595357

```
#MEAN FUNCTION
def mean(data_list):
  n = len(data_list)
  x = 0
  for i in data_list:
    x= x+i
  return x/n

mean(sales)
```

    3052.714285714286

```
#VARIENCE FUNCTION
def var(data_list):
  n = len(data_list)
  x = mean(data_list)
  v = []
  for i in data_list:
    v= np.append([v],[(i-x)**2])
  a = v.copy()
  b = 0
  for i in a:
    b = b+i
  return b/(n-1)

var(sales)
```

    214387.90476190473

```
#STANDARD DEVIATION FUNCTION
def stdv(data_list):
  n = len(data_list)
  x = mean(data_list)
  v = []
  for i in data_list:
    v= np.append([v],[(i-x)**2])
  a = v.copy()
  b = 0
  for i in a:
    b = b+i

  return (b/(n-1))**(1/2)

stdv(sales)
```

    463.0204150595357

```
#SHOWING THE SPREAD WITH MY FUNCTIONS
print('MEAN:',mean(sales))
print('VAR:',var(sales))
print('STD:',stdv(sales))
```

    MEAN: 3052.714285714286
    VAR: 214387.90476190473
    STD: 463.0204150595357

## 1.2 Find the covariance between last week's sales numbers and the number of customers that entered the store last week: [127, 80, 105, 92, 120, 115, 93] (you may use librray functions for calculating the covariance since we didn't specifically talk about its formula)

```
customers = [127,80,105,92,120,115,93]
```

```
print('MEAN:',mean(customers))
print('VAR:',var(customers))
print('STD:',stdv(customers))
```

    MEAN: 104.57142857142857
    VAR: 290.95238095238096
    STD: 17.057326313123664

```
#COVARIANCE
data = {'sales': sales, 'customers': customers}
df = pd.DataFrame(data)
df
```

      sales
      customers

      0
      3505
      127

      1
      2400
      80

      2
      3027
      105

      3
      2798
      92

      4
      3700
      120

      5
      3250
      115

      6
      2689
      93

```
np.cov(sales,customers)
```

    array([[214387.9047619 ,   7604.35714286],
           [  7604.35714286,    290.95238095]])

```
var_cov = df.cov()
var_cov
```

      sales
      customers

      sales
      214387.904762
      7604.357143

      customers
      7604.357143
      290.952381

```
print('Covariance:',var_cov.sales[1])
```

    Covariance: 7604.357142857142

## 1.3 Find the standard deviation of customers who entered the store last week. Then, use the standard deviations of both sales and customers to standardize the covariance to find the correlation coefficient that summarizes the relationship between sales and customers. (You may use library functions to check your work.)

```
print('sigma: \u03C3')
```

    sigma: σ

```
σx = stdv(sales)
σy = stdv(customers)
σxy = var_cov.sales[1]
```

```
#CORRELATION COEFFICIENT
r = σxy / (σx * σy)
print('Correlation:',r)
```

    Correlation: 0.9628339778148909

## 1.4 Use pandas to import a cleaned version of the titanic dataset from the following link: [Titanic Dataset](https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv)

## Calculate the variance-covariance matrix and correlation matrix for the titanic dataset's numeric columns. (you can encode some of the categorical variables and include them as a stretch goal if you finish early)

```
titanic = pd.read_csv('https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv')
titanic = titanic.drop('Unnamed: 0', axis=1)
titanic.head()
```

      pclass
      survived
      name
      sex
      age
      sibsp
      parch
      ticket
      fare
      cabin
      embarked
      boat
      body
      home.dest
      has_cabin_number

      0
      1.0
      1.0
      Allen, Miss. Elisabeth Walton
      female
      29.0000
      0.0
      0.0
      24160
      211.3375
      B5
      S
      2
      NaN
      St Louis, MO
      1

      1
      1.0
      1.0
      Allison, Master. Hudson Trevor
      male
      0.9167
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      11
      NaN
      Montreal, PQ / Chesterville, ON
      1

      2
      1.0
      0.0
      Allison, Miss. Helen Loraine
      female
      2.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      NaN
      NaN
      Montreal, PQ / Chesterville, ON
      1

      3
      1.0
      0.0
      Allison, Mr. Hudson Joshua Creighton
      male
      30.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      NaN
      135.0
      Montreal, PQ / Chesterville, ON
      1

      4
      1.0
      0.0
      Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
      female
      25.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      NaN
      NaN
      Montreal, PQ / Chesterville, ON
      1

```
titanic.describe()
```

      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      count
      1309.000000
      1309.000000
      1310.000000
      1309.000000
      1309.000000
      1308.000000
      121.000000
      1310.000000

      mean
      2.294882
      0.381971
      29.881135
      0.498854
      0.385027
      33.295479
      160.809917
      0.225191

      std
      0.837836
      0.486055
      12.878277
      1.041658
      0.865560
      51.758668
      97.696922
      0.417867

      min
      1.000000
      0.000000
      0.166700
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000

      25%
      2.000000
      0.000000
      22.000000
      0.000000
      0.000000
      7.895800
      72.000000
      0.000000

      50%
      3.000000
      0.000000
      29.881135
      0.000000
      0.000000
      14.454200
      155.000000
      0.000000

      75%
      3.000000
      1.000000
      35.000000
      1.000000
      0.000000
      31.275000
      256.000000
      0.000000

      max
      3.000000
      1.000000
      80.000000
      8.000000
      9.000000
      512.329200
      328.000000
      1.000000

```
titanic2 = titanic[['pclass','survived','age','sibsp','parch','fare','body','has_cabin_number']]
titanic2
```

      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      0
      1.0
      1.0
      29.000000
      0.0
      0.0
      211.3375
      NaN
      1

      1
      1.0
      1.0
      0.916700
      1.0
      2.0
      151.5500
      NaN
      1

      2
      1.0
      0.0
      2.000000
      1.0
      2.0
      151.5500
      NaN
      1

      3
      1.0
      0.0
      30.000000
      1.0
      2.0
      151.5500
      135.0
      1

      4
      1.0
      0.0
      25.000000
      1.0
      2.0
      151.5500
      NaN
      1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1305
      3.0
      0.0
      29.881135
      1.0
      0.0
      14.4542
      NaN
      0

      1306
      3.0
      0.0
      26.500000
      0.0
      0.0
      7.2250
      304.0
      0

      1307
      3.0
      0.0
      27.000000
      0.0
      0.0
      7.2250
      NaN
      0

      1308
      3.0
      0.0
      29.000000
      0.0
      0.0
      7.8750
      NaN
      0

      1309
      NaN
      NaN
      29.881135
      NaN
      NaN
      NaN
      NaN
      0

1310 rows × 8 columns

```
#TITANIC VARIANCE COVARIANCE MATRIX
titanic2.cov()
```

      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      pclass
      0.701969
      -0.127248
      -3.954605
      0.053090
      0.013287
      -24.227788
      -2.876653
      -0.249992

      survived
      -0.127248
      0.236250
      -0.314343
      -0.014088
      0.034776
      6.146023
      0.000000
      0.061406

      age
      -3.954605
      -0.314343
      165.850021
      -2.559806
      -1.459378
      114.416613
      81.622922
      1.463138

      sibsp
      0.053090
      -0.014088
      -2.559806
      1.085052
      0.336833
      8.641768
      -8.708471
      -0.003946

      parch
      0.013287
      0.034776
      -1.459378
      0.336833
      0.749195
      9.928031
      4.237190
      0.013316

      fare
      -24.227788
      6.146023
      114.416613
      8.641768
      9.928031
      2678.959738
      -179.164684
      10.976961

      body
      -2.876653
      0.000000
      81.622922
      -8.708471
      4.237190
      -179.164684
      9544.688567
      3.625689

      has_cabin_number
      -0.249992
      0.061406
      1.463138
      -0.003946
      0.013316
      10.976961
      3.625689
      0.174613

# Orthogonality

## 2.1 Plot two vectors that are orthogonal to each other. What is a synonym for orthogonal?

```
#Perpendicular
import matplotlib.pyplot as plt

a = [0,1]
b = [1.25,0]

plt.arrow(0,0,a[0],a[1],head_width=.05, head_length=0.05)
plt.arrow(0,0,b[0],b[1],head_width=.05, head_length=0.05)

plt.xlim(-1,2)
plt.ylim(-1,2)
plt.show()
```

```
a = np.array([0,1])
b = np.array([1.25,0])
np.dot(a,b)
```

    0.0

## 2.2 Are the following vectors orthogonal? Why or why not?

\begin{align}
a = \begin{bmatrix} -5 \\ 3 \\ 7 \end{bmatrix}
\qquad
b = \begin{bmatrix} 6 \\ -8 \\ 2 \end{bmatrix}
\end{align}

```
a = np.array([-5,3,7])
b = np.array([6,-8,2])
```

```
# Vectors a and b are orthogonal because their dot product is zero.
np.dot(a,b)
```

    -40

## 2.3 Compute the following values: What do these quantities have in common?

## What is $||c||^2$?

## What is $c \cdot c$?

## What is $c^{T}c$?

\begin{align}
c = \begin{bmatrix} 2 & -15 & 6 & 20 \end{bmatrix}
\end{align}

```
#NORM SQUARED
c = np.array([2,-15,6,20])
```

```
np.linalg.norm(c)**2
```

    665.0

```
#DOT PRODUCT
np.dot(c,c)
```

    665

```
#TRANSPOSED
c.T * c
```

    array([  4, 225,  36, 400])

```
#cT . c
np.dot(c.T,c)
```

    665

# Unit Vectors

## 3.1 Using Latex, write the following vectors as a linear combination of scalars and unit vectors:

\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix}
\qquad
e = \begin{bmatrix} 2 \\ 11 \\ -8  \end{bmatrix}
\end{align}

\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix} = 7
\begin{bmatrix} 1 \\ 0 \end{bmatrix} + 12
\begin{bmatrix} 0\\ 1 \end{bmatrix} = 7 î + 12 ĵ
\end{align}

\begin{align}
e = \begin{bmatrix} 2 \\ 11\\-8 \end{bmatrix} = 2
\begin{bmatrix} 1 \\ 0 \\0 \end{bmatrix} + 11
\begin{bmatrix} 0\\ 1 \\0 \end{bmatrix} + -8
\begin{bmatrix} 0 \\ 0 \\1 \end{bmatrix} = 2 î + 11 ĵ -8 k̂
\end{align}

```
print('i\u0302')
print('j\u0302')
print('k\u0302')
print('f\u0302')
```

    î
    ĵ
    k̂
    f̂

## 3.2 Turn vector $f$ into a unit vector:

\begin{align}
f = \begin{bmatrix} 4 & 12 & 11 & 9 & 2 \end{bmatrix}
\end{align}

```
#DIVIDE EACH ELEMENT BY IT'S NORM
f = np.array([[4,12,11,9,2]])
fnorm = np.linalg.norm(f)
fnorm
```

    19.131126469708992

```
f̂ = f / fnorm
f̂
```

    array([[0.20908335, 0.62725005, 0.57497921, 0.47043754, 0.10454167]])

# Linear Independence / Dependence

## 4.1 Plot two vectors that are linearly dependent and two vectors that are linearly independent (bonus points if done in $\mathbb{R}^3$).

```
#LINEARLY DEPENDENT
a = [1,1.8]
b = [.5,.9]

plt.arrow(0,0,a[0],a[1],head_width=.1, head_length=0.05, color='blue')
plt.arrow(0,0,b[0],b[1],head_width=.1, head_length=0.05)

plt.xlim(-1,2)
plt.ylim(-1,2)
plt.show()
```

```
#LINEARLY DEPENDENT
a = [.8,.4]
b = [1.6,0]

plt.arrow(0,0,a[0],a[1],head_width=.1, head_length=0.05, color='blue')
plt.arrow(0,0,b[0],b[1],head_width=.1, head_length=0.05)

plt.xlim(-1,2)
plt.ylim(-1,2)
plt.show()
```

# Span

## 5.1 What is the span of the following vectors?

\begin{align}
g = \begin{bmatrix} 1 & 2 \end{bmatrix}
\qquad
h = \begin{bmatrix} 4 & 8 \end{bmatrix}
\end{align}

```
#THE SPAN OF THESE VECTORS IS ONE (1) BECAUSE THE ARE LINEARLY DEPENDENT
g = [1,2]
h = [4,8]
plt.arrow(0,0,h[0],h[1],head_width=.15, head_length=0.15, color = 'pink')
plt.arrow(0,0,g[0],g[1],head_width=.15, head_length=0.15, color='blue')
plt.xlim(0,5)
plt.ylim(0,9)
plt.show()
```

## 5.2 What is the span of $\{l, m, n\}$?

\begin{align}
l = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}
\qquad
m = \begin{bmatrix} -1 & 0 & 7 \end{bmatrix}
\qquad
n = \begin{bmatrix} 4 & 8  & 2\end{bmatrix}
\end{align}

```
#THE SPAN OF THESE VECTORS IS THREE (3)
lmn = np.array([[1,2,3],
              [-1,0,7],
              [4,8,2]])

np.linalg.matrix_rank(lmn)

```

    3

# Basis

## 6.1 Graph two vectors that form a basis for $\mathbb{R}^2$

```
a = [0,1]
b = [1,0]

plt.arrow(0,0,a[0],a[1],head_width=.05, head_length=0.05, color ='#fcba03')
plt.arrow(0,0,b[0],b[1],head_width=.05, head_length=0.05, color ='#03fca5')

plt.xlim(-.2,1.2)
plt.ylim(-.2,1.2)
plt.show()
```

## 6.2 What does it mean to form a basis?

Basis  - every vector in the vector space is linearly independent and can fill the entire vector space.

# Rank

## 7.1 What is the Rank of P?

\begin{align}
P = \begin{bmatrix}
1 & 2 & 3 \\
 -1 & 0 & 7 \\
4 & 8  & 2
\end{bmatrix}
\end{align}

```
lmn = np.array([[1,2,3],
              [-1,0,7],
              [4,8,2]])

np.linalg.matrix_rank(lmn)
```

    3

## 7.2 What does the rank of a matrix tell us?

The rank of this matrix tell us that demensions of the vector space.

# Linear Projections

## 8.1 Line $L$ is formed by all of the vectors that can be created by scaling vector $v$
\begin{align}
v = \begin{bmatrix} 1 & 3 \end{bmatrix}
\end{align}

\begin{align}
w = \begin{bmatrix} -1 & 2 \end{bmatrix}
\end{align}

## find $proj_{L}(w)$

## graph your projected vector to check your work (make sure your axis are square/even)

```
v = np.array([1,3])
w = np.array([-1,2])
#L =

plt.arrow(0,0,v[0],v[1],head_width=.3,
          head_length=0.2, color ='#fcba03') #vector v
plt.arrow(0,0,w[0],w[1],head_width=.3,
          head_length=0.2, color ='#03fca5') #vector w

plt.xlim(-4,4)
plt.ylim(-4,4)
plt.hlines(0,-4,4)
plt.vlines(0,-4,4)
import matplotlib.patches as mpatches
gold_patch = mpatches.Patch(color='#fcba03', label='v')
teal_patch = mpatches.Patch(color='#03fca5', label='w')
plt.legend(handles=[gold_patch,teal_patch])
plt.show()
```

```
np.linalg.norm(v)**2
```

    10.000000000000002

```
projection = (np.dot(w,v) / (np.linalg.norm(v)**2))*v
projection
```

    array([0.5, 1.5])

```
#projL(w)
plt.xlim(-4,4)
plt.ylim(-2,4)
plt.hlines(0,-4,4)
plt.vlines(0,-4,4)

v = np.array([1,3])
w = np.array([-1,2])
x= np.linspace(-5,5)
y=3*x
plt.plot(x,y,color = 'dimgray', linestyle = '--') #Line L

plt.arrow(0,0,v[0],v[1],head_width=.3, head_length=0.2,
          color ='#fcba03', width=.05) #vector v
plt.arrow(0,0,w[0],w[1],head_width=.3, head_length=0.2,
          color ='#03fca5', width=.05) #vector w
plt.arrow(0,0,projection[0],projection[1],head_width=.3,
          head_length=0.2, color ='#e834eb',
          width=.05) #vector w projected on v (line L)
plt.plot((.5,-1),(1.5,2), linestyle='--')

gold_patch = mpatches.Patch(color='#fcba03', label='v')
teal_patch = mpatches.Patch(color='#03fca5', label='w')
purple_patch = mpatches.Patch(color='#e834eb', label='p')
pink_patch = mpatches.Patch(color='dimgray', label='L')
plt.legend(handles=[gold_patch,teal_patch,purple_patch,pink_patch])

plt.show()
```

# Stretch Goal

## For vectors that begin at the origin, the coordinates of where the vector ends can be interpreted as regular data points. (See 3Blue1Brown videos about Spans, Basis, etc.)

## Write a function that can calculate the linear projection of each point (x,y) (vector) onto the line y=x. run the function and plot the original points in blue and the new projected points on the line y=x in red.

## For extra points plot the orthogonal vectors as a dashed line from the original blue points to the projected red points.

```
import pandas as pd
import matplotlib.pyplot as plt

# Creating a dataframe for you to work with -Feel free to not use the dataframe if you don't want to.
x_values = [1, 4, 7, 3, 9, 4, 5 ]
y_values = [4, 2, 5, 0, 8, 2, 8]

data = {"x": x_values, "y": y_values}

df = pd.DataFrame(data)

df.head()

plt.scatter(df.x, df.y)
plt.show()
```

```

```