# Soil Moisture Sensor Design

## Summary
This notebook outlines a design for a capacitive soil moisture sensor. It uses a co-planar capacitor to detect capacitiance of the material around the sensor - in this case we are interested in soil and the water content. As the soil moisture content changes, the permitivity of the material changes. For example, the relative permitivity of air is around 1 and the permitivity of water is around 80. We will attempt to calculate the different capacitances expected for these changes and design a circuit to detect them.

### Inspiration
The inspiration for this project comes from existing soil capacitance sensors - with the added challenge of creating an improved circuit that outputs a rail-to-rail voltage directly proportional to moisture - in other words, a voltage of 0 corresponds to the sensor in air, and and voltage of Vdd when the sensor is submerged in water.

## References:
* [Co-Planar Capacitance Calculator](https://www.emisoftware.com/calculator/coplanar-capacitance/)
* [Capacitive Soil Moisture Sensor](http://www.gikfun.com/belong-to-you-modules-c-68/capacitive-soil-moisture-sensor-corrosion-resistant-for-arduino-p-824.html)

```python
import sympy
import math
import numpy as np
import matplotlib.pyplot as plt

sympy.init_printing()
```

```python
# create the co-planar capacitance functions using [this website](https://www.emisoftware.com/calculator/coplanar-capacitance/) as a guide
s, w, l, ξr = sympy.symbols('s w l ξr')

# s is the distance in m between the plates
# w is the width in m of the plates
# l is the length in m of the plates
condition = s / (s + 2 * w)

# speed of light in a vacuum, m/s
v0 = 299792548

eq = sympy.root(1 - (s**2/(s + 2*w)**2), 4)
denom = eq - 1
numer = eq + 1

cap1 = (ξr * l * sympy.ln((-2 / denom) * numer)) / (377 * sympy.pi * v0)
cap2 = (ξr * l) / (120 * v0 * sympy.ln((-2 * (sympy.root(condition, 2) + 1) ) / (sympy.root(condition, 2) - 1)))

def cp_cap(s, w, l, ξr):
    """
    Calculates the capacitance of a co-planar capacitor

    :param s: distance separating the plates in metres
    :param w: width of the plates in metres
    :param l: length of the plates in metres
    :param ξr: relative permitivity of the material surrounding the capacitor
    :return: capacitance in Farads
    """
    cond = condition.evalf(subs={'s': s, 'w': w, 'l': l, 'ξr': ξr})
    if cond < 0 or cond > 1:
        raise ValueError(f'We are expecting the condition to be between 0 and 1, but found {cond}')

    if cond <= (1 / math.sqrt(2)):
        return cap1.evalf(subs={'s': s, 'w': w, 'l': l, 'ξr': ξr})
    else:
        return cap2.evalf(subs={'s': s, 'w': w, 'l': l, 'ξr': ξr})
```

```python
# plot the expected capacitance between air and water
s_val = 5e-4 # 0.5mm seperating the plates
w_val = 5e-3 # 5mm width of plates
l_val = 0.1 # 10cm length of plates

x_vals = np.linspace(1.0, 80.0)
y_vals = [cp_cap(s_val, w_val, l_val, x) for x in x_vals]

plt.figure()

plt.plot(x_vals, y_vals)
plt.xlabel('Relative Permitivity')
plt.ylabel('Capacitance (F)')
plt.grid(True)
plt.title('Capacitance from Air to Water')

plt.show()
```

```python
# good news, it appears to be linear
air_cp = cp_cap(s_val, w_val, l_val, 1.0006)
water_cp = cp_cap(s_val, w_val, l_val, 80.0)
dry_soil_cp = cp_cap(s_val, w_val, l_val, 3.0)
print(f'Expected range from air to water: [{air_cp}, {water_cp}] F')
```

    Expected range from air to water: [2.49693140512721E-12, 1.99634731571234E-10] F

We can estimate capacitance by detecting the gain across the sensor in an RC circuit when applying a square wave ([RC Circuit - Wikipedia](https://en.wikipedia.org/wiki/RC_circuit)).

```python
ω, R, C = sympy.symbols('ω R C')

Gc = 1 / (sympy.sqrt(1 + (ω * R * C)**2))

# bode plots to determine what frequency we should use
freqs = np.logspace(3, 10)

# resistance to use in the RC circuit for measurement
res = 10e3

gain_air = [Gc.evalf(subs={'R': res, 'C': air_cp, 'ω': f}) for f in freqs]
gain_water = [Gc.evalf(subs={'R': res, 'C': water_cp, 'ω': f}) for f in freqs]
gain_dry_soil = [Gc.evalf(subs={'R': res, 'C': dry_soil_cp, 'ω': f}) for f in freqs]

# bode plots
plt.plot(freqs, gain_air, 'b-', label='Gain Air')
plt.plot(freqs, gain_water, 'r-', label='Gain Water')
plt.plot(freqs, gain_dry_soil, 'g-', label='Gain Dry Soil')
plt.xscale('log')
plt.yscale('log')
plt.xlabel('Frequency (Hz)')
plt.ylabel('Gain')
plt.legend()

plt.grid(True)
plt.title('Gain')
plt.show()
```

From the above plot, we can choose a frequency that provides us with the most gain difference between air and water - somewhere around 10 MHz. This is based on ideal conditions and no other capacitances affecting the sensor.

To test the theory, I first created a very basic co-planar capacitory using copper tape and paper:

After I had proved the basic concept, I then created a PCB version (which I waterproofed with a varathane spray):

I then used a function generator and an oscilliscope to measure the gain loss of the signal when the sensor was submerged in water and also when it was in open air.

```python
# actual measurements from paper prototype
test_freqs = np.array([1e2, 1e3, 1e4, 2e4, 3e4, 4e4, 5e4, 6e4, 7e4, 8e4, 9e4, 1e5, 2e5, 3e5, 4e5, 5e5, 6e5, 7e5, 8e5, 9e5, 1e6])

test_air_v = np.array([4.92, 4.92, 4.92, 4.86, 4.82, 4.78, 4.68, 4.64, 4.6, 4.54, 4.46, 4.36, 3.38, 2.54, 1.90, 1.4, 1.04, .8, .6, .4, .2])
test_air_g = np.divide(test_air_v, 4.92)

test_water_v = np.array([4.92, 4.90, 4.76, 4.54, 4.3, 3.98, 3.76, 3.44, 3.24, 3.02, 2.76, 2.62, 1.54, 1.04, 0.7, 0.5, 0.4, 0.36, 0.28, 0, 0])
test_water_g = np.divide(test_water_v, 4.92)

plt.plot(test_freqs, test_air_g, 'b-', label='Gain Air')
plt.plot(test_freqs, test_water_g, 'r-', label='Gain Water')
plt.xscale('log')
plt.yscale('log')
plt.xlabel('Frequency (Hz)')
plt.ylabel('Gain (Test)')
plt.legend()

plt.grid(True)
plt.title('Gain')
plt.show()
```

At this point, we can use Numpy functions to determine the frequency at which we see the largest gain difference:

```python
dist = np.absolute(test_water_g - test_air_g)
max_index = np.argmax(dist)
print(f'Frequency to use: {test_freqs[max_index]} Hz')
```

    Frequency to use: 200000.0 Hz

200 KHz is a huge difference from 10 MHz calculated in the ideal case. I suspect that my PCB sensors have manufacturing inaccuracies, as well as some parasitic capacitances. I am not a electronics engineer though and these reasons are speculative, so any feedback as to why this is the case would be welcomed!

## Circuit Design
Please refer to the following schematic for the next section:

### Square Wave Generator
To generate a square wave frequency at 200 KHz, we can use a [555 timer (Wikipedia)](https://en.wikipedia.org/wiki/555_timer_IC). I used a rail-to-rail 555 timer, [TLC555CP (Digikey)](https://www.digikey.ca/en/products/detail/texas-instruments/TLC555CP/277502). Using a capacitance of 470pF, and a resistor combination of 1kΩ and 4.7kΩ, I found this resulted in a square wave with an actual frequency of 210kHz (calculators may give you closer to 300kHz but these timers behave different than calculated at higher frequencies) and a duty cycle close to 50%. The generated wave was between 0 volts and Vdd, where Vdd is approximately 5.5V from the power supply I was using.

### Sensor
The sensor is a capacitor, so we pass the generated signal through an RC circuit - I chose a 10kΩ resistor. The output is a typical RC response that varies based on the capacitance change when the moisture surround the sensor changes. The higher the moisture, the lower the peak to peak voltage becomes. The voltage is centred around half of Vdd.

### Rectification
The output is then passed through a diode and low pass filter to extract a constant voltage that varies from about 2.9V (when submerged in water) and 4.8V when in open air. This gives us a voltage range of about 2V to work with.

### Voltage Buffers
The measured voltage is then fed through a voltage buffer circuit, using an op-amp.

**NOTE:** I used a quad op-amp [MCP6004 (Digikey)](https://www.digikey.ca/en/products/detail/microchip-technology/MCP6004-I-P/523060) of which I made use of all 4.

Another op-amp is used as a voltage buffer as well, but this one samples half of Vdd using a voltage divider, and is used in the next circuit.

### Differential Amplifier
This amplifier simply takes the measured voltage, and subtracts off half of Vdd in order to get a signal that now varies from approximately 0.15 to 2.05 volts.

### Inverting Amplifier
Finally, we pass in the previous signal into an inverting amplifier that amplifies the inverted voltage by 1.8 times (from experimentation). We can now measure the voltage output and we will find that we have a voltage that varies from around 0 to Vdd, directly proportinal to the amount of moisture about the sensor.

## Board Design
**NOTE:** In the following image, it appears there is no gap between the capacitor cathode and anode, when in reality there is. Please see the Eagle board and schematic files for finer details.

## Results
Images of the final PCB sensor (again, the sensor portion was water-proofed using a varathane spray):

Images of the sensor at the extremes:

## Other Notes
If using this sensor from a micro-controller, it is required that you only energize this circuit when you wish to sample the value. Otherwise, the operational amplifiers will saturate over time, and the output voltage will be Vdd. I recommended energizing for a few seconds, and then taking a sample, before shutting the sensor down until you need the next sample.


# Implementing FIR filters

In real-time filtering applications, filters are implemented by using some variation or other of their constant-coefficient difference equation (CCDE), so that one new output sample is generated for each new input sample. If all input data is available in advance, as in non-real-time (aka "offline") applications, then the CCDE-based algorithm is iteratively applied to all samples in the buffer.

In the case of FIR filters, the CCDE coefficients correspond to the impulse response and implementing the CCDE is equivalent to performing a convolution sum. In this notebook we will look at different ways to implement FIR filters.

```python
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
```

## Online implementation

The classic way to implement a filter is the one-in one-out approach. We will need to implement a persistent delay line. In Python we can either define a class or use function attributes; classes are tidier and reusable:

```python
class FIR_loop():
    def __init__(self, h):
        self.h = h
        self.ix = 0
        self.M = len(h)
        self.buf = np.zeros(self.M)

    def filter(self, x):
        y = 0
        self.buf[self.ix] = x
        for n in range(0, self.M):
            y += self.h[n] * self.buf[(self.ix+self.M-n) % self.M]
        self.ix = (self.ix + 1) % self.M
        return y
```

```python
# simple moving average:
h = np.ones(5)/5

f = FIR_loop(h)
for n in range(0, 10):
    print(f.filter(n), end=", ")
```

    0.0, 0.2, 0.6, 1.2, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0,

While there's nothing wrong with the above implementation, when the data to be filtered is known in advance, it makes no sense to explicitly iterate over its element and it's better to use higher-level commands to perform the convolution. In Numpy, the command is `convolve`; before we use it, though, we need to take border effects into consideration.

## Offline implementations: border effects

When filtering a finite-length data vector with a finite-length impulse response, we need to decide what to do with the "invalid" shifts appearing in the terms of the convolution sum. Remember that, in the infinite-length case, the output is defined as

$$
    y[n] = \sum_{k=-\infty}^{\infty} h[k]x[n-k]
$$

Let's say that the impulse response is $M$ points long, so that $h[n]$ is nonzero only between $0$ and $M-1$; this means that the sum is reduced to

$$
    y[n] = \sum_{k=0}^{M-1} h[k]x[n-k]
$$

Now assume that $x[n]$ is a length-$N$ signal, so it is defined only for $0 \leq n \le N$ (we can safely consider $N > M$, otherwise exchange the roles of $x$ and $h$). In this case, the above sum is properly defined only for $M - 1 \le n \le N-1$; for any other value of $n$, the sum will contain an element $x[n-k]$ outside of the valid range of indices for the input.

So, if we start with an $N$-point input, we can only formally compute $N-M+1$ output samples. While this may not be a problem in some applications, it certainly is troublesome if repeated filtering operations end up "chipping away" at the signal little by little.

The solution is to "embed" the finite-length input data signal into an infinite-length sequence and, as always, the result will depend on the method we choose: finite support or periodization. (Note that the impulse response is already an infinite sequence since it's the response of the filter to the infinite sequence $\delta[n]$).

However, the embedding will create "artificial" data points that are dependent on the chosen embedding: these data points are said to suffer from **border effects**.

Let's build a simple signal and a simple FIR filter:

```python
# let's use a simple moving average:
M = 5
h = np.ones(M)/float(M)

# let's build a signal with a ramp and a plateau
x = np.concatenate((np.arange(1, 9), np.ones(5) * 8, np.arange(8,0,-1)))
plt.stem(x);
print('signal length: ', len(x))
```

### 1) No border effects

We may choose to accept the loss of data points and use only the $N-M+1$ output samples that correspond to a full overlap between the input data and the impulse response. This can be achieved by selecting `mode='valid'` in `correlate`:

```python
y = np.convolve(x, h, mode='valid')
print('signal length: ', len(y))
plt.stem(y);
```

### 2) finite-support extension

By embedding the input into a finite-support signal, the convolution sum is now well defined for all values of $n$, which now creates a new problem: the output will be nonzero for all values of $n$ for which $x[n-k]$ is nonzero, that is for $0 \le n \le N+M-1$: we end up with a *longer* support for the output sequence. This is the default in `correlate`  and corresponds to `mode='full'`:

```python
y = np.convolve(x, h, mode='full')
print('signal length: ', len(y))
plt.stem(y);
```

If we want to preserve the same length for input and output, we need to truncate the result. You can keep the *first* $N$ samples and discard the tail; this corresponds to the online implementation of the FIR filter. Alternatively, you can discard half the extra samples from the beginning and half from the end of the output and distribute the border effect evenly; this is achieved in `correlate`  by setting `mode='same'`:

```python
y = np.convolve(x, h, mode='same')
print('signal length: ', len(y))
plt.stem(y);
```

### 3) Periodic extension

As we know, the other way of embedding a finite-length signal is to build a periodic extension. The convolution in this case will return an $N$-periodic output:

$$
    \tilde{y}[n] = \sum_{k=0}^{M-1} h[k]\tilde{x}[n-k]
$$

We can easily implement a circular convolution using `convolve` like so: since the overlap between time-reversed impulse response and input is already good for the last $N-M$ points in the output, we just need to consider two periods of the input to compute the first $M$:

```python
def cconv(x, h):
    # as before, we assume len(h) < len(x)
    L = len(x)
    xp = np.concatenate((x,x))
    # full convolution
    y = np.convolve(xp, h)
    return y[L:2*L]
```

```python
y = cconv(x, h)
print('signal length: ', len(y))
plt.stem(y);
```

OK, clearly the result is not necessarily what we expected; note however that in both circular and "normal" convolution, you still have $M-1$ output samples "touched" by border effects, it's just that the border effects act differently in the two cases.

Interestingly, you can still obtain a "normal" convolution using a circular convolution if you zero-pad the input signal with $M-1$ zeros:

```python
y = cconv(np.concatenate((x, np.zeros(M-1))), h)
print('signal length: ', len(y))
plt.stem(y);
# plot in red the difference with the standard conv
plt.stem(y - np.convolve(x, h, mode='full'), markerfmt='ro');
```

Why is this interesting? Because of the DFT....

## Offline implementations using the DFT

The convolution theorem states that, for infinite sequences,

$$
    (x\ast y)[n] = \mbox{IDTFT}\{X(e^{j\omega})Y(e^{j\omega})\}[n]
$$

Can we apply this result to the finite-length case? In other words, what is the inverse DFT of the product of two DFTs? Let's see:

\begin{align}
    \sum_{k=0}^{N-1}X[k]Y[k]e^{j\frac{2\pi}{N}nk} &= \sum_{k=0}^{N-1}\sum_{p=0}^{N-1}x[p]e^{-j\frac{2\pi}{N}pk}\sum_{q=0}^{N-1}y[q]e^{-j\frac{2\pi}{N}qk} \,e^{j\frac{2\pi}{N}nk} \\
    &= \sum_{p=0}^{N-1}\sum_{q=0}^{N-1}x[p]y[q]\sum_{k=0}^{N-1}e^{j\frac{2\pi}{N}(n-p-q)k} \\
    &= N\sum_{p=0}^{N-1}x[p]y[(n-p) \mod N]
\end{align}

The results follows from the fact that $\sum_{k=0}^{N-1}e^{j\frac{2\pi}{N}(n-p-q)k}$ is nonzero only for $n-p-q$ multiple of $N$; as $p$ varies from $0$ to $N-1$, the corresponding value of $q$ between $0$ and $N$ that makes $n-p-q$ multiple of $N$ is $(n-p) \mod N$.

So the fundamental result is: **the inverse DFT of the product of two DFTs is the circular convolution of the underlying time-domain sequences!**

To apply this result to FIR filtering, the first step is to choose the space for the DFTs. In our case we have a finite-length data vector of length $N$ and a finite-support impulse response of length $M$ with $M<N$ so let's operate in $\mathbb{C}^N$ by zero-padding the impulse response to size $N$. Also, we most likely want the normal convolution, so let's zero-pad both signals by an additional $M-1$ samples

```python
def DFTconv(x, h, mode='full'):
    # we want the compute the full convolution
    N = len(x)
    M = len(h)
    X = np.fft.fft(x, n=N+M-1)
    H = np.fft.fft(h, n=N+M-1)
    # we're using real-valued signals, so drop the imaginary part
    y = np.real(np.fft.ifft(X * H))
    if mode == 'valid':
        # only N-M+1 points, starting at M-1
        return y[M-1:N]
    elif mode == 'same':
        return y[int((M-1)/2):int((M-1)/2)+N]
    else:
        return y
```

Let's verify that the results are the same

```python
y = np.convolve(x, h, mode='valid')
print('signal length: ', len(y))
plt.stem(y);
y = DFTconv(x, h, mode='valid')
print('signal length: ', len(y))
plt.stem(y, markerfmt='ro');
```

```python
y = np.convolve(x, h, mode='same')
print('signal length: ', len(y))
plt.stem(y);
y = DFTconv(x, h, mode='same')
print('signal length: ', len(y))
plt.stem(y, markerfmt='ro');
```

Of course the question at this point is: why go through the trouble of taking DFTs if all we want is the standard convolution? The answer is: **computational efficiency.**

If you look at the convolution sum, each output sample requires $M$ multiplications (and $M-1$ additions but let's just consider multiplications). In order to filter an $N$-point signal we will need $NM$ multiplications. Assume $N \approx M$ and you can see that the computational requirements are on the order of $M^2$. If we go the DFT route using an efficient FFT implementation we have approximately:

 * $M\log_2 M$ multiplication to compute $H[k]$
 * $M\log_2 M$ multiplication to compute $X[k]$
 * $M\log_2 M$ multiplication to compute $X[k]H[k]$
 * $M\log_2 M$ multiplication to compute the inverse DFT

Even considering that we now have to use complex multiplications (which will cost twice as much), we can estimate the cost of the DFT based convolution at around $8M\log_2M$, which is smaller than $M^2$ as soon as $M>44$.

In practice, the data vector is much longer than the impulse response so that filtering via standard convolution requires on the order of $MN$ operations. Two techniques, called [Overlap Add](https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method) and [Overlap Save](https://en.wikipedia.org/wiki/Overlap%E2%80%93save_method)
can be used to divide the convolution into $N/M$ independent convolutions between $h[n]$ and an $M$-sized piece of $x[n]$; FFT-based convolution can then be used on each piece. While the exact cost per sample of each technique is a bit complicated to estimate, as a rule of thumb **as soon as the impulse response is longer than 50 samples, it's more convenient to use DFT-based filtering.**

```python

```

```python

```

Digital Signal Processing 2 Filtering/implementation.ipynb


# Tutorial 1 Part 2: Supervised Learning

In this tutorial, we will cover:

* Basics of supervised learning
    - Definitions
    - Optimizing with SGD
    - Types of errors
* Classic ML example: Binary logistic regression from scratch using numpy
* Multiclass logistic regression from scratch with PyTorch and autograd
    - Optimization
    - Writing training loops

```python
# Setup
%matplotlib inline
import numpy as np
import pandas as pd
import sklearn
import matplotlib.pyplot as plt
```

```python
plt.rcParams['font.size'] = 20
```

## Theory Reminders

### The supervised learning framework

1. `Labeled dataset` of $N$ labelled samples: $\mathcal{S} = \{ (\vec{x_i},y_i) \}_{i=1}^N$, where:
- $\vec{x_i} = \left(x_i^1, \dots, x_i^D\right) \in \mathcal{X}$  is a **sample** or **feature vector**.
- $y_i \in \mathcal{Y}$ is the **label**.
- For classification with $C$ classes, $\mathcal{Y} = \{0,\dots,C-1\}$, so each $y_i$ is a **class label**.
- Probabilistic perspective: assume each labeled sample $(\vec{x}^i,y^i)\sim\mathcal{D}$,
  i.e. each labeled sample is drawn from a joint distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}$.

2. `model` is a parametrized set of functions $\mathcal{H}\subseteq \mathcal{Y}^{\mathcal{X}}$.
($\mathcal{Y}^{\mathcal{X}}$ is the set of all functions from $\mathcal{X}$ to $\mathcal{Y}$).

From a probabilistic perspective, a model is a posterior: $P_{Y|\vec{X}}(y, \vec{x})$.

For example, the following hypothesis class

$$
\mathcal{H} =
\left\{ h: \mathcal{X}\rightarrow\mathcal{Y}
~\vert~
h(\vec{x}) = \varphi(\vec{w}\vec{x}+b); \vec{w}\in\set{R}^D,~b\in\set{R}\right\}
$$
where $\varphi(\cdot)$ is some nonlinear function, is known as the **perceptron** model.

3. `pointwise loss function` is $\ell:\mathcal{Y}\times\mathcal{Y}\to{R}$
if and only if it satisfies the following four conditions:
- Non-negativity: $\ell(y_i, y_j) ≥ 0$, for any two distinct observations i,j.
- Symmetry: $\ell(y_i, y_j) = \ell(y_j, y_i)$ for all p and q.
- Triangle Inequality: $\ell(y_i, y_j) ≤ \ell(y_i, y_k) + \ell(y_k, y_j)$ for all i,j,k.

Regression Loss Functions such as **Mean Squared Error** (MSE), **Log MSE** etc..

Binary Classification Loss Functions such as **Binary Cross-Entropy**, **Hinge** etc..

Multi-Class Classification Loss Functions such as **Multi-Class Cross-Entropy**, **Kullback Leibler Divergence**

**The task**: Given the training set $\mathcal{S} = \{ (\vec{x}^i,y^i) \}_{i=1}^N \sim \mathcal{D}$, find
a predictor (hypothesis) $h: \mathcal{X}\to\mathcal{Y}$ which minimizes **population loss**:

$$
L_{\mathcal{D}}(h) =  E[(\vec{x},y)\sim\mathcal{D}]{\ell(y, h(\vec{x})}.
$$

This is also known as the **out-of-sample** loss. It is a deterministic quantity.

Can we solve this problem?

Joint-distribution $\mathcal{D}$ is unknown!

Instead, we define an **empirical (in-sample) loss** $L_{\mathcal{S}}(h)$ as the measure of how well a function $h\in\mathcal{H}$ fits the data $\mathcal{S}$, for example
$$
L_{\mathcal{S}}(h) = \frac{1}{N} \sum_{i=1}^{N} \ell(h(\vec{x}^i), y^i) + R(h)
$$
where

- $\ell(y,\hat{y})$ is some pointwise loss (depends on the data).
- $R(h)$ is a regularization term (depends only on the model).

Note that the in-sample loss is a random variable.

Moreover, minimizing the empirical loss might not give us the best outcome, since our dataset can be [imbalanced](https://developers.google.com/machine-learning/data-prep/construct/sampling-splitting/imbalanced-data?fbclid=IwAR1v65XxQcl_3lb9kSh5WWs_-TWad4tyGoc_kWUP6eHxoI4m7JXeHLqAaFc)
or we can have other desires such as [fairness](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing),
or low false positive rate

and many more

*note that if we have time in the end of this tutorial i will dive deeper into some of this intresting topics*

### Optimization

The model is **trained** by updating it's parameters to improve its performance on some data.

We wish to find a parametrization $\vec{w}^\ast$ of our model $h^{\ast}_{\mathcal{S}}(\vec{x};\vec{w}^\ast)\in\mathcal{H}$ such that
$$
h^{\ast}_{\mathcal{S}} = \arg\min_{h\in\mathcal{H}} L_{\mathcal{S}}(h)
$$

Usual approach: descent-based optimization (will be covered in next lecture)
$$
\vec{w}_{k+1} \leftarrow \vec{w}_{k} - \eta \vec{d}_{k}
$$

Where $\vec{d}_k$ is a **descent direction** and $\eta$ is the step size.

Most common choice for $\vec{d}_k$?

Gradient descent: $\vec{d}_k = \nabla_{\vec{w}_{k}} L(\vec{w}_{k})$

Will we find the solution, $\vec{w}^\ast$, though?

Generally the loss is non-convex and we have no guarantee of converging to the global optimum!
$\Rightarrow$ **Optimization Error**

Gradient descent (GD) has the advantage of being extremely simple to apply, and requires only first-order derivatives.

However it provides no guarantees for convergence on non-convex objectives.

Later in the course we'll learn about variants of GD which perform well in practice even on the non-convex objectives we'll face in deep learning.

### Generalization and Expressiveness

We've talked about optimization error.

What are **other** sources of errors in our learning approach (besides optimization)?

We train to minimize our empirical loss $L_\mathcal{S}$ instead of the population loss $L_\mathcal{D}$ $\Rightarrow$ **Generalization Error**

We only consider a limited set of possible functions, $\mathcal{H}$ $\Rightarrow$ **Approximation Error**

How do we mitigate these errors in the practice of machine learning?

- Optimization error: Mini batches; GD variants like stochastic gradient, momentum, Adam, etc (we'll see later in the course).

- Generalization error: Get more data; get data which better represents $\mathcal{D}$; train-test splits; cross validation; early stopping; regularization.

- Approximation error: Use a powerful hypothesis class (e.g. DNN); more parameters; tailoring to the domain (e.g. CNN for images).

### Binary Logistic Regression

Actually a **classification** model. We're trying to classify data into 2 classes.

- Domain: $\vec{x}^i \in {R}^D$
- Target: $y^i \in \{0,1\}$
- Model: $\hat{y} = h(\vec{x}) = \sigma(\vectr{w}\vec{x}+b)$, where $\sigma(\vec{z})$ is the **logistic function**:
    $$\sigma(\vec{z}) = \frac{1}{1+e^{-\vec{z}}}.$$
    This function maps the real line onto $[0,1]$.
- Probabilistic interpretation: $\hat{y}=P(\rvar{Y}=1|\rvec{X}=\vec{x})$.

  This is a posterior probability function for our target variable $\rvar{Y}$.

```python
def logistic(z):
    return 1 / (1 + np.exp(-z))

x = np.arange(-5, 5, .01)
y_hat = logistic(x)
```

```python
plt.plot(x, y_hat, label='$\sigma(z)$')
plt.grid(True); plt.xlabel('z'); plt.legend(); plt.title('The logistic function');
```

To optimize, we minimize the **negative log-likelihood** (equivalent to maximizing likelihood, i.e. MLE) of the parameters $\vec{w}$:

$$
\bb{w}^\ast = \mathrm{arg}\max_{ \bb{w}} \prod_{i=1}^n P(y_i | \bb{x}_i;\vec{w}) = \mathrm{arg}\min_{ \bb{w}} \sum_{i=1}^n -\log P(y_i | \bb{x}_i; \vec{w})
$$

Based on this we define our loss $L({w})$ (what we minimize) as,

$$
\begin{align}
L({w}) &\triangleq \sum_{i=1}^n -\log P(y_i | \bb{x}_i)\\
&= \sum_{i=1}^n -y_i \log P(y_i=1 | \bb{x}_i) - (1-y_i) \log P(y_i=0 | \bb{x}_i) \\
&= \sum_{i=1}^n -y_i \log \hat{y}_i - (1-y_i) \log(1-\hat{y}_i)
\end{align}
$$

The resulting pointwise loss is also known as **cross-entropy**:
$$
\begin{align}
\ell(y, \hat{y}) &=  - y \log(\hat{y}) - (1-y) \log(1-\hat{y}) \\
\end{align}
$$

(The "cross" here is between the distribution of the samples $y^i$ and the distribution of the predictions $\hat{y}^i$).

Optimization scheme: No closed form solution, but loss is **convex** so gradient-based approach leads to global optimum.

$$
\begin{align}
\ell(y, \hat{y}) &=  - y \log(\hat{y}) - (1-y) \log(1-\hat{y}) \\
\end{align}
$$

Let's plot this loss function:

```python
loss_y0 = -np.log(1-y_hat)
loss_y1 = -np.log(y_hat)
plt.plot(y_hat, loss_y0, label='$\ell(y=0,\hat{y})$')
plt.plot(y_hat, loss_y1, label='$\ell(y=1,\hat{y})$')
plt.grid(True); plt.xlabel('$\hat{y}$'); plt.legend(); plt.title('Cross entropy loss');
```

To apply gradient descent, we need to know the gradient of the loss w.r.t. the parameters $\vec{w}$.

Exercise: Prove that for the cross entropy loss with binary logistic regression, it holds that:
$$
\pderiv{\ell(y,\hat{y})}{\vec{w}}= (\hat{y}-y)\vec{x}.
$$

Proof:

First, we apply the chain-rule
$$
\pderiv{\ell(y,\hat{y})}{\vec{w}}=\pderiv{\ell}{\hat{y}}\cdot\pderiv{\hat{y}}{z}\cdot\pderiv{z}{\vec{w}},
$$
where $z=\vectr{w}\vec{x}$.

Now, just by definition,
$$
\begin{align}
\pderiv{\ell}{\hat{y}}&=-\frac{y}{\hat{y}}+\frac{1-y}{1-\hat{y}}=\frac{\hat{y}-y}{\hat{y}(1-\hat{y})}\\
\pderiv{\hat{y}}{z}&=\frac{e^{-z}}{(1+e^{-z})^2}=
\frac{e^{-z}}{1+e^{-z}}\cdot\frac{1}{1+e^{-z}}=(1-\sigma(z))\sigma(z)=(1-\hat{y})\hat{y}\\
\pderiv{z}{\vec{w}}&=\vec{x}
\end{align}
$$

## Part 1: Binary Logistic Regression

As a warm up exercise, let's see a quick example of implementing this algorithm and training it from scratch using just `numpy` (and a toy dataset from `sklearn`).

This is a classic and very simple example of implementing and training a machine learning algorithm.

## Dataset

The `scikit-learn` library comes with a few [toy datasets](http://scikit-learn.org/stable/datasets/index.html#toy-datasets) that are fun to quickly train small models on.

As an example we'll load the Wisconsin-breast cancer database:
- 569 samples of cancer patients
- 30 features: various properties of tumor cells extracted from images
- 2 classes: Tumor is either [Benign or Malignant (שפיר או ממאיר)](https://www.verywellhealth.com/what-does-malignant-and-benign-mean-514240)

We'll apply the basic machine learning approach we saw above: binary logistic regression.

```python
import sklearn.datasets

ds_cancer = sklearn.datasets.load_breast_cancer()
feature_names = ds_cancer.feature_names
target_names = ds_cancer.target_names
n_features = len(feature_names)

print(f'{n_features=}')
print(f'feature names: {feature_names}')
print(f'target  names: {target_names}')
```

    n_features=30
    feature names: ['mean radius' 'mean texture' 'mean perimeter' 'mean area'
     'mean smoothness' 'mean compactness' 'mean concavity'
     'mean concave points' 'mean symmetry' 'mean fractal dimension'
     'radius error' 'texture error' 'perimeter error' 'area error'
     'smoothness error' 'compactness error' 'concavity error'
     'concave points error' 'symmetry error' 'fractal dimension error'
     'worst radius' 'worst texture' 'worst perimeter' 'worst area'
     'worst smoothness' 'worst compactness' 'worst concavity'
     'worst concave points' 'worst symmetry' 'worst fractal dimension']
    target  names: ['malignant' 'benign']

```python
X, y = ds_cancer.data, ds_cancer.target
n_samples = len(y)

print(f'X: {X.shape}')
print(f'y: {y.shape}')
```

    X: (569, 30)
    y: (569,)

```python
# Load into a pandas dataframe and show some samples
y_names = np.full_like(y, target_names[0].upper(), dtype=target_names.dtype)
y_names[y==1] = target_names[1].upper()

df_cancer = pd.DataFrame(data=X, columns=ds_cancer.feature_names)
df_cancer = df_cancer.assign(CLASS=y_names)
df_cancer.iloc[100:110, 0::3]
```

      mean radius
      mean area
      mean concavity
      mean fractal dimension
      perimeter error
      compactness error
      symmetry error
      worst texture
      worst smoothness
      worst concave points
      CLASS

      100
      13.610
      582.7
      0.08625
      0.05871
      2.8610
      0.014880
      0.01465
      35.27
      0.1265
      0.11840
      MALIGNANT

      101
      6.981
      143.5
      0.00000
      0.07818
      1.5530
      0.010840
      0.02659
      19.54
      0.1584
      0.00000
      BENIGN

      102
      12.180
      458.7
      0.02383
      0.05677
      1.1830
      0.006098
      0.01447
      32.84
      0.1123
      0.07431
      BENIGN

      103
      9.876
      298.3
      0.06154
      0.06322
      1.5280
      0.021960
      0.01609
      26.83
      0.1559
      0.09749
      BENIGN

      104
      10.490
      336.1
      0.02995
      0.06481
      2.3020
      0.022190
      0.02710
      23.31
      0.1219
      0.03203
      BENIGN

      105
      13.110
      530.2
      0.20710
      0.07692
      2.4100
      0.029120
      0.01547
      22.40
      0.1862
      0.19860
      MALIGNANT

      106
      11.640
      412.5
      0.07070
      0.06520
      2.1550
      0.023100
      0.01565
      29.26
      0.1688
      0.12180
      BENIGN

      107
      12.360
      466.7
      0.02643
      0.06066
      0.8484
      0.010470
      0.01251
      27.49
      0.1184
      0.08442
      BENIGN

      108
      22.270
      1509.0
      0.42640
      0.07039
      10.0500
      0.086680
      0.03112
      28.01
      0.1701
      0.29100
      MALIGNANT

      109
      11.340
      396.5
      0.05133
      0.06529
      1.5970
      0.015570
      0.01568
      29.15
      0.1699
      0.08278
      BENIGN

```python
# Split into train and test sets
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

print(f'train: X={X_train.shape} y={y_train.shape}')
print(f'test : X={X_test.shape} y={y_test.shape}')
```

    train: X=(398, 30) y=(398,)
    test : X=(171, 30) y=(171,)

```python
# Standardize the features

# Note: each feature is standardized individually:
mu_X = np.mean(X_train, axis=0) # (N, D) -> (D,)
sigma_X = np.std(X_train, axis=0)

# Note: Broadcasting (N, D) with (D,) -> (N, D)
X_train_sc = (X_train - mu_X) / sigma_X

# Note: Test set must be transformed identically to training set
X_test_sc = (X_test - mu_X) / sigma_X

print(f'{mu_X.shape=}, {sigma_X.shape=}')
```

    mu_X.shape=(30,), sigma_X.shape=(30,)

## Model Implementation

We'll implement based on the above definitions.

The model will be implemented as a class with an API that conforms to the `sklearn` models, specifically see
`sklearn`'s [`LogisticRegression`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression) class.

```python
class BinaryLogisticRegression(object):
    def __init__(self, n_iter=100, learn_rate=0.1):
        self.n_iter = n_iter
        self.learn_rate = learn_rate
        self._w = None

    def _add_bias(self, X: np.ndarray):
        # Add a bias term column
        ones_col = np.ones((X.shape[0], 1))
        return np.hstack([ones_col, X])

    def predict_proba(self, X: np.ndarray, add_bias=True):
        X = self._add_bias(X) if add_bias else X

        # Apply logistic model
        z = np.dot(X, self.weights) # (N, D) * (D,)
        return logistic(z) # shape (N,)

    def predict(self, X: np.ndarray):
        proba = self.predict_proba(X)

        # Apply naive threshold of .5
        return np.array(proba > .5, dtype=np.int)

    def fit(self, X: np.ndarray, y: np.ndarray):
        n, d = X.shape # X is (N, D), y is (N,)

        # Initialize weights
        self._w = np.random.randn(d + 1) * 0.1

        Xb = self._add_bias(X)

        # Training loop
        self._losses = []
        for i in range(self.n_iter):
            # Predicted probabilities
            y_hat = self.predict_proba(Xb, add_bias=False)

            # Pointwise loss
            loss = -y.dot(np.log(y_hat)) - ((1 - y).dot(np.log(1 - y_hat)))

            # See Exercise for gradient derivation
            loss_grad = 1/n * Xb.T.dot(y_hat - y)  # dl/dw: (D+1, N) * (N,)

            # Optimization step
            self._w += -self.learn_rate * loss_grad
            self._losses.append(loss)

        return self

    @property
    def weights(self):
        if self._w is None:
            raise ValueError("Model is not fitted")
        return self._w
```

```python
# Fit the model to the training data
lr_model = BinaryLogisticRegression(n_iter=500, learn_rate=0.01)
lr_model.fit(X_train_sc, y_train)

plt.plot(lr_model._losses, label='$L_{\mathcal{S}}$');
plt.xlabel('training iteration'); plt.legend(); plt.grid(True); plt.title('Train loss');
```

```python
y_train_pred = lr_model.predict(X_train_sc)
train_acc = np.mean(y_train_pred == y_train)
print(f'train set accuracy: {train_acc*100:.2f}%')

y_test_pred = lr_model.predict(X_test_sc)
test_acc = np.mean(y_test_pred == y_test)
print(f'test set accuracy: {test_acc*100:.2f}%')
```

    train set accuracy: 96.98%
    test set accuracy: 95.32%

    <ipython-input-11-99026b2ed7a5>:23: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
    Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
      return np.array(proba > .5, dtype=np.int)

## Part 2: Multiclass Logistic Regression

What if we have $C$ classes? Can we still use logistic regression?

A naïve approach is to train $C$ binary logistic regression classifiers, for example in a One vs. Rest scheme,
and then predict based on the classifier returning the greatest probability.

One major drawback of this approach is that it doesn't model the probability distribution over the possible classes, $P_{\vec{Y}|\vec{X}}$.

For example, a sample might be classified as class A with probability $0.8$ and class $B$ with $0.7$ since nothing constrains the different classifiers.

### The softmax function

**Softmax** is a function which can generate a probability distribution for our $C$ classes given raw prediction scores. It's defined as follows:

$$
\mathrm{softmax}(\vec{z}) = \frac{e^{\vec{z}}}{\sum_{j=1}^{C} e^{z_j}}
$$

note that this is a vector valued, multivariate function. The exponent in the enumerator operates elementwise.

The result of softmax is a vector with elements in $[0,1]$ that all sum to $1$.

### The multiclass model

Our model can now be defined as

$$\hat{\vec{y}} = h(\vec{x}) = \mathrm{softmax}(\mattr{W}\vec{x}+\vec{b})$$

where,
- $\hat{\vec{y}}$ is a $C\times 1$ vector of class probabilities.
- ${\vec{x}}$ is a $D\times 1$ sample.
- $  {W}$ is a $D\times C$ matrix representing the per-class weights.
- $\vec{b}$ is a per-class bias vector of length $C$.

Probabilistic interpretation: $\hat{y}_j = P(\rvar{Y}=j|\rvec{X}=\vec{x})$.

While not very powerful on it's own, this type of model is commonly found at the end of deep neural networks performing classification tasks.

To train such a model we need our labels to also be singleton probability distributions instead of simply the number of the correct class.

We'll transform our labels to a 1-hot encoded vector corresponding to a delta distribution.
For example, if $y^i=j$ then we'll create
$$
\vec{y}^i = [0,\dots,0,\underbrace{1}_{j\mathrm{th\ component}},0,\dots,0]^\top
$$

### Cross-Entropy loss

After defing the 1-hot label vectors, the multiclass extension of the binary cross-entropy is straightforward:

$$
\ell(\vec{y}, \hat{\vec{y}}) = - \vec{y} \log(\hat{\vec{y}})
$$

Note that only the probability assigned to the correct class affects the loss.

Minimizing this cross entropy can be interpreted as trying to move the probability distribution of model predictions towards the singleton distribution of the appropriate class.

### Dataset

This time we'll tackle an image classification task, the MNIST database of handwritten digits.

Today this is also considered a toy dataset, even though it was used in the past to benchmark classification algorithms.

```python
import os
import torch
import torch.autograd
import torch.utils.data
import torchvision
import torchvision.transforms
import plot_utils

from torch import Tensor
```

```python
# Define the transforms that should be applied to each image in the dataset before returning it
tf_ds = torchvision.transforms.ToTensor()

batch_size = 64
data_root = os.path.expanduser("~/.pytorch-datasets")

# Training and test datasets
ds_train, ds_test = [
    torchvision.datasets.MNIST(root=data_root, download=True, train=train, transform=tf_ds)
    for train in [True, False]
]

# Data loaders
dl_train = torch.utils.data.DataLoader(ds_train, batch_size, shuffle=True)
dl_test = torch.utils.data.DataLoader(ds_test, batch_size=len(ds_test))

x0, y0 = ds_train[0]
n_features = torch.numel(x0)
n_classes = 10
```

```python
# Show first few samples
print(f'x0: {x0.shape}, y0: {y0}')
plot_utils.dataset_first_n(ds_train, 10, cmap='gray');
```

Note that when training, we're actually working with **batches** of samples (we'll learn about SGD in the sebsequent lectures/tutorials):

```python
x0, y0 = next(iter(dl_train))
x0.shape
```

    torch.Size([64, 1, 28, 28])

### Model Implementation

This time we'll use `pytorch` tensors and its [`autograd`](https://pytorch.org/docs/stable/autograd.html) functionality to implement our model.

This means we wont have to implement any gradient calculations!

First, let's implement $\mathrm{softmax}(\cdot)$. We need a small numerical trick to prevent large numbers from exploding the exponentiation. You can verify that this doesn't influence the result.

```python
def softmax(z: Tensor) -> Tensor:
    """ Softmax of z (N,C), a batch of class scores per sample """

    # normalization trick to prevent numerical instability:
    # shift so that the highest class score (per sample) is 0
    zmax, _ = torch.max(z, dim=1, keepdim=True)
    z = z - zmax # note broadcasting: (N,C) - (N,1)

    exp_z = torch.exp(z) # (N, C)
    sum_exp = torch.sum(exp_z, dim=1, keepdim=True) # (N, 1)
    return exp_z / sum_exp # probabilities, (N,C)
```

Let's test our softmax and calculate its derivative with `autograd`.

```python
z = torch.randn(size=(4,3), requires_grad=True)
y = softmax(z)
y
```

    tensor([[0.6005, 0.0097, 0.3898],
            [0.3947, 0.1850, 0.4203],
            [0.0709, 0.0711, 0.8580],
            [0.1064, 0.7703, 0.1233]], grad_fn=<DivBackward0>)

```python
y = softmax(z)
L = torch.sum(y) # scalar function of z

# Calculate gradient: dL/dz
torch.autograd.grad(L, z)
```

    (tensor([[0., 0., 0.],
             [0., 0., 0.],
             [0., 0., 0.],
             [0., 0., 0.]]),)

Instead of calling `autograd.grad()` directly with specific input tensors, `pytorch` provides us with a way to calculate the derivative of a tensor w.r.t. all the tensors which are **leaves** in it's computation graph ($\vec{z}$ ini this case).

This can be done by calling `.backward()` on a scalar tensor.
As a result, the `.grad` property of leaf tensors will be populated with the gradient:

```python
y = softmax(z)
L = torch.sum(y) # scalar function of z
L.backward()     # Calculate derivative w.r.t. all leaves
z.grad           # The leaf z will have its .grad populated
```

    tensor([[0., 0., 0.],
            [0., 0., 0.],
            [0., 0., 0.],
            [0., 0., 0.]])

Here's the resulting computation graph:

```python
import torchviz
torchviz.make_dot(L, params=dict(z=z))
```

The next ingredient of the solution is the cross-entropy loss function.

Recall that we need to encode our ground-truth labels as one-hot vectors to apply the multiclass cross-entropy.

```python
def cross_entropy_loss(y: Tensor, y_hat: Tensor):
    """
    y_hat (N,C) is a batch of probabilities
    y (N, C) is onehot ground-truth labels
    """
    return torch.sum( - y * torch.log(y_hat + 1e-6) )
```

```python
def onehot(y: Tensor, n_classes: int) -> Tensor:
    """ Encodes y of shape (N,) as one-hot of shape (N,C) """

    y = y.reshape(-1, 1) # Reshape y to (N,1)
    zeros = torch.zeros(size=(len(y), n_classes), dtype=torch.float32) # (N,C)
    ones = torch.ones_like(y, dtype=torch.float32)

    # scatter: put items from 'src' into 'dest' at indices correspondnig to 'index' along 'dim'
    y_onehot = torch.scatter(zeros, dim=1, index=y, src=ones)

    return y_onehot # result has shape (N, C)
```

```python
onehot(torch.tensor([1, 3, 5, 0]), n_classes=10)
```

    tensor([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
            [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
            [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

Our model itself will just hold the parameters $\mat{W}$ and $\vec{b}$ and apply them to an input batch $\mat{X}$.

```python
class MCLogisticRegression(object):
    def __init__(self, n_features: int, n_classes: int):
        # Define our parameter tensors: notice that now W and b are separate
        # Specify we want to track their gradients with autograd
        self.W = torch.randn(n_features, n_classes, requires_grad=True)
        self.b = torch.randn(n_classes, requires_grad=True)
        self.params = [self.W, self.b]

    def __call__(self, *args):
        return self.forward(*args)

    def forward(self, X: Tensor):
        """
        X (N, D) contains a batch of samples
        output (N, C) is a batch of class probabilities
        """
        # X is (N, D), W is (D, C), b is (C,)
        z = torch.mm(X, self.W) + self.b
        y_hat = softmax(z)
        return y_hat # (N, C)
```

Let's try out the model and loss on the first batch.

Note that we naïvely treat each pixel as a separate feature. We'll learn how to properly work with images in a future tutorial.

```python
model = MCLogisticRegression(n_features, n_classes)

# Flatten images and convert labels to onehot
x0_flat = x0.reshape(-1, n_features)
y0_onehot =  onehot(y0, n_classes)

print(f'x0_flat: {x0_flat.shape}')
print(f'y0_onehot: {y0_onehot.shape}\n')
```

    x0_flat: torch.Size([64, 784])
    y0_onehot: torch.Size([64, 10])

```python
# Forward pass and compute loss
y0_hat = model(x0_flat)
loss = cross_entropy_loss(y0_onehot, y0_hat)
print('loss = ', loss)

# Backward pass to populate .grad on leaf nodes
loss.backward()
```

    loss =  tensor(652.7525, grad_fn=<SumBackward0>)

Since we specified `require_grad=True` for our model parameters, every operation performed on these tensors is recorded, and a **computation graph** can be built, which included the model and loss calculation.
Notice that the **leaves** in this graph are our parameters $\mat{W}$ and $\vec{b}$.

```python
import torchviz
torchviz.make_dot(loss, params=dict(W=model.W, b=model.b))
```

This graph is what allows efficient implementation of the **back-propagation** algorithm which you'll learn about in the next lecture.

By calling `.backward()` from the final loss tensor, pytorch automatically populated the `.grad` property of all leaves in this graph, without us having to explicitly specify them (`W` and `b`).

### Training

The optimization will be as before, but now we'll take the gradients from the `grad` property of our parameter tensors.

Therefore, the optimizer (in this case just a funcion) only needs access to the parameter tensors from the model.
Later you'll see that `pytorch`'s `Optimizer` classes work in the same way.

```python
def sgd_optimizer(params: [Tensor, ...], learn_rate: float):
    # params is a list of Tensors
    with torch.autograd.no_grad(): # Don't track gradients for this operation
        for param in params:
            param -= learn_rate * param.grad
            param.grad.zero_()
```

Inference and prediction accuracy:

```python
def evaluate_accuracy(dataloader, model, max_batches=None):
    n_correct = 0.
    n_total = 0.
    for i, (X, y) in enumerate(dataloader):
        X = X.reshape(-1, n_features) # flatten images into vectors

        # Forward pass
        with torch.autograd.no_grad():
            y_hat = model(X)

        predictions = torch.argmax(y_hat, dim=1)
        n_correct += torch.sum(predictions == y).type(torch.float32)
        n_total += X.shape[0]

        if max_batches and i+1 >= max_batches:
            break

    return (n_correct / n_total).item()

test_set_acc = evaluate_accuracy(dl_test, MCLogisticRegression(n_features, n_classes))
print(f'Test set accuracy pre-training: {test_set_acc*100:.2f}%')
```

    Test set accuracy pre-training: 8.87%

#### The training loop

This is a crucial part of any ML pipeline where model parameters get updated iteratively.

One pass over the entire training data is called an **epoch**.
When using `pytorch`, your training loop will generally contain the following steps:

- Each epoch:
    - Split training data into batches
    - For each batch
        - Forward pass: Compute predictions and build computation graph
        - Calculate loss
        - Set existing gradients to zero
        - Backward pass: Use back-propagation algorithm to calculate the gradients
        - Optimization step: Use the gradients to update the parameters
    - Evaluate accuracy on validation set

```python
# Define some training hyper-parameters
epochs = 10
max_batches = 50  # limit batches so training is fast (just as a demo)
learn_rate = .005
num_samples = len(ds_train)

# Instantiate the model we'll train
model = MCLogisticRegression(n_features, n_classes)
```

```python
# Epoch: traverse all samples
for e in range(epochs):
    cumulative_loss = 0

    # Loop over randdom batches of training data
    for i, (X, y) in enumerate(dl_train):

        X = X.reshape(-1, n_features)
        y_onehot = onehot(y, n_classes)

        # Forward pass: predictions and loss
        y_hat = model(X)
        loss = cross_entropy_loss(y_onehot, y_hat)

        # Backward pass: calculate gradients
        loss.backward()

        # Update model using the calculated gradients
        sgd_optimizer(model.params, learn_rate)

        cumulative_loss += loss.item()
        if i+1 > max_batches:
            break

    # Evaluation
    test_accuracy = evaluate_accuracy(dl_test, model, max_batches)
    train_accuracy = evaluate_accuracy(dl_train, model, max_batches)

    avg_loss = cumulative_loss/num_samples
    print(f"Epoch {e}. Avg Loss: {avg_loss:.3f}, Train Acc: {train_accuracy*100:.2f}, Test Acc: {test_accuracy*100:.2f}")
```

    Epoch 0. Avg Loss: 0.272, Train Acc: 54.37, Test Acc: 52.99
    Epoch 1. Avg Loss: 0.120, Train Acc: 66.22, Test Acc: 66.92
    Epoch 2. Avg Loss: 0.091, Train Acc: 72.75, Test Acc: 73.01
    Epoch 3. Avg Loss: 0.073, Train Acc: 77.06, Test Acc: 76.22
    Epoch 4. Avg Loss: 0.060, Train Acc: 77.44, Test Acc: 78.31
    Epoch 5. Avg Loss: 0.055, Train Acc: 78.84, Test Acc: 79.93
    Epoch 6. Avg Loss: 0.056, Train Acc: 80.41, Test Acc: 81.58
    Epoch 7. Avg Loss: 0.050, Train Acc: 80.03, Test Acc: 82.57
    Epoch 8. Avg Loss: 0.047, Train Acc: 83.66, Test Acc: 83.07
    Epoch 9. Avg Loss: 0.046, Train Acc: 83.03, Test Acc: 83.23

#### Final notes

- This is a very naive implementation, for example because
    - We didn't treat the images properly.
    - We didn't include any regularization.

- PyTorch provides many functions and classes that we could have used, for example:
  - Perceptron fully connected layer with model parameters
  - Softmax
  - SGD and many other optimizers
  - Cross entropy loss

  however the purpose here was to show an (almost) from-scratch implementation using only tensors,
  in order to see whats "under the hood" (more or less) of the PyTorch functions.

### Thanks!

**Further reading**

[Evaluation Metrics for Classification Models](https://www.analyticsvidhya.com/blog/2020/10/how-to-choose-evaluation-metrics-for-classification-model/)

[google explain imbalanced data](https://developers.google.com/machine-learning/data-prep/construct/sampling-splitting/imbalanced-data?fbclid=IwAR1v65XxQcl_3lb9kSh5WWs_-TWad4tyGoc_kWUP6eHxoI4m7JXeHLqAaFc)

[Deal with imbalanced data](https://towardsdatascience.com/methods-for-dealing-with-imbalanced-data-5b761be45a18)

[Fairness in ML book](http://www.fairmlbook.org)

[Fairness methods review](https://arxiv.org/abs/2012.15816)

**Credits**

This tutorial based on Technion IIT cs-236781 tutorial that written by [Aviv A. Rosenberg](https://avivr.net)
And modeified by Moshe Kimhi.

Some images in this tutorial were taken and/or adapted from the following sources:
- Fundamentals of Deep Learning, Nikhil Buduma, Oreilly 2017


```python
from sympy import *
from util import *
```

MMS
===

```python
x, y = symbols('x y')
lam = Function('lam')(x,y)
mu = Function('mu')(x,y)
```

```python
u = Array([cos(pi * x) * cos(pi * y), 0.0])
I = Array([[1, 0], [0, 1]])
```

```python
', '.join([lua_code(ui) for ui in u])
```

    'math.cos(math.pi*x)*math.cos(math.pi*y), 0.0'

```python
Ju = permutedims(derive_by_array(u, [x, y]), (1, 0))
```

```python
strain = 0.5 * simplify(Ju + Ju.transpose())
stress = simplify(lam * tensorcontraction(strain, (0, 1)) * I + 2.0 * mu * strain)
```

```python
force = -simplify(tensorcontraction(derive_by_array(stress, [x, y]), (0, 2)))
force
```

$\displaystyle \left[\begin{matrix}\pi \left(\pi \left(1.0 \operatorname{lam}{\left(x,y \right)} + 2.0 \mu{\left(x,y \right)}\right) \cos{\left(\pi x \right)} \cos{\left(\pi y \right)} + \left(1.0 \frac{\partial}{\partial x} \operatorname{lam}{\left(x,y \right)} + 2.0 \frac{\partial}{\partial x} \mu{\left(x,y \right)}\right) \sin{\left(\pi x \right)} \cos{\left(\pi y \right)} + 1.0 \pi \mu{\left(x,y \right)} \cos{\left(\pi x \right)} \cos{\left(\pi y \right)} + 1.0 \sin{\left(\pi y \right)} \cos{\left(\pi x \right)} \frac{\partial}{\partial y} \mu{\left(x,y \right)}\right) & - 1.0 \pi \left(\pi \operatorname{lam}{\left(x,y \right)} \sin{\left(\pi x \right)} \sin{\left(\pi y \right)} + \pi \mu{\left(x,y \right)} \sin{\left(\pi x \right)} \sin{\left(\pi y \right)} - \sin{\left(\pi x \right)} \cos{\left(\pi y \right)} \frac{\partial}{\partial y} \operatorname{lam}{\left(x,y \right)} - \sin{\left(\pi y \right)} \cos{\left(\pi x \right)} \frac{\partial}{\partial x} \mu{\left(x,y \right)}\right)\end{matrix}\right]$

Constant coefficients
=================

```python
f1 = force.subs(mu, 1).subs(lam, 2).simplify()
f1
```

$\displaystyle \left[\begin{matrix}5.0 \pi^{2} \cos{\left(\pi x \right)} \cos{\left(\pi y \right)} & - 3.0 \pi^{2} \sin{\left(\pi x \right)} \sin{\left(\pi y \right)}\end{matrix}\right]$

```python
', '.join([lua_code(fi) for fi in f1])
```

    '5.0*math.pi^2*math.cos(math.pi*x)*math.cos(math.pi*y), -3.0*math.pi^2*math.sin(math.pi*x)*math.sin(math.pi*y)'

Variable coefficients
================

```python
f2 = force.subs(mu, (x+y)**3).subs(lam, (x+y)**4).simplify()
f2
```

$\displaystyle \left[\begin{matrix}\pi \left(\pi \left(x + y\right)^{3} \left(1.0 x + 1.0 y + 2.0\right) \cos{\left(\pi x \right)} \cos{\left(\pi y \right)} + 1.0 \pi \left(x + y\right)^{3} \cos{\left(\pi x \right)} \cos{\left(\pi y \right)} + 3.0 \left(x + y\right)^{2} \sin{\left(\pi y \right)} \cos{\left(\pi x \right)} + \left(4.0 \left(x + y\right)^{3} + 6.0 \left(x + y\right)^{2}\right) \sin{\left(\pi x \right)} \cos{\left(\pi y \right)}\right) & - 1.0 \pi \left(\pi \left(x + y\right)^{4} \sin{\left(\pi x \right)} \sin{\left(\pi y \right)} + \pi \left(x + y\right)^{3} \sin{\left(\pi x \right)} \sin{\left(\pi y \right)} - 4 \left(x + y\right)^{3} \sin{\left(\pi x \right)} \cos{\left(\pi y \right)} - 3 \left(x + y\right)^{2} \sin{\left(\pi y \right)} \cos{\left(\pi x \right)}\right)\end{matrix}\right]$

```python
', '.join([lua_code(fi) for fi in f2])
```

    'math.pi*(math.pi*(x + y)^3*(1.0*x + 1.0*y + 2.0)*math.cos(math.pi*x)*math.cos(math.pi*y) + 1.0*math.pi*(x + y)^3*math.cos(math.pi*x)*math.cos(math.pi*y) + 3.0*(x + y)^2*math.sin(math.pi*y)*math.cos(math.pi*x) + (4.0*(x + y)^3 + 6.0*(x + y)^2)*math.sin(math.pi*x)*math.cos(math.pi*y)), -1.0*math.pi*(math.pi*(x + y)^4*math.sin(math.pi*x)*math.sin(math.pi*y) + math.pi*(x + y)^3*math.sin(math.pi*x)*math.sin(math.pi*y) - 4*(x + y)^3*math.sin(math.pi*x)*math.cos(math.pi*y) - 3*(x + y)^2*math.sin(math.pi*y)*math.cos(math.pi*x))'

Oscillatory material
=================

```python
freq = symbols('freq')
f3 = force.subs(mu, 1 + 0.25*sin(freq*pi*x)).subs(lam, 1 + 0.25*sin(freq*pi*y)).simplify()
f3
```

$\displaystyle \left[\begin{matrix}\pi \left(0.5 \pi freq \sin{\left(\pi x \right)} \cos{\left(\pi y \right)} \cos{\left(\pi freq x \right)} + 1.0 \pi \left(0.25 \sin{\left(\pi freq x \right)} + 1\right) \cos{\left(\pi x \right)} \cos{\left(\pi y \right)} + \pi \left(0.5 \sin{\left(\pi freq x \right)} + 0.25 \sin{\left(\pi freq y \right)} + 3.0\right) \cos{\left(\pi x \right)} \cos{\left(\pi y \right)}\right) & - 1.0 \pi \left(- 0.25 \pi freq \sin{\left(\pi x \right)} \cos{\left(\pi y \right)} \cos{\left(\pi freq y \right)} - 0.25 \pi freq \sin{\left(\pi y \right)} \cos{\left(\pi x \right)} \cos{\left(\pi freq x \right)} + \pi \left(0.25 \sin{\left(\pi freq x \right)} + 1\right) \sin{\left(\pi x \right)} \sin{\left(\pi y \right)} + \pi \left(0.25 \sin{\left(\pi freq y \right)} + 1\right) \sin{\left(\pi x \right)} \sin{\left(\pi y \right)}\right)\end{matrix}\right]$

```python
', '.join([lua_code(fi) for fi in f3])
```

    'math.pi*(0.5*math.pi*freq*math.sin(math.pi*x)*math.cos(math.pi*y)*math.cos(math.pi*freq*x) + 1.0*math.pi*(0.25*math.sin(math.pi*freq*x) + 1)*math.cos(math.pi*x)*math.cos(math.pi*y) + math.pi*(0.5*math.sin(math.pi*freq*x) + 0.25*math.sin(math.pi*freq*y) + 3.0)*math.cos(math.pi*x)*math.cos(math.pi*y)), -1.0*math.pi*(-0.25*math.pi*freq*math.sin(math.pi*x)*math.cos(math.pi*y)*math.cos(math.pi*freq*y) - 0.25*math.pi*freq*math.sin(math.pi*y)*math.cos(math.pi*x)*math.cos(math.pi*freq*x) + math.pi*(0.25*math.sin(math.pi*freq*x) + 1)*math.sin(math.pi*x)*math.sin(math.pi*y) + math.pi*(0.25*math.sin(math.pi*freq*y) + 1)*math.sin(math.pi*x)*math.sin(math.pi*y))'

```python

```


# DCGAN for Image Generation

[](https://colab.research.google.com/github/EmoryMLIP/DeepGenerativeModelingIntro/blob/main/examples/DCGAN.ipynb)

## Some References

- Original paper on GANs by [Goodfellow et al. (2014)](https://arxiv.org/abs/1406.2661)
- Tutorial by [Goodfellow (2017)](https://arxiv.org/abs/1701.00160)
- We use a Deep Convolutional GAN proposed by [Salimans et al. (2016)](https://arxiv.org/abs/1606.03498)
- Section 5.1 of our paper  [Introduction to Deep Generative Modeling](https://arxiv.org/abs/2103.05180)

## Short Mathematical Description

In training we approximately solve the saddle point problem

\begin{equation}
\newcommand{\E}{\mathbb{E}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\bftheta}{\boldsymbol{\theta}}
\newcommand{\bfphi}{\boldsymbol{\phi}}
\newcommand{\calX}{\mathcal{X}}
\newcommand{\calZ}{\mathcal{Z}}
\newcommand{\bfx}{\mathbf{x}}
\newcommand{\bfz}{\mathbf{z}}
\newcommand{\bfy}{\mathbf{y}}
\min_{\bftheta}\max_{\bfphi} J_{\rm GAN}(\bftheta, \bfphi) = \E_{\bfx\sim \calX} \left[\log(d_{\bfphi}(\bfx)) \right] + \E_{\bfz\sim \calZ} \left[ \log\left(1 - d_{\bfphi}(g_{\bftheta}(\bfz))\right) \right].
\end{equation}

where

- $g_{\bftheta} : \R^2 \to \R^{784}$ is the generator (here a convolutional neural network)
- $d_{\bfphi} : \R^{784} \to \R^2$ is the discriminator (here a convolutional neural network)

```python
# install requirements  (Colab only)
import sys,os
if 'google.colab' in sys.modules:
    dgm_dir = '/content/DeepGenerativeModelingIntro/'
    if not os.path.exists(dgm_dir):
        !git clone git://github.com/EmoryMLIP/DeepGenerativeModelingIntro.git

    sys.path.append(os.path.dirname(dgm_dir))#%%
```

```python
import torch.nn.functional as F
import torch
import torch.nn as nn
import torchvision
import numpy as np
import matplotlib.pyplot as plt
from os import path

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

plt.rcParams.update({'font.size': 16})
plt.rcParams.update({'image.interpolation' : None})
# plt.rcParams['figure.figsize'] = [10, 6]
# plt.rcParams['figure.dpi'] = 200
```

## Prepare Image Data

```python
batch_size = 64
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST

img_transform = transforms.Compose([
    transforms.ToTensor()
])

train_dataset = MNIST(root='./data/MNIST', download=True, train=True, transform=img_transform)
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

test_dataset = MNIST(root='./data/MNIST', download=True, train=False, transform=img_transform)
test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

x,_ = next(iter(train_dataloader))
plt.Figure()
plt.imshow(torchvision.utils.make_grid(x,16,padding=5,pad_value=1.0).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.show()
```

## Select GAN Architecture

```python
class Generator(nn.Module):
    def __init__(self,w,q):
        """
        :param w: number of channels on the finest level
        :param q: latent space dimension
        """
        super(Generator, self).__init__()
        self.w = w
        self.fc = nn.Linear(q, w * 2 * 7 * 7)
        self.conv2 = nn.ConvTranspose2d(w * 2, w, kernel_size=4, stride=2, padding=1)
        self.conv1 = nn.ConvTranspose2d(w, 1, kernel_size=4, stride=2, padding=1)

        self.bn1 = nn.BatchNorm2d(w)
        self.bn2 = nn.BatchNorm2d(2*w)

    def forward(self, z):
        """
        :param z: latent space sample
        :return: g(z)
        """
        gz = self.fc(z)
        gz = gz.view(gz.size(0), self.w * 2, 7, 7)
        gz = self.bn2(gz)
        gz = F.relu(gz)
        gz = self.conv2(gz)
        gz = self.bn1(gz)

        gz = F.relu(gz)
        gz = torch.sigmoid(self.conv1(gz))
        return gz

class Discriminator(nn.Module):
    def __init__(self, w,useSigmoid=True):
        """
        Discriminator for GANs
        :param w: number of channels on finest level
        :param useSigmoid: true --> DCGAN, false --> WGAN
        """
        super(Discriminator, self).__init__()
        self.w = w
        self.useSigmoid = useSigmoid
        self.conv1 = nn.Conv2d(1, w, kernel_size=4, stride=2, padding=1)
        self.conv2 = nn.Conv2d(w, w * 2, kernel_size=4, stride=2, padding=1)
        self.fc = nn.Linear(w * 2 * 7 * 7, 1)
        self.bn1 = nn.BatchNorm2d(w)
        self.bn2 = nn.BatchNorm2d(2*w)

    def forward(self,x):
        """
        :param x: MNIST image or generated image
        :return: d(x), value of discriminator
        """
        x = (x-0.5)/0.5 #
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.leaky_relu(x,0.2)
        x = self.conv2(x)
        x = self.bn2(x)

        x = F.leaky_relu(x,0.2)
        x = x.view(x.shape[0],-1)
        x = self.fc(x)
        if self.useSigmoid:
            x = torch.sigmoid(x)
        return x

q = 2 # latent space dimension
width_disc = 32 # width of discriminator
width_dec = 32 # width of decoder

g = Generator(width_dec,q).to(device)
d = Discriminator(width_disc,useSigmoid=True).to(device)
```

## Train the Generator and Discriminator

We alternate between ADAM steps of the generator and the discriminator.

```python
optimizer_g = torch.optim.Adam(params=g.parameters(), lr=0.0002, betas=(0.5, 0.999))
optimizer_d = torch.optim.Adam(params=d.parameters(), lr=0.0002, betas=(0.5, 0.999))

his = np.zeros((0,3))

num_steps = 5 # number of training steps
plot_interval = 2  # plot solution every so many steps
init_g = "./results/VAEmnist-q-2-batch_size-%d-w_enc-%d-w_dec-%d-g.pt" % (batch_size,width_disc,width_dec) # path to .pt file that contains weights of a trained generator
out_file = "./results/DCGANmnist-q-2-batch_size-%d-w_disc-%d-w_dec-%d" % (batch_size,width_disc,width_dec)

retrain = False

if init_g is not None and path.exists(init_g):
    print("initialize g with weights in %s" % init_g)
    g.load_state_dict(torch.load(init_g ,map_location=torch.device(device)))

if retrain == False and path.exists(out_file + "-g.pt") and path.exists(out_file + "-d.pt") and path.exists(out_file + ".mat"):
    g.load_state_dict(torch.load("%s-g.pt" % (out_file),map_location=torch.device(device)))
    d.load_state_dict(torch.load("%s-d.pt" % (out_file),map_location=torch.device(device)))
    from scipy.io import loadmat
    his_file = loadmat("%s.mat" % (out_file))
    his = his_file['his']
    print("loaded results from %s" % out_file)

else:
    print((3*"--" + "device=%s, q=%d, batch_size=%d, num_steps=%d, w_disc=%d, w_dec=%d," + 3*"--") % (device, q, batch_size, num_steps, width_disc, width_dec))
    if out_file is not None:
        import os
        out_dir, fname = os.path.split(out_file)
        if not os.path.exists(out_dir):
            os.makedirs(out_dir)
        print((3*"--" + "out_file: %s" + 3*"--") % (out_file))
    print((4*"%7s    ") % ("step","J_GAN","J_Gen","ProbDist"))

    from epsTest import epsTest

    train_JGAN = 0.0
    train_JGen = 0.0
    train_epsTest = 0.0
    num_ex = 0

    def inf_train_gen():
        while True:
            for images, targets in enumerate(train_dataloader):
                yield images,targets

    get_true_images = inf_train_gen()

    for step in range(num_steps):
        g.train()
        d.train()
        # update discriminator using - J_GAN = - E_x [log(d(x)] - E_z[1-log(d(g(z))]
        x = get_true_images.__next__()[1][0]
        x = x.to(device)
        dx = d(x)
        z = torch.randn((x.shape[0],q),device=device)
        gz = g(z)
        dgz = d(gz)
        J_GAN = - torch.mean(torch.log(dx)) - torch.mean(torch.log(1-dgz))
        optimizer_d.zero_grad()
        J_GAN.backward()
        optimizer_d.step()

        # update the generator using J_Gen = - E_z[log(d(g(z))]
        optimizer_g.zero_grad()
        z = torch.randn((x.shape[0], q), device=device)
        gz = g(z)
        dgz = d(gz)
        # J_Gen = -torch.mean(torch.log(dgz))
        J_Gen = torch.mean(torch.log(1-dgz))
        J_Gen.backward()
        optimizer_g.step()

        # update history
        train_JGAN -= J_GAN.item()*x.shape[0]
        train_JGen += J_Gen.item()*x.shape[0]
        train_epsTest += epsTest(gz.detach(), x)
        num_ex += x.shape[0]

        if (step + 1) % plot_interval == 0:
            train_JGAN /= num_ex
            train_JGen /= num_ex

            print(("%06d   " + 3*"%1.4e  ") %
                  (step + 1, train_JGAN, train_JGen, train_epsTest))
            his = np.vstack([his, [train_JGAN, train_JGen, train_epsTest]])

            plt.Figure()
            img = gz.detach().cpu()
            img -= torch.min(img)
            img /=torch.max(img)
            plt.imshow(torchvision.utils.make_grid(img, 16, 5,pad_value=1.0).permute((1, 2, 0)))
            plt.title("trainDCGANmnist: step=%d" % (step + 1))
            if out_file is not None:
                plt.savefig(("%s-step-%d.png") % (out_file, step + 1))
            plt.show()

            train_JGAN = 0.0
            train_JGen = 0.0
            train_epsTest = 0.0
            num_ex = 0

    if out_file is not None:
        torch.save(g.state_dict(), ("%s-g.pt") % (out_file))
        torch.save(d.state_dict(), ("%s-d.pt") % (out_file))

        from scipy.io import savemat
        savemat(("%s.mat") % (out_file), {"his":his})
```

    initialize g with weights in ./results/VAEmnist-q-2-batch_size-64-w_enc-32-w_dec-32-g.pt
    loaded results from ./results/DCGANmnist-q-2-batch_size-64-w_disc-32-w_dec-32

## Convergence

We plot the GAN loss and the loss of the generator on the left.
Since we are solving a saddle point problem this plot can be difficult to read.
Therefore, we also plot the results of an $\epsilon$ test for equal distribution as an alternative way to meausre the quality of the generator (see [Szekely and Rizzo (2004)](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.226.377))

```python
plt.Figure()
plt.subplot(1,2,1)
plt.plot(his[:,0:2])
plt.legend(("JGAN","JGen"))
plt.title("GAN Objectives")

plt.subplot(1,2,2)
plt.plot(his[:,2])
plt.title("eps test")
```

## Show Samples

```python
ncol = 8
x, _ = next(iter(test_dataloader))
plt.Figure()
plt.subplot(1,2,1)
plt.imshow(torchvision.utils.make_grid(x,ncol,1,pad_value=1).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.title("original images")

z = torch.randn((64,q),device=device)
gz = g(z)
plt.subplot(1,2,2)
plt.imshow(torchvision.utils.make_grid(gz.cpu().detach(),ncol,1,pad_value=1).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.title("generated images")
plt.show()
```

## Interpolate in Latent Space

```python
z1 = -torch.ones(1,2,device=device)
z2 = torch.ones(1,2,device=device)
with torch.no_grad():

    lam = torch.linspace(0,1,20,device=device).reshape(-1,1)
    zinter = z1*(1-lam) + z2*lam

    gz = g(zinter)

plt.Figure()
plt.imshow(torchvision.utils.make_grid(gz.cpu(),len(lam),2,pad_value=1).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.show()
```


## Kálmán filter in 2D
GPS+IMU fusion

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm
from sympy import Symbol, Matrix
from sympy.interactive import printing
```

Example

```python
example = np.random.normal(12, 4, 500)
plt.hist(example, 20, density = True, label="example sensor readings")
e_mu, e_std = norm.fit(example)
print("mean:          ", e_mu)
print("std deviation: ", e_std)
xmin, xmax = plt.xlim()
e_x = np.linspace(xmin, xmax, 100)
p = norm.pdf(e_x, e_mu, e_std)
plt.plot(e_x, p, 'k', linewidth=2, label="norm")
plt.show()

```

```python

x = np.matrix([[0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]).T
n = x.size  # States
P = np.matrix(
    [
        [10.0, 0.0, 0.0, 0.0, 0.0, 0.0],
        [0.0, 10.0, 0.0, 0.0, 0.0, 0.0],
        [0.0, 0.0, 10.0, 0.0, 0.0, 0.0],
        [0.0, 0.0, 0.0, 10.0, 0.0, 0.0],
        [0.0, 0.0, 0.0, 0.0, 10.0, 0.0],
        [0.0, 0.0, 0.0, 0.0, 0.0, 10.0],
    ]
)
#print(P)

fig = plt.figure(figsize=(6, 6))
im = plt.imshow(P, interpolation="none", cmap=plt.get_cmap("BuGn"))
plt.title("Initial Covariance Matrix $P$")
ylocs, ylabels = plt.yticks()
# set the locations of the yticks
plt.yticks(np.arange(7))
# set the locations and labels of the yticks
plt.yticks(
    np.arange(6),
    ("$x$", "$y$", "$\dot x$", "$\dot y$", "$\ddot x$", "$\ddot y$"),
    fontsize=22,
)

xlocs, xlabels = plt.xticks()
# set the locations of the yticks
plt.xticks(np.arange(7))
# set the locations and labels of the yticks
plt.xticks(
    np.arange(6),
    ("$x$", "$y$", "$\dot x$", "$\dot y$", "$\ddot x$", "$\ddot y$"),
    fontsize=22,
)

plt.xlim([-0.5, 5.5])
plt.ylim([5.5, -0.5])

from mpl_toolkits.axes_grid1 import make_axes_locatable

divider = make_axes_locatable(plt.gca())
cax = divider.append_axes("right", "5%", pad="3%")
plt.colorbar(im, cax=cax)
plt.tight_layout()
```

```python
dt = 0.1  # Time Step between Filter Steps

A = np.matrix(
    [
        [1.0, 0.0, dt, 0.0, 1 / 2.0 * dt ** 2, 0.0],
        [0.0, 1.0, 0.0, dt, 0.0, 1 / 2.0 * dt ** 2],
        [0.0, 0.0, 1.0, 0.0, dt, 0.0],
        [0.0, 0.0, 0.0, 1.0, 0.0, dt],
        [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
        [0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
    ]
)
"""
fig = plt.figure(figsize=(6, 6))
im = plt.imshow(A, interpolation="none", cmap=plt.get_cmap("BuGn"))
plt.title("Matrix $A$")
divider = make_axes_locatable(plt.gca())
cax = divider.append_axes("right", "5%", pad="3%")
plt.colorbar(im, cax=cax)
plt.tight_layout()
"""
```

    '\nfig = plt.figure(figsize=(6, 6))\nim = plt.imshow(A, interpolation="none", cmap=plt.get_cmap("BuGn"))\nplt.title("Matrix $A$")\ndivider = make_axes_locatable(plt.gca())\ncax = divider.append_axes("right", "5%", pad="3%")\nplt.colorbar(im, cax=cax)\nplt.tight_layout()\n'

```python
H = np.matrix(
    [
        [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
        [0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
        [1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
        [0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
    ]
)
```

```python
ra = 10.0 ** 2
rp = 2.0 ** 2

R = np.matrix(
    [[ra, 0.0, 0.0, 0.0], [0.0, ra, 0.0, 0.0], [0.0, 0.0, rp, 0.0], [0.0, 0.0, 0.0, rp]]
)
```

```python
sj = 0.1
dts = Symbol("\Delta t")
Q = (
    np.matrix(
        [
            [(dt ** 6) / 36, 0, (dt ** 5) / 12, 0, (dt ** 4) / 6, 0],
            [0, (dt ** 6) / 36, 0, (dt ** 5) / 12, 0, (dt ** 4) / 6],
            [(dt ** 5) / 12, 0, (dt ** 4) / 4, 0, (dt ** 3) / 2, 0],
            [0, (dt ** 5) / 12, 0, (dt ** 4) / 4, 0, (dt ** 3) / 2],
            [(dt ** 4) / 6, 0, (dt ** 3) / 2, 0, (dt ** 2), 0],
            [0, (dt ** 4) / 6, 0, (dt ** 3) / 2, 0, (dt ** 2)],
        ]
    )
    * sj ** 2
)
"""
fig = plt.figure(figsize=(6, 6))
im = plt.imshow(Q, interpolation="none", cmap=plt.get_cmap("BuGn"))
plt.title("Matrix $Q$")
divider = make_axes_locatable(plt.gca())
cax = divider.append_axes("right", "5%", pad="3%")
plt.colorbar(im, cax=cax)
plt.tight_layout()
print(Q)
"""
```

    '\nfig = plt.figure(figsize=(6, 6))\nim = plt.imshow(Q, interpolation="none", cmap=plt.get_cmap("BuGn"))\nplt.title("Matrix $Q$")\ndivider = make_axes_locatable(plt.gca())\ncax = divider.append_axes("right", "5%", pad="3%")\nplt.colorbar(im, cax=cax)\nplt.tight_layout()\nprint(Q)\n'

Used sources:
- https://github.com/udacity/robot_pose_ekf
- https://github.com/TUMFTM
- https://github.com/cggos/imu_x_fusion
- https://github.com/PRBonn/LiDAR-MOS


# Ellipsometer Basics

**Scott Prahl**

*April 2020*

This notebook reviews the basic equations used in ellipsometry.  It also demonstrates how the Fresnel reflection equations are related to the ellipsometry parameter $\rho = \tan\psi \cdot e^{j\Delta}$.

References

* Archer, *Manual on Ellipsometry* 1968.
* Azzam, *Ellipsometry and Polarized Light*, 1977.
* Fujiwara, *Spectroscopic Ellipsometry*, 2007.
* Tompkins, *A User's Guide to Ellipsometry*, 1993
* Tompkins, *Handbook of Ellipsometry*, 2005.
* Woollam, *A short course in ellipsometry*, 2001.

```python
import sympy
import numpy as np
import matplotlib.pyplot as plt

import pypolar.fresnel as fresnel
import pypolar.ellipsometry as ellipse

sympy.init_printing(use_unicode=True)
```

## Ellipsometry

### Layout
A basic ellipsometer configuration is shown below where the lights hits the sample at an angle $\theta$ from a normal to the surface.  The  incident electric field $E_{ip}$ is parallel to the plane-of-incidence (which contains the incoming vector and the vector normal to the surface).  The electric field $E_{is}$ is perpendicular to the plane of incidence

Usually, the incident light $\mathbf{E}_i$ is linearly polarized but does not need to be.  The reflected light $\mathbf{E}_r$ is, in general, elliptically polarized.

$$
\mathbf{E}_i =
\begin{bmatrix}
E_{ip}e^{j\delta_{ip}}\\
E_{is}e^{j\delta_{is}}\\
\end{bmatrix}
\qquad\mbox{and}\qquad
\mathbf{E}_r =
\begin{bmatrix}
E_{rp}e^{j\delta_{rp}}\\
E_{rs}e^{j\delta_{rs}}\\
\end{bmatrix}
$$

### $\Delta$ and $\tan\psi$

The effect of reflection is characterized by the angle $\Delta$, defined as the change in phase, and the angle $\psi$, the arctangent of the factor by which the amplitude ratio changes.

$$
\Delta = (\delta_{rp} - \delta_{rs}) - (\delta_{ip}-\delta_{is})
$$

and

$$
\tan\psi = \frac{E_{is}}{E_{ip}} \cdot \frac{E_{rp}}{E_{rs}}
$$

In the special (but common case) of a smooth surface, there will be no mixing of parallel and perpendicular light, i.e.,

$$
\begin{bmatrix}
E_{rp}e^{j\delta_{rp}}\\
E_{rs}e^{j\delta_{rs}}
\end{bmatrix}
=
\begin{bmatrix}
r_p & 0 \\
0 & r_s
\end{bmatrix}
\begin{bmatrix}
E_{ip}e^{j\delta_{ip}}\\
E_{is}e^{j\delta_{is}}
\end{bmatrix}
=
\begin{bmatrix}
r_p E_{ip}e^{j\delta_{ip}}\\
s_s E_{is}e^{j\delta_{is}}
\end{bmatrix}
$$

the $\tan\psi$ equation simplifies to

$$
\tan\psi = \frac{r_p}{r_s}
$$

These overall change in field is often written as a single complex number $\rho$

$$
\rho = \tan\psi e^{j\Delta}
$$

Ellipsometry is the science of measuring and interpreting $\Delta$ and $\psi$ for a surface. A simple use of ellipsometry is to determine the complex index of refraction $m$ for a thick uniform flat substrate.  More elaborate ellisometric techniques allow one to determine one or more coatings on the substrate.

### Refractive index

When light reflects off a surface, the amplitude and phase of the electric changes.  These changes depend on the wavelength $\lambda$ of the light, the angle of incidence $\theta$, the complex refractive index of the material $m= n(1-j \kappa)$, and the polarization state of the incident beam:

* The plane of incidence contains the incident electric field propagation vector and the normal to the surface.

* The angle of incidence is the angle beween these two directions

* The real part of the refractive index $n$ determines the speed of light in the material

* The imaginary part of the refractive index $\kappa$ determines the light absorption of the material

* Linearly polarized light parallel to the plane of incidence is **p-polarized**.

* Linearly polarized light perpendicular to the plane of incident is **s-polarized**.

* The phase shift and amplitude change is different for p and s- polarized light.

* For dielectrics like glass, the amount of reflected light is determined by a single number, the index of refraction $n$.

* Semi-conductors and metals have a complex index of refraction  $m = n(1 - j \kappa)$.

### Fresnel Reflection

The Fresnel formulas for light incident from a vacuum onto a flat surface at an angle $\theta$ from the normal with refractive index $m$ varies with the orientation of the electric field.  The plane of incidence is defined as containing both the incident direction and the normal to the surface.  If the incident field is parallel to the plane of incidence then
$$
r_p=r_\parallel ={m^2\cos\theta-\sqrt{m^2-\sin^2\theta}\over
                        m^2\cos\theta+\sqrt{m^2-\sin^2\theta}}
$$
If the incident field is perpendicular to the plane of incidence then
$$
r_s=r_\perp ={\cos\theta-\sqrt{m^2-\sin^2\theta}\over
                    \cos\theta+\sqrt{m^2-\sin^2\theta}}
$$

### Fundamental Equation of Ellipsometry

Ellipsometers are used to determine the parameters $\psi$ and $\Delta$ which can be used to calculate $\rho$
$$
\rho = {r_p(\theta)\over r_s(\theta)} = \tan\psi \cdot e^{j \Delta}
$$

The graph below shows how $\psi$ and $\Delta$ vary with the incidence angle

### Determining the complex index of refraction

A convenient formula for an isotropic, uniform sample is given in
[Measurement of the Thickness and Refractive Index of Very Thin Films and the Optical Properties of Surfaces by Ellipsometry](https://archive.org/details/jresv67An4p363) so that when $\rho$ and $\theta$ are known, the complex index of refraction can be calculated.
$$
m = \tan\theta \sqrt{1-{4\rho\sin^2\theta\over (1+\rho)^2}}
$$

```python
def plot_rho(m):
    angles = np.linspace(0.001,90,91)
    rho = ellipse.rho_from_m(m,angles, deg=True)

    plt.plot(angles, np.abs(rho), color='red')
    plt.plot(angles, np.angle(rho),color='blue')

    plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
    plt.ylabel(r'$\tan\psi$ or $\Delta$')
    plt.title('Refractive Index %.3f%.3fj' % (m.real,m.imag))

    plt.xlim(0,90)
    plt.text(30, 3, r'$\Delta$', color='blue')
    plt.text(30, 0.8, 'tan($\psi$)', color='red')

m = 1.5
plot_rho(m)
plt.show()

m      = 1.5-1.0j
plot_rho(m)
plt.show()

m      = 3-3j
plot_rho(m)
plt.show()
```

Replicate figures 2.28 and 2.29 from Woollam's Short Course.

```python
def plot_rho2(m1,m2):
    angles = np.linspace(65,85,181)
    rho1 = ellipse.rho_from_m(m1,angles,deg=True)
    rho2 = ellipse.rho_from_m(m2,angles,deg=True)

    psi1 = np.degrees(np.arctan(np.abs(rho1)))
    psi2 = np.degrees(np.arctan(np.abs(rho2)))

    Delta1 = np.degrees(np.angle(rho1))
    Delta2 = np.degrees(np.angle(rho2))

    plt.subplots(2,1,figsize=(8,8))
    plt.subplot(2,1,1)

    plt.plot(angles, psi1, color='red')
    plt.plot(angles, psi2,color='blue')
    plt.ylabel(r'$\psi$ (degrees)')
    plt.title('%.3f%+.3fj and %.3f%+.3fj' % (m1.real,m1.imag,m2.real,m2.imag))
    plt.grid(True)

    plt.subplot(2,1,2)
    plt.plot(angles, Delta1, color='red')
    plt.plot(angles, Delta2,color='blue')
    plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
    plt.ylabel(r'$\Delta$ (degrees)')
    plt.grid(True)

m1 = 4.516 - 0.249j  # amorphous silicon
m2 = 3.875 - 0.023j  # crystalline silicon
plot_rho2(m1,m2)
plt.show()
```

## Extracting Refractive Index from $\rho$

First we will calculate $\rho$ for a known complex index of refraction.

```python
m = 1.5-1.3j
theta_i = np.linspace(1,89,15)
rho = ellipse.rho_from_m(m, theta_i, deg=True)

plt.plot(theta_i, np.abs(rho), 'o', color='red')
plt.plot(theta_i, np.angle(rho), 'o', color='blue')

plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
plt.ylabel(r'$\tan\psi$ or $\Delta$')
plt.title('Refractive Index %.3f%.3fj' % (m.real,m.imag))

plt.xlim(0,90)
plt.text(30, 3, r'$\Delta$', color='blue')
plt.text(30, 1.0, 'tan($\psi$)', color='red')

plt.show()
```

Then, we will see if we can recover $m$ using $\rho$ and $\theta_i$.
We will test with incidence angles from 1° to 89°.  We avoid 0° and 90° because these angles are either impossible or do not contain sufficient information to invert.

```python
m = 1.5-1.3j
theta_i = np.linspace(1,89,15)
rho = ellipse.rho_from_m(m, theta_i, deg=True)

m2 = ellipse.m_from_rho(rho, theta_i, deg=True)

plt.plot(theta_i, m2.real, 'o', color = 'blue')
plt.plot(theta_i, m2.imag, 'o', color='red')

plt.text(theta_i[0], m2[0].real-0.05, r'n', color='blue', va="top")
plt.text(theta_i[0], m2[0].imag+0.1, r'$n \kappa$', color='red')

plt.axhline(m.real,color='blue')
plt.axhline(m.imag,color='red')
plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
plt.ylabel('Real or Imaginary part of Refractive Index')
plt.title('Refractive Index = %.4f %.4fj'%(m.real,m.imag))

plt.show()
```

Quantitatively show that things work at an incidence angle of 70°

```python
theta_i = 70
m = 1.5 - 2.0j
rho = ellipse.rho_from_m(m, theta_i, deg=True)
print('Incidence angle  = %.1f°'%(theta_i))
print('rho              = %.3f%+.3fj'%(rho.real,rho.imag))
print('Refractive index = %.3f%+.3fj'%(m.real,m.imag))
print()
m2 = ellipse.m_from_rho(rho, theta_i, deg=True)
print('Extracted index  = %.3f%+.3fj using rho'%(m2.real,m2.imag))
```

    Incidence angle  = 70.0°
    rho              = 0.006+0.541j
    Refractive index = 1.500-2.000j

    Extracted index  = 1.500-2.000j using rho

```python

```


# Day 3. Interactive Controls

* [IPython Widgets](#1)
* [Bokeh](#2)

```python
from IPython.display import IFrame
from IPython.display import YouTubeVideo
from ipywidgets import interact, interactive, fixed
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
from cryptography.fernet import Fernet
import base64
def encrypt(string, key):
    keygen = lambda x : base64.urlsafe_b64encode(x.encode() + b' '*(32 - len(x)))
    cipher = Fernet(keygen(key))
    return cipher.encrypt(string.encode())
def decrypt(string, key):
    keygen = lambda x : base64.urlsafe_b64encode(x.encode() + b' '*(32 - len(x)))
    cipher = Fernet(keygen(key))
    return print(cipher.decrypt(string.encode()).decode())
```

# IPython Widgets

```python
IFrame(src='https://api.kaltura.nordu.net/p/310/sp/31000/embedIframeJs/uiconf_id/23449977/partner_id/310?iframeembed=true&playerId=kaltura_player&entry_id=0_18ipnucr&flashvars[streamerType]=auto&flashvars[localizationCode]=en&flashvars[leadWithHTML5]=true&flashvars[sideBarContainer.plugin]=true&flashvars[sideBarContainer.position]=left&flashvars[sideBarContainer.clickToClose]=true&flashvars[chapters.plugin]=true&flashvars[chapters.layout]=vertical&flashvars[chapters.thumbnailRotator]=false&flashvars[streamSelector.plugin]=true&flashvars[EmbedPlayer.SpinnerTarget]=videoHolder&flashvars[dualScreen.plugin]=true&&wid=0_m97dwa52', width=608, height=402)
```

## Task 1
Create an interactive function that returns the number of words in a text. The user interface should consist of a text area. Hint: use [split](https://docs.python.org/3/library/stdtypes.html#str.split).

```python
answer = 'gAAAAABcBE3nc21nV9chE1Qim80hFg5iHLw0-5jQcV1gtsFX4MN9_by8PWa7XcN80CGvShXFmh33AOaB6YCuTmTnNDGJmjQQJD4ip_bRIpdrRYSQTngImF4o4bi2Wv5fOLhk1dparYfliwieqb-yVi6b2jkTDhim6Ot9ipQ2hXgv9S_qFwweovdbN5VKr9zHgFCsrXDjQPRJ5WGrb7Ye55-1UhtkF2gHUvKhjIgxvzKI07qWfKTamLLVXns96RtSK6a6-gVQWlDS'

```

```python
decrypt(answer,'Skurup')
```

```python
@interact(text='hello')
def f(text):
    print(text)
```

```python
@interact(text='',)
def f(text):
    n = 0
    if len(text)>0:
        n = len( text.split(' ') )
    print('Number of words:',n)
```

## Task 2
Create an interactive function that returns the number of words and the number of characters in a text. The user interface should consist of a dropdown menu and a text area. Hint: use [split](https://docs.python.org/3/library/stdtypes.html#str.split) and str.replace().

```python
answer = 'gAAAAABcBFWm9q_kCXaTRs_seAM7b8XtKB182n80Hojlw-eyY0eqK-W9ZukhIeWf_4cP1XNEZ2273RH0Rcp3q59ex-sYl6kafEeJV0cJnIqnzt3h0BKYVFHH_g5qPhOFPT9Kf3sLf4ZX_pUU-jELD_7MTtI2yMeBhA9aTKg55ylQOOKb-YG6X9uD80VF1VdqUX2QpT0HJIAxaotyuTU_l5OCsr4ZbjbjNROXy-zU9UgfmMlBxvSH02wD-VJzWY0PB5ZkgK-oi0mJI5NhN7tWnauQ3Upn2S7AhlDqEjTHgfrMT2J8aV2PLgU3efkO_njxrZDATJuipghPaVbpPG-a5p-vz6IW_oseWsjQAUyAj1mWWZwAYSTbQrzd1Hz2o4ChxrrseY-vMW3JLA8oAZNsrmftmYxaIBhBMFXRcJ12yrVNV5RrBo6MnaqacI29Pnqb5vVkRH42QeQgkvcpQLPzWb8zzJDBPDXYXg=='

```

```python
decrypt(answer,)
```

## Task 3

Make an interactive plot of the following function with two sliding bars to control $\nu$ and $\Gamma$

\begin{equation}
I(t) = \cos{(2 \pi \nu t)} \exp{(-t 2 \pi \Gamma)}
\end{equation}

1. Define a function of $\nu$ and $\Gamma$ that plots $I(t)$
2. Use interactive to generate the sliders coupled to the parameters $\nu$ and $\Gamma$
3. Modify the description of the children (the two sliders)

Complete the following code

```.py
def plot_cos_decay(freq=1,gamma=.2):

    def cos_decay(time,freq,gamma):
        cos = np.cos(2*np.pi*freq*time)
        exp = np.exp(-time*2*np.pi*gamma)
        return exp*cos

    time = np.arange(0,10,.001)

    fig = plt.figure(figsize=(3.5,4))
    ax = plt.axes()
    ax.plot(,,color=plt.get_cmap('tab10')(3), lw=1)
    ax.set_ylabel('$I(t)$',fontsize=12)
    ax.set_xlabel(r'Time',fontsize=12)
    ax.set_xlim(0,10)

interactive_plot = interactive( )
interactive_plot.children[0].description=r'$f$' # slider
interactive_plot```

```python
answer = 'gAAAAABcBvmULdmxrFtMQ1AVeXU44jtvp8aF3jqh8G4ot7SRg6STnut7NJ7zKy57iriNHxyaHG0P6Ut8z5W5sC3QSvVLfh4C63Bg2xv2OLZJjjqUNsPRNv_KiAK1qU1y8qBJYAY_zvkLQcQyEozLf0bq_0DRpwIEmM_8Q6PTpq_K51D5rM2cDxloCh1xE_Lqysuo9PAclHsIbYR6aJVvoK2s8vD42Sx5O5QMNvxssbYgXvU2KZCazLYu9Y9Ks35E0reqmu77OORCQZjJKFpc_g7YfTj_nv3w7jeW43I17s8xENkGgcQtsXAM1UO-B6w8MtaNGtszv1uPaYVjJ-APl6nCaChqSCNLBFXLoxuGjOuLF5txCzRf8SiDRgHgEIx6Ni74AYRx8hftB53jGBwwNV12MjVis5l7gL2rH_6DOL3ltJiGobNUsZLoos0VKsNvZn49rnO7Z_fqWonq0gK8UfzKzceSJcMpufmDOH4KZW4wCB6iJSp0lzNQXTUxRt4_V99ky2DNrb7ReHeU-phWt1-vEeTD1o1GwT2NYSKAXEFIhjRszNkWwbIxLMxLOqlDKZhO-YcvG5KObDbIzKIsI3Sth0HLhKK4CPE0ZPL7uPWh6eQv-KmsvV_NMungPShD9185P5uYdbqZ-23BOdBtwCZdBWi7H-GgvmG_NpINSQ_jeLv3qgkNgEbQvbOkrzRDCYxWfneVH7z3U3sSU3zK_Qlnjheo4SaByUWGtRXRZsJgFDHrhoFPpAxh88ulpGjXUBhtBHc_ypfX6NGNxjPjpV1_vWYV4mookP-xgkSziaHCc1biCBT0K3N3WcCCDUj6Imy1ll3aT-eaMeA0rrr3OvLcNYobt08iC6JqX-XoJ62twDNJUfZOsuIzHdwVjq3q3p8pGcadfO1Vz81sFp3NabqoyZVof79UwW0AUChsYFvpqjwrz4xuA8Jftynyq8_n4anKwyUqen3BD3xt1epnWCl6QYMd5y9CDA=='

```

```python
decrypt(answer,)
```

## Task 4

Add an inset in the previous plot showing the Fourier Transform of the decaying cosine
\begin{equation}
I(\nu) = \int_0^\infty \mathrm{d}t \cos{(2 \pi \nu t)}\; I(t)
\end{equation}
The plot in the inset should also be coupled to the sliders.

```python
answer = 'gAAAAABcBvkEo2TnSlwfLSy7cy53MY3K7wBcVtKvQEww8lWVRrra5u4TWq7ro9o1IoBRq1V-CVPp5QIrBlFWv7XHClZtUkk14YzW9ChEKponzkQo7-L9xXUPSEMlqckftQAAxiF5FBz1Od4azI7bYwLFQb7RZSCG2MS82x1OEXKv9BfSp3yfN_x-gb5Cnl8QsCTlkun8nBJv5j6xIL-dLpfJU4IO5Pwifcsu_eJj4QDYlVNwB3u_6mWvQ5rWox2LTt8eJHJkz9GbGFxGylGhApKyPsbItb7KDrCxRgW4gh1w0IwEInF8vNbkUwlQoXjhpb0jL6f2cx4we_mVz_OfuePnW9R-gET0JXm7R9hW4yuSR9_r0cSSIZyNREqJDA4cLqAnkhEVupw6LSvG926oQp7dYtuyEUYmq1OgIQ2jW6CYFPSwYecTN-nxVdBnUgXnwOWiKK52-O9tQ46U167LOt78vVa4hJm5cKiJIjQtV1LzHuT4rC8nTAghUxKZLustFRKt72s3Hx7w5D2XuIkyig152nzWoGGJyoRC-kwS9y5-MfB5a1OMAdriTZIAjfbrYcBNZA9X41poZIdCrhwRgvxP_I3HC4Y8jgFHohN4wb0PwFhmrKFC5r1OaqqLpA4HMyCkywNPa9FgXhzGx0P8Fzy6pzJztmzeSPoBiZmCqxs3P0PUfi07IesCcXqE_-AizVUpkG6iZyP7V_INEd15niD9aY1g_kg8evb0qAf8UN57CBNs7A8naLPJ4P0n3xtazkLL84CGIAfQx6ivZv1HclHs92ogo3Ayv_eE4RNcUdq0vxO0761-wcO6RFfyNkKK5MnzWsb-a8tGMZUmgk04klLZ9XUbjNEPnRFWVQSapDfWkzsRNfTMHC-ajJ7QOlNaS13EWRou3QvhyvNcwxhw-oQ4pMpvkMa0liBfp1l8YI6X8r0HI-hyunfF7seHwbkkqry4FlojP8lIHuyqlcaqvYuZec5kU12Gm7Mync-4X1bXBjU0VE3qvZBJqHMHr_zs00Z_qXSXZkeyL1CqvXgzXLpomSYiSVqzh7Qzbo378uafjH_uCJBE_g_3zDgy2VCg4tB2tgbVCZ7GqGsCctLr70KQhy0i2StHUYUtTTug1z4CSdC852IRx_8VlREuDSBVlJas0vMLKJo2asUrcZwb62WEwH_3ru0nJLjJ8m_CK97mMmXnxkvQL_QZZGGReJRxI7rNZakI8g6xE3zKw4xSzgSZ0piFlv30-_IxuDnwOXpKXihv_M8CmJKAWLdslIhSK0eX67BuIcrfoQ0v2cmhVH252GK0Z6NeC3bspLFJzdIvG5a9vM9ZEDaEE7MHx-jaRNIBrjSWM1j7ivTohT6rXXLTVqGDUadsv8Yk1nbzFLxNpvl5gJ6CAUiKNIfxbY5FgIEEfvJOgofStaP8zGyOdpIwxTJv-kDwGeSfzpVEaLmiAlmadbst1mzsbpKHQr1BTRHtH2c9FeehuoSR85-Y5qjVNSRgAR5aXzqLTDATjO7T6uXQutHXP530-COsA0PlMoivnglelE-j9ghFntUPAsfrIr5m5pJGL5tX_xTUlMI5WJkC851RHGHXF_MY2KxnLWmDvSP0S38rucob7Hpq6RUqpZKzocKCmwaVLPaYdYYyBvGSI4T_09M9YJ_ZufaS4Fbab11M_1gFKn1cnsF7igRBULMHrhUkbV9l73jldb-Cbice9DXGUsa61xGbWvrj5a-KXZKfh7jJ09lZe5I8SaWozRM22MQDaUoG7I9QzesfZ_-fYb9QUJiSe256JK7jiBKddNc4fQF5G-QqY3DJ7bXLbH0C73R4t9qQI20VNRruPLwo2q9fccHgbP2VbHzB5Vv2MzeQoX2gM2lpz_7Wje7D1zPVmiKTXhHAI9tDZSrSnJo8EDCwCn1I1JE-7ijdy8GKRS_IVKnz'

```

```python
decrypt(answer,)
```

# Bokeh

```python
YouTubeVideo('oLU5eIO7b84')
```

## Task 5
Create an animated plot showing the data on CO$_2$ release from the [ICOS data portal](https://www.icos-cp.eu/) in Hohenpeißenberg (HPB), Hyltemossa (HTM), and Norunda (NOR).

In the following cells, some steps are already implemented:
1. We read the data sets into `pandas` DataFrames
2. We average the emissions over years, days, and minutes to obtain the average emission throughout a day, hour by hour, for the different months
3. We concatenate the DataFrames
4. We create a static Bokeh plot with a tooltip inspection tool. The stations are represented by circles on a map. The radius of the circle is proportional to the CO$_2$ emission.
Now let's add 2 slider widgets to control the month and the hour, following the approach shown in this [video lecture](https://youtu.be/oLU5eIO7b84).

```python
!cat data/ICOS_ATC_L2_L2-2018.1_NOR_32.0_462.CO2 | grep "LATITUDE"
!cat data/ICOS_ATC_L2_L2-2018.1_NOR_32.0_462.CO2 | grep "LONGITUDE"
!cat data/ICOS_ATC_L2_L2-2018.1_HPB_93.0_382.CO2 | grep "LATITUDE"
!cat data/ICOS_ATC_L2_L2-2018.1_HPB_93.0_382.CO2 | grep "LONGITUDE"
!cat data/ICOS_ATC_L2_L2-2018.1_HTM_70.0_461.CO2 | grep "LATITUDE"
!cat data/ICOS_ATC_L2_L2-2018.1_HTM_70.0_461.CO2 | grep "LONGITUDE"
```

```python
data_hpb = pd.read_csv('data/ICOS_ATC_L2_L2-2018.1_HPB_93.0_382.CO2',header=0,skiprows=38,sep=';')
data_hpb[data_hpb.co2<0] = np.nan
data_hpb.head()
```

```python
data_hpb = data_hpb[['Year','Month','Day','Hour','Minute','co2','SamplingHeight']]
data_hpb.set_index(['Year','Month','Day','Hour','Minute'],inplace=True)
data_hpb = data_hpb.groupby(['Month','Hour'],level=(1,3)).mean()
data_hpb.reset_index(inplace=True)
data_hpb['co2'] /= 10
data_hpb['lat'] = 47.8011
data_hpb['lon'] = 11.0246
data_hpb['station'] = 'Hohenpeißenberg'
data_hpb.head()
```

```python
data_nor = pd.read_csv('data/ICOS_ATC_L2_L2-2018.1_NOR_32.0_462.CO2',header=0,skiprows=38,sep=';')
data_nor[data_nor.co2<0] = np.nan
data_nor = data_nor[['Year','Month','Day','Hour','Minute','co2','SamplingHeight']]
data_nor.set_index(['Year','Month','Day','Hour','Minute'],inplace=True)
data_nor = data_nor.groupby(['Month','Hour'],level=(1,3)).mean()
data_nor.reset_index(inplace=True)
data_nor['co2'] /= 10
data_nor['lat'] = 60.0864
data_nor['lon'] = 17.4794
data_nor['station'] = 'Norunda'
data_nor.head()
```

```python
data_htm = pd.read_csv('data/ICOS_ATC_L2_L2-2018.1_HTM_70.0_461.CO2',header=0,skiprows=38,sep=';')
data_htm[data_htm.co2<0] = np.nan
data_htm = data_htm[['Year','Month','Day','Hour','Minute','co2','SamplingHeight']]
data_htm.set_index(['Year','Month','Day','Hour','Minute'],inplace=True)
data_htm = data_htm.groupby(['Month','Hour'],level=(1,3)).mean()
data_htm.reset_index(inplace=True)
data_htm['co2'] /= 10
data_htm['lat'] = 56.0976
data_htm['lon'] = 13.4189
data_htm['station'] = 'Hyltemossa'
data_htm.head()
```

```python
df = pd.concat([data_nor,data_hpb,data_htm])
df.set_index(['Month','Hour'],inplace=True)
df.sort_index(inplace=True)
df.head()
```

```python
def make_data(df,month,hour):
    data = df.loc[month,hour].copy()
    return data.reset_index()
```

```python
make_data(df,1,4)
```

```python
import geopandas as gpd
from bokeh.plotting import figure
from bokeh.io import show, output_notebook
from bokeh.palettes import Category10
from bokeh.models import GeoJSONDataSource, ColumnDataSource, HoverTool
from bokeh.transform import factor_cmap
from bokeh.models.widgets import Slider
from bokeh.layouts import column, row, WidgetBox

source = ColumnDataSource( data = make_data(df,1,4) )
```

```python
location = { 'HPB': {'lat':47.8011,'lon':11.0246},
           'NOR': {'lat':60.0864,'lon':17.4794} }

europe = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
geo_source = GeoJSONDataSource(geojson=europe.to_json())

p = figure(title="CO\u2082 Release in Europe", x_range=(-30,60), y_range=(30,85),
          tools='pan,reset')

p.patches('xs', 'ys', fill_alpha=0,
    line_color=Category10[10][7], line_width=1.5, source=geo_source)

glyph = p.scatter('lon','lat',
              source = source, size = 'co2', fill_alpha = .5, hover_fill_alpha = 1.0,
              marker = 'circle',
              legend_label = 'station',
              color = factor_cmap( 'station', 'Category10_5', source.to_df().station.unique() ),
              hover_line_color = factor_cmap( 'station', 'Category10_5', source.to_df().station.unique() ),
              hover_fill_color = factor_cmap( 'station', 'Category10_5', source.to_df().station.unique() ) )

glyph_hover = HoverTool(renderers=[glyph],
                         tooltips=[('Longitude','@lon E'), ('Latitude','@lat N'), ('Sation','@station'),
                                  ('Month','@Month'), ('Hour','@Hour'), ('CO2','@co2'),
                                  ('Sampling Height','@SamplingHeight')])
p.add_tools(glyph_hover)

p.title.align = 'center'
p.legend.location = 'top_left'

output_notebook()
show(p)
```

```python
answer = 'gAAAAABcBvoKW3LiJnuzl-rUkXnWXDyKdCa1TtA-6Zp5eHeGiSp42LcPzqyChSRfFel5HeESU-z8_OLoZp2MFw8VAOWoCNSkIacf758tpY_HOggr6j8rYC9e1pTfzus0HXEsdOBReP58YQ8eLw63dR-la2nC4TcRXezGAQ2IKOGav8WM11nMlkhNXPV126xh4jsBT0nRxmNUaQejldIJC_q4WPDsWuQAHl3b5e-sraepPnmkONRTq4VqlSrFpd62CBsmCo8KbOuCRXMjBHv_xCmt34htey7DQ65WnNauZqOMLpPze_6d58jjA3hzFth9oqEspzsP5J_VrK55vqdB8LZM1dnu7g97QEVWWe2XsKVpXEa2TXPUqACN4AKlhfSy-zfS78v5qWzgFZLscMHa2ifgFZGgPXoWVRhcxLq5AL644559F8g1jCAGnXNaYmMXoUZ6xZCdMFliYiFuL7V8dhEtaHawX441rIEtoIb2XS0ZK0lPTav5eIHhceQLuTm4lCmppAXvkLkrJnkBzlom3bYPZb00whL4nyMN39c5el4TNo5Sf-v-3GZl4uSWubhh5pPE3opFA8NC444CBvU9VBn2ZklTyeziByeKhdCOYHlKXECXWKRqAEK0ScBuZOTFNm83Kj18XcXmEt5-v2vjMGfM-rVSAcPyBNU3gOKhnFYNbtNgwL69F3ehIBQlkH5rGkgvilbmlgcJx0hPS3aVyFybxN9xGUlFcSs8DmDdGF5hxAs4vZWNgy_PjAXNeW-hdMUGY3wIU3JPC96DNIBTyMQV_BtzGVgpNm4kJ8M7ljNq563pVnf0q3vXwPSYnFUS6-aL0GarW47U2MQzCg8VerMKMR5-WTfNcz0euna78dg7hb_ZqeJgmESiL0gqFQdHNMriGMNJeLfd-sMlgI1WBLV9TSeFqbbOoar0IjzzzmoRmla8WFrSULbK0zMGz_DdiQOLQCWoEBRVDkwcVPmobdYaS4xy2DE2w_rBQOkCM68k-sNqg04_hZ652Pq7KnOAG1BwL3D4Pz73ZaE-uDL1vbnr6auI7dsxcPgJX3_b0pIqIjwcPwd0iaoU2SdEVU2OpGBnY6KsSl5MrwX7P-dYVhWwJ4FfYP59l7HfTJrw7t03jkA399adGWF70wA7v-2bHHzCTf66wgCnFmQtLL8ukmrkImiAzCMA3af47hq8GPKbFK74UsnwyjabLv26N0WZcuc2P3Tib-qMtc0ebuHTXWFtmio0RSDCjKVOehakMwoeIKfIAWZAoYfAg5NFQWa8Xw8ybsHvwCrDwt4j7qb-hdoyIQ2cMEFfXXx4ZcL-GRJy6jlwyP0C90JjvWv8ono8ffdhuuSD_ylCHCt0c58UxibJXUkvHzgA40rlWRH7Pmh6iw1zeG5lkxkHZFt6AaKpxNhtu0WDsHLUxqMoa1PrjJjjP1VwjVRKpeaIWMui9j-Wg3gQB0Bt9b7iqvgPW8R5OLAXOm_84E8ozBFn1DWdJaMyutqgh3SsHiL_bOlWeJfm9-G_YeVaXgjsj4sSAf01VzBxIe_bdgY44OUiobZ5aoF0O4Ya7AVMhXPeJHi8f4GVeacI7KJAiGFH3Vmm_X-cgM8axMPthxtC7vBYgw_a4TOikWYewPh5uU9f4mjs4rKS9xC0H8tjp5ICrDhdYFWqviVnnuunSWR9bmbRKYVIsmIJHYZLZpci6bOz8jKVyxjp4YgD9-3IYe5EPy_6uF9XbNZesyeJkeY_q5P1qt2iigQFi8hD88lt4nvFjV5yXdGwfeyLIm0YaS_SE9mAu2NnQgejRKrcsmahQAvYUy64DD38ktJevblcNRy90xMd6GXHr69ZPVz65QsZAroqHy_GXjjQhJpRS-aajoL6LcolcFvrvzq46Hv_JIscJscnPHkqwWbKgS1QLD4HdUC_VVK_9GmfbjgI8Y05hOEPF2c_0Nhm1-F3dkTBrIXLgHnd88FHHU9feobDBNerd9_sqQctTE7d2FHxrcvNZRVlhHZiFyFuer1cz29keRSJm0Gp035WFhieYECzaM-_7NMbRkUdwAf3yxzUjNy1WKyehC-ZawcXwsyTOVvMugfHZ4olSImjGvVKsd7Ho5oTip2fZEv9ic2GdGFxnzP3bhpBhp4fLj8QeDEKYkeGNmDumo-yPJo_NVr30-rNUdeNDIQtOKB8rvv-HKFsz8t_nvctKEMOpe4Iq0nI1QadWCQtkyUAG7kIfsby6jtgANtyLjddZ5HZAvpbpros4jr3fanKxr4Qzr8kDzOw09oid1it8SetOWnBM9YzEbHf2-Rm1ZFhkAfjaf0_wSGBeMXinAr66Gk11pTmCOHqOFzSZRcE6-bGkSBJXe4gf17EY9Ot8S2CjDNqSO9s1eDSh5Xrw_SbPq9OCYDk58CY1RihwEr6O6u_tdpB-i-PEaPUc1te-Sk9dOSLrRE97K-AHjjW1Zxb8lEL4k2d9-NQUUg0jACC7eYACWOgoUwnoq3QrYR-lbhoM0EZSLIEjuP5fhPQWMpVUQ3iaXTtl-j1cfXjDT8_bPqE4oD2IdO_MIaiAV0NsBjM9s5MR_w2ssJ9vexRnaQ92k7pL4OTjMQ4Cgu0IriH7roFIil1UQAaH-gZFWnpKYJrD6zIAQrdgZ_Cb2LNSwgcybA7SFfzFbAOitp5foBGlHr1UtGdlN30SmKQ105mLeBHE3zFdqP4SazKCL1Gyo6XNGyS_n_Qj-yYl66GLdegIBNtQMY6apVCpAHoSCzAXzEHw2ARW3LIZMjjeJn5UFlmAyuW7elMGV-rKSdxlszOXAryXBgl3GIbiYXAnZ_LL04EqXydNaDdtNL4r6_cQnMw3ykTLC6HW997onodrnWuVRswO6KR_0kJ97KUwPULxs1Q8ANcMoV4MsLqDsDPJBi1txw0PJKfzONAun4Wbbj5w-NDn3yckR7cakFieDcWqblAMdAoyFw-Cx7JZyuqWXaQQh6PzD3k2FOIQoH8qUegVabq7iW1bZt_xgv6HxGixf_u_RvQ3ViL8nsL_SiPGBTqC_sjVKlXkO9JJZd3DBOWgt-tG7ft99QacBpfz3n6L6j7c759G7uIspqCRD2PvpXcykk9MRPMzBPe7uzFkir4z6I61wnwFaxurKo6Sps_DHGAQZuYC3PemsWh9yNjVruTV0ZNTwlG-PgEbGNfI_eiIt7K0Mw6BjCY7co5SyPPP6KSU64dLPjjaO--0F1Uc-7mzdjy5PyfalG4FnPymXnhLg6_vHjtLgtw1Vh_jL2Bwda6hc_Z-zU2ZXvsAe1dkvgEdhJ5W0MSzHPi3qo8YLgpVw5UAwTWqzI-INJpMqh7CZOkNP31qLI7x1gpV2hWGM-UzW3FGuJdKMPpwEE5ppDlbXgM3zscGiNYTqVlpl58MKoh7OCFFGhxM1aaX4tLIEonRCEH-UapKpqU-mArBIFM0AHy5zLeHHal72jBpQC4OlTf4F8cY_sFCf52SjcayIAR4wlvI6yQKDOSTWS1nEl5JrAuWzR4RIBVtuT_ohT_Gw-8i1CzocFLi2e_CWoHWBvrkAcOI9RiyiidmhyaJrQdfmSdpEFC-Kv5tlTuKxypxP5Rwi4vCh-rvYAOpu_mucOhzKEshoTXunlfarmsJiFk5neSEuZ20LmsVi_Nt7S9HNwgM5RA-k5xxxm-sLzMLCFWwAuHU9vZjhH5s44So3xyGhssI9uyGCJC4QA0M41ihZUOmcQKeEighxgXvVmY'
```

```python
decrypt(answer,)
```