# Assignment 3 - basic classifiers

Math practice and coding application for main classifiers introduced in Chapter 3 of the Python machine learning book.

## Weighting

Note that this assignment is more difficult than the previous ones, and thus has a higher weighting 3 and longer duration (3 weeks). Each one of the previous two assignments has a weighting 1.

Specifically, the first 3 assignments contribute to your continuous assessment as follows:

Assignment weights: $w_1 = 1, w_2 = 1, w_3 = 3$

Assignment grades: $g_1, g_2, g_3$

Weighted average: $\frac{1}{\sum_i w_i} \times \sum_i \left(w_i \times g_i \right)$

Future assignments will be added analogously.

# RBF kernel (20 points)

Show that a Gaussian RBF kernel can be expressed as a dot product:
$$
K(\mathbf{x}, \mathbf{y})
= e^\frac{-|\mathbf{x} - \mathbf{y}|^2}{2}
= \phi(\mathbf{x})^T \phi(\mathbf{y})
$$
by spelling out the mapping function $\phi$.

For simplicity
* you can assume both $\mathbf{x}$ and $\mathbf{y}$ are 2D vectors
$
x =
\begin{pmatrix}
x_1 \\
x_2
\end{pmatrix}
, \;
y =
\begin{pmatrix}
y_1 \\
y_2
\end{pmatrix}
$
* we use a scalar unit variance here

even though the proof can be extended for vectors $\mathbf{x}$ $\mathbf{y}$ and general covariance matrices.

Hint: use Taylor series expansion of the exponential function

## Answer
We denote $e^x$ as exp($x$). Since $
\mathbf x =
\begin{pmatrix}
x_1 \\
x_2
\end{pmatrix}
, \;
\mathbf y =
\begin{pmatrix}
y_1 \\
y_2
\end{pmatrix}
$, we have

$$
\begin{align}
K(\mathbf{x}, \mathbf{y}) = \text{exp}(\frac{-||\mathbf{x} - \mathbf{y}||^2}{2}) & =  \text{exp}(\frac{-||(x_1-y_1, x_2-y_2)||^2}{2} )\\
& = \text{exp}(\frac{-(x_1-y_1)^2-(x_2-y_2)^2}{2})\\
& = \text{exp}(\frac{-{x_1}^2-{x_2}^2-{y_1}^2-{y_2}^2+2 x_1 y_1+2 x_2 y_2}{2}) \\
& = \text{exp}(\frac{-||\mathbf{x}||^2}{2}) \text{ exp}(\frac{-||\mathbf{y}||^2}{2}) \text{ exp}(x_1 y_1 + x_2 y_2)
\end{align}
$$

By Taylor series of $f(x)$ on $a$, $e^x$ at $a=0$ can be expressed as $\sum_{n=0}^{\infty} \frac{x^n}{n!} = 1 + x + \frac{x^2}{2!} +$ ... for $x \in \mathbb{R}^1$. Therefore,

$$K(\mathbf{x}, \mathbf{y}) = \text{exp}(\frac{-||\mathbf{x}||^2}{2}) \text{ exp}(\frac{-||\mathbf{y}||^2}{2}) \sum_{n=0}^{\infty} \frac{(x_1 y_1 + x_2 y_2)^n}{n!}$$

By binomial expansion, we have

$$ (x_1 y_1 + x_2 y_2)^n = \sum_{i = 0}^{n} \binom{n}{i} (x_1 y_1)^{n-i} (x_2 y_2)^i = \sum_{i = 0}^{n} \sqrt{\binom{n}{i}} (x_1^{n-i} x_2^i) \sqrt{\binom{n}{i}} (y_1^{n-i} y_2^i)$$

We let $\xi_{n}(\mathbf{x}) = \xi_{n}(x_1, x_2) = \left[\sqrt{\binom{n}{i}} (x_1^{n-i} x_2^i) \right] = \left[\sqrt{\binom{n}{0}} (x_1^{n} x_2^0), \sqrt{\binom{n}{1}} (x_1^{n-1} x_2^1), ..., \sqrt{\binom{n}{n}} (x_1^{0} x_2^n)\right] \in \mathbb{R}^{\text{n}}$.

Hence we have

$$ \left\{
\begin{aligned}
\phi(\mathbf{x}) & = \text{exp}(\frac{-||\mathbf{x}||^2}{2}) \left[1, \frac{\xi_{1}(\mathbf{x})}{\sqrt{1!}}, \frac{\xi_{2}(\mathbf{x})}{\sqrt{2!}} , \frac{\xi_{3}(\mathbf{x})}{\sqrt{3!}} , ... \right]^T \\
\phi(\mathbf{y}) & = \text{exp}(\frac{-||\mathbf{y}||^2}{2}) \left[1, \frac{\xi_{1}(\mathbf{y})}{\sqrt{1!}}, \frac{\xi_{2}(\mathbf{y})}{\sqrt{2!}} , \frac{\xi_{3}(\mathbf{y})}{\sqrt{3!}} , ... \right]^T
\end{aligned}
\right.
$$

The mapping function is therefore $\phi(\mathbf{x}) = \text{exp}(\frac{-||\mathbf{x}||^2}{2}) \left[1, \frac{\xi_{1}(\mathbf{x})}{\sqrt{1!}}, \frac{\xi_{2}(\mathbf{x})}{\sqrt{2!}} , \frac{\xi_{3}(\mathbf{x})}{\sqrt{3!}} , ... \right]^T$, where $\xi_{n}(\mathbf{x}) = \left[\sqrt{\binom{n}{0}} (x_1^{n} x_2^0), \sqrt{\binom{n}{1}} (x_1^{n-1} x_2^1), ..., \sqrt{\binom{n}{n}} (x_1^{0} x_2^n)\right]$.

# Kernel SVM complexity (10 points)

How would the complexity (in terms of number of parameters) of a trained kernel SVM change with the amount of training data, and why?
Note that the answer may depend on the specific kernel used as well as the amount of training data.
Consider specifically the following types of kernels $K(\mathbf{x}, \mathbf{y})$.
* linear:
$$
K\left(\mathbf{x}, \mathbf{y}\right) = \mathbf{x}^T \mathbf{y}
$$
* polynomial with degree $q$:
$$
K\left(\mathbf{x}, \mathbf{y}\right) =
(\mathbf{x}^T\mathbf{y} + 1)^q
$$
* RBF with distance function $D$:
$$
K\left(\mathbf{x}, \mathbf{y} \right) = e^{-\frac{D\left(\mathbf{x}, \mathbf{y} \right)}{2s^2}}
$$

## Answer

For all examples, we assume $\mathbf{x}, \mathbf{y} \in \mathbb{R}^\text{d}$.

### Linear:
For linear kernal, the mapping function $\phi(\mathbf{x}) = \mathbf{x}$, which mapps $\mathbb{R}^\text{d}$ to $\mathbb{R}^\text{d}$, therefore the size of data is unchanged.

There are not explicit parameters, therefore the time cost increase linearly with the dimension of data, or the amount of data increase $n$ times, the time cost simply increase $O(n)$ time. Both changes in dimension or data amount will not afftect any parameters.

### Polynomial with degree $q$:
For simplicity we write $1 = x_{d+1} y_{d+1}$. Then

$$K\left(\mathbf{x}, \mathbf{y}\right) =(\mathbf{x}^T\mathbf{y} + 1)^q = (\sum_{i=1}^{d+1} x_i y_i)^q = \sum_{k_1 + k_2 + ... + k_{d+1} = q} \binom{q}{k_1, k_2, ..., k_{d+1}} \prod_{t=1}^{d+1} (x_t y_t)^{k_t} = \sum_{\sum_{i=1}^{d+1} k_i = q} \frac{q!}{\prod_{i=1}^{d+1} k_i!} \prod_{t=1}^{d+1} (x_t y_t)^{k_t}$$

by Multinomial theorem. Therefore the mapping function is

$$\phi(\mathbf{x}) = \left[\sqrt{\frac{q!}{\prod_{i=1}^{d+1} k_i!}} \prod_{t=1}^{d+1} (x_t)^{k_t}\right]_{\sum_{i=1}^{d+1} k_i = q}^T,$$

which maps $\mathbb{R}^\text{d}$ to $\mathbb{R}^\binom{p+(d+1)-1}{(d+1)-1} = \mathbb{R}^\binom{p+d}{d} = \mathbb{R}^\frac{(p+d)!}{p! d!}$, computed using the stars and bars method.

* If $p=1$, only one useless dimension is added, where $x_{d+1} = 1$. In this case the actual dimension remains.

* If $p>2$, then the dimension increases from $d$ to $\binom{p+d}{d}$, where actural dimension is $\binom{p+d}{d} - 1$ since we always have a $x_{d+1}^q = 1$ term.

Now we consider the parameters.
* For each entry in $K\left(\mathbf{x}, \mathbf{y}\right)$, we have a parameter $\frac{q!}{\prod_{i=1}^{d+1} k_i!} = \binom{q}{k_1, k_2, ..., k_{d+1}}$, which takes $O(q \prod_{t=1}^{d+1} k_t)$ to compute in brute force. Considering the dimension analysis we discuss above, the greater the dimension or the greater $q$ is, the more parameters and greater time complexity we will have in the kernal function.

* However, since $q$ and $k_i$ are identical for any set of input data, increasing amount of data will not change number of parameter to be calculated (because they only need to be calculated once), although multiplying them to each term of $x$ and $y$ takes constant time.

* If we do $\mathbf{x}^T \mathbf{y} + 1$ first and then do the power function, then the parameter analysis is the same as the linear function, except that we need an extra power $q$ after the $\mathbf{x}^T \mathbf{y} + 1$.

### RBF with distance function $D$:

Assume $D(\mathbf{x}, \mathbf{y}) = \omega(\mathbf{x}) \omega(\mathbf{y})$. For $K(\mathbf{x}, \mathbf{y} ) = e^{-\frac{D\left(\mathbf{x}, \mathbf{y} \right)}{2s^2}} = e^{-\frac{\omega(\mathbf{x}) \omega(\mathbf{y})}{2s^2}} = e^{-\frac{1}{2s^2} \omega(\mathbf{x}) \omega(\mathbf{y})}$, we have the mapping function as

$$
\phi(\mathbf{x}) = e^{-\frac{1}{4s^2}} \left[1, \frac{\omega(\mathbf{x})}{\sqrt {1!}}, \frac{\omega(\mathbf{x})^2}{\sqrt {2!}}, \frac{\omega(\mathbf{x})^3}{\sqrt {3!}}, ... \right]^T,
$$

which maps $\mathbb{R}^\text{d}$ to $\mathbb{R}^\infty$. That is, RBF essentially projects the original vector to an infinite dimensional space.

Now we consider the parameters.
* We first clarify that although RBF maps $\mathbb{R}^\text{d}$ to $\mathbb{R}^\infty$, the dimension actually used is determined by the explicit function $K(\mathbf{x}, \mathbf{y})$, because we don't have to separate the mapping function. Instead, we can just compute the kernal $K$ directly.

* As calculating exp($\mathbf{x}$) is simply mapping to an exponential function, the main cost in terms of dimension of data is at the distance function $D(\mathbf{x}, \mathbf{y})$, which varies with different distance functions we choose. For example, if we choose simple metrics such as Taxicab distance and Euclidean distance, the cost is relatively small. However, if we choose complex metrics for some reason, then the time cost could be huge.

* Also, if all input data share the same set of parameters, then the parameters only need to be computed once, and applyed to each set of input data with constant time. However, if parameters change with different sets of input data in a specific kernal, then the number of parameters as well as time complexity also increase linearly with the increase of amount of data.

# Gaussian density Bayes (30 points)

$$
p\left(\Theta | \mathbf{X}\right)
=
\frac{p\left(\mathbf{X} | \Theta\right) p\left(\Theta\right)}{p\left(\mathbf{X}\right)}
$$

Assume both the likelihood and prior have Gaussian distributions:

$$
\begin{align}
p(\mathbf{X} | \Theta)
&=
\frac{1}{(2\pi)^{N/2}\sigma^N} \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)
\\
p(\Theta)
&=
\frac{1}{\sqrt{2\pi}\sigma_0} \exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right)
\end{align}
$$

Derive $\Theta$ from the dataset $\mathbf{X}$ via the following methods:

### ML (maximum likelihood) estimation
$$
\Theta_{ML} = argmax_{\Theta} p(\mathbf{X} | \Theta)
$$

### MAP estimation
$$
\begin{align}
\Theta_{MAP}
&=
argmax_{\Theta} p(\Theta | \mathbf{X})
\\
&=
argmax_{\Theta} p(\mathbf{X} | \Theta) p(\Theta)
\end{align}
$$

### Bayes estimation

$$
\begin{align}
\Theta_{Bayes}
&=
E(\Theta | \mathbf{X})
\\
&=
\int \Theta p(\Theta | \mathbf{X}) d\Theta
\end{align}
$$

## Answer

### 1. ML (maximum likelihood) estimation

To maximize $p(\mathbf{X} | \Theta)$, we set $\nabla_\Theta p(\mathbf{X} | \Theta) = \nabla_\Theta \left(\frac{1}{(2\pi)^{N/2}\sigma^N} \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)\right) = 0$.

By Chain rule we get

$$
\begin{align}
\nabla_\Theta p(\mathbf{X} | \Theta) & = \frac{1}{(2\pi)^{N/2}\sigma^N} \frac{\partial \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)}{\partial \left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)} \frac{\partial \left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)}{\partial \Theta}\\
0 & = \frac{1}{(2\pi)^{N/2}\sigma^N} \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right) \left( - \frac{\sum_{t=1}^N -2(\mathbf{x}^{(t)} - \Theta)}{2\sigma^2} \right) \\
\end{align}
$$
Note that $p(\mathbf{X} | \Theta) = \frac{1}{(2\pi)^{N/2}\sigma^N} \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)$ is non-zero, because $e^y$ is always positive for $y \in \mathbb{R}$, and the constant $\frac{1}{(2\pi)^{N/2}\sigma^N}$ is positive.

Then we have

$$
\begin{align}
0 & = \frac{\sum_{t=1}^N 2(\mathbf{x}^{(t)} - \Theta)}{2\sigma^2} = \frac{\sum_{t=1}^N \mathbf{x}^{(t)} - \Theta}{\sigma^2} \\
0 & = \frac{(\sum_{t=1}^N \mathbf{x}^{(t)}) - N\Theta}{\sigma^2} \\
N\Theta & = \sum_{t=1}^N \mathbf{x}^{(t)} \\
\Theta & = \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}
\end{align}
$$
Hence $$\Theta_{ML} = argmax_{\Theta} p(\mathbf{X} | \Theta) = \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}.$$

### 2. MAP estimation
To maximize $p(\mathbf{X} | \Theta) p(\Theta)$, we set $\nabla_\Theta (p(\mathbf{X} | \Theta) p(\Theta)) = p(\Theta)\nabla_\Theta p(\mathbf{X} | \Theta) + p(\mathbf{X} | \Theta)\nabla_\Theta p(\Theta) = 0$.

We get $p(\Theta)\nabla_\Theta p(\mathbf{X} | \Theta) = - p(\mathbf{X} | \Theta)\nabla_\Theta p(\Theta) $ and therefore

$$
\begin{align}
\frac{1}{\sqrt{2\pi}\sigma_0} \exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right) \nabla_\Theta \left(\frac{1}{(2\pi)^{N/2}\sigma^N} \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)\right) & \\
= - \frac{1}{(2\pi)^{N/2}\sigma^N} \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right) \nabla_\Theta & \left(\frac{1}{\sqrt{2\pi}\sigma_0} \exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right)\right)
\end{align}
$$

By Removing the constants we get

$$
\begin{align}
\exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right) \nabla_\Theta \left(\exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)\right) & = - \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right) \nabla_\Theta \left(\exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right)\right) \\
\end{align}
$$

By Chain rule we get

$$
\begin{align}
\exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right) \frac{\partial \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)}{\partial \left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)} \frac{\partial \left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)}{\partial \Theta} & = - \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right) \frac{\partial \exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right)}{\partial \left(-\frac{(\Theta - \mu_0)^2}{2\sigma_0^2}\right)} \frac{\partial \left(-\frac{(\Theta - \mu_0)^2}{2\sigma_0^2}\right)}{\partial \Theta}\\
\exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right) \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right) \left( - \frac{\sum_{t=1}^N -2(\mathbf{x}^{(t)} - \Theta)}{2\sigma^2} \right) & = -\exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right) \exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right) \left(-\frac{2(\Theta - \mu_0)}{2\sigma_0^2}\right)
\end{align}
$$

Since $e^y$ is always positive for $y \in \mathbb{R}$, the first two terms of both sides are non-zero. Dividing them on both sides we have

$$
\begin{align}
- \frac{\sum_{t=1}^N -2(\mathbf{x}^{(t)} - \Theta)}{2\sigma^2} & = \frac{2(\Theta - \mu_0)}{2\sigma_0^2}\\
\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)}{\sigma^2} & = \frac{\Theta -\mu_0}{\sigma_0^2}\\
\frac{(\sum_{t=1}^N \mathbf{x}^{(t)}) - N\Theta}{\sigma^2} & = \frac{\Theta -\mu_0}{\sigma_0^2}\\
(\sum_{t=1}^N \mathbf{x}^{(t)})\sigma_0^2 - N\Theta\sigma_0^2 & = \sigma^2 \Theta - \sigma^2 \mu_0 \\
(\sum_{t=1}^N \mathbf{x}^{(t)})\sigma_0^2 + \sigma^2 \mu_0 & = (N\sigma_0^2 + \sigma^2)\Theta \\
\end{align}
$$

Hence

$$
\begin{align}
\Theta_{MAP} & = argmax_{\Theta} p(\Theta | \mathbf{X}) \\
& = argmax_{\Theta} p(\mathbf{X} | \Theta) p(\Theta) \\
& = \frac{(\sum_{t=1}^N \mathbf{x}^{(t)})\sigma_0^2 + \sigma^2 \mu_0}{N\sigma_0^2 + \sigma^2} \\
\end{align}
$$

Furthurmore, since $\Theta_{ML} = \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}$, we have

$$\Theta_{MAP} = \frac{(\sum_{t=1}^N \mathbf{x}^{(t)})\sigma_0^2 + \sigma^2 \mu_0}{N\sigma_0^2 + \sigma^2}
= \frac{N\Theta_{ML}\sigma_0^2 + \sigma^2 \mu_0}{N\sigma_0^2 + \sigma^2} = \frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma_0^2} \Theta_{ML} +\frac{1/\sigma_0^2}{N/\sigma^2 + 1/\sigma_0^2} \mu_0$$

### 3. Bayes estimation

For $\Theta_{Bayes} = E(\Theta | \mathbf{X}) = \int \Theta p(\Theta | \mathbf{X}) d\Theta \\$, since $p(\Theta | \mathbf{X}) = \frac{p(\mathbf{X}| \Theta) p(\Theta)}{p(\mathbf{X})}$ and $p(\mathbf{X})$ is a constant for given $\mathbf{X}$, our interest is in $p(\mathbf{X}| \Theta) p(\Theta)$. We denote $\phi(x, \mu, \sigma^2)$ as the normal distribution with input $x$, mean $\mu$ and standard deviation $\sigma$. Then  $$p(\mathbf{X}| \Theta) p(\Theta) = \phi(\Theta, \mu_0, \sigma_0^2) \prod_{i=1}^N \phi(\Theta, \mathbf{x}^{(i)}, \sigma^2).$$

Notice that $$\phi(x, \mu_1, \sigma_1^2) \phi(x, \mu_2, \sigma_2^2) = \phi(\mu_1, \mu_2, \sigma_1^2 + \sigma_2^2) \phi(x, \mu_i, \sigma_i^2)$$ where $$\mu_i = \frac{1 / \sigma_1^2}{1/ \sigma_1^2 + 1/ \sigma_2^2}\mu_1 + \frac{1 / \sigma_2^2}{1/ \sigma_1^2 + 1/ \sigma_2^2}\mu_2 \text{ and } \sigma_i^2 = \frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2 + \sigma_2^2}.$$

We will prove the formula later on. Using this formuma, we have $$\phi(\Theta, \mathbf{x}^{(a)}, \sigma^2) \phi(\Theta, \mathbf{x}^{(b)}, \sigma^2) = \phi(\mathbf{x}^{(a)}, \mathbf{x}^{(b)}, 2\sigma^2) \phi(\Theta, \frac{\mathbf{x}^{(a)}+\mathbf{x}^{(b)}}{2}, \frac{\sigma^2}{2}) = C_0 \phi(\Theta, \frac{\mathbf{x}^{(a)}+\mathbf{x}^{(b)}}{2}, \frac{\sigma^2}{2}),$$ where $C_0$ is some constant since all variables of $\phi(\mathbf{x}^{(a)}, \mathbf{x}^{(b)}, 2\sigma^2)$ are set. Following similar steps we get

$$\prod_{i=1}^N \phi(\Theta, \mathbf{x}^{(i)}, \sigma^2) = C_1 \phi(\Theta,  \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}, \frac{\sigma^2}{N}),$$ where $C_1$ is some constant.

Hence, $$p(\mathbf{X}| \Theta) p(\Theta) = C_1 \phi(\Theta, \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}, \frac{\sigma^2}{N}) \phi(\Theta, \mu_0, \sigma_0^2) =  C_2 \phi(\Theta, \mu_\text{new}, \sigma_\text{new}^2),$$ where $C_2$ is some constant and by the formula, $\mu_\text{new} = \frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma_0^2} \Theta_{ML} +\frac{1/\sigma_0^2}{N/\sigma^2 + 1/\sigma_0^2} \mu_0 $, where $\Theta_{ML} = \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}.$

Notice that for a given normal distribution, multiplying the probability density function by a constant will not change its mean value. Therefore the expectation of $p(\Theta | \mathbf{X})$ is exactly the expectation of the non-constant normal distribution part. Hence,
$$
E(\Theta | \mathbf{X}) = \mu_\text{new} = \frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma_0^2} \Theta_{ML} +\frac{1/\sigma_0^2}{N/\sigma^2 + 1/\sigma_0^2} \mu_0 = \Theta_{MAP},
$$
where $\Theta_{ML} = \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}.$

Finally we want to prove the formula $\phi(x, \mu_1, \sigma_1^2) \phi(x, \mu_2, \sigma_2^2) = \phi(\mu_1, \mu_2, \sigma_1^2 + \sigma_2^2) \phi(x, \mu_i, \sigma_i^2)$:

$$
\begin{align}
\phi(x, \mu_1, \sigma_1^2) \phi(x, \mu_2, \sigma_2^2) & = \frac{1}{\sqrt{2\pi}\sigma_1} \exp\left( -\frac{(x - \mu_1)^2}{2\sigma_1^2} \right) \frac{1}{\sqrt{2\pi}\sigma_2} \exp\left( -\frac{(x - \mu_2)^2}{2\sigma_2^2} \right) \\
& =  \frac{1}{2\pi \sigma_1 \sigma_2} \exp\left( -\frac{(x - \mu_1)^2}{2\sigma_1^2} - \frac{(x - \mu_2)^2}{2\sigma_2^2}\right) \\
& = \frac{1}{2\pi \sigma_1 \sigma_2} \exp\left( -\frac{(\sigma_1^2+\sigma_2^2) x^2 -2(\mu_1 \sigma_2^2 +\mu_2 \sigma_1^2)x +(\mu_1^2 \sigma_2^2+\mu_2^2 \sigma_1^2)}{2\sigma_1^2 \sigma_2^2}    \right)\\
& = \frac{1}{2\pi \sigma_1 \sigma_2} \exp\left( -\frac{x^2 -2\frac{\mu_1 \sigma_2^2 +\mu_2 \sigma_1^2}{\sigma_1^2+\sigma_2^2}x + \frac{\mu_1^2 \sigma_2^2+\mu_2^2 \sigma_1^2}{\sigma_1^2+\sigma_2^2}}{2\frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2+\sigma_2^2}}    \right) \\
& = \frac{1}{2\pi \sigma_1 \sigma_2} \exp\left( -\frac{x^2 -2\frac{\mu_1 \sigma_2^2 +\mu_2 \sigma_1^2}{\sigma_1^2+\sigma_2^2}x + \frac{\mu_1^2 \sigma_2^4+\mu_2^2 \sigma_1^4 + 2\mu_1 \sigma_2^2 \mu_2 \sigma_1^2}{(\sigma_1^2+\sigma_2^2)^2}}{2\frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2+\sigma_2^2}} \right) \\
& \times \exp\left(\frac{ - (\sigma_1^2+\sigma_2^2)(\mu_1^2 \sigma_2^2+\mu_2^2 \sigma_1^2) +  (\mu_1^2 \sigma_2^4+\mu_2^2 \sigma_1^4 + 2\mu_1 \sigma_2^2 \mu_2 \sigma_1^2) }{2 \sigma_1^2 \sigma_2^2 (\sigma_1^2+\sigma_2^2)}\right) \\
& = \frac{1}{2\pi \sqrt{\frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2+\sigma_2^2}} \sqrt{\sigma_1^2+\sigma_2^2}} \exp\left( -\frac{(x -\frac{\mu_1 \sigma_2^2 +\mu_2 \sigma_1^2}{\sigma_1^2+\sigma_2^2})^2}{2\frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2+\sigma_2^2}} \right) \exp\left( -\frac{(\mu_1 - \mu_2)^2}{2 \sigma_1^2 + \sigma_2^2} \right)\\
& = \frac{1}{\sqrt{2\pi} \sqrt{\frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2+\sigma_2^2}}} \exp\left( -\frac{(x -\frac{\mu_1 \sigma_2^2 +\mu_2 \sigma_1^2}{\sigma_1^2+\sigma_2^2})^2}{2\frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2+\sigma_2^2}} \right) \frac{1}{\sqrt{2\pi} \sqrt{\sigma_1^2+\sigma_2^2}} \exp\left( -\frac{(\mu_1 - \mu_2)^2}{2 \sigma_1^2 + \sigma_2^2} \right)\\
& = \phi(x, \mu_i, \sigma_i^2) \phi(\mu_1, \mu_2, \sigma_1^2 + \sigma_2^2)\\
\end{align}
$$
where $$\mu_i = \frac{1 / \sigma_1^2}{1/ \sigma_1^2 + 1/ \sigma_2^2}\mu_1 + \frac{1 / \sigma_2^2}{1/ \sigma_1^2 + 1/ \sigma_2^2}\mu_2 \text{ and } \sigma_i^2 = \frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2 + \sigma_2^2}.$$

Hence we complete the proof and we validate that
$$\Theta_{Bayes} = \mu_\text{new} = \frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma_0^2} \Theta_{ML} +\frac{1/\sigma_0^2}{N/\sigma^2 + 1/\sigma_0^2} \mu_0.$$

# Hand-written digit classification (40 points)

In the textbook sample code we applied different scikit-learn classifers for the Iris data set.

In this exercise, we will apply the same set of classifiers over a different data set: hand-written digits.
Please write down the code for different classifiers, choose their hyper-parameters, and compare their performance via the accuracy score as in the Iris dataset.
Which classifier(s) perform(s) the best and worst, and why?

The classifiers include:
* perceptron
* logistic regression
* SVM
* decision tree
* random forest
* KNN
* naive Bayes

The dataset is available as part of scikit learn, as follows.

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,pandas,matplotlib,scipy,sklearn
%matplotlib inline
```

    last updated: 2016-10-17

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.1
    pandas 0.18.1
    matplotlib 1.5.1
    scipy 0.17.1
    sklearn 0.18

```python
# Added version check for recent scikit-learn 0.18 checks
from distutils.version import LooseVersion as Version
from sklearn import __version__ as sklearn_version
```

## Load data

```python
from sklearn.datasets import load_digits
digits = load_digits()

X = digits.data # training data
y = digits.target # training label

print(X.shape)
print(y.shape)
```

    (1797, 64)
    (1797,)

## Visualize data

```python
import matplotlib.pyplot as plt
import pylab as pl

num_rows = 4
num_cols = 5

fig, ax = plt.subplots(nrows=num_rows, ncols=num_cols, sharex=True, sharey=True)
ax = ax.flatten()
for index in range(num_rows*num_cols):
    img = digits.images[index]
    label = digits.target[index]
    ax[index].imshow(img, cmap='Greys', interpolation='nearest')
    ax[index].set_title('digit ' + str(label))

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
plt.show()
```

## Date Preprocessing
Hint: How you divide training and test data set? And apply other techinques we have learned if needed.
You could take a look at the Iris data set case in the textbook.

```python
from sklearn.preprocessing import StandardScaler
if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split
print ('scikit-learn version: ' + str(Version(sklearn_version)))

# 1. Standardize features by removing the mean and scaling to unit variance
X_std = StandardScaler().fit_transform(X) # fit_transform(X) will fit to data, then transform it.
print ('1. Complete removing the mean and scaling to unit variance.')

# 2. splitting data into 70% training and 30% test data:
split_ratio = 0.3
X_train, X_test, y_train, y_test = train_test_split(X_std, y, test_size=split_ratio, random_state=0)
print('2. Complete splitting with ' + str(y_train.shape[0]) + \
      '(' + str(int((1-split_ratio)*100)) +'%) training data and ' + \
      str(y_test.shape[0]) + '(' + str(int(split_ratio*100)) +'%) test data.')

```

    scikit-learn version: 0.18
    1. Complete removing the mean and scaling to unit variance.
    2. Complete splitting with 1257(70%) training data and 540(30%) test data.

## Classifier #1 Perceptron

```python
from sklearn.linear_model import Perceptron
from sklearn.metrics import accuracy_score

# Training
ppn = Perceptron(n_iter=800, eta0=0.1, random_state=0)
ppn.fit(X_train, y_train)
# Testing
y_pred = ppn.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))
```

    Misclassified samples: 39 out of 540
    Accuracy: 0.928

## Classifier #2 Logistic Regression

```python
from sklearn.linear_model import LogisticRegression

# Training
lr = LogisticRegression(C=1.0, random_state=0) # we observe that changing C from 0.0001 to 1000 has ignorable effect
lr.fit(X_train, y_train)
# Testing
y_pred = lr.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))
```

    Misclassified samples: 25 out of 540
    Accuracy: 0.954

## Classifier #3 SVM

```python
from sklearn.svm import SVC

# 1. Using linear kernel

# Training
svm = SVC(kernel='linear', C=1.0, random_state=0)
svm.fit(X_train, y_train)
# Testing
y_pred = svm.predict(X_test)
# Results
print('1. Using linear kernel:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))

# 2. Using rbf kernel

# Training
svm = SVC(kernel='rbf', C=1.0, random_state=0)
svm.fit(X_train, y_train)
# Testing
y_pred = svm.predict(X_test)
# Results
print('2. Using rbf kernel:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))
```

    1. Using linear kernel:
       Misclassified samples: 14 out of 540
       Accuracy: 0.974
    2. Using rbf kernel:
       Misclassified samples: 8 out of 540
       Accuracy: 0.985

## Classifier #4 Decision Tree

```python
from sklearn.tree import DecisionTreeClassifier

# 1. Using entropy criterion

# Training
tree = DecisionTreeClassifier(criterion='entropy', random_state=0)
tree.fit(X_train, y_train)
# Testing
y_pred = tree.predict(X_test)
# Results
print('1. Using entropy criterion:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))

# 2. Using Gini criterion

# Training
tree = DecisionTreeClassifier(criterion='gini', random_state=0)
tree.fit(X_train, y_train)
# Testing
y_pred = tree.predict(X_test)
# Results
print('2. Using Gini criterion:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))
```

    1. Using entropy criterion:
       Misclassified samples: 71 out of 540
       Accuracy: 0.869
    2. Using Gini criterion:
       Misclassified samples: 77 out of 540
       Accuracy: 0.857

## Classifer #5 Random Forest

```python
from sklearn.ensemble import RandomForestClassifier

# 1. Using entropy criterion

# Training
forest = RandomForestClassifier(criterion='entropy', n_estimators=10, random_state=1, n_jobs=2)
forest.fit(X_train, y_train)
# Testing
y_pred = forest.predict(X_test)
# Results
print('1. Using entropy criterion:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))

# 2. Using Gini criterion

# Training
forest = RandomForestClassifier(criterion='gini', n_estimators=10, random_state=1, n_jobs=2)
forest.fit(X_train, y_train)
# Testing
y_pred = forest.predict(X_test)
# Results
print('2. Using Gini criterion:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))
```

    1. Using entropy criterion:
       Misclassified samples: 32 out of 540
       Accuracy: 0.941
    2. Using Gini criterion:
       Misclassified samples: 27 out of 540
       Accuracy: 0.950

## Classifier #6 KNN

```python
from sklearn.neighbors import KNeighborsClassifier

# Training
knn = KNeighborsClassifier(n_neighbors=5, p=2, metric='minkowski')
knn.fit(X_train, y_train)
# Testing
y_pred = knn.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))
```

    Misclassified samples: 15 out of 540
    Accuracy: 0.972

## Classifier #7 Naive Bayes

```python
from sklearn.naive_bayes import GaussianNB

# Training
gnb = GaussianNB()
gnb.fit(X_train, y_train)
# Testing
y_pred = gnb.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))
```

    Misclassified samples: 123 out of 540
    Accuracy: 0.772

Hence in this example, the accuracy of the predictions is ranked as (in descending order):
* SVM (rbf kernel) - 0.985

* SVM (linear kernel) - 0.974

* KNN - 0.972

* Logistic Regression - 0.954

* Random forest (Gini criterion) - 0.950

* Random forest (entropy criterion) - 0.941

* Perceptron - 0.928

* Decision Tree (entropy criterion) - 0.869

* Decision Tree (Gini criterion) - 0.857

* Naive Bayes - 0.772

The best is SVM (using rbf kernel). Because SVM maximize margins to nearest samples (called support vectors), which is considered as an effective way of classifying spacially separated samples. Also, SVM is more robust against outliers and offers slack variables as a solution to not linearly-separable samples. Moreover, using rbf kernel, the dimension is mapped to infinity and therefore samples are highly likely to be separated by some hyperplanes.

The worst is Naive Bayes. Because it assumes that all data are independent, which leads to high bias and low variance. When the input samples are not generally independent, the assumption fails and therefore the accuracy is low. Also, Naive Bayes cannot deal with outliers or noise, therefore unideal samples may not be correctly classified.


```python
import numpy
from matplotlib import pyplot, cm
from mpl_toolkits.mplot3d import Axes3D ##library for 3d projection plots
%matplotlib inline
```

# High-Level Waste : Technology

Having covered the policy related to high level waste, we will now discuss the technology.

## Learning Objectives

At the end of this lesson, you will be equipped to:

- Recognize the relationship between decay and repository heat burden.
- Calculate the decay heat of UNF as a function of reactor power and time since discharge.
- Calculate radiotoxicity of a mixture of isotopes over time.
- Explain the characteristics and purpose of _the reference man_.
- Recognize the barriers between a disposed waste form and the reference man.
- List mechanisms involved in radionuclide contaminant transport.
- Estimate the radionuclide transport impacts of solubility, sorption, retardation, diffusion, and advection.
- Differentiate between reducing and oxidizing geologic host media.

## Radioactivity

Recall: The SI unit of activity is the becquerel (Bq), equal to one reciprocal second.

\begin{align}
A(i) &= -\frac{dN_i}{dt}\\
&= \lambda_iN_i
\end{align}

And, given these decays, we also know that

\begin{align}
N(t) = N_0e^{t/\tau}
\end{align}

## Decay Heat

How do we get from radiation to heat?

\begin{align}
f(t) &= \sum_i\left( \bar{E}_{\beta,i} + \bar{E}_{\gamma, i} + \bar{E}_{\alpha, i}\right)A(i)\\
     &= \sum_i\left( \bar{E}_{\beta,i} + \bar{E}_{\gamma, i} + \bar{E}_{\alpha, i}\right)\lambda_iN_i(t)\\
\end{align}

Recall that we know how to get the populations $N_i$ of all isotopes built up in the reactor.

```python
from pyne import data

data.decay_const('u238')

from pyne.material import Material
lwr_snf_ma = Material({'U234': 0.12,
                    'U236': 4.18,
                    'Np237': 0.75,
                    '236Pu': 9.2E-6,
                    '238Pu': 0.22,
                    '239Pu': 5.28,
                    '240Pu': 2.17,
                    '241Pu': 1.02,
                    '242Pu': 0.35,
                    '241Am': 0.05,
                    '243Am': 0.09,
                    '242Cm': 4.9E-3,
                    '244Cm': 3.3E-2},
                   1000)

#print(lwr_snf_ma.comp)
#print(lwr_snf_ma.activity())
#print(lwr_snf_ma.decay_heat())
D=lwr_snf_ma.decay_heat()

import matplotlib.pylab as plt
plt.bar(range(len(D)), D.values(), align='center')
plt.xticks(range(len(D)), D.keys(), rotation='vertical')

plt.show()
```

## Decay Heat and Geologic Heat Capacity

## Toxicity

One possibility is to use toxicity as a classifying metric. It captures the

\begin{align}
\mbox{Toxicity}(i,k) &= \frac{A(i)}{DAC(i,k)}\\
i &= \mbox{isotope}\\
k &= \mbox{medium index, air or water}\\
DAC &= \mbox{derived air concentration of isotope i in air or water.}
\end{align}

### Example: Toxicity

Which is more dangerous, 7.2mg of $^{90}Sr$ or 0.8g of $^{239}Pu$?

```python

def tox(activity, dac):
    """returns the radiotoxicity of a sample
    :param activity: activity of the sample in Bq
    :param dac: derived air concentration for the active isotope in the sample
    """
    return activity/dac
```

To solve this, we'll need to go look up the DAC for each isotope.

\begin{align}
DAC_{^{90}Sr} &= 8\times 10^{-9}\left[\frac{\mu Ci}{cm^3}\right]\\
DAC_{^{239}Pu} &= 3\times 10^{-12}\left[\frac{\mu Ci}{cm^3}\right]\\
\end{align}

```python
dac_90_sr = 8E-9
dac_239_pu = 3E-12
```

And, we'll need to calculate the activity of the sample. First, we'll need the number of atoms as a function of the mass of each isotope.

\begin{align}
N_i &= N_A\frac{m_i}{A}\\
N_i &= \mbox{number of atoms in the sample}\\
N_A &= \mbox{Avogadro's number, atoms per mol}\\
m_i &= \mbox{mass of isotope i in the sample}\\
A &= \mbox{Atomic mass}
\end{align}

```python
def n_from_mass(iso, g):
    from pyne import data
    from scipy import constants as sc
    a = data.atomic_mass(iso)
    avo = sc.Avogadro
    return (g/a)*avo
```

```python
## Test it
n_from_mass('1H', 1)
```

    5.9753832901678335e+23

```python
def activity(iso, n):
    """returns the activity of some N atoms of an isotope
    :param iso: the isotope
    :param n: number of atoms in the sample
    """
    from pyne import data
    lam = data.decay_const(iso)
    return n*lam
```

**BUT: All of this has been done in Bq. Recall that our toxicity calculation needs Ci. Let's report our activities in Ci.'**

```python
def bq_to_ci(bq):
    return bq/3.7E10

sr_activity =  bq_to_ci(activity('Sr90',
                                 (n_from_mass('Sr90',
                                              0.8))))
pu_activity = bq_to_ci(activity('Pu239',
                                (n_from_mass('Pu239',
                                             0.8))))

print("90Sr activity [Ci] = ", sr_activity)
print("239Pu activity [Ci] = ", pu_activity)
```

    90Sr activity [Ci] =  110.48974430129589
    239Pu activity [Ci] =  0.04962160155426739

```python
print("90Sr toxicity = ", '%e' % tox(sr_activity, dac_90_sr), "\n"
"239Pu toxicity = ", '%e' % tox(pu_activity, dac_239_pu))
```

    90Sr toxicity =  1.381122e+10
    239Pu toxicity =  1.654053e+10

```python
dac_99_tc = 3E-7
dac_137_cs = 6E-8

tc_activity =  bq_to_ci(activity('Tc99',
                                 (n_from_mass('Tc99',
                                              1))))
cs_activity = bq_to_ci(activity('Cs137',
                                (n_from_mass('Cs137',
                                             0.001))))

print("99Tc activity [Ci] = ", tc_activity)
print("137Cs activity [Ci] = ", cs_activity)
print("99Tc toxicity = ", '%e' % tox(tc_activity, dac_99_tc), "\n"
"137Cs toxicity = ", '%e' % tox(cs_activity, dac_137_cs))
```

    99Tc activity [Ci] =  0.017122166117922367
    137Cs activity [Ci] =  0.08680943113648998
    99Tc toxicity =  5.707389e+04
    137Cs toxicity =  1.446824e+06

### Question: Units?

What were the units of toxicity, and how should we interperet those units?

## Empirical Activity Fits

An aside... note that your book gives the following empirical equation for post-reactor burnup activity.

\begin{align}
R(t) &=
\begin{cases}
 A_1e^{\left(\frac{1}{A_2 + A_3t}\right)} & t<30y\\
 B_1t^{-a}\left[1+ \frac{B_2}{1+\left(\frac{t}{B_3}\right)^4}\right] & 30<t<1\times10^5y\\
 \end{cases}\\
\end{align}
For a 1250 Mwe PWR with $\eta=0.33$, average specific power of 37.5 MWth/tHM, and 33,000 MWd/tHM burnup, the coefficients are:

\begin{align}
A_1 &= 1.42\times10^5\\
 A_2 &= 0.296\\
 A_3 &= 7.22\times10^{-2}\\
 a &= 0.2680\\
 B_1 &= 1.98\times10^5\\
 B_2 &= 10\\
 B_3 &= 88\\
\end{align}

## Release

**Source Term** Radionuclide mass flux to the environment
- Safety and Risk Metric
- EPA Regulation

**Radionuclide transport** is a function of
- Geochemistry
  - chemically induced material degradation
  - radionuclide solubility limits
  - sorption
  - colloid mobility, etc.
- Hydrology
  - water induced material degradation,
  - water movement (diffusion and advection) through pores and fractures,
  - dissolved contaminant dispersion.
- Thermal Effects
  - thermally induced material degradation,
  - thermal hydrological effects.

## Diffusion
[Diffusion code below from 12 steps to Navier Stokes, Lorena Barba.](http://nbviewer.jupyter.org/github/barbagroup/CFDPython/blob/master/lessons/09_Step_7.ipynb)

We begin with the 2D-diffusion equation:

$$\frac{\partial u}{\partial t} = \nu \frac{\partial ^2 u}{\partial x^2} + \nu \frac{\partial ^2 u}{\partial y^2}$$

Without going into details, just note that it is possible to discretize second order derivatives. The following scheme uses a _forward difference_ in time and two second-order derivatives.

$$\frac{u_{i,j}^{n+1} - u_{i,j}^n}{\Delta t} = \nu \frac{u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n}{\Delta x^2} + \nu \frac{u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n}{\Delta y^2}$$

This method requires that you reorganize the discretized equation and solve for $u_{i,j}^{n+1}$
\begin{align}
u_{i,j}^{n+1} = u_{i,j}^n &+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n) \\
&+ \frac{\nu \Delta t}{\Delta y^2}(u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n)
\end{align}

```python
### variable declarations
nx = 31
ny = 31
nt = 17
nu = 0.05
dx = 2 / (nx - 1)
dy = 2 / (ny - 1)
sigma = .25
dt = sigma * dx * dy / nu

x = numpy.linspace(0, 2, nx)
y = numpy.linspace(0, 2, ny)

u = numpy.ones((ny, nx))  # create a 1xn vector of 1's
un = numpy.ones((ny, nx))

### Assign initial conditions
# set hat function I.C. : u(.5<=x<=1 && .5<=y<=1 ) is 2
u[int(.5 / dy):int(1 / dy + 1),int(.5 / dx):int(1 / dx + 1)] = 2

fig = pyplot.figure()
ax = fig.gca(projection='3d')
X, Y = numpy.meshgrid(x, y)
surf = ax.plot_surface(X, Y, u, rstride=1, cstride=1, cmap=cm.viridis,
        linewidth=0, antialiased=False)

ax.set_xlim(0, 2)
ax.set_ylim(0, 2)
ax.set_zlim(1, 2.5)

ax.set_xlabel('$x$')
ax.set_ylabel('$y$');
```

\begin{align}
u_{i,j}^{n+1} = u_{i,j}^n &+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n) \\
&+ \frac{\nu \Delta t}{\Delta y^2}(u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n)
\end{align}

```python
###Run through nt timesteps
def diffuse(nt):
    u[int(.5 / dy):int(1 / dy + 1),int(.5 / dx):int(1 / dx + 1)] = 2

    for n in range(nt + 1):
        un = u.copy()
        u[1:-1, 1:-1] = (un[1:-1,1:-1] +
                        nu * dt / dx**2 *
                        (un[1:-1, 2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                        nu * dt / dy**2 *
                        (un[2:,1: -1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))
        u[0, :] = 1
        u[-1, :] = 1
        u[:, 0] = 1
        u[:, -1] = 1

    fig = pyplot.figure()
    ax = fig.gca(projection='3d')
    surf = ax.plot_surface(X, Y, u[:], rstride=1, cstride=1, cmap=cm.viridis,
        linewidth=0, antialiased=True)
    ax.set_zlim(1, 2.5)
    ax.set_xlabel('$x$')
    ax.set_ylabel('$y$');

```

```python
diffuse(0)
```

```python
diffuse(50)
```

[6] J. A. Blink, T. A. Buscheck, W. G. Halsey, and T. Wolery.
Disposal systems evaluations and tool Development-Engineered barrier system evaluation
work package LL1015080425.
Technical report, Lawrence Livermore National Laboratory (LLNL), Livermore, CA, 2010.

## Hydrology 101: Adevection Dispersion Equation

 In a saturated, reducing environment, contaminants are transported by
**diffusion**, **dispersion,** and **advection**.

\begin{align}
      J &= J_{dis} + J_{adv}\nonumber\\
      &= -\theta(D_{mdis} + \tau D_m)\nabla C + \theta vC\nonumber\\
      &= -\theta D\nabla C + \theta vC \nonumber\\
      J_{dis} &= \mbox{ Total Dispersive Mass Flux }[kg/m^2/s]\nonumber\\
      J_{adv} &= \mbox{ Advective Mass Flux }[kg/m^2/s]\nonumber\\
      \tau &= \mbox{ Toruosity }[-] \nonumber\\
      \theta &= \mbox{ Porosity }[-] \nonumber\\
      D_m &= \mbox{ Molecular diffusion coefficient }[m^2/s]\nonumber\\
      D_{mdis} &= \mbox{ Coefficient of mechanical dispersivity}[m^2/s]\nonumber\\
      D &= \mbox{ Effective Dispersion Coefficient }[m^2/s]\nonumber\\
      C &= \mbox{ Concentration }[kg/m^3]\nonumber\\
      v &= \mbox{ Fluid Velocity in the medium }[m/s].\nonumber
\end{align}

(From F. W. Schwartz and H. Zhang. Fundamentals of ground water. Environmental Geology, 45:10371038, 2004.)

One can express this slightly more simply as:

\begin{align}
  \frac{\partial n C}{\partial t} & = - \nabla \cdot  (F_c + F_{dc} + F_d) + m
\end{align}

\begin{align}
      n &= \mbox{solute accessible porosity } [\%]\nonumber\\
      C &= \mbox{ concentration } [kg \cdot m^{-3}]\nonumber\\
      t &= \mbox{ time } [s]\nonumber\\
      F_c &= \mbox{ advective transport} [kg \cdot m^{-2}\cdot s^{-1}]\nonumber\\
      &= nvC \nonumber \\
      F_{dc} &= \mbox{ dispersive transport} [kg \cdot m^{-2}\cdot s^{-1}]\nonumber\\
      &= \alpha nv \nabla C  \nonumber\\
      F_d &= \mbox{ diffusive transport} [kg \cdot m^{-2}\cdot s^{-1}]\nonumber\\
      &= D_e \nabla C\nonumber
    \end{align}
    \begin{align}
      m &= \mbox{ solute source } [kg \cdot m^{-3}\cdot s^{-1}].\nonumber\\
      v &= \mbox{ pore velocity } [m\cdot s^{-1}] \nonumber\\
      \alpha &= \mbox{ dispersivity } [m]\nonumber\\
      D_e &= \mbox{ effective diffusion coefficient } [m^2\cdot s^{-1}]\nonumber\\
      n\cdot v &= \mbox{ Darcy velocity } [m\cdot s^{-1}].
    \end{align}

## Dispersion

Dispersion captures the impact of the tortuous path fluids must take through porous media:

It is customary to define the combination of molecular diffusion, $D_e$ and mechanical dispersion, $\alpha v$, as $D$
\begin{align}
  D = \alpha v + D_e
\end{align}
such that the mass conservation equation becomes:

\begin{align}
  D = \alpha v + D_e
\end{align}
such that the mass conservation equation becomes:
\begin{align}
  \frac{\partial(nC)}{\partial t} &= \nabla \left( nD\nabla C \right) - \nabla \left( nvC \right)
  \end{align}

Adding sorption, by accounting for a change in mass storage,

\begin{align}
  \frac{\partial(nC)}{\partial t}  + \frac{\partial(s\rho_b)}{\partial t} &=
  \nabla \left( nD\nabla C \right) - \nabla \left( nvC \right) \\
  s &= \mbox{sorption coefficient}\nonumber\\
  \rho_b &= \mbox{ bulk (dry) density }[kg/m^3].\nonumber
\end{align}

## Sorption

If it is assumed that sorption can be approximated as a linear equilibrium,
reversible reaction,
\begin{align}
  \frac{\partial(s\rho_b)}{\partial t} &= \left( R_f - 1
  \right)\frac{\partial(nC)}{\partial t}\\
  R_f\frac{\partial(nC)}{\partial t} &=
  \nabla \left( nD\nabla C \right) - \nabla \left( nv \right) \\
  R_f &= \mbox{retardation factor}\nonumber\\
  &= 1+\frac{\rho_bK_d}{n}\\
  \rho_b &=\mbox{bulk density of the rock matrix}\nonumber\\
  K_d &= \mbox{species distribution coefficient.}\nonumber
\end{align}

### Question: Should we be more concerned about high or low sorption elements?

## Solubility

Elemental solubility limits are based on the maximum concentration of an  element which can exist in solution. A limit on concentration in the void  volume may be expressed as

\begin{align}
      m_{1i}(t)&\le v_{1i}(t)C_{sol}\\
      m_{li} &= \mbox{dissolved kg of radionuclide i}\nonumber\\
      v_{li} &= \mbox{void volume}\nonumber\\
      C_{sol} &= \mbox{solubility limit}\nonumber
\end{align}

### Question: Should we be more concerned about high or low solubility elements?

## Wrap up

At the end of this lesson, you should be able to:

- Recognize the relationship between decay and repository heat burden.
- Calculate the decay heat of UNF as a function of reactor power and time since discharge.
- Calculate radiotoxicity of a mixture of isotopes over time.
- Explain the characteristics and purpose of _the reference man_.
- Recognize the barriers between a disposed waste form and the reference man.
- List mechanisms involved in radionuclide contaminant transport.
- Estimate the radionuclide transport impacts of solubility, sorption, retardation, diffusion, and advection.
- Differentiate between reducing and oxidizing geologic host media.

## A note on the future

- EPA: Transition team being led by Myron Ebell.
- DOE: Considering fracking mogul Harold Hamm as Secretary of Energy.
- Clean Power Plan: Can immediately stop enforcing.
- International Climate Agreements: Takes 4 years to back out.
- Coal: possible "clean coal" subsidies will compete with nuclear baseload.

A more optimistic view b

From ANS Focus on Communications Panel, November 9, 2016.

```python

```


```python
from Arbie.Variables.pool import Pool
from Arbie import Token
```

```python
size = 100

# Setup tokens
dai = Token('dai')
eth = Token('eth')

pool1 = Pool([dai, eth],[400*size, 1*size],[0.51, 0.49], fee=0.003)
print(pool1.spot_price(dai, eth))
pool1
```

    384.3137254901961

    Amm(
      Tokens: [Token(Name: dai, Address: 0x6850A5d7c21A8449886130A946d14d3f37Cc18AE), Token(Name: eth, Address: 0x704cE91D485E70d9a886e0Ebd717F659215a2436)],
      Balances: [Balance(Token: Token(Name: dai, Address: 0x6850A5d7c21A8449886130A946d14d3f37Cc18AE), Value: 40000), Balance(Token: Token(Name: eth, Address: 0x704cE91D485E70d9a886e0Ebd717F659215a2436), Value: 100)],
      Weights: [Balance(Token: Token(Name: dai, Address: 0x6850A5d7c21A8449886130A946d14d3f37Cc18AE), Value: 0.51), Balance(Token: Token(Name: eth, Address: 0x704cE91D485E70d9a886e0Ebd717F659215a2436), Value: 0.49)],
      Fee: 0.003
      Address: 0xf530E737266e350882D25DCf228e1e075b8F60d5)

```python
pool2 = Pool([dai, eth],[410*size, 1*size],[0.51, 0.49], fee=0.005)
print(pool2.spot_price(dai, eth))
pool2
```

    393.92156862745094

    Amm(
      Tokens: [Token(Name: dai, Address: 0x6850A5d7c21A8449886130A946d14d3f37Cc18AE), Token(Name: eth, Address: 0x704cE91D485E70d9a886e0Ebd717F659215a2436)],
      Balances: [Balance(Token: Token(Name: dai, Address: 0x6850A5d7c21A8449886130A946d14d3f37Cc18AE), Value: 41000), Balance(Token: Token(Name: eth, Address: 0x704cE91D485E70d9a886e0Ebd717F659215a2436), Value: 100)],
      Weights: [Balance(Token: Token(Name: dai, Address: 0x6850A5d7c21A8449886130A946d14d3f37Cc18AE), Value: 0.51), Balance(Token: Token(Name: eth, Address: 0x704cE91D485E70d9a886e0Ebd717F659215a2436), Value: 0.49)],
      Fee: 0.005
      Address: 0x0013B7cc84C81E14821c5f62a1801549B714A1C2)

```python
# We can see that pool2 has a higher price for eth then pool1. If we have dai we can buy eth from pool1 and sell to pool2
# The only question is how much should we buy and sell?
```

```python
from sympy import *
from sympy.plotting import plot
init_printing()
```

```python
# We can plot our return function
x = symbols('x')
expr = pool1.out_given_in_expr(dai, eth)
plot(expr, (x, 0, 50000))
expr = pool1.out_given_in_expr(eth, dai)
plot(expr, (x, 0, 200))
```

```python
# How can we found if there is an arbitrage opertunity between pool1 and pool2?
from Arbie.Actions.arbitrage import arbitrage_expr, arbitrage_diff_expr, TradeOpertunity

trade = TradeOpertunity([pool1, pool2], dai, eth)
arb_expr = arbitrage_expr(trade)
arb_expr
```

```python
# If we plot we can clearly see that there seams to be some profit to be made!
plot(arb_expr, (x, 0, 400))
```

```python
darb_expr = arbitrage_diff_expr(trade)
darb_expr
```

```python
from Arbie.Actions.arbitrage import find_arbitrage
find_arbitrage(trade)
```

    Balance(Token: Token(Name: dai, Address: 0x6850A5d7c21A8449886130A946d14d3f37Cc18AE), Value: 165.009948076539)

```python
# If we have two pools that dosn't have an arbitrage opertunity what happens then?
bad_trade = TradeOpertunity([pool2, pool1], dai, eth)
bad_expr = arbitrage_expr(bad_trade)
plot(bad_expr, (x, 0, 400))
```

```python

```


Author: [Greg Nordin](http://ece.byu.edu/faculty/nordin), [github](https://github.com/gregnordin)<br>
Nov. 12, 2016

# Spectrometer Measurement

## Purpose

Measure absorption spectrum for 3D printer resins.

## Experimental Setup

We use an Ocean Optics QE65-ABS scientific-grade spectrometer ([download pdf datasheet](http://oceanoptics.com/wp-content/uploads/OEM-Data-Sheet-QE65Pro.pdf)).

Need figure with:

- Spectrometer
- Light source
- Sample holder
- Set up with sample and no-resin
- Set up with sample and resin

## Measurement Method

1. Determine a spectrometer integration time, $\tau_{int}$, that uses most of the instrument's dynamic range for a given resin or set of resins. Use this integration time for measurements.
1. Measure dark spectrum $N$ times: $d^i$ with $i \in [1,N]$. This results in $N$ data files.
1. Measure transmission spectrum of sample with no resin $N$ times: $u^i$ with $i \in [1,N]$. This results in $N$ data files.
1. Measure transmission spectrum of sample with resin $N$ times: $m^i$ with $i \in [1,N]$. This results in $N$ data files.

Note that $N$ does not have to be the same for the dark, no resin, and resin measurements. However, for convenience we typically choose the same $N$ for each case.

## Transmittance and Absorbance

The transmittance of a sample, $T$, is the fraction of incident light that is transmitted through a sample, which is a function of wavelength,

$$ T(\lambda) = I_t(\lambda)/I_i(\lambda), $$

where $I_i$ is the incident irradiance (power per unit area, W/cm$^2$) and $I_t$ is the transmitted irradiance. The transmittance can be written in terms of the absorbance, $A$, as

$$ T(\lambda) = 10^{-A(\lambda)}. $$

We can solve for the absorbance as

\begin{align}
A(\lambda) &= -log_{10}(T(\lambda) \\
&= -log_{10} \left( \frac{I_t(\lambda)}{I_i(\lambda)} \right)  \\
\end{align}

## Absorbance Calculation

The absorbance $A_j$ at a given wavelength, $\lambda_j$, is calculated according to

\begin{align}
b_j &= \frac{1}{N}\sum_{i=1}^N{m^i_{j}} - d_j \\
T_j &= \frac{b_j}{u_j} \\
A_j &= -\log_{10}(T_j)
\end{align}

where

\begin{align}
b_j &= \text{dark-corrected average spectrum measurement at } j^{th} \text{wavelength} \\
T_j &= \text{Transmittance at } j^{th} \text{wavelength} \\
A_j &= \text{Absorbance at } j^{th} \text{wavelength} \\
m^i_j &= \text{measurement at } j^{th} \text{wavelength in } i^{th} \text{file} \\
d_{j} &= \frac{1}{N}\sum_{i=1}^{N}{d^i_{j}} = \text{average dark measurement at } j^{th} \text{wavelength} \\
u_{j} &= \frac{1}{N}\sum_{i=1}^{N}{u^i_{j}} - d_j = \text{average dark-corrected measurement at } j^{th} \text{wavelength prior to filling sample with resin} \\
\end{align}

Note that it is important to subtract the measured dark signal at each wavelength from the corresponding resin and no-resin measurements. The dark signal originates from an accumulation of electrons in each spectrometer detector pixel during the integration time even when no light is incident on the pixel. As such it represents a spurious signal in the spectrum measurement (in the form of a constant offset for a given integration time), which is easily corrected by simply subtracting the average dark signal from each measurement.

# Implementation

## Directory Structure

Here is an example directory structure:

    ├── 2016-06_resin_paper
    │   ├── 161109_process_data.ipynb
    │   ├── absorbance.py
    │   ├── processed_data
    │   │   ├── dark_2015-06-27.csv
    │   │   ├── fslclear_2015-07-30.csv
    │   │   ├── irgacure819_2015-06-30.csv
    │   │   ├── martius_05_2015-06-30.csv
    │   │   ├── plasclear_2015-06-27.csv
    │   │   ├── pr48_2015-07-27.csv
    │   │   ├── pr48noA_2015-07-30.csv
    │   │   ├── pr48noPA_2015-07-30.csv
    │   │   ├── sudanI_015_2015-06-27.csv
    │   │   ├── sudanI_01_2015-06-27.csv
    │   │   ├── sudanI_02_2015-06-27.csv
    │   │   └── sudanI_04_2015-06-27.csv
    │   └── raw_data
    │       ├── dark_2015-06-27
    │       ├── fslclear_2015-07-30
    │       ├── irgacure819_2015-06-30
    │       ├── martius_05_2015-06-30
    │       ├── plasclear_2015-06-27
    │       ├── pr48_2015-07-27
    │       ├── pr48noA_2015-07-30
    │       ├── pr48noPA_2015-07-30
    │       ├── sudanI_015_2015-06-27
    │       ├── sudanI_01_2015-06-27
    │       ├── sudanI_02_2015-06-27
    │       └── sudanI_04_2015-06-27
    ├── Absorbance_information_and_methods.ipynb
    └── source_spectra
        ├── 161112_compare_asiga_visitech.ipynb
        ├── asiga_pico_plus_27
        │   ├── 161112_asiga_spectrum.ipynb
        │   ├── asiga_spectrum.png
        │   ├── ave_raw_data
        │   └── processed_data
        ├── compare_visitech_asiga_spectra.png
        ├── spectrum_helper_calcs.py
        └── visitech_luxbeam_LRS_WQ
            ├── 161112_visitech_source_spectrum.ipynb
            ├── Collect_Spectra.ipynb
            ├── processed_data
            ├── raw_data
            └── visitech_spectrum.png

The main directory contains a directory, `source_spectra`, dedicated to emission spectrum measurements for various 3D printers. It also contains a directory for various resin spectrum data sets (`2016-06_resin_paper`, which corresponds to our paper ["Optical Approach to Resin Formulation for 3D Printed Microfluidics"](https://www.ncbi.nlm.nih.gov/pubmed/26744624)). New resin spectrum data sets are anticipated to be included in the future.

Each data set is organized by date and purpose in its own directory named according to YYYY-MM_NAME. Inside the directory are folders for raw data and processed data calculated from the raw data. Any code and/or Jupyter notebooks used in processing the data are included inside the data set directory. The intention is that processed data can always be recreated from the raw data without worrying about conflicting source code versions (for example, `absorbance.py` in `2016-06_resin_paper`).

## Raw Data Files

In the `raw_data` directory of `2016-06_resin_paper` spectrometer measurements for individual resins are organized as follows:

    └── sudanI_04_2015-06-27
        ├── meas_04_1.txt
        ├── meas_04_10.txt
        ├── meas_04_100.txt
        ├── meas_04_11.txt
        ├── meas_04_12.txt
        ├── <etc.>
        ├── std_04_1.txt
        ├── std_04_10.txt
        ├── std_04_100.txt
        ├── std_04_11.txt
        ├── std_04_12.txt
        ├── <etc.>

Data file directory names in `raw_data` have the form, `resin_YYYY-MM-DD`, where `YYYY-MM-DD` is the date the data was taken. In each resin directory, data files end in `.txt`. Files that start with `meas_` are spectrum measurements of a sample with resin while files that start with a different name are spectrum measurements of the sample with no resin.

There are generally `N` files that start with `meas_` for a given material, and the same number that do not start with `meas_` for samples prior to filling with resin.

Each data file is generated by a single spectrometer measurement. Files are in the following space-separated-value (SSV) format:

    wavelength1 wavelength2 wavelength3 wavelength4 ...
    measuremen1 measuremen2 measuremen3 measuremen4 ...

Here is an example:

    1.907908162221780231e+02 1.915984207987377772e+02 1.924059541587151330e+02 ...
    2.400000000000000000e+03 2.399000000000000000e+03 2.395000000000000000e+03 ...

Wavelength is in nanometers, and measurements are counts from the spectrometer's 16-bit analog-to-digital converter (range of values: [0, 65535]).

## Create CSV files with processed absorbance spectra

### Integration Time

The integration time for the spectrometer measurements in `2016-06_resin_paper` is 10,000 $\mu$s.

### Dark-Correct Raw Data

Each average dark-corrected value, $b_j$, for a material depends on using the correct dark value, $d_j$. All of the material measurements shown in the `2016-06_resin_paper/raw_data` directory above use the same dark data, which is in the processed data file `processed_data/dark_2015-06-27.csv`.

### File format

For each material, we want to create a new CSV data file that has the following format:

    # {date_measured : 2015-06-01, thickness_um : XX, description : "", concentration_percent : YY, ... }
    wavelength1,value1,stdev1
    wavelength2,value2,stdev2
    wavelength3,value3,stdev3

where the first line is a python dict containing relevant information about the measurement, and each following line contains comma-separated wavelength and absorbance (dark- and blank-corrected) values, and possibly the corresponding standard deviation.

### Wavelength range

In each spectrum measurement file in `2016-06_resin_paper/raw_data` there are 1044 measurements, each corresponding to a different wavelength in the range [190.7908, 993.5721] nm. The measurements near the shortest and longest wavelengths tend to be the most noisy, so we will throw them away and restrict the range to [231.0837 960.6043] nm, which corresponds to wavelengths[50:998] in the original data files, i.e., 948 data points.

### Processing steps

1. Create average dark data, $d_j$
1. Get names of data sets (i.e., names of folders in `raw_data`)
2. Choose data sets for which to create processed absorbance files
3. Loop over these data sets. For each:
    1. Open & read resin and no resin files
    1. Calculate absorbance using the method above
    1. Save data as csv file (with sample resin thickness, integration time, and dark data file name)

# Acknowledgments

Spectrum measurements and resin & sample preparation by Hua Gong, Steven Perry, and Mike Beauchamp.

```python

```